Re-evaluating GPT-4’s bar exam performance
Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of find...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Science and Business Media LLC
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/1721.1/153986 |
| _version_ | 1824457955138863104 |
|---|---|
| author | Martínez, Eric |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Martínez, Eric |
| author_sort | Martínez, Eric |
| collection | MIT |
| description | Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that indicate that OpenAI’s estimates of GPT-4’s UBE percentile are overinflated. First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was below the 69th percentile, and
$$\sim$$
∼
48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against first-time test takers is estimated to be
$$\sim$$
∼
62nd percentile, including
$$\sim$$
∼
42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to
$$\sim$$
∼
48th percentile overall, and
$$\sim$$
∼
15th percentile on essays. In addition to investigating the validity of the percentile claim, the paper also investigates the validity of GPT-4’s reported scaled UBE score of 298. The paper successfully replicates the MBE score, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the reported essay score. Finally, the paper investigates the effect of different hyperparameter combinations on GPT-4’s MBE performance, finding no significant effect of adjusting temperature settings, and a significant effect of few-shot chain-of-thought prompting over basic zero-shot prompting. Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI. |
| first_indexed | 2024-09-23T09:33:50Z |
| format | Article |
| id | mit-1721.1/153986 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:18:13Z |
| publishDate | 2024 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1539862024-12-23T05:37:38Z Re-evaluating GPT-4’s bar exam performance Martínez, Eric Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Law Artificial Intelligence Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that indicate that OpenAI’s estimates of GPT-4’s UBE percentile are overinflated. First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was below the 69th percentile, and $$\sim$$ ∼ 48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against first-time test takers is estimated to be $$\sim$$ ∼ 62nd percentile, including $$\sim$$ ∼ 42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to $$\sim$$ ∼ 48th percentile overall, and $$\sim$$ ∼ 15th percentile on essays. In addition to investigating the validity of the percentile claim, the paper also investigates the validity of GPT-4’s reported scaled UBE score of 298. The paper successfully replicates the MBE score, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the reported essay score. Finally, the paper investigates the effect of different hyperparameter combinations on GPT-4’s MBE performance, finding no significant effect of adjusting temperature settings, and a significant effect of few-shot chain-of-thought prompting over basic zero-shot prompting. Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI. 2024-04-01T18:23:21Z 2024-04-01T18:23:21Z 2024-03-30 2024-03-31T03:17:06Z Article http://purl.org/eprint/type/JournalArticle 0924-8463 1572-8382 https://hdl.handle.net/1721.1/153986 Martínez, E. Re-evaluating GPT-4’s bar exam performance. Artif Intell Law (2024). PUBLISHER_CC en 10.1007/s10506-024-09396-9 Artifcial Intelligence and Law Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf Springer Science and Business Media LLC Springer Netherlands |
| spellingShingle | Law Artificial Intelligence Martínez, Eric Re-evaluating GPT-4’s bar exam performance |
| title | Re-evaluating GPT-4’s bar exam performance |
| title_full | Re-evaluating GPT-4’s bar exam performance |
| title_fullStr | Re-evaluating GPT-4’s bar exam performance |
| title_full_unstemmed | Re-evaluating GPT-4’s bar exam performance |
| title_short | Re-evaluating GPT-4’s bar exam performance |
| title_sort | re evaluating gpt 4 s bar exam performance |
| topic | Law Artificial Intelligence |
| url | https://hdl.handle.net/1721.1/153986 |
| work_keys_str_mv | AT martinezeric reevaluatinggpt4sbarexamperformance |