- The University of Minnesota Law School, ChatGPT took their law school exams alongside real students.
- The chatbot was a C student, with essay aptitude better than multiple choice.
A group of faculty at the University of Minnesota Law School put the AI bot ChatGPT through four trials alongside real students and found that the bot — credited as the world’s most advanced example of generative artificial intelligence — is a C student on paper .
Curious about how ChatGPT could be used to both help students cheat and help teachers teach, professors decided to test its capabilities.
Associate Law School professor Jon Choi ran exam questions through ChatGPT, reformatted them to fit the exam, and blended the resulting exams with the students’ tests.
ChatGPT scored a C+ average and “a low but passing grade” in the areas of constitutional law, employee benefits, tax and tort, according to Choi. (According to the paper, it ranked 36th out of 40 students in Con Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and bottom out of 75 out of 75 students in Torts).
It performed better on essays than on multiple-choice questions. The bot performed particularly poorly on multiple-choice questions involving math,”who bombed ChatGPT.”
In essays, it could recite legal rules and describe cases correctly (without citations), but failed to “detect problems” and provide deep arguments or analysis, Choi said.
Of the experiment’s three human raters, all of whom scored the tests blindly, at least two told Choi they had an idea which tests came from AI — and they were right. For example, ChatGPT replies used perfect grammar and were more repetitive. (Choi’s co-authors—Kristin E. Hickman, Amy Monahan, and Daniel Schwarcz—served as the three raters).
At the moment it is relatively easy to tell the difference in the exams. “I think professors would almost always get it right,” he told Insider. But a rough draft created by ChatGPT that a student is revising would be “much harder to spot,” he said.
While the exams are graded along a curve and ChatGPT is ranked as one of the worst students in the class, Choi pointed out that he finds the University of Minnesota Law School students to be relatively stiff competition. According to the university, a whopping 97% of first-time test-takers pass the Minnesota bar exam, and as of fall 2020, the pass rate for Minnesota law graduates who pass the bar for the first time was the third-best in the United States, behind Harvard and Yale.
“I would say just by knowing the law and being able to summarize and present legal arguments, Minnesota students will be among the best in the country,” Choi told Insider. “ChatGPT comes in at the bottom of a class where 99% of students pass the bar exam. Almost all students will be successful in their careers. So it’s a tough comparison.” While the results would have landed ChatGPT on academic probation, it would probably still be good enough to earn a law degree.
“Overall, ChatGPT wasn’t a great law student who acted alone,” Choi said tweeted on 23“but we expect that working with humans, language models like ChatGPT would be very useful for law students taking exams and practicing lawyers.”
Choi believes ChatGPT could be a boon to his students and colleagues if used intentionally. Choi used it to write code for the newspaper’s statistics and graphs, and said it worked “VERY well” and noted that it “generated simple code that would be tedious to write (e.g. bootstrapping p-values, creating chart grids).”
While some school districts have banned ChatGPT, some teachers have embraced it as an educational tool.
Instead of pretending ChatGPT doesn’t exist, Choi focused on how law schools can adapt to a changing environment. To do this, professors may have to forgo exams that primarily test basic legal knowledge.
“As lawyers start using these tools to expand their skills as lawyers, we need to change our teaching and testing accordingly,” Choi told Insider.
“When they invented calculators, they didn’t force students to continue doing calculations by hand,” Choi said. “They allowed them to use calculators and just changed the nature of the test. I think that is the task of legal education.”
OpenAI, the company responsible for ChatGPT, recently received a staggering “multi-billion dollar investment” from Microsoft. In a post, the company said the investment “will allow us to continue our independent research and develop AI that continues to become safer, more useful and more powerful.”
E-mail tips for everything to do with the Internet [email protected].