Judges around the world have made headlines for illicitly using generative AI in their work. But in Pakistan, a large-scale trial of a specially designed AI tool for judges found the technology–together with appropriate training–boosted the number of cases resolved by 6.3% with no obvious drop in the quality of judgements.
With a backlog of 2.26 million cases and fewer than two judges per 100,000 people – compared to 22 in the EU and 8 in Brazil – Pakistan’s judiciary was in sore need of help. So, in consultation with the judiciary, economist Sultan Mehmood, of the New Economic School in Moscow, Russia, and collaborators tested whether AI could ease the burden.
They built a custom tool combining OpenAI’s GPT-4 large language model (LLM) with a knowledge base of nearly 130,000 Pakistani judicial opinions and statutes, to help judges with legal research and drafting judgements. They began offering the tool in 2024 to 1,559 trial judges – roughly half the country’s justices.
“We do find an increase in cases resolved, and we don’t find any corresponding decrease in decision quality,” Mehmood says.
First of its kind
“It’s pretty amazing that he’s able to pull this off,” says David Autor, an economics professor at the Massachusetts Institute of Technology. “It’s not easy to do large-scale field experiments in civil service, but especially where the stakes are so high.” The 6.3% productivity boost is not overwhelming, he says, but it’s credible and likely to improve as the tool is more widely used.
AI tools for judges are already being rolled out in Brazil and India, and prominent U.S. law professor Eric Posner has compared LLM judgments to human judgements in a single case study, but until now there have been no major independent assessment of ongoing judicial use of AI. The new study focused on Pakistan’s trial courts; Mehmood says judges there were enthusiastic from the start.
“They were more techno-optimist than we were,” he says. “The delays are so huge this is something which they thought was worth trying anyway to reduce people’s suffering.”
Some judges were also already using AI chatbots, Mehmood says, but commercial offerings performed poorly on Pakistani legal queries, frequently hallucinating case law. So the team built a tool tailored to the Pakistani context, called JudgeGPT.
They used retrieval-augmented generation (RAG), which allowed the model to query a database of 128,292 Pakistani judicial opinions and 943 statutes. Responses included footnotes linking to cases and laws.
“It turns out that actually the way to fix [hallucinations] isn’t just more intelligent models,” says study co-author Elliott Ash, an associate professor of law, economics, and data science at ETH Zurich in Switzerland. “It’s to attach the models to a tool that can do a search and verify the sources.” However, the researchers do not report hallucination rates.
The team also put 1,197 judges through six 90-minute Zoom training sessions where Ash covered how LLMs work, their limitations, the risk of bias and hallucinations and the importance of verifying outputs. Another 180 judges only underwent general training on technology in legal research, while a final group got no training.
By the time 487 judges had been through the program, the median district saw a jump of 6.3% resolved cases, and the more trained judges in a district, the bigger the effect. Appeal rates also fell slightly, suggesting faster resolution wasn’t leading to sloppier decisions.
The JudgeGPT can be used to surface relevant case law with a simple text query and results provide links to the full text of the related judgements.Sultan Mehmood, Christoph Goessmann, and Elliott Ash
Addressing limitations
The team also assessed the quality of judgements. Having legal experts evaluate large numbers of judgements was infeasible, Mehmood says, so the team asked OpenAI’s GPT-5-mini to choose between pairs of judgements from the same judge before and after training. The LLM chose post-training judgements 59% of the time. Two experienced Pakistani lawyers also evaluated the model’s analysis of 90 judgement pairs. They agreed with GPT-5-mini 70.6% of the time, compared to 73% agreement with each other.
Training turned out to be vital. On average, JudgeGPT-trained judges logged in 56 times and sent 212 prompts over the study period, compared to 10 logins and 25 prompts after generic training. Those who had no training tended to use the tool for around a month and then drop off entirely, Mehmood says. “Just giving people the technology does not necessarily make them use it persistently,” he says.
A 6.3% increase sounds modest, but the researchers calculated that a trained judge was resolving 38.5 more cases a month than the baseline, translating to roughly US $38.50 saved in judicial costs for every dollar spent running the tool. Ash also notes that these figures come from a nine-month period at the start of the trial, and that they’ve since updated both the underlying AI model and the
[…]
Content was trimmed to protect the source. Please visit the original article for the full text.
Read the original article: