Concerns regarding AI hallucinations within the legal system have been renewed after a high court case involved a senior crown prosecutor citing a fake legal precedent. Court records show that Ravinder Dhillon, who was acting for the Crown Prosecution Service (CPS), mentioned a completely fabricated case while arguing if a child should face prosecution for raping his younger sister.
The error in court
Working within the CPS’s rape and serious sexual offences unit as a senior crown prosecutor, Dhillon supported the child’s prosecution by providing a specific reference to CPS – R v J [2024] EWCA Crim 3529, a case that was later found to be nonexistent. In April, Dhillon decided to move forward, a choice the chief crown prosecutor subsequently approved. By October, Dhillon acknowledged the citation was mistaken. She explained that her intention was to refer to an actual decision from 15 years prior.
Two high court judges of senior rank described the explanation for using the fake case as “unsatisfactory,” while a specialist in law and AI termed it a “serious failure.” It is understood by The Guardian that the citation originated from an AI-generated answer during a search on the internet.
“This was used as a legal source to argue for the prosecution of a child and that legal source doesn’t exist: that’s a serious failure,” stated Brian Flanagan, an associate professor of law at Ireland’s Maynooth University who studies how judges employ AI. “It calls into question norms of trust in the law where we treat authorities as precious things. You can’t make them up; it’s out of bounds. AI increases the risk that is going to happen because of the time you can save.”
Related Post: Supreme Court to weigh federal limits on state election rules
Impact on the case
According to the high court, the outcome of the boy’s challenge against the prosecution decision was not impacted by the CPS citing a fake case; the trial is scheduled for the youth courts later this week. The boy, aged 11 during the alleged rape of his seven-year-old sister, was represented by lawyers who contested the CPS’s choice to prosecute. These lawyers claimed that the prosecution ignored input from a forensic psychologist, youth workers, and social workers.
According to the CPS, they maintain a strict AI policy “which limits its use so that it is not used to make legal decisions and there must be human checks for accuracy”. A representative for the agency stated that the CPS “identified an error had occurred from conducting legal research using the internet”.
“As soon as this error was identified, it was remedied and an internal review was carried out to establish how this had occurred and to check that no other cases have been affected,” the spokesperson added. “Following the rapid review we shared learnings, including that case law must be cited from trusted legal sources.”
It is believed that an AI-produced summary containing the made-up case information was the result of the internet search.
Related Post: Court to rule on late defense claims
Sanctions and Future Guidance
Dame Victoria Sharp warned of “serious implications for the administration of justice and public confidence in the justice system if artificial intelligence is misused” and said lawyers misusing AI could face sanctions, from public admonishment to contempt of court proceedings and referral to the police. The high court responded to AI hallucinations in housing and banking cases by calling on the Bar Council and the Law Society to take steps to address the problem. The CPS said it has a strict policy limiting AI use so it does not make legal decisions and requires human checks for accuracy.
Wider context
Most courtroom errors linked to AI this year have involved self-represented individuals without legal qualifications, frequently in employment tribunal matters, unlike this incident involving a professional. This marks the second instance involving the CPS—the primary public body for criminal prosecutions in Wales and England—which acknowledged in February that a lawyer in an extradition matter concerning two Romanians convicted of minor crimes had cited fake authorities, probably due to AI.
These fictional cases were discovered only after opposing counsel failed to find them; consequently, the CPS determined “the risk of recurrence was low”.
A UK tracker of disputed, suspected, and confirmed AI errors and false citations is kept by Matthew Lee, a barrister with Doughty Street Chambers. Lee suggested that the recorded instances might be the “tip of the iceberg” because many decisions from lower courts are not made public. He further noted that since the CPS manages highly sensitive data, using AI presents risks to confidentiality and privacy. Additionally, he cautioned that AI is increasing the workload for barristers, as they must now verify the validity of legal authorities in documents where AI might have assisted in the drafting.
