Key takeaways Nevada County prosecutors used AI software that generated fake legal citations and misinterpreted laws in criminal court filings. Defense lawyers identified errors bearing AI hallucinati...
A Northern California prosecutor's office has confirmed using artificial intelligence software to write criminal court filings that contained fabricated legal citations and nonexistent case precedents, raising concerns about the technology's impact on criminal justice.
Nevada County District Attorney Jesse Wilson acknowledged that a prosecutor used AI to prepare a filing in the case of Kalen Turner, who faced five felony and two misdemeanor drug charges.
The brief included what AI researchers call "hallucinations"—information invented by the software rather than based on actual legal precedent.
"A prosecutor recently used artificial intelligence in preparing a filing, which resulted in an inaccurate citation," Wilson stated to The Sacramento Bee. "Once the error was discovered, the filing was immediately withdrawn."
The admission came after defense lawyers discovered what they describe as AI-related errors in multiple cases from Wilson's office.
Attorneys representing Kyle Kjoller, a 57-year-old welder charged with illegal gun possession, identified similar mistakes in at least four separate criminal cases.
When Kjoller was ordered held without bail in April, prosecutors filed an 11-page brief opposing his release.
His lawyers contend the document was filled with errors characteristic of generative AI, including wholesale misinterpretations of law and quotations that don't actually appear in the cited texts.
"The Nevada County District Attorney's Office has, in at least three criminal cases in recent weeks, filed a brief citing fabricated (legal) authority," wrote lawyers from the nonprofit Civil Rights Corps, which is representing Kjoller along with a Nevada County public defender.
Kjoller's attorneys filed a petition with the California Third District Court of Appeal requesting sanctions against the prosecutor's office, including reimbursement of $23,000, which they said was necessary to identify and challenge the errors.
The court denied the sanctions request but ordered a new bail hearing for Kjoller, who was ultimately convicted on firearms charges at trial.
On Friday, a coalition of 22 legal and technology scholars filed a brief with the California Supreme Court supporting Kjoller's case and warning about the dangers of unchecked AI use in criminal prosecutions.
The group includes Barry Scheck, co-founder of the Innocence Project, which has helped exonerate more than 250 people; Chesa Boudin, former district attorney of San Francisco; and Katherine Judson, executive director of the Center for Integrity in Forensic Sciences.
"Prosecutors' reliance on inaccurate legal authority can violate ethical rules, and represents an existential threat to the due process rights of criminal defendants and the legitimacy of the courts," Kjoller's lawyers wrote in their petition.
The petition notes that AI hallucinations in legal research occur between 17% and 82% of the time, depending on the tool and task.
When prosecutors cite fabricated legal authority, judges may rely on these misrepresentations in their rulings, potentially leading to wrongful convictions or improper detention.
Wilson maintains that only one case involved AI errors, with mistakes in other cases stemming from human error rather than artificial intelligence.
He disputed that AI was used in Kjoller's case or in the case of Taylor McGrath, who was charged with felony child endangerment and drug offenses.
"Just because the Turner case involved AI, it cannot now be assumed that every citation error stems from the use of artificial intelligence," Wilson said.
However, Kjoller's attorneys contend the errors across multiple cases share distinctive characteristics of AI-generated content.
The petition includes examples of six cases cited in briefs that appear to be hallucinations.
Wilson's chief deputy, Lydia Stuart, denied threatening defense attorneys with counter-sanctions, though she referenced California law prohibiting frivolous sanction requests in an email to Kjoller's lawyers.
Following the revelations, Wilson's office has taken steps to prevent future AI-related errors. The district attorney said all attorneys in the office were reminded to verify legal citations independently.
"All of the attorneys in the office were reminded to verify all legal citations independently and not rely on AI-generated material without confirmation from reliable sources," Wilson stated.
The Nevada County DA's office began using Westlaw Advantage, a specialized AI legal research tool, in September—the same day it filed the last of the four flawed briefs identified by Kjoller's lawyers.
"Prosecutors work diligently and in good faith under heavy caseloads and time constraints, with a deep sense of responsibility to the law, the Court, and the community we serve," Wilson said. "At no time was there any intent to mislead the court, and any characterization to the contrary misstates the facts."
A database maintained by researcher Damien Charlotin at HEC Paris tracks over 590 cases worldwide where courts have detected AI hallucinations in legal filings.
More than half involved people representing themselves, and two-thirds occurred in United States courts.
Until now, only one case involved a prosecutor using AI—an Israeli case.
The California incidents appear to be among the first documented instances of American prosecutors relying on AI-generated content in criminal filings.
The California Supreme Court has not yet indicated whether it will consider Kjoller's petition requesting an investigation into whether the errors represent a wider pattern of prosecutors seeking rulings against defendants based on nonexistent citations and legal holdings.
Deloitte Faces Second AI Citation Scandal In Million-dollar Government Report
Law Enforcement Agencies Turn to AI for Non-Emergency Call Management
Amazon Directs Engineers To Use In-house Kiro Over Third-party AI Coding Tools
Key takeaways OpenAI has signed a multi-year, $10 billion agreement with AI chipmaker Cerebras Systems to secure computing infrastructure. The deal will deliver 750 megawatts of computing power throug...
KEY TAKEAWAYS: xAI implemented restrictions preventing Grok from editing images of real people in revealing clothing after global backlash California Attorney General Rob Bonta launched investigation...
Key takeaways TSMC reported fourth-quarter net income of NT$505.74 billion, beating analyst expectations and setting a new record. Fourth-quarter revenue reached NT$1.046 trillion, up 20.5% year over...
