AI Hallucination in Legal Cases Remain a Problem

One would think an artificial intelligence company would be sensitized to the risk of AI hallucination in legal citations. One would be wrong.

In Concord Music Group, Inc. v. Anthropic PBC, Magistrate Judge Susan Van Keulen of the U.S. District Court for the Northern District of California found that Anthropic had submitted an expert declaration that included a citation to a non-existent article — an “AI hallucination” produced when using the large language model (LLM) Claude.ai to format references.[1] Although the underlying real article was eventually found, the AI-generated error called into question the reliability of the expert’s overall submission. In her May 23, 2025 Order, the court viewed the incident as a “serious concern,” though not as egregious as outright reliance on fabricated content without verification.

This is just one recent example of litigation participants falling victim to AI hallucination. While typographical and cite-checking errors will occur from time to time as the result of honest mistakes, reliance on an LLM for research, writing, or editing can easily result in a subtle or overt hallucination being relied on in a brief, declaration, or other filing. Indeed, some courts — including the judge who will preside over the Concord Music case — require litigants to identify when AI has been used in relation to a filing or certify that a human being has personally verified the accuracy of the filing’s contents.

An attempt to track the extent of the hallucination problem in litigation can be found at AI Hallucination Cases. The website, run by Damien Charlotin (a research fellow at HEC Paris and lecturer at Sciences Po), currently identifies 129 cases worldwide in which a court decision discusses generative AI having producing hallucinated content that was cited to the court. Since the website does not include incidents of AI hallucination that were not the topic of a decision, its list necessarily underplays the extent of the problem. Notably, the number of such cases is accelerating rapidly: 32 are from May 2025 alone.

Given the convenient presentation of these cases, we have gone through the list and further characterized the jurisdictions where hallucinations were identified and what types of party submitted what types of hallucinations, as well as other factors.

• • • The vast majority of the cases were in the United States, a whopping 91 of 129. Other jurisdictions include Israel (14), U.K. (eight), Canada (three), Australia (three), Brazil (two), South Africa (two), The Netherlands (two), Ireland (one), Trinidad and Tobago (one), Italy (one), and Spain (one).
    • Lawyers (including prosecutors and public defenders) provided the hallucinations to the court in 59 of the cases. Interesting, a pro se litigant was the culprit in 62 cases, suggesting that both professionals trained to do legal research and pro se litigants likely untrained in legal research are vulnerable to hallucination.[2]
    • The vast majority of cited hallucinations took the form of citations to fictitious cases, improperly cited cases, or misrepresentations of cited cases. Other hallucinations were alleged based on “suspicious signs of generative AI use,” unverifiable AI-based calculations, and the like.
    • In the U.S. cases, monetary penalties were imposed 15 times on one or more individuals providing the hallucinatory material. The average was $4,713 and ranged from $100 to $31,100.

While LLMs have been in public use for less than three years, their integration into research and writing in all fields — including legal services — continues to grow. AI is a useful tool that can expedite and simplify the process of drafting, but has significant limitations, especially in distinguishing between fact and fiction. The risk of submitting a filing with an AI hallucination, or even just relying on such a hallucination, can expose a lawyer to serious professional, legal, and reputational risks. In order to reduce the likelihood of such an undesirable outcome, the following best practices should be followed.

• • • • Educate yourself on the basics of how LLMs work and their proper uses. When used correctly, they can provide significant efficiency and insight, but they are not experts and cannot be relied upon in the same way that one would a primary source, an academic paper, or even a Wikipedia article.
      • Review all LLM content for accuracy, bias, and completeness. Double check all citations and read the cited material to verify that it is properly represented in your writing. [3]
      • Use AI mainly to develop a first draft or to suggest edits to an early draft. Final drafts should be thoroughly proofread and checked by a designated human. The failure to identify one individual as the person responsible for proofing and cite-checking has led even large law firms to submit hallucinated cases or citations.
      • When required by court rules, disclose your use of AI. Consider doing so proactively even in the absence of court rules in order to establish transparency.
      • When subject to a looming deadline, you may be highly motivated to use AI to help you beat the clock. This is also when you are most vulnerable to missing a hallucination or other AI slop. Be extra careful when under pressure.
      • Maintain a healthy skepticism toward AI-generated “facts” or quotations. If a passage sounds too perfectly tailored to your argument or includes an unusually precise or obscure citation, see that as a red flag and investigate its source.
      • Keep a log or record of your AI interactions in case questions arise later. Saving prompt-response pairs used in substantive drafting can help reconstruct your process, demonstrate diligence, and defend against accusations of recklessness or misconduct. It may save you thousands of dollars in sanctions.
      • Train your colleagues, staff, and co-counsel on these best practices.

By treating LLMs as fallible tools rather than authoritative sources, lawyers can take advantage of their benefits while avoiding their pitfalls. Courts are increasingly scrutinizing AI-assisted filings, and the professional consequences of failing to catch an AI-generated hallucination can be severe. Adhering to careful use and review protocols after building institutional knowledge around responsible AI use, are essential steps for minimizing risk. As these technologies evolve, so too must our standards of diligence and verification.

[1] The order can be found at https://ecf.cand.uscourts.gov/doc1/035125784551.

[2] The overall rate of pro se litigants is difficult to obtain, but in Federal Civil cases from 2000-2019 it was 27% with prisoner civil rights petitions being 69% of all pro se cases. See https://www.uscourts.gov/data-news/judiciary-news/2021/02/11/just-facts-trends-pro-se-civil-litigation-2000-2019.

[3] Courts have cited Rules 1.1 (Competence), 1.3 (Diligence), 3.3 (Candor Toward the Tribunal), 3.4 (Fairness to Opposing Party and Counsel), 4.1 (Truthfulness in Statements to Others), and 5.1 (Responsibilities of Partners, Managers, and Supervisory Lawyers) of the Rules of Professional Conduct in finding that submission of hallucinated materials is not only improper, but also unethical.