AI Hallucination Cases

Father admitted using ChatGPT to generate case citations; 14 cited cases were non-existent or misattributed. Court treated them as AI hallucinations, disregarded them, awarded costs, and ordered declarations for future AI use.

Counsel's Amended Motion cited a non-existent case (In re Berry Good, LLC) generated by AI. Court found the citation fabricated, counsel admitted AI use and negligence, implemented safeguards, and court declined sanctions while transmitting matter to disciplinary office. The court's reflections are worth quoting in full.

"When examined, paragraph 15 is the result of a flawed AI search that suggested an incorrect generalization falsely supported by a specific but non-existent case. Under any scenario, a suspect argument buttressed with a case from whole cloth violates the expressed and tacit norms governing the practice of law as it intersects with the courts. Simply, you cannot make stuff up to convince a court to do something.

In simpler times, that would end the inquiry with draconian consequences. The times, however, are no longer simple; thus, the inquiry does not end here.

AI interposes additional considerations that defy easy categorization or scrutiny when examining the traditional norms of practice and advocacy. Threshold, two fields of inquiry present in any AI debate. The first is whether exposition artificially generated by a computer is acceptable in what has contextually been a forum for human critical analysis, thought, and advocacy. That is not the instance here. Rather, it is the second; that is, when AI sacrifices accuracy to satisfy the consumer and that inaccuracy is advocated before the court.

[...]

The only acceptable conclusion is that citing a made-up case is a false representation to the court that violates the expressed norms of practice, is actionable, and should bear consequences. The answer, however, to the second question —whether an artificial component alters the calculus—is more difficult.

Here, our traditional norms are challenged. In context, the traditional advocacy process begins with black letter law in the form of statutes, codes, and regulations. Then, there are judicial opinions and orders interpreting black letter law in the context of justiciable issues based on discrete facts. Lawyers cite cases for their binding or persuasive authority and have done so ever since someone started committing judicial decisions to stone, parchment, or paper. The ability to find and cite cases has evolved over time from laborious research in courthouses, to compendiums, indices, digests, reporters, advance sheets, key numbers, and now the internet.

The internet changed this dynamic in two ways. First, it streamlined, refined, and improved the research process. Users could select fields, topics, and key numbers to find and locate cases. Eventually, more specific tools, such as key words or phrases, enhanced the process. Second, and more pertinent to the present issue, AI is creating a process where instead of just finding authority, the computer also does the supplicant's thinking and analysis for them. The first innovation continues to evolve; the second is new, likewise continues to evolve, but constitutes an entirely new dimension, rather than a continuum, in legal practice. Finding has evolved into finding and thinking.

Travel back to research, the first changed dynamic. From stone to the internet, how lawyers found and used case law took many forms. Some would carefully read every case they found; some would read every page of select cases they thought pertinent; others might read the digest entry and then read only that section of the case; others might read only the digest entry; others barely read anything at all and cited anything that looked like it might support their position. Each was a personal decision by the lawyer for which he or she is accountable under the traditional norms of practice both officially—through codes of conduct and Rule 9011—and unofficially per the norms and expectations of courts.

AI compresses this historical dynamic. The research engine is now the search and thinking engine. AI finds case law and tells the subscriber that it supports their position. All the personalities described above—and known oh-so-well to us all—have become one.

Except, however, AI is flawed. In its infancy—albeit one that may be in college soon—it is immature and does not always provide accurate information or correct analysis; it may even make things up.

A tool that potentially supplants your advocacy by doing the finding and thinking for you is enticing. 3 But in the end, it is only a tool. Despite our near total and unfortunate faith in the internet, it does not relieve the attorney of his or her responsibility to make sure that the information and analysis are correct.

Reliance on AI can, however, mitigate intent. Here, Counsel is responsible for their negligence and misplaced reliance on AI, but there is no indication—and this court does not believe —that they purposely misled the court. They did not decide to make up a case. Misplaced reliance or negligence does not always equal intentional misrepresentation."

In her latter response to the Court, Counsel apologised for her use of a "a non-firm approved general AI service".

Magistrate Judge found Plaintiff's Reply contained multiple citations that do not exist, quotes not found in the cited authorities, and material mischaracterizations of cases; court concluded counsel likely used generative AI and failed to verify outputs and recommended the District Court consider Rule 11 sanctions.

The Agency raised concerns that several expert reports were prepared with assistance of an artificial intelligence program and contained unverified citations and inserted text. The Tribunal noted admissions by at least one practitioner (Ms McPhee) that AI assisted drafting and found instances where citations had been corrected by another witness and where the author could not confirm whether AI added certain phrases. The Tribunal criticised the lack of independent verification and gave the reports reduced weight but did not impose sanctions.

Counsel filed a motion containing case citations that did not exist; counsel admitted the citations were generated by Microsoft CoPilot and not independently verified. The court found a Rule 9011 violation, declined to impose monetary sanctions because of procedural limits and dismissal, and ordered CLE focused on AI ethics.

" Opposing counsel, and the court, should not have to parse case citations and parentheticals to discern whether cases exist, and if so, if they stand for the propositions asserted. "

Although the tribunal did not address it as a case of hallucinations or misuse of artificial intelligence, the details make clear that this was very likely at issue.

While the decision is not public, further details have been reported by, uh, me:

🔗 Damien Charlotin, Niger’s proposal to disqualify Fernando Mantilla-Serrano from uranium mining arbitration is rejected; challenge procedure is marred by citations and authorities that couldn’t be borne out when scrutinized by co-arbitrators (Investment Arbitration Reporter, 28 August 2025).

"Counsel should take seriously its obligation to provide the Court with an accurate description of the law. See, e.g., United States v. Hayes, 763 F. Supp. 3d 1054 (E.D. Cal. 2025) (levying $1,500 in monetary sanctions against counsel personally for fictitious cases and quotations that the court suspected were produced using artificial intelligence),reconsideration denied, No. 2:24-CR-0280-DJC, 2025 WL 1067323 (E.D. Cal. Apr. 9, 2025); Grant v. City of Long Beach, 96 F.4th 1255 (9th Cir. 2024) (striking an appellant's brief and dismissing an appeal for materially misrepresenting or fabricating case citations). After New Materials freely accused opposing counsel of misstating the law, New Materials’ submission of non-existent quotes is troubling (Dkt. 30 at 5 (“Chenco's argument for remand collapses under the weight of its own misreading of the law”); id.at 7 (“Chenco fundamentally misrepresents the applicable removal standard”); Dkt. 34 at 1 (“The proposed sur-reply ... is necessary to address new legal misstatements ....”); id. at 2-3 (“Chenco's failure to address this standard ... misstates controlling law and warrants correction.”)). Accordingly, the Court reminds counsel of their duties to act according to the Idaho Rules of Professional Conduct."

The self-represented Applicant admitted using generative AI (Google Gemini and ChatGPT) to prepare submissions. NCAT Procedural Direction 7 requires verification of citations when Gen AI is used; the Applicant did not verify citations. The Tribunal accepted the submissions into the record but placed greater weight on primary records and noted non-compliance with the verification requirement.

""101    The use of AI technology in the Courts has been the subject of judicial observations, particularly regarding legal practitioners who are subject to professional and ethical obligations and responsibilities. However, as Bell CJ observed in May v Costaras [2025] NSWCA 178 at [15], with whom Payne JA and McHugh JA agreed, in the context of considering the use of AI in the preparation of submissions, that “(a)ll litigants are under a duty not to misled the court or their opponent.” The reliance on unverified materials produced by generative AI does have the potential to misled the Court.

102    Although the termed used in relation to erroneously generated references by AI is “hallucinations”, this is a term which seeks to legitimise the use of AI. More properly, such erroneously generated references are simply fabricated, fictional, false, fake and as such could be misleading.

103    All persons appearing before the Court have a duty to verify that the case law and legislation referred to and relied on, is accurate and that such materials actually exist. The references in Ayinde at [85] and [86] and in Costaras at [14]-[15], to matters involving litigants who are acting in person who rely on AI generated material clearly supports the position that all are required to verify the submissions made to the Court. There are many publicly available legal research websites, some which are accessible without a fee. Further, and without attempting to be exhaustive, the Queensland Supreme Court library is open and available to the public.

104    The use of generative AI to prepare submissions that may include fake authorities will nearly always introduce added costs, complexity and add to the burden of other parties and to the Court: Costaras at [16] and [49]. I gratefully adopt the observation from Ayinde at [9] that “(t)here are serious implications for the administration of justice and public confidence in the justice system if artificial intelligence is misused.” The Court in Ayinde observed from [10] to [31] the existing guidance, regulatory duties of the profession, referrals and the Court’s powers. Matters which are within the Court’s own domain include in the most serious of cases contempt of Court. The observations in Ayinde at [26]-[28] regarding contempt of court are not limited to legal practitioners.

105    As the reasons above demonstrate, the circumstances of this case involved many fake authorities, fabricated quotes and false propositions. It is unhelpful for the Court to be referred to, and for parties to rely on, such matters."

"The case at bar epitomizes the concern. Inordinate judicial resources were expended on reviewing cases cited by Plaintiff that did not exist. No doubt Plaintiff’s opponent in this litigation was forced to expend similar energies. Here, too, as noted, certain cases Plaintiff cited in support of her arguments stood for the opposite result from that which Plaintiff stated in her briefs. This kind of activity not only wastes precious and limited judicial resources, but it also drives up the cost of litigation unnecessarily for those who must defend against or seek to prosecute claims on behalf of paying clients, given the underpinnings of the American Rule that attaches to most civil litigation in this country."

The debtor, Sonja Helvig DeRosa-Grund, engaged in substantial abuse of the Chapter 13 process by filing numerous frivolous motions and making false allegations against opposing counsel. The debtor repeatedly cited non-existent case law and fabricated quotes from existing cases, despite being warned about this behavior. The court dismissed the case with prejudice for one year, terminated the debtor's ECF filing privileges, and imposed additional protective measures to prevent future abuse. The court found that the debtor violated Bankruptcy Rule 9011(b) by making arguments based on non-existent case law and misquoting cases.

The court issued an Order to Show Cause, and the plaintiff's Counsel acknowledged responsibility but did not explicitly admit to using AI. The court determined that the counsel violated Rule 11(b)(2) by failing to verify the accuracy of the citations and imposed several sanctions:

• "The pro hac vice status of Counsel shall be revoked and Counsel will be removed from this case;
• Plaintiff’s Opening Brief shall be stricken;
• Counsel will be ordered to promptly serve a copy of this Order on Plaintiff, who will in turn be afforded time to engage new counsel or proceed as a self-represented litigant;
• Counsel will be ordered to write a letter to the three Judges to whom she attributed fictitious cases, [...], notifying them of her use of fake cases with their respective names attached;
• Counsel will be ordered to transmit a copy of this Order to every Judge who presides over any case in which Counsel is attorney of record; and
• The Clerk of Court’s Office will be directed to serve a copy of this Order on the Washington State Bar Association, of which Counsel is a member. If Counsel is a member of any other state’s bar, she shall serve a copy of this Order on that state’s bar office."

The court identified issues with the use of artificial intelligence in preparing written submissions. The submissions contained fabricated citations and fictitious quotes, which were initially filed as joint submissions by the defense and prosecution. Upon discovery, the defense counsel took responsibility, citing the use of AI without proper verification. The court allowed revised submissions to be filed, emphasizing the importance of accuracy in legal documents and the responsible use of AI. No professional sanctions or monetary penalties were imposed, but the court reiterated the need for adherence to guidelines on AI use in litigation.

The court imposed sanctions under Rule 11, requiring Mr. Owoc to complete 10 hours of community service and to certify the accuracy of legal citations in future filings if AI is used. No monetary penalty was imposed.

"Second, the protester's explanation--that it was “manual mismatches in secondary summaries” that caused the citation errors (Protester's Resp., Aug. 8, 2025, at 1)--does not meaningfully explain the number of citation errors in the protester's filings. Indeed, Oready's patently erroneous citations are far removed from mere typographical or scrivener's errors, and instead, bear the hallmarks of the use of a large-language model or other artificial intelligence (AI) without adequate verification that the generated results were accurate. "

" We have the authority to sanction Wife under Florida Rule of Appellate Procedure 9.410(a) for failure to comply with Florida Rule of Appellate Procedure 9.210(c). See Gutierrez v. Gutierrez, 399 So. 3d 1185, 1188 (Fla. 3d DCA 2024). However, we decline to do so. Our decision is tempered by the fact that Wife is defending a judgment on appeal in an unrepresented capacity, Husband has not sought the imposition of sanctions, and Wife has not brought any other meritless or frivolous filings in this Court. See id. at 1187–88; Al-Hamim, 564 P.3d at 1125–26. Instead, we admonish Wife for her counterfeit brief and warn her that the Court will not regard similar infractions as mildly in the future. "