AI Hallucination Cases

In May, the court asked Counsel to show cause why they should not be sanctioned for filing briefs with hallucinations - especially since they continued filing hallucinated submissions after being warned about it.

In their Answer, Counsel revealed that "specific citations and quotes in question were inadvertently derived from internal draft text prepared using generative AI research tools designed to expedite legal research and brief drafting".

In the Order, the court noted that Counsel "was not candid to the Court when confronted about his use of AI, stating that some of these documents were “prepared under time constraints,” when he had nearly two more weeks before the deadline to submit his responses." The judge was also unimpressed by Counsel's attempt to shift the blame to a paralegal.

Finally, in the fee dispute order, the court cited this database to point out that its approach to award costs and fees was appropriate, especially given the egregiousness of the claimant's conduct in this case. The judge also took into account "the significant, if indirect, monetary losses that may arise from nonmonetary sanctions in other cases, such as loss of business and loss of reputation, or the monetary loss borne by a client when a motion or even an entire case is adversely decided due to counsel’s misuse of AI."

Plaintiff's Counsel cited nine non-existent cases in a response to a motion to dismiss, which were generated using AI software. The court found this to be a violation of Rule 11, as the citations were not checked for accuracy. Counsel admitted the error, apologized, and explained the circumstances, including staff transitions and the use of AI. The court imposed a $1000 sanction, required Mr. Braddy to attend a CLE course on AI ethics, and ordered reimbursement of Defendant's attorney fees and costs.

Counsel confessed having Claude Sonnet 4 to draft a legal submission, which included fabricated quotations from legal authorities. Counsel said he was pressed by time.

After holding that there is "no reason to distinguish between the submission of fabricated cases and the submission of fabricated quotations from real cases. In both postures, the attorney seeks to persuade the Court using legal authority that does not exist", the court held that Mr. Desmarais had violated Rule 11 of the Federal Rules of Civil Procedure by failing to verify the accuracy of the AI-generated content. Counsel was found to have acted in subjective bad faith, as he was aware of the potential for AI to hallucinate legal citations and failed to take corrective action even after the government pointed out the errors.

The court imposed a $1,000 monetary sanction and required Counsel to complete a CLE course on the ethical use of AI in legal practice and notify his client of the issue.

Prior Order to Show Cause available here.

After reviewing - and dismissing - the factual allegations made by Counsel, and noting that they had submitted errata in parallel cases (dealing with other fabricated citations), the court swiftly concluded that they "have violated Rule 11 because they were not reasonable in certifying that the claims, defenses, and other legal contentions contained in Defendants’ Opposition to Motion in Limine [Doc. 283] were warranted by existing law or by a nonfrivolous argument for extending, modifying, or reversing existing law or for establishing new law."

Both counsel were sanctioned with a 3,000 USD fine, payable to the court.

"Further, these inaccuracies signal that Plaintiff's counsel may have used AI to assist in the drafting of Plaintiff's Response (or otherwise counsel produced exceptionally sloppy work). In this regard, this Court's Chambers Rules include “Disclosure and Certification Requirements” for use of “Generative Artificial Intelligence” and expressly provide that an attorney or party must disclose in any document to be filed with the Court “that AI was used and the specific AI tool that was used” and to “certify in the document that the person has checked the accuracy of any portion of the document drafted by generative AI, including all citations and legal authority.” See id.6 No such disclosure and certification has been made in this case. The Court's Rules further provide that an attorney will be responsible for the contents of any documents prepared with generative AI, in accordance with Rule 11 of the Federal Rules of Civil Procedure, and that the failure to make the disclosure and certification “may result in the imposition of sanctions.”"

The court noted that:

"both Lehr-Guthrie's and McDonald's briefs are replete with errors in their citations to case authority, such as repeated citation errors, references to nonexistent quotes, and incorrect statements about the cases (for instance, as noted above, the two cases McDonald cited for a proposition relating to the duty of impartiality don't even reference that duty). This suggests to us that the briefs may have been drafted with the use of generative artificial intelligence (GAI). “[U]sing a GAI tool to draft a legal document can pose serious risks if the user does not thoroughly review the tool's output.” Al-Hamim v. Star Hearthstone, LLC, 2024 COA 128, ¶ 32. Self-represented litigants must be particularly careful, as they “may not understand that a GAI tool may confidently respond to a query regarding a legal topic ‘even if the answer contains errors, hallucinations, falsehoods, or biases.’ ” Id. (citation omitted).4 We advise the parties that errors caused by GAI in future filings may result in sanctions. See id. at ¶ 41. "

The court warned that future errors caused by AI could result in sanctions.

"In a prior order, we struck portions of Walker's brief that included citations to nonexistent cases. We note that the arguments raised would not have changed the outcome of the appeal in any event. "

"Compounding the problem, what generative AI lacks in precision, it more than makes up for in speed. Litigants who simply file the material that AI tools generate, without carefully reviewing it first for accuracy, have the potential to swamp courts with what appear at first glance to be legal arguments built on law and precedent, but which are in fact nothing of the sort. And not only are these problems in their own right, but they also heighten the two concerns the Court highlighted above—that defendants will be forced to spend more time and incur more costs parsing through copious baseless filings to defend an action, and that Courts will waste precious time doing the same in ruling on motions and moving matters along."

(Plaintiff acknowledged use of ChatGPT in a subsequent filing)

Plaintiff was eventually designated as vexatious litigant and his case dismissed with prejudice.

Defendant Holly Nelson, appearing pro se, filed a motion to dismiss that included a fabricated case citation, Finch v. Smith, which does not exist. The court inferred that Nelson used AI to draft the motion without verifying the accuracy of the citations. The court issued an order to show cause, requiring Nelson to justify why her filing does not violate Rule 11, or alternatively, to withdraw her motion. If she fails to do so, the court intends to strike her motion entirely.

Fake citation, in this brief, was to : Moms Against Poverty v. Dep’t of State, 2022 WL 17951329, at *3 . Case docket can be found here.

Counsel later confirmed having used ChatGPT and apologised.