AI Hallucination Cases

The applicant's representative (Mr M Archer) cited various cases in closing submissions; the Deputy President formed the view he may have used artificial intelligence, asked him, and he admitted doing so. He was asked to supply a list of cited cases with links; the Commission stated it would disregard any cases for which citations/links could not be produced.

The court eventually granted some costs in a subsequent decision (see here).

Misuse of generative AI was later confirmed by the registrar.

The appellant admitted using generative AI to prepare her Amended Grounds of Appeal. The Appeal Panel and respondent identified numerous incorrect or inapplicable citations and legal propositions generated by AI; the panel noted the appellant failed to verify those citations contrary to NCAT Procedural Direction 7 and treated the Amended Grounds as unreliable in parts when assessing whether questions of law were raised.

The appellant admitted using generative AI to prepare his Notice of Appeal and Summary of Argument. The Court found several cited authorities could not be located (concluding they were fictitious) and held that deploying unverified AI-generated research that cites non-existent cases breaches duties not to mislead the court and risks contravening Pt XIVB of the Family Law Act 1975 (Cth) (on confidentiality of proceedings).

The Agency raised concerns that several expert reports were prepared with assistance of an artificial intelligence program and contained unverified citations and inserted text. The Tribunal noted admissions by at least one practitioner (Ms McPhee) that AI assisted drafting and found instances where citations had been corrected by another witness and where the author could not confirm whether AI added certain phrases. The Tribunal criticised the lack of independent verification and gave the reports reduced weight but did not impose sanctions.

The self-represented Applicant admitted using generative AI (Google Gemini and ChatGPT) to prepare submissions. NCAT Procedural Direction 7 requires verification of citations when Gen AI is used; the Applicant did not verify citations. The Tribunal accepted the submissions into the record but placed greater weight on primary records and noted non-compliance with the verification requirement.

""101    The use of AI technology in the Courts has been the subject of judicial observations, particularly regarding legal practitioners who are subject to professional and ethical obligations and responsibilities. However, as Bell CJ observed in May v Costaras [2025] NSWCA 178 at [15], with whom Payne JA and McHugh JA agreed, in the context of considering the use of AI in the preparation of submissions, that “(a)ll litigants are under a duty not to misled the court or their opponent.” The reliance on unverified materials produced by generative AI does have the potential to misled the Court.

102    Although the termed used in relation to erroneously generated references by AI is “hallucinations”, this is a term which seeks to legitimise the use of AI. More properly, such erroneously generated references are simply fabricated, fictional, false, fake and as such could be misleading.

103    All persons appearing before the Court have a duty to verify that the case law and legislation referred to and relied on, is accurate and that such materials actually exist. The references in Ayinde at [85] and [86] and in Costaras at [14]-[15], to matters involving litigants who are acting in person who rely on AI generated material clearly supports the position that all are required to verify the submissions made to the Court. There are many publicly available legal research websites, some which are accessible without a fee. Further, and without attempting to be exhaustive, the Queensland Supreme Court library is open and available to the public.

104    The use of generative AI to prepare submissions that may include fake authorities will nearly always introduce added costs, complexity and add to the burden of other parties and to the Court: Costaras at [16] and [49]. I gratefully adopt the observation from Ayinde at [9] that “(t)here are serious implications for the administration of justice and public confidence in the justice system if artificial intelligence is misused.” The Court in Ayinde observed from [10] to [31] the existing guidance, regulatory duties of the profession, referrals and the Court’s powers. Matters which are within the Court’s own domain include in the most serious of cases contempt of Court. The observations in Ayinde at [26]-[28] regarding contempt of court are not limited to legal practitioners.

105    As the reasons above demonstrate, the circumstances of this case involved many fake authorities, fabricated quotes and false propositions. It is unhelpful for the Court to be referred to, and for parties to rely on, such matters."

The court identified issues with the use of artificial intelligence in preparing written submissions. The submissions contained fabricated citations and fictitious quotes, which were initially filed as joint submissions by the defense and prosecution. Upon discovery, the defense counsel took responsibility, citing the use of AI without proper verification. The court allowed revised submissions to be filed, emphasizing the importance of accuracy in legal documents and the responsible use of AI. No professional sanctions or monetary penalties were imposed, but the court reiterated the need for adherence to guidelines on AI use in litigation.