In a motion filed in the Southern District of New York, attorney Robert Schwartz submitted a 14-page legal brief in support of his client's personal injury claim against Avianca Airlines. The brief cited six prior court decisions to support its arguments.
All six were fabricated by ChatGPT.
The cases had plausible names — Varghese v. China Southern Airlines, Martinez v. Delta Air Lines. They referenced real courts and included properly formatted citation numbers. A judge reading the brief would have had no immediate reason to question them.
Opposing counsel could not locate the cases. When the court ordered Schwartz to produce copies of the decisions he had cited, he turned back to ChatGPT and asked it to confirm the cases were real. It confirmed they were.
"The court is presented with an unprecedented circumstance," wrote Judge P. Kevin Castel in his order. "A submission replete with citations to non-existent cases."
Schwartz was sanctioned. He told the court he had "no intent to deceive" and was "unaware that the content could be false."
This case, while now well-documented, represents a category of AI failure that is growing faster than the legal profession's ability to address it. Autominous has identified 14 additional instances across US, UK, and Australian courts in the past 18 months where AI-generated legal filings contained fabricated citations.
The pattern is consistent: the AI generates text that is structurally correct but factually invented. The confidence of the output — no hedging, no uncertainty markers — makes it indistinguishable from accurate research to a user who does not independently verify every citation.
Legal AI companies including Casetext, Harvey, and CoCounsel have introduced citation-verification features. But adoption is uneven, and the underlying models still hallucinate at rates between 3% and 12% depending on the complexity of the legal question, according to a Stanford HAI study published last month.
What we know for certain
Attorney Robert Schwartz submitted fabricated citations generated by ChatGPT in a federal court filing. He was sanctioned. At least 14 similar incidents have occurred in other jurisdictions since 2024.
What we are inferring
The rate of AI-generated legal hallucinations entering court filings is likely higher than documented, as cases where fabricated citations are caught before filing are not publicly recorded.
What we couldn't verify
The exact hallucination rates of current legal AI tools in real-world use. Published benchmarks test controlled scenarios that may not reflect the complexity of actual legal research.