Over 2,000 Court Decisions on Fake AI Citations: What the Sanction Cases Teach Law Firms About Using AI Safely

ByBNW Staff

In 2023, two New York lawyers filed a brief in Mata v. Avianca that cited six court decisions. None of them existed. ChatGPT had invented them, quotes and all. The judge fined the lawyers and their firm $5,000, and the case became the profession’s cautionary tale.

Most people assumed it would be a one-off. It was a beginning.

Researcher Damien Charlotin keeps a public database of court decisions where a judge found, or clearly implied, that a filing relied on AI-generated fabrications. In mid-2025 it held around 200 cases. By September 2026 it had passed 2,000, with roughly 1,400 from the United States. Most come from self-represented litigants, but hundreds involve qualified lawyers, including at large and well-resourced firms.

This article is for law firm partners and practice managers who don’t build software but have to decide how their firm uses it. The lesson from 2,000 cases is not “ban AI.” It is about how the work is set up.

The cases are not getting less serious

The early sanctions were small fines and embarrassment. They have become more severe.

  • In 2025, a federal judge in Wyoming fined lawyers from Morgan & Morgan, one of the largest personal injury firms in the US, over AI-invented citations in a case against Walmart. The drafting lawyer’s permission to appear in that case was revoked.
  • Also in 2025, a federal judge in Alabama disqualified lawyers from the firm Butler Snow from further work on a case after fabricated citations appeared in their filings, and referred the matter to the state bar.

The pattern in the judges’ written opinions is consistent. Courts are rarely angry that a lawyer used AI. They are angry that nobody checked. The duty to verify what you sign has not changed since long before AI existed.

The uncomfortable finding: legal AI tools hallucinate too

The common response from firms is “we don’t use ChatGPT, we use a legal research product built for lawyers.” That helps. It does not solve the problem.

In 2024, researchers at Stanford tested leading legal AI research tools from the major legal publishers. These products are built specifically to reduce hallucinations by grounding answers in real case law. They still produced incorrect or misleading answers between roughly 17% and 33% of the time, depending on the tool.

That is far better than a general chatbot. But it means that one in every handful of answers can still be wrong, and the wrong answers look just as confident as the right ones.

So the real question for a firm is not “which AI tool never makes mistakes?” No such tool exists. The question is: “What in our process catches the mistake before a judge does?”

Why AI invents cases (in plain English)

A language model doesn’t look things up the way a paralegal does. It predicts what text should come next, based on patterns. Ask it for a case supporting your argument and it knows what a case citation looks like: party names, a reporter, a volume, a page, a year. If no real case fits, it can produce one that looks exactly right.

From building AI systems for clients, I’d add one thing that is rarely said: these models do not give the same answer twice. Even with identical questions and the most cautious settings, the output can vary. A tool that cited correctly in your test last month can invent something next Tuesday. Any safeguard that depends on “we tried it and it worked” is not a safeguard.

 The design lesson: split the job into checkable steps

When we do AI agent development for regulated work, we follow one rule: don’t ask one AI model to do everything and then trust its single answer. Break the job into steps, and make the risky steps ones that a computer or a person can check with a clear yes or no.

For legal drafting, that looks like this:

  1. The AI helps write the argument. This is where it is genuinely useful: structure, first drafts, summarising long documents.
  2. Citation extraction. Every citation in the draft is automatically pulled into a list. This is a simple, mechanical step. No judgement involved.
  3. Citation verification. Each citation is checked against a real legal database. This step is not AI deciding whether a case “sounds right.” Either the case exists at that reporter, volume and page, or it does not. Anything that fails, or matches only loosely, is flagged.
  4. Human review. A lawyer reviews every flagged citation, and reads the cited passage of every case the argument depends on.

The value of splitting it up is that a mistake in step one gets caught in step three, by a check that cannot be talked into agreeing. One AI model checking its own work is just the same pattern-matcher marking its own homework.

Two more risks firms are not watching for

Hidden instructions inside documents. If your firm uses AI to read incoming documents, emails or discovery material, be aware of “prompt injection.” Someone can hide text in a document (white text on a white background, or tiny print) that a human never sees but the AI reads and may follow. In legal work, where you routinely handle documents from opposing parties, treat everything the AI reads from outside as untrusted, never as instructions.

The difference between a search tool and a working system. Many “legal AI agents” on the market are a fairly thin layer: your documents loaded into a search index, connected to a general AI model, with a dashboard on top. That can be useful for finding things. It is not a system that checks its own output. Ask any vendor: “What happens when the tool can’t find a supporting authority?” A good answer describes a fallback, such as flagging the gap to a human. A weak answer talks about better prompts.

A practical policy any firm can adopt this month

You don’t need new software to cut most of the risk. You need a written process that people actually follow:

  • No citation leaves the firm unverified. Every case, statute and quote is checked against a primary source by a named person, whatever tool produced the draft.
  • Record AI use on each matter. Note which tool was used and for what, so you can answer a court’s question honestly if one comes.
  • Check court and client rules. A growing number of courts require disclosure of AI use, and some clients restrict it in their engagement terms.
  • Train on the failure, not the feature. Show your team real sanction opinions. They teach more than any vendor demo.
  • Test with real work before you trust a tool. In our experience, the first hundred real uses of any AI system reveal several failure modes that weeks of internal testing missed. Run a controlled trial with full review before anyone relies on it. If you want an outside review of a tool before that trial, AI consulting is a sensible place to start.

Where AI genuinely earns its place in a law firm

None of this means firms should stay away. Used with proper checks, AI is excellent at intake, document summaries, first drafts, organising discovery and handling routine client updates. The firms that get burned use it as an oracle. The firms that benefit use it as a fast junior whose work is always reviewed.