In March 2023 lawyers for a man injured on an Avianca flight filed a brief citing decisions that ChatGPT had supplied on request. Six of them did not exist. The lawyers stood by the opinions after court orders questioned them. One later declared that the first order prompted him to ask ChatGPT directly whether a case was real, and it said yes. A New York federal judge sanctioned both lawyers and their firm, with a US$5,000 penalty.1Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), opinion and order on sanctions. The opinion notes that “no date is given for the queries”.
ChatGPT had done what it was built to do. A large language model, the kind of system behind it, produces statistically likely text, and in a legal brief likely text has a party, a court and a page number. Nothing in the output marks the difference between what is likely and what is true. That makes these models a poor default for business decisions: useful where someone checks the work, and costly where the checking is assumed.
The judge wrote that there is “nothing inherently improper” in using a reliable AI tool for assistance. His complaint was with what the lawyers did around it. Anyone signing a decision built on a model’s output should expect to have to explain it.
Blaming the machine has been tried. In February 2024 a tribunal in British Columbia held Air Canada liable after its website chatbot gave a passenger wrong advice on bereavement fares. The airline’s position, as the tribunal put it, amounted to saying the chatbot was “a separate legal entity that is responsible for its own actions”. The tribunal called that “a remarkable submission” and ordered the airline to pay C$812.02.2Moffatt v. Air Canada, 2024 BCCRT 149, quoted by Barry Sookman. The sum includes interest and fees. The Canadian Bar Association’s magazine, National, reports that this chatbot was not AI-driven.3Doug Beazley, “Proving AI’s misrepresentations”, National, March 2024. The case therefore only hints at how a language model would be treated.
The confident guesser
A model asked for a pricing strategy produces what one tends to look like, and the result resembles a strategy in much the way a wax apple resembles lunch. Its output drifts towards the generic, the statistical average of everything it read.
People fall back on habit too. An analyst, though, also carries what no document records: the client meeting that went sideways, the faces in the room when last quarter’s numbers were presented. A model has only patterns, and those patterns come from the past. The world gives no notice when the past stops being a guide.
Fabricated material stated with confidence, known as hallucination, comes from the same mechanism: the most probable-sounding output is sometimes wrong. A new release can reduce that without closing it.
Tracing a fault
Conventional software can be traced: find the flawed logic and fix it. A language model’s behaviour is spread across billions of parameters, and the same prompt can give a different answer tomorrow. Nobody can step through it to learn why one answer went wrong.
That limit is real, and narrower than it sounds. Outputs can be tested against known answers, logged, tied to their sources and measured over time. Such testing estimates how often a system fails and where, even when it cannot say why one answer failed. It takes engineers, test data and a budget. Without it, the remaining method is to adjust the prompt and hope, and hope is not an engineering methodology.
The best case for the model
The strongest reply is that a bare model is the wrong thing to judge. Underlying models have become a commodity, the argument runs, and the value lies in what surrounds them. The main tool is retrieval-augmented generation, which makes a model answer from a chosen set of documents. A 2025 study in JMIR Cancer put 62 cancer questions to six chatbot configurations. With a curated cancer-information source, hallucination rates fell from about 40% to between 0% and 6%, though those chatbots answered fewer than half the questions.4“Reducing Hallucinations and Trade-Offs in Responses in Generative AI Chatbots for Cancer Information”, JMIR Cancer, 2025. The questions were in Japanese. The rest they declined, and a system that admits ignorance has something to recommend it.
Price is on its side too. Stanford’s 2025 AI Index reports that the cost of running a system at the level of GPT-3.5 fell more than 280-fold between November 2022 and October 2024.5Stanford HAI, The 2025 AI Index Report. For high-volume, standardised work such as drafting and summarising, the sums can now work.
What the fixes leave behind
Both points hold, and neither removes the problem. Retrieval makes a model’s output more likely to resemble a firm’s own documents. That narrows the range of wrong answers without ending them, and a wrong answer that quotes the firm’s own policy can be harder to spot than one that does not. Retrieval weights the coin in the firm’s favour, which is worth having. It is still a coin. And while the toss has become cheap, the checking is priced separately.
On the defenders’ own account, it works if the firm builds the right architecture, maintains the data, keeps audit trails, trains its overseers and measures the error rate. That is a programme of work, and it sits with the adopter. The vendor sells the transformation; the buyer builds it.
The safeguard that wears
The usual safeguard is human review, and review depends on a skill that may fade with disuse. A study published in The Lancet Gastroenterology & Hepatology in August 2025 examined 1,443 colonoscopies performed without AI at four Polish centres. After an AI polyp-detection tool arrived, the rate at which experienced endoscopists found precancerous growths in such procedures fell from 28.4% to 22.4%.6Budzyń et al., “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study”, 2025. A polyp-detection tool is a different thing from a language model, and the study was observational, so it proves nothing about compliance review. Any firm whose control is a person reading a machine’s work still needs a plan for how that person stays good at it. A review that stops at the quality of the prose checks nothing.
Reversing the default
The marketing implies that the only wrong AI strategy is not having one. None of this is a reason to ignore the technology. It is a reason to reverse the default, so that a model earns its place one decision at a time.
Start with the problem, not the technology. You may find it better served by a training programme or another spreadsheet. If a model does fit, treat it like any other business risk. Use it where a person reviews the work anyway, for first-pass triage or a draft with its sources attached. Keep the decision with a named person. Find out how often it is wrong and what an error costs, then decide whether you accept that cost. A measured error rate in a low-stakes content pipeline is a reasonable trade. An unmeasured one in a pricing decision is not.
Answers at GPT-3.5’s level have become 280 times cheaper. The most expensive technology investment is still the one that solves the wrong problem with great confidence.
Drafted with AI assistance and reviewed by Ilja Orlovs before publication.
Correction, 5 November 2026: this article was rewritten and shortened. Earlier versions attributed the tribunal’s words in the Air Canada case to the airline, and stated several figures and sources inaccurately.