Where AI Falls Short

The professional problem

Most knowledge workers meet language models as a box that answers questions. That interface is misleading. A search engine finds and ranks pages that already exist. A language model generates an answer from learned patterns and from the context you provide. The output can look like research, analysis or legal reasoning because it is fluent. Fluency is not a method. It is a surface.

The organisations that get burned treat AI as a junior analyst who speaks with the composure of a partner. A model that writes in complete sentences, cites sources and never hesitates will pass a first reading. Failure appears later: a number that cannot be reconciled, a case that does not exist, a policy clause that was never in the document, a client fact that leaked into a consumer tool. By then the work has already left the room.

This chapter is not an argument against using AI at work. It is an argument against using it as a library, a calculator, a records system or a witness. Know where the model is composing rather than retrieving, where confidence is a stylistic habit rather than accuracy, and where the organisation still has to supply grounding, verification and a named person accountable for what gets sent.

AI is not a search engine

Search and generation solve different problems. Search starts with an index of existing pages, documents or records. It matches a query, ranks the matches, and returns pointers to sources you can open. The engine does not author the answer. When search fails, it usually misses a relevant page or ranks a weak one too highly. You can still inspect the underlying document.

A language model does the opposite. It predicts a continuation that is statistically consistent with its training and with the prompt. It does not look an answer up unless you have connected it to retrieval tools. Left to itself, it composes a plausible paragraph. That paragraph may coincide with facts in the world. It may also coincide with a common error, a stale figure, a lookalike citation or a blend of two similar cases. The model has no native way to tell those apart. Plausibility is the objective it was trained for.

Retrieval-augmented generation, usually abbreviated RAG, sits between the two. The system retrieves trusted documents first, then asks the model to generate an answer grounded in those passages. Done well, RAG changes the job from composing something that sounds right to composing something supported by nominated evidence. Done poorly, it still looks like search to the user, while the model fills gaps, merges conflicting passages, or answers from memory when the retrieved text is thin.

Confidence is not accuracy

Language models are trained to produce helpful, complete-looking text. They are not trained to be silent when they do not know. The default behaviour is to continue. That continuation often carries the grammar of certainty: “the report shows,” “the court held,” “revenue grew 12 per cent,” “according to the 2022 study.” Those constructions are writing habits, not calibrated probabilities. A confident sentence can be false. A hedged sentence can be true. Tone is not evidence.

The practical failures cluster. Models generate incorrect facts: the wrong date, party, regulator or statutory threshold. They invent references and citations that look like the apparatus of scholarship or case law. They misinterpret data you pasted, treating a comment as a result or a forecast as an actual. They make arithmetic mistakes a spreadsheet would catch in a second. They misunderstand context, answering last quarter’s question with this year’s numbers, or applying a US rule to a UK entity. They produce reasoning that sounds sequential while smuggling in a non sequitur.

Fabricated or unsupported information presented as fact is a hallucination. Keep the definition tight. It is not merely an opinion you dislike, and it is not every rounding error. It is a claim the model presents as established when it has no warrant in the prompt, in retrieved sources, or in a tool result. Inventing a paper is a hallucination. Stating a number that does not appear in the attached workbook is a hallucination. Summarising a clause that is not in the contract is a hallucination. Guessing a missing assumption and writing it as if it were in the brief is a hallucination dressed as helpfulness.

A worked example: the invented citation

Citation fabrication is the failure mode that most clearly shows the gap between search and generation. The model has seen millions of references. It has learned the shape of a Harvard Business Review article, a Nature paper, a High Court judgment, an OECD working paper. When you ask for a source, it can emit that shape without retrieving a record. The result is often more persuasive than a real but messy citation, because the invented one is tidy.

Suppose a strategy analyst is writing a short briefing on AI in procurement and asks a general-purpose model for a canonical article.

Find a 2019 Harvard Business Review article on governing AI in
procurement. Give me a full citation and a three-sentence summary
I can drop into a board paper.

A plausible invented reply might read as follows. Chen, L. and Okonkwo, M. (2019) “Governing Algorithmic Procurement,” Harvard Business Review, 97(4), pp. 112–121. The article argues that category managers should retain exception rights when models score suppliers, that audit logs must capture the features used in each award, and that boards should review model drift annually. The prose is competent. The volume and page range look right. The authors’ names are ordinary. Nothing in the surface of the text tells you that the article does not exist.

Verification is mechanical. Search the HBR archive by title. Search by the alleged authors. Check a library index. Look for a DOI. If the article existed, those paths would converge on a page you can open. If they do not, you do not have a source. You have a well-formed fiction. The correct next step is not to ask the model whether it is sure. Discard the citation. Attach a real paper you have already read, or run a literature search in a tool that returns links.

Why lawyers and analysts get burned

Lawyers and analysts are exposed for structural reasons. Both are paid to produce text that other people will treat as reliable. Both work under time pressure. Both have house styles that reward complete-looking work: tables of authorities, footnotes, exhibits, sensitivity cases. A language model is unusually good at filling those templates. Reviewers who skim for form will often pass it.

In legal work the canonical injury is the fabricated authority. A brief that cites a case the court cannot find is a representation to a tribunal, not a drafting slip. Courts have sanctioned lawyers who filed model-generated citations they had not checked. The lesson is not that language models are forbidden. It is that the verification duty does not move. If your name is on the filing, you are representing that the authorities exist and say what you claim. Quote from the instrument, not from the model’s memory of instruments like it.

Analysts fail more quietly. The model misreads a table, swaps units, treats a forecast column as actuals, or invents a “peer median” that was never in the pack. Because the surrounding narrative is fluent, the error survives a first read and becomes a slide title the room remembers. Unlike a fake case, a fake number may never be litigated. It can still move capital, headcount or a go or no-go decision.

Why hallucinations happen

Hallucinations are not a mysterious personality trait of the model. They are the predictable result of how generation works when information is missing, instructions are vague, or the system is asked to sound finished. The causes overlap. A single bad answer often has more than one.

Missing information is the leading cause because the model will not, by default, stop. If the prompt asks for a market size, a legal test or a headcount figure that is not in the context, generation still proceeds. The gap is filled with a number or a doctrine that fits the genre. Vague instructions have a similar effect. “Write a report about our project” does not tell the model what it is allowed to know. It therefore borrows from generic project-management language. You experience that as invention. The model experiences it as being helpful.

No trusted grounding means there is no document, database or tool result that the answer must answer to. Conflicting sources make the problem worse: the model may average, pick the more fluent passage, or write as if the conflict did not exist. Statistical generation is the underlying mechanism rather than a separate bug. The model predicts plausible wording, not a verified fact. Hidden assumptions slip in when audience, jurisdiction, time period or definition is unspecified. Long or complex context increases the chance that a constraint on the first page is ignored by the last. Model limits complete the list: training cut-offs, no live access unless tools are attached, and finite attention over a long thread.

When RAG still hallucinates

RAG is the right architecture when answers must be grounded in a corpus you trust: policies, contracts, manuals, prior filings, a knowledge base. It is not a guarantee. Teams that deploy RAG and then relax verification are repeating the search-engine mistake with better branding. The model can still generate unsupported text. It simply does so in the presence of documents.

Retrieval can fetch the wrong chunks. A query about termination for convenience may retrieve a termination for cause clause because both passages contain “termination.” Incomplete chunks are common when documents are split for indexing: the definition sits on the previous page, the exception on the next, and the model answers from the middle. Conflicting sources in the index produce blended answers unless the prompt forces the model to surface the conflict. The model may also ignore retrieved context and answer from prior knowledge, especially if the retrieved text is thin or the corpus does not cover the question.

Gap-filling is the distinctive RAG hallucination. The retrieved passages cover most of the question. The model writes a complete answer, including the missing middle, in the same voice as the quoted material. Readers then believe the whole paragraph came from the knowledge base. The fix is procedural: require citations to chunk identifiers; require the model to say when the corpus is silent; keep a human on any answer that will be acted on. Stale indexes add a further risk. If the policy was updated last month and the index was not, RAG will ground you confidently in the old world.

Reducing hallucinations in three stages

You do not need a research lab to cut the rate of bad answers. Treat generation as one step in a work process. The habit has three stages: before you ask, in the prompt, and before you send. Each is cheap compared with a correction after the work has circulated.

Before you ask

Paste the actual document, policy, data extract or notes. Never rely on the model’s memory of “typical” policies or “usual” market sizes. Memory is where invention lives. If the file is too long, paste the relevant sections and say what you omitted. Tell the model to answer only from what you supplied and to say so if the answer is not there. Name the audience, the format and the depth. Also name the time period, the entity, the jurisdiction and the definition of any metric that could be ambiguous. Those are the hidden assumptions in the chart above.

In the prompt

Make reasoning inspectable. Ask the model to identify the source behind each claim, to state assumptions in a separate list, and to flag uncertainty rather than smooth it over. Ask for quotations with locations when the task is to interpret a document. Ask it not to invent missing information. These instructions do not make the model truthful in a philosophical sense. They change the shape of the output so that a reviewer can see where the ice is thin.

Answer only from the policy text I pasted. If the policy is silent,
write “Not in the supplied policy” and stop. Do not use outside knowledge.

For every requirement you state:
1. Quote the exact sentence.
2. Give the section heading.
3. State any assumption you needed to apply it to this case.

List residual uncertainties in a final bullet list. Do not bury them
in the narrative.

The same pattern works for numbers. Require the model to point at the cell, row label or sentence from which a figure comes, and to show arithmetic you can retype. If it cannot point, the figure is not ready to travel.

Before you send

Verify names, numbers, dates and quotations. Redo arithmetic in a calculator or spreadsheet. Open the cited document. Check that the quote exists and that it means what the surrounding sentence claims. Keep a named person accountable. “The AI wrote it” is not a control. It is a description of how the draft appeared. The organisation still needs someone who will stand behind the issued version. If a memo cannot be sent without a source pass, people will do the source pass. If a slide can be pasted into the pack from chat, they will not.

A verification checklist

Use this as a gate for any AI-assisted output that will leave your desk. Skip items only when the work is explicitly a private brainstorm and cannot be mistaken for a fact pack.

Verification is not re-prompting. Asking the same model to “check its work” can catch local inconsistencies. It cannot create a source that was never retrieved. Match the gate to the blast radius of being wrong.

Bias that arrives with the model

Training data is a record of what was published, digitised and retained, with all the skew that implies. Models inherit associations, defaults and blind spots from that record. They may describe some occupations, regions or family structures as typical and others as exceptional. They may propose examples that centre a US corporate context when your organisation is not one. They may be less accurate, less specific or less careful for groups that were under-represented in the data. None of this requires a malicious prompt. It is the default texture of a system trained on the public internet and a large share of English professional text.

In workplace use, inherited bias shows up as uneven quality rather than as an overt slur. A hiring-screen summary that rewards fluency in a particular dialect. A customer-complaint classifier that treats some writing styles as more “aggressive.” A market scan that under-weights firms that do not appear in English-language coverage. A medical or HR draft that uses population averages as if they applied to the person in front of you. Use the model to surface options and language. Keep humans, and better data, on anything that allocates opportunity, credit, discipline or care.

Privacy versus productivity

The productivity case is strongest when the prompt contains the work: the email thread, the contract, the board paper, the customer complaint. That is also when the privacy case is weakest. Consumer tools may store, log, review or reuse prompts. Even when a vendor says it does not train on your data, the text still leaves your environment. Client confidentiality, employee data and NDAs do not pause because the interface was convenient.

The tension is real. A sanitised prompt produces a generic draft. A complete prompt produces a useful draft and a potential leak. Organisations that pretend there is no tension get either no adoption or uncontrolled adoption. The workable settlement is classification before paste. Public and low-sensitivity material can go to a consumer tool if policy allows it. Client names, unpublished results, personal data, credentials, M&A materials and anything under NDA stay in an approved environment or do not go to a model at all. Productivity then comes from using AI on the work you are allowed to use it on, not from feeding it everything you have.

Data leakage is not only a vendor-training story. It is also a colleague story. People paste output into shared decks without noticing that the prompt contained a salary band, a patient detail or a supplier’s unpublished price. The model is a copier with initiative. Whatever you give it can reappear, slightly rewritten, in a form that looks original enough to share. Treat prompts as disclosures.

Ownership, copyright and attribution

Generated text and images arrive without a reliable chain of title. You usually do not know which training fragments shaped a sentence. You do not get a bibliography unless you demand sources and then verify them. Every factual claim still needs checking, because the model does not attach provenance.

Ownership of the output is a contract question layered on an unsettled legal one. Many commercial terms assign or licence the output to the customer, which is useful and is not the end of the analysis. If the output reproduces a substantial part of a copyrighted work, a vendor licence does not wash the underlying right. If the output is purely generated with minimal human arrangement, some jurisdictions are reluctant to treat it as a copyright work at all. Client deliverables still need human authorship, selection and records of what was used.

In professional settings the practical rules are conservative. Do not treat generated logos or long-form copy as cleared for brand use without a review. Do not paste a third party’s proprietary document into a model to “rewrite it in our voice”; that can be both a leak and a copyright problem. Do not cite the model as a source of fact. Keep the human contribution visible: the brief, the attached sources, the edits, the verification, and the judgement about what to issue.

Build verification into the workflow

Individual virtue does not scale. If catching errors depends on everyone remembering this chapter on a busy Thursday, some share of errors will ship. Put the checks where the work already passes: attached sources for policy questions, a “not in the document” path, a named reviewer for client-facing text, numbers that originate in a workbook, and an approved tool for anything above a stated data class. Supervisors should inspect that process, not eloquence. A fluent memo is not evidence that the checklist ran.

None of this requires hostility to the tools. It requires the same stance you already take to a fast junior colleague: useful, unsupervised only on work you can afford to redo, never the signatory. The next chapter turns to prompting. Once you know where the model will invent, you can brief it so that invention is constrained, visible and cheaper to catch. Prompting does not replace verification. It determines how much verification you will need.

Key takeaways

• A language model composes a plausible answer. It does not look one up unless you connect it to sources.
• Search finds and ranks existing pages. RAG retrieves nominated documents and then generates.
• Confidence is a writing style. Accuracy is a property of claims you can check.
• Hallucination is fabricated or unsupported information presented as fact. Invented citations are the clearest example.
• RAG still hallucinates when retrieval is wrong, chunks are incomplete, sources conflict, or the model fills gaps.
• Reduce risk in three stages: ground the question, make uncertainty visible in the prompt, verify before you send.
• Keep a named human accountable. Redo arithmetic. Open every citation.
• Inherited bias, data leakage and unclear ownership are parallel limitations. Consumer tools do not override NDAs.
• Risk is managed in the workflow. It is not eliminated by a better model or a sterner prompt.