Getting AI to answer from your own documents

The single most useful thing a small organisation can do with AI is make its own documents answerable. Policies, manuals, past quotes, meeting notes, the procedure somebody wrote in 2021 that three people still reference from memory.

The technique has a name, retrieval augmented generation, and the name is unhelpful. Here is what it actually is, and where it actually goes wrong.

How it works, in four steps

1. Your documents get split into pieces. A page at a time, roughly, because whole documents are too large to hand a model at once.
2. Each piece is converted into a list of numbers that represents its meaning, and stored. Two pieces about the same subject end up with similar numbers even when they share no words.
3. A question gets converted the same way, and the closest pieces are retrieved.
4. The model is given the question and those pieces and told to answer using only them, with references.

That is the whole idea. The model is not learning your business or being trained on it. It is being handed the relevant page at the moment of the question, exactly as you would hand a new employee the right binder.

Why the answers are usually bad at first

When one of these systems gives a poor answer, people assume the model is not good enough. It almost never is. The failure is at step three: the right piece was never retrieved, so the model was answering without it.

The common causes, in order:

The documents contradict each other. Three versions of the same policy, two of them obsolete, none marked as such. Retrieval returns all three, the model picks one, and it is a coin flip whether the answer is current.

The answer is in a table or a scan. Text extraction from PDFs is far worse than people expect, especially for tables and anything scanned. If the extracted text is garbage, everything downstream is garbage.

Documents were split mid-idea. A procedure whose steps land in two different pieces retrieves as half a procedure, and a half procedure confidently delivered is worse than nothing.

Context is missing from the piece itself. A paragraph saying “this does not apply to seasonal staff” is meaningless without knowing what “this” is. Pieces need their document title and section carried with them.

Note that four of the four are filing problems, not AI problems. The quality of a document AI system is set almost entirely by the quality of the documents going into it.

What to fix first

Before building anything:

• Establish which version is current for anything with more than one. This single step improves answers more than any amount of tuning.
• Delete or clearly archive the obsolete ones. Not “keep them just in case in the same folder,” which is how they get retrieved.
• Check what your PDFs actually extract to. Open a few, copy the text out, read it. If tables come out scrambled, they need converting properly or those answers will always be wrong.
• Give every document a title and a date that a human would recognise. Retrieval uses these, and so does the person checking a citation.

Doing this well is unglamorous and it is most of the project. It also has value even if you never build the AI system, which is a rare property.

The design decisions that matter

Always cite. Every answer should link to the source pieces it used. This is what makes the system trustworthy and, more practically, what makes bad answers diagnosable instead of mysterious.

Refuse when nothing relevant was found. A system that says “I could not find this in your documents” is more useful than one that always produces something. Getting this behaviour reliably takes deliberate work, because the default inclination is to answer anyway.

Respect permissions at retrieval. If some documents are restricted, the filter has to happen before retrieval, not after. Do not build a system where the model sees a confidential document and is instructed not to mention it.

Log the questions. The list of what people ask, especially what they ask and do not get answers to, is worth more than the system. It tells you which documents to write next.

What it costs

Less than most people expect to run, and more than most expect to set up. The recurring cost is dominated by re-processing documents when they change and by storage, not by the questions themselves. The arithmetic, with real numbers, is in what a document AI system actually costs to run.

The setup cost is proportional to how tidy your documents already are, which is why the honest first estimate always starts with looking at them.

Where to start

One document set, one team, read-only, with citations. A staff handbook or a procedures folder is ideal: bounded, frequently asked about, and low risk if an answer is imperfect because a human can check the cited page in seconds.

Get that working and used before extending it to anything customer facing. The step from internal to external is a much bigger step than it looks, because it changes who pays for a wrong answer.

If you want to know whether your documents are in good enough shape for this, tell us what they are and where they live. That assessment is usually a short conversation.

Related: where AI projects go wrong, what MCP actually is, and our AI tools work.

Further reading

• The original retrieval augmented generation paper
• Anthropic on contextual retrieval