What a document AI system actually costs to run
Real monthly figures for retrieval systems over your own documents, what drives the cost, and the two mistakes that inflate it most.
“Retrieval-augmented generation” is a mouthful for something simple: instead of hoping the AI knows the answer, you find the relevant part of your own documents first and hand it over with the question.
It is the most useful AI pattern for most businesses, and the one people are most nervous about costing. Here are real numbers.
Where the money goes
Three components, and only one of them is usually significant.
Storing and searching your documents. A vector database holding a few thousand documents costs somewhere between nothing and about $70 a month depending on the provider. Several have free tiers that comfortably fit a small business.
Turning documents into something searchable. A one-time cost when you load your documents, plus a small ongoing cost as new ones arrive. For a few thousand documents this is usually single-digit dollars, once.
Answering the questions. This is where nearly all the ongoing cost sits, and it scales with how much you use it.
A concrete example
A support team of five, each asking the assistant roughly twenty questions a day. Each question retrieves a few relevant passages and generates a paragraph or two in reply.
That is about 2,000 questions a month. Depending on model choice and how much context each question carries, expect somewhere in the range of $20 to $150 a month in model costs, plus the storage above.
For a system replacing “interrupt a colleague to ask where something is”, that is not a difficult number to justify.
Monthly model cost as usage grows
The band is the same $20 to $150 spread from the example above, scaled by how many questions get asked.
- Model calls
Dollars per month
Scaled linearly from the worked example: 2,000 questions a month at $20 to $150. Real pricing is per token rather than per question, so treat this as the shape of the curve rather than a quote. Vector storage sits on top of every point and does not grow with question volume.
View as a table
| Dollars per month | Model calls, low | Model calls, high |
|---|---|---|
| 500 a month | $5 | $38 |
| 1,000 | $10 | $75 |
| 2,000 | $20 | $150 |
| 4,000 | $40 | $300 |
| 8,000 a month | $80 | $600 |
The width of that band is the point. Two businesses asking the same number of questions can differ sevenfold in bill, and the difference is entirely in the two mistakes below.
The two mistakes that inflate it
Stuffing whole documents into every request. The entire point of retrieval is sending the relevant paragraphs rather than the whole manual. Systems that skip the retrieval step and paste everything can cost ten times more for worse answers, because the relevant fact gets lost in the noise.
Using the largest model for everything. Model tiers differ by roughly five times in price. Plenty of tasks, especially “summarise these three passages”, run perfectly well on a smaller, faster model. Reserve the expensive one for questions that genuinely need it.
Between them, these two account for most of the horror stories about AI running costs.
What makes it cheaper
Caching the stable part. If every request carries the same instructions, those can be cached so you pay a fraction for them after the first call. On a system with a large fixed prompt this is most of the saving available.
Answering only what needs answering. A surprising share of questions have a fixed answer. Those can be served from a lookup, at no model cost at all.
Rate limits. Not as a cost control so much as an abuse control, but worth having on anything public.
What it does not cost
There is no per-seat licence, no minimum commitment, and no platform fee unless someone sells you one. If a quote includes a large fixed monthly platform charge on top of usage, ask precisely what it buys.
The build is a real cost and the running is usually modest. Be suspicious of the inverse.
Before you build
Write down the number you expect to move: hours saved per week, tickets deflected, days off the processing time. If you cannot name one, the project has no definition of success and will be judged on whether it felt impressive, which is a bad way to spend money.
And check whether the problem needs AI at all. Our test for that rules out more projects than it approves, deliberately.
If you want a straight estimate for your situation, describe what your team keeps asking.
Further reading
Topics in this article
Get new articles by email
One email when something new goes up, roughly twice a month. Plain writing on what software costs and what is worth building. No sequences, no sales calls, and one click to leave.
Keep reading on the same subject
-
Getting AI to answer from your own documents
How document AI actually works, why answer quality is a filing problem rather than a model problem, and what to fix before you build anything.
See how retrieval works -
When does your business actually need AI?
A practical test for whether an AI feature is worth building, the four cases where it genuinely pays, and the ones where it quietly wastes money.
Find out if you do -
Where AI projects go wrong in small businesses
The five failure patterns behind most abandoned AI pilots, and the cheap checks that catch each one before you have spent a budget finding out.
Avoid the five traps