Skip to content
AI Tools

What a document AI system actually costs to run

Real monthly figures for retrieval systems over your own documents, what drives the cost, and the two mistakes that inflate it most.

4 min read
What a document AI system actually costs to run. An article by Athabasca Solutions.

“Retrieval-augmented generation” is a mouthful for something simple: instead of hoping the AI knows the answer, you find the relevant part of your own documents first and hand it over with the question.

It is the most useful AI pattern for most businesses, and the one people are most nervous about costing. Here are real numbers.

Where the money goes

Three components, and only one of them is usually significant.

Storing and searching your documents. A vector database holding a few thousand documents costs somewhere between nothing and about $70 a month depending on the provider. Several have free tiers that comfortably fit a small business.

Turning documents into something searchable. A one-time cost when you load your documents, plus a small ongoing cost as new ones arrive. For a few thousand documents this is usually single-digit dollars, once.

Answering the questions. This is where nearly all the ongoing cost sits, and it scales with how much you use it.

A concrete example

A support team of five, each asking the assistant roughly twenty questions a day. Each question retrieves a few relevant passages and generates a paragraph or two in reply.

That is about 2,000 questions a month. Depending on model choice and how much context each question carries, expect somewhere in the range of $20 to $150 a month in model costs, plus the storage above.

For a system replacing “interrupt a colleague to ask where something is”, that is not a difficult number to justify.

Monthly model cost as usage grows

The band is the same $20 to $150 spread from the example above, scaled by how many questions get asked.

  • Model calls

Dollars per month

$0 $200 $400 $600 500 a month 1,000 2,000 4,000 8,000 a month

Scaled linearly from the worked example: 2,000 questions a month at $20 to $150. Real pricing is per token rather than per question, so treat this as the shape of the curve rather than a quote. Vector storage sits on top of every point and does not grow with question volume.

View as a table
Dollars per month Model calls, low Model calls, high
500 a month $5 $38
1,000 $10 $75
2,000 $20 $150
4,000 $40 $300
8,000 a month $80 $600

The width of that band is the point. Two businesses asking the same number of questions can differ sevenfold in bill, and the difference is entirely in the two mistakes below.

The two mistakes that inflate it

Stuffing whole documents into every request. The entire point of retrieval is sending the relevant paragraphs rather than the whole manual. Systems that skip the retrieval step and paste everything can cost ten times more for worse answers, because the relevant fact gets lost in the noise.

Using the largest model for everything. Model tiers differ by roughly five times in price. Plenty of tasks, especially “summarise these three passages”, run perfectly well on a smaller, faster model. Reserve the expensive one for questions that genuinely need it.

Between them, these two account for most of the horror stories about AI running costs.

What makes it cheaper

Caching the stable part. If every request carries the same instructions, those can be cached so you pay a fraction for them after the first call. On a system with a large fixed prompt this is most of the saving available.

Answering only what needs answering. A surprising share of questions have a fixed answer. Those can be served from a lookup, at no model cost at all.

Rate limits. Not as a cost control so much as an abuse control, but worth having on anything public.

What it does not cost

There is no per-seat licence, no minimum commitment, and no platform fee unless someone sells you one. If a quote includes a large fixed monthly platform charge on top of usage, ask precisely what it buys.

The build is a real cost and the running is usually modest. Be suspicious of the inverse.

Before you build

Write down the number you expect to move: hours saved per week, tickets deflected, days off the processing time. If you cannot name one, the project has no definition of success and will be judged on whether it felt impressive, which is a bad way to spend money.

And check whether the problem needs AI at all. Our test for that rules out more projects than it approves, deliberately.

If you want a straight estimate for your situation, describe what your team keeps asking.

Further reading

Sections covered in What a document AI system actually costs to run: Where the money goes, A concrete example, The two mistakes that inflate it, What makes it cheaper, What it does not cost, Before you build
The shape of the argument, in order.

Get new articles by email

One email when something new goes up, roughly twice a month. Plain writing on what software costs and what is worth building. No sequences, no sales calls, and one click to leave.

We use it for the newsletter and nothing else. Unsubscribe any time.

Have something you need built?

Tell us what the problem is. You will get an honest read on whether it is worth building, what it would take, and roughly what it would cost. No pitch deck, no pressure.

Replies within one business day. Mon to Fri, 9am to 5pm MT.

Call Start a project