AI agents and language models

How to build an AI knowledge base so the agent answers from your documents, not from a guess

An AI agent is only as good as the source it answers from. This guide is entirely about the source: what goes into a knowledge base, how to write a document so it can be cited down to the section, how to settle two conflicting versions of the same file, and who makes sure that a quarter later the base still tells the truth.

9 minutes read

5 to 10 secondshow long BOND, our own company AI agent, takes to answer from company data and name the source
300 to 800 tokensthe size of one chunk we cut a document into, roughly one to three paragraphs
1 question per sectionthe rule for writing a document for the base: one heading, one answer, version and date in a fixed place
30 and 90 dayswhen we repeat the control set of questions, because a document added after go-live changes the answers

Short answer

How do you build an AI knowledge base step by step?

Start from questions, not from documents. Write down the 30 questions your team asks most often and mark, next to each one, where the answer lives today: in a file on a drive, in the CRM, in one person's inbox, or in nobody's head at all. A knowledge base is that set of sources, not a copy of the company drive.

Then get the sources ready for a machine to read: one topic per document, a heading phrased as a question, one answer under the heading, version and effective date in a fixed place. The agent must state which document and which section the answer came from. Without that, nobody can tell whether the answer came from your company or from the model's general knowledge.

Finally, give every document an owner and a review date. A knowledge base does not break on day one. It breaks in month three, when the March procedure sits next to the June version and nobody marked which one applies.

What goes into an AI agent's knowledge base?

A knowledge base is not a folder of files. It is a list of sources the agent is allowed to answer from, each one with an owner and a date. Some of them are documents. Some are systems the agent queries only at the moment somebody asks.

The split follows how fast the data moves. Content that stands still for weeks (a procedure, terms of sale, a product description, commercial conditions) goes in as a document. A number that changes during the day (stock level, account balance, deal stage in the CRM) is fetched from the system, because a copy of it is out of date the moment it is saved.

How an agent reaches into a live system has its own guide: the MCP standard describes how tools get connected. What matters here is the other half. BOND, our own company AI agent, answers from company data in 5 to 10 seconds and names the source, but somebody has to put that source in order first.

Source	What you can get out of it	The trap
Documents on a drive (Google Workspace, Microsoft 365)	procedures, terms, quotes, product descriptions, instructions	several versions of the same file sitting side by side, and scans with no text layer that the model cannot read
SQL database and warehouse (BigQuery, the shop database)	today's numbers: sales, stock levels, balances, due dates	column names only the author understands; with no table descriptions the agent counts something other than what you asked about
CRM (HubSpot, Pipedrive, Salesforce)	contact history, deal stage, what was agreed with the customer, who owns the relationship	notes written in shorthand and fields every rep fills in their own way
Email and chat (Microsoft 365, Slack)	decisions nobody ever moved into a document	private threads and personal data, plus decisions reversed in the next message
Ticket history (Zendesk, Intercom, the support inbox)	the real questions customers ask and the answers that worked	wrong answers, or answers quoting an old price list, repeated by the agent forever after
ERP and invoicing (SAP Business One, Microsoft Dynamics, Xero)	sales documents, payment terms, conditions agreed per customer	billing data visible to the whole company unless you set the access scope

Order the connections by how many questions from your list of thirty each source will answer. Usually the first two close more of that list than the other four together, and every extra source is a separate piece of work on permissions.

How do you prepare a document so the agent can cite it?

The model reads text, not layout. A document built from boxes, with a table split across columns and a scanned signature at the bottom, is worse for it than a plain text file with headings. What works: one topic per document, a heading phrased as a question, one answer under that heading.

The difference shows up at the moment of citation. Without a prepared source the answer reads: payment terms are 14 days. With one: payment terms are 14 days, Terms of Sale, version 3, section “Payments”, in force since 1 March 2026. The first has to be verified from scratch. The second takes fifteen seconds, so somebody actually does it.

Forcing a citation also changes what the agent will not write. When the instruction says answer only from the passages provided and name the document, a question with no cover in the base ends in a refusal instead of a smooth sentence out of general knowledge. We cover that separately in the guide on AI hallucinations.

One topic per document, a heading phrased as a question, one answer under the heading
Version, effective date and owner in a fixed place in the file, at the top or at the bottom
Abbreviations and internal names spelled out on first use, because the person asking will use a different word than the author
Full text in the document instead of “see attachment”: the agent will not open the attachment for you
A scan with no text layer put through text recognition or retyped, otherwise it is invisible

Why split documents into chunks?

The agent does not send the model your whole drive. For each question it picks a handful of the best matching passages, and the answer is written from those. That is why a document gets cut into chunks, and how you cut decides whether the agent lands on the right section or returns a generality from page one.

In our own deployments a chunk runs from 300 to 800 tokens, roughly one to three paragraphs, and carries the document name and the section heading with it. Without the heading, the chunk “the term is 14 days” does not say which term or which contract.

A chunk of 300 to 800 tokens: shorter loses the context, longer dilutes the question
Document name and section heading attached to every chunk, so the answer can be cited
Cut on headings, not every so many characters: a procedure cut mid step gives half an answer
A table stays whole or goes into the database as data; a row without its column headers is useless

Search by meaning or search by keyword?

These are two different mechanisms and each one fails on its own. Keyword search looks for an exact string: it lands on an invoice number and finds nothing when the person asks about time off at short notice while the procedure calls it unplanned leave. Meaning-based search does the opposite: it handles the paraphrase and loses the number.

So in practice you run both and merge the results. It costs one extra query against the database and removes the most irritating class of error: an agent that cannot find a document you know exists.

Question from the team	Search by meaning	Search by keyword
How many days do we have to handle a complaint?	lands on the section about returns even when the word complaint is not in it	finds only documents containing that one word
What are the terms on invoice INV/2026/0912?	returns random invoices with similar wording	lands on the exact document number
How do I claim a business trip?	finds the procedure called “Domestic travel expenses”	finds nothing, because the wording does not overlap
Who owns the Smith & Co Ltd account?	confuses companies with similar sounding names	lands on the exact name as written in the CRM
Can we give a 15 percent discount?	finds the discount policy even though the question is about a number	lands only if the policy states that exact figure

Draw the conclusion from your own list of questions. Questions with proper names, document numbers and product codes need keyword search; questions about how something is done need meaning-based search. Most lists carry both kinds, so both mechanisms stay switched on.

Which version of the document is in force?

This is mistake number one in every base built by pointing an agent at the company drive. Three price lists sit there: “price list”, “price list new” and “price list new FINAL 2”. The agent finds all three and picks the one that matches the question better, because nothing in the content says which one counts.

The fix is not technical, it is organisational. Somebody has to say which version applies, and the rest have to drop out of the base. A backup copy is welcome to sit on the drive. It has no business being in the agent's knowledge base.

Effective date inside the document, not in the file name: anyone who clicks copy changes the name
One document in force per topic, the rest outside the base or in an archive the agent cannot reach
An expired document marked as expired, so the agent can say the terms no longer apply
A contradiction escalated to a person: the agent shows both versions and asks instead of choosing for you
A new version enters the base on the day it takes effect, not a quarter later

Who should see which document?

A knowledge base inherits a problem the company already had: files on the shared drive that half the team should not be opening. The agent exposes it within a week, because it answers anyone who asks, and it does so faster than anyone could ever browse the folder by hand.

Scope is set on the sources and on the person asking, not in the instruction given to the model. The sentence “do not answer questions about salaries” is a suggestion, and a differently worded question gets around it. No access to the HR folder is a fact. The GDPR calls this minimisation: you process only the data necessary for the purpose (art. 5(1)(c) of Regulation 2016/679).

The agent sees what the person asking sees, not what the integration's technical account sees
HR, payroll and employee matters in a separate scope, with their own list of people and their own log
Customer personal data only where it is needed for the answer, the rest stripped before it enters the base
A log of questions and answers, together with which document was shown and to whom
A refusal on permissions sounds like a refusal, not like missing data: the person asking should know who to go to

Who updates the knowledge base, and how often?

A knowledge base is not a project with an end date, it is a register with owners. If after three months nobody has added a single document and nobody has withdrawn anything, that means the team stopped using it, not that the company stood still.

Document register
One table: document name, owner, date of last review, date of the next one. A document with no owner does not enter the base, because there is nobody to ask on the day it stops being true.
Review at the pace of change
Price lists and commercial terms monthly, procedures quarterly, terms and policies whenever the underlying rules change. The date sits in the owner's calendar, not in a note from the workshop.
Gaps taken from real questions
Questions where the agent found no document land on one list. Once a week somebody decides: write a new document, fix an existing one, or leave the topic out of scope.
Control at 30 and 90 days
The same set of questions as before go-live, asked again against the same sources. An answer that changed for no reason means a document nobody approved got into the base.

Common mistakes

Six mistakes that make a knowledge base answer wrongly

Connecting the entire company drive at once
The agent cites copies, working drafts and meeting notes from two years ago. Answers contradict each other, and after a week the team goes back to asking the colleague at the next desk.
Documents kept as scans and slide decks
A scan with no text layer is a blank page to the agent. A slide with three bullet points produces an answer made of three bullet points, and the person asking has to open the file anyway.
No version and no effective date in the content
The agent has nothing to decide which price list counts, so it picks the one that matches the question better. The answer is correctly cited and untrue.
An answer with no document and no section named
Checking an answer like that takes a quarter of an hour, so nobody checks it. After the first mistake the team stops trusting the whole base, including the part that was fine.
A knowledge base with no document owners
After a quarter part of the content is out of date and responsibility dissolves between IT and whoever wrote the files. Nobody knows who has the right to withdraw a procedure.
Permissions set through an instruction to the model
A differently worded question is enough to make the agent show a document from the HR folder. The limit only holds when it sits on access to the source.

Checklist

Checklist before you connect a knowledge base to an AI agent

If any line is a no, the agent will answer, but the answer will not stand up to scrutiny.

A list of 30 real questions from the team, each marked with where the answer lives today
Every document in the base has an owner, a version and an effective date in its content
Old versions pushed out of the base, one document in force per topic
Scans put through text recognition or retyped by hand
Chunks carry the document name and the section heading, and the agent quotes them in the answer
Keyword search switched on alongside meaning-based search
The agent's scope equals the scope of the person asking, HR and payroll kept separate
The review date written into the owner's calendar, not into a note from the workshop

Sources

Let us start with your documents

A knowledge base is built on real questions, not on a list of folders

On a free 30-minute consultation we take the ten questions your team asks most often and check where the answer lives today. The process workshop costs EUR 3,500 net per week and ends with a list of sources, owners and order in your documents, which stays with you whether or not we build anything. We run at most two implementations at a time, so we say plainly when there is room.

Book a free consultation

Frequently asked questions

What is an AI knowledge base?

It is the list of sources an AI agent is allowed to answer from: documents, procedures, quotes, ticket history, and systems queried live such as a CRM or a data warehouse. Every source has an owner and a date. It is not the same as a folder on a drive, because only the version in force gets in.

What format should documents be in for a knowledge base?

Plain text with structure works best: a heading phrased as a question, one answer under the heading, version and date in a fixed place. The model reads content, not layout, so boxes, columns and embedded images get in the way. A scan with no text layer has to be recognised or retyped first.

Why is a document split into chunks?

The agent does not hand the model a whole file, only the handful of passages that best match the question. So the document is cut on headings into chunks of 300 to 800 tokens, each carrying the document name and the section heading. That way the answer can be cited down to the section rather than to the file.

What is the difference between meaning-based search and keyword search?

Meaning-based search matches the sense of the question to the sense of the passage, so a question about claiming a trip finds a procedure called Domestic travel expenses. Keyword search looks for an exact string and is the only one that lands on an invoice number or a product code. In practice you run both and merge the results.

What do you do when the base holds two conflicting versions of a document?

A person decides, not the agent. One version stays as the one in force, with a date and an owner, and the rest leave the base for the archive. If the contradiction has to stay, because two markets run on different terms for example, the agent shows both and asks which case you mean.

Should every employee see the whole knowledge base?

No. Scope is set at the level of the sources and the person asking, not through an instruction to the model, because a differently worded question gets around an instruction. HR, payroll and personal data go into a separate scope with their own list of people, in line with the minimisation rule in art. 5 of Regulation 2016/679.

How often should a knowledge base be updated?

At the pace the data changes: price lists and commercial terms monthly, procedures quarterly, policies whenever the underlying rules change. Put the review date in the document owner's calendar. At 30 and 90 days after go-live, repeat the control set of questions and check which answers moved.

Do you need a separate vector database for a knowledge base?

Not always. With a few hundred documents, a vector extension in the database you already run, or the mechanism built into the agent platform, is enough. A separate tool earns its place on large collections and heavy traffic. Choosing the tool is the last decision, not the first: order in the sources comes before it.