Build answers you can check
How to build an AI knowledge base that can show where its answers came from
An AI knowledge base connects a model to information it can retrieve when answering a question. A useful answer also needs a source you can inspect and a rule for when the information is insufficient. This tutorial gives you a small downloadable example that you can run locally, including tests for old, missing and contradictory records.
Keep permission and evidence checks in software, carry source information into the answer, and test the questions the system should decline.
Build a small answer with a checkable source
The sample answers one structured question: how often does a particular client receive a report? It holds invented records for two clients, an older agreement and a pair of contradictory budget decisions. It uses exact topic lookup, which keeps the example small enough to inspect.
A larger implementation may search documents by meaning, connect live sources and answer more varied questions. Those additions require their own retrieval tests. This sample demonstrates the evidence boundary; it does not establish the quality of a document search engine.
Application
Identify the client and topic
Use the supplied example identity; a real service must obtain it from an authenticated session.
Software
Select authorised, current records
Check client, team, validity dates, collection date and complete source information.
Software
Check agreement between sources
Return unknown when the available current facts disagree.
Software or optional model
Propose and validate an answer
Accept only the selected structured value with a matching record ID and exact quotation.
The final answer includes origin and dates from the records. The model cannot supply replacement metadata.
Run the example before changing it
Unzip the download into a folder and open a terminal there. Check your Node.js version, run the tests and request the default answer:
node --version node --test demo.test.mjs node demo.mjs
The default answer is “monthly”, with the quotation “The reporting frequency is monthly.” Its source origin is sample://client-a/agreement-v2. The validity start is 2026-09-01, the validity end is 2026-12-31 and the collection date is 2026-09-02. These are separate fields.
The sample deliberately uses the fixed query date 2026-10-01. Changing the computer’s date does not change the result. You can pass a different date as the last argument when testing expiry.
node demo.mjs client-a budget-approved account-a 2026-10-01 node demo.mjs client-b reporting-frequency account-a 2026-10-01 node demo.mjs client-a reporting-frequency account-a 2027-01-01
These three requests return unknown for different reasons: contradictory evidence, no authorised evidence for that client identity, and expired information. Inspect sources.json and the tests to see which records create each condition.
Connect a local model if you already use one
The default answer is deterministic. The download also includes an optional adapter for an already installed Ollama model. With Ollama running locally, replace the placeholder with the name of your installed model:
OLLAMA_MODEL=YOUR_INSTALLED_MODEL node demo.mjs
The adapter uses Ollama’s current chat API. It sends the selected synthetic records to the local endpoint and requests JSON. It does not download a model or require a paid API key.
The same software validates the model’s output. A changed quotation, invented source ID or different answer produces unknown. Because the fact is structured, the answer must exactly match its selected value. That strict check does not extend to arbitrary prose.
Adapt the records before connecting real documents
Said describes our current build as carrying provenance through every trace: source origin, when information was valid and when it was recorded. Confidence falls if a record is not seen again by its validity date. When the system cannot provide reviewable evidence, it must say it does not know.
The sample makes a stricter teaching choice: expired records are withheld completely. It does not calculate a confidence probability. Keep that difference in mind when adapting it.
For real use, obtain identity and access from trusted software rather than prompt text. Preserve permission checks through retrieval and output. Add secure storage, ingestion, review ownership and an audit trail. Test misleading or malicious source content as well as ordinary questions.
A matching quotation establishes traceability. Someone still needs to check whether the source is authoritative and whether it answers the question. For open-ended answers, test claim-by-claim support rather than accepting any answer containing a real citation.
Keep a record of the question and its evidence
Worked example
The downloadable reporting question
Invented teaching example. No client outcome is claimed.
SOURCES
Two invented reporting agreements for client A, with separate validity periods.
SELECTION
The query identifies client A, reporting-frequency, account-a and 2026-10-01.
Only the current authorised agreement is selected.
ANSWER
monthly; source report-a-current; exact quotation and original dates included.
NEGATIVE TESTS
An account-a request for client B returns unknown.
The two current budget decisions disagree and require a person.
A request after all reporting records expire returns unknown.
Further reading
Written by Said Lopez, founder of LeverageAI and author of Lean AI. Our guides draw on the method in the book and our work with clients.