An AI company-knowledge system should answer from the right documents, for the right user, in a way that can be checked. Adding search to a folder is insufficient. Source selection, content ownership, permissions and evaluation against everyday questions all matter.
Retrieval-augmented generation, or RAG, combines retrieval of relevant material with answer generation. It does not automatically resolve contradictory policies, old versions or unclear permissions. Our AI Knowledge Systems service addresses these prerequisites before selecting models and tools.
Define the questions it should serve
Collect questions from the intended users. A salesperson looking for approved specifications has different needs from an employee checking an internal procedure. For each question, record what a useful answer looks like and which document is authoritative.
Include questions the system should not answer: information outside the organisation, another team's restricted details and unapproved policies. The system needs to recognise its boundaries. Success does not mean generating prose for every input.
Clean sources against explicit criteria
Include documents that have a clear purpose. Identify duplicates, obsolete versions, drafts and scanned files with unreadable text. A file import can succeed technically even when its contents have not been extracted correctly.
When two sources conflict, a content owner must decide which applies. Do not expect a model to invent an organisational hierarchy. Add title, version, owner, effective date and status. Metadata have operational meaning beyond search performance.
Preserve meaning when splitting documents
Large documents are often divided into retrieval chunks. Each needs enough context to keep a condition attached to its limitations. A paragraph saying something “is allowed” may be misleading without the preceding conditions.
Inspect tables, footnotes and headings. In product instructions, the measurement unit or specific model may appear only in the heading. Evaluate chunking settings against real questions instead of treating one size as universally ideal.
Apply permissions before answering
Retrieval should be restricted to material the current user is permitted to see. Retrieving everything and then asking the model to conceal restricted material is insufficient. The same principle applies to previews and titles that may reveal sensitive information.
Test the same question with two users holding different permissions. Also test role changes, access revocation and shared caches. Rules from the original document system must transfer clearly into the new application rather than disappear during copying.
Require checkable sources
Answers should identify the relevant document and let users inspect the supporting passage. Links need to work for that user. A reference to an entire folder does little to support a particular claim.
Distinguish what the source supports from what it leaves unanswered. If a policy describes an application process without specifying approval time, the answer should not invent a typical deadline. Prompt design defines this behaviour alongside the presentation of sources and gaps.
Evaluate retrieval and generation separately
When an answer is wrong, first inspect whether the right material was retrieved. If the authoritative source is absent, changing the prompt's tone will not solve the problem. If retrieval is correct but the answer omits a restriction, examine generation and context.
Test synonyms, abbreviations, follow-up questions and known version conflicts. Record expected sources and required answer points. Repeat the set after significant updates and add failures observed during use.
Organise updates and expiry
The knowledge base needs ownership and an update cadence suited to each source type. Frequently changing procedures differ from historical records. Record when a new version was received and when its availability in search was verified.
Removing a source should also affect its derived data. Define withdrawal, reimport and investigation processes for outdated answers. AI Automation can connect source updates to checks while retaining human responsibility for approving content.
Frequently asked questions
Should we upload every file?
No. A first version benefits from a limited, reliable collection covering specific questions. Expand through a controlled process.
Does RAG eliminate incorrect answers?
No. It can ground answers in company sources, but evaluation, permissions and missing-information handling remain necessary.
What happens when policies conflict?
The source owner determines which applies. The system should flag an unresolved conflict instead of choosing arbitrarily.
Can it connect to an agent?
Yes, where useful. Access to knowledge and permission to perform actions are separate concerns and need separate design.
Further reading and next steps
Anthropic's framework for effective agent applications provides technical context for systems combining retrieval and tools. Describe your sources and their intended users without sending confidential documents in the initial enquiry.