Build an Internal AI Assistant: The Hard Parts

An internal AI assistant can help employees find trusted company information and complete routine tasks without searching across scattered documents. The language model is only one component: success depends on useful workflows, reliable data, access controls, evaluation, and ongoing ownership. Before starting, choose a narrow employee problem and involve the people responsible for the relevant systems and information.

What You’ll Need

• A specific business use case, a process owner, and a group of representative employees for testing.
• Approved access to the source material, such as policies, support articles, or team documents, plus permission to use it for this purpose.
• An approved model and hosting approach, with documented data-handling, retention, and access requirements.
• An application environment for the interface and backend, identity-provider integration, logging, and a way to collect feedback.
• A small evaluation set: real or carefully anonymized questions, expected answer points, and examples the assistant should refuse or escalate.

Step-by-Step

1. Choose one job, not a general-purpose bot. Find a frequent, bounded task where people lose time—for example, locating the current travel policy or summarizing an internal procedure. Write down who asks, what information they need, what a good answer enables, and what the assistant must not do. Identify a human owner who can decide whether the result is useful.
2. Set measurable success and safety criteria. Establish a baseline before building: time to find an answer, rate of successful task completion, or volume of routine requests. Define acceptable answer quality, response time, and escalation behavior. Include a “do not answer” criterion for missing, conflicting, sensitive, or out-of-scope information. A demo that sounds convincing is not evidence of dependable performance.
3. Map the data and permissions. List the systems the assistant may consult, who owns each source, how often it changes, and which employee groups can see it. Use approved, current material; exclude documents with unclear ownership or permission. Decide how to handle restricted records and personal information. The assistant must not reveal a document simply because it was indexed or included in a prompt.
4. Pick the simplest architecture that meets the need. For questions about changing internal content, retrieval-augmented generation (RAG) is often a practical starting point: retrieve relevant authorized passages, then ask the model to answer from them. A typical system has an employee interface, an authenticated backend, a retrieval layer, and a model endpoint. Fine-tuning can help with consistent style or repeated patterns, but it does not automatically keep facts current or enforce access permissions.
5. Build ingestion before polishing the chat. Extract text from approved sources, preserve useful headings and metadata, and divide long documents into coherent sections. Store source identifiers, owners, dates, and access labels alongside each section. Remove duplicates and stale copies. Test retrieval directly: for a known question, check whether the right passage appears near the top. If retrieval is poor, changing the model prompt will not fix the underlying data problem.
6. Enforce identity and authorization in the backend. Authenticate employees through the organization’s existing identity system. Apply permissions during retrieval, using the caller’s identity and document-level access rules; do not rely on the model to decide who is allowed to see content. Keep model credentials and data-store secrets server-side. Limit logs to what is needed, restrict log access, and set a retention policy with the relevant owners.
7. Design answers for verification. Instruct the assistant to use retrieved material, say when evidence is insufficient, and ask a clarifying question when needed. Show citations that point to accessible source documents or passages. Make the source date visible where freshness matters. Do not present generated explanations as policy or fact without a way for employees to verify them.
8. Test with realistic and adversarial cases. Create a fixed set of representative questions, including typos, ambiguous requests, unanswerable questions, outdated-source conflicts, and attempts to get restricted information. Have subject-matter owners review factual correctness and citations. Test access with accounts that have different permissions. Track incorrect answers, missed retrievals, inappropriate disclosures, and unnecessary refusals separately.
9. Pilot with a small group and a clear fallback. Explain what the assistant can and cannot do, how to report an issue, and where to go for a definitive answer. Monitor usage, latency, costs, user feedback, and failure categories. Keep the original process available while the pilot is evaluated. Pause or narrow the pilot if there is a privacy incident, repeated harmful misinformation, or a permissions failure.
10. Assign ongoing ownership before launch. Name people responsible for source updates, access reviews, evaluation, model or prompt changes, and incident response. Re-run the evaluation set after changes to documents, retrieval, prompts, or models. Track changes so a regression can be diagnosed and rolled back. Expand to another workflow only when the first one meets its agreed quality and safety goals.

Common Mistakes

• Starting with “answer anything.” A broad scope makes evaluation and safe escalation difficult. Start with a specific workflow and expand based on evidence.
• Indexing everything. More data can mean more stale, duplicated, or unauthorized material. Use an approved source list and preserve permissions and freshness metadata.
• Assuming a prompt prevents leaks or hallucinations. Prompts can guide behavior, but authorization belongs in the application and answers still require evaluation.
• Launching without citations or an escalation route. Employees need a way to check claims and reach a human when the assistant is uncertain.
• Measuring only satisfaction or fluent answers. Include task completion, factual accuracy, retrieval quality, access control, and failure rates.

Tips

• Keep a small, versioned test set and include examples of good answers, not just prompts.
• Use metadata filters and document permissions before sending context to the model.
• Make the assistant’s scope visible in the interface, with a clear way to report wrong or outdated content.
• Start with read-only answers. Add actions or write access only when there is a separate approval, confirmation, and audit design.
• Prefer a dependable answer with a source and a sensible “I don’t know” over a polished guess.

Troubleshooting

Answers are plausible but wrong: inspect the retrieved passages first. If they are irrelevant, improve source quality, chunking, metadata, or retrieval. If they are relevant but the answer misstates them, tighten instructions, require citations, and add the failure to the evaluation set.

It misses known information: check whether the document was successfully ingested, whether its text extracted correctly, and whether filters or permissions exclude it. Test a direct search before changing models.

Users see information they should not: disable the affected feature or pilot, preserve relevant audit records, and follow the organization’s incident process. Review authorization at retrieval time, cached results, logs, and shared links before restoring service.

Responses are slow or costly: measure retrieval, model, and application time separately. Reduce unnecessary context, avoid repeated work where safe, and set usage limits. Do not improve speed by removing access checks or essential citations.

Employees do not trust or use it: compare answers with the real workflow, review feedback with users, and fix the most common failure. Clarify scope and limitations; do not market the assistant as authoritative when it is not.

Frequently Asked Questions

Do we need to train our own model?
Usually not for a first internal assistant. An approved existing model with well-managed retrieval is often enough; consider customization only after evaluation identifies a specific gap.

Is RAG the same as fine-tuning?
No. RAG supplies relevant source material at answer time. Fine-tuning changes model behavior through training examples and is not a substitute for current sources or access control.

Can the assistant use confidential documents?
Only if the organization has approved the use and the system enforces each user’s existing permissions. Confirm data handling, retention, and logging with the responsible teams.

How can we tell whether it is working?
Use a reviewed test set and measure task success, factual accuracy, citation quality, access-control failures, refusals, latency, and user feedback against agreed targets.

Should it take actions, such as updating a record?
Begin with read-only use. For actions, add explicit user confirmation, narrowly scoped permissions, validation, audit trails, and a way to undo or escalate mistakes.

What should it do when sources disagree?
It should show the conflict, cite the relevant sources, and avoid choosing a definitive answer unless an owner has identified which source takes precedence.

Related

• AI Language Models and DNA: Accuracy and Limits
• Meta VR Glasses Storage Full: Move or Delete Media
• Local AI vs Cloud AI: What Works Offline?