How to Build a Private AI Assistant Using Company Data
Secure Enterprise AI Implementation Guide

Connecting a chat interface to every company file may produce a quick demo, but it doesn't create dependable or properly governed access to company knowledge. Without identity and permission checks, employees may see documents they shouldn't. Poor source control can surface outdated policies or files named "final\_v7b." And when important systems aren't indexed or answers arrive without usable citations employees can't tell whether the response is complete, current, or trustworthy.

A production-ready assistant should work differently. It should retrieve information only from approved sources, respect each user's permissions, and provide clickable citations that employees can verify. It should say when it's unsure, route to humans when needed, and follow clear review and escalation rules. In short: a secure enterprise AI assistant that fits your operating model, not a side project.

Building that capability requires more than choosing a model. You need a focused use case, approved company data, permission-aware retrieval, appropriate hosting, measurable quality controls, and clearly assigned ownership. This guide explains how those pieces fit together from initial scoping to a governed pilot.

Start With a Narrow Use Case, Users, and Acceptance Criteria

Start with a narrow scope so you can test the assistant rigorously. A credible pilot should answer one recurring category of business questions well before it attempts company-wide knowledge access. A Build MVP Fast Software Services approach can help teams validate the core use case, data readiness, security requirements, and user value through a focused initial release before investing in a broader enterprise rollout. Suitable pilot use cases might include finding an approved SOP and summarizing its steps, locating the latest product specification, or explaining which part of a company policy applies to a particular employee.

• Choose one question category: Select a frequent, time-consuming task that relies on approved information and has reasonably clear correct and incorrect outcomes. Example (hypothetical): "Locate the current PTO policy, highlight eligibility differences by country, and link to the source."
• Identify the intended users: Identify the employees or teams included in the pilot and document how they currently access the relevant information. Define excluded users, repositories, document categories, and data types just as clearly.
• Set measurable acceptance criteria: Decide what "good" looks like before build:
  • Usefulness: for example, ≥80% of pilot questions resolved without human re‑search; median response under 15 seconds.
  • Authorization behavior: zero instances of cross‑permission leakage in test; answers only from permitted sources.
  • Citation usability: every answer includes 1-3 clickable citations that open at the paragraph/section level and are readable by the requester.
  • Escalation: the assistant flags uncertainty or missing permissions and routes to a human playbook within one click.
  • Operational ownership: named owners can reindex sources, revoke content, and review logs without engineering tickets.

Example: HR Policy Assistant Pilot

An HR operations team could pilot an assistant that answers employee questions using approved policy documents in SharePoint. Draft policies, legal cases, performance records, and employee-specific documents would remain excluded. The assistant would need to retrieve the current policy, respect country-level permissions, provide paragraph-level citations, and escalate ambiguous questions to HR.

These targets are examples. Each organization should set thresholds based on the risk and complexity of its use case.

• Draw the review line: Define where the assistant may provide read‑only answers or drafts, and where a qualified person must review before use. Contracts, regulated decisions, exports, or consequential workflow actions (for example, provisioning, customer notifications) require human approval. The assistant may prepare drafts, but a qualified person should make or approve consequential decisions.
• Assign accountable owners: Name the business sponsor, data steward, security lead, legal/records contact, technical owner, and a pilot‑approval authority. Put decision rights in writing so governance isn't implicit.

Your project has a credible boundary when you can clearly state the problem, intended users, approved sources, excluded information, operating rules, and success measures. The next step is preparing the underlying company data for reliable retrieval.

Prepare Approved Company Data Before Connecting an Assistant

Once the pilot scope is clear, prepare the information the assistant will be allowed to retrieve. This work should happen before documents are indexed or sent to a model.

• Start with authoritative, business-owned sources: Inventory policies, SOPs, product docs, price lists, service catalogs, FAQs, contracts, and operational runbooks. Then approve only sources that have a named content owner, a clear business purpose for the pilot, and an existing access model you can enforce. Exclude personal drives, unmanaged folders, and repositories without accountable owners. If no one can confirm the source's purpose, accuracy, or lifecycle, it isn't ready for the pilot.
• Classify documents at the level where permissions actually change: Mark sensitivity and restrictions per document or page, not just per repository. Capture project codes, country/region limits, export controls, legal‑hold flags, and time‑bound embargoes (for example, "valid after 2025‑01‑01"). Don't assume role‑based access alone reflects every rule. Example: a global PTO policy may be public to employees, but a country appendix might be restricted to local HR. Repository-level or role-based access may not be sufficient. A single document can contain sections governed by different regions, projects, contracts, or confidentiality rules. Where necessary, preserve those restrictions at the page, section, record, or chunk level.
• Clean and fix source quality first: Remove duplicate and obsolete versions, and establish one authoritative location for each document. Establish a simple versioning rule (for example, "For example, the approved PDF in the policy library may be the authoritative source, while working drafts remain excluded in Confluence until publication."). Decide how to handle scanned PDFs, diagrams, and images what to OCR, what to transcribe, and what to exclude until checked. If you plan to extract structured data from PDFs, see AI data extraction from PDFs: automate securely. To tighten business decision‑making with better inputs, also review Data cleaning: impact on business decisions.
• Require traceable metadata: Require metadata that supports retrieval, authorization, freshness, and auditing. At minimum, capture the content owner, authoritative source URL, publication or effective date, document version, intended audience, retention status, and relevant permission attributes. This keeps retrieval grounded, citations clickable, and audit trails intact. If a document lacks basics like owner or effective date, it's not ready for your secure enterprise AI assistant.
• Define lifecycle and connector rules: Document how each connector authenticates, how often it checks for changes, and when it reindexes. Define how deletions, archival changes, access revocations, and legal holds propagate from the source system to indexes, caches, backups, and generated previews. Require source‑change notifications to data owners. For logs and feedback, set employee‑privacy rules: limit who can see queries, mask sensitive strings, and define retention. That's how a private RAG implementation avoids "shadow archives" of sensitive prompts. For a sense of the operational plumbing you'll need, see the Robotic Process Automation and Data Management Platform case study.

Good data preparation improves both security and answer quality. It also makes permission-aware retrieval easier to implement regardless of whether the assistant runs through a managed service, private cloud, hybrid architecture, or on-premises environment.

Build a Permission-Aware Private RAG Architecture

Design the retrieval architecture so access rules remain attached to the content and are enforced throughout ingestion, indexing, retrieval, response generation, and downstream delivery.

Here's the baseline flow your team should implement:

• Approved sources → connectors: Only pre‑approved systems connect. Connectors authenticate through your identity provider (IdP) and request the minimum scopes.
• Parse and chunk: Normalize, section, and chunk documents. Keep logical structure (titles, headings, tables) so chunks carry context.
• Label with metadata and permissions: Each chunk should inherit synchronized metadata from its authoritative source. Permissions, ownership, effective dates, and retention status can change, so they should not be described as permanently immutable.
• Index: Write chunks to an index that supports both semantic (vector) and lexical search. Store permissions with each record.
• Retrieve for an authorized user: At query time, the user is authenticated, authorization filters are applied, and only permitted chunks are retrieved.
• Ground the answer and cite: The model uses retrieved chunks as grounding material. It returns an answer with citations to source passages so reviewers can check the work.

A private retrieval‑augmented generation (RAG) implementation retrieves relevant, authorized material before any answer is generated. Citations make an answer easier to verify, but they do not prove that it is correct, complete, current, or appropriate for every audience. Treat citations as verification and audit aids not as substitutes for authorization, evaluation, or human review.

Preserve source‑system permissions with layered enforcement. Never let the model decide whether a user may retrieve a source.

• Identity (SSO/IdP): All users and services authenticate via your IdP. Tokens include attributes you'll use for policy decisions (roles, groups, regions).
• Connectors: Enforce least privilege, propagate ACLs, and block sources per legal‑hold or embargo rules. A connector should not index everything visible to a broad service account. It should preserve the source system's user, group, project, region, and document-level access rules.
• Indexes: Store ACLs with each chunk. Support policy evaluation server‑side. Consider per‑tenant namespaces to isolate teams or business units.
• Retrieval queries: Apply allow/deny filters before ranking results. Attribute‑ and time‑based controls (for example, "US‑only until 2026‑01‑01") should execute here.
• Answer rendering: Redact or suppress snippets a user can't see, and avoid summarizing restricted context. If no permitted evidence remains, return "insufficient rights."
• Downstream integrations: Re‑check permissions before posting to Slack, tickets, or email. Log only what policy allows; never leak excerpts in telemetry. Protect audit logs with least‑privileged access, encryption, and policy‑defined retention.

Design choices that affect relevance, permissions, and maintainability:

• Chunk size: A range of approximately 200-500 words can be a starting hypothesis, but it should not become a fixed rule. Policies, tables, contracts, support articles, and technical manuals may require different chunking strategies. Evaluate chunk size using retrieval quality, context preservation, citation precision, and permission boundaries.
• Document structure: Carry headings, lists, tables, captions, and figure references into chunk metadata. Structure boosts retrieval quality and safer summarization.
• Metadata quality: Incomplete or stale metadata can weaken both retrieval quality and access-control enforcement.
• Retrieval filters: Combine semantic similarity with keyword, field, and date filters. Add business rules (for example, "prefer current policy over archived").
• Index tenancy: Choose between a shared index with record-level authorization, separate indexes for stronger domain or regional isolation, or a hybrid approach. The decision should reflect tenant boundaries, regulatory requirements, operational complexity, and the consequences of a filtering failure.
• Reindexing strategy: Rebuild on schema or policy changes; run incremental updates for content changes; propagate access revocations and applicable deletion or erasure requirements across indexes, caches, and generated artifacts.

When built this way, your internal knowledge chatbot answers from company knowledge AI assets while honoring the same controls your systems already enforce whether you run a managed service, a private‑cloud setup, or an on-premises AI assistant.

Choose Managed, Private-Cloud, Hybrid, or On-Premises Deployment

You've defined a permission‑aware architecture. Now decide where it runs. Deployment location matters, but it doesn't determine privacy on its own. Privacy depends on identity controls, data handling, encryption, contractual commitments, monitoring, retention, and day-to-day operational discipline.

"Private cloud" can refer to dedicated or logically isolated resources operated in an organization's cloud environment or datacenter. Because vendors use the term differently, define the required network, tenancy, key-management, logging, and administrative boundaries contractually.

• Data residency: Where data and logs may live; cross‑border flows.
• Contractual terms: DPAs, training prohibitions, IP ownership, audit rights, sub processor list.
• Model access: API‑only versus bring‑your‑own‑model, fine‑tuning options, GPU availability.
• Integration needs: Network paths to source systems, SSO/IdP, proxies, egress rules.
• Operational capacity: Who patches, monitors, rotates keys, and answers pagers.
• Recovery expectations: Target RTO/RPO, multi‑region failover, backups and key escrow.
• Observability: Metrics, traces, redactable logs, access audits exportable to your SIEM.
• Cost drivers: See list below; avoid "per‑seat" guesses that hide real usage costs.

When to use each model

• Managed (SaaS): Control over residency, networking, model configuration, and observability varies significantly by provider and service tier. A managed platform may support both pilots and production workloads when its controls and contractual commitments satisfy organizational requirements.
• Private‑cloud: A private-cloud model may be appropriate when the organization requires isolated networking, custom connectors, customer-controlled encryption keys, detailed audit exports, or tighter control over data flows.
• Hybrid: Keep sensitive indexes, caches, or PII processing in controlled environments while using managed services for less‑sensitive layers (for example, model inference). Use only with explicit interface contracts, data minimization, encryption, and clear responsibility boundaries.
• On‑premises: On-premises deployment can provide greater control over infrastructure, data paths, and physical environments. However, stronger control does not automatically produce stronger security; the organization remains responsible for patching, identity management, monitoring, model operations, backups, and incident response.

Cost drivers that are easy to underestimate

Model usage is only one part of the total cost. Data preparation, connector maintenance, security controls, testing, monitoring, and ongoing ownership can be equally important. Before scaling the investment, evaluate AI Automation ROI to Build a Credible Business Case by comparing these implementation and operating costs with measurable time savings, productivity gains, and business outcomes.

• Document handling: Parsing, OCR, conversions.
• Embeddings and indexing: Initial build and re‑indexing cadence.
• Storage: Object store, vector DB, backups, snapshots.
• Query volume: Retrieval calls, cache hits/misses.
• Model inference: Tokens, context windows, reranking, guards.
• Connectors: Licenses, throttling, custom builds.
• Security controls: KMS/HSM, network egress, secrets management.
• Monitoring and observability: Logs, traces, SIEM ingest.
• Support and maintenance: SRE/on‑call, upgrades, incident drills, and custom AI solutions.

Pick the simplest model that satisfies your residency, contractual, and recovery needs with the operating capacity you actually have. Then prove it with a pilot before you scale your AI chatbot using company data.

Design Security and AI Governance Into Daily Use

You've picked where it runs. Now make sure people can trust it every day. A private AI assistant over company data is only "private" when security and governance show up in every prompt, retrieval, and action.

Treat every prompt and retrieved content item as potentially untrusted even when the content comes from an approved internal repository. Treat user prompts, retrieved passages, scanned PDFs, diagrams, screenshots, and other multimodal files as potential prompt‑injection or unsafe‑content sources even if they come from an approved repository. That's where a secure enterprise AI assistant earns its name: by never trusting content by default.

Content scanning can reduce risk, but it cannot reliably detect every prompt-injection technique. The system should therefore combine detection with strict tool permissions, independent authorization, constrained actions, output validation, and human approval where required.

Use layered defenses so no single miss becomes a breach:

• Prevent: The model may suggest an answer or action, but the surrounding platform should independently decide whether the requested data access or action is permitted.
  • Approve sources before ingestion.
  • Enforce least-privileged connector access.
  • Separate external, partner-provided, and internal content.
  • Restrict tools and workflow actions through allow lists.
• Detect
  • Scan extracted content for hidden instructions, secrets, malware indicators, and restricted data.
  • Monitor unusual retrieval patterns and repeated access denials.
  • Test the system using prompt-injection and corpus-probing scenarios.
• Contain
  • Apply permission filters before retrieval and ranking.
  • Limit retrieval scope, file types, time windows, and result counts.
  • Validate structured outputs and workflow parameters before execution.
  • Require approval for consequential actions.
• Respond
  • Maintain playbooks for pausing connectors, rotating keys, revoking access, notifying owners, and restoring service.
  • Preserve policy-compliant evidence for investigation.
  • Review incidents and update controls before reopening affected functionality.

Apply input and output controls for secrets, personal information, regulated data, and policy violations. Configure the controls according to the actual use case because overly broad filtering can also block legitimate business questions.

The governance program can use the NIST AI Risk Management Framework to organize risk responsibilities, while OWASP's guidance for large language model applications can inform threat modeling and adversarial testing. Refer to relevant cloud-provider documentation (e.g., NIST Zero Trust Architecture) for infrastructure-level security guidance.

Keep sensitive levers outside the model's discretion. Don't let it decide authorization, mint credentials, set retrieval filters, export data, or trigger consequential workflow actions. Use independent policy engines and, where the action is material (for example, data export, customer contact, ticket closure), require human approval.

Apply least privilege everywhere users, service accounts, connectors, model gateways, and admins. Review access regularly. Record retrieval events, denied requests, policy exceptions, and suspected corpus‑probing. Rate‑limit and alert on unusual patterns. Protect logs with access controls and retention that meet policy and employee‑privacy standards.

Governance needs clear owners and records. Define acceptable use, change control for prompts/models/tools, incident handling, retention rules, deletion propagation across indexes, caches, and embeddings, and legal holds. Protect employee privacy for query logs and feedback; set retention, masking, and approved analytics. Do this consistently and, whether it runs as an on‑premises AI assistant or a managed setup, trust will come from your day‑to‑day controls not the server room.

Test Answer Quality, Human Review, and Workflow Integrations

Before connecting the assistant to operational workflows, test it using representative company data, realistic user permissions, and clearly defined success criteria.

1. Offline evaluation before the pilot

Use a controlled test set to evaluate retrieval, grounding, correctness, permissions, citations, safety, and escalation behavior before employees depend on the assistant.

Build a representative test set

Start with realistic questions and the exact data the assistant should use. Include easy wins and messy edge cases so you don't get surprised later.

• Real user questions: Pull top tasks from your pilot group; avoid synthetic prompts when possible.
• Expected sources: Pair each question with the passages or records the answer should cite.
• Hard exceptions: Add ambiguous asks, multi‑document tasks, and conflicting answers.
• Stale content: Include outdated policies to verify the system prefers the latest version.
• Restricted content: Seed items only some testers can access to validate permission scoping.
• Prompt injection: Add adversarial inputs to confirm guardrails hold.

For a quick refresher on structuring automated checks and coverage, see Automated testing: benefits and use cases.

What to measure (and how to pass)

Score each run and set thresholds that match your risk and use case. Don't chase a universal pass rate; define yours and hold to it.

• Retrieval quality: Are the right chunks or records pulled for each query?
• Groundedness: Does the answer stick to retrieved sources rather than inventing?
• Relevance and completeness: Is it on‑topic and does it cover all required parts?
• Citation usefulness: Are links/snippets precise and quick to verify?
• Correctness: Do domain reviewers validate the final answer as right?
• Authorization behavior: Are restricted items hidden and denials consistent?
• Safety handling: Are toxic, PII, or regulated elements filtered or masked per policy?
• Escalation performance: When uncertain or unsafe, does it defer, ask for more info, or route to a human?

That's where a strong private RAG implementation earns trust: great retrieval plus verifiable grounding over approved company knowledge.

Permission and lifecycle checks

Test the messy realities of enterprise content across connectors, indexes, retrieval, rendering, and logs.

• Permission changes: When access is revoked, does the assistant immediately stop retrieving those items (and stop showing cached snippets)?
• Deleted records and legal holds: Do deletions propagate? Are legal-hold rules correctly enforced for retrieval, deletion, modification, and export?
• Conflicting versions: Does it prioritize the current record and surface version/last‑updated?
• Restricted documents: Are previews, citations, and logs free of leaked details?

(Hypothetical) Suppose an approved policy was updated yesterday while an older version remains in an archive. The assistant should prefer the current authoritative version, disclose its effective date, and cite it even if the archived version is a stronger keyword match.

Human review and safe integrations

• Require human review: For high‑impact answers, contract interpretation, regulated decisions, sensitive exports, and any action that changes records or invokes external services.
• Start read‑only: Begin with retrieval and draft creation. Add controlled actions only after you demonstrate deterministic validation, independent authorization, and a clear approval path.

Once your secure enterprise AI assistant clears these gates with company knowledge, you're ready to launch a governed pilot and expand integrations with confidence.

2. Live pilot monitoring

Once the assistant is available to pilot users, monitor unanswered questions, citation use, user feedback, permission anomalies, source freshness, latency, and operating costs.

Launch a Governed Pilot and Scale It Deliberately

Launch the pilot with one user group, one approved source set, and one primary category of questions. Assign named owners, provide a clear support route, and keep integrations read-only until the quality and security gates have been met.

Expand only when pre‑agreed acceptance criteria are met and residual risks are accepted by the right owners. For example, require minimum retrieval coverage, zero unauthorized access events, defined per‑answer cost ceilings, and positive user feedback in the target workflow. Each new source, user group, language, modality, or workflow action should pass its own readiness gate: permissions validated, evaluations re‑run, runbooks updated, alerts set, and budgeted costs confirmed. That's how you scale a secure enterprise AI assistant deliberately.

If you need help connecting data readiness (including messy PDFs), custom AI, systems integration, and governance into a scoped private RAG implementation, Moltech Solutions can support that effort.

Conclusion

A private AI assistant becomes dependable when it operates as a governed knowledge-access system not simply as a chatbot connected to company files. Start with one valuable use case, approve the underlying sources, preserve permissions throughout retrieval, and establish measurable quality and security gates before launch.

From here, keep it concrete. Document a single use case, define clear data boundaries, and design a permission‑aware retrieval‑augmented generation approach that cites what it retrieves. Decide what "good" means before launch answer quality, security gates, and auditability and test against it. Assign owners to keep connectors healthy, indexes current, identity and authorization in sync, and logs reviewed. That's how a secure enterprise AI assistant moves from a flashy demo to a reliable internal knowledge chatbot.

If you want a sounding board, Moltech Solutions Inc. can run a fit and readiness assessment covering approved data, required integrations, governance controls, hosting choices (including on‑premises or private cloud), and day-two operations so you can judge whether a pilot is feasible now and what must be true to proceed. Reach out through your approved Moltech contact pathway to start the conversation.