Private RAG: Search Company Documents Safely

Every company has knowledge locked in documents: contracts, policies, proposals, support notes and technical manuals. Retrieval-augmented generation, usually shortened to RAG, lets people ask questions in plain language and get answers grounded in those documents. The catch is that the documents are often confidential, and a careless setup can show a salesperson the board minutes or send sensitive text to a service you do not control. A private RAG system keeps the data inside an environment you govern and makes sure every answer respects the same access rules as the original files. This guide explains the architecture we use, where leaks usually happen, and a worked example of permission-aware retrieval.

What makes a RAG system private

A RAG system has three jobs: index documents into searchable chunks, retrieve the chunks relevant to a question, and pass them to a language model that writes the answer. Calling it private means four things in practice:

• Data stays in infrastructure you control, or with providers whose data-processing terms you have reviewed and accepted
• Every chunk carries the access rules of its source document, and retrieval enforces them for each user
• Prompts, retrieved text and answers are logged securely, with sensitive values masked where needed
• Nothing is used to train external models unless you have explicitly agreed to it

A private RAG design is less about one product and more about controlling every place the text travels.

The architecture, step by step

Ingestion

Connectors pull documents from file shares, document management systems, wikis or the CRM. Each document is parsed to text, split into chunks of a few paragraphs, and tagged with metadata: source, owner, date and, most importantly, who is allowed to read it.

Embeddings

Each chunk is converted into a vector that represents its meaning. You can run an open embedding model on your own servers, or use a hosted model under an agreement that covers confidentiality and retention. Either way, the choice should be deliberate and documented.

Vector storage

Vectors and metadata are stored in a vector index. For many companies, PostgreSQL with the pgvector extension is a practical choice: it keeps vectors next to normal relational data, supports standard backups and access controls, and avoids adding another service.

Retrieval with permissions

When a user asks a question, the system embeds the question, searches for similar chunks and, crucially, filters out every chunk the user is not allowed to see before anything reaches the model.

Generation

The permitted chunks and the question go to a language model, either self-hosted or a private endpoint, with instructions to answer only from the supplied text and to cite sources.

Where private RAG systems leak

Leak pathHow it happensControlMissing access filtersRetrieval searches every chunk regardless of who asksCopy document permissions into chunk metadata and filter on every queryStale permissionsAccess changes in the source system but not in the indexRe-sync permissions on a schedule and on change eventsPrompt injectionA document contains instructions that make the model ignore its rulesTreat retrieved text as data, restrict tools the model can call, review outputsLogs and analyticsFull prompts and answers stored in plain text or sent to third-party toolsMask sensitive values, restrict log access, set retention periodsExternal model providersText sent to a service that retains or reuses itSelf-host, or use providers with contractual data protectionsOver-broad answersThe model combines several permitted chunks into something sensitiveLimit scope per use case and review high-risk domains

The OWASP Top 10 for LLM Applications lists prompt injection and sensitive information disclosure among its main risks, and is a useful checklist when reviewing a design.

Worked example: permission-aware retrieval with pgvector

The illustrative schema below stores each chunk with the groups allowed to read it. The query only considers chunks whose groups overlap with the user's groups, so restricted text is never retrieved for the wrong person.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
    id              bigserial PRIMARY KEY,
    doc_id          text NOT NULL,
    source_url      text NOT NULL,
    content         text NOT NULL,
    allowed_groups  text[] NOT NULL,          -- copied from the source system
    updated_at      timestamptz NOT NULL DEFAULT now(),
    embedding       vector(1024) NOT NULL     -- match your embedding model's size
);

CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON doc_chunks USING gin (allowed_groups);

The retrieval function then passes the user's groups into that filter, so the permission check happens inside the query itself:

import psycopg

def retrieve(conn, question_vector, user_groups, k=8):
    sql = """
        SELECT doc_id, source_url, content
        FROM doc_chunks
        WHERE allowed_groups && %(groups)s          -- user must share a group
        ORDER BY embedding <=> %(qvec)s::vector     -- cosine distance
        LIMIT %(k)s
    """
    with conn.cursor() as cur:
        cur.execute(sql, {"groups": user_groups, "qvec": str(question_vector), "k": k})
        return cur.fetchall()

The user's groups must come from your identity provider at request time, never from the browser. With approximate indexes, a strict filter can return fewer results than requested; the pgvector documentation describes options for filtered searches, and it is worth testing with realistic permission patterns.

Keeping the index in sync

Documents change, get deleted and change hands. A private RAG system needs:

• Incremental updates when documents are added or edited
• Deletion that removes every chunk of a deleted document promptly
• Permission re-syncs, so a person who loses access to a folder loses access in search too
• A record of which version of each document was indexed

Handling sensitive values inside documents

Some documents are broadly shareable but contain a few sensitive values, such as bank details, personal phone numbers or salary figures. Options include masking those values during ingestion, splitting the document so the sensitive section gets stricter permissions, or excluding the document entirely. Decide the rule for each collection before indexing, not after a user stumbles on something.

Evaluating answers before go-live

Build a test set of real questions with known answers and known restrictions. Check three things: whether the answer is correct, whether it cites the right sources, and whether users without access are refused. Repeat the tests after every change to chunking, models or prompts.

Choosing where to start

Start with one well-defined collection, such as policies and procedures or product documentation, where permissions are simple and the value is easy to see. Add sensitive collections such as contracts or HR documents only after the access controls have been tested. Expanding gradually keeps the risk small while the team learns how people actually use the system.

Private RAG often sits alongside other document automation. Our guides to AI invoice data extraction and automating data entry into your CRM cover how structured data is pulled from the same kinds of documents.

Running it responsibly

Agree who owns each collection, who can add sources, and how users report wrong or inappropriate answers. Show sources with every answer so people can check them, and make it clear that answers are generated and may be incomplete. For decisions with legal or financial consequences, the source document remains the authority.

Frequently Asked Questions

What is private RAG?

It is retrieval-augmented generation run in an environment you control, where documents, embeddings and prompts stay protected and every answer respects the user's access rights.

Can we use a hosted language model and still keep RAG private?

Yes, if the provider's terms cover confidentiality, retention and training use, and only permitted chunks are sent. Some organisations prefer self-hosted models for the most sensitive collections.

How do we stop users seeing documents they should not?

Store each document's permissions with its chunks, filter retrieval by the user's groups from your identity provider, and keep permissions in sync with the source systems.

Do we need a separate vector database?

Not always. PostgreSQL with pgvector works well for many collections and keeps vectors alongside your existing data, backups and access controls.

Build a private RAG system with us

We design private RAG systems with permission-aware retrieval, secure logging and evaluation built in. See our AI automation services, or talk to us about your documents.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.