Building RAG over twenty years of grant compliance

Most RAG demos answer questions about a folder of PDFs. The system I built answers questions about a compliance platform: thousands of structured reports, their revision history, the documents attached to them, and the engineering context around the software itself. That changes almost every decision.

The question nobody could answer quickly

The platform replaces a 20-year-old grant and housing-compliance system used by 24 California cities and counties. A single record can carry 200 to 500 fields, and records change through revisions over years.

The questions people actually ask look like this:

Why was this program's budget amended, what changed in the reports filed after it, and who approved it?

Before, the answer lived across a report, a revision snapshot, a field-level diff, an attached PDF, a ticket, and someone's memory. A generic "chat with your documents" bot does not help. The value is in connecting those pieces and showing the evidence.

What an answer looks like

The output is not a paragraph. It is a small, checkable report:

Question: Why was Program X's budget amended?

Answer:   The amendment followed a revised allocation from the agency...

Evidence:
  - Revision of the grant record, field `budget.total` (diff)
  - Expense report filed the following quarter, section 3
  - Attached allocation letter (PDF), page 2

Confidence: Medium. No approval note found for the revision.

Every line under "Evidence" has to resolve to a real record the user is allowed to see. If it cannot, it is not printed.

The architecture

reports · revision diffs · PDFs · Jira · Confluence · GitHub · Slack
                              │
                  ingestion + chunking workers
                              │
                 record-shaped chunks + metadata
                              │
           Titan Text Embeddings (Amazon Bedrock)
                              │
            ┌─────────────────┴─────────────────┐
     pgvector (dense)                  Elasticsearch (BM25)
            └─────────────────┬─────────────────┘
               tenant filter, then hybrid fusion
                              │
                 Cohere Rerank (Amazon Bedrock)
                              │
            Claude on Bedrock, numbered context blocks
                              │
            answer + citations checked against sources

I owned the whole path: ingestion and chunking, retrieval, answer generation with citations, and the evaluation and monitoring around it. Keeping everything on Bedrock meant model calls stayed inside the AWS account the platform already runs in, under the same IAM and network boundaries.

Decision 1: tenant scoping happens before retrieval

This was the one I would not compromise on. The platform already enforces per-resource permissions and query-level tenant scoping, and the retrieval layer had to inherit that rather than approximate it.

Every chunk carries its tenant and the resource it came from. The filter is applied inside the pgvector and Elasticsearch queries, never to the results afterwards. Filtering after retrieval fails in two ways: another city's text can reach the model's context, and a heavily filtered top-k can quietly come back empty.

Decision 2: chunk by record structure, not by token count

Fixed token windows are fine for prose. Compliance data is not prose. A report is a set of sections and fields whose meaning lives in their labels, and cutting one mid-field produces a chunk that says amount: 48,000 with no idea what the amount is for.

So chunks follow the shape of the source:

• reports: one chunk per section, with field labels rendered inline
• revisions: one chunk per diff, stating what changed, from what, to what
• PDFs and long text: split by page and paragraph, with overlap
• tickets, pages, PRs and threads: one chunk per item or per logical section

Each chunk keeps the metadata a person would use to trust it: jurisdiction, program, reporting period, revision, source system and a link back to the original. For reports, the chunker reads the form definition instead of guessing at structure.

Decision 3: hybrid retrieval, because IDs and numbers matter

Dense embeddings are good at "why did the budget change". They are bad at an exact report period, a program code or a ticket key. Real questions mix both.

Each question runs a vector query in pgvector and a BM25 query in Elasticsearch in parallel, both tenant-filtered. The two ranked lists are merged with reciprocal rank fusion, which needs no score calibration between two very different systems. Cohere Rerank then narrows the merged candidates to the few that go into the prompt, with a cap per source so one long report cannot crowd out everything else.

Decision 4: citations are verified, not trusted

Claude receives context blocks numbered by chunk ID and must cite those IDs in structured output. Before anything is rendered:

• every cited ID must exist in the retrieved set
• every cited ID must still pass the user's permission check
• a claim whose citations fail is dropped, not reworded

If too much is dropped, the answer says so. "I found the revision but not the reason" is a useful answer. A fluent guess is not.

Decision 5: evaluate retrieval separately from generation

When an answer is wrong, the first question is whether the right evidence was ever retrieved. So evaluation runs against a golden set of real questions, each paired with the records that must be cited, and measures the stages separately:

• retrieval recall: did the right records make it into context?
• citation precision: does every cited chunk support its claim?
• refusal correctness: when the evidence is missing, does it say so?
• tenant leakage: cross-tenant questions that must return nothing

Monitoring tracks the same pipeline in production: what was retrieved, what was reranked in, what was cited, and latency and cost per stage. Reranking and generation are where both go.

What I left out

• Agents and multi-step planning. Most of these questions are one good retrieval plus one good synthesis step.
• A graph database. Revision history already encodes the timeline, so a sort by revision date gets most of the way.
• Fine-tuning. Retrieval quality was the bottleneck long before the model.

The interesting part of RAG over a system like this is not the model. It is making sure the right record, from the right tenant, at the right revision, is what the model reads, and that every sentence it writes can be traced back to it.