Build RAG for legislation and policy document search

Problem

Government agencies and law firms manage thousands of policy documents, contracts, and legislation files. Keyword search returns irrelevant results or misses documents that use different phrasing for the same concept. Employees waste hours manually searching folders, or miss critical information because they did not use the right terms. Cross-referencing between related legislation is entirely manual.

Solution

Build a RAG system that chunks legal documents intelligently and uses Claude for semantic retrieval.

Step 1: Chunk documents with legal-aware splitting

from langchain.text_splitter import RecursiveCharacterTextSplitter

legal_splitter = RecursiveCharacterTextSplitter(
    separators=[
        "\n## ",        # Major section headers
        "\nArticle ",   # Article boundaries
        "\nSection ",   # Section boundaries
        "\nClause ",    # Clause boundaries
        "\n\n",         # Paragraph breaks
        ". ",           # Sentence boundaries
    ],
    chunk_size=1500,
    chunk_overlap=200,
)

Step 2: Create embeddings and store in pgvector

import anthropic

client = anthropic.Anthropic()

def embed_and_store(documents, session):
    for doc in documents:
        response = client.embeddings.create(
            model="voyage-3", input=doc["text"]
        )
        chunk = DocumentChunk(
            content=doc["text"],
            embedding=response.data[0].embedding,
            metadata=doc["metadata"],
        )
        session.add(chunk)
    session.commit()

Step 3: Query with semantic search and LLM synthesis

def query_policies(question: str, session) -> str:
    q_embedding = client.embeddings.create(
        model="voyage-3", input=question
    ).data[0].embedding

    results = session.query(DocumentChunk).order_by(
        DocumentChunk.embedding.cosine_distance(q_embedding)
    ).limit(10).all()

    context = "\n---\n".join(
        f"[{r.metadata.get('document_title')}]\n{r.content}"
        for r in results
    )

    response = client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=2000,
        messages=[{
            "role": "user",
            "content": f"""Answer based on these policy documents.
Include citations with document names and clauses.
If not found, say so explicitly.

Context:\n{context}\n\nQuestion: {question}""",
        }],
    )
    return response.content[0].text

Why It Works

Legal-aware chunking preserves document structure so each chunk contains a coherent section rather than cutting mid-clause. Semantic embeddings match questions to passages even when terminology differs. The LLM synthesis step combines information from multiple chunks into a direct answer with citations, saving users from reading dozens of search results.

Context

For law firms, this answers questions like "When does the McDonald's contract expire?" across hundreds of contracts
Consider Copilot Studio with Azure Search for organizations already in the Microsoft ecosystem
Chunking strategy matters significantly for legal text; splitting mid-clause degrades retrieval quality
Add metadata extraction (dates, parties, contract values) as a preprocessing step for structured queries
Public domain government documents avoid copyright concerns that apply to proprietary legal databases