A retrieval-augmented assistant that answers questions over your own docs and code — hybrid search, reranking, and answers that cite the exact source they came from.
hybrid search + reranka citation per claimrefuses when unsure
Overview
Overview
Librarian is a retrieval-augmented generation system built to do the unglamorous part of RAG well: not the chat box, but the pipeline behind it.
Documents are chunked semantically, embedded, and indexed for hybrid search; queries pull candidates by both meaning and keyword, a reranker sharpens the top results, and the model answers only from what was retrieved — with inline citations back to the source chunk. If it can't ground an answer, it says so instead of guessing.
The challenge
The demo version of RAG falls apart on real corpora.
Pure vector search misses exact terms like error codes and API names; naive chunking splits a table from its header; and a model handed weak context will happily fill the gaps with confident fiction. The hard problems in RAG live in retrieval quality and grounding discipline, long before the model is even called.
Exact termserror codes and API names
Intact chunkstables keep their headers
Grounded answersno confident fiction
The solution
A pipeline where every stage earns its place.
Semantic chunking keeps sections and their headers together. Hybrid retrieval combines dense vector similarity with keyword search so exact identifiers aren't lost. A reranker reorders candidates by true relevance to the query rather than raw embedding distance. The generation prompt is strict: answer only from the provided context, cite the chunk for every claim, and refuse when the context doesn't support an answer. A small eval set keeps retrieval honest as the corpus grows.
RetrieveRerankCite
Architecture
Two pipelines, one index.
An ingest pipeline and a query pipeline sharing one index. Retrieval quality is treated as the product; generation is the last, most constrained step.
Ingest
Semantic chunking
Header-aware splitting
Embedding + metadata
Index
pgvector dense index
Keyword / full-text index
Per-document access scope
Retrieve
Hybrid dense + sparse
Reranking
Top-k with scores
Generate
Grounded-only prompting
Inline citations
Refuse when unsupported
One request, start to finish
Docs are chunked, embedded, and indexed
A question runs hybrid retrieval
Candidates are reranked by true relevance
The model answers only from retrieved context
Every claim links back to its source chunk
Decisions
Four calls and why I made them.
Call 01
Hybrid retrieval, not pure vectors
Vector search alone kept missing exact strings — an error code, a function name, a SKU. Adding keyword search back in and fusing the rankings recovered the precise-term matches that embeddings blur, without losing semantic recall.
Call 02
Rerank before you generate
Embedding distance is a rough proxy for relevance. A dedicated reranking pass over the top candidates consistently pulled the actually-useful chunk into the context window, and it moved answer quality more than any prompt change I tried.
Call 03
Citations are mandatory, refusal is allowed
An answer without a source is a liability in a knowledge tool. Requiring a citation per claim — and letting the model say "not in the docs" — is what makes the output trustworthy instead of merely fluent.
Call 04
An eval set from day one
Retrieval quality silently rots as the corpus grows and chunking changes. A small golden set of question/expected-source pairs turns 'it feels worse' into a number I can regression-test against.
Reflections
What I took away.
RAG is a retrieval problem wearing a generation costume
Every time an answer was wrong, I instinctively reached for the prompt. Almost every time, the real fix was upstream: the right chunk never made it into context. Once I started treating retrieval as the product and generation as a thin, constrained layer on top, quality stopped being mysterious and started being measurable.
Chunking is where quality is won or lost
The least interesting-sounding decision — how to split a document — had the biggest effect on everything downstream. Splitting a table from its header, or a heading from its section, poisons retrieval in ways no reranker recovers. Getting chunking right did more for accuracy than swapping models ever did.
Outcomes
What shipped.
RetrievalHybridDense + keyword + rerank
AnswersCitedSource per claim
UngroundedRefusedNo context, no answer
QualityEval-gatedGolden retrieval set
Stack
Technology
AI
Claude API
Embeddings
Reranking
Grounded prompting
Retrieval
Hybrid search
Semantic chunking
Top-k scoring
Data
Supabase
Postgres
pgvector
Full-text search
Quality
Golden eval set
Citation checks
Regression tests
Features
What it does
Semantic, header-aware chunking that keeps sections intact for meaningful retrieval
Hybrid retrieval fusing dense vector similarity with keyword search so exact terms aren't lost
A reranking pass that reorders candidates by true query relevance before generation
Grounded-only generation: answers cite the exact source chunk and refuse when context is insufficient
pgvector index on Postgres with per-document access scoping for multi-user corpora
A golden evaluation set that regression-tests retrieval quality as documents and chunking evolve