Librarian
RAG knowledge assistant
Live Demo
Overview
Librarian is a retrieval-augmented generation system built to do the unglamorous part of RAG well: not the chat box, but the pipeline behind it. Documents are chunked semantically, embedded, and indexed for hybrid search; queries pull candidates by both meaning and keyword, a reranker sharpens the top results, and the model answers only from what was retrieved — with inline citations back to the source chunk. If it can't ground an answer, it says so instead of guessing.
The Challenge
The demo version of RAG — embed some text, stuff the top matches into a prompt — falls apart on real corpora. Pure vector search misses exact terms like error codes and API names; naive chunking splits a table from its header; and a model handed weak context will happily fill the gaps with confident fiction. The hard problems in RAG live in retrieval quality and grounding discipline, long before the model is even called.
The Solution
A pipeline where every stage earns its place. Semantic chunking keeps sections and their headers together. Hybrid retrieval combines dense vector similarity with keyword search so exact identifiers aren't lost. A reranker reorders candidates by true relevance to the query rather than raw embedding distance. The generation prompt is strict: answer only from the provided context, cite the chunk for every claim, and refuse when the context doesn't support an answer. A small eval set keeps retrieval honest as the corpus grows.
Architecture
An ingest pipeline and a query pipeline sharing one index. Retrieval quality is treated as the product; generation is the last, most constrained step.
Key Decisions & Tradeoffs
Hybrid retrieval, not pure vectors
Why: Vector search alone kept missing exact strings — an error code, a function name, a SKU. Adding keyword search back in and fusing the rankings recovered the precise-term matches that embeddings blur, without losing semantic recall.
Rerank before you generate
Why: Embedding distance is a rough proxy for relevance. A dedicated reranking pass over the top candidates consistently pulled the actually-useful chunk into the context window, and it moved answer quality more than any prompt change I tried.
Citations are mandatory, refusal is allowed
Why: An answer without a source is a liability in a knowledge tool. Requiring a citation per claim — and letting the model say "not in the docs" — is what makes the output trustworthy instead of merely fluent.
An eval set from day one
Why: Retrieval quality silently rots as the corpus grows and chunking changes. A small golden set of question/expected-source pairs turns 'it feels worse' into a number I can regression-test against.
What I took away from this project
Lessons that still shape how I build — in my own words.
RAG is a retrieval problem wearing a generation costume
Every time an answer was wrong, I instinctively reached for the prompt. Almost every time, the real fix was upstream: the right chunk never made it into context. Once I started treating retrieval as the product and generation as a thin, constrained layer on top, quality stopped being mysterious and started being measurable.
Chunking is where quality is won or lost
The least interesting-sounding decision — how to split a document — had the biggest effect on everything downstream. Splitting a table from its header, or a heading from its section, poisons retrieval in ways no reranker recovers. Getting chunking right did more for accuracy than swapping models ever did.
Impact
Technology Stack
AI
Retrieval
Data
Quality
Key Features
Related case studies
Want something like this, built for you?
If the Librarian story sounds close to what you're trying to ship, a 30-min call is the fastest way to find out if we're a fit.