Librarian's owl holding a stack of books and a glowing open book

AI productAI LabCase study

Librarian

RAG knowledge assistant.

A retrieval-augmented assistant that answers questions over your own docs and code — hybrid search, reranking, and answers that cite the exact source they came from.

Role
Design and engineering
When
2025 to now
Team
Solo build
Read
About 3 min
Librarian's owl holding a stack of books and a glowing open book
Librarian console in pen and ink: a question filed on an index card, its score sheet, and the catalogue cards pulled from the drawers with the cited one marked

Overview

Overview

Librarian is a retrieval-augmented generation system built to do the unglamorous part of RAG well: not the chat box, but the pipeline behind it.

Documents are chunked semantically, embedded, and indexed for hybrid search; queries pull candidates by both meaning and keyword, a reranker sharpens the top results, and the model answers only from what was retrieved — with inline citations back to the source chunk. If it can't ground an answer, it says so instead of guessing.

The challenge

The demo version of RAG falls apart on real corpora.

Pure vector search misses exact terms like error codes and API names; naive chunking splits a table from its header; and a model handed weak context will happily fill the gaps with confident fiction. The hard problems in RAG live in retrieval quality and grounding discipline, long before the model is even called.

  • Exact termserror codes and API names
  • Intact chunkstables keep their headers
  • Grounded answersno confident fiction

The solution

A pipeline where every stage earns its place.

Semantic chunking keeps sections and their headers together. Hybrid retrieval combines dense vector similarity with keyword search so exact identifiers aren't lost. A reranker reorders candidates by true relevance to the query rather than raw embedding distance. The generation prompt is strict: answer only from the provided context, cite the chunk for every claim, and refuse when the context doesn't support an answer. A small eval set keeps retrieval honest as the corpus grows.

Architecture

Two pipelines, one index.

An ingest pipeline and a query pipeline sharing one index. Retrieval quality is treated as the product; generation is the last, most constrained step.

Ingest
  • Semantic chunking
  • Header-aware splitting
  • Embedding + metadata
Index
  • pgvector dense index
  • Keyword / full-text index
  • Per-document access scope
Retrieve
  • Hybrid dense + sparse
  • Reranking
  • Top-k with scores
Generate
  • Grounded-only prompting
  • Inline citations
  • Refuse when unsupported

One request, start to finish

  1. Docs are chunked, embedded, and indexed
  2. A question runs hybrid retrieval
  3. Candidates are reranked by true relevance
  4. The model answers only from retrieved context
  5. Every claim links back to its source chunk

Decisions

Four calls and why I made them.

Call 01

Hybrid retrieval, not pure vectors

Vector search alone kept missing exact strings — an error code, a function name, a SKU. Adding keyword search back in and fusing the rankings recovered the precise-term matches that embeddings blur, without losing semantic recall.

Call 02

Rerank before you generate

Embedding distance is a rough proxy for relevance. A dedicated reranking pass over the top candidates consistently pulled the actually-useful chunk into the context window, and it moved answer quality more than any prompt change I tried.

Call 03

Citations are mandatory, refusal is allowed

An answer without a source is a liability in a knowledge tool. Requiring a citation per claim — and letting the model say "not in the docs" — is what makes the output trustworthy instead of merely fluent.

Call 04

An eval set from day one

Retrieval quality silently rots as the corpus grows and chunking changes. A small golden set of question/expected-source pairs turns 'it feels worse' into a number I can regression-test against.

Reflections

What I took away.

RAG is a retrieval problem wearing a generation costume

Every time an answer was wrong, I instinctively reached for the prompt. Almost every time, the real fix was upstream: the right chunk never made it into context. Once I started treating retrieval as the product and generation as a thin, constrained layer on top, quality stopped being mysterious and started being measurable.

Chunking is where quality is won or lost

The least interesting-sounding decision — how to split a document — had the biggest effect on everything downstream. Splitting a table from its header, or a heading from its section, poisons retrieval in ways no reranker recovers. Getting chunking right did more for accuracy than swapping models ever did.

Outcomes

What shipped.

  • RetrievalHybridDense + keyword + rerank
  • AnswersCitedSource per claim
  • UngroundedRefusedNo context, no answer
  • QualityEval-gatedGolden retrieval set

Stack

Technology

AI

  • Claude API
  • Embeddings
  • Reranking
  • Grounded prompting

Retrieval

  • Hybrid search
  • Semantic chunking
  • Top-k scoring

Data

  • Supabase
  • Postgres
  • pgvector
  • Full-text search

Quality

  • Golden eval set
  • Citation checks
  • Regression tests

Features

What it does

  • Semantic, header-aware chunking that keeps sections intact for meaningful retrieval
  • Hybrid retrieval fusing dense vector similarity with keyword search so exact terms aren't lost
  • A reranking pass that reorders candidates by true query relevance before generation
  • Grounded-only generation: answers cite the exact source chunk and refuse when context is insufficient
  • pgvector index on Postgres with per-document access scoping for multi-user corpora
  • A golden evaluation set that regression-tests retrieval quality as documents and chunking evolve

Keep reading

More case studies

Your project

Building something like this?

AI, voice, privacy constraints or a mobile app that can’t afford to break. Free 30-minute call, no pitch, clear next steps either way.

Email
Back to
All work