Retrieval demonstrations work on clean public documents. Company documents are scanned, inconsistent, contradictory and confidential. Notes on making source-grounded answering survive contact with a real internal archive.
A retrieval demonstration on a public corpus is close to a solved problem. Point a standard pipeline at a set of well-formed documents, ask questions with answers in them, and the results are good enough to be unremarkable.
The same pipeline pointed at a company archive produces answers that are fluent, plausible and unreliable, and the failure is not in the model. It is in every assumption the pipeline makes about its documents.
Public benchmark corpora are clean, consistent, self-contained, and written to be read. Internal archives are none of those. They are scanned at varying quality, written by people who assumed context the reader would have, full of documents that supersede each other without saying so, and organised by whatever filing convention was in force that decade.
A pipeline tuned on the first will report high confidence on the second and be wrong in ways nobody catches, because the answer sounds exactly like the answers that were right.
Fixed-size chunking assumes prose. A scanned form is not prose. A table split across a page boundary loses its header, and a chunk containing the numbers without the header is not merely useless, it is actively harmful: it retrieves well and means nothing.
What works is layout-aware segmentation that treats a table, a form region and a paragraph as different objects, and carries the header context into every fragment of a table it produces.
Internal archives are full of contradictions, and most of them are not errors. They are the same policy in three revisions, with no field anywhere saying which is current.
Retrieval that returns the highest-similarity passage will confidently return a superseded one. The mitigation is not better embeddings. It is extracting effective dates and version markers as first-class metadata during ingestion, and treating a contradiction between two retrieved passages as a signal to surface both with their dates rather than to pick one.
Every claim in an answer traces to a page. Not to a document, to a page, and where possible to a region of that page, with the image available beside the text.
This is a hard constraint rather than a feature. In an internal system, an answer nobody can verify is an answer nobody will act on, and a system nobody acts on is a system that quietly stops being used in the third week.
The value is not that the system answers. It is that a person can check the answer in five seconds instead of forty minutes.
Most of this work happens where documents cannot leave. That rules out hosted APIs and it changes the architecture: models sit behind an interface, weights are deployed as artefacts, and the pipeline has to be installable and upgradable by an administrator with no external package access.
The discipline this imposes is worth having even where it is not required. A pipeline that can swap its model backend without a rewrite is a pipeline that survives the next eighteen months of the field.
Benchmark scores are the wrong measure here. The right one is a set of questions the business actually asks, collected from the people who ask them, with answers agreed in advance by someone who knows.
Fifty of those are worth more than any public leaderboard, and building the set is usually the first week of the engagement rather than the last.
A large share of the questions people bring to a document system are not document questions at all. "How many of these did we buy last year" is a query against a schema, and retrieval will answer it badly by finding a document that happens to mention a number.
Recognising that split early is most of the value of the discovery phase. Retrieval for the things that genuinely live in prose, a schema for the things that should have been records all along.
Every operation has documentation that stopped being true at some point. This is a method for finding that point: extracting entities from records, building a graph of what should be, and reading the divergences against what is.
The pilot worked. A year later nothing is in production. The failure is almost never model quality. It is data nobody owns, permissions nobody granted, and a workflow nobody changed.
A seaweed supply operation running from Korean farms and factories to U.S. warehouses kept its truth in four disconnected spreadsheets. Discovery found 25 places the sources disagreed and 34 questions nobody had answered.