LangChain finds linked evidence improves AI answers in a synthetic insurance test

Giving an AI agent a summary of documents helped it connect scattered rules. Giving it explicit paths back to the evidence helped further. In an October 28 evaluation, LangChain says adding those links to its OpenWiki memory system improved accuracy by 27% and cut cost per correct answer by 58% on cross-document questions in a synthetic insurance library.

A map back to the source

OpenWiki is LangChain’s open-source tool for compiling source material into linked, cited wiki pages. Those pages preserve explanations and relationships an agent would otherwise need to reconstruct through search. Each assertion links to supporting source lines, so the wiki is more than a collection of standalone summaries.

But a written citation still leaves an agent with navigation work. LangChain added an evidence graph: a map of explicit relationships between wiki claims, pages and source passages. Rather than searching again for a connection, the agent can follow a link that identifies what the connection means.

1. Make passages addressable. The system splits each source document into paragraphs and gives each paragraph a unique identifier. Claims can then point to exact supporting passages instead of whole documents.
2. Label the connections. Links identify relationships such as a claim citing a paragraph, a section referring to another document, or a newer document superseding an older one.

Two tools let the agent use this map: locate finds a starting point, while expand follows connections to evidence and related documents. LangChain says graph construction took under a second and required no additional model calls. The links still depend on the accuracy of the wiki’s model-generated citations.

Testing rules that cross document boundaries

The company built 111 synthetic insurance documents around a ledger of 972 facts. An AI model generated the surrounding prose, and automated checks verified that required facts appeared. The library deliberately included outdated versions, contradictions and indirect reference chains, making it harder to answer by finding one relevant passage.

LangChain compared keyword search over the original documents with access to the compiled wiki, keeping the model, instructions, documents and questions constant. It then evaluated the graph-enhanced wiki. The test included 80 simple lookups and 50 questions requiring connections across documents, with each asked three times.

On cross-document questions, the accuracy score rose from 0.10 with search to 0.52 with the wiki. Yet the wiki’s cited evidence overlapped by only 0.25 across repeated runs. That measure tracks whether answers cite the same evidence—not whether they are correct. The graph improved both accuracy and repeatability without making retrieval fully deterministic.

The savings depend on reuse

For connected questions, the wiki alone reduced cost per correct answer from about $0.55 to $0.19 before the graph’s additional savings. Those figures cover retrieval and answering, not memory construction or updates. Compiling the initial library cost $4.15; adding 17 documents later cost about $0.09 each.

Simple lookups offered little reason to add the wiki. Search scored 0.84 for accuracy against the wiki’s 0.83, while answer time increased from nine to 12 seconds. LangChain argues that repeated questions spanning the same sources offer the stronger business case: reuse can offset the expense of compiling and maintaining memory.

The company’s next proposed step is to compile and retain only the context agents need for cross-source questions, rather than entire repositories. That would target the upfront work left outside this benchmark’s answer-cost figures.