Lemmalog Turns Agent Memory Into Datalog State, Cutting LongMemEval Context 38-Fold

Long-running AI agents often face a subtle failure: an old observation is disproved, yet its downstream conclusions survive in the model’s reasoning. Lemmalog is an experimental memory engine designed to make those conclusions retractable, treating an investigation as maintained logical state rather than a transcript an LLM must repeatedly reread.

Jordy Zomer built the system for LLM-assisted vulnerability research, where agents may work through source code, debugger output and hypotheses over hours. The core split is deliberate: the LLM extracts messy material into structured observations, while a Datalog engine applies explicit rules to determine what follows from them.

From recalled notes to supported conclusions

The distinction is more than a storage choice. Semantic retrieval can answer which past material seems relevant to a question. It does not inherently establish whether a statement was later disproved or which other conclusions relied on it. Lemmalog’s aim is the second task: determining what remains supported now.

What changes when a fact changes

• An LLM can turn an observation, such as a pointer relationship found in debugging, into a structured fact for the engine to use.
• Rules derive conclusions from those facts. When an input fact is retracted, Lemmalog can invalidate the conclusions that depended on it.
• A conclusion with another independent derivation remains in place, rather than disappearing merely because one supporting path failed.
• Provenance and validity intervals let the system show why a conclusion was supported and distinguish a current belief from an earlier one.

~104,000Full-context prompting

Zomer reported roughly 104,000 tokens passed to the answering model under full-context prompting.

~2,700Lemmalog prompting

Zomer reported roughly 2,700 tokens passed per question with Lemmalog, about 38 times less context than the full-context setup.

A smaller prompt is not the same as a stronger memory system

Zomer tested Lemmalog on LongMemEval and LoCoMo using standardized reader models and evaluation procedures; Claude Sonnet 4.6 handled extraction during ingestion. On the described LongMemEval split, it posted 0.463 plus or minus 0.010 F1 and 0.575 plus or minus 0.004 accuracy across three runs.

That score sits below published results cited for PropMem at 0.550 F1 and SimpleMem at 0.480, but above the cited Full Context result of 0.222. The performance profile fits the design goal: a representative knowledge-updates result reached 0.579, while the reported multi-session score was 0.211, versus 0.582 for PropMem.

The deterministic layer starts after an uncertain step

Lemmalog does not eliminate hallucinations at the point where they enter the system. If the extraction model misunderstands evidence or fails to convert an event into a fact, the rule engine has no correct input to maintain. The project also combines its maintained state with hybrid retrieval, rather than positioning logical state as a total replacement for semantic search.

The results are project-reported and have not been independently validated. They also come from conversational-memory benchmarks, not a real vulnerability investigation—the setting that motivated the tool. The consequential next test is whether provenance and automatic retractions keep abandoned exploit paths from returning during that kind of extended work.