Agent Memory Challenge Will Open a Shared Test for Long-Running AI Agents

The Agent Memory Challenge will open its second cycle on September 20 with a shared evaluation for systems that give AI agents long-term memory. The challenge covers text, coding and multimodal work, with rules intended to test whether an agent retrieves evidence that is useful now rather than merely recalling something it saw before.

A second pass at a growing technical problem

The Agent Memory Leaderboard is framing the problem narrowly: long-running agents must distinguish an earlier fact or decision from one that has since changed. Pulling stale context into a later task can be worse than finding no context at all, according to the organizer’s description of the challenge.

Cycle 2 follows the program’s first leaderboard, published August 12. That cycle drew 136 team applications, and 67 representative memory systems completed official evaluation. The new cycle is not a claim to identify one universally best memory system; results apply to the submitted version, track, suite and evaluation conditions.

Three ways memory can fail

The textual track covers long conversations and cross-session history, including fact recall, temporal events, personalization and rule following. Its new Streaming Memory component makes Add and Search calls as events unfold, so systems must use information that was available at the appropriate moment rather than search a finished archive.

What the other tracks put under pressure

• Coding Memory includes 150 software-engineering tasks, each run with relevant history and noisy history, creating 300 task-condition units. It tests whether a system can separate useful engineering experience from distracting repository context.
• Multimodal Memory tests systems that write, retrieve and use text-and-image memory. Participants can process original images, captions or both, and must disclose which media their systems use.

A strict contract around the memory layer

The standardization comes with operational constraints. Entrants need stable, publicly reachable APIs and per-track smoke tests; submitted versions must remain fixed for formal evaluation. The rules also require user isolation, immediate searchability after a synchronous Add call, and idempotent request handling so network retries do not create duplicate memory.

Those requirements leave room for sharply different designs, including raw-history retrieval, summaries, structured facts, graphs and temporal reasoning. But they aim to hold the downstream answering and scoring process constant, making a result easier to interpret as a test of the memory layer rather than its surrounding stack.

The next test opens September 20

The challenge is open globally to research groups, open-source maintainers, commercial teams, individual developers and cross-organization teams. Each track separates open-source methods from commercial products; eligible open-source entrants compete for a total RMB 150,000 prize pool, while commercial products receive separate rankings without prize eligibility.

Registration, API integration, smoke tests, full evaluations and review are scheduled through October 31, with evaluations closing November 4 and official results planned for mid-November. The unresolved question is whether a shared boundary can capture the messier reality of memory: not just preserving experience, but recognizing what should no longer guide an agent’s next action.