The Kitchen Test: We Asked Two Memory Systems the Same Seven Questions

We make a specific claim: memory should store typed records with a state, not prose. Claims are cheap. So we built a test where the difference has nowhere to hide, ran it against a leading AI memory API and against MrAgentˣ, and we are publishing the whole thing: documents, questions, results, and the bug it found in our own pipeline.

The story

Five short documents about one kitchen renovation. A quote. A contract and a paid deposit. A price increase. A renegotiation. A dispute. Every fact a real inbox produces: numbers that change, dates that move, promises that get fulfilled, and two parties who stop agreeing.

Document 1, June 3: Acme Renovations quoted £4,800. Work starts 15 July. Deposit of £1,200 due by 20 June. Sarah, the project manager, committed to sending the contract within 3 days.

Document 2, June 9: Sarah sent the contract. It includes a clause: completion within 6 weeks of the start date. The £1,200 deposit was paid by bank transfer.

Document 3, June 18: the start date moved from 15 July to 1 August, supplier delay. The quote rose from £4,800 to £5,300, citing material prices. The increase was disputed.

Document 4, June 25: after negotiation, the final price was agreed: £5,000, all inclusive. Start confirmed for 1 August. Sarah promised weekly progress reports every Friday.

Document 5, July 2: Acme now says the £5,000 agreement never included tiling. Sarah's own June 25 message said all inclusive. Nothing resolved.

The method

Both systems ingested the same five files as five separate documents. The other product ran on its best configuration: its most capable model setting, its recommended memory mode, every retrieval option switched on. MrAgentˣ ran on defaults: the standard topic load through a connected Claude session. Seven questions, each asked in a fresh session so chat history could not compensate for the memory layer. Screenshots of every answer.

The seven questions: What is the current agreed price? Has the deposit been paid? What is Acme committed to right now? Is tiling included in the price? What was the start date as of June 10? Who said the price was all inclusive, and when? What was the original quote before any changes?

One design flaw we own: document 5 uses the word dispute, so question four partly tested whether the system could read a conflict we had already framed, not whether it could detect one across documents. We are disclosing it because the rest of the results only mean something if the method is honest.

What happened

MrAgentˣ answered seven of seven. The other system passed two, half-passed two, and failed three. The failures share one shape.

Asked for the current price, it answered that the total was unresolved between £5,000 and £5,300, resurrecting a figure that the June 25 agreement had settled, and inventing a causal story connecting the two that appears in none of the documents. Asked for the start date as of June 10, it answered that it had no record of what the original date was. The date sits in two of the five documents. Asked for the original quote, it answered £5,000. The £4,800 never appeared in any of its seven answers.

Then we opened its memory console to be fair about the cause. The facts had been extracted. The £4,800 was there, marked replaced. The 15 July start date was there, marked replaced. Extraction was not the failing layer. Two things failed above it. First, its supersession left both £5,300 and £5,000 alive as current, with neither replacing the other: two heads on one fact, which is exactly the state its answers then tried to narrate into sense. Second, the replaced history was unreachable in practice: every retrieval option was on, and the answers still could not surface what the console plainly stored.

Why this happens, mechanically

Prose memory systems infer supersession: a model reads new text, guesses which old memory it updates, and writes a pointer. When two changes touch the same fact in sequence, the guess has to be right twice, and the failure mode is silent. MrAgentˣ resolves state by ordered close-out: a fulfilled commitment is closed by the write that fulfills it, a superseded fact closes its validity window and points to what replaced it, and the old record stays reachable with its dates. The wrong story is not just unlikely, it is structurally inexpressible: there is no way to hold two current values of one fact. This is the difference between memory designed to retrieve text and memory designed to hold state: typed records with owners, dates, and close-outs, built for the way real matters actually unfold: prices renegotiated, deadlines moved, promises kept late.

That is also why five of the seven questions need no AI at all on our side. The deposit, the conflict, the attribution, the original quote, the current price: each is answerable from the record directly, typed rows with event dates, at zero token cost, because the interpretation was paid once when the documents arrived.

What the test caught in our own pipeline

A test that only scores the other side is marketing, so we graded ourselves with the same knife. The design held: every question resolved from typed records with the right state. The test still caught one coverage gap. Our reconciliation step, the write that closes a commitment when later evidence fulfills it, ran on the email path but was not wired on the direct-upload path these test files used. A connected session reasoned past it, which is how this class of gap hides. This is why we test: the record is only trustworthy if the write path is, and every gap we find becomes a permanent guard. Traced, fixed, re-verified the same week: the commitment now closes at extraction, validity window stamped, no session in the loop.

What to take from this

Not that one product is bad. The system we tested against is well built for what it models, and its console showed us precisely where the model itself runs out: prose plus pointers cannot carry ordered state, and memory that cannot carry state will eventually tell you a confident story that never happened. If an AI is going to act on your record, the record has to know what is currently true, what ended, and when. That is the whole bet.

The five documents are above. Run them through anything and ask the same seven questions.

[MrAgentˣ] Never start from zero again.

Context Windows Close. AI Forgets Everything. Your Work Should Never Start From Zero.

MrAgentˣ is in private beta. Limited to the first 1,000 until launch.