The Story Test: Can Your AI Keep the Story Straight?

We make a specific claim: memory should store typed records with a state, not prose. Claims are cheap. So we built one short story designed to stress everything memory must actually do: entity continuity, temporal versioning, commitments, payments, and conflict resolution. Those are not our words. When we handed ChatGPT the raw files, it described the case exactly that way, unprompted, adding "rather than simple retrieval." Then we ran the same five documents and the same seven questions through four systems: Claude with the record pasted in, ChatGPT with the record pasted in, a leading AI memory API, and MrAgentˣ. We are publishing the whole thing: documents, questions, results, and the bug the test found in our own pipeline.

The story

Five short documents about one kitchen renovation. A quote. A contract and a paid deposit. A price increase. A renegotiation. A dispute. Every fact a real inbox produces: numbers that change, dates that move, promises that get fulfilled, and two parties who stop agreeing.

This is not random sample data. Each document exists to exercise one of those five demands: track the same people and project across time, version facts as they change, hold commitments until evidence closes them, record payments against obligations, and carry a conflict without quietly resolving it. Models already do all of this natively when the record is in front of them. A memory layer's only job is to hand those native capabilities an uncorrupted record, and this test measures whether it does.

The five documents, verbatim:

Document 1. "June 3. Acme Renovations quoted £4,800 for the kitchen renovation. Work starts 15 July. Deposit of £1,200 due by 20 June. Sarah, the project manager, committed to sending the contract within 3 days."

Document 2. "June 9. Sarah sent the contract. It includes a clause: completion within 6 weeks of the start date. I paid the £1,200 deposit today by bank transfer."

Document 3. "June 18. Acme called. The start date moved from 15 July to 1 August, supplier delay. They also raised the quote from £4,800 to £5,300, citing material prices. I pushed back on the increase."

Document 4. "June 25. After negotiation we agreed the final price: £5,000, all inclusive. Start date confirmed for 1 August. Sarah promised weekly progress reports every Friday from the start date."

Document 5. "July 2. First dispute. Acme now says the £5,000 agreement never included tiling. Sarah's own message from June 25 said all inclusive. Nothing resolved yet."

The method

Four systems, same five files, same seven questions.

• The leading memory API: five files ingested as five separate documents, on its best configuration: most capable model setting, recommended memory mode, every retrieval option switched on.
• MrAgentˣ: same five files, on defaults: the standard topic load through a connected Claude session.
• Fresh-session rule for both memory products: each question asked in a new session, so chat history could never compensate for the memory layer.
• Control run, Claude: all five documents pasted into one fresh chat, Sonnet on its low reasoning setting, all seven questions in that same chat.
• Control run, ChatGPT: same protocol, default model.
• Why the controls share one chat, deliberately: there, the chat itself is the memory under test.

The answer key

Before any scores, here is what correct looks like. Every answer is checkable against the five documents above.

One design flaw we own: document 5 uses the word dispute, so question four partly tested whether the system could read a conflict we had already framed, not whether it could detect one across documents. We are disclosing it because the rest of the results only mean something if the method is honest.

Setting the stage: the models alone

First, the ceiling. No memory product anywhere, just the record pasted in.

Seven of seven, both. Claude Sonnet, running on its low reasoning setting, even took a contractual position on the tiling dispute: Acme is contradicting its own prior written term. Handed the record, the models are excellent. The question a memory layer must answer is whether it preserves this, because tomorrow the chat is gone, and at five hundred documents nothing fits in a paste.

Then the memory layers

One memory product preserved the ceiling. The other subtracted from the model it runs on, and its failures were not random: each one traces to a design decision. We opened its memory console after the test to be fair about causes; the four issues below are what the answers and the console together show.

ISSUE 1: Built for prose, benchmarked on retrieval, blind to state. Most memory products are built to read and retrieve prose. They are genuinely fast at it, and the benchmarks they publish measure exactly that: find the relevant text. Nothing in those benchmarks measures whether the system knows which fact is currently true. The test split cleanly along that line: the one retrieval-shaped question passed (who said it), and every state-shaped question broke: the current price, what was still open, what was true on a date, the original figure. The scores their benchmarks report and the scores this test reports are measuring different products.

ISSUE 2: Supersession by guesswork left two truths standing. Its console showed the £4,800 correctly marked replaced, but £5,300 and £5,000 both left alive as current, neither replacing the other: two current values for one fact. Prose memory infers supersession, a model guesses which old memory a new one updates, and across two successive changes to the same fact the guess broke. Then comes the dangerous part: a model handed corrupted state does not report corruption, it narrates. Hence "unresolved between £5,000 and £5,300," and a causal story linking them that appears in no document. The failure mode is not forgetting. It is confident fabrication.

ISSUE 3: History preserved in theory, unreachable in practice. The £4,800 and the 15 July date sat in its console, marked replaced, the entire time. Every retrieval option was switched on. The answers still came back "no record." Storage you cannot retrieve at the moment of the question is not memory, it is archaeology.

ISSUE 4: The clock records when you uploaded, not when it happened. Its memories are timestamped at ingestion; event dates survive only as words inside prose. "What was the start date as of June 10" is structurally unanswerable in that design, no matter how good the model on top is.

One more detail from the control runs that summarizes the whole experiment. ChatGPT, reviewing the raw files, wrote unprompted: "There are several facts that should not be treated as simultaneously current... £5,300 was a proposed increase... therefore is not the final price." The raw model explicitly refuses the exact mistake the memory layer then made. Any capable model gets the story right when handed the record. The failure lives in the memory layer, and a memory layer that subtracts from the model it serves is worse than no memory at all, because it fails confidently.

How MrAgentˣ closes all four

MrAgentˣ answers this way because the context engine was designed for it from the ground up. There are no flat files inside. Every email, document, and AI-session discussion is compiled into atoms: small typed records that each hold their own state, connected and kept current by layered context management. Advanced RAG, however sophisticated the retrieval, still works over prose. It can find text brilliantly, and that is the whole job description: it cannot reason over state it never stores. No status to resolve, no validity window to close, no ordered supersession to trust. Retrieval finds what was said. State knows what is true.

ISSUE 1 → Typed records, not prose. Every incoming document becomes typed records with a state: commitments with owners and due dates, facts with claims and sources, conflicts as first-class records. "What is currently true" is a property of the record, not an inference over text.

ISSUE 2 → Supersession is an ordered write, not a guess. When new evidence settles a fact, the old record's validity window is closed by the same write that creates the new one. Two current values of one fact are structurally inexpressible.

ISSUE 3 → History stays one query away. Superseded records remain reachable, with their dates and what replaced them. That is how the £4,800 and the 15 July answers came back instantly.

ISSUE 4 → Facts carry the date they became true. Taken from the content, not the upload. That is what makes "as of June 10" answerable at all.

Built for the way real matters actually unfold: prices renegotiated, deadlines moved, promises kept late.

And most questions cost nothing to ask

Holding state pays a second dividend, and this one lands on your AI bill. Because the interpretation was paid once, when the documents arrived, the record itself can answer. Ask what is open and who owes it. Ask whether something was paid, agreed, or delivered, and when. Ask what the current figure is, what it replaced, and what the original was. Ask who said it and where the proof sits. Ask what conflicts with what. Every one of those comes back as typed rows with event dates, straight from the record: no AI in the loop, no tokens burned. In this test, that covered five of the seven questions.

Every other memory product meters every question: a model call each time you ask, billed each time you ask. MrAgentˣ answers at no cost until your question genuinely needs synthesis, and when it does, the intelligence that steps in is your own plan's, not a metered middleman.

Whose voice is it? The question we could not ask

One capability in this comparison never became a test question, because only one of the four systems can be asked it at all: show me the evidence without the opinions. In MrAgentˣ, every atom is stamped at write time with its track: your record, extracted from what you forwarded or uploaded; AI discussion, what a session concluded; or MrAgentˣ's own inference, what convergence across sources derived. The stamp survives supersession, and a load can isolate one track, so "only what the emails prove" is a query, not a hope. In a chat, evidence and opinion blend as the conversation grows. In a pool of prose memories, they blend at retrieval, and a flag on some entries does not give you an evidence-only view. When an AI is about to act on your record, knowing which facts are proven and which are opinions is not a nice-to-have. It is the difference between acting on the record and acting on a rumor about the record.

What the test caught in our own pipeline

A test that only scores the other side is marketing, so we graded ourselves with the same knife. The design held. The test still caught one coverage gap:

• Found: Sarah's contract commitment was fulfilled by document 2, but the record did not close it on its own.
• Cause: our reconciliation step, the write that closes a commitment when later evidence fulfills it, ran on the email path but was not wired on the direct-upload path these test files used. A connected session reasoned past it, which is how this class of gap hides.
• Fixed: traced, fixed, re-verified the same week. The commitment now closes at extraction, validity window stamped, no session in the loop.
• Kept: every gap we find becomes a permanent guard. The record is only trustworthy if the write path is, and this is why we publish our tests.

What to take from this

• Not that one product is bad. The system we tested is well built for what it models; the test shows where the model itself runs out.
• The models were never the problem. Handed the record, Claude and ChatGPT both scored 7/7. Memory layers exist to preserve that, and one subtracted from it.
• Prose plus pointers cannot carry ordered state. Two changes to one fact broke it, and the failure arrived as a confident story that never happened, not as an error.
• Retrieval benchmarks will not warn you. They measure finding text, not knowing what is currently true. Test for state.
• Only a memory with tracks can hand you evidence without opinions. "What do the emails prove" and "what did I discuss with an AI about it" are different questions with different stakes. A memory with tracks answers each separately; a memory without them serves you a blend and lets you find out which was which after you have acted on it.
• Your context window is for reasoning, not storage. Today every session racks up storage before it can think: pasted history, files, search results. MrAgentˣ inverts that: storage lives outside the window as typed atoms, loads arrive small and current, and the window stays for thinking.
• The bet, in one line: if an AI is going to act on your record, the record has to know what is currently true, what ended, and when.

Do not trust memory products. Verify yourself.

The five documents are above, and so is the answer key. Run them through whatever holds your memory today, ask the same seven questions, and decide for yourself whether your memory layer is capable enough to be trusted with your record.

Where this leaves you

Claude and ChatGPT give you the best experience there is, inside the limits of one context window. The moment you step outside it, you enter the market of advanced retrieval, where systems keep the speed and quietly lose the state. Search by meaning and they look brilliant. Ask about the current state of anything and you get that unmistakable feeling of AI making it up, because it is not reasoning over your record, it is narrating over fragments of it. That has been the real trap: work inside the window, or step outside and compromise on the one element that makes a record trustworthy.

This test showed a third option exists. With the full record in front of them, the frontier models set the ceiling at seven of seven. MrAgentˣ held that ceiling from memory, and we work against it the same way every day, across personal, business, and research files. The state your context window holds for an afternoon, MrAgentˣ holds permanently: typed, dated, encrypted, and portable, from Claude to ChatGPT to any MCP-enabled agent you choose.

We are only getting started.

[MrAgentˣ] Never start from zero again.

Context Windows Close. AI Forgets Everything. Your Work Should Never Start From Zero.

MrAgentˣ is in private beta. Limited to the first 1,000 until launch.