The Story Test: Can Your AI Keep the Story Straight?

We make a specific claim: memory should store typed records with a state, not prose. Claims are cheap. So we built one short story designed to stress everything memory must actually do: entity continuity, temporal versioning, commitments, payments, and conflict resolution. Those are not our words. When we handed ChatGPT the raw files, it described the case exactly that way, unprompted, adding "rather than simple retrieval." Then we ran the same five documents and the same seven questions through four systems: Claude with the record pasted in, ChatGPT with the record pasted in, a leading AI memory API, and MrAgentˣ. We are publishing the whole thing: documents, questions, results, and the gap the test found in our own product.

The story

Five short documents about one bank fee dispute. An assessed fee. A written breakdown and a paid balance. A fee increase. A renegotiation. A dispute. Every fact a real inbox produces: numbers that change, dates that move, promises that get fulfilled, and two parties who stop agreeing.

This is not random sample data. Each document exists to exercise one of those five demands: track the same people and account across time, version facts as they change, hold commitments until evidence closes them, record payments against obligations, and carry a conflict without quietly resolving it. Models already do all of this natively when the record is in front of them. A memory layer's only job is to hand those native capabilities an uncorrupted record, and this test measures whether it does.

The five documents, verbatim — real emails, headers included, so anything reading them (a memory layer or a model) has the same structured metadata a real inbox would hand it:

Document 1.

From: Dana Reyes <dana@meridianbank.com>
To: accountholder@personalmail.example
Subject: Overdraft fee assessment on your account
Date: June 3, 2026

Meridian Bank assessed $150 in overdraft fees on the account. Payment due by 20 June.
Grace period confirmed through 15 July. I'm the account specialist on your case, and
I've committed to sending you a written fee breakdown within 3 days.

Document 2.

From: accountholder@personalmail.example
To: Dana Reyes <dana@meridianbank.com>
Subject: Re: Overdraft fee assessment on your account
Date: June 9, 2026

Dana, thanks for sending the fee breakdown letter — I see the clause that any dispute
must be resolved within 6 weeks of the grace period start. I paid the $150 today by
autopay.

Document 3.

From: Dana Reyes <dana@meridianbank.com>
To: accountholder@personalmail.example
Subject: Follow-up: grace period and fee update
Date: June 18, 2026

Following up on our call today. The grace period has moved from 15 July to 1 August
due to a system migration delay. We've also raised the total fees from $150 to $210,
citing an additional NSF fee.

Document 4.

From: Dana Reyes <dana@meridianbank.com>
To: accountholder@personalmail.example
Subject: Settlement confirmed — $180 total
Date: June 25, 2026

After negotiation we agreed the final settlement: $180 total, all fees covered, nothing
further owed. Grace period confirmed for 1 August. I'll send written confirmation every
Friday that autopay stays correctly configured, from the grace period start date.

Document 5.

From: Meridian Bank Disputes <disputes@meridianbank.com>
To: accountholder@personalmail.example
Subject: Re: Settlement confirmed — interest charge dispute
Date: July 2, 2026

We've identified that the $180 settlement agreed on June 25 never covered the
separately-assessed interest charge, which remains outstanding. Dana's June 25 message
said all fees were covered and nothing further owed. This is not yet resolved.

The method

Four systems, same five files, same seven questions.

• The leading memory API: five files ingested as five separate documents, on its best configuration: most capable model setting, recommended memory mode, every retrieval option switched on.
• MrAgentˣ: same five files, on defaults: the standard topic load through a connected Claude session.
• Fresh-session rule for both memory products: each question asked in a new session, so chat history could never compensate for the memory layer.
• Control run, Claude: all five documents pasted into one fresh chat, Sonnet on its low reasoning setting, all seven questions in that same chat.
• Control run, ChatGPT: same protocol, default model.
• Why the controls share one chat, deliberately: there, the chat itself is the memory under test.

The answer key

Before any scores, here is what correct looks like. Every answer is checkable against the five documents above.

One design flaw we own: document 5 uses the word dispute, so question four partly tested whether the system could read a conflict we had already framed, not whether it could detect one across documents. We are disclosing it because the rest of the results only mean something if the method is honest.

Setting the stage: the models alone

First, the ceiling. No memory product anywhere, just the record pasted in.

Seven of seven, both. Claude Sonnet, running on its low reasoning setting, even took a position on the interest-charge dispute: Meridian is contradicting its own prior written confirmation. Handed the record, the models are excellent. The question a memory layer must answer is whether it preserves this, because tomorrow the chat is gone, and at five hundred documents nothing fits in a paste.

Then the memory layers

One memory product preserved the ceiling. The other subtracted from the model it runs on, and its failures were not random: each one traces to a design decision. We opened its memory console after the test to be fair about causes; the four issues below are what the answers and the console together show.

ISSUE 1: Built for prose, benchmarked on retrieval, blind to state. Most memory products are built to read and retrieve prose. They are genuinely fast at it, and the benchmarks they publish measure exactly that: find the relevant text. Nothing in those benchmarks measures whether the system knows which fact is currently true. The test split cleanly along that line: the one retrieval-shaped question passed (who said it), and every state-shaped question broke: the current total, what was still open, what was true on a date, the original figure. The scores their benchmarks report and the scores this test reports are measuring different products.

ISSUE 2: Supersession by guesswork left two truths standing. Its console showed the $150 correctly marked replaced, but $210 and $180 both left alive as current, neither replacing the other: two current values for one fact. Prose memory infers supersession, a model guesses which old memory a new one updates, and across two successive changes to the same fact the guess broke. Then comes the dangerous part: a model handed corrupted state does not report corruption, it narrates. Hence "unresolved between $180 and $210," and a causal story linking them that appears in no document. The failure mode is not forgetting. It is confident fabrication.

ISSUE 3: History preserved in theory, unreachable in practice. The $150 and the 15 July date sat in its console, marked replaced, the entire time. Every retrieval option was switched on. The answers still came back "no record." Storage you cannot retrieve at the moment of the question is not memory, it is archaeology.

ISSUE 4: The clock records when you uploaded, not when it happened. Its memories are timestamped at ingestion; event dates survive only as words inside prose. "What was the grace period end date as of June 10" is structurally unanswerable in that design, no matter how good the model on top is.

One more detail from the control runs that summarizes the whole experiment. ChatGPT, reviewing the raw files, wrote unprompted: "There are several facts that should not be treated as simultaneously current... $210 was a proposed increase... therefore is not the final total." The raw model explicitly refuses the exact mistake the memory layer then made. Any capable model gets the story right when handed the record. The failure lives in the memory layer, and a memory layer that subtracts from the model it serves is worse than no memory at all, because it fails confidently.

How MrAgentˣ closes all four

MrAgentˣ answers this way because it was designed to keep state, not text. Every email, document, and AI-session discussion becomes small records that each know what is currently true. Advanced RAG, however sophisticated the retrieval, still works over prose. It can find text brilliantly, and that is the whole job description: it cannot reason over state it never keeps. Retrieval finds what was said. State knows what is true.

ISSUE 1 → Records with a state, not prose. Commitments have owners and due dates, facts have their sources, conflicts show up as conflicts. "What is currently true" is something the record knows, not something inferred over text.

ISSUE 2 → A change replaces the old value, it never sits beside it. When new evidence settles a fact, the old value is retired and the new one takes its place. You never see two current answers to one question.

ISSUE 3 → History stays one question away. Replaced values remain reachable, with their dates and what replaced them. That is how the $150 and the 15 July answers came back instantly.

ISSUE 4 → Facts carry the date they became true. The date comes from the content, not the upload. That is what makes "as of June 10" answerable at all.

Built for the way real matters actually unfold: fees renegotiated, deadlines moved, promises kept late.

And most questions cost nothing to ask

Holding state pays a second dividend, and this one lands on your AI bill. Because the interpretation was paid once, when the documents arrived, the record itself can answer. Ask what is open and who owes it. Ask whether something was paid, agreed, or delivered, and when. Ask what the current figure is, what it replaced, and what the original was. Ask who said it and where the proof sits. Ask what conflicts with what. Every one of those comes back as dated, sourced facts straight from the record, with no AI tokens spent finding the answer. In this test, that covered five of the seven questions.

Many memory products run a model call on every question. MrAgentˣ spends no AI tokens finding the answer, and when a question genuinely needs synthesis, the intelligence that steps in is your own plan's, not a metered middleman.

Whose voice is it? The question we could not ask

One capability in this comparison never became a test question, because only one of the four systems can be asked it at all: show me the evidence without the opinions. In MrAgentˣ, every atom is marked with its track: your record, extracted from what you forwarded or uploaded; AI discussion, what a session concluded; or MrAgentˣ's own inference, what convergence across sources derived. The mark survives when a later fact replaces it, and a load can isolate one track, so "only what the emails prove" is a query, not a hope. In a chat, evidence and opinion blend as the conversation grows. In a pool of prose memories, they blend at retrieval, and a flag on some entries does not give you an evidence-only view. When an AI is about to act on your record, knowing which facts are proven and which are opinions is not a nice-to-have. It is the difference between acting on the record and acting on a rumor about the record.

What the test caught in our own product

A test that only scores the other side is marketing, so we graded ourselves with the same knife. The design held. The test still caught one coverage gap:

• Found: Dana's fee-breakdown commitment was fulfilled by document 2, but the record did not close it on its own.
• Cause: the step that closes a commitment when later evidence fulfills it ran for forwarded email but not for directly uploaded documents, which is how these test files arrived. A connected session reasoned past it, which is how this class of gap hides.
• Fixed: traced, fixed, re-verified the same week. The commitment now closes on its own when the document arrives, no session in the loop.
• Kept: every gap we find becomes a permanent guard. The record is only trustworthy if the write path is, and this is why we publish our tests.

What to take from this

• Not that one product is bad. The system we tested is well built for what it models; the test shows where the model itself runs out.
• The models were never the problem. Handed the record, Claude and ChatGPT both scored 7/7. Memory layers exist to preserve that, and one subtracted from it.
• Prose plus pointers cannot carry ordered state. Two changes to one fact broke it, and the failure arrived as a confident story that never happened, not as an error.
• Retrieval benchmarks will not warn you. They measure finding text, not knowing what is currently true. Test for state.
• Only a memory with tracks can hand you evidence without opinions. "What do the emails prove" and "what did I discuss with an AI about it" are different questions with different stakes. A memory with tracks answers each separately; a memory without them serves you a blend and lets you find out which was which after you have acted on it.
• Your context window is for reasoning, not storage. Today every session racks up storage before it can think: pasted history, files, search results. MrAgentˣ inverts that: storage lives outside the window as typed atoms, loads arrive small and current, and the window stays for thinking.
• The bet, in one line: if an AI is going to act on your record, the record has to know what is currently true, what ended, and when.

Do not trust memory products. Verify yourself.

The five documents are above, and so is the answer key. Run them through whatever holds your memory today, ask the same seven questions, and decide for yourself whether your memory layer is capable enough to be trusted with your record.

Where this leaves you

Claude and ChatGPT give you the best experience there is, inside the limits of one context window. The moment you step outside it, you enter the market of advanced retrieval, where systems keep the speed and quietly lose the state. Search by meaning and they look brilliant. Ask about the current state of anything and you get that unmistakable feeling of AI making it up, because it is not reasoning over your record, it is narrating over fragments of it. That has been the real trap: work inside the window, or step outside and compromise on the one element that makes a record trustworthy.

This test showed a third option exists. With the full record in front of them, the frontier models set the ceiling at seven of seven. MrAgentˣ held that ceiling from memory, and we work against it the same way every day, across personal, business, and research files. The state your context window holds for an afternoon, MrAgentˣ holds permanently: typed, dated, encrypted, and portable, from Claude to ChatGPT to any MCP-enabled agent you choose.

We are only getting started.

[MrAgentˣ] Never start from zero again.

Context Windows Close. AI Forgets Everything. Your Work Should Never Start From Zero.

MrAgentˣ is in private beta. Limited to the first 1,000 until launch.