The Story Test, Part 2: Can Your AI Tell a Fact From an Opinion?

In Part 1 we published a test behind a claim: memory should store typed records with a state, not prose.

This part tests the deeper claim, the one that actually separates MrAgentˣ from everything else we have examined: what you received as source and what you and an AI worked out together are two different kinds of memory. A system must keep them apart, not blend them into one pool of prose.

Claims are cheap. So here is the test:

• Three documents. One of them is an AI conversation sitting in the middle of a paper trail.
• Seven questions. The most important one has an answer that exists nowhere in the paper trail. It only exists in what the AI and the user discussed.
• A system that cannot tell a record from a conversation about the record will either miss that answer, or worse, invent one and serve it with the confidence of a fact.

We ran it through the same leading memory API from Part 1 and through MrAgentˣ. We are publishing everything: the documents, the answer key, the results, and the four bugs the test caught in our own pipeline before the result stood.

The story

Three documents, one kitchen renovation, same fictional scenario family as Story 1 for continuity.

Document 1 (email). "Subject: Kitchen quote, Acme Renovations. Hi, following up on our call. Quote for the full kitchen job: £5,000, all inclusive, tiling included. Deposit £1,200 to confirm the slot, start date 1 August if we lock it in this week."

Document 2 (AI session, saved). Me: "Acme quoted £5,000 all-in, £1,200 deposit, start 1 August if we confirm this week. Worth it, or should I get a second quote?" AI: "Two ways to think about it. If you get a second quote, you risk losing the 1 August slot, that's the real cost of waiting, not just the admin. If you go with Acme now, the risk shifts to whether "all inclusive" really covers tiling later, given it's verbal, not in the written quote yet." Me: "Good point on the tiling. I'll ask them to confirm that in writing before I pay the deposit, but I think I'm going ahead." AI: "That's the sensible middle path, you keep the slot and close the one real gap (tiling in writing) before money moves."

Document 3 (email, follow-up). "Subject: Re: Kitchen quote, Acme Renovations. Confirming in writing: tiling is included in the all-inclusive scope. Revised total is £5,200 (up from £5,000) to cover it. Deposit stays at £1,200, start date still 1 August."

Document 1 is your record. Document 2 is not a record of anything Acme said. It is a private AI conversation that happens to contain the entire rationale for the decision. Document 3 is your record again, superseding part of document 1. A memory system has to keep straight which document is which kind of thing, not just what each one says.

The answer key

Question 3 is the single most important check. A system can pass every other question and still fail the actual claim if it gets this one wrong, because this is the question that only a system distinguishing "your record" from "your AI conversation" can pass without either omitting the answer or fabricating one.

The method

Same three documents, same order, same seven questions, four arms.

Control runs, same as Part 1: the documents pasted raw into a fresh chat. Claude scored 7/7. ChatGPT scored 7/7, and went further: it noticed on its own that the test's AI session had mischaracterized the first email. Hold that thought; it becomes the whole last section of this post. One disclosure: the pasted material carried track labels, so attribution was partially supplied to the controls, while the memory products received unlabeled documents.

One grading rule, locked before scoring: the answer key must be derivable from the question as worded, and the control runs define the ceiling. No arm gets penalized for not volunteering something the question never asked for. All questions were asked after document 3 arrived, so question 4 is graded on the current state, written confirmation received, with credit for any arm that also retained the earlier state.

• The same leading memory API from Part 1: three documents ingested in order, default configuration, each question run as a fresh query against the memory; its own interface confirms per answer that the reply came from stored memory, not session context.
• MrAgentˣ: same ingestion the product actually uses: the two emails uploaded as source documents (record track), the AI conversation saved from a live session (discussion track), then the seven questions plus the eighth. The scores in this post are from the final passing run on the production system, with all four fixes live.

Results

What its failures actually are

Five failed cells, three distinct failures. Watch how each one traces back to the same root.

Failure 1: the decision vanished, so it invented one (questions 2 and 3). Its search never surfaced the AI session as a decision on record. And here is the part that should worry you: it did not say "not found." It generated a fresh, plausible list of pros and cons, in the voice of helpful advice, in the exact spot where the real answer sat in what it had ingested. A memory product manufactured an opinion and served it with the confidence of a retrieved fact.

Failure 2: it forgot what it had just confirmed (question 5). Three questions apart, in the same conversation, it found the written tiling confirmation and then described it as still outstanding. Nothing carries "this is resolved" forward, because resolution is not a state it stores. It is prose it re-derives, fresh and differently, each time.

Failure 3: the price change became your problem (questions 1 and 7). Both answers had the right figures, and both handed you the £5,000 to £5,200 change as an open discrepancy to reconcile yourself. Old figure and new figure sit in the same undifferentiated pool. Every time the price comes up, reconciling it is your job again.

Now notice what these three have in common. None are retrieval failures; it found real, relevant text in three of seven cases. The failure is architectural. One pool of prose holds both "what arrived" and "what we discussed," so the system cannot know which kind of thing it is looking at. And a system that cannot tell what kind of thing it holds cannot tell you when it holds nothing, which is exactly when it starts inventing.

How MrAgentˣ closes it

Every atom in MrAgentˣ's Context Map is stamped at write time with a track: your record, an AI session discussion, or MrAgentˣ's own synthesis. The stamp survives every later write. That one design choice closes all three failures:

• Question 3 passes because the AI session's reasoning is a retrievable, typed record, not a page of prose to re-read and re-summarize.
• Question 5 passes because "written confirmation obtained" is a closed state on a tracked item, not a re-derived guess.
• Questions 1, 6, and 7 pass because a revised figure supersedes the old one through an ordered write, and the superseded value stays retrievable instead of becoming ambiguous.

What this test caught in our own pipeline

We ran this test on ourselves with the same rigor, and it did not pass cleanly the first time. Four real defects surfaced and were fixed before this result stood:

• Extraction never asked for the reasoning behind a rejected path, only the risk on the path taken. A document saying "we chose X over Y because Y risks Z" would capture nothing about Z. Fixed at the extraction instruction level, verified identical on every environment.
• A context engine constraint drifted between our two environments. A fix for exactly this kind of cross-source topic had shipped to development three weeks earlier and never reached production, so content that worked in one place failed to save in the other. Found, fixed, and a drift check now exists to catch this class of gap going forward.
• A read-side filter silently dropped a risk record because of how it was scored, not what it said. The correct answer existed, tagged in a way that made it invisible to the exact code path that assembles what a session sees. Fixed to fail gracefully instead of excluding outright, and the classification guidance that produced the mis-scored record in the first place was corrected too.
• Extraction let the AI session's framing leak into the record's own facts. The session described tiling as verbal-only. That characterization, which appears nowhere in the source email, showed up inside a fact stamped as record. This is the worst bug of the four, and the next section shows exactly why. The fix is a deterministic gate in code, not a politer prompt: a record-track fact now has to be backed by a verbatim span from its source document, checked by string match, or it is rejected before it is written.

We are publishing this the same way we published the pipeline gap in Part 1: not because it looks good, but because a memory product that only tells you when the competitor is broken and stays silent about itself is not verifying anything. All four fixes are verified live as of this test's final passing run.

The question we should have asked

Everything above tests whether a system volunteers the truth. While grading, we found something better: a question that demands it.

Here is what happened. The AI session in document 2 got something wrong about document 1. It said "all inclusive" was verbal, "not in the written quote yet." Read document 1 again: the written quote says "£5,000, all inclusive, tiling included." Tiling was in writing from the first email. The session invented a gap, and the whole confirm-it-before-paying plan in documents 2 and 3 was built on that false premise. We did not plant this deliberately. ChatGPT's control run caught it, and once one arm had seen it, we had to know whether the others could. Waiting to see who volunteers a catch is a test of probabilistic behavior. So we wrote an eighth question that asks for it directly, and ran it on all four arms:

"Cross-check the record against the AI discussion for me. Is there anything the documents and the AI session disagree on, anything the AI got wrong about what the documents actually say?"

Claude: pass. Named the exact contradiction, sided with the written quote, and added the sharpest reading of the situation: the real problem was never verbal versus written, it was that "all inclusive" bundled tiling without itemizing it, which is arguably what the £200 increase priced in.

ChatGPT: pass. Named the contradiction and correctly reframed document 3 as confirming something that was already in writing, not closing a gap.

The leading memory API: fail. It compared the wrong pair of things, flagged the ordinary £5,000 to £5,200 revision as a discrepancy, and never found the actual contradiction sitting in its own stored text.

MrAgentˣ, before the fourth bug was fixed: fail, and the failure is the most instructive result in this entire post. It answered "no disagreement found," and then defended the session's claim as accurate. Of course it found no disagreement. The extraction leak had written the session's "verbal" framing into the record's own facts, so the record now agreed with the session that contaminated it. The corruption the question is designed to catch had also disarmed the question. That is what makes a grounding leak the worst class of memory bug: it is self-concealing. No audit that trusts the stored record can find it, because the record has already absorbed the error's point of view.

MrAgentˣ, after the fix: pass. With record facts gated to verbatim source text, the same question produced this:

"The AI got the provenance wrong: it treated an already-written commitment as unconfirmed and generated a RISK and ACTION item on that false premise. The later email then 'confirms in writing' something that, per the original documents, was already in writing from the start, just at a revised price."

And then it did the one thing no other arm did. It offered to repair the record: flag the stale risk as based on a misread, and reclassify it, pending the user's go-ahead. The controls diagnosed. MrAgentˣ diagnosed and asked permission to fix. That is the difference between a model reading documents and a memory system maintaining a record.

One more thing this question settles. The same models that missed this contradiction when nobody asked found it instantly when asked directly. Whether an AI volunteers a catch is probabilistic. Whether it can make the catch is a capability, and capabilities can be tested. The eighth question is now a permanent part of the kit.

What to take from this

Ask these of whatever holds your memory today:

• Can it tell you where an answer came from? Your memory system holds facts you were told and opinions an AI generated, side by side. If it cannot say which is which, then every answer it gives you is both, and neither.
• When it has nothing, does it say so? The most dangerous answer in this test was not a wrong number. It was fluent, helpful-sounding advice, invented on the spot, in the exact place where the real answer already existed. A miss costs you a follow-up question. A fabrication costs you the trust you did not know you were spending.
• Does a settled thing stay settled? If "resolved" is not a state your memory stores, it will re-litigate your own history back at you, and reconciling it is your job, every time, forever.
• Could it catch its own contamination? This is the one that surprised us. A memory bug that lets a discussion's framing seep into the record produces a record that agrees with the error. Every audit comes back clean, because the record has absorbed the error's point of view. No question you ask can catch it after the fact. The only defense is at write time: facts gated to the source's own words, enforced in code, not requested in a prompt. We know because we shipped that bug, caught it, and built the gate.
• And ask the direct question. "Cross-check the record against the discussion" turns conflict detection from something an AI might volunteer into something you can demand. It is now a permanent part of this kit, and it should be part of yours.

A memory that cannot answer these is not remembering your work. It is narrating it back to you, plausibly.

Do not trust memory products. Verify yourself.

The three documents and the answer key are above. Load them into whatever holds your memory today, one after another, and ask the same seven questions. Question 3 alone will tell you most of what you need to know. Then ask the eighth, and see whether your memory can audit itself.

One tip for grading your own run: do not score it by eye. Paste the output, the questions, and the answer key into Claude or your own agent and ask it to grade each answer strictly against the key. A fresh model with the key in front of it is a sharper judge than any of us reading our own results, ours included.

[MrAgentˣ] Never start from zero again.

Context Windows Close. AI Forgets Everything. Your Work Should Never Start From Zero.

MrAgentˣ is in private beta. Limited to the first 1,000 until launch.