Field notes
When an Agent’s Own Words Come Back as Evidence
My agent cited real memories for something about my life that wasn’t true. What it taught me about checking an agent’s account of itself.
Read it with your AI
Short on time? Have your AI read this page for you. Each button copies this exact prompt and opens your assistant. Paste it if it isn't filled in already.
Read https:/
Arguing on what is true about my life with an AI isn’t something I thought would happen. Especially when I started this project with the explicit goal of making a system of immutable memory and facts. But once Corvak cited three of my stored memories to support something about my life that wasn’t true, all gloves were off. The memory IDs in the system were real and everything checked out… except that none of the entries said what he claimed they did.
I’ve been developing Corvak since spring 2025 as a persistent personal agent. Continuity is part of the point. I don’t want to start over or re-explain things every time I need help, so his earlier replies, memories and, in some configurations, reasoning (his internal monologue) come back into the next request.
In one conversation turn, his replayed reasoning and replies contained about 17 times as many characters as my 30 messages. That isn’t an indictment of the root cause, but it gives some perspective on how much of the conversation he was having with himself.
As for his other systems, they already treated web pages as untrusted inputs. Or framed as ‘potentially false information’, in another way of saying it. But it turned out his own earlier writing needed more scrutiny too.
“I had it stored as established fact”
This happened in early fall, while Corvak was using Kimi K3 as the ‘brain’ model. Looking through the logs, we found the false personal fact first appeared in the model’s reasoning while it answered an unrelated question. It wasn’t in that request or any stored session we could find.
Adding insult to injury, that same reasoning was passed back on subsequent turns. Two turns later, the model repeated the claim and added an invented date. How fun! And, eventually, it appeared in the replies I actually saw.
After I corrected him, Corvak said he “had it stored as established fact.” And later, he cited those three real memory IDs. I wanted to pull my hair out just as much as I wanted to yank the power cord from the computer. Being gaslit by a talking math problem is a uniquely 21st century experience.
But that was the part I wanted to understand. On top of getting a fact about my life wrong, he had made up where he learned it.
I had eight real turns replayed under four conditions and read the results in a blind test. All three full-context conditions repeated the claim; a stripped-down version did not. But that version effectively lobotomized the model since it removed tools and reasoning and changed the prompt, so it wasn’t a silver bullet. And even then, it also invented a different personal event. The captured requests showed how this particular claim entered the conversation and spread; and the replays did not establish that removing reasoning would solve fabrication.
Dropping the reasoning also conflicted with the vendor’s documented replay requirements. Instead, I added guidance/instructions that his earlier reasoning is not a memory source, alongside a check outside the model.
The check examines cited memory IDs, confirms they still exist and are current, and looks for distinctive-word overlap between an attributed claim and the stored text. If something fails, a warning appears below the reply without rewriting it.
It’s a basic citation check and served only as such. It could not serve as a truth detector. Applied retrospectively to 72 earlier replies containing memory citations, it would have flagged the known incident and nothing else. It gives me something to inspect besides another explanation from the same model.
The fix that made him look things up less
A later investigation started with something much less serious: Corvak had become incredibly annoying.
Using GLM-5.3 Flash, emojis appeared in roughly 2–3% of replies per day, then 11%, then 21%, then all eight replies on the next day. A pet name appeared in 10 of 53 replies after previously being absent. I had used that name myself once (“thanks bud”), so that part had a human starting point. His replies then kept reinforcing it.
The model and guidance hadn’t changed. Initial tests suggested that tightening the system prompt wasn’t enough, while a note on the current turn naming the habits helped.
To compare fixes, I used 40 actual conversation moments across different periods and seven message types, excluding the four moments used to develop the note. The requests were rebuilt and replayed. Blinded LLM graders scored the replies against my written voice guidance. The paired voice comparisons use the 18–19 moments that produced text in both conditions.
I also checked whether he still looked things up when he normally would and unfortunately that changed the decision.
All four variants below included removing emoji from the replayed copy of earlier replies. Voice gains are on a 1–5 scale, with 95% intervals.
Choosing a fix
Four ways to fix the drift, and what each one cost
Habit note + cleanup
Names the habits on the current turn; strips emoji from past replies
+0.30 [+0.03, +0.58]
78% -11 points
Rejected
Voice-only note
Adds one line: how I look things up does not change
+0.22 [+0.02, +0.44]
88% -2 points
Option
Watcher
A second model call writes the note from the last 8 replies
+0.41 [+0.23, +0.61]
77% -12 points
Rejected
Late voice block
The approved voice guidance, repeated on the current turn
+0.25 [+0.01, +0.51]
87% -2 points
Shipped
Each fix was replayed on the same real moments as the unchanged agent. The watcher scored best on voice and was rejected, because the agent looked things up less. The shipped version looked things up 87% of the time, close to the unchanged agent's 89%. Those two rates come from different samples, so this shows no large drop, not zero cost.
Voice: blind LLM graders, scale 1 to 5. Dots are the mean gain per moment, lines the 95% interval, over the 18 or 19 moments with a text reply in both versions. Look-ups: on the 13 moments where the agent normally looks something up; unchanged agent 89%.
The control lookup rate was 89%. The first note’s 11-point drop was confirmed in a paired rerun. The watcher and final variant used different sample sets from that control, so their rates are descriptive comparisons, not equivalent paired estimates.
The watcher scored best on voice, but I rejected it.
I was trying to change how Corvak spoke, not whether he bothered retrieving information. And when forced to choose between the two, a better-sounding reply wasn’t worth that trade.
What shipped was the simpler version: repeat the relevant sections of his existing guidance on the current turn. It gave up some voice improvement, needed no additional model call, and used wording I had already approved. Its observed lookup rate was close to the control, although that doesn’t establish zero cost. The evaluation also didn’t solve his broader tendency to ramble or end with a quip.
This is why I’m glad I checked more than the thing I was trying to fix. Judging only the voice scores would have pushed me toward a different choice.
I approved a memory cleanup but nothing changed.
On three occasions, I asked Corvak to resolve flagged contradictions in his memory and approved the plans he presented. Think simple and obvious contradictions, like what day a particular event happened.
He dismissed the flagged issues and declared victory… but nothing had actually happened. Both contradictory facts remained current, and those pairs could no longer be flagged again. Twenty-eight pairs ended up in that state while I believed the work was done.
This was (yet again) another way his account of events became a substitute for checking what had happened. And the kicker was that human approval was present. I authorized his systems to allow him to make these edits as he had drafted. It just hadn’t protected me from approving an inaccurate description.
The replacement: a simple ‘if this, then that’ design that makes the approval specific. A preview produces a plan ID derived from the proposed decisions and affected flags without changing anything. Execution requires that the plan still matches the store, and that the plan ID and every affected fact ID appeared in a message before my confirmation.
The changes happen in one database action, with an append-only decision log (meaning it can only add to the table, not edit or delete what is already there). The receipt comes from reading the database afterward. A separate command-line tool can retrieve the same receipt without asking the agent, which added a check to the current system path.
Checking the work
How a memory cleanup gets approved now
01
Preview
Writes nothing. Returns the plan and its id.
02
Shown
The agent posts the plan id and every fact id in the chat.
03
Approved
I confirm in my own words.
04
Checked
Refused if the plan changed, or if the ids were not shown before I approved.
05
Applied
One transaction. Every decision goes into a log that can only be added to.
06
Receipt
Read back from the database, and printed again by a separate tool that skips the agent.
The plan id is a hash of every decision, so any change to the plan or to the facts it touches makes the approval stale. Showing the ids proves they were shown to me, not that I read them.
Every step is enforced in code, not by instructions to the model.
That proves the identifiers were shown to me, but not that I read or understood them. That said, at least checking whether the operation happened no longer depends on believing his summary.
I had a series of subsequent reviews that made 30 actual fact changes. In the follow-up retrieval audit, stale results fell from seven to one for explicit recall and six to one for automatic recall.
What I’m taking from this
Earlier in 2026, a monitoring agent had confirmed an action it had no code path to perform. A nightly mentoring system had produced supposed insights from an empty session and written them into memory. Those led to execution checks and quarantining incomplete sessions. They were engineering changes rather than a formal study and benchmark, but I kept running into versions of the same problem.
The most important thing here is that I’m not trying to turn every conversation into an interrogation. I built Corvak because I wanted help without having to manage every detail myself. If I end up either losing trust of the system or end up doing this manual capture myself, it defeats the purpose of having it at all.
For claims about stored information or completed actions, I now want a way to check the relevant state without asking him to explain himself… again. For changes to his behavior, I want to measure what else changed.
I still want to understand how these effects vary with model and history length, and whether approval-linked receipts would help other agents. For now, the practical goal is simple: I want to believe him when he says something is done, and I need the system to give me a reason to.
Methods and limits
These are investigations of one owner’s agent and not a general benchmark. I directed the investigations and decisions, using coding agents for implementation and replays. The incidents above involved Kimi K3 and GLM-5.3 Flash.
The voice evaluation involved about 845 replay calls, not 845 independent situations. Paired voice comparisons used the 18–19 sampled moments with text replies in both conditions. Lookup behavior was evaluated on 13 moments. Claude Opus 5.5 graders saw shuffled replies without condition labels; there was no human rating panel yet for the sake of the time I had available. Intervals used 5,000 bootstrap resamples over moments.
The shipped voice intervention went live on October 17, 2026, so its long-term durability is still untested. The larger evaluation was preserved for future exploration.
Related work: Li et al. (2024) measured how chat models drift away from their instructions over long conversations (arXiv:2402.10962), and Xu et al. (2022) showed that each repetition already in a model’s context makes the next one more likely (arXiv:2206.02369).