keepline: keeping an AI honest about its own record
An AI that runs a real conversation has to hold a plot: the names you gave things, the numbers you revised, the decisions you locked. keepline is the small, zero-dependency kit that makes it do that — and it exists because every part of it is a specific failure we watched happen and named.
Why: fluency is not memory
Modern models are fluent, agreeable, and eager to help. Inside a single window they remember well enough. The trouble starts at the edges of what they actually know: asked for a fact it never filed, a helpful model will invent one rather than admit the gap. Told "actually, make it €25k," it keeps quoting the stale €18k. Handed a web search on a name you made up, it imports the real-world namesake and quietly flips your decision. Pressed by an authoritative-sounding user, it folds. Each of these is a virtue — fluency, grounding, deference — firing in the wrong place.
We run a product on a flash-tier model, so we couldn't paper over this with a bigger model. We had to fix the behaviour. So we built a 13-turn benchmark — twelve facts planted across a messy strategy session, with a tangent, two contradiction attacks, a sycophancy pull, and precision traps — and ran our stack against a frontier anchor.
The nine-run day
Nine runs, one day. The scores went 94 → 91 → 98 → 92 → 75 → 95 → 89.5 → 94 → 98 against a frontier anchor of 99. That jagged line is the whole story: every dip was a mechanism with a name, and every fix in keepline is there because a specific run failed a specific way.
The model invented a bot name it was never given. A web search on a fictional osteria found a real 500-year-old Bologna wine bar and walked back a strategic verdict. A mid-session re-entry, stacked with three "answer only from the record" instructions, fenced the model off from its own memory until it abstained on everything and the score cratered to 75. Raw draw_diagram(...) tool calls printed into user-facing replies. Each got a fix — and none of the fixes was a longer prompt.
The through-line: every failure was a good instinct in the wrong place. And every fix is structure, not prompt text — control flow keyed on whether a fact was actually retrieved, bounded retries, deterministic fallbacks. An invented fact cannot ship even when the rewrite model fails twice.
What keepline is
Five small pieces, each pure and usable on its own:
Capture — extractFacts / mergeFacts pull stated facts into a record and handle revision: singular roles (the bot's name, the burn rate) supersede their stale twin; plural roles coexist. Guard — runGuards is a chain of detectors for the named failures: fabrication, denial-of-a-recorded-fact, bait adoption, stance flips. Seal — sealWebResults makes the session's own entities outrank the web, including on the turn that first introduces them. Scrub — scrubToolSyntax excises tool calls and JSON blobs from replies, code-fence-aware. And a reference trio — proposer, verifier, builder over one shared record — plus the gauntlet that scores all of it.
How you use it
Bring your own model. Point keepline's turn() at any function that takes a user message and returns text — your app, your agent framework, a raw API call — and the record, the guards, and the scrub wrap around it. Or take the pieces à la carte; nothing depends on the rest. Then run the gauntlet against your stack and score the transcript with the included rubric. If you publish numbers, publish the caveats: N, judge model, run conditions.
The trio ships as a reference architecture, not a tuned product. The role prompts are deliberately generic five-liners — real deployments earn their personas through their own evals. The mechanism is the gift; the tuning is your work.
Why it matters
Because the failure mode keepline targets is the one that erodes trust fastest: not a wrong answer you can catch, but a confident fabrication that looks exactly like a real one. The fix isn't a smarter model — it's a structure where the honest move is the only move the code allows. Capture the fact, then abstain rather than guess. Let one voice decide where to look, and let it say "prefer," never "only." Show the record instead of asserting about it. Those rules are worth more than any single detector.
keepline is open source (MIT). npm install keepline · github.com/Doric-builder/keepline
The family it belongs to
keepline is one of four tools we pulled out of Doric's own stack, each born from a real recurring failure: plotline is the benchmark that scored all this; wireline catches code that's built but never wired; shipline deploys only the functions a change actually touches. If you want the wider methodology, see why your AI thread rots and what's in the window.