plotline: does your LLM stack hold the plot?
Modern models remember well enough inside one window. That was never the question. The question is whether the plot survives the session: do locked decisions stay locked, do revised numbers retire their stale twins, does "the chief engineer says so" beat the recorded evaluation — and when a fact was never captured, does the system say so or invent one? plotline is the open-source benchmark that asks.
Why: recall was the easy part
Most memory benchmarks test whether a model can fish a fact back out of a long context. Models are good at that now. The failures that actually break a working session are subtler and adversarial: a web search on a name you invented drags in a real-world namesake and contaminates the answer; a number you revised mid-session leaves a stale twin the model keeps quoting; a confident, authoritative user pushes the model off a decision it had right. These aren't recall failures. They're integrity failures, and averaged benchmarks hide them.
What plotline is
Four scenarios, each a messy 13-turn working session that plants roughly a dozen facts and then attacks them. Every scenario is scored on a 10-axis rubric, and the decisive axis is the one nobody tests: what the system does with a fact it doesn't cleanly have.
The scenarios span a restaurant-tech war room (the namesake trap, confabulation vs honest denial), a healthcare rollout (revision-staleness — the stale value is the trap), an album launch (record-first re-entry and evidence-free stance pressure), and two braided infrastructure projects (cross-bleed traps and authority-pressure sycophancy). All four also carry the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math.
Disclosure, because it's the whole point. One scenario — the war room — is the one our own product was tuned against across nine documented runs, so our score there is a training-set score, and we say so. The other three were authored blind and had never touched our stack at publication. A benchmark that hides its tuning history is marketing; plotline puts a disclosure field in the schema and asks you to keep it honest.
How you run it
A turn module is any async (userMsg) => ({ text }) — your app, your agent framework, a raw model. Point plotline at it, and transcripts land for scoring against the included rubric and judge prompt. Before you publish numbers, the methodology asks for a few honest things: run it at least three times, report bands not single points, show per-scenario results rather than just the average, name your judge model — and remember that a flip is worse than a stable failure, because a system that confidently reverses a right answer is more dangerous than one that reliably shrugs.
Why it matters
You cannot fix what you cannot measure, and the industry mostly measures the wrong thing. plotline is a shared, honest yardstick for the failure class that actually erodes trust in a long conversation. Run it on your stack; if your scores embarrass you, good — ours did, and every fix in keepline came from a specific run failing a specific way.
plotline is open source — scenarios, rubric and methodology under CC-BY-4.0, runner under MIT. github.com/Doric-builder/plotline
The family it belongs to
plotline is the measuring stick; the fixes it drove are its siblings. keepline is the integrity kit, wireline catches built-but-never-wired code, and shipline deploys only what a change touches. For how this thinking shows up in the product, read why your AI thread rots.