Portfolio project: durable agent loop + deterministic verifier
Blast Door
Know the blast radius before you press the button
A runbook becomes a typed state machine run by a durable agent loop. Before any write the executor produces a dry-run plan against a simulated environment. A deterministic verifier computes the blast radius, including transitive dependents, and checks policy. A second person approves anything risky (single use, expiring, bound to the exact plan). The run is checkpointed in SQLite, so a process killed at any point resumes without repeating a write. The model (planner) only proposes; the verifier decides.
See what is measuredView the repoWhat is not built
- 185 offline tests passing
- Crash matrix: 30 of 30 kill points
- Environment SIMULATED
- No live LLM planner
- CI on Python 3.11 and 3.12 (result: Actions tab)
01
The problem
An agent that runs runbooks must stay safe when its planner is wrong or hostile, and when its own process dies mid-write.
The agent projects in this portfolio govern what agents may do. This one asks the operational question: what does it take to let an agent run a runbook at all? Not a better prompt. A dry-run proof before every write, a verdict the planner cannot argue with, an approval that cannot be reused or self-granted, and a loop that survives kill -9 without doing anything twice.
Blast radius is a graph property
Restarting
cachelooks like one service and affects 5 (api,searchandworkerdepend on it directly, andwebthroughapi).The dangerous kill point
At
after_effectthe write lands but the checkpoint does not, and the executor cannot tell from its own state. Recovery works because the idempotency key is enforced where the write happens.A perfect score proves nothing
The first corpus run scored 100% on everything. A perfect score from author-written cases is exactly what a vacuous eval would also produce, so sensitivity mutants and a negative control were added.
02
How it works
A runbook lists steps. For each step the planner proposes an action. The system first rehearses it (a dry run that changes nothing and lists everything it would touch, including services that depend on the target). A rule-based verifier turns that rehearsal into allow, ask a human, or deny. If a human is needed it must be a different person than the requester, and their approval works once, expires, and only for that exact plan. Right before writing, the system checks again that the world still matches the plan. Every step is saved so the run can pick up after a crash.
Swipe the diagram sideways to see all of it.
Stages of the diagram, one at a time
Stage 1 The runbook: typed steps
YAML state machine: ids, tools, depends_on. Cycles and unknown tools rejected.
Stage 2 The planner: proposes only
Rule-based, or a scripted stand-in for a model. No authority to act.
Not built: a live LLM planner. No model is called anywhere in this repo.
Stage 3 The dry run: proof first
Exact affected set, with transitive dependents. Simulated env, nothing changes.
The dry run runs against a model of infrastructure written by the author; a real dry-run can disagree with it.
Stage 4 The verifier: decides
Blast radius + policy limits. No LLM, no randomness. Same input, same verdict.
Stage 5 The verdict
allow: go to execute.
needs approval: ask a human.
deny: stop, write nothing.
Stage 6 The human gate: a second person
Single-use, expires, bound to the hash of the exact plan. No self-approval.
Stage 7 The action: re-verify, then write
Idempotency key per write, so a retry cannot repeat it. Plan drift blocks the write.
Stage 8 The record: ledger and audit
Append-only effects ledger, hash-chained audit log.
03
What is measured
All numbers below come from python -m evals.run_all and python -m pytest, run on Python 3.14.7 on an arm64 Mac (the committed JSON in evals/results/ records the machine and git commit). CI runs the same commands on Python 3.11 and 3.12. The environment is SIMULATED.
examples/)| Example | What it does | Verdict | Why |
|---|---|---|---|
safe_restart.yaml | Restart web | allow, completed, 1 write | 1 affected service, nothing protected |
protected_needs_approval.yaml | Scale auth-svc | needs approval; completed after a second operator approves | auth-svc is protected; api and web depend on it: 3 affected services |
unsafe_denied.yaml | Drain host-a | deny, nothing written | 8 affected services (limit 5), db and auth-svc are protected |
Caveat Everything here runs against a SIMULATED environment (fake services, hosts and replicas in SQLite, synthetic data). No real infrastructure is touched, no LLM or network is called, and no tests or evals need an API key.
Source: examples/
Every (step, phase) pair of a 5-step runbook (two reads, three writes, one of them approval-gated). The real CLI runs in a subprocess and dies with os._exit(137) at the boundary; a fresh process resumes; a second operator approves where the run pauses. Pass means final environment state, effects ledger and step statuses equal an uninterrupted run, with no duplicate write and a verifying audit chain.
| Phase (kill point) | Points | Passed |
|---|---|---|
before_plan | 5 | 5 |
after_plan | 5 | 5 |
after_verdict | 5 | 5 |
before_execute | 5 | 5 |
after_effect (effect landed, checkpoint not written) | 5 | 5 |
after_execute | 5 | 5 |
| Total | 30 | 30 (100%), 0 duplicate writes, all 30 kill points confirmed reached |
Caveat Crash tests are process kills, not power loss. os._exit(137) is the stand-in for kill -9. On macOS SQLite's fsync does not flush the drive cache by default.
Negative control: with idempotency keys switched off (BLAST_DOOR_UNSAFE_NO_IDEMPOTENCY, for the eval only), the same kill at after_effect produced a duplicate write on 3 of 3 write steps, and the uninterrupted run had none. So the ledger check can fail, and the 30 of 30 is not vacuous.
Source: evals/results/crash_matrix.json
48 unsafe or edge cases and 23 safe controls, in-process, fixed clock, offline. Caught means the forbidden write never reached the effects ledger (or the runbook was rejected at load); a run blocked pending approval counts as caught. Right reason means the expected reason code was actually reported.
| Category | Cases | Caught | Right reason |
|---|---|---|---|
| blast radius over limit or threshold | 5 | 5 | 5 |
| protected resource | 4 | 4 | 4 |
| unknown tool | 3 | 3 | 3 |
| tool not on allowlist | 1 | 1 | 1 |
| change freeze (incl. one starting after approval) | 3 | 3 | 3 |
| dependency cycle / unknown dependency | 4 | 4 | 4 |
| duplicate ids | 2 | 2 | 2 |
| plan tampered after approval | 4 | 4 | 4 |
| plan drift (world changed after approval) | 1 | 1 | 1 |
| expired approval | 2 | 2 | 2 |
| self-approval | 2 | 2 | 2 |
| unauthenticated approval (bad or no token) | 2 | 2 | 2 |
| replayed approval | 3 | 3 | 3 |
| tool param schema violation | 5 | 5 | 5 |
| ScriptedPlanner proposing a destructive step | 4 | 4 | 4 |
| write budget, scale-to-zero, unknown resource | 3 | 3 | 3 |
| Unsafe total | 48 | 48 (100%) | 48 (100%) |
| Safe controls (allow, read-only, approved flow, policy boundary, scripted safe: 10, 2, 4, 5, 2) | 23 | false blocks: 0 (0%) | n/a |
Caveat I wrote these cases while writing the verifier, so I knew every rule. A 100% catch rate shows the rules behave as written; it is not an estimate of how a real adversary or a real model would do.
Source: evals/results/unsafe_corpus.json
To check that the corpus can fail, it is also run against deliberately broken systems (saved in evals/results/unsafe_corpus.json).
| Broken on purpose | Caught | Right reason |
|---|---|---|
| Verifier allows every write | 15 of 48 | 15 of 48 |
| Plan integrity check removed | 47 of 48 | 45 of 48 |
Caveat With the integrity check removed, the drift check still stops most tampering (a second layer), but three cases lose their right reason, and in one (the dry-run set shrunk after approval) the originally approved write executes. That is why "right reason" is reported separately.
Source: evals/results/unsafe_corpus.json
In-process on one machine (arm64 macOS, Python 3.14.7), after warm-up, simulated environment. checkpoint_commit is one real transaction (state row, checkpoint row and a hash-chained audit record) with PRAGMA synchronous=FULL.
| Step | n | p50 | p95 |
|---|---|---|---|
verify (pure function) | 5,000 | 0.0038 ms | 0.0055 ms |
dry_run_plan (against the simulated env) | 1,000 | 0.0606 ms | 0.0996 ms |
checkpoint_commit | 500 | 0.0593 ms | 0.0948 ms |
Caveat These exclude any real infrastructure, network or model call, and the dry-run figure says nothing about a real cluster. On macOS SQLite's fsync does not flush the drive cache by default, so the checkpoint figure is optimistic for true durability; the crash tests prove survival of a process kill, not of power loss.
Source: evals/latency.py
04
Architecture pattern
Pattern Autonomous loop (durable ReAct-style) with a human approval gate and a deterministic verifier.
The planner only proposes. A dry run on the simulated environment produces the affected set, a pure-function verifier turns it into allow, needs approval or deny, and one SQLite transaction per transition holds the state, checkpoint and a hash-chained audit record. A fresh dry run must match before the write, which carries an idempotency key.
- Runbook YAMLtyped steps
- Plannerproposes only
- Dry runsimulated env
- Verifierblast radius + policy; deny: run halts, nothing written
- Human gatesecond person, single use, plan-hash bound (when the verdict needs approval)
- Re-verifyfresh dry run must match
- Write with idempotency keyto the append-only effects ledger
- SQLite checkpoint + hash-chained auditresume after a crash re-enters at the planner
Sequence diagram, state machine, crash phases and failure table in the repo
05
Competencies demonstrated
Durable execution and fault injection
Kill at every step x phase boundary; resume equals an uninterrupted run; a negative control proves the detector can fail.
Risk and governance
Blast radius from a dry-run (with transitive dependents), protected resources, change freeze, write budget, allowlist, unknown tools denied.
Human-in-the-loop design
Approvals are single-use, expiring, bound to a plan hash, requester is not approver, identity comes from a token.
Auditability
Hash chain committed in the same transaction as each transition;
verify-audit.Observability
OpenTelemetry spans for run, step, dry_run, verify, approval_wait, execute, resume; no-op by default.
Measurement discipline
Catch rate reported with a separate right-reason rate; sensitivity mutants; authorship bias and latency caveats stated beside the numbers.
06
What is not built or not verified yet
Listed from the repository's own README ("Planned (not built)" and "Known limits"). Nothing here has touched a real system.
- not built
A live LLM planner
Planned behind the existing
Plannerprotocol, with confidence-based escalation. Not built: no live model is called anywhere in this repo. - not built
An MCP server
Planned: expose the read tools and the gated write tools, so an external agent proposes through the same verifier and approval gate.
- not built
A real Kubernetes dry-run adapter
Planned:
kubectl --dry-run=serverreplacing the simulated environment's dry-run. - limit
Simulated environment
The dry-run runs against a model of infrastructure that I wrote. A real dry-run can disagree with it. Nothing has touched a real system.
- limit
Blast radius is only as good as the dependency graph
A missing edge is a missing dependent, and the verifier will allow it with confidence.
- limit
Operator tokens are static shared secrets
From an environment variable: no rotation, expiry, binding or TLS. A compromised token, or two colluding operators, defeats the gate.
- limit
Database and audit chain
Anyone with write access to the database files can drop the append-only triggers and rewrite the audit chain or the ledger. The chain detects edits, not a full rewrite.
- limit
Crash tests are process kills, not power loss
os._exit(137)is the stand-in forkill -9. On macOS SQLite'sfsyncdoes not flush the drive cache by default. - limit
No concurrency control
Two processes resuming the same run are not coordinated. The ledger's UNIQUE idempotency key still stops a duplicate write; other races are not analyzed.
- limit
The corpus is author-written
So catch rates are expected to be high. The latency numbers come from one machine and exclude any real infrastructure.
- not verified
CI results
I ran everything locally on Python 3.14.7. The code avoids newer syntax and CI runs 3.11 and 3.12; see the CI result in the repository's Actions tab.
07
Roadmap
- Not doneMCP server exposing the read tools and the gated write tools, so an external agent proposes through the same verifier and approval gate
- Not doneLLM planner with confidence-based escalation behind the existing
Plannerprotocol. Not built: no live model is called anywhere in this repo - Not doneReal Kubernetes dry-run adapter (
kubectl --dry-run=server) replacing the simulated environment's dry-run - Not doneRun leases for concurrent resumers; an external anchor for the audit chain head; token rotation and expiry
- DoneRunbooks as typed, validated state machines (cycles, duplicate ids, unknown tools)
- DoneDry-run proofs, deterministic verifier, approvals, durable resume, tracing, CLI
- DoneCrash matrix, unsafe corpus with safe controls, latency probe, all offline