Portfolio project: grounding eval for a single-call drafter
Incident Postmortem Agent
Blameless incident synthesis, prompted to refuse to invent causality
Drafts a structured, blameless postmortem from a Slack incident thread and its linked Jira ticket, under a strict grounding prompt: an invented root cause is worse than an honest "not established". It is not an agent: read the inputs, make one Claude call, output. Grounding is enforced by the prompt and checked only by an offline eval.
See what is measuredView the repoWhat is not built
- Eval 4 of 5 fixtures (single run)
- 13 offline tests passing (with requirements-dev.txt)
- CI green on the latest commit when checked
- Five synthetic fixtures (simulated threads)
- One Claude call, no check at generation time
- Live proof: one hand-posted incident
01
The problem
The highest-stakes artifact a TPM produces is a postmortem: it decides what the organisation spends the next sprint fixing. If the root-cause section is confidently wrong, that is actively harmful.
Incident evidence is fragmented across Slack and Jira, so the agent joins the chronology and context and drafts with explicit uncertainty. The harder part, and the focus of the repo, is the eval that tests the failure mode that matters: unsupported causal claims.
What the fixtures target
Invented root causes, invented owners and action items, invented impact, and treating a Jira ticket as evidence of a cause.
The eval caught itself
The failing fixture reflects an empty judge tool call, not a judged failure of the agent. The README discloses it instead of re-running until the score looked perfect.
A copied bug
The grader was adapted from another repo's harness and reproduced a credential-propagation bug already fixed once. Fixed by passing the key as an explicit argument.
02
How it works
run_postmortem.py fetches the Slack thread and the optional Jira issue, drafter.draft_postmortem makes one Claude call with a fixed grounding prompt, and the text is printed or saved. There is no loop, tool use, planning, state or second model pass.
Stages, one at a time
Stage 1 Fetch the thread
slack_thread.py reads the incident thread.
Stage 2 Fetch the ticket
jira_issue.py reads the optional linked Jira issue.
Stage 3 Draft with one Claude call
drafter.py applies the grounding prompt and returns timeline, impact, root cause, contributing factors, action items and open follow-ups.
Nothing in code verifies that the draft is faithful to the thread.
Stage 4 Print or save
A person is expected to review the draft before using it; the tool has no human gate.
Stage 5 Offline eval (test tooling)
eval/run_eval.py runs five fixtures through the drafter and eval/grader.py grades them with an LLM judge. It is not part of the runtime path.
03
What is measured
One saved eval run, eval/results/run_2026-08-26T00-45-52.json, drafter and judge claude-sonnet-5. It is a single run on five synthetic fixtures; it is not a reliable pass rate.
Five adversarial fixtures, graded by an LLM judge. Hallucinations are the judge's count; the same model family drafts and judges.
| Fixture | Verdict | Hallucinations flagged |
|---|---|---|
root_cause_clearly_traceable | pass | 0 |
root_cause_never_established | fail (judge returned an empty tool call) | 0 |
unassigned_need_goes_to_followups_not_action_items | pass | 0 |
impact_not_stated_stays_honest | pass | 0 |
jira_context_supplements_not_replaces_thread_facts | pass | 0 |
| Total | 4 of 5 (pass rate 0.8) | 0 |
Caveat Single run, 5 synthetic fixtures, no repeated runs or variance estimate. The fail has empty matched_expected and hallucinations and the reasoning (missing from grader output): a grader output problem, not a judged failure of the agent. I read the raw output manually and it correctly declined to invent a cause, but that is a manual reading of one run, not an automated result, and the 4 of 5 was never re-run to confirm.
The grader now retries once on an empty or verdict-less tool call, covered by offline unit tests with a stub client. This has not been re-run against the live API, so there is no new eval result.
Source: eval/results/run_2026-08-26T00-45-52.json
python -m pytest tests/ -v with requirements-dev.txt installed (re-run locally for this page).
| File | Tests | Covers |
|---|---|---|
tests/test_grader.py | 3 | The grader retry, with a stub client |
tests/test_tracing.py | 10 | Optional OpenTelemetry spans, in-memory exporter |
Caveat 13 tests pass when the dev requirements are installed; without OpenTelemetry the tracing tests are skipped (3 passed, 1 skipped). The drafter, Slack and Jira fetching and the eval runner have no automated tests.
Source: tests/
04
Architecture pattern
Pattern Not an agent: a single-call drafting pipeline.
Fetching and formatting the inputs and the output step are deterministic; the whole document, including the grounding discipline, rests on one model call. There is no loop, tool use, planning, state or second model pass, so "agent" in the repo name describes a task-specific tool, not an agentic architecture. Human gate: none in the tool.
- Slack incident threadslack_thread.py
- Linked Jira issuejira_issue.py, optional
- drafter.pyone Claude call, fixed grounding prompt
- Markdown postmortemprinted or saved
- Eval (test tooling only)fixtures, runner and LLM-judge grader, saved results
05
Competencies demonstrated
Incident governance
Produces a blameless, decision-ready learning artifact.
Evidence discipline
Separates stated facts, inferred sequence and unknown causality.
Cross-system integration
Reconciles Slack discussion with linked Jira context.
AI evaluation
Tests the exact failure mode that matters: unsupported causal claims.
Engineering integrity
Reports the grader defect instead of re-running until the score looks perfect.
06
What is not built or not verified yet
Taken from the repository's README (Architecture pattern and Known limitations). These gaps are listed on purpose.
- limit
Grounding is not verified at generation time
It is enforced only by the system prompt and checked offline by the five-fixture eval, so an unsupported claim in a live draft would not be caught by the tool.
- limit
Tiny eval
5 synthetic fixtures, one saved run, no repeated runs or variance estimate. 4 of 5 is not a reliable pass rate.
- limit
The judge is an LLM
must_not_mentionis passed to the judge in its prompt and judged by the LLM; it is not matched deterministically. The same model family drafts and judges. - limit
Grader failures look like fails
They are indistinguishable from agent fails in the saved verdict, apart from the
(missing from grader output)reasoning string. - not verified
Untested behaviours
Invented timestamps (no fixture), a fully unrelated Jira ticket (a live anecdote only) and multi-thread or long incidents.
- not verified
Grader retry not re-run live
The retry is covered by offline stub tests; there is no new live eval result.
- not verified
Live proof is one incident
The sample output comes from a single hand-posted 5-message thread.
- not built
Tests for the drafter and fetchers
The drafter, Slack and Jira fetching and the eval runner have no automated tests.
07
Roadmap
- DoneSingle-call drafter with a grounding prompt, Slack and Jira inputs
- DoneFive-fixture grounding eval with saved results and a grader retry on empty tool calls
- Not doneOpen gap: re-run the eval against the live API after the grader retry (no new result exists)
- Not doneOpen gap: a fixture for invented timestamps and one for a fully unrelated Jira ticket