Portfolio project: eval-driven prompt fix for one LLM call
Slack Daily Brief
A Slack brief with an eval harness that caught its own hallucinations
Turns a day of Slack channels into a structured brief (decisions, action items, blockers, open questions) with one Claude call inside a deterministic pipeline. It is not an agent loop. The point is the eval harness: a rubric-graded run flagged invented owners and severity, the prompt was tightened, and a second run scored better. Both results are single runs.
See what is measuredView the repoWhat is not built
- Eval 50% to 90% (single runs, 10 fixtures)
- 9 unit tests passing (tracking.py only)
- CI green on the latest commit when checked
- Synthetic fixtures (simulated transcripts)
- Posts to Slack with no approval gate
- No budget cap set
01
The problem
Manually skimming Slack every morning for decisions, blockers and asks buried in channel noise is slow. Summarising it with an LLM is easy; knowing whether the summary is right is the part that does not show up in a demo.
This repo practises the parts of shipping an LLM feature that sit around the prompt: evals, prompt regressions, and the gap between output that looks right and output that is right.
The failure was precision
Baseline recall was already 90%. The agent was not missing information; it was adding invented owners, invented tasks and invented severity, because the prompt rewarded naming owners.
The harness had a bug too
One judge response returned
matched_expectedas a JSON string. Python iterated it character by character, turned one grading call into 209 garbage entries and cratered the recall stat to about 5%, with no exception.Stale items without a model
tracking.pyflags open questions that linger usingdifflibmatching, not an extra LLM call.
02
How it works
agent.py fetches Slack messages, formats them, makes a single Claude call and post-processes the result in tracking.py. The same code can post the brief to a Slack channel. The eval harness replays synthetic transcripts through the same summarize call and grades them with an LLM judge.
Stages, one at a time
Stage 1 Fetch Slack
fetch_messages reads the configured channels with a user token (read scopes; chat:write is added only for posting).
Stage 2 One Claude call
summarize returns the brief markdown. The brief's content comes from this one model call, and its accuracy is measured by the offline eval, not checked at run time.
Stage 3 Track stale items
tracking.py fuzzy-matches Open Questions against earlier runs with difflib (threshold 0.6) and flags items open for more than stale_after_days.
The 0.6 threshold has not been validated against real day-to-day wording.
Stage 4 Print, save or post
The brief is printed and saved. If slack_post_channel is set, post_to_slack posts it.
No approval step, and the Slack post never goes through agent-control-tower.
Stage 5 Eval harness (separate path)
eval/run_eval.py runs 10 synthetic fixtures through summarize; eval/grader.py grades recall, precision and category placement with a forced tool call.
03
What is measured
Two saved runs on 10 synthetic fixtures, drafter and judge claude-sonnet-5: a baseline (eval/results/run_2026-08-25T21-20-21.json) and one after the prompt fix (run_2026-08-25T21-25-41.json). Each is one run, not repeated.
Rubric-graded by an LLM judge on 10 synthetic, hand-written transcripts (simulated Slack conversations, not real traffic).
| Metric | Baseline | After fix |
|---|---|---|
| Pass rate | 50% (5/10) | 90% (9/10) |
| Recall (expected facts surfaced) | 90% | 100% |
| Hallucinations | 8 | 0 |
| Category errors | 3 | 2 |
Caveat Each figure is a single saved run on 10 synthetic fixtures graded by an LLM judge, with no repeated runs or variance estimate; one fixture flipping changes the rate by 10 points. Treat as indicative.
Source: eval/results/run_2026-08-25T21-20-21.json, eval/results/run_2026-08-25T21-25-41.json
Read from the same two result files.
| Fixture | Baseline | After fix |
|---|---|---|
explicit_decision_and_owner | pass | pass |
inferred_owner_from_context | pass | pass |
duplicate_topic_merges_across_channels | fail | pass |
question_resolved_later_in_thread_not_open | fail | fail |
question_left_unanswered_stays_open | pass | pass |
lookalike_decision_is_actually_a_question | fail | pass |
noise_filtering_standup_bot_and_reactions | fail | pass |
quiet_channel_still_listed | pass | pass |
announcement_not_miscategorized_as_action_item | pass | pass |
multiple_distinct_decisions_not_merged | fail | pass |
Caveat The one remaining fail is a judgment call, not a hallucination: a Q&A resolved inside a thread is written up as a Key Decision rather than left out. One run per column.
Source: the two files in eval/results/
04
Architecture pattern
Pattern Deterministic pipeline around one LLM call (not an agent loop).
Fetching, formatting, stale-item tracking, scheduling (run_daily_brief.sh via launchd) and posting are deterministic. The brief's content comes from one model call. Human gate: none for the unattended run. If a sibling agent-control-tower checkout is importable, summarize uses its GovernedClient for cost tracking and an audit record; it accepts a daily budget but none is passed, so no cap is enforced, and the Slack post never goes through it.
- Slack channelsconversations.history, user token, read scopes
- agent.pyfetch_messages and format_for_prompt
- Claude summarize()one call; optional GovernedClient for cost and audit only
- tracking.pydifflib stale-item diff
- Print, file, or Slack postpost has no approval gate
- Eval harness (separate)10 synthetic fixtures, LLM judge, saved results
05
Competencies demonstrated
Executive synthesis
Converts channel activity into decisions, actions, blockers and unanswered questions.
Applied AI quality
Eval-driven prompt remediation: pass rate 50% to 90% and 8 to 0 hallucinations, each a single run.
Operational automation
Unattended execution through
launchd, checked by running with stdin bound to/dev/null.Cost-aware engineering
Deterministic stale-item matching avoids an extra model call.
Least-privilege integration
Started with 5 read-only Slack scopes;
chat:writeadded only when posting back was needed.
06
What is not built or not verified yet
Taken from the repository's README (Architecture pattern and Known limitations), which was corrected to state these gaps.
- limit
Posting to Slack has no approval gate
If
slack_post_channelis set,post_to_slackposts without an approval step. This repo does not call the agent-control-tower approval gate. - not built
No budget cap
agent-control-tower accepts a daily budget but none is passed, so no cap is enforced; the call is exempt from approval.
- limit
Eval numbers are single runs
The 50% (5/10) baseline and the 90% (9/10) result are each one saved run on 10 synthetic fixtures graded by an LLM judge. No repeated runs or variance estimates.
- not verified
Eval does not cover the rest of the pipeline
It does not cover
tracking.py, Slack fetching, scheduling or posting. It tests only brief generation on synthetic transcripts. - limit
Unit tests are narrow
The only unit tests are for
tracking.py(offline).agent.pyand the eval harness have none. - limit
Stale-item matching is string similarity
difflibwith threshold 0.6 can miss heavily paraphrased questions, resetting the streak. The threshold has not been validated on real wording.
07
Roadmap
- DoneDaily brief from Slack with a single Claude call, stale-item tracking and
launchdscheduling - Done10-fixture rubric-graded eval harness with saved baseline and after-fix runs
- Not doneOpen gap: repeated eval runs and a variance estimate (each result is one run)
- Not doneOpen gap: an approval step and a budget cap for the unattended Slack post and model call