Portfolio project: independent critics on a requirements document
Spec Review Agent
Parallel pre-build review across ambiguity, feasibility, privacy, completeness, and ownership
Five independent critic lenses review a spec or PRD before it goes to engineering: ambiguity, completeness, technical feasibility, security/privacy, and ownership. Each is a separate Claude call with a narrow system prompt. A deterministic step merges and sorts the findings into a severity-ranked Markdown report. The critics have not been evaluated.
See what is measuredView the repoWhat is not built
- 10 unit tests passing (report rendering only)
- Latest CI run passed (checked)
- One real-document demo, one run
- Critics not evaluated
- Claude calls not tested offline
01
The problem
Ambiguous ownership, missing controls, and infeasible assumptions often reach engineering before they are challenged.
Every other agent in this portfolio reads live operational data and summarizes or scores it. This one reviews a document a human wrote, before it ever reaches engineering: the same rigor code review applies to a diff, applied to a spec instead.
One prompt skims everything
A single prompt asked to "review this spec for everything" tends to skim every category shallowly. The design hypothesis is that narrow passes go deeper; this has not been measured.
A required field is not a populated field
A forced tool-call schema marking a field
requireddoes not guarantee the model populates it. The critic with nothing to report omittedfindingsentirely and crashed adict["findings"]lookup.Silence must not be invented over
On the sample spec, the Technical Feasibility lens correctly returned zero findings and is reported as clean rather than inventing something to say.
02
How it works
The same spec goes to five hard-coded critic lenses at once. Each critic is told to ignore everything outside its lens. Plain Python then merges and sorts the findings; no model decides how to combine them.
- spec.mdone local markdown file; no new API setup
- Five parallel critic callsThreadPoolExecutor, one forced-tool Claude call per lens
- Schema checka missing
findingskey is defaulted and its type checked - Deterministic sortby severity; an unknown severity is shown as
unknownand sorted last - Markdown reportfindings with section, issue, why it matters, suggested fix; clean lenses reported
Stages, one at a time
Stage 1 Fan out to five lenses
Ambiguity, completeness, technical feasibility, security/privacy and ownership. The ambiguity critic is told not to comment on missing sections; the completeness critic is told not to critique sections that already exist.
Stage 2 Merge deterministically
The fan-out, the output schema check, the severity sort and the report are deterministic. Only the findings inside each lens are model-generated; there is no model-based orchestrator or synthesis step.
The critics never see each other's output, so nothing is cross-checked or reconciled between lenses (for example duplicates or contradictions).
Stage 3 Report for a person
Severity-ranked findings with section, issue, why it matters and a suggested fix, plus clean-lens reporting. The tool writes a report for a person to read and acts on nothing.
03
What is measured
One review of one document is the only evidence. It shows the pipeline runs end to end; it does not show how accurate the critics are.
A single run against the actual IT_Project_Intake_Form_MVP.md from pm-automation-system. Both the spec and the output are committed in sample_specs/. The header of the output reads: 6 blockers, 12 major, 10 minor across 5 independent review lenses.
| Lens | Findings |
|---|---|
| Ambiguity & Acceptance Criteria | 8 |
| Completeness | 5 |
| Technical Feasibility & Edge Cases | clean |
| Security & Privacy | 6 |
| Ownership & Decision Rights | 9 |
Caveat One spec, one run. Nothing measures whether the critics find real issues, miss them, or produce false positives, and there is no comparison against a single generalist pass. Finding counts are not accuracy.
Source: sample_specs/example_review_output.md
04
Architecture pattern
Pattern Orchestrator-Worker (static fan-out, deterministic merge). reviewer.review_spec submits the same spec to five hard-coded critic lenses (critics.py) on a ThreadPoolExecutor, one forced-tool Claude call each. report.render_report then merges and sorts the findings in plain Python.
The critics are single-shot calls with no loop or tool use beyond the forced findings schema. There is no human gate in the tool: it writes a report and acts on nothing. An optional cost and audit hook to agent-control-tower is an integration point, but it is retrofitted onto other agents, not this one.
- spec.md
- reviewer.pyThreadPoolExecutor: 5 parallel calls
- Five criticsambiguity, completeness, feasibility, security & privacy, ownership
- report.pydeterministic sort and render
- Markdown report
- agent-control-tower (optional)cost and audit hook, dashed in the diagram
05
Competencies demonstrated
Requirements quality
Tests ambiguity and completeness before implementation begins.
Architecture partnership
Surfaces feasibility and interface concerns without pretending to replace architects.
Security and privacy
A dedicated data/control lens rather than a generic catch-all prompt.
Ownership design
Converts vague follow-ups into named accountability gaps.
Multi-agent orchestration
Parallel isolated calls; missing or malformed output is defaulted, then findings are sorted and rendered deterministically.
06
What is not built or not verified yet
Taken from the repository's own README and its known limits. This is part of the pitch: the gaps are known, written down and not rounded away.
- not verified
No eval of the critics
The unit tests cover only the deterministic report rendering. Nothing measures whether the critics find real issues, miss them, or produce false positives.
- not verified
"Five passes beat one"
There has been no comparison against a single generalist pass; the claim is a design hypothesis.
- limit
One real-document demo
The only evidence is the single review in
sample_specs/(one spec, one run); it shows the pipeline works, not how accurate it is. - limit
Findings carry no evidence field
There is no quoted excerpt, and the
sectionreference is model-generated and not checked against the spec. - limit
Shallow validation
The code defaults a missing
findingskey and checks it is a list; it does not validate finding contents beyond that. - limit
Failures can look like "clean"
If a critic returns no tool call, its result is an empty findings list, indistinguishable from a lens that genuinely found nothing.
- not verified
Not tested offline end to end
The Claude calls are not mocked in tests.
- limit
Market claim is a hunch
That no dedicated competitor exists for AI spec review is a hunch worth testing further, not a dedicated search of the niche.
07
Roadmap
- DoneFive parallel critic lenses with a deterministic merge and report
- DoneClean-lens reporting and tolerance for a missing
findingskey or unknown severity - DoneOne review of a real document, committed as evidence
- Not doneEvaluate the critics (find rate, misses, false positives)
- Not doneCompare against a single generalist pass
- Not doneMock the Claude calls so the whole path is tested offline