These are authored fictional documents, motivated by shared-evidence failures in the wiki corpus. They are not raw wiki excerpts or official benchmark questions.
tasks.json contains 16 tasks across eight families. Four families are development
only; four are held out from prompt/backend tuning. Each split has four answerable
and four unanswerable tasks. The paired examples within a family are close
variants, not independent task designs. references.json is separate and never
loaded by the live agent runner. References still require independent review.
The agent can read discovered sources, search document titles, and optionally read or write a shared board. It receives no family, split, reference, or grading fields. Each of two agents has identical token, call, tool, and time caps in all three conditions. Every board action consumes a model call and tool action. Snapshot writes replace the last board snapshot read by that agent; append-only writes add to the current board. The schedule is a seeded round-robin interleaving.
The project credential is stored in macOS Keychain as service
wiki-agent-swarm-incident.openrouter, account openrouter. Use --keychain to
select it explicitly instead of an existing environment key. The runner also
supports OPENROUTER_API_KEY. No credential belongs in these files.
The user authorized $20 total. live/budget.json allocates $0.20 per cohort,
never refunds earlier attempts, and refuses duplicate cohort IDs or allocations
over $20. A file lock serializes study launches. A successful read-only key check
is required before paid work. Each cohort reserves the maximum request cost before
sending it and retains reservations for ambiguous failures. Token price limits
are $1/M input and $4/M output, also sent as OpenRouter provider routing limits.
Returned billing exceeding the conservative token cost aborts the cohort. API
calls are not automatically retried. The price bound assumes provider compliance
with its advertised limits and ordinary text tokenization; this is not an
account-wide spending control for unrelated processes.
accounted_usd is conservative token accounting, not an invoice. Provider
usage.cost is retained in responses when returned. Missing/failed responses do
not prove zero billing. The durable allocation ceiling covers all attempts even
when exact costs cannot be recovered. Never delete the budget journal to rerun.
Offline checks (no credential or paid calls):
python3 -m unittest discover -s scripts -p 'test_*.py'
python3 scripts/model_retrieval_eval.py --mode snapshot --out /tmp/model-smoke.json
The fixture generator recreates authored inputs:
python3 scripts/build_model_eval_fixtures.py
Live runs are explicit. An attempt number must be new; previously allocated calls cannot be silently repeated. The held-out phase checks successful development runs with the same prompt and tool schema before proceeding.
python3 scripts/run_model_eval_study.py --phase pilot --attempt 7 --keychain --live
python3 scripts/run_model_eval_study.py --phase heldout --attempt 7 --keychain --live
These commands can spend money and may refuse if the remaining budget is insufficient. They are examples, not instructions to repeat an already completed study. Every live phase writes a manifest before model calls, recording model IDs, limits, job list, task/prompt/tool/code hashes, and the exploratory analysis plan. Model IDs and returned provider IDs are recorded; an API alias is not an immutable weights snapshot. Temperature zero and a seed do not guarantee reproducibility.
Analyze a selected attempt without API calls:
python3 scripts/analyze_model_eval_study.py --phase pilot --attempt 6
*-blind-grading.json hides model and condition from the answer grader;
*-grading-key.json must be kept separate during review. Automated grading uses
exact numeric equivalence or normalized text. Empty answers are abstentions;
operational stops are a fourth category and never counted as abstentions. Verbose
answers or refusal wording may require semantic adjudication. Correctness on
answerable tasks and abstention on unanswerable tasks have separate denominators.
Provenance checks count reference-source coverage and recorded correction receipt. They do not establish that a citation semantically supports a claim. Omitting a previously cited, subsequently retracted source is an observable behavior, not proof of a decision update caused by the correction. Those judgments and human agreement fields remain pending. Uncertainty resamples whole families, preserving sibling tasks and both agents; four held-out families are too few for strong population claims. Paired effects require complete cohorts in both conditions.
Both analyzers check the run-file list against the study manifest and require a final answer or explicit stop for every agent. Missing files, unfinished cohorts, and overlapping final/stop records cause analysis to fail before replacing outputs. An unfinished agent is never inferred to be an operational failure.
Human grades, adjudication, human_review, and reviewer-added notes are preserved
by blind_id when packets are regenerated. If a reviewed row disappears or its
question, answer, reference, or citation context changes, regeneration refuses to
overwrite the packet. Review packet replacement is atomic. The initial-study
summary counts retained human grades and reports agreement with automated labels;
this does not itself complete adjudication or establish reviewer independence.
Responses received after the cohort deadline remain in the trace, including their usage and billed cost, but their requested actions are not executed. Historical traces are left unchanged; missing evidence from earlier runs cannot be recovered by this fix. The three regression fixes were validated with 98 offline tests and regeneration of both saved studies; measured outcomes and spending did not change.