RFC 0033: Evidence-based quality review

Status: experimental implementation on codex/quality-reviewer-rfc; not released

Author: Repo2RLEnv maintainers, with Codex assistance

Created: 2026-10-16

Design baseline: origin/main at 27d04ee8; extends RFC 0027

SummaryCopy Anchor Link

Extend the shared Harbor quality loop with auditable findings and an evaluation suite for the reviewer itself. Keep generation, task rewards and the existing repair engine separate: the reviewer must identify a concrete task defect before repair can act, and its own accuracy must be measured against independently audited cases. All pipeline integration code remains owned by Repo2RLEnv.

The first deliverable is an opt-in review profile and a CPU-first, 20-family pilot. Recommended external API/cloud cap: $150, with $70–$120 expected under the assumptions below. The user approved this separate cap with “go ahead” on 16 October 2026; the campaign authorization and ledger are retained locally.

MotivationCopy Anchor Link

The existing loop already checks task/verifier alignment, reads bounded evidence, executes wrong-solution and valid-alternative controls, distinguishes rollout outcomes and preserves repaired revisions. More review calls alone do not tell us whether the reviewer is correct. We need to measure false accusations as well as missed defects, and retain the reasoning behind accepted corrections.

Micro1's Flow-QC article motivates this work: it describes evidence-grounded task and rollout review, clean/defective task pairs, adjudicated feedback and offline instruction optimization. This is an independent adaptation of public ideas. We do not have their internal implementation, complete prompts or evaluation corpus, and do not claim to reproduce their production results.

The argument against a new subsystem is strong: the existing quality loop already handles most execution. Therefore this RFC adds contracts and reviewer evaluation around that loop, not another generator, agent harness, reward model or cloud abstraction.

DesignCopy Anchor Link

InputCopy Anchor Link

Accept an immutable Harbor task directory from any recipe or Tasksmith, plus optional baseline, oracle, probe and solver receipts. Existing hash/checksum binding rules remain authoritative. A review may inspect instruction.md, task.toml, environment/build inputs, private tests, reference solution and bound rollout artifacts. Missing material is recorded, never invented.

Optional author context records the intended public behavior, PR provenance and why a test exists. Context helps investigation but cannot silently add requirements to the learner's instructions. Keep this material outside learner-visible files.

The pilot supports the existing binary quality contract. Graded optimization tasks, including FrontierSmith, may receive static findings but cannot receive full acceptance through this profile. Their feasible alternatives need a different score contract; changing success_reward alone does not supply one.

InterfaceCopy Anchor Link

Retain repo2rlenv quality run and quality show. Add an explicit profile option so existing callers continue using practical-generation-v1 by default. The new profile is named evidence-review-v2.

Review with existing evidence; model calls, no new solver or probes:

repo2rlenv quality run ./tasks/example \
  --profile evidence-review-v2 \
  --campaign ./workspace/qc-pilot \
  --out ./workspace/qc-pilot/reviews/example \
  --rollout ./jobs/example/trial/result.json \
  --review-model anthropic/claude-sonnet-4-6

Review and bounded repair with fresh remote validation:

repo2rlenv quality run ./tasks/example \
  --profile evidence-review-v2 \
  --campaign ./workspace/qc-pilot \
  --out ./workspace/qc-pilot/repaired/example \
  --repair --provider daytona \
  --runtime-wheel ./dist/repo2rlenv-0.9.3-py3-none-any.whl \
  --review-model anthropic/claude-sonnet-4-6 \
  --max-repairs 3 --max-spend-usd 5

The wheel path is illustrative; use the wheel built from the implementation checkout. A per-task cap is a stop condition, not a guarantee that repair finishes. --provider modal uses the same CPU execution contract. Existing native GPU support remains Modal-only; provider capability checks happen before allocation. Remote tools, tests and builds never execute on the local controller.

Implemented commands (see the usage guide):

quality show DIR displays the exact task/reviewer revisions, criterion decisions, evidence coverage, issue locations, costs and missing checks using the shared Rich console. --json is the stable machine interface. Preserve existing exit codes; benchmark run distinguishes completion from whether a candidate qualifies for promotion. It must not claim success because it merely produced reports.

Keep QualityLoop.run() as the Python entry point. Profile selection is an additive LoopOptions field. New result types are versioned; the legacy parser must continue reading old files. Benchmark and feedback code call existing model, budget and remote adapters rather than adding alternate provider clients.

AlgorithmCopy Anchor Link

1. Freeze task bytes and imported evidence before review. Bind reviewer identity to prompt/skill hashes, code, model route, generation settings and tool limits.
2. Build a bounded evidence view, prioritizing instruction and actual failed assertions. Existing citations and document readers remain in use.
3. Ask for material defects, permitted counterexamples and missing evidence. Review-only mode can request execution but cannot silently allocate a worker.
4. Validate quoted excerpts and evidence identities mechanically. A model cannot declare that a probe ran; only a completed, bound receipt establishes execution.
5. Derive assessment/disposition in code. Valid solver failure is compatible with a sound task. Infrastructure failure or missing assertions leave relevant checks unresolved. Optional polish is advisory, not a rejection.
6. Route confirmed task defects to the existing bounded repair loop when enabled. Re-run affected validation; keep original task, failed attempts and costs.
7. Persist feedback independently of task disposition. A disputed finding does not become false merely because someone disagrees. Adjudication records evidence, reviewer identity and unresolved disagreements; no silent deletion of findings.
8. Evaluate a candidate reviewer against the unchanged baseline before promotion. Benchmark labels and previous baseline decisions are hidden from both reviewers.

Finding and evidence contractsCopy Anchor Link

Use one stable ID per issue within an immutable review. Track later occurrences through explicit links rather than guessing that similar wording identifies the same defect. Extend existing categories without changing old serialized records.

Start with requirement alignment, feasibility/assets, reference correctness, verifier coverage, valid alternatives, reward gaming, leakage and rollout/grade agreement. These are Repo2RLEnv criteria, not a claim to copy Micro1's private rubric. Keep current task/verifier/leakage aggregates as a compatibility view.

Invariants: every failed criterion has a grounded blocking finding; a passing criterion cannot own a blocking finding; unsupported hypotheses remain unresolved; all linked evidence must exist and match its digest. Effective ratings are derived: a grounded blocking finding overrides a proposed pass; an unsupported failure becomes unknown. Raw responses and these corrections remain auditable. Advisories live separately from defect findings. New fields on historical reviews remain explicitly absent; do not infer confidence, attribution or observed evidence during migration.

OutputCopy Anchor Link

Review output lives beside the task, under the campaign, not in the learner image:

reviews/CASE/
  result.json              # versioned final result and compatibility view
  findings.json            # pointer to latest accepted review
  reviews/KEY/
    findings.json          # criteria and stable issue IDs
    evidence.json          # exact excerpts, file hashes and bound trials
    investigation.json     # hypotheses and supported/dismissed/unresolved conclusions
    reviewer.json          # pinned code, prompts, model and settings
  calls/                   # immutable metered requests and response receipts
  revisions/              # existing immutable repair outputs, if any

CLI summaries and a human-readable report link to these records. Reuse existing [metadata.repo2env.evaluation] labels only through their current publication contract; a static review does not automatically promote a task to verified. No new metadata is needed in the Harbor task for the first pilot. Published task bytes and historical grades remain intact unless an explicit new revision ships.

VerificationCopy Anchor Link

Two different kinds of mutationCopy Anchor Link

Existing semantic probes mutate a solver submission to test the verifier. The new benchmark mutates a task or verifier to test the reviewer. Keep the APIs, manifests and labels separate so a successful probe cannot be mistaken for evidence that the reviewer is accurate.

For the initial binary profile, retain the baseline/reference contrast and require the reference to reach the task's declared success reward, normally 1. A valid alternative must preserve public behavior; a wrong-solution control must fail for the intended behavioral reason, not import/setup damage. Unchanged private tests run outside the learner. Never weaken a legitimate test to increase solver success.

Pilot corpus and protocolCopy Anchor Link

Target 20 CPU task families, preferably four from each HF ML repository where independently auditable candidates exist. Do not fill a repository quota with uncertain labels. Reuse existing frozen tasks and saved rollouts where hashes match; do not rerun all fifty solvers. GPU traces can inform later cases, but new GPU execution and graded-reward tasks are outside this pilot budget.

Each family has one independently audited clean task and two separate variants, each introducing exactly one defect: 60 cases total. Candidate defects include undocumented exception wording, private-state overfitting, a missing required asset, ordering/tolerance mismatch, answer exposure, stale-output acceptance and removal of a central behavioral assertion. Existing real false-rejection examples are preferred where their labels can be independently established.

Do not equate an old verified label or an oracle pass with a clean label. A second reviewer checks each proposed clean case and each mutation rationale. A separate model can assist triage, but unresolved cases remain outside the scored clean set. Retain exclusions and reasons; they affect achieved corpus size.

Freeze 10 development / 5 selection / 5 final-test families before tuning. Group by PR/commit lineage and shared feature or near-duplicate implementation, not just task ID. All variants and correction descendants stay in the same group. This is not a claim of unseen-repository generalization.

Compare the baseline and one frozen candidate using the same pinned model, tools and budget, with two independent runs per case/profile: at most 60 × 2 × 2 = 240 review sessions. Selection results choose the candidate; final test runs occur only after that choice. Repeated sessions must bypass model-response cache reuse; immutable task/probe evidence may still be reused. Resuming the same session reuses its receipts and never charges for duplicate dispatch.

The scorer never exposes clean/defective labels, mutation patches, expected criterion/location, meaningful variant names or adjudication notes to the reviewer. Only allowlisted task bytes and scenario-appropriate evidence enter its context. After tuning uses final-test feedback, that set is retired as a fresh holdout.

Metrics and promotionCopy Anchor Link

Report counts and denominators for defect detection at the correct shared category and location (the legacy reviewer has no seven-criterion IDs), blocking findings on clean tasks, independently adjudicated finding precision, abstentions, execution errors, schema violations, tokens, cost and latency. Report definite decisions and total attempted coverage separately; unknown cases cannot disappear from the denominator. Wrong-location findings do not count as catching the planted defect. Genuine additional defects trigger label adjudication.

Detection rate is correctly localized planted defects divided by all eligible defective-case attempts. The initial scorer reports blocking on proposed-clean cases divided by all eligible clean-case attempts. Unexpected findings need adjudication before this is interpreted as a false-positive rate; provisional labels are reported separately. Decision coverage is decidable completed attempts divided by all scheduled attempts. Finding precision uses all adjudicated findings in the pilot, with counts for any unadjudicated remainder. Repeat samples stay clustered by family in comparisons.

Compute uncertainty by resampling task families while retaining variants and repeated runs together. Twenty families provide a pilot signal, not a precise production accuracy claim. A candidate must pass deterministic contract tests and known regression cases, reduce false blocking findings without observed detection regression on selection/final cases, and stay within the declared budget. If the comparison is inconclusive, keep it opt-in and retain the current default. Do not translate a small observed win into a universal accuracy guarantee.

Required implementation checksCopy Anchor Link

• Old result files and the current default profile still work; v2 is explicit.
• Fabricated quotations, cross-task receipts and inconsistent findings are rejected.
• A solver timeout cannot manufacture a task defect; an undisclosed exact-message test cannot be treated as an implicit public requirement.
• Benchmark labels and lineage variants cannot leak into reviewer inputs or splits.
• Probe setup failure is unresolved instrumentation, not successful defect detection.
• Resume is idempotent; uncertain provider charges remain held; cancellation cleans up owned workers without modifying caller-owned worker lifecycle.
• Mutation fixtures prove the intended one-defect change; both clean and defective cases are tested. No local execution of generated task code is required.

Anti-contaminationCopy Anchor Link

Private reference solutions and tests are visible only to the trusted reviewer and verifier, never the blind solver. No author feedback or benchmark labels are copied into the Harbor learner image. Treat task files, logs and prior findings as untrusted data; instructions inside them cannot override the review protocol. Probe execution uses the existing offline remote boundary with controller-held credentials.

Separate task content hashes from report annotations. A repaired task invalidates incompatible execution evidence and requires fresh checks. No historical score is silently relabeled after a repair or an accepted feedback entry.

LLM use and budgetCopy Anchor Link

Models are used at review and optional repair, not added to deterministic reward computation. Baseline and candidate use one explicit route. Start with the existing anthropic/claude-sonnet-4-6 integration for a controlled comparison; this is not a claim that it is the latest or best model. Other supported routes remain configurable. Changing the model is a separate experiment, not a reviewer-prompt improvement.

Cost assumptions, checked 16 October 2026Copy Anchor Link

Anthropic lists Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens. A planning call with 20,000 input and 2,000 billed output tokens costs $0.06 + $0.03 = $0.09. At two to three calls per review, 240 reviews imply $43.20–$64.80 of inference. These token volumes are assumptions, not measurements. All billed reasoning, context expansion, retries and adjudication calls count; no cache discount is assumed.

Daytona's posted CPU rates imply about $0.3312/hour for 4 vCPU and 8 GiB, before storage: 4 × $0.0504 + 8 × $0.0162. A planning 20–50 aggregate sandbox-hours costs $6.62–$16.56 before build/storage overhead. The exact worker resources must satisfy the task. Modal remains an alternative; use its current rates when dispatching there.

Expected total: $70–$120; the hard cap is a ceiling, not a spend target or a promise to finish if cases require substantially more investigation. Code, RFC and local synthetic tests need no additional provider/cloud spend; this excludes the user's existing coding-assistant subscription and human review time.

After a five-family smoke, recompute the estimate from actual tokens and cloud receipts before dispatching the remaining families. Stop if the remaining work does not fit the cap. Start with four concurrent model reviews and two remote CPU trials, bounded by actual provider quotas. Record known costs and unresolved holds separately. Do not reuse old campaign authorization or silently increase the cap.

For later routine usage, the same assumptions suggest $18–$27 per 100 review-only tasks, excluding probes, repair, escalation and retries. This is a projection to replace with measured per-task economics, not a quoted production price.

Optional GEPA optimization is a later experiment: provisionally $100–$200 extra for a bounded search and a new comparison. It is excluded from the $150 pilot and requires a separate measured plan. A useful reviewer and benchmark do not depend on enabling it.

Yield and repository suitabilityCopy Anchor Link

This component produces review evidence, not new environments. Report attempted, completed, decidable and independently adjudicated reviews separately. The target is 20 audited families, not 20 assumed-good tasks; achieved corpus size depends on label audit. Existing Harbor tasks from every generator can use the review API. The initial scored benchmark deliberately limits execution to CPU/binary tasks.

No quality target is defined as a solver success rate. Difficult tasks remain valuable when the public contract, reference and verifier agree. A reward-zero rollout can therefore receive a sound-task assessment.

Dependencies and code placementCopy Anchor Link

Reuse Pydantic, the shared console, campaign ledger, event journal and remote execution adapters. No new mandatory dependency or upstream research-repository installation. GEPA, if later adopted, is an optional pinned library used offline; the shipped review profile remains usable without it. Preserve license/source metadata on real tasks; do not commit entire third-party repositories or credentials.

Alternatives consideredCopy Anchor Link

• Another generation pipeline: duplicates existing quality integration and does not evaluate reviewer errors. Extend the shared quality command instead.
• A stronger reviewer model first: could help, but confounds model choice with protocol changes. Establish a fixed-model baseline first.
• Automatic prompt optimization immediately: optimizes uncertain labels before we know the benchmark is trustworthy. Build and audit cases first.
• One combined quality score: hides the difference between task defects, missing evidence and infrastructure failure. Keep criterion decisions and execution state.

Rollout planCopy Anchor Link

Stages 1–3 are roughly 2–4 focused engineering days, with case adjudication potentially adding 1–2 days. Paid execution should be scheduled from smoke measurements rather than promised from review latency alone. These are planning estimates, not an unattended completion guarantee.

Tasksmith and all recipes call the same opt-in quality profile. Do not add per-recipe review implementations. No task dataset is republished, benchmark result announced or default switched merely because this RFC or its code has merged.

Open questionsCopy Anchor Link

• Budget: a separate $150 cap is approved. Paid pilot evidence stays in the ignored campaign directory; no previous campaign allowance is reused.
• Independent labels: unresolved clean-task disputes need a maintainer or domain reviewer; model agreement alone is not ground truth. Record this labor separately.
• Corpus size: confirm twenty sufficiently independent CPU families exist after audit; report a smaller defensible pilot if they do not.
• Optional optimization: decide on GEPA only after the first comparison and observed costs; retain a fresh holdout for further iterations.

References and creditCopy Anchor Link

• Rumi Elias Calles and Tyler Houchin, micro1, Teaching the reviewer: Introducing flow QC, 13 October 2026. Method inspiration; no upstream source code, proprietary prompts or evaluation data copied.
• GEPA and its paper: possible later offline instruction optimization. Not a runtime dependency or part of this pilot's claimed results.
• RFC 0027 and the quality-loop guide: existing owned implementation.
• RFC 0032: explains why graded optimization tasks need a separate quality acceptance contract.

ImplementationCopy Anchor Link

Implementation boundariesCopy Anchor Link

The first benchmark runner compares static task reviews; ordinary quality run continues to accept bound rollouts and run remote validation. Benchmark-side rollout ingestion and remote probes are deferred. Feedback adjudication produces a model second opinion and two regression suggestions; it does not automatically create, audit or promote cases. Findings keep source excerpts and inventory hashes; derived claims require an explanation but are not automatically recomputed. The source reviewer identity and requests are pinned, while final acceptance remains the existing quality execution contract. See the usage guide for exact commands.

Initial measurements (16 October 2026)Copy Anchor Link

The development corpus contains five CPU task families from Accelerate, PEFT and TRL: one original plus two controlled verifier mutations per family. Every label is provisional. An original exported task is not automatically a clean benchmark case, even when its reference and baseline previously passed execution controls.

The last comparison used identical model, context and spending settings within the pair. It does not isolate the effect of changing models from earlier runs, establish general accuracy, or justify promoting either profile. The subsequent unit-tested correction also preserves existing entrypoint citations when deriving a failed criterion; it has not been measured in a new paid comparison.

Two engineering lessons became regression tests. First, reviewing a grader's contents is insufficient: inspect whether the selected entrypoint actually runs it. Second, a model can emit a pass rating before finding a defect later in its structured response. Derive the effective failure from the grounded finding; asking it to retry for consistency can erase the defect instead.

Inference accounted across these comparisons is $20.136099. The initial Sonnet smoke cost $11.186025 for 60 scheduled review sessions. The separate Astra comparison cost $4.270449 for six sessions, averaging approximately $0.53 per evidence-profile session and $0.90 per legacy session. These include bounded reads and protocol retries, and are small development samples, not production pricing. Remote checks, failed-request holds and adjudication are tracked separately in the ignored campaign ledger under the approved $150 cap.

Focused Modal execution subsequently confirmed the report-forgery finding in the sampled original. The exact verifier Dockerfile was built remotely, with a fresh 1 CPU / 2 GiB sandbox for each check and outbound network blocked at execution:

All five Modal sandboxes terminated. These checks directly execute the verifier entrypoint, not a full Harbor solver trial. The result establishes an exploit in this exported task; it does not establish the affected count across the published suite. The shared JUnit trust boundary needs a separate investigation and repair. Passing reference/baseline controls alone does not establish resistance to reward hacking.

The feedback CLI also completed a live second opinion on the unconditional-reward finding: upheld, with exact citations and regression suggestions, at $0.246680. An earlier provider failure remains reserved. Total accounted model usage is $20.382779, with $6.30 held for uncertain model requests and remote billing; these are ledger values, not reconciled provider invoices. No paid work remains running at the end of this smoke.

The twenty-family benchmark remains pending independent label audit. Preserve the original manifest and report; add investigation evidence and a new corpus revision rather than rewriting the earlier scores. The original case's blocked provisional clean label is not evidence of a false positive. No reviewer default or published task acceptance label changes on the strength of this smoke test.

Follow-up scope and gate decisions (17 October 2026)Copy Anchor Link

The user retained the $150 total hard cap and requested equal OpenAI/Sonnet allocations. All earlier charges and uncertain reservations remain in the same ledger. The expanded experiment schedules 80 static reviews per model: 20 HF ML families, each original plus one mutation, on both reviewer profiles. Ten families receive unconditional-reward mutations and ten receive hidden-gate mutations. One repeat per model replaces the original two repeats on one model; model strata are reported separately. This is an exploratory development corpus, not the planned audited 10/5/5 development/selection/final split. No final split was consumed and no default promotion is justified.

The corpus could not honestly supply twenty clean originals: static inspection found the shared JUnit-report boundary in all 50 local HF exports. The latest local research snapshots contain the same pattern in 650 of 1,330 tasks: 100 each from SWE-smith, SWE-gen, SWE-next, SWE-flow, R2E and R2E-Gym, plus 50 Tasksmith tasks. The HF set overlaps those 50. This inventory establishes shared source exposure, not individually demonstrated exploits or current remote Hub state. Every original label remains provisional; blocking it is not a proven false alarm.

All 160 scheduled attempts have receipts; 126 produced a completed review. An unresolved review stays in the denominator:

The Astra stratum cost $33.876137 and Sonnet $12.614294, or approximately $0.42 and $0.16 per scheduled review, respectively. Both profiles share each stratum's fixed model/settings. The bounded experiment used 50,000 context characters, one additional read round, 6,000 output tokens and a $35 route cap. Most incomplete reviews exhausted evidence/context or citation-correction limits; two Astra sessions also encountered reservation pressure at the route cap. The results are not a general model ranking: the strict context allowance affects larger tasks, original labels are unaudited, and the planted defect distribution is narrow. They support preserving evidence and investigating misses, not default promotion or a claim of calibrated false-positive rates.

Executed repairCopy Anchor Link

The concatenation repair moves assertions into a private parent and serializes worker results as data. The reusable standalone primitive runs submitted code at a reserved UID in a dedicated Linux verifier, sets no_new_privs, bounds output and time, and kills/reaps descendants before accepting a JSON value. The parent never imports the submitted module and never accepts a worker's test verdict. Network isolation still belongs to the outer sandbox.

Ten fresh Modal checks gave the intended outcomes: reference and valid alternative earned 1; unchanged source, wrong values, the original JUnit forgery, fabricated verdict JSON, private-file access, timeout, detached-child and malformed-output submissions earned 0. No worker-UID processes remained after any check. Full Harbor oracle/nop controls on the corrected migration then returned 1 and 0. The worked migration is reproducible from the existing export. Its first full trial found a historical grader-API mismatch that direct checks against the maintained helper had not exercised; the migration now copies the maintained submission validator explicitly.

This is a task-specific observable-value adapter, not a universal pytest repair. JSON does not establish arbitrary Python object identity, and the new checks are not identical to the historical suite. Other exports require deliberate assertion migration and their own execution controls. Published tasks and acceptance labels remain unchanged; the old revisions and failed runs are retained.

Bounded optimization and label provenanceCopy Anchor Link

The owned optimizer uses one to three named audited development families. It evaluates stock instructions with both models, asks each model for one short addendum, and evaluates both addenda with both models. No candidate may worsen false blocks, detection, decidability or execution errors on either model; stock wins ties. Proposal failures are recorded for both routes and prevent selection. Inputs, prompts, source identity, guidance and receipts are pinned. Final promotion still requires separate selection and final evaluations. This is not GEPA.

A separate, small DataArc report-writing family supplies calibration controls. Its private verifier parses only a bounded regular text artifact in a fresh environment; it executes no learner code. The fixed-input task's exact report format makes externally checked outputs appropriate. Eight direct Modal checks accepted the reference and two newline alternatives, while rejecting wrong counts, wrong reasons, forged verdict text, absent output and oversized output. Full Harbor oracle/nop controls returned 1 and 0. Two mutated entrypoints then demonstrated unconditional reward for nop and missing reward for a valid oracle blocked by an undocumented sentinel.

Labels are machine-assisted and execution-backed, with controller adjudication; they are not independent human ground truth. OpenAI and Sonnet independently reviewed source. Both initially missed that the separate verifier Dockerfile omitted test.sh; the full Harbor run exposed it. Sonnet also missed the hidden gate and once miscounted fixture totals. Those responses remain in the evidence, and executed controls resolve the disagreements. The final image includes the entrypoint. In Harbor 0.22 separate mode, tests are baked into the verifier image; the verifier is invoked with test upload skipped. This is why source agreement and manually staged direct checks cannot replace an end-to-end trial.

The optimizer uses only this calibration family. Its result is a development candidate decision, not evidence of improved performance on HF repository tasks. The broader independent-label and heldout gates remain unmet, so the existing reviewer default remains unchanged.

The search pilot completed stock and OpenAI-guidance comparisons on both models: 12 review sessions per model, including both reviewer profiles. Astra's evidence profile located both mutations with either prompt; Sonnet located one with either prompt. Sonnet's decidable reviews rose from one to three, a tiny development observation that does not establish a general improvement.

The Sonnet proposal copied the exact private sentinel path from the benchmark. Controller inspection rejected it and stopped its partly started evaluation; two additional legacy review results and uncertain in-flight requests are retained. No guidance was selected or promoted. The shipped optimizer now rejects exact defect quotations as well as case IDs before candidate evaluation, with regression tests that ensure both proposals are recorded and no candidate is evaluated after a rejection. The original search source snapshot and raw reports remain available; the stopped run is not rewritten as a successful optimized benchmark.

Final pilot accounting is $70.868357 accounted plus $27.10 held, totaling $97.968357 committed against the unchanged $150 cap. The remaining allowance is $52.031643. Holds include prior uncertain calls, interrupted model requests and remote execution awaiting billing reconciliation. They are not assumed free. The validation Modal application has zero active tasks, and no local paid controller remains running. Unit/contract checks, package build and documentation checks accompany the PR; no version bump or public dataset change is included.