Quality and verification

An exported task is a generation result: it passed the checks its generator runs, and nothing more. Quality is established afterward, by controls (the oracle scores 1 and the nop scores 0) and by the review and repair loop. The outcome is recorded as an evaluation label bound to the task's exact content. This page explains each layer and what it can and can't tell you.

Generated vs verifiedCopy Anchor Link

ControlsCopy Anchor Link

Static validationCopy Anchor Link

repo2rlenv validate reads task files and metadata. It never builds or runs anything.

repo2rlenv validate ./tasks            # each task.toml parses and names its task
repo2rlenv validate ./tasks --deep     # plus instruction, environment, tests/test.sh, verifier files, reproducibility
repo2rlenv validate ./tasks --oracle   # --deep, plus solution/solve.sh and solution/patch.diff

Use it on native output. For recipe and Tasksmith bundles, rely on the Harbor controls below.

Nop and oracle runsCopy Anchor Link

The nop agent does nothing, and the oracle agent runs solution/solve.sh. Run both before you trust a dataset:

harbor run -p ./tasks -a nop --env docker      # every task should score 0
harbor run -p ./tasks -a oracle --env docker   # every task should score 1

SCALER tasks score −1 and +1 instead of 0 and 1. Controls are necessary but not sufficient: a verifier that checks too little passes both of them, which is why the loop below also probes with a wrong solution.

Review and repairCopy Anchor Link

repo2rlenv quality run takes an existing Harbor task and establishes whether it is sound. It is a shared component, not a generator, and it never edits its input: each repair produces a new task directory.

The flags choose how far the loop goes:

repo2rlenv campaign init ./workspace/my-campaign --budget-usd 25
repo2rlenv quality run ./tasks/swe-smith-4f1c2a \
  --campaign ./workspace/my-campaign \
  --out ./workspace/reviews/swe-smith-4f1c2a \
  --repair --provider modal \
  --runtime-wheel ./dist/repo2rlenv-0.9.3-py3-none-any.whl \
  --max-repairs 3 --max-spend-usd 15
repo2rlenv quality show ./workspace/reviews/swe-smith-4f1c2a

Every run ends in one disposition, stored as status in result.json:

The command exits 0 for usable and reviewed, 1 for the other dispositions, and 2 for setup errors. Automation that needs validated tasks must check status == "usable", not the exit code. Code derives the disposition from evidence, so a model declaring success cannot override a failed control or probe. See Review and repair a Harbor task for models, budgets, resumption and the output layout, and the CLI reference for every flag.

Evaluation labelsCopy Anchor Link

Every emitter writes an [metadata.repo2env.evaluation] table into task.toml at export time, with status unverified, stage generation and reason code validation_not_run. The label changes only when evidence changes it.

A quality result maps onto a label like this:

verified can only come from a saved quality result. The importer rechecks that result against the task's bundle hash, the nop and oracle contrast, both probe kinds, the reviewed rollout and the digest of every trial file. If any of that evidence is missing or altered, the task cannot be labeled verified. The attempt is kept with an evidence_unavailable reason instead.

quality run writes a labeled copy of its final revision under OUT/labeled/<result-digest>/<task-name>. Use repo2rlenv tasks to label, inspect and filter tasks yourself:

# Import a saved quality result as a label on a new copy
repo2rlenv tasks label ./workspace/reviews/swe-smith-4f1c2a/revisions/r1/swe-smith-4f1c2a \
  --out ./catalog/swe-smith-4f1c2a \
  --quality-result ./workspace/reviews/swe-smith-4f1c2a/result.json

# Record an operational diagnosis without evidence
repo2rlenv tasks label ./attempt/swe-smith-9b07d3 --out ./catalog/swe-smith-9b07d3 \
  --status blocked --stage bootstrap --reason-code runtime_incompatible \
  --detail "Verifier used the wrong interpreter; rebuild and rerun controls."

repo2rlenv tasks show ./catalog/swe-smith-4f1c2a
repo2rlenv tasks list ./catalog --status needs_repair

tasks label always writes a new directory and never overwrites one. The label is excluded from the bundle hash, so labeling doesn't change a task's identity. It does change Harbor's own directory checksum, because that covers task.toml. --status verified is deliberately unavailable. See Evaluation labels for the stored fields and the identity rules.

LimitsCopy Anchor Link

• The loop's remote execution (controls, probes, rollout and repair) currently accepts only single-container, no-network Dockerfile tasks. That is the shape of recipe and Tasksmith bundles. Native tasks allow network access, and the runtime ones ship a compose overlay, so they can get a review-only pass but not a usable result yet. Run their controls with Harbor directly.
• Probes are two targeted counterexamples, not an exhaustive adversarial suite. A verifier can still miss a requirement that neither probe exercises.
• The campaign budget is an accounting limit, not a bill cap enforced by the provider.
• A label describes the evidence it references. Reading a label does not prove that the evidence is still present, so keep quality results and trial files with any labeled corpus you distribute.