code_instruct

Each task asks the agent to write a new module, task_module.py, that builds on the target library's API. An LLM invents the problem, a pytest test and a reference solution from a random snippet of the repository in one call. The pipeline keeps the task only if the test fails without the solution and passes with it inside the repository's Docker image; the hidden test is the reward.

At a glanceCopy Anchor Link

QuickstartCopy Anchor Link

repo2rlenv generate \
  --repo pallets/click \
  --pipeline code_instruct \
  --pipeline-opt limit=5 \
  --pipeline-opt seed=42 \
  --llm anthropic/claude-sonnet-4-6 \
  --out ./tasks/click-code-instruct

Here limit is the number of tasks to emit; the pipeline samples up to five snippets per requested task before it gives up. seed makes snippet sampling repeatable. The first run bootstraps the repository (capped by --max-spend-usd, default 5.0, and cached). Each task lands in ./tasks/click-code-instruct/pallets__click-cinst-<hash>/. Run the oracle, which should score 1.0:

harbor run -p ./tasks/click-code-instruct -a oracle --env docker

How it worksCopy Anchor Link

1. Bootstrap the repository once and shallow-clone it at --ref. See Bootstrap.
2. Find the package. Read [project].name from pyproject.toml, or look for src/<repo>/__init__.py or <repo>/__init__.py, and list the package's top-level symbols.
3. Sample a seed. Pick a random file matching file_glob and not exclude_glob, then a random window of seed_min_loc to seed_max_loc lines. Windows that are at least 80% blank lines, comments or imports are resampled.
4. Synthesize. One call asks for three sections, [Problem Description], [Test] and [Solution]. The prompt requires the problem to build on the package's public API, the test to import from task_module with 3–6 assertions including a pytest.raises, and the solution to import and use the package.
5. Gate the candidate. Reject it if it matches a known benchmark phrase, if the test doesn't import task_module, if the solution doesn't import and use the package, if a top-level name in the solution already exists in the package, if the test is weak, or if it duplicates an earlier task. A rejected candidate is retried on the same seed up to max_attempts_per_seed times.
6. Verify in the sandbox. With only the test file in place, pytest must not pass. With the solution written to task_module.py, it must pass.
7. Emit the task. The instruction is the generated problem plus a delivery contract: write the implementation to /workspace/task_module.py, don't modify existing files, and expose the names the problem mentions.

OptionsCopy Anchor Link

Pass each option with --pipeline-opt key=value.

OutputCopy Anchor Link

• task.toml: Harbor 1.0 task. reference links to the seed lines; [metadata.repo2env.code_instruct] records the seed path and line range, the test file name, the bootstrap image and llm_cost_usd. The evaluation label starts as unverified.
• instruction.md: the generated problem statement and the delivery contract.
• environment/Dockerfile: FROM the bootstrap image, with the repository as bootstrapped.
• solution/patch.diff: creates task_module.py (the oracle); solve.sh applies it.
• tests/test_r2e_<hash>.py: the generated test. Harbor delivers tests/ at verify time for every agent.
• task.toml sets [verifier] environment_mode = "separate" and collects only /workspace/task_module.py: Harbor grades in a fresh container from the task image with that one file copied in.
• tests/test.sh: copies the test into /workspace, runs python -m pytest on it as an unprivileged user with a JUnit report, and calls tests/verifier.py (see pr_runtime for the launcher and time limits).

RewardCopy Anchor Link

tests/test.sh writes 1.0 to /logs/verifier/reward.txt when the test file passes and 0.0 otherwise. There is no partial credit. A no-op agent scores 0, because the test can't import task_module. See Rewards.

The verifier reads the JUnit report only after every process of the test user is gone. Reward is 1.0 only if pytest exits 0 and the report holds exactly the test IDs the reference solution passed at generation (tests/expected.json, root-only), every one passed: a skipped, missing or extra test fails, so a report the submission writes itself and exits early with scores 0. Generation applies the same verdict. reward-details.json records the counts and parse_status; see the reward schema. Forged results from inside the test process (a patched report, the expected IDs read from the test file and reported as passing) are caught by the mutant check: the test also runs with the expected value of a few of its fail-to-pass assertions changed (for equivalence_tests, a recorded reference result), and a contradiction between the runs scores 0 as tampered (see pr_runtime).

Yield and costCopy Anchor Link

The reference generation log records 136 sampled seeds for 100 exported tasks, a 73.5% yield. Model retries happen within a seed, so they don't add to the count.

The most common rejections were solutions that failed their own tests and duplicate tasks. Recorded synthesis cost was $3.78 for the 100 tasks, about $0.04 per task, including retries but not bootstrap, compute or solver runs. In a five-task sample, one task per repository, Claude Sonnet 4.6 in Claude Code scored 1.0 on four and 0 on one, for $0.27 in model usage. That's a small sample, not a solve rate. See native results.

What moves yield: the model's skill at writing a test and a solution that agree, seed size (very small or very large windows give weaker tasks), and whether the package can be detected. Cost scales with limit × max_attempts_per_seed calls. Unlike the runtime pipelines, the repository's own test suite doesn't need to pass, but the repository still has to bootstrap.

LimitsCopy Anchor Link

• A generated task isn't a verified environment. Run the controls: the oracle should score 1.0 and a no-op agent (-a nop) 0. Record the outcome as an evaluation label; see Quality.
• The model invents the problem. The generated test is the only specification, so it can encode choices the instruction doesn't state. Review instructions against their tests.
• Anchoring depends on package detection. If no package is found, the repository-anchoring and name-collision gates are skipped, and tasks can drift into generic exercises.
• Decontamination is a short phrase list, not a benchmark index.
• Python and pytest only.
• llm_cost_usd is cumulative for the run, not the cost of that task.
• Tasks depend on a local image until you publish them with repo2rlenv push.

RelatedCopy Anchor Link

• RFC 0004: code_instruct
• Reference dataset and its generation evidence
• equivalence_tests: tasks grounded in a real function instead of an invented one
• Tasks, Rewards and Run with Harbor
• Adapted from OSS-Instruct in Magicoder (Wei et al., 2024): seeds come from one repository instead of a corpus, and every task ships an executable test that the reference solution must pass. No code is copied.

Implementation notesCopy Anchor Link

Source: pipelines/code_instruct.py and pipelines/_oss_instruct.py.

The gates and the skip reasons they record:

• The gold patch creates only task_module.py. The test ships in tests/ rather than the patch, because Harbor stages solution/ only for its oracle agent, so other agents would otherwise run pytest against a missing file.
• The delivery contract exists because agents often wrote correct code to a naturally named file, and pytest then failed with ModuleNotFoundError: No module named 'task_module'.
• Sandbox checks reset the working tree with git reset --hard and git clean before each candidate, and decide pass or fail from pytest's summary line.