pr_diff

Each task is a merged pull request, rewound to its base commit: the agent gets the PR's title and description and edits the repository to resolve it. A verifier compares the agent's diff with the merged diff and, when you pass it a key, asks an LLM judge whether the patch addresses the issue. Generation needs no Docker and no LLM, so pr_diff is the cheapest way to get many tasks out of a repository.

At a glanceCopy Anchor Link

QuickstartCopy Anchor Link

repo2rlenv generate \
  --repo pallets/click \
  --pipeline pr_diff \
  --pipeline-opt limit=5 \
  --out ./tasks/click-pr-diff

limit caps how many merged PRs are listed, so you get at most five tasks, one directory each (./tasks/click-pr-diff/pallets__click-<pr>/). The command exits with status 1 if every PR is filtered out. Check the tasks, then run the oracle, which applies the merged diff:

repo2rlenv validate ./tasks/click-pr-diff --oracle
harbor run -p ./tasks/click-pr-diff -a oracle --env docker

To score a real agent, pass the judge key to the verifier with --ve; --ae only reaches the agent:

harbor run -p ./tasks/click-pr-diff \
  -a claude-code -m anthropic/claude-sonnet-4-6 \
  --ak max_budget_usd=2.00 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --ve ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --env docker

How it worksCopy Anchor Link

1. List merged PRs. gh pr list --state merged on GitHub, or the merge-request API on GitLab, newest first. The listing over-fetches three times limit so that draft and since/until filtering still leaves up to limit PRs.
2. Filter on metadata. Skip drafts (skip_drafts), PRs with no changed files or more than max_files_per_pr, and PRs without a merge timestamp.
3. Fetch the diff with gh pr diff (or the GitLab equivalent). Empty diffs and fetch failures are skipped.
4. Filter on content. Skip diffs that only touch tests or only touch docs, titles that start with Revert , diffs with fewer than min_loc_changed changed lines, and PRs whose description is empty after stripping and whose title has fewer than five words.
5. Write the instruction from the PR title and description, with closing keywords, issue and PR links, GitHub URLs, commit trailers and squash suffixes removed (see Implementation notes).
6. Calibrate. Compute the reward an empty patch would get (the no-op baseline) and a difficulty bucket from the diff size.
7. Emit the task. The environment is python:3.12-slim with git and the repository checked out at the base commit, with git history past that commit removed. The oracle and verifier go in tests/, which Harbor uploads only at verify time, so the agent's image never contains the answer.

OptionsCopy Anchor Link

Pass each option with --pipeline-opt key=value.

OutputCopy Anchor Link

• task.toml: Harbor 1.0 task (name = "<org>/pallets__click-<pr>", agent timeout 1,800 s, verifier timeout 300 s) with provenance, calibration and an evaluation label under [metadata.repo2env]. The label starts as unverified.
• instruction.md: the stripped PR title and description, plus a short task statement.
• environment/Dockerfile: python:3.12-slim, git, and the repository at the base commit. An optional GITHUB_TOKEN or GITLAB_TOKEN build argument clones private repositories; the remote URL is reset afterwards.
• solution/patch.diff: the merged diff (the oracle).
• solution/solve.sh: applies patch.diff; Harbor's oracle agent runs it.
• tests/test.sh: captures the agent's edits with git add -A and git diff --cached <base>, then runs the verifier.
• tests/verifier.py: the standalone six-component scorer (Python standard library only).
• tests/oracle.patch: the reference diff the verifier compares against.
• tests/instruction.md: the verifier's copy of the instruction, for the judge.

The per-task metadata looks like this:

[metadata.repo2env]
pipeline = "pr_diff"
repo = "pallets/click"
ref = "<base commit sha>"
reference = "https://github.com/pallets/click/pull/<pr>"
reward_kinds = ["diff_similarity"]

[metadata.repo2env.pr_diff]
pr_merged_at = "<timestamp>"
diff_format = "unified"
context_files = ["src/click/…", "tests/…"]

[metadata.repo2env.reward_calibration]
baseline_reward = 0.0
loc_changed = 95
difficulty = "large"

Multi-component rewardCopy Anchor Link

The verifier scores the agent's diff against the oracle with six components and combines them as a weighted average:

The result is clipped to [0, 1]. If size_sanity is below 0.10, the reward is capped at 0.40, so a lenient judge can't inflate a patch that is wildly the wrong size. An empty patch scores 0.

When the judge is unavailable (no key or model configured, or a network or API error) or disabled, the verifier writes evaluation_status = "ungraded", eval_trustworthy = false and final_reward = null to reward-details.json and writes no reward file. Harbor raises RewardFileNotFoundError, classes the trial as VERIFIER_FAILED with reward_valid = false and does not retry it; the trial's result.json shows verifier_result: null and the exception under exception_info (the failure cause and reward_valid are computed from it, not stored). RL consumers drop the rollout rather than training on a 0, while Harbor's job mean and pass@k count it as 0. A provider reply that isn't JSON or has the wrong shape (judge_status parse) is ungraded too, as is a crashed verifier (verifier_status: "error"). A provider-reported output or context limit (output_limit) is also ungraded, even if the truncated answer contains a score. Retry with a suitable judge budget or configuration; truncation is not a correctness judgment. When the judge completes without a valid score (missing_score or invalid_score), including a refusal, a content filter or an empty reply, the run is a graded 0 instead: the agent's patch is part of the judge prompt, so it can provoke those replies. A predicted patch over 10x the oracle's changed lines (and over 1,000) or over 16 MiB is a graded 0 too (predicted_status: "predicted_too_large"), so one huge file can't slow the verifier into a timeout.

A judge that never returns a valid score, such as a self-hosted model that doesn't follow the JSON format or spends its 200-token budget on reasoning, now gives a graded 0 on every rollout rather than an ungraded run. Check that the oracle run's judge_status is ok before training on a new judge setup. Retry grading the saved patch from logs/verifier/predicted.patch when the judge is available. Do not rerun the coding agent just to recover a judge outage. diagnostic_similarity retains the weighted mean of the five deterministic components for inspection, without turning it into training reward.

Setting R2E_W_JUDGE=0 disables the judge and leaves the evaluation ungraded. A valid judge score makes this a graded heuristic; it does not prove functional correctness. Use executable tests when that stronger guarantee is needed. Override any weight per run with R2E_W_FORMAT, R2E_W_SIZE, R2E_W_FILE, R2E_W_REGION, R2E_W_SIM or R2E_W_JUDGE, passed with harbor run --ve.

The verifier removes any existing reward files, then writes the score to /logs/verifier/reward.txt (graded runs only) and the breakdown (final_reward, components, weights, judge_model, judge_endpoint, judge_status, evaluation_status, eval_trustworthy, diagnostic_similarity) to /logs/verifier/reward-details.json.

Patch capture and trust boundaryCopy Anchor Link

The verifier captures edits with a fresh Git configuration and index, using only the cloned repository's object store for the pinned base. It includes new files without running workspace-configured clean filters, external diff programs, textconv drivers, fsmonitor or hooks. A capture error is graded 0, because the agent can cause it (for example by removing the Git object store).

This does not protect against an agent that can replace system binaries, alter the object store or write verifier files while grading runs. Strong adversarial grading requires a separate trusted verifier environment with an immutable base snapshot. The LLM judge also processes untrusted patch text; it is not a security boundary. These remaining threats are tracked in issue #219 and issue #214.

Previously exported tasks embed their verifier. Run repo2rlenv migrate pr-diff on a copy to inspect required repairs; review changed assets and revalidate before publishing a new dataset revision. Old scores do not become trustworthy just because the installed package was upgraded.

LLM judgeCopy Anchor Link

judge_status records the outcome: ok, no_api_key, no_judge_model, empty_predicted, network, parse, missing_score, invalid_score or disabled. To use a self-hosted judge, start a server on the host and route the verifier to it:

harbor run -p ./tasks/click-pr-diff \
  -a claude-code -m anthropic/claude-sonnet-4-6 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --ve R2E_JUDGE_ENDPOINT=http://host.docker.internal:8000/v1 \
  --ve R2E_JUDGE_MODEL=Qwen/Qwen3.5-4B \
  --env docker

host.docker.internal resolves on Docker Desktop (macOS, Windows, WSL2). On a Linux Docker daemon, use the host's LAN IP instead.

Calibration baselineCopy Anchor Link

reward_calibration.baseline_reward is the reward an empty patch gets against the task's oracle, computed without the judge. Normalize with calibrated = (raw - baseline) / (1 - baseline); a negative value means the agent did worse than doing nothing. With the current components the baseline is 0.

Difficulty bucketingCopy Anchor Link

reward_calibration.difficulty buckets the oracle by changed lines: trivial (5 or fewer), small (6–20), medium (21–80) and large (more than 80), with the raw count in loc_changed. metadata.difficulty in task.toml uses the same buckets, with trivial written as easy.

Scoring outside HarborCopy Anchor Link

For text-only training loops, score a predicted diff directly. The single-component SWE-RL-style similarity is:

from repo2rlenv.reward import calculate_diff_similarity_reward

reward, meta = calculate_diff_similarity_reward(oracle_diff, predicted_diff)

For the full six-component score, import the functions in _pr_diff_verifier.py. See Rewards for how this compares with the test-based rewards.

Yield and costCopy Anchor Link

No complete candidate count or generation cost was recovered for the reference dataset, so there's no measured yield. Yield depends on the filters alone: there's no execution gate, so every listed PR that survives them becomes a task. The run summary counts each skip reason (draft, too_many_files, test_only_diff, docs_only_diff, revert_pr, diff_too_small, instruction_too_thin, and so on). Generation makes no model calls; the only model cost is one judge call per scored attempt.

An earlier 23-trial Claude Sonnet 4.6 pilot on the first 100-task release had a median reward of 0.71 (range 0.16–0.98). These are continuous scores, not a solve rate. See native results.

LimitsCopy Anchor Link

• A generated task isn't a verified environment. Run the controls: the oracle should score 1.0 (exactly 1.0 without a judge) and a no-op agent (-a nop) 0. Record the outcome as an evaluation label; see Quality.
• The published dataset predates oracle isolation. Tasks generated before #145, including the 181-task reference dataset, bake the oracle into the agent's image. Repairing that dataset is tracked in #155. Regenerate tasks, and rebuild cached images, before you train or evaluate on them.
• Similarity isn't correctness. A valid fix that differs from the merged one scores lower on the deterministic components. The judge carries half the weight to offset this, but judges are noisy, especially small self-hosted ones.
• No egress guard. Git history is removed, but pr_diff doesn't ship the network denylist the runtime pipelines use (see contamination defenses). An agent with web access can look up the merged PR.
• The instruction is written by the fixer. Stripping removes links and references, not prose, so some descriptions still explain the approach.
• GitLab listing sees at most the newest 100 merged MRs, so a limit above about 33 won't reach older ones.

RelatedCopy Anchor Link

• RFC 0001: pr_diff
• Reference dataset and its release history
• pr_runtime: the same PRs, verified by running their tests
• Tasks, Rewards and Run with Harbor
• Inspired by SWE-RL (Wei et al., 2025). The verifier is an independent reimplementation.

Implementation notesCopy Anchor Link

Source: pipelines/pr_diff.py and pipelines/_pr_diff_verifier.py.

Instruction info-leak stripCopy Anchor Link

The instruction builder removes these pattern families from the PR title and body. Composite forms run first, so no orphaned Closes keywords or empty [text]() brackets are left behind:

Environment and verifier detailsCopy Anchor Link

• The Dockerfile clones over HTTPS (SSH and http:// inputs are rewritten), fetches the base commit, runs git reset --hard and git clean, then scrubs history: it removes origin, deletes every other branch and tag, expires the reflog and garbage-collects, keeping only the base commit reachable. The clone is partial (--filter=blob:none), and gc keeps unreachable objects from partial-clone packs, so the scrub then repacks just what the base reaches and drops the old packs.
• No agent tooling is installed. Harbor's agent adapter installs what it needs when the container starts, so the same task works with any harness.
• test.sh stages new files before diffing (git add -A). Without that, files the agent creates wouldn't count toward file targeting or region overlap.
• The judge prompt truncates the instruction, oracle and prediction to 4,000 characters each and asks for a JSON score. The default Anthropic route keeps the provider's sampling defaults, which the weights were tuned against; custom endpoints are pinned to temperature 0.
• The default weights were retuned after a 23-task pilot: format_valid was 1.0 on every evaluated task (no signal), and similarity correlated strongly with region_overlap, so the judge took the freed weight.