pr_diff¶

SWE-RL-inspired PR mining with a Harbor-runnable multi-component verifier. Each emitted task is a real merged GitHub PR; the agent edits the repo and the verifier scores the diff with five deterministic components plus an LLM-as-judge.

What it does¶

flowchart TD
    A[Repo URL<br/>e.g. pallets/click] --> B[gh pr list<br/>--state merged]
    B --> C{For each PR}
    C --> D{Filters<br/>drafts? file count?<br/>test-only? docs-only?<br/>revert? min LOC?}
    D -- skip --> Z[Count skip reason<br/>continue loop]
    D -- pass --> E[gh pr diff #N]
    E --> F[Strip info-leak from<br/>instruction title + body]
    F --> G[Compute baseline reward<br/>+ difficulty bucket]
    G --> H[Build Harbor task]
    H --> I[task.toml<br/>+ instruction.md<br/>+ solution/patch.diff<br/>+ solution/solve.sh<br/>+ environment/Dockerfile<br/>+ tests/test.sh]

For each merged PR within scope:

1. List PRs via gh pr list, apply structural filters (date, draft, file count) + quality filters (drop test-only, docs-only, reverts, trivially-small diffs).
2. Fetch the unified diff via gh pr diff.
3. Strip leakage patterns from the PR title + body (eight pattern families — see Instruction info-leak strip below).
4. Compute the calibration baseline (the score an empty patch would get against this oracle) and the difficulty bucket by LOC changed.
5. Emit a Harbor-spec task: instruction.md, solution/{patch.diff, solve.sh}, environment/Dockerfile, tests/test.sh, task.toml.

The environment is a thin, agent-agnostic python:3.12-slim image with git + the repo checked out at base_commit — no agent CLI is pre-installed. Harbor's agent adapter (-a claude-code, -a openhands, -a codex, -a aider, …) drops in the runtime its agent needs when the container starts. The verifier (tests/test.sh) runs after the agent and computes the multi-component reward.

Private repos work the same way — the Dockerfile clones via an optional GITHUB_TOKEN build arg. Public repos need nothing; for a private source, the consumer passes --build-arg GITHUB_TOKEN=$GITHUB_TOKEN to harbor run. The token is used only for the clone and the remote is scrubbed afterward, so it never persists in the image. See reference/AUTH.md.

Multi-component reward¶

The verifier captures the agent's edits as a unified diff against base_commit, then scores it against the oracle (gold) diff using six components, weighted sum:

Final reward is clipped to [0, 1]. A catastrophic-size hard cap clamps the final to ≤ 0.40 when size_sanity < 0.10 — stops a charitable judge from inflating scores on patches that are wildly the wrong size.

The verifier writes both /logs/verifier/reward.txt (single float, Harbor reads this) and /logs/verifier/reward.json (full breakdown for downstream inspection / re-weighting).

Weights are overridable per-task via task.toml.metadata or per-run via R2E_W_{FORMAT,SIZE,FILE,REGION,SIM,JUDGE} env vars passed to harbor run --ve.

Calibration baseline¶

Each task carries task.toml.metadata.repo2env.reward_calibration.baseline_reward — the reward an empty predicted diff would get against this oracle. Consumers can normalize:

calibrated = (raw - baseline) / (1 - baseline)

calibrated < 0 means the agent did worse than no-op. Useful for cross-task comparability since trivial 5-line fixes and 200-line refactors are no longer the same number.

Difficulty bucketing¶

Each task carries task.toml.metadata.difficulty ∈ {easy, small, medium, large} based on oracle LOC changed (≤ 5, 6 – 20, 21 – 80, > 80 respectively), plus a raw loc_changed int. Lets training scripts weight or filter by difficulty.

Instruction info-leak strip¶

PR descriptions often contain pointers to the answer. The pipeline strips eight pattern families from instructions before they reach the agent:

Composite patterns are stripped before piece-wise patterns so we don't leave orphaned Closes keywords or empty [text]() markdown brackets behind.

Options¶

class PRDiffOptions(BaseModel):
    limit: int = 50
    since: date | None = None
    until: date | None = None
    state: Literal["merged", "all"] = "merged"
    diff_format: Literal["unified", "search_replace"] = "unified"
    max_files_per_pr: int = 5
    skip_drafts: bool = True
    emit_harbor_env: bool = True
    min_loc_changed: int = 3

Skip reasons¶

A PR may not become a task. The pipeline records counts per reason:

Yield¶

Yield = emitted tasks ÷ merged PRs examined. pr_diff is the highest-yield pipeline (~80–95%) because there is no execution gate — any merged PR with a non-trivial diff and a usable problem statement becomes a task. Nothing is dropped for failing to build or run.

The only structural drops are the buckets in the table above (test/docs-only, reverts, too-small, too-thin) — typically 5–20% combined on a healthy repo.

Worked example: to ship 100 tasks at ~90% yield you need ~110 merged PRs — one mid-sized repo (or two small ones) clears it in a single pass. This is why the reference dataset (AdithyaSK/repo2rlenv-pr-diff, 100 envs) was built from just a handful of repos.

Example invocations¶

CLI¶

# Generate one env from a single repo
repo2rlenv generate \
  --repo pallets/click \
  --pipeline pr_diff \
  --pipeline-opt limit=5 \
  --out ./datasets/click-prdiff

# Run it through harbor with the oracle adapter (must score reward=1.0)
harbor run -p ./datasets/click-prdiff -a oracle --env docker -n 1

# Run it through harbor with a real agent.
# The verifier's LLM judge also needs an API key — pass via --ve so it
# reaches the verifier container (the --ae key only reaches the agent).

# Example 1: claude-code + Sonnet 4.6 (what we used to verify the
# reference dataset).
harbor run \
  -p ./datasets/click-prdiff \
  -a claude-code -m anthropic/claude-sonnet-4-6 \
  --ak max_budget_usd=2.00 --ak max_turns=30 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --ve ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --env docker -n 1

# Example 2: same env, different agent — openhands + GPT-4o.
harbor run \
  -p ./datasets/click-prdiff \
  -a openhands -m openai/gpt-4o \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  --ve ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --env docker -n 1

# Harbor ships 25+ agent harnesses you can swap in here:
#   claude-code · openhands / openhands-sdk · codex · aider · gemini-cli
#   copilot-cli · opencode · cursor-cli · qwen-coder · kimi-cli · goose
#   mini-swe-agent · swe-agent · nemo-agent · terminus-2 · trae-agent · ...
# Each one expects its own provider env var via `--ae`. Run
# `harbor run --help` to see the full list.

# Generate locally, then push to HF Hub
repo2rlenv push ./datasets/click-prdiff <your-org>/<dataset-name>

Python¶

from pathlib import Path
from repo2rlenv.spec.input import (
    GenerationInput, RepoSpec, PipelineSpec, OutputSpec, PipelineName,
)
from repo2rlenv.spec.options import PRDiffOptions
from repo2rlenv.pipelines.pr_diff import PRDiffPipeline

g = GenerationInput(
    repo=RepoSpec(url="pallets/click", access="auto"),
    pipeline=PipelineSpec(name=PipelineName.PR_DIFF, options={}),
    output=OutputSpec(destination="./out", org="myorg", dataset_name="click-prdiff"),
)
options = PRDiffOptions(limit=5, max_files_per_pr=10)

pipeline = PRDiffPipeline(g, options)
result = pipeline.run(Path("./out"))

print(result.candidates, result.emitted, result.skip_reasons)

Pulling the reference dataset¶

A verified reference dataset is published on HF Hub:

https://huggingface.co/datasets/AdithyaSK/repo2rlenv-pr-diff

repo2rlenv pull AdithyaSK/repo2rlenv-pr-diff /tmp/pr_diff-ref
repo2rlenv validate /tmp/pr_diff-ref

# Smoke-check with the oracle (must score reward=1.0)
harbor run -p /tmp/pr_diff-ref -a oracle --env docker -n 1

You can also browse it interactively via the Harbor Visualiser badge on the dataset card.

[metadata.repo2env] schema¶

Each emitted task carries:

[metadata.repo2env]
pipeline = "pr_diff"
pipeline_version = "0.3.0"
repo = "pallets/click"
ref = "<base_commit_sha>"
reference = "https://github.com/pallets/click/pull/3508"
built_at = "2026-05-26T..."
reward_kinds = ["diff_similarity"]

[metadata.repo2env.pr_diff]
pr_merged_at = "2026-05-23T..."
diff_format = "unified"
context_files = ["src/click/shell_completion.py", "tests/test_shell_completion.py"]

[metadata.repo2env.reward_calibration]
baseline_reward = 0.0
loc_changed = 95
difficulty = "large"

Consuming the reward at training time¶

Two paths:

(a) Per-task reward via Harbor. Use harbor run against the emitted task directory. The verifier writes reward.txt (single float) and reward.json (full breakdown). This is the production path for evals.

(b) Pure-text reward for SWE-RL-style training rollouts. Score a candidate prediction against the oracle directly:

from repo2rlenv.reward import calculate_diff_similarity_reward

oracle = (task_dir / "solution" / "patch.diff").read_text()
reward, meta = calculate_diff_similarity_reward(oracle, prediction_diff)

This is the (single-component, dense) signal SWE-RL trained on. For higher fidelity, run the predicted diff through the same six-component verifier — it's a pure-stdlib module at _pr_diff_verifier.py that can be imported directly.

Acknowledgments¶

The text-only PR-as-task formulation + the diff-similarity reward shape are inspired by SWE-RL (Wei et al., NeurIPS '25, arXiv:2502.18449). The LLM-as-judge component follows the constitutional-AI / RLAIF pattern. No code is copied from SWE-RL; the reward function is an independent reimplementation against the Python standard library and is released under Apache-2.0.