Quickstart

Install Repo2RLEnvCopy Anchor Link

pip install repo2rlenv

Then sign in to GitHub. Repo2RLEnv uses gh to list merged PRs and fetch their diffs.

gh auth login

If you'd rather not sign in interactively, set GITHUB_TOKEN to a read-scoped token instead. Installation covers extras and other credentials.

Generate tasks from three PRsCopy Anchor Link

repo2rlenv generate \
  --repo pallets/click \
  --pipeline pr_diff \
  --pipeline-opt limit=3 \
  --out ./tasks

This lists the three most recently merged PRs, drops those that make weak tasks (more than five changed files, test-only or docs-only diffs, reverts, diffs under three changed lines), and writes one task per remaining PR to ./tasks. The summary tells you what happened:

candidates  3
emitted     1
skipped     2 ({'too_many_files': 2})

limit counts PRs listed, not tasks emitted, so you often get fewer tasks than the limit. If nothing is emitted, the command exits with status 1. Raise the limit (for example, limit=10) and run it again.

Look at what you madeCopy Anchor Link

Each task is a directory named after its source PR:

Harbor uploads tests/ only when it verifies, so the agent never sees the reference diff. The reward is a weighted diff similarity between 0 and 1. It combines file targeting, region overlap, line similarity and patch size, plus an LLM judge. Without a judge, the run is ungraded and writes no reward file; a completed judge answer without a valid score is a graded 0, while a reported output limit stays ungraded. Deterministic similarity remains diagnostic only. pr_diff documents the weights.

Validate the tasksCopy Anchor Link

repo2rlenv validate ./tasks --oracle

This is a static check. It confirms each task.toml parses, the instruction, environment and verifier files exist, and solution/solve.sh and solution/patch.diff are present. It doesn't build the image or run anything. The next step does.

✓ default/pallets__click-<pr>
✓ all 1 tasks valid

Run the tasks with HarborCopy Anchor Link

Install Harbor and run the oracle agent first. It applies the reference solution, so a successfully graded task should score 1.0. PR-diff tasks need a correctness judge: set ANTHROPIC_API_KEY and pass it to the verifier, including for the controls. A missing or unreachable judge leaves the result ungraded: no reward file, so Harbor reports the trial as VERIFIER_FAILED. Diagnose that separately from a graded oracle failure.

uv tool install harbor
harbor run -p ./tasks -a oracle --env docker \
  --ve ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY"

The nop agent changes nothing, so every task should score 0.0. Together these two runs are the task's controls.

harbor run -p ./tasks -a nop --env docker \
  --ve ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY"

Now run a real agent, Claude Code with Sonnet:

export ANTHROPIC_API_KEY=sk-ant-...
harbor run -p ./tasks \
  -a claude-code -m anthropic/claude-sonnet-4-6 \
  --ak max_budget_usd=2.00 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --ve ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --env docker

Harbor writes results to ./jobs/<timestamp>/, with each trial's verifier/reward.txt and verifier/reward-details.json. Browse them with harbor view jobs. To run in a cloud sandbox instead of local Docker, replace --env docker with --env modal, daytona, e2b or runloop.

Push to the Hub (optional)Copy Anchor Link

hf auth login
repo2rlenv push ./tasks <you>/click-pr-diff

push creates the dataset repository (add --private for a private one) and uploads the tasks with a dataset card, a manifest.json and a Harbor-compatible registry.json. pr_diff tasks build from a public base image, so no container registry is involved. Anyone can download the dataset again with repo2rlenv pull <you>/click-pr-diff.