commit_runtime: filter / leak / artifact fixes + 52-env reference dataset (Arc 3)

This release sharpens commit_runtime end-to-end and ships 52 oracle-verified environments as the reference dataset. Kept experimental for now: the underlying mining problem (commits without linked issues yield thinner instructions than PRs do) hasn't fully gone away, but every other lever has been pulled.

Published datasetCopy Anchor Link

https://huggingface.co/datasets/AdithyaSK/repo2rlenv-commit-runtime (added to the Verifiable RL Environments collection).

• 52 environments, all oracle-verified (reward == 1.0 with the gold patch).
• Resolution split: 52 resolved (tracked SWE-bench-style) · 47 command_resolved (clean test command) · 47 eval_grade (command_resolved + non-empty P2P regression guard).
• 12 source repos: urfave/cli 15 · gin-gonic/gin 8 · pallets/click 7 · gorilla/mux 6 · python-attrs/attrs 4 · stretchr/testify 3 · spf13/cobra 3 · pocketbase/pocketbase 2 · pallets/werkzeug · encode/httpx · sirupsen/logrus · psf/requests (1 each).
• Go-heavy by design. See Audit § Repo skew below.
• Same shape as the pr_runtime reference dataset (manifest with per-task build/oracle status + checksums + dataset commit; plain tests/verifier.py + tests/f2p.json + tests/p2p.json artifacts; inline environment/Dockerfile from public bases).

What changedCopy Anchor Link

commit_runtime already reuses pr_runtime's validation harness, so most of Arc 2's wins came along for free. Arc 3 was about everything upstream of validation (the mining filters and the instruction text) plus one critical Arc-2 inheritance bug.

1. Non-bugfix conventional-commit-type rejectionCopy Anchor Link

Before Arc 3, _CC_PREFIX_RE only stripped the conventional-commit type from a subject. So chore: bump version, docs: typo, feat: add X, refactor: rename Y, style: black format, test: cover Z, ci: pin actions, build: drop py3.10, perf: hot loop, and revert: … all sailed past the filter, got bootstrapped, and burned validation cycles before finally getting rejected at the F2P stage. The new _NON_BUG_TYPE_RE returns non_bugfix_type from _metadata_filter for those, mirroring pr_runtime._NON_BUG_TITLE_RE.

2. Bugfix positive-signal filterCopy Anchor Link

A commit with no type prefix had no positive bugfix signal either, so plain-subject feature commits ("Update README", "Improve performance") slipped through. Now _metadata_filter requires at least one of:

• fix: prefix
• A Closes #N / Fixes #N issue trailer
• A bugfix keyword in the subject (fix / bug(fix|s)? / regression / crash(ed|ing)? / broken / incorrect(ly)? / wrong(ly)? / fail(s|ed|ing|ure)? / defect / hotfix / patch(ed)?)

3. Issue-fetch fallback for the instructionCopy Anchor Link

The biggest single quality lever, ported from Arc 2's PR-body leak fix. Commit messages routinely name the function being fixed ("Fix off-by-one in Pager.advance"), describe the fix approach, and link to companion fix-PRs or commit SHAs. The bug report doesn't usually do any of that.

When a commit has Closes #N, build_instruction_from_commit now fetches the issue body (github.fetch_issue, same path pr_runtime uses) and renders that as the problem statement. Commits without a linked issue fall back to the commit subject + body, run through _strip_info_leak and _reflow_pr_body. The smoke test on pallets/click showed the first emitted task's instruction came verbatim from the GitHub issue, not the commit message, which is exactly the design.

4. Solution-leak strip + extended patternsCopy Anchor Link

Imported _strip_info_leak from pr_runtime and applied it to commit subject + body. Plus two new patterns covering the audit's residual soft leaks:

• Trailing (#NNNN) squash trailers: Pallets/Werkzeug/Flask/httpx all squash-merge PRs with (#1234) appended to the subject; that's a direct lookup back to the merged fix.
• Cross-repo issue refs without a closes keyword (gorilla#739 style): the existing _LEAK_PATTERNS only caught the owner/repo#N form when paired with a closes keyword.

Both shared with pr_runtime since the patterns are universally applicable.

5. Body reflowCopy Anchor Link

Ported _reflow_pr_body (drop HTML template comments, stop at checklist headers, collapse blank runs, length cap). Commit bodies can be just as verbose as PR bodies, especially when authors paste in CI output or sign-off lines.

6. reward_calibration stamping (parity with pr_runtime)Copy Anchor Link

Emitted tasks now carry [metadata.repo2env.reward_calibration] with f2p_count, p2p_count, source_files, loc_changed, and a bucketed difficulty, and the HarborTask.difficulty is set from the bucket rather than the hard-coded "medium". The launch-side manifest enricher (plans/build_enriched_manifest.py) reads these to compute the eval_grade flag without re-parsing the diff.

7. The critical bug: emit graded test.sh + plain artifacts (Arc 2 inheritance miss)Copy Anchor Link

Arc 2 refactored pr_runtime to ship tests/{verifier.py,f2p.json,p2p.json} as plain task artifacts and to call them from test.sh. commit_runtime never inherited that: it called pr_runtime.build_eval_script without fail_to_pass / pass_to_pass, which falls back to the binary exit-code path. Result: emitted test.sh only wrote reward.txt (binary 1.0/0.0), reward.json was never written, and tracked / command_resolved / the full F2P/P2P breakdown were silently lost across the whole dataset.

Two fixes:

• Pass fail_to_pass=…, pass_to_pass=… to build_eval_script so the graded path triggers.
• Pass aux_files=_runtime_aux_files(fail_to_pass, pass_to_pass) to the emitted HarborTask so the three artifacts ship with the task dir.

The oracle gate caught this immediately: 0/56 reward.json files on the first run vs 52/52 reward.json after the fix.

Validation evidenceCopy Anchor Link

Full 100-task-style oracle gate run on the 52-env dataset:

• 52/52 tracked-resolved (gold patch always satisfies the F2P+P2P sets: the oracle invariant).
• 47/52 command_resolved. The 5 that aren't: 3× spf13/cobra (one untracked failure each: sibling test in the same file that the gold patch doesn't touch) + 2× stretchr/testify (22 untracked failures each: Go-subtest parser over-counts; see Limitations below).
• 47/52 eval_grade. All 47 command_resolved tasks also have p2p_count > 0, so eval_grade matches command_resolved exactly on this dataset.

A separate qualitative audit on 9 stratified tasks (one per repo, eval_grade + edge cases): 5/9 gold-standard (clean issue body sourced from GitHub, well-scoped F2P/P2P, real source fix), 3/9 mediocre (thin commit-message-fallback instructions when no linked issue), 1/9 weak (270-char title-only prompt, also one of the 4 broken-oracle tasks that was dropped pre-publish). The mediocre tail is the structural commit-runtime limitation, not a fixable bug.

4 broken-oracle tasks dropped from the 56 generated: psf__requests-6404f3, stretchr__testify-15f682, urfave__cli-195aaf, urfave__cli-b4f42d. All scored 0.97-0.999 on the gold patch: single flaky P2P each, same shape as Arc 2's urfave__cli-2290/TestCompletionShell. Could be salvaged with the same flaky-P2P prune Arc 2 used; left for a future polish pass.

LimitationsCopy Anchor Link

• Thin instructions when no linked issue. ~30% of tasks have no Closes #N and fall back to the commit subject + body. When the commit subject is a one-liner ("Fix RFC 2069 mode digest authentication"), the agent has very little to work with. This is the structural difference vs pr_runtime and the reason commit_runtime stays experimental.
• Go-subtest parser over-counts untracked failures. The verifier's Go-test log parser treats each subtest line as a tracked test, so when a subtest fails inside a non-F2P parent the count of "untracked failed" balloons (22 for stretchr/testify even though the test command exited 0). Affects command_resolved / eval_grade flagging only; resolved and the graded reward are correct. Subtest-aware grouping is a follow-up.
• Lower yield on PR-driven repos. Pallets/Werkzeug/Flask/httpx all merge-commit their PRs; commit_runtime correctly rejects those merge commits and yields ~0 directly mineable bugfix commits. Use pr_runtime for those repos. That's its turf.
• Go-heavy reference dataset. 39/52 envs are Go (urfave/gin/mux/cobra/testify/logrus) because Go projects in the bootstrap-cached set commit directly + squash-merge more than the Python projects do. This isn't a commit_runtime defect. It's the pipeline's natural fit shining through. Documented up-front for any eval that wants language balance.
• No commit_stream continuous variant. Same call as Arc 2: if continuous mining ever becomes a hard requirement, it's flags on commit_runtime (since=auto + state-file=…), not a separate pipeline.

How it was generatedCopy Anchor Link

# Per repo: bootstrap (cached from Arc 2) + mine/validate commits
repo2rlenv generate --repo <owner>/<repo> --pipeline commit_runtime \
  --pipeline-opt limit=120 --pipeline-opt clone_depth=400 \
  --llm anthropic/claude-sonnet-4-6 \
  --out ./stage/<repo>

# Oracle-gate (keep reward 1.0), drop broken oracles, push
harbor run -p ./pool -a oracle --env docker -n 5
repo2rlenv push ./dataset AdithyaSK/repo2rlenv-commit-runtime --inline-dockerfile

Scale driver: plans/arc3_scale.py (gitignored, launch-side).