RFC 0006: cve_patches¶

Status: implemented (experimental) Author: @adithya-s-k Created: 2026-05-15 (retrospective — pipeline shipped in v0.7.0; hardened + PoC agent added in v0.8.5; RFC written 2026-07-15 as archival record)

Summary¶

OSV-driven security-task pipeline: for a target repo, look up its CVE fixes in the OSV database, map each vulnerability to the fixing commit, and emit a Harbor task whose oracle is the upstream security patch. Reuses pr_runtime's F2P/P2P verifier. When the CVE ships no regression test (common), an agentic PoC-test synthesis step (v0.8.5) lets an LLM with shell access inside the vulnerable sandbox write a regression test that catches the vulnerability.

Motivation¶

Two existing corpora — PatchSeeker (5K CVEs), PATCHEVAL (1K) — are datasets of CVE-fix pairs. What didn't exist was a reusable pipeline that takes any repo + OSV and turns the pair into Repo2RLEnv-shaped tasks: paper-only artifacts, no library. cve_patches closes that gap.

Beyond the practical "pipeline over dataset" reason, CVEs are qualitatively different from ordinary bugs — they're security-relevant, human-triaged, and have documented reproduction paths. An RL agent that can fix a CVE (not just any test failure) is a more compelling story than one that fixes an off-by-one.

The counter-argument at v0.7 time was scope: "we don't need a CVE-specific pipeline; pr_runtime would emit these too if the fix PR was in the mined set." Our answer: OSV gives us an authoritative source of which commits are the security fixes, and the CVE→fix mapping is data that pr_runtime doesn't have. Plus the follow-up shape (LLM-synthesized PoC tests when the fix ships no test) is CVE-specific and doesn't belong in pr_runtime.

Design¶

Input¶

• Source — GitHub (needs the GitHub commit API + OSV; local + GitLab don't apply).
• Trigger — repo2rlenv generate --pipeline cve_patches --repo <owner>/<name> --pipeline-opt limit=20 --pipeline-opt osv_ecosystem=PyPI --llm anthropic/claude-sonnet-4-6 ...
• Options model — CvePatchesOptions: limit, since, min_severity, osv_ecosystem (auto-guessed per owner for known ecosystems), require_fail_to_pass, synthesize_poc_test (default on in v0.8.5), poc_agent_max_iterations, LLM knobs.

Algorithm¶

1. Query OSV for vulnerabilities affecting the repo's package.
2. For each vulnerability: resolve the fixing commit URL (OSV's references[].url).
3. git show <fix_commit> → split into (source_patch, test_patch).
4. Branch A — CVE ships a test: proceed like pr_runtime, validate F2P/P2P in the sandbox.
5. Branch B — CVE has no test (majority case, ~60–70% of OSV entries): run the PoC agent (_poc_agent.py, v0.8.5). An LLM with shell access inside the pre-fix vulnerable container writes a regression test that: (a) fails on the pre-fix state, (b) passes on the post-fix state, © doesn't reference the fix's SHA or CVE ID.
6. Validate the resulting (patch, test_patch) tuple in the sandbox.
7. Emit Harbor task with the shared shape.

Output¶

Same shape as pr_runtime. [metadata.repo2env] adds a cve_patches subtable: cve_id, ghsa_id, severity, fix_commit, osv_ecosystem, poc_agent_used (bool), poc_agent_iterations (if used).

Verification¶

• Reward kinds — test_execution + diff_similarity. Uses _pr_runtime_verifier.py verbatim (upgraded to graded F2P/P2P in v0.8.5; was binary whole-suite before that pass, which spuriously scored the gold patch 0.0 on unrelated suite failures).
• Oracle invariant — gold security patch scores reward=1.0, resolved=True on every emitted task.
• Non-tamper — same as pr_runtime.

Anti-contamination¶

CVE tasks have the highest contamination risk in the set. The fix has a CVE ID, a GHSA ID, a public advisory, a PyPI release note, a patched wheel published to PyPI, and often a PoC in the advisory. Multiple leak paths.

• Git-history scrub + egress guard — mandatory. PyPI is blackholed so pip download <pkg>==<fixed_version> fails (the original observed leak that drove _env_guard.py's creation).
• Leak-stripped instruction — v0.8.5 strips CVE / GHSA IDs, PR/commit URLs, "fixed in vX.Y" phrases, Closes #N trailers. The prompt only carries the symptom.
• PoC agent's no_fix_reference guard — the agent that synthesizes a regression test is prompted + regex-checked to not include the fix commit SHA or CVE ID in the emitted test.

The principle: the environment enforces, the prompt never asks. For CVE tasks, prompt-only defenses are insufficient — the network isolation is doing the real work.

LLM use¶

• at bootstrap (cached) — one call chain per repo, standard.
• at synthesis (branch B, per CVE without a shipped test) — the PoC agent iterates in a shell inside the vulnerable sandbox. Multi-turn, budget-bounded (poc_agent_max_iterations, default 8). Cost: ~$0.10–0.50 per PoC synthesis. Skipped for branch A CVEs.

Yield & repo suitability¶

• 5–25% yield — the lowest in the set. Dominant factors:
• Does the CVE fix actually have a verifiable test (shipped or synthesizable)?
• Does the repo's suite collect in a slim container?
• Real numbers from repo scouting (plans/cve_repo_scout.py): urllib3 → 0/16 CVEs (tests need network, don't collect in slim); sqlparse → 2/4 (suite runs clean); werkzeug, requests, httpx, flask → mid-yield.
• What works: library-shaped Python repos with pytest-clean suites and CVE-rich histories.
• Budget many repos: at ~15% yield, 100 CVE tasks ≈ 15–20 CVE-rich, test-clean repos.

Dependencies¶

• pr_runtime — full validation harness + verifier + Docker file emitter.
• bootstrap/ — cache.
• _env_guard.py — critical here.
• OSV API — public, no auth needed.
• GitHub commit API — to resolve fixing commits.
• _poc_agent.py — new in v0.8.5, wraps LiteLLM + a bash-tool loop inside a DockerSandbox.

Alternatives considered¶

• Static PoC extraction from CVE advisories — rejected. Advisory PoCs are inconsistent in format and often not runnable (curl-y HTTP dumps that don't map to a test).
• Third-party CVE-fix datasets (PatchSeeker, PATCHEVAL) as input — the pipeline can consume them (via a hand-curated URL list, once pr_to_env lands as RFC 0007) but the direct-mine path is OSV.
• Skip CVEs without shipped tests entirely — considered pre-v0.8.5. Would have cut yield to ~5%. PoC-agent synthesis expanded the addressable set 3–4×.

Rollout plan¶

Historic. v0.7.0 shipped experimental (binary whole-suite reward, no PoC synthesis). v0.8.5 hardening pass in PR #69 added: - Graded F2P/P2P verifier (was binary; fixed the "gold patch scores 0.0 on unrelated suite failures" bug) - Leak-stripped instruction - Anti-contamination bake (git-history scrub + egress guard applied via emitter) - Agentic PoC-test synthesis - 19-env reference dataset published

Stays experimental until a graded-reward audit shows the reward is meaningful across a broader CVE pool.

Open questions¶

• NVD-direct fallback for CVEs OSV hasn't curated. Rely on OSV's pre-resolved fix URLs today; a PatchSeeker-style LLM+embedding fallback that resolves NVD CVEs to fix commits is on the roadmap.
• PoC synthesis distribution shape — should we distribute PoC tests as part of the published dataset, or does that carry a "we're publishing exploit code" concern? Currently included. Worth revisiting with the community.

References¶

• OSV: osv.dev.
• PatchSeeker: hungkien05/PatchSeeker.
• CVE-Bench (NAACL '25): uiuc-kang-lab/cve-bench.
• Anti-contamination writeup: plans/reward_hacking_writeups.md.

Implementation¶