RFC 0006: cve_patches

Status: implemented (experimental) Author: @adithya-s-k Created: 2026-05-15 (retrospective: pipeline shipped in v0.7.0; hardened + PoC agent added in v0.8.5; RFC written 2026-07-15 as archival record)

SummaryCopy Anchor Link

OSV-driven security-task pipeline: for a target repo, look up its CVE fixes in the OSV database, map each vulnerability to the fixing commit, and emit a Harbor task whose oracle is the upstream security patch. Reuses pr_runtime's F2P/P2P verifier. When the CVE ships no regression test (common), an agentic PoC-test synthesis step (v0.8.5) lets an LLM with shell access inside the vulnerable sandbox write a regression test that catches the vulnerability.

MotivationCopy Anchor Link

Two existing corpora, PatchSeeker (5K CVEs) and PATCHEVAL (1K), are datasets of CVE-fix pairs. What didn't exist was a reusable pipeline that takes any repo + OSV and turns the pair into Repo2RLEnv-shaped tasks: paper-only artifacts, no library. cve_patches closes that gap.

Beyond the practical "pipeline over dataset" reason, CVEs are qualitatively different from ordinary bugs: they're security-relevant, human-triaged, and have documented reproduction paths. An RL agent that can fix a CVE (not just any test failure) is a more compelling story than one that fixes an off-by-one.

The counter-argument at v0.7 time was scope: "we don't need a CVE-specific pipeline; pr_runtime would emit these too if the fix PR was in the mined set." Our answer: OSV gives us an authoritative source of which commits are the security fixes, and the CVE→fix mapping is data that pr_runtime doesn't have. Plus the follow-up shape (LLM-synthesized PoC tests when the fix ships no test) is CVE-specific and doesn't belong in pr_runtime.

DesignCopy Anchor Link

InputCopy Anchor Link

• Source: GitHub (needs the GitHub commit API + OSV; local + GitLab don't apply).
• Trigger: repo2rlenv generate --pipeline cve_patches --repo <owner>/<name> --pipeline-opt limit=20 --pipeline-opt osv_ecosystem=PyPI --llm anthropic/claude-sonnet-4-6 ...
• Options model: CvePatchesOptions: limit, since, min_severity, osv_ecosystem (auto-guessed per owner for known ecosystems), require_fail_to_pass, synthesize_poc_test (default on in v0.8.5), poc_agent_max_iterations, LLM knobs.

AlgorithmCopy Anchor Link

1. Query OSV for vulnerabilities affecting the repo's package.
2. For each vulnerability: resolve the fixing commit URL (OSV's references[].url).
3. git show <fix_commit> → split into (source_patch, test_patch).
4. Branch A: CVE ships a test. Proceed like pr_runtime, validate F2P/P2P in the sandbox.
5. Branch B: CVE has no test (majority case, ~60–70% of OSV entries). Run the PoC agent (_poc_agent.py, v0.8.5). An LLM with shell access inside the pre-fix vulnerable container writes a regression test that: (a) fails on the pre-fix state, (b) passes on the post-fix state, (c) doesn't reference the fix's SHA or CVE ID.
6. Validate the resulting (patch, test_patch) tuple in the sandbox.
7. Emit Harbor task with the shared shape.

OutputCopy Anchor Link

Same shape as pr_runtime. [metadata.repo2env] adds a cve_patches subtable: cve_id, ghsa_id, severity, fix_commit, osv_ecosystem, poc_agent_used (bool), poc_agent_iterations (if used).

VerificationCopy Anchor Link

• Reward kinds: test_execution + diff_similarity. Uses _pr_runtime_verifier.py verbatim (upgraded to graded F2P/P2P in v0.8.5; was binary whole-suite before that pass, which spuriously scored the gold patch 0.0 on unrelated suite failures).
• Oracle invariant: gold security patch scores reward=1.0, resolved=True on every emitted task.
• Non-tamper: same as pr_runtime.

Anti-contaminationCopy Anchor Link

CVE tasks have the highest contamination risk in the set. The fix has a CVE ID, a GHSA ID, a public advisory, a PyPI release note, a patched wheel published to PyPI, and often a PoC in the advisory. Multiple leak paths.

• Git-history scrub + egress guard: mandatory. PyPI is blackholed so pip download <pkg>==<fixed_version> fails (the original observed leak that drove _env_guard.py's creation).
• Leak-stripped instruction: v0.8.5 strips CVE / GHSA IDs, PR/commit URLs, "fixed in vX.Y" phrases, Closes #N trailers. The prompt only carries the symptom.
• PoC agent's no_fix_reference guard: the agent that synthesizes a regression test is prompted + regex-checked to not include the fix commit SHA or CVE ID in the emitted test.

The principle: the environment enforces, the prompt never asks. For CVE tasks, prompt-only defenses are insufficient. The network isolation is doing the real work.

LLM useCopy Anchor Link

• at bootstrap (cached): one call chain per repo, standard.
• at synthesis (branch B, per CVE without a shipped test): the PoC agent iterates in a shell inside the vulnerable sandbox. Multi-turn, budget-bounded (poc_agent_max_iterations, default 8). Cost: ~$0.10–0.50 per PoC synthesis. Skipped for branch A CVEs.

Yield & repo suitabilityCopy Anchor Link

• 5–25% yield: the lowest in the set. Dominant factors:
  • Does the CVE fix actually have a verifiable test (shipped or synthesizable)?
  • Does the repo's suite collect in a slim container?
• Real numbers from repo scouting (plans/cve_repo_scout.py): urllib3 → 0/16 CVEs (tests need network, don't collect in slim); sqlparse → 2/4 (suite runs clean); werkzeug, requests, httpx, flask → mid-yield.
• What works: library-shaped Python repos with pytest-clean suites and CVE-rich histories.
• Budget many repos: at ~15% yield, 100 CVE tasks ≈ 15–20 CVE-rich, test-clean repos.

DependenciesCopy Anchor Link

• pr_runtime: full validation harness + verifier + Docker file emitter.
• bootstrap/: cache.
• _env_guard.py: critical here.
• OSV API: public, no auth needed.
• GitHub commit API: to resolve fixing commits.
• _poc_agent.py: new in v0.8.5, wraps LiteLLM + a bash-tool loop inside a DockerSandbox.

Alternatives consideredCopy Anchor Link

• Static PoC extraction from CVE advisories: rejected. Advisory PoCs are inconsistent in format and often not runnable (curl-y HTTP dumps that don't map to a test).
• Third-party CVE-fix datasets (PatchSeeker, PATCHEVAL) as input: the pipeline can consume them (via a hand-curated URL list, once pr_to_env lands as RFC 0007) but the direct-mine path is OSV.
• Skip CVEs without shipped tests entirely: considered pre-v0.8.5. Would have cut yield to ~5%. PoC-agent synthesis expanded the addressable set 3–4×.

Rollout planCopy Anchor Link

Historic. v0.7.0 shipped experimental (binary whole-suite reward, no PoC synthesis). v0.8.5 hardening pass in PR #69 added:

• Graded F2P/P2P verifier (was binary; fixed the "gold patch scores 0.0 on unrelated suite failures" bug)
• Leak-stripped instruction
• Anti-contamination bake (git-history scrub + egress guard applied via emitter)
• Agentic PoC-test synthesis
• 19-env reference dataset published

Stays experimental until a graded-reward audit shows the reward is meaningful across a broader CVE pool.

Open questionsCopy Anchor Link

• NVD-direct fallback for CVEs OSV hasn't curated. Rely on OSV's pre-resolved fix URLs today; a PatchSeeker-style LLM+embedding fallback that resolves NVD CVEs to fix commits is on the roadmap.
• PoC synthesis distribution shape: should we distribute PoC tests as part of the published dataset, or does that carry a "we're publishing exploit code" concern? Currently included. Worth revisiting with the community.

ReferencesCopy Anchor Link

• OSV: osv.dev.
• PatchSeeker: hungkien05/PatchSeeker.
• CVE-Bench (NAACL '25): uiuc-kang-lab/cve-bench.
• Anti-contamination writeup: plans/reward_hacking_writeups.md.

ImplementationCopy Anchor Link