Repo2RLEnv

Pipeline RFCs

Edit on GitHub

Design docs for new synthesis pipelines. One RFC per pipeline. Written before the code lands; kept in the repo after the pipeline ships as a permanent record of why the pipeline exists in the shape it does.

Why RFCs

A pipeline is 100–300 LOC of code, but the decisions behind it (repo shape it fits, verification approach, contamination story, LLM use, yield expectations) are much harder to reconstruct from a merged diff months later. Every new pipeline in this repo has had non-obvious design choices (issue-fetch fallback in commit_runtime, PoC-agent for cve_patches, anti-contamination compose overlay, graded vs. binary reward). Those choices deserve a durable home separate from the implementation.

An RFC also front-loads the audit: writing the "how does contamination get in?" section forces you to think about it before you've shipped 100 published envs.

When to write one

  • Always for a new pipeline (new entry in PipelineName).
  • Optional for meaningful reshapes of an existing pipeline (e.g. adding LLM synthesis to commit_runtime in v0.8.4, retrospectively RFC-worthy).
  • Skip for polish / bug-fix work that fits in a PR description.

Retrospective RFCs. Pipelines that shipped before this process existed (RFCs 0001–0006) have RFCs written after their initial merge, as archival records. They're status: implemented from day one and their Implementation sections link back to the initial PR, source file, doc page, and reference dataset. Retro RFCs are lighter on the "alternatives considered" front (memory decays) and heavier on cross-referencing the current authoritative doc (docs/pipelines/<name>.md) as the source of truth. Do not write retro RFCs for pipelines that have been withdrawn (mutation_bugs, refactor_synthesis). Git history is enough.

Process

  1. Pick a candidate from plans/candidate_pipelines.md, or propose a new one.
  2. Copy TEMPLATE.md to docs/rfcs/NNNN-<name>.md (next unused 4-digit number, kebab-case name).
  3. Fill in every section: write "n/a" if a section genuinely doesn't apply, don't just delete it. If you don't know an answer yet, mark it TBD and open it in "Open questions."
  4. PR the RFC alone first: reviews want to look at the design without the implementation blur. RFCs at this stage should carry the label rfc:draft (add it via gh pr edit --add-label rfc:draft).
  5. Iterate on the RFC based on review. Update the status header as it moves through the lifecycle (see below).
  6. Once accepted, implement the pipeline in a follow-up PR that references the RFC number and follows docs/contributing/ADDING_A_PIPELINE.md.
  7. After merge, update the RFC's status to implemented, add the merge commit + PR link, and link the reference dataset from the "Rollout" section.

Lifecycle

RFCs have a Status: header line that moves through these states:

  • draft: being written; not yet ready for design review.
  • review: ready for feedback; PR open.
  • accepted: design signed off; implementation can begin.
  • implemented: code has landed on main. RFC is now archival.
  • withdrawn: decided against. Kept for posterity; explain why in a final "Withdrawal" section.
  • superseded: replaced by a later RFC (link both directions).

Numbering

Sequential. 0001-<name>.md, 0002-<name>.md, …. Never reuse a number. If an RFC is withdrawn, the number is retired with it.

Index

#PipelineStatusRFCReference dataset
0001pr_diffimplemented (stable)0001-pr-diff.mdrepo2rlenv-pr-diff (181)
0002pr_runtimeimplemented (stable)0002-pr-runtime.mdrepo2rlenv-pr-runtime (100)
0003commit_runtimeimplemented (stable)0003-commit-runtime.md…commit-runtime (100)
0004code_instructimplemented (experimental)0004-code-instruct.md…code-instruct (100)
0005equivalence_testsimplemented (experimental)0005-equivalence-tests.md…equivalence-tests (100)
0006cve_patchesimplemented (experimental)0006-cve-patches.md…cve-patches (19)
0007pr_to_envdraft0007-pr-to-env.mdNone
0008env_setupdraft0008-env-setup.mdNone
0009test_synthesisdraft0009-test-synthesis.mdNone
0010issue_runtimedraft0010-issue-runtime.mdNone
0011owned recipe contractimplementation in PR #1090011-owned-recipes.mdRelease inventory
0012repo_mutate / swe_smithexperimental; PR #1090012-swe-smith-recipe.md100 tasks
0013terminal_synth / seta_seed2synthexperimental; PR #1090013-seta-seed2synth-recipe.md100 tasks
0014task_evolve / seta_evolexperimental; PR #1090014-seta-evol-recipe.md100 tasks
0015pr_to_env / swe_genexperimental; PR #1090015-swe-gen-recipe.md100 tasks
0016repo_reconstruct / swe_flowexperimental; PR #1090016-swe-flow-recipe.md100 tasks
0017equivalence_tests / r2eexperimental; PR #1090017-r2e-recipe.md100 tasks
0018terminal_synth / tmaxexperimental; PR #1090018-tmax-recipe.md55 tasks
0019terminal_reconstruct / terminalworldexperimental; PR #1090019-terminalworld-recipe.md100 tasks
0020terminal_synth / endless_terminalsexperimental; PR #1090020-endless-terminals-recipe.md100 tasks
0021env_repair / cli_gymexperimental; PR #1090021-cli-gym-recipe.md25 tasks
0022terminal_synth / dataarcexperimental; PR #1090022-dataarc-terminal-recipe.md100 tasks
0023pr_runtime / swe_nextexperimental; PR #1090023-swe-next-recipe.md100 tasks
0024commit_runtime / r2e_gymexperimental; PR #1090024-r2e-gym-recipe.md100 tasks
0025cve_patches / sec_benchdeferred0025-sec-bench-recipe.mdExcluded
0026reasoning_synth / scalerexperimental; PR #1090026-scaler-recipe.md100 tasks
0027Harbor review and repairimplementation in PR #1090027-harbor-quality-loop.mdn/a
0028Tasksmith PR pilotcompleted pilot; PR #1090028-tasksmith-pr-pilot.mdHF ML Tasksmith: 50 tasks
0029HF bootstrap and Tasksmith scalerecorded campaign; PR #1090029-tasksmith-hf-scale.mdHF ML Tasksmith: 50 tasks
0030Campaign expansion and Harbor releasesimplementation in PR #1090030-campaign-expansion-and-release.mdRelease inventory

| 0031 | repo_reconstruct / codemidas | implemented in 0.9.2; 100 local tasks | 0031-codemidas-recipe.md | Local campaign; dataset unpublished |

On this page