terminal_reconstruct / terminalworld¶
TerminalWorld reconstructs executable tasks from real terminal recordings. Its tests are generated from the public goal and observed reference execution.
Pipeline, step by step¶
flowchart TD
S["info.json + recording.txt"] --> F["Deterministic transcript screening"]
F --> P1["P1 · Score value and runtime feasibility"]
P1 --> C["Add deterministic context score; apply threshold"]
C -->|"Unsupported or too little signal"| X["Record filter reason"]
C --> P2["P2 · Extract solution script"]
P2 --> P3["P3 · Refine script"]
P3 --> P4["P4 · Write outcome-oriented instruction"]
P4 --> P5["P5 · Reconstruct environment and reference"]
P5 --> E["Remote reference replay; record filesystem changes"]
E --> P6["P6 · Write tests from observed changes"]
P6 --> Q["Q1 · Optional draft consistency review"]
Q --> R["Fresh Harbor nop + oracle"]
Q -->|"Blocking issue"| FB
E -->|"Replay failure"| FB["Bounded materializer feedback"]
R -->|"Wrong reward or error"| FB
FB --> P5
R -->|"0 / 1"| H["Export reconstructed task"]P1, P2, … identify actual model calls. Unlabelled stages are code or remote execution.
Screen the input. Credential/PII patterns filter transcripts before any model call. P1 scores three axes from 0–3, reports tools, command count and supported. A bounded public-link probe supplies the fourth 0–3 context score.
Recover intent and actions. Require supported=true, at least three commands and total score ≥ min_score. Extract and refine the script, then write the instruction from metadata and the refined script.
Observe before verifying. The environment builder sees transcript evidence and produces real dependencies, starting fixtures and a reference. Only after a successful replay with persistent changes does the test author see bounded initial/final paths, contents and stdout.
Every prompt and its data¶
Six authoring calls on a successful first attempt: score, extract, refine, instruction, environment, tests. Filtered inputs use zero or one call. With review_drafts: true, the shared runner adds Q1 consistency review after P6. Blocking issues return to the materializer's existing bounded loop.
| Call | System prompt composition | User / input material | Output | Retry or branch |
|---|---|---|---|---|
| P1 · Score | score_long_prompt.md for >40 transcript lines; otherwise score_short_prompt.md | Screened metadata and transcript. | RecordingScore: three scores, reasoning, supported, command_count, required_tools | Default total threshold four out of twelve; not a final quality score. |
| P2 · Extract | extract_prompt.md | Full screened seed. | Script: solution_shell | One call after filtering. |
| P3 · Refine | refine_prompt.md | Extracted Script only. | Script: solution_shell | A separate call. |
| P4 · Instruction | instruction_prompt.md | Title, description and refined solution. | Instruction: instruction | No tests exist yet. |
| P5 · Environment / repair | environment_prompt.md + owned environment adaptation | RecordingDesign, including transcript evidence, and accumulated feedback. | EnvironmentBuild: setup, files, solution_shell, self_review | Reference is executed remotely before P6. |
| P6 · Tests | tests_prompt.md + observed-state adaptation | Instruction, reference script and execution_snapshot. | TestProgram: code | Repeated when a repaired environment reaches replay successfully. |
Read the complete terminalworld prompt reference for every retained template, appended instruction, substitution, example and output schema. The shared prompt guide explains how to inspect the fully resolved request from a real run.
Follow one task¶
Illustration: a recorded Git workflow is turned into a self-contained initial repository fixture. The reference performs the recorded transformation; tests inspect the observed resulting files or Git state. The original transcript is authoring evidence, not the learner prompt.
What repeats, what is checked¶
A materialization retry repeats environment construction and, if replay succeeds, test generation. Extraction and instruction are not regenerated by this loop. Opaque TUI, external accounts, GPU and unsupported service requirements are input filters. A provider timeout is an uncertain request, not a low-quality-task verdict.
An exported bundle is a generation result. Independent leakage review, shortcut probes and blind solver traces belong to the later quality campaign.
Implementation map¶
terminalworld/source.pyterminalworld/privacy.pyterminalworld/recipe.pyterminalworld/materialize.pyterminalworld/worker.pyterminal/runner.py
Run and supported profile¶
Provide a directory containing one or more native recording folders, each with info.json and recording.txt. Metadata includes title, description, id and url. To acquire public source recordings from numeric IDs:
python -m repo2rlenv.pipelines.recipes.terminalworld.source \
--ids-json workspace/recording-ids.json --out workspace/recordings
repo2rlenv generate --config examples/owned-terminalworld.yaml
The ID file is a JSON list such as ["100135"]. Acquisition fetches text and metadata only, checks robots.txt and records download failures. Existing inputs are reused. Export the generated task bundles rather than the raw recordings.
To discover fresh sources without using an upstream task dataset, index bounded public explore pages first:
python -m repo2rlenv.pipelines.recipes.terminalworld.discovery \
--feeds recent featured popular --pages-per-feed 5 --out workspace/recording-index
python -m repo2rlenv.pipelines.recipes.terminalworld.source \
--ids-json workspace/recording-index/ids.json --out workspace/recordings
Discovery uses the native public/recent/featured/popular feed URLs. It keeps only numeric recording IDs and page receipts, bounds pages and response sizes, checks robots.txt, waits between requests and stops a feed on empty or repeated pages. Completed pages are reused on restart. New inputs still pass the same privacy and feasibility filters; finding a recording does not accept a task. Use one acquisition process per recording directory to preserve its receipt.
You can also provide a JSON list of explicit public profile paths, such as ["/~example"], through --profiles-json workspace/profiles.json. Profile paths are restricted to Asciinema; the combined discovery run permits at most 200 pages. This is an owned source-curation extension to the native explore feeds. The scale campaign selected public profiles linked from earlier usable source recordings after the explore feeds began repeating. It therefore samples related workflows; it does not establish a random or representative terminal benchmark.
Acquisition writes acquisition.json with a running, interrupted or completed state. Completion is bound to the current retrieval.json hash. A controller can wait for a useful batch of new recordings, then consume a smaller final batch only after acquisition completes. A download failure remains in the retrieval receipt and is not a successful input.
The first runtime profile supports a single offline CPU Linux container. It installs real dependencies during build and can synthesize missing input files when the recorded workflow provides enough evidence, as permitted by the native builder. It excludes opaque TUI, GPU, privileged networking, multi-service and external-account workflows. The value score is an upstream input-selection stage, separate from the later task-quality audit.
An empty environment_files list is valid when the learner creates the requested deliverables from scratch. The owned Dockerfile and dependencies still define the environment. Requiring a placeholder fixture caused a recorded compilation task to exhaust a repair reservation despite a valid starting state; the schema now accepts that shape without changing reference or verifier requirements.
Options include the shared terminal generation bounds and min_score (default four out of twelve, the native bronze threshold). The snapshot records file changes and bounded content prefixes. The test author consumes that execution evidence before the fresh baseline/reference trials.
Measured results and limits¶
The dataset contains 100 tasks. The measured sample produced 80 new exports from 1,293 recorded candidate IDs, including recordings rejected during design screening. Generation cost averaged $1.26 per new task, including estimated compute and failed attempts. This is not the conversion rate among pre-approved designs. See economics for the counting rules and sample scope.
Three published tasks are labeled needs_repair: two verifiers do not execute required scripts, and one checks C source keywords and executable format without establishing the requested process behavior. The other 97 remain unverified for independent quality acceptance. One task used an assisted recovery of saved authoring output after an empty-fixture schema fix; its provenance records that intervention. Passing baseline/reference controls does not establish resistance to reward shortcuts or replace independent review and blind solver rollouts.
Credit: TerminalWorld, Apache-2.0, commit 784698ba93735470ce1664bff2ec44bcd7b28e15. See RFC 0019 and packaged recipes/terminalworld/provenance.md for the exact source map and adaptations.