Contents
Why should an AI engineer care about karpathy/autoresearch?
The repo is a deliberately small harness where an LLM agent edits exactly one Python file and is graded on a single scalar, and it has already accumulated 97,819 stars and 13,566 forks on the GitHub API. That is not a benchmark suite, a framework, or a paper; it is a concrete pattern for closing the loop between a coding agent and a training run, with the evaluation code frozen and the surface the agent can touch intentionally tiny (README). For anyone designing an agent harness that touches real ML code rather than toy scripts, the design choices inside the loop are the actual artifact.
How This Review Was Researched
This review is based on the repository source, README, program.md, and the GitHub API — we did not run it hands-on. Every numeric claim about the repo is anchored to either the GitHub API snapshot, the README, or program.md. Where the sources disagree, we flag the disagreement rather than smooth it over. The parent training code lives at karpathy/nanochat, which the autoresearch project explicitly identifies as its upstream.
What is the three-file architecture?
The repository is split into three files with strictly separated roles, and the split is the central design decision (README). prepare.py is read-only and owns the constants, data preparation, the BPE tokenizer, the dataloader, and the evaluation; train.py is the only file the agent edits and contains the GPT model definition, the Muon + AdamW optimizers, and the training loop (program.md). The training code is itself a simplified single-GPU version of nanochat, documented in its README. program.md is the instruction document, written by the human, that the agent reads at the start of every iteration. Treating evaluation as a frozen oracle means the agent cannot tune the metric, and that is what makes the scalar comparable across runs.
What contract does the agent have to follow?
The agent’s instructions are a short ruleset spelled out in program.md, and the rules double as a research-method discipline (program.md). It may edit train.py only; it may not modify prepare.py, because evaluate_bpb is “the ground truth metric”; and it may not add dependencies beyond pyproject.toml. The metric it is graded on is val_bpb, validation bits per byte, which is lower-is-better and vocab-size-independent, so an agent that widens the embedding table to chase loss cannot game it. A change that buys a tiny gain at the cost of ugly complexity is rejected; a change that removes code with equal results is a keep; VRAM is a soft constraint rather than a hard wall. The README does not describe seed averaging, and program.md does not require it, so any single result is a single seed.
How does the NEVER STOP loop actually run?
The loop is intentionally trivial and intentionally unattended, and that triviality is the point (program.md). The agent edits train.py, commits, runs uv run train.py > run.log 2>&1, greps for ^val_bpb: to extract the metric, and on a crash tails the log and attempts a fix for a few attempts before giving up on that edit. A run that exceeds 10 minutes is killed and treated as a failure, so the agent cannot quietly burn GPU time on a runaway experiment. The fixed five-minute wall-clock budget, excluding startup and compilation, is the shared unit of cost that makes results comparable, and the README quotes a throughput of roughly 12 experiments per hour, on the order of 100 while you sleep (README). The headline behavioral rule, printed in program.md, is that the loop is not allowed to pause to ask the human once it has started; it runs until the human interrupts it.
How does the ratchet keep the agent honest?
The keep/reset protocol is a git-level invariant, and it is the mechanism that makes the run reproducible later (program.md). Each iteration lives on a branch named autoresearch/<tag>, and a commit is kept on the branch only when val_bpb improves over the previous best; otherwise the working tree is reset and the failed edit is discarded. Results are appended to results.tsv, a tab-separated log that is untracked by git, so the experiment history survives branch churn but never enters the diff history. That separation is deliberate: the branch records the trajectory of the best model, the TSV records the trajectory of every measurement, and the two never get conflated.
What platform constraints shape every experiment?
The harness is single-GPU by construction, and the README reports that it was tested on an H100 (README). There is no distributed training, no multi-node orchestration, and no gradient accumulation scheme designed to span devices, so every comparison the agent makes happens on the same machine. The README concedes that the fixed five-minute budget makes experiments comparable on the same platform, but the resulting numbers are explicitly NOT comparable across different compute platforms. Anyone reading the results.tsv needs to remember that an H100 number and a 4090 number, even on identical code, are not the same experiment.
Where are the known risks and edge cases?
The biggest risk is that val_bpb is a single seed and a single checkpoint, and the README does not describe seed averaging, so a lucky run can be kept and replayed. A second risk is license provenance: the README and the repo page describe the project as MIT, but the GitHub API license field is null at the time of writing, and the discrepancy is something to verify before adopting the code rather than something this review can resolve. A third risk is platform portability; the community has already produced forks for MacOS (miolini/autoresearch-macos, trevin-creator/autoresearch-mlx), Windows (jsegov/autoresearch-win-rtx), and AMD (andyluo7/autoresearch), which is a useful signal that the single-GPU assumption is brittle in practice (README). Finally, the agent can edit any line of train.py, including the loss function, so reviewers need to read the kept commits, not just the TSV.
How does autoresearch compare to adjacent patterns?
| Dimension | karpathy/autoresearch | karpathy/nanochat | Generic agent-with-eval loop | Hyperparameter-search baseline |
|---|---|---|---|---|
| Mutable surface | One file, train.py (program.md) | Full repo, human-driven | Whatever the harness permits | A search space defined in config |
| Evaluation trust | Frozen evaluate_bpb in read-only prepare.py (program.md) | Same eval, but editable | Agent often writes its own eval | Fixed eval, often a held-out split |
| Budget model | Fixed 5-minute wall clock excluding startup (README) | Unbounded, human-paced | Often per-iteration token or step budget | Often trial-count or wall-clock |
| Stopping criterion | Human interruption only (program.md) | Human terminates | Convergence, budget, or human | Trial budget, patience, or convergence |
| Results log | Append-only results.tsv, untracked (program.md) | Human-written notes | Often stdout or a custom DB | Optuna/MLflow/W&B logs |
| Reviewability | Git branch autoresearch/<tag> ratchets kept commits only | Single human PR | Mixed; commits vary | Search-space file plus run logs |
| License posture | MIT per README, API null (README, API) | Parent project, see nanochat | Depends on harness | Depends on framework |
| Platform | Single NVIDIA GPU, tested on H100 (README) | Single GPU, similar posture | Usually compute-agnostic | Usually compute-agnostic |
| Cross-platform comparability | Not comparable across platforms (README) | Not stated in README | Not stated in README | Often normalized by hardware |
The two rightmost columns are synthesized from typical harness practice, not from a source in this repo; treat them as a reference frame, not as facts about a specific system.
How does the example output in program.md relate to real runs?
The program.md file includes an example val_bpb value and example results.tsv rows, and the file marks them as illustrative rather than measured (program.md). Do not cite the example number as a benchmark, and do not cite the example rows as a log of real experiments; the only measured data point the sources support is the structural one, that a kept commit is one where the agent’s val_bpb beat the previous best on the same machine.
FAQ
Is autoresearch a benchmark or a harness?
It is a harness, with the agent plus the frozen evaluate_bpb evaluation acting as the benchmark surface. The prepare.py evaluation is read-only and treated as the ground truth metric, so the benchmark is the function the agent cannot touch rather than a fixed dataset (program.md). Two agents run on the same machine are comparable; two agents on different GPUs are not (README).
Can the agent just overfit val_bpb by editing the loss?
The metric is val_bpb computed inside the frozen evaluate_bpb function in prepare.py, which the agent is not allowed to modify. The agent can edit the model and the optimizer in train.py, and VRAM is treated as a soft constraint rather than a hard wall, so the lever is architecture and training dynamics, not the metric itself (program.md).
Why a five-minute budget rather than a step count?
A wall-clock budget makes experiments comparable on the same platform, since every trial gets the same time on the same machine, but the README is explicit that the same code on different GPUs is not comparable (README). Combined with the keep/reset ratchet on the autoresearch/<tag> branch, the budget becomes the unit of cost the agent optimizes against (program.md).
The Bottom Line
autoresearch is worth studying as a concrete pattern for closing a coding agent around a real ML training loop, with a frozen evaluator and a single editable file. The interesting design choices are the read-only split between prepare.py and train.py, the git ratchet on autoresearch/<tag>, and the NEVER STOP rule that forces unattended iteration. Treat it as a harness template to argue with, not a benchmark to quote.
