Scaling Autoresearch with SkyPilot: 9× Faster AI Research

Answer: By coupling Andrej Karpathy’s Autoresearch agent with the open‑source SkyPilot orchestration layer, researchers can run ~910 experiments in under 8 hours on a 16‑GPU Kubernetes cluster—delivering a 9× speed‑up and a 2.9 % validation loss improvement for a fraction of the cost of a single‑GPU baseline.

Why Scaling Autoresearch Matters for Modern AI Labs

AI research teams constantly wrestle with a paradox: the most promising ideas require massive compute, yet budget constraints force them to run experiments one‑by‑one on a single GPU. The recent SkyPilot‑powered scaling experiment shatters this bottleneck, turning a night‑long, 70‑experiment marathon into an 8‑hour sprint that explores every corner of the hyper‑parameter space.

In this article we unpack the technical blueprint, walk through the five research phases, and give you a reproducible, step‑by‑step tutorial so you can replicate the results on your own cloud or on‑prem GPU farm.

Project Overview: Autoresearch Meets SkyPilot

Autoresearch, introduced by Andrej Karpathy, is a self‑modifying coding agent that iteratively edits train.py, runs a 5‑minute training job, and keeps changes that lower the validation bits‑per‑byte (val_bpb) metric. In its vanilla form the agent is limited to a single GPU, yielding roughly 10–12 experiments per hour.

SkyPilot is an open‑source cloud‑agnostic orchestration tool that abstracts away the provisioning of GPU clusters across Kubernetes, AWS, GCP, Azure, and more. By embedding a SkyPilot “skill” into the agent’s instruction set, the agent learns to launch, monitor, and tear down GPU jobs without human intervention.

The combined system was tested on a Kubernetes cluster backed by CoreWeave, using a mix of NVIDIA H100 (80 GB) and H200 (141 GB) accelerators. Over 8 hours the agent submitted ~910 experiments across 16 GPUs, discovered a superior model width, and reduced val_bpb from 1.003 to 0.974.

Phase‑by‑Phase Breakdown & GPU Config

Phase 1 – Hyperparameter Sweeps (≈200 experiments)

• Batch size, Adam betas, weight decay, window patterns, learning‑rate schedules.
• Parallelism allowed 10‑13 experiments per 5‑minute wave, mapping the space in ~30 minutes instead of 5 hours.
• Key finding: halving batch size to 2ⁱ⁸ increased optimizer steps, improving val_bpb to 0.981.

Phase 2 – Architecture Discovery (≈200‑420 experiments)

The agent tested six aspect‑ratio (AR) values (48‑112) simultaneously. The parallel grid revealed that widening the model to AR = 96 (model_dim = 768) outperformed every hyper‑parameter tweak from Phase 1.

Phase 3 – Fine‑Tuning the Wider Model (≈420‑560 experiments)

With the optimal width locked, the agent explored warm‑down schedules, matrix learning rates, and Newton‑Schulz steps for the Muon optimizer. The best configuration achieved val_bpb = 0.975 on H200 hardware.

Phase 4 – Optimizer Tuning (≈560‑700 experiments)

A systematic sweep of muon_beta2 values (0.95‑0.99) across 10 clusters in a single wave identified beta2 = 0.98 as the most effective, shaving another 0.001 off the loss.

Phase 5 – Diminishing Returns (≈700‑910 experiments)

The agent performed combinatorial sweeps of final LR fraction, warm‑down ratio, and scalar LR. Improvements fell below 0.0001 per experiment, indicating that the low‑hanging fruit had been exhausted.

GPU Allocation & Heterogeneous Strategy

SkyPilot automatically allocated 13 × H100 and 3 × H200 GPUs. The agent observed that H200s completed ~9 % more training steps in the same 5‑minute window, prompting a two‑tier strategy:

1. Screen dozens of hypotheses on cheap H100s.
2. Promote the top‑performing candidates to H200s for final validation.

This emergent behavior mirrors what human researchers would design, but it arose purely from the agent’s data‑driven observations.

Performance Gains & Cost Breakdown

The cost per 0.01 improvement in val_bpb dropped from >$1,000 in the single‑GPU scenario to under $30 with parallelism—a compelling ROI for research budgets.

For organizations that already use the Enterprise AI platform by UBOS, the SkyPilot workflow can be integrated as a native Workflow automation studio job, enabling cost‑controlled scaling across on‑prem and cloud resources.

How to Replicate the Scaling Experiment

Follow these commands on a machine with git, python, and kubectl installed.

1️⃣ Clone the repositories

git clone https://github.com/karpathy/autoresearch.git
git clone https://github.com/skypilot-org/skypilot.git
cd autoresearch

2️⃣ Copy the SkyPilot experiment definition

cp ../skypilot/examples/autoresearch/experiment.yaml .
cp ../skypilot/examples/autoresearch/instructions.md .

3️⃣ Install dependencies locally (one‑time)

pip install uv
uv sync
uv run prepare.py

4️⃣ Install the SkyPilot skill for your coding agent

The skill teaches the agent how to invoke sky launch and sky exec. Detailed docs are available on the UBOS partner program page.

5️⃣ Point your agent at instructions.md

Whether you use Claude Code, OpenAI’s OpenAI ChatGPT integration, or a custom LLM, the agent will read the file, fetch the SkyPilot skill, and start launching GPU jobs automatically.

6️⃣ Monitor progress with SkyPilot logs

sky logs gpu-01 -f   # tail logs of a specific cluster
sky status            # view all active clusters

7️⃣ Collect final results

After the run completes, the agent writes a summary CSV to the shared workdir. You can feed this into the UBOS templates for quick start to generate a visual report or feed it into the AI SEO Analyzer for further insights.

Tip: Adjust the resources.accelerators field in experiment.yaml to match your own cloud provider’s pricing. SkyPilot will automatically select the cheapest available GPU type, preserving the cost‑effective nature of the workflow.