BrokenML

BrokenML benchmark tests how much cluster access AI agents need to fix broken ML jobs

Alex Zhebrak (BrokenML) published this on Oct 12, 2026 (source: zhebrak.io).

BrokenML runs nine models against 46 broken distributed ML jobs, PyTorch over Gloo on a two-node Slurm cluster or a llama.cpp server in Docker Compose, with config and code only, read-only live logs, or root on the cluster, and a separate verifier re-runs each fix on a fresh cluster. More access improved every agent and made fixes up to 8x cheaper, and live logs nearly matched full cluster access for the strongest models. The author found GPT-6 Luna gave the best price-performance at about 5% of Sonnet 5.5's cost per run, and cautions that each task was run only once, on CPUs.