Taste-Bench

Taste-Bench: the best AI agent picks the better path at long-horizon decision forks only 59.7% of the time in the authors' tests

Posted on arXiv on Oct 18, 2026 (source: arxiv.org).

Researchers from City University of Hong Kong and Microsoft introduced Taste-Bench in an arXiv paper: 502 questions that stop a real agent trajectory at a decision fork and ask a model which of two next directions the hidden rest of the run shows to be better. The forks are mined automatically, without human annotation, from GPT-5.4 and GPT-5.5 agent rollouts on SWE-bench Pro tasks and from research-task runs in METR's public MALT transcripts, and a question counts only if the model answers it correctly in both option orders, so random guessing scores 25%. In the authors' own tests of 14 models, GPT-5.6 Sol did best at 59.7%, just ahead of GPT-5.5 at 59.5%, and a larger reasoning budget did not improve accuracy. The authors report that distilling a teacher that has seen the outcome into a student model improved its decisions on unseen tasks and its end-to-end success on held-out SWE-bench Pro tasks.