RealCLI

RealCLI benchmark tests CLI agents against a scripted user who changes instructions mid-task, and the best setup averages 49.2%

Posted on arXiv on Oct 19, 2026 (source: arxiv.org).

RealCLI is a paired benchmark for command-line agents: RealCLI-Exec gives the full specification upfront, while RealCLI-Collab adds a scripted user who delivers information step by step, corrects the agent, revises constraints and makes authorization decisions. Collaboration is scored on five dimensions broken into 22 sub-dimensions and 51 interaction scenario templates. The paper reports that the best configuration averages only 49.2% and peaks at 37.6% on the joint collaboration score, and the authors say frontier agents consistently struggle with steerability, trust calibration and safety boundaries. The paper has not yet been peer-reviewed.