GPT-2-small

Researchers train GPT-2-small models on 96 synthetic languages to test how language models learn action nominal constructions

Posted on arXiv on Oct 19, 2026 (source: arxiv.org).

Iskar Deng and Shane Steinert-Threlkeld built 96 synthetic target languages that vary clause and noun-phrase word order, alignment, complement systems and action nominal construction strategy, trained a GPT-2-small model from scratch on each, and tested them with minimal pairs covering 34 grammatical phenomena. The authors report typological parallels in basic clause configurations but divergences in complement systems and nominal strategies, with models also learning combinations that are rare or unattested in natural languages. They say this suggests natural language distributions reflect both general learning pressures and functional demands that model learning experiments do not fully capture. The paper has not yet been peer-reviewed.