MetaOthello: A Controlled Study of Multiple World Models in Transformers

How do transformers represent the world when different latent processes can generate the same data? To answer this question, we construct a set of toy “worlds” based on the game Othello, each using different rules but operating on the same board and syntax. We train small GPT models on sequences generated from these mixtures. Finally, we show that the models learn shared world-state representations and localize a steerable mechanism by which they resolve ambiguity.