Social conventions are the foundation of social coordination, shaping how individuals come together to form a society. In this talk, I will present theoretical and experimental findings that demonstrate the spontaneous emergence of social norms in human groups, as well as the existence of tipping points in social convention. I will then focus on the case of populations of large language models (LLMs). As AI agents increasingly communicate using natural language, understanding how they develop conventions is crucial for interpreting and managing their collective behaviour. I will show that LLM populations can establish social conventions and highlight how collective biases can emerge even when individual agents appear unbiased. I will conclude by stressing how the ability of AI agents to develop norms without explicit programming has significant implications for designing AI systems that align with human values and societal goals.