Graph Neural Networks (GNNs) have become a cornerstone for the application of deep learning to relational data on complex networks. However, we increasingly have access to time-resolved data that not only capture which nodes are connected to each other, but also when and in which temporal order those connections occur. A number of works have shown how the timing and ordering of links shapes the causal topology of networked systems, i.e. which nodes can possibly influence each other over time. Those works have shed light on the question how the time dimension of dynamic graphs influences node centralities, community structures, or the evolution of dynamical processes. However, we lack methods to incorporate those insights into state-of-the-art deep learning models. Addressing this gap, we introduce a time-aware graph neural network architecture for temporal graphs. We propose a novel notion of temporal graph isomorphism and develop a temporal generalization of the Weisfeiler-Leman algorithm to heuristically distinguish non-isomorphic temporal graphs. Building on this foundation, we derive a novel message passing scheme for temporal graph neural networks that operates on the event graph representation of temporal graphs. Our approach accounts for temporal-topological patterns that unfold via causal walks, i.e. temporally ordered sequences of connections by which nodes can causally influence each other over time.