In this talk, I will focus on the problem of constructing useful graph representations from noisy, multi-modal and temporal measurements. I will take the perspective that the downstream task should inform and guide the learning process and present two feedback-driven learning algorithms. I will discuss results and insights on various downstream tasks on both synthetic and real datasets. I will show that local topological metrics are often sufficient to guide the learning process towards better graph representations and that the quality of the graph does not transfer from one downstream task to the other. This is joint work with Jeremy Kun and Benjamin Fish.