Modern machine learning systems are increasingly developed through multi-stage training pipelines—pretraining, fine-tuning, and alignment—and deployed in settings where failures can be costly. Despite their impressive performance, these systems can exhibit brittle or unsafe behaviors, raising a central accountability question: when a model fails, which components of the learning pipeline or input features are responsible, and how can we reliably diagnose and address them? In this talk, I will present our recent work that develops statistical foundations for auditing and accountability in modern ML systems at two complementary levels. First, auditing training pipelines (stage-level accountability): we formalize the effect of removing or altering a training stage as a causal estimand using the potential outcomes framework. We define counterfactual stage effects and derive retraining-free, efficient estimators that approximate these effects while accounting for realistic optimization dynamics, including learning-rate schedules, momentum, and weight decay. The resulting stage-effect representations enable scalable audits of model behavior across diverse tasks and evaluation metrics. We further provide explicit worst-case bounds on approximation error, yielding formal guarantees on the fidelity of these estimates. Second, auditing model behavior locally (feature-level accountability): I will show that many widely used feature attribution methods can be interpreted as instances of local function approximation, with key differences driven by their neighborhood distributions and loss functions. Building on this perspective, I will describe Bayesian formulations that quantify uncertainty in feature attributions and derive principled sampling strategies for reliable estimation. Finally, I will discuss prescriptive feature-level interventions (counterfactual explanations) and how to ensure their robustness when the underlying model evolves. I will conclude by outlining future directions for strengthening the theoretical and practical foundations of auditing and improving modern machine learning systems.