Explainability is often a legal requirement: US adverse-action rules (ECOA / Regulation B, per CFPB Circular 2022-03) require specific principal reasons even when a complex algorithm is involved, and GDPR Article 22 gives a right to human review of solely automated decisions.
A useful distinction is global (how the model behaves overall) versus local (why this one decision was made) — adverse-action law is about local explanations.