We will add additional monitoring and alerting to ensure that we would catch similar service discovery related problems earlier.We are working with Google Cloud to better understand how issues with Traffic Director resulted in a large outage affecting Spotify’s users.Timelineġ8:12 UTC / 13:12 ET – Reports of users being logged out of the different client apps start to surface.ġ8:39 UTC / 13:39 ET – Remediations were being put in place to restore the affected systemsĢ0:35 UTC / 15:35 ET – Incident fully mitigated at Spotify Where do we go from here? This in coordination with a bug in a client (gRPC) library caused the Spotify outage that affected many of our users: if you were logged out of a Spotify app, you were unable to log back in.Īs soon as the problem was discovered, we rolled out configuration changes to revert our affected systems back to use our DNS-based service discovery and saw it recover gradually. On March 8, Google Cloud Traffic Director experienced an outage. Most of our services are using a DNS based service discovery system however, some of our services use an xDS based traffic control plane and discovery system called Traffic Director. For microservices to be able to find each other, we utilize multiple service discovery technologies. The Spotify backend consists of multiple microservices that communicate with each other.

This outage affected our users and we apologize for the inconvenience it may have caused. We were made aware of issues with login at 18:12 UTC / 13:12 ET and started implementing fixes to critical systems at 18:39 UTC / 13:39 ET. On March 8, we experienced a global outage triggered by issues in a cloud-hosted service discovery system used at Spotify.

Spotify down march 8 2022