Total customer request outage to AWS Frankfurt

Incident Report for ESS (Public)

Resolved

Due to a misconfigured deployment job, rather than removing a percentage of the proxies from the load balancer, the job removed all proxies from the load balancer. All traffic in aws-eu-central-1 (AWS Frankfurt) was impacted from Tues May 21 2019 20:09 to 20:15 UTC. The deployment job was manually updated to decrease deployment time. We're taking immediate action to prevent such a thing from happening again by not changing the default for this deployment job in the future and will add a second layer of verification when it changes. We apologize for the mistake.

Posted May 21, 2019 - 20:43 UTC

This incident affected: GCP Frankfurt (europe-west3) (Elasticsearch connectivity: GCP europe-west3).