Azure - potential issues with plan changes and creation of new deployments
Incident Report for ESS (Public)
Resolved
This incident has been resolved.
Posted Dec 12, 2019 - 07:07 UTC
Update
We've rolled out a fix for the original issue and deployment hosts continue to be stable. Customers should not experience issues creating new deployments or applying changes to the existing ones. We will keep monitoring the situation over the next 6 hours, before closing this incident to make sure the issue has been permanently resolved.
Posted Dec 12, 2019 - 00:37 UTC
Monitoring
Rollback has been completed. Allocator components seem to be stable now and customers should not experience issues creating new deployments or applying changes to the existing ones. We will keep monitoring the situation in the coming hours to make sure the issue has been permanently resolved.
Posted Dec 11, 2019 - 15:31 UTC
Update
Rollback is currently in progress. We expect it to be completed in 30-60 minutes.
Posted Dec 11, 2019 - 14:48 UTC
Identified
We suspect the issue has been introduced with a new code base deployed recently and the currently working on a rollback. We will provide an update within 1 hour.
Posted Dec 11, 2019 - 13:47 UTC
Investigating
We have noticed an increase in the rate of allocator component restarts due to out of memory exceptions following a recent deployment in Azure. We are working on rolling back to the previous version. Customers might experience issues creating new deployments and applying plan changes to their existing deployments.
Posted Dec 11, 2019 - 13:44 UTC
This incident affected: Azure Singapore (azure-southeastasia) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-southeastasia), Azure Virginia (azure-eastus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2), Azure Netherlands (azure-westeurope) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope), and Azure Washington (azure-westus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2).