Unable to upgrade previously externally monitored clusters to 7.5

Incident Report for ESS (Public)

Resolved

This incident has been resolved.

Posted Dec 12, 2019 - 07:08 UTC

Monitoring

The fix for the 7.5 upgrade issue was rolled to all regions in Azure, along with AWS and GCP. We will monitor the issue for the next 6 hours before closing the incident.

Posted Dec 12, 2019 - 00:16 UTC

Update

We have rolled out the fix for the 7.5 upgrade issue to all regions in AWS and GCP. The fix for Azure was rolled back in the other currently open incident. The estimated time to fix for Azure will be within the next 24 hours. We will update when Azure is resolved as well.

Posted Dec 11, 2019 - 18:14 UTC

Update

We've identified a fix for this issue and are currently testing before deploying the fix into production. We expect to have this fix deployed by the end of next week (December 13th).

We will update this status page when the fix has been deployed and validated in production, or if the situation changes.

Posted Dec 06, 2019 - 06:30 UTC

Update

We are continuing to work on a fix for this issue.

We will update this incident within the next 12 hours with further details on when this issue will be fixed.

If you encounter any other issues, or need further help upgrading your clusters to 7.5 please contact us via support@elastic.co

Posted Dec 05, 2019 - 19:32 UTC

Identified

We've uncovered an issue with upgrading a small percentage of clusters to the latest 7.5 release.

Clusters which have previously been monitored by a different Cloud deployment, but currently have monitoring disabled or are self monitored, will fail to elect new master nodes when attempting to upgrade to 7.5. Clusters with monitoring currently enabled, or which have never had external monitoring enabled will not be impacted by this issue.

We're working on fixing this workflow, and will update this incident when we have an estimate on a fix becoming available. In the meantime, customers can upgrade affected clusters by re-enabling external monitoring before attempting the upgrade.

As part of this issue, customers are unable to disable external monitoring on 7.5 clusters via the Cloud console. External monitoring can be disabled in 7.5 clusters by:

* Disabling monitoring in the Cloud Console
* Sending the following request in the API Console


PUT _cluster/settings
{
    "persistent": {
        "xpack.monitoring.collection.enabled": false,
        "xpack.monitoring.exporters.found-user-defined": {
            "type": "http",
            "enabled": false
        }
    }
}

We will update this incident within the next 12 hours with further details on when this issue will be fixed.

If you encounter any other issues, or need further help upgrading your clusters to 7.5 please contact us via support@elastic.co

Posted Dec 05, 2019 - 04:55 UTC

This incident affected: GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2), Azure Singapore (azure-southeastasia) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-southeastasia), AWS Tokyo (ap-northeast-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-northeast-1), AWS Ireland (eu-west-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-1), GCP Oregon (us-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1), Azure Virginia (azure-eastus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2), GCP Tokyo (asia-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast1), Azure Netherlands (azure-westeurope) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope), AWS Sydney (ap-southeast-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-southeast-2), AWS Oregon (us-west-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-2), AWS N. California (us-west-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-1), AWS London (eu-west-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-2), GCP N. Virginia (us-east4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east4), GCP Iowa (us-central1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1), AWS Frankfurt (eu-central-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-central-1), GCP Sydney (australia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP australia-southeast1), Azure Washington (azure-westus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2), AWS São Paulo (sa-east-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS sa-east-1), AWS Singapore (ap-southeast-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-southeast-1), and AWS N. Virginia (us-east-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-1).