Event Report: CDN Service Downtime

Date: March 30, 2023

Summary:

On March 30, 2023, our CDN (Content Delivery Network) service experienced a downtime from 6:41:18 AM UTC to 6:46:17 AM UTC, lasting for approximately 5 minutes. The downtime occurred during an attempt to resolve a redirection issue. Due to an error in our automatically generated configuration files, the service could not start properly, which led to the disruption.

Initial Issue:

The primary objective of the maintenance was to fix a redirection problem within the CDN service. However, during the maintenance process, we encountered an issue with our automatically generated configuration files. The files contained duplicate settings, causing the service to fail to start normally.

Temporary Solution:

Upon identifying the problematic configuration file, our team quickly deleted it and confirmed that the service could start properly. We then applied the corrected configuration to all CDN nodes to ensure the service resumed normal functionality.

Follow-up Actions:

To prevent similar issues from occurring in the future, we will:

1. Modify our software to ensure that duplicate configuration files are not generated.
2. Implement a staged deployment process for future updates, starting with a small number of nodes before expanding the update to the entire network.

By taking these measures, we aim to minimize the impact of future maintenance and updates on our CDN service, ultimately providing a more reliable experience for our users.

Customer Sites recovered.

Customer Sites went down.