Partial Service Disruption
October 15, 2024 6:13PM UTC
[Resolved] Resolved Sept 29
September 29, 2024 6:57PM UTC
[Monitoring] Current Status:
We have resolved all issues related to SiteBacker/Traffic Controller/Simple Failover failover/failback at this time. We will continue to monitor.
Previous Impact:
15:30 - 16:50 UTC: Manual failover/failback for SiteBacker/Traffic Controller/Simple Failover pools were delayed.
15:30 - 18:50 UTC: Probe-driven failover/failback for SiteBacker/Traffic/Simple Failover pools were delayed.
While the issue was ongoing, there would have been propagation delays to any SiteBacker/Traffic Controller/Simple Failover whose serving records needed to change as a result of either manual or probe-driven changes.
There was no impact to SiteBacker/Traffic Controller/Simple Failover pools that didn't need to make record updates and no impact to DNS records that do not use SiteBacker/Traffic Controller/Simple Failover.
September 29, 2024 6:19PM UTC
[Monitoring] Current Status:
Probe-driven (automated) failovers/failbacks for SiteBacker/Traffic Controller/Simple Failover are restored in some regions but still facing delays in a few regions. We are working to restore to normal propagation times in remaining regions and will update once this issue has been resolved in all locations.
Previous Impact:
15:30 - 16:50 UTC: Manual failovers for SiteBacker/Traffic Controller/Simple Failover were delayed.
September 29, 2024 5:48PM UTC
[Monitoring] Start Time: 15:30 GMT
Current Status:
We see SiteBacker/Traffic Controller/Simple Load Balancing force fail/force active updates processing again.
However, we are still seeing issues with new probes and updates to probe definitions for existing pools.
Previous Impact:
All failover/failback events for Sitebacker/Traffic Controller and Simple Load Balancing pools were delayed.
September 29, 2024 5:03PM UTC
[Monitoring] Start Time: 15:30 GMT
End Time: 16:50 GMT
Current Status:
We have mitigated the root cause and see SiteBacker/Traffic Controller/Simple Load Balancing failover/failback updates processing as normal again. We will continue to monitor.
Previous Impact:
All failover/failback events for Sitebacker/Traffic Controller and Simple Load Balancing pools were delayed.
September 29, 2024 5:00PM UTC
[Monitoring] We are failing over service to USWEST, and things are coming up. We will share updates in next 15 mins.
September 29, 2024 4:41PM UTC
[Identified] Starting from 15:30 AM GMT, all failover/failback events for Sitebacker/Traffic Controller and Simple Load Balancing pools are delayed. We have identified the root cause and are working on mitigation.