NA-W POP hardware failures

Minor incident Global Network and Backbone North America - West
2020-08-14 18:58 UTC · 3 days, 4 hours, 36 minutes

Updates

Resolved

Engineers have replaced additional hardware in the cluster operating the US-W region. We have placed this POP back into service and are continuing to monitor the infrastructure for additional errors.

We are considering this incident resolved at this time, however we will reopen should further issues arise.

November 4, 2021 · 15:24 UTC
Update

Engineers are working on additional hardware replacements at this time. Intermittent or delayed UI/API changes may result for the next 60 minutes. We apologize in advance for these issues and are working as quickly as possible to restore full service to this region.

November 4, 2021 · 15:24 UTC
Update

We have seen a reoccurrence of hardware issues in the US-W cluster. Customer traffic was automatically rerouted to alternate POPS. Engineers are investigating at this time.

November 4, 2021 · 15:24 UTC
Monitoring

Working with our vendor, Total Uptime engineers have replaced a number of hardware components in an effort to isolate the issues affecting our US-W ADC cluster.

After exhaustive testing, we have returned this region to service and will continue to monitor for any further issues.

November 4, 2021 · 15:24 UTC
Update

Total Uptime engineers continue to work with our hardware vendor’s Advanced TAC team to isolate the the issues that are impacting our US-W ADC cluster.

No services are significantly affected by this incident. All ADCaaS (Load Balancing, WAF, etc.) and DNS service traffic is being served by other POPs in nearby regions.

The next update will be provided when there is a significant change to report.

November 4, 2021 · 15:24 UTC
Update

Total Uptime engineers are working with our hardware vendor to better troubleshoot the issue. We have engaged their Advanced TAC support and will continue working until we have determined a root cause for this issue.

No services are significantly affected by this incident. All ADCaaS, DNS, Load Balancing, WAF etc. traffic is being served by other POPs in nearby regions.

November 4, 2021 · 15:24 UTC
Investigating

We have seen a reoccurrence of hardware issues in the US-W cluster. Customer traffic was automatically rerouted to alternate POPS. Engineers are investigating at this time.

November 4, 2021 · 15:24 UTC
Resolved

Engineers have replaced additional hardware. We have placed this POP back into service and are continuing to monitor the infrastructure for additional errors.

We are considering this incident resolved at this time, however we will reopen should further issues arise.

November 4, 2021 · 15:24 UTC
Update

We continue to see hardware issues in the US-W POP and will proactively remove it from service to perform further troubleshooting.

November 4, 2021 · 15:24 UTC
Monitoring

Working with our hardware vendor, we have replaced a suspected faulty hardware component in the cluster. We have returned this POP to normal status and it is now processing traffic at this time.

Engineers will continue to monitor systems for further incidents for the next 72 hours.

November 4, 2021 · 15:24 UTC
Update

Total Uptime engineers are working with our hardware vendor to better troubleshoot the issue. We have engaged their Advanced TAC support and will continue working until we have determined a root cause for this issue.

No customer traffic is affected by this incident. All traffic is being served by other POPs.

November 4, 2021 · 15:24 UTC
Issue

We are experiencing repetitive hardware issues with our ADC cluster in the NA-W region. Traffic was immediately routed to alternate locations at the time of failure by our automation systems. Engineers continue to investigate this issue for resolution. We will post updates here as they become available. Thank you for your patience as we work to restore service to normal levels as quickly as possible.

November 4, 2021 · 15:24 UTC

← Back