Network Degradation via Los Angeles

This issue has been resolved. Our summary is below:

At 11:36 PM UTC on Friday March 8, 2024 the NOC received alarms for SSL/TLS processing in our Los Angeles (LAX) POP. Engineers began investigating immediately and were unable to quickly determine the cause of the processing issues. Traffic was diverted away from the LAX POP at 11:43 PM UTC and those customers who were affected by the SSL/TLS processing issue were resolved as a result of this diversion since traffic was then processed successfully by neighboring POPs. At 12:21 a SSL/TLS offload appliance was determined to be the root cause and it was removed from service for later analysis by our vendor. Additional tests were performed against the remaining infrastructure at the LAX POP and it was brought back into service gracefully at approximately 01:30 AM UTC on Saturday March 9th, 2024.

We sincerely apologize for this issue. It has been determined that approximately 18% of our customers, and only those who had traffic traversing the LAX POP, were affected by this incident. Traffic via other POPs was not affected. We have many automated systems to detect and remediate these types of issues, but the way the errors were presented were somewhat unique in this case and not one our automation scripts were able to detect due to unique error keywords not seen before. We will be following-up with our vendor to determine the root cause of this issue and also work to automate a resolution to a repeat of this exact issue.

Since all traffic has been moved from this POP, customers are no longer affected. We are downgrading this from a Major to a Minor incident at this time.

Engineers have identified an issue in the Los Angeles POP with SSL processing. This POP has been removed from service while further investigations are completed. Customer traffic should no longer be impacted in this region.

We are currently investigating alarms for network availability issues in the Los Angeles region. Engineers are actively investigating the issue at this time. We will post updates here as they become available. Thank you for your patience as we work to restore service to normal levels as quickly as possible.