April 6, 2022
We’ve experienced an incident on our platform in Amsterdam that caused an outage.
The incident was short but affected many customers. 74,5% of all the BGP-sessions on the platform were interrupted. The incident occurred between 10:20 and 10:45. The incident is now resolved.
The incident was caused during preparations for a routing test prior to a standard planned maintenance on the platform. Due to an unfortunate set of circumstances, an action by an AMS-IX engineer set the wrong configurations on one of the ports, which created a loop on the production network. This loop disrupted the switching of packets across the AMS-IX platform, causing connected customers to experience outages.
The issues were resolved by disabling the ports that were causing the loop. Our customers and members were updated on the root cause via our tech-l mailing list yesterday.
We are reviewing our internal procedures to make sure that we will limit the possibility of a similar event happening in the future. We sincerely apologize for any inconvenience caused by the outage.