Loss of Peerings and Transits with Amsterdam POP...

Major Incident Cloud Virtual Private Cloud Single-Site Schinnen Virtual Private Cloud Multi-Site Managed VPS - Windows Managed VPS - Linux Connectivity Autonomous System AS15966 Datacenter Shared Colocatie Schinnen Network Core Network Inter-Datacenter VLANs PoP Location Equinix AM7 Security IP-VPN Shared Firewall Firewall (Managed) Online Back-up VEEAM Cloud Connect Backup Overige / Shared SMTP Servers Spamfilter Webhosting (Windows) Webhosting (Linux)
2020-11-15 08:35 CEST · 5 hours, 55 minutes, 25 seconds

Update

Resolved

Dear customer,

The supplier did a cold start of a module in the backbone network at 11:45. Since then the connection looks stable again. It’s been running without issues for 2,5 hours now, so we have re-enabled all BGP peers towards Amsterdam.

As everything looks stable and the network is back in normal operation, we’ll close this issue as resolved.

Emango NOC

November 15, 2020 · 14:30 CEST
Investigating

Dear customer,

Since 11:15 the connection between Amsterdam and Maastricht is completely down again. To prevent the continues BGP recalculation due to the link flapping, we have shutdown all BGP peers towards Amsterdam until further notice. All traffic is routed through Luxemburg.

Emango NOC

November 15, 2020 · 11:50 CEST
Investigating

Dear customer,

The connection has just gone down again. The fiber supplier has been notified again.

Emango NOC

November 15, 2020 · 11:00 CEST
Monitoring

Dear customer,

Our BGP peers towards Amsterdam have recovered. There was no further impact during this process.

The fiber supplier is still investigating the outage of this morning.

The link is stable, but we’ll continue monitoring.

Inter-Datacenter VLAN’s are also available again.

All services have recovered.

Emango NOC

November 15, 2020 · 10:42 CEST
Update

Dear customer,

The fiber supplier has confirmed there is no connectivity to/from Maastricht due to malfunctioning hardware in the managed connection towards Maastricht. A lot of parties are affected by this outage, including our transport carrier.

The fiber supplier is investigating the issue.

We’ll publish a new update around 10:30.

Emango NOC

November 15, 2020 · 09:46 CEST
Issue

Dear customer,

This morning between 08:35 and 08:37 we experienced a short - partial - traffic loss on our BGP network for our routes towards Amsterdam. This was due to a problem in our transport’s supplier network. Due to the recalculation of the BGP tables, this has caused packet loss for traffic flowing over these peers. The stable peers haven’t been affected by packet loss. The issue has been confirmed by them under ticket T2011 124 and they are currently investigating the issue.

At 08:51 we experienced an outage again and then lost all our BGP peers towards Amsterdam. All traffic is now flowing through our Luxemburg route.

Customers who are using an IP-VPN might be affected by this outage as Inter-Datacenter VLAN’s aren’t available. In this case, users can connect to their cloud environment through a public internet link as a fallback scenario. Customers with a redundant link aren’t affected.

We are waiting for more details coming from their NOC.

As soon as we have more information, we’ll publish an update.

Should you have any questions regarding this incident, please feel free to contact Emango @ 088-4435200 or e-mail support@emango.nl.

We apologize for the inconvenience.

Emango NOC

November 15, 2020 · 09:37 CEST

← Back