21/01/2020
Incident Report - International Inbound Routing Issue
Some customers would not have been able to access Internet resources and also issues for some Australian based voice customers.
SUMMARY OF INCIDENT
Summary:
At 1101hrs on the 21st of January 2020 we suffered an outage with our direct NZ to Sydney connectivity. This outage had an unintended outcome of causing a blackhole effect for inbound traffic from third party networks on the Sydney exchanges. Example of included third parties are, but not limited to; Facebook, Google, Twitter.
Cause:
The initial cause was a break in service with our NZ to Sydney connectivity. This caused an isolation between our New Zealand based network and our Sydney based network. Our Sydney network was still operating and was advertising our IP ranges into the Sydney peering exchanges. In turn this lead to inbound traffic for our ranges from third party networks on those exchanges to enter our network in Sydney. As Sydney was isolated this caused the traffic to be blackholed.
Impact:
Some customers would not have been able to access Internet resources (websites, voice, gaming etc.) for some third party networks where those providers connected to either MegaIX Sydney or NSW-IX. Supplementary to this there were also issues for some Australian based voice customers who would have had no voice during this time. This was due to the previously mentioned routing issue. The total outage was for 38 minutes.
Mitigation:
We resolved the routing issue by removing the advertisements of our IP addressing to those Sydney Exchanges. This caused traffic from those networks to route to us via alternative redundant paths in New Zealand. Once Sydney to NZ connectivity was restored just after midday we completed some configuration changes within Sydney to ensure this situation doesn't happen again. These changes included removing the "anchoring" of our IP addresses in Sydney. Anchoring of IP addressing is a standard procedure within an ISP configuration. However going forward Sydney will now only advertise our ranges to those exchanges when connectivity to New Zealand is up and working.
Resolution:
We have restored routing and access to the Sydney exchanges via our Trans Tasman connectivity. This will ensure a shorter path for our customers to those third parties that are on those exchanges. Through implementation of the mitigation steps we have ensured a similar scenario will not happen again. As mentioned, what we had in place was standard procedure and configuration for ISPs, but, we have raised a requirement to review this design and work out if there is a better way to do it.
We apologise for these issues and appreciate the patience our customers had while we restored service.