Vasileios Giotsas
Georgios Smaragdakis
ABSTRACT
Peering infrastructures, namely, colocation facilities and Internet exchange points, are located in every major city, have hundreds of network members, and support hundreds of thousands of interconnections around the globe. These infrastructures are well provisioned and managed, but outages have to be expected, e.g., due to power failures, human errors, attacks, and natural disasters. However, little is known about the frequency and impact of outages at these critical infrastructures with high peering concentration.
In this paper, we develop a novel and lightweight methodology for detecting peering infrastructure outages. Our methodology relies on the observation that BGP communities, announced with routing updates, are an excellent and yet unexplored source of information allowing us to pinpoint outage locations with high accuracy. We build and operate a system that can locate the epicenter of infrastructure outages at the level of a building and track the reaction of networks in near real-time. Our analysis unveils four times as many outages as compared to those publicly reported over the past five years. Moreover, we show that such outages have significant impact on remote networks and peering infrastructures. Our study provides a unique view of the Internet's behavior under stress that often goes unreported.
Supplemental Material
- B. Ager, N. Chatzis, A. Feldmann, N. Sarrar, S. Uhlig, and W. Willinger. 2012. Anatomy of a Large European IXP. In ACM SIGCOMM. Google Scholar
Digital Library
- AMS-IX. 2016. Connected Parties. https://ams-ix.net/connected_parties. (2016).Google Scholar
- Ars Technica. 2016. BT, other ISPs hit by second major Internet outage-power failure blamed. goo.gl/fGx6nF. (July 2016).Google Scholar
- ATNOG Mailing List. 2015. AMSIX Heute. https://atnog.at/pipermail/atnog/2015-May/000022.html. (May 2015).Google Scholar
- R. Banerjee, A. Razaghpanah, L. Chiang, A. Mishra, V. Sekar, Y. Choi, and P. Gill. 2015. Internet Outages, the Eyewitness Accounts: Analysis of the Outages Mailing List. In PAM.Google Scholar
- G. D. Battista, T. Refice, and M. Rimondini. 2006. How to extract BGP peering information from the internet routing registry. In SIGCOMM workshop on Mining network data. ACM, 317--322. Google ScholarDigital Library
- A. Beccaris, D. Quinn, D. Barroso, H. Adollarsson, and M. Walster. 2016. PINDER: peer speed-dating. RIPE NCC IXP tools hackathon. http://accel.waffle.sexy/pinder.pdf. (October 2016).Google Scholar
- K. Benson, A. Dainotti, K. Claffy, and E. Aben. 2012. Gaining Insight into AS-level Outages through Analysis of Internet Background Radiation. In TMA.Google Scholar
- R. Beverly and L. Alt. 2014. On the Potential for Mining Unstructured Public Data to Aid Network Intelligence. (2014).Google Scholar
- S. Bird. 2006. NLTK: The Natural Language Toolkit. In COLING-ACL.Google ScholarDigital Library
- S. Brito, M. Santos, R. Fontes, D. Perez, and C. Rothenberg. 2016. Dissecting the Largest National Ecosystem of Public Internet eXchange Points in Brazil. In PAM. Google ScholarCross Ref
- Broadband Internet Technical Advisory Group Report (BITAG). 2014. Interconnection and Traffic Exchange on theInternet. (2014).Google Scholar
- C. Partridge, P. Barford, D. D. Clark, S. Donelan, V. Paxson, J. Rexford, and M. K. Vernon. 2003. The Internet Under Crisis Conditions: Learning from September 11. The National Academy Press.Google Scholar
- X. Cai, J. Heidemann, B. Krishnamurthy, and W. Willinger. 2010. Towards an AS-to-Organization Map. In ACM IMC.Google Scholar
- CAIDA. 2016. Archipelago (Ark) Measurement Infrastructure. http://www.caida.org/projects/ark/. (2016).Google Scholar
- I. Castro, J. C. Cardona, S. Gorinsky, and P. Francois. 2014. Remote Peering: More Peering without Internet Flattening. In CoNEXT.Google ScholarDigital Library
- R. Chandra, P. Traina, and T. Li. 1996. BGP Communities Attribute. IETF RFC 1997. (1996).Google ScholarDigital Library
- N. Chatzis, G. Smaragdakis, A. Feldmann, and W. Willinger. 2013. There is More to IXPs than Meets the Eye. ACM CCR 45, 5 (2013). Google ScholarDigital Library
- K. Chen, D. Choffnes, R. Potharaju, Y. Chen, F. Bustamante, D. Pei, and Y. Zhao. 2009. Where the sidewalk ends: Extending the Internet AS graph using traceroutes from P2P users. In CoNEXT.Google Scholar
- K. Cho, C. Pelsser, R. Bush, and Y. Won. 2011. The Japan Earthquake: the impact on traffic and routing observed by a local ISP. In ACM CoNEXT SWID workshop. Google ScholarDigital Library
- B. Claise, B. Trammell, and P. Aitken. 2013. RFC 7011: Specification of the IPFIX Protocol for the Exchange of Flow Information. (2013).Google Scholar
- Cymru. 2016. BGP Bogon Refence. https://goo.gl/An2cdU. (2016).Google Scholar
- A. Dainotti, R. Amman, E. Aben, and kc claffy. 2012. Extracting Benefit from Harm: Using Malware Pollution to Analyze the Impact of Political and Geophysical Events on the Internet. ACM CCR 42, 1 (2012).Google Scholar
- A. Dainotti, C. Squarcella, E. Aben, KC Claffy, M. Chiesa, M. Russo, and A. Pescape. 2011. Analysis of Country-wide Internet Outages Caused by Censorship. In ACM IMC. Google ScholarDigital Library
- Data Center Dynamics. 2016. http://www.datacenterdynamics.com/. (2016).Google Scholar
- Data Center Knowledge. 2016. http://www.datacenterknowledge.com/. (2016).Google Scholar
- Data Center Map. 2016. http://www.datacentermap.com/. (2016).Google Scholar
- DE-CIX. 2016. Connected networks in DE-CIX Frankfurt. https://goo.gl/DnPz6s. (2016).Google Scholar
- DE-CIX. 2016. Frankfurt enabled sites. https://goo.gl/BG2yWv. (August 2016).Google Scholar
- Department of Homeland Security. 2010. Communications Sector-Specific Plan: An Annex to the National Infrastructure Protection Plan. (2010).Google Scholar
- C. Dietzel, A. Feldmann, and T. King. 2016. Blackholing at IXPs: On the Effectiveness of DDoS Mitigation in the Wild. In PAM.Google Scholar
- F. Dobrian, A. Awan, D. Joseph, A. Ganjam, J. Zhan, V. Sekar, I. Stoica, and H. Zhang. 2011. Understanding the Impact of Video Quality on User Engagement. In ACM SIGCOMM. Google ScholarDigital Library
- B. Donnet and O. Bonaventure. 2008. On BGP communities. ACM CCR 38, 2 (Mar 2008), 55--59. Google ScholarDigital Library
- R. Durairajan, P. Barford, J. Sommers, and W. Willinger. 2015. InterTubes: A Study of the US Long-haul Fiber-optic Infrastructure. In ACM SIGCOMM. Google ScholarDigital Library
- Z. Durumeric, E. Wustrow, and J. A. Halderman. 2013. ZMap: Fast Internet-Wide Scanning and its Security Applications. In USENIX Security Symposium.Google ScholarDigital Library
- Equinix. 2016. Equinix Expands Data Center Leadership Position with Close of Telecity Acquisition. https://goo.gl/vbXDCv. (2016).Google Scholar
- Equinix. 2016. Investor Relations: Annual and Quarter Results. http://investor.equinix.com/. (2016).Google Scholar
- European Internet Exchange Association. 2016. https://www.euro-ix.net. (2016).Google Scholar
- European Union Agency for Network and Information Security. 2010. Critical Infrastructures and Services, Internet Infrastructure: Internet Interconnections. https://goo.gl/SJMJfJ. (2010).Google Scholar
- R. Fanou, P. Francois, and E. Aben. 2015. On the Diversity of Interdomain Routing in Africa. In PAM. Google ScholarCross Ref
- P. Faratin, D. Clark, S. Bauer, W. Lehr, P. Gilmore, and A. Berger. 2008. The Growing Complexity of Internet Interconnection. Communications and Strategies (2008).Google Scholar
- A. Feldmann, O. Maennel, Z. M. Mao, A. Berger, and B. Maggs. 2004. Locating Internet routing instabilities. In ACM SIGCOMM. Google ScholarDigital Library
- J. R. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Annual Meeting on Association for Computational Linguistics. Google ScholarDigital Library
- K. Foster. 2003. Application of BGP Communities. The Internet Protocol Journal 6, 2 (Sep 2003).Google Scholar
- France-IX. 2017. Interconnection with other IXPs. https://goo.gl/gBYcRH. (2017).Google Scholar
- A. Gerber and R. Doverspike. 2011. Traffic Types and Growth in Backbone Networks. In OFC/NFOEC. Google ScholarCross Ref
- M. Ghobadi and R. Mahajan. 2016. Optical Layer Failures in a Large Backbone. In IMC. Google ScholarDigital Library
- V. Giotsas, A. Dhamdhere, and k. claffy. 2016. Periscope: Unifying Looking Glass Querying. In PAM.Google Scholar
- V. Giotsas, M. Luckie, B. Huffaker, and k. claffy. 2014. Inferring Complex AS Relationships. In IMC.Google Scholar
- V. Giotsas, G. Smaragdakis, B. Huffaker, M. Luckie, and k. claffy. 2015. Mapping Peering Interconnections at the Facility Level. In CoNEXT.Google Scholar
- V. Giotsas and S. Zhou. 2013. Improving the Discovery of IXP Peering Links through Passive BGP Measurements. In Global Internet.Google Scholar
- V. Giotsas, S. Zhou, M. Luckie, and kc claffy. 2013. Inferring Multilateral Peering. In CoNEXT.Google Scholar
- Google. 2016. Google Maps Geocoding API. https://goo.gl/mvDy17. (2016).Google Scholar
- S. P. Gorman and E. J. Malecki. 2002. Fixed and Fluid: Stability and Change in the Geography of the Internet. Telecommunications Policy 26, 7--8 (2002), 389--413.Google ScholarCross Ref
- A. Gupta, M. Calder, N. Feamster, M. Chetty, E. Calandro, and E. Katz-Bassett. 2014. Peering at the internet's frontier: A first look at isp interconnectivity in africa. In PAM. Google ScholarDigital Library
- J. Heidemann, L. Quan, and Y. Pradkin. 2012. A Preliminary Analysis of Network Outages During Hurricane Sandy. Technical Report ISI-TR-2008-685b. USC/Information Sciences Institute.Google Scholar
- N. Hilliard, E. Jasinska, R. Raszuk, and N. Bakker. 2016. Internet Exchange BGP Route Server Operations. IETF RFC 7948. (September 2016).Google Scholar
- Bradley Huffaker, Marina Fomenkov, et al. 2014. DRoP: DNS-based router positioning. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 5--13. Google ScholarDigital Library
- Init7 NOC. 2016. BGP Communities For Init7 customers. http://as13030.net/static/pdf/as13030_bgp_communities.pdf. (January 2016).Google Scholar
- E. Katz-Bassett, H. V. Madhyastha, J. P. John, A. Krishnamurthy, D. Wetherall, and T. Anderson. 2008. Studying Black Holes in the Internet with Hubble. In NSDI.Google Scholar
- R. Kloti, B. Ager, V. Kotronis, G. Nomikos, and X. Dimitropoulos. 2016. A Comparative Look into Public IXP Datasets. ACM CCR 46, 1 (2016). Google ScholarDigital Library
- C. Labovitz, S. Lekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian. 2010. Internet Inter-Domain Traffic. In ACM SIGCOMM. Google ScholarDigital Library
- C. Lees, J. Paussa, and A. Fenioux. 2016. Peer Match-making. RIPE NCC IXP tools hackathon. https://goo.gl/g1uZKA. (October 2016).Google Scholar
- C. Lévy-Bencheton, L. Marinos, R. Mattioli, T. King, C. Dietzel, and J. Stumpf. 2015. Threat Landscape and Good Practice Guide for Internet Infrastructure. EU Agency for Network and Information Security (ENISA) (2015).Google Scholar
- LINX. 2016. LINX From Anywhere. https://goo.gl/gHhfn1. (2016).Google Scholar
- LINX. 2016. LINX Virtual PoP Programme. https://goo.gl/eABGhC. (2016).Google Scholar
- Outage Reporting List. 2016. https://puck.nether.net/pipermail/outages/. (2016).Google Scholar
- A. Lodhi, N. Larson, A. Dhamdhere, C. Dovrolis, and K. Claffy. 2014. Using PeeringDB to Understand the Peering Ecosystem. ACM CCR 44, 2 (2014). Google ScholarDigital Library
- H. V. Madhyastha, T. Isdal, M. Piatek, C. Dixon, T. Anderson, A. Kirshnamurthy, and A. Venkataramani. 2006. iPlane: An information plane for distributed systems. In ACM OSDI.Google Scholar
- E. J. Malecki. 2002. The Economic Geography of the Internet's Infrastructure. Economic Geography 78, 4 (2002), 399--424. Google ScholarCross Ref
- E. J. Malecki. 2012. Internet networks of world cities: agglomeration and dispersion. International Handbook of Globalization and World Cities (2012), 117.Google Scholar
- R. Motamedi, B. Chandrasekaran, B. Maggs, R. Rejaie, and W. Willinger. 2014. On the Geography of X-Connects. Technical Report CIS-TR-2014-02. University of Oregon.Google Scholar
- W. Muhlbauer, A. Feldmann O. Maennel, M. Roughan, and S. Uhlig. 2006. Building an AS-Topology Model that Captures Route Diversity. In ACM SIGCOMM.Google Scholar
- NANOG. 2016. Mailing List and Archives. https://www.nanog.org/list/archives. (2016).Google Scholar
- NOC Incident Reporting websites. 2017. http://goo.gl/iNRlRY. Version 1. (2017).Google Scholar
- G. Nomikos and X. Dimitropoulos. 2016. traIXroute: Detecting IXPs in traceroute paths. In PAM.Google Scholar
- W. B. Norton. 2010. The Art of Peering: The Peering Playbook. (2010).Google Scholar
- W. B. Norton. 2014. European vs. U.S. Internet Exchange Points. https://goo.gl/qvsgG4. (2014).Google Scholar
- C. Orsini, A. King, D. Giordano, V. Giotsas, and A. Dainotti. 2016. BGPStream: a software framework for live and historical BGP data analysis. In IMC. Google ScholarDigital Library
- V. Paxson. 1997. End-to-End Routing Behavior in the Internet. IEEE/ACM Transactions on Networking 5, 5 (1997), 601--615. Google ScholarDigital Library
- PeeringDB. 2016. IXPs and colocation database. https://www.peeringdb.com. (2016).Google Scholar
- I. Poese, B. Frank, G. Smaragdakis, S. Uhlig, A. Feldmann, and B. Maggs. 2012. Enabling Content-aware Traffic Engineering. ACM CCR 42, 5 (2012). Google ScholarDigital Library
- L. Quan, J. Heidemann, and Y. Pradkin. 2012. Detecting Internet Outages with Precise Active Probing. Technical Report ISI-TR-701. USC/Information Sciences Institute.Google Scholar
- L. Quan, J. Heidemann, and Y. Pradkin. 2013. Trinocular: Understanding Internet Reliability Through Adaptive Probing. In ACM SIGCOMM.Google ScholarDigital Library
- B. Quoitin, C. Pelsser, L. Swinnen, O. Bonaventure, and S. Uhlig. 2003. Interdomain Traffic Engineering with BGP. IEEE Communications Magazine 41, 5 (2003), 122--128. Google ScholarDigital Library
- V.R. Raja, A. Dhamdhere, A. Scicchitano, S. Shakkottai, and S. Leinen. 2014. Volume-Based Transit Pricing: Is 95 the Right Percentile?. In PAM.Google Scholar
- The Register. 2016. BT internet outage was our fault, says Equinix. https://goo.gl/YBNYwF. (July 2016).Google Scholar
- Y. Rekhter, T. Li, and S. Hares. 2006. A Border Gateway Protocol 4 (BGP-4). IEFT RFC 4271. (2006).Google Scholar
- P. Richter, G. Smaragdakis, A. Feldmann, N. Chatzis, J. Boettger, and W. Willinger. 2014. Peering at Peerings: On the Role of IXP Route Servers. In ACM IMC.Google Scholar
- RIPE NCC. 2016. RIPE Atlas. https://atlas.ripe.net/. (2016).Google Scholar
- RIPE NCC. 2016. RIPE Atlas Rate Limits. https://goo.gl/8Wr6S6. (2016).Google Scholar
- M. Roughan, W. Willinger, O. Maennel, D. Pertouli, and R. Bush. 2011. 10 Lessons from 10 Years of Measuring and Modeling the Internet's Autonomous Systems. IEEE J. on Sel. Areas in Comm. 29, 9 (2011). Google ScholarCross Ref
- S. Sangli, D. Tappan, and Y. Rekhter. 2006. BGP Extended Communities Attribute. IETF RFC 4360. (2006).Google Scholar
- A. Sapegin and S. Uhlig. 2013. On the extent of correlation in BGP updates in the Internet and what it tells us about locality of BGP routing events. Computer Communications 36, 15--16 (2013). Google ScholarDigital Library
- R. Singh and P. Gill. 2016. PathCache: A Path Prediction Toolkit. In SIGCOMM Poster and Demo Session. Google ScholarDigital Library
- R. Stapleton-Gray and W. Woodcock. 2011. National Internet Defense -- Small States on the Skirmish Line. Comm. of the ACM 54, 3 (2011). Google ScholarDigital Library
- Fortune Tech. 2015. Digital Realty Trust to buy Telx in $1.88 billion deal. https://goo.gl/Bq4nJj. (2015).Google Scholar
- D. Turner, K. Levchenko, A. C. Snoeren, and S. Savage. 2010. California Fault Lines: Understanding the Causes and Impact of Network Failures. In ACM SIGCOMM.Google Scholar
- L. Wang, X. Zhao, D. Pei, R. Bush, D. Massey, A. Mankin, S. F. Wu, and L. Zhang. 2002. Observation and Analysis of BGP Behavior under Stress. In ACM IMW. Google ScholarDigital Library
- B. Woodcock and M. Frigino. 2016. Survey of Internet Carrier Interconnection Agreements. Packet Clearing House.. (November 2016).Google Scholar
Index Terms
- Detecting Peering Infrastructure Outages in the Wild
Recommendations
Peer the peers: an overlay id assignment service at internet exchange points
Co-Next Student Workshop '09: Proceedings of the 5th international student workshop on Emerging networking experiments and technologiesP2P networks enable end users to establish services relying neither on a dedicated infrastructure nor on an ISP deployment of enhanced services at the network layer. Regrettably, overlay traffic is not optimal with respect to native connections and ...
Meta-peering: towards automated ISP peer selection
ANRW '21: Proceedings of the Applied Networking Research WorkshopWe introduce meta-peering, a term that encompasses the set of tools needed to ease and automate the Internet Service Provider (ISP) peering process; starting with identifying a list of ISPs that are likely to peer, generating respective BGP ...
An adaptive peer-to-peer live streaming system with incentives for resilience
Recently, there have been a lot of research efforts on peer-to-peer (P2P) live streaming services. P2P systems can be easily deployed since a participating peer's resources (i.e., upload link bandwidth) can be exploited to distribute contents. However, ...
Comments