BGPstream

BGPstream
Name	BGPstream
Developer	CAIDA
Released	2014
Programming language	C, Python
Operating system	Linux, macOS
License	BSD

BGPstream BGPstream is an open-source framework for processing and analyzing large-scale BGP measurement data. It provides tools and libraries to ingest, filter, and correlate streaming and historical Border Gateway Protocol feeds from multiple measurement infrastructures, enabling research and operational workflows for routing, security, and Internet topology studies.

Overview

BGPstream was developed by the Center for Applied Internet Data Analysis team and is used alongside projects and institutions such as CAIDA, RIPE NCC, RIPE Atlas, RouteViews, RIPE RIS, Packet Clearing House, Team Cymru, and USC Information Sciences Institute. It integrates with research initiatives and events including MENOG, ENOG, APRICOT, IETF, NSF programs, and datasets referenced in venues like SIGCOMM, USENIX, NDSS, IEEE INFOCOM, and ACM IMC. BGPstream supports analysis patterns employed by operators at organizations such as Cloudflare, Google, Amazon Web Services, Facebook, and Microsoft and is relevant to studies citing works from Vint Cerf, Jon Postel, Van Jacobson, and research groups at MIT, Stanford University, UC Berkeley, Princeton University, Carnegie Mellon University, Georgia Tech, ETH Zurich, University College London, and Tsinghua University.

Architecture and Components

The BGPstream architecture comprises a core C library, Python bindings, command-line tools, and processing modules that interact with measurement collectors like RouteViews and RIPE RIS. The core design parallels toolchains used in projects such as SiLK, Bro/Zeek, Snort, Suricata, and nfdump, and interoperates with storage and processing systems like Hadoop, Apache Kafka, Apache Spark, Elasticsearch, and Prometheus. Components include parsers for MRT and BMP formats standardized by IETF working groups, adapters for feeds produced by BGPmon, OpenBMP, and export mechanisms compatible with dashboards from Grafana and visualization systems inspired by Gephi and Cytoscape.

Data Sources and Collection

BGPstream consumes feeds from prominent collectors and measurement platforms including RouteViews, RIPE RIS, RIPE Atlas, Packet Clearing House, BGPmon, OpenBMP, and archives maintained by research centers like CAIDA and APNIC. It processes MRT archives and real-time streams produced under standards and discussions in IETF meetings and integrates with traceroute infrastructures such as Archipelago and DIMES to correlate path data. BGPstream has been used to analyze events documented in historical datasets tied to incidents involving networks operated by Level 3 Communications, NTT Communications, Akamai Technologies, Verizon Business, CenturyLink, and autonomous systems monitored in studies at Los Alamos National Laboratory and Lawrence Berkeley National Laboratory.

Use Cases and Applications

Researchers employ BGPstream for outage detection and routing anomaly studies cited in publications at ACM IMC, USENIX Security, NDSS, and IEEE S&P. Operators use BGPstream-derived insights for incident response at companies like Akamai, Cloudflare, Google, Microsoft Azure, Amazon Web Services, and network operators participating in regional forums such as NANOG, RIPE, APNIC, LACNIC, and AFRINIC. Security analysts leverage it to study prefix hijacks, leaks, and route oscillations previously analyzed in case studies about events involving YouTube Pakistan, Indosat, and large-scale routing incidents that affected services like Facebook and Amazon. BGPstream supports forensic workflows for compliance and research tied to projects and grants by NSF, DARPA, IARPA, and collaborations with institutions including University of Michigan, Rutgers University, Duke University, and Pennsylvania State University.

Performance and Scalability

BGPstream is optimized for performance with streaming parsers and incremental filtering to handle high-volume feeds comparable to production loads seen by Tier 1 ISP operators such as NTT, CenturyLink, AT&T, Deutsche Telekom, and Telefonica. Its design allows horizontal scaling using message buses like Apache Kafka and batch processing on platforms such as Apache Hadoop and Apache Spark. Benchmarks reported by CAIDA and academic groups compare throughput characteristics with other analysis frameworks used in SIGCOMM research and evaluate trade-offs relevant to deployments at major cloud providers including Google Cloud Platform, Amazon EC2, and Microsoft Azure.

Development and Community Support

Development is coordinated by researchers and engineers at CAIDA with contributions from universities and industry partners including USC Information Sciences Institute, UC San Diego, ETH Zurich, RIPE NCC, RouteViews, and community participants from NANOG and IETF mailing lists. Documentation, tutorials, and workshops have been presented at conferences like IMC, SIGCOMM', USENIX, Black Hat, and in academic courses at MIT, Stanford University, and UC Berkeley. The project benefits from collaborations with standards bodies like IETF and regional registries including ARIN and APNIC and is cited in theses and dissertations from institutions such as Columbia University, Cornell University, Brown University, and Imperial College London.

Category:Internet measurement tools