Waymo Open Dataset

Waymo Open Dataset
Name	Waymo Open Dataset
Developer	Waymo
Released	2019
Latest release	2020
Domain	Autonomous vehicles, computer vision, robotics
License	Research and non-commercial (varies by release)

Waymo Open Dataset The Waymo Open Dataset is a large-scale multimodal dataset originating from an autonomous driving program operated by Waymo, intended to support research in perception, tracking, and mapping for self-driving systems. It provides synchronized sensors, labeled objects, and benchmarks that are used by academic institutions, industrial laboratories, and standards bodies to develop algorithms for detection, segmentation, and motion prediction. Leading research groups and technology companies use the dataset alongside datasets like KITTI, nuScenes, and Argoverse to advance methods in machine learning, robotics, and computer vision.

Overview

The dataset was produced by Waymo as part of a broader self-driving vehicle initiative that involves partners and stakeholders such as Alphabet, Google, Cruise, Uber ATG, Tesla, Mobileye, NVIDIA, Intel, Bosch, General Motors, FCA, Toyota Research Institute, Ford, Aurora, Zoox, Lyft, BMW, Volkswagen, Siemens, Honda, and Continental. Data collection occurred across metropolitan areas including Phoenix, Chandler, San Francisco, Mountain View, Los Angeles, and Austin, using sensor suites installed on modified vehicles built by engineering teams at Jaguar Land Rover and Chrysler. The project intersects with research traditions exemplified by datasets and efforts like KITTI, Cityscapes, Berkeley DeepDrive, ApolloScape, BDD100K, Mapillary, Oxford RobotCar, CMU, MIT CSAIL, Stanford AI Lab, Carnegie Mellon University, University of Michigan, Cornell, UC Berkeley, ETH Zurich, Max Planck Institute, Facebook AI Research, Microsoft Research, Amazon, Baidu, Tencent, Huawei, and SenseTime.

Dataset Composition

The dataset comprises high-resolution sensor streams captured by LiDAR systems and camera arrays produced by hardware vendors such as Velodyne, Hesai, Luminar, Ouster, Velodyne Lidar, Riegl, ZF, Bosch Sensortec, Continental AG, and ON Semiconductor. It includes 3D point clouds, synchronized 2D images from multi-view cameras, pose and calibration metadata, and HD map annotations that align with mapping efforts from HERE Technologies, TomTom, OpenStreetMap, Garmin, and Esri. Data spans diverse traffic actors and environmental conditions, referencing object classes familiar to robotics and perception labs: vehicles, pedestrians, cyclists, buses, trucks, motorcycles, traffic signals, and signage. Contributors and consumer institutions employing the data include Stanford, MIT, Oxford, Cambridge, Imperial College London, ETH Zurich, Tsinghua University, Peking University, Fudan University, Zhejiang University, Shanghai Jiao Tong University, KAIST, Seoul National University, University of Toronto, University of Waterloo, McGill, University of British Columbia, University of Illinois Urbana–Champaign, Purdue University, Georgia Institute of Technology, and University of Michigan.

Annotation and Ground Truth

Annotations were produced through a pipeline integrating human labelers, semi-automated tools, and internal verification used by teams at Waymo and collaborators, including annotation platforms akin to those from Scale AI, Appen, Alegion, Labelbox, and CloudFactory. Ground truth includes per-frame 3D bounding boxes, object tracking IDs, velocity vectors, occlusion and truncation flags, and semantic segmentation masks informed by mapping partners like HERE and TomTom as well as standards from SAE International, ISO, and IEEE working groups. Quality assurance involved practices typical of datasets curated by institutions such as DARPA, NASA, NIST, and the U.S. Department of Transportation, and auditing comparable to processes at Apple, Samsung Research, Alibaba DAMO Academy, DeepMind, Google Research, and Baidu Research.

Benchmarking and Evaluation Tasks

Waymo Open Dataset defines benchmarks for tasks frequently studied at conferences and venues including CVPR, ICCV, ECCV, NeurIPS, ICRA, IROS, RSS, ICML, AAAI, BMVC, and ACM Multimedia. Standard evaluation tasks comprise 3D object detection, 3D object tracking, motion prediction, and multi-view 2D-3D fusion, with metrics analogous to those used by KITTI, nuScenes, and Argoverse challenges. Research teams from companies and universities—such as Tesla Autopilot researchers, Toyota Research Institute, NVIDIA Drive Labs, Intel Mobileye, Cruise Automation, Uber ATG, Waymo Research, Google Brain, Facebook AI Research, Microsoft Research, Amazon Robotics, DeepMind, OpenAI, and Baidu—use these benchmarks to compare algorithms including variants of PointNet, PointPillars, PV-RCNN, SECOND, CenterPoint, SSD, Faster R-CNN, Mask R-CNN, YOLO, DeepSORT, and DETR3D.

Usage and Tools

Users access the dataset through hosting portals and software stacks that interoperate with machine learning frameworks and tooling from TensorFlow, PyTorch, JAX, ROS, CARLA, LGSVL, Apollo, Open3D, PDAL, PCL, Eigen, Ceres Solver, gRPC, Protobuf, and ONNX. Community-developed tooling for visualization, annotation conversion, and benchmarking has been contributed by academic labs and open-source projects such as OpenCV, Detectron2, MMDetection3D, TensorRT, Ray, DALI, Horovod, Kubeflow, and MLflow. Industrial adopters integrate the dataset into development pipelines alongside simulation platforms created by Unity Technologies, Unreal Engine, AirSim, and NVIDIA Isaac.

Releases and Version History

Initial releases occurred in 2019 and 2020 with subsequent dataset expansions and challenge updates following timelines similar to major dataset efforts like KITTI (2012), Cityscapes (2016), nuScenes (2019), and Argoverse (2019). Release notes and versioning practices reflect collaboration patterns seen in open datasets from Google, Microsoft, Facebook, Amazon, Apple, Baidu, Tencent, and Alibaba. Academic workshops and challenge tracks hosted at conferences such as CVPR, ICCV, ECCV, NeurIPS, and ICRA have referenced specific releases when organizing leaderboards and reproducibility tracks involving teams from institutions including Stanford, MIT, CMU, Oxford, and ETH Zurich.

Impact and Applications

The dataset has influenced research and product development across autonomous vehicle stacks and has been cited in work from researchers at Google Brain, DeepMind, Microsoft Research, NVIDIA Research, Intel Labs, Amazon Science, Baidu Research, SenseTime, Huawei Noah’s Ark Lab, Tencent AI Lab, and academic groups at Stanford, MIT, CMU, and Oxford. Applications extend to perception modules in self-driving systems, robotics research in perception and planning, HD mapping efforts by HERE and TomTom, simulation and virtual testing by Unity and NVIDIA, and regulatory and safety analyses referencing SAE, ISO, and NHTSA guidance. The dataset has helped benchmark advances in deep learning architectures and has accelerated collaborations between universities, research institutes, and corporations such as Alphabet, Google, Ford, General Motors, Toyota, BMW, Uber, Lyft, and Cruise.

Category:Datasets