TPU

TPU. A Tensor Processing Unit (TPU) is a category of application-specific integrated circuit (ASIC) developed by Google specifically to accelerate artificial neural network computations, primarily for machine learning and deep learning applications. It is optimized for the high-volume, low-precision matrix operations central to neural network inference and neural network training, serving as a key component within Google Cloud Platform and powering many of the company's services like Google Search and Google Translate.

Overview

The core function of a TPU is to execute operations defined within the TensorFlow framework with high efficiency, acting as a co-processor alongside conventional central processing unit (CPU) and graphics processing unit (GPU) resources. Its architecture is fundamentally designed around a systolic array, a network of processing elements that efficiently performs the massive matrix multiplication operations required for models such as convolutional neural network (CNN) and recurrent neural network (RNN). By specializing in this domain, TPUs offer significant improvements in performance per watt for targeted workloads compared to more general-purpose hardware, enabling faster iteration on complex models like BERT and AlphaGo.

Architecture and design

The architectural hallmark of the TPU is its large, two-dimensional systolic array which forms the matrix multiply unit. This array is fed by a high-bandwidth memory subsystem, historically using DDR3 SDRAM in earlier generations, to keep the computational units saturated with data. Control is managed by a host CPU via the Peripheral Component Interconnect Express (PCIe) interface, with the TPU executing instructions from an internal very long instruction word (VLIW) program. Key design optimizations include the use of 8-bit integer precision for inference tasks, a streamlined pipeline that minimizes datapath overhead, and dedicated hardware for activation function computations like rectified linear unit (ReLU).

Later generations, such as those developed by teams at Google Research, introduced significant advancements. The Cloud TPU v2 and Cloud TPU v3 incorporated floating-point units for higher precision training and adopted a pod configuration, linking multiple chips via a high-speed interconnect fabric. The Edge TPU, a smaller variant, was designed for on-device inference in Internet of things (IoT) and embedded system applications, emphasizing low power consumption. The most recent architectures have continued to scale performance and memory bandwidth, integrating HBM2 and supporting frameworks like JAX and PyTorch alongside TensorFlow.

Performance and applications

TPUs deliver exceptional throughput for both training and inference phases of deep learning, particularly on large-scale models. They have been instrumental in achieving record-breaking results on benchmarks like ImageNet and in reducing the training time for transformative models including GPT-3 and Vision Transformer. Within Google Cloud, they power services such as Google Photos for image recognition and are available for researchers through programs like the TensorFlow Research Cloud. Their performance profile makes them ideal for applications in natural language processing, computer vision, recommendation system development, and scientific computing projects at institutions like CERN.

History and development

The TPU project originated from a need at Google to more efficiently run DeepMind's AlphaGo models and to improve the computational demands of Google Street View and RankBrain. Development began around 2015 under engineers including Norman Jouppi, with the first-generation chip (now referred to as the Cloud TPU v1) deployed in Google data centers in 2016. Its existence was publicly revealed in a paper at the International Symposium on Computer Architecture (ISCA) in 2017. Subsequent generations have been unveiled at events like Google I/O, with continuous development driven by the Google Brain team and collaborations with semiconductor manufacturing partners.

Comparison with other processors

Compared to general-purpose CPUs from Intel or AMD, TPUs offer orders of magnitude higher efficiency for tensor operations but lack flexibility for other computational tasks. Against GPUs from NVIDIA, such as the Tesla V100 or A100, TPUs are more specialized; GPUs offer broader programmability for graphics and general-purpose computing on graphics processing units (GPGPU), while TPUs can achieve higher sustained utilization on their targeted workloads. Other specialized accelerators like the Graphcore Intelligence Processing Unit (IPU) or Cerebras Wafer-Scale Engine offer different architectural approaches to similar problems, often competing on metrics of throughput and latency for specific model architectures. The Field-programmable gate array (FPGA) provides a reconfigurable alternative but typically at a lower peak performance for dedicated neural network tasks.

Category:Google hardware Category:Computer hardware Category:Artificial intelligence