Tensor Processing Unit

Tensor Processing Unit
Name	Tensor Processing Unit
Developer	Google
Type	Application-specific integrated circuit
Released	2016

Tensor Processing Unit. A Tensor Processing Unit is a class of application-specific integrated circuit developed by Google specifically to accelerate machine learning workloads. First deployed in 2016 within Google Cloud Platform data centers, these processors are optimized for the high-volume matrix multiplication and convolution operations fundamental to neural network inference and training. Their design represents a significant shift from general-purpose central processing unit and graphics processing unit architectures towards hardware tailored for artificial intelligence.

Overview

The primary function of these units is to execute the computational graphs defined by frameworks like TensorFlow with high efficiency and low latency. They are integral to powering a vast array of Google services, including Google Search, Google Photos, and Google Translate, by handling deep learning predictions. By offloading these intensive tasks from traditional server hardware, they enable more responsive applications and reduce the overall energy consumption of data centers. Their deployment marked a key milestone in the industrialization of AI accelerator technology.

Architecture

Architecturally, these processors feature a systolic array as their core computational unit, which is highly efficient for the large-scale matrix operations prevalent in neural network models. This design minimizes data movement by streaming inputs through a fixed network of processing elements, a concept advanced by researchers like H. T. Kung. Memory hierarchy is carefully tuned, with high-bandwidth on-chip memory placed close to the compute units to feed the array continuously. Different generations, such as those detailed at the International Solid-State Circuits Conference, have introduced support for floating-point formats like bfloat16 and enhanced interconnect technologies for scaled-out deployments.

Development and history

Development was initiated by a team at Google led by engineers including Norman Jouppi, driven by the growing computational demands of projects like RankBrain. The first-generation chip was revealed publicly at the 2016 Google I/O developer conference and was already in use within the Google Cloud Platform. Subsequent generations have been unveiled at events like the Google Next cloud conference, showcasing evolving capabilities for both inference and training workloads. This progression reflects the broader industry trend seen with competitors like NVIDIA with its Tesla (microarchitecture) and AMD with its Instinct series.

Performance and applications

In terms of performance, these processors offer orders-of-magnitude improvements in throughput and performance per watt for specific machine learning models compared to contemporary general-purpose computing on graphics processing units. This efficiency is critical for applications at scale, such as real-time image recognition in Google Street View or language model queries in Google Assistant. Their performance characteristics are often benchmarked against offerings from Intel and Xilinx in industry reports. Within scientific computing, they have been applied to challenges in fields like computational biology and climate modeling.

Comparison with other processors

When compared to a graphics processing unit from NVIDIA, such as the Volta (microarchitecture)-based Tesla V100, these units typically sacrifice programmability for higher efficiency on a narrower set of tensor operations. Unlike field-programmable gate array solutions from Xilinx or Intel, they are fixed-function ASICs, offering lower flexibility but superior performance and power metrics for their target domain. The approach contrasts with the vector processor designs used in classical supercomputers like those from Cray. The competitive landscape also includes other AI accelerator startups like Graphcore and Cerebras Systems.

Software and programming

Software support is primarily channeled through Google's TensorFlow ecosystem, with specific compiler stacks like XLA optimizing computational graphs for execution on the hardware. Programmers using Google Cloud Platform can access these units via services like Cloud AI Platform and Cloud TPU without managing the underlying infrastructure. Lower-level programming can utilize frameworks such as JAX, developed by teams at Google Research. This integrated software approach is similar in philosophy to NVIDIA's CUDA platform but is tailored for a more specialized hardware target.

Category:Google hardware Category:Application-specific integrated circuits Category:Artificial intelligence accelerators