Pandas (software)

Pandas (software)
Marc Garcia · BSD · source
Name	Pandas
Title	Pandas
Author	Wes McKinney
Developer	NumFOCUS
Released	2008
Programming language	Python
Operating system	Cross-platform
License	BSD-3-Clause

Pandas (software) is an open-source data analysis and manipulation library for Python designed to provide high-performance, easy-to-use data structures and data analysis tools. It integrates with a wide range of scientific and data engineering projects, facilitating workflows across NumPy, SciPy, Scikit-learn, Matplotlib and Jupyter Notebook. Widely used in industry and academia, it supports time series, statistics, and tabular data processing for users of Google, Facebook, Amazon, Microsoft and many research institutions.

Overview

Pandas provides key abstractions such as the DataFrame and Series, built atop NumPy arrays and designed to interoperate with libraries like SciPy, Scikit-learn, Matplotlib, Seaborn, Plotly, and Bokeh. It is part of the scientific Python ecosystem that includes IPython, Jupyter Notebook, Anaconda, Conda, pip, and PyPI, enabling reproducible analysis for projects in organizations such as Netflix, Airbnb, Uber, Spotify, and Stripe. The library adheres to a BSD-style license and is stewarded by the nonprofit NumFOCUS, with governance and community contributions coordinated through platforms like GitHub and GitLab.

History and development

Pandas originated in 2008, created by Wes McKinney while working on quantitative finance problems at AQR Capital Management; its development was influenced by earlier projects such as R's data.frame, S-Plus, and the Python numeric stack exemplified by NumPy and SciPy. Its public growth accelerated with adoption by academic labs at Harvard University, MIT, Stanford University, University of California, Berkeley, and industry teams at Two Sigma, Goldman Sachs, and J.P. Morgan. Major releases introduced enhancements influenced by standards from PEP 8, PEP 257, and collaboration with contributors from Microsoft Research, Google Research, Facebook AI Research, and open-source projects like Apache Arrow and Dask. Governance moved towards community model under NumFOCUS with release management via maintainers and working groups that include representatives from Continuum Analytics, Anaconda, Inc., and corporate users.

Features and architecture

Pandas implements labeled, tabular data structures: the two primary types are Series (1-D) and DataFrame (2-D), leveraging NumPy for contiguous memory and numeric operations while exposing APIs familiar to users of R, SAS, SPSS, and Stata. It supports data alignment, handling of missing data, group operations inspired by SQL (structured query language), merge and join semantics analogous to relational algebra and PostgreSQL, and time-series functionality comparable to xts and zoo. Pandas integrates I/O capabilities for formats and systems like CSV, JSON, Parquet, Feather, HDF5, SQL databases, SQLite, MySQL, PostgreSQL, cloud services such as Amazon S3, and big-data engines including Apache Spark and Apache Arrow. Its Cython and C extensions, combined with vectorized operations and the use of Numba for JIT compilation, optimize inner loops, while interoperability with Apache Arrow improves zero-copy data exchange.

Usage and ecosystem

Pandas is commonly used within workflows orchestrated by Jupyter Notebook, JupyterLab, Visual Studio Code, PyCharm, Spyder, and deployed in production with tools such as Docker, Kubernetes, Airflow, Luigi, Prefect, and Great Expectations. It interfaces with machine learning pipelines using Scikit-learn, TensorFlow, PyTorch, and XGBoost, and supports visualization through Matplotlib, Seaborn, Altair, and Plotly. Educational resources and community content are hosted on platforms like Stack Overflow, GitHub, Medium, YouTube, and MOOCs from institutions like Coursera, edX, DataCamp, Udacity, and Kaggle. The ecosystem includes companion projects such as Dask, Modin, Vaex, Polars, PyArrow, and data validation or profiling tools created by companies like IBM and Google.

Performance and scalability

Pandas emphasizes ease of use but faces challenges with very large datasets; solutions include parallel and distributed frameworks like Dask, Modin, Ray, Apache Spark, and columnar engines such as Apache Arrow. Performance improvements have come via integration with Numba for accelerating user-defined functions, Cython optimizations, and efforts to offload operations to vectorized primitives present in BLAS and OpenBLAS. Projects such as Polars and Vaex offer alternative columnar approaches inspired by lessons from Pandas, while cloud-scale data platforms like Snowflake, BigQuery, Databricks, AWS Redshift, and Google Cloud Platform influence design choices for I/O, parallelism, and storage formats like Parquet and ORC. Benchmarking and profiling are performed using tools from perf, pyinstrument, and Intel VTune in enterprise settings at Bloomberg, Goldman Sachs, Morgan Stanley, and Deutsche Bank.

Adoption and applications

Pandas is used across industries for tasks in finance, genomics, web analytics, scientific research, and public policy by organizations such as NASA, NOAA, CDC, World Health Organization, European Space Agency, Bureau of Labor Statistics, Federal Reserve System, Fidelity Investments, Goldman Sachs, Morgan Stanley, Bloomberg, Netflix, Uber, Airbnb, Spotify, Siemens, Boeing, General Electric, Pfizer, Moderna, Roche, Johnson & Johnson, Procter & Gamble, Unilever, and academic groups at Harvard University, MIT, Cambridge University, Oxford University, and University of California, Berkeley. It underpins reproducible research workflows in journals and conferences such as Nature, Science, NeurIPS, ICML, KDD, SIGMOD, and VLDB, and supports data journalism projects at outlets like The New York Times, The Guardian, ProPublica, and FiveThirtyEight.

Category:Python (programming language) libraries