EMC Greenplum

EMC Greenplum
Jgreenplum · CC BY-SA 3.0 · source
Name	EMC Greenplum
Developer	EMC Corporation, Pivotal Software, VMware
Initial release	2005
Latest release	2019
Programming language	C (programming language), C++, Python (programming language), SQL
Operating system	Linux
Genre	Data warehouse, Massively parallel processing
License	Proprietary

EMC Greenplum is a commercially supported, massively parallel processing (MPP) data warehouse platform designed for analytic workloads on large datasets. It integrates a distributed shared-nothing architecture with a suite of management tools and extensions to support business intelligence, data science, and big data integration. The platform has been deployed by organizations that also use technologies from Oracle Corporation, IBM, Microsoft Corporation, Amazon Web Services, and Google Cloud Platform to address petabyte-scale analytics and complex queries.

Overview

Greenplum is built to provide scalable, high-throughput analytic processing across commodity hardware, competing with systems such as Teradata, Vertica, Snowflake (company), and SAP HANA. It supports standard Structured Query Language through an extended implementation compatible with PostgreSQL semantics, positioning it alongside PostgreSQL-based distributions and projects like Amazon Aurora and TimescaleDB. The product targets use cases similar to implementations at Facebook, Netflix, Walmart, Capital One, and Verizon Communications where complex ad-hoc analytics, reporting, and machine learning workflows coexist.

Architecture and Components

Greenplum’s architecture follows a master/segment model influenced by MPP designs found in systems like HP NonStop and research platforms from Massachusetts Institute of Technology and University of California, Berkeley. A single Greenplum Master node coordinates SQL planning and metadata, while multiple segment instances store data and execute queries in parallel, resembling architectures used by Google BigQuery and Amazon Redshift. Components include the master service, segment daemons, interconnect network layer, fault monitors, and client utilities that interoperate with tools from Tableau Software, MicroStrategy, Qlik, SAS Institute, and Apache Hadoop ecosystems such as Apache HDFS, Apache Hive, and Apache Spark.

Data Storage and Management

Data in Greenplum is distributed across segments using hashing or range partitioning, similar to distribution strategies in Cassandra (Apache Cassandra), HBase, and MongoDB. On-disk storage uses a columnar storage option and row-oriented storage, enabling workload-specific trade-offs akin to ClickHouse and Amazon Redshift Spectrum. Greenplum integrates with external storage and data integration services from Cloudera, MapR, Informatica, Talend, and IBM InfoSphere to ingest data from sources like Oracle Database, Microsoft SQL Server, SAP ASE, and cloud object stores operated by Amazon S3 and Google Cloud Storage. It supports table partitioning, table inheritance, and secondary indexing strategies influenced by Postgres-XC and Postgres-XL research.

Query Processing and Optimization

Query planning in Greenplum leverages a cost-based optimizer and parallel execution engine inspired by work from University of Wisconsin–Madison and Stanford University database research. The system implements distributed query planning, motion operations, and gather/broadcast patterns comparable to execution strategies used in Teradata and Vertica. The optimizer uses statistics, histograms, and join cardinality estimates similar to those in PostgreSQL, and benefits from extensions and tools developed in collaboration with teams at Pivotal Software and contributors from the Open Source community. Support for user-defined functions and procedural languages allows integration with analytic libraries from R Project, Python (programming language), and GNU Scientific Library.

Deployment, Scaling, and High Availability

Greenplum can be deployed on-premises across clusters built with hardware from Dell Technologies, Hewlett Packard Enterprise, and Cisco Systems, or in cloud environments managed by Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Scaling is achieved by adding segments (nodes) to the cluster and redistributing data, a model comparable to scaling strategies used by Citus Data and Cockroach Labs. High availability is provided via standby masters, mirroring of segment instances, and integration with orchestration and monitoring tools from Kubernetes (software), Ansible, Puppet, Nagios, and Prometheus.

Security and Administration

Security features include role-based access control, authentication integration with LDAP, Kerberos (protocol), and Active Directory, encryption at-rest and in-flight using standards implemented by OpenSSL Project, and auditing compatible with compliance regimes such as procedures used by PCI DSS, HIPAA, and SOX. Administration is supported through management utilities and GUIs that interact with systems like EMC Isilon, VMware vSphere, and enterprise backup solutions from Commvault and Veritas Technologies. Regular maintenance tasks include vacuuming, statistics collection, and segment health checks following practices common to PostgreSQL administrators and enterprise DBAs at organizations like Goldman Sachs and JP Morgan Chase.

History and Development

Greenplum originated from a startup leveraging research in parallel databases and was later acquired and developed by EMC Corporation before stewardship transitioned to Pivotal Software and later associations with VMware. The project’s evolution paralleled shifts in the industry driven by contributions from academic institutions such as University of California, Berkeley and companies including Amazon.com, Inc., Google LLC, and Facebook, Inc. that influenced large-scale analytics. Greenplum’s roadmap and community engagements brought together engineers and researchers who previously contributed to projects at PostgreSQL Global Development Group, Apache Software Foundation, and commercial analytics vendors, shaping its features through collaborations and acquisitions in the enterprise data management landscape.

Category:Data warehousing