This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| BDA Architecture | |
|---|---|
| Name | BDA Architecture |
| Type | Data architecture |
| Industry | Information technology |
| Introduced | 21st century |
| Components | Data ingestion; storage; processing; analytics; governance |
BDA Architecture BDA Architecture describes system designs and patterns for Big Data Analytics stacks that integrate ingestion, storage, processing, governance, and serving layers to support large-scale data-driven applications. It connects technologies and institutions across data science, cloud computing, and enterprise information systems to enable batch, streaming, and interactive analytics. Practitioners draw on lessons from projects at organizations such as Google, Facebook, Amazon (company), Netflix, and research from University of California, Berkeley, Massachusetts Institute of Technology, and Stanford University.
BDA Architecture situates components like ingestion, storage, processing, and serving in logical tiers influenced by platforms from Hadoop, Spark (software), Flink, Kafka (software), and Druid (database). Influential systems design patterns reference works associated with MapReduce, Lambda architecture, Kappa architecture, and scholarly contributions from Doug Cutting, Jeff Dean, Sanjay Ghemawat, Matei Zaharia, and Neha Narkhede. Early commercial deployments at Cloudera, Hortonworks, MapR, and cloud providers such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure shaped reference architectures used by enterprises including Walmart, Uber, Airbnb, and LinkedIn.
Origins trace to distributed computing milestones like Google File System, Bigtable, and MapReduce that influenced projects at Yahoo!, Facebook, and Twitter. The rise of streaming and real-time needs spurred work from LinkedIn on Apache Kafka and from Twitter on Heron (stream processing), while batch-processing evolved via Apache Hadoop initiatives driven by contributors such as Doug Cutting and Mike Cafarella. Academic advances at Berkeley Data Analytics Stack and initiatives like AMPLab produced Spark (software), with commercialization via Databricks. Architectural debates around Lambda vs Kappa referenced implementations at Pinterest, Spotify, and NASA big-data programs. Standards and governance discussions engaged ISO, NIST, GDPR regulators in the European Union, and industry consortia including Open Data Center Alliance.
Core components include ingestion systems (e.g., Apache NiFi, Flume (software), Logstash), message buses (Apache Kafka, RabbitMQ), storage layers (e.g., Hadoop Distributed File System, Amazon S3, Google BigQuery, Snowflake (company), Cassandra (database), HBase), processing engines (Apache Spark, Flink, Storm (software), Beam (software)), serving layers (Presto (SQL query engine), Druid (database), Elasticsearch), and orchestration tools (Apache Airflow, Oozie). Metadata and governance use Apache Atlas, AWS Glue, Collibra, and identity controls from Okta, Active Directory. Observability integrates Prometheus, Grafana, Datadog, and Splunk.
Pipelines often adopt models inspired by Lambda architecture and Kappa architecture to combine batch and stream processing with technologies like Spark Streaming, Structured Streaming, Apache Flink, and Kafka Streams. ETL/ELT patterns leverage Talend, Informatica, Fivetran, and dbt (data build tool) for transformation, while feature stores developed by Uber and Tecton (company) support machine learning pipelines using TensorFlow, PyTorch, scikit-learn, and orchestration from Kubeflow and MLflow (software). Data lineage and change data capture patterns reference Debezium and Apache NiFi integrations for CDC-based ingestion from databases like PostgreSQL, MySQL, MongoDB, and Oracle Database.
Implementation patterns include the data lake architecture exemplified by Amazon S3 and Delta Lake, the data warehouse pattern implemented by Snowflake (company), Google BigQuery, and Amazon Redshift, and hybrid lakehouse approaches promoted by Databricks and Apache Hudi. Containerization and orchestration use Docker (software), Kubernetes, and service meshes like Istio for microservices. Security and compliance integrate OAuth 2.0, OpenID Connect, TLS, and enterprise IAM from Okta and Azure Active Directory; audit trails rely on Apache Ranger and AWS CloudTrail. Edge and IoT ingestion reference deployments from Cisco Systems, Siemens, and Bosch using MQTT and EdgeX Foundry.
Common applications include real-time analytics for Uber-style dispatch, recommendation systems like those at Netflix and Amazon (company), fraud detection engines used by Visa and Mastercard, telemetry analytics in NASA missions and European Space Agency programs, genomic and biomedical analytics in projects at Broad Institute and Illumina, and smart city platforms implemented by municipalities collaborating with Siemens and IBM. Advertising-tech stacks at Google and The Trade Desk and financial risk systems at Goldman Sachs and JPMorgan Chase frequently adopt BDA reference patterns. Media companies such as The New York Times and The Guardian use similar architectures for personalization and analytics.
Scalability strategies reference horizontal scaling patterns employed by Google, Amazon (company), and Facebook including sharding used in Cassandra (database) and partitioning in Kafka (software). Performance tuning invokes resource managers like Apache YARN, cluster autoscaling from Kubernetes, and query optimizers in Presto (SQL query engine), Spark (software), and Snowflake (company). Security and privacy draw upon GDPR, HIPAA, PCI DSS, and best practices from NIST, with encryption using AES standards, key management via HashiCorp Vault, and threat detection using Splunk and CrowdStrike. Disaster recovery and durability follow multi-region replication strategies used by AWS, Google Cloud Platform, and Microsoft Azure.
Category:Data architectures