Concourse CI

Concourse CI
Name	Concourse CI
Developer	Pivotal Software; later community contributors
Released	2014
Programming language	Go
Operating system	Cross-platform
License	Apache License 2.0

Concourse CI Concourse CI is an open-source continuous integration and continuous delivery system designed for pipeline-as-code workflows and reproducible build environments. It emphasizes declarative pipelines, containerized tasks, and simple primitives for resource management to support software delivery for teams at scale. Widely adopted in cloud-native, platform engineering, and DevOps contexts, it integrates with container runtimes, orchestration platforms, and version control systems to automate build, test, and deployment processes.

Overview

Concourse CI provides a minimal set of primitives—pipelines, jobs, resources, and tasks—implemented to enable repeatable automation across distributed systems. Influenced by concepts from Docker, Kubernetes, BOSH, Cloud Foundry, and HashiCorp tooling, it targets organizations seeking immutable, observable pipelines. Its architecture separates orchestration from execution; components such as the web UI and worker pools are designed to run on platforms including Linux, Amazon Web Services, Google Cloud Platform, Microsoft Azure, and private data centers. Projects and teams in companies like Pivotal Software, VMware, Intel, Netflix-adjacent ecosystems, and cloud-native communities have used it alongside tools such as Jenkins, GitLab CI/CD, Travis CI, and CircleCI.

Architecture

Concourse CI's architecture centers on a centralized coordinator (the "ATC") and ephemeral workers that execute tasks inside containers. The design draws on principles from Unix pipelines, Erlang fault-tolerance ideas, and the container model popularized by Docker Engine and containerd. Core components include a web UI, a scheduler, a database backend (often PostgreSQL), and worker processes that can run on virtual machines managed by Kubernetes, Nomad (software), or directly on Amazon EC2. Artifact handling is performed through versioned resources with support for external stores such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. Concourse CI uses container images from registries like Docker Hub, Quay.io, and private registries; it orchestrates container lifecycle and resource caching to provide reproducible builds.

Pipeline Concepts and Configuration

Pipelines in Concourse CI are declared as YAML files that describe jobs, resources, and tasks. Jobs chain resource inputs and task steps; resources represent external systems such as GitHub, Bitbucket, GitLab, Artifactory, Maven Central, and container registries. Tasks run within containers specified by images and can execute arbitrary scripts or languages including Python (programming language), Go (programming language), Java (programming language), Ruby (programming language), and Node.js. The pipeline model supports triggers, manual approvals, and fan-in/fan-out job graphs similar to directed acyclic graphs used in Apache Airflow and workflow engines like Argo Workflows. Configuration-as-code enables pipelines to be versioned alongside application code in repositories such as GitHub and promoted through environments like staging and production with traceability comparable to practices advocated by The Twelve-Factor App methodology proponents.

Deployment and Operations

Operators deploy Concourse CI on infrastructure managed by orchestration systems such as Kubernetes, Terraform, and Helm charts; provisioning often leverages Ansible or platform-specific installers. High-availability setups employ multiple ATC instances with a shared PostgreSQL store and external blobstores; scaling is achieved by increasing worker counts, often using autoscaling groups on Amazon EC2 Auto Scaling or Google Compute Engine. Monitoring and observability integrate with systems like Prometheus, Grafana, ELK Stack (Elasticsearch, Logstash, Kibana), and tracing solutions like Jaeger. Backup and disaster recovery typically involve continuous snapshots of PostgreSQL and object storage, combined with immutable pipeline definitions stored in GitHub or GitLab.

Integrations and Ecosystem

Concourse CI integrates with version control systems, container registries, artifact repositories, cloud providers, and authentication providers. Common integrations include GitHub, GitLab, Bitbucket, Docker Hub, Quay.io, Harbor (software) registries, Nexus Repository, JFrog Artifactory, AWS IAM, Google Cloud IAM, and OAuth 2.0 providers like Okta and Auth0. Community-contributed resources and tasks extend functionality with adapters for Slack, PagerDuty, Datadog, Snyk, SonarQube, and HashiCorp Vault. The ecosystem contains reusable task images and resource types published in registries and curated by contributors from companies and open-source organizations including Cloud Native Computing Foundation-affiliated projects.

History and Development

Concourse CI originated in the mid-2010s within the ecosystem around Pivotal Software as an attempt to redesign CI/CD runtime primitives for cloud-native deployments. Early design discussions referenced practices from Continuous Delivery (book), 12-factor app, and containerization patterns emerging from Docker Inc. and CoreOS. Over time the project transitioned from a single-vendor-maintained codebase to a broader open-source community with contributions from engineers at VMware, Cloud Foundry Foundation, and independent maintainers. Releases have introduced multi-platform worker support, improved authentication, and enhanced pipeline primitives; the project has been discussed at conferences such as KubeCon, Velocity Conference, and DockerCon.

Security and Compliance

Security in Concourse CI focuses on container isolation, credential management, and auditable pipelines. Credential handling commonly leverages secret backends such as HashiCorp Vault, cloud native IAM services like AWS IAM and Google Cloud IAM, and OAuth providers including Okta. Role-based access and authentication are integrated through external identity providers, while network segmentation and runtime isolation are implemented via container runtime security features from runc and orchestration controls provided by Kubernetes RBAC. Compliance-conscious deployments adopt practices from standards such as NIST Cybersecurity Framework and enterprise policies enforced by teams using SIEM platforms like Splunk.

Category:Continuous integration