Custom Resource Definitions

Custom Resource Definitions

Custom Resource Definitions provide a mechanism to extend the Kubernetes API surface by declaring new API kinds that behave like built-in resources. Originally introduced to enable operators and platform teams to model domain-specific concepts, they integrate with controllers, schedulers, and tooling across the Kubernetes ecosystem while leveraging existing API machinery such as API aggregation, storage, and watch semantics.

Overview

Custom Resource Definitions (CRDs) allow cluster operators and developers to add new resource types to a Kubernetes Kubernetes (container orchestration) cluster without modifying the core server. They complement controller frameworks such as Operator pattern implementations used by projects like Operator Framework and Red Hat OpenShift operators; they are central to platforms like Rancher and distribution projects including k3s. CRDs follow the Kubernetes API conventions and are discoverable through the cluster's API server and OpenAPI-based tooling, enabling integration with client libraries such as client-go and command-line tools like kubectl.

Architecture and Components

A CRD is an API object defined in the apiextensions.k8s.io API group and stored in etcd alongside native kinds. The primary components include the CRD schema, subresources (e.g., status, scale), versions, and the CustomResourceDefinition object itself. Controllers—often implemented with frameworks like Kubebuilder or controller-runtime—watch custom resources and reconcile desired state into actual cluster state, interacting with resources like Deployments, DaemonSets, and Services. CRD-related admission flows include validating and mutating admission webhooks, which integrate with Admission Controller chains and the API Aggregation Layer. The API discovery model relies on OpenAPI Specification snippets generated from CRD schemas.

Usage and Examples

Common uses include implementing operators for databases (e.g., PostgreSQL operators), application lifecycle management for platforms like Istio or Linkerd, and multi-tenant services in platforms such as Google Kubernetes Engine and Amazon EKS. Example CRDs model resources like backups, a cluster-scoped Prometheus scrape target, or application manifests consumed by controllers to create ConfigMaps and Secrets. Tooling examples: Helm charts may install CRDs for operators packaged as charts; Flux and Argo CD manage CRD-backed custom resources as part of GitOps pipelines. Community projects such as Metacontroller and Crossplane demonstrate advanced composition patterns built atop CRDs.

Versioning and Schema Validation

CRDs support versioning via the versions field, enabling gradual API evolution and conversion between versions using conversion webhooks or conversion strategies. Schema validation uses structural schemas expressed in OpenAPI v3-compatible JSON Schema to validate spec and status fields; this validation occurs server-side and prevents invalid resources from persisting in etcd. Conversion webhooks and served versions interact with API discovery and client libraries to support rolling upgrades similar to versioning practices in SemVer-oriented projects. Careful versioning is essential for controller backward compatibility and for integrations with CI/CD services like Jenkins and Tekton.

Admission and Lifecycle Management

Admission control for CRDs includes validating and mutating admission webhooks, which can enforce invariants, inject defaults, or augment status fields during create/update operations. Lifecycle management spans installation, upgrades, and garbage collection: CRD installation order matters when resources reference each other, and finalizers on custom resources help implement safe deletion semantics. Operators commonly implement leader election patterns using libraries from client-go and handle reconciliation loops that respond to events from the API server or external systems such as AWS APIs or Azure resources. Migration strategies often combine conversion webhooks with controller-managed reconciliation to transition live objects safely.

Best Practices and Design Patterns

Recommended practices include designing idempotent controllers, modeling resources as declarative desired state documents, and keeping spec/state separation by writing status only from controllers. Use structural schemas to limit invalid inputs, provide defaulting via defaulting webhooks or CRD defaulting, and adopt semantic versioning for API changes. Composition patterns include higher-level composite resources as used in Crossplane, composing managed resources and using controllers to reconcile children. Avoid embedding dense operational logic in CRD schemas; instead, encode intent and let controllers perform imperative steps. Testing patterns involve unit tests with fake clients and integration tests using lightweight clusters such as kind or k3s.

Security and RBAC Considerations

CRDs introduce expanded attack surface; controllers and admission webhooks require least-privilege Role-Based Access Control rules granted via RBAC Role and ClusterRole bindings. Secure webhook endpoints using TLS certificates managed by solutions like cert-manager and restrict API access through admission policies and namespace scoping when appropriate. Auditability and traceability rely on integrating with cluster tracing and logging stacks such as Prometheus for metrics, Grafana for dashboards, and Fluentd/Elasticsearch stacks for logs. Operators should follow supply-chain security guidance from projects like In-Toto and Sigstore when publishing CRD-backed controllers and images.

Category:Kubernetes