Understanding Operators in Kubernetes
When it comes to operating Kubernetes, it's essential to grasp the role of operators and how they extend the cluster's capabilities to encompass resources that aren’t natively understood by the system. At the core of Kubernetes’ functionality are built-in controllers, which manage familiar resources like Deployments and Services. But what happens when you need to handle something unique to your environment? Enter the operator.
An operator enhances the Kubernetes ecosystem by bringing custom resources into the fold. Essentially, it adopts the same declarative management style Kubernetes employs for its standard resources but applies it to custom entities that it doesn't recognize out of the box. Operators allow users to manage these external systems in an intuitive way, similar to how one manages all other resources within the cluster.
To help navigate this space, the discussion will be structured into four key sections:
1. Defining what an operator is and how it fits into Kubernetes.
2. Analyzing the anatomy of an operator, exploring its key components.
3. Walking through the practical steps of building an operator from scratch.
4. Examining considerations for deploying operators in production environments.
This structured approach underscores not just the mechanics of operators but also their application and implications within Kubernetes, ultimately aiming for a more seamless integration of external systems.
What Exactly Is an Operator?
At its core, an operator acts to bridge the gap between desired and actual states in a Kubernetes environment. This reconciliation process is fundamental: a controller pulls the necessary information, compares the desired state defined by the user with the observed actual state, and adjusts as needed. Think of it as a constant loop—observe, compare, act—that ensures the system's health and correctness.
The resilience of this loop hinges on its nature; it doesn't rely on perceiving every event at once. It simply requires periodic invocation, which allows it to react to changes no matter when they occur.
Operators extend this mechanism to custom resources. By defining a Custom Resource and constructing a controller to reconcile this new domain, operators afford a consistent method for managing resources that are vital to specific applications or infrastructures.
Clearing the Confusion: Operator, Controller, and CRD
In the Kubernetes vernacular, terms like “operator,” “controller,” and “CRD” often get thrown around interchangeably, leading to some confusion. However, each term serves a distinct role in the Kubernetes architecture:
- **CRD (Custom Resource Definition)**: At a basic level, a CRD is a way to register a new resource type with the Kubernetes API server. It defines how resources of this type are structured, but on its own, it's inactive—merely a template.
- **Controller**: This term broadly refers to any software that runs a reconciliation loop against a resource type. This could include built-in controllers like the Deployment controller, as well as those created for custom applications.
- **Operator**: More specifically, an operator represents a tailored controller or set of controllers that manage the lifecycle of a custom resource. It encapsulates knowledge about that resource’s domain, handling tasks like provisioning, upgrades, and failure recoveries without human intervention.
One key distinction is that while every operator is indeed a controller, not every controller qualifies as an operator. A CRD that isn't paired with an active controller remains a dormant definition, lacking functionality.
Why Operators Are Superior to Helm Charts, CronJobs, and Scripts
You might wonder why go through the effort of constructing an operator when simpler tools like Helm charts or CronJobs are available. Here's the reality: Helm charts are static—they configure and deploy resources once without monitoring ongoing changes. If something goes awry after deployment, Helm won't act unless manually re-invoked.
CronJobs introduce a level of periodicity, yet they still fall short on granularity and state management. Scripts, reliant as they are on manual initiation or CI tools, revert to a reactive model. Again, if state changes between executions, those changes can get overlooked.
What distinguishes an operator is its event-driven, continuous nature. It reacts immediately to any changes in associated custom resources, constantly reconciling the state throughout the resource's lifecycle. This proves invaluable for systems that require constant attention and can evolve beyond their initial specifications.
However, be advised: creating and operating an operator comes with its complexities. An operator is a long-term process that demands robust Role-Based Access Control (RBAC) configurations, clear error management strategies, and efficient monitoring capabilities. If your task merely involves applying a YAML file once, then sure—stick with a Helm chart. But when the requirement is to continuously ensure accuracy over a more complex lifecycle, that's where operators justify their development overhead.
Diving into the Mechanics: The Anatomy of an Operator
In the subsequent sections, we'll delve deeper into the specific components that make the operational loop functional. This involves exploring the anatomy of custom resources, the inner workings of change detection mechanisms, and the comprehensive management required to run an operator effectively.
By unpacking these details, you’ll gain a solid understanding of how to effectively implement and benefit from operators within your Kubernetes configurations.go
func (in *VirtualMachine) DeepCopyObject() runtime.Object {
out := VirtualMachine{
TypeMeta: in.TypeMeta,
ObjectMeta: *in.ObjectMeta.DeepCopy(), // ObjectMeta has its own copy logic
Spec: in.Spec, // Spec has no pointers or slices, so a straightforward copy suffices
Status: in.Status,
}
return &out
}
```
Next up, we have the CRD manifest that familiarizes the API server with our custom resource:
```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: virtualmachines.compute.example.com
spec:
group: compute.example.com
scope: Namespaced
names:
kind: VirtualMachine
listKind: VirtualMachineList
plural: virtualmachines
singular: virtualmachine
shortNames: [vm] # allows shorthand: `kubectl get vm` instead of typing the full name
versions:
- name: v1
served: true
storage: true
subresources:
status: {} # separates status, allowing distinct updates, see Part 2
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [image, cpu, memory]
properties:
image: { type: string }
cpu: { type: integer }
memory: { type: string }
status:
type: object
properties:
phase: { type: string }
id: { type: string }
```
The `subresources.status` section is especially significant, enabling status management as a standalone subresource. This design choice clearly delineates what user clients can manipulate and the areas reserved solely for the controller's operations.
Moving to the `names` block, it serves as the foundation for resolving `kubectl` commands. For instance, when you run `kubectl get virtualmachines`, it works seamlessly thanks to the plural designation, while the `shortNames` property enables the quick `kubectl get vm` command, streamlining interactions akin to `kubectl get po` for Pods.
### Reconciler
The reconciler's duties may seem straightforward at a glance: observe a `VirtualMachine` and ensure a corresponding VM is active in the provider, all while its status mirrors the real-world state. To keep the reconciler readable, we encapsulate the provider's HTTP API through a concise client:
```go
func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var vm computev1.VirtualMachine
if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err) // if the object was deleted, we’re done here
}
if vm.Status.ID == "" {
// No VM exists yet; this is our initial encounter with the object
created, err := r.Provider.Create(ctx, vm.Spec.Image)
if err != nil {
return ctrl.Result{}, err
}
vm.Status.ID = created.ID
vm.Status.Phase = created.Phase
if err := r.Status().Update(ctx, &vm); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // we check back soon to prevent blocking
}
// The VM already exists; we should poll the provider for updates
current, err := r.Provider.Get(ctx, vm.Status.ID)
if err != nil {
return ctrl.Result{}, err
}
vm.Status.Phase = current.Phase
if err := r.Status().Update(ctx, &vm); err != nil {
return ctrl.Result{}, err
}
if current.Phase != "Running" {
return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // keep polling while provisioning
}
return ctrl.Result{}, nil
}
```
Two points deserve emphasis. First, our reconciler accesses the `VirtualMachine` through a registered watch. This feature allows the API server to notify us of any creation or modification, thus initiating the reconcile process effectively.
Second, every section concludes with a write to `vm.Status`, translating provider insights to our resource directly. Notably, Kubernetes never communicates with the provider directly. Awareness of a running VM stems solely from the reconciler documenting it in the status.
### Failure Handling & Retries
Observe that the reconciler intentionally avoids retrying operations within itself. When calls to `r.Provider.Create` or `r.Provider.Get` fail, perhaps due to transient network issues, it simply returns an error. This approach leverages controller-runtime’s built-in mechanism to requeue requests with exponential backoff.
However, not all failures warrant the same treatment. A temporary timeout while contacting the provider is worth retrying. Conversely, if a specific `VirtualMachine` has a `spec.image` deemed unacceptable by the provider, perpetual retries would result in a never-ending loop, wasteful and unproductive.
For now, we’ll hone in on this distinction through status conditions in forthcoming exercises. As it stands, the reconciler has only one failure scenario to tend to, since the mock provider operates without explicitly rejecting requests.
### Finalizer
When a `VirtualMachine` is deleted, Kubernetes would traditionally purge the object but could leave behind an orphaned VM still recognized by the provider. This is where a finalizer comes into play. It acts as a safeguard, instructing Kubernetes: "do not delete this resource until I confirm it's safe to do so."
```go
const vmFinalizer = "compute.example.com/vm-cleanup"
func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var vm computev1.VirtualMachine
if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
if !vm.DeletionTimestamp.IsZero() {
// Handling deletion; conduct cleanup with the provider first
if controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
if vm.Status.ID != "" {
if err := r.Provider.Delete(ctx, vm.Status.ID); err != nil {
return ctrl.Result{}, err
}
}
controllerutil.RemoveFinalizer(&vm, vmFinalizer) // safe to proceed with deletion now
return ctrl.Result{}, r.Update(ctx, &vm)
}
return ctrl.Result{}, nil
}
if !controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
controllerutil.AddFinalizer(&vm, vmFinalizer) // register it before provisioning starts
return r.Update(ctx, &vm) // update the state of the VM first
}
// ... provisioning logic follows
return ctrl.Result{}, nil
}
```
Executing `kubectl delete` on a `VirtualMachine` that has this finalizer does not lead to its immediate removal. Instead, it marks `deletionTimestamp` and waits for additional instructions.
During the next reconciliation call, the reconciler manages deprovisioning of the VM via the provider and subsequently removes the finalizer. At this point, Kubernetes proceeds with the actual deletion of the object. Without this finalizer, there's a risk of cleanup tasks never being executed, which could lead to inconsistencies.
### Predicate
There's a subtle flaw in the reconciler's logic. Each time it calls `r.Status().Update`, that action counts as a change which triggers the reconciliation process anew. While this won't lead to an infinite loop—because the same status is being re-evaluated until it stabilizes—it's still inefficient.
To avoid unnecessary work from these self-initiated reconciliations, utilizing a predicate can filter events that queue reconciliation before reaching the core logic:
```go
func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). // ignores status-only events
Complete(r)
}
```
The `generation` only advances when there's a change to the `spec`. Updates to the status do not alter it. Employing the `GenerationChangedPredicate` ensures we eliminate events that arise solely from status updates, helping us focus on substantive changes or explicitly requeueing requests.
At this point, we return to a state where reconciliation happens only when meaningful changes occur or when invoked explicitly.
### Owned Resources
Recognizing that a standalone `VirtualMachine` in a `Running` state isn’t particularly helpful, we broaden the operator’s functionality to include the creation of a `Secret` that contains the VM's connection details:
```go
func (r *VirtualMachineReconciler) reconcileConnectionSecret(ctx context.Context, vm *computev1.VirtualMachine) error {
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: vm.Name + "-connection",
Namespace: vm.Namespace,
},
StringData: map[string]string{"id": vm.Status.ID},
}
if err := controllerutil.SetControllerReference(vm, secret, r.Scheme); err != nil {
return err // ties the lifecycle of the Secret to the VirtualMachine
}
return r.Patch(ctx, secret, client.Apply, client.ForceOwnership, client.FieldOwner("vmoperator")) // creation or update
}
```
The use of `SetControllerReference` establishes this resource as an **owned resource**, effectively linking it back to the `VirtualMachine`.
This ownership structure provides two significant benefits. First, deleting the `VirtualMachine` automatically cascades to the `Secret`, allowing Kubernetes to perform garbage collection effortlessly without needing a finalizer since both exist within the cluster. Second, if `Owns(&corev1.Secret{})` is added alongside `For(&computev1.VirtualMachine{})` in `SetupWithManager`, any modifications to the `Secret` will trigger the reconciliation of the associated `VirtualMachine`.
By implementing this same technique for a second owned resource—a `Service` that interfaces with the VM—one `VirtualMachine` can manage multiple resource types. This concept of **multiple owned resources** simply involves repeating this procedural pattern for different resource classifications, creating a cohesive structure for resource management.
### Cross-Resource Reconciliation
As it stands, every `VirtualMachine` currently interacts with a singular, hardcoded provider endpoint. For real-world applications, flexibility is key, and most environments house numerous VMs that might share the same endpoint and credentials.
Instead of directly embedding an `endpoint` field within `VirtualMachineSpec`, which would necessitate individual updates across potentially many `VirtualMachine` entries, we can encapsulate the endpoint within its own object. This way, multiple `VirtualMachine` instances can reference that configuration by name, allowing a solitary edit to cascade throughout all related instances.
Consider this second lightweight CRD:
```go
type ProviderConfigSpec struct {
Endpoint string `json:"endpoint"`
}
```
Alongside a `providerRef` field within `VirtualMachineSpec` that points to an instance of this new object by its name. The novelty lies not in the introduction of additional types but in the ramifications of modifying a `ProviderConfig`.
A `VirtualMachine` doesn’t watch `ProviderConfig` directly, and without ownership links, a simple `Owns()` won’t suffice. Instead, we'll implement a watch for this type and map each event to all `VirtualMachine` instances that reference it:
```go
func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
Owns(&corev1.Secret{}).
Owns(&corev1.Service{}).
Watches(
&computev1.ProviderConfig{}, // not owned, so Owns() won’t capture its variations
handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig),
).
Complete(r)
}
func (r *VirtualMachineReconciler) findVirtualMachinesForProviderConfig(ctx context.Context, obj client.Object) []reconcile.Request {
var vms computev1.VirtualMachineList
if err := r.List(ctx, &vms, client.InNamespace(obj.GetNamespace())); err != nil {
return nil
}
var requests []reconcile.Request
for _, vm := range vms.Items {
if vm.Spec.ProviderRef == obj.GetName() { // this filters to only include relevant VMs
requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&vm)})
}
}
return requests
}
```
This approach exemplifies **cross-resource reconciliation**: a change in one resource inadvertently triggering the reconciliation of an unrelated resource, based solely on a positional field rather than traditional ownership.
This aligns with a larger theme prevalent in this project. The `ProviderConfig` could embody credentials and settings for connections to AWS, Azure, or GCP. So, while the mock provider serves as our stand-in, it highlights the importance of boundary management within the architecture of this operator.
### RBAC
All these integrations hinge on the correct permissions being in place. The manifest needs to detail all resources being interacted with: `VirtualMachine` and `ProviderConfig`, along with access to the `VirtualMachine` status subresource and the creation of `Secret` and `Service` objects:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: vmoperator-manager-role
rules:
- apiGroups: ['compute.example.com']
resources: ['virtualmachines', 'providerconfigs']
verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
- apiGroups: ['compute.example.com']
resources: ['virtualmachines/status'] # separate rule because it is a different subresource
verbs: ['get', 'update', 'patch']
- apiGroups: ['']
resources: ['secrets', 'services']
verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vmoperator-manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: vmoperator-manager-role
subjects:
- kind: ServiceAccount
name: vmoperator-controller-manager
namespace: vmoperator-system
```
**Note:** Although VMOperator governs resources externally via a custom-built HTTP client, this isn’t a unique method. Other tools like [Crossplane](https://www.crossplane.io/), [AWS Controllers for Kubernetes](https://aws-controllers-k8s.github.io/community/), and [Cluster API](https://cluster-api.sigs.k8s.io/) similarly handle reconciliation of external states through custom resources.
**Another note:** We're intentionally narrowing VMOperator's initial scope. Future capabilities could include features like resizing VMs, providing snapshots, and supporting multiple real providers behind `ProviderConfig`, progressing naturally once the core functionality is firmed up.
### Part 4: Production & Deployment
With VMOperator successfully working, let's transition into packaging, deploying, and scaling it for production environments.
### Packaging & Deployment
Thus far, the operator has run as a local binary, executed with `go run` against any cluster that `kubectl` is pointed toward.
Now, we need a `Deployment` that references an image instead, which leads us to a multi-stage `Dockerfile`: the first stage builds the application, while a second, minimal image serves to execute it:
```dockerfile
FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /vmoperator ./cmd/manager
FROM gcr.io/distroless/static-debian12
COPY --from=build /vmoperator /vmoperator
USER 65532:65532 # nonroot user that aligns with the Deployment's security context
ENTRYPOINT ["/vmoperator"]
```
The build stage contains the entirety of the Go toolchain and all source files, unnecessary for distribution once compiled. The final image, containing just the binary, enhances container security by minimizing potential attack surfaces—there’s no shell for access even before `runAsNonRoot` is in play.
```bash
docker build -t registry.example.com/vmoperator:v0.1.0 .
docker push registry.example.com/vmoperator:v0.1.0
```
This image will be referenced in the `Deployment` manifest found under `config/manager/`. With it pushed to a location accessible by the cluster, we can now apply the rest of the necessary manifests: CRDs, RBAC permissions, and the operator's `Deployment`, alongside a `Deployment` and `Service` for the mock provider. This marks a shift away from personal machines as execution hosts:
```bash
kubectl apply -f config/crd/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/
```
Prioritizing the installation of CRDs is essential. If the operator’s `Deployment` starts without these definitions, it will crash-loop upon encountering an unrecognized resource type that the API server doesn’t recognize yet.
Adjusting schemas is something our managed manifests will bring to the forefront. Adding an attribute to `VirtualMachineSpec` is straightforward, but overhauls or renaming existing fields present challenges; every saved `VirtualMachine` has been serialized from the previous format.
The `versions` field in the CRD cleverly accommodates this, allowing multiple versions to be simultaneously `served`. One version is marked as the `storage` version, meaning it's the format persisted in the database. Should clients request a different version, a conversion webhook translates requests accordingly.
At present, VMOperator only has `v1` as its existing version, but this foresight in allowing for potential adjustments is why the `versions` field was structured to permit a list from the outset.
None of this supersedes the manual execution of `kubectl apply`. Establishing a CI pipeline to build, push the operator's image, and automate the application of manifests upon merging code to the main branch represents a sensible next step. This CI/CD development is commonplace, devoid of any operator-specific intricacies once the manifests are committed to a version control system.
### Performance & Resilience
By default, a controller processes only one reconciliation request at a time. While that works fine during initial testing phases, handling hundreds of `VirtualMachine` objects means many are left in the work queue, idly waiting for their turn, despite no blocking concerns when reconciling others.
To address this issue, the `MaxConcurrentReconciles` setting can be adjusted:
Final Thoughts
As we wrap this guide, it’s apparent that crafting a Kubernetes operator is more than just building a service; it’s about navigating the complexities of asynchronous operations, state management, and security. Throughout these sections, we’ve dissected important aspects of designing a VirtualMachine operator and highlighted key takeaways that extend well beyond the specific task at hand.
Here’s the thing: while we’ve delved into the technical nuts and bolts—like leader election, caching mechanisms, and observability metrics—the essential concepts are universally applicable across various operator scenarios. Whether you're orchestrating VMs, databases, or any other Kubernetes resource, the lessons learned here about handling external dependencies and maintaining robust state can be a guiding light.
The observed asymmetry in operational costs between in-cluster reads and external HTTP calls poses real challenges. Sure, Kubernetes helps facilitate in-memory caching for quick lookups, but the uncoordinated HTTP requests to external services can easily escalate into overwhelmed systems under high load. If you’re pushing to scale operations, considering additional layers like rate limiting becomes not just beneficial, but necessary.
Moreover, security is paramount. Tightening RBAC policies and safeguarding sensitive data—like API keys—must be prioritized in any Kubernetes operator setup. Without these precautions, you risk exposing critical information, creating vulnerabilities in the system.
Looking ahead, there are endless possibilities for further improvement. Future iterations of this operator could integrate more in-depth metrics, deeper observability features, or even adopt a more sophisticated caching strategy to enhance performance.
All of this points to one undeniable truth: the journey doesn’t stop here. The foundational concepts explored in this guide will aid anyone venturing deeper into Kubernetes operator development. For those formulating their own, remember the challenging domains that can crop up. Prepare to address them head-on, and your efforts will not only yield immediate results but will also build a more resilient and secure operator capable of evolving as needs change.
In a nutshell, the next adventure starts now. Review the provided resources, consider your design choices, and take the next step with confidence. The Kubernetes community continues to grow and innovate, and there’s a steadfast place for your contributions within it.
