Strimzi Drain Cleaner

Strimzi Drain Cleaner coordinates safe rolling restarts of Strimzi-managed Kafka Pods during node drains by intercepting eviction requests and delegating Pod management to the Strimzi Cluster Operator.

Strimzi-managed Kafka Pods may be evicted when Kubernetes worker nodes are cordoned and drained. This happens during cluster maintenance, Kubernetes upgrades, or pod rescheduling. Platform teams typically initiate draining with kubectl drain or cluster maintenance tooling.

Without Strimzi Drain Cleaner, Kubernetes handles eviction according to its own rules and PodDisruptionBudget settings. The Strimzi Cluster Operator has no chance to coordinate when and how Kafka Pods restart. Strimzi Drain Cleaner changes this behavior. It intercepts eviction requests and hands control to the Strimzi Cluster Operator so Kafka Pods are rolled in a controlled sequence instead of being evicted directly by Kubernetes.

After installation, Strimzi Drain Cleaner runs continuously in the background. It requires no day-to-day operator action and responds automatically when Pod evictions occur. You do not trigger Strimzi Drain Cleaner as part of routine Kafka operations. Platform teams drain nodes as part of maintenance, and Strimzi Drain Cleaner handles matching Strimzi Kafka Pods on those nodes.

How it works

Strimzi Drain Cleaner registers a validating admission webhook that receives pods/eviction CREATE requests from the Kubernetes API. When something tries to evict a matching Strimzi-managed Kafka Pod, the webhook adds the strimzi.io/manual-rolling-update annotation to that Pod. With default chart values, the webhook then denies the eviction request.

Denying the eviction request prevents Kubernetes from restarting the Pod on its own. The Strimzi Cluster Operator detects the annotation during reconciliation and performs a controlled rolling update of the Pod. The Cluster Operator rolls Pods one at a time using its own algorithms. This helps keep partition replicas in sync and Kafka available while worker nodes are drained.

When a worker node is drained, the drain command may fail or report errors when the webhook denies eviction of Kafka Pods on that node. This is expected. Strimzi Drain Cleaner and the Strimzi Cluster Operator handle those Pods automatically. The Cluster Operator rolls each affected Pod off the node one at a time. The node stays cordoned during this process.

Whoever runs the drain command, typically the platform team using kubectl drain or cluster maintenance tooling, may need to run it again on the same cordoned node after rolled Pods have left it. If multiple affected Pods were on the node, more than one retry may be needed as the Cluster Operator rolls them in sequence. The drain completes when no matching Kafka Pods remain on the node.

Affected pods

Strimzi Drain Cleaner acts on Strimzi-managed Kafka Pods. This includes broker Pods and KRaft controller Pods deployed by the Strimzi Cluster Operator.

The webhook matches Pods labeled with strimzi.io/kind=Kafka and a strimzi.io/name label whose value ends with -kafka. These are the Kafka broker and controller Pods created from your Kafka and KafkaNodePool resources.

Strimzi Drain Cleaner does not act on other Strimzi workloads, such as Kafka Connect, the Entity Operator, or Kafka Exporter Pods. It also does not manage Pods that are not deployed and labeled by Strimzi.

Default webhook configuration

With default Helm chart values, Strimzi Drain Cleaner registers a validating admission webhook automatically. No manual TLS configuration is required when cert-manager is installed.

Strimzi Drain Cleaner installs a cluster-scoped ValidatingWebhookConfiguration resource named strimzi-drain-cleaner. The resource registers the strimzi-drain-cleaner.strimzi.io webhook with the Kubernetes API.

The webhook intercepts pods/eviction CREATE operations in namespaced scope. When a matching eviction request is received, the Kubernetes API forwards the request to the Strimzi Drain Cleaner Service on port 443 at the /drainer path.

The following example shows the key webhook settings deployed by the Strimzi Drain Cleaner Helm chart with default values:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: strimzi-drain-cleaner
webhooks:
  - name: strimzi-drain-cleaner.strimzi.io
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods/eviction"]
        scope: Namespaced
    clientConfig:
      service:
        namespace: strimzi-drain-cleaner
        name: strimzi-drain-cleaner
        path: /drainer
        port: 443
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Ignore
    timeoutSeconds: 5

With default chart values, cert-manager creates TLS certificates for the webhook and injects the CA bundle into the ValidatingWebhookConfiguration automatically. You do not need to configure TLS manually after installation.

Testing Strimzi Drain Cleaner with a node drain

Walk through a node drain to see how Strimzi Drain Cleaner annotates Strimzi-managed Kafka Pods and how the Strimzi Cluster Operator performs a rolling update.

After installation, Strimzi Drain Cleaner requires no day-to-day operator action. It responds automatically when Pod evictions occur.

Use this procedure once, or when validating a new installation, to understand the eviction, annotation, and rolling update sequence. In production, platform teams drain nodes as part of maintenance. Strimzi Drain Cleaner handles Strimzi-managed Kafka Pods automatically. You do not need to repeat these steps each time a node is drained.

Strimzi Drain Cleaner is installed. For installation steps, see Installing Strimzi Drain Cleaner with Helm.
A Kafka cluster is deployed with appropriate replication settings, including a replication factor greater than 1 and min.insync.replicas lower than the replication factor. For an example, see Deploying a Kafka cluster.
If you use anti-affinity rules for Kafka Pods, consider adding a spare worker node to your cluster. Spare capacity helps ensure that rolled Pods can be rescheduled while a node is drained.
Kafka Pods must use a StorageClass backed by network-attached storage, such as cloud block storage or a distributed storage solution. If Kafka Pods use node-local storage, their PersistentVolumes are bound to the specific node where the data resides. In that case, evicted Pods cannot be rescheduled on another node and the drain cannot complete.

Drain a Kubernetes worker node that hosts Strimzi-managed Kafka Pods.
```
kubectl get nodes
```
```
kubectl drain [***NODE NAME***] --delete-emptydir-data --ignore-daemonsets --timeout=6000s --force
```
The first drain attempt may fail or report that the validating admission webhook denied eviction of Kafka Pods. This is expected behavior.

Check the Strimzi Drain Cleaner log for eviction webhook and annotation events.

Look for log lines similar to the following examples:

INFO ... Received eviction webhook for Pod my-cluster-kafka-0 in namespace my-project
INFO ... Pod my-cluster-kafka-0 in namespace my-project will be annotated for restart
INFO ... Pod my-cluster-kafka-0 in namespace my-project found and annotated for restart

Check the Strimzi Cluster Operator log for rolling update reconciliation events.

Look for log lines similar to the following examples:

INFO  PodOperator:68 - Reconciliation #13(timer) Kafka(my-project/my-cluster): Rolling Pod my-cluster-kafka-0
INFO  AbstractOperator:500 - Reconciliation #13(timer) Kafka(my-project/my-cluster): reconciled

Retry the drain command on the same node after affected Pods have been rolled off the node.
```
kubectl drain [***NODE NAME***] --delete-emptydir-data --ignore-daemonsets --timeout=6000s --force
```
Wait until the Strimzi Cluster Operator has finished rolling affected Pods off the node, then run the drain command again on the same cordoned node. If the node hosted multiple affected Pods, repeat this step until the drain completes.