Upgrading and rolling back Cloudera Streams Messaging - Kubernetes Operator

Learn about upgrading and rolling back Cloudera Streams Messaging - Kubernetes Operator.

Upgrades🔗

Upgrading Cloudera Streams Messaging - Kubernetes Operator consists of upgrading Strimzi, Kafka, and Kafka Connect.

Upgrading Strimzi involves upgrading the Strimzi Custom Resource Definitions (CRD) and upgrading the Strimzi Cluster Operator using Helm.

Upgrading Kafka and Kafka Connect involves updating your Kafka and KafkaConnect resources so that they specify the latest supported Kafka version. Upgrading Kafka and Kafka Connect is done following a Strimzi upgrade. Upgrading Kafka and Kafka Connect is:

Strongly recommended by Cloudera after every Strimzi upgrade.
Mandatory if you are upgrading from a maintenance version or if you are using a custom Kafka image.

Upgrading Strimzi might affect the Kafka and KafkaConnect resources in the cluster. All Kafka and Kafka Connect clusters that specify the version of the cluster but not the image are restarted when you upgrade Strimzi. This is due to the fact that the default image of all versions changes with the Strimzi upgrade, triggering a restart. This restart is safe, that is, all healthy topics with a proper replication factor and minimum ISR are kept online during the restart.

Rollbacks🔗

A rollback involves rolling back Kafka and Kafka Connect as well as Strimzi to a previous version.

A rollback is only possible if the Kafka version you are upgrading from is still supported by the Cloudera Streams Messaging - Kubernetes Operator version you are upgrading to. If the Kafka version is not supported, upgrading is possible, but rollback is not. If you are upgrading to a version that does not support your current Kafka version and you want to have the ability to roll back, you might need to carry out multiple upgrades.

Upgrading Cloudera Streams Messaging - Kubernetes Operator🔗

To upgrade Cloudera Streams Messaging - Kubernetes Operator, you upgrade Strimzi as well as your Kafka and Kafka Connect clusters.

Ensure that your Kubernetes environment meets requirements listed in System requirements.
Ensure that you have access to your Cloudera credentials (username and password). Credentials are required to access the Cloudera Archive and Cloudera Docker registry where upgrade artifacts are hosted.
If you are upgrading from a maintenance version, check that the bug fixes provided in the maintenance version are available in the newer Kafka supported by Cloudera Streams Messaging - Kubernetes Operator. If certain fixes are not available, be aware that upgrading will result in regressed functionality.
If you are using a custom Kafka image, build new custom images that are based on the Kafka images shipped with the Cloudera Streams Messaging - Kubernetes Operator version you are upgrading to.

Cloudera Streams Messaging - Kubernetes Operator ships with two Kafka images, one for the latest supported version of Kafka, and one for the previously supported version of Kafka. Build new custom images based on both.

The custom image based on the latest supported version is required for the upgrade. The custom image based on the previously supported version is required for rollbacks.
note
If you have Ranger integration set up, you are using a custom image. Building new custom images is required.
The following steps instruct you to set inter.broker.protocol.version during the upgrade to keep Kafka in backward compatible state during the upgrade. This is only necessary if the protocol version differs between your current and new versions.

If there is no change in the protocol version, you also do not finalize the upgrade. This means that for these types of upgrades, a rollback is possible even after you completed the upgrade.

You can find the Kafka protocol version in Component versions.

Update the Strimzi CRDs.
You do this by replacing the currently installed CRDs with the new version. The CRDs are published as a single YAML on the Cloudera Archive in /p/csm-operator/1.3/install.
```
curl -s --user [***USERNAME***] \
  https://archive.cloudera.com/p/csm-operator/1.3.0/install/strimzi-crds-0.45.0.1.3.0-b52.yaml \
| kubectl replace --filename -
```
Enter your Cloudera password when prompted.
If you are upgrading from a maintenance version or are using a custom Kafka image, pause reconciliation for all your Kafka clusters.
important
During the period when reconciliation is paused, your cluster is not managed by Strimzi. Even if a pod crashes, it is not restarted. Ensure that you prepare for this time window to avoid accidental issues.
You pause reconciliation by setting the strimzi.io/pause-reconciliation annotation to true in the Kafka resource.
```
kubectl annotate kafka [***KAFKA CLUSTER NAME***] /
  --namespace [***KAFKA NAMESPACE***] /
   strimzi.io/pause-reconciliation="true"
```
Pausing reconciliation ensures that the Kafka cluster does not get automatically updated during the upgrade of Strimzi. Reconciliation is only paused temporarily. You re-enable it after the Strimzi upgrade. Ensure that you add the annotation to all your Kafka clusters.
If you are upgrading from a maintenance version, or you are using a custom Kafka image, pause reconciliation for all your Kafka Connect clusters.

important
During the period when reconciliation is paused, your cluster is not managed by Strimzi. Even if a pod crashes, it is not restarted. Ensure that you prepare for this time window to avoid accidental issues.

You pause reconciliation by setting the strimzi.io/pause-reconciliation annotation to true in the KafkaConnect resource.
```
kubectl annotate kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] /
  --namespace [***KAFKA CONNECT NAMESPACE***] /
   strimzi.io/pause-reconciliation="true"
```
Pausing reconciliation ensures that the Kafka Connect cluster does not get automatically updated during the upgrade of Strimzi. Reconciliation is only paused temporarily. You reenable it after the Strimzi upgrade. Ensure that you add the annotation to all your Kafka Connect clusters.
Log in to the Cloudera Docker registry with helm.
```
helm registry login container.repository.cloudera.com
```
Enter your Cloudera credentials when prompted.
Upgrade Strimzi using helm upgrade.
This step upgrades the Strimzi Cluster Operator. Under the hood, the Strimzi Cluster Operator deployment is updated by changing the image used by the Strimzi Cluster Operator.
```
helm upgrade strimzi-cluster-operator \
  --namespace [***STRIMZI CLUSTER OPERATOR NAMESPACE***] \
  --atomic \
  oci://container.repository.cloudera.com/cloudera-helm/csm-operator/strimzi-kafka-operator \
--version 1.3.0-b52
```
- The string strimzi-cluster-operator is the Helm release name of the chart installation. This is an arbitrary, user-defined name. Replace this string if you used a different name when you installed Strimzi.
- The --atomic option makes the command wait for the upgrade to complete or roll back if the upgrade fails.
note
Usually, you do not need to set new Helm chart properties during upgrade, the properties that you configured during installation are preserved. However, If you set even a single property using the --set options in your helm upgrade command, properties set previously might be ignored. If you decide to set properties during the upgrade, ensure that you do one of the following.
- Set all your properties (including the ones that were set during installation) during upgrade. This ensures that all required properties are set explicitly.
- Set the properties you want to update and use the --reset-then-reuse-values option. Using this option preserves previously set properties. This option is only available in the more recent versions of Helm.
Verify that the Strimzi upgrade is successful.
To do this, check that the Strimzi Cluster Operator deployment is in a healthy state.
```
kubectl get deployments strimzi-cluster-operator \
  --namespace [***STRIMZI CLUSTER OPERATOR NAMESPACE***]
```
Upgrade Kafka.
To do this, update your Kafka resources for each cluster.
```
kubectl edit kafka [***KAFKA CLUSTER NAME***] \
  --namespace [***KAFKA NAMESPACE***]
```
1. Set spec.kafka.version to the latest version.
2. If you are using a custom image, set your new custom image that corresponds with the new Kafka version in spec.kafka.image.
3. Add the old inter.broker.protocol.version in spec.kafka.config.
```
#...
kind: Kafka
spec:
  kafka:
    version: 3.9.0.1.3
    config:
      inter.broker.protocol.version: “3.8”
```
  Specifying inter.broker.protocol.version keeps Kafka in a backward-compatible state following the upgrade.
4. Save the changes made to the Kafka resource.
5. If you paused reconciliation of the Kafka cluster, remove the strimzi.io/pause-reconciliation annotation.
  This re-enables reconciliation for the cluster.
```
kubectl annotate kafka [***KAFKA CLUSTER NAME***] /
  --namespace [***KAFKA NAMESPACE***] /
  strimzi.io/pause-reconciliation-
```
Verify that the Kafka upgrade is successful.
Upgrade is successful once the Kafka resources are in a ready state.
```
kubectl get kafkas \
  --namespace [***KAFKA NAMESPACE***] \
  --watch
```
Upgrade Kafka Connect.
To do this, update your KafkaConnect resources for each cluster.
```
kubectl edit kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] \
  --namespace [***KAFKA CONNECT NAMESPACE***]
```
1. Set spec.version to the latest version.
2. If you are using a custom image, set your new custom image that corresponds with the new Kafka version in spec.image.
3. Save the changes made to the KafkaConnect resource.
4. If you paused reconciliation of the Kafka Connect cluster, remove the strimzi.io/pause-reconciliation annotation.
  This re-enables reconciliation for the cluster.
```
kubectl annotate kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] /
  --namespace [***KAFKA CONNECT NAMESPACE***] /
  strimzi.io/pause-reconciliation-
```
Verify that the Kafka Connect upgrade is successful.
Upgrade is successful once the KafkaConnect resources are in a ready state.
```
kubectl get kafkaconnects \
  --namespace [***KAFKA CONNECT NAMESPACE***] \
  --watch
```
Finalize the Kafka upgrade by removing the old inter.broker.protocol.version from your Kafka resources.
After this step, rollback is no longer possible.

Rolling back Cloudera Streams Messaging - Kubernetes Operator🔗

To roll back Cloudera Streams Messaging - Kubernetes Operator to a previous version, you roll back your Kafka and Kafka Connect clusters as well as Strimzi.

Rollbacks are only possible if:
- The Kafka version you upgraded from is still supported by the Cloudera Streams Messaging - Kubernetes Operator version that you upgraded to.
- The Kafka upgrade is not finalized. A Kafka upgrade is finalized if inter.broker.protocol.version for the Kafka cluster is set to the current protocol version. That is, the protocol version matches the version of Kafka.
If you are using a custom Kafka image:
- Ensure that you have a custom Kafka image that is based on the latest base image with the previous Kafka version. This image is only set and used temporarily during the rollback.
  
  For example, Cloudera Streams Messaging - Kubernetes Operator 1.3 ships two Kafka base images. One for Kafka 3.9.0.1.3 (latest/current) and one for Kafka 3.8.0.1.2 (previous). In order to rollback, you need a custom image based on the 3.8.0.1.2 image shipped with 1.3.
  
  The base image used to build the custom image must be an image shipped in the Cloudera Streams Messaging - Kubernetes Operator version you upgraded to. Even though earlier Cloudera Streams Messaging - Kubernetes Operator versions might have shipped the same Kafka version, you cannot use a base image from a previous Cloudera Streams Messaging - Kubernetes Operator release even if the Kafka version is the same.
- Ensure that you have access to the original custom image that you used before you upgraded. You will be rolling back your clusters to use this image.
- If you are rolling back to a maintenance version, it can happen that during the rollback your cluster temporarily runs with a Kafka image that does not have all fixes included in the maintenance version.
  
  This is because a rollback is done in two stages. First, you roll back Kafka and Kafka Connect, then you roll back Strimzi. After rolling back Kafka or Kafka Connect your clusters are automatically restarted and use a Kafka image that includes the previous version of Kafka. However, that image might not contain all fixes that are in the maintenance version you are rolling back to. This is only temporary, once you rollback Strimzi, your cluster will fully revert to using the initial maintenance version you upgraded from.

Roll back Kafka.
To do this, edit your Kafka resources.
```
kubectl edit kafka [***KAFKA CLUSTER NAME***] \
  --namespace [***KAFKA NAMESPACE***]
```
1. Set spec.kafka.version to the previous version.
2. If you are using a custom Kafka image, set spec.kafka.image to a custom Kafka image that is based on the latest base image with the previous Kafka version.
  
  The image you set here is not the original custom image that was used before you completed the upgrade. Strimzi at this stage is still upgraded and is unable to manage your original image. Because of this, you must specify a custom Kafka image that is based on the latest base image with the previous Kafka version.
3. Save the changes made to the Kafka resource.
After the changes are saved, the Kafka cluster is restarted in a rolling fashion.
Roll back Kafka Connect.
To do this, edit your KafkaConnect resources.
```
kubectl edit kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] \
  --namespace [***KAFKA CONNECT NAMESPACE***]
```
1. Set spec.version to the previous version.
2. If you are using a custom Kafka image, set spec.image to a custom Kafka image that is based on the latest base image with the previous Kafka version.
  
  The image you set here is not the original custom image that was used before you completed the upgrade. Strimzi at this stage is still upgraded and is unable to manage your original image. Because of this, you must specify a custom Kafka image that is based on the latest base image with the previous Kafka version
3. Save the changes made to the KafkaConnect resource.
After the changes are saved, the Kafka Connect cluster is restarted in a rolling fashion.
If you are using a custom Kafka image, pause reconciliation of the Kafka clusters.
To do this, add the strimzi.io/pause-reconciliation="true" annotation to your Kafka resources.
```
kubectl annotate kafka [***KAFKA CLUSTER NAME***] /
  --namespace [***KAFKA NAMESPACE***] /
   strimzi.io/pause-reconciliation="true"
```
If you are using a custom Kafka image, pause reconciliation of the Kafka Connect clusters.
To do this, add the strimzi.io/pause-reconciliation="true" annotation to your KafkaConnect resources.
```
kubectl annotate kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] /
  --namespace [***KAFKA CONNECT NAMESPACE***] /
   strimzi.io/pause-reconciliation="true"
```
Roll back Strimzi with helm.
```
helm rollback strimzi-cluster-operator \
  --namespace [***STRIMZI CLUSTER OPERATOR NAMESPACE***]
```
note
If you did not pause reconciliation, both Kafka and Kafka Connect clusters managed by the Strimzi Cluster Operator are restarted in a rolling fashion when you roll back Strimzi.
Verify that the Strimzi rollback is successful.
To do this, check that the Strimzi Cluster Operator deployment is in a healthy state.
```
kubectl get deployments strimzi-cluster-operator \
  --namespace [***STRIMZI CLUSTER OPERATOR NAMESPACE***]
```
If you are using a custom Kafka image, set spec.kafka.image in your Kafka resources to the original custom Kafka image that was in use before you completed the upgrade.
If you are using a custom Kafka image, set spec.image in your KafkaConnect resources to the original custom Kafka image that was in use before you completed the upgrade.
If you paused reconciliation for Kafka clusters, remove the strimzi.io/pause-reconciliation annotation from your Kafka resources.
This re-enables reconciliation for the cluster.
```
kubectl annotate kafka [***KAFKA CLUSTER NAME***] /
  --namespace [***KAFKA NAMESPACE***] /
   strimzi.io/pause-reconciliation-
```
If you paused reconciliation for Kafka Connect clusters, remove the strimzi.io/pause-reconciliation annotation from your KafkaConnect resources.
This re-enables reconciliation for the cluster.
```
kubectl annotate kafkaconnect [***KAFKA CONNECT CLUSTER NAME***] /
  --namespace [***KAFKA CONNECT NAMESPACE***] /
   strimzi.io/pause-reconciliation-
```
Verify that both Kafka and Kafka Connect rollbacks are successful.
Rollback is successful once your Kafka and KafkaConnect resources are in a ready state.
```
kubectl get kafkas \
  --namespace [***KAFKA NAMESPACE***] \
  --watch
```
```
kubectl get kafkaconnects \
  --namespace [***KAFKA CONNECT NAMESPACE***] \
  --watch
```