Upgrade FreeIPA

To ensure that your FreeIPA nodes are running with the latest patches, you should periodically upgrade your FreeIPA cluster.

FreeIPA is the core component of an environment that runs on one or more instances.

To ensure that your FreeIPA nodes are running with the latest patches, you should periodically upgrade your FreeIPA cluster. CDP currently allows you to upgrade all FreeIPA clusters, updating OS-level security patches on the cluster nodes. The upgrade process includes launching entirely new instances with the new OS image.

The upgrade process is a rolling upgrade consisting of the following steps:

  1. A new instance is provisioned from a newer image.

  2. CDP validates that the new node is reachable, all services are up, and that replication works.

  3. The old instance is removed.

  4. If there is more than one instance in the cluster, the previous 3 steps are repeated for the second instance only after the first instance has been successfully upgraded.

Your environment remains functional during the upgrade process, with an exception of a short (one minute or less) downtime. During and after the FreeIPA upgrade, the status of the environment changes to Update Started as the environment is refreshing all the clusters to the new FreeIPA configurations (IP addresses for DNS resolution). Refreshing the clusters in the environment can take some time. You can minimize the time of the FreeIPA upgrade by not initiating any cluster related activities such as scaling and provisioning.

During FreeIPA provisioning a full backup is performed and there are periodic incremental backups afterwards. These backups can be used in case a recovery is needed.

Use either the CDP web interface or CDP CLI to initiate an upgrade. The upgrade process takes about one hour; The duration varies depending on the cloud provider and the number of nodes.

Performing a FreeIPA upgrade

You can trigger the FreeIPA upgrade via CDP web interface or CDP CLI.

Prerequisites

  • If the upgrade involves upgrading from CentOS to RHEL, review the Prerequisites for upgrading from CentOS to RHEL.

  • On Azure, before you run the FreeIPA upgrade, make sure that the resource group has neither a DELETE nor a READ-ONLY lock applied.

  • Before performing the FreeIPA upgrade for Data Hub clusters where autoscaling is enabled, you must disable autoscaling and start all compute nodes to ensure the healthy state of Data Hub clusters. After upgrading FreeIPA, autoscaling can be enabled again. For more information, see the Autoscaling must be stopped before performing FreeIPA upgrade description in the Known issue.

Required roles: EnvironmentAdmin or Owner of the environment

Steps

  1. Log in to the CDP web interface.

  2. From the navigation pane, select Environments.

  3. Click on your environment.

  4. Click on the FreeIPA > Upgrade tab.

  5. From the Target image dropdown, select a target image. If you would like to upgrade from CentOS to RHEL, ensure that you select a RHEL image. Once you have reviewed the information related to your current base image and the target base image that will be used for provisioning new instances during the upgrade process. Once you’ve reviewed the information, click on Upgrade:
  6. The upgrade process starts. During the upgrade, FreeIPA status switches to Update in progress, but the environment remains available. A new instance is provisioned, then the old instance is removed. If more than one instance is present, the process is repeated for the additional instances.
  7. Once the upgrade is completed, the FreeIPA status changes to Running.
Use the following command to trigger FreeIPA upgrade (including a CentOS to RHEL upgrade):
cdp environments upgrade-freeipa --environment-name <ENVIRONMENT_NAME_OR_CRN>
For example:
cdp environments upgrade-freeipa --environment-name my-env

The command will return a JSON including target and original image information and the operation ID.

{
    "targetImage": {
        "catalog": "https://gist.githubusercontent.com/lacikaaa/c096c999ade874a60bf15be897220eed/raw/e30ed12b14c77175b82ceb9caf26500ab1340f26/freeipa-test-catalog.json",
        "id": "9c1c8959-86a7-4b7d-af5a-be252f8b395d",
        "os": "centos7",
        "imageName": "ami-0dfafedeed3a4474f",
        "date": "2021-06-10"
    },
    "originalImage": {
        "catalog": "https://gist.githubusercontent.com/lacikaaa/c096c999ade874a60bf15be897220eed/raw/e30ed12b14c77175b82ceb9caf26500ab1340f26/freeipa-test-catalog.json",
        "id": "0b73e149-6e22-4667-acc5-1d0ba82f8245",
        "os": "centos7",
        "imageName": "ami-030e1d907cfa5ca33",
        "date": "2021-06-04"
    },
    "operationId": "28432209-a4ff-4aea-a8ec-4604d67cdb7c"
}
Use the following command to track the progress of the upgrade process:
cdp environments get-repair-freeipa-status --operation-id <OPERATION-ID-FROM-UPGRADE-FREEIPA-COMMAND-OUTPUT>
The operation ID can be obtained from the output of the upgrade-freeipa command.

What to do next

After upgrading FreeIPA, autoscaling can be enabled again. For more information, see the Autoscaling must be stopped before performing FreeIPA upgrade description in the Known issue.

In rare cases, the FreeIPA upgrade process might fail. In such cases, you should trigger a retry of the FreeIPA upgrade.

Retry a FreeIPA upgrade

In rare cases, the FreeIPA upgrade process might fail. In such cases, you should trigger a retry of the FreeIPA upgrade.

Steps

If you are performing the upgrade via CDP web interface, you can find the retry option on the same FreeIPA tab where the upgrade option is located:

The Retry option is grayed out if there is no reason to trigger the repair and is only enabled when FreeIPA upgrade encounters problems.

If you would like to trigger the repair via CDP CLI, use the following command:
cdp environments retry-freeipa –environment <MY-ENVIRONMENT-NAME-OR-CRN>
The command returns an operation ID that you can pass in the following command to track the status of the repair:
cdp environments get-repair-freeipa-status --operation-id <OPERATION-ID-FROM-UPGRADE-FREEIPA-COMMAND-OUTPUT>

Troubleshooting

Issue Workaround
After performing a FreeIPA upgrade, CDW VMs are broken due to the new FreeIPA server instances' IP addresses not being reflected in CDW. Manually restart every cdp-coredns-updater pod running in your CDW Kubernetes cluster namespace.
After performing a FreeIPA upgrade, CML workspaces are broken because the IP addresses of the new FreeIPA server instances are not reflected in CML. Restart the cdp-coredns-updater pod by running kubectl delete -n kube-system $(kubectl get pods -n kube-system -l app.kubernetes.io/name=cdp-coredns-updater --no-headers -o name) against your CML kubernetes cluster as soon as the FreeIPA upgrade has completed.