Updating instance metadata to IMDSv2

Cloudera can use IMDSv2 or IMDSv1 for accessing EC2 instance metadata from a running instance.

Cloudera currently uses IMDSv2 for accessing EC2 instance metadata on all newly created Data Lakes, FreeIPA clusters, and Cloudera Data Hub clusters, as long as an IMDSv2-compatible image is used. Prior to Cloudera supporting IMDSv2, Data Lakes, FreeIPA clusters, and Cloudera Data Hub clusters used IMDSv1; These clusters created with IMDSv1 can now be updated to IMDSv2 as long as an IMDSv2-compatible image was used to create the cluster.

Update an existing cluster to IMDSv2

You can update an existing Data Lake, FreeIPA cluster and Cloudera Data Hub cluster that is currently using IMDSv1 to IMDSv2. This is a zero downtime operation and does not disrupt any existing processes or jobs

Prerequisites

The following prerequisites needs to be met:

Image must be compatible with IMDSv2
You can only update to IMDSv2 if the image used for creating the cluster is compatible with IMDSv2. If the image is not compatible or if a cluster is already using IMDSv2, the Update to IMDSv2 button is grayed out:

If you would like to check image compatibility manually, see Checking if cluster image is compatible with IMDSv2.
You may need to update recipes.
If you are using recipes, you need to update them first to ensure that they are compatible with IMDSv2. See:
For a quick check, you can search for usage of "169.254.169.254" IP in the recipe content (as this is the IP pointing to the AWS IMDS). You can find an example in the Retrieve instance metadata documentation. You may also want to review How Instance Metadata Service Version 2 works.

Required roles: EnvironmentAdmin or Owner of the environment

Steps

CDP UI
CDP CLI

In the Cloudera Management Console, navigate to Data Lake, FreeIPA or Data Hub details.
Navigate to Nodes.
The option to Update to IMDSv2 is available as follows:
The update should happen within a few seconds. You can track the status in event history.

If you would like to verify that the update happened correctly, navigate to one of the EC2 instances in your AWS console.

Once the update is complete, the Update to IMDSv2 button is grayed out.

Steps CLI

Use the following commands to update a cluster from IMDSv1 ro IMDSv2:

Data Lake:

cdp datalake update-to-aws-imds-v2 --crn <SPECIFY_CRN>

FreeIPA:

cdp environments update-freeipa-to-aws-imds-v2 --environment-crn <SPECIFY_CRN>

Cloudera Data Hub:

cdp datahub update-to-aws-imds-v2 --crn <SPECIFY_CRN>

The update should happen within a few seconds.

Checking if cluster image is compatible with IMDSv2

To check if the image is compatible, follow these steps:

Steps

CDP UI
CDP CLI

In the Cloudera Management Console, navigate to your cluster.
Navigate to the Image details tab.
Click on the image ID.
Scroll down to Package Versions details of the image:
Package versions include a variable called “imds”. Find this variable and ensure that its value is “v2

You can use the describe CDP CLI command of the given cluster (FreeIPA, Data Lake, Cloudera Data Hub. The commands are as follows:
- cdp datahub describe-cluster
- cdp datalake describe-datalake
- cdp environment describe-environment

The response should contain image and image catalog JSON details.

Here is an example from image catalog where an image is compatible (note the "package-versions" section, which includes “imds” version. If the value is “v2”, the image is compatible. For example, see the highlighted “imds” section in the following image catalog file:

{
        "created": 1709942743,
        "published": 1709951597,
        "date": "2024-03-09",
        "description": "Official Cloudbreak image",
        "images": {
          "aws": {
            "eu-central-1": "ami-0faa3e25092764091",
            "us-west-1": "ami-043b94655fb95edf1",
            "us-west-2": "ami-0f9ed7da8f775d00e"
          }
        },
        "os": "redhat8",
        "os_type": "redhat8",
        "uuid": "9b84a914-a15a-4856-94a9-3cfda722b0b8",
        "package-versions": {
          "blackbox-exporter": "0.19.0",
          "cdp-logging-agent": "1.3.2_b1",
          "cdp-minifi-agent": "1.22.07",
          "cdp-prometheus": "2.36.2",
          "cdp-request-signer": "0.2.4",
          "cdp-telemetry": "1.3.2_b1",
          "cloudbreak_images": "8cf7cd165b58343011091c4908f9f796d4dceb92",
       "imds": "v2",
          "inverting-proxy-agent": "3.0.7-b1",
          "inverting-proxy-agent_gbn": "41924420",
          "java": "8",
          "java11": "11.0.21",
          "java17": "17.0.9",
          "java21": "21.0.2",
          "java8": "1.8.0_392",
          "metering_agent": "2.0.0",
          "node-exporter": "1.3.1",
          "psql": "14",
          "psql11": "11.22",
          "psql14": "14.11",
          "python36": "3.6.8-38.module+el8.5.0+12207+5c5719bc",
          "python38": "3.8.16-1.module+el8.8.0+18967+20d359ae.1",
          "python39": "3.9.16-1.module+el8.8.0+20025+f2100191.2",
          "salt": "3001.8",
          "salt-bootstrap": "0.13.6-2022-05-20T08:57:17",
          "source-image": "ami-039ce2eddc1949546"
        },
        "tags": {
          "fips-mode": "disabled",
          "hardening": "cis_server_l1"
        }
      },