Create role and policy used to deploy Cloudera environments for Cloudera Data Flow

The Cloudera Data Flow control plane requires a role and policies to create Cloudera environments. In this step, you create the compute infrastructure restricted IAM policy - a common policy for all data services deployed on Cloudera.

To enable the Cloudera Data Flow experience after the environment has been created, an Administrator needs to attach the Compute Restricted IAM policy and the Cloudera Data Hub restricted policy with the Cloudera cross-account role associated with the environment.

To use Cloudera Data Flow with the most restrictive options available, you need to create a custom Customer Managed Key (CMK) defining it on Environment level. The Compute Restricted IAM policy provided here requires the existence of such a key. If you want to use Cloudera Data Flow with less restrictive options, you can modify the policy in a way that allows Cloudera to create and manage the key for you.

Option #1: During environment creation

The Cloudbreak environment creation UI should be set up as shown here:

note

For the AWS IAM policy mentioned in the “Create Cross-account Access Policy” section, use the Compute infrastructure Restricted IAM policy below and create as a new policy in AWS IAM. There may be one or more restricted policies already attached to the cross-account role, in addition to the Compute infrastructure restricted policy. For example, there may also be a Cloudera Data Hub restricted policy.
For the “Create Cross-account Access Role” section, create the cross-account role as instructed (or update the role if one already exists) and attach the newly created Compute infrastructure Restricted IAM policy. Finally, update the cross-account role to use it.

Option #2: Prior to enabling Cloudera AI data service

If the Cloudbreak environment has already been created, you can create and attach the Compute infrastructure Restricted IAM policy to the existing cross-account role associated with the environment.

To view the existing cross-account role, in the Environments section of the Cloudera Management Console, on the Summary tab, see Credentials.

Compute (Liftie) Restricted IAM policy

Replace the following placeholders in the JSON file:

[***YOUR-ACCOUNT-ID***] with your account ID in use.
[***YOUR-IAM-ROLE-NAME***] with the IAM restricted role associated with this policy.
[***'YOUR-SUBNET-ARN-*'***] supplied during the Cloudera Environment(s) creation.
note
Provide all the subnets present in all the Cloudera Environment(s) that you intend to use for the experience. If at any point a new Cloudera Environment is created or an existing one is updated for subnets, provide it here.
[***YOUR-IDBROKER-ROLE-NAME***] with the ID Broker Role name in use.
[***YOUR-LOG-ROLE-NAME***] with the Log Role name in use.
[***YOUR-KMS-CUSTOMER-MANAGED-KEY-ARN***] with KMS key ARN.

{
  "Version": "2012-10-17",
  "Id": "ComputePolicy_v12",
  "Statement": [
    {
      "Sid": "SimulatePrincipalPolicy",
      "Effect": "Allow",
      "Action": [
        "iam:SimulatePrincipalPolicy"
      ],
      "Resource": [
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/[YOUR-IAM-ROLE-NAME]"
      ]
    },
    {
      "Sid": "RestrictedPermissionsViaClouderaRequestTag",
      "Effect": "Allow",
      "Action": [
        "cloudformation:CreateStack",
        "cloudformation:CreateChangeSet",
        "ec2:createTags",
        "eks:TagResource"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:RequestTag/Cloudera-Resource-Name": [
            "crn:{{ .CRNPartition }}:*"
          ]
        }
      }
    },
    {
      "Sid": "RestrictedPermissionsViaClouderaResourceTag",
      "Effect": "Allow",
      "Action": [
        "autoscaling:DeleteTags",
        "autoscaling:DetachInstances",
        "autoscaling:ResumeProcesses",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:SuspendProcesses",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup",
        "cloudformation:DeleteChangeSet",
        "cloudformation:DeleteStack",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStacks",
        "cloudformation:CancelUpdateStack",
        "cloudformation:ContinueUpdateRollback",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DescribeStackResource",
        "cloudformation:DescribeStackResources",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:ListStacks",
        "cloudwatch:deleteAlarms",
        "cloudwatch:putMetricAlarm",
        "ec2:AttachVolume",
        "ec2:CreateNetworkInterface",
        "ec2:CreateVolume",
        "ec2:DeleteVolume",
        "ec2:RunInstances",
        "eks:DescribeUpdate",
        "eks:ListUpdates",
        "eks:UpdateClusterConfig",
        "eks:UpdateClusterVersion",
        "iam:GetRolePolicy",
        "iam:ListInstanceProfiles",
        "iam:ListRoleTags",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:TagRole",
        "iam:UntagRole",
        "logs:DescribeLogStreams",
        "logs:FilterLogEvents"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:ResourceTag/Cloudera-Resource-Name": [
            "crn:{{ .CRNPartition }}:*"
          ]
        }
      }
    },
    {
      "Sid": "RestrictedPermissionsViaCloudFormation",
      "Effect": "Allow",
      "Action": [
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:CreateLaunchConfiguration",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DeleteLaunchConfiguration",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeScheduledActions",
        "autoscaling:DescribeTags",
        "dynamodb:DescribeTable",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateSecurityGroup",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeletePlacementGroup",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeInstances",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribePlacementGroups",
        "ec2:DescribeRegions",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "eks:CreateCluster",
        "eks:DeleteCluster"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringEquals": {
          "aws:CalledVia": [
            "cloudformation.amazonaws.com"
          ]
        }
      }
    },
    {
      "Sid": "RestrictedEC2PermissionsViaClouderaResourceTag",
      "Effect": "Allow",
      "Action": [
        "ec2:RebootInstances",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "ForAnyValue:StringLike": {
          "ec2:ResourceTag/Cloudera-Resource-Name": [
            "crn:{{ .CRNPartition }}:*"
          ]
        }
      }
    },
    {
      "Sid": "RestrictedIamPermissionsToClouderaResources",
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": [
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/[YOUR-IDBROKER-ROLE-NAME]",
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/[YOUR-LOG-ROLE-NAME]",
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/liftie-*-eks-service-role",
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/liftie-*-eks-worker-nodes",
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/cdp-eks-master-role",
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/cdp-liftie-instance-profile"
      ]
    },
    {
      "Sid": "RestrictedKMSPermissionsUsingCustomerProvidedKey",
      "Effect": "Allow",
      "Action": [
        "kms:CreateGrant",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*"
      ],
      "Resource": [
        "[YOUR-KMS-CUSTOMER-MANAGED-KEY-ARN]"
      ]
    },
    {
      "Sid": "AllowCreateDeleteTagsForSubnets",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags",
        "ec2:DeleteTags"
      ],
      "Resource": [
        "arn:{{ .ARNPartition }}:ec2:[YOUR-SUBNET-REGION]:[YOUR-ACCOUNT-ID]:subnet/*"
      ]
    },
    {
      "Sid": "ModifyInstanceAttribute",
      "Effect": "Allow",
      "Action": [
        "ec2:ModifyInstanceAttribute"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "StringEquals": {
          "ec2:Attribute": "SourceDestCheck"
        }
      }
    },
    {
      "Sid": "OtherPermissions",
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:CreatePlacementGroup",
        "ec2:DeleteKeyPair",
        "ec2:DeleteNetworkInterface",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcAttribute",
        "ec2:DescribeVpcs",
        "ec2:ImportKeyPair",
        "ec2:UpdateSecurityGroupRuleDescriptionsIngress",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeCluster",
        "eks:CreateAccessEntry",
        "eks:DeleteAccessEntry",
        "eks:ListAccessEntries",
        "eks:DescribeAccessEntry",
        "eks:AssociateAccessPolicy",
        "eks:DisassociateAccessPolicy",
        "eks:ListAssociatedAccessPolicies",
        "elasticloadbalancing:DescribeLoadBalancers",
        "iam:GetRole",
        "iam:ListRoles",
        "iam:GetInstanceProfile"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Sid": "AllowSsmParams",
      "Effect": "Allow",
      "Action": [
        "ssm:DescribeParameters",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParameterHistory",
        "ssm:GetParametersByPath"
      ],
      "Resource": [
        "arn:aws:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"
      ]
    },
    {
      "Sid": "CfDeny",
      "Effect": "Deny",
      "Action": [
        "cloudformation:*"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "ForAnyValue:StringLike": {
          "cloudformation:ImportResourceTypes": [
            "*"
          ]
        }
      }
    },
    {
      "Sid": "ForAutoscalingLinkedRole",
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/aws-service-role/autoscaling-plans.amazonaws.com/AWSServiceRoleForAutoScalingPlans_EC2AutoScaling"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "autoscaling-plans.amazonaws.com"
        }
      }
    },
    {
      "Sid": "ForEksLinkedRole",
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:{{ .ARNPartition }}:iam::[YOUR-ACCOUNT-ID]:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForEKS"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "eks.amazonaws.com"
        }
      }
    }
  ]
}

CMKs

Depending on whether you created a custom Customer Managed Key (CMK) to be used for EBS encryption or you want Cloudera to generate and manage the CMK for you, select one of the following options:

Provide own CMK - if you want to use your own custom CMK
Let Cloudera generate CMK - if you want to allow Cloudera to generate and manage your CMK

Provide own CMK
Let CDP generate CMK

If you have not already created it, create a custom CMK. Verify that the policy (this is different from the IAM policy) for CMK at KMS has the required additional permissions blocks defined.
For more information, see Using CMKs with Cloudera Data Flow.
Provide the KMS CMK for volume encryption in the policy section with Sid: RestrictedKMSPermissionsUsingCustomerProvidedKey.

Replace the RestrictedKMSPermissionsUsingCustomerProvidedKey policy section with the following:


{
 "Sid": "AllCreateAndManageKMS",
 "Effect": "Allow",
 "Action": [
   "kms:Create*",
   "kms:Describe*",
   "kms:Enable*",
   "kms:List*",
   "kms:Put*",
   "kms:Update*",
   "kms:Revoke*",
   "kms:Disable*",
   "kms:Get*",
   "kms:Delete*",
   "kms:ScheduleKeyDeletion",
   "kms:CancelKeyDeletion"
  ],
  "Resource": "*"
}