Known issues

This section lists known issues that you might run into while using the Management Console service.

CB-11925: Knox Load Balancer API requests initiated from Knox Gateway hosts can fail with Connection timeout error

Problem: When logged into a Data Lake or Data Hub node that has a Knox Gateway service instance configured on it, making Knox API calls through the Knox load balancer can result in a connection timeout error. This is because for security reasons, the IP address of the request is preserved in the traffic passed through the load balancer. Preserving the IP address means that the load balancer will reject "loopback" traffic, meaning traffic that originates and is directed back to the same node.

Workaround: If Knox API calls need to be made while logged into a Knox gateway node, use the hostname of the node instead of the load balancer hostname in the API call.

The Knox load balancer hostname can be identified by the "-gateway" suffix in the first clause of the hostname with no numeric identifier. For example: <cluster-name>-gateway.<env-shortname>.<hash> is the load balancer hostname, and <cluster-name>-gateway0.<env-shortname>.<hash> is a direct node hostname.

OPSAPS-59129: CM reports the HDFS warning "Secure DataNode configuration is valid, but not recommended."

On Runtime versions 7.2.9 and earlier, Data Lake clusters that include the HDFS service show this warning in Cloudera Manager: "Secure DataNode configuration is valid, but not recommended." This warning is benign and can be ignored.

CDPSDX-2879 Ranger import fails when you create a Hive replication policy for a medium duty Data Lake cluster

When you create a Hive replication policy with the Include Sentry Permissions with Metadata or Skip URI Privileges option for a medium duty Data Lake cluster, Ranger import fails. Before you choose the Include Sentry Permissions with Metadata option for a Hive replication policy for a medium duty Data Lake cluster, contact Cloudera Support.

CB-10535, CB-10372 Generate CLI command for existing environment should show 3 commands instead of 1

Problem: If you try to obtain the CDP CLI commands from an existing environment > Actions > Show CLI commands, only the create environment command is displayed instead of all three commands required for registering an environment from CDP CLI.

Workaround: You can obtain the command for creating a Data Lake from Data Lake details. The command to obtain the set IDbroker mappings can be obtained from an existing environment or from CDP CLI help, but you need to modify it manually to set the same mappings as in the source environment.

CB-10706 SSO is not working for Solr/Namenode UI links

Problem: SSO login to an environment with a medium duty Data Lake breaks access to Solr and Namenode UI links.

Workaround: After you deploy a medium duty Data Lake, login to Gateway0 and run:

openssl rand -base64 12

Then login as root to both gateway nodes and run:

export KNOX_GATEWAY_DATA_DIR=/var/lib/knox/gateway/data
/opt/cloudera/parcels/CDH/lib/knox/bin/ create-alias pac4j.password --cluster knoxsso --value “the value from above"

Then in Cloudera Manager, restart the Knox service.

DWX-6635 Tags are not being added to S3 buckets

Problem: S3 buckets that are part of your AWS environment registered in CDP are not being tagged during environment registration. This is because the PutBucketTagging policy is missing from the cross-accunt IAM role that CDP requires you to create for your environment’s credential.

Workaround: You can:
  • Manually add tags to your S3 buckets used for existing environments.
  • Add the PutBucketTagging policy to the IAM role used for your provisioning credential so that any environments registered in CDP in the future can automatically add S3 bucket tags.

CB-6924 Workaround for ZooKeeper external volume bug

Problem: In the current version of CDP, ZooKeeper might be configured to write to CDP's root disk which is too small to accommodate the ZooKeeper data. To correct this issue, you need to reconfigure ZooKeeper to write to an external volume and move any ZooKeeper data to that volume.

Workaround: To check if ZooKeeper is configured to use an external volume, complete the following:

  1. Open ZooKeeper and navigate to: ZooKeeper menu item -> Configuration tab -> Filter to Server.
  2. If the dataDir and dataLogDir fields contain /hadoopfs/fs1/zookeeper you do not need to do anything.
  3. If the fields contain any other values, you must reconfigure ZooKeeper.

To reconfigure ZooKeeper, complete the following:

  1. ssh into the machine where the ZooKeeper server is running .
  2. Run the following command to change the user:

    sudo -su zookeeper

  3. Run the following command:

    cp -R /var/lib/zookeeper/ /hadoopfs/fs1/zookeeper

  4. Open the cluster from the Cloudbreak user interface.
  5. Log into the Clouder Manager user interface.
  6. Find ZooKeeper on the Cloudera Manager page and navigate to the configuration with either the Search box or select it from the side menu: ZooKeeper menu item -> Configuration tab -> Filter to Server.
  7. Change the following properties:
    dataDir: /hadoopfs/fs1/zookeeper
    dataLogDir: /hadoopfs/fs1/zookeeper
  8. Save your changes
  9. Restart the Stale configuration

    You do not need to redeploy ZooKeeper.

CB-3876 Data Warehouse and Machine Learning create security groups

Problem: When during environment registration you choose to use your own security groups, the Data Warehouse and Machine Learning services do not use these security groups but create their own.

Workaround: For instructions on how to restrict access on the security groups created by the Data Warehouse service, refer to Restricting access to endpoints in AWS environments.

CRB-971 Data Warehouse creates IAM, S3, and DynamoDB resources

Problem: The Data Warehouse service creates its own S3 buckets, DynamoDB tables, and IAM roles and policies. It does not use the environment's S3 bucket(s), DynamoDB table, and IAM roles and policies.

Workaround: There is no workaround.

CB-4176 Data Lake cluster repair fails after manual stop

Problem: Data Lake cluster repair fails after an instance has been stopped manually via AWS console or AWS CLI.

Workaround: After stopping a cluster instance manually, restart it manually via the AWS console or AWS CLI, and then use the Sync option in CDP to sync instance state.

CB-2813 Environment with ML workspaces in it can be deleted

Problem: When deleting an environment that uses a customer-created VPC and subnets, there is no mechanism in place to check for any existing ML workspaces running within the environment. As a result, an environment can be deleted when ML workspaces are currently running in it.

Workaround: If using an environment created within your existing VPC and subnets, prior to deleting an environment, ensure that there are no ML workspaces running within the environment.

CB-3459 Subnet dependency error when deleting an environment

Problem: When deleting an environment that uses a VPC and subnets created by CDP, the environment deletion fails with an error similar to: AWS CloudFormation stack reached an error state: DELETE_FAILED reason: The subnet 'subnet-05606fd72fda58c8c' has dependencies and cannot be deleted. (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation; Request ID: da9a7fe0-ac43-467e-9942-94f10e6bd2b7)
This error occurs if there are resources such as instances used for Data Warehouse, or Machine Learning cluster nodes that were not deleted prior to environment termination.

Workaround: Prior to terminating an environment, you must terminate all clusters running within that environment.

CB-4248 Expired certificate causes untrusted connection warning

Problem: CDP automatically generates an SSL certificate for every Data Lake and Data Hub cluster. There are two possibilities:
  • By default, CDP generates a trusted certificate valid for 3 months.
  • If generating a trusted certificate fails, CDO generates a self-signed certificate valid for 2 years.

In the first case, if your cluster stays active for over 3 months, the trusted certificate will expire and you will see an "untrusted connection", warning when trying to access cluster UIs from your browser.

Workaround: To fix this, you should generate a new certificate by using the following steps:
  1. Use the Renew certificate UI option:
    • For Data Lake - Click the Renew certificate button on the Data Lake details page.
    • For Data Hub - Click Actions > Renew certificate on Data Hub cluster details page.

    During certificate renewal, several related messages will be written to Event History. Once the certificate renewal has been completed, the following message appears: "Renewal of the cluster's certificate finished."

  2. Additionally, if your cluster was created prior to December 19, you need to perform the following manual steps:
    1. SSH to the Knox gateway host on your cluster.
    2. Run the hostname command to get your domain name.
    3. Run the following commands (just replace the domain name with your correct, fully-qualified domain name):
      sudo sh -c '/opt/salt_2017.7.5/bin/salt --out=newline_values_only '' pillar.get gateway:userfacingcert > /etc/certs-user-facing/server.pem'
      sudo systemctl reload nginx.service