Cloudera Docs

Cloudera Manager - memory tuning issue

Cloudera Data Hub and Data Lake services on JDK 17 clusters might fail after upgrading to Cloudera Runtime 7.3.1.500 or lower versions due to a problem in fine memory tuning with JDK17.

Condition

After upgrading to Cloudera Runtime 7.3.1.500 or lower versions, Cloudera Data Hub and Data Lake services can fail with Not enough space error if the JDK version of the cluster is 17.

Cause

In certain cases, Cloudera Manager does not fully consider the memory already assigned to management service roles when calculating memory for Cloudera Data Hub and Data Lake service roles (such as Atlas) that are co-located on the same node. As a result, the services might attempt to acquire more memory than is physically available on the host, causing the process to fail with the following error: java.lang.OutOfMemoryError: Not enough space.

Remedy

  1. Use the following API to manually trigger the Cloudera Manager memory allocation algorithm on the Data Lake or Cloudera Data Hub host.

    You can call this API through the Cloudera Manager Swagger UI or any API client. You need to have EnvironmentAdmin resource role on the environment to call the API.

    API Endpoint: POST /api/v57/hosts/reallocateMemory
    Include the host name in the API request body. This ensures that memory assignments are recomputed correctly, taking all roles on the host into account.
    Using the reallocateMemory API, Cloudera Manager automatically runs memory configuration rules during cluster setup to allocate memory across all roles. The reallocateMemory API provides a mechanism to rerun those rules to recalculate memory configuration parameters for roles on specified hosts and optionally apply the changes after the initial cluster deployment. For more information about the reallocateMemory API, see the Cloudera Manager API documentation.
  2. Restart the cluster.