Cloudera Docs

Displaying your costs associated with an environment

When a cluster is created, detailed summary reports of the costs and resource usage for the environment are also generated, which enable you to analyze the costs and the cost breakdown associated with the environment. You can view the current and historical costs and the resource usage associated with your environments.

Steps on how to view the detailed summary reports associated with an environment.

The reports visually display the tracked resource consumption and usage costs associated with the environment for a specific time-period that you select from the Date Range and Granularity list.

Within each report, you can drill down further:
  • To view by users, resource pools, jobs, queries, projects, teams, and application types with the highest costs. Note that projects and teams are only available in the Cloudera AI environment, and resource pools are not applicable for Cloudera AI.
  • To view the health of a job or query of interest.
  • To optimize costs by using the Cloudera Observability prescriptive guidance and recommendations that enable you to improve performance and resource usage.
  • Enabling the Show Resource Usage checkbox allows you to track resource usage, including CPU, GPU, memory, and data read/write. For example, CPU and GPU usage can be viewed in core-hours.
  1. Verify that you are logged in to the Cloudera Observability web UI.
    1. Log in to Cloudera in a supported browser.
      The Cloudera Cloud web interface landing page opens.
    2. From the Your Enterprise Data Cloud landing page, select the Observability tile.
      The Cloudera Observability landing page opens.
  2. From the Cloudera Observability Main navigation panel, select Financial Governance > Environments.
    The Cloudera Observability Environments page opens, which displays:
    • The total provisioned cost, and the provisioned and utilized cost of CPU, memory, and GPU cost for all of your environments. Data read and written values for all of your cost centers.

      Provisioned cost = Provisioned CPU + Provisioned Memory + Provisioned GPU + Data read + Data write

    • Lists your existing environments that use the current criteria settings and displays the provisioned costs, CPU, GPU, and Memory cost and data read and written cost associated with each environment.
    • The provisioned cost, CPU, GPU, and memory usage, and data read and written values for users and pools that are in the Other Engines section.
  3. To display an environment's detailed report that includes costs for each chosen environment, do the following:
    1. From the Date Range and Granularity list, select a time-period that meets your requirements.
    2. From the Environments page, click inside the environment row that requires analysis.
      The environment's report page opens, which displays the following:
      • The provisioned costs, CPU, GPU, and Memory usage, and Data Read and Data Written cost per GB for the environment.
      • Lists the environments that are associated with the environments and displays their provisioned costs, CPU, GPU, and Memory cost, and data read and written cost.
  4. To display the users, pools, jobs, queries, and application types that created the highest costs on a specific cluster of interest, do the following:
    1. Click inside the cluster row that requires more analysis.
      The cluster report overview page opens, which displays the following:
      • The top 500 users whose jobs created the highest costs.
      • The top 500 pools whose jobs created the highest costs.
      • The top 500 jobs or queries that created the highest costs.
    2. To gain more insights on the health of a job or query, click the name of the job or query listed in the Top 500 Jobs panel that requires more investigation.

      The job or query's summary page opens.

    3. Select the Health Checks and Execution Details tabs for more insights and if available read the optimization recommendations.
  5. To view a full list of users, including their job costs, CPU, GPU, and Memory usage, select the Users tab.
    The Users report opens, which displays the following:
    • The name of the user.
    • The total cost that the user incurred.
    • The number of jobs that the user ran.
    • The CPU, GPU, and Memory hourly usage.
    • The cost per GB for the data read and written by your application.
    • The total job costs that the Cloudera Observability engines incurred: AI Jobs, Impala, Hive, Spark, MapReduce, and Oozie.
  6. To view a full list of pools, including their job costs, CPU, GPU, and Memory usage, select the Pools tab.
    The Pools report opens, which displays the following:
    • The name of the pool.
    • The environment that the pool is associated with.
    • The total cost that the pool incurred.
    • The number of jobs that the pool ran.
    • The CPU, GPU, and Memory usage.
    • The cost per GB for the data read and written by your application.
    • The total job costs that the Cloudera Observability engines incurred: AI Jobs, Impala, Hive, Spark, MapReduce, and Oozie.
  7. To view historical costs, change the time-period currently displayed in the Date Range and Granularity list fields. For information on how to change the time-period, click the Related Information link below.