Managing costs in the cloud

Cost optimization in cloud environments is a top priority for enterprises. Eighty percent of cloud costs are determined by the number of compute instances you use. Compute instances can be virtual machines or containers. If you can closely monitor and control the number of compute instances, you can control your overall cloud spend. Cloudera Data Warehouse service provides methods to monitor and control the number of cloud resources you use. In addition, configuring your usage limits with your cloud provider can also help you control your costs.

Setting resource limits with Cloudera Data Warehouse service

Cloudera Data Warehouse service provides the following ways to manage your cloud costs:

  • Choose Virtual Warehouse size: Virtual Warehouse size specifies the number of
 executor nodes used by the Virtual Warehouse, which translates to compute instances.
 Before you create a Virtual Warehouse, determine the number of concurrent queries
 or users your Virtual Warehouse must serve during peak periods. This information helps you
 determine what size of Virtual Warehouse you need. Choose the size based on the number of
 nodes you typically use for clusters in an on-premises deployment.
  • Set auto-scaling thresholds: When you create a Virtual Warehouse, you can define
 auto-scaling, which sets limits on how many cloud resources can be consumed to meet
 workload demands. In addition, you can also set the maximum time a Virtual Warehouse idles
 before shutting down. Both settings ensure that you only use cloud resources that you need
 when you need them, helping you to manage your costs in the cloud.

Setting resource limits with your cloud provider

Another way to manage your cloud costs is by setting effective resource limits with your
 cloud provider. Cloud providers offer ways to set the overall limit on numbers of
 virtual machines or containers that can be used for your account. You can also save on
 cloud spend by shutting down resources when they are not in use. CDP has built-in functionality to shut down cloud resources when not in use. If necessary, you can also terminate resources to save on costs related to disk snapshots, reserved IP addresses, and so on. In addition, cloud providers might offer
 tools to help you save. For example, AWS offers
 AWS Trusted Advisor and CloudWatch.