Usage-Based Billing

Cloudera Director 2.1 and higher includes an automated metering service that enables usage-based billing, so that you only pay for the services you use. This section describes how usage-based billing works in Cloudera Director.

Prerequisites

The following are required for usage-based billing:
  • Cloudera Director 2.1 or higher
  • A billing ID provided by Cloudera. Your billing ID ensures that the Cloudera Manager instance and the clusters it manages are associated with your customer account, so that metering of your cluster usage is accurate.
  • A Cloudera Enterprise license. When you provide a Cloudera Enterprise license and a billing ID during deployment of Cloudera Manager, usage-based billing is enabled for all clusters created with that Cloudera Manager instance. If you do not add a billing ID, usage-based billing is not enabled, and you are charged for your clusters under normal node-based billing.
  • An account on a cloud service supported by Cloudera Director to deploy Cloudera Manager and CDH.

  • Outbound HTTPS connectivity from Cloudera Director to Cloudera's metering service at https://metering.cloudera.com and the endpoints within AWS where usage information is collected. If outbound internet connectivity is restricted by your organization's security policies, then HTTPS connectivity can be narrowed to the AWS IP address ranges.

  • At least 2 GB of free disk space should be available on the Cloudera Director server to store usage information until it can be transmitted to the metering service.

How Usage-Based Billing Works

When usage-based billing is enabled, Cloudera Director collects cluster usage information at regular intervals in the form of usage bundles. The usage bundles are sent to a metering service that aggregates the information and determines the total bill.

The price for usage-based billing is determined by three factors:
  • The Cloudera hourly rate, which is determined by two factors:
    • Instance type
    • CDH services enabled on the cluster
  • Number of instances
  • Number of hours

Hours billed are based on the time the instance or service starts, not on the time of day. Portions of an hour are rounded up to the next full hour. For example, an instance that runs from 1:40 pm. to 2:20 p.m. is charged for one hour.

Charging for instances in a cluster begins when bootstrapping is complete and the appropriate components have been installed and started on that cluster. The applicable rate is determined by the components that are deployed on the cluster for a given hour, so the price can change when a component is added or removed that would affect the rate.

There is no charge for instances in a cluster where none of the services are running, and billing stops for all instances in the cluster if the cluster is stopped or terminated. Billing and collection of usage information also stops if Cloudera Director is stopped. Billing resumes when Cloudera Director is started, but the billing hour for all billable clusters is reset from when Cloudera Director restarts.

The price charged for a running cluster depends partly on the CDH services it contains. The following table shows the five types of clusters defined for billing purposes, from least to most expensive.
Basic Data Engineering Operational DB Analytic Database Data Hub
"Core Hadoop" "Core Hadoop" + Spark, Search "Core Hadoop" + HBase, Spark, Search "Core Hadoop" + Impala All Capabilities

Usage-based billing only applies to your use of Cloudera Director, Cloudera Manager, and CDH services in the cloud. You are billed directly by your cloud provider for all cloud provider services, such as the virtual instances and databases used by your clusters.

Contact Cloudera for additional details about pricing with usage-based billing.

Deploying Cloudera Manager and CDH with Usage-Based Billing

When you create an instance of Cloudera Manager with a Cloudera Enterprise license and a billing ID, usage-based billing is enabled for all clusters you launch through that Cloudera Manager instance.

You can deploy Cloudera Manager and create clusters with usage-based billing either through the Cloudera Director server web UI or with the Cloudera Director client and the bootstrap-remote command, as described in this section.

Enabling Usage-Based Billing with the Cloudera Director Server web UI

The procedure for deploying Cloudera Manager and CDH through the Cloudera Director web UI is described in Deploying Cloudera Manager and CDH on AWS. To enable usage-based billing, follow the procedure as described there, but be sure to provide a Cloudera Enterprise license and a billing ID as described in the steps for the Add Cloudera Manager screen.

If you choose Cloudera Enterprise, the License Key and Billing ID fields are displayed. The Billing ID field is optional. Enter a valid license key, but do not enter a billing ID if you want your clusters to include Cloudera Enterprise features but without usage-based billing.

Enabling Usage-Based Billing with bootstrap-remote

The procedure for deploying Cloudera Manager and CDH through the Cloudera Director client using the bootstrap-remote command is described in Submitting a Cluster Configuration File.

There is a sample Cloudera Director CLI configuration file for remote bootstrapping a cluster on AWS with usage-based billing enabled. This configuration file will create a basic cluster with a Cloudera Enterprise license and billing ID. Edit the file to provide your license and billing ID, your credentials for your cloud provider, and configurations for your desired cluster services.

Managing Billing IDs with an Existing Deployment

To manage billing IDs for an existing deployment of Cloudera Manager, click Manage Billing on the admin menu in the upper right of the Cloudera Director web UI.


The Manage Billing page displays information about Cloudera Manager instances and environments managed by Cloudera Director.

If a Cloudera Manager instance has a Cloudera Enterprise license and a billing ID, the billing ID is displayed on this page in redacted form, as shown here for the Cloudera Manager instance CM01:


If a Cloudera Manager deployment has a Cloudera Enterprise license but does not have a billing ID, as shown above for the deployment CM02, the value of the Billing ID for that instance is Not Assigned and usage-based billing is not enabled. You can add a billing ID for that Cloudera Manager deployment to enable usage-based billing. To add a billing ID to an existing Cloudera Manager deployment:
  1. On the Manage Billing page, click Assign Billing ID to open the Update Billing ID dialog.
  2. Enter a valid billing ID.
  3. Click Update.
To replace a billing ID with a different one:
  1. Click Update Billing ID.
  2. In the Update Billing ID dialog, enter the new billing ID.
  3. Click Update.

Troubleshooting Network Connectivity for Usage-Based Billing

If Cloudera Director is unable to connect to or upload usage information to the metering service, or is unable to connect to Cloudera Manager to obtain the usage information, an alert appears under the bell icon at the upper right of the top banner in the Cloudera Director web UI, and the bell icon turns red. Click the icon to see the alert.

If Cloudera Director is unable to connect to or upload usage information to the metering service, the alert will say:
  • Cloudera Director is unable to send usage data to Cloudera’s billing service at https://metering.cloudera.com. Check that your network is configured to allow sending of usage data and that Cloudera’s billing service is running.
If Cloudera Director is unable to connect to Cloudera Manager, the alert will say, for example (with actual values for the names of your Cloudera Manager instance and environment, and time elapsed):
  • Unable to connect to cm1 in env2 for at least 2 minutes 18 seconds. Check your deployment status. The deployment might have failed or might have a connectivity issue.
When an alert appears, check the network and security configuration where Cloudera Director is running:
  • Check that the firewall rules for your Cloudera Director instance (for example, the security group for an AWS EC2 instance) are configured to permit network access to the internet.
  • Check that the subnet for the Cloudera Director instance has a route to the internet.
  • Check in the Cloudera Director web UI to ensure that Cloudera Director is able to connect to the Cloudera Manager instance.
  • Open a shell on the Cloudera Director instance and try to ping a publicly-accessible URL, such as www.cloudera.com.
  • Using a machine in your local network environment (outside of the network environment where Cloudera Director is running), send a ping request from a web browser to the collection service ping endpoint at this URL: https://metering.cloudera.com/api/v1/ping. If the metering service is not reachable, the service might be down. Contact Cloudera Support.