Azure requirements for Cloudera DataFlow

Understand your Azure account requirements for Cloudera Public Cloud

Understand the Cloudera DataFlow requirements

Networking requirements:

Determine your networking option:

• Use your existing VNet and subnets
• Have Cloudera create a new VNet and subnets

Review firewall exceptions for Azure AKS

Determine your load balancer option:

• public
• private (internal)

Verify that the following services are available in your environment for Cloudera DataFlow to use:

Azure environments used for the Cloudera DataFlow service must have the following resources available in the specific Azure region where the environment is registered. Currently, there is no cross-regional support for Cloudera DataFlow service.

Azure Kubernetes Service (AKS)

Azure Database for PostgreSQL

Azure Data Lake Storage Gen2

Virtual machine scale sets

Dsv4-series

Azure Availability Zones (optional)

Understand Cloudera DataFlow role requirements

There are two Cloudera user roles associated with the Cloudera DataFlow service: DFAdmin and DFUser. Any Cloudera user with the EnvironmentAdmin (or higher) access level must assign these roles to users who require access to the Cloudera DataFlow console within their environment.

Register an Azure Environment in Cloudera

Once you have met cloud provider requirements and have created the Azure provisioning credential for Cloudera, you may proceed to register an Azure Environment.

Instructions: Register an Azure environment

Use only app-based credentials

For the Cloudera DataFlow service, you must only use an app-based credential, which requires the Contributor role to create a new service principal. For more information about creating an app-based credential for the environment you want to use for the Cloudera DataFlow service, see Create an app-based credential. If you need to change your environment credential, see Change environment's credential. Both of these references are in the Management Console documentation.

App must have the Contributor role at the subscription level

For environments that you plan to use for the Cloudera DataFlow service, you must ensure that the application you create in Azure has the built-in Contributor Azure role at the Azure subscription level. For more information, see the description of app-based credentials in Credential options on Azure.

Created Azure app must have access to the storage account used during environment registration

Ensure that the application, which the Azure app-based credentials are attached to, must have access to the ADLS Gen2 storage location that is specified when you register the Azure environment. This is the storage location specified in Step 6 in the Register an Azure environment topic. Also see ADLS Gen2 and managed identities for information about storage accounts for Azure environments. See Minimal setup for cloud storage for further details. These references are in the Cloudera Management Console documentation.

Azure subscription should be in a similar region as the resources

Ensure that your Azure subscription is in a relatively similar region as the region where your resources are deployed. Particularly, be careful that the regions are governed by the same regulatory laws. For more information, see Azure region requirements in the Cloudera Management Console documentation. In that topic it specifies that "Cloudera Public Cloud requires that the ADLS Gen2 storage location provided during environment registration must be in the same region as the region selected for the environment." In addition, please review Azure geographies in the Microsoft documentation.