Cloudera Docs

Enabling a fully private network for a Cloudera Data Engineering service for Azure (Tech Preview)

Learn how to enable a fully private network setup for a Cloudera Data Engineering service for Azure services in Cloudera. Additionally, you can learn how to add User Defined Routing (UDR) in the UI or CLI. The UDR helps from exposing public IP addresses in your service.

This feature ensures that all Azure services used by Cloudera Data Engineering are provisioned as private (private Azure Kubernetes Service (AKS), MySQL, and Storage Accounts). The Azure cluster is deployed as a fully private network cluster when you enable a Cloudera Data Engineering service and enables VNet access through private endpoints and private links. Lastly, Cloudera Data Engineering on Microsoft Azure does not currently support SSD or Spot instances.

Before you begin

  • Ensure that you have created and enabled a Cloudera Data Engineering service. Additionally, the Cloudera must communicate with the Cloudera Data Engineering service on a private network in order to manage the Cloudera Data Engineering service lifecycle. This communication occurs using the Cluster Connectivity Manager v2; therefore, to enable this feature, the Cloudera environment must be enabled with the CCMv2. Once the CCMv2 is enabled at the Cloudera environment level, the Enable Private Network option displays in the Cloudera Data Engineering user interface when you enable a service. For more information on how to enable a Cloudera Data Engineering service and set up CCMv2, refer to the links below.

UI steps for enabling a private network and enabling UDR

  1. While enabling a Cloudera Data Engineering service for an Azure environment, select Enable Private Network. Optionally, once you've enabled a private network on Microsoft Azure, you can select the User Defined Routing checkbox. Use this to avoid exposing public IP addresses in your service by using a user defined routing (UDR) table. After, you'll need to specify a Subnet.
  2. Click Enable.
You can enable the user defined routing (UDR) with the CDP CLI using the –network-outbound-type CLI switch with a value of “UDR”. See the example command: 
./clients/cdpcli/cdp.sh de enable-service --name "test-service-cdpcli" --env "dex-priv-env" --instance-type "Standard_D8s_v4" --minimum-instances 0 --maximum-instances 10 --enable-private-network --subnets dex-dev.internal.19.westus2 --network-outbound-type UDR