Cloudera Docs

Create Azure Files Storage Account and File Share

Azure Files NFS v4.1 is a managed, POSIX compliant NFS service on Azure. The file share is used to store files for the Cloudera Machine Learning infrastructure and Cloudera Machine Learning Workspaces. This is the recommended NFS service for use with Cloudera Machine Learning.

Consult the Azure Files for NFS documentation, below, for detailed information on how to create an NFS share. We outline the sequence of steps here.

  1. Create a FileStorage storage account.
  2. Disable secure transfer.
  3. See Create a private endpoint, below, to create a private endpoint between the FileStorage storage account for each of the subnets that you plan to use for creating the Cloudera Machine Learning Workspace. Without this step, Cloudera Machine Learning Workspace nodes will not have connectivity to the NFS server.
  4. Create NFS file shares, one for each Cloudera Machine Learning Workspace. Create each file share with an initial capacity of at least 100 GB. Ensure that the “No Root Squash” export policy is set. This is the default setting.
  5. Create a Linux VM in the subnet in which you are going to create the Cloudera Machine Learning Workspace, and ensure that you can successfully mount the NFS share in the VM. This step will ensure that there are no connectivity issues to the NFS server from hosts in this subnet.
  6. Create a directory within the NFS share to be used as the Cloudera Machine Learning Workspace’s projects folder. For example, if the NFS export path is ‘azurenfsv4.file.core.windows.net:/myshare’, and is mounted as ‘/mnt/myshare’, run ‘sudo mkdir /mnt/myshare/ml-workspace-1; sudo chown 8536:8536 /mnt/myshare/ml-workspace-1’ to set ownership to the ‘cdsw’ user ID and group ID.
  7. Go to the mount instructions page to copy the server name and export path details. Enter this into the Cloudera Machine Learning control plane when prompted for an existing server. Due to a known issue in the control plane UI, the “nfs://” prefix must be specified if you are using the domain name of the file share server, e.g., “nfs://azurenfsv4.file.core.windows.net:/myshare/ml-workspace-1”. Remember to select the NFS protocol version as 4.1.
  8. The Linux VM created above can be deleted if there are no issues with mounting the NFS share.