List of required configuration parameters for the ADLS to Databricks ReadyFlow
When deploying the ADLS to Databricks ReadyFlow, you have to provide the following parameters. Use the information you collected in Prerequisites.
| Parameter Name | Description |
|---|---|
| CDP Workload User | Specify the CDP machine user or workload user name that you want to use to authenticate to the object stores and to the schema registry. Ensure this user has the appropriate access rights to the object store locations and to the schema registry in Ranger or IDBroker. |
| CDP Workload User Password | Specify the password of the CDP machine user or workload user you are using to authenticate against the object stores and the schema registry. |
| CDPEnvironment | The CDP Environment configuration resources. |
| Destination ADLS File System | Specify the name of the ADLS data container you want to write to. The full path is constructed from: abfs://#[***Destination ADLS File System***]@#[***Destination ADLS Storage Account***].dfs.core.windows.net/#[***Destination ADLS Path***] |
| Destination ADLS Path | Specify the path within the ADLS data container where you want to write to. Make sure
that the path starts with "/". The path has to end with the destination
Databricks Table Id. The full path is constructed from:
abfs://#[***Destination ADLS File
System***]@#[***Destination ADLS Storage
Account***].dfs.core.windows.net/#[***Destination ADLS
Path***] |
| Destination ADLS Storage Account | Specify the destination ADLS storage account name. |
| Partition Column | Specify the name of the column used to partition your destination Databricks table. This ReadyFlow only supports a single partition column. |
| Partition Column Exists | Specify whether the destination Databricks column is partitioned. The default value is YES. |
| Schema Name | Specify the schema name to be looked up in the Schema Registry used to parse the source files. |
| Schema Name 2 | If your Databricks table is partitioned, specify the name of the modified schema to be looked up in the Schema Registry. This schema should not include the partition column field. |
| Schema Registry Hostname | Specify the hostname of the Schema Registry you want to connect to. This must be the direct hostname of the Schema Registry itself, not the Knox Endpoint. |
| Source ADLS File System | Specify the name of the ADLS data container you want to read from. The full path is constructed from: abfs://#[***Source ADLS File System***]@#[***Source ADLS Storage Account***].dfs.core.windows.net/#[***Source ADLS Path***] |
| Source ADLS Path | The full path will be constructed from: abfs://#[***Source ADLS File System***]@#[***Source ADLS Storage Account***].dfs.core.windows.net/#[***Source ADLS Path***] |
| Source ADLS Storage Account | Specify the source ADLS storage account name. |
