Deploying the JDBC to S3/ADLS ReadyFlow
Learn how to use the Deployment wizard to deploy the JDBC to S3/ADLS ReadyFlow using the information you collected using the prerequisites checklist.
The CDF Catalog is where you manage the flow definition lifecycle, from initial import, through versioning, to deploying a flow definition.
In DataFlow, from the left navigation pane, click
Flow definitions available for you to deploy are displayed, one definition per row.
Launch the Deployment wizard.
- Click the row to display the flow definition details and versions.
- Click a row representing a flow definition version to display flow definition version details and the Deploy button.
- Click Deploy to launch the Deployment wizard.
Select the environment to which you want to deploy this version of your
flow definition, and click Continue.
In the Overview, give your flow deployment a unique
You can use this name to distinguish between different versions of a flow definition, flow definitions deployed to different environments, and similar.
In NiFi Configuration:
- Select a NiFi Runtime Version for your flow deployment. Cloudera recommends that you always use the latest available version, if possible.
- Autostart Behavior is on by default, allowing your flow to start automatically after successful deployment. You can clear selection if you do not want the automatic start.
In Parameters, specify parameter values like connection
strings, usernames and similar, and upload files like truststores, and similar.
For parameters specific to this ReadyFlow, see the Example with the configuration parameters table below.
Specify your Sizing & Scaling configurations.
- NiFi node sizing
- You can adjust the size of your cluster from Extra Small to Large
- Number of NiFi nodes
- You can set the number of nodes from 1 to 32.
In Key Performance Indicators, you can set up your
metrics system with specific KPIs to track the performance of a deployed
flow. You can also define when and how to receive alerts about the KPI
See Working with KPIs for more information about the KPIs available and how you can monitor them.
- Review the summary of the information you provided in the Deployment wizard and make any necessary edits by clicking Previous. When you are finished, complete your flow deployment by clicking Deploy.
Once you click Deploy, you are redirected to the Alerts tab in the Flow Deployment Detail view where you can track its progress.
For the JDBC to S3/ADLS Readyflow, the following parameters are required. Use the information you collected in the Meeting the prerequisites section.
|CDP Workload User||Specify the CDP machine user or workload username that you want to use to authenticate to the object stores. Ensure this user has the appropriate access rights to the object store locations in Ranger or IDBroker.|
|CDP Workload User Password||Specify the password of the CDP machine user or workload user you are using to authenticate against the object stores (via IDBroker).|
|CSV Delimiter||If your output data is CSV, specify the delimiter here.|
|Data Output Format||Specify the desired format for your output data. You can use "CSV", "JSON" or "AVRO" with this ReadyFlow.|
|Database Connection URL||Specify the JDBC database connection URL used to connect to your source database. May contain database system name, host, port, database name and some parameters. The exact syntax of a database connection URL is specified by your DBMS. For example, jdbc:mysql://hostname:port/database for MySQL databases.|
|Database Driver||Upload the database driver jar file for your source database.|
|Database Driver Class Name||Specify the source database driver class name. For example, com.mysql.jdbc.Driver for MySQL databases.|
|Database Incremental Fetch Column||Specify the column that is to be used to track state as new data lands in the source table.|
|Database Password||Specify the source database password.|
|Database Table Name||Specify the source database table name.|
|Database User||Specify the source database user.|
|Destination S3 or ADLS Path||Specify the name of the destination S3 or ADLS path you want to write to. Make sure that the path starts with "/".|
|Destination S3 or ADLS Storage Location||Specify the name of the destination S3 bucket or ADLS Container you want to
For S3, enter a value in the form: s3a://[Destination S3 Bucket]
For ADLS, enter a value in the form: abfs://[Destination ADLS File System]@[Destination ADLS Storage Account].dfs.core.windows.net
|Filter Rule||Specify the filter rule expressed in SQL to filter database records. Records matching the filter are written to the destination. The default value forwards all records.|