Cloudera Docs

Adding custom DDL templates and connectors

You can add custom templates for the existing catalogs, and also add new custom templates and connector to SQL Stream Builder to further customize the SQL Stream job for your production environment.

You can customize SSB based on your requirements with the following available scenarios for Flink DDL:
  • Add custom DDL templates for default connectors
  • Add custom DDL templates with custom connectors
  • Add custom connectors to use in SSB
You can specify the new template by uploading the template file to the User Defined DDL Templates directory. If you need to add a custom connector that can be used with the template, you need to upload the corresponding connector jar file to the Flink SQL Connector Jar directory. Both directories can be configured through the SQL Stream Builder service in Cloudera Manager.
  1. Navigate to the Streaming SQL Console.
    1. Navigate to Management Console > Environments, and select the environment where you have created your cluster.
    2. Select the Streaming Analytics cluster from the list of Data Hub clusters.
    3. Select Cloudera Manager from the services.
      You are redirected to the Cloudera Manager user interface.
    4. Click Configuration.
  2. Select Local Directories and Files.
  3. Select sql_stream_builder from Scope to filter down the configuration properties.
    The Flink SQL Connector Jar Directory and User Defined DDL Templates Directory information is displayed.
    You need to upload the custom template and connector to these directories.
  4. Check the predefined value for the configurations.
  5. Connect to the host with ssh where the Streaming SQL Console is running.
    For example:
    ssh <your_workload_username>@<master_node_FQDN>
Password:<your_workload_password>
  6. Upload the custom template and the connector jar to the corresponding directories using the command line tool.
    For example:
    scp path/flink-faker-template <your_workload_username>@<master_node_FQDN>:.
  7. Review that the template and connector file is in the directory.
    For example:
    ll /usr/share/flink-ddl-templates/
    If you see the template and connector file in the directories, you must restart the Streaming SQL Console service.
  8. Select Clusters > sql_stream_builder.
  9. Click on Instances tab.
  10. Check the Streaming SQL Console role type checkbox.
  11. Click on Actions > Restart.
    After the Streaming SQL Console service restarted, the new template and connector should be available for Flink DDL.
  12. Click on SQLStreamBuilder Console.
  13. Click on Tables from the main tabs.
  14. Select Add table > Flink DDL.
    The Flink DDL window appears.
  15. Click on Available templates drop-down menu.
    The custom template is listed after the predefined templates.