Cloudera Docs

Executing SQL jobs in production mode

As by default the SQL jobs are running in a session cluster, there is a risk in case of a cluster failure that every job is affected within that cluster. However, you can set a per-job production mode in SQL Stream Builder to create a dedicated environment for your production jobs.

Production mode means that the deployed SQL job (Flink job) runs in a separate SQL session and on a dedicated YARN cluster configured specifically for that particular job. You can set the execution mode in the Job Settings window, although the YARN execution target can be overwritten using a SET statement in the SQL Editor.

Setting production mode in Job Settings

To set the production mode for your SQL jobs in the Jobs Setting page, you need to select the execution mode with the following steps:
  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Click on SQL Stream Builder from the list of Services.
    3. Click on the SQLStreamBuilder Console.
    The Streaming SQL Console opens in a new window.
  2. Click Create Job or select a previous job on the Getting Started page.

    You are redirected to the Console page.

  3. Click Job Settings.
  4. Select PROD - Job runs in isolated Cluster at Execution Mode.
  5. Add a SQL statement to the SQL Editor you want to execute.
  6. Click Execute.

Setting YARN execution target in SQL Editor

To set the YARN execution mode for your SQL jobs in the SQL Editor, you need to use the SET statement, and execute the SQL query after defining the execution target and properties in the same window:
set 'execution.target' = 'yarn-per-job';
set 'logging.configuration.file' = '/tmp/log4j.properties';
select * from datagen_table_1631781644;

In the above example, the execution target is set to per-job to create a new YARN application for the job. Setting the execution target to per-job allows you to have an individual cluster for the specific job. The additional properties that you configure using the SET statement overwrites the properties that are configured for the running session. However, when you set properties for the production mode, the settings of the session cluster are not affected.

  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Click on SQL Stream Builder from the list of Services.
    3. Click on the SQLStreamBuilder Console.
    The Streaming SQL Console opens in a new window.
  2. Click Create Job or select a previous job on the Getting Started page.

    You are redirected to the Console page.

  3. Set the execution mode to per-job.
    set 'execution.target' = 'yarn-per-job';
  4. Add additional configuration to the production job.
  5. Add a SQL statement you want to execute.
    Example:
    set 'execution.target' = 'yarn-per-job';
set 'state.backend' = 'rocksdb';
select * from faker_table_1631781644;
  6. Click Execute.