Cloudera Docs

Managing SQL jobs

You can configure your SQL jobs with Flink features in the Streaming SQL Console or through REST API using the API Explorer.

Configuring SQL job settings

If you need to further customize your SQL Stream job, you can add more advanced features to configure the job restarting method and time, threads for parallelism, sample behavior, exactly once processing and restoring from savepoint.

Before running a SQL query, you can configure advanced features by clicking on the Job Settings button at the SQL Editor.
Job parallelism (threads)
The number of threads to start to process the job. Each thread consumes a slot on the cluster. When the Job Parallelism is set to 1, the job consumes the least resources. If the data source supports parallel reads, increasing the parallelism can raise the maximum throughput. For example, when using Kafka as a data source, setting the parallelism to the equal number as the partitions of the topic can be a starting point for performance tuning.
Sample Count
The number of sample entries shown under the Results tab. To have an unlimited number of sample entries, add 0 to the Sample Count value.
Sample Window Size
The number of sample entries to keep in under the Results tab. To have an unlimited number of sample entries, add 0 to the Sample Window Size value.
Sample Behavior
You have the following options to choose the behavior of the sampled data under the Results tab:
  • Sample all messages
  • Sample one message every second
  • Sample one message every five seconds
Restore From Savepoint
You can enable or disable restoring a SQL job from a Flink savepoint after stopping it.
Enable Checkpointing
You can enable and disable checkpointing for a SQL job. By default the checkpointing is enabled.
Checkpoint Mode
Switching between checkpointing modes. You can choose between At Least Once or Exactly Once.
Checkpoint Interval
The time in milliseconds between checkpointing attempts.
Checkpoint Timeout
The maximum time in milliseconds until a checkpointing attempt is timed out.
Tolerable Checkpoint Failures
The number of checkpointing attempts until the job is aborted.
Failure Restart Strategy
Switching between restarting strategies when checkpointing is failed. You can choose between enabling and disabling auto recovery. By default auto recovery is enabled for checkpointing.

Managing SQL jobs

You can manage your SQL jobs and reload them regardless of success to the SQL Editor to make further and additional changes, and execute them again.

You can manage the SQL Jobs on the Jobs tab if you need to stop them, restart them, delete them, duplicate them, review their events and also reload them to the SQL Editor in case any change is needed to their configuration. To reload a SQL job to the SQL Editor, you can simply click on it from the list of jobs or use > Edit. The details of the job are reloaded to the SQL Editor with the latest saved configuration and Materialized Views.

Loading SQL queries to SQL Editor

You can reuse an already submitted SQL query from SQL History view of Streaming SQL Console.

When you need to reuse a SQL query for a new job, you can load it and execute it in the SQL Editor. You can use the SQL History view of Streaming SQL Console.
You can view the full submitted statement by hovering the mouse over the items. You can load the query to the editor by simply clicking on the query and the statement will appear in the SQL Editor next to it.

Using SQL Stream Builder REST API

You can use the REST API to monitor, manage and configure the SQL Stream jobs with GET, POST and DELETE HTTP methods. You can use the SQL Stream Builder (SSB) REST API in command line, import them to REST API Tools or by accessing the Swagger UI.

The following HTTP methods are available for SSB:
  • GET to query information about the specified endpoint
  • POST to create resources for the specified endpoint
  • PUT to update existing resources for the specified endpoint as a whole
  • PATCH to partially update existing resources for the specified endpoint
  • DELETE to remove objects from the specified endpoint

The REST API Reference document contains the available endpoints for SQL Stream Builder.

You can also reach the SSB REST API reference document from the main menu of Streaming SQL Console using the API Explorer.

The Streaming SQL Engine API details the following operations for SQL Stream Builder and Flink:
  • Heartbeat
  • Admin Operations
  • User Operations
  • User Keytab Operations
  • Data Source Operations
  • Table Operations
  • Project Operations
  • Project Environment Operations
  • Project Sync Operations
  • Project Invitation Operations
  • Project Permission Operations
  • SQL Operations
  • SSB Session Operations
  • Job Operations
  • Sampling Operations
  • Flink Job Operations
  • Flink Session Cluster Operations
  • Artifact Operations
  • UDF Artifact Operations
  • UDF Operations
  • API Key Operations
  • Connector Operations
  • Data Format Operations
  • Diagnostic Operations

Materialized View Pagination

You can set a limit and order the results of a Materialized View query by adding values to the limit and offset configurations.

After creating a Materialized View, you can order and limit the results after opening the Materialized View. The default limit of the results is 100 entries. You can set the limit and offset using the REST URL. You need to modify the URL address in your browser by setting the value and the offset value as shown in the following example:
localhost:18131/api/v1/query/5275/summary_query?key=6ef33a7a-c82a-4a72-946b-5fe0b33e033c&limit=20&offset=3

When you edit the Materialized View, the set limit and offset change based on the new filters set for the Materialized View query. If you set an invalid number for the limit and offset, an error message is displayed.