Cloudera SQL Stream Builder OverviewPDF version

Cloudera SQL Stream Builder architecture

The Cloudera SQL Stream Builder service is integrated on Cloudera and connected to Flink and its services. The Cloudera SQL Stream Builder architecture includes SQL Stream Engine and Materialized View Engine. These main components within Cloudera SQL Stream Builder are responsible for executing jobs, populating topics, creating metadata and querying data that happens in the background.

Cloudera SQL Stream Builder consists of the following main components:
  • SQL Stream Engine
  • Materialized View Engine

The primary point of user interaction for Cloudera SQL Stream Builder is the Streaming SQL Console User Interface running on the Streaming SQL Engine. When a query is submitted on the Streaming SQL Console, the Streaming SQL Engine automatically creates a Flink job in the background on the cluster. Cloudera SQL Stream Builder also requires a Kafka service on the same cluster. This mandatory Kafka service is used to automatically populate topics for the Server-Side Events (SSE) output. The SSE output is needed for sampling data to the Console, and when no table is added to output the results of the SQL query.

When a Materialized View query is submitted, Flink generates the data to the Materialized View database from which the Materialized View Engine queries the required data.

Cloudera SQL Stream Builder uses databases in the following cases:
  • To store metadata of SQL jobs
  • To store data for creating Materialized Views
  • As a connector for Flink SQL

The Streaming SQL Console and the Materialized Views need databases where the metadata of SQL jobs are stored and from which the Materialized View Engine queries data to create the views. Cloudera SQL Stream Builder supports MySQL/ MariaDB, PostgreSQL and Oracle as databases for the Streaming SQL Engine. You can only use PostgreSQL for the Materialized View Engine. When using the JDBC connector in Flink SQL, you can use MySQL and PostgreSQL from the supported databases.

When configuring the databases for Cloudera SQL Stream Builder, you only set the databases that are used for metadata management. When you want to use the JDBC connector, you need to add the database information to the CREATE TABLE statement using the JDBC connection parameter.