SQL Stream Builder architecture

The architecture for the SQL Stream Builder that is integrated with Cloudera Data Platform (CDP) typically consists of a Console component through with you can submit queries, a SQL Stream Engine (not sure wht it does, please include, and a Materialized View Engine (again, please include what it does).

The SQL Stream Builder (SSB) service is integrated on the Cloudera platform and connected to Flink and its services. The SSB architecture includes Streaming SQL Console, SQL Stream Engine and Materialized View Engine. These main components within SSB are responsible for executing jobs, populating topics, creating metadata and querying data that happens in the background.

SSB consists of the following main components:
  • SQL Stream Engine
  • Streaming SQL Console
  • Materialized View Engine

The primary point of user interaction for SQL Stream Builder is the Console component. When you submit a query using the Streaming SQL Console, a Flink job is automatically created in the background on the cluster. SSB also requires a Kafka service on the same cluster. This mandatory Kafka service is used to automatically populate topics for the websocket output. The websocket output is needed for sampling data to the Console, and when no table is added to output the results of the SQL query.

When a Materialized View query is submitted, Flink generates the data to the Materialized View database from which the Materialized View Engine queries the required data.

The Streaming SQL Console and the Materialized Views need databases where the metadata of SQL jobs are stored and from which the Materialized View Engine queries data to create the views. SSB supports MySQL/ MariaDB, PostgreSQL and Oracle as databases. For the Streaming SQL Console, you can choose MySQL/MariaDB, PostgreSQL or Oracle DB. However, you must install PostgreSQL to be able to create Materialized Views.