Fixed issues and improvements

Learn about the fixed issues, improvements, and changes functionality in this release of Cloudera Streaming Analytics - Kubernetes Operator.

Fixed issues

CSA-5622: Project sync doesn't sync Kubernetes config for jobs: Fixes an issue where the project synchronization mechanism in Cloudera Streaming Analytics - Kubernetes Operator did not correctly sync Kubernetes configurations for Flink jobs.
CSA-5751: Bug in SecuredPropertyConverter: Fixes an issue where the SecuredPropertyConverter class caused an infinite encryption loop when its convertToDatabaseColumn method mutated the attribute argument, resulting in exceptions when the data became too long.
CSA-5770: Fix intermittent Job Not Found error while deploying Flink jobs: Fixes an issue with prematurely failed Flink jobs due to recoverable errors to ensure jobs only fail if they reach a globally terminal state or encounter an unrecoverable error.
CSA-5780: Upgrade jackson-databind in flink-cdc to version 2.13.4.2 or above: Upgrades outdated jackson-databind versions in transitive dependencies for Flink artifacts.
CSA-5703: Secure environment variables are broken: Fixes an issue with secure environment variables inadvertently overwriting their original values masking process, leading to data source validation failures and infinite encryption loops.
CSA-5711: Environment importing loader is stuck when there is an error with importing: Fixes an issue where the environment importing loader remained stuck after an import operation failed and users confirmed the error pop-up.
CSA-5658: Postgresql does not start when installing the operator on RKE2: Fixes an issue where the Postgresql init container failed to start for non-root users when installing Cloudera Streaming Analytics - Kubernetes Operator operator on RKE2 due to lack of permissions for the ssb_admin user.

Improvements

Unify default Flink job log4j rolling configuration

For jobs deployed through both Cloudera SQL Stream Builder and Flink, the log4j configuration now by default:

Applies compression
Keeps the last 5 entries
Triggers rollover when size reaches 100MB

For more information on configuring logging, see Adjusting logging configuration in Advanced Settings.

Configurable checkpoint and savepoint directories in the Cloudera SQL Stream Builder UI

The savepoint and checkpoint paths are displayed and editable in the Job Settings window.

Ensure OpenTelemetry dependencies are included in Cloudera Streaming Analytics - Kubernetes Operator

Cloudera Streaming Analytics - Kubernetes Operator includes all required dependencies to configure the OTel exporter.

To enable the Flink OpenTelemetry Metrics Exporter for a working Flink Java application, launch the application with the following additional FlinkDeployment options:

spec:
  flinkConfiguration:
    metrics.reporters: otel
    metrics.reporter.otel.factory.class: org.apache.flink.metrics.otel.OpenTelemetryMetricReporterFactory
    metrics.reporter.otel.exporter.endpoint: [*** ENDPOINT ***]:[*** PORT ***]
    metrics.reporter.otel.service.name: [*** SERVICE NAME ***]
    metrics.reporter.otel.service.version: [*** VERSION ***]

To enable the Flink OpenTelemetry Metrics Exporter for a job in Cloudera SQL Stream Builder, add the following SET commands to the job:

set metrics.reporters=otel;
set metrics.reporter.otel.factory.class=org.apache.flink.metrics.otel.OpenTelemetryMetricReporterFactory;
set metrics.reporter.otel.exporter.endpoint=[*** ENDPOINT ***]:[*** PORT ***];
set metrics.reporter.otel.service.name=[*** SERVICE NAME ***];
set metrics.reporter.otel.service.version=[*** VERSION ***];

For more information, see Using the OpenTelemetry Collector [Technical Preview].

More flexible error handling when executing jobs in Cloudera SQL Stream Builder

Implementing more flexible error handling for jobs executing on Kubernetes. Unrecoverable error checks parse errors to determine if the Flink job's main method was the source of an exception. Cloudera SQL Stream Builder jobs are only marked as failed if their state reaches a globally terminal state or if the error explicitly indicates that they're unrecoverable.

Configurable Kafka sampling settings at runtime in Cloudera SQL Stream Builder

Users can add the settings of the sampling Kafka instance on the global configuration UI in Cloudera SQL Stream Builder, previously only configurable through configuration files, which will be passed to Flink jobs.

The settings are only configurable in Cloudera SQL Stream Builder UI, and settings added to the configuration YAML files will be ignored.

To learn how to configure sampling in Cloudera SQL Stream Builder, see Sampling.

Temporary table and view support

Users can create and manage temporary tables and views in Cloudera SQL Stream Builder to define temporary objects suitable for session-scoped operations or intermediate results. Temporary tables and views simplify complex SQL queries and making job definitions more modular without affecting the global catalog.

To create a temporary table:

CREATE TEMPORARY TABLE [*** TABLE NAME ***] (
    id INT,
    name STRING,
    event_time TIMESTAMP(3)
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1',
    'fields.id.kind' = 'sequence',
    'fields.id.start' = '1',
    'fields.id.end' = '100',
    'fields.name.length' = '10'
);

To create a temporary view:

CREATE TEMPORARY VIEW [*** VIEW NAME ***] AS
SELECT
    order_id,
    SUM(amount) AS total_amount
FROM
    [*** PERSISTENT TABLE NAME ***]
GROUP BY
    order_id;