Fixed Issues in Spark Atlas Connector

This section lists the issues in Spark Atlas Connector that are fixed in Cloudera Runtime 7.3.1 release, its service packs and cumulative hotfixes.

Cloudera Runtime 7.3.1.600 SP3 CHF1

CDPD-92198: Union of a partitioned table could result in redundant inputs in Spark process entity: 7.3.1.600 SP3 CHF1; This issue is now fixed by Spark generating only unique input paths for filtered and unioned partitioned tables. This reduces redundant entity creation and improves Atlas data accuracy.
CDPD-92197: Filtered queries on partitioned data show all inputs: 7.3.1.600 SP3 CHF1; This issue is fixed by refactoring the table input path collection to return only the filtered subset of data.
CDPD-90666: Atlas failing to show column lineage for Spark queries: 7.3.1.600 SP3 CHF1; This issue is fixed by updating the Spark Atlas Connector to ensure proper lineage reporting for all Spark SQL operations, including the insert overwrite operation.

There are no new fixed issues in this release.

CDPD-73828: SAC - spark_process entity contains huge strings in its fields: 7.3.1.400 SP2; Adds a new configuration property in Cloudera Manager to allow disabling the fields details and sparkPlanDescription in the spark_process entity, which could cause Atlas OOM errors if they contained too large strings.

There are no new fixed issues in this release.

There are no new fixed issues in this release.

There are no new fixed issues in this release.

There are no new fixed issues in this release.