You can review the changes in certain features or functionalities of components that
have resulted in a change in behavior from the previously released version to this version of
Cloudera Runtime 7.2.18.1101.
Atlas
- Summary: Spark plan is optional in Spark process entity sent to Atlas
- Previous behavior:
- Atlas would experience an Out-of-Memory error during processing if the
spark_process entity's details and sparkPlanDescription fields contained too large strings.
- New behavior:
- A new configuration property
atlas.spark.plan.enabled was added in Cloudera Manager to enable users to enable or disable the details and sparkPlanDescription fields. (Default value: true.)
- When set to
true (default), the fields can be used as before. When set to false, it reduces memory usage, but removes the fields from the Atlas UI.
- Summary:
A new option to avoid sending details
and sparkPlanDescription in the Spark process entity is introduced.
- Previous behavior:
The spark_process entity
attributes details and sparkPlanDescription are populated
with query plan details, which can contain a large amount of text, often in megabytes. This
amount of data can incur unnecessary processing costs.
- New behavior:
The atlas.spark.plan.enabled is set
to true by default. Set it to false to send the details and
sparkPlanDescription attributes in the Spark process entity. When these
attributes are not sent, the cost of having large amount of data processed in Atlas is
avoided.
