Rounding in arithmetic operations
Arithmetic operations between decimals return a rounded value, instead of NULL, if an exact representation is not possible.
Type of change: Property/Spark SQL changes
Spark 1.6
Arithmetic operations between decimals return a NULL value if an exact representation is not possible.
Spark 2.4
The following changes have been made:
- Updated rules determine the result precision and scale according to the SQL ANSI 2011.
- Rounding of the results occur when the result cannot be exactly represented with the specified precision and scale instead of returning NULL.
- A new config
spark.sql.decimalOperations.allowPrecisionLosswhich default to true (the new behavior) to allow users to switch back to the old behavior. For example, if your code includes import statements that resemble those below, plus arithmetic operations, such as multiplication and addition, operations are performed using dataframes.
from pyspark.sql.types import DecimalType
from decimal import DecimalAction Required
If precision and scale are important, and your code can accept a NULL value (if exact representation is not possible due to overflow), then set the following property to false. spark.sql.decimalOperations.allowPrecisionLoss = false
