Spark 1.6 to Spark 2.4 changes

A description of the change, the type of change, and the required refactoring provide the information you need for migrating from Spark 1.6 to Spark 2.4.

New Spark entry point SparkSession
There is a new Spark API entry point: SparkSession.
Dataframe API registerTempTable deprecated
The Dataframe API registerTempTable has been deprecated in Spark 2.4.
union replaces unionAll
The dataset and DataFrame API unionAll has been deprecated and replaced by union.
Empty schema not supported
Writing a dataframe with an empty or nested empty schema using any file format, such as parquet, orc, json, text, or csv is not allowed.
Referencing a corrupt JSON/CSV record
In Spark 2.4, queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column.
Dataset and DataFrame API explode deprecated
Dataset and DataFrame API explode has been deprecated.
CSV header and schema match
Column names of csv headers must match the schema.
Table properties support
Table properties are taken into consideration while creating the table.
Managed table location
Creating a managed table with nonempty location is not allowed.
Write to Hive bucketed tables
Rounding in arithmetic operations
Arithmetic operations between decimals return a rounded value, instead of NULL, if an exact representation is not possible.
Precedence of set operations
Set operations are executed by priority instead having equal precedence.
HAVING without GROUP BY
HAVING without GROUP BY is treated as a global aggregate.
CSV bad record handling
How Spark treats malformations in CSV files has changed.
Spark 2.4 CSV example
A CSV example illustrates the CSV-handling change in Spark 2.4.

Parent topic: Spark 1.6 to Spark 2.4 Refactoring