Spark 2.3 to Spark 2.4 changes

A description of the change, the type of change, and the required refactoring provide the information you need for migrating from Spark 2.3 to Spark 2.4.

Empty schema not supported
Writing a dataframe with an empty or nested empty schema using any file format, such as parquet, orc, json, text, or csv is not allowed.
CSV header and schema match
Column names of csv headers must match the schema.
Table properties support
Table properties are taken into consideration while creating the table.
Managed table location
Creating a managed table with nonempty location is not allowed.
Precedence of set operations
Set operations are executed by priority instead having equal precedence.
HAVING without GROUP BY
HAVING without GROUP BY is treated as a global aggregate.
CSV bad record handling
How Spark treats malformations in CSV files has changed.
Spark 2.4 CSV example
A CSV example illustrates the CSV-handling change in Spark 2.4.

Parent topic: Spark 2.3 to Spark 2.4 Refactoring