Custom Spark SQL Warehouse Directory

If required, you must configure the Spark SQL warehouse location in the upgraded cluster.

If Applicable: Configure Custom Spark SQL Warehouse Directory

Since Spark 2.0.0, Spark references spark.sql.warehouse.dir as the default Spark SQL Warehouse location. In HDP 2.6.5.x, the standard value for the spark.sql.warehouse.dir property is /apps/hive/warehouse. On an Ambari cluster with HDP 2.6.5.x, this property must be set manually in both Custom spark2-defaults and Custom spark2-thrift-sparkconf. When upgrading from HDP 2.6.5.x to HDP intermediate bits, Ambari migrates the spark.sql.warehouse.dir property to Advanced spark2-defaults and Advanced spark2-thrift-sparkconf, and changes the value of this property to /apps/spark/warehouse. This is done to accommodate the separate Spark and Hive catalogs introduced in the HDP intermediate bits. If you used a custom setting for the spark.sql.warehouse.dir property in HDP 2.6.5.x, the Ambari upgrade to HDP intermediate bits ignores the custom setting and sets the value of the spark.sql.warehouse.dir property to /apps/spark/warehouse in both Advanced spark2-defaults and Advanced spark2-thrift-sparkconf.

If you want to use a custom Spark SQL warehouse after upgrading to the HDP intermediate bits, select Spark2 > Configs, then use Add Property in Advanced spark2-defaults and Advanced spark2-thrift-sparkconf to update the value of the spark.sql.warehouse.dir property with the custom setting.

If Hive is configured to run as a custom user, you must change the ownership of your new Hive warehouse directory to be owned by that custom user. hdfs dfs -chown cstm-hive /warehouse/tablespace/managed/hive/