Closing HiveWarehouseSession operations
You need to know how to release locks that Apache Spark operations puts on Apache Hive resources. An example shows how and when to release these locks.
Spark can invoke operations, such as
rdd(), on a DataFrame you
obtain from running a HiveWarehouseSession
.sql(). The Spark operations can lock Hive resources. You can
release any locks and resources by calling the HiveWarehouseSession
close() invalidates the
HiveWarehouseSession instance and you cannot perform any further operations on the
close()when you finish running all other operations on the instance of HiveWarehouseSession.
import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession._ val hive = HiveWarehouseSession.session(spark).build() hive.setDatabase("tpcds_bin_partitioned_orc_1000") val df = hive.sql("select * from web_sales") . . . //Any other operations .close()
You can also call
close()at the end of an iteration if the application is designed to run in a microbatch, or iterative, manner that does not need to share previous states.
No more operations can occur on the DataFrame obtained by