Hive on Spark Executor Nodes

Component Memory CPU Disk
Hive-on-Spark
  • Minimum: 16 GB
  • Recommended: 32 GB for larger data sizes

Individual executor heaps should be no larger than 16 GB so machines with more RAM can use multiple executors.

  • Minimum: 4 cores
  • Recommended: 8 cores for larger data sizes
Disk space requirements are driven by scratch space requirements for Spark spill.
For more information on how to reserve YARN cores and memory that will be used by Spark executors, refer to the Hive documentation.