Accessing Spark SQL through the Spark shell
Use the following steps to access Spark SQL using the Spark shell.
The following sample command launches the Spark shell on a YARN cluster:
spark-shell --num-executors 1 --executor-memory 512m --master
To read data directly from the file system, construct a SQLContext. For an example that uses SQLContext and the Spark DataFrame API to access a JSON file, see Using the Spark DataFrame API.
To read data by interacting with the Hive Metastore, construct a HiveContext instance
(HiveContext extends SQLContext). For an example of the use of HiveContext (instantiated as
val sqlContext), see "Accessing ORC Files from Spark" in this guide.