Here is a sample command that launches the Spark shell on a YARN cluster:

./bin/spark-shell --num-executors 1 --executor-memory 512m --master yarn-client

To read data directly from the filesystem, construct a SQLContext. For an example that uses SQLContext and the Spark DataFrame API to access a JSON file, see Using the Spark DataFrame API.

To read data by interacting with the Hive MetaStore, construct a HiveContext. (HiveContext extends SQLContext.) For an example of the use of HiveContext (instantiated as val sqlContext), see Accessing ORC Files from Spark.