Using Spark 2 from R

R users can access Spark 2 using sparklyr. Although Cloudera does not ship or support sparklyr, we do recommend using sparklyr as the R interface for Cloudera Machine Learning.

The spark_apply() function requires the R Runtime environment to be pre-installed on your cluster. This will likely require intervention from your cluster administrator. For details, refer the RStudio documentation.
  1. Install the latest version of sparklyr:
  2. Optionally, connect to a local or remote Spark 2 cluster:
    ## Connecting to Spark 2
    # Connect to an existing Spark 2 cluster in YARN client mode using the spark_connect function.
    system.time(sc <- spark_connect(master = "yarn-client"))
    # The returned Spark 2 connection (sc) provides a remote dplyr data source to the Spark 2 cluster.
    For a complete example, see Importing Data into Cloudera Machine Learning.