Cloudera Docs

Using PySpark

Apache Spark provides APIs in non-JVM languages such as Python. Many data scientists use Python because it has a rich variety of numerical libraries with a statistical, machine-learning, or optimization focus.

  • Running PySpark in a virtual environment
  • Running Spark Python applications