Creating and updating Apache Spark jobs using the CLI

The following example demonstrates how to create a Spark application in Cloudera Data Engineering (CDE) using the command line interface (CLI).

Make sure that you have downloaded the CLI client. For more information, see Using the Cloudera Data Engineering command line interface.
  1. Run the cde application create command as follows:
    cde job create --application-file <path_to_application_jar> --class <application_class> --name <job_name> --num-executors <num_executors> --type spark
    To see the full command syntax and supported options, run cde job create --help.
  2. Run cde application describe to verify that the application was created:
    cde job describe --name <job_name>
  3. If you want to update the application configuration, use the cde application update command.
    For example, to change the number of executors:
    cde job update --name test_job --num-executors 15
    To see the full command syntax and supported options, run cde job update --help.
  4. To verify the updated configuration, run cde job describe again:
    cde job describe --name <job_name>