Using spark-submit drop-in migration toolPDF version

Installing the cde-env tool

You can install the cde-env tool as an Administrator or as a normal user. Cloudera recommends you to install the tool as an Administrator in the /opt/cloudera/bin folder so that all the users in the host can access the tool.

  1. Install the tool by copying the required binary and script files to the /opt/cloudera/bin folder so that the migration tool can run in the current gateway host.
    $ ./cde-env.sh enable-spark-submit-proxy
  2. Enable the installed binary and script files by granting access to those files for all users on the host using one of the following options.
    • Link the existing /usr/bin/spark-submit and /usr/bin/spark3-submit.
      $ ln -s /opt/cloudera/bin/cde /usr/bin/cde
      $ ln -s /opt/cloudera/bin/cde-env.sh /usr/bin/cde-env.sh
      $ ln -s -f /opt/cloudera/bin/spark-submit /usr/bin/spark-submit
      $ ln -s -f /opt/cloudera/bin/spark3-submit /usr/bin/spark3-submit

      or

    • Update PATH to point to the new installation location at the host level.
      $ export PATH=/opt/cloudera/bin:$PATH