Experiments

The rest of this section describes the different stages in the lifecycle of an experiment - from launch to completion.

1. Launch Experiment

   In this step you will select a script from your project that will be run as part of the experiment, and the resources (memory/GPU) needed to run the experiment. The engine kernel will be selected by default based on your script. For detailed instructions on how to launch an experiment, see Running an Experiment (QuickStart).

2. Build

   When you launch the experiment, Cloudera Data Science Workbench first builds a new versioned engine image where the experiment will be executed in isolation. This new engine includes:

   • the base engine image used by the project (check Project > Settings)
   • a snapshot of the project filesystem
   • environmental variables inherited from the project.
   • packages explicitly specified in the project's build script (cdsw-build.sh)

     It is your responsibility to provide the complete list of dependencies required for the experiment via the cdsw-build.sh file. As part of the engine's build process, Cloudera Data Science Workbench will run the cdsw-build.sh script and install the packages or libraries requested there on the new image.

   Note that custom mounts or environment variables configured in cdsw.conf (such as NO_PROXY, HTTP(S)_PROXY, etc.) are still not passed to the container builds for experiments and models (even though they are applied to sessions, jobs, and deployed models/experiments).

   For details about the build process and examples on how to specify dependencies, see Engines for Experiments and Models.

3. Schedule

   Once the engine is built the experiment is scheduled for execution like any other job or session. Once the requested CPU/GPU and memory have been allocated to the experiment, it will move on to the execution stage.

   Note that if your deployment is running low on memory and CPU, your runs may spend some time in this stage.

4. Execute

   This is the stage where the script you have selected will be run in the newly built engine environment. This is the same output you would see if you had executed the script in a session in the Workbench console.

   You can watch the execution in progress in the individual run's Session tab.

   You can also go to the project Overview > Experiments page to see a table of all the experiments launched within that project and their current status.

   Run ID: A numeric ID that tracks all experiments launched on a Cloudera Data Science Workbench deployment. It is not limited to the scope of a single user or project.

The following steps describe how to launch an experiment from the Workbench console. In this example we are going to run a simple script that adds all the numbers passed as arguments to the experiment.

1. Go to the project Overview page.
2. Click Open Workbench.

3. Create/modify any project code as needed. You can also launch a session to simultaneously test code changes on the interactive console as you launch new experiments.

   As an example, you can run this Python script that accepts a series of numbers as command-line arguments and prints their sum.

   add.py

   import sys
   import cdsw
   
   args = len(sys.argv) - 1  
   sum = 0
   x = 1
   
   while (args >= x): 
       print ("Argument %i: %s" % (x, sys.argv[x]))
       sum = sum + int(sys.argv[x])
       x = x + 1
       
   print ("Sum of the numbers is: %i." % sum)

   To test the script, launch a Python session and run the following command from the workbench command prompt:

   !python add.py 1 2 3 4

4. Click Run Experiment. If you're already in an active session, click Run > Run Experiment. Fill out the following fields:

   • Script - Select the file that will be executed for this experiment.

   • Arguments - If your script requires any command line arguments, enter them here.

   • Engine Kernel and Engine Profile - Select the kernel and computing resources needed for this experiment.

   For this example we will run the add.py script and pass some numbers as arguments.

   
   

   
   
   

5. Click Start Run.

6. To track progress for the run, go back to the project Overview. On the left navigation bar click Experiments. You should see the experiment you've just run at the top of the list. Click on the Run ID to view an overview for each individual run. Then click Build.

   On this Build tab you can see realtime progress as Cloudera Data Science Workbench builds the Docker image for this experiment. This allows you to debug any errors that might occur during the build stage.

   
   

   
   
   

7. Once the Docker image is ready, the run will begin execution. You can track progress for this stage by going to the Session tab.

   For example, the Session pane output from running add.py is:
   

   
   
   

8. (Optional) The cdsw library that is bundled with Cloudera Data Science Workbench includes some built-in functions that you can use to compare experiments and save any files from your experiments.

   For example, to track the sum for each run, add the following line to the end of the add.py script.

   cdsw.track_metric("Sum", sum)

   This will be tracked in the Experiments table:
   

   
   
   

   For more details, see Tracking Metrics and Saving Files.

The function accepts input in the form of key value pairs.

cdsw.track_metric(key, value)

cdsw.track_metric(“R_squared”, 0.79)

R

cdsw::track.metric(“R_squared”, 0.62)

cdsw.track_file('model.pkl')

cdsw::track.file('model.pkl')