Cloudera Docs

Workers API

This section lists the functions available as part of the workers API.

Launch Workers

Launches worker engines into the cluster.

Syntax
launch_workers(n, cpu, memory, nvidia_gpu=0, kernel="python3", script="", code="", env={})
Parameters
  • n (int) - The number of engines to launch.
  • cpu (float) - The number of CPU cores to allocate to the engine.
  • memory (float) - The number of gigabytes of memory to allocate to the engine.
  • nvidia_gpu (int, optional) - The number of GPU's to allocate to the engine.
  • kernel (str, optional) - The kernel. Can be "r", "python2", "python3" or "scala". This parameter is only available for projects that use legacy engines.
  • script (str, optional) - The name of a Python source file the worker should run as soon as it starts up.
  • code (str, optional) - Python code the engine should run as soon as it starts up. If a script is specified, code will be ignored.
  • env (dict, optional) - Environment variables to set in the engine.
Example Usage

Python

import cdsw
workers = cdsw.launch_workers(n=2, cpu=0.2, memory=0.5, code="print('Hello from a CDSW Worker')")

R

library("cdsw") 
              workers <- launch.workers(n=2, cpu=0.2, memory=0.5, env="", code="print('Hello from a Cloudera AI Worker')")

List Workers

Returns all information on all the workers in the cluster.

Syntax
list_workers()

Stop Workers

Stops worker engines.

Syntax
stop_workers(*worker_id)
Parameter
  • worker_id (int, optional) - The ID numbers of the worker engines that must be stopped. If an ID is not provided, all the worker engines on the cluster will be stopped.