Running Distributed ML Workloads on YARN
Cloudera Data Science Workbench 1.6 (and higher) allows you to run distributed machine learning workloads on the CDH/HDP cluster with frameworks such as TensorFlowOnSpark, H2O, XGBoost, and so on. This is similar to what you can already do with Spark workloads that run on the attached CDH/HDP cluster.
To support this, Cloudera Data Science Workbench now forwards three extra
ports from the host to each engine. The ports numbers for these ports are
stored in the following environmental variables:
-
CDSW_HOST_PORT_0 -
CDSW_HOST_PORT_1 -
CDSW_HOST_PORT_2
CDSW_IP_ADDRESS
and the host's IP address is stored in
CDSW_HOST_IP_ADDRESS.The information in these environmental variables can be used to make services running in the engine available to services running in the CDH cluster.
