Cloudera Data Science Workbench 1.4.x Requirements and Supported Platforms

A gateway node that is dedicated to running Cloudera Data Science Workbench must use one of the aforementioned supported versions even if the remaining CDH hosts in your cluster are running any of the other operating systems supported by Cloudera Enterprise.

The entire CDH cluster, including Cloudera Data Science Workbench gateway nodes, must use Oracle JDK. OpenJDK is not currently supported by Cloudera Data Science Workbench.

For Red Hat/CentOS deployments in particular, Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction must be enabled on the Cloudera Data Science Workbench gateway nodes.

For more specifics on the versions of Oracle JDK recommended for CDH and Cloudera Manager clusters, and instructions on how to install the Java Cryptography Extension, see the Cloudera Product Compatibility Matrix - Supported JDK Versions.

• All Cloudera Data Science Workbench gateway hosts must be part of the same datacenter and use the same network. Hosts from different data-centers or networks can result in unreliable performance.
• A wildcard subdomain such as *.cdsw.company.com must be configured. Wildcard subdomains are used to provide isolation for user-generated content.
• Disable all pre-existing iptables rules. While Kubernetes makes extensive use of iptables, it’s difficult to predict how pre-existing iptables rules will interact with the rules inserted by Kubernetes. Therefore, Cloudera recommends you use the following commands to disable all pre-existing rules before you proceed with the installation.

  sudo iptables -P INPUT ACCEPT
  sudo iptables -P FORWARD ACCEPT
  sudo iptables -P OUTPUT ACCEPT
  sudo iptables -t nat -F
  sudo iptables -t mangle -F
  sudo iptables -F
  sudo iptables -X

• Cloudera Data Science Workbench sets the following sysctl options in /etc/sysctl.d/k8s.conf:
  • net.bridge.bridge-nf-call-iptables=1
  • net.bridge.bridge-nf-call-ip6tables=1
  • net.ipv4.ip_forward=1
  Underlying components of Cloudera Data Science Workbench (Docker, Kubernetes, and NFS) require these options to work correctly. Make sure they are not overridden by high-priority configuration such as /etc/sysctl.conf.
• SELinux must either be disabled or run in permissive mode.
• Multi-homed networks are supported only with Cloudera Data Science Workbench 1.2.2 (and higher).
• No firewall restrictions across Cloudera Data Science Workbench or CDH hosts.
• Non-root SSH access is not allowed on Cloudera Data Science Workbench hosts.
• localhost must resolve to 127.0.0.1.
• Cloudera Data Science Workbench does not support DNS servers running on 127.0.0.1:53. This IP address resolves to the container localhost within Cloudera Data Science Workbench containers. As a workaround, use either a non-loopback address or a remote DNS server.

Cloudera Data Science Workbench does not support hosts or clusters that do not conform to these restrictions.

Cloudera Data Science Workbench hosts are added to your CDH cluster as gateway hosts. The recommended minimum hardware configuration for Cloudera Data Science Workbench gateway hosts is:

• CPU for Master and Worker Hosts: 16+ CPU (vCPU) cores

• RAM for Master and Worker Hosts: 32+ GB RAM

• Disk for Master Host
  • Root Volume: 100+ GB.

    The Cloudera Data Science Workbench installer temporarily decompresses the engine image file located in /etc/cdsw/images to the /var/lib/docker/tmp/ directory. If you are going to partition the root volume, make sure you allocate at least 20 GB to /var/lib/docker/tmp so that the installer can proceed without running out of space.

  • Application Block Device (mounted to /var/lib/cdsw): 1 TB
  • Docker Image Block Device: 1 TB
• Disk for Worker Hosts
  • Root Volume: 100+ GB.

    The Cloudera Data Science Workbench installer temporarily decompresses the engine image file located in /etc/cdsw/images to the /var/lib/docker/tmp/ directory. If you are going to partition the root volume, make sure you allocate at least 20 GB to /var/lib/docker/tmp so that the installer can proceed without running out of space.

  • Docker Image Block Device: 1 TB

    The Application Block Device mount is not required on worker hosts.

The default Cloudera Data Science Workbench engine currently includes Python 2.7.11 and Python 3.6.1. To use PySpark with lambda functions that run within the CDH cluster, the Spark executors must have access to a matching version of Python. For many common operating systems, the default system Python will not match the minor release of Python included in Data Science Workbench.

To ensure that the Python versions match, Python can either be installed on every CDH node or made available per job run using Spark’s ability to distribute dependencies. Given the size of a typical isolated Python environment and the desire to avoid repeated uploads from gateway hosts, Cloudera recommends installing Python 2.7 and 3.6 on the cluster if you are using PySpark with lambda functions.

You can install Python 2.7 and 3.6 on the cluster using any method and set the corresponding PYSPARK_PYTHON environment variable in your project. Cloudera Data Science Workbench 1.3 (and higher) include a separate environment variable for Python 3 sessions called PYSPARK3_PYTHON. Python 2 sessions continue to use the default PYSPARK_PYTHON variable. This will allow you to run Python 2 and Python 3 sessions in parallel without either variable being overridden by the other.

For an example on distributing Python dependencies dynamically, see Example: Distributing Dependencies on a PySpark Cluster.

Cloudera Data Science Workbench only supports the versions of Docker and Kubernetes that are shipped with each release. Upgrading Docker or Kubernetes, or running on third-party Kubernetes clusters is not supported.

On AWS, Cloudera Data Science Workbench must be used with persistent/long-running Apache Hadoop clusters only.