Known Issues and Limitations in Cloudera Data Science Workbench 1.7.2

Pods not ready in cluster default ['role/<pod_name>'].

Cloudera Bug: DSE-6855

• Third-party security and orchestration software (such as McAfee, Tanium, Symantec) can lead to CDSW crashing randomly

  Workaround: Disable all third-party security agents on CDSW hosts.

  Cloudera Bug: DSE-8550

• High I/O utilization on the application block device can cause the application to stall or become unresponsive. Users should read and write data directly from HDFS rather than staging it in their project directories.

• Installing ipywidgets or a Jupyter notebook into a project can cause Python engines to hang due to an unexpected configuration. The issue can be resolved by deleting the installed libraries from the R engine terminal.

• Logs generated by a browser IDE do not appear within the IDE. They are displayed in the Logs tab for the session.

  Cloudera Bug: DSE-6570

• Sessions with Browser IDEs running do not adhere to the limit set in IDLE_MAXIMUM_MINUTES. Session logs show the warning message that states that the idle session will timeout, but the timeout does not occur. The session continues to run and consume resources until the timeout set in SESSION_MAXIMUM_MINUTES is reached. Ensure that you manually stop a session after you are finished, so that the resources are available to other users.

  Cloudera Bug: DSE-6651

• Sessions with Browser IDEs running time out with no warning after the time limit set in SESSION_MAXIMUM_MINUTES is reached, regardless of whether or not the session is idle. Periodically stop the browser IDE and session manually to avoid reaching SESSION_MAXIMUM_MINUTES.

  Cloudera Bug: DSE-6652

• The lack of a ROOT CA certificate can cause issues with terminals and the Jupyter editor after upgrading CDSW.

  Problem: After upgrading from CDSW version 1.5 to version 1.7.1, the terminal does not open for any kernel, and the Jupyter notebook does not work.

  Workaround: In CDSW, go to Admin > Security, and paste the internal CA root certificate file contents directly into the Root CA configuration field. You should be able to launch a new session and start the terminal or launch the Jupyter editor. It is not necessary to restart CDSW. This procedure is described at Configuring Custom Root CA Certificate

• Configuring duplicate mount points in the site admin panel (Admin > Engines > Mounts) results in sessions crashing in the workbench.

  Cloudera Bug: DSE-3308

• Spawning remote workers fails in R when the env parameter is not set. For more details, see Distributed Computing with Workers.

  Cloudera Bug: DSE-3384

• Autofs mounts are not supported with Cloudera Data Science Workbench.

  Cloudera Bug: DSE-2238

• When using Conda to install Python packages, you must specify the Python version to match the Python versions shipped in the engine image (2.7.11 and 3.6.1). If not specified, the conda-installed Python version will not be used within a project. Pip (pip and pip3) does not face this issue.

• When engine version 8 (or higher) is used, and the Allow containers to run as root property is disabled, the creation of containers that run with root privileges is prevented. Additionally, the elevation of privileges from the cdsw user to root (for example, using a setuid binary) is also prevented.

  As a result, running the ping command, which is actually a setuid binary, will fail in engine 8 (or higher) when Allow containers to run as root property is disabled.

  $ ping www.google.com
  Ping: icmp open socket: Operation not permitted.

• (If quotas are enabled) Experiments that are stuck in the Scheduled state due to lack of resources do not automatically start even if you free up existing resources.

  Workaround: Stop the experiment that is stuck in the Scheduled state. Then manually reschedule the experiment.

  Cloudera Bug: DSE-8736

• Experiments do not store snapshots of project files. You cannot automatically restore code that was run as part of an experiment.

• Experiments will fail if your project filesystem is too large for the Git snapshot process. As a general rule, any project files (code, generated model artifacts, dependencies, etc.) larger than 50 MB must be part of your project's .gitignore file so that they are not included in snapshots for experiment builds.

• Experiments cannot be deleted. As a result, be conscious of how you use the track_metrics and track_file functions.

  • Do not track files larger than 50MB.
  • Do not track more than 100 metrics per experiment. Excessive metric calls from an experiment may cause Cloudera Data Science Workbench to hang.

• The Experiments table will allow you to display only three metrics at a time. You can select which metrics are displayed from the metrics dropdown. If you are tracking a large number of metrics (100 or more), you might notice some performance lag in the UI.

• Arguments are not supported with Scala experiments.

• The track_metrics and track_file functions are not supported with Scala experiments.

• The UI does not display a confirmation when you start an experiment or any alerts when experiments fail.

• Job notification emails fail intermittently when attachments are included. Emails are delivered either with blank attachments or no attachments at all.

  Cloudera Bug: DSE-9469, DSE-8806

• Cloudera Data Science Workbench does not support changing your API key, or having multiple API keys.

• Currently, you cannot use the Jobs API to create a job, stop a job, or get the status of a job.

• Jobs pipeline visualization in CML and CDSW no longer shows dependent jobs. Dependencies only show the first step in the chain. Previously (before upgrade to 1.7), the UI displayed the whole chain of jobs. Jobs still run in the correct order, but the UI is no longer clear. Associated Bug: DSE-8003

Applications time out after the number of minutes specified by SESSION_MAXIMUM_MINUTES have passed.

• Known Issues with Model Builds and Deployed Models

  • Unable to create a model with the name of a deleted Model.

    Workaround: For now, Models shall have unique names across the lifespan of the cluster installation.

    Cloudera Bug: DSE-4237

  • Re-deploying or re-building models results in model downtime (usually brief).

  • Re-starting Cloudera Data Science Workbench does not automatically restart active models. These models must be manually restarted so they can serve requests again.

    Cloudera Bug: DSE-4950

  • Model deployment will fail if your project filesystem is too large for the Git snapshot process. As a general rule, any project files (code, generated model artifacts, dependencies, etc.) larger than 50 MB must be part of your project's .gitignore file so that they are not included in snapshots for model builds.

  • Model builds will fail if your project filesystem includes a .git directory (likely hidden or nested). Typical build stage errors include:

    Error: 2 UNKNOWN: Unable to schedule build: [Unable to create a checkpoint of current source: [Unable to push sources to git server: ...

    To work around this, rename the .git directory (for example, NO.git) and re-build the model.

    Cloudera Bug: DSE-4657

  • JSON requests made to active models should not be more than 5 MB in size. This is because JSON is not suitable for very large requests and has high overhead for binary objects such as images or video. Call the model with a reference to the image or video, such as a URL, instead of the object itself.

  • Any external connections, for example, a database connection or a Spark context, must be managed by the model's code. Models that require such connections are responsible for their own setup, teardown, and refresh.

  • Model logs and statistics are only preserved so long as the individual replica is active. Cloudera Data Science Workbench may restart a replica at any time it is deemed necessary (such as bad input to the model).

• Limitations

  • Scala models are not supported.

  • Spawning worker threads is not supported with models.

  • Models deployed using Cloudera Data Science Workbench are not highly-available.

  • Dynamic scaling and auto-scaling are not currently supported. To change the number of replicas in service, you will have to re-deploy the build.

• CDSW cannot launch sessions due to connection errors resulting from a segfault

  Sample error:

  transport: Error while dialing dial tcp 100.77.93.252:20051: connect: connection refused

  Workaround: Enable IPv6 on all CDSW hosts

  1. Double-check that IPv6 is currently disabled during boot time, i.e. ipv6.disable should be equal to 1.

     $ dmesg 
     [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.el7.x86_64 root=UUID=3e109aa3-f171-4614-ad07-c856f20f9d25 ro console=tty0 crashkernel=auto console=ttyS0,115200 ipv6.disable=1

     $ cat /proc/cmdline
     .....ipv6.disable=1

  2. Edit /etc/default/grub and delete the ipv6.disable=1 entry from GRUB_CMDLINE_LINUX. For example:

     GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel=auto rd.lvm.lv=rhel/root"

  3. Run the grub2-mkconfig command to regenerate the grub.cfg file:

     grub2-mkconfig -o /boot/grub2/grub.cfg

     Alternatively, on UEFI systems, you would run the following command:

     grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

  4. Follow the above steps for both CDSW Master and Worker nodes.
  5. Stop the Cloudera Data Science Workbench service.
  6. Reboot all the Cloudera Data Science Workbench hosts to enable IPv6 support.
  7. Start the Cloudera Data Science Workbench service. Run dmesg on the CDSW hosts to ensure there are no segfault errors seen.

  Cloudera Bug: DSE-7238, DSE-7455

• Custom /etc/hosts entries on Cloudera Data Science Workbench hosts do not propagate to sessions and jobs running in containers.

  Cloudera Bug: DSE-2598

• Initialisation of Cloudera Data Science Workbench (cdsw init) will fail if localhost does not resolve to 127.0.0.1.

• Cloudera Data Science Workbench does not support DNS servers running on 127.0.0.1:53. This IP address resolves to the container localhost within Cloudera Data Science Workbench containers. As a workaround, use either a non-loopback address or a remote DNS server.
• Kubernetes throws the following error when /etc/resolv.conf lists more than three domains:

  Resolv.conf file '/etc/resolv.conf' contains search line consisting of more than 3 domains!

  Due to a limitation in the libc resolver, only two DNS servers are supported in /etc/resolv.conf. Kubernetes uses one additional entry for the cluster DNS.

• If custom quota for a user is enabled, and the quotas feature is then disabled, the custom quota setting continues to remain in effect. That is, even if the quotas feature is disabled, the user will still see the cpu/memory limit reached error when they reach the previously set custom quota limit.

  Workaround: If you want to disable quotas, first manually delete each custom quota row and then switch Quotas toggle to OFF. To remove custom quotas, click the vertical ellipses at the end of each custom quota row and choose Remove.

  Cloudera Bug: DSE-9063

• (If quotas are enabled) Experiments that are stuck in the Scheduled state due to lack of resources do not automatically start even if you free up existing resources.

  Workaround: Stop the experiment that is stuck in the Scheduled state. Then manually reschedule the experiment.

  Cloudera Bug: DSE-8736

• Environment variables with the dollar ($) character are not parsed correctly by CDSW. For example, if you set PASSWORD="pass$123" in the project environment variables, and then try to read it using the echo command, you see the following output: pass23

  Workaround: Use one of the following commands to print the $ sign:

  echo 24 | xxd -r -p
  or
  echo JAo= | base64 -d

  Insert the value of the environment variable by wrapping it in the command substitution using $() or ``. For example, if you want to set the environment variable to ABC$123, specify:

  ABC$(echo 24 | xxd -r -p)123
  or
  ABC`echo 24 | xxd -r -p`123

• You see the HTTP 404 error on navigating to other tabs within the CDSW web UI after updating the project name on the Project Settings page. This is because the backend API still uses the older project name.

  Workaround: After you save the new project name, manually reload the webpage before navigating to the other tabs or pages within the CDSW web UI.

  Cloudera Bug: DSE-11911

• In some cases, the application switcher (grid icon) does not show any other applications, such as Hue or Ranger.

  Cloudera Bug: DSE-865

• Scala sessions hang when running large scripts (longer than 100 lines) in the Workbench editor.

  Workaround 1:

  Execute the script in manually-selected chunks. For example, highlight the first 50 lines and select Run > Run Line(s).

  Workaround 2:

  Restructure your code by moving content into imported functions so as to bring the size down to under 100 lines.

• The R engine is unable to display multi-byte characters in plots. Examples of multi-byte characters include languages such as Korean, Japanese, and Chinese.

  Workaround: Use the showtext R package to support more fonts and characters. For example, to display Korean characters:

  install.packages('showtext')
  library(showtext)
  font_add_google("Noto Sans KR", "noto")
  showtext_auto()

  Cloudera Bug: DSE-7308

• In a scenario where 100s of users are logged in and creating processes, the nproc and nofile limits of the system may be reached. Use ulimits or other methods to increase the maximum number of processes and open files that can be created by a user on the system.

• When rebooting, Cloudera Data Science Workbench hosts can take a significant amount of time (about 30 minutes) to become ready.

• Long-running operations such as fork and clone can time out when projects are large or connections outlast the HTTP timeouts of reverse proxies.

• The Scala kernel does not support auto-complete features in the editor.

• Scala and R code can sometimes indent incorrectly in the workbench editor.

  Cloudera Bug: DSE-1218

• The Cloudera Data Visualization application quits when it hits a session timeout.

  Workaround: Upgrade to CDSW 1.8.x or later.

  Cloudera Bug: DSE-8943