Cloudera Data Science Workbench Release Notes

This release fixed various bugs that are listed in Issues Fixed in Cloudera Data Science Workbench 1.8.1.

• Fixed R working directory in Rprofile.site. This issue is resolved by using Engine 13.
• Fixed an upgrade issue impacting database upgrade part when exceeding maxBuffer length can prevent cluster from coming up.

  Cloudera Bug: DSE-10120

• Fixed an issue where Applications can stuck in a stopped state after upgrading to CDSW 1.8.0.

  Cloudera Bug: DSE-12374

• Fixed an issue where an Application could be stuck with status starting and a user might not be able to restart or delete it.

  Cloudera Bug: DSE-12732

• Fixed an issue handling large responses from Model.

  Cloudera Bug: DSE-12310

• Fixed Admin Usage page refresh issue.

  Cloudera Bug: DSE-12370

• Fixed issue displaying latest engine image in case engine9 or earlier versions are also present.

  Cloudera Bug: DSE-12372

• Resolved issue with python scripts not being able to interact with git while running as Jobs.

  Cloudera Bug: DSE-12624

• The Options tab on the Project Settings page is now only viewable to the Administrator. This is because the Private and Team visibility options on the Options page are only viewable by the Administrator.

  Cloudera Bug: DSE-12027

• Fixed an issue that prevented Terminal Access in case of Sessions on a cluster behind firewall and/or with proxy settings.

  Cloudera Bug: DSE-12867

• Production Machine Learning

  Machine Learning (ML) lifecycle focused functionality enabling ML Engineers and Data Scientists to cut time to production for ML models from weeks to minutes and scale ML use cases without compromising enterprise security, maintainability, and governance standards.

• Ability to monitor model metrics

  CDSW allows you to track individual model predictions and analyze the metrics using custom code. Monitoring models’ functional and business performance requires specialized tooling and CDSW now includes native functionality to enable the storage and access of custom and arbitrary model metrics. Included as well is the ability to track individual predictions to ground truth ensuring models are performing optimally and compliantly.

• Improved model security

  Model REST endpoints now have additional security features that allow user-level access control to prevent unauthorized users from accessing the endpoints. This enables models to be served in a production ML environment without compromising security.

• Quota management is generally available

  Resource quotas provide constraints that limit the aggregate CPU, memory, and GPU resource consumption per user. Default quotas can be configured for a Workspace, which can be overridden on a per-user basis with Custom quotas. Quota management is now generally available.

• New Web Browser Timeout Settings for Admin and User

  The default Admin user timeout is one hour, regardless of activity. For users, the default timeout is 24 hours. Five minutes before the timeout is reached, a message displays, asking if the user wants to remain logged in. If the user clicks Dismiss or does not see the reminder, they have to log back in when their session expires. The timeout settings can be changed in Admin > Security.

• Updated Session Start view

  The user experience around starting a new CDSW session has been enhanced.

• Ability to add CDSW session metadata information

  Administrators can make it mandatory for the users to enter additional metadata before starting a CDSW session. Administrators can configure up to three fields to collect such additional metadata information for compliance purposes and can make these mandatory to be filled in. This metadata is captured and stored in the user_events database table and in the audit log as key-value pairs.

• Base Engine 13 support

  CDSW supports and deploys Base engine 13 that ships with both Python versions 2.7.18 and 3.6.10, and R version 3.6.3.

• Ability to use custom command-line arguments for sessions and jobs

  CDSW jobs run scripts, and the scripts need you to pass arguments while they are being executed. You can set these command-line arguments in the Engine’s Command Line Arguments field when you create a job. These command-line arguments are shared by CDSW sessions, jobs, and experiments in a project. The arguments can be accessed inside your scripts the same way CLI arguments are normally used.

• CDSW displays engine total resource consumption

  The Engine Profile now reflects the total resource consumption required to run engines. This means that apart from the “effective” resources used for computation, the additionally required “sidecar” resources are also counted, and enforced when using quotas.

• Syntax highlighting support for additional programming languages

  CDSW now supports syntax highlighting for the following languages when you are previewing a file: XML, CSS, JS, HTML, CoffeeScript, SQL, and CSV.

• Ability to configure Windows line endings for Git integration

  Previously, when CDSW Brackets editor interacted with files having Windows line endings, the Brackets editor converted all the Windows line endings to Linux line endings distorting the git log. The behavior to use Windows line endings is now configurable by the Administrator on a site level.

• Cross-Origin Resource Sharing (CORS) is disabled by default

  CORS is now disabled by default. You can enable CORS if you have web applications on different domains that need cross-domain communication with the CDSW API.

• OS support changes

  CentOS 7.2 is no longer supported.

• Improved platform stability and performance
  • Fixed a number of potential security vulnerabilities in the base operating systems of our Kubernetes pods
  • Upgraded Kubernetes to 1.14
  • Fixed the Sweet32 Birthday attack vulnerability within the Kubernetes infrastructure that supports CDSW

    Cloudera Bug: DSE-9061

• A number of CDSW pods now run as non-root users

  Cloudera Bug: DSE-10493, DSE-7357

• Changed how the web sessions are handled
  • CDSW provides an option to refresh the session five minutes before your web session expires so that your active users can continue their session without the cookie expiring.

    Cloudera Bug: DSE-11773

  • Administrators can now disable the ability to have multiple concurrent sessions. This prevents users from logging in to the same account with multiple browsers or computers. By default, users will still be able to have concurrent web browser sessions. This may change in a future release.

    Cloudera Bug: DSE-8109

• Accessibility fixes
  • The CDSW UI has been polished with many accessibility fixes. The main application strives to be WCAG2AA compliant with the exception of the workbench editor.

    Cloudera Bug: DSE-4490

• Re-starting Cloudera Data Science Workbench does not automatically restart active models. These models must be manually restarted so they can serve requests again.

  Cloudera Bug: DSE-4950

• Fixed the CDSW restart issue on multi-node deployments.

  Cloudera Bug: DSE-9587, DSE-9663

• Cloudera Manager now generates an "https" link for the CDSW web UI instead of "http" if you have enabled TLS for your CDSW cluster.

  Cloudera Bug: DSE-8698

• (If quotas are enabled) Experiments no longer remain stuck in the Scheduled state due to lack of resources. You no longer need to manually reschedule the experiment.

  Cloudera Bug: DSE-8736

• Job notification emails no longer fail intermittently when attachments are included.

  Cloudera Bug: DSE-9469, DSE-8806

• Fixed an issue on multi-node CDSW 1.7.1 deployments where the CDSW Web UI would not automatically come up after upgrading to CDSW 1.7.1.

  Cloudera Bug: DSE-9587

• Fixed an issue where environmental variables set at the site admin level and at the project level would not get passed down to experiments and models at container build time.

  Cloudera Bug: DSE-6708

• Fixed an issue where new users could not log in when the Require invitation to sign up checkbox was enabled. Additionally, the Test LDAP Configuration form did not return any error message if the user being tested wasn't already synced to the local CDSW database.

  Cloudera Bug: DSE-3829

• Fixed an issue where license files could not be uploaded to CDSW through the UI.

  Cloudera Bug: DSE-9874, DSE-8865

• Bring Your Own Editor

  You can now take advantage of all the benefits of Cloudera Data Science Workbench while using an editor you are familiar with. This feature supports third-party IDEs that run on your local machine like PyCharm and browser-based IDEs such as Jupyter. Base Image v8 ships with Jupyter preconfigured and can be selected from the Start Session menu.

  For details, see Editors.

• Multiple Cloudera Data Science Workbench Deployments

  You can now have multiple Cloudera Data Science Workbench CSD deployments associated with one instance of Cloudera Manager.

  For details, see Multiple Cloudera Data Science Workbench Deployments.

• Audits

  Cloudera Data Science Workbench logs specific events, such as user logins and sharing, that you can view by querying a database. For more information, see Monitoring User Events and Tracked User Events.

• Expanded Support for Distributed Machine Learning

  Cloudera Data Science Workbench 1.6 (and higher) allows you to run distributed workloads with frameworks such as TensorFlowOnSpark, H2O, XGBoost, and so on. This is similar to what you can already do with Spark workloads that run on the attached CDH/HDP cluster. For details, see Running Distributed ML Workloads on YARN.

• cdswctl CLI Client

  The cdswctl client provides an additional way to interact with your Cloudera Data Science Workbench deployment to perform certain actions. For example, you can use the cdswctl client to start an SSH-endpoint on your local machine and then connect a local IDE, such as PyCharm, to Cloudera Data Science Workbench.

  You can download cdswctlfrom the Cloudera Data Science Workbench web UI and use it from your local machine. Note that this client differs from the cdsw CLI tool used to run commands such as cdsw status, which exists within the Cloudera Data Science Workbench deployment.

  For details, see cdswctl Command Line Interface Client.

• Status and Validate Commands

  The CDSW service in Cloudera Manager now includes two new commands that can be used to assess the status of your Cloudera Data Science Workbench deployment: Status and Validate. They are the equivalent of the cdsw status and cdsw validate commands that are available via the CLI.

  For details, see Checking the Status of the CDSW Service.

• Experiments

  • If your cluster has been equipped with GPUs, you can now use GPUs to run experiments on Cloudera Data Science Workbench.
  • Tracked experiment files now refresh and appear automatically on the Overview page for a run of an experiment. Previously, you had to manually refresh the page after an experiment completes.

• Command Line Interface (CLI) Changes - RPM Deployments only

  • The cdsw reset command has been removed and replaced by the cdsw stop command.
  • The cdsw init command has been removed and replaced by the cdsw start command.

  For details on how these commands behave on the master and worker hosts, refer to the Cloudera Data Science Workbench Command Line Reference.

• Kubernetes and Weave

  Kubernetes has been upgraded to version 1.13.5. Weave Net has been upgraded to version 2.5.1. This upgrade resolves Weave issue #2934.

• Logs

  • Staging Directory

    You can now configure the temporary directory that Cloudera Data Science Workbench uses to stage logs when collecting a diagnostic bundle. Old logs in the directory are deleted when a new diagnostic bundle is collected or when the size grows larger than 10 MB.

  • Logs tab

    Running sessions now display a Logs tab. This tab displays engine logs and, if applicable, Spark logs for the running session. Previously, if you wanted to access these logs, that required logging into the Cloudera Data Science Workbench host(s) and the Spark server.

    For details, see Diagnostic Bundles.

• Operating System

  Cloudera Data Science Workbench 1.6 supports RHEL and CentOS 7.6.

• Workload Scheduling Changes

  • Starting with version 1.6, Cloudera Data Science Workbench allows you to specify a list of CDSW gateway hosts that are labeled as Auxiliary Nodes. These hosts will be deprioritized during workload scheduling. That is, they will be chosen to run workloads that can’t be scheduled on any other hosts. For example, sessions with very large resource requests, or when the other hosts are fully utilized.

    For details, see Customize Workload Scheduling.

  • Reserve Master Host

    Cloudera Data Science Workbench 1.4.3 introduced a new feature that allowed you to reserve the CDSW Master host for running internal application components. Starting with version 1.6, this feature can be enabled on CSD-based deployments using the Reserve Master Host property in Cloudera Manager. Safety valves are no longer needed.

    For details, see Reserving the Master Host for Internal CDSW Components.

• Security
  • FreeIPA Support

    In addition to MIT Kerberos and Active Directory, Cloudera Data Science Workbench now also supports FreeIPA as an identity management system. For details, see Configure FreeIPA.

  • New User Role - Operator

    Version 1.6 includes a new access role called Operator. When a user is assigned the Operator role on a project, they will be able to start and stop pre-existing jobs and will have view-only access to project code, data, and results.

  • Restricting User-Controlled Kubernetes Pods

    Cloudera Data Science Workbench 1.6 includes three new properties that allow you to control the permissions granted to user-controlled Kubernetes pods. An example of a user-controlled pod is the engine pod, which provides the environment for sessions, jobs, etc. These pods are launched in a per-user Kubernetes namespace. Since the user has the ability to launch arbitrary pods, these settings restrict what those pods can do.

    For details, see Restricting User-Controlled Kubernetes Pods.

  • LDAP/SAML Configuration Changes

    Previously, if you wanted to grant the site administrator role to users of an LDAP/SAML group, that group had to be listed under 2 properties: LDAP/SAML Full Administrator Groups and LDAP/SAML User Groups. If a group was only listed under LDAP/SAML Full Administrator Groups, and not under LDAP/SAML User Groups, users of that group would not be able to log in to CDSW.

    With version 1.6, you do not need to list the admin groups under both properties. Users belonging to groups listed under LDAP/SAML Full Administrator Groups will be able to log in and have site administrator access to Cloudera Data Science Workbench as expected.

  • Project and Team Creation

    Site administrators can now restrict whether or not users can create projects or teams with the following properties on the Settings page:

    • Allow users to create projects
    • Allow users to create teams

    For details, see User Access to Features.

  • Session Tokens

    The method by which the Cloudera Data Science Workbench web UI session tokens are stored has been hardened. Users must log out of the Cloudera Data Science Workbench web UI and back in after upgrading to version 1.6.0.

  • Sharing

    Site administrators can now control whether consoles can be shared with the Allow console output sharing property on the Admin > Security page. Disable this property to remove the Share button from the project workspace and workbench UI as well as disable access to all shared console outputs across the deployment. Note that re-enabling this property does not automatically grant access to previously shared consoles. You will need to manually share each console again.

  • TLS/SSL

    Cloudera Data Science Workbench now defaults to using TLS 1.2. The default cipher suites have also been upgraded to Mozilla's Modern cipher suites.

• IPv6 Requirement

  Cloudera Data Science Workbench 1.6.x requires you to enable IPv6 on all CDSW gateway hosts. For instructions, refer the workaround provided here: Known Issue: CDSW cannot start sessions due to connection errors.

• Spark UI

  The Spark UI is now available as a tab within running sessions that use Spark.

For a complete list, see Known Issues and Limitations in Cloudera Data Science Workbench 1.8.1.

• Cloudera Enterprise 6.1 Support

  Cloudera Data Science Workbench is now supported with Cloudera Manager 6.1.x (and higher) and CDH 6.1.x (and higher). For details, see Cloudera Manager and CDH Requirements.

• Cloudera Data Science Workbench on Hortonworks Data Platform (HDP)

  Cloudera Data Science Workbench can now be deployed on HDP 2.6.5 and HDP 3.1.0. For an architecture overview and installation instructions, see Deploying Cloudera Data Science Workbench 1.8.1 on Hortonworks Data Platform.

• Security Enhancements

  • Allow Site Administrators to Enable/Disable Project Uploads and Downloads - By default, all Cloudera Data Science Workbench users are allowed to upload and download files to/from a project. Version 1.5 introduces a new feature flag that allows site administrators to hide the UI features that let users upload and download project files.

    Note that this feature flag only removes the relevant features from the Cloudera Data Science Workbench UI. It does not disable the ability to upload and download files through the backend web API.

    For details on how to enable this feature, see Disabling Project File Uploads and Downloads.

• OpenJDK Support

  Cloudera Data Science Workbench now supports Open JDK 8 on Cloudera Enterprise 5.16.1 (and higher). For details, see Product Compatibility Matrix - Supported JDK Versions.

• Engines

  • Base engine upgraded with a new version of R - 3.5.1 (Base Image v7)

  • Debugging Improvements - Previously, engines and their associated logs were deleted immediately after an exit or a crash. With version 1.5, engines are now available for about 5 minutes after they have ended to allow you to collect the relevant logs.

    Additionally, when an engine exits with a non-zero status code, the last 50 lines from the engine's logs are now printed to the Workbench console. Note that a non-zero exit code and the presence of engine logs in the Workbench does not always imply a problem with the code. Events such as session timeouts and out-of-memory issues are also assigned non-zero exit codes and will display engine logs.

• Installation and Upgrade

  • New Configuration Parameters - Version 1.5 includes three new configuration parameters that can be used to specify the type of distribution you are running, the directory for the installed packages/parcels, and the path where Anaconda is installed (for HDP only).
    • DISTRO
    • DISTRO_DIR
    • ANACONDA_DIR
    Details and sample values for these properties have been added to the relevant installation topics for CDH and HDP.
  • DOCKER_TMPDIR changed to /var/lib/cdsw/tmp/docker - Previously the Cloudera Data Science Workbench installer would temporarily decompress the base engine image file to the /var/lib/docker/tmp directory. Starting with version 1.5, the installer will use the /var/lib/cdsw/tmp/docker directory instead. Make sure you have an Application block device mounted to /var/lib/cdsw as recommended so that installation/upgrade can proceed without issues.
  • Improved Validation Checks - Improved the validation checks run by the installer and the error messages that are displayed during the installation process. Cloudera Data Science Workbench now:
    • Checks that space is available on the root directory, the Application Block Device and the Docker Block Device(s).
    • Checks that DNS forward and reverse lookup works for the Cloudera Data Science Workbench Domain and Master IP address provided.
    • Displays better error messages for the cdsw status and cdsw validate commands for easier debugging.

• Command Line

  • cdsw logs - Previously, the cdsw logs command generated two log bundles - one in plaintext and one with sensitive information redacted. With version 1.5, the command now generates only a single bundle that has all the sensitive information redacted by default.

    To turn off redaction of log files for internal use, you can use the new --skip-redaction option as follows:

    cdsw logs --skip-redaction

• Networking

  • Cloudera Data Science Workbench now uses DNS hostnames (not IP addresses) for internal communication between components. As a result, the wildcard DNS hostname configured for Cloudera Data Science Workbench must now be resolvable from both, the CDSW cluster, and your browser.

  • Cloudera Data Science Workbench now enables IPv4 forwarding (net.ipv4.conf.default.forwarding) during the installation process.

Deprecated Property - CDH Parcel Directory

• Fixed an issue with RPM installations where NO_PROXY settings were being ignored.

  Cloudera Bug: DSE-4444

• Fixed an issue where CDSW would not start because of IP issues with web pods. Version 1.5 fixes this by enabling IPv4 forwarding at startup.

  Cloudera Bug: DSE-4609

• Fixed an issue where engines would get deleted immediately after an exit/crash and engine logs did not persist which made it difficult to debug issues with crashes or auto-restarts.

  Cloudera Bug: DSE-4008, DSE-4417

• Fixed intermittent issues with starting and stopping Cloudera Data Science Workbench on CSD deployments.

  Cloudera Bug: DSE-4426, DSE-4829

• Fixed an issue where Cloudera Data Science Workbench was reporting incorrect file sizes for files larger than 2 MB.

  Cloudera Bug: DSE-4531, DSE-4532

• Fixed an issue where the Run New Experiment dialog box did not include the file selector and the Script name had to be typed in manually.

  Cloudera Bug: DSE-3650

• Fixed an issue where underlying Kubernetes processes were running out of resources leading to Out of Memory (OOM) errors. Cloudera Data Science Workbench now reserves compute resources for Kubernetes components.

  Cloudera Bug: DSE-4896, DSE-5001

• Fixed an issue where the PYSPARK3_PYTHON environment variable was not working as expected for Python 3 workloads.

  Cloudera Bug: DSE-4329

• Fixed an issue where Docker commands would fail on Cloudera Data Science Workbench engines that are not available locally (such as custom engine images) when an HTTP/HTTPS proxy was in use.

  Cloudera Bug: DSE-4427

• Fixed an issue where installation of the XML package would fail in the R kernel.

  Cloudera Bug: DSE-2201

For a complete list of the current known issues and limitations in Cloudera Data Science Workbench 1.5.x, see Known Issues in Cloudera Data Science Workbench 1.5.x.

• Reserve Master Host for Internal Application Components

  Cloudera Data Science Workbench now allows you to reserve the master host for running internal application components and services such as Livelog, the PostgreSQL database, and so on, while user workloads run exclusively on worker hosts.

  By default, the master host runs both, user workloads as well as the application's internal services. However, depending on the size of your CDSW deployment and the number of workloads running at any given time, it's possible that user workloads might dominate resources on the master host. Enabling this feature will ensure that CDSW's application components always have access to the resources they need on the master host and are not adversely affected by user workloads.

  For details on how to enable this feature, see Reserving the Master Host for Internal CDSW Components.

• Allow Only Session Creators to Execute Commands in Active Sessions

  By default, project contributors, project administrators, and site administrators have the ability to execute commands within your actively running sessions in the Workbench. Cloudera Data Science Workbench 1.4.3 introduces a new feature that allows site administrators to restrict this ability. When this feature is enabled, only the user that creates the session will be able to execute commands in that session. No other users, regardless of their permissions in the team or as project collaborators/administrators, will be able to execute commands on active sessions that were not created by them.

  For details on how to enable this feature, see Restricting Collaborator and Administrator Access to Active Sessions.

For a complete list of the current known issues and limitations in Cloudera Data Science Workbench 1.4.x, see Known Issues and Limitations in Cloudera Data Science Workbench 1.8.1.

• Operating System: Added support for RHEL / CentOS / Oracle Linux RHCK 7.5.

• Engines
  • Mounts - By default, host mounts (specified at Admin > Engines > Mounts) are loaded into engine containers with read-only permissions. With version 1.4.2, a new checkbox allows you to make these mounted directories available in engine containers with read-write permissions instead.
  • Engine upgrade (Base Image v6)
• Models
  • In Cloudera Data Science Workbench 1.4.0, model request sizes were limited to 100 KB. In version 1.4.2, this limit has now been increased to 5 MB. To take advantage of this higher threshold, you will need to upgrade to Cloudera Data Science Workbench 1.4.2 and rebuild your existing models.
• Security
  Added three new properties to the Admin > Security page that allow you to customize HTTP headers accepted by Cloudera Data Science Workbench.

  • Enable HTTP security headers
  • Enable cross-origin resource sharing (CORS)
  • Enable HTTP Strict Transport Security (HSTS)

  For details, see Configuring HTTP Headers for Cloudera Data Science Workbench.

For a complete list of the current known issues and limitations in Cloudera Data Science Workbench 1.4.x, see Known Issues and Limitations in Cloudera Data Science Workbench 1.8.1.

• Models and Experiments - Cloudera Data Science Workbench 1.4 extends the machine learning platform experience from research to production. Now you can use Cloudera Data Science Workbench to build, train, and deploy models in a unified workflow.

  • Experiments - Train and compare versioned, reproducible models

  • Models - Deploy and manage models as REST APIs to serve predictions

• External Authentication

  • LDAP/SAML users can now restrict access to Cloudera Data Science Workbench to specific LDAP/SAML groups. Additionally, you can now specify groups that should automatically be granted site administrator privileges when they log in to Cloudera Data Science Workbench. For details, see Configuring External Authentication with LDAP and SAML.

  • Cloudera Data Science Workbench now supports multiple identity provider signing certificates for SAML 2.0 authentication.

  • Cloudera Data Science Workbench now supports SAML 2.0 Errata 05 E43 for SAML 2.0 authentication.

• Projects and Workbench

  • Site administrators can now disable individual built-in template projects by using a checkbox in the Project Templates table at Admin > Settings. Only enabled project templates will be displayed in the dropdown menu when creating a new project.

  • The default .gitignore file that is created with each new project has been updated to:

    R
    node_modules
    *.pyc
    .*
    !.gitignore

  • Added support for multiple Terminal windows within a single session.

• Networking

  • Cloudera Data Science Workbench now supports DNS resolution of localhost to non-local IP address (not 127.0.0.1).

  • Cloudera Data Science Workbench now appends the following default values to the NO_PROXY parameter if any of the following properties are configured: HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY.

    "127.0.0.1,localhost,100.66.0.1,100.66.0.2,100.66.0.3,
    100.66.0.4,100.66.0.5,100.66.0.6,100.66.0.7,100.66.0.8,100.66.0.9,
    100.66.0.10,100.66.0.11,100.66.0.12,100.66.0.13,100.66.0.14,
    100.66.0.15,100.66.0.16,100.66.0.17,100.66.0.18,100.66.0.19,
    100.66.0.20,100.66.0.21,100.66.0.22,100.66.0.23,100.66.0.24,
    100.66.0.25,100.66.0.26,100.66.0.27,100.66.0.28,100.66.0.29,
    100.66.0.30,100.66.0.31,100.66.0.32,100.66.0.33,100.66.0.34,
    100.66.0.35,100.66.0.36,100.66.0.37,100.66.0.38,100.66.0.39,
    100.66.0.40,100.66.0.41,100.66.0.42,100.66.0.43,100.66.0.44,
    100.66.0.45,100.66.0.46,100.66.0.47,100.66.0.48,100.66.0.49,
    100.66.0.50,100.77.0.10,100.77.0.128,100.77.0.129,100.77.0.130,
    100.77.0.131,100.77.0.132,100.77.0.133,100.77.0.134,100.77.0.135,
    100.77.0.136,100.77.0.137,100.77.0.138,100.77.0.139"

• Installation Validation Checks - Improved validation checks run during the installation process. Cloudera Data Science Workbench now:
  • Verifies that the wildcard DNS subdomain has been configured.
  • Verifies that resolv.conf is not pointing to 127.0.0.1.
  • Validates iptables chains to ensure there are no custom rules being set.
  • Throws a warning if you are using a self-signed TLS certificate, an expired certificate, or if the certificate is not valid for the wildcard domain used for Cloudera Data Science Workbench.
• Command Line - Added a verbose option to the cdsw status command.

  cdsw status [-v|--verbose]

• Kubernetes has been upgraded to version 1.8.12.

Host Mounts are now Read-Only in Engines - Previously, mounts (specified at Admin > Engines > Mounts) were loaded into engine containers with read-write permissions.

Starting with version 1.4.0, mount points are now loaded into engines with read-only permissions.

• Fixed an issue where Git would timeout when cloning a project took too long. The timeout has now been increased to 60 seconds when creating a new project from Git.

  Cloudera Bug: DSE-3363

• Fixed an issue where manual parcel deployments could not detect parcel hash files with a .sha1 extension.

  Cloudera Bug: DSE-3301

• Fixed several usability issues (file create, save, and so on) with Internet Explorer 11.

  Cloudera Bug: DSE-3426, DSE-3434

• Fixed an issue where CSD installations would fail to recognize Oracle Linux 7.3 as a supported operating system.

  Cloudera Bug: DSE-3257

• Fixed an issue where Cloudera Data Science Workbench would hang with 100 percent CPU utilization.

  Cloudera Bug: DSE-3450

• Fixed a SAML 2.0 configuration issue where uploading the identity provider metadata XML file did not update identity provider signing certificate and/or SSO URL on Cloudera Data Science Workbench correctly.

  Cloudera Bug: DSE-3076

• Fixed an issue with SAML 2.0 authentication where the identity provider’s signature was not being validated correctly.

  Cloudera Bug: DSE-3694

• Fixed the Save As functionality in the project Workbench.

  Cloudera Bug: DSE-3870

• Fixed an issue where if a user had some files opened in the Workbench in a previous session, and those files no longer existed in the project filesystem, a File Not Found error would occur when opening the Workbench.

  Cloudera Bug: DSE-3835

For a complete list of the current known issues and limitations in Cloudera Data Science Workbench 1.4.x, see Known Issues and Limitations in Cloudera Data Science Workbench 1.8.1.

For a list of the current known issues and limitations, refer the documentation for version 1.3.x at Cloudera Data Science Workbench 1.3.x.

• Added support for SUSE Linux Enterprise Server 12 SP3.

• Site administrators can now add template projects that are customized for their organization's use-cases.

• Version 1.3 introduces a new environment variable for Python 3 sessions called PYSPARK3_PYTHON. Python 2 sessions will continue to use the default PYSPARK_PYTHON variable. This will allow you to configure distinct variables for Python 2 and Python 3 applications.

• In the Cloudera Manager CDSW service, the Wildcard DNS Domain property has been renamed to Cloudera Data Science Workbench Domain.

• Output for the cdsw version command now includes the type of deployment you are running – RPM or CSD.

• Added log4j and spark-defaults sample configuration to the PySpark and Scala template projects.

• Fixed an issue where the cdsw status command failed to run all the required system checks.

  Cloudera Bug: DSE-3070

• Session lists now include additional metadata to help distinguish between different sessions.

  Cloudera Bug: DSE-2814

• Pre-install validation checks have been improved to detect issues with iptables modules and Java settings.

  Cloudera Bug: DSE-2293

• Fixed an issue with the cdsw status command output when TLS is enabled.

  Cloudera Bug: DSE-3182

• CDS 2.2 Release 2 fixes the issue where a PySpark application could only be run once per active Workbench session.

  Cloudera Bug: CDH-58475

• Fixed an issue that prevented Bokeh plots from rendering.

  Cloudera Bug: DSE-3134

• Fixed an issue in Cloudera Data Science Workbench 1.2.2 that prevented WebSocket re-connections and caused console hangs.

  Cloudera Bug: DSE-3085

• Improved CDSW service restart performance for CSD deployments.

  Cloudera Bug: DSE-2937

For a list of the current known issues and limitations, refer the documentation for version 1.3.x at Cloudera Data Science Workbench 1.3.x.

For a list of known issues and limitations, refer the documentation for version 1.2.x at Cloudera Data Science Workbench 1.2.x.

Upgrading from Cloudera Data Science Workbench 1.2.0 to 1.2.1 on CSD-based deployments

After upgrading from Cloudera Data Science Workbench 1.2.0 to 1.2.1 on a CSD-based deployment, CLI commands might not work as expected due to missing binaries in the environment. Note that this issue does not affect fresh installs.

For a list of known issues and limitations, refer the documentation for version 1.2.x at Cloudera Data Science Workbench 1.2.x.

• Cloudera Data Science Workbench is now available as an add-on service for Cloudera Manager. To this end, Cloudera Data Science Workbench is now distributed in a parcel that integrates with Cloudera Manager using a Custom Service Descriptor (CSD). You can use Cloudera Manager to install, upgrade, and monitor Cloudera Data Science Workbench. Diagnostic data bundles can be generated and submitted to Cloudera through Cloudera Manager.
• Cloudera Data Science Workbench now enables secure sharing of job and session consoles. Additionally, site administrators can disable anonymous sharing from the Site Administrator dashboard (Admin > Security). See Sharing Job and Session Console Outputs.
• The Admin > Usage page now includes graphs for monitoring usage activity such as number of CPUs or GPUs used, memory usage, and total session runs, over customizable periods of time.
• Cloudera Data Science Workbench now lets you configure session, job, and idle timeouts. These can be configured using environmental variables either for the entire deployment or per-project.
• The cdsw enable and disable commands are no longer needed. The master host will now automatically detect the IP addresses of worker hosts joining or leaving Cloudera Data Science Workbench. See the revised Cloudera Data Science Workbench Command Line Reference.
• The Kudu Python client is now included in the Cloudera Data Science Workbench base engine image.
• Interactive session names can now be modified by project contributors and admins. By default, session names are set to 'Untitled Session'.
• All-numeric usernames are now accepted.
• Kubernetes has been upgraded to version 1.6.11.

Proxy Configuration Change: If you are using a proxy server, you must ensure that the IP addresses for the web and Livelog services are skipped from the proxy.

100.77.0.129
100.77.0.130

These have also been added to the installation instructions.

For a list of known issues and limitations, refer the documentation for version 1.2.x at Cloudera Data Science Workbench 1.2.x.

• Keytab Authentication - With version 1.1.1, you can now authenticate yourself to the CDH cluster by uploading your Kerberos keytab to Cloudera Data Science Workbench. To use this feature, go to the top-right dropdown menu, click Account settings > Hadoop Authentication, enter your Kerberos principal and click Upload Keytab.

For a list of known issues and limitations, refer the documentation for version 1.1.x at Cloudera Data Science Workbench 1.1.x.

• Added support for RHEL/CentOS 7.3 and Oracle Linux 7.3.

• Cloudera Data Science Workbench now allows you to run GPU-based workloads. For more details, see Using NVIDIA GPUs for Cloudera Data Science Workbench Projects.

• For Cloudera Manager and CDH clusters that are not connected to the Internet, Cloudera Data Science Workbench now supports fully offline installations. See the installation guide for more details.

• Web UIs for processing frameworks such as Spark 2, Tensorflow, and Shiny, are now embedded in Cloudera Data Science Workbench and can be accessed directly from active sessions and jobs. For more details, see Accessing Web User Interfaces from Cloudera Data Science Workbench.

• Added support for a Jobs REST API that lets you orchestrate jobs from 3rd party workflow tools. See Cloudera Data Science Workbench Jobs API.

• DataFrames are now scrollable in the workbench session output pane. For examples, see the section on Grid Displays.

• Added support for rich visualizations in Scala engine using Jupyter jvm-repr. For an example, see HTML Visualizations - Scala.

• JAVA_HOME is now set in cdsw.conf, and not from the Site Administrator dashboard (Admin > Engines).

• Improved support for dynamic data visualizations in Python, including Bokeh.

• Fixed issues with the Python template project. The project now supports offline mode and will therefore work on airgapped clusters.

• Fixed issues related to cached responses in Internet Explorer 11.

• Fixed issues with Java symlinks outside of JAVA_HOME.

• The cdsw status command can now be run on worker hosts.

• Removed unauthenticated localhost access to Kubernetes.

• Fixed Kerberos authentication issues with specific enc-types and Active Directory.

• Removed restrictions on usernames with special characters for better compatibility with external authentication systems such as Active Directory.

• Fixed issues with LDAP configuration validation that caused application crashes.

• Improved LDAP test configuration form to avoid confusion on parameters being sent.

• Upgrading from version 1.0.x to 1.1.x

  During the upgrade process, you will encounter incompatibilities between the two versions of cdsw.conf. This is because even though you are installing the latest RPM, your previous configuration settings in cdsw.conf will remain unchanged. Depending on the release you are upgrading from, you will need to modify cdsw.conf to ensure it passes the validation checks run by the 1.1.x release.

  Key changes to note:

  • JAVA_HOME is now a required parameter. Make sure you add JAVA_HOME to cdsw.conf before you start Cloudera Data Science Workbench.
  • Previous versions allowed MASTER_IP to be set to a DNS hostname. If you are still using a DNS hostname, switch to an IP address.
• Python engine updated in version 1.1.x

  Version 1.1.x includes an updated base engine image for Python which no longer uses the deprecated pylab mode in Jupyter to import the numpy and matplotlib functions into the global scope. With version 1.1.x, engines will now use built-in functions like any rather than the pylab counterpart, numpy.any. As a result of this change, you might see certain behavioral changes and differences in results between the two versions.

  Also note that Python projects originally created with engine 1 will be running pandas version 0.19, and will not auto-upgrade to version 0.20 by simply selecting engine 2. You will also need to manually install version 0.20.1 of pandas when you launch a project session.

For a list of known issues and limitations, refer the documentation for version 1.1.x at Cloudera Data Science Workbench 1.1.x.

• Fixed a random port conflict that could prevent Scala engines from running.

• Improved formatting of validation, and visibility of some errors.

• Fixed an issue with Firefox that was resulting in duplicate jobs on job creation.

• Removed the Mathjax external dependency on CDN.

• Improved PATH and JAVA_HOME handling that previously broke Hadoop CLIs.

• Fixed an issue with Java security policy files that caused Kerberos issues.

• Fixed an issue that caused git clone to fail on some repositories.

• Fixed an issue where updating LDAP admin settings deactivated the local fallback login.

• Fixed an issue where bad LDAP configuration crashed the application.

• Fixed an issue where job environmental variable settings did not persist.

For a list of known issues and limitations, refer the documentation for version 1.0.x at Cloudera Data Science Workbench 1.0.x.