Homepage/Data Science Workbench1.10.5

(latest)

Search Documentation

▶︎Cloudera
1. Reference Architectures
▶︎Cloudera Public Cloud
▶︎Cloudera Private Cloud
▶︎Cloudera Manager
1. Cloudera Manager
▼Applications
▶︎Legacy
▶︎

Cloudera Streaming Community Edition
Data Science Workbench
Data Visualization
Edge Management
Observability SaaS
Observability on premises
Workload XM On-Prem

«

Filter topics

Data Science Workbench
▶︎Release Notes
▶︎ML Runtimes Release Notes
1. ▶︎ML Runtimes What's New
2. ML Runtimes Known Issues and Limitations
▶︎ML Runtimes Pre-installed Packages
1. ▶︎Pre-Installed Packages in ML Runtimes
▶︎Product Overview
1. Cloudera Data Science Workbench Overview
2. Key Differences - CML vs. CDSW
▶︎Planning
1. ▶︎Architecture Overview
  1. ▶︎Architecture Overview
2. ▶︎Requirements and Supported Platforms
▶︎Installation & Upgrade
▼How To
▶︎Troubleshooting
1. ▶︎Troubleshooting Cloudera Data Science Workbench
▶︎API
1. Cloudera Data Science Workbench API v2
2. API v2 Usage
▶︎CLI Reference
1. ▶︎Command Line Reference
▶︎FAQs
1. ▶︎Cloudera Data Science Workbench FAQs
▶︎Glossary
1. Cloudera Data Science Workbench Glossary
▶︎Videos
1. ▶︎Cloudera Data Science Workbench Videos

(Optional) Configure the Sync Between Cloudera Data Science Workbench and PyCharm
(Optional) Generate an SSH Public/Private Key
(Optional) Install Cloudera Data Science Workbench on Worker Hosts
(Optional) Install Cloudera Data Science Workbench on Worker Hosts
(Optional) Install Cloudera Data Science Workbench on Worker Hosts
(Optional) Using VS Code with Git integration
(Optional) Using VS Code with Jupyter
(Optional) Using VS Code with Python
(Optional) Using VS Code with R
(Python Only) Using a Requirements File
404 Not Found Error
Access Control for Teams and Projects
Access the Terminal
Accessing Cloudera Data Science Workbench from Cloudera Manager
Accessing Data from Apache HBase
Accessing Data from Apache Hive
Accessing Data from Apache Impala
Accessing Data from HDFS
Accessing Data in Amazon S3 Buckets
Accessing Environmental Variables from Projects
Accessing Environmental Variables from Projects
Accessing External SQL Databases
Accessing Local Data from Your Computer
Accessing Spark 2 from the Scala Engine
Accessing the Grafana Dashboard
Accessing the Site Administrator Dashboard
Accessing Web User Interfaces from Cloudera Data Science Workbench
Add a catalog
Add and Set Up the New Hosts
Add and Set Up the New Hosts
Add Cloudera Data Science Workbench as an Interpreter for VS Code
Add Docker registry credentials
Add Docker Registry Credentials
Add Gateway Hosts for Cloudera Data Science Workbench to Your HDP Cluster
Add Role Instances for the New Hosts
Add the Cloudera Data Science Workbench Service
Add the Cloudera Data Science Workbench Service
Add the new ML Runtime
Adding a Worker Host Using Cloudera Manager
Adding a Worker Host Using Packages
Adding and Removing Worker Hosts
Adding and Removing Worker Hosts
Adding CDSW Session Metadata Information
Adding Collaborators
Adding New Users
Adding Remote or Local JARs
Adding Remote Packages
Adding SSH Key to GitHub
Adding the Cloudera Data Science Workbench Service
Additional Usage Notes
Airgapped Installations
Allow "privileged" pod containers
Allow containers to run as root
Allow pod containers to mount unsupported volume types
AMP Project Specification
AMPs
Analytical Applications
Analytical Applications
Apache Spark
API
API Key Authentication
API v2
API v2 Usage
Application Block Device or Mount Point
Applications
Applied ML Prototypes (AMPs)
Applied ML Prototypes (AMPs)
Architecture Overview
Architecture Overview
Assigning the Site Administrator Role to an Existing User
Authentication
Authorization
Available Grafana Dashboards
Await Workers
Backup and Disaster Recovery for Cloudera Data Science Workbench
Backup Application Data
Backup Application Data
Base Engine 10
Base Engine 13
Base Engine 15
Base Engine 16
Base Engine 2
Base Engine 3
Base Engine 4
Base Engine 5
Base Engine 6
Base Engine 7
Base Engine 8
Base Engine Image Security
Basic Concepts and Terminology
Build Image
Build the New Docker Image
Build the New Image
Building
Business Users and CDSW
Calling a Model
Can I run Cloudera Data Science Workbench on hosts shared with other Hadoop services?
Can I run Cloudera Data Science Workbench on my own Kubernetes cluster?
Catalog File Specification
CDH Integration
CDS 2.x Powered by Apache Spark
CDSW 1.10.5 Download Information
CDSW in Cloudera Manager
CDSW-on-HDP Architecture Overview
cdswctl Command Line Interface Client
Changing the Domain Name Using Cloudera Manager
Changing the Domain Name Using Packages
Checking the Status of the CDSW Service
CLI Reference
Cloudera Altus Director Support (AWS and Azure Only)
Cloudera Data Science Workbench 1.0.0
Cloudera Data Science Workbench 1.0.1
Cloudera Data Science Workbench 1.1.0
Cloudera Data Science Workbench 1.1.1
Cloudera Data Science Workbench 1.10.0
Cloudera Data Science Workbench 1.10.1
Cloudera Data Science Workbench 1.10.2
Cloudera Data Science Workbench 1.10.3
Cloudera Data Science Workbench 1.10.4
Cloudera Data Science Workbench 1.2.0
Cloudera Data Science Workbench 1.2.1
Cloudera Data Science Workbench 1.2.2
Cloudera Data Science Workbench 1.3.0
Cloudera Data Science Workbench 1.3.1
Cloudera Data Science Workbench 1.4.0
Cloudera Data Science Workbench 1.4.2
Cloudera Data Science Workbench 1.4.3
Cloudera Data Science Workbench 1.5.0
Cloudera Data Science Workbench 1.6.0
Cloudera Data Science Workbench 1.6.1
Cloudera Data Science Workbench 1.7.1
Cloudera Data Science Workbench 1.7.2
Cloudera Data Science Workbench 1.8.0
Cloudera Data Science Workbench 1.8.1
Cloudera Data Science Workbench 1.9.0
Cloudera Data Science Workbench 1.9.1
Cloudera Data Science Workbench 1.9.2
Cloudera Data Science Workbench API v2
Cloudera Data Science Workbench Email Notifications
Cloudera Data Science Workbench Engine Versions and Packaging
Cloudera Data Science Workbench FAQs
Cloudera Data Science Workbench Gateway Host Security
Cloudera Data Science Workbench Glossary
Cloudera Data Science Workbench Jobs API
Cloudera Data Science Workbench Overview
Cloudera Data Science Workbench Scaling Guidelines
Cloudera Data Science Workbench Security Guide
Cloudera Data Science Workbench Videos
Cloudera Data Science Workbench Web Application
Cloudera Data Visualization for ML
Cloudera Enterprise License
Cloudera Manager
Cloudera Manager and CDH Requirements
Cloudera Manager Integration
Cloudera Manager, Hue, and the Spark History Server
Cluster Authorization
Cluster Management
Cluster Management
Cluster Management
Cluster Monitoring with Grafana
Code Autocomplete
Collaborate
Collaborating on Projects with Cloudera Data Science Workbench
Collaborating with Git
Command Line Reference
Concepts
Concepts and Terminology
Concepts and Terminology
Configuration Options
Configure a Browser IDE as an Editor
Configure a Browser IDE at the Legacy Engine Level
Configure a Browser IDE at the Legacy Engine Level
Configure a Browser IDE at the Project Level
Configure a SSH Gateway to Use Local IDEs
Configure and Use a Local IDE
Configure Apache Spark 2
Configure Apache Spark 2
Configure Apache Spark 2 on CDH 6 or CDP Data Center 7
Configure Apache Spark 2 on CDH 6 or CDP Data Center 7
Configure Block Devices
Configure FreeIPA
Configure Gateway Hosts Using Cloudera Manager
Configure Gateway Hosts Using Cloudera Manager
Configure JAVA_HOME
Configure JAVA_HOME
Configure Jupyter Notebook in a Customized Engine Image
Configure PyCharm as a Local IDE
Configure VS Code as a Local IDE
Configuring CDS 2.x Powered by Apache Spark 2
Configuring Cloudera Data Science Workbench Deployments Behind a Proxy
Configuring Cloudera Data Science Workbench Engines
Configuring Cloudera Data Science Workbench Properties
Configuring Custom Root CA Certificate
Configuring Engine Environments for Experiments and Models
Configuring External Authentication with LDAP and SAML
Configuring External Termination
Configuring Global Properties Using Cloudera Manager
Configuring Hostnames to be Skipped from the Proxy
Configuring HTTP Headers for Cloudera Data Science Workbench
Configuring Internal Termination
Configuring LDAP/Active Directory Authentication
Configuring SAML Authentication
Configuring Spark Environment Variables Using Cloudera Manager
Configuring spark-port-forwarder ports
Configuring the Engine Environment
Configuring the Reverse Proxy Server
Connecting to Spark 2
Copy the DNS Nameserver to the new host
Copy the DNS Nameserver to the New Host
Copy the JDK to the new host
Copy the JDK to the new host
Copy the Kerberos Configurations
Copy the Kerberos Configurations
Crashes and stops responding
Create a Chart to Track Disk Usage on the Application Block Device
Create a Custom CUDA-capable Engine Image
Create a Dockerfile for the Custom Runtime Image
Create a Dockerfile for the New Custom Image
Create a model using the CLI
Create a Project
Create a Project from a Built-in Template
Create a Trigger to Notify Cluster Administrators when Free Space Runs Low
Create HDFS User Directories
Create the Administrator Account
Create the Administrator Account
Create the Administrator Account
Create the Administrator Account
Create the Site Administrator Account
Creating a Backup for Disaster Recovery for Cloudera Data Science Workbench
Creating a Certificate Signing Request (CSR) and Key/Certificate Pair
Creating a Certificate Signing Request (CSR) and Key/Certificate Pair
Creating a Customized Engine Image
Creating a Customized Engine with the Required Package(s)
Creating a Job
Creating a model
Creating a Pipeline
Creating a Project with Legacy Engine Variants
Creating a Project with ML Runtimes Variants
Creating a Team With an Associated LDAP Group
Creating a Team Without an Associated LDAP Group
Creating an Extensible Engine with Conda
Creating and Deploying a Model (QuickStart)
Creating Customized ML Runtimes
Creating Diagnostic Bundles
Creating New AMPs
Creating Resource Profiles
Creating SSH Tunnels
Cross-Origin Resource Sharing (CORS)
CSD Deployments
CSD Deployments
CSD Deployments
CSD Installation on CDH
Custom AMP Catalog
Custom Legacy Engine Images
Custom Template Projects
Customize Workload Scheduling
Customized Engine Images
Customized Runtimes
Data Access
Data Collection
Data Collection in Cloudera Data Science Workbench
Data Science Workbench
Data Visualization
Debug Login URL
Debugging Issues with Experiments
Debugging Issues with Models
Delete CDSW Roles from Existing Hosts
Deleting a Model
Deleting a Project
Dependencies
Deploy a New Build for a Model
Deploy CDSW on HDP
Deploy the Model
Deployed
Deploying
Deployment Considerations
Diagnostic Bundles
Disable Untrusted SSH Access
Disable Usage Tracking
Disabling Project File Uploads and Downloads
Disabling the Experiments Feature
Disabling the Models Feature
Disabling Usage Metrics
Disabling User Accounts
Distribute the Image
Distribute the Image
Distributed Computing with Workers
Distributed ML
Docker and Kubernetes Support
Docker Block Device
Documenting Your Analysis
Does Cloudera Data Science Workbench support REST API access?
Download and Configure the cdswctl
Download and Install the Cloudera Data Science Workbench CSD
Download and Install the Cloudera Data Science Workbench CSD
Download cdswctl and Add an SSH Key
Download cdswctl and Add an SSH Key
Download cdswctl and Add an SSH Key
Editor Customization
Editors
Editors
Embedded Web Apps
Enable GPU Support in Cloudera Data Science Workbench
Enable HTTP Security Headers
Enable HTTP Strict Transport Security (HSTS)
Enabling Authentication
Enabling Cloudera Data Science Workbench to use GPUs
Enabling Custom Quotas for Specific Users
Enabling Default Quotas for all CDSW Users
Enabling Model Governance
Enabling Model Metrics
Enabling TLS/SSL for Cloudera Data Science Workbench
End-to-End Example: MeCab
Engine Configuration
Engine Dependencies
Engine Environment Variables
Engine Environment Variables
Engine Limitations
Engine Upgrade 1.1.0
Engine Upgrade 1.2.0
Engine Upgrade 1.2.2
Engine Upgrade 1.4.0
Engine Upgrade 1.4.2
Engine Upgrade 1.5.0
Engine Upgrade 1.6.0
Engine Upgrade 1.6.1
Engine Upgrade 1.7.1
Engines for Experiments and Models
Engines Packaging
Environment Variables
Environment Variables from Cloudera Manager
Environmental Variables
Error Encountered Trying to Load Images when Initializing Cloudera Data Science Workbench
Example: A Shiny Application
Example: A Tensorboard Application
Example: Distributing Dependencies on a PySpark Cluster
Example: H2O
Example: Locating and Adding JARs to Spark 2 Configuration
Example: Montecarlo Estimation
Example: Read Files from the Cluster Local Filesystem
Example: Using External Packages by Adding Jars or Dependencies
Example: Worker Network Communications
Experiments
Experiments
Experiments Demo
Export Session List
Extensible Engines
External Authentication - LDAP and SAML
External Communications
External Termination
FAQs
Forking Projects
Frequently Asked Questions (FAQs)
General
General Troubleshooting Solutions
Generating a Model API Key
Getting Started with a New Project on Cloudera Data Science Workbench
Getting Started with Cloudera Data Science Workbench
Glossary
GPU Issues
GPU Support
GPU Support
GPUs
Grid Displays
Hadoop Authentication - Kerberos
Hadoop Authentication with Kerberos for Cloudera Data Science Workbench
Hardware Requirements
Health Tests
How do I contact Cloudera for issues regarding Cloudera Data Science Workbench?
How does Cloudera Data Science Workbench use Docker and Kubernetes?
How Login Works with LDAP Group Settings Enabled
How Login Works with SAML Group Settings Enabled
HTML Visualizations
HTTP Headers
IFrame Visualizations
Importing a Project From Git
Importing Data into Cloudera Data Science Workbench
Incompatible Changes in Cloudera Data Science Workbench 1.1.0
Incompatible Changes in Cloudera Data Science Workbench 1.2.0
Incompatible Changes in Cloudera Data Science Workbench 1.2.1
Incompatible Changes in Cloudera Data Science Workbench 1.3.0
Incompatible Changes in Cloudera Data Science Workbench 1.4.0
Incompatible Changes in Cloudera Data Science Workbench 1.5.0
Incompatible Changes in Cloudera Data Science Workbench 1.6.0
Increasing Docker storage
Information Collected in Diagnostic Bundles
Initialize an SSH Connection to Cloudera Data Science Workbench
Initialize an SSH Connection to Cloudera Data Science Workbench
Initialize an SSH Connection to Cloudera Data Science Workbench for VS Code
Install Cloudera Data Science Workbench on New Master Host
Install Cloudera Data Science Workbench on the Master Host
Install Cloudera Data Science Workbench on the Master Host
Install Cloudera Data Science Workbench on the Master Host
Install the Cloudera Data Science Workbench Parcel
Install the Cloudera Data Science Workbench Parcel
Install the NVIDIA Driver on GPU Hosts
Installation
Installation
Installing a Jupyter extension
Installing a Jupyter kernel
Installing Additional ML Runtimes Packages
Installing Additional Packages
Installing Cloudera Data Science Workbench 1.10.3 on HDP
Installing Cloudera Data Science Workbench 1.10.4
Installing Cloudera Data Science Workbench 1.10.4 Using Cloudera Manager
Installing Cloudera Data Science Workbench 1.10.4 Using Cloudera Manager
Installing Cloudera Data Science Workbench 1.10.4 Using Packages
Installing Cloudera Data Science Workbench 1.10.4 Using Packages
Installing Cloudera Data Science Workbench on CDP Private Base
Installing Cloudera Data Science Workbench on CDP Private Base
Installing Packages Directly Within Projects
Installing sparklyr
Installing the Reverse Proxy Server
Internal Communications
Internal Termination
Introduction to Production Machine Learning
Issues Fixed in Cloudera Data Science Workbench 1.0.1
Issues Fixed in Cloudera Data Science Workbench 1.1.0
Issues Fixed In Cloudera Data Science Workbench 1.1.1
Issues Fixed in Cloudera Data Science Workbench 1.10.0
Issues Fixed in Cloudera Data Science Workbench 1.10.1
Issues Fixed in Cloudera Data Science Workbench 1.10.2
Issues Fixed in Cloudera Data Science Workbench 1.10.3
Issues Fixed in Cloudera Data Science Workbench 1.10.4
Issues Fixed in Cloudera Data Science Workbench 1.2.0
Issues Fixed In Cloudera Data Science Workbench 1.2.1
Issues Fixed In Cloudera Data Science Workbench 1.2.2
Issues Fixed in Cloudera Data Science Workbench 1.3.0
Issues Fixed in Cloudera Data Science Workbench 1.3.1
Issues Fixed in Cloudera Data Science Workbench 1.4.0
Issues Fixed in Cloudera Data Science Workbench 1.4.2
Issues Fixed in Cloudera Data Science Workbench 1.4.3
Issues Fixed in Cloudera Data Science Workbench 1.5.0
Issues Fixed in Cloudera Data Science Workbench 1.6.0
Issues Fixed in Cloudera Data Science Workbench 1.6.1
Issues Fixed in Cloudera Data Science Workbench 1.7.1
Issues Fixed in Cloudera Data Science Workbench 1.7.2
Issues Fixed in Cloudera Data Science Workbench 1.8.0
Issues Fixed in Cloudera Data Science Workbench 1.8.1
Issues Fixed in Cloudera Data Science Workbench 1.9.0
Issues Fixed in Cloudera Data Science Workbench 1.9.1
Issues Fixed in Cloudera Data Science Workbench 1.9.2
Java Requirements
JDK Requirements
Jobs
Jobs and Pipelines
Jupyter Magic Commands
Key Differences - CML vs. CDSW
Key Points to Note
Known Issues and Limitations
Known Issues and Limitations
Known Issues and Limitations in Cloudera Data Science Workbench 1.0.x
Known Issues and Limitations in Cloudera Data Science Workbench 1.1.0
Known Issues and Limitations in Cloudera Data Science Workbench 1.1.1
Known Issues and Limitations in Cloudera Data Science Workbench 1.2.0
Known Issues and Limitations in Cloudera Data Science Workbench 1.2.1
Known Issues and Limitations in Cloudera Data Science Workbench 1.2.2
Known Issues and Limitations in Cloudera Data Science Workbench 1.3.0
Known Issues and Limitations in Cloudera Data Science Workbench 1.3.1
Known Issues and Limitations in Cloudera Data Science Workbench 1.4.0
Known Issues and Limitations in Cloudera Data Science Workbench 1.4.2
Known Issues and Limitations in Cloudera Data Science Workbench 1.4.3
Known Issues and Limitations in Cloudera Data Science Workbench 1.5.0
Known Issues and Limitations in Cloudera Data Science Workbench 1.6.0
Known issues for previous versions
Labeling Auxiliary Hosts for CSD Deployments
Labeling Auxiliary Hosts for RPM Deployments
Launch a Session to Run the Project
Launch Workers
LDAP General Settings
LDAP Group Settings
LDAP Over SSL (LDAPS)
Legacy Engines
Legacy Engines
Legacy Engines
Legacy Engines
Limitations
Limitations
Limitations
Limitations for Analytical Applications
Limitations with Port Availability
Limiting files in Explorer view
Linking an Existing Project to a Git Remote
List Workers
Listing runtimes
Load Data into HBase Table
Loading CSV Data into an Impala Table
Log in to cdswctl
Managing CDSW
Managing Cloudera Data Science Workbench Hosts
Managing Cloudera Data Science Workbench Users
Managing Cloudera Data Science Workbench Worker Hosts
Managing Dependencies for Spark 2 Jobs
Managing Dependencies for Spark 2 Projects
Managing Engine Dependencies
Managing Engine Images
Managing Engines
Managing Files
Managing Hosts
Managing Jobs and Pipelines in Cloudera Data Science Workbench
Managing License Keys for Cloudera Data Science Workbench
Managing Memory Available for Spark Drivers
Managing ML Runtimes
Managing Model API Keys
Managing Projects in Cloudera Data Science Workbench
Managing Resource Profiles
Managing Team Accounts
Managing the Cloudera Data Science Workbench Service in Cloudera Manager
Managing Users as a Site Administrator
Managing your Personal Account
Master Host
Metadata for Custom ML Runtimes
Migrating a CSD Deployment
Migrating a Deployment to New Hosts
Migrating an RPM Deployment
Migrating from an RPM-based Deployment to the latest CSD
ML Runtimes
ML Runtimes 2020.11
ML Runtimes 2020.11
ML Runtimes 2021.02
ML Runtimes 2021.02
ML Runtimes 2021.04
ML Runtimes 2021.04
ML Runtimes 2021.06
ML Runtimes 2021.06
ML Runtimes 2021.09
ML Runtimes 2021.09
ML Runtimes 2021.12
ML Runtimes 2022.04
ML Runtimes 2022.11
ML Runtimes 2023.05
ML Runtimes 2023.05
ML Runtimes 2023.08
ML Runtimes 2023.08.2
ML Runtimes 2023.12.1
ML Runtimes 2024.02.1
ML Runtimes 2024.05.1
ML Runtimes Environment Variables
ML Runtimes Environment Variables List
ML Runtimes Known Issues and Limitations
ML Runtimes Nvidia GPU Edition
ML Runtimes NVIDIA RAPIDS Edition
ML Runtimes Pre-installed Packages
ML Runtimes Release Notes
ML Runtimes Version 2020.11
ML Runtimes Version 2021.02
ML Runtimes Version 2021.04
ML Runtimes Version 2021.06
ML Runtimes Version 2021.09
ML Runtimes Version 2021.09.02
ML Runtimes Version 2021.12
ML Runtimes Version 2021.12
ML Runtimes Version 2022.04
ML Runtimes Version 2022.11
ML Runtimes Version 2022.11.2
ML Runtimes Version 2023.05.1
ML Runtimes Version 2023.05.2
ML Runtimes Version 2023.08.1
ML Runtimes Version 2023.08.2
ML Runtimes Version 2023.12.1
ML Runtimes Version 2024.2.1
ML Runtimes versus Legacy Engine
ML Runtimes versus Legacy Engines
ML Runtimes What's New
Model Artifacts
Model Code
Model Governance
Model Monitoring and Administration
Model Training and Deployment - Iris Dataset
Models
Models
Models
Models Demo
Modifying Default and Custom Quotas
Modifying Project Settings
Modifying Team Account Settings
Monitoring
Monitoring All Active Models
Monitoring Cloudera Data Science Workbench Usage
Monitoring Individual Models
Monitoring User Events
Monitoring Users
Monitoring/Reducing CDSW disk space
Mounting Additional Dependencies from the Host
Move Backup to New Master
Move Backup to the New Master
Multiple Cloudera Data Science Workbench Deployments
Network and Security Requirements
Networking
Networking and Security Requirements
New Features and Changes in Cloudera Data Science Workbench 1.1.0
New Features and Changes in Cloudera Data Science Workbench 1.10.0
New Features and Changes in Cloudera Data Science Workbench 1.10.1
New Features and Changes in Cloudera Data Science Workbench 1.10.2
New Features and Changes in Cloudera Data Science Workbench 1.10.3
New Features and Changes in Cloudera Data Science Workbench 1.10.4
New Features and Changes in Cloudera Data Science Workbench 1.2.0
New Features and Changes in Cloudera Data Science Workbench 1.2.2
New Features and Changes in Cloudera Data Science Workbench 1.3.0
New Features and Changes in Cloudera Data Science Workbench 1.4.2
New Features and Changes in Cloudera Data Science Workbench 1.4.3
New Features and Changes in Cloudera Data Science Workbench 1.5.0
New Features and Changes in Cloudera Data Science Workbench 1.6.0
New Features and Changes in Cloudera Data Science Workbench 1.6.1
New Features and Changes in Cloudera Data Science Workbench 1.7.1
New Features and Changes in Cloudera Data Science Workbench 1.7.2
New Features and Changes in Cloudera Data Science Workbench 1.8.0
New Features and Changes in Cloudera Data Science Workbench 1.8.1
New Features and Changes in Cloudera Data Science Workbench 1.9.0
New Features and Changes in Cloudera Data Science Workbench 1.9.1
New Features and Changes in Cloudera Data Science Workbench 1.9.2
New Features in Cloudera Data Science Workbench 1.1.1
New Features in Cloudera Data Science Workbench 1.3.1
New Features in Cloudera Data Science Workbench 1.4.0
Next Steps
Next Steps
Next Steps
Next Steps
Next Steps
Older Releases
Onboarding Business Users
Operating System Requirements
Operating System Requirements
Overview of Deploying CDSW 1.10.3 on HDP Overview
Package Libraries
Parallel Computing
Personal Key
Platform Requirements
Ports Required by Cloudera Data Science Workbench
Ports Used By Cloudera Data Science Workbench
Pre-Installed Packages in ML Runtimes
Pre-Installed Packages in ML Runtimes
Prepare to manage models using the model CLI
Prerequisites
Prerequisites
Prerequisites
Prerequisites
Prerequisites
Private Key and Certificate Requirements
Product Overview
Project Admins: Enable the CUDA Engine for your Project
Project Code Files
Project Collaborators
Project Environments
Projects
Projects
Proxy Configuration
Purpose
Push the Image to a Public Registry such as DockerHub
Push the Image to Your Company's Docker Registry
Pushing
Python
Python
Python 2 Libraries in Base Engine 2
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for Conda
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for JupyterLab
Python 3.10 Libraries for Workbench
Python 3.10 Libraries for Workbench
Python 3.10 Libraries for Workbench
Python 3.10 Libraries for Workbench
Python 3.10 Libraries for Workbench
Python 3.10 Libraries for Workbench
Python 3.11 Libraries for JupyterLab
Python 3.11 Libraries for JupyterLab
Python 3.11 Libraries for JupyterLab
Python 3.6 Libraries for JupyterLab
Python 3.6 Libraries for JupyterLab
Python 3.6 Libraries for JupyterLab
Python 3.6 Libraries for JupyterLab
Python 3.6 Libraries for Workbench
Python 3.6 Libraries for Workbench
Python 3.6 Libraries for Workbench
Python 3.6 Libraries for Workbench
Python 3.6.12 Libraries for JupyterLab
Python 3.6.12 Libraries for JupyterLab
Python 3.6.12 Libraries for Workbench
Python 3.6.12 Libraries for Workbench
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for JupyterLab
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7 Libraries for Workbench
Python 3.7.9 Libraries for JupyterLab
Python 3.7.9 Libraries for JupyterLab
Python 3.7.9 Libraries for Workbench
Python 3.7.9 Libraries for Workbench
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for JupyterLab
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8 Libraries for Workbench
Python 3.8.6 Libraries for JupyterLab
Python 3.8.6 Libraries for JupyterLab
Python 3.8.6 Libraries for Workbench
Python 3.8.6 Libraries for Workbench
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for JupyterLab
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9 Libraries for Workbench
Python 3.9.6 Libraries for JupyterLab
Python 3.9.6 Libraries for JupyterLab
Python 3.9.6 Libraries for JupyterLab
Python 3.9.6 Libraries for JupyterLab
Python 3.9.6 Libraries for JupyterLab
Python Libraries in Base Engine 10
Python Libraries in Base Engine 13
Python Libraries in Base Engine 15
Python Libraries in Base Engine 16
Python Libraries in Base Engine 3
Python Libraries in Base Engine 4
Python Libraries in Base Engine 5
Python Libraries in Base Engine 6
Python Libraries in Base Engine 7
Python Libraries in Base Engine 8
Python Supported Versions
Query Data Using HappyBase
Querying the engine type
Quickstart
Quickstart Demo
Quotas
R
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 3.6 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.0 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.1 Libraries
R 4.3 Libraries
R 4.3 Libraries
R 4.3 Libraries
R 4.3 Libraries
R 4.4 Libraries
R Libraries in Base Engine 10
R Libraries in Base Engine 13
R Libraries in Base Engine 15
R Libraries in Base Engine 16
R Libraries in Base Engine 2
R Libraries in Base Engine 3
R Libraries in Base Engine 4
R Libraries in Base Engine 5
R Libraries in Base Engine 6
R Libraries in Base Engine 7
R Libraries in Base Engine 8
RAPIDS Runtime PIP Python 3.7.8 Libraries for JupyterLab
RAPIDS Runtime PIP Python 3.7.8 Libraries for JupyterLab
RAPIDS Runtime PIP Python 3.7.8 Libraries for Workbench
RAPIDS Runtime PIP Python 3.7.8 Libraries for Workbench
RAPIDS Runtime PIP Python 3.8.6 Libraries for JupyterLab
RAPIDS Runtime PIP Python 3.8.6 Libraries for JupyterLab
RAPIDS Runtime PIP Python 3.8.6 Libraries for Workbench
RAPIDS Runtime PIP Python 3.8.6 Libraries for Workbench
Re-deploy an Existing Build
Recommended Configuration on Amazon Web Services (AWS)
Recommended Configuration on Microsoft Azure
Recommended Hardware Configuration
Registering training data lineage using a linking file
Related Resources
Related Resources
Release Notes
Remove Cloudera Data Science Workbench from Existing Hosts
Removing a Worker Host Using Cloudera Manager
Removing a Worker Host Using Packages
Required Pre-Installation Steps
Requirements and Supported Platforms
Reserving the Master Host for CSD Deployments
Reserving the Master Host for Internal CDSW Components
Reserving the Master Host for RPM Deployments
Resouce Consumption and Scaling
Restart a Model
Restricting Collaborator and Administrator Access to Active Sessions
Restricting User-Controlled Kubernetes Pods
Restrictions for upgrading R and Python packages
Reverse Proxy Overview
Roles Associated with the Cloudera Data Science Workbench Service
Rollback Cloudera Data Science Workbench to an Older Version
RPM Deployments
RPM Deployments
RPM Deployments
RPM Installation on CDH
Run Code
Run Experiment / Deploy Model
Run the Prepare Node command on the New Hosts
Running an Experiment (Quick Start)
Running Distributed ML Workloads on YARN
Running Queries on Impala Tables
Running Spark Jobs on an HDP Cluster
Runtimes
Runtimes
Sample Job Run
Saved Images
Saving Files
Scala
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala 2.11 Libraries for Workbench
Scala in Base Engine 10
Scala in Base Engine 2
Scala in Base Engine 3
Scala in Base Engine 4
Scala in Base Engine 5
Scala in Base Engine 6
Scala in Base Engine 7
Scala in Base Engine 8
Securing Models using Model API Key
Security
Security
Security
Security Considerations
Security Model
Set Up a Wildcard DNS Subdomain
Set Up the Operating System and Kernel
Setting Environmental Variables
Setting Up a PySpark Project
Setting Up an HTTP Proxy for Spark 2
Setting up VS Code
Sharing Data Visualizations
Sharing Job and Session Console Outputs
Sharing Personal Projects
Sign up
Simple Plots
Site Administration
Site Admins: Add the Custom CUDA Engine to your Cloudera Data Science Workbench Deployment
Snapshot Code
Spark 2 Web UIs (CDSW_SPARK_PORT)
Spark 3 on CDSW
Spark Configuration
Spark Configuration Files
Spark Logging Configuration
Spark on ML Runtimes
SSH Keys
SSH Keys
Start a New Session
Start the CDSW Service
Starting a Job Run Using Python
Starting a Job Run Using the API
Starting sessions and creating SSH endpoints
Starting, Stopping, and Restarting the Service
Stop a Model
Stop a Session
Stop Cloudera Data Science Workbench
Stop the CDSW Service
Stop Workers
Storage
Supporing a TLS-Enabled Proxy Server
Supported Browsers
Supported Platforms and Requirements
Team Key
Teams
TensorBoard, Shiny, and others (CDSW_APP_PORT or CDSW_READONLY_PORT)
Test a Browser IDE in a Session Before Installation
Test LDAP Configuration
Test the CUDA Runtime
Test whether Cloudera Data Science Workbench can Detect GPUs
Testing Applications before deployment
Testing ML Runtime GPU Setup
Testing ML Runtime GPU Setup
Third-party Editors
TLS and SSL Encryption
Tracked User Events
Tracking Disk Usage on the Application Block Device
Tracking Metrics
Tracking Model Metrics
Train the Model
Trial License
Troubleshooting
Troubleshooting Access Errors
Troubleshooting Airgapped Clusters
Troubleshooting Cloudera Data Science Workbench
Troubleshooting Engine Issues
Troubleshooting for Analytical Applications
Troubleshooting Installation Issues
Troubleshooting Issues with Models and Experiments
Troubleshooting Issues with Workloads
Troubleshooting Network Issues
Troubleshooting TLS/SSL Errors
UI Behavior for Non-Kerberized Clusters
Uninstalling CPM Deployments
Uninstalling RPM Deployments
Update DNS Records for the New Master
Update DNS Records for the New Master
Updating Active Models
Upgrade
Upgrades
Upgrading CDH and CDP
Upgrading CDSW 1.7.2 or higher from CDH 6 to CDP Private Cloud Base 7.x
Upgrading CDSW using Cloudera Manager
Upgrading cdsw-csd files for Cloudera Manager
Upgrading Cloudera Data Science Workbench
Upgrading Cloudera Data Science Workbench using packages
Upgrading to Cloudera Data Science Workbench 1.10.3 on HDP
Upgrading to the latest version of CDSW on CDP Private Cloud Base
Uploading License Keys
Usability
Usage Guidelines
Usage Tracking
User Access to Features
User Contexts
User Guide
User Role Authorization
User Roles
User Sign up Process
User-Controlled Kubernetes Pod
Using CDS 2.x Powered by Apache Spark
Using Cloudera Manager
Using Conda Runtime
Using Conda with Cloudera Data Science Workbench
Using Editors for ML Runtimes
Using Git to Collaborate on Projects
Using GPUs with Legacy Engines-Technical Preview
Using JupyterLab with ML Runtimes
Using ML Runtimes with cdswctl
Using NVIDIA GPUs for Cloudera Data Science Workbench Projects
Using Runtime Catalog
Using Spark 2 from Python
Using Spark 2 from R
Using Spark 2 from Scala
Using the Command Line
Using the Workbench
Validating Sessions on New Worker Host
Videos
View replica logs for a model using the CLI
Viewing Job History
Viewing Licensed Users
Viewing lineage for a model deployment in Atlas
Visualize Report
Web Applications Embedded in Cloudera Data Science Workbench
Web session timeouts
Web UI
What are the software and hardware requirements for Cloudera Data Science Workbench?
What's New
Where can I get a sample project to try out Cloudera Data Science Workbench?
Whitelist the Image in Cloudera Data Science Workbench
Wildcard DNS Subdomain Requirement
Wire Encryption
Workbench
Workbench editor file types
Worker Hosts
Workers API

«

Filter topics

Using Spark 2 from Python

Spark 3 on CDSW
Using CDS 2.x Powered by Apache Spark
▶︎Configuring CDS 2.x Powered by Apache Spark 2
Setting Up an HTTP Proxy for Spark 2
▼Using Spark 2 from Python
▶︎Using Spark 2 from R
1. Installing sparklyr
2. Connecting to Spark 2
▶︎Using Spark 2 from Scala

»

Spark Configuration

Using Spark 2 from Python

Cloudera Data Science Workbench supports using Spark 2 from Python via PySpark.

Setting Up a PySpark Project
The default Cloudera Data Science Workbench engine currently includes Python 2.7.18 and Python 3.6.10.
Spark on ML Runtimes
Only certain Spark versions are supported on ML Runtimes.
Example: Montecarlo Estimation
Within the template PySpark project, pi.py is a classic example that calculates Pi using the Montecarlo Estimation.
Example: Locating and Adding JARs to Spark 2 Configuration
This example shows how to discover the location of JAR files installed with Spark 2, and add them to the Spark 2 configuration.
Example: Distributing Dependencies on a PySpark Cluster
Although Python is a popular choice for data scientists, it is not straightforward to make a Python library available on a distributed PySpark cluster. To determine which dependencies are required on the cluster, you must understand that Spark code applications run in Spark executor processes distributed throughout the cluster. If the Python code you are running uses any third-party libraries, Spark executors require access to those libraries when they run on remote executors.

© 2019–2023 by Cloudera, Inc. All rights reserved.

This site uses cookies and related technologies, as described in our privacy policy, for purposes that may include site operation, analytics, enhanced user experience, or advertising. You may choose to consent to our use of these technologies, or

1.10
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
1.0