Test your connectivity to the CDP-DC cluster

Test that you can create a Project in your ML Workspace and access data that is stored in the data center cluster.

Create a new Project, using the PySpark template.
Create a new file called testdata.txt (use this exact filename).
Add 2-3 lines of any text in the file to serve as sample data.

Run the following Spark commands to test the connection.

from pyspark.sql import SparkSession
                        
# Instantiate Spark-on-K8s Cluster
spark = SparkSession\
.builder\
.appName("Simple Spark Test")\
.config("spark.executor.memory","8g")\
.config("spark.executor.cores","2")\
.config("spark.driver.memory","2g")\
.config("spark.executor.instances","2")\
.getOrCreate()

# Validate Spark Connectivity
spark.sql("SHOW databases").show()
spark.sql('create table testcml (abc integer)').show()
spark.sql('insert into table testcml  select t.* from (select 1) t').show()
spark.sql('select * from testcml').show()

# Stop Spark Session
spark.stop()

Run the following direct HDFS commands to test the connection.

# Run sample HDFS commands
# Requires an additional testdata.txt file to be created with sample data in project home dir
!hdfs dfs -mkdir /tmp/testcml/
!hdfs dfs -copyFromLocal /home/cdsw/testdata.txt /tmp/testcml/
!hdfs dfs -cat /tmp/testcml/testdata.txt

If you get errors, then check with your Admin to make sure that your user ID is set up in the Hadoop Authentication settings to access the CDP-DC cluster, and that the correct Ranger permissions have been applied.