CDS Powered by Apache Spark Known Issues

When disk encryption is enabled, PySpark broadcast variables fail with the following stack trace:

Traceback (most recent call last): File "broadcast.py", line 37, in <module>
words_new.value File "/pyspark.zip/pyspark/broadcast.py", line 137, in value
File "pyspark.zip/pyspark/broadcast.py", line 122, in load_from_path File
"pyspark.zip/pyspark/broadcast.py", line 128, in load EOFError: Ran out of input
      

Workaround: None

Affected Versions: CDS 2.2 Release 4

Apache Issue: SPARK-26201

Cloudera Issue: CDH-76116

Even if user is configured via Sentry to not have read permission to a Hive table, a Spark SQL job running as that user can still read the table's metadata directly from the Hive metastore.

Cloudera Bug: CDH-33658

After a Parquet table is created by the saveAsTable() function, Spark SQL queries against the table return an empty result set. The issue is caused by the "path" property of the table not being written to the Hive metastore during the saveAsTable() call.

Cloudera Bug: CDH-60037

Affects: CDS 2.2 Release 1

Severity: High

Workaround: You can set the path manually before the call to saveAsTable():

val options = Map("path" -> "/path/to/hdfs/directory/containing/table")
df.write.options(options).saveAsTable("db_name.table_name")

Or you can add the path to the metastore when the table already exists, for example:

spark.sql("alter table db_name.table_name set SERDEPROPERTIES ('path'='hdfs://host.example.com:8020/warehouse/path/db_name.db/table_name')")
spark.catalog.refreshTable("db_name.table_name")

Resolution: Upgrade to CDS 2.2 release 2, which contains the fix.

Although Spark 1 and Spark 2 can coexist in the same CDH cluster, you cannot use multiple Spark 2 versions simultaneously in the same Cloudera Manager instance. All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release. Make sure to install or upgrade the CDS 2 service descriptor and parcels across all machines of all clusters at the same time.

Spark Standalone is not supported for Spark 2.

The HiveOnSpark module is a CDH 5 component that has a dependency on Apache Spark 1.6. Because CDH 5 components do not have any dependencies on Spark 2, the HiveOnSpark module does not work with CDS Powered by Apache Spark. You can still use Spark 2 with Hive using other methods.

The SparkOnHBase module is a CDH 5 component that has a dependency on Apache Spark 1.6. Because CDH 5 components do not have any dependencies on Spark 2, the SparkOnHBase module does not work with CDS Powered by Apache Spark. You can still use Spark 2 with HBase using other methods.

If you are using Spark Streaming, Cloudera recommends that you disable dynamic allocation by setting spark.dynamicAllocation.enabled to false when running streaming applications.

The Oozie Spark action is a CDH component that has a dependency on Spark 1.6. Because CDH components do not have any dependencies on Spark 2, the Oozie Spark action does not work with Spark 2.

SparkR is not supported for Spark 2. (SparkR is also not supported in CDH with Spark 1.6.)

GraphX is not supported for Spark 2. (GraphX is also not supported in CDH with Spark 1.6.)

The Thrift JDBC/ODBC server is not supported for Spark 2. (The Thrift server is also not supported in CDH with Spark 1.6.)

The Spark SQL CLI is not supported for Spark 2. (The Spark SQL CLI is also not supported in CDH with Spark 1.6.)

Rolling upgrades are not possible from Spark 1.6 bundled with CDH, to CDS 2 Powered by Apache Spark.

CDS 2 Powered by Apache Spark is only installable as a parcel.

The spark-avro library is not integrated into the CDS 2 Powered by Apache Spark.x parcel.

Spark does not support accessing multiple clusters in the same application.

This feature, part of the GPL Extras package for CDH, is not supported with CDS 2 Powered By Apache Spark. This feature is supported for Spark 1.6.

The Cost Based Optimization feature is not supported in Spark 2.2. Do NOT set the spark.sql.cbo.enabled configuration option to true.

The spark2-submit script's --principal and --keytab arguments do not work with Spark-on-YARN's client mode. Use cluster mode instead.

If you submit a long-running app on a secure cluster using the --principal and --keytab options in cluster mode, and a failure causes the driver to restart after 7 days (the default maximum HDFS delegation token lifetime), the new driver fails with an error similar to the following:

Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token <token_info> can't be found in cache

Workaround: None

Affected Versions: All CDS 2.0, 2.1, and 2.2 releases

Fixed Versions: CDS 2.3 Release 2

Apache Issue: SPARK-23361

Cloudera Issue: CDH-64865