In-Place Upgrade CDH 5 to CDP Private Cloud Base
Preparing for your upgrade
Assessing the Impact of an Upgrade
How much time should I plan for to complete my upgrade?
About using this online Upgrade Guide
CDP Private Cloud Base Pre-upgrade transition steps
Set log level for KeyTrustee KMS to INFO
Transitioning from MapReduce 1 to MapReduce 2
Upgrading an MRv1 installation using Cloudera Manager
Major changes when migrating to MapReduce 2
Configuration changes between MRv1 and MRv2
Transitioning Cloudera Search configuration
Before you begin
Create HDFS backup directory
Set Solr configuration properties
Download the configuration
Back up Solr configuration and data
Transition Solr configuration
Cloudera Manager versions 7.1.1 to 7.2.4
Cloudera Manager versions 7.3.1 or higher
Validate the configuration
Test the configuration
Copy the transitioned configuration to the upgrade metadata directory
About Solr configuration transformation script
Transitioning from Sentry Policy Files to the Sentry Service
Transitioning the Sentry service to Apache Ranger
Configuring a Ranger or Ranger KMS Database: MySQL/MariaDB
Configuring a Ranger Database: PostgreSQL
Configuring a Ranger or Ranger KMS Database: Oracle
Transitioning Navigator content to Atlas
Transition process
Assumptions and prerequisites
Installing Atlas in the Cloudera Manager upgrade wizard
Transitioning Navigator data using customized scripts
Mapping Navigator metadata to Atlas
Transitioning Navigator audits
What's new in Atlas for Navigator Users?
Preparing the backend HMS database for upgrade
Migrating Hive 1-2 to Hive 3
Hive Configuration Changes Requiring Consent
Check SERDE Definitions and Availability
Handle Missing Table or Partition Locations
Remove transactional=false from Table Properties
Checking Apache HBase
Remove PREFIX_TREE Data Block Encoding
Validate HFiles
Check co-processor classes
CDH cluster upgrade requirements for Replication Manager
Upgrading the JDK
Using AES-256 Encryption
Configuring a Custom Java Home Location
Tuning JVM Garbage Collection
Upgrading the Operating System
Step 1: Getting Started
Step 2: Backing Up
Backing up Cloudera Manager databases
Step 3: Before You Upgrade
Step 4: After You Upgrade
Upgrading Cloudera Manager 5
Step 1: Getting Started Upgrading Cloudera Manager 5
Step 2: Backing Up
Collect Information
Back Up Cloudera Manager Agent
Back Up the Cloudera Management Service
Back Up Cloudera Navigator Data
Stop Cloudera Manager Server & Cloudera Management Service
Back Up the Databases
Back Up Cloudera Manager Server
(Optional) Start Cloudera Manager Server & Cloudera Management Service
Step 3: Upgrading the Server
Step 4: Upgrading the Agents
Step 5: After You Upgrade
Upgrade Key Trustee Server to 7.1.x
Upgrade Navigator Encrypt to 7.1.x
Upgrading Cloudera Navigator Key HSM
Troubleshooting
Reverting a Failed Upgrade
Validate TLS configurations
Expediting the Hive upgrade
Overview of the expedited Hive upgrade
Preparing tables for migration
Check SERDE Definitions and Availability
Handle Missing Table or Partition Locations
Managed Table Location Mapping
Make Tables SparkSQL Compatible
Configuring HSMM to prevent migration
Understanding the Hive upgrade
Installing dependencies for Hue
Installing the psycopg2 Python package for PostgreSQL database
Installing MySQL client for MySQL databases
Installing MySQL client for MariaDB databases
Upgrading a CDH 5 Cluster
Step 1: Getting Started
Step 2: Review Notes and Warnings
Step 3: Backing Up the Cluster
Step 4: Back Up Cloudera Manager
Collect Information
Back Up Cloudera Manager Agent
Back Up the Cloudera Management Service
Back Up Cloudera Navigator Data
Stop Cloudera Manager Server & Cloudera Management Service
Back Up the Databases
Back Up Cloudera Manager Server
(Optional) Start Cloudera Manager Server & Cloudera Management Service
Step 5: Complete Pre-Upgrade steps for upgrades to CDP Private Cloud Base
Run Hue Document Cleanup
Check Oracle Database Initialization
Step 6: Access Parcels
Step 7: Configure Streams Messaging Manager
Step 8: Configure Schema Registry
Step 9: Enter Maintenance Mode
Step 10: Run the Upgrade Cluster Wizard
Fair Scheduler to Capacity Scheduler transition
Plan your scheduler transition
Scheduler transition limitations
Placement Rules transition
Auto-converted Fair Scheduler properties
Fair Scheduler features and conversion details
Use the fs2cs conversion utility
CLI options of the fs2cs conversion tool
Manual configuration of scheduler properties
Manually add the configurations of yarn-site.xml
Use YARN Queue Manager UI to configure scheduler properties
Use Cloudera Manager Safety Valves to configure scheduler properties
Configure TLS/SSL for Ranger in a manually configured TLS/SSL environment
Step 11: Finalize the HDFS or Ozone Upgrade
Step 12: Complete Post-Upgrade steps for upgrades to CDP Private Cloud Base
Step 13: Exit Maintenance Mode
Troubleshooting
Manual upgrade to CDP Private Cloud Base
Rolling back a CDP Private Cloud Base 7 upgrade to CDH 5
Configuring a Local Package Repository
Configuring a Local Parcel Repository
CDH 5 to CDP Private Cloud Base post-upgrade transition steps
Update permissions for Replication Manager service
Migrating Spark workloads to CDP
Spark 1.6 to Spark 2.4 Refactoring
Handling prerequisites
Spark 1.6 to Spark 2.4 changes
New Spark entry point SparkSession
Dataframe API registerTempTable deprecated
union replaces unionAll
Empty schema not supported
Referencing a corrupt JSON/CSV record
Dataset and DataFrame API explode deprecated
CSV header and schema match
Table properties support
Managed table location
Write to Hive bucketed tables
Rounding in arithmetic operations
Precedence of set operations
HAVING without GROUP BY
CSV bad record handling
Spark 2.4 CSV example
Configuring storage locations
Querying Hive managed tables from Spark
Compiling and running Spark workloads
Compiling and running a Java-based job
Compiling and running a Scala-based job
Running a Python-based job
Running a job interactively
Post-migration tasks
Spark 2.3 to Spark 2.4 Refactoring
Handling prerequisites
Spark 2.3 to Spark 2.4 changes
Empty schema not supported
CSV header and schema match
Table properties support
Managed table location
Precedence of set operations
HAVING without GROUP BY
CSV bad record handling
Spark 2.4 CSV example
Configuring storage locations
Querying Hive managed tables from Spark
Compiling and running Spark workloads
Post-migration tasks
Apache Hive Expedited Migration Tasks
Preparing tables for migration
Check SERDE Definitions and Availability
Handle Missing Table or Partition Locations
Managed Table Location Mapping
Make Tables SparkSQL Compatible
Creating a list of tables to migrate
Migrating tables to CDP
Apache Hive Changes in CDP
Hive Configuration Property Changes
LOCATION and MANAGEDLOCATION clauses
Handling table reference syntax
Add Backticks to Table References
Key semantic changes and workarounds
Casting timestamps
Casting invalid dates
Changing incompatible column types
Understanding CREATE TABLE behavior
Configuring legacy CREATE TABLE behavior
Handling the Keyword APPLICATION
Disabling Partition Type Checking
Dropping partitions
Handling output of greatest and least functions
Renaming tables
TRUNCATE TABLE on an external table
Hive unsupported interfaces and features
Changes to CDH Hive Tables
Changes to HDP Hive tables
Apache Hive Post-Upgrade Tasks
Customizing critical Hive configurations
Setting Hive Configuration Overrides
Hive Configuration Requirements and Recommendations
Configuring HiveServer for ETL using YARN queues
Removing Hive on Spark Configurations
Configuring authorization to tables
Making the Hive plugin for Ranger visible
Setting up access control lists
Configure encryption zone security
Configure edge nodes as gateways
Spark integration with Hive
Configure HiveServer HTTP mode
Configuring HMS for high availability
Installing Hive on Tez and adding a HiveServer role
Updating Hive and Impala JDBC/ODBC drivers
Getting the JDBC driver
Getting the ODBC driver
Apache Impala changes in CDP
Set ACLs for Impala
Impala Configuration Changes
Interoperability between Hive and Impala
Revert to CDH-like Tables
Authorization Provider for Impala
Data Governance Support by Atlas
Handling Data Files
Hue post-upgrade tasks
Updating group permissions for Hive query editor
Adding Security Browser to the blocked list of applications
Adding Query Processor service to a cluster
Importing Sentry privileges into Ranger policies
Apache Knox - create plugin audit directory
Apache Ranger TLS Post-Upgrade Tasks
Migrating ACLs from Key Trustee KMS to Ranger KMS
Key Trustee KMS operations not supported by Ranger KMS
ACLs supported by Ranger KMS and Ranger KMS Mapping
Cloudera Search post-upgrade tasks
Bootstrapping Solr collections
Recreating aliases
Upload aliases.json to the upgraded cluster
Reindexing Solr collections
Apache Hadoop YARN default value changes
Apache ZooKeeper ACLs: YARN
Upgrade Notes for Apache Kudu 1.12 / CDP 7.1
Apache HBase post-upgrade tasks
Configure SMM to monitor SRM replications
Configure SMM's service dependency on Schema Registry
Apache Sqoop Changes
Applications Upgrade