Setting up Cloudera Iceberg REST Catalog for data sharing

Installation and Upgrade Scenarios

Different versions of Cloudera Runtime and Data Lakes require different steps to enable data sharing.

• Fresh 7.3.2.20000 or Higher Cloudera Base on premises Installation
• Upgrade from a lower version of Cloudera Base on premises to a 7.3.2.20000 or Higher version of Cloudera Base on premises

Fresh 7.3.2.20000 or Higher Cloudera Base on premises Installation

1. Installing the Metering V2 Service

   Install and configure the Metering V2 service. This is a prerequisite for Cloudera Data Sharing on Ozone storage. If Metering V2 is TLS-enabled, also import the Metering V2 certificate authority into the Hive Metastore truststore.

2. HMS REST Catalog configuration for Ozone storage

   Enable the Iceberg REST Catalog in the Hive Metastore and point it at your Ozone Object Manager and S3 Gateway so Cloudera Data Sharing can vend temporary S3 credentials through the gateway. Only Iceberg tables whose storage locations use s3a:// paths are shareable; tables registered with ofs:// locations only cannot be shared this way.

3. Installing the Knox IDBroker

   Add the IDBroker role to the Knox service. On premises deployments do not provision the IDBroker automatically, so you must install it manually before you can configure it for Ozone S3 credential vending.

4. Configuring Ozone STS for on premises Data Sharing

   Enable the Ozone Security Token Service (STS) and the Ranger action-matcher condition, and then complete the Ranger role, Knox IDBroker, certificate, and topology configuration required so that REST Catalog can vend temporary Ozone S3 credentials.

5. Creating the Ranger role for Ozone STS

   Create the sts-assume-role Ranger role and grant the hive user Assume Role access so that Ozone STS can generate session credentials for the REST Catalog.

6. Configuring the Knox IDBroker for Ozone S3 environments

   Configure the Knox IDBroker STS endpoint override, session policy template, and AWS user mapping so that IDBroker can vend temporary Ozone S3 credentials to external clients.

7. Trusting the Ozone STS certificate in the Knox IDBroker truststore

   Import the cluster's SCM Local CA into the Knox IDBroker truststore so that IDBroker can call the Ozone STS and S3 Gateway endpoint over HTTPS.

8. Declaring Knox topologies for on premises Data Sharing

   Declare the cdp-datashare-access and cdp-share-management Knox topologies so external clients can reach REST Catalog through Knox and administrators can generate the CLIENT_ID and CLIENT_SECRET used by external clients.

9. Registering external clients for on premises deployments

   Generate a CLIENT_ID and CLIENT_SECRET for the external client through the cdp-share-management Knox topology, create the Ranger group and role that represent the client, and then create the Data Share policy in Managing Ranger policies to grant that role SELECT access to the shared Iceberg tables.

Upgrade from a lower version of Cloudera Base on premises to a 7.3.2.20000 or Higher version of Cloudera Base on premises

1. Installing the Metering V2 Service

   Install and configure the Metering V2 service. This is a prerequisite for Cloudera Data Sharing on Ozone storage. If Metering V2 is TLS-enabled, also import the Metering V2 certificate authority into the Hive Metastore truststore.

2. HMS REST Catalog configuration for Ozone storage

   Enable the Iceberg REST Catalog in the Hive Metastore and point it at your Ozone Object Manager and S3 Gateway so Data Sharing can vend temporary S3 credentials through the gateway. Only Iceberg tables whose storage locations use s3a:// paths are shareable; tables registered with ofs:// locations only cannot be shared this way.

3. Installing the Knox IDBroker

   Add the IDBroker role to the Knox service. On premises deployments do not provision the IDBroker automatically, so you must install it manually before you can configure it for Ozone S3 credential vending.

4. Configuring Ozone STS for on premises Data Sharing

   Enable the Ozone Security Token Service (STS) and the Ranger action-matcher condition, and then complete the Ranger role, Knox IDBroker, certificate, and topology configuration required so that REST Catalog can vend temporary Ozone S3 credentials.

5. Creating the Ranger role for Ozone STS

   Create the sts-assume-role Ranger role and grant the hive user Assume Role access so that Ozone STS can generate session credentials for the REST Catalog.

6. Configuring the Knox IDBroker for Ozone S3 environments

   Configure the Knox IDBroker STS endpoint override, session policy template, and AWS user mapping so that IDBroker can vend temporary Ozone S3 credentials to external clients.

7. Trusting the Ozone STS certificate in the Knox IDBroker truststore

   Import the cluster's SCM Local CA into the Knox IDBroker truststore so that IDBroker can call the Ozone STS and S3 Gateway endpoint over HTTPS.

8. Declaring Knox topologies for on premises Data Sharing

   Declare the cdp-datashare-access and cdp-share-management Knox topologies so external clients can reach REST Catalog through Knox and administrators can generate the CLIENT_ID and CLIENT_SECRET used by external clients.

9. Registering external clients for on premises deployments

   Generate a CLIENT_ID and CLIENT_SECRET for the external client through the cdp-share-management Knox topology, create the Ranger group and role that represent the client, and then create the Data Share policy in Managing Ranger policies to grant that role SELECT access to the shared Iceberg tables.