Supported REST Catalog APIs for accessing the data

To use the Iceberg data sharing in Cloudera, you must employ the REST client to perform specific operations.

The REST APIs from the specification defined by Apache Iceberg are available in the REST Catalog open API specification. Cloudera currently supports the REST APIs that allow read operations on Iceberg tables:

On premises gateway endpoints and Ozone S3 credential verification

For Cloudera on premises deployments, external clients and administrators reach the data-sharing gateway paths through the Knox topology instead of a cloud load balancer:

  • https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces — client ID and client secret generation.
  • https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/... — REST Catalog namespace, table, and credentials endpoints.

When a table is stored on Cloudera Object Store (powered by Apache Ozone) S3, load the table and confirm that the response contains a s3a:// credential prefix that matches the translated Ozone link bucket, along with a session token:

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables/<table>

A successful response includes a config block similar to the following:

{
  "s3.access-key-id": "ASIA...",
  "s3.secret-access-key": "...",
  "s3.session-token": "...",
  "s3.endpoint": "https://[***OZONE_S3_GATEWAY_HOST***]:9879",
  "s3.path-style-access": "true",
  "s3.remote-signing-enabled": "false"
}

Retrieving the access token

Retrieving the access token works the same for all supported endpoints:
[***MY-TOKEN-NAME***]=$(curl -k -X POST -H  "Content-Type: application/x-www-form-urlencoded" -d "client_id=[***CLIENT ID***]&client_secret=[***CLIENT SECRET***]&grant_type=client_credentials" "https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/knoxtoken/api/v2/token"  | jq -r '.access_token')
Using the endpoints must be always preceded by retrieving the access token.

List Databases: /v1/{prefix}/namespaces

List all namespaces at a certain level, optionally starting from a given parent namespace.

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces
My-token-name=$(curl -k -X POST -H  "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token"  | jq -r '.access_token')

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces

List Tables: /v1/{prefix}/namespaces/{namespace}/tables

List all table identifiers under the specified namespace.

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables

Example:

My-token-name=$(curl -k -X POST -H  "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token"  | jq -r '.access_token')

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces/hive_rest_airline_orc/tables

Load Tables: /v1/{prefix}/namespaces/{namespace}/tables/{table}

Load a table from the catalog.

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables/icebergtable

Example:

My-token-name=$(curl -k -X POST -H  "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token"  | jq -r '.access_token')

curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces/hive_rest_airline_orc/tables/airport_iceberg_external