Supported REST Catalog APIs for accessing the data
To use the Iceberg data sharing in Cloudera, you must employ the REST client to perform specific operations.
The REST APIs from the specification defined by Apache Iceberg are available in the REST Catalog open API specification. Cloudera currently supports the REST APIs that allow read operations on Iceberg tables:
On premises gateway endpoints and Ozone S3 credential verification
For Cloudera on premises deployments, external clients and administrators reach the data-sharing gateway paths through the Knox topology instead of a cloud load balancer:
https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces— client ID and client secret generation.https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/...— REST Catalog namespace, table, and credentials endpoints.
When a table is stored on Cloudera Object Store (powered by Apache Ozone) S3, load the
table and confirm that the response contains a s3a:// credential
prefix that matches the translated Ozone link bucket, along with a session
token:
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables/<table>
A successful response includes a config block similar to the
following:
{
"s3.access-key-id": "ASIA...",
"s3.secret-access-key": "...",
"s3.session-token": "...",
"s3.endpoint": "https://[***OZONE_S3_GATEWAY_HOST***]:9879",
"s3.path-style-access": "true",
"s3.remote-signing-enabled": "false"
}
Retrieving the access token
[***MY-TOKEN-NAME***]=$(curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=[***CLIENT ID***]&client_secret=[***CLIENT SECRET***]&grant_type=client_credentials" "https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/knoxtoken/api/v2/token" | jq -r '.access_token')Using
the endpoints must be always preceded by retrieving the access token.List Databases: /v1/{prefix}/namespaces
List all namespaces at a certain level, optionally starting from a given parent namespace.
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces
My-token-name=$(curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token" | jq -r '.access_token')
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces
List Tables: /v1/{prefix}/namespaces/{namespace}/tables
List all table identifiers under the specified namespace.
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables
Example:
My-token-name=$(curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token" | jq -r '.access_token')
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces/hive_rest_airline_orc/tables
Load Tables: /v1/{prefix}/namespaces/{namespace}/tables/{table}
Load a table from the catalog.
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $[***MY-TOKEN-NAME***]" https://[***KNOX-GATEWAY-HOST***]/[***DATALAKE-NAME***]/cdp-datashare-access/iceberg-rest/v1/namespaces/<namespace>/tables/icebergtable
Example:
My-token-name=$(curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "client_id=b9efc3dd-3695-4867-9e4c-523389c8a78b&client_secret=WWpsbFptTXpaR1F0TXpZNU5TMDBPRFkzTFRsbE5HTXROVEl6TXpnNVl6aGhOemhpOjpNRGt5TmpneE1HUXRZak00TmkwMFpqTXhMVGczTTJZdFptTXhOekl6TmpVNFlqazI=&grant_type=client_credentials" "https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/knoxtoken/api/v2/token" | jq -r '.access_token')
curl -ivk -X GET -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $My-token-name" https://my-cluster-gateway.example.cldr.work/my-cluster/cdp-datashare-access/iceberg-rest/v1/namespaces/hive_rest_airline_orc/tables/airport_iceberg_external
