Best practices for Iceberg in CDP

Based on large scale TPC-DS and performance testing and real-world experiences, Cloudera recommends several best practices when using Iceberg.

Follow the best practices listed below when using Iceberg:

Use Iceberg as intended for analytics.
The table format is designed to manage a large, slow-changing collection of files. For more information, see the Iceberg spec.
Reduce read amplification
Monitor the growth of positional delta files, and perform timely compactions.
Speed up drop table performance, preventing deletion of data files by using the following table properties:
```
Set external.table.purge=false and gc.enabled=false
```
To improve concurrency on writes and reduce commit failures, tune the table properties: commit.retry.num-retries (default is 4),commit.retry.min-wait-ms (default is 100)
Tune the following table properties to improve concurrency on writes and reduce commit failures: commit.retry.num-retries (default is 4),commit.retry.min-wait-ms (default is 100)
Maintain a relatively small number of data files under the iceberg table/partition directory for efficient reads. To alleviate poor performance caused by too many small files, run the following queries:
```
TRUNCATE TABLE target;
INSERT OVERWRITE TABLE target select * from target FOR SYSTEM_VERSION AS OF <preTruncateSnapshotId>; 
```
To minimize the number of delete files and file handles and improve performance, ensure that the Spark write.distribution.mode table property value is “hash” (the default setting for Spark Iceberg 1.2.0 onwards).