Cloudera Docs

Using the Spark shell

Using Spark, you can create an Iceberg table followed by schema evolution, partition specification, and partition evolution.

You must configure the Spark shell as such you have included the valid Spark runtime version.

Run the following command in your Spark shell to create a new Iceberg table

  1. spark.sql("CREATE TABLE spark_catalog.default.sample_1 ( id bigint COMMENT 'unique id', data string) USING iceberg");
  2. Navigate accordingly in the Atlas UI to view the changes.
    The following images provide information about Iceberg table creation process.

    Run the following command in your Spark shell to create a Schema Evolution in a new table. For example - sample_2.

  3. spark.sql("CREATE TABLE spark_catalog.default.sample_2 ( id bigint COMMENT 'unique id', data string) USING iceberg");
  4. Navigate accordingly in the Atlas UI to view the changes.
    The following image provide information about Iceberg schema evolution process.

    Run the following command in your Spark shell to include a column:

  5. spark.sql("ALTER TABLE spark_catalog.default.sample_2 ADD COLUMN  (add_col_1 string )");
  6. Navigate accordingly in the Atlas UI to view the changes.
    The following images provide information about Iceberg schema creation process.

    Run the following command in your Spark shell to include the second column:

  7. spark.sql("ALTER TABLE spark_catalog.default.sample_2 ADD COLUMN  (add_col_2 string )");
  8. Navigate accordingly in the Atlas UI to view the changes.
    The following image provide information about Iceberg schema creation process.

    Run the following command in your Spark shell to create a Partition Specification in a new table (sample_3):

  9. spark.sql("CREATE TABLE spark_catalog.default.sample_3 (id bigint,data string,category string,ts timestamp) USING iceberg PARTITIONED BY (bucket(16, id), days(ts), category)");
  10. Navigate accordingly in the Atlas UI to view the changes.
    The following images provide information about Iceberg partition specification process.
    Run the following command in your Spark shell to create a Partition Evolution in a new table (sample_3):
  11. spark.sql("ALTER TABLE spark_catalog.default.sample_3 ADD PARTITION FIELD years(ts)");
  12. Navigate accordingly in the Atlas UI to view the changes.
    The following images provide information about Iceberg partition evolution process.