Cloudera Docs

Managing Ozone storage [Technical Preview]

Cloudera Migration Assistant provides comprehensive management of Apache Ozone storage structures on Cloudera Base clusters. You can scan existing Ozone storage, browse volumes and buckets, create new storage structures manually or by importing a YAML layout definition, and migrate data between HDFS and Ozone or between Ozone locations.

Ozone is a scalable, distributed object store optimized for big data workloads. In CMA, Ozone storage is organized in the following three-level hierarchy:

  • Volumes – Top-level containers that provide namespace isolation. Each volume has an owner, a namespace quota (number of buckets), and a space quota.

  • Buckets – Storage containers within a volume that hold keys (files and folders). Buckets have configurable layout types, replication settings, quotas, encryption, and snapshot options.

  • Folders – Directory structures within a bucket that organize data. Folders can be nested to create complex hierarchies.

CMA allows you to manage this hierarchy through the UI and API, track changes through server-sent events (SSE), and apply the storage layout to the actual Ozone cluster through the CMA Agent.

  • You must set up Cloudera Migration Assistant correctly. For instructions, see Deploying Cloudera Migration Assistant.

  • The destination cluster must have Apache Ozone installed and the Ozone service must be running.

  • You must register a Cloudera Base in CMA. If you do not have a cluster yet, for instructions, see Registering clusters.

  • You must configure Kerberos authentication correctly if the cluster is secured. CMA uses the configured migration user principal for Ozone operations.

  1. Scan Ozone storage to discover existing volumes, buckets, and folders.
    1. Click on the cluster you want to scan on the Clusters page.
    2. Click Start Scanning to open Scan Settings.
    3. Select Ozone scan.
    4. Click Scan selected.

      You are redirected to the scanning progress page where you can monitor whether the scan completed successfully or encountered an error.

  2. Browse the Ozone cluster in the cluster view.

    The scan results display volumes, buckets, and folder hierarchies. For each bucket, CMA displays the layout type, replication settings, quotas, and size information. For each volume, CMA displays the namespace quota, space quota, and owner.

    1. View all volumes on the cluster with the (owner, quotas, and bucket count attributes.
    2. Drill into a volume to view its buckets with the layout, replication type, quotas, encryption key, and GDPR enforcement attributes.
    3. Drill into a bucket to browse its folder hierarchy, including file sizes and ownership.
    4. View Ozone snapshots for buckets that have snapshots enabled.
    5. Filter folders by keyword, label, or related service such as Hive tables or databases that reference specific Ozone paths.
  3. Create a volume.
    1. Go to the Ozone storage view for the cluster.
    2. Click Add Volume.
    3. Provide the following properties:
      Table 1. Volume configuration properties
      Property Description
      Name Name of the volume (required)
      Namespace quota Maximum number of buckets allowed in the volume
      Space quota Maximum storage space for the volume, for example, 10TB
      User Owner of the volume
    4. Click Create.
  4. Update a volume.
    1. Select the volume you want to modify.
    2. Edit the mutable properties (namespace quota, space quota, user).
    3. Click Save.
  5. Delete a volume.
    1. Select the volume you want to delete.
    2. Click Delete.
  6. Create a bucket.
    1. Go to the volume where you want to create the bucket.
    2. Click Add Bucket.
    3. Provide the following properties:
      Table 2. Bucket configuration properties
      Property Description
      Name Name of the bucket (required)
      Layout Bucket layout type. Possible values are FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, or LEGACY
      Replication type Type of the replication. Can be RATIS or EC (Erasure Coding)
      Replication For RATIS: ONE or THREE. For EC: codec-data-parity-chunksize format (for example, RS-3-2-1024k)
      Namespace quota Maximum number of keys allowed in the bucket
      Space quota Maximum storage space for the bucket (for example, 1TB)
      User Owner of the bucket
      Bucket encryption key Name of the encryption key registered in the Ozone Key Management Server
      Enforce GDPR Enable Transparent Data Encryption for GDPR compliance
      Create snapshot Enable snapshot creation for the bucket
    4. Click Create.
  7. Update a bucket.
    1. Select the bucket you want to modify.
    2. Edit the mutable properties.
    3. Click Save.
  8. Delete a bucket.
    1. Select the bucket you want to delete.
    2. Click Delete.
  9. Create a folder.
    1. Go to the bucket where you want to create a folder.
    2. Click Add Folder.
    3. Provide the folder name.
    4. Click Create.
      Non-existent intermediate folders are created automatically.
  10. Rename a folder.
    1. Select the folder you want to rename.
    2. Provide the new name.
    3. Click Save.
  11. Delete a folder.
    1. Select the folder you want to delete.
    2. Click Delete.

Known limitations

No input validation

CMA does not validate layout definitions for volumes, buckets, or quotas.

No feasibility pre-check

CMA does not verify whether the specified settings are feasible on the target cluster.

Limited migration testing

Ozone Layout Provisioning has not been tested as part of the end-to-end migration workflow.

Use this feature for evaluation purposes only. Behavior and APIs may change in future releases.

CMA provides a Storage Layout Editor for designing and deploying Ozone storage structures.

Both the Storage Layout Editor and the Ozone browser use colors and text styles to indicate the status of volumes, buckets, and folders. The two views display different stages of the same lifecycle.

The Storage Layout Editor displays both unsaved (local) changes and changes that were saved to the database but not yet deployed to the cluster.

Table 3. Storage Layout Editor statuses
Status Color Style Description
New or locally modified (unsaved) Light red Normal Node was created or edited in the editor but not yet saved to the database.
Locally deleted (unsaved) Light red Strikethrough Node was deleted in the editor but the deletion has not been saved yet.
Saved, pending CREATE Red Normal Node was saved to the database and is queued for deployment to the cluster.
Saved, pending MODIFY Default Italic Node was saved to the database and is queued for deployment to the cluster.
Saved, pending DELETE Default Strikethrough Node was saved as a deletion and is queued for removal from the cluster.
Saved, no pending event Default Normal Node is deployed and up to date.

The Ozone browser only displays nodes that have been saved to the database. Unsaved editor changes are not visible here. If you have unsaved changes in the editor, the browser shows the last saved state of those nodes until you save.

Table 4. Ozone browser statuses
Status Color Style Description
Pending CREATE (saved, not yet deployed) Red Normal Node was created and saved, awaiting deployment to the cluster.
Pending MODIFY (saved, not yet deployed) Default Italic Node was modified and saved, awaiting deployment to the cluster.
Pending DELETE (saved, not yet deployed) Default Strikethrough Node is queued for deletion. All children of a pending-delete node are also displayed with strikethrough.
No pending event Default Normal Node is deployed and up to date.
  1. Plan and deploy an Ozone storage layout using the Storage Layout Editor.
    1. Go to the Layout tab in the cluster view to access the Storage Layout Editor.

      Design the layout structure in the Plan tab and apply the layout to the cluster in the Deploy tab.

      The Plan tab displays the following panels.

      • Left panel (Available Templates) — Contains reusable volume structures you can drag into the editor.

      • Right panel (Editor) — Is the working area where you build the target Ozone structure.

  2. Add volumes and buckets manually.
    1. Select the cluster on the Clusters page.
    2. Go to the Layout tab.
    3. In the Plan tab, click the Add Volume button at the top of the editor panel.
    4. Provide the volume properties in the drawer that opens, then click Create.
    5. To add buckets to a volume, select the volume node and click Add Bucket.
    6. To add folders to a bucket, select the bucket node and click Add Folder.
    7. Click Save Plan in the toolbar to save all pending changes to the database.
The Available Templates panel on the left displays reusable volume structures. You can use the following template types:

  • System templates — Built-in templates provided by Cloudera Migration Assistant. These cannot be deleted.

  • User templates — Templates you create from volumes in the editor. These can be deleted.

  1. Search and filter templates.

    Use the Search field at the top of the Available Templates panel to filter templates by name. Use the Type drop-down list to filter by template type.

  2. Drag a template into the editor.
    1. In the Available Templates panel, click the template name to expand it and see its volumes, buckets, and folders.
    2. Drag the node you want (volume, bucket, or folder) by its drag handle into the editor panel on the right.

      You can drag any node from a template directly into the editor to add it to the layout. Volumes, buckets, and folders can all be dragged individually using the drag handle (the grip icon on the right side of each node). When you drag a node, the full subtree (all its children) moves with it.

  3. Save a volume as a template.

    You can save any volume in the editor as a template to reuse it later. Saving a volume as a template stores the entire volume structure, including its buckets and folders, in the Available Templates panel.

    1. In the editor, hover over the volume node you want to save.
    2. Click Add to Template that is the download icon that is displayed on hover.
    3. Provide a name for the template.
    4. Click Save.

      The template is displayed in the Available Templates panel and is stored in the database.

  4. Delete a user template.
    1. In the Available Templates panel, hover over the template name.
    2. Click the Delete icon next to the template name.
    3. Click Delete to confirm the deletion.
  5. Import a layout from YAML or JSON.

    You can define a complete Ozone storage structure in a YAML or JSON file and import it into the editor.

    This example shows a YAML file defining volumes, buckets, and folders in a hierarchical structure.
    volumes:
          - name: "my-volume"
            namespace-quota: 100
            space-quota: "10TB"
            user: "admin"
            buckets:
              - name: "my-bucket"
                create-snapshot: true
                namespace-quota: 1000
                space-quota: "1TB"
                user: "admin"
                layout: "FILE_SYSTEM_OPTIMIZED"
                replication-type: "RATIS"
                replication: "THREE"
                bucketkey: "my-encryption-key"
                enforcegdpr: false
                folders:
                  - name: "data"
                    folders:
                      - name: "raw"
                      - name: "processed"
                  - name: "logs"
    1. In the Plan tab, click Import Structure in the toolbar.
    2. Upload the YAML or JSON file or paste the YAML or JSON content.
    3. Click Upload.

      CMA validates the structure and loads the volumes, buckets, and folders into the editor. Validation errors are displayed if the YAML or JSON is malformed or contains invalid values.

    4. Click Save Plan to save the imported structure to the database.
  6. Deploy the layout to the cluster.
    1. Click the Deploy tab in the Layout page.

      The Deployment Queue table displays all saved volumes, buckets, and folders that have pending changes. The CREATE, MODIFY, or DELETE events are not yet applied to the cluster.

    2. Review the pending changes in the Volumes, Buckets, and Folders tables.
    3. Click Deploy to Ozone.
    CMA sends the deploy command to the CMA Agent on the cluster. Monitor progress in the Commands tab on the cluster view. The agent processes the changes in the following order:
    1. Create, update, or delete volumes.
    2. Create, update, or delete buckets.
    3. Create, update, or delete folders.
  7. Use collections with Ozone locations.

    You can organize Ozone locations into collections using labels for visual organization. This is useful when preparing for migration.

    1. Go to the Ozone storage view and navigate to the folders you want to organize.
    2. Select the Ozone locations.
    3. Click Add to collection and select or create a collection.

      Collections with Ozone locations can be added to migration plans for HDFS-to-Ozone or Ozone-to-Ozone migration.

  8. Migrate HDFS to Ozone.
    CMA supports migrating data from HDFS to Ozone storage using Cloudera Replication Manager policies. The migration process consists of the following sequence:

    1. Ensure that the source cluster HDFS data is scanned and the destination cluster Ozone storage is set up.

    2. Create a migration plan that includes HDFS locations from the source cluster.

    3. In the structure mapping, map the source HDFS paths to destination Ozone paths (volume/bucket/folder).

    4. Create an execution from the plan and click Run.

    CMA uses the Cloudera Replication Manager Java API to create replication policies that copy data from HDFS to the mapped Ozone locations on the destination cluster.

  9. Create folders on the destination cluster during migration planning.
    1. In the migration plan split view, go to the destination cluster Ozone storage.
    2. Click Create Folder on the destination side.
    3. Provide the folder path.
    4. Click Create.

    The folder is created in the CMA data model and applied to the cluster during execution.

Ozone Storage Layout Editor — UI reference

Known limitations

No input validation

CMA does not validate layout definitions for volumes, buckets, or quotas.

No feasibility pre-check

CMA does not verify whether the specified settings are feasible on the target cluster.

Limited migration testing

Ozone Layout Provisioning has not been tested as part of the end-to-end migration workflow.

Use this feature for evaluation purposes only. Behavior and APIs may change in future releases.

Internal reference. This section documents detailed UI interactions for the Ozone Storage Layout Editor. For the main workflow, see the steps above.

Using templates

The Available Templates panel shows reusable volume structures. System templates are built-in and cannot be deleted. User templates are created from volumes in the editor and can be deleted.

Use Search to filter templates by name and the Type dropdown to filter by template type. Drag any node (volume, bucket, or folder) by its drag handle into the editor; the full subtree moves with it.

To save a volume as a template, hover over the volume, click Add to Template, provide a name, and click Save. To delete a user template, hover over the template name and click the delete icon.

Importing a layout from YAML or JSON

Define volumes, buckets, and folders in a hierarchical YAML or JSON file and import it into the editor from the Plan tab using Import Structure. Upload a file or paste content, then click Upload. CMA validates the structure and loads nodes into the editor. Click Save Plan to persist the imported structure.

Deploying the layout to the cluster

After saving a plan, open the Deploy tab. The Deployment Queue table lists volumes, buckets, and folders with pending CREATE, MODIFY, or DELETE events. Review pending changes, then click Deploy to Ozone.

CMA sends the deploy command to the CMA Agent, which processes volumes, then buckets, then folders. Monitor progress in the Commands tab on the cluster view.

Node status colors and styles

The Storage Layout Editor and Ozone browser use colors and text styles to indicate node status. In the editor, light red indicates unsaved local changes; red or italic or strikethrough styles indicate saved changes pending deployment. In the browser, red indicates pending CREATE; italic indicates pending MODIFY; strikethrough indicates pending DELETE.