Restoring a Cloudera Data Engineering Service with configuration replacement

Cloudera Data Engineering supports configuration replacement while restoring a Cloudera Data Engineering Service, allowing you to make changes to the recorded backup configuration, for example, to modify the deprecated instance type.

The existing service-property-override option remains available, but you can specify only one of the following options in the restore command:

service-property-override - it only modifies a part of the configuration, and merges the modified part to the original configuration.
service-config-replacement - it covers all the configurations, and it is a complete replacement rather than a merge operation.

Install one of the following:
- To use the CLI, install and configure CDP CLI. For more information, see CLI client setup.
- To use the API, install cdpcurl. For more information, see cdpcurl.
Install the JQ JSON processor. For more information see JQ.

Get the restore configuration.
- Use the CLI by running the following command:
```
# specify CDP_CMD that contains common configs like profile and extra auth info
export CDP_CMD=cdp --profile xxx ...

$CDP_CMD de describe-backup --backup-id $[***BACKUP-ID***]  --include-restore-request > [***OLD-REQUEST.JSON***]
```
- Use the API by running the following command:
  To get the Base URL that is needed for this command, use one of the following methods:
  - Obtain the BASE-URL by copying the console URL up to the first forward slash (/).
    Example for the console URL: https://console.us-west-1.cdp.cloudera.com/dex/home
    Example for the BASE-URL, which changes according to the region: https://console.us-west-1.cdp.cloudera.com
  - Alternatively:
    1. In the Cloudera Data Engineering UI, click Administration on the left navigation menu.
    2. Right click on the Administration page and select Inspect from the list of options.
    3. Click the Console tab.
    4. Type window.location.origin and press Enter.
```
cdpcurl -X POST -f string -d '{"filter":["id[eq][***BACKUP-ID***]"]}' [***BASE-URL***]/dex/api/v1/cluster/backup/list| jq '.BackupInfo[0].restoreRequest' > [***OLD-REQUEST.JSON***]
```

Modify the configuration.

Copy and change the configuration that is under the config and app_instances fields. Other fields, such as id and env are ignored.

note
Consult Cloudera Support if you are not sure which fields need to be modified.

note
JQ is recommended for large-scale changes, or for auditing changes. For minor changes, you can also make manual modifications. Write the result to a new file and review it before proceeding to the next step.

Check Basic filters to get more information on the available JQ filters. Consider the following common use cases:

# merge partial json config with ". *" syntax. All the specified json will be merged with the original configuration.
jq '. * {"config":{"properties":{"custom.property":"new-value"},"resources":{"max_instances":"20","min_instances":"3"}}}' [***OLD-REQUEST.JSON***] > [***NEW-REQUEST.JSON***]

# modify instance_type and subnets with jq = syntax
jq '.config.properties.subnets = "[***EXAMPLE***]"
  | .config.resources.instance_type = "Standard_D8s_v5"
  | .config.resources.allp_instance_group_details.instance_type = "Standard_D8s_v5"
' [***OLD-REQUEST.JSON***] > [***NEW-REQUEST.JSON***]

# modify blocks like instance size with jq += syntax 
jq '.config.resources.allp_instance_group_details += {"max_instances":"20","min_instances":"2"} | .config.resources += {"max_instances":"20","min_instances":"2"}' [***OLD-REQUEST.JSON***] > [***NEW-REQUEST.JSON***]

# modify one array item
jq '(.app_instances[] | select(.name=="Default-Virtual-Cluster")).config.resources += {"cpu_requests":"8","mem_requests":"32Gi"}
' [***OLD-REQUEST.JSON***] > [***NEW-REQUEST.JSON***]

Perform the restore operation with configuration replacement.

Use the CLI by running the following command:

$CDP_CMD de restore-service --backup-id $[***BACKUP-ID***] --environment-crn $[***ENV-CRN***]
  --service-config-replacement "$(cat [***NEW-REQUEST.JSON***])"

Use the API by running the following command:

cdpcurl -X POST -v -f string \
  -d "$(jq -n \
    --argjson override "$(< [***NEW-REQUEST.JSON***])" \
    '{
      backup_id: [***BACKUP-ID***],
      environment_crn: "[***ENV-CRN***]",
      config_override: $override
    }')" \
  [***BASE-URL****]/dex/api/v1/cluster/backup/restore