Understanding Cloudera Data Warehouse backup and restore

This topic explains how to resolve backup and restore job failures caused by unpatched inline scripts on private cloud clusters running older Cloudera Data Warehouse operator versions.

Condition

Backup and restore jobs on older Cloudera Data Warehouse versions embed shell scripts inline within the job specifications. Two issues cause these jobs to fail:

WebHDFS active namenode probe failure: Hue and Viz backup or restore scripts contain a get_active_namenode() function that uses GETCONTENTSUMMARY to probe WebHDFS namenodes. On secured clusters, this returns HTTP 403, causing the probe and the job to fail.
Missing Hue database failure: The Database Catalog level Hue backup script (backup-hue.sh) runs pg_dump against the _huedb database. If no Virtual Warehouses have been created yet, this database does not exist, and the backup job fails.

Solution

Deploy a mutating admission webhook as a temporary workaround. The webhook intercepts created Job specifications in Cloudera Data Warehouse namespaces (warehouse-, viz-, cdw-hue-, hive-, impala-, trino-) and patches the scripts in memory before execution.

Prerequisites:

kubectl CLI access with cluster-admin privileges.
openssl installed locally to generate TLS certificates.

To set environment variables, define the deployment parameters and retrieve the matching container image tag present on your cluster.
1. Query all container images on the cluster and extract the autoscaler image tag matching your Cloudera Data Warehouse operator version:
```
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | \
  grep impala-autoscaler-webui-metrics | sort -u
```
2. Set the environment variables in your terminal session, making sure WEBHOOK_NS is your target namespace, WEBHOOK_NAME is the resource prefix, and WEBHOOK_IMAGE is the image path retrieved in the previous step:
```
WEBHOOK_NS="cdp"
WEBHOOK_NAME="dwx-backup-script-patcher"
WEBHOOK_IMAGE="docker-private.infra.cloudera.com/cloudera/impala-autoscaler-webui-metrics:1.11.0-h4-b1"
```
3. Confirm that your terminal loaded the full image path correctly into the environment:
```
echo $WEBHOOK_IMAGE
```

To generate TLS certificates, Kubernetes requires mutating webhooks to serve HTTPS. Use openssl to generate a Certificate Authority (CA) and server certificates.

Generate the authority private key and certificate:

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 30 \
  -subj "/CN=${WEBHOOK_NAME}-ca" -out ca.crt

Generate the server private key and Certificate Signing Request (CSR):

openssl genrsa -out server.key 2048
openssl req -new -key server.key \
  -subj "/CN=${WEBHOOK_NAME}.${WEBHOOK_NS}.svc" \
  -out server.csr

Create the extension configuration file and sign the server certificate with Subject Alternative Names (SAN):

cat > server-ext.cnf <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:${WEBHOOK_NAME}.${WEBHOOK_NS}.svc,DNS:${WEBHOOK_NAME}.${WEBHOOK_NS}.svc.cluster.local
EOF

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 30 -sha256 \
  -extfile server-ext.cnf

Encode the base64 CA certificate string into an environment variable:
```
CA_BUNDLE=$(base64 < ca.crt | tr -d '\n')
```
Confirm that the required certificate output files exist in your working directory:
```
ls -l server.crt server.key ca.crt
```

To create the TLS secret, upload the generated certificate and server key to a Secret in the target namespace.
1. Create the TLS secret by uploading the key pair to Kubernetes:
```
kubectl -n "${WEBHOOK_NS}" create secret tls "${WEBHOOK_NAME}-tls" \
  --cert=server.crt --key=server.key
```
2. Confirm the Kubernetes secret in your target namespace:
```
kubectl -n "${WEBHOOK_NS}" get secret "${WEBHOOK_NAME}-tls"
```

To deploy the Python HTTPS webhook server, create a ConfigMap containing the script patching logic and then apply the corresponding Deployment and Service manifests.

Apply the ConfigMap manifest containing the Python script that implements both op=GETFILESTATUS and database existence checks:

kubectl -n "${WEBHOOK_NS}" apply -f - <<'CM_EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: dwx-backup-script-patcher-code
data:
  webhook.py: |
    #!/usr/bin/env python3
    import base64, http.server, json, ssl, sys

    VALID_NS_PREFIXES = ("warehouse-", "viz-", "cdw-hue-", "hive-", "impala-", "trino-")

    OLD_OP = "op=GETCONTENTSUMMARY"
    NEW_OP = "op=GETFILESTATUS"
    OLD_OUTPUT = '-o /tmp/output -w "%{http_code}" -X GET "$PROTOCOL://$namenode/webhdfs'
    NEW_OUTPUT = '-o /tmp/probe_output -w "%{http_code}" -X GET "$PROTOCOL://$namenode/webhdfs'
    OLD_IF_BLOCK = (
        '    if [ $HTTP_STATUS -eq 200 ]; then\n'
        '        host=$namenode\n'
        '    fi'
    )
    NEW_IF_BLOCK = (
        '    if [ $HTTP_STATUS -eq 200 ]; then\n'
        '        host=$namenode\n'
        '    else\n'
        '        echo "Namenode $namenode returned HTTP $HTTP_STATUS: $(cat /tmp/probe_output)" 1>&2\n'
        '    fi'
    )

    def patch_webhdfs(script):
        if OLD_OP not in script:
            return script, False
        patched = script.replace(OLD_OP, NEW_OP)
        patched = patched.replace(OLD_OUTPUT, NEW_OUTPUT)
        patched = patched.replace(OLD_IF_BLOCK, NEW_IF_BLOCK)
        return patched, patched != script

    DB_CHECK_ANCHOR = 'CHECKSUM_FILE="${BACKUP_TEMP_PATH}/checksum.md5"\n\ngraceful_exit_on_error()'
    DB_CHECK_REPLACEMENT = (
        'CHECKSUM_FILE="${BACKUP_TEMP_PATH}/checksum.md5"\n'
        '\n'
        'DB_CHECK_STDERR="${BACKUP_TEMP_PATH}/hue_db_check_err.log"\n'
        'PGPASSFILE="${POSTGRES_PWD_FILE}" psql \\\n'
        '  --host="${HUE_RDS_HOST}" --port="${HUE_RDS_PORT}" --username="${HUE_RDS_USER}" --dbname="${HUE_DB}" --no-password \\\n'
        '  -c "SELECT 1" > /dev/null 2>"${DB_CHECK_STDERR}"\n'
        'DB_CHECK_RC=$?\n'
        '\n'
        'if [ "${DB_CHECK_RC}" -ne 0 ]; then\n'
        "  if grep -q 'database .* does not exist' \"${DB_CHECK_STDERR}\"; then\n"
        '    echo "Database ${HUE_DB} does not exist, skipping backup" > "${SUCCESS_FILE}"\n'
        '    rm -f "${DB_CHECK_STDERR}"\n'
        '    exit 0\n'
        '  fi\n'
        '  echo "Failed to connect to database ${HUE_DB}:" 1>&2\n'
        '  cat "${DB_CHECK_STDERR}" 1>&2\n'
        '  mv "${DB_CHECK_STDERR}" "${FAILURE_FILE}"\n'
        '  exit 0\n'
        'fi\n'
        'rm -f "${DB_CHECK_STDERR}"\n'
        '\n'
        'graceful_exit_on_error()'
    )

    def patch_db_check(script):
        if 'pg_dump' not in script or 'HUE_DB' not in script:
            return script, False
        if 'DB_CHECK_STDERR' in script:
            return script, False
        if DB_CHECK_ANCHOR not in script:
            return script, False
        patched = script.replace(DB_CHECK_ANCHOR, DB_CHECK_REPLACEMENT)
        return patched, patched != script

    def mutate(review):
        req = review["request"]
        obj = req["object"]
        namespace = req.get("namespace", "")
        uid = req["uid"]

        if not any(namespace.startswith(p) for p in VALID_NS_PREFIXES):
            return {
                "apiVersion": "admission.k8s.io/v1",
                "kind": "AdmissionReview",
                "response": {"uid": uid, "allowed": True},
            }

        patches = []
        spec = obj.get("spec", {}).get("template", {}).get("spec", {})

        for section in ("initContainers", "containers"):
            for i, c in enumerate(spec.get(section, [])):
                args = c.get("args", [])
                if not args or not isinstance(args[0], str):
                    continue
                script = args[0]
                changed = False

                script, fix1 = patch_webhdfs(script)
                changed = changed or fix1

                script, fix2 = patch_db_check(script)
                changed = changed or fix2

                if changed:
                    patches.append({
                        "op": "replace",
                        "path": f"/spec/template/spec/{section}/{i}/args/0",
                        "value": script,
                    })

        resp = {
            "apiVersion": "admission.k8s.io/v1",
            "kind": "AdmissionReview",
            "response": {"uid": req["uid"], "allowed": True},
        }
        if patches:
            resp["response"]["patchType"] = "JSONPatch"
            resp["response"]["patch"] = base64.b64encode(json.dumps(patches).encode()).decode()
        return resp

    class Handler(http.server.BaseHTTPRequestHandler):
        def do_POST(self):
            body = self.rfile.read(int(self.headers["Content-Length"]))
            review = json.loads(body)
            result = mutate(review)
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps(result).encode())

    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain("/tls/tls.crt", "/tls/tls.key")
    server = http.server.HTTPServer(("0.0.0.0", 8443), Handler)
    server.socket = ctx.wrap_socket(server.socket, server_side=True)
    server.serve_forever()
CM_EOF

Apply the deployment workload and service endpoint to expose the webhook server on port 8443:

kubectl -n "${WEBHOOK_NS}" apply -f - <<DEPLOY_EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${WEBHOOK_NAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${WEBHOOK_NAME}
  template:
    metadata:
      labels:
        app: ${WEBHOOK_NAME}
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
      - name: webhook
        image: ${WEBHOOK_IMAGE}
        command: ["python3", "/app/webhook.py"]
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: tls
          mountPath: /tls
          readOnly: true
        - name: code
          mountPath: /app
          readOnly: true
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
      volumes:
      - name: tls
        secret:
          secretName: ${WEBHOOK_NAME}-tls
      - name: code
        configMap:
          name: ${WEBHOOK_NAME}-code
---
apiVersion: v1
kind: Service
metadata:
  name: ${WEBHOOK_NAME}
spec:
  selector:
    app: ${WEBHOOK_NAME}
  ports:
  - port: 443
    targetPort: 8443
DEPLOY_EOF

Monitor the deployment rollout status until the pod becomes ready:

kubectl -n "${WEBHOOK_NS}" rollout status deployment/${WEBHOOK_NAME} --timeout=60s

If the webhook is deployed into a namespace protected by a deny-all NetworkPolicy, explicitly allow incoming requests from the Kubernetes API server on port 8443 by configuring a dedicated ingress rule.
1. Run the following command to apply a NetworkPolicy that allows ingress traffic on port 8443:
```
kubectl -n "${WEBHOOK_NS}" apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ${WEBHOOK_NAME}-allow-apiserver
  namespace: ${WEBHOOK_NS}
spec:
  podSelector:
    matchLabels:
      app: ${WEBHOOK_NAME}
  policyTypes:
  - Ingress
  ingress:
  - ports:
    - protocol: TCP
      port: 8443
EOF
```
2. Confirm that the NetworkPolicy resource has been registered successfully:
```
kubectl -n "${WEBHOOK_NS}" get networkpolicy "${WEBHOOK_NAME}-allow-apiserver"
```

Execute the following command to register the mutating webhook configuration rules:

kubectl apply -f - <<EOF
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ${WEBHOOK_NAME}
webhooks:
- name: ${WEBHOOK_NAME}.${WEBHOOK_NS}.svc
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Ignore
  timeoutSeconds: 5
  rules:
  - apiGroups: ["batch"]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["jobs"]
  clientConfig:
    caBundle: ${CA_BUNDLE}
    service:
      name: ${WEBHOOK_NAME}
      namespace: ${WEBHOOK_NS}
      path: /mutate
      port: 443
EOF

Confirm that the MutatingWebhookConfiguration resource is active:
```
kubectl get mutatingwebhookconfiguration "${WEBHOOK_NAME}"
```

Trigger a test backup job and confirm that created scripts were modified by the webhook.

Execute the cluster backup operation using the CDP CLI:
```
cdp dw backup-cluster --cluster-id <cluster-id>
```
Inspect the webhook pod logs to verify that incoming admission requests are being processed:
```
kubectl -n "${WEBHOOK_NS}" logs -l app=${WEBHOOK_NAME} --tail=20
```

Run the Python validation script against the newly created backup job to verify its specification:

BACKUP_NS="<warehouse-namespace>"
JOB_NAME=$(kubectl -n "${BACKUP_NS}" get jobs -l backup-job-id \
  --sort-by=.metadata.creationTimestamp -o name | tail -1)

kubectl -n "${BACKUP_NS}" get "${JOB_NAME}" -o json | \
  python3 -c "
import json, sys
job = json.load(sys.stdin)
for section in ('initContainers', 'containers'):
    for c in job['spec']['template']['spec'].get(section, []):
        script = (c.get('args') or [''])[0]
        name = c['name']
        if 'get_active_namenode' in script:
            has_old = 'GETCONTENTSUMMARY' in script
            print(f\"{name} (webhdfs): {'NEEDS PATCH' if has_old else 'OK (patched)'}\")
        if 'pg_dump' in script and 'HUE_DB' in script:
            has_check = 'DB_CHECK_STDERR' in script
            print(f\"{name} (db-check): {'OK (patched)' if has_check else 'NEEDS PATCH'}\")"

Ensure that the verification output prints OK (patched) for both container types:
```
backup-shipper (webhdfs): OK (patched)
backup-hue (db-check): OK (patched)
```

After completing the backup or upgrading your Cloudera Data Warehouse operator, remove the temporary webhook resources:

Delete the active mutating webhook configuration to stop job interception immediately:
```
kubectl delete mutatingwebhookconfiguration "${WEBHOOK_NAME}"
```

Remove all supporting Kubernetes objects created for the workaround:

kubectl -n "${WEBHOOK_NS}" delete deployment "${WEBHOOK_NAME}"
kubectl -n "${WEBHOOK_NS}" delete service "${WEBHOOK_NAME}"
kubectl -n "${WEBHOOK_NS}" delete configmap "${WEBHOOK_NAME}-code"
kubectl -n "${WEBHOOK_NS}" delete secret "${WEBHOOK_NAME}-tls"
kubectl -n "${WEBHOOK_NS}" delete networkpolicy "${WEBHOOK_NAME}-allow-apiserver" --ignore-not-found

Clean up the local certificate artifacts from your machine:
```
rm -f ca.key ca.crt ca.srl server.key server.csr server.crt server-ext.cnf
```

Debugging and diagnostics

Webhook pod not starting

Run kubectl -n "WEBHOOK_NS" describepod-l app={WEBHOOK_NAME} and kubectl -n "WEBHOOK_NS" logs-l app={WEBHOOK_NAME} to review pod events.
Check if the TLS secret name is correct or if namespace Security Context constraints are blocking the pod.

Job still has GETCONTENTSUMMARY

Verify registration using kubectl get mutatingwebhookconfiguration ${WEBHOOK_NAME}.
Review logs to see if incoming requests reach the pod using kubectl -n "WEBHOOK_NS" logs -l app={WEBHOOK_NAME} --tail=50.
Check if a deny-all NetworkPolicy exists in the webhook namespace without the allow policy from Step 5.
Confirm that the target job namespace starts with a valid prefix (warehouse-, viz-, cdw-hue-, hive-, impala-, trino-).

Webhook is blocking job creation

Execute kubectl delete mutatingwebhookconfiguration ${WEBHOOK_NAME} to immediately disable interception.