Cloudera Docs

Troubleshooting Istio service mesh errors and pod failures

Diagnose and resolve common deployment errors, cross-namespace mTLS communication failures, and pod startup issues, and review relevant service mesh logs.

Cause and resolution

Use the following table to identify root causes and steps to resolve common Istio service mesh errors, deployment issues, and pod failures based on your active mesh mode:

Symptom Active Mode Cause Resolution
Waypoint proxy status is 0/1 with error violates PodSecurity "restricted:latest" Ambient Missing seccompProfile in istiod Helm values. Configure gateways.seccompProfile.type: RuntimeDefault in the istiod Helm chart values.
Trino worker in CrashLoopBackOff with BindException on port 35000. Sidecar Port conflict between Trino metrics and Istio Prometheus merge whenenablePrometheusMerge: True. Sidecar mode is not supported for Trino. Switch to ambient mode.
Pods missing istio-proxy sidecar after mode change. Sidecar Virtual Warehouse was not rebuilt after changing the mode. Rebuild the Virtual Warehouse in the Cloudera Data Warehouse UI to execute UpdateNamespace and apply labels.
Cross-namespace communication failures occur. Ambient or Sidecar mTLS configuration mismatch caused by incomplete rebuilds. Rebuild all remaining Database Catalogs and Virtual Warehouses to ensure uniform label alignment across namespaces.
Namespace displays incorrect or missing istio labels. Ambient or Sidecar Cloudera Data Warehouse server or worker deployments were not restarted, or components were not rebuilt.
  1. Restart Cloudera Data Warehouse server and worker deployments.
  2. Rebuild the affected Database Catalogs and Virtual Warehouses.
Hive Virtual Warehouse fails to start with Error 7080, hiveserver2 stuck in CrashLoopBackOff, HikariCP connection timeout, or error:
Host '<ip>' is blocked because of many connection errors; unblock with
                    'mysqladmin flush-hosts'
 Sidecar MySQL max_connect_errors limit exceeded on the external Metastore database due to aborted TCP handshakes from Istio sidecars during pod churn.
  1. Unblock host connections and update the limit on the MySQL server:
    • MySQL 8.4 or higher, run 
      TRUNCATE TABLE performance_schema.host_cache;
SET GLOBAL max_connect_errors = 100000;
    • MySQL 8.0.23 or older, run 
      FLUSH HOSTS;
  2. To persist this configuration across database restarts, add max_connect_errors = 100000 (or higher) under [mysqld] in the MySQL configuration file (/etc/my.cnf).