CPU based auto-scaling

When you turn this feature on, Cloudera DataFlow uses the Kubernetes Horizontal Pod Autoscaler (HPA) to increase or decrease the number of NiFi pods in a deployment based on resource utilization.

HPA monitors CPU utilization across all NiFi pods of a flow deployment. When CPU utilization reaches 75%, a new NiFi pod gets created and is added to the NiFi cluster. HPA continues to add additional NiFi pods until the aggregated CPU utilization for a deployment becomes stable around the 75% threshold. Once the new NiFi pod has joined the NiFi cluster, it starts receiving and processing data.

A similar logic is applied to automatically scale down and decrease the number of NiFi pods. HPA monitors CPU utilization and when it falls below 75% for five minutes it starts removing NiFi pods. Before the NiFi pod is terminated, Cloudera DataFlow offloads all existing data from the NiFi pod that has been selected for termination. Offloading data distributes data that is currently being processed by the affected node to the remaining nodes in the cluster. This offloading procedure ensures that NiFi pods are terminated gracefully and avoids data loss.

Use cases

CPU based auto-scaling is great for data flows that require CPU intensive transformations like compressing or decompressing data and have to deal with bursts of data every now and then.