Flow Management cluster layout

The Data Hub service includes two default Flow Management cluster definitions. These are the Flow Management: Light Duty and Flow Management: Heavy Duty definitions. Learn about the layout, capacity, and components of these definitions.

Flow Management: Light Duty cluster layout

You can use a Flow Management: Light Duty cluster definition in development, testing, or proof of concept scenarios. Light duty clusters include the following nodes and components (services):

Each of the cluster nodes includes:

• NiFi and ZooKeeper co-located on all instances

• For each node hosting NiFi and ZooKeeper:

  • AWS: m5.2xlarge
  • Azure: D8_v3
  • GCP: e2-standard-8
• Storage requirements per NiFi node:

  • AWS: 4 x 500 GB EBS ST1
  • Azure 4 x 500 GB Standard SSD
  • GCP: 4 x 500 GB PD-Standard
• Each NiFi node hosts the following repositories

  • FlowFile repository
  • Content repository
  • Provenance repository
  • Log and Database repository

For more information, see the Instance types and Storage information specific to your cloud provider.

Flow Management: Heavy Duty cluster layout

You can use the Flow Management: Heavy Duty cluster definition in production scenarios. Heavy duty clusters include the following nodes and components (services):

The cluster definition includes:

• NiFi and ZooKeeper on separate nodes

• NiFi nodes scale independently of ZooKeeper

• For each ZooKeeper node:

  • AWS – m5.2xlarge
  • Azure – D8_v3
  • GCP: e2-standard-8
• For each NiFi node:

  • AWS – m5.2xlarge
  • Azure – F16sv2
  • GCP: e2-standard-8
• Storage requirements per NiFi node:

  • AWS – 4 x 1 TB EBS GP2
  • Azure – 4 x 1 TB Premium SSD
  • GCP: 4 x 1 TB PD-SSD
• Each NiFi node hosts the following repositories

  • FlowFile repository
  • Content repository
  • Provenance repository
  • Log and Database repository

For more information, see the Instance types and Storage information specific to your cloud provider.