Cloudera Docs

Setting up Parameter Context inheritance

Follow these steps to configure inherited Parameter Contexts.

Now that you have the individual Parameter Contexts set up, you can begin grouping them together, and create higher-level Parameter Contexts to build out your inheritance hierarchy and use them with the two Process Groups (Source ABC Kafka to S3 and Source DEF Kafka to GCS). The goal is to have one Parameter Context for each Process Group, composed of smaller units.

In this example, site properties and Kafka parameters are needed for both Process Groups. Additionally, each Process Group will have a source-related and a destination-related Parameter Context.

You have created two Process Groups in Apache NiFi:
  • Source ABC Kafka to S3 that needs parameters for your Kafka brokers, Source ABC’s topic, and for the S3 bucket
  • Source DEF Kafka to GCS that needs parameters for the same Kafka brokers, Source DEF’s topic, and for the GCP bucket
  • In both Process Groups, you would like to use some common parameters tagging flowfiles with site-specific information
  1. Design the inheritance logic.

    In this example, the following structure is used:

    • Kafka and Site - inherits from Kafka and Site Properties
    • ABC Kafka and Site - inherits from Kafka and Site and Source ABC
    • DEF Kafka and Site - inherits from Kafka and Site and Source DEF
    • ABC Kafka to S3 - inherits from ABC Kafka and Site and S3
    • DEF Kafka to GCS - inherits from DEF Kafka and Site and GCP Cloud Storage

    Based on the above structure, the hierarchy for the two top-level Parameter Contexts looks like this:

    ABC Kafka to S3 Parameter Context
    • ABC Kafka and Site Parameter Context
      • Kafka and Site Parameter Context
        • Kafka Parameter Context
          • Kafka Brokers parameter with value localhost:9092
          • Kafka Group ID parameter with value MyGroup
        • Site Properties Parameter Context
          • Site Identifier parameter with value 1234
          • Site Data Manager parameter with value MyDM
      • Source ABC Parameter Context
        • Kafka Topic parameter with value source-abc-topic
    • S3 Parameter Context
      • S3 Bucket parameter with value my-bucket
      • S3 Region parameter with value us-west-2

    DEF Kafka to GCS Parameter Context

    • DEF Kafka and Site Parameter Context
      • Kafka and Site Parameter Context
        • Kafka Parameter Context
          • Kafka Brokers parameter with value localhost:9092
          • Kafka Group ID parameter with value MyGroup
        • Site Properties Parameter Context
          • Site Identifier parameter with value 1234
          • Site Data Manager parameter with value MyDM
      • Source DEF Parameter Context
        • Kafka Topic parameter with value source-def-topic
    • GCP Cloud Storage Parameter Context
      • GCP Project ID parameter with value my-project
      • GCP Bucket parameter with value my-bucket
  2. Configure Parameter Context inheritance in NiFi for ABC Kafka to S3.
    1. Create a new Parameter Context named Kafka and Site and go to the INHERITANCE tab.
      You can see all of the individual Parameter Contexts.
    2. Drag Kafka and Site Properties to the right under Selected Parameter Context.
    3. Click APPLY, and then edit the Kafka and Site Parameter Context to see the new parameters.

      All four parameters from the two Parameter Contexts are now available in the view, and each of them has an (arrow) icon. You can click one of these to navigate to the Parameter Context in which the parameter is defined and edit its value.

    4. Repeat steps 1-3 to create ABC Kafka and Site Parameter Context inheriting all parameters from the Kafka and Site and the Source ABC Parameter Contexts.
    5. Finally, repeat steps 1-3 to create ABC Kafka to S3 Parameter Context inheriting all parameters from the ABC Kafka and Site and the S3 Parameter Contexts.
    This is how ABC Kafka to S3 looks like after the full setup:
  3. Configure Parameter Context inheritance for DEF Kafka to GCS. Follow the steps you completed when creating the ABC Kafka to S3 Parameter Context.