Homepage
/
Cloudera Flow Management
2.1.2
(Private Cloud)
Search Documentation
▶︎
Cloudera
Reference Architectures
▶︎
Cloudera Public Cloud
Getting Started
Patterns
Preview Features
Data Catalog
Data Engineering
DataFlow
Data Hub
Data Warehouse
Data Warehouse Runtime
Cloudera AI
Management Console
Operational Database
Replication Manager
DataFlow for Data Hub
Runtime
▼
Cloudera Private Cloud
Getting Started
Base
Upgrade
Flow Management
Streaming Analytics
▶︎
Cloudera Manager
Cloudera Manager
▶︎
Applications
Cloudera Streaming Community Edition
Data Science Workbench
Data Visualization
Edge Management
Observability SaaS
Observability on premises
Workload XM On-Prem
▶︎
Legacy
Cloudera Enterprise
Flow Management
Stream Processing
HDP
HDF
Streams Messaging Manager
Streams Replication Manager
▶︎
Getting Started
Base
Upgrade
Flow Management
Streaming Analytics
«
Filter topics
Cloudera Flow Management
▶︎
Release Notes
▶︎
Support Matrix
Review System Requirements
Supported operating systems
Supported NiFi Registry databases
Supported NiFi Processors
Supported NiFi Controller Services
Supported NiFi Reporting Tasks
▶︎
Download Locations
Download from the CFM Repository
▶︎
Release Notes
What's new in this release?
Component support
▶︎
Unsupported Features
Technical Preview Features
Unsupported Customizations
Behavioral Changes
Known Issues
Fixed Issues
Common Vulnerabilities and Exposures
Download from the CFM Repository
▶︎
Apache NiFi Overview
What is Apache NiFi?
The core concepts of NiFi
NiFi Architecture
Performance Expectations and Characteristics of NiFi
High Level Overview of Key NiFi Features
▶︎
Sizing your Flow Management Cluster
Sizing your Flow Management cluster
Data flow design
NiFi design
Cluster layout
Disk configuration
Resource intensive processors
Recommendations
▶︎
Installation & Upgrade
▶︎
Deployment
CFM deployment workflow
Understanding your deployment scenario
▶︎
Preparing your CDP Private Cloud Base cluster
Install the JDK
▶︎
Install and configure a database for NiFi Registry
Install MySQL
Configuring NiFi Registry Metadata Stores in MySQL
Install PostgreSQL
Configuring NiFi Registry Metadata Stores in PostgreSQL
Install Cloudera Manager and a CDP Private Cloud Base cluster
Install the CFM parcel from the repository
Download the CFM Custom Service Descriptor files
▶︎
Installing NiFi and NiFi Registry on your Base cluster
Understand your Base cluster layout
Add the NiFi service
Add the NiFi Registry service
Connect NiFi to NiFi Registry
Add users or groups to Ranger policies
Deselect unwanted NiFi Registry dependencies
▶︎
Installing NiFi and NiFi Registry on a Compute cluster
Understand your Compute cluster layout
Create a Shared Data Context
Add the NiFi and NiFi Registry groups to Ranger in the Base cluster
Create a Compute cluster
Add the NiFi and NiFi Registry services to a Compute cluster
Connect NiFi to NiFi Registry
Add users or groups to Ranger policies
Deselect unwanted NiFi Registry dependencies
▶︎
Installing NiFi on a Compute cluster and NiFi Registry on a Base cluster
Understand your mixed cluster layout
Install NiFi Registry on your Base cluster
Create a Shared Data Context
Add NiFi and NiFi Registry groups to Ranger
Create Compute cluster
Add NiFi to the Compute cluster
Setting up NiFi node users in NiFi Registry
Connect NiFi to NiFi Registry
Add users or groups to Ranger
Deselect unwanted NiFi Registry dependencies
▶︎
Migration
Before you begin
▶︎
Preserve source cluster files and directories
NiFi files to preserve
Preserve custom processors/NARs
NiFi Registry files to preserve
▶︎
Install CFM 2.1.2
Add and configure the NiFi service
Add and configure the NiFi Registry service
Verify CFM 2.1.2
Clear activity and shut down source services
Migrate the NiFi data directories
▶︎
Migrate the NiFi flow.xml.gz file
Remove unnecessary reporting tasks
Update the Registry Client
Update references to cluster nodes
Update a flow with sensitive properties
▶︎
Migrate authorization policies
Migrate NiFi Ranger-based policies
Migrate NiFi Registry Ranger-based policies
Migrate NiFi file-based policies
Migrate NiFi Registry file-based policies
Migrate NiFi state and custom components
▶︎
Migrate NiFi Registry data storage
Migrate the metadata database
Migrate flow storage
Migrate bundle storage configurations
Post migration steps
▶︎
Migrate file-based authorization to Ranger
Migrate NiFi File-Based Authorization to Ranger
Migrate NiFi Registry File-Based Authorization to Ranger
▶︎
In-place Upgrade
In-place Upgrade
▶︎
Before you upgrade
Installing Solr service
Checking cluster services
Checking service accounts
▶︎
Collect data for upgrade
Collect Ranger passwords
Collect Nifi Registry database password
Extracting Kafka broker IDs
▶︎
Downloading Ambari blueprint
Extending the JSON file
▶︎
Cloudera Manager installation and setup
Checking pre-installation setup
Configuring Cloudera Manager repository
Installing Cloudera Manager server and agents
Configuring database for Cloudera Manager
Starting Cloudera Manager server and adding license
Configuring Cloudera agents and hosts
▶︎
Adding Cloudera Management services
Modifying host monitor port number
▶︎
Upgrading HDF to CFM on CDP Private Cloud Base
Configuring parcel
Deploying Cloudera Manager
Adding CFM parcel in Cloudera Manager
Activating parcel
Troubleshooting HDF upgrade
▶︎
Post-upgrade steps on CDP
Enable security
Setting core configuration service
Starting Zookeeper service
▶︎
Configuring NiFi Registry settings
Setting database password for NiFi Registry
Configuring Kerberos for NiFi Registry
▶︎
Configuring Ranger for NiFi Registry
Modifying the service name in Ranger
Migrating NiFi Registry directories
Verifying Ranger configurations
Configuring Ranger settings
Configuring Solr settings
Initializing Solr
▶︎
Configuring NiFi settings
Configuring Kerberos for NiFi
▶︎
Configuring Ranger for NiFi
Modifying the service name in Ranger
Migrating LDAP authentication configuration
Migrating file-based user handling and policies
▶︎
Post-upgrade steps on CDP for HDF on HDP
Configuring YARN settings
▶︎
Kafka in-place migration with Ranger
Migrating Kafka Ranger policies
▶︎
Upgrade Paths
CFM upgrade and migration paths
▶︎
Upgrade
Before you upgrade
Upgrading to CFM 2.1.2 from CFM 2.x
▶︎
Upgrading to CFM 2.1.2 from CFM 1.1.0
Turn off TLS regeneration
Back up NiFi keystore and truststore settings
Back up NiFi Registry keystore and truststore settings
Upgrading to CFM 2.1.2
Restore NiFi keystore and truststore settings
Restore your NiFi Registry keystore and truststore settings
Turn off identity mapping
▶︎
Additional post-upgrade steps for some upgrade scenarios
Enable Auto-TLS for CFM
Create a Ranger user for the Initial Admin Identity
Manually integrate with Atlas
Integrate with Atlas when Auto-TLS is enabled
Start your NiFi and NiFi Registry services
▶︎
Component Installation and Upgrade
▶︎
Installing NiFi on Linux
Installing NiFi on Linux
Installing NiFi as a Service
Starting and Stopping NiFi on Linux
Launching the User Interface
Installing and Starting NiFi Registry Manually
▶︎
Installing NiFi on Windows
Installing NiFi using the MSI file
Using a Local User for NiFi Windows Service
Using a Domain User for NiFi Windows Service
Starting and Stopping NiFi on Windows
▶︎
Upgrading NiFi Manually
Getting ready to upgrade
▶︎
Preserve Customizations Prior to Upgrade
Preserve your custom processors
Preserve your custom NAR files
Install the new NiFi version
Update the Configuration Files for Your New NiFi Installation
Migrating a dataflow with sensitive properties
Restarting the dataflow after upgrade
▶︎
Upgrading NiFi Registry Manually
Install the New Version of NiFi Registry
Update the Configuration Files
▶︎
How To
▶︎
Tuning your DataFlow
Tuning your dataflow
▶︎
Timer and event driven thread pools
Viewing the total number of threads in a cluster
Viewing the total number of active threads
Viewing the number of cores
Configuring thread pool size
Concurrent tasks
Run duration
Recommendations
▶︎
Moving Data using NiFi Site-to-Site
▶︎
Moving data from CDP Private Cloud Base to Public Cloud with NiFi site-to-site
Understand the use case
Prepare your clusters
Set up your network configuration
Configure your truststores
Define your CDP Public Cloud dataflow
Configure Ranger policies for site-to-site communication
Define your CDP Private Cloud Base dataflow
▶︎
Exchanging Data with External Systems
▶︎
Moving Data in and Out of Snowflake
Pushing data to and moving data from Snowflake using Apache NiFi
▶︎
Moving data out of Snowflake
Before you begin
Downloading the Snowflake JDBC driver jar file
Adding Snowflake CA certificates to NiFi truststore
Building your dataflow
Creating Controller Services for your dataflow
Configuring your source processor
Configuring your target processor
Confirming your dataflow success
▶︎
Pushing data into Snowflake
Before you begin
Adding Snowflake CA certificates to NiFi truststore
Building your dataflow
Configuring your Controller Services
Configure your source processor
Configuring your target processor
Confirming your dataflow success
Next steps
▶︎
Using Apache NiFi
Introduction
▶︎
Browser Support
Unsupported Browsers
Viewing the UI in Variably Sized Browsers
Terminology
NiFi User Interface
Accessing the UI with Multi-Tenant Authorization
Logging In
▶︎
Building a DataFlow
Adding Components to the Canvas
▶︎
Component Versions
Sorting and Filtering Components
Changing Component Versions
Understanding Version Dependencies
▶︎
Configuring a Processor
Settings Tab
Scheduling Tab
Properties Tab
Comments Tab
Additional Help
▶︎
Configuring a Process Group
General Tab
Controller Services
▶︎
Parameters
Parameter Contexts
Adding a Parameter to a Parameter Context
Parameters and Expression Language
Assigning a Parameter Context to a Process Group
Referencing Parameters
Accessing Parameters
▶︎
Using Custom Properties with Expression Language
Variables
Referencing Custom Properties via nifi.properties
▶︎
Controller Services
Adding Controller Services for Reporting Tasks
Adding Controller Services for Dataflows
Enabling/Disabling Controller Services
Reporting Tasks
▶︎
Connecting Components
Details Tab
Settings
Changing Configuration and Context Menu Options
Bending Connections
Processor Validation
▶︎
Site-to-Site
Configure Site-to-Site client NiFi instance
Configure Site-to-Site Server NiFi Instance
Example Dataflow
▶︎
Command and Control of the DataFlow
Starting a Component
Stopping a Component
Terminating a Component's tasks
Enabling/Disabling a Component
▶︎
Remote Process Group Transmission
Individual Port Transmission
▶︎
Navigating within a DataFlow
Component Linking
▶︎
Component Alignment
Align Vertically
Align Horizontally
▶︎
Search Components in DataFlow
Filters
Keywords
▶︎
Monitoring of DataFlow
Anatomy of a Processor
Anatomy of a Process Group
Anatomy of a Remote Process Group
Queue Interaction
Summary Page
Historical Statistics of a Component
▶︎
Versioning a DataFlow
Connecting to a NiFi Registry
Version States
Import a Versioned Flow
Start Version Control
▶︎
Managing Local Changes
Show Local Changes
Revert Local Changes
Commit Local Changes
Change Version
Stop Version Control
Nested Versioned Flows
Parameters in Versioned Flows
Variables in Versioned Flows
▶︎
Restricted Components in Versioned Flows
Restricted Controller Service Created in Root Process Group
Restricted Controller Service Created in Process Group
▶︎
Templates
Creating a Template
Importing a Template
Instantiating a Template
▶︎
Managing Templates
Exporting a Template
Removing a Template
▶︎
Data Provenance
Provenance Events
Searching for Events
Details of an Event
Replaying a FlowFile
▶︎
Viewing FlowFile Lineage
Find Parents
Expanding an Event
▶︎
Write Ahead Provenance Repository
Backwards Compatibility
Older Existing NiFi Version
Bootstrap.conf
System Properties
Encrypted Provenance Considerations
▶︎
Encrypted Provenance Repository
What is it?
How does it work?
Writing and Reading Event Records
Potential Issues
▶︎
Encrypted Content Repository
What is it?
▶︎
How does it work?
StaticKeyProvider
FileBasedKeyProvider
Key Rotation
Writing and Reading Content Claims
Potential Issues
▶︎
Encrypted FlowFile Repository
What is it?
▶︎
How does it work?
StaticKeyProvider
FileBasedKeyProvider
Key Rotation
Writing and Reading FlowFiles
Potential Issues
Experimental Warning
Other Management Features
▶︎
Using the Apache NiFi Toolkit
▶︎
Overview
Prerequisites for Running in a Secure Environment
▶︎
NiFi CLI
Usage
Property/Argument Handling
▶︎
Security Configuration
Example - Secure NiFi Registry without Proxied-Entity
Example - Secure NiFi Registry with Proxied-Entity
Interactive Usage
Output
Back-Referencing
Adding Commands
▶︎
Encrypt-Config Tool
▶︎
Usage
NiFi
NiFi Registry
▶︎
Examples
NiFi
NiFi Registry
Sensitive Property Key Migration
▶︎
File Manager
Usage
▶︎
Expected Behavior
Backup
Install
Restore
▶︎
Flow Analyzer
Usage
▶︎
Node Manager
Usage
▶︎
Expected Behavior
Status
Disconnect
Connect
Remove
▶︎
Notify
Usage
▶︎
S2S
Usage
▶︎
TLS Toolkit
▶︎
Wildcard Certificates
Potential issues with wildcard certificates
▶︎
Operation Modes
Standalone
Client/Server
▶︎
Using An Existing Intermediate Certificate Authority (CA)
nifi-cert.pem
nifi-key.key
Signing with Externally-signed CA Certificates
Additional Certificate Commands
▶︎
ZooKeeper Migrator
Usage
▶︎
Migrating Between Source and Destination ZooKeepers
ZooKeeper Migration Steps
▶︎
Using Apache NiFi Registry
Introduction
▶︎
Browser Support
Unsupported Browsers
Viewing the UI in Variably Sized Browsers
Terminology
NiFi Registry User Interface
Logging In
▶︎
Manage Flows
▶︎
View a Flow
Sorting & Filtering Flows
Delete a Flow
▶︎
Manage Buckets
Sorting & Filtering Buckets
Create a Bucket
Delete a Bucket
Delete Multiple Buckets
Edit a Bucket Name
Make a Bucket Publicly Visible
Allow Bundles in a Bucket to be Overwritten
▶︎
Bucket Policies
Create a Bucket Policy
Delete a Bucket Policy
▶︎
Manage Users & Groups
Sorting & Filtering Users/Groups
Add a User
Delete a User
Delete Multiple Users
Edit a User Name
▶︎
Special Privileges
Grant Special Privileges to a User
▶︎
Manage Groups
Add an Empty Group
Add User to a Group
Create a New Group with Selected Users
▶︎
Remove a User from a Group
User Window
Group Window
Other Group Level Actions
▶︎
Manage Bundles
Upload Bundle
▶︎
Download Bundle
Bundle Coordinates
Bundle Id
Additional Actions
▶︎
Security Recommendations
Recommendations
Identity and policies in NiFi
▶︎
TLS/SSL Configuration
Enable Auto-TLS
▶︎
Manually configure TLS/SSL
TLS/SSL certificate requirements and recommendations
Configure TLS/SSL encryption manually for NiFi and NiFi Registry
NiFi TLS/SSL Properties
NiFi Registry TLS/SSL properties
▶︎
Authentication
▶︎
Kerberos Authentication
Customizing Kerberos principal
LDAP Authentication
SAML Authentication
OpenID Connect Authentication
Identity-Mapping Properties
▶︎
Authorization
▶︎
User group providers
▶︎
LDAP Integration
LDAP and Ranger Policies
LDAP and File-Based Policies
LDAP User Group Provider Properties
Pairing LDAP with a Composite Group Provider
▶︎
Access Policies providers
▶︎
Ranger Authorization
Understanding the Ranger Authorization Process for CFM
Predefined Ranger Access Policies for Apache NiFi
Predefined Ranger Access Policies for Apache NiFi Registry
Predefined component-level policies for NiFi
NiFi Restricted Components
Before you begin
Add user to a pre-defined Ranger access policy
Create a custom Ranger access policy
Authorization example
Enable access to Knox and NiFi
File-Based Authorization
▶︎
Migrate file-based authorization to Ranger
Migrate NiFi File-Based Authorization to Ranger
Migrate NiFi Registry File-Based Authorization to Ranger
▶︎
Environment Variables
Kerberos credentials
Local file system access
▶︎
Network
Default Ports for NiFi and NiFi Registry
▶︎
Cryptography
▶︎
FIPS 140-2 compliance
Encrypting NiFi sensitive properties with FIPS 140-2 approved algorithm
▶︎
Integrations
▶︎
Integrate NiFi and Atlas
Manually Integrate with Atlas when Auto-TLS is not Enabled
Manually Integrate with Atlas when Auto-TLS is Enabled
Integrate NiFi and NiFi Registry with Knox
Enhance or Overwrite Properties in Cloudera Manager
▼
Reference
▶︎
Property Enhancements
▶︎
Property Enhancements
XML Property Enhancements
Bootstrap and Properties File Enhancements
▶︎
Apache NiFi REST API
Apache NiFi REST API Reference
▶︎
Apache NiFi Record Path Reference
Apache NiFi RecordPath Overview
Structure of a RecordPath
Child Operator
Descendant Operator
Filters
▶︎
Function Usage
Arrays
Maps
Predicates
Functions
▶︎
Standalone Functions
substring
substringAfter
substringAfterLast
substringBefore
substringBeforeLast
replace
replaceRegex
concat
fieldName
toDate
toString
toBytes
coalesce
format
trim
toUpperCase
toLowerCase
base64Encode
base64Decode
hash
padLeft
padRight
uuid5
▶︎
Filter Functions
contains
matchesRegex
startsWith
endsWith
not
isEmpty
isBlank
▶︎
Apache NiFi Expression Language Guide
Apache NiFi Expression Language Overview
▶︎
Structure of a NiFi Expression
Expression Language Hierarchy
▶︎
Expression Language in the Application
Escaping Expression Language
Expression Language Editor
▶︎
Functions
Data Types
Boolean Logic
String Manipulation
Encode/Decode Functions
Searching
Mathematical Operations and Numeric Manipulation
Date Manipulation
Type Coercion
Subjectless Functions
Evaluating Multiple Attributes
▶︎
Apache NiFi Admin Guide
System Requirements
How to install and start NiFi
▶︎
Port Configuration
NiFi
Embedded ZooKeeper
Configuration Best Practices
Recommended Antivirus Exclusions
▶︎
Security Configuration
TLS Generation Toolkit
▶︎
User Authentication
Lightweight Directory Access Protocol (LDAP)
Kerberos
OpenId Connect
SAML
Apache Knox
▶︎
Multi-Tenant Authorization
Authorizer Configuration
▶︎
Authorizers.xml Setup
FileUserGroupProvider
LdapUserGroupProvider
ShellUserGroupProvider
Composite Implementations
FileAccessPolicyProvider
StandardManagedAuthorizer
FileAuthorizer
Initial Admin Identity (New NiFi Instance)
Legacy Authorized Users (NiFi Instance Upgrade)
Cluster Node Identities
▶︎
Configuring Users & Access Policies
Creating Users and Groups
Access Policies
Viewing Policies on Users
Access Policy Configuration Examples
▶︎
Encryption Configuration
▶︎
Key Derivation Functions
NiFi Legacy KDF
OpenSSL PKCS#5 v1.5 EVP_BytesToKey
Bcrypt
Scrypt
PBKDF2
None
Argon2
▶︎
Salt and IV Encoding
NiFi Legacy
OpenSSL PKCS#5 v1.5 EVP_BytesToKey
Bcrypt, Scrypt, PBKDF2, Argon2
Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies
Allow Insecure Cryptographic Modes
Encrypted Passwords in Flows
Encrypted Passwords in Configuration Files
NiFi Toolkit Administrative Tools
▶︎
Clustering Configuration
Zero-Leader Clustering
Why Cluster?
Terminology
Communication within the Cluster
▶︎
Managing Nodes
Disconnect Nodes
Offload Nodes
Delete Nodes
Decommission Nodes
NiFi CLI Node Commands
Flow Election
Basic Cluster Setup
Troubleshooting
▶︎
State Management
Configuring State Providers
Embedded ZooKeeper Server
ZooKeeper Access Control
▶︎
Securing ZooKeeper with Kerberos
Kerberizing Embedded ZooKeeper Server
Kerberizing NiFi's ZooKeeper Client
Troubleshooting Kerberos Configuration
Securing ZooKeeper with TLS
Embedded ZooKeeper with TLS
ZooKeeper Migrator
Bootstrap Properties
▶︎
Notification Services
Email Notification Service
HTTP Notification Service
Proxy Configuration
▶︎
Kerberos Service
Notes
Analytics Framework
▶︎
System Properties
Upgrade Recommendations
Core Properties
State Management
H2 Settings
FlowFile Repository
Write Ahead FlowFile Repository
Encrypted Write Ahead FlowFile Repository Properties
Volatile FlowFile Repository
RocksDB FlowFile Repository
Swap Management
Content Repository
File System Content Repository Properties
Encrypted File System Content Repository Properties
Volatile Content Repository Properties
Provenance Repository
Write Ahead Provenance Repository Properties
Encrypted Write Ahead Provenance Repository Properties
Persistent Provenance Repository Properties
Volatile Provenance Repository Properties
Status History Repository
Site to Site Properties
▶︎
Site to Site Routing Properties for Reverse Proxies
Site to Site protocol sequence
Reverse Proxy Configurations
Site to Site and Reverse Proxy Examples
Web Properties
Security Properties
Identity Mapping Properties
Cluster Common Properties
Cluster Node Properties
ZooKeeper Properties
Kerberos Properties
Analytics Properties
Referencing Custom Properties via nifi.properties
▶︎
Upgrading NiFi
Preserve Custom Processors
Preserve Modified NARs
Clear Activity and Shutdown Existing NiFi
Install the new NiFi Version
▶︎
Update the Configuration Files for Your New NiFi Installation
Migrating a Flow with Sensitive Properties
Start New NiFi
▶︎
Processor Locations
Available Configuration Options
Installing Custom Processors
Autoloading Custom Processors
▶︎
Apache NiFi Developer Guide
NiFi Developer Guide Introduction
NiFi Components
▶︎
Processor API
▶︎
Supporting API
FlowFile
ProcessSession
ProcessContext
PropertyDescriptor
Validator
ValidationContext
PropertyValue
Relationship
StateManager
ProcessorInitializationContext
ComponentLog
▶︎
AbstractProcessor API
Processor Initialization
Exposing Processor's Relationships
Exposing Processor Properties
Validating Processor Properties
Responding to Changes in Configuration
Performing the Work
When Processors are Triggered
▶︎
Component Lifecycle
@OnAdded
@OnEnabled
@OnRemoved
@OnScheduled
@OnUnscheduled
@OnStopped
@OnShutdown
▶︎
Component Notification
@OnPrimaryNodeStateChange
Restricted
▶︎
State Manager
Scope
Storing and Retrieving State
Unit Tests
Reporting Processor Activity
▶︎
Documenting a Component
Documenting Properties
Documenting Relationships
Documenting Capability and Keywords
Documenting FlowFile Attribute Interaction
Documenting Related Components
Advanced Documentation
Provenance Events
▶︎
Common Processor Patterns
Data Ingress
Data Egress
Route Based on Content (One-to-One)
Route Based on Content (One-to-Many)
Route Streams Based on Content (One-to-Many)
Route Based on Attributes
Split Content (One-to-Many)
Update Attributes Based on Content
Enrich/Modify Content
▶︎
Error Handling
Exceptions within the Processor
Exceptions within a callback: IOException, RuntimeException
Penalization vs. Yielding
Session Rollback
▶︎
General Design Considerations
Consider the User
Cohesion and Reusability
Naming Conventions
Processor Behavior Annotations
Data Buffering
▶︎
Controller Services
Developing a ControllerService
Interacting with a ControllerService
▶︎
Reporting Tasks
Developing a Reporting Task
▶︎
UI Extensions
Custom Processor UIs
Content Viewers
Command Line Tools
▶︎
Testing
Instantiate TestRunner
Add ControllerServices
Set Property Values
Enqueue FlowFiles
Run the Processor
Validate Output
Mocking External Resources
Additional Testing Capabilities
NiFi Archives (NARs)
Per-Instance ClassLoading
Deprecating a Component
▶︎
How to contribute to Apache NiFi
Technologies
Where to Start?
Supplying a contribution
Contact Us
▶︎
Apache NiFi Registry Admin Guide
System Requirements
How to install and start NiFi Registry
Recommended Antivirus Exclusions
Security Configuration
▶︎
User Authentication
Lightweight Directory Access Protocol (LDAP)
Kerberos
▶︎
Authorization
Authorizer Configuration
▶︎
Authorizers.xml Setup
StandardManagedAuthorizer
UserGroupProvider
AccessPolicyProvider
Initial Admin Identity (New NiFi Registry Instance)
▶︎
Access Policies
Bucket Policies
Special Privilege Policies
▶︎
Encrypted Passwords in Configuration Files
Encrypt-Config Tool
Sensitive Property Key Migration
Bootstrap Properties
Proxy Configuration
▶︎
Kerberos Service
Notes
▶︎
System Properties
Web Properties
Security Properties
Identity Mapping Properties
Providers Properties
Alias Properties
Database Properties
Extension Directories
Kerberos Properties
▶︎
Metadata Database
H2
Postgres
MySQL
Schema Differences & Limitations
▶︎
Persistence Providers
▶︎
Flow Persistence Providers
FileSystemFlowPersistenceProvider
GitFlowPersistenceProvider
DatabaseFlowPersistenceProvider
Switching from other Flow Persistence Provider
Data model version of serialized Flow snapshots
▶︎
Bundle Persistence Providers
FileSystemBundlePersistenceProvider
S3BundlePersistenceProvider
▶︎
Event Hooks
Shared Event Hook Properties
ScriptEventHookProvider
LoggingEventHookProvider
URL Aliasing
▶︎
Backup & Recovery
Metadata Database
▶︎
Persistence Providers
Flow Persistence
Bundle Persistence
Configuration Files
▼
Apache NiFi Registry REST API
▶︎
1. Overview
1.1. Version information
1.2. Contact information
1.3. License information
1.4. URI scheme
1.5. Tags
▶︎
2. Security
2.1. Authorization
2.2. BasicAuth
▶︎
3. Resources
▶︎
3.1. Access
3.1.1. Get access status
3.1.2. Performs a logout for other providers that have been issued a JWT.
3.1.3. Redirect/callback URI for processing the result of the OpenId Connect login sequence.
3.1.4. Retrieves a JWT following a successful login sequence using the configured OpenId Connect provider.
3.1.5. Performs a logout in the OpenId Provider.
3.1.6. Initiates a request to authenticate through the configured OpenId Connect provider.
3.1.7. Create token trying all providers
3.1.8. Create token using identity provider
3.1.9. Test identity provider
3.1.10. Get identity provider usage
3.1.11. Create token using kerberos
3.1.12. Create token using basic auth
▶︎
3.2. Bucket Bundles
3.2.1. Get extension bundles by bucket
3.2.2. Create extension bundle version
▶︎
3.3. Bucket Flows
3.3.1. Create flow
3.3.2. Get bucket flows
3.3.3. Get bucket flow
3.3.4. Update bucket flow
3.3.5. Delete bucket flow
3.3.6. Get bucket flow diff
3.3.7. Create flow version
3.3.8. Get bucket flow versions
3.3.9. Get latest bucket flow version content
3.3.10. Get latest bucket flow version metadata
3.3.11. Get bucket flow version
▶︎
3.4. Buckets
3.4.1. Create bucket
3.4.2. Get all buckets
3.4.3. Get bucket fields
3.4.4. Get bucket
3.4.5. Update bucket
3.4.6. Delete bucket
▶︎
3.5. Bundles
3.5.1. Get all bundles
3.5.2. Get all bundle versions
3.5.3. Get bundle
3.5.4. Delete bundle
3.5.5. Get bundle versions
3.5.6. Get bundle version
3.5.7. Delete bundle version
3.5.8. Get bundle version content
3.5.9. Get bundle version extensions
3.5.10. Get bundle version extension
3.5.11. Get bundle version extension docs
3.5.12. Get bundle version extension docs details
▶︎
3.6. Config
3.6.1. Get configration
▶︎
3.7. Extension Repository
3.7.1. Get extension repo buckets
3.7.2. Get extension repo groups
3.7.3. Get extension repo artifacts
3.7.4. Get extension repo versions
3.7.5. Get extension repo version
3.7.6. Get extension repo version content
3.7.7. Get extension repo extensions
3.7.8. Get extension repo extension
3.7.9. Get extension repo extension docs
3.7.10. Get extension repo extension details
3.7.11. Get extension repo version checksum
3.7.12. Get global extension repo version checksum
▶︎
3.8. Extensions
3.8.1. Get all extensions
3.8.2. Get extensions providing service API
3.8.3. Get extension tags
▶︎
3.9. Flows
3.9.1. Get flow fields
3.9.2. Get flow
3.9.3. Get flow versions
3.9.4. Get latest flow version
3.9.5. Get latest flow version metadata
3.9.6. Get flow version
▶︎
3.10. Items
3.10.1. Get all items
3.10.2. Get item fields
3.10.3. Get bucket items
▶︎
3.11. Policies
3.11.1. Create access policy
3.11.2. Get all access policies
3.11.3. Get available resources
3.11.4. Get access policy for resource
3.11.5. Get access policy
3.11.6. Update access policy
3.11.7. Delete access policy
▶︎
3.12. Tenants
3.12.1. Create user group
3.12.2. Get user groups
3.12.3. Get user group
3.12.4. Update user group
3.12.5. Delete user group
3.12.6. Create user
3.12.7. Get all users
3.12.8. Get user
3.12.9. Update user
3.12.10. Delete user
▼
4. Definitions
4.1. AccessPolicy
4.2. AccessPolicySummary
4.3. AllowableValue
4.4. Attribute
4.5. BatchSize
4.6. Bucket
4.7. BucketItem
4.8. BuildInfo
4.9. Bundle
4.10. BundleInfo
4.11. BundleVersion
4.12. BundleVersionDependency
4.13. BundleVersionMetadata
4.14. ComponentDifference
4.15. ComponentDifferenceGroup
4.16. ConnectableComponent
4.17. ControllerServiceAPI
4.18. ControllerServiceDefinition
4.19. CurrentUser
4.20. DeprecationNotice
4.21. DynamicProperty
4.22. DynamicRelationship
4.23. Extension
4.24. ExtensionBundle
4.25. ExtensionFilterParams
4.26. ExtensionMetadata
4.27. ExtensionMetadataContainer
4.28. ExtensionRepoArtifact
4.29. ExtensionRepoBucket
4.30. ExtensionRepoGroup
4.31. ExtensionRepoVersion
4.32. ExtensionRepoVersionSummary
4.33. ExternalControllerServiceReference
4.34. Fields
4.35. JaxbLink
4.36. Permissions
4.37. Position
4.38. Property
4.39. ProvidedServiceAPI
4.40. RegistryConfiguration
4.41. Relationship
4.42. Resource
4.43. ResourcePermissions
4.44. Restricted
4.45. Restriction
4.46. RevisionInfo
4.47. Stateful
4.48. SystemResourceConsideration
4.49. TagCount
4.50. Tenant
4.51. User
4.52. UserGroup
4.53. VersionedConnection
4.54. VersionedControllerService
4.55. VersionedFlow
4.56. VersionedFlowCoordinates
4.57. VersionedFlowDifference
4.58. VersionedFlowSnapshot
4.59. VersionedFlowSnapshotMetadata
4.60. VersionedFunnel
4.61. VersionedLabel
4.62. VersionedParameter
4.63. VersionedParameterContext
4.64. VersionedPort
4.65. VersionedProcessGroup
4.66. VersionedProcessor
4.67. VersionedPropertyDescriptor
4.68. VersionedRemoteGroupPort
4.69. VersionedRemoteProcessGroup
▶︎
Getting Started with Cloudera Flow Management
What is Cloudera Flow Management?
Who is this guide for?
Flow management terminology
▶︎
Set up Cloudera Flow Management
Download NiFi and NiFi Registry
Install NiFi and NiFi Registry
Start NiFi and NiFi Registry
Build your first data flow
Learn about NiFi processors
▶︎
Work with data flow templates
Create a template
Download a template
Upload a template
Use a template
Manage templates
Connect NiFi to NiFi Registry
Create a bucket in NiFi Registry
▶︎
Version data flows
Start version control
Commit local changes
Change the version of your data flow
Manage versioned data flows
Monitor data flows
Trace Data Lifecycle
▶︎
Getting Started with Apache NiFi Registry
Who is This Guide For?
Terminology Used in This Guide
Downloading and Installing NiFi Registry
▶︎
Starting NiFi Registry
For Linux/Unix/Mac OS X users
Installing as a Service
▶︎
I Started NiFi Registry. Now What?
Create a Bucket
Connect NiFi to the Registry
Start Version Control on a Process Group
Save Changes to a Versioned Flow
Import a Versioned Flow
Where To Go For More Information
1. Overview
1.1. Version information
1.2. Contact information
1.3. License information
1.4. URI scheme
1.5. Tags
2. Security
2.1. Authorization
2.2. BasicAuth
3. Resources
3.1. Access
3.1.1. Get access status
3.1.10. Get identity provider usage
3.1.11. Create token using kerberos
3.1.12. Create token using basic auth
3.1.2. Performs a logout for other providers that have been issued a JWT.
3.1.3. Redirect/callback URI for processing the result of the OpenId Connect login sequence.
3.1.4. Retrieves a JWT following a successful login sequence using the configured OpenId Connect provider.
3.1.5. Performs a logout in the OpenId Provider.
3.1.6. Initiates a request to authenticate through the configured OpenId Connect provider.
3.1.7. Create token trying all providers
3.1.8. Create token using identity provider
3.1.9. Test identity provider
3.10. Items
3.10.1. Get all items
3.10.2. Get item fields
3.10.3. Get bucket items
3.11. Policies
3.11.1. Create access policy
3.11.2. Get all access policies
3.11.3. Get available resources
3.11.4. Get access policy for resource
3.11.5. Get access policy
3.11.6. Update access policy
3.11.7. Delete access policy
3.12. Tenants
3.12.1. Create user group
3.12.10. Delete user
3.12.2. Get user groups
3.12.3. Get user group
3.12.4. Update user group
3.12.5. Delete user group
3.12.6. Create user
3.12.7. Get all users
3.12.8. Get user
3.12.9. Update user
3.2. Bucket Bundles
3.2.1. Get extension bundles by bucket
3.2.2. Create extension bundle version
3.3. Bucket Flows
3.3.1. Create flow
3.3.10. Get latest bucket flow version metadata
3.3.11. Get bucket flow version
3.3.2. Get bucket flows
3.3.3. Get bucket flow
3.3.4. Update bucket flow
3.3.5. Delete bucket flow
3.3.6. Get bucket flow diff
3.3.7. Create flow version
3.3.8. Get bucket flow versions
3.3.9. Get latest bucket flow version content
3.4. Buckets
3.4.1. Create bucket
3.4.2. Get all buckets
3.4.3. Get bucket fields
3.4.4. Get bucket
3.4.5. Update bucket
3.4.6. Delete bucket
3.5. Bundles
3.5.1. Get all bundles
3.5.10. Get bundle version extension
3.5.11. Get bundle version extension docs
3.5.12. Get bundle version extension docs details
3.5.2. Get all bundle versions
3.5.3. Get bundle
3.5.4. Delete bundle
3.5.5. Get bundle versions
3.5.6. Get bundle version
3.5.7. Delete bundle version
3.5.8. Get bundle version content
3.5.9. Get bundle version extensions
3.6. Config
3.6.1. Get configration
3.7. Extension Repository
3.7.1. Get extension repo buckets
3.7.10. Get extension repo extension details
3.7.11. Get extension repo version checksum
3.7.12. Get global extension repo version checksum
3.7.2. Get extension repo groups
3.7.3. Get extension repo artifacts
3.7.4. Get extension repo versions
3.7.5. Get extension repo version
3.7.6. Get extension repo version content
3.7.7. Get extension repo extensions
3.7.8. Get extension repo extension
3.7.9. Get extension repo extension docs
3.8. Extensions
3.8.1. Get all extensions
3.8.2. Get extensions providing service API
3.8.3. Get extension tags
3.9. Flows
3.9.1. Get flow fields
3.9.2. Get flow
3.9.3. Get flow versions
3.9.4. Get latest flow version
3.9.5. Get latest flow version metadata
3.9.6. Get flow version
4. Definitions
4.1. AccessPolicy
4.10. BundleInfo
4.11. BundleVersion
4.12. BundleVersionDependency
4.13. BundleVersionMetadata
4.14. ComponentDifference
4.15. ComponentDifferenceGroup
4.16. ConnectableComponent
4.17. ControllerServiceAPI
4.18. ControllerServiceDefinition
4.19. CurrentUser
4.2. AccessPolicySummary
4.20. DeprecationNotice
4.21. DynamicProperty
4.22. DynamicRelationship
4.23. Extension
4.24. ExtensionBundle
4.25. ExtensionFilterParams
4.26. ExtensionMetadata
4.27. ExtensionMetadataContainer
4.28. ExtensionRepoArtifact
4.29. ExtensionRepoBucket
4.3. AllowableValue
4.30. ExtensionRepoGroup
4.31. ExtensionRepoVersion
4.32. ExtensionRepoVersionSummary
4.33. ExternalControllerServiceReference
4.34. Fields
4.35. JaxbLink
4.36. Permissions
4.37. Position
4.38. Property
4.39. ProvidedServiceAPI
4.4. Attribute
4.40. RegistryConfiguration
4.41. Relationship
4.42. Resource
4.43. ResourcePermissions
4.44. Restricted
4.45. Restriction
4.46. RevisionInfo
4.47. Stateful
4.48. SystemResourceConsideration
4.49. TagCount
4.5. BatchSize
4.50. Tenant
4.51. User
4.52. UserGroup
4.53. VersionedConnection
4.54. VersionedControllerService
4.55. VersionedFlow
4.56. VersionedFlowCoordinates
4.57. VersionedFlowDifference
4.58. VersionedFlowSnapshot
4.59. VersionedFlowSnapshotMetadata
4.6. Bucket
4.60. VersionedFunnel
4.61. VersionedLabel
4.62. VersionedParameter
4.63. VersionedParameterContext
4.64. VersionedPort
4.65. VersionedProcessGroup
4.66. VersionedProcessor
4.67. VersionedPropertyDescriptor
4.68. VersionedRemoteGroupPort
4.69. VersionedRemoteProcessGroup
4.7. BucketItem
4.8. BuildInfo
4.9. Bundle
@OnAdded
@OnEnabled
@OnPrimaryNodeStateChange
@OnRemoved
@OnScheduled
@OnShutdown
@OnStopped
@OnUnscheduled
AbstractProcessor API
Access Policies
Access Policies
Access Policies providers
Access Policy Configuration Examples
Accessing Parameters
Accessing the UI with Multi-Tenant Authorization
AccessPolicyProvider
Activating parcel
Add a User
Add an Empty Group
Add and configure the NiFi Registry service
Add and configure the NiFi service
Add ControllerServices
Add NiFi and NiFi Registry groups to Ranger
Add NiFi to the Compute cluster
Add the NiFi and NiFi Registry groups to Ranger in the Base cluster
Add the NiFi and NiFi Registry services to a Compute cluster
Add the NiFi Registry service
Add the NiFi service
Add User to a Group
Add user to a pre-defined Ranger access policy
Add users or groups to Ranger
Add users or groups to Ranger policies
Add users or groups to Ranger policies
Adding a Parameter to a Parameter Context
Adding CFM parcel in Cloudera Manager
Adding Cloudera Management services
Adding Commands
Adding Components to the Canvas
Adding Controller Services for Dataflows
Adding Controller Services for Reporting Tasks
Adding Snowflake CA certificates to NiFi truststore
Adding Snowflake CA certificates to NiFi truststore
Additional Actions
Additional Certificate Commands
Additional Help
Additional post-upgrade steps for some upgrade scenarios
Additional Testing Capabilities
Advanced Documentation
Alias Properties
Align Horizontally
Align Vertically
Allow Bundles in a Bucket to be Overwritten
Allow Insecure Cryptographic Modes
Analytics Framework
Analytics Properties
Anatomy of a Process Group
Anatomy of a Processor
Anatomy of a Remote Process Group
Apache Knox
Apache NiFi Admin Guide
Apache NiFi Developer Guide
Apache NiFi Expression Language Guide
Apache NiFi Expression Language Overview
Apache NiFi Overview
Apache NiFi Record Path Reference
Apache NiFi RecordPath Overview
Apache NiFi Registry Admin Guide
Apache NiFi Registry REST API
Apache NiFi REST API
Apache NiFi REST API Reference
Argon2
Arrays
Assigning a Parameter Context to a Process Group
Authentication
Authorization
Authorization
Authorization example
Authorizer Configuration
Authorizer Configuration
Authorizers.xml Setup
Authorizers.xml Setup
Autoloading Custom Processors
Available Configuration Options
Back up NiFi keystore and truststore settings
Back up NiFi Registry keystore and truststore settings
Back-Referencing
Backup
Backup & Recovery
Backwards Compatibility
base64Decode
base64Encode
Basic Cluster Setup
Bcrypt
Bcrypt, Scrypt, PBKDF2, Argon2
Before you begin
Before you begin
Before you begin
Before you begin
Before you upgrade
Before you upgrade
Behavioral Changes
Bending Connections
Boolean Logic
Bootstrap and Properties File Enhancements
Bootstrap Properties
Bootstrap Properties
Bootstrap.conf
Browser Support
Browser Support
Bucket Policies
Bucket Policies
Build your first data flow
Building a DataFlow
Building your dataflow
Building your dataflow
Bundle Coordinates
Bundle Id
Bundle Persistence
Bundle Persistence Providers
CFM deployment workflow
CFM upgrade and migration paths
Change the version of your data flow
Change Version
Changing Component Versions
Changing Configuration and Context Menu Options
Checking cluster services
Checking pre-installation setup
Checking service accounts
Child Operator
Clear activity and shut down source services
Clear Activity and Shutdown Existing NiFi
Client/Server
Cloudera Flow Management
Cloudera Manager installation and setup
Cluster Common Properties
Cluster layout
Cluster Node Identities
Cluster Node Properties
Clustering Configuration
coalesce
Cohesion and Reusability
Collect data for upgrade
Collect Nifi Registry database password
Collect Ranger passwords
Command and Control of the DataFlow
Command Line Tools
Comments Tab
Commit Local Changes
Commit local changes
Common Processor Patterns
Common Vulnerabilities and Exposures
Communication within the Cluster
Component Alignment
Component Installation and Upgrade
Component Lifecycle
Component Linking
Component Notification
Component support
Component Versions
ComponentLog
Composite Implementations
concat
Concurrent tasks
Configuration Best Practices
Configuration Files
Configure Ranger policies for site-to-site communication
Configure Site-to-Site client NiFi instance
Configure Site-to-Site Server NiFi Instance
Configure TLS/SSL encryption manually for NiFi and NiFi Registry
Configure your source processor
Configure your truststores
Configuring a Process Group
Configuring a Processor
Configuring Cloudera agents and hosts
Configuring Cloudera Manager repository
Configuring database for Cloudera Manager
Configuring Kerberos for NiFi
Configuring Kerberos for NiFi Registry
Configuring NiFi Registry Metadata Stores in MySQL
Configuring NiFi Registry Metadata Stores in PostgreSQL
Configuring NiFi Registry settings
Configuring NiFi settings
Configuring parcel
Configuring Ranger for NiFi
Configuring Ranger for NiFi Registry
Configuring Ranger settings
Configuring Solr settings
Configuring State Providers
Configuring thread pool size
Configuring Users & Access Policies
Configuring YARN settings
Configuring your Controller Services
Configuring your source processor
Configuring your target processor
Configuring your target processor
Confirming your dataflow success
Confirming your dataflow success
Connect
Connect NiFi to NiFi Registry
Connect NiFi to NiFi Registry
Connect NiFi to NiFi Registry
Connect NiFi to NiFi Registry
Connect NiFi to the Registry
Connecting Components
Connecting to a NiFi Registry
Consider the User
Contact Us
contains
Content Repository
Content Viewers
Controller Services
Controller Services
Controller Services
Core Properties
Create a Bucket
Create a Bucket
Create a bucket in NiFi Registry
Create a Bucket Policy
Create a Compute cluster
Create a custom Ranger access policy
Create a New Group with Selected Users
Create a Ranger user for the Initial Admin Identity
Create a Shared Data Context
Create a Shared Data Context
Create a template
Create Compute cluster
Creating a Template
Creating Controller Services for your dataflow
Creating Users and Groups
Cryptography
Custom Processor UIs
Customizing Kerberos principal
Data Buffering
Data Egress
Data flow design
Data Ingress
Data model version of serialized Flow snapshots
Data Provenance
Data Types
Database Properties
DatabaseFlowPersistenceProvider
Date Manipulation
Decommission Nodes
Default Ports for NiFi and NiFi Registry
Define your CDP Private Cloud Base dataflow
Define your CDP Public Cloud dataflow
Delete a Bucket
Delete a Bucket Policy
Delete a Flow
Delete a User
Delete Multiple Buckets
Delete Multiple Users
Delete Nodes
Deploying Cloudera Manager
Deployment
Deprecating a Component
Descendant Operator
Deselect unwanted NiFi Registry dependencies
Deselect unwanted NiFi Registry dependencies
Deselect unwanted NiFi Registry dependencies
Details of an Event
Details Tab
Developing a ControllerService
Developing a Reporting Task
Disconnect
Disconnect Nodes
Disk configuration
Documenting a Component
Documenting Capability and Keywords
Documenting FlowFile Attribute Interaction
Documenting Properties
Documenting Related Components
Documenting Relationships
Download a template
Download Bundle
Download from the CFM Repository
Download from the CFM Repository
Download Locations
Download NiFi and NiFi Registry
Download the CFM Custom Service Descriptor files
Downloading Ambari blueprint
Downloading and Installing NiFi Registry
Downloading the Snowflake JDBC driver jar file
Edit a Bucket Name
Edit a User Name
Email Notification Service
Embedded ZooKeeper
Embedded ZooKeeper Server
Embedded ZooKeeper with TLS
Enable access to Knox and NiFi
Enable Auto-TLS
Enable Auto-TLS for CFM
Enable security
Enabling/Disabling a Component
Enabling/Disabling Controller Services
Encode/Decode Functions
Encrypt-Config Tool
Encrypt-Config Tool
Encrypted Content Repository
Encrypted File System Content Repository Properties
Encrypted FlowFile Repository
Encrypted Passwords in Configuration Files
Encrypted Passwords in Configuration Files
Encrypted Passwords in Flows
Encrypted Provenance Considerations
Encrypted Provenance Repository
Encrypted Write Ahead FlowFile Repository Properties
Encrypted Write Ahead Provenance Repository Properties
Encrypting NiFi sensitive properties with FIPS 140-2 approved algorithm
Encryption Configuration
endsWith
Enhance or Overwrite Properties in Cloudera Manager
Enqueue FlowFiles
Enrich/Modify Content
Environment Variables
Error Handling
Escaping Expression Language
Evaluating Multiple Attributes
Event Hooks
Example - Secure NiFi Registry with Proxied-Entity
Example - Secure NiFi Registry without Proxied-Entity
Example Dataflow
Examples
Exceptions within a callback: IOException, RuntimeException
Exceptions within the Processor
Exchanging Data with External Systems
Expanding an Event
Expected Behavior
Expected Behavior
Experimental Warning
Exporting a Template
Exposing Processor Properties
Exposing Processor's Relationships
Expression Language Editor
Expression Language Hierarchy
Expression Language in the Application
Extending the JSON file
Extension Directories
Extracting Kafka broker IDs
fieldName
File Manager
File System Content Repository Properties
File-Based Authorization
FileAccessPolicyProvider
FileAuthorizer
FileBasedKeyProvider
FileBasedKeyProvider
FileSystemBundlePersistenceProvider
FileSystemFlowPersistenceProvider
FileUserGroupProvider
Filter Functions
Filters
Filters
Find Parents
FIPS 140-2 compliance
Fixed Issues
Flow Analyzer
Flow Election
Flow management terminology
Flow Persistence
Flow Persistence Providers
FlowFile
FlowFile Repository
For Linux/Unix/Mac OS X users
format
Function Usage
Functions
Functions
General Design Considerations
General Tab
Getting ready to upgrade
Getting Started with Apache NiFi Registry
Getting Started with Cloudera Flow Management
GitFlowPersistenceProvider
Grant Special Privileges to a User
Group Window
H2
H2 Settings
hash
High Level Overview of Key NiFi Features
Historical Statistics of a Component
How does it work?
How does it work?
How does it work?
How to contribute to Apache NiFi
How to install and start NiFi
How to install and start NiFi Registry
HTTP Notification Service
I Started NiFi Registry. Now What?
Identity and policies in NiFi
Identity Mapping Properties
Identity Mapping Properties
Identity-Mapping Properties
Import a Versioned Flow
Import a Versioned Flow
Importing a Template
In-place Upgrade
In-place Upgrade
Individual Port Transmission
Initial Admin Identity (New NiFi Instance)
Initial Admin Identity (New NiFi Registry Instance)
Initializing Solr
Install
Install and configure a database for NiFi Registry
Install CFM 2.1.2
Install Cloudera Manager and a CDP Private Cloud Base cluster
Install MySQL
Install NiFi and NiFi Registry
Install NiFi Registry on your Base cluster
Install PostgreSQL
Install the CFM parcel from the repository
Install the JDK
Install the new NiFi version
Install the new NiFi Version
Install the New Version of NiFi Registry
Installing and Starting NiFi Registry Manually
Installing as a Service
Installing Cloudera Manager server and agents
Installing Custom Processors
Installing NiFi and NiFi Registry on a Compute cluster
Installing NiFi and NiFi Registry on your Base cluster
Installing NiFi as a Service
Installing NiFi on a Compute cluster and NiFi Registry on a Base cluster
Installing NiFi on Linux
Installing NiFi on Linux
Installing NiFi on Windows
Installing NiFi using the MSI file
Installing Solr service
Instantiate TestRunner
Instantiating a Template
Integrate NiFi and Atlas
Integrate NiFi and NiFi Registry with Knox
Integrate with Atlas when Auto-TLS is enabled
Integrations
Interacting with a ControllerService
Interactive Usage
Introduction
Introduction
isBlank
isEmpty
Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies
Kafka in-place migration with Ranger
Kerberizing Embedded ZooKeeper Server
Kerberizing NiFi's ZooKeeper Client
Kerberos
Kerberos
Kerberos Authentication
Kerberos credentials
Kerberos Properties
Kerberos Properties
Kerberos Service
Kerberos Service
Key Derivation Functions
Key Rotation
Key Rotation
Keywords
Known Issues
Launching the User Interface
LDAP and File-Based Policies
LDAP and Ranger Policies
LDAP Authentication
LDAP Integration
LDAP User Group Provider Properties
LdapUserGroupProvider
Learn about NiFi processors
Legacy Authorized Users (NiFi Instance Upgrade)
Lightweight Directory Access Protocol (LDAP)
Lightweight Directory Access Protocol (LDAP)
Local file system access
Logging In
Logging In
LoggingEventHookProvider
Make a Bucket Publicly Visible
Manage Buckets
Manage Bundles
Manage Flows
Manage Groups
Manage templates
Manage Users & Groups
Manage versioned data flows
Managing Local Changes
Managing Nodes
Managing Templates
Manually configure TLS/SSL
Manually integrate with Atlas
Manually Integrate with Atlas when Auto-TLS is Enabled
Manually Integrate with Atlas when Auto-TLS is not Enabled
Maps
matchesRegex
Mathematical Operations and Numeric Manipulation
Metadata Database
Metadata Database
Migrate authorization policies
Migrate bundle storage configurations
Migrate file-based authorization to Ranger
Migrate file-based authorization to Ranger
Migrate flow storage
Migrate NiFi File-Based Authorization to Ranger
Migrate NiFi File-Based Authorization to Ranger
Migrate NiFi file-based policies
Migrate NiFi Ranger-based policies
Migrate NiFi Registry data storage
Migrate NiFi Registry File-Based Authorization to Ranger
Migrate NiFi Registry File-Based Authorization to Ranger
Migrate NiFi Registry file-based policies
Migrate NiFi Registry Ranger-based policies
Migrate NiFi state and custom components
Migrate the metadata database
Migrate the NiFi data directories
Migrate the NiFi flow.xml.gz file
Migrating a dataflow with sensitive properties
Migrating a Flow with Sensitive Properties
Migrating Between Source and Destination ZooKeepers
Migrating file-based user handling and policies
Migrating Kafka Ranger policies
Migrating LDAP authentication configuration
Migrating NiFi Registry directories
Migration
Mocking External Resources
Modifying host monitor port number
Modifying the service name in Ranger
Modifying the service name in Ranger
Monitor data flows
Monitoring of DataFlow
Moving data from CDP Private Cloud Base to Public Cloud with NiFi site-to-site
Moving Data in and Out of Snowflake
Moving data out of Snowflake
Moving Data using NiFi Site-to-Site
Multi-Tenant Authorization
MySQL
Naming Conventions
Navigating within a DataFlow
Nested Versioned Flows
Network
Next steps
NiFi
NiFi
NiFi
NiFi Architecture
NiFi Archives (NARs)
NiFi CLI
NiFi CLI Node Commands
NiFi Components
NiFi design
NiFi Developer Guide Introduction
NiFi files to preserve
NiFi Legacy
NiFi Legacy KDF
NiFi Registry
NiFi Registry
NiFi Registry files to preserve
NiFi Registry TLS/SSL properties
NiFi Registry User Interface
NiFi Restricted Components
NiFi TLS/SSL Properties
NiFi Toolkit Administrative Tools
NiFi User Interface
nifi-cert.pem
nifi-key.key
Node Manager
None
not
Notes
Notes
Notification Services
Notify
Offload Nodes
Older Existing NiFi Version
OpenId Connect
OpenID Connect Authentication
OpenSSL PKCS#5 v1.5 EVP_BytesToKey
OpenSSL PKCS#5 v1.5 EVP_BytesToKey
Operation Modes
Other Group Level Actions
Other Management Features
Output
Overview
padLeft
padRight
Pairing LDAP with a Composite Group Provider
Parameter Contexts
Parameters
Parameters and Expression Language
Parameters in Versioned Flows
PBKDF2
Penalization vs. Yielding
Per-Instance ClassLoading
Performance Expectations and Characteristics of NiFi
Performing the Work
Persistence Providers
Persistence Providers
Persistent Provenance Repository Properties
Port Configuration
Post migration steps
Post-upgrade steps on CDP
Post-upgrade steps on CDP for HDF on HDP
Postgres
Potential Issues
Potential Issues
Potential Issues
Potential issues with wildcard certificates
Predefined component-level policies for NiFi
Predefined Ranger Access Policies for Apache NiFi
Predefined Ranger Access Policies for Apache NiFi Registry
Predicates
Prepare your clusters
Preparing your CDP Private Cloud Base cluster
Prerequisites for Running in a Secure Environment
Preserve Custom Processors
Preserve custom processors/NARs
Preserve Customizations Prior to Upgrade
Preserve Modified NARs
Preserve source cluster files and directories
Preserve your custom NAR files
Preserve your custom processors
ProcessContext
Processor API
Processor Behavior Annotations
Processor Initialization
Processor Locations
Processor Validation
ProcessorInitializationContext
ProcessSession
Properties Tab
Property Enhancements
Property Enhancements
Property/Argument Handling
PropertyDescriptor
PropertyValue
Provenance Events
Provenance Events
Provenance Repository
Providers Properties
Proxy Configuration
Proxy Configuration
Pushing data into Snowflake
Pushing data to and moving data from Snowflake using Apache NiFi
Queue Interaction
Ranger Authorization
Recommendations
Recommendations
Recommendations
Recommended Antivirus Exclusions
Recommended Antivirus Exclusions
Referencing Custom Properties via nifi.properties
Referencing Custom Properties via nifi.properties
Referencing Parameters
Relationship
Release Notes
Remote Process Group Transmission
Remove
Remove a User from a Group
Remove unnecessary reporting tasks
Removing a Template
replace
replaceRegex
Replaying a FlowFile
Reporting Processor Activity
Reporting Tasks
Reporting Tasks
Resource intensive processors
Responding to Changes in Configuration
Restarting the dataflow after upgrade
Restore
Restore NiFi keystore and truststore settings
Restore your NiFi Registry keystore and truststore settings
Restricted
Restricted Components in Versioned Flows
Restricted Controller Service Created in Process Group
Restricted Controller Service Created in Root Process Group
Reverse Proxy Configurations
Revert Local Changes
Review System Requirements
RocksDB FlowFile Repository
Route Based on Attributes
Route Based on Content (One-to-Many)
Route Based on Content (One-to-One)
Route Streams Based on Content (One-to-Many)
Run duration
Run the Processor
S2S
S3BundlePersistenceProvider
Salt and IV Encoding
SAML
SAML Authentication
Save Changes to a Versioned Flow
Scheduling Tab
Schema Differences & Limitations
Scope
ScriptEventHookProvider
Scrypt
Search Components in DataFlow
Searching
Searching for Events
Securing ZooKeeper with Kerberos
Securing ZooKeeper with TLS
Security Configuration
Security Configuration
Security Configuration
Security Properties
Security Properties
Security Recommendations
Sensitive Property Key Migration
Sensitive Property Key Migration
Session Rollback
Set Property Values
Set up Cloudera Flow Management
Set up your network configuration
Setting core configuration service
Setting database password for NiFi Registry
Setting up NiFi node users in NiFi Registry
Settings
Settings Tab
Shared Event Hook Properties
ShellUserGroupProvider
Show Local Changes
Signing with Externally-signed CA Certificates
Site to Site and Reverse Proxy Examples
Site to Site Properties
Site to Site protocol sequence
Site to Site Routing Properties for Reverse Proxies
Site-to-Site
Sizing your Flow Management Cluster
Sizing your Flow Management cluster
Sorting & Filtering Buckets
Sorting & Filtering Flows
Sorting & Filtering Users/Groups
Sorting and Filtering Components
Special Privilege Policies
Special Privileges
Split Content (One-to-Many)
Standalone
Standalone Functions
StandardManagedAuthorizer
StandardManagedAuthorizer
Start New NiFi
Start NiFi and NiFi Registry
Start Version Control
Start version control
Start Version Control on a Process Group
Start your NiFi and NiFi Registry services
Starting a Component
Starting and Stopping NiFi on Linux
Starting and Stopping NiFi on Windows
Starting Cloudera Manager server and adding license
Starting NiFi Registry
Starting Zookeeper service
startsWith
State Management
State Management
State Manager
StateManager
StaticKeyProvider
StaticKeyProvider
Status
Status History Repository
Stop Version Control
Stopping a Component
Storing and Retrieving State
String Manipulation
Structure of a NiFi Expression
Structure of a RecordPath
Subjectless Functions
substring
substringAfter
substringAfterLast
substringBefore
substringBeforeLast
Summary Page
Supplying a contribution
Support Matrix
Supported NiFi Controller Services
Supported NiFi Processors
Supported NiFi Registry databases
Supported NiFi Reporting Tasks
Supported operating systems
Supporting API
Swap Management
Switching from other Flow Persistence Provider
System Properties
System Properties
System Properties
System Requirements
System Requirements
Technical Preview Features
Technologies
Templates
Terminating a Component's tasks
Terminology
Terminology
Terminology
Terminology Used in This Guide
Testing
The core concepts of NiFi
Timer and event driven thread pools
TLS Generation Toolkit
TLS Toolkit
TLS/SSL certificate requirements and recommendations
TLS/SSL Configuration
toBytes
toDate
toLowerCase
toString
toUpperCase
Trace Data Lifecycle
trim
Troubleshooting
Troubleshooting HDF upgrade
Troubleshooting Kerberos Configuration
Tuning your DataFlow
Tuning your dataflow
Turn off identity mapping
Turn off TLS regeneration
Type Coercion
UI Extensions
Understand the use case
Understand your Base cluster layout
Understand your Compute cluster layout
Understand your mixed cluster layout
Understanding the Ranger Authorization Process for CFM
Understanding Version Dependencies
Understanding your deployment scenario
Unit Tests
Unsupported Browsers
Unsupported Browsers
Unsupported Customizations
Unsupported Features
Update a flow with sensitive properties
Update Attributes Based on Content
Update references to cluster nodes
Update the Configuration Files
Update the Configuration Files for Your New NiFi Installation
Update the Configuration Files for Your New NiFi Installation
Update the Registry Client
Upgrade
Upgrade Paths
Upgrade Recommendations
Upgrading HDF to CFM on CDP Private Cloud Base
Upgrading NiFi
Upgrading NiFi Manually
Upgrading NiFi Registry Manually
Upgrading to CFM 2.1.2
Upgrading to CFM 2.1.2 from CFM 1.1.0
Upgrading to CFM 2.1.2 from CFM 2.x
Upload a template
Upload Bundle
URL Aliasing
Usage
Usage
Usage
Usage
Usage
Usage
Usage
Usage
Use a template
User Authentication
User Authentication
User group providers
User Window
UserGroupProvider
Using a Domain User for NiFi Windows Service
Using a Local User for NiFi Windows Service
Using An Existing Intermediate Certificate Authority (CA)
Using Apache NiFi
Using Apache NiFi Registry
Using Custom Properties with Expression Language
Using the Apache NiFi Toolkit
uuid5
Validate Output
Validating Processor Properties
ValidationContext
Validator
Variables
Variables in Versioned Flows
Verify CFM 2.1.2
Verifying Ranger configurations
Version data flows
Version States
Versioning a DataFlow
View a Flow
Viewing FlowFile Lineage
Viewing Policies on Users
Viewing the number of cores
Viewing the total number of active threads
Viewing the total number of threads in a cluster
Viewing the UI in Variably Sized Browsers
Viewing the UI in Variably Sized Browsers
Volatile Content Repository Properties
Volatile FlowFile Repository
Volatile Provenance Repository Properties
Web Properties
Web Properties
What is Apache NiFi?
What is Cloudera Flow Management?
What is it?
What is it?
What is it?
What's new in this release?
When Processors are Triggered
Where To Go For More Information
Where to Start?
Who is this guide for?
Who is This Guide For?
Why Cluster?
Wildcard Certificates
Work with data flow templates
Write Ahead FlowFile Repository
Write Ahead Provenance Repository
Write Ahead Provenance Repository Properties
Writing and Reading Content Claims
Writing and Reading Event Records
Writing and Reading FlowFiles
XML Property Enhancements
Zero-Leader Clustering
ZooKeeper Access Control
ZooKeeper Migration Steps
ZooKeeper Migrator
ZooKeeper Migrator
ZooKeeper Properties
«
Filter topics
4. Definitions
▶︎
1. Overview
1.1. Version information
1.2. Contact information
1.3. License information
1.4. URI scheme
1.5. Tags
▶︎
2. Security
2.1. Authorization
2.2. BasicAuth
▶︎
3. Resources
▶︎
3.1. Access
3.1.1. Get access status
3.1.2. Performs a logout for other providers that have been issued a JWT.
3.1.3. Redirect/callback URI for processing the result of the OpenId Connect login sequence.
3.1.4. Retrieves a JWT following a successful login sequence using the configured OpenId Connect provider.
3.1.5. Performs a logout in the OpenId Provider.
3.1.6. Initiates a request to authenticate through the configured OpenId Connect provider.
3.1.7. Create token trying all providers
3.1.8. Create token using identity provider
3.1.9. Test identity provider
3.1.10. Get identity provider usage
3.1.11. Create token using kerberos
3.1.12. Create token using basic auth
▶︎
3.2. Bucket Bundles
3.2.1. Get extension bundles by bucket
3.2.2. Create extension bundle version
▶︎
3.3. Bucket Flows
3.3.1. Create flow
3.3.2. Get bucket flows
3.3.3. Get bucket flow
3.3.4. Update bucket flow
3.3.5. Delete bucket flow
3.3.6. Get bucket flow diff
3.3.7. Create flow version
3.3.8. Get bucket flow versions
3.3.9. Get latest bucket flow version content
3.3.10. Get latest bucket flow version metadata
3.3.11. Get bucket flow version
▶︎
3.4. Buckets
3.4.1. Create bucket
3.4.2. Get all buckets
3.4.3. Get bucket fields
3.4.4. Get bucket
3.4.5. Update bucket
3.4.6. Delete bucket
▶︎
3.5. Bundles
3.5.1. Get all bundles
3.5.2. Get all bundle versions
3.5.3. Get bundle
3.5.4. Delete bundle
3.5.5. Get bundle versions
3.5.6. Get bundle version
3.5.7. Delete bundle version
3.5.8. Get bundle version content
3.5.9. Get bundle version extensions
3.5.10. Get bundle version extension
3.5.11. Get bundle version extension docs
3.5.12. Get bundle version extension docs details
▶︎
3.6. Config
3.6.1. Get configration
▶︎
3.7. Extension Repository
3.7.1. Get extension repo buckets
3.7.2. Get extension repo groups
3.7.3. Get extension repo artifacts
3.7.4. Get extension repo versions
3.7.5. Get extension repo version
3.7.6. Get extension repo version content
3.7.7. Get extension repo extensions
3.7.8. Get extension repo extension
3.7.9. Get extension repo extension docs
3.7.10. Get extension repo extension details
3.7.11. Get extension repo version checksum
3.7.12. Get global extension repo version checksum
▶︎
3.8. Extensions
3.8.1. Get all extensions
3.8.2. Get extensions providing service API
3.8.3. Get extension tags
▶︎
3.9. Flows
3.9.1. Get flow fields
3.9.2. Get flow
3.9.3. Get flow versions
3.9.4. Get latest flow version
3.9.5. Get latest flow version metadata
3.9.6. Get flow version
▶︎
3.10. Items
3.10.1. Get all items
3.10.2. Get item fields
3.10.3. Get bucket items
▶︎
3.11. Policies
3.11.1. Create access policy
3.11.2. Get all access policies
3.11.3. Get available resources
3.11.4. Get access policy for resource
3.11.5. Get access policy
3.11.6. Update access policy
3.11.7. Delete access policy
▶︎
3.12. Tenants
3.12.1. Create user group
3.12.2. Get user groups
3.12.3. Get user group
3.12.4. Update user group
3.12.5. Delete user group
3.12.6. Create user
3.12.7. Get all users
3.12.8. Get user
3.12.9. Update user
3.12.10. Delete user
▼
4. Definitions
4.1. AccessPolicy
4.2. AccessPolicySummary
4.3. AllowableValue
4.4. Attribute
4.5. BatchSize
4.6. Bucket
4.7. BucketItem
4.8. BuildInfo
4.9. Bundle
4.10. BundleInfo
4.11. BundleVersion
4.12. BundleVersionDependency
4.13. BundleVersionMetadata
4.14. ComponentDifference
4.15. ComponentDifferenceGroup
4.16. ConnectableComponent
4.17. ControllerServiceAPI
4.18. ControllerServiceDefinition
4.19. CurrentUser
4.20. DeprecationNotice
4.21. DynamicProperty
4.22. DynamicRelationship
4.23. Extension
4.24. ExtensionBundle
4.25. ExtensionFilterParams
4.26. ExtensionMetadata
4.27. ExtensionMetadataContainer
4.28. ExtensionRepoArtifact
4.29. ExtensionRepoBucket
4.30. ExtensionRepoGroup
4.31. ExtensionRepoVersion
4.32. ExtensionRepoVersionSummary
4.33. ExternalControllerServiceReference
4.34. Fields
4.35. JaxbLink
4.36. Permissions
4.37. Position
4.38. Property
4.39. ProvidedServiceAPI
4.40. RegistryConfiguration
4.41. Relationship
4.42. Resource
4.43. ResourcePermissions
4.44. Restricted
4.45. Restriction
4.46. RevisionInfo
4.47. Stateful
4.48. SystemResourceConsideration
4.49. TagCount
4.50. Tenant
4.51. User
4.52. UserGroup
4.53. VersionedConnection
4.54. VersionedControllerService
4.55. VersionedFlow
4.56. VersionedFlowCoordinates
4.57. VersionedFlowDifference
4.58. VersionedFlowSnapshot
4.59. VersionedFlowSnapshotMetadata
4.60. VersionedFunnel
4.61. VersionedLabel
4.62. VersionedParameter
4.63. VersionedParameterContext
4.64. VersionedPort
4.65. VersionedProcessGroup
4.66. VersionedProcessor
4.67. VersionedPropertyDescriptor
4.68. VersionedRemoteGroupPort
4.69. VersionedRemoteProcessGroup
»
Apache NiFi Registry REST API
4. Definitions
4.1. AccessPolicy
4.2. AccessPolicySummary
4.3. AllowableValue
4.4. Attribute
4.5. BatchSize
4.6. Bucket
4.7. BucketItem
4.8. BuildInfo
4.9. Bundle
4.10. BundleInfo
4.11. BundleVersion
4.12. BundleVersionDependency
4.13. BundleVersionMetadata
4.14. ComponentDifference
4.15. ComponentDifferenceGroup
4.16. ConnectableComponent
4.17. ControllerServiceAPI
4.18. ControllerServiceDefinition
4.19. CurrentUser
4.20. DeprecationNotice
4.21. DynamicProperty
4.22. DynamicRelationship
4.23. Extension
4.24. ExtensionBundle
4.25. ExtensionFilterParams
4.26. ExtensionMetadata
4.27. ExtensionMetadataContainer
4.28. ExtensionRepoArtifact
4.29. ExtensionRepoBucket
4.30. ExtensionRepoGroup
4.31. ExtensionRepoVersion
4.32. ExtensionRepoVersionSummary
4.33. ExternalControllerServiceReference
4.34. Fields
4.35. JaxbLink
4.36. Permissions
4.37. Position
4.38. Property
4.39. ProvidedServiceAPI
4.40. RegistryConfiguration
4.41. Relationship
4.42. Resource
4.43. ResourcePermissions
4.44. Restricted
4.45. Restriction
4.46. RevisionInfo
4.47. Stateful
4.48. SystemResourceConsideration
4.49. TagCount
4.50. Tenant
4.51. User
4.52. UserGroup
4.53. VersionedConnection
4.54. VersionedControllerService
4.55. VersionedFlow
4.56. VersionedFlowCoordinates
4.57. VersionedFlowDifference
4.58. VersionedFlowSnapshot
4.59. VersionedFlowSnapshotMetadata
4.60. VersionedFunnel
4.61. VersionedLabel
4.62. VersionedParameter
4.63. VersionedParameterContext
4.64. VersionedPort
4.65. VersionedProcessGroup
4.66. VersionedProcessor
4.67. VersionedPropertyDescriptor
4.68. VersionedRemoteGroupPort
4.69. VersionedRemoteProcessGroup
This site uses cookies and related technologies, as described in our
privacy policy
, for purposes that may include site operation, analytics, enhanced user experience, or advertising. You may choose to consent to our use of these technologies, or
manage your own preferences.
Accept all
4.0.0
2.1
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.0
2.0.4
2.0.1
