Homepage/Cloudera DataFlow for Data Hub7.2.2

(Public Cloud)

Search Documentation

▶︎Cloudera
1. Reference Architectures
▼Cloudera Public Cloud
▶︎Cloudera Private Cloud
▶︎Cloudera Manager
1. Cloudera Manager
▶︎Applications
▶︎Legacy
▶︎

Getting Started
Patterns
Preview Features
Data Catalog
Data Engineering
DataFlow
Data Hub
Data Warehouse
Data Warehouse Runtime
Cloudera AI
Management Console
Operational Database
Replication Manager
Cloudera Manager
CDF for Data Hub
Runtime

«

Filter topics

CDF for Data Hub
▶︎Release Notes
▶︎Concepts
▶︎Planning
▼How To: Flow Management
▶︎How To: Streams Messaging
▶︎How To: Streaming Analytics
▶︎Reference
▶︎Learning & Training
1. ▶︎Getting Started with Apache NiFi
2. ▶︎Getting Started with Apache NiFi Registry

.NET client
@OnAdded
@OnEnabled
@OnPrimaryNodeStateChange
@OnRemoved
@OnScheduled
@OnShutdown
@OnStopped
@OnUnscheduled
AbstractProcessor API
Accessing Parameters
Accessing the UI with Multi-Tenant Authorization
Add a User
Add an Empty Group
Add ControllerServices
Add Ranger policies
Add Ranger policies
Add the user or group to a pre-defined access policy
Add the user to predefined Ranger access policies
Add User to a Group
Adding a new schema
Adding a Parameter to a Parameter Context
Adding a Processor
Adding and Configuring Record Reader and Writer Controller Services
Adding Components to the Canvas
Adding Controller Services for Dataflows
Adding Controller Services for Reporting Tasks
Adding Functionality to Apache NiFi
Adding User-Defined Attributes
Additional Actions
Additional Help
Additional Testing Capabilities
Advanced Documentation
Alert Policies Overview
Align Horizontally
Align Vertically
Amazon Web Services
Analyzing data with Apache HBase in CDP Public Cloud
Analyzing data with Apache Kafka in CDP Public Cloud
Analyzing data with Apache Kudu in CDP Public Cloud
Analyzing your data with HBase
Analyzing your data with Kafka
Analyzing your data with Kudu
Anatomy of a Process Group
Anatomy of a Processor
Anatomy of a Remote Process Group
Apache Flink Overview
Apache Kafka
Apache Kafka Overview
Apache NiFi
Apache NiFi Overview
Apache NiFi Registry REST API Reference
Apache NiFi Registry REST API Reference
Apache NiFi REST API Reference
Apache NiFi REST API Reference
Apache patch information
Appendix - Schema example
Assign resource roles
Assign the EnvironmentUser role
Assigning a Parameter Context to a Process Group
Assigning administrator level permissions
Assigning selective permissions to user
Attribute Extraction
Authentication
Authorization
Authorization example
Authorization workflow
Authorizing Flow Management Cluster Access in CDP Public Cloud
Backwards Compatibility
Basics
Bending Connections
Bootstrap.conf
Broker garbage log collection and log rotation
Broker log management
Broker migration
Broker Tuning
Brokers
Browser Support
Browser Support
Bucket Policies
Build the data flow
Build the data flow
Build the data flow
Build the data flow
Build the data flow
Build the data flow
Building a DataFlow
Building an Apache NiFi DataFlow
Building Cloudera Manager charts with Kafka metrics
Bulletins
Bundle Coordinates
Bundle Id
CDF for Data Hub
Change Version
Change Version
Changing Component Versions
Changing Configuration and Context Menu Options
Choosing the number of partitions for a topic
Client and broker compatibility across Kafka versions
Client authentication using delegation tokens
Client examples
Client examples
Cluster sizing
Cohesion and Reusability
Command and Control of the DataFlow
Command Line Tools
Command Line Tools
Comments Tab
Commit Local Changes
Commit Local Changes
Common Attributes
Common Processor Patterns
Common vulnerabilities and exposures
Compatibility Policies
Component Alignment
Component Lifecycle
Component Linking
Component Notification
Component Statistics
Component support
Component Types and Metrics for Alert Policies
Component Versions
ComponentLog
Configure clients on a producer or consumer level
Configure clients on an application level
Configure JMX ephemeral ports
Configure Kafka brokers
Configure Kafka brokers
Configure Kafka clients
Configure Kafka clients
Configure Ranger policies for site-to-site communication
Configure Site-to-Site client NiFi instance
Configure Site-to-Site Server NiFi Instance
Configure the Controller Service
Configure the controller services
Configure the HBase client service
Configure the processor for merging records
Configure the processor for merging records
Configure the processor for your data source
Configure the processor for your data source
Configure the processor for your data source
Configure the processor for your data source
Configure the processor for your data source
Configure the processor for your data source
Configure the processor for your data target
Configure the processor for your data target
Configure the processor for your data target
Configure the processor for your data target
Configure the processor for your data target
Configure the processor for your data target
Configure the resource-based Ranger service used for authorization
Configure the service account
Configure your truststores
Configure Zookeeper TLS/SSL support for Kafka
Configuring a Processor
Configuring a Processor
Configuring Apache Kafka
Configuring Kafka ZooKeeper chroot
Configuring log levels for command line tools
Connect NiFi to the Registry
Connecting Components
Connecting Kafka Clients to CDP Public Cloud Clusters
Connecting Kafka clients to Data Hub provisioned clusters
Connecting Kafka clients to Data Hub provisioned clusters
Connecting Processors
Connecting to a NiFi Registry
Connecting to a NiFi Registry
Consider the User
Contact Us
Content Viewers
Controller Services
Controller Services
Create a Bucket
Create a Bucket
Create a Bucket Policy
Create a custom access policy
Create a Custom Access Policy
Create a New Group with Selected Users
Create Atlas entity type definitions
Create controller services for your data flow
Create controller services for your data flow
Create controller services for your data flow
Create IDBroker mapping
Create IDBroker mapping
Create IDBroker mapping
Create IDBroker mapping
Create the HBase target table
Create the Hive target table
Create the Kudu target table
Create your cluster
Create your cluster
Create your cluster
Create your streaming clusters
Creating a Kafka Topic
Creating a Notifier
Creating an Alert Policy
Creating checkpoints and savepoints in Flink
Creating your first Flow Management cluster
Creating your First Flow Management Cluster in CDP Public Cloud
Creating your first Streaming Analytics cluster
Creating your First Streaming Analytics Cluster in CDP Public Cloud
Creating your first Streams Messaging cluster
Creating your First Streams Messaging Cluster in CDP Public Cloud
Custom Processor UIs
Data Buffering
Data Egress
Data Egress / Sending Data
Data Hub cluster definitions
Data Ingestion
Data Ingress
Data Provenance
Data Provenance
Data querying with SQL Client
Data Transformation
Database Access
Define your CDP Private Cloud Base dataflow
Define your CDP Public Cloud dataflow
Delegation token based authentication
Delete a Bucket
Delete a Bucket Policy
Delete a Flow
Delete a User
Delete Multiple Buckets
Delete Multiple Users
Deleting a Kafka Topic
Deleting a Notifier
Deleting a schema
Deleting an Alert Policy
Deployment scenarios
Deployment scenarios
Deprecating a Component
Details of an Event
Details Tab
Developing a ControllerService
Developing a Reporting Task
Developing Apache Kafka Applications
Disabling an Alert Policy
Disk management
Disk Removal
Disk Replacement
Documenting a Component
Documenting Capability and Keywords
Documenting FlowFile Attribute Interaction
Documenting Properties
Documenting Related Components
Documenting Relationships
Download Bundle
Downloading and Installing NiFi
Downloading and Installing NiFi Registry
Edit a Bucket Name
Edit a User Name
Enable authorization in Kafka with Ranger
Enable or disable authentication with delegation tokens
Enabling an Alert Policy
Enabling Interceptors
Enabling/Disabling a Component
Enabling/Disabling Controller Services
Encrypted Content Repository
Encrypted FlowFile Repository
Encrypted Provenance Considerations
Encrypted Provenance Repository
End to End Latency Overview
End to End Latency Use Cases
Enqueue FlowFiles
Enrich/Modify Content
Error Handling
Essential metrics to monitor
Event Details
Event-driven applications with Flink
Evolving a schema
Example Dataflow
Examples of Interacting with Schema Registry
Exceptions within a callback: IOException, RuntimeException
Exceptions within the Processor
Expanding an Event
Experimental Warning
Exposing Processor Properties
Exposing Processor's Relationships
Expression Language / Using Attributes in Property Values
Extracting Attributes
File descriptor limits
FileBasedKeyProvider
FileBasedKeyProvider
Filesystems
Find Parents
Fixed issues
Flink Streaming Applications
Flow Management cluster definitions
Flow Management cluster layout
FlowFile
For Linux/Mac OS X users
For Linux/Unix/Mac OS X users
For Windows Users
General Design Considerations
Getting More Info for a Processor
Getting Started with Apache NiFi
Getting Started with Apache NiFi Registry
Give users access to your cluster
Give users access to your cluster
Give users access to your cluster
Grant Special Privileges to a User
Granularity of Metrics
Group Window
Groups and fetching
Handling disk failures
Handling large messages
Handling state in Flink
High Level Overview of Key NiFi Features
Historical Statistics of a Component
How does it work?
How does it work?
How does it work?
How to contribute to Apache NiFi
HTTP
I Started NiFi Registry. Now What?
I Started NiFi. Now What?
Import a Versioned Flow
Import a Versioned Flow
Import a Versioned Flow
Individual Port Transmission
Ingesting data into Amazon S3
Ingesting Data into Amazon S3 Buckets
Ingesting Data into Apache HBase in CDP Cloud
Ingesting Data into Apache HBase in CDP Public Cloud
Ingesting Data into Apache Hive in CDP Public Cloud
Ingesting Data into Apache Hive in CDP Public Cloud
Ingesting data into Apache Kafka
Ingesting Data into Apache Kafka in CDP Public Cloud
Ingesting Data into Apache Kudu in CDP Public Cloud
Ingesting Data into Apache Kudu in CDP Public Cloud
Ingesting Data into Azure Data Lake Storage
Ingesting data into Azure Data Lake Storage
Ingesting Data into CDP Public Cloud
Installing as a Service
Installing as a Service
Instantiate TestRunner
Integrating Kafka and Schema Registry
Integrating Kafka and Schema Registry Using NiFi Processors
Integrating with Kafka
Integrating with NiFi
Integrating with Schema Registry
Interacting with a ControllerService
Introduction
Introduction
ISR management
Java client
JBOD
JBOD Disk migration
JBOD setup
Job monitoring with Flink Dashboard
Job monitoring with Flink Dashboard
Job monitoring with Flink Dashboard
JVM and garbage collection
Kafka Architecture
Kafka brokers and Zookeeper
Kafka clients and ZooKeeper
Kafka consumers
Kafka FAQ
Kafka Introduction
Kafka producers
Kafka public APIs
Kafka security hardening with Zookeeper ACLs
Kafka Streams
kafka-*-perf-test
kafka-configs
kafka-console-consumer
kafka-console-producer
kafka-consumer-groups
kafka-delegation-tokens
kafka-log-dirs
kafka-reassign-partitions
kafka-topics
Kafka-ZooKeeper performance tuning
Kerberos authentication
Key Rotation
Key Rotation
Known issues and limitations
LDAP authentication
Leader positions and in-sync replicas
Lineage Graph
Log cleaner
Logging In
Logging In
Logs and log segments
Manage Buckets
Manage Bundles
Manage Flows
Manage Groups
Manage individual delegation tokens
Manage Users & Groups
Management basics
Managing Alert Policies
Managing Alert Policies and Notifiers
Managing an Apache NiFi DataFlow
Managing Apache Kafka
Managing Local Changes
Managing Local Changes
Managing Topics
Managing topics across multiple Kafka clusters
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Meet the prerequisites
Metadata governance with Atlas
Metadata governance with Atlas
Metadata governance with Atlas
Metrics
Migrate brokers by modifying broker IDs in meta.properties
Mocking External Resources
Modify a Kafka Topic
Monitoring
Monitoring an Apache NiFi DataFlow
Monitoring Brokers
Monitoring Clusters
Monitoring Consumers
Monitoring End to End Latency
Monitoring End-to-end Latency
Monitoring Kafka Clusters
Monitoring NiFi
Monitoring of DataFlow
Monitoring Producers
Monitoring Topics
Monitoring your data flow
Monitoring your data flow
Monitoring your data flow
Moving data from CDP Private Cloud Base to Public Cloud with NiFi site-to-site
Moving Data using NiFi Site-to-Site
Naming Conventions
Navigating an Apache NiFi DataFlow
Navigating within a DataFlow
Nested Versioned Flows
Nested Versioned Flows
Network and I/O threads
Networking parameters
Next steps
Next steps
Next steps
Next steps
Next steps
Next steps
Next steps
Next steps
Next steps
NiFi Architecture
NiFi Archives (NARs)
NiFi Components
NiFi patches
NiFi Registry patches
NiFi Registry User Interface
NiFi User Interface
Notifiers
Obtain HBase connection details
Obtain Hive connection details
Older Existing NiFi Version
Operating system requirements
Other Components
Other Group Level Actions
Other Management Features
PAM Authentication
Parameter Contexts
Parameters
Parameters in Versioned Flows
Parameters in Versioned Flows
Partitions
Penalization vs. Yielding
Per-Instance ClassLoading
Performance considerations
Performance Expectations and Characteristics of NiFi
Performant .NET producer
Performing the Work
Planning your Flow Management deployment
Planning your Streaming Analytics deployment
Planning your Streams Messaging deployment
Potential Issues
Potential Issues
Potential Issues
Pre-defined Access Policies for Schema Registry
Predefined Ranger Access Policies for Apache NiFi
Predefined Ranger Access Policies for Apache NiFi Registry
Prepare your clusters
Prepare your environment
ProcessContext
Processor API
Processor Behavior Annotations
Processor Initialization
Processor Validation
ProcessorInitializationContext
ProcessSession
Properties Tab
PropertyDescriptor
PropertyValue
Protocol between consumer and broker
Provenance Events
Provenance Events
Querying a schema
Queue Interaction
Quotas
Ranger
Reassigning replicas between log directories
Reassignment examples
Rebalancing partitions
Recommendations for client development
Record management
Record order and assignment
Records
Referencing Custom Properties via nifi.properties
Referencing Parameters
Relationship
Release Notes
Remote Process Group Transmission
Remove a User from a Group
Replaying a FlowFile
Reporting Processor Activity
Reporting Tasks
Reporting Tasks
Responding to Changes in Configuration
Restrict access to Kafka metadata in Zookeeper
Restricted
Restricted Components in Versioned Flows
Restricted Components in Versioned Flows
Restricted Controller Service Created in Process Group
Restricted Controller Service Created in Process Group
Restricted Controller Service Created in Root Process Group
Restricted Controller Service Created in Root Process Group
Retries
Retrieve keytab file
Retrieving log directory replica assignment information
Revert Local Changes
Revert Local Changes
Rotate the master key/secret
Route Based on Attributes
Route Based on Content (One-to-Many)
Route Based on Content (One-to-One)
Route Streams Based on Content (One-to-Many)
Routing and Mediation
Routing on Attributes
Run the Processor
Save Changes to a Versioned Flow
Scheduling Tab
Schema Entities
Schema Registry
Schema Registry Authorization through Ranger Access Policies
Schema Registry Component Architecture
Schema Registry Concepts
Schema Registry Overview
Schema Registry Overview
Schema Registry Use Cases
Scope
Searching for Events
Securing Apache Kafka
Securing Schema Registry
Security examples
Security examples
Security for Flow Management Clusters and Users in CDP Public Cloud
Session Rollback
Set Property Values
Set Ranger policies
Set up MirrorMaker in Cloudera Manager
Set up your network configuration
Set workload password
Setting up the HortonworksSchemaRegistry Controller Service
Setting user limits for Kafka
Settings
Settings Tab
Settings to avoid data loss
Show Local Changes
Show Local Changes
Simple .NET consumer
Simple .NET producer
Simple Java consumer
Simple Java producer
Site-to-Site
Sizing estimation based on network and disk message throughput
Sophisticated windowing in Flink
Sorting & Filtering Buckets
Sorting & Filtering Flows
Sorting & Filtering Users/Groups
Sorting and Filtering Components
Special Privileges
Split Content (One-to-Many)
Splitting and Aggregation
Start the data flow
Start the data flow
Start the data flow
Start Version Control
Start Version Control
Start Version Control on a Process Group
Start your data flow
Start your data flow
Start your data flow
Starting a Component
Starting and Stopping Processors
Starting NiFi
Starting NiFi Registry
State Manager
Stateless Mode and High Availability
StateManager
StaticKeyProvider
StaticKeyProvider
Status Bar
Step 1: Generate keys and certificates for Kafka brokers
Step 2: Create your own certificate authority
Step 3: Sign the certificate
Step 4: Configure Kafka brokers
Step 5: Configure Kafka clients
Stop Version Control
Stop Version Control
Stopping a Component
Storing and Retrieving State
Streaming Analytics cluster layout
Streaming Analytics Data Hub cluster definitons
Streaming Analytics deployment scenarios
Streaming use cases with Flink
Streams Messaging
Streams Messaging cluster layout
Streams Messaging Manager
Streams Messaging Manager Overview
Streams Messaging Manager Overview
Subscribing to a topic
Summary Page
Supplying a contribution
Supporting API
System Interaction
System Level Broker Tuning
System Properties
Technologies
Terminology
Terminology
Terminology Used in This Guide
Terminology Used in This Guide
Testing
The core concepts of NiFi
TLS
Tool usage
Topics
Tuning Apache Kafka Performance
UI Extensions
Understand the use case
Understand the use case
Understand the use case
Understand the use case
Understand the use case
Understand the use case
Understand the use case
Understand the use case
Understanding NiFi Record Based Processing
Understanding the kafka-run-class Bash Script
Understanding Version Dependencies
Unit Tests
Unlock Kafka metadata in Zookeeper
Unsupported Apache NiFi extensions
Unsupported Browsers
Unsupported Browsers
Unsupported command line tools
Unsupported features
Unsupported Flow Management features
Unsupported Streams Messaging features
Update Attributes Based on Content
Updating a Notifier
Updating an Alert Policy
Upload Bundle
Use Case 1: Registering and Querying a Schema for a Kafka Topic
Use Case 2: Reading/Deserializing and Writing/Serializing Data from and to a Kafka Topic
Use Case 3: Dataflow Management with Schema-based Routing
Use cases
Use rsync to copy files from one broker to another
User Authorization
User Window
Using Apache NiFi Provenance Tools
Using Apache NiFi Registry
Using Apache NiFi Templates
Using Custom Properties with Expression Language
Using Kafka's inter-broker security
Using Record-Enabled Processors
Using Schema Registry
Using the Apache NiFi Interface
Using watermark in Flink
Validate Output
Validating Processor Properties
ValidationContext
Validator
Variables
Variables in Versioned Flows
Variables in Versioned Flows
Verify data flow operation
Verify data flow operation
Verify data flow operation
Verify that you can write data to Kudu
Verify your data flow
Verify your data flow
Version States
Version States
Versioning a DataFlow
Versioning a DataFlow
Versioning an Apache NiFi DataFlow
View a Flow
Viewing FlowFile Lineage
Viewing the UI in Variably Sized Browsers
Viewing the UI in Variably Sized Browsers
Virtual memory handling
What is Apache Flink?
What is Apache NiFi?
What is it?
What is it?
What is it?
What Processors are Available
What's New in Cloudera Data Flow for Data Hub
When Processors are Triggered
Where to Start?
Who is This Guide For?
Working With Attributes
Working With Templates
Write Ahead Provenance Repository
Writing and Reading Content Claims
Writing and Reading Event Records
Writing and Reading FlowFiles
zookeeper-security-migration

«

Filter topics

Ingesting data into Amazon S3

▼Ingesting data into Amazon S3
Monitoring your data flow
Next steps

»

Ingesting Data into Amazon S3 Buckets

Ingesting data into Amazon S3

You can use an Apache NiFi data flow to ingest data into Amazon S3 object stores in CDP Public Cloud by following these steps.

Understand the use case
Learn how you can use a Flow Management cluster connected to a Streams Messaging cluster to build an end-to-end flow that ingests data to Amazon S3 storage. This example use case shows you how to use Apache NiFi to move data from Kafka to S3 buckets.
Meet the prerequisites
Use this checklist to make sure that you meet all the requirements before you start building your data flow.
Build the data flow
Learn how you can create an ingest data flow to move data from Kafka to S3 buckets. This involves opening Apache NiFi in your Flow Management cluster, adding processors and other data flow objects to your canvas, and connecting your data flow elements.
Create IDBroker mapping
Learn how you can create the IDBroker mapping for the PutHDFS processor for your data flow. To enable your CDP user to utilize the central authentication features CDP provides and to exchange credentials for AWS access tokens, you have to map your CDP user to the correct IAM role.
Create controller services for your data flow
Learn how you can create and configure controller services for an S3 ingest data flow in CDP Public Cloud. Controller services provide shared services that can be used by the processors in your data flow. You will use these Controller Services later when you configure your processors.
Configure the processor for your data source
Learn how you can configure the ConsumeKafkaRecord_2_0 data source processor for your S3 ingest data flow. You can set up a data flow to move data to Amazon S3 from many different locations. This example assumes that you are streaming data from Kafka and shows you the configuration for the relevant data source processor.
Configure the processor for merging records
Learn how you can configure the MergeRecord processor for your S3 ingest data flow. You can use it to merge together multiple record-oriented flow files into a large flow file that contains all records of your Kafka data input.
Configure the processor for your data target
Learn how you can configure the data target processor for your S3 ingest data flow. This example assumes that you are moving data to AWS S3 and shows you how to configure the corresponding processors.
Start the data flow
When your flow is ready, you can begin ingesting data into Amazon S3 buckets. Learn how to start your S3 ingest data flow.
Verify data flow operation
Learn how you can verify the operation of your S3 ingest data flow.

© 2019–2021 by Cloudera, Inc. All rights reserved.

7.3.1
7.2
7.1.0
7.0.2

This site uses cookies and related technologies, as described in our privacy policy, for purposes that may include site operation, analytics, enhanced user experience, or advertising. You may choose to consent to our use of these technologies, or