Example Config File
Below are two example config YAML files. The first tails the minifi-app.log, send the tailed log and provenance data back to a secure instance of NiFi. The second uses a series of processors to tail the app log, routes off only lines that contain "WriteAheadFlowFileRepository" and puts it as a file in the "" directory.
MiNiFi Config Version: 1 Flow Controller: name: MiNiFi Flow comment: Core Properties: flow controller graceful shutdown period: 10 sec flow service write delay interval: 500 ms administrative yield duration: 30 sec bored yield duration: 10 millis FlowFile Repository: partitions: 256 checkpoint interval: 2 mins always sync: false Swap: threshold: 20000 in period: 5 sec in threads: 1 out period: 5 sec out threads: 4 Provenance Repository: provenance rollover time: 1 min Content Repository: content claim max appendable size: 10 MB content claim max flow files: 100 always sync: false Component Status Repository: buffer size: 1440 snapshot frequency: 1 min Security Properties: keystore: /tmp/ssl/localhost-ks.jks keystore type: JKS keystore password: localtest key password: localtest truststore: /tmp/ssl/localhost-ts.jks truststore type: JKS truststore password: localtest ssl protocol: TLS Sensitive Props: key: algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL provider: BC Processors: - name: TailFile class: org.apache.nifi.processors.standard.TailFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 1 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: Properties: File to Tail: logs/minifi-app.log Rolling Filename Pattern: minifi-app* Initial Start Position: Beginning of File Connections: - name: TailToS2S source name: TailFile source relationship name: success destination name: 8644cbcc-a45c-40e0-964d-5e536e2ada61 max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer Remote Processing Groups: - name: NiFi Flow comment: url: https://localhost:8090/nifi timeout: 30 secs yield period: 10 sec Input Ports: - id: 8644cbcc-a45c-40e0-964d-5e536e2ada61 name: tailed log comments: max concurrent tasks: 1 use compression: false Provenance Reporting: comment: scheduling strategy: TIMER_DRIVEN scheduling period: 30 sec destination url: https://localhost:8090/ port name: provenance originating url: http://${hostname(true)}:8081/nifi use compression: true timeout: 30 secs batch size: 1000
Flow Controller: name: MiNiFi Flow comment: Core Properties: flow controller graceful shutdown period: 10 sec flow service write delay interval: 500 ms administrative yield duration: 30 sec bored yield duration: 10 millis max concurrent threads: 1 FlowFile Repository: partitions: 256 checkpoint interval: 2 mins always sync: false Swap: threshold: 20000 in period: 5 sec in threads: 1 out period: 5 sec out threads: 4 Content Repository: content claim max appendable size: 10 MB content claim max flow files: 100 always sync: false Component Status Repository: buffer size: 1440 snapshot frequency: 1 min Security Properties: keystore: /tmp/ssl/localhost-ks.jks keystore type: JKS keystore password: localtest key password: localtest truststore: /tmp/ssl/localhost-ts.jks truststore type: JKS truststore password: localtest ssl protocol: TLS Sensitive Props: key: algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL provider: BC Processors: - name: TailAppLog class: org.apache.nifi.processors.standard.TailFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 10 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: Properties: File to Tail: logs/minifi-app.log Rolling Filename Pattern: minifi-app* Initial Start Position: Beginning of File - name: SplitIntoSingleLines class: org.apache.nifi.processors.standard.SplitText max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - failure - original Properties: Line Split Count: 1 Header Line Count: 0 Remove Trailing Newlines: true - name: RouteErrors class: org.apache.nifi.processors.standard.RouteText max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - unmatched - original Properties: Routing Strategy: Route to 'matched' if line matches all conditions Matching Strategy: Contains Character Set: UTF-8 Ignore Leading/Trailing Whitespace: true Ignore Case: true Grouping Regular Expression: WALFFR: WriteAheadFlowFileRepository - name: PutFile class: org.apache.nifi.processors.standard.PutFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - failure - success Properties: Directory: ./ Conflict Resolution Strategy: replace Create Missing Directories: true Maximum File Count: Last Modified Time: Permissions: Owner: Group: Connections: - name: TailToSplit source name: TailAppLog source relationship name: success destination name: SplitIntoSingleLines max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer - name: SplitToRoute source name: SplitIntoSingleLines source relationship name: splits destination name: RouteErrors max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer - name: RouteToS2S source name: RouteErrors source relationship name: matched destination name: PutFile max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer Provenance Reporting: comment: scheduling strategy: TIMER_DRIVEN scheduling period: 30 sec destination url: https://localhost:8080/ port name: provenance originating url: http://${hostname(true)}:8081/nifi use compression: true timeout: 30 secs batch size: 1000