-
Processors
- AttributeRollingWindow
- AttributesToCSV
- AttributesToJSON
- CalculateParquetOffsets
- CalculateParquetRowGroupOffsets
- CalculateRecordStats
- CaptureChangeDebeziumDB2
- CaptureChangeDebeziumMongoDB
- CaptureChangeDebeziumMySQL
- CaptureChangeDebeziumOracle
- CaptureChangeDebeziumPostgreSQL
- CaptureChangeDebeziumSQLServer
- CaptureChangeMySQL
- CompressContent
- ConnectWebSocket
- ConsumeAMQP
- ConsumeAzureEventHub
- ConsumeBoxEnterpriseEvents
- ConsumeBoxEvents
- ConsumeElasticsearch
- ConsumeGCPubSub
- ConsumeIMAP
- ConsumeJMS
- ConsumeKafka
- ConsumeKafka_2_6
- ConsumeKafka2CDP
- ConsumeKafka2RecordCDP
- ConsumeKafkaRecord_2_6
- ConsumeKinesisStream
- ConsumeMQTT
- ConsumePLC
- ConsumePOP3
- ConsumeSlack
- ConsumeTwitter
- ConsumeWindowsEventLog
- ControlRate
- ConvertAvroToParquet
- ConvertCharacterSet
- ConvertProtobuf
- ConvertRecord
- CopyAzureBlobStorage_v12
- CopyS3Object
- CountText
- CreateHadoopSequenceFile
- CryptographicHashContent
- DebugFlow
- DecryptContentAge
- DecryptContentPGP
- DeduplicateRecord
- DeleteAzureBlobStorage_v12
- DeleteAzureDataLakeStorage
- DeleteByQueryElasticsearch
- DeleteCDPObjectStore
- DeleteDynamoDB
- DeleteFile
- DeleteGCSObject
- DeleteGridFS
- DeleteHBaseCells
- DeleteHBaseRow
- DeleteHDFS
- DeleteMongo
- DeleteS3Object
- DeleteSFTP
- DeleteSQS
- DetectDuplicate
- DistributeLoad
- DuplicateFlowFile
- EncodeContent
- EncryptContentAge
- EncryptContentPGP
- EnforceOrder
- EvaluateJsonPath
- EvaluateXPath
- EvaluateXQuery
- ExecuteGraphQuery
- ExecuteGraphQueryRecord
- ExecuteGroovyScript
- ExecuteProcess
- ExecuteScript
- ExecuteSparkInteractive
- ExecuteSQL
- ExecuteSQLRecord
- ExecuteStreamCommand
- ExtractAvroMetadata
- ExtractDocumentText
- ExtractEmailAttachments
- ExtractEmailHeaders
- ExtractGrok
- ExtractHL7Attributes
- ExtractImageMetadata
- ExtractMediaMetadata
- ExtractRecordSchema
- ExtractText
- FetchAzureBlobStorage_v12
- FetchAzureDataLakeStorage
- FetchBoxFile
- FetchBoxFileInfo
- FetchBoxFileRepresentation
- FetchCDPObjectStore
- FetchDistributedMapCache
- FetchDropbox
- FetchFile
- FetchFTP
- FetchGCSObject
- FetchGoogleDrive
- FetchGridFS
- FetchHBaseRow
- FetchHDFS
- FetchParquet
- FetchPLC
- FetchS3Object
- FetchSFTP
- FetchSmb
- FilterAttribute
- FlattenJson
- ForkEnrichment
- ForkRecord
- GenerateFlowFile
- GenerateRecord
- GenerateTableFetch
- GeoEnrichIP
- GeoEnrichIPRecord
- GeohashRecord
- GetAsanaObject
- GetAwsPollyJobStatus
- GetAwsTextractJobStatus
- GetAwsTranscribeJobStatus
- GetAwsTranslateJobStatus
- GetAzureEventHub
- GetAzureQueueStorage_v12
- GetBoxFileCollaborators
- GetBoxGroupMembers
- GetCouchbaseKey
- GetDynamoDB
- GetElasticsearch
- GetFile
- GetFileResource
- GetFTP
- GetGcpVisionAnnotateFilesOperationStatus
- GetGcpVisionAnnotateImagesOperationStatus
- GetHBase
- GetHDFS
- GetHDFSEvents
- GetHDFSFileInfo
- GetHDFSSequenceFile
- GetHubSpot
- GetJiraIssue
- GetMongo
- GetMongoRecord
- GetS3ObjectMetadata
- GetS3ObjectTags
- GetSFTP
- GetShopify
- GetSlackReaction
- GetSmbFile
- GetSNMP
- GetSnowflakeIngestStatus
- GetSolr
- GetSplunk
- GetSQS
- GetTCP
- GetWorkdayReport
- GetZendesk
- HandleHttpRequest
- HandleHttpResponse
- IdentifyMimeType
- InvokeGRPC
- InvokeHTTP
- InvokeScriptedProcessor
- ISPEnrichIP
- JoinEnrichment
- JoltTransformJSON
- JoltTransformRecord
- JSLTTransformJSON
- JsonQueryElasticsearch
- ListAzureBlobStorage_v12
- ListAzureDataLakeStorage
- ListBoxFile
- ListBoxFileInfo
- ListCDPObjectStore
- ListDatabaseTables
- ListDropbox
- ListenBeats
- ListenFTP
- ListenGRPC
- ListenHTTP
- ListenNetFlow
- ListenOTLP
- ListenSlack
- ListenSyslog
- ListenTCP
- ListenTrapSNMP
- ListenUDP
- ListenUDPRecord
- ListenWebSocket
- ListFile
- ListFTP
- ListGCSBucket
- ListGoogleDrive
- ListHBaseRegions
- ListHDFS
- ListS3
- ListSFTP
- ListSmb
- LogAttribute
- LogMessage
- LookupAttribute
- LookupRecord
- MergeContent
- MergeRecord
- ModifyBytes
- ModifyCompression
- MonitorActivity
- MoveAzureDataLakeStorage
- MoveHDFS
- Notify
- PackageFlowFile
- PaginatedJsonQueryElasticsearch
- ParseEvtx
- ParseNetflowv5
- ParseSyslog
- ParseSyslog5424
- PartitionRecord
- PublishAMQP
- PublishGCPubSub
- PublishJMS
- PublishKafka
- PublishKafka_2_6
- PublishKafka2CDP
- PublishKafka2RecordCDP
- PublishKafkaRecord_2_6
- PublishMQTT
- PublishSlack
- PutAccumuloRecord
- PutAzureBlobStorage_v12
- PutAzureCosmosDBRecord
- PutAzureDataExplorer
- PutAzureDataLakeStorage
- PutAzureEventHub
- PutAzureQueueStorage_v12
- PutBigQuery
- PutBoxFile
- PutCassandraQL
- PutCassandraRecord
- PutCDPObjectStore
- PutClouderaHiveQL
- PutClouderaHiveStreaming
- PutClouderaORC
- PutCloudWatchMetric
- PutCouchbaseKey
- PutDatabaseRecord
- PutDistributedMapCache
- PutDropbox
- PutDynamoDB
- PutDynamoDBRecord
- PutElasticsearchJson
- PutElasticsearchRecord
- PutEmail
- PutFile
- PutFTP
- PutGCSObject
- PutGoogleDrive
- PutGridFS
- PutHBaseCell
- PutHBaseJSON
- PutHBaseRecord
- PutHDFS
- PutIceberg
- PutIcebergCDC
- PutIoTDBRecord
- PutJiraIssue
- PutKinesisFirehose
- PutKinesisStream
- PutKudu
- PutLambda
- PutMongo
- PutMongoBulkOperations
- PutMongoRecord
- PutParquet
- PutPLC
- PutRecord
- PutRedisHashRecord
- PutS3Object
- PutSalesforceObject
- PutSFTP
- PutSmbFile
- PutSnowflakeInternalStage
- PutSNS
- PutSolrContentStream
- PutSolrRecord
- PutSplunk
- PutSplunkHTTP
- PutSQL
- PutSQS
- PutSyslog
- PutTCP
- PutUDP
- PutWebSocket
- PutZendeskTicket
- QueryAirtableTable
- QueryAzureDataExplorer
- QueryCassandra
- QueryDatabaseTable
- QueryDatabaseTableRecord
- QueryIoTDBRecord
- QueryRecord
- QuerySalesforceObject
- QuerySolr
- QuerySplunkIndexingStatus
- RemoveRecordField
- RenameRecordField
- ReplaceText
- ReplaceTextWithMapping
- ResizeImage
- RetryFlowFile
- RouteHL7
- RouteOnAttribute
- RouteOnContent
- RouteText
- RunMongoAggregation
- SampleRecord
- SawmillTransformJSON
- SawmillTransformRecord
- ScanAccumulo
- ScanAttribute
- ScanContent
- ScanHBase
- ScriptedFilterRecord
- ScriptedPartitionRecord
- ScriptedTransformRecord
- ScriptedValidateRecord
- SearchElasticsearch
- SegmentContent
- SelectClouderaHiveQL
- SendTrapSNMP
- SetSNMP
- SignContentPGP
- SplitAvro
- SplitContent
- SplitExcel
- SplitJson
- SplitPCAP
- SplitRecord
- SplitText
- SplitXml
- StartAwsPollyJob
- StartAwsTextractJob
- StartAwsTranscribeJob
- StartAwsTranslateJob
- StartGcpVisionAnnotateFilesOperation
- StartGcpVisionAnnotateImagesOperation
- StartSnowflakeIngest
- TagS3Object
- TailFile
- TransformXml
- TriggerClouderaHiveMetaStoreEvent
- UnpackContent
- UpdateAttribute
- UpdateByQueryElasticsearch
- UpdateClouderaHiveTable
- UpdateCounter
- UpdateDatabaseTable
- UpdateDeltaLakeTable
- UpdateJiraIssue
- UpdateRecord
- ValidateCsv
- ValidateJson
- ValidateRecord
- ValidateXml
- VerifyContentMAC
- VerifyContentPGP
- Wait
-
Controller Services
- AccumuloService
- ActiveMQJMSConnectionFactoryProvider
- ADLSCredentialsControllerService
- ADLSCredentialsControllerServiceLookup
- ADLSIDBrokerCloudCredentialsProviderControllerService
- AmazonGlueSchemaRegistry
- ApicurioSchemaRegistry
- AvroReader
- AvroRecordSetWriter
- AvroSchemaRegistry
- AWSCredentialsProviderControllerService
- AWSIDBrokerCloudCredentialsProviderControllerService
- AzureBlobIDBrokerCloudCredentialsProviderControllerService
- AzureBlobStorageFileResourceService
- AzureCosmosDBClientService
- AzureDataLakeStorageFileResourceService
- AzureEventHubRecordSink
- AzureServiceBusJMSConnectionFactoryProvider
- AzureStorageCredentialsControllerService_v12
- AzureStorageCredentialsControllerServiceLookup_v12
- CassandraDistributedMapCache
- CassandraSessionProvider
- CdpCredentialsProviderControllerService
- CdpOauth2AccessTokenProviderControllerService
- CEFReader
- CiscoEmblemSyslogMessageReader
- ClouderaAttributeSchemaReferenceReader
- ClouderaAttributeSchemaReferenceWriter
- ClouderaEncodedSchemaReferenceReader
- ClouderaEncodedSchemaReferenceWriter
- ClouderaHiveConnectionPool
- ClouderaSchemaRegistry
- CMLLookupService
- ConfluentEncodedSchemaReferenceReader
- ConfluentEncodedSchemaReferenceWriter
- ConfluentSchemaRegistry
- CouchbaseClusterService
- CouchbaseKeyValueLookupService
- CouchbaseMapCacheClient
- CouchbaseRecordLookupService
- CSVReader
- CSVRecordLookupService
- CSVRecordSetWriter
- DatabaseRecordLookupService
- DatabaseRecordSink
- DatabaseTableSchemaRegistry
- DBCPConnectionPool
- DBCPConnectionPoolLookup
- DeveloperBoxClientService
- DistributedMapCacheLookupService
- EBCDICRecordReader
- ElasticSearchClientServiceImpl
- ElasticSearchLookupService
- ElasticSearchStringLookupService
- EmailRecordSink
- EmbeddedHazelcastCacheManager
- ExcelReader
- ExternalHazelcastCacheManager
- FreeFormTextRecordSetWriter
- GCPCredentialsControllerService
- GCSFileResourceService
- GenericPLC4XConnectionPool
- GrokReader
- HadoopCatalogService
- HadoopDBCPConnectionPool
- HazelcastMapCacheClient
- HBase_2_ClientMapCacheService
- HBase_2_ClientService
- HBase_2_RecordLookupService
- HikariCPConnectionPool
- HiveCatalogService
- HttpRecordSink
- ImpalaConnectionPool
- IPFIXReader
- IPLookupService
- JASN1Reader
- JdbcCatalogService
- JettyWebSocketClient
- JettyWebSocketServer
- JiraRecordSink
- JMSConnectionFactoryProvider
- JndiJmsConnectionFactoryProvider
- JsonConfigBasedBoxClientService
- JsonPathReader
- JsonRecordSetWriter
- JsonTreeReader
- Kafka3ConnectionService
- KafkaRecordSink_2_6
- KerberosKeytabUserService
- KerberosPasswordUserService
- KerberosTicketCacheUserService
- KuduLookupService
- LivySessionController
- LoggingRecordSink
- MapCacheClientService
- MapCacheServer
- MongoDBControllerService
- MongoDBLookupService
- Neo4JCypherClientService
- ParquetReader
- ParquetRecordSetWriter
- PEMEncodedSSLContextProvider
- PhoenixThickConnectionPool
- PhoenixThinConnectionPool
- PostgreSQLConnectionPool
- PropertiesFileLookupService
- ProtobufReader
- ProxyPLC4XConnectionPool
- RabbitMQJMSConnectionFactoryProvider
- ReaderLookup
- RecordSetWriterLookup
- RecordSinkServiceLookup
- RedisConnectionPoolService
- RedisDistributedMapCacheClientService
- RedshiftConnectionPool
- RESTCatalogService
- RestLookupService
- S3FileResourceService
- ScriptedLookupService
- ScriptedReader
- ScriptedRecordSetWriter
- ScriptedRecordSink
- SetCacheClientService
- SetCacheServer
- SimpleCsvFileLookupService
- SimpleDatabaseLookupService
- SimpleKeyValueLookupService
- SimpleRedisDistributedMapCacheClientService
- SimpleScriptedLookupService
- SiteToSiteReportingRecordSink
- SlackRecordSink
- SmbjClientProviderService
- SnowflakeComputingConnectionPool
- StandardAsanaClientProviderService
- StandardAzureCredentialsControllerService
- StandardDatabaseDialectService
- StandardDropboxCredentialService
- StandardFileResourceService
- StandardHashiCorpVaultClientService
- StandardHttpContextMap
- StandardJiraCredentialService
- StandardJsonSchemaRegistry
- StandardKustoIngestService
- StandardKustoQueryService
- StandardOauth2AccessTokenProvider
- StandardPGPPrivateKeyService
- StandardPGPPublicKeyService
- StandardPLC4XConnectionPool
- StandardPrivateKeyService
- StandardProxyConfigurationService
- StandardRestrictedSSLContextService
- StandardS3EncryptionService
- StandardSnowflakeIngestManagerProviderService
- StandardSSLContextService
- StandardWebClientServiceProvider
- Syslog5424Reader
- SyslogReader
- TinkerpopClientService
- UDPEventRecordSink
- VolatileSchemaCache
- WindowsEventLogReader
- XMLFileLookupService
- XMLReader
- XMLRecordSetWriter
- YamlTreeReader
- ZendeskRecordSink
-
Reporting Tasks
- AzureLogAnalyticsProvenanceReportingTask
- AzureLogAnalyticsReportingTask
- ControllerStatusReportingTask
- MonitorDiskUsage
- MonitorMemory
- QueryNiFiReportingTask
- ReportLineageToAtlas
- ScriptedReportingTask
- SiteToSiteBulletinReportingTask
- SiteToSiteMetricsReportingTask
- SiteToSiteProvenanceReportingTask
- SiteToSiteStatusReportingTask
-
Parameter Providers
- AwsSecretsManagerParameterProvider
- AzureKeyVaultSecretsParameterProvider
- CyberArkConjurParameterProvider
- DatabaseParameterProvider
- EnvironmentVariableParameterProvider
- GcpSecretManagerParameterProvider
- HashiCorpVaultParameterProvider
- KubernetesSecretParameterProvider
- OnePasswordParameterProvider
- PropertiesFileParameterProvider
-
Flow Analysis Rules
- DisallowComponentType
- DisallowConsecutiveConnectionsWithRoundRobinLB
- DisallowDeadEnd
- DisallowDeprecatedProcessor
- DisallowExtractTextForFullContent
- RecommendRecordProcessor
- RequireHandleHttpResponseAfterHandleHttpRequest
- RequireMergeBeforePutIceberg
- RestrictBackpressureSettings
- RestrictComponentNaming
- RestrictConcurrentTasksVsThreadPoolSizeInProcessors
- RestrictFlowFileExpiration
- RestrictProcessorConcurrency
- RestrictSchedulingForListProcessors
- RestrictThreadPoolSize
- RestrictYieldDurationForConsumeKafkaProcessors
FetchPLC 2.3.0.4.10.0.0-147
- Bundle
- com.cloudera | nifi-cdf-plc4x-processors-nar
- Description
- Processor able to read data from industrial PLCs using Apache PLC4X
- Tags
- plc4x, read, source
- Input Requirement
- ALLOWED
- Supports Sensitive Dynamic Properties
- false
-
Additional Details for FetchPLC 2.3.0.4.10.0.0-147
FetchPLC
Purpose
This processor is supposed to read specific register values from a PLC device, and to generate NiFi records from these readings.
General Concept
A minimum setup requires an enabled PLC4X Connection Pool, a Record Writer, and any number of registers defined in the Address Map (dynamic properties). During execution, it compiles the Address Map into an Avro schema, and connects to the device, specified by the Connection Pool. Then reads the addresses from the device, and converts the results into a NiFi record, on success. Failing to read any of the specified registers count as an error, and in this case no record is generated, even if there is a partial result.
User can supply input FlowFile to this processor, by connecting it after another one. In this case, the attributes or the contents of the input can be referred in the Address Map, using Expression Language. Please note that, if Expression Language is present in the Address Map, the address syntax is not validated.
Address Map
Address Map is defined as Dynamic Properties in the processor settings. Validation fails, if there is none defined. The generated record consists of fields, and the mapping defines which field should contain which PLC register’s value. The name (key) defines the record field, the value defines the PLC register’s address. (Please refer to the Driver documentation, to find the required address format.) You can define any number of fields, all of these are fetched in a single pass.
Schema Cache
The processor needs to compile the Address Map into an Avro Schema, which uses noticeable amount of the system resources for a short duration (a spike). Normally this is done only once, when the processor starts up.
With Expression Language, you can make the Address Map dynamic, that can require multiple schema versions to be generated, during execution. The Schema Cache stores these compilations in an associative FiFo array, so that the schemas are generated only once, if there are no more variations than the specified cache size.
Input
Providing input is not necessary. In case you do, the processor will only read data, when an input FlowFile is available, and you can refer to the attributes of this file, when specifying the Address Map.
If input is not provided, the processor reads data as per Run Schedule.
Output
In case of a successful reading, the emitted FlowFile contain a single NiFi record, written by the specified Record Writer. In case of failure, there is no output, if the processor has no ingress connection.
The failure relationship is only used, if the processor is driven by an input FlowFile, and was unable to read all the specified registers. In this case, the input FlowFile is forwarded as-is.
Note: Partial read results are always dropped.
Type Mapping
Due to the limitations of Avro, some data types are translated into plain string value, and the correct type information is not carried over in the generated records. You can find the exact mapping below:
Mapping of PLC4X data types to Avro field types
PLC Type Name Numeric Code (HEX) Avro Type NULL
0x00
stringType
BOOL
0x01
booleanType
BYTE
0x02
intType
WORD
0x03
intType
DWORD
0x04
longType
LWORD
0x05
stringType
USINT
0x11
intType
UINT
0x12
intType
UDINT
0x13
longType
ULINT
0x14
stringType
SINT
0x21
intType
INT
0x22
intType
DINT
0x23
intType
LINT
0x24
longType
REAL
0x31
floatType
LREAL
0x32
doubleType
CHAR
0x41
stringType
WCHAR
0x42
stringType
STRING
0x43
stringType
WSTRING
0x44
stringType
TIME
0x51
stringType
LTIME
0x52
stringType
DATE
0x53
stringType
LDATE
0x54
stringType
TIME_OF_DAY
0x55
stringType
LTIME_OF_DAY
0x56
stringType
DATE_AND_TIME
0x57
stringType
LDATE_AND_TIME
0x58
stringType
Struct
0x61
stringType
List
0x62
array (of first element type)
RAW_BYTE_ARRAY
0x71
bytesType
PLC4X Documentation
PLC4X Support
-
Connection Pool Service
The Controller Service that is used to obtain connection to the PLC device.
- Display Name
- Connection Pool Service
- Description
- The Controller Service that is used to obtain connection to the PLC device.
- API Name
- connection-pool-service
- Service Interface
- com.cloudera.nifi.plc.api.PLC4XConnectionPool
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Record Writer
Specifies the Controller Service to use for writing results to a FlowFile. The Record Writer may use Inherit Schema to emulate the inferred schema behavior, i.e. an explicit schema need not be defined in the writer, and will be supplied by the same logic used to infer the schema from the column types.
- Display Name
- Record Writer
- Description
- Specifies the Controller Service to use for writing results to a FlowFile. The Record Writer may use Inherit Schema to emulate the inferred schema behavior, i.e. an explicit schema need not be defined in the writer, and will be supplied by the same logic used to infer the schema from the column types.
- API Name
- record-writer
- Service Interface
- org.apache.nifi.serialization.RecordSetWriterFactory
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Request Timeout
The maximum amount of time allowed for reading data from PLC. Fraction of milliseconds are rounded up.
- Display Name
- Request Timeout
- Description
- The maximum amount of time allowed for reading data from PLC. Fraction of milliseconds are rounded up.
- API Name
- request-timeout
- Default Value
- 10 seconds
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Schema Cache Size
Maximum number of entries in the cache. Can improve performance when addresses change dynamically.
- Display Name
- Schema Cache Size
- Description
- Maximum number of entries in the cache. Can improve performance when addresses change dynamically.
- API Name
- schema-cache-size
- Default Value
- 1
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Record field name
Address Map for linking record fields to PLC registers.
- Name
- Record field name
- Description
- Address Map for linking record fields to PLC registers.
- Value
- Address of the register to be read from
- Expression Language Scope
- FLOWFILE_ATTRIBUTES
Name | Description |
---|---|
success | Successfully processed |
failure | An error occurred while processing |
Name | Description |
---|---|
plc4x.read.row.count | Number of rows written into the output FlowFile |