Execute Spark Code over a Livy-managed HTTP session to a live Spark context. Supports cached RDD sharing.
spark, livy, http, execute
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Livy Controller Service | exec-spark-iactive-livy-controller-service | Controller Service API: LivySessionService Implementation: LivySessionController | The controller service to use for Livy-managed session(s). | |
| Code | exec-spark-iactive-code | The code to execute in the session. This property can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain valid code to be issued by the processor to the session. Note that Expression Language is not evaluated for flow file contents. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | ||
| Character Set | exec-spark-iactive-charset | UTF-8 | The character set encoding for the incoming flow file. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | |
| Status Check Interval | exec-spark-iactive-status-check-interval | 1 sec | The amount of time to wait between checking the status of an operation. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Name | Description |
|---|---|
| success | FlowFiles that are successfully processed are sent to this relationship |
| wait | FlowFiles that are waiting on an available Spark session will be sent to this relationship |
| failure | FlowFiles are routed to this relationship when they cannot be parsed |