Flink application structure

You must understand the parts of application structure to build and develop a Flink streaming application. To create and run the Flink application, you need to create the application logic using the DataStream API.

A Flink application consists of the following structural parts:
  • Creating the execution environment
  • Loading data to a source
  • Transforming the initial data
  • Writing the transformed data to a sink
  • Triggering the execution of the program

StreamExecutionEnvironment

StreamExecutionEnvironment env =
    StreamExecutionEnvironment.getExecutionEnvironment();
...

env.execute("My Flink job")

The getExecutionEnvironment() static call guarantees that the pipeline always uses the correct environment based on the location it is executed on. When running from the IDE, a local execution environment, and when running from the client for cluster submission, it returns the YARN execution environment. The rest of the main class defines the application sources, processing flow and the sinks followed by the execute() call. The execute call triggers the actual execution of the pipeline either locally or on the cluster. The StreamExecutionEnvironment class is needed to configure important job parameters for maintaining the behavior of the application and to create the DataStream.