You can use the design component of Workflow Manager to create and graphically lay out your workflow. You can select action nodes and control nodes and implement them in a defined sequence to ingest, transform, and analyze data. Action nodes trigger a computation or processing task. Control nodes control the workflow execution path. You can add Oozie actions and other operations as action nodes on the workflow, and you can add fork and decision points as control nodes to your workflow. You can include a maximum of 400 nodes in a workflow.

From the designer, you can also schedule workflows using coordinators, and group multiple coordinators into bundles. A coordinator enables you to automate the recurring execution of a workflow based on time, an external event, or the availability of specific data files. The coordinator can define parameters only around a single workflow. Coordinators make it easy to manage concurrency and keep track of the completion of each workflow instance. The coordinator controls how many instances of a workflow can execute in parallel, as well as controlling how many instances are brought into a waiting state before the next workflow instance can run.

Coordinator jobs can be combined to create a chain of jobs that can be managed as a single entity. This logical chaining together of multiple, interdependent coordinator jobs is referred to as a bundle, also called a data application pipeline. By using bundled coordinator jobs, the output of one workflow that runs at one frequency can be used as the initial input for another job that runs at a different frequency. You can even bundle the same workflow with different coordinator schedules. To create a bundle, you must configure one or more workflows as coordinator jobs, then configure the coordinator jobs as part of a bundle.

​

Figure 1.1. Relationship of Workflows, Coordinators, and Bundles

More Information

You can read further details about workflows, coordinators, and bundles in the following Apache documentation:

The dashboard provides monitoring capabilities for jobs submitted to Workflow Manager. There are three types of jobs in Workflow Manager: Workflow jobs, coordinator jobs, and bundle jobs. Each job type is viewed in a separate table in the dashboard.

You can perform actions such as run, kill, suspend, and resume on individual jobs, or you can select multiple jobs of a single type and perform these actions on several jobs at once.

From the dashboard, you can access details about a job, view job logs, and view the XML structure of the job.

Workflow jobs are launched from a client and run on the server engine. The Oozie metadata database contains the workflow definitions, variables, state information, and other data required to run the workflow jobs.

Each action in a workflow represents a job to be run. For example, a Hive action runs a Hive job. Most actions in a workflow are executed asynchronously, so they have to wait until the previous action completes. However, file system operations on HDFS are executed synchronously.

Identical workflows can be run concurrently when properly parameterized. The workflow engine can detect completion of computation and processing actions. Each action is given a unique callback URL. The action invokes the URL when its task is complete. The workflow engine also has a mechanism to poll actions for completion, for cases in which the action fails to invoke the callback URL or the type of task cannot invoke the callback URL upon completion.

Workflow action types

More Information

For more information about actions, see the following Apache documentation: