The Workflow Coordination Spectrum

What does it mean to coordinate workflows?

As data practitioners, we have many ways of referring to the things we build. We may call them “jobs”, or “pipelines”, or “DAGs”, or maybe even “streams”. While each term implies a different implementation, to Prefect, they’re all workflows. A workflow is any recurring, usually data-intensive, process. Different workflows have related, but variable, needs. You may know Prefect as the best workflow orchestrator around, but not every workflow needs to be orchestrated.

Just as workflow is a broad concept for any way of moving or processing data, coordination is an equally broad concept for any way of ensuring a workflow behaves as expected. You can think of coordination as a spectrum from observability to orchestration.

Observability

Observability is the ability to see the status of a workflow. It’s passive. Through observability, you can get a sense of the state of a workflow, but can’t manipulate that state. A fully observable workflow has telemetry at every level: the infrastructure it’s running on, the code it’s executing, the data it’s handling, and the systems it’s interfacing with. If you’re used to traditional orchestration, observability alone may sound inferior, but observability alone has several benefits:

• Lightweight - Unlike Schrodinger's cat, you can reap the benefits of being able to observe almost any workflow without impacting its state or performance.
• Flexibility - Observability should sit “outside” of your workflow implementation. Often, you can change that implementation without any consideration for the observability layer.
• Speed - You can add observability to a workflow very quickly. With Prefect, it takes just a few lines of code.

Observability typically implies that there’s an observer - a person, probably you, that reviews the information at some point. When the observer reviews the information, they can see if something isn’t behaving as expected and intervene manually to address it. Often, that’s enough - especially for newer, less critical workflows. If and when manual intervention is too time-consuming or slow, you can progress along the coordination spectrum toward orchestration as the needs arise.

Orchestration

Orchestration is the automated control of the execution of a workflow. It’s active. Through orchestration, you can automate part of your role as the observer. The orchestrator reviews the observability information as it’s generated and makes decisions according to pre-specified rules. These rules specify the constraints for at what time and under which conditions specific code can run. A fully orchestrated workflow requires that every unit of code, down to the function, receive explicit permission to run. While orchestration demands deeper integration with your workflows, it has several benefits:

• Configurability - Orchestration enables you to manipulate your workflow’s execution through configuration, without manipulating its implementation directly.
• Robustness - Orchestration can prevent your workflow from entering known failure modes, and help it recover from unexpected failures.
• Reactiveness - Orchestration makes your workflow execution responsive to external, contextual factors that may be out of your control.

The more “mission critical” a workflow is, the more likely it is to benefit from orchestration. A fully orchestrated workflow should be portable, highly configurable, extremely robust, context aware, and observable in high fidelity.

There is no global optimum

There is no ideal point on the coordination spectrum for every workflow at all times. Different workflows have different needs that call for different approaches. Whatever you call your workflows and however you implement them, Prefect can meet you where you are and grow with you as your workflows’ needs change. Prefect enables you to build to the optimal point on the coordination spectrum for each flow, while still managing them through a single pane of glass - the Prefect control plane.