Prefect 2.0

Today, our team is very pleased to announce the public release of Prefect 2.0 and Prefect Cloud 2.0, along with our expanded mission to coordinate the world’s dataflow. Prefect 2.0 is the foundation of an entirely new product suite, designed to enhance operations and visibility across the entire data stack, that we call the coordination plane.

Prefect software has always been designed to be simple but powerful, and with 2.0 we’ve really taken that to another level. Our philosophy of incremental adoption means that you never have to use or learn more about our tools than you need; in fact, adding a single @flow decorator to your python script is sufficient to drive many of our headline features. Similarly, our belief in “code as workflows” means that you won’t have to rewrite all your code to take advantage of even our most advanced orchestration functionality. We’ve reimagined the entire deployment experience, built on top of our patented hybrid execution model, so that the whole process of packaging dependencies and executing a flow from remote storage takes less time than reading this paragraph. Blocks, our brand-new approach to configuration management, make it easy to set up and configure business logic without leaving Prefect 2.0’s redesigned UI.

And that’s just what you can do today. One of the most important things about Prefect 2.0 is that it’s been designed for a highly dynamic and rapidly changing world. You may have noticed hints of this in Orion, our second-generation workflow engine. In the near future, we’ll release new features that extend Prefect far beyond the workflow orchestration that was the bread and butter of our last-generation product. When you set up your applications in Prefect 2.0, it will be capable of automatically surfacing activity and events from across your entire stack, even for code and systems that Prefect isn’t orchestrating itself. For the first time, you’ll be able to visualize your entire data stack, express and evaluate expectations about its behavior, and respond in real-time.

Thanks to a new observability API, Prefect 2.0 will be ready to trigger actions and workflows in response to dataflow activity from across your stack, without requiring orchestration (or code!) at all. This approach is so powerful and flexible that Prefect Cloud’s own audit log feature is built on top of it.

Our mascot is a blue rubber duck named Marvin. You may be familiar with the practice of "rubber ducking," in which engineers keep a toy duck on their desks to help them work through their hardest problems. Similarly, we hope that Prefect earns a permanent role as our users' constant partner. As one of our favorite books says: "One's never alone with a rubber duck."

Get started with Prefect 2.0 today by checking out the docs or joining the community!

The Coordination Plane

The Cambrian explosion of applications in the modern data stack has become meme-worthy. But it would be a mistake to confuse choice with complexity. Obviously, no company uses more than a handful of the seemingly infinite options that the MDS offers, but this rich menu has enabled their data stacks to become far more streamlined and purpose-built than ever before. While a few core components have emerged as household names, every company enhances them with a heterogeneous combination of applications that best achieves their unique business objectives. What makes the modern data stack “modern” is that each application presents an API that allows it to be added to the stack in a flexible manner.

The result is a stack as unique as the company that built it, each one far more than just the sum of its logos. The way in which applications interact and exchange data forms a dynamic chain of activity that we call dataflow. Ultimately, dataflow is what binds an otherwise disparate set of applications together. In this way, dataflow is a key characteristic of modern data architectures, as composing the same applications in different ways would give rise to very different outcomes. For example, consider “ETL” and “reverse ETL,” which represent wildly different business objectives (and users!) despite involving the same nominal stack. The major observable difference is the direction of dataflow. In a post-hoc, sanitized environment, this idea might be captured in something like data lineage. But data lineage captures information about static relationships: where data started and where it ended. It doesn’t reflect the potentially messy and error-prone processes that actually funnel data through the stack.

Therefore, the unique challenge of working with modern data is coordinating and monitoring the motion of that data through the entire stack. Each tool exposes powerful APIs for its own domain, and data lineage might help us understand which tools collaborated to produce a specific output, but we need a universal window into dataflow as it’s happening. And yet, there is no modern API for monitoring dataflow across the entire stack. Ironically, because of the explosive diversity of modern data applications, there’s no clear way to ask what feels like a simple question: “What’s happening in my data stack right now?”

A few years ago, Prefect launched a new category of dataflow automation tools with the introduction of the first dataflow-native orchestrator, Prefect 1.0. An orchestrator is a tool that runs workflows by explicitly controlling their execution at each step. By writing workflow code that talked to multiple applications or APIs, Prefect users could start to visualize that dataflow for the first time. Unsurprisingly, this proved very popular, and people started eagerly producing code that gave the orchestrator as much control of their dataflow as possible. To such users, it might seem like the orchestrator is a natural place — and perhaps the only place — to solve the problem of global dataflow visibility.

But it’s unreasonable to presume that a single orchestration plane will ever be able to control dataflow across an entire stack. This would require us to believe that an orchestrator can become the sole access point and single source of truth for all dataflow or, equivalently, that there are teams of engineers eager to write code that translates the entire data stack back into a form the orchestrator can consume. Not only would this be an extraordinary waste of time, but this forced re-bundling of every data application back into the orchestrator would be profoundly un-modern.

It turns out that the solution to this problem isn’t to redefine and force all dataflow to pass through the orchestrator, but rather to enable a more passive mode of collecting information, one that lets the software observe dataflow as it moves through the stack without controlling it. By working together with other applications and leveraging their APIs, we can quickly form a complete picture of everything that’s happening and guide dataflow to successful outcomes. To achieve this, we introduce the idea of a coordination plane that encompasses a full spectrum of activity from orchestration to observation.

ATC for the MDS

To understand the coordination plane, consider one you’re already very familiar with: air traffic control (ATC). The primary function of ATC is to coordinate the movement of aircraft, and every year the global ATC network employs a combination of active orchestration and passive observation to ensure safe and efficient air travel for billions of passengers. This has immediate and obvious similarities to moving data through a modern stack.

• Tower controllers based at airports are in direct communication with planes on the ground and in close proximity to the airport. They ensure safe operations by instructing planes when and where to move. Nothing - including ground vehicles - can travel through the airport without their awareness and permission. This represents the orchestration end of the spectrum.
• En-route controllers guide aircraft between airports. They ensure safe travel by closely monitoring traffic, but are unlikely to issue specific instructions or overrule a pilot unless they become aware of a significant bottleneck. This is most similar to observability.
• Approach and departure controllers handle air traffic at medium-to-short distances from the airport. They may not always issue explicit instructions to aircraft, but they will post advisories and generally guide pilots at medium range to or from the airport. As circumstances including weather or traffic change, they will take action to ensure that airport operations are minimally disrupted. This represents a blend of orchestration and observation that is arguably the most interesting opportunity for a true coordination plane: translating an observation about the world into a cleanly orchestrated response.

As a passenger, you may not be aware of just how much effort goes into ensuring the safety and efficiency of your journey. At a minimum, you may be aware that your plane is in contact with the control tower to coordinate its own takeoff or landing. But you might not realize how much of your journey is affected by ATC’s knowledge of other aircraft and factors. For example: traffic jams, bad weather causing delays, a plane making an emergency landing, or a runway under construction could all impact your journey even though they have no first-order connection to your plane. This is because all air traffic shares a limited set of resources, and it’s only by tracking all activity that we can understand how any specific plane is affected.

Despite the clear metaphor, there is no ATC for the modern data stack. Our systems operate like planes with no way to coordinate, loosely hoping that they'll get their passengers to the right airport in time to make their connections. True, most applications deliver their own set of metrics, schedulers, and APIs - which are generally excellent, but only within the confines of that application. To understand the whole, we have to choose between tools that inundate us with in-flight metrics (at the data infrastructure level), and others that tell us, after-the-fact, whether every passenger got to their destination (such as data lineage). But what helps us ensure all planes take off on time? Interface with pilots in need of emergency assistance? Identify delays that will affect future flights? Measure airport activity to plan future expansion? We need a coordination plane that can monitor all activity and let us take action when needed. In short: our stacks need a control tower.

Prefect 2.0 is the foundation for executing this vision. It includes Orion, our powerful, second-generation orchestration engine. Orion itself is a modern data application, delivered as an API and designed for maximum flexibility. But though orchestration was historically our entire product, it is now just one form of dataflow coordination that Prefect 2.0 offers. We are also introducing the building blocks of a new observability API that can be used to automatically connect, observe, and react to dataflow, even for applications not orchestrated by Prefect. Note that neither of these is sufficient on its own: pure orchestration requires more control than some applications can give (or some engineers care to write); pure observation leaves us unable to react or coordinate good outcomes. By unifying orchestration and observation in a single platform, Prefect 2.0 will deliver the modern data stack’s first coordination plane: a single layer that enables you to observe, alert, orchestrate, and respond to dataflows between and across your applications.

New and Improved

Prefect 2.0 is the foundation of the new coordination plane that we’re building, and, as a ground-up rewrite, provides an extraordinary number of new and exciting features that reflect years of experience and customer feedback. Since entering beta early this year, Prefect 2.0 usage has grown by nearly 10% every week. To meet this extraordinary demand, we haven’t only focused on building product. We have fantastic new docs that are perfect for new and experienced users alike. Our sales, success, and community engineering teams are extremely knowledgeable and ready to assist. Most of our partners — and more than a few of our customers! — are producing native Prefect 2.0 integrations. We’ve also published a helpful guide to upgrading for Prefect 1.0 users.

Rather than produce an exhaustive list of what Prefect 2.0 can do, we’ll focus on four key enhancements. To learn more about the product, please visit the docs, the product page, or join us (and 20,000 of our closest friends) in Slack.

Code as Workflows: The Python API

Prefect embraces a product philosophy of eliminating negative engineering. While most orchestrators require users to rewrite their workflows to comply with the orchestrator’s own semantics, Prefect attempts to be minimally invasive. We’ve embraced “code as workflows,” acknowledging that whenever an engineer writes code, that code is already the best possible representation of their workflow objectives. It necessarily describes ordering and dependencies, parameterization, data passing, control flow, and conditions. Any further modifications required by the orchestrator are a tax on time and energy. Therefore, Prefect strives to achieve any goal with as few changes as possible.

That’s why, to turn any function into a fully-observable Prefect flow all you have to do is add a @flow decorator to it. You will automatically gain access to error management and retries, and can see logs and state changes in the UI - no matter when or how the function was run (if you like your cron, you can keep it!). If you want more granular detail, add @task decorators to nested functions to gain orchestration and capture them in the UI as well. Each of these functions still behaves like a function: you call it to run it and you can examine its result. You can even use breakpoints to debug it! When you’re ready to expose your flow as a schedule-able and configurable API, build a deployment and let Prefect take care of of the rest.

Building Blocks: The Configuration API

Blocks are a powerful new way to configure and share business logic. We like them so much that many other Prefect 2.0 features are built directly on top of their API, including flow and result storage, notifications, deployments, and infrastructure configuration.

Historically, one of the most difficult things about workflow management was handling configuration. We’ve seen Prefect users jump through many hoops to make context values, API keys, and other important information available to their workflows and infrastructure. Worst of all, changing this configuration almost always requires redeploying the flow or rebuilding its environment.

Blocks make configuration management easy by moving it to the UI, where each block exposes a structured, validated schema, thanks to OpenAPI support. Once a block is created, users can load it anywhere in their code by name. For example, S3.load("dev") might load a block with your “dev” configuration, while S3.load("prod") would retrieve your production setup. This means that swapping configuration at runtime is as easy as specifying a different block name!

But blocks do much more than just load configuration. Most are enhanced with business logic, meaning our S3 blocks are immediately ready to .read() or .write() data. Since blocks implement interfaces, you can swap an S3 block for any other filesystem, such as your local machine, GCS, or Azure. Other block types include notifications like Slack or email; generic containers like JSON and Secret for holding arbitrary - and possibly sensitive - information; and infrastructure configuration like Kubernetes job manifests and Docker containers. Want to share templated SQL with your team? There’s a block for that.

Our goal is to create blocks for every part of the data ecosystem, providing a user experience similar to “Plaid for the modern data stack:” choose your applications or needs from a large menu, enter any credentials or configuration, and immediately take advantage of new functionality. Teams can also write custom blocks for proprietary use cases. These custom blocks can still be created from the UI (even in Prefect Cloud!) and used in flows, exactly the same as “official” ones.

Lastly, blocks introduce a much-needed separation of concerns. Suppose a workflow author loads data with an S3 block that in turn loads its credentials from an AWS Credentials block. In this setup, a DevOps engineer could independently rotate the credentials, which would be picked up by all other blocks. Meanwhile, a data engineer could change the S3 bucket and know that every flow using the block will load the new configuration. Despite all that, the workflow author wouldn’t have to change a thing, while Prefect’s audit log could help the team lead track changes and usage.

Ship It: The Deployments API

In Prefect 2.0, you can run flows interactively against the API without preregistering, packaging, or deploying them at all (incremental adoption FTW!). This makes testing and debugging identical to your normal Python experience. When it’s time to hand your flow over to the Prefect API for remote scheduling and execution within your environment of choice, you’ll create a deployment. Based on extensive feedback from Prefect 1.0 and throughout the Prefect 2.0 beta, we’ve significantly streamlined the deployment experience while somehow making it way more powerful.

First, you build the deployment by specifying where the code is and how to execute it. This is done with a single CLI command that uses a storage block(local file, S3, etc.) for the flow’s location and an infrastructure block (subprocess, Docker, K8s, etc.) for execution. Prefect will automatically use the storage block to upload your project into a portable structure that can be run within any compatible Python environment (no more “module not found” errors for your relative imports!).

The output of building a deployment is a descriptive YAML file that you can send to the server at any time to create or update the deployment: prefect deployment apply. This offers a lightning-fast CLI workflow and also transforms the experience of working with Prefect in CI. In addition, separating the build step from the creation step means that users can add workflows to their workspaces entirely from the UI! By pointing at a known storage location and choosing a compatible infrastructure type, new deployments can be created or configured without ever visiting the command line.

1# build a deployment in my team's dev S3 bucket
2prefect deployment build ./flows.py:marvin \
3  --storage-block s3/dev \
4  --name "Marvin's First Deployment" \
5  --work-queue duck
6
7
8# send the deployment to the API
9prefect deployment apply ./deployment.yaml

view rawdeployment.sh hosted with ❤ by GitHub

Now you can start an agent on any computer and run the deployed flow:

1# start an agent and work queue to run "duck"-tagged deployments
2prefect agent start -q duck

view rawagent.sh hosted with ❤ by GitHub

Whenever you create or schedule a run from the UI, the agent will use the storage block to connect to S3, download the flow, and execute it.

We’re especially grateful to our partners at Intel, who inspired and directly sponsored many Prefect 2.0 features, including the new deployment experience.

Sens-O-Matic: The Observability API

Prefect 2.0’s dynamic engine and block-based configuration are enough to fully orchestrate any stack. But in order to deliver a complete coordination plane, we must also introduce a new observability API that lets us extend Prefect through the entire stack without needing to write orchestration code.

The observability API introduces the concepts of resources and events. Resources represent any objects or systems that interact with your dataflow, such as data warehouses, database tables, Kubernetes clusters, dbt models, S3 buckets, and even Prefect flows. Any resource can emit events, which Prefect also captures. Events can describe actions, operations, heartbeats, or even relationships.

The combination of resources and events gives Prefect a universal vocabulary for describing dataflow, whether orchestrated or only observed. This lets us do useful things like correlate Kubernetes cluster events to workflow failures; discover how recently database tables were updated; and collect heartbeats from critical services.

But just being able to visualize and measure dataflow isn’t enough. Earlier, we wrote that the most interesting opportunity for the coordination plane is neither pure orchestration nor observation, but a hybrid of the two. For this reason, we’re also introducing triggers. Triggers listen for specific event and resource criteria and create actions in response. For example, a trigger could send a notification, start a workflow, or even create a new event. Triggers can also operate on the absence of events, allowing users to set up dataflow SLAs that span multiple applications: SLAs can be created for workflow activity, database updates, heartbeats or any custom event.

Why is this important? Because one of the most important jobs of a coordination plane isn't just making something happen, but informing users when it doesn't happen. Consider a product that runs an important workflow at 9am. This product is certainly useful, but what would make it truly valuable is if it could also inform users that an expected 9am run never happened. The ability to express expectations and discover when they aren't met is a principle product objective of the coordination plane and a major way that it delivers confidence to users.

Best of all, Prefect will do most of the heavy lifting! For example, when a user adds a block to Prefect, the system can use that information to automatically discover related resources and events on an ongoing basis. We are also working with our partners to create native integrations for their most critical activity. For example, failing a quality check in Monte Carlo or Great Expectations could result in concurrency limits or disabled scheduling for Prefect workflows that hit the same database. SLAs on warehouse table updates could result in Slack notifications or new support tickets in Monday or Jira. A file landing in S3 could launch one of your orchestrated workflows. You could even implement Prefect 1.0's extremely popular "flow-of-flows" pattern right from the UI.

Between blocks and observability, Prefect can access and interact with the entire stack in a whole new way. In great part, these features are driven by the needs of the fastest growing segment of Prefect’s user base: data analysts. These users are often not workflow authors, but they frequently configure workflow runs and are almost always dependent on their output. Ensuring our product is maximally useful for them is critical to Prefect’s continued growth, and we are excited to explore this new direction with them.

Prefect’s first observability features will be released this quarter.

One's Never Alone with a Rubber Duck

Today’s release is the first of many for our second-generation products line. It represents an extraordinary effort by every member of the Prefect team and a large number of community members, as well. We are especially grateful to our Club42 ambassadors and the latest graduates from the Prefect Associate Certification course, all of whom provided helpful insight that guided many of our decisions.

With a team nearly four times larger than a year ago, a user community that shows no signs of slowing, and a company operating as efficiently as ever, we are working incredibly hard to deliver the best possible experience to our users. We’re privileged to work with incredible people and partners on a dazzling array of use cases, and appreciate the trust you show in us.

We hope that Prefect 2.0’s coordination plane quickly earns its place in your data stack and we’re excited to help your workflows take off.

After all, every airport needs a control tower.

Happy engineering!