Reducing the Workflow Hangover in 2023

No matter how amazing a job is, everyone has to hit a mental "hard reset" to log back in the morning of January 2nd. And what might you find when you do this?

A slew of errors coming from your pipelines in Slack. And so it begins: curing the workflow hangover.

Here's a recap of what we added to Prefect in 2023 that can make you more efficient.

Understand exactly what's failing

In 2023, you get a more efficient way to triage workflows.

When you see the dreaded red alarm Slack alert, there's only one question to answer: what happened and why? You need context - enter the Prefect dashboard.

You're most desperate for information from Prefect when execs are pounding on your door about stale data. What you need at your finger tips:

• Which workflows are failing
• Summarized errors (even better with AI) so you know exactly what to fix
• Infrastructure state - if your work pools are down, nothing else matters
• An easy way to look at past errors to ensure you're meeting data SLAs

Even if it's just one task in a flow failing, all downstream tasks will be off. Getting to the bottom of who's affected by an error means a clear view of dependencies that adapt per run as your data changes.

What about infrastructure?

It's easier to get something right the first time - particularly when productionizing - than to do it over again.

In 2023, we really took this mantra to heart by provisioning infrastructure.

After you write code, you need to quickly:

• Provision custom infrastructure, like an ECS cluster, with as little config as possible
• Deploy your code to this infrastructure, like with the new .deploy() method on a flow function
• Make your infrastructure connect to your code remotely, not on your machine

A question remains, to operate lean: how can you spin up infrastructure only when it's needed?

The minimum viable schedule

If there's no data to update, you don't need to run your workflow. Your finance team doesn't need another reason to be on your back about your cloud bill.

Sometimes the minimum viable schedule is no schedule at all.

In 2023, we brought you: events, automations, and webhooks.

Run a workflow only when something actually happens. An event driven pipeline:

• Shouldn't be a black box like a cloud function - alert on lack of executions, handle dependencies, etc.
• Integrated with your other work so you can debug and track dependencies with context

And those were only the highlights of what Prefect shipped in 2023.

Let's get stoked for 2024.

Try Prefect Cloud for free for yourself, download our open source package, join our Slack community, or talk to one of our engineers to learn more.