Background Tasks: Why They Matter in Prefect

Web applications and data pipelines probably have more in common than you think.

A data engineer’s right of passage is struggling, at least once, with a scheduler that keeps running tasks twice at 9am instead of once. Why? Because all data engineers, in some capacity or another, write tasks that validate incoming data, transform it, and move it from one place to another. And they need to schedule those tasks, which is often harder than it should be.

But what if I told you that engineers building web applications and APIs struggle with the exact same thing? Unlike data engineers, however, web engineers often lack access to the powerful orchestration tools that make scheduling and managing tasks easier.

Tasks for the non-data software engineer

It’s me, Andrew Brookins. I’m the author, and a Lead Staff Software Engineer at Prefect. I’ve never held the title “data engineer,” and before working here, I hadn’t heard of orchestration as a product category. I’m a software engineer who builds web applications: full-stack, backend, from a different world, another planet.

And yet. I’ve earned the data engineer’s right of passage because I, too, have spent years building tasks—background tasks—and running them on various schedulers. But before going deeper, we need to dive into what a “background task” is.

What is a background task?

Any application with a user interface uses some form of background task.

📖 Background task: Code designed to run in a background computer process, without blocking the main program and without user intervention.

This is because user-facing applications have a foreground where all the action happens—a web page, mobile app, or desktop window.

Users are constantly clicking, typing, and swiping, and the application has to keep up with those actions quickly because users perceive delays with the deepest loathing. So any hard work an application does? That happens in the background. In the shadows, in deepest darkness.

Software engineers need better background task tools

Data engineers rely on tools like Prefect and Airflow to orchestrate complex workflows, offering features like observability, detailed UIs, and robust error handling. These tools make managing dependencies, failures, and retries straightforward.

But when it comes to background tasks in web apps, tools like Celery and RQ fall short. They’re difficult to operate, provide limited visibility into task failures, and lack the comprehensive UIs that make debugging and managing workflows easier.

If data engineers get these features for task orchestration, shouldn’t web engineers also have access to the same capabilities for background tasks?Prefect fills this gap, bringing those same powerful features to web applications, making it easier to manage and observe background tasks with confidence.

Prefect 3.0 is the best way to schedule, observe, and run functions in the background

Prefect 3.0 introduces support for background tasks, so you can build modern web applications alongside your data pipelines and AI workflows. As an engineer who came to Prefect because I thought it was already 10x better two years ago than any other background task tools, I want to tell you why I’m even more excited about it.

Compared to Celery and RQ, Prefect offers an alternative designed to make background tasks easy to use while scaling up to complex workflows, with a rich UI and thoughtful features that help you manage failure proactively.

Want to see some code? Check out the example FastAPI and Flask apps in our GitHub examples repo or read our Background Tasks Docs.

Tasks, as far as the eye can see

With Prefect 3.0, you can compose data pipelines and AI workflows with tasks, just like you always could. But we know web apps need the same background task features that make pipelines resilient. That’s why in Prefect 3.0, we redesigned Prefect tasks to support the unique constraints of web apps:

• The main reason web apps use background tasks is to keep HTTP request times short, so triggering a task should have minimal overhead.
• Web apps don’t usually need task results immediately, but might need them later, so applications should be able to look up results by task ID.
• Users want to know whether a task is in progress or completed, so web apps need to be able to report the status of a task.

To help with these and other constraints particular to background tasks, we’ve added delay() and apply_async() methods to @task functions, so you can trigger tasks to run on any infrastructure, using a task worker.

Built-in Observability

Prefect gives you real-time insights into task statuses, detailed logs, and failure info. Even better, you can set up notifications and automations that react to status changes immediately.

Handle Failure Proactively

Prefect’s proactive approach to handling failure extends even farther. You can configure task caching, transactional behavior, custom retry logic, and failure hooks to ensure your system is well-prepared. Check out our task docs to see what’s possible.

Support for Complex Requirements

Prefect scales with the complexity of your tasks, not just their volume. When you need to design fan-out/fan-in architectures, add rate-limiting, or manage task dependencies, Prefect gives you powerful tools to get the job done. Unlike many task frameworks, Prefect was designed for the hard parts first, from enterprise-grade scheduling to human-in-the-loop interaction.

Try it out for yourself

Using Prefect in your web application—or anywhere else—is straightforward. You add a decorator that makes your functio a task, call the delay() or apply_async() methods on the task, and run a task worker that executes task runs. If you use Celery, you might not have to change much. Our docs and examples on GitHub make it easy to get up and running quickly. Plus, the Prefect community is always there to help if you run into any issues.

Ready to take Prefect for a spin? Dive into our Background Tasks Docs and explore example applications on our GitHub repository to get started. Join our community and see how Prefect can streamline your workflows today!