Prefect or Airflow?
Instead of manually coding task dependencies and using complex sensors, write Python and let Prefect's event-based and dynamic architecture do the rest
Airflow code isn't DRY and requires significant boilerplate config. With @flow decorators on existing Python functions, Prefect structures your workflow order at runtime automatically.
1from prefect import flow
2from datetime import datetime
3
4@task
5def get_datetime():
6 return datetime.now()
7
8@flow
9def greetings():
10 now = get_datetime()
11 print(f"Hello, World! It's {now}")
12
13if __name__ == "__main__":
14 greetings()Prefect Cloud does not rely on clunky sensors—it is event-based at its core. This means flows can run as a response to internal or external webhook events, which are first-class observable objects within the Prefect dashboard.
1triggers:
2 - enabled: true
3 match:
4 prefect.resource.id: prefect.flow-run.*
5 expect:
6 - prefect.flow-run.Completed
7 match_related:
8 prefect.resource.name: prefect.flow.etl-flow
9 prefect.resource.role: flow
10 parameters:
11 param_1: "{{ event }}"Instead of managing a cluster of Airflow instances per team, customize infrastructure needs down to each workflow. Prefect allows you to operate lean—cut your infrastructure bill and development time when collaborating with devops.
1deployments:
2- name: deployment-1
3 entrypoint: flows/hello.py:my_flow
4 work_pool: *my_docker_work_pool
5 # Use default Docker build
6 build: *docker_build
7
8- name: deployment-2
9 entrypoint: flows/goodbye.py:my_other_flow
10 work_pool: *my_docker_work_pool
11 build:
12 - prefect_docker.deployments.steps.build_docker_image:
13 <<: *docker_build_config
14 # Override with custom Dockerfile
15 dockerfile: Dockerfile.customPrefect vs. Airflow
Hear from developers who chose Prefect over Airflow
It addresses many of the pain points common to more complicated tools like Airflow. Specifically, Prefect lets you turn any Python function into a task using a simple Python decorator.
I can go from thought to production 5x faster.
What would have taken 2-3 months to get running in Airflow took us only 1 month.
Our Data Engineering Platform used to be a liability. Now it’s a strength. Saving us days on DAG design vs. Airflow.
Airflow is heavier from an infra perspective - from an infra and identity perspective, it was much more bloated and inflexible.
We have been able stay on top of the data flows we've moved to Prefect easily. Seeing failures, successes, and outages in a timely and clear fashion has let us inform stakeholders what's up with the data flows.
