1from prefect import flow, task
2from prefect_dbt.cloud import DbtCloudCredentials, DbtCloudJob
3
4@task
5def load_data():
6 # Your existing data loading code
7 pass
8
9@flow
10def analytics_pipeline():
11 # Load raw data
12 raw_data = load_data()
13
14 # Transform with dbt
15 dbt_job = DbtCloudJob(
16 dbt_cloud_credentials=DbtCloudCredentials.load("dbt-creds"),
17 job_id="daily-transformations"
18 )
19 dbt_job.run()Why data engineers choose Prefect
Transform brittle ETL jobs into resilient data pipelines. Integrate seamlessly with dbt while ensuring data quality and timely insights delivery.
Accelerate time to production
Deploy and refresh analytics pipelines quickly with self-service capabilities that minimize maintenance overhead.
Maintain data quality
Automate data quality checks and dependency management across data pipelines with custom alerts and comprehensive failure notifications for end-to-end observability.
Build trust in your data
Monitor analytics pipelines comprehensively with automated recovery, clear audit trails, and SLA tracking.
Native integrations
Connect to the whole analytics stack seamlessly across dbt, data warehouses, and BI tools to streamline ETL workflows.
1from prefect import flow
2from prefect_dbt.cli.tasks import from prefect import flow
3from prefect_dbt.cli.commands import trigger_dbt_cli_command, dbt_build_task
4
5
6@flow
7def dbt_build_flow():
8 trigger_dbt_cli_command(
9 command="dbt deps", project_dir="/Users/test/my_dbt_project_dir",
10 )
11 dbt_build_task(
12 project_dir = "/Users/test/my_dbt_project_dir",
13 create_summary_artifact = True,
14 summary_artifact_key = "dbt-build-task-summary",
15 extra_command_args=["--select", "foo_model"]
16 )Team enablement
Scale across the whole team securely with collaborative debugging and fine-grained object-level access control (RBAC & SCIM).
Hear from data engineering teams
Prefect is like an air traffic controller for your data. It supports Amazon, Databricks, and Adobe, which I use daily. The availability of prebuilt flows and infrastructure blocks like Git, Docker, and ECS not only saves time but also makes it easier to deploy containers locally. - G2 Crowd
We orchestrate our ETL and analytics workflows with Prefect. The UI gives us real-time visibility into data processing, ensuring that every task is executed correctly. This has significantly improved our data pipeline reliability and debugging process. - G2 Crowd
Prefect has given us visibility into our automated processes. We can now see, at a glance, the health of our entire ecosystem in a dashboard. With Prefect Cloud, we can monitor and troubleshoot issues in real-time from anywhere without connecting to a VPN. - G2 Crowd
Prefect stays as close to python best practices as is possible, which makes it easy to get started and to experiment. Prefect's hybrid nature allows for easy integrations and secure data handling. - G2 Crowd
We run all of our data pipelines and analytics model workflows through Prefect. It makes orchestrating our hundreds of pipelines easy. With Prefect, I spend less time troubleshooting and more time focusing on business insights. - G2 Crowd
With Prefect, we’ve handed over the scheduling of Python script execution to non-technical business users, enabling them to manage their analytics workflows autonomously. Our administrators can easily spot errors and fix them before they impact business operations. - G2 Crowd
Prefect makes scheduling, executing, and monitoring data pipelines incredibly easy and low-maintenance. The hybrid model allows for much flexibility in scaling execution infrastructure, allowing economical use of resources. - G2 Crowd
At first, we just got Prefect because we were running simple jobs on Linux machines with cron. Our team thought this wasn’t best practice. We were basically using it as a glorified Linux box. However, we recently figured out that the orchestration capabilities of Prefect would be great for managing database actions. - G2 Crowd
Before Prefect, we had long-running analytics tasks that would sometimes stop running for hours, and we had no visibility into the issue. Prefect eliminated this problem and allowed us to automate new use cases with ease. - G2 Crowd