Modern Orchestration: Endpoint’s evolution from Airflow to Prefect

“Prefect’s support feels like a true collaboration, significantly increasing our team’s efficiency and satisfaction.” - Sunny

Identifying Airflow hurdles

Endpoint, a digital title and escrow company, develops technology that streamlines home closing for agents, buyers, and sellers. As Endpoint’s data engineering and ML workflows began to scale rapidly, Sunny Pachunuri, Endpoint’s Data Engineering and Platform Manager, began evaluating orchestration solutions. Initially, the organization chose Astronomer (built on Airflow) because it allowed for quick environment setups, enabling rapid development. It seemed promising at first, but as the team began onboarding teams and developing complex pipelines, they encountered several unforeseen challenges that quickly became insurmountable obstacles.

“Airflow’s interface wasn’t as user-friendly as we expected, and we often waited for essential features that didn’t meet our needs.” - Sunny

Complex and costly workflow retrofitting

Astronomer’s rigid Directed Acyclic Graph (DAG) structure didn’t work well with Endpoint’s existing Python code, requiring substantial modifications, which absorbed time and increased Machine Learning turnaround time by 3x.

High infrastructure overheads

To manage different compute types in Astronomer, Endpoint had to create and maintain multiple production environments, leading to increased complexity and costs.

Poor user experience

The team struggled with Astronomer’s user interface, which lacked the intuitive design and timely updates needed for smooth adoption. “The interface wasn’t as user-friendly as we expected, and we often waited for essential features that didn’t meet our needs,” Sunny said.

After doing their best to adapt to the many challenges Astronomer posed, Sunny began evaluating alternative solutions that could better meet the needs of his team. After exploring several alternatives, including Dagster and Mage, Endpoint selected Prefect, which stood out for its robust error handling and fault tolerance, effortless workflow deployment, advanced observability and monitoring, and Pythonic approach.

Upgrading to Prefect

After determining that Prefect was the best solution for their needs, Sunny emphasized the smooth implementation and migration process. In contrast to their previous experience with Astronomer —which required extensive code modifications to fit its Directed Acyclic Graph (DAG) and task structures, while also imposing significant deployment limitations such as Celery, Kubernetes, and EKS— Prefect’s lightweight and user-friendly setup allowed for a unified environment. This eliminated the need for retrofitting, facilitating seamless integration and enabling immediate productivity.

Prefect’s technical team collaborated with Endpoint to review their architecture and proposed design changes that would best leverage Prefect's capabilities. Once the new architecture was finalized, the migration was successfully completed in under two and a half months. This process involved setting up infrastructure using Terraform and configuring a unified environment. With just one Data Engineer and one ML Engineer, Endpoint was able to transition 58 Data Engineering pipelines and 14 ML pipelines, ultimately streamlining the team’s entire workflow.

“The feedback from the team was overwhelmingly positive [after migrating to Prefect],” Sunny emphasized. “Both the Data Engineering and ML Ops teams were impressed by the significant time saved and the elimination of retrofitting requirements, which had previously been a major bottleneck.” This streamlined migration process enabled the teams to collectively build an additional 78 pipelines within a single quarter, each tailored to unique compute resource requirements—an efficiency that would have been unattainable with Astronomer’s rigid code adaptation and deployment limitations.

“The Data Engineering and ML Ops teams were impressed by the significant time Prefect saved and the elimination of retrofitting requirements.” - Sunny

Unlocking efficiency and observability

Following the migration, Endpoint quickly began to appreciate the advantages of using Prefect over Astronomer.

Streamlined implementation

During a proof of concept (POC) involving two Data Engineering pipelines and four ML pipelines, the team found that Prefect's decorators, like @Flow and @Task, saved substantial time and improved efficiency.

Unified compute environment

Prefect’s compute model allowed Endpoint to operate a single environment for diverse workflows. “Our Machine Learning team could use Docker and Kubernetes while the Data Engineering team used Celery and EKS, all within one environment. It simplified our DataOps process significantly,” Sunny explained.

Predictable cost structure

Prefect’s fixed price per Prefect Cloud Workspace pricing structure enabled Endpoint to maintain multiple environments (DEV, STAGING, INTEGRATION, and PROD) without incurring extra costs. “Our expenses became predictable, eliminating surprises and enabling precise annual budget estimates,” Sunny noted. In addition to gaining this predictability, Endpoint was also able to cut orchestration costs by over 73% by switching to Prefect.

"Switching from Astronomer to Prefect resulted in a 73.78% reduction in invoice costs alone.” - Sunny

Enhanced observability and monitoring

With Prefect, Endpoint finally had comprehensive monitoring capabilities with Prefect to monitor job performance and get actionable alerts. Sunny added, “We also run DBT jobs within Prefect, centralizing all observability and monitoring.”

Building a lasting partnership

Since adopting Prefect nearly two years ago, Sunny and his team at Endpoint Closing have not only transformed their orchestration capabilities but have also benefited from exceptional support. “I suggested an enhancement for a Slack alert, and within three days, Prefect released the feature,” Sunny recounted. “Prefect’s support feels like a true collaboration, significantly increasing our team’s efficiency and satisfaction.”

By migrating to Prefect, Endpoint Closing has effectively addressed the challenges presented by Astronomer, enhancing their data strategy and allowing them to focus on what they do best: simplifying and speeding up the home closing process for their customers.

“Prefect’s support feels like a true collaboration, significantly increasing our team’s efficiency and satisfaction.” - Sunny

Experience the difference

Explore our Prefect vs. Airflow page to learn more about why teams like Endpoint are switching to Prefect. Or, speak with an expert and let us show you by booking a demo with our team.