Crumbl and Prefect: A Five Star Recipe

About

Crumbl is a dessert startup sensation that uses fresh, high-quality ingredients to deliver delicious gourmet cookies for takeout and delivery. With a weekly rotating menu, Crumbl’s highly complex and time-sensitive operations are fueled by a large software engineering team that, in turn, relies on clean, precise data from their small-but-mighty data team.

Challenge

Crumbl’s operational complexity requires customization, granularity, and a high level of security that most OOTB data platforms can't support. Thankfully, their 45-person engineering team is more than up to the challenge of building a unique solution that supports every bite of the business, from marketing to franchise operations to inventory management. However, as Crumbl has grown from a scrappy one-store startup to an international franchiser, the existing processes of relying on individually scheduled AWS Lambdas has become too unwieldy to troubleshoot and inaccessible to the growing data team.

“So we asked ourselves, ‘How do we make our data pipelines clean, maintainable, accessible for analysts, and most of all, secure?’ This is where Prefect excels—in being able to monitor a plethora of custom ETL pipelines to see where things fail and where things break, all while knowing that the sensitive data stays secure.” Chandler Soto, Data Engineer, Crumbl

Solution

Crumbl needed a data workflow automation tool to replace their usage of Lambda functions. The new automation would have to support their fast-paced sprint speeds without contributing to code sprawl. Most importantly, they needed the visibility to not just see which jobs are running and when, but how to diagnose issues if they arise.

When Crumbl first implemented their data delivery and transformation system, they quickly leveraged a PostgreSQL-based warehouse and Lambda jobs to get the system up and running. However, as their business expanded, Crumbl knew they'd need a more resilient and observable system. So, they redesigned their entire stack with Prefect in mind, taking advantage of easy integrations to Snowflake and dbt.

With Prefect in place, the Crumbl team can easily observe the status of their various ETL jobs. Chandler, the sole data engineer supporting this entire system, can easily monitor success or failure, and even debug and rerun jobs from his phone. “It’s been a huge weight off my shoulders being able to monitor our jobs and ensure that they’re running successfully,” Chandler said.

Result

Because of Crumbl’s migration to a modern data platform powered by Snowflake, Prefect, and dbt, analysts and downstream stakeholders alike can benefit from the added confidence of knowing that the information they need is accurate and always available. The observability features in Prefect allow Crumbl to confidently support their ever-expanding rotating menu, which sometimes requires inventory and purchasing decisions a year in advance.

“Our team supports the models for different ingredients and various forecasts, and our team can support this business-critical function with confidence because of Perfect.” Chandler Soto, Data Engineer, Crumbl

Additionally, Prefect’s observability features allow ‌passive monitoring while also enabling the Crumbl team to react and respond to issues at record speed. The increase in efficiency in time to resolution for their data team is incalculable.

“Issues and tickets that previously took weeks of developer time to prioritize officially, diagnose across disparate systems, and implement a solution can now be resolved in just minutes.” Chandler Soto, Data Engineer, Crumbl

Checkout Prefect Cloud for yourself!