Cox Automotive Meets Dynamic Demands with Workforce Analytics Solutions Powered by Prefect

Empowering worksites with optimal staffing and equipment hinges on making data-driven decisions. This is especially true at Manheim, a Cox Automotive brand, North America's leading provider of vehicle remarketing services. As the operator of the largest wholesale used vehicle auction network, Cox Automotive's success depends on Manheim's efficient management of over 70 auction sites across the United States, and their 18,000 employees facilitate auctions for millions of used vehicles every year.

John Kang, Director of Planning Analytics, and his team look to tackle the following challenge:

How do we support our sites with the right decisions driven by the right data to support staffing and equipment investments?

Evolving data pipelines: Why Cox Automotive shifted from cron jobs to Prefect

Initially, the team implemented cron jobs to trigger Python scripts that would call out to virtual machines that Cox managed to run large data conversions. Many scenario planning questions were also fielded by spreadsheets in Excel. John and the team found this approach inadequate in several key areas, especially as their data infrastructure continued to grow, which led to their adoption of Prefect:

Flexibility 🧘‍♂️

It was difficult to verify their hourly data ingestion scripts were completing successfully or that they moved the intended data with cron. If a script was unsuccessful at refreshing data, retrying on failures was a manual process that created hurdles for non-technical staff.

Prefect Cloud's user-friendly interface lets everyone on the team, regardless of coding experience, see the health of Cox Automotive's data pipelines. This includes real-time visibility into a flow status and automated retries. The Planning Analytics team was able to unlock these observability and repeatability features just by adding Prefect’s @task and @flow decorators to existing Python scripts.

Scalability ☁️

Cox Automotive evaluated some legacy and GUI-based orchestrators and found them suitable for prototyping but lacking in the flexibility or reliability required to support their data applications. Their self-hosted virtual machines for data transformations limited their ability to scale resources efficiently.

Cox Automotive's Planning Analytics team now achieves significant efficiency gains in their data infrastructure by leveraging Prefect's flexible deployment options. Prefect Deployments were a catalyst that enabled a transition from self-hosted virtual machines to a cloud-based container infrastructure on Google Kubernetes Engine (GKE). Their new containerized setup allows the team to orchestrate hundreds of data pipelines, optimize their compute environment, and scale up or down to meet the needs of their data conversion jobs.

Self-service 🎾

Self-service views of data pipelines would help team members who were not fluent programmers, but this was not possible with the team’s old approach. Keeping everything in Python would be necessary for those on the team who were fluent Python programmers.

New team members now have a simple way to quickly learn how to deploy their data pipelines, without wrestling with infrastructure management or provisioning they used to have to perform. Prefect’s integration with GitHub has allowed new and existing team members to accelerate their time to production and build robust CI/CD pipelines that speed flow deployment.

Trust 🤝

The volume and diversity of Cox Automotive’s auction business require an assurance that the applications running their planning analytics be deployed nationwide with high availability and security.

Cox Automotive already had extensive data and IT security tools in place to separate concerns for orchestration, code, and data. Prefect's robust security features, including granular access controls (ACLs), comprehensive logging, secrets management, and execution options integrated with their existing toolchain work with Prefect in authenticating users, easing their security audit process and alleviating their security concerns.

Cox Automotive’s new planning analytics stack

The transition from cron to Prefect has resulted in a framework for building resilient Python workflows that empower Cox Automotive to improve workflow execution visibility, optimize resource utilization, speed pipeline deployment, and instill confidence in results. As John explains:

We were at first relying upon cron, but we did not have the observability and retries that I really like with Prefect. We started using Prefect within our application to refresh our Redis and Postgres caches and it's doing a wonderful job.

Their new workforce analytics stack utilizes the following key tools:

• Data warehouse - Snowflake
• Data caches - Redis and Postgres
• Data transformations - dbt
• Compute platform - Google Kubernetes Engine
• Dashboards - Plotly Dash Enterprise
• Orchestration platform - Prefect

The Planning Analytics team at Cox Automotive leveraged Prefect's comprehensive learning resources, including Associate Certification Courses, extensive documentation and examples, and Prefect’s Slack community and customer support, to seamlessly transition from manual Python scripts. Soon, the team had one of their most important applications, a workforce and equipment scenario planner, powered by data and insights driven by Prefect.

Workforce and equipment planning, powered by Prefect

As demands and availability shift across their network of auction sites, Cox Automotive’s data applications empower corporate users and site leaders to make informed decisions based on the latest data. Prefect orchestrates ELT jobs that efficiently ingest and transform data in Snowflake for these applications, replacing costly workflows reliant on external databases with large flat files. Additionally, Prefect manages data hydration jobs that keep the team's Redis and Postgres caches fresh, ensuring real-time updates for dashboards like the one pictured below.

Discover the future of data at Prefect Summit

Organizations like Cox Automotive are looking to build their next generation of data tools to take advantage of innovation without compromising the trust of their users. You can find more information on how Prefect is enabling this trust with transactional, flexible, and portable orchestration, including John’s talk on Cox Automotive’s journey to Prefect, on our Prefect Summit recap page.