How LiveEO Harnesses Petabytes of Satellite Data with Prefect
Everyone is happy using Prefect and it has improved our time-to-customer value through faster feature development, bug fixes, and product iteration. -Toby Rahloff, Director of Solution Architecture, LiveEO
Transforming Infrastructure Management with Satellite Data
LiveEO, a Berlin-based SaaS company, is transforming how infrastructure operators process satellite data. By analyzing massive satellite datasets, LiveEO delivers actionable insights across critical sectors like utility maintenance, third-party risk monitoring, and supply chain compliance. Their work bridges the gap between science-heavy academia and practical civil applications, helping to mitigate risks and optimize maintenance strategies for infrastructure, and ensuring adherence to the environmental regulation, the EU Deforestation Regulation (EUDR).
The Challenge: From Homegrown Platform to Modern Orchestration
Toby Rahloff, the Director of Solution Architecture at LiveEO, oversees the organization’s ML, data, and analytics teams, which are responsible for managing and processing satellite data on behalf of their customers. Due to the nature of the satellite work, a reliable infrastructure is essential. “To monitor risks, optimize maintenance, and ensure environmental compliance, we must analyze satellite images, which are humongous and require optimization and scalability,” Rahloff said.
At first, LiveEO managed its growing data operations through a homegrown solution built on Airflow, custom scripts, and manual processes. This setup served them well initially, but as LiveEO scaled, the team began to face bottlenecks due to resource inefficiencies, fragmented monitoring, and the burden of workflow maintenance. These limitations quickly became too significant to ignore.
- Resource Inefficiency: The platform lacked dynamic scaling, which meant that processing high-resolution satellite imagery was both inefficient and costly. Teams struggled to manage compute resources effectively as data volumes grew.
- Fragmented Monitoring: Different teams relied on a patchwork of monitoring systems, making it challenging to ensure workflows ran smoothly. This fragmentation made it difficult to maintain visibility across their entire operation.
- High Maintenance Burden: Frequent manual maintenance was required to keep workflows operational, draining valuable engineering resources that could have been better spent on innovation.
When LiveEO turned to Airflow to address these challenges, they encountered new obstacles that made their situation even more complex.
- Static Resource Provisioning: Processing satellite imagery required thousands of compute instances, but Airflow's architecture couldn't scale dynamically to meet these demands.
- Rigid DAG Structures: Developers found themselves frustrated by Airflow's rigid DAG structures, which introduced unnecessary complexity and cognitive overhead.
- Limited Observability: Limited observability made it difficult to troubleshoot and optimize workflows effectively.
Transforming Critical Operations with Prefect
In light of these insurmountable challenges, LiveEO began searching for a platform that could better support their dynamic, data-heavy workflows while providing more control, scalability, and observability. After evaluating several options, they chose Prefect for its modern, developer-friendly design and its ability to address their unique challenges with ease.
Prefect delivered capabilities that aligned perfectly with LiveEO's needs.
Flexible Microservices-Ready Workflows
Prefect enabled LiveEO to transform workflows into instantly deployable microservices, reducing cognitive overhead and ensuring that every piece of business logic (and thus, Python code) could operate as a standalone, composable service.
"Before Prefect, we had a dispersed array of approaches, causing cognitive overhead. With Prefect, you write your business logic, and it becomes a data processing microservice." - Rahloff
Developer Experience and Productivity
Prefect's Pythonic workflows eliminated the need for rigid DAG structures, empowering LiveEO's engineers to design workflows that simply integrated with the existing codebase yet remained powerful, eliminating the burden of retrofitting and forced adaptation efforts.
With Prefect’s intuitive design, LiveoEO engineers no longer had to worry about rigid structures or complex configurations, and could dedicate more of their time to building workflows in Python. This, as well as real-time processing capabilities, streamlined the software development lifecycle and accelerated LiveEO’s delivery of insights to clients.
"After implementing Prefect, we quickly saw improved developer experience, velocity, and resilience. We were able to triple our development speed with Prefect – resulting in faster development, bug fixes, and product iterations – which also improved our time-to-customer value." - Rahloff
Dynamic Scaling and Cost Control
Through seamless AWS integration (including Lambda and Kubernetes), Prefect allowed LiveEO to dynamically provision and decommission compute resources, ensuring efficient satellite data processing. The ability to scale to meet processing demands steeply reduced compute costs, reducing LiveEO’s AWS spending by 63%. In addition to Prefect’s dynamic scaling, other features like validation mechanisms helped LiveEO to optimize their resources, allowing them to keep costs in check.
"Prefect’s impact, reliability, and scalability directly influence cloud costs. After switching to Prefect, we saw a 63% reduction in AWS spending." - Rahloff
Enhanced Reliability
Centralized monitoring and real-time task performance insights like error validation made it easier for LiveEO to troubleshoot and optimize workflows at scale.
"Another huge win for us was input validation. For example, we often trigger flows that require thousands of CPUs and GPUs on AWS. Prefect helps validate inputs before infrastructure is provisioned, reducing costs and errors." - Rahloff
Lasting Impact
LiveEO relies on Prefect’s orchestration platform to scale operations, streamline workflows, and maintain cost efficiency while expanding its geospatial data analysis capabilities. With Prefect in place, LiveEO’s ML, data, and analytics teams were able to quickly optimize their productivity and performance, allowing LiveE to better serve customers.
Key Highlights:
- 63% Reduction in AWS Costs: Dynamic scaling significantly lowered infrastructure expenses.
- 3x Faster Development Speeds: Prefect's developer-friendly tools enabled faster time-to-production.
- Microservices at Scale: Prefect has turned every workflow into a composable, reusable microservice.
Results Across Sectors
Prefect and LiveEO’s partnership has also created a lasting impact across the globe, enabling LiveEO to deliver more efficient and effective insights to their customers. Some key examples being:
- Predictive Maintenance: With Prefect’s scalable resources, LiveEO rapidly processes satellite imagery to identify vegetation risks near power lines and third-party risks near pipelines.
- Supply Chain Compliance: LiveEO effectively monitors deforestation with Prefect by efficiently processing high volumes of satellite imagery that help to solve compliance with the EU Deforestation Regulation (EUDR).
Experience the Difference
Learn more about how Prefect empowers businesses like LiveEO on our Prefect vs. Airflow page, or schedule a demo to see how we can transform your workflows.
Related Content
