Built to Fail: Design Patterns for Resilient Data Pipelines

In the ever-evolving world of data engineering, one truth stands out: change is constant and failure is inevitable. But what if we could build systems that not only anticipate failure but embrace it as a pathway to resilience? Let's explore how principles from resilience engineering can revolutionize how we build data pipelines.

The True Nature of Software Design

Before diving into resilience patterns, we need to challenge a fundamental misconception about software design. Many believe that design is about accomplishing tasks or meeting requirements. However, consider this thought experiment: if you had to build an application with completely known, unchanging requirements that would never need modification, would you need design patterns at all? You could simply brute force the solution without concern for structure or architecture.

This reveals a crucial insight: software design isn't about the present—it's about the future. As Sandy Metz eloquently puts it, "practical design does not anticipate what will happen. It merely accepts that something will, and that in the present you cannot know what."

The Unique Challenges of Data Engineering

In data engineering, change isn't just common—it's relentless. Consider the typical scenarios:

• Data schemas evolve constantly, both upstream and downstream
• Data sources appear and disappear without warning
• Data volumes fluctuate unpredictably
• External system dependencies shift beneath our feet
• Stakeholder expectations transform daily

As Heraclitus might say if he were a data engineer: "No person ever steps in the same data stream twice, for it's not the same stream, and they're not the same person."

Learning from Resilience Engineering

Resilience engineering emerged from studying major industrial disasters like Three Mile Island and the Challenger explosion. Two fundamental principles from Charles Perrow's Normal Accident Theory (1984) are particularly relevant:

1. In complex systems, failure is 100% inevitable
2. There's rarely a single root cause—failures emerge from multiple, interacting factors

This field evolved through studying high-reliability organizations in the 1990s and entered software engineering in the 2000s, pioneered by Netflix's chaos engineering and Google's site reliability engineering practices.

Core Principles for Resilient Data Pipelines

1. Reliability is Emergent

A system's reliability isn't just the sum of its parts. Two reliable components can create an unreliable system, while unreliable components can sometimes combine to create reliability. Think of a memory-leaking application that becomes reliable through scheduled Kubernetes restarts.

2. Design for the Unknown

When building data pipelines, we can't predict every failure mode. However, we can implement four key strategies to handle unexpected issues:

• Functional Observability: Monitor at the business logic level, not just infrastructure. Your Datadog metrics might show perfect system health even while your pipeline is failing because they're monitoring the wrong abstraction layer.
• Shift Left Practices: Don't wait until data reaches your warehouse to validate it. Integrate quality checks throughout your pipeline, starting with data ingestion.
• Adaptive Capacity: Build systems that scale and adjust automatically based on changing demands through auto-scaling and serverless architectures.

3. Graceful Extensibility

Systems should be designed to bend rather than break when reaching their limits. This means balancing two principles:

• Graceful Degradation: Continue processing valid records even when some fail. Use patterns like dead letter queues to handle problematic data without halting the entire pipeline.
• Software Extensibility: Design configuration-driven pipelines that can be modified without code changes. Keep processing logic independent of data schemas to accommodate change.

4. The Human Element

The "irony of automation" tells us that adding automation increases system complexity and potential failure modes. To counter this:

• Maintain manual operational knowledge
• Ensure pipelines can run locally for debugging
• Establish a strong incident response culture
• Conduct blameless postmortems
• Document learnings for future reference

Embracing Change

As Grace Hopper famously warned, "The most dangerous phrase in the English language is 'we've always done it this way.'" Building resilient data pipelines isn't about preventing all failures—it's about creating systems that can adapt, recover, and learn from failures.

The key is to embrace change as a constant companion rather than an unwelcome guest. By applying principles from resilience engineering, we can build data pipelines that don't just survive change—they thrive on it.

Inspired by the talk at Big Data London 2024 - watch it below.