Microservices Orchestration: What It Is, How to Use It

For modern data teams, microservices just may be one of the best software inventions of the 21st century. A microservice, briefly, is an independent software component that focuses on doing just one thing. It has its own codebase and environment. Teams can stitch microservices together to create more complex workflows while maintaining microservice independence. In contrast, in a monolithic architecture, all software components are developed and deployed from the same codebase in the same environment.

Microservices greatly simplify data pipelines. Teams can break down complicated data workflows into distinct steps, each built and deployed as a separate service. A common use case occurs when users upload data and companies process the data to take action. For example, a fintech company may need a user to upload a pay stub for income verification in a loan application. After the user uploads and saves the document, the company has to parse the document, check if the income levels are acceptable, decide to accept or reject the application, and finally, notify the user.

In a monolithic approach, the entire process would run from the same codebase. A single failure would necessitate rerunning the entire process. The mere mention of the word scaling would induce panic. In contrast, in a microservices approach each step can be run by itself, failures don’t crash the entire process, and each component can scale independently of the others. Additionally, separating out pieces of this workflow into separate microservices means they can be reused. There are likely additional processes in which a paystub needs to be uploaded and processed that don’t deal with loan applications.

The issue, however, is that creating robust workflows out of microservices takes a lot of work. Figuring out which microservice failed in the workflow (and restarting it), making sure microservices execute in the proper order, and ensuring each microservice is scaled appropriately—these problems can almost make microservices not worth it. Almost.

Thankfully, there is a solution to these problems: microservices orchestration.

What is Microservices Orchestration?

Microservices orchestration is the deployment and coordination of a network of microservices within a distributed system. It is a layer over them that ensures they work together nicely in data workflows. Microservices orchestration handles a few key tasks:

• Ensuring microservices run when they’re supposed to, and in the correct order
• Retrying failed microservices immediately or rolling back to a clean slate (without affecting the rest of the pipeline)
• Scaling each microservice to handle its workload
• Adding observability to microservices to debug issues

Without microservice orchestration tools automating these steps, data teams would be overwhelmed to do each of these steps for each of their many data workflows.

Why you Need Microservices Orchestration

Without a microservice orchestrator, a lot can go wrong with microservice data workflows—and you’ll have to fix it all manually. A few big failure points and how orchestration helps:

🤪 Microservice execution can go haywire

Without orchestration, microservices may fire or not fire without rhyme or reason. If the first microservice in a data workflow fails, the second microservice may still try to run with incorrect data. Suddenly, data states are out of sync and the problem is compounding. A tempting solution would be to make microservices check the status of other microservices before running. But then, they’re no longer independent of each other: that’s a one-way ticket to monolith land.

Microservice orchestration handles the failures and execution flows. The microservices don’t need to know about other services; they run when the orchestrator tells them to. With a good orchestrator, that means they run exactly as expected.

❌Data workflows can fail silently

Data workflows often have multiple microservices that contribute to the final result. Like in the fintech example, paystub parsing and loan approval could be two separate services. When the workflow fails, it can be tough to figure out which microservice failed. It’ll also take time to restart that microservice and continue the workflow. In the worst case, failures can cascade through the workflow.

Microservice orchestration makes for resilient and debuggable workflows. A good orchestrator can automatically re-run a failed workflow, keeping it moving. If a re-run fails, the orchestrator can roll the microservice (and workflow) back to a clean slate and run from there. Orchestrators also give you visibility into the internal workings of the workflow. With logs and metrics, you can identify which microservice failed, why it failed, and what you need to do to fix it.

📈Scaling can cause over- or under-provisioning

Different microservices need different resources. An ingestion microservice may need network bandwidth, a processing microservice may need CPU and RAM, and a storage microservice may need disk I/O. Without an orchestrator, you’d need to manually scale each resource for each microservice. If workflow usage spikes, even a few minutes of scaling delay can cause a workflow breakdown, and a good day with high usage will turn into a bad one with churning users.

An orchestrator pools all the resources by type and allocates them to each microservice based on need. If usage spikes, the orchestrator will immediately scale up the resources to ensure a steady throughput, and it will scale down when the spike is gone.

Use Cases for Microservices Orchestration

Microservice orchestration is a good fit for workflows with these properties:

• Has many subtasks (e.g., ingestion, processing, storage, notification)
• Has its own custom infrastructure (e.g., databases, virtual servers, pipelines)
• Is event-triggered (usage can spike rapidly, and failures need to be quickly dealt with)
• Needs to be called by other parties, either internally or externally, and usually via API

Here are some common use cases that fit these criteria (to make the examples concrete, we’ll center them around e-commerce).

📊Data analytics

A data analytics workflow in e-commerce could be calculating the average purchase volume for a given month. First, an ingestion service would log each transaction with user IDs, total price, and date. Second, a processing service would collect all purchases for a given month and average the volume. Third, a storage service would save the last month’s average volume in a metric datastore. Finally, a reporting service would return all the values for the metric in a report or dashboard. In reality, this workflow would need to cover multiple different data sources, metrics, and outputs.

Great microservice orchestrators would ensure each service runs only when the prior service has correctly completed. Microservice orchestrators would also re-run failing components or alert the user if errors pile up. And they would enable easy addition of new microservices that pull from different data sources or push to different outputs.

⚡Machine learning

The quintessential use of machine learning in e-commerce is product recommendations. Here, there are two phases to consider: model training and inference. In training, there would be microservices for data collection, data cleaning, feature selection, training on the data, training validation, and versioning. In inference, there would be microservices for user data ingestion, data processing, feature extraction, model inference, and logging. That’s a lot of microservices, even for a simple model.

Microservice orchestrators are vital in managing complex machine-learning workflows for e-commerce recommendations. They ensure correct execution order, from data collection through model training to inference, while handling data flow between services. They also handle scaling resources, such as GPU resources during training and high-inference loads.

🤖AI agents

AI agents are actually also microservices! Each agent has a single job to do, independent of the others. It’s triggered programmatically. And each agent runs frequently. AI agent workflows can also get messy, with hundreds of agent calls in a single execution. In e-commerce, an AI agent workflow might focus on helping users decide which items to buy. One agent may be responsible for finding the top 10 results for a given query. Another agent may gather information about a particular result. Yet another agent could figure out an item's most efficient shipping method.

Microservice orchestrators play a crucial role in managing AI agent workflows. They ensure proper sequencing of agent calls, handle data flow between agents, and manage resource allocation for each agent. Additionally, orchestrators can monitor agent performance, retry failed agent calls, and provide detailed logging for debugging and optimization of the AI agent workflow.

Tools and Technologies to Enable Microservices Orchestration

There are many options for microservice orchestration—you could even create your own orchestrator (probably not worth the trouble though). Great microservice orchestrators stand out because they treat your data workflows as first-class citizens. These orchestrators make it easy to visualize your workflow runs and see execution flow. In addition, great microservice orchestrators support metrics and logging for fine-grained observability. Here are a few common choices for microservice orchestrators:

Kubernetes

Kubernetes is a container orchestration system. Containers are lightweight, independent software packages that contain everything needed to run. These containers can then be run in any environment. Data teams could put each microservice into its own container and then use Kubernetes to manage the microservices.

Kubernetes offers automated retries and rollbacks, auto-scaling based on load, and logging.

However, Kubernetes’ main drawback is that it doesn’t directly support workflows. Each microservice can be managed but it’s not possible to set execution dependencies or ensure data format consistency between microservices. Dashboarding, debugging visuals, and GUIs for workflows are also missing as a result.

Amazon ECS

Amazon Elastic Container Service (ECS) is a container orchestration solution within AWS. As with Kubernetes, data teams can put each microservice into a container and manage them via ECS.

ECS handles deployment, scheduling, scaling, and load balancing for each container. ECS doesn’t support workflow orchestration directly either, but other AWS services (such as Step Functions) can complement ECS. The main drawback with ECS is it’s really complicated to set up. Developers generally need to know the technical details of the system to get it to work without issues. Dependency management between multiple containers is missing, making debugging and understanding system state quite difficult. Dashboarding and helpful GUIs are also limited.

Prefect

Prefect treats your data workflows like the first-class citizens they are. With Prefect, you can orchestrate and automate any microservice workflow by simply adding a few decorators to your Python workflow code. Prefect runs on an open, easily extensible framework with many prebuilt integrations, so it will work with almost any tech stack or use case, including yours.

Once you’ve added Prefect to your workflow code, orchestration becomes very simple very quickly. Prefect does the following for you:

• Manages the scheduling of microservices, making sure they run in the right order and for the right triggers.
• Retries or rolls back failing microservices to protect the integrity and throughput of the workflow.
• Provides full observability support; on the GUI, you can visualize each run for every workflow and debug what went wrong.
• Supports both schedule-driven and event-driven workflows.
• Handles microservice resource scaling. With resource pools, Prefect can allocate resources to the microservices that most need them. Prefect also allows for caching so that similar results don’t need to be computed every single time.

From script to scale, Prefect's Python-based workflow orchestration platform delivers data workflows you can actually trust because they’re resilient to failure and adaptable to change.

Getting Started with Microservices Orchestration

Alright, we’ve given you the reasons why Prefect is great for microservice-driven workflow orchestration. Now it’s time to try it out for yourself: create a free account and follow this tutorial.