A platform approach to workflow orchestration
Customized workflow orchestration for self-service data teams
At Prefect, we’ve seen how a platform approach to data engineering can massively increase developer velocity. This approach abstracts away the complexity of infrastructure while giving engineers the power to build resilient, complex workflows. For developers this means: reduced mental overhead, less development time, and fewer headaches to get their code up and running quickly.
For platform teams, it means protecting your infrastructure with a minimally invasive architecture while meeting the needs of the teams you support.
Prefect is perfect for platform people
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to whatever workflows they are building. It just so happens that Prefect is also a highly composable application framework that can be customized by platform teams to better serve developers and stakeholders.
With Prefect, platform teams can:
- Customize a platform that developers can use to build and observe their workflows
- Set appropriate levels of access to infrastructure and other resources
- Create and scale opinionated interfaces to infrastructure, without significant overhead
- Provide infrastructure abstractions appropriate for the skill level of the team they support
Developers, the workflow builders
“Everything fails, all the time”
-Amazon CTO, Werner Vogels
Code breaks, it is a fact of life. APIs change, infrastructure fails unexpectedly, and any other of an infinite number of problems can occur. Every developer wants to make their workflows resilient to failure.
Developers...
-want to schedule and observe their code
-don't want to rewrite
-need help with infrastructure abstractions such as containers, Kubernetes, AWS ECS, etc.
With no standardization or platform team helping them, developers are left figuring out how to deploy on their own. While this is possible for some, it isn't for many.
Platform teams, the infrastructure owners
Enter platform teams, engineers responsible for the infrastructure of an organization.
Compared to a data engineer who is often building one-off data pipelines or trying to piece things together themselves, platform teams care about ensuring that the business has the tools needed to achieve its goals at scale. To this end, platform teams build and expose foundational frameworks to help other teams build faster (particularly with respect to infrastructure) while maintaining compliance and access control standards.
Platform teams...
-are responsible for the sanctity of a company's infrastructure
-help developer teams put their work into production using this infrastructure
By providing a standardized surface area for how developers interact with data, infrastructure, and other systems, platform teams can help workflow builders create faster and scale safely.
At Prefect, we see this platform approach to data engineering frequently, with organizations ranging from Fortune 100 companies to smaller tech firms who want to move as quickly as possible.
Creating a platform with Prefect
Platform teams can use the modular architecture of Prefect to create a custom platform that helps data engineers, data scientists, or anyone creating workflows to build faster while meeting infrastructure and access requirements.
Give developers code interfaces and a UI
Prefect turns every object into discrete API automatically, accessible by your users. Create deployment templates for your team to submit work via an API, or give them the ability to build their own highly custom workflows.
The Prefect UI gives teams a UI to track and observe their work, or even to submit templated workflows. Even non-technical teams can submit workflows with parameters, and track their progress.
Customize your infrastructure concepts
Prefect workers and work pools allow platform teams to customize their infrastructure definitions in Prefect:
-Create opinionated infrastructure patterns
-Tightly control who uses what infrastructure with RBAC + deployments
-Dynamically assign infrastructure, switching between cloud providers and services easily
Keep it minimally invasive
Prefect's hybrid model means that execution and orchestration are separate concepts, meaning you can keep your data and code on your own premises. With Prefect Cloud, isolate and scale the infrastructure behind your workflow, without needing to worry about the orchestration server.
Platform user patterns
Let’s see some examples of how Prefect provides clear interfaces to create guardrails for different developer personas:
Deployment creators
Work pools can be used to lock down deployment infrastructure options. For example, you can remove a field you don’t want deployment creators to adjust or hard code a value for it.
Prefect Block types are Python classes that store configuration and code. Blocks have a structured HTML form for input and can be used in workflow code. Alternatively, they can be defined in Python code. Blocks are great for sharing configuration, as they live on the server.
Blocks are extremely flexible. Custom block types can be created to do whatever your team needs. Instead of needing to fork Prefect, you can just create a block with the capabilities you need and have your team use it. We see many platform teams using blocks to create
With Prefect’s team management, granular infrastructure control options, and RBAC options including custom roles, platform teams can make sure right people have access to the right resources.
Deployment runners
Through role-based access controls (RBAC), Prefect Cloud teams can limit users to run workflows but not create them.
Additionally, workflow authors can specify the data types of parameters required for the entry point function. This way at runtime, the deployment runner must provide a parameter value of an acceptable type to run the workflow.
Under the hood, Prefect uses Pydantic, so parameters can be limited to a list of choices through enums. Authors can even create a custom Pydantic class to require that a value in a valid email address format is provided or ensure a maximum or minimum numeric value.
Deployment viewers
Prefect Cloud administrators can further limit user permissions to viewing the results of runs, but not running them. The principle of least privilege states that each entity should be granted the minimum resources and authorizations needed to perform its function. Sometimes the ability to check in is all you need.
Prefect Cloud also allows for automations that fire PagerDuty notifications, emails, Slack messages, or other notifications in response to event triggers you specify. So if a flow run fails or data doesn’t look how you expect, the right people can be automatically alerted.
Additionally, if a Prefect team is using error summaries powered by MarvinAI, they can quickly see the source of workflow failures at a glance in the UI.
As you can see, Prefect provides data platform teams the ability to create self-service guardrails for personas of any type.
Build a platform for your developers
We see more and more companies adopting a platform approach to data engineering. This approach empowers developers to build faster while providing infrastructure and access safeguards.
A platform team can tailor Prefect however they choose: from adding permissions to infrastructure environments, to heavily opinionated deployment patterns. Developer teams will move faster, and your infrastructure will run smoother than ever.
For examples of teams using Prefect as a platform for a variety of engineering teams, check out the following case studies:
Prefect makes complex workflows simpler, not harder. Try Prefect Cloud for free for yourself, download our open source package, join our Slack community, or talk to one of our engineers to learn more.
Related Content
