Chloe Maraina stands at the forefront of the modern data revolution, blending her deep expertise in business intelligence with a forward-thinking vision for data science and systems integration. As an expert who sees data not just as numbers but as a narrative waiting to be told, she has spent years navigating the shift from rigid, legacy automation to the fluid, intelligent ecosystems we see today. Her perspective is particularly vital as we approach 2026, a year that promises to redefine how humans and machines collaborate through increasingly sophisticated orchestration layers. With a background that spans from traditional data management to the cutting edge of AI-driven workflows, she provides a unique lens on how enterprises can maintain their competitive edge while ensuring their digital infrastructure remains resilient, compliant, and scalable.
This discussion explores the rapid evolution of workflow engines, moving away from simple task automation toward comprehensive platforms that manage microservices, cloud-native deployments, and intricate data pipelines. We delve into the critical transition from manual scripting to the adoption of directed acyclic graphs (DAGs), which allow for more modular and visible dependency management. A significant theme of our conversation focuses on the “agentic” shift—the merging of large language models and autonomous agents with traditional deterministic processes to create more adaptive systems. We also weigh the pros and cons of different architectural philosophies, such as the code-centric nature of tools like Apache Airflow versus the business-process-focused designs of Camunda, while highlighting how platforms like Argo and Kestra are simplifying the complexities of Kubernetes and multi-cloud environments for the next generation of IT leaders.
Traditional scripting has evolved into complex models using directed acyclic graphs (DAGs) to manage intricate dependencies; how has this shift changed the daily reality for teams trying to maintain high-uptime data pipelines?
The shift from the old-school cron jobs and linear bash scripts to the world of Directed Acyclic Graphs, or DAGs, has been nothing short of a paradigm shift for how we visualize and control the flow of information. Back when we relied on manual scripting, a single failure in a sequence often felt like a black box; you’d spend hours digging through logs just to figure out which link in the chain snapped. With the rise of platforms like Apache Airflow, which pioneered this DAG-centric approach, teams now have a programmatic and visual map of every dependency, allowing them to author, schedule, and monitor workflows with surgical precision. The release of Airflow 3.0 in April 2025 marked a massive milestone here, introducing unified orchestration that handles both data and AI infrastructure in one breath. By using DAGs, we can isolate tasks and version them, ensuring that if a patch management process or a security audit fails, we aren’t flying blind. It turns a chaotic web of scripts into a structured, visible architecture where you can see the pulse of your entire system in real-time, significantly reducing the “mean time to recovery” when things inevitably go sideways.
With the rise of “agentic AI” and tools like Agentspan, how are workflow engines transitioning from purely deterministic systems to platforms that can handle non-deterministic agent behavior without losing control?
We are moving into a fascinating, albeit slightly nerve-wracking, era where our workflows are no longer just “if this, then that” sequences. Traditional systems were deterministic—they did exactly what they were told, every single time—but AI agents introduce a layer of non-deterministic behavior where the machine might take different paths to reach a goal. To keep this from turning into digital chaos, engines like Conductor, through its new Agentspan runtime, are creating a “durable execution layer” that gives these agents a safe space to operate while still allowing us to call tools via internal APIs. Even legacy-focused platforms like Camunda have pivoted, blending their strict Business Process Model and Notation (BPMN) logic with agentic orchestration to ensure that while the AI might be creative in its execution, it still follows the compliance and approval guardrails set by the business. It feels like we are finally building the “brain” for the automation “muscle,” using things like the Model Context Protocol (MCP) to let agents communicate with our existing systems. This allows us to operationalize AI in the real world, where things like failure modes and residency constraints are very real, rather than just keeping AI as a toy in a laboratory setting.
For organizations that are deeply embedded in the Kubernetes ecosystem, why is a container-native approach like the one found in Argo Workflows superior to more general-purpose orchestration tools?
If your entire world is built on Docker and Kubernetes, using an orchestrator that doesn’t “speak” that language natively is like trying to fit a square peg in a round hole. Argo Workflows stands out because it treats every step of a process as a container, which is a massive win for consistency and scaling. When an IT engineer or a data scientist defines a workflow using a YAML file in Argo, they are speaking the same language as the rest of their infrastructure-as-code stack, making version control and CI/CD integration seamless. This container-native DNA allows Argo to excel at complex provisioning across major cloud providers like AWS, Azure, and Google Cloud Platform while maintaining the governance assurance that comes with being a CNCF-backed project. You aren’t just running a script; you are orchestrating a fleet of containers that can spin up, do their job—whether it’s training a machine learning model or running a batch data process—and then vanish, optimizing your resource spend. The beauty of it lies in that native integration, where the workflow engine and the underlying infrastructure are essentially two sides of the same coin, providing a level of resilience that general-purpose tools struggle to match in a cloud-native environment.
How do declarative, YAML-first platforms like Kestra manage to satisfy both the high-code requirements of developers and the need for a unified control plane across different business domains?
The “declarative” revolution, which Kestra is a prime example of, is really about lowering the barrier to entry without sacrificing the power that developers crave. By using a YAML-first approach, you’re essentially creating a universal translator; a developer can work in their favorite code editor and commit changes to Git, while a business analyst or a security auditor can look at the same workflow through a graphical interface and actually understand what’s happening. Kestra has built this massive ecosystem of over 1,300 plugins that cover everything from cloud databases to SaaS tools, which means you aren’t stuck writing custom glue code for every new integration. This platform also leans heavily into the future with its AI copilot, allowing users to describe a workflow in natural language and have it generated into an executable YAML structure. It creates a single control plane where data pipelines, AI memory management, and standard IT operations tasks all live together. It feels much more like an organized library than a messy workshop, giving everyone from the infrastructure team to the finance department a clear, audited view of the processes that drive the company.
When a company decides to scale their automation from a small, one-off project to a full-scale enterprise operation, what are the most common pitfalls they face regarding long-term maintenance?
One of the biggest traps is what I call “the accidental architect” syndrome, where a simple automation script that was meant to be a temporary fix ends up becoming the backbone of a mission-critical process without any of the necessary governance. When you start scaling, you quickly realize that if you haven’t extended your Infrastructure-as-Code (IaC) principles to your workflows, you’re looking at a maintenance nightmare where nobody knows which version of a task is running or how to roll it back if it breaks. You need to treat your workflows like production-grade software, with rigorous testing, versioning, and disaster recovery plans in place from day one. There’s also the challenge of “tool fatigue”—if you have one team using Airflow for data, another using Argo for Kubernetes, and a third using Camunda for business processes, you end up with silos that are impossible to monitor collectively. The goal should be to find a tool that provides “durable execution,” meaning that even if the underlying system restarts or a network blip occurs, the workflow can pick up exactly where it left off. Scaling isn’t just about doing more; it’s about doing more with the discipline to ensure that a failure at 3:00 AM doesn’t turn into a catastrophic outage that requires a week of forensic analysis to solve.
What is your forecast for the role of workflow engines in the enterprise ecosystem by the end of 2026?
By the end of 2026, I believe the distinction between “data orchestration” and “AI orchestration” will completely vanish, leaving us with a unified layer of intelligent execution. We are already seeing the signs with Apache Airflow 3.0 and the way Dagster is moving toward asset-aware models that treat data sets and machine learning artifacts as first-class citizens. The real breakthrough will be the normalization of the Model Context Protocol, which will allow these workflow engines to act as the central nervous system for a company’s AI agents, giving them the ability to call internal APIs and interact with legacy databases as easily as a human operator would. I expect we will see a massive shift toward “hybrid execution” models, where the control plane is managed in the cloud for ease of use, but the actual execution of tasks stays within the customer’s secure environment to satisfy the growing demands of data residency and compliance. The platforms that win will be the ones that can prove their resilience—those that can manage millions of tasks across thousands of containers while still offering a “natural language” interface for the business teams. It’s a future where the workflow engine isn’t just a background utility, but the primary interface through which we manage the complex dance between human decision-making and machine-speed execution.
