Chloe Maraina is a specialist in the architecture of real-time data systems and a passionate advocate for the intersection of data science and business intelligence. With her deep background in building scalable visual stories from massive datasets, she currently focuses on the evolution of multimodal streaming engines and their role in fueling the next generation of artificial intelligence. In this conversation, she explores how the transition from simple event brokering to comprehensive stream processing is redefining the unit economics of the cloud and the fundamental reliability of AI-driven automation.
The following discussion examines the strategic shift toward unified data foundations, the technical trade-offs between speed and storage cost, and the emerging role of SQL in bridging the gap between live streams and historical records.
Traditionally, organizations managed separate clusters for observability, CRM, and ERP data. How does shifting to a multimodal engine change the daily responsibilities of data architects, and what specific operational efficiencies should they expect when moving beyond simple event brokering into full stream processing?
The shift to a multimodal engine like Redpanda One fundamentally redefines the role of the data architect by moving the control plane from the cluster level to the topic level. Instead of spending their days as “cluster managers” worrying about the overhead of maintaining separate silos for ERP, CRM, and observability data, architects can now focus on the strategic design of data flows. This transition from simple event brokering—which is just moving data from point A to point B—to true stream processing allows for filtering and analyzing data in real time within a single environment. The operational efficiency is massive; organizations no longer need to manage three or four specialized environments to stay competitive in event-driven automation. By unifying these workloads, teams can reduce the complexity of their infrastructure while gaining the ability to tune every single data feed for either speed or storage density depending on the business need.
High cross-availability zone networking fees often kill massive streaming projects before they start. When writing message contents directly to cloud object storage while only using the expensive network for metadata, what is the step-by-step impact on unit economics and total cost of ownership?
The impact on unit economics is a “fundamental shift” because networking fees are often the hidden killer of large-scale streaming deployments. By adopting a cloud-first approach where message contents are written directly to object storage, such as AWS S3, you effectively bypass the bulk of those expensive cross-AZ networking charges. In this model, the high-cost network is reserved only for small bits of metadata, which represent a tiny fraction of the total data volume. This specific action allows organizations to greenlight massive projects that were previously deemed too expensive due to projected egress and transfer costs. For the total cost of ownership, this means you are pairing the durability and low cost of object storage with the performance of a streaming engine, significantly lowering the “per-message” cost of data ingestion.
AI agents and chatbots require massive volumes of up-to-date contextual data to avoid hallucinations and remain accurate. How does providing a unified real-time data foundation change the way developers build these applications, and what specific metrics indicate a successful integration of streaming data into AI workflows?
Providing a unified real-time foundation changes the developer experience by moving away from fragmented batch updates toward a continuous “Agentic Data Plane.” Developers can now feed AI agents the high volumes of high-quality, up-to-date contextual data they require to perform autonomously without the lag that typically leads to hallucinations. A successful integration is measured by the agent’s ability to reach accurate outcomes using vast data aggregates rather than just a few isolated data points. We also look at the seamlessness of the connection between agents and data sources, often facilitated by protocols like MCP and Agent2Agent (A2A). Ultimately, the metric for success is the reduction in “data motion” complexity—if a developer can query the entire data lifecycle instantly to inform a chatbot, the integration is working.
Automatically converting data streams into Apache Iceberg tables allows for real-time access without traditional ETL pipelines. What are the primary technical trade-offs when choosing between in-memory write caching and tiered storage, and how should teams decide which format fits their performance-critical applications?
The choice between these formats is a classic trade-off between absolute speed and cost-effective scale. In-memory Write Caching is designed for performance-critical applications where sub-millisecond latency is the priority, allowing users to build high-speed workflows that operate entirely in a high-performance storage layer. On the other hand, Tiered Storage balances speed and cost by intelligently distributing data across local disks and cloud object storage, which is better for long-term retention and historical analysis. Teams should opt for Write Caching for real-time fraud detection or high-frequency trading where every microsecond counts. For broader analytics or building a lakehouse via Iceberg Topics, Tiered Storage is the better fit, as it avoids the “ETL tax” while keeping data accessible for direct queries without the high price tag of constant in-memory residency.
New capabilities allow for performing SQL joins across live streaming data and historical tables. How will the ability to query the entire data lifecycle instantly change the way highly regulated industries handle audit trails, and what role does automated data lineage play in maintaining trust?
For highly regulated industries like banking and healthcare, the ability to perform SQL joins across live streams and historical Iceberg tables is a game-changer for transparency. It bridges the gap between what is happening right now and what happened a year ago, making the entire data lifecycle instantly queryable for auditors. Automated data lineage plays a crucial role here because as agentic AI moves into the mainstream, the primary concern for these industries is not the model itself, but the “trust” in the data feeding it. By embedding lineage and quality tools directly into the console, every data stream becomes traceable and continuously validated. This ensures that an audit trail isn’t just a static report, but a live, verifiable record of how data influenced a specific AI-driven business decision.
What is your forecast for streaming engines optimized for AI?
I forecast that the industry will move rapidly toward “streaming intelligence,” where observability, access control, and governance are no longer post-processing steps but are embedded directly into the data engine. We are moving toward a future where Redpanda SQL and similar tools will allow for a complete data platform that treats live streams and long-term storage as a single, searchable entity. My prediction is that by 2026, the distinction between a “streaming platform” and a “database” will continue to blur, as the demand for real-time AI context forces every data engine to support multimodal workloads by default. Success will be defined by how well these engines can manage risk, performance, and compliance automatically as data is generated, rather than after it has been processed in batches.
