With an aptitude for data science and a vision for the future of data integration, Business Intelligence expert Chloe Maraina has built her career on creating compelling visual stories from big data. As companies grapple with the monumental task of quantifying the return on their AI investments, we sat down with Chloe to cut through the noise. This conversation explores the shift from simple A/B tests to sophisticated financial frameworks, delves into the hidden operational costs that can derail ROI calculations, and illuminates why “squishy” cultural metrics are often the most powerful predictors of long-term success.
Agustina Branz suggests A/B testing AI versus human output. Beyond conversions, what specific KPIs do you track for AI-generated content, and how do you ensure the test isolates AI’s true impact without external factors blurring the results? Please share an example.
That’s the fundamental question, isn’t it? It’s not just about whether AI is faster, but whether it’s better—or at least more cost-effective. Beyond just looking at raw conversions, a key KPI for us is the cost per qualified outcome. It’s a fantastic metric because it forces you to ask how much less it costs to get the same quality of result you were getting before. For example, we ran a test on AI-generated keyword clusters for a marketing campaign. We didn’t just look at which version drove more clicks. We tracked the entire funnel—traffic, engagement, and ultimately, conversions—for both the AI-generated set and the human-curated one. The key to isolating the impact was treating AI performance as a directional metric for optimization, not as a final, absolute judgment. It showed us where the AI excelled and where human intuition was still needed, allowing us to refine our approach rather than just picking a winner.
John Atalla’s framework includes “productivity uplift” and “value-realization speed.” Could you walk me through a project where you measured how quickly benefits appeared in the first 90 days? What were the biggest challenges in tracking that immediate financial impact?
I remember an early project where we were so focused on “productivity uplift”—essentially, time saved and capacity released—that we almost missed the bigger picture. We deployed an AI to assist a legal research team, and within the first month, we saw a clear reduction in the time it took to complete routine tasks. That was the easy part. The real challenge in that first 90-day window was measuring the “value-realization speed” of the secondary benefits. We started noticing an improvement in decision quality and even a spike in staff engagement, as the team felt more empowered. The biggest hurdle was attributing that directly back to the AI. There’s always so much noise in a business. We had to create a feedback loop where we could directly tie those improvements in customer experience and reduced error rates, which had a measurable financial impact, back to the specific processes the AI had streamlined.
Adrian Dunkley discusses “impact chaining” to map out an AI’s downstream value. Can you provide a step-by-step example of how you would apply this to forecast the ROI of a new AI tool for a sales team, tracing its effects beyond initial time savings?
Impact chaining is a brilliant concept, borrowed from climate research, and it’s perfect for forecasting AI’s true value. Imagine a new AI tool for a sales team. The first link in the chain is obvious: the AI automates CRM data entry and reporting, saving each salesperson, let’s say, five hours a week. That’s the efficiency gain. But we don’t stop there. The second link is what the sales team does with that reclaimed time; they can make more calls and send more personalized follow-ups. The third link is the direct result: a measurable increase in qualified leads. The fourth link is an improved win rate because the AI also provides better insights on which leads to prioritize. So, the final ROI isn’t based on the value of those initial saved hours. It’s calculated from the concrete revenue generated by the higher win rate. This chain allows you to trace a direct line from a simple automation to a significant impact on the company’s bottom line.
Salome Mikadze advises shifting from “model accuracy” to “what changed in the business.” How do you establish a clean pre-AI baseline for a complex process, and what are the top three metrics you track to show a clear, undeniable business impact?
That advice is spot-on and gets to the heart of the matter. Executives don’t really care about model accuracy percentages; they care about results. To get a clean baseline, you have to be disciplined. Before any rollout, we baseline the existing process meticulously. For a customer support team, for instance, we’d track first-response times, resolution times, and customer satisfaction scores over a full quarter. Then, we do a controlled rollout to a segment of the team. My top three metrics to demonstrate undeniable impact are always: first, time-to-value, because an unused model has zero ROI. Second is adoption by active users—if people aren’t using it, it’s a failure, period. And third, and this is crucial, is task completion without human rescue. This metric tells you if the AI is truly autonomous and reliable or if it’s just creating more work for your team to clean up.
The article notes that AI’s variable operational costs challenge traditional software pricing. When calculating an AI’s Total Cost of Ownership (TCO), what are the most commonly overlooked expenses, like model drift or data labeling, that can dramatically alter the final ROI?
This is where so many companies get caught off guard. They’re used to the old SaaS model: expensive to build, but with near-zero marginal costs to run. AI completely flips that script. The most overlooked expenses are absolutely the ongoing operational ones. I’ve seen TCO calculations that completely omit the cost of continuous data labeling needed to keep the model sharp, or the spend on prompts and retrieval, which can fluctuate wildly. Then there’s the cost of monitoring for things like model drift, which forces expensive retraining, and the very real human cost of change management to get people to trust and use the tool effectively. When you start adding up integration work, vendor fees, and evaluation harnesses, you realize that the initial development cost is often just the tip of the iceberg.
Michael Domanic distinguishes between “hard” and “squishy” ROI. How do you translate a “squishy” metric, like improved employee sentiment or experimentation, into a hard number that an executive will value? Could you share a specific anecdote where this was done successfully?
It’s all about connecting the “squishy” leading indicators to the “hard” lagging ones. I worked with a company that rolled out an internal AI tool, and at first, the only feedback we had was anecdotal. Employees were excited and felt more innovative. To capture this, we used self-reported data, and found that 73% of early adopters said the new tool improved their productivity. That was our “squishy” metric. But that perception created a powerful, virtuous cycle of adoption. To translate it into a hard number, we tracked the project completion times for that group of early adopters versus a control group. After six months, the data was undeniable: the teams who embraced the tool were delivering features 20% faster. We were able to go to the executives and say, “The positive sentiment you heard about wasn’t just talk; it’s directly responsible for accelerating our time to market.”
What is your forecast for how organizations will measure and price the value of AI in the next three to five years?
I believe we’re on the cusp of a fundamental shift away from legacy pricing models. The seat-based or feature-based SaaS model simply doesn’t make sense for AI, where value is tied to accomplishment, not logins. In the next three to five years, I forecast a massive move toward outcome-based and usage-based pricing. We’ll see more models like Zendesk’s, which charges per case resolution. Companies will stop buying “access to AI” and start paying for concrete results—a percentage of savings, a fee per successful transaction, or a share of the revenue generated. Consequently, the way we measure ROI will become far more sophisticated. We’ll move toward a standard of risk-adjusted ROI, where the gross benefit is discounted by reliability signals like hallucination rates and data-leak incidents. The squishy will become measurable, and the measurable will become truly transformative.
