Chloe Maraina is a seasoned software engineer who has spent her career at the intersection of Big Data and high-stakes infrastructure. Having worked on the recommendation engines powering Instagram Reels at Meta and now contributing her expertise to Google, she specializes in making massive AI systems more efficient. Her work focuses on the “invisible plumbing” of machine learning—the complex data lifecycles that determine whether a recommendation system is a high-performing asset or an environmental and financial liability.
In this conversation, we explore the intricate architecture of late-stage ranking, the technical strategies for eliminating “ghost data” through lazy logging, and how optimizing storage schemas and feature audits can lead to megawatt-scale energy savings in global data centers.
Recommendation systems typically use a funnel-based architecture that culminates in a feature-dense late-stage ranking phase. What specific bottlenecks emerge during this final stage, and how can engineers determine if a model’s complexity has started to exceed the physical limits of their hardware and energy capacity?
The primary bottleneck in late-stage ranking is the sheer density of features that must be processed and logged simultaneously. At this stage, we are often managing complex deep learning models, such as two-tower architectures, which might evaluate hundreds of dense and sparse features for a set of 50 to 100 items. The system hits a wall when the “write amplification” of these features—essentially the cost of serializing and storing them for potential training—starts to consume excessive CPU cycles and network bandwidth. Engineers can see they’ve hit a limit when data centers begin to run hot and operating costs climb into the millions of dollars due to the energy required just to move this “ghost data” around. At Meta, we realized we were hitting this ceiling when the infrastructure was effectively DDoS-ing itself just to keep up with the serialization requirements of these massive feature vectors.
Maintaining data consistency often requires logging feature vectors into a transitive key-value store to prevent online-offline skew. What are the operational risks of managing these high-throughput stores, and how do you mitigate the massive network bandwidth consumption generated by serializing these high-dimensional vectors?
The biggest operational risk is “online-offline skew,” which occurs if you try to fetch features at the moment of a user’s click rather than at the moment of the original recommendation; since features like follower counts are mutable, using fresh data with old labels poisons the training set. To mitigate this, we use a transitive key-value (KV) store with a short time-to-live (TTL) to freeze the features at the moment of inference. However, writing petabytes of these high-dimensional vectors to a distributed store is incredibly taxing on network bandwidth. We addressed this by rethinking the “write” side of the equation—specifically by moving away from eager logging for every ranked item and instead focusing on the items the user actually sees. This reduction in write throughput significantly lowers the serialization CPU load and prevents the network from being overwhelmed by data that would otherwise just expire untouched.
Many systems default to eager logging for all ranked items, even if a user only views the first few. How does shifting to a lazy logging approach change the serving pipeline, and what are the technical hurdles when implementing client-triggered pagination for these asynchronous data updates?
Shifting to lazy logging transforms the pipeline from a “push-all” model to a “just-in-time” model where we only persist features for the “head load,” such as the top six items a user is likely to see. The technical hurdle lies in the coordination between the client and the server; the client must fire a lightweight “pagination” signal as the user scrolls past the initial content. This signal then triggers the server to asynchronously serialize and log the next batch of features, such as items 7 through 15. This decoupling allows us to maintain a deep ranking buffer of 100 items for quality purposes while only paying the “storage tax” for the content that has a high probability of generating a label. Implementing this requires a highly responsive, asynchronous feedback loop to ensure that when a user scrolls quickly, the data is logged before any interaction occurs.
Standard storage schemas often repeat redundant user data across multiple item impressions within a single request. What steps are involved in moving toward a batched storage model to de-duplicate this information, and what impact does this have on CPU performance and bandwidth availability for training?
Moving to a batched storage model involves a fundamental shift from a tabular format, where every row is a single impression, to a structured format where user-specific and item-specific data are separated. Instead of writing a user’s age, location, and follower count 15 different times for 15 recommended videos, we store the user features once per request and link them to a list of unique item features. This simple de-duplication step reduced our storage footprint by more than 40% at Meta. Because storage in these massive systems isn’t passive—it requires CPU for compression, replication, and management—slashing the footprint directly frees up CPU resources. Furthermore, it significantly increases bandwidth availability for the distributed workers that need to fetch this data for training, creating a more fluid and efficient ecosystem.
Large-scale recommendation engines frequently accumulate thousands of features that may provide negligible predictive value over time. How do you systematically audit these features without degrading model accuracy, and what are the long-term benefits for inference latency when pruning these insignificant inputs?
Systematic auditing involves analyzing the weights the model assigns to each of the tens of thousands of registered features to identify those with statistically insignificant predictive value. We found that features like “recently liked content” are far more impactful than a user’s “age,” yet both consume the same resources to compute and store. By initiating a large-scale program to prune these low-value features, we were able to clean up a “digital hoarding” problem of over 100,000 distinct inputs. The long-term benefit is a noticeable reduction in inference latency, as the model has fewer inputs to process during the critical milliseconds of a request. This “spring cleaning” ensures that every byte processed is actually contributing to the user experience, rather than just wasting energy.
Beyond simply building smaller models, how can rethinking the “invisible” data plumbing layer lead to megawatt-scale energy savings? Could you walk through the process of identifying which parts of the data lifecycle—from computation to storage—are the most wasteful in a high-traffic environment?
Megawatt-scale savings come from identifying “write amplification” and redundant data processing in the lifecycle. The process starts by mapping out the journey of a single training example, from the moment a feature is fetched to the moment it is joined with a label and stored in a data lake like Hive. We identified that the most wasteful part was “eager logging,” where we were serializing data for 100 items when users only viewed six. By switching to lazy logging and de-duplicating storage schemas, we eliminated the energy wasted on CPU cycles and network I/O for data that was destined to expire. These “unsexy” plumbing optimizations allowed us to reduce annual operating expenses by eight figures and saved megawatts of power without needing to compromise the model’s actual intelligence.
What is your forecast for the future of AI energy efficiency as generative models continue to demand more computing power?
I believe we are entering an era where “efficiency” will be just as important a metric as “accuracy” or “engagement.” While the industry is currently obsessed with the massive energy demand of training larger generative models, the long-term sustainability of AI will depend on smarter engineering of the data lifecycle rather than just better hardware. We will see a shift toward more “aware” infrastructure that only computes and stores what is absolutely necessary, moving away from the “collect everything” mentality of the last decade. Ultimately, sustainable AI will be defined by our ability to optimize the invisible layers of the stack—turning the plumbing of data into a lean, high-performance system that supports growth without skyrocketing carbon footprints.
