Tag: Data observability

January 19, 2024

Do you know the costs of poor data quality? Below, I explore the significance of data observability, how it can mitigate the risks of bad data, and ways to measure its ROI. By understanding the impact of bad data and […]

November 15, 2023

“Data observability” can be described as the practice of monitoring the “health and state” of data pipelines in your system. This practice encompasses some technologies and activities that enable business operators to identify, examine, and solve data-related problems in near […]

June 19, 2023

In part one of this article, we discussed how data testing can specifically test a data object (e.g., table, column, metadata) at one particular point in the data pipeline. While this technique is practical for in-database verifications – as tests […]

June 14, 2023

Dealing with data and databases is laden with a multitude of challenges, often characterized by the questions, “What happened to my data?” and “Why is this data all wrong?” Whether data is stale or unreliable, the solution lies within the […]

June 1, 2023

You can’t manage what you can’t measure. Just as software engineers need a comprehensive picture of the performance of applications and infrastructure, data engineers need a comprehensive picture of the performance of data systems. In other words, data engineers need […]

May 26, 2023

Suppose you’re in charge of maintaining a large set of data pipelines from cloud storage or streaming data into a data warehouse. How can you ensure that your data meets expectations after every transformation? That’s where data quality testing comes […]

April 28, 2023

In our rapidly evolving microservices and digital landscape, keeping track of what is happening with these services and making well-informed decisions is paramount for business success. With observability, organizations can navigate complex challenges and unlock their full potential in the […]

March 2, 2023

Just as vendors rely on U.S. mail or UPS to get their goods to customers, workers count on data pipelines to deliver the information they need to gain business insights and make decisions. This network of data channels, operating in […]

February 24, 2023

It’s a new year, but we’re still facing many of the same challenges as in 2022. Businesses are preparing for greater economic uncertainty, often looking for creative ways to keep costs low while still meeting goals for the year. By […]

October 17, 2022

Data observability is a relatively new discipline in the fields of data engineering and data management. While many are familiar with the longstanding concepts of observability and monitoring in enterprise IT networks and infrastructure, data observability has only really come […]