The modern business world is increasingly recognizing that data is a vital asset for decision-making. However, time is not always allocated to implementing processes that ensure data quality until decisions are made based on inaccurate information, triggering strategies to prevent repeating the same mistake. We must learn from other organizations’ experiences and take the initiative to improve these processes before they negatively impact the company’s future direction.

Implementing data quality involves much more than simply having a tool with quality indicators. It’s a process that encompasses a strategy, a change in the culture of system users, and ultimately, the implementation of tools to calculate and monitor quality.

Without the first two components, it becomes merely a technological process that yields no significant results or provides no competitive advantage.

Establishing a plan within the company’s data strategy, defining ownership, roles, and continuous validation processes, ensures that data is truly useful for decision-making, changing the direction of a company, and simultaneously enabling continuous innovation through new products.

The technical definition involves that Data Quality measures how well a dataset is suited to its specific purpose, and measuring data quality levels can help organizations identify data errors that need to be resolved and assess whether the data in their IT systems is fit to serve its intended purpose. In the last layer inside a data warehouse, high-quality data is crucial for making informed decisions, conducting reliable analyses, and generating accurate reports.

When working and measuring data quality, we need to consider the dimensions. They are measurable characteristics used to assess and improve the fitness of data for its intended use on specific topics.

Using these dimensions helps organizations:

• Identify and address data issues systematically
• Ensure data meets business requirements and regulatory standards
• Build trust in data-driven decisions by providing a clear framework for evaluating and comparing datasets

Each dimension targets a specific aspect of quality.

• Completeness: Are there blank, null, or missing values? It measures whether all necessary information is present in a dataset at the record or attribute level and whether it does not contain missing values. We can view this metric from two angles: at the level of a specific column, or at the instance level, which includes a specific group of columns.
• Accuracy: Does the data reflect the real world? Data should be free from errors and reflect the true state of the information. Essentially, you’re evaluating how much the information accurately reflects the event or object described. The accuracy of your data is essential to avoid transaction processing issues and to generate accurate analytics results.
• Consistency: Is there a difference when comparing several representations of a thing against a definition? Maintaining synchronization between different databases is essential. This metric checks that different values of the same record conform to a given rule and that there are no contradictions. Data consistency ensures that the same information is represented uniformly across different records.
• Validity (Conformity): Does data conform to the syntax of its definition (format, type, range)? Validity is significant because if you store many invalidly formatted data, you won’t be able to analyze or compare it easily.

Examples:

% of values that match the date format.

% of values in an email column without the format xxx@domain

% of values in a phone number column with less than 9 digits

• Timeliness: Does the data represent reality from the required point in time? As the name implies, it refers to how up-to-date information is. It is the level at which data represents reality at the required time within an acceptable time frame and duration.  The timeliness of information is an important data quality characteristic because information that isn’t timely can lead to people making the wrong decisions, which costs organizations time, money, and reputational damage. When working with this dimension, an additional piece of information that is always useful is the number of records outside the time range, as well as the minimum and maximum dates.
• Integrity: Maintain all data quality metrics when moving information between systems (CRM-ERP-DWH-DASHBOARDS) or over time. This is important to measure because if your integrity degrades, your decision-making will suffer too.
• Uniqueness: Are there items recorded more than once? Confirms that the values in a column or for an entity are unique. Data uniqueness is the degree to which data values are distinct and non-repetitive. Or simply identifying and removing duplicate information and records. An event or entity should be recorded only once. Duplicate data should be avoided, as it can lead to double-counting or misreporting.
• Compliance: Does our data comply with international data standards in terms of both information and security?

There are various tools on the market for managing data quality; however, I want to mention one I’ve been working with for some clients: Qlik Talend. It has several features that streamline data governance, but I want to highlight that within the Qlik Data Catalog, you can define different metrics and dimensions to evaluate data quality. Additionally, the platform allows you to assign a Trust Score to the dataset or table.

The Trust Score is “a comprehensive measure of an organization’s AI-readiness, calculated based on six key data principles: Diversity, Timeliness, Accuracy, Security, Discoverability, and ML/LLM Consume-ability. By evaluating these factors, the Qlik Talend Trust Score ensures that data is not only high-quality and secure but also optimized for AI-driven insights”.

It can be updated at any time to revalidate the metrics and values, and most importantly, it updates within seconds. While manual calculation is possible, a tool that performs this calculation for any dataset and provides visualizations would be of interest to many companies seeking a tool that allows them to move data while simultaneously offering data governance capabilities.

Whether you are just starting out in these processes or have already made progress and need advice, we would be happy to discuss further details related to tools, processes, or the data strategy section focused on data quality.

Victor Gomez

Principal Data Consultant