Data Quality: Best Practices for Accurate Insights

Data quality refers to the usability and applicability of data used for an organization’s priority use cases — including AI and machine learning initiatives. Data quality is usually one of the goals of effective data management and data governance. Yet too often organizations treat it like an afterthought.

Why is data quality important to the organization? In part because poor data quality costs organizations at least $12.9 million a year on average, according to Gartner research from 2020.

A number of common challenges hinder organizations and make it difficult for them to address their data quality issues, such as:

• Growing regulatory requirements from government or industry. These include GDPR and CCPA, which restrict how organizations manage personal data and make them accountable for any personal data they hold.
• Inconsistency in data across sources. Named as the most challenging data quality problem, according to Gartner, is the result of having data stored and maintained in silos with significant overlaps, gaps or inconsistencies. If they are not connected, data standardization becomes much harder.
• Lack of resources. Organizations may have a data quality program in one department or in one data domain but cannot scale it due to a scarcity of skills, experience and resources.
• Lack of ownership. While business leaders agree that data quality matters, they do not view it as their responsibility, nor do they necessarily understand how data related to their domain relates to broader enterprise outcomes. Yet data quality is a business discipline. How business users enter data, use data and manage the data are the foundations of data quality. Ownership and collaboration among stakeholders are essential.

Defining the scope of your data quality program is essential for setting expectations and establishing priorities. Data and analytics leaders cannot and should not aim for data quality everywhere because not all data is equally important. Setting the scope too wide taxes resources without producing business benefits.

Instead, set the scope of your data quality efforts using the following two views into your organization’s data use cases:

Map the organization’s business use cases and data resources along the dimensions of value and risk. Those with the potential to produce high value from high data quality, and face severe risks from low data quality, get priority from the data quality program across a wide scope of activities.

Categorize data sources based on whether the organization’s use cases leverage the data at a centralized, regional or local level.

According to Gartner surveys, 59% of organizations do not measure data quality. That makes it difficult to know what a lack of quality is costing you and how much improvement your data quality management program is producing.

So how do you measure and improve data quality?

By taking steps to understand the dimensions of data quality, choosing the dimensions and corresponding metrics that matter to you, applying data profiling and analyzing the results.

Organizations measure data quality based on the data quality dimensions that matter to them. The nine most common are:

• Accessibility. Data is available, easily retrieved and integrated into business processes.

• Accuracy. Data value accurately reflects the real-world objects or events that the data is intended to model.

• Completeness. Records are not missing fields, and datasets are not missing instances.

• Consistency. Data that exists in multiple locations is similarly represented and structured.

• Precision. Data is recorded with the precision required by business processes. 

• Relevancy. Data is applicable to one or more business processes or decisions.

• Timeliness. Data is updated with sufficient frequency to meet business requirements.

• Uniqueness. Each data record is unique based on how it is identified.

• Validity. Data conforms to the defined business rules/requirements and comes from a verifiable source.

Data profiling is the process of measuring data quality by reviewing source data and understanding structure, content and interrelationships to identify potential data quality problems.

There are two types of data profiling:

• Column-based profiling provides statistical information about data.

• Rule-based profiling validates data against business logic or rules.

Data profiling capabilities are available in commercial data quality tools or in open-source data profiling tools. Organizations can also write their own custom-built queries to conduct data profiling on their data sources.

The market for data quality technology has evolved in recent years to include an array of solutions that enable organizations to accelerate time to value for their data assets, reduce risk and increase competitive advantage. In particular, augmented data quality solutions represent a fundamental shift in solving data quality issues using active metadata, AI (e.g., NLP) and graph technologies.

Data and analytics leaders should note that data quality tools do not exist alone. Instead, organizations deploy them to support a broader set of data management processes or use cases, like data integration or master data management. Thus when evaluating which solution is right for your needs, data and analytics leaders should weigh the relative importance of each capability to the use cases they hope to support.

Overall, Gartner emphasizes the following 10 critical capabilities organizations should consider when evaluating for data quality solutions:

• Profiling

• Parsing, standardizing and cleansing

• Analytics and interactive visualization

• Matching, linking and merging

• Multidomain support

• Business-driven workflow and issue resolution

• Rule management and data validation

• Metadata and lineage

• Monitoring and detection

• Automation and augmentation