Comprehensive Guide to Data Integration: Strategies and Tools

Data integration is a discipline whose goal is to meet the data consumption requirements of business applications and end users. It does that by leveraging tools, architectural techniques and best practices to achieve consistent access to and delivery of data across a wide spectrum of data sources and types in the enterprise.

Data integration is not a monolithic, independent component of the data engineering practice. It includes six facets representing a mix of technological and organizational dimensions:

• Strategy: How does your organization perceive data integration? Is it considered a liability or an asset? How do you leverage it to deliver added value and lead growth?

• Organizational model: What is the organizational structure supporting the data integration practice and its processes? How are responsibilities distributed?

• Styles and architecture: What data integration styles are available? How are they employed? Are they used separately or jointly to support multiple use cases? What is the data integration architecture?

• Technology and tools: What data integration tools are available? How are they selected and assigned to use cases?

• Governance: What governance framework is used for data integration? How can data integration be leveraged to perform governance tasks?

• Metadata: What metadata is collected? How is it organized? How is it used to improve and maintain data pipelines?

These areas are tightly connected, and all of them contribute to success in data integration.

To begin maturing your data integration practice, consider and assess each of the six dimensions on the following general model. (Gartner clients can access the more detailed Gartner Data Integration Maturity Model.) It is common for different dimensions to be at different levels of maturity.

• Level 1 — Ad hoc. The organization has no defined best practices and does not yet understand the importance of the data integration practice.

• Level 2 — Enlighted. The company understands the centrality of data integration but still has an immature practice, usually managed on a project basis, not at an enterprise level.

• Level 3 — Centralized. A central team manages data integration and selects data integration tools properly and cohesively. The practice develops best practices but the central approach creates challenges to scaling over time.

• Level 4 — Balanced. Data integration starts to be a common practice between teams, which use more distributed architectures and tools. Processes are optimized but still rely on human effort for expected results.

• Level 5 — Augmented. The practice leverages automated processes, using activated metadata to drive ML/AI models. Non-technical users are data literate and becoming more autonomous.

Visualize the results in a spider diagram to clearly see where you have more and less mature capabilities.

With results in hand, focus on achieving a baseline data integration maturity level 3 for each of the six dimensions before trying to add more advanced capabilities. This way, less developed dimensions don’t undermine more mature ones. Data integration programs in particular need to be at least a level 3 in strategy, styles and architecture, and technology and tools.

One argument for elevating the maturity level for the styles and architecture dimension — with corresponding technology and tools — is that doing so can produce concrete results quickly and with relatively little effort for common use cases.

Historically, data integration professionals have been aligned to one of three specific integration styles: data, events or applications. When approaching an integration project, teams default to their preferred integration approach and platform, and often waste time and effort dealing with constraints that would not exist if they had chosen the platform based on the needs of the use case.

Effective integration — delivered by more mature practices — creates the flexibility to choose the style that delivers best against the requirements of the use case.

For example:

• Data-centric integration is about moving data from one place to another and converting from one data model to another. It is most commonly associated with batch extraction, transformation and loading (ETL) tools, however it can deliver data in near real time. Data-centric integration focuses on maintaining data consistency, integrity and relationships across large datasets, so it typically works well on large volumes of data.

• Event-centric integration focuses on delivering events, or streams of events, to the endpoints where you consume them. Events are unidirectional, traveling from sources to sinks (data users). Event sources do not know what sinks are consuming their events, and sinks may not know which source emitted an event they consume.

• Application-centric integration focuses on invoking and composing functionality, rather than accessing data. Application integration flows usually have payloads that contain only the information needed for a single interaction or process, which makes the payloads significantly smaller than those used in data-centric integration. Application-centric integration can be implemented using request-response or messaging communication patterns, and can be synchronous or asynchronous.

Different approaches offer different levels of support for common enterprise data integration use cases, such as:

• Master data management. Synchronizing master data repositories and communicating changes.

• Application data synchronization. Synchronizing data between application stores.

• Reporting and data pipelines. Creating semantic layers in reporting platforms or moving data from one location to another within a defined time period.

• Stream processing and analytics. Looking for patterns within a stream of data, such as potentially fraudulent activity in a stream of financial transactions. Maintaining the position of an event in the stream and the content of nearby events is as important as the integrity of individual data items.

• Business-to-business integration. Exchanging information with external organizations and business partners.

• Application composition and API mediation. Creating new applications by orchestrating a set of calls to existing services.

• Digital process automation. Automating business processes to improve process speed, compliance and accuracy.

At some point, level 3 maturity — which organizations often reach using a centralized model — will begin to show its limitations. The central team won’t be able to cope with the volume of integration requests, resulting in a longer delivery cycle and suboptimal outcomes.

Progressing to level 4 maturity

To progress to level 4, focus on the organizational model and on data integration governance. Adopt a hub-and-spoke model that balances capabilities between a highly skilled central team and cross-functional satellite teams composed of data, technology and business talent.

The central-versus-local balance also applies to the types of tasks each team performs. The central team focuses on best practices and foundational implementations; the satellite teams focus on independently developing data pipelines. Implement specific programs to increase data literacy across the organization and use data integration tools to empower citizen users.

Adopt a federated governance model, partially decentralizing some responsibilities while keeping centralized ownership of shared policies. The goal is to manage the risks of decentralizing data integration.

Progressing to level 5 maturity

Level 5 maturity revolves around data integration efficiency and optimization. Organizations that achieve this level of maturity lower costs of implementing the data integration practice.

Metadata activation is the backbone of level 5. Activating metadata —data that describes the various facets of data assets — is a precondition to reaching level 5 capabilities for the other five dimensions of data integration. Activation starts by learning about the events underway and understanding their consequences to provide actionable recommendations about data pipelines and enable progress toward fully automated machine-managed data integration orchestration and optimization.

A second element of level 5 data integration maturity is a data fabric design and use for data integration. The goal of a data fabric is to make data management flexible, reusable and augmented through leveraging metadata. With a data fabric and its optimization features, data integration achieves lower operating costs in part by requiring less effort from those who design, deploy and maintain data pipelines.

These improvements make it possible to introduce advanced capabilities (such as ML, AI and GenAI), which results in higher independence from deep technical knowledge.