What is the role of a data warehouse in data integration? Data warehouses have evolved since the days of the all encompassing enterprise data warehouse in the early 1990s. Rather than defined as a sealed database and data warehouse, we should think of it as an overarching architectural approach and the processes enabling information access and delivery.
It encompasses the workflow of data from wherever it is created to where ever it is consumed by business people when they perform analysis or review reports. It still plays a vital role in data integration, but data integration can happen without a data warehouse today.
What about the definitions of Master Data Management, or MDM, and Customer Data Integration, or CDI? Well, master data management, which is also referred to as referenced data management, is an old concept with a new name. We have been dealing with the inconsistent reference data such as products and parts and customers for over a decade now. Now those efforts to make those data consistent is called MDM, or master data management.
| #1 Ranking: Read how InetSoft was rated #1 for user adoption in G2's user survey-based index |
|
Read More |
Consistent Customer Data Across The Enterprise
That is also being called conforming dimensions for those that have been involved in data warehousing and data modeling. CDI, or Customer Data Integration, is consistent customer data across the enterprise. Again, this is a not new concept, but it is a new name, CDI. Remember 360 degrees around the customer, customer relationship management folks? The CRM folks keep inventing new terms of integrating customer data, because they have to deal with customer applications such as Call Center Campaign Management and Sales Force Automation that tend to keep customer data fragmented.
Metadata, what does that mean? Metadata is data about data. Basically what it is, is defining what the data means both in the technical sense as well as in business terms. It defines the data from when it was created from its data sources through its transformation into information and then finally how it is consumed by business.
Lastly, how about data quality in the integration space today? Often times people assume data quality simply means eliminating bad data. The data is missing or inaccurate or incorrect. Now that is certainly important, but is not the only issue that most people get into with data integration and data quality. It is not usually the case that data is missing, but that it is not comprehensively consistent across various applications that the data is sourceed from.
Okay, so we have just gone through a lot of terms here, and people can go back may be and listen if they have missed anything. Next I would like to talk about what drives a company to pick one solution over another. Why would somebody chose to use ETL over EII or EAI or data mashup?
 |
View a 2-minute demonstration of InetSoft's easy, agile, and robust BI software. |
Picking an ETL Tool
When you pick an ETL tool, then you are dealing with large numbers of source systems. When you have inconsistent data, especially reference data or master data across the source systems, or when you need to accumulate historical data for trending or analysis, all of those conditions really imply physical integration of the data, so ETL would be your choice.
Now when you are trying to exchange information between applications is to get real time information, such as getting the information from your banks or Amazon or FedEx, you probably use EII, EAI, or SOA and really the reason you pick one of the other would be which one is supported in the application that you are trying to integrate.
There are a ton of data integration tools and vendors out there. Without naming names what should people look for as they assess data integration technology? All right, there are three areas that you should look at. First off, quite simply, the data integration job has to get done so pick the tool that handles all the data sources and application sources as well as the targets that you need to handle in your enterprise. You want a tool that makes your data integration efforts much more productive than custom coding.
So you want to look for products that have workgroup code management so more than one person can manage the code. Second, you want the ability to create documentation of the work flow of your data integration. And finally you want to have products provide logging or loading and restart capabilities. And all these are important aspects of the operational flow of your data integration efforts.
Finally, the third area, you want to look for vendors who are starting to incorporate an overall data integration platform. Several vendors are incorporating ETL, EII, and EAI under the covers so that you don’t have to go out and pick individual tools. You can get them all in one platform. Also these vendors are starting to add data quality, data profiling and meta data tools as part of the overall platform, It’s just like getting an office suite, where you’re getting a data integration suite and platform that helps and improves your efforts overall.
What advice would I give people who face with data integration challenges or about the plan in integration project?
I can think of a few areas that I will have them consider. First, they need to have a program that is business driven. IT needs to drive the “how,” but business needs to be involved to drive the “what” you are trying to do. The business and IT groups have to work closely together.
You should develop an overall data integration program determining the right steps and as far as processes go, data governance is achieved. The business needs to own the data and responsibility for data definitions, how to consolidate the data and make the inconsistent data consistent, with IT knowing how to get it implemented. Finally, you want to use the data integration tool not custom code. Achieving data integration probably seems overwhelming to many companies, but it is a journey that you take one step at a time, but it is a journey that provides significant business benefits. It’s well worth the long road.