Dwarfing the Cost of the Original ETL Tool

This is the continuation of the transcript of the DM Radio show "Avoiding Bottlenecks and Hurdles in Data Delivery.". InetSoft's Principal Technologist, Abishek Gupta, joined industry analysts and other data management software vendors for a discussion about current issues and solutions for information management.

Jim Ericson: Or is this spike useful even? I mean, I don’t want to hijack the conversation, but for example, you know, you see Sprint advertising that they won't slow down your telephone data downloads like the other providers will or charge you more for them, but that really has no bearing on whether this data is being used constructively, or if it's useful to begin with. I mean it's just the flip side of looking at bandwidth as a way of rationalizing the value of data or something I don’t know.

Eric Kavanagh: Yeah. Well that’s a good point. So David I have just another couple of questions here and there for the rest of the segment. What are some of the biggest mistakes you have seen organizations make when trying to circumvent a bottleneck that they find? Maybe batch processes or what I used to think about batch windows being missed, that kind of thing, what are some mistakes that people make when trying to resolve those problems?

View a 2-minute demonstration of InetSoft's easy, agile, and robust BI software.

Why an ETL Process Can Become Very Slow

David Inbar: It’s so interesting. There are several things you can often point to which are the originators of the problems. One of them, and I will just lay it straight out there, is poorly written software. The process runs slowly, and they just assume because it runs slowly it must be the hardware that’s the problem, and so they go out and write bigger checks for bigger hardware and hope to get around the problem that way.

They assume that it just can't run any faster or can't be made to run any faster. The other one very often originates earlier in the whole process where they really didn’t understand what was in the data or the quality or reliability of the data. And some what they think of is normal starts to creep in over time and potentially hijack the ETL processes in a non expected way, but that really they just didn’t see as bad at the beginning.

Eric Kavanagh: Yeah, that’s a good point. And, yeah, the other thing I was thinking of is, I remember we had a show I think it was the beginning of this year, in fact, Rick Sherman was on it from Athena Solutions, and he made a really good point that a lot of times when hand-coding ETL, people will find that everything works fine at first, but then as you start adding more systems, and the data volume grows, and you have more people who want things over time, that hand-coding can get very, very brittle. That’s why he was suggesting that really if you are dealing with enterprise caliber systems and real business needs, you need to look at some of the data management tools on the marketplace and realize that hand-coding although effective in many cases can be real danger down the road, right?

David Inbar: Yes, it is usually is, and it is partly because of the scalability but also because the people who originally wrote it have moved on, and you are then maintaining what was originally a fine concept somewhere in that wetware that you referenced between somebody’s ears, and which is now far from transparent and far from obvious. But the downstream costs of things like ETL and data integration, synchronization and everything else are enormous for most organizations and usually dwarf the cost of the original either custom development or ETL tool acquisition.

Avoiding Bottlenecks in Data Delivery

Eric Kavanagh: Talking all about avoiding bottlenecks in data delivery and data access, to Jim’s point, the consumption side, business users want information, and they get spoiled right? When you get the information you want, you get real happy, and then when you don’t get it, if somebody changes a process, and now you are not getting that data feed anymore that can be rather disruptive I am sure.

Jim Ericson: More, more, more.

Eric Kavanagh: That’s it. More, more, more, baby, just give it to me. Jim Ericson: I think I had pushed that to a dead-end when I was talking about the consumption side of this, but it's interesting how people are treating data now. As you know, I look at my Amazon bill for this stuff I store up there. I looked at the cell bills that I run up when I accidentally have my roam on in Toronto, and you know AT&T is charging $15 a megabyte. You start to see the supply and demand of some of this as it starts to play out commercially because it's been, you know, the internet is free, and the access is free, but we are starting to put a value on some of that, too. So it's a good sign, right?

Eric Kavanagh: They are the phone company. They always find a way to get you.

Jim Ericson: That’s right. They are the phone company. That’s the answer. That is the answer, oh, they are the phone company.

Eric Kavanagh: It's a circular argument, great sign for a debate on that topic.