The Trend of Big Data Analytics

This is the continuation of the transcript of the DM Radio show "Avoiding Bottlenecks and Hurdles in Data Delivery." InetSoft's Principal Technologist, Byron Igoe, joined industry analysts and other data management software vendors for a discussion about current issues and solutions for information management.

Eric Kavanagh: Let’s bring in our good friend Philip Russom from TDWI.

Philip Russom: Hey, Eric.

Eric Kavanagh: So you have this great concept, I know, I have done some webcasts with you on it, and we have talked about it on DM Radio before, but you track mega trends, which are these big broad trends that impact all sorts of different bits and pieces of the market place. And one of them has to do with data integration, right? Do you want to talk about that?

Philip Russom: Yeah, you know, actually the mega trends are really broad, and they tend to influence everything, not just data integration, but databases, applications, just presently hit everything in IT. And for years one of the leading mega trends, at least one that was the most pressing issue, and for some organizations it was actually a crisis level, that would be the increase of data volumes. And if you think back to when the first explosions of data volumes hit us in the late 90s, early 2000s, you know a lot of organizations weren’t prepared for that, and a lot of the technology for managing data wasn’t there.

Data Flow Bottlenecks

We are here to talk about bottlenecks, and you know, one of the biggest bottlenecks is just trying to move very large data sets through a processing data flow and into whatever the delivery mechanism is, whether it’s a report or some of the new steps Jim was just talking about.

I would add to Jim’s list. I would add mobile BI. We are seeing mobile BI really picking up right now. So you are probably seeing old cartoons where the snake swallowed an elephant. You see this big lump in the snake that looks like an elephant. Well you know it’s kind of like your network cable, right? The big data is coming out through the cable.

So that one of the mega trends is just simply about a kind of scalability crisis for some organizations around data volume, but I have to say that crisis has been subsiding in recent years, about the last five years. And it’s ironic, we are now in the age of Big Data, but for organizations that have kept up-to-date with their IT and data infrastructure, there are modern technologies for that which can scale and deal with very large data sets. So giant data volumes is not the crisis it was, but it’s still a big issue, and it’s still probably the number one mega trends that’s creating the most bottlenecks.

Read what InetSoft customers and partners have said about their selection of Style Scope for their solution for dashboard reporting.

Real-Time Data Integration

Eric Kavanagh: It’s interesting the change in the marketplace. We actually talked a few weeks ago in a fairly philosophical show about how things really seem to be coming around in a fairly significant way these days such that organizations are increasingly looking to kind of move out from that traditional world of offline analytics and instead tie those analytics back into our personal systems. We all know things like Netflix with next best offer and stuff like that, but there are lots of other things you can do I mean think in terms of supply chain management when a particular part, for example, with the tragedy out there in Japan, is not going to be available because there was an earthquake or tsunami or something like that. If you are really on your game in terms of real-time data integration or tracking various things, various feeds, you can respond a lot more quickly than other organizations that are not, right?

Philip Russom: Yeah, it’s true and, you know, TDWI had a conference about two weeks ago on Big Data Analytics, and we had a lot of users stand up and talk about what they are doing. Pretty much what you were saying, Eric, that there are lot of ways to collect and use data and actually turn it around and turn it into an actual product or service, right. So this is typically done by some Internet-based company, right? Some old brick and mortar company typically doesn’t have the nimbleness to do this.

For instance, we had a guy from LinkedIn talk. You know we are all members of LinkedIn, and you know how LinkedIn can make recommendations for people that you probably know or you are removed from by a couple of degrees of separation, and therefore you should propose a link with them, right? But they are actually processing massive amounts of data.

They have a profile of every user as they call us. They do update your profile in terms of possible links in a near real time fashion. So they are processing terabytes of Web data a day to update these recommendations. So there is a service you can see right there. And so you know they are trying all kinds of technologies to make this data move quickly, process it analytically like you said.

Read the top 10 reasons for selecting InetSoft as your BI partner.

And then also something else we can talk about, it’s just closing the loop. A lot of times your analytics has some output. Can you really close the loop and bring that and apply it immediately. So yeah, a lot of big challenges are there, and it involves processing and moving a lot of data and then delivering some sort of analytic product or other database product.