The Data Avalanche

In the context of “Big Data”, many things have been said about the volume of data and its exponential growth. But what does it mean for us if data grows exponentially? The closest most of us come to grasping the explosion of data is when it comes to our home media library. Imagine your child is about to be born, and you decide to buy a DSR camera in order to capture every moment you consider worthwhile. The resulting library will grow to tens of thousands of pictures by the time your child reaches their fourth birthday. It will continue to grow with every month and every event.

As the camera collects high-resolution images, every single picture may account for more than 10 MB. The next media category in your media library is music: ever since online music stores became available, you have been purchasing music online and also converted your CD library into digital files. This library has grown to tens of thousands of songs where each song accounts for 3-5 MB. However, these songs are nothing compared to the TV shows you purchase digitally, as well as the family videos you shot with your HD camera. Each of the HD videos accounts for multiple gigabytes in your library. Overall, it is very easy for an average household to end up with several terabytes of data. And you don’t want to run the risk of losing any of your precious pictures, music or movies – so you start creating backups of your media library. Most of us are startled to realize how big the library has grown when you figure out how long it takes to copy your library over a network (one gigabit per second) to a backup storage device.

If you assume a maximum throughput of 80% of the bandwidth on the network, it will take you almost three hours to copy a single disk (1 TB = 1.000.000 MB; 1 Gbps = 125 MB per second; (1.000.000 MB / 0.8 * 125 MB/sec) / 60 seconds = 166.66 minutes). So no matter how fast your hard disk may read or write, it will take 3 hours to copy your disk over the 1 Gbs network to a backup server. If the backup server is on the Internet, it might take one week under the best of conditions to upload the data – if you have a good Internet provider with at least a 16 Mbs upload speed.

So much for private data, now let’s take a closer look at the enterprise: the data problems obviously compound exponentially Not only do businesses generate billions of documents every day, they also execute billions of transactions. Breweries running SAP software, for example, produce 150 billion liters of beer every year – generating billions of records originating from the business transactions, and thousands of sensors along the production process, which capture throughput, production quantity and quality, temperature, energy consumption, and generated waste. For every order from a customer, there is an invoice, a delivery document, a payment, and potentially many other documents surrounding the transaction.

Once the products have been shipped, people consuming the beer voice their opinion on Twitter or Facebook, which again represents interesting information for companies to collect and analyze. Multiple enterprise data streams add up to an incredible data avalanche, caused by data, triggering new data, which in itself triggers new data leading to exponential growth. If you have ever seen an avalanche racing down the slope of a mountain, you know that the power this avalanche unleashes is incredible. Now imagine you could leverage the incredible power that lies in the data avalanche rolling along with your business. What if you could put all the information that your enterprise generates into context and derive insights that help you improve your business along the way?

However, avalanches are not only impressive they are also incredibly dangerous: besides being built up by large amounts of snow, they are also incredibly fast! Before you even realize that there is something coming your way, the avalanche might have already rolled over you, and you are history. Looking at today’s corporate world, you can easily come up with names of companies that seem to have been overrun by an avalanche, before they realized that something was happening. Small companies may be entirely wiped out – do you remember the video store around the corner? While others might suffer significant pain – did you own a Walkman from Sony, or a Blackberry from RIM? The obvious question is whether these companies might have seen what was coming if they understood their environment and adapted to the changing business conditions.