Big Data challenges and benefits

In last week’s post – the first of a two part discussion on Big Data – an introduction to Big Data was made, including the main sources, currently, of Big Data. After explaining the state of the art of this field, in this second post the challenges and possible benefits of the new reality of Big Data will be tackled.

Data storing

Hard drive capacity is not increasing fast enough to keep up with the explosion of digital data world wide. A 50-fold increase in global data is forecast by 2020, but hard drives are likely to grow only by a factor of 15, even considering all the latest advances in data systems. No storage technology has been developed that can scale up to the pegabyte and beyond. Today piling conventional hard drive upon hard drive is the common procedure to handle data sets of this size.

However, there are a couple of research lines addressing this problem, both in their infancy:

Diamonds and quantum computers

Diamonds are not just for jewellery. Researchers at the Max Planck Institute of Quantum Optics and Caltech (Harvard), were able to store a quantum state in a diamond crystal for more than second, at room temperature. Doesn’t sound like much, but in quantum physics, that’s a lifetime, and a big step toward building a quantum computer with magnificent storage capacity.

Bacteria

The Chinese University of Hong Kong has recently discovered how to store encrypted data in the DNA of E. coli bacteria. Such “biostorage” could be used for future Big Data storages, especially considering a single gram of the bacteria could hold as much as 450 conventional 2-terabyte hard drives.

Big Data benefits

As explained in the first Big Data post, living in a world where economies, political freedom, social welfare and cultural growth increasingly depend on our technological capabilities, big data management and, most importantly, the knowledge that can be obtained from it, has enormous potential to benefit individual organizations. While in production teams it is increasingly asked to “do more with less”, in relation to data we are asked to “do more with more”.

There remains much unexplored terrain in Big Data and traditional databases and analytical platforms are not able to meet the challenges required of it. Capturing, filtering, storing and analysing Big Data flows has huge potential outcomes: Innovative new products, services and business models, better decision making, better productivity and higher revenues.