Big Data – unravel the big mystery

Contributed by: Sourabh Agarwala, CFA

Jim Barksdale, former Netscape CEO quipped, “If we have data, let’s look at data. If all we have are opinions, let’s go with mine”. Yahoo says data scientist is the sexiest job of 21^st century.

Big Data and Analytics are redefining the meaning of intelligence; transforming business, social interactions & the future of our society but is still unknown for many of us. IAIP was pleased to host Prof. Charanpreet Singh to unravel this big mystery for our members/candidates in our event held on 22^nd of March 2014 in Kolkata. The Professor kicked off the session by engaging the participants in the search for Flight MH370. He explained how Tomnod.com rallied the crowd to search satellite images and shapes for signs of the flight and DigitalGlobe processed geospatial big data to identify shapes like aircraft within 90 minutes to aid rescuers. This got the participants thinking and gave us a glimpse of how world and businesses are evolving.

Focusing on the massive deluge of data in the world of data, Professor noted that the amount of electronic data that exists in the world today is a phenomenal 2.7 zettabytes, equivalent to the storage required for more than 200 billion high-definition movies and is increasing at exponential rates. Other stunning facts shared were, there are 50 million tweets per day, 700 billion minutes spent on facebook each month, 72.9 products ordered on Amazon per second, 20 hours of video uploaded on Youtube every minute and so on. He demonstrated the power of Big Data and Analytics by interspersing the session with engaging real world examples of how analytics are used to transform a business or industry. The social media and analytics team of President Obama analyzed data on a large scale to model behavior of potential voters, create personalized communications, predicted donor behaviors to increase hit rate and mobilized volunteers to get people out to vote.

Amongst Professor’s favourite stories was about Harrah, an underperforming gaming casino company, which used predictive analytics to ascertain ‘pain threshold’ for each customer and intervened real time, before dissatisfaction happens – to keep the customer in the casino. Another case was about the largest SMB Lending Bank in US which analysed credit card transaction history, information from e-commerce web sites, social chatter, web site, rating sites like BBB etc. to develop an innovative risk assessment model driving a new line of business. Farecast developed machine learning technologies over big data to predict whether the online airfare will go UP or DOWN or STAY SAME in the future and was subsequently taken over by Microsoft,. These stories kept the group captivated in the world of data and analytics.

Professor apprised the participants on the evolution of data from Megabytes to Petabytes and from ERP (Enterprise Data) to CRM (Customer focused data) to WEB and now to Big Data. He stressed that “Big data” are huge in volume, is in various forms unstructured/structured/semi structured, is real time and highly complex. Explaining that data analysis deals with known unknowns while data analytics deals with unknown unknowns, Professor emphasized it is beyond the ability of typical database software tools to capture, store, manage, and analyze Big Data. Different analytical tools and language like Hadoop, R, MapReduce, bigML etc ere also discussed briefly. McKinsey estimates that there is a shortage of 140,000 to 190,000 people with deep analytical skills to fill the demand of jobs in the United States by 2018. Referring to these huge opportunities presented and skill sets required to tap it, Professor noted that data scientist is a person who is better in statistics than any software engineer and better at software engineering than any statistician. Professor also touched upon the usage of Marketing Analytics, Credit Risk Analytics, Fraud Analytics and Financial Analytics in the field of Banking, Insurance and Finance.

The amount of data available to us has truly reached unprecedented levels. The event made us aware of the need to appreciate the importance of tools to analyse such massive real time data and gave us a sneak peek on how to use data and analytics to give an edge to our career and our life.

– S A