Big Data refers to the data that is quite simply so large that it can not be processed by using traditional techniques and methods. The concept of big data gained momentum in the early 2000s when industry analyst Dough Laney segmented the definition of big data as three V’s:
- Volume: Volume of data is the amount of data. It is estimated tat around 2.5 quintillion bytes of data is created per day, which would result in around 40 zettabytes of data being created by 2020 which would be almost 300 times from that of 2005. This is also evident from the fact that nowadays it is not uncommon for large companies to have terabytes or petabytes of data in servers and storage devices. This data helps to shape the future of a company and its actions, all while tracking its progress.
- Velocity: The growth and impact of data has changed the way we see it. Velocity essentially means the rate at which data is being gathered.
- Variety: In earlier times data could not be captured structurally. In the world of big data, data usually comes in various unstructured forms like social media posts, server log data, photos, audio, video etc.
The above definition was further modified to include four more V’s which are:
- Variability: The meaning of words in unstructured data can change based on the context.
- Veracity: Veracity deals with exploring a dataset for quality data and systematically cleaning the data for data analysis.
- Visualization: After the data is analysed 3D and 2D plots are drawn for end-users to understand and act upon.
- Value: Data must be combined with rigorous processing and analysis to be useful.
Importance of big data:
- Some big data tools such as Hadoop can help with cost reductions for businesses when it comes to storing large amounts of data.
- By analysis of big data, one can understand current market conditions.
- High-speed tools and in-memory analytics help identify new sources of data which help organizations analyze data immediately and make quick decisions based on what they have learnt.
- Big data tools can help with sentiment analysis.
- Big data analytics can affect all business operations. It includes meeting customer expectations and ensuring marketing campaigns are impactful.
- Big data can help businesses innovate and re-develop their products.
Applications of Big Data:
Big Data has many application spread across various industries, some of which are listed below:
- The SEC(Securities Exchange Commission) uses big data to monitor financial market activity.
- Retail traders, hedge funds etc. use big data for trade analytics in high-frequency trading, sentiment measurement etc.
- Spotify uses Hadoop to collect data from its millions of users worldwide and give personalized recommendations to each of its users.
- Some hospitals are using data collected from cell phone apps to allow doctors to use evidence-based medicine as opposed to administering several lab tested medicines to patients that visit the hospital.
- An Australian university has deployed a learning and Management System which tracks when a student logs onto the system, how much time is spent on different pages in the system as well as the overall progress of a student over time.
- Big data is being used in solving manufacturing challenges and to gain competitive advantage.
- In the public service sector, big data is used in energy exploration, financial market analysis, fraud detection, health-related research and environmental protection.
- The FDA(Food and Drug Association) uses big data to study patterns of food-related disease and illness.
- During insurance claims management, predictive analysis from big data is used to offer faster service since massive amounts of data can be analysed mainly in the underwriting stage.
Evidently, Big data has become a huge part of our daily lives. Data Science and Analytics are evolving fields with huge potential. It is essential for professionals to be aware of big data and the terms related to it.