s5.07.09 - introduction to big data
Explore Big Data's Volume, Velocity, Variety. Understand why relational databases fail, and analyse its real-world implications.
Hello future data scientists! Today, we are stepping into the realm of 'Big Data'. Think about how much information is generated every second - from your Instagram likes and TikTok scrolls to weather sensors and scientific experiments. Traditional databases, which you've mastered, simply cannot cope with this tsunami of information. In this lesson, we'll discover what makes this data 'Big' and why it demands a whole new way of thinking about storage and processing. This understanding is critical for anyone aiming to become a Data Analyst or AI Engineer.
Learning Outcomes
Factual (Knowing that)
Conceptual (Knowing why)
Procedural (Knowing how)
Recall the definition of Big Data.
State the three main characteristics (the '3 Vs') of Big Data: Volume, Velocity, and Variety.
Identify examples of different data types (structured, semi-structured, unstructured) that contribute to Big Data.
Conceptual (Knowing why)
Explain why traditional Relational Database Management Systems (RDBMS) are often insufficient for managing Big Data.
Analyse how each of the '3 Vs' presents specific challenges for data storage, processing, and analysis.
Distinguish between scenarios where traditional databases are appropriate and where Big Data solutions are required.
Procedural (Knowing how)
Apply knowledge of the '3 Vs' to characterise different real-world data scenarios.
Evaluate the suitability of a relational database for a given large-scale data problem.
The Deep Dive
Introduction to Big Data
The term Big Data refers to extremely large data sets that may be analysed computationally to reveal patterns, trends, and associations, especially relating to human behaviour and interactions. However, it is not just about the volume of data; it also encompasses the speed at which it is generated and processed (velocity), and the diverse forms it takes (variety). Traditional relational databases (RDBMS), which you have studied extensively, were designed for structured, well-defined data sets that fit neatly into tables with predefined schemas. When confronted with the scale and complexity of Big Data, these systems often hit severe limitations.
The Three Vs of Big Data
The defining characteristics of Big Data are often summarised by the '3 Vs': Volume, Velocity, and Variety. Understanding these three aspects is crucial for grasping why new approaches to data management became necessary.
NUMBER 1: Volume
Volume refers to the sheer amount of data generated and stored. In the era of Big Data, this is often measured in petabytes (1015 bytes) and even exabytes (1018 bytes), far exceeding the capacity of a single server or traditional data warehouse. Consider the data generated by a major social media platform like X (formerly Twitter). Every tweet, like, retweet, and direct message contributes to a massive data pool that grows by terabytes daily. Storing and processing such quantities requires distributed storage and processing systems that can scale horizontally, meaning they add more machines to handle the load, rather than upgrading a single, more powerful machine.
Volume
NUMBER 2: Velocity
Velocity refers to the speed at which data is generated, collected, and processed. In many Big Data applications, data arrives continuously and needs to be analysed in near real-time. Examples include financial trading systems, where market data streams in at high speed and needs immediate analysis to make trading decisions, or IoT (Internet of Things) sensors sending back readings every second from millions of devices. Processing this data often requires stream processing architectures rather than traditional batch processing, where data is collected over time and processed in large chunks. The challenge is not just storing the data, but making sense of it as it flows in.
Velocity
NUMBER 3: Variety
Variety describes the diverse forms of data that exist. Unlike traditional databases that primarily handle highly structured data (data that fits neatly into predefined tables and columns, like a spreadsheet), Big Data often includes:
Structured data: Data that resides in a fixed field within a record or file, such as data in a relational database.
Semi-structured data: Data that does not conform to a formal data model but has some organisational properties, like XML or JSON documents. These tags or markers can separate semantic elements and enforce hierarchies of records and fields within the data.
Unstructured data: Data that has no predefined organisational structure and cannot be stored in a traditional database format, such as text documents, emails, social media posts, audio, images, and video files.
The challenge with variety is that traditional RDBMS are designed for structured data and struggle to store, query, and analyse unstructured or even semi-structured data efficiently. A Software Engineer might use different tools and techniques for each type of data.
Variety
Why Traditional Relational Databases Fail
Relational Database Management Systems (RDBMS) like MySQL or PostgreSQL, while excellent for many applications, face significant hurdles when confronted with Big Data:
Rigid Schema: RDBMS require a predefined schema (a blueprint of tables, columns, and relationships) before data can be stored. This rigidity is problematic for Variety, especially with unstructured data or rapidly evolving data schemas. Modifying schemas in a large, active RDBMS is complex and time-consuming.
Scalability Limits: While RDBMS can scale vertically (adding more CPU, RAM, storage to a single server), this eventually hits physical limits and becomes extremely expensive. Horizontal scaling (adding more servers) is challenging for RDBMS due to the complexities of maintaining data consistency and transactional integrity across distributed machines. This impacts handling extreme Volume.
Performance with High Velocity: The ACID (Atomicity, Consistency, Isolation, Durability) properties that ensure data integrity in RDBMS can become performance bottlenecks at very high data ingestion rates. The overhead of ensuring strict consistency for every transaction can slow down systems that need to process millions of operations per second.
Inefficiency with Unstructured Data: RDBMS are simply not designed to natively store and query unstructured data like images, audio, or free-form text. While they can store pointers to external files, the actual content cannot be directly searched or processed within the database itself, limiting the scope for comprehensive analysis.
This inability of RDBMS to gracefully handle the 3 Vs led to the development of alternative data storage and processing paradigms, often grouped under the umbrella term NoSQL databases (Not Only SQL), which we will explore in later lessons. For a Data Analyst, knowing when to choose a relational vs. a Big Data solution is a critical skill.
Don't just accept the text. Write down three high-level 'Why' or 'What if' questions about these notes that aren't explicitly answered here. Use these to challenge your teacher or guide your independent study.
Comprehension Questions
Answer these questions on your lesson sheet.1
What is the primary definition of 'Big Data'?
2
List and briefly define the '3 Vs' that characterise Big Data.
3
Explain two reasons why traditional Relational Database Management Systems (RDBMS) struggle to manage Big Data effectively.
4
In the context of Big Data, what does 'Volume' specifically refer to, and why is it a challenge?
5
Describe what 'Variety' means in relation to Big Data, giving examples of different data forms.
Task Characterising Real-World Data
0
Get Organised!
Ensure you have your lesson sheet for recording your analysis.
Access to a web browser for quick research on real-world data volumes.
1
Understand the Scenarios
Look closely at the three distinct data-generating scenarios below. For each scenario, you need to identify and explain which of the '3 Vs' (Volume, Velocity, Variety) are prominent.
Consider a network of millions of weather sensors worldwide, each transmitting temperature, humidity, and pressure data every 5 seconds. This data is collected continuously.
On your lesson sheet, identify and explain the 'Vs' present in this scenario.
Would a traditional relational database be suitable for storing and processing this data? Justify your answer with reference to the limitations discussed in the notes.
Consider a university with 50,000 students, each having records for personal details, course enrolments, grades, and fee payments. This data is updated periodically (e.g., at the start of term, after exams).
On your lesson sheet, identify and explain the 'Vs' present in this scenario.
Would a traditional relational database be suitable for storing and processing this data? Justify your answer with reference to the limitations discussed in the notes.
Imagine a platform similar to YouTube or TikTok, where users upload millions of videos daily, along with comments, likes, view counts, and user profiles. Videos can be of varying lengths and formats.
On your lesson sheet, identify and explain the 'Vs' present in this scenario.
Would a traditional relational database be suitable for storing and processing this data? Justify your answer with reference to the limitations discussed in the notes.
3
Compare and Contrast
Compare your analysis for the Weather Sensor Network and the University Student Information System. Explain why one is a stronger candidate for Big Data solutions than the other, focusing on the combination of the '3 Vs'.
Outcome: You will have analysed three distinct data scenarios, identified their Big Data characteristics, and justified the suitability of relational databases for each, enhancing your critical data architecture decision-making skills.
Hungry for more?
Delve deeper into the practical implications of Big Data and the technologies designed to handle it.
Search the web: Real-world Big Data case studies Volume Velocity Variety
Ask an AI: Click on the carefully written prompt below to learn more...
Explain what a NoSQL database is and how it differs from a relational database, specifically addressing its ability to handle the Volume, Velocity, and Variety of Big Data. Limit response to 200 words. Audience: A-Level Computer Science student. Tone: Informative, technical. Constraints: Explain key differences and advantages for Big Data. NO intro, NO outro, NO deviation from the topic, NO follow-up questions
The Big Data Ecosystem: Research popular Big Data technologies like Hadoop, Spark, and MongoDB. What problems do they solve?
Sticky Note Challenge
On a sticky note, summarise the single biggest challenge that Big Data poses to a traditional relational database, and which of the '3 Vs' is most responsible for that challenge.
Last modified: November 5th, 2026
