Ch1. Big Data Fundamentals — The 5Vs, Hadoop, and the Data Pipeline

What Is Big Data?

Big data refers to datasets so large, fast-moving, or diverse that traditional database tools cannot capture, manage, and process them effectively.

The 5Vs

Volume: Data too large for conventional databases. Think petabytes and exabytes.

Velocity: Data generated and processed at high speed — real-time streams, not batch overnight.

Variety: Three data types:

• Structured: Fixed schema — relational databases, CSV
• Semi-structured: Flexible schema — JSON, XML, HTML
• Unstructured: No schema — text, images, video, audio

Veracity: Trustworthiness and quality of data.

Value: The business insights extractable from the data.

---

The Big Data Pipeline

Collection → Storage → Processing → Analysis → Visualization → Decision

Collection

• Batch: Periodic bulk collection (Sqoop, Flume)
• Streaming: Continuous real-time ingestion (Kafka, Kinesis)
• Web scraping: Collecting public web data

Storage

• HDFS: Hadoop Distributed File System
• Data Lake: Raw data in original form (AWS S3, Azure ADLS)
• Data Warehouse: Structured analytical data (Redshift, BigQuery)
• NoSQL: Unstructured/semi-structured (MongoDB, Cassandra, HBase)

Processing

• Batch: Hadoop MapReduce, Apache Spark
• Streaming: Spark Streaming, Apache Flink, Kafka Streams

---

The Hadoop Ecosystem

Component	Role
HDFS	Distributed file storage
MapReduce	Distributed batch processing
YARN	Cluster resource management
Hive	SQL-like queries on HDFS
Spark	In-memory high-speed processing
HBase	NoSQL column-family database
Sqoop	RDB ↔ HDFS data transfer
Kafka	Message streaming

---

Hadoop vs. Spark

	Hadoop MapReduce	Apache Spark
Storage	Disk-based	In-memory
Speed	Baseline	~100x faster
Cost	Lower hardware cost	More RAM required
Status	Legacy	Current standard

---

Key Concept Cards

Big Data 5Vs ★★★★★ : Volume, Velocity, Variety, Veracity, Value. The 3Vs (first three) are the foundation.

Structured vs. Unstructured ★★★★★ : Structured=RDB/CSV (clear schema), semi-structured=JSON/XML, unstructured=text/images. 80% of real data is unstructured.

Hadoop vs. Spark ★★★★★ : Hadoop=disk-based batch (stable, cheap), Spark=in-memory (100x faster). Spark is the current industry standard.

---

Practice Quiz

Q1. What type of data are social media posts, images, and videos?

Unstructured data. They have no fixed schema and exist in diverse formats. Analyzing them requires specialized techniques: natural language processing (NLP) for text, computer vision for images and video.

Q2. IoT sensors generate thousands of readings per second. What architecture is appropriate for collection and processing?

A streaming architecture. Use Kafka for message collection (high-throughput, low-latency ingestion), Spark Streaming or Flink for real-time processing, and HBase (for fast writes) or S3 (for long-term archiving) for storage. Batch processing cannot respond to real-time requirements.