Apache Hadoop Overview

7 min read

Backlinks

Apache Hadoop Overview

Note

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Apache Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. Originating from Google’s GFS and MapReduce papers (2003–2004), Hadoop became the foundation of the modern big data ecosystem.

“Move computation to data, not data to computation.” — Core design philosophy of Hadoop

Core Architecture

Hadoop is composed of four primary modules that work together as a unified platform.

┌─────────────────────────────────────────────┐
│              Hadoop Ecosystem               │
│                                             │
│   ┌─────────────┐   ┌─────────────────┐     │
│   │    HDFS     │   │   MapReduce     │     │
│   │  (Storage)  │   │  (Processing)   │     │
│   └──────┬──────┘   └────────┬────────┘     │
│          │                   │              │
│   ┌──────┴───────────────────┴──────┐       │
│   │         YARN (Scheduler)        │       │
│   └──────────────────┬──────────────┘       │
│                      │                      │
│   ┌──────────────────┴──────────────┐       │
│   │      Hadoop Common (Utilities)  │       │
│   └─────────────────────────────────┘       │
└─────────────────────────────────────────────┘
ModuleRole
HDFSDistributed file system — splits files into blocks, replicates across nodes
MapReduceProgramming model — parallel batch processing of large datasets
Yarn（Yet Another Resource Negotiator）Resource manager — allocates CPU/memory, schedules jobs on the cluster
Hadoop CommonShared utilities, RPC, serialisation (Writable), configuration APIs

HDFS — Hadoop Distributed File System

HDFS stores files by splitting them into fixed-size blocks (default 128 MB) and distributing replicas across DataNodes. This design tolerates hardware failure automatically.

Key Roles

Client
  │
  ▼
NameNode  ←─── stores block metadata (namespace, block locations)
  │
  ├── DataNode 1  [Block A][Block C][Block E]
  ├── DataNode 2  [Block A][Block B][Block D]   ← replica distribution
  └── DataNode 3  [Block B][Block C][Block E]
  • NameNode — single master that manages the file-system namespace and block-to-node mapping. Does not store data.
  • DataNode — worker nodes that physically store blocks and serve read/write requests.
  • Secondary NameNode — periodically merges the edit log with the fsimage checkpoint. Not a hot standby.

HDFS Properties

PropertyDefaultNotes
Block size128 MBConfigurable per file
Replication factor3Configurable per file
Write modelWrite-once, read-manyNo random writes
Rack awarenessYesReplicas spread across racks
High availabilityOptionalActive/Standby NameNode via ZooKeeper

HDFS Read Path

  1. Client calls open() on DistributedFileSystem
  2. NameNode returns block locations sorted by proximity
  3. Client opens a DFSInputStream and reads blocks directly from DataNodes
  4. On failure, client transparently retries the next replica

HDFS Write Path

  1. Client calls create() — NameNode creates a new file entry
  2. Client writes data to a pipeline of DataNodes (rack-aware selection)
  3. Each DataNode forwards data to the next in the pipeline
  4. Acknowledgements flow back upstream to the client
  5. Client calls close() — NameNode finalises block metadata

MapReduce — Distributed Processing Model

MapReduce expresses computation as two user-defined functions applied to key–value pairs across the cluster.

Execution Phases

HDFS Input
    │
    ▼
[InputFormat]  →  InputSplits (1 per ~128 MB)
    │
    ▼
[Mapper × N]   →  map(k, v) → emit(k2, v2)     per record
    │
    ▼
[Combiner]     →  optional local pre-aggregation  (reduces network I/O)
    │
    ▼
[Partitioner]  →  hash(key) % R  →  assigns keys to reducers
    │
    ▼
[Shuffle & Sort]  →  network transfer + k-way merge sort
    │
    ▼
[Reducer × R]  →  reduce(k2, [v2…]) → emit(k3, v3)
    │
    ▼
[OutputFormat] →  HDFS output  part-r-00000 … part-r-NNNNN

Word Count Example

// Mapper — emit (word, 1) for every token
public class TokenizerMapper
    extends Mapper<Object, Text, Text, IntWritable> {
 
  private final static IntWritable one = new IntWritable(1);
  private Text word = new Text();
 
  public void map(Object key, Text value, Context context)
      throws IOException, InterruptedException {
    StringTokenizer itr = new StringTokenizer(value.toString());
    while (itr.hasMoreTokens()) {
      word.set(itr.nextToken());
      context.write(word, one);          // emit (word, 1)
    }
  }
}
 
// Reducer — sum all 1s for each word
public class IntSumReducer
    extends Reducer<Text, IntWritable, Text, IntWritable> {
 
  private IntWritable result = new IntWritable();
 
  public void reduce(Text key, Iterable<IntWritable> values, Context context)
      throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) sum += val.get();
    result.set(sum);
    context.write(key, result);          // emit (word, count)
  }
}

Job Configuration

Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);   // local optimisation
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);

MapReduce Data Types

TypeJava EquivalentDescription
TextStringUTF-8 serialisable string — used as word key
IntWritableintSerialisable integer — used as count value
LongWritablelongByte offset key from TextInputFormat
NullWritablenullPlaceholder when key or value is not needed

YARN — Yet Another Resource Negotiator

YARN separates resource management from job scheduling, allowing Hadoop to run workloads beyond MapReduce (Spark, Flink, Tez, etc.).

YARN Components

ResourceManager (master)
  ├── Scheduler          — allocates Container resources (CPU + RAM)
  └── ApplicationsManager — tracks running applications

NodeManager (per worker node)
  └── ContainerExecutor  — launches and monitors containers

ApplicationMaster (per job)
  └── Negotiates containers from ResourceManager
      Coordinates task execution
      Reports progress

Job Submission Flow

  1. Client submits job to ResourceManager
  2. RM launches an ApplicationMaster in a container on a NodeManager
  3. AM requests containers from RM’s Scheduler
  4. RM allocates containers on available NodeManagers
  5. AM launches mapper/reducer tasks inside those containers
  6. AM reports completion back to RM

YARN Scheduling Policies

SchedulerBehaviour
FIFOSimple queue — first submitted runs first
CapacityMultiple queues with guaranteed capacity fractions
FairAll running jobs share resources equally over time

Hadoop Ecosystem — Key Tools

Hadoop is a platform, not a complete solution. These projects extend it:

ToolLayerPurpose
HiveSQLHiveQL → MapReduce/Tez — SQL on HDFS
PigScriptingPig Latin dataflow language for ETL
HBaseStorageColumnar NoSQL on top of HDFS (random read/write)
SparkProcessingIn-memory DAG engine — 10–100× faster than MapReduce
FlinkProcessingStreaming-first distributed engine
SqoopIngestBulk import/export between RDBMS and HDFS
FlumeIngestStreaming log ingestion into HDFS
OozieOrchestrationWorkflow scheduler for Hadoop jobs
ZooKeeperCoordinationDistributed configuration and leader election
AmbariOperationsCluster management and monitoring UI

Fault Tolerance

Hadoop is designed to run on commodity hardware that fails regularly.

HDFS Fault Tolerance

  • DataNode heartbeats every 3 seconds to NameNode
  • Missing heartbeat after 10 minutes → node declared dead
  • NameNode triggers re-replication of under-replicated blocks
  • Checksums on every block detect silent corruption
  • Rack-aware placement ensures 1 replica survives a full rack failure

MapReduce Fault Tolerance

  • YARN monitors task containers via heartbeat
  • Failed task → automatically retried on a different node (default 4 attempts)
  • Failed ApplicationMaster → YARN restarts it; tasks may be re-run
  • Speculative execution: slow tasks are duplicated on idle nodes

Deployment Modes

ModeDescriptionUse case
LocalSingle JVM, no HDFSUnit testing, development
Pseudo-distributedSingle machine, all daemons run as separate processesIntegration testing
Fully-distributedMulti-node cluster, production HDFSProduction
Cloud (EMR / Dataproc / HDInsight)Managed Hadoop on cloud VMsProduction, elastic scaling

Configuration Files

FileControls
core-site.xmlfs.defaultFS — NameNode URI
hdfs-site.xmlReplication factor, block size, data directories
mapred-site.xmlMapReduce framework (yarn), memory settings
yarn-site.xmlResourceManager address, NodeManager memory/vCPUs
hadoop-env.shJAVA_HOME, heap sizes for daemons

Key Design Decisions & Trade-offs

What Hadoop does well

  • Batch processing of petabyte-scale datasets
  • Fault tolerance on cheap commodity hardware
  • Write-once, read-many workloads (log analysis, ETL, archival)
  • Linear horizontal scaling — add nodes to increase capacity

Hadoop limitations

LimitationBetter alternative
High latency (minutes–hours per job)Apache Spark (seconds), Apache Flink (milliseconds)
No random writes in HDFSHBase, Apache Kudu
Complex operational overheadCloud-managed services (EMR, Dataproc)
MapReduce verbositySpark (Scala/Python/SQL), Hive
Real-time streamingApache Flink, Apache Kafka Streams

Quick Reference — Important Defaults

HDFS block size           128 MB      (dfs.blocksize)
HDFS replication          3×          (dfs.replication)
Map output buffer         100 MB      (mapreduce.task.io.sort.mb)
Map spill threshold       80%         (mapreduce.map.sort.spill.percent)
Default reducers          1           (mapreduce.job.reduces)
Max map task retries      4           (mapreduce.map.maxattempts)
Max reduce task retries   4           (mapreduce.reduce.maxattempts)
NodeManager memory        8192 MB     (yarn.nodemanager.resource.memory-mb)
Container min memory      1024 MB     (yarn.scheduler.minimum-allocation-mb)

References

Download

Documents