Q21Cloud Computing

The Google File System (GFS) architecture was designed to meet the rapidly growing demands of Google's data processing needs. It consists of a single Master node and multiple Chunkservers. GFS assumes that component failures are the norm rather than the exception (given clusters of thousands of commodity machines), that files are huge by traditional standards (multi-gigabyte files are common), and that the dominant workload consists of large streaming reads and appends rather than small random writes, and the architecture is optimized around these assumptions rather than trying to be a general-purpose POSIX file system.

1. Files and Chunks

Files are divided into fixed-size chunks (typically 64 MB). Each chunk is identified by an immutable and globally unique 64-bit chunk handle assigned by the Master at the time of chunk creation. The large chunk size is a deliberate design decision: it reduces the number of client-master interactions needed for large files, reduces network overhead since a client can perform many operations on a chunk over a single persistent TCP connection, and reduces the size of the metadata stored on the Master.

2. GFS Master

The single master maintains all file system metadata. This includes the namespace, access control information, the mapping from files to chunks, and the current locations of chunks. It periodically communicates with each Chunkserver in HeartBeat messages to give it instructions and collect its state. All this metadata is kept in the Master's memory for speed, while the namespace and file-to-chunk mapping are also persisted to an operation log on disk for durability, with periodic checkpoints to bound recovery time after a crash.

3. Chunkservers

These store the chunks on local disks as standard Linux files and read/write chunk data specified by a chunk handle and byte range. For reliability, each chunk is replicated on multiple Chunkservers (default is 3 replicas), and the Master ensures replicas are spread across different racks so that a single rack power failure cannot take out every copy of a chunk.

Read and Write Data Flow

Data flows directly from Chunkservers to the client, while the Master only provides metadata, preventing the Master from becoming a bottleneck. For a read, the client asks the Master for the chunkservers holding the relevant chunk, then reads the data directly from the nearest chunkserver. For a write, the Master designates one replica as the 'primary,' which orders all mutations to that chunk; the client pushes data to all replicas in a pipelined fashion, and only after all replicas acknowledge does the primary confirm the write, ensuring consistency across replicas.

Significance

GFS's single-master design (with metadata operations kept lightweight and off the data path) directly inspired the NameNode/DataNode architecture of the Hadoop Distributed File System (HDFS), making GFS one of the foundational architectures underpinning modern big-data and cloud storage systems used across the industry today. Google itself later moved beyond a single-master bottleneck with successor systems like Colossus, but the core lesson of separating metadata management from bulk data transfer remains standard practice in distributed storage design.