Q2Cloud Computing

(a) Service Layers in Layered Architecture of Cloud

Cloud computing architecture is typically conceptualized as a stack of layers, each providing a progressively higher level of abstraction and managed functionality, building upon the capabilities of the layer immediately beneath it.

Physical/Data Center Layer

The lowest layer, comprising the actual physical hardware — servers, storage systems, and networking equipment — housed within the cloud provider's data centers, along with the physical facility infrastructure (power, cooling, physical security) needed to operate this hardware reliably at scale. This layer is entirely managed by the cloud provider and is not directly visible to or configurable by cloud consumers.

Infrastructure Layer — Infrastructure as a Service (IaaS)

This layer virtualizes the underlying physical hardware (using hypervisor-based server virtualization, as discussed elsewhere in this paper) to present consumers with virtualized computing resources — virtual machines, virtual storage volumes, and virtual networks — that consumers can provision, configure, and manage largely as if they were dedicated physical resources, but without needing to own, house, or directly maintain any physical hardware themselves. Example: Amazon EC2, providing virtual server instances that a customer configures with their own choice of operating system and installed software.

Platform Layer — Platform as a Service (PaaS)

Building upon the infrastructure layer, this layer provides a complete, ready-to-use application development and deployment platform — including the operating system, programming language runtime, database, and web server environment — allowing application developers to build, deploy, and run their applications without needing to individually provision, configure, or manage any of this underlying infrastructure or platform software themselves. Example: Google App Engine, allowing developers to simply upload their application code, which the platform then automatically deploys, scales, and manages.

Application/Software Layer — Software as a Service (SaaS)

The highest, most abstracted layer, providing complete, ready-to-use software applications delivered directly to end users over the network, requiring no software installation, infrastructure provisioning, or platform management whatsoever by the end user — the entire application, from underlying infrastructure through to the finished user-facing software functionality, is fully managed by the SaaS provider. Example: Salesforce (customer relationship management software) or Google Workspace (office productivity applications), both accessed entirely through a web browser with no local software installation required.

(b) Hadoop

Apache Hadoop is an open-source software framework, originally inspired by Google's internal MapReduce and Google File System technologies, designed for the reliable, scalable, distributed storage and processing of very large datasets across large clusters of commodity (relatively inexpensive, standard) hardware. Its two core components are the Hadoop Distributed File System (HDFS), which stores data by splitting large files into fixed-size blocks distributed and replicated across many nodes in the cluster (providing both scalability, since total storage capacity grows simply by adding more nodes, and fault tolerance, since each data block is stored redundantly on multiple nodes, allowing continued data availability even if some individual nodes fail); and the MapReduce processing engine, which allows large-scale data processing jobs to be automatically distributed across the cluster following the Map/Reduce paradigm discussed elsewhere in this paper, with the framework automatically scheduling tasks close to the actual physical location of the data they need to process wherever possible (a technique called data locality optimization, reducing network transfer overhead), and automatically detecting and recovering from individual node failures during job execution. Hadoop has become a foundational technology for big-data processing in cloud computing environments, with cloud providers commonly offering managed Hadoop cluster services (such as Amazon EMR or Azure HDInsight) that handle the underlying cluster provisioning and management complexity on behalf of customers wishing to run Hadoop-based data processing workloads.

HDFS architecture detail: an HDFS cluster follows a master-worker (NameNode-DataNode) architecture — a single NameNode (or an active/standby pair, for high availability) maintains the filesystem's metadata (the directory tree, and which specific DataNodes hold each block of every stored file), while a large number of DataNodes actually store the physical data blocks themselves and periodically report their block status back to the NameNode; a client wishing to read or write a file first contacts the NameNode to determine which DataNodes hold (or should hold) the relevant blocks, and then communicates directly with those DataNodes for the actual bulk data transfer, keeping the NameNode itself free of the heavy data-transfer load and able to scale to managing metadata for very large clusters. The default replication factor of 3 (each data block stored on three separate DataNodes, ideally spread across different physical server racks) provides substantial fault tolerance against both individual node failures and entire rack-level failures (such as a shared network switch or power circuit failing), at the cost of tripling the effective storage capacity consumed compared to storing each block only once.

Ecosystem tools built on Hadoop: the broader Hadoop ecosystem includes numerous additional tools built on top of the core HDFS/MapReduce foundation, addressing specific big-data processing needs — Apache Hive provides a SQL-like query interface allowing analysts familiar with traditional relational databases to query data stored in HDFS without needing to write raw MapReduce code directly; Apache Pig provides a higher-level scripting language for expressing complex data transformation pipelines more concisely than raw MapReduce; Apache HBase provides a distributed, NoSQL column-oriented database built on top of HDFS for applications requiring fast, random real-time read/write access to individual records (which HDFS itself, being optimized for large sequential reads/writes, does not efficiently support); and Apache Spark, while not strictly part of the original Hadoop project, is commonly deployed alongside Hadoop (often using YARN, Hadoop's resource-management layer, for cluster resource allocation) and has become a widely-used, generally faster alternative processing engine to classic MapReduce for many big-data workloads, particularly those involving iterative algorithms (such as machine learning) that benefit from Spark's in-memory data processing model rather than MapReduce's disk-based intermediate storage between processing stages.

Relationship between Hadoop and the layered cloud architecture in part (a): Hadoop clusters can themselves be deployed at, and consumed through, different layers of the cloud architecture stack discussed in part (a) — an organization might deploy and manage its own Hadoop cluster directly on IaaS-provisioned virtual machines (retaining full control over cluster configuration and tuning, at the cost of managing the cluster software itself), or instead consume Hadoop capability as a managed PaaS offering (such as Amazon EMR), where the underlying cluster provisioning, scaling and maintenance is handled by the cloud provider, letting the customer focus purely on submitting and running their data-processing jobs — illustrating how the same underlying big-data processing technology can be delivered to different customers at whichever layer of the cloud stack best matches their own operational preference for control versus convenience.