Q19Big Data Analytics

YARN (Yet Another Resource Negotiator) improves upon Hadoop 1.x by separating resource management from job scheduling. Its primary components include: 1. ResourceManager (RM): The master daemon running on a master node. It arbitrates resources among competing applications in the cluster. It contains two main parts: a Scheduler (allocates resources based on capacity/queues) and an ApplicationsManager (accepts job submissions and negotiates the first container to start the ApplicationMaster). 2. NodeManager (NM): The slave daemon running on every compute node. It is responsible for launching application containers, monitoring their resource usage (CPU, memory, disk, network), and reporting this to the ResourceManager. 3. ApplicationMaster (AM): A per-application framework-specific entity. It negotiates resources from the ResourceManager and works with the NodeManager(s) to execute and monitor component tasks. 4. Container: Represents a fraction of the cluster's capacity (CPU, memory) allocated by the RM to an application.