Q17Big Data Analytics
Question
4 marks
How does K-Means clustering work on large datasets?
Answer
K-Means groups data into K distinct clusters by iteratively updating cluster centroids to minimize the variance within each cluster.
For large datasets, distributed frameworks like Spark MLlib implement K-Means. Initially, K centroids are chosen randomly. In the Map phase, each data point is assigned to the nearest centroid. In the Reduce phase, new centroids are calculated as the mean of all points assigned to that cluster. This process iterates until the centroids stabilize or a maximum number of iterations is reached.