Q17Big Data Analytics
Question
4 marks
Explain the Machine learning lifecycle using Spark MLlib.
Answer
The Spark MLlib lifecycle involves data ingestion, feature engineering, model training, evaluation, and deployment via Pipelines.
Spark MLlib standardizes the ML process using the Pipeline API.
1. Data Preparation: Loading raw data into Spark DataFrames.
2. Feature Engineering: Using Transformers to extract and convert features (e.g., Tokenization, StringIndexing, VectorAssembler).
3. Model Training: Using Estimators (algorithms like Random Forest or Logistic Regression) that train on the DataFrame to produce a Model (a Transformer).
4. Evaluation: Testing the model's accuracy on validation data.
5. Tuning: Using tools like CrossValidator to optimize hyperparameters.