Iris Classification with PySpark MLlib

Overview • Dataset • Methodology • Results • Run Project

Academic Project — STQD6324 Data Management

A multiclass classification project using PySpark MLlib to compare Logistic Regression, Decision Tree and Random Forest models on the Iris dataset, with cross-validation and hyperparameter tuning.

---

Overview

This project demonstrates an end-to-end multiclass classification workflow using PySpark MLlib:

flowchart LR
    A[Data Loading & Preprocessing] --> B[Feature Engineering]
    B --> C[Train-Test Split]
    C --> D[Model Training & Hyperparameter Tuning]
    D --> E[Performance Evaluation]
    E --> F[Model Comparison]

Loading

Dataset

Iris Dataset (Fisher, 1936)

• Source: GitHub↗️
• Samples: 150 iris flowers
• Features: 4 continuous measurements
  • Sepal length (cm)
  • Sepal width (cm)
  • Petal length (cm)
  • Petal width (cm)
• Target: species

Setosa	Versicolor	Virginica

Methodology

Models

Three classification algorithms were implemented and compared:

Model	Purpose
Logistic Regression	Linear classification model used as a baseline
Decision Tree	Tree-based model that captures nonlinear decision boundaries
Random Forest	Ensemble of decision trees designed to improve predictive performance and reduce overfitting

Model Tuning

Each model was optimized using 5-fold cross-validation and grid search over selected hyperparameters:

• Logistic Regression: regParam, elasticNetParam, maxIter
• Decision Tree: maxDepth, impurity, minInstancesPerNode
• Random Forest: numTrees, maxDepth, maxBins

Evaluation Metrics

Models were evaluated using Accuracy, Weighted Precision, Weighted Recall and F1-Score to assess overall classification performance across the three classes.

🎯 Accuracy   🔍 Weighted Precision   📈 Weighted Recall   ⚖️ F1-Score

Notebook Structure

iris_classification_pyspark.ipynb
├── 1. Environment Setup and Imports
├── 2. Load Iris Dataset
├── 3. Data Preprocessing
├── 4. Train-Test Split
├── 5. Model Implementation with Hyperparameter Tuning
│   ├── Logistic Regression
│   ├── Decision Tree
│   └── Random Forest
├── 6. Model Evaluation
├── 7. Predictions on Test Data
├── 8. Comparative Analysis
└── 9. Cleanup

Results

Model	Accuracy	Precision	Recall	F1-Score
Logistic Regression	0.97	0.97	0.97	0.97
Decision Tree	0.91	0.94	0.91	0.91
Random Forest	0.97	0.97	0.97	0.97

Note

Key Findings

• Top Performers: Logistic Regression and Random Forest achieved the strongest overall test performance.
• Feature Importance: Petal length and petal width were the most influential features for classification.
• Class Separability: Setosa was easily separated, while most classification errors occurred between Versicolor and Virginica.

Confusion Matrix

The confusion matrices show the classification performance of each model across the three Iris species. Most errors occur between Versicolor and Virginica, while Setosa is more easily distinguished.

Logistic Regression

Decision Tree

Random Forest

Feature Importance

Random Forest feature importance indicates that petal measurements contributed most strongly to the classification, while the sepal measurements had comparatively lower importance.

Run Project

This project was developed in Google Colab using PySpark, providing a convenient environment for running Apache Spark without a local Spark installation.

Environment Used

Google Colab

1. Download iris_classification_pyspark.ipynb.
2. Open Google Colab.
3. Upload the notebook.
4. Connect to a runtime using Runtime → Connect.
5. Install the required libraries:

!pip install pyspark pandas matplotlib seaborn

6. Run the notebook cells sequentially.

---

Alternative: Local Jupyter Notebook

For local execution, ensure Java JDK 8 or 11 is available for the Spark runtime.

1. Download the notebook file : iris_classification_pyspark.ipynb
2. Open the notebook in Jupyter
3. Install the required libraries :

!pip install pyspark pandas matplotlib seaborn

---

This project was completed as part of the STQD6324 Data Management course at Universiti Kebangsaan Malaysia.