UCI Forest Deep Learning

1. The Challenge

• Context: The US Forest Service needs to classify forest cover types (e.g., Spruce/Fir, Aspen) for resource management. Doing this manually via on-site surveys is expensive and slow. The goal was to automate this using cartographic variables (elevation, soil type, sunlight).
• The Obstacle: The dataset (UCI Covertype) contains 581,012 instances but is severely imbalanced. Two classes (Spruce/Fir and Lodgepole Pine) account for ~85% of the data, while others (like Cottonwood/Willow) are less than 1%. A standard model would simply guess the majority class and achieve high "fake" accuracy while failing the business objective.

2. The Solution Architecture

I moved beyond simple decision trees and implemented a Feed-Forward Deep Neural Network (DNN) using TensorFlow/Keras to capture non-linear relationships in the geological data.

1. Preprocessing: Tabular data normalization using StandardScaler (critical for Neural Network convergence).
2. Architecture: A Multi-Layer Perceptron (MLP) with decreasing density (128 -> 64 -> 32 units).
3. Key Decisions:
   • Keras over Scikit-Learn: While Random Forest is good for tabular data, I chose Keras to experiment with custom loss functions and fine-grained control over learning rates to squeeze out higher accuracy.
   • He Initialization: Used He Normal weight initialization to prevent vanishing gradients in the deep ReLU layers.

3. Implementation Highlights

A. Fighting Imbalance with Weighted Loss

Instead of deleting valuable data (undersampling), I computed class weights. This forces the model to pay 10x-50x more attention to rare tree types during backpropagation.

from sklearn.utils import class_weight
import numpy as np

# Calculate weights inversely proportional to class frequencies
weights = class_weight.compute_class_weight(
    class_weight='balanced',
    classes=np.unique(y_train),
    y=y_train
)

# Create a dictionary to pass into Keras
class_weight_dict = dict(enumerate(weights))
print(f"Weight for rare class (Cottonwood): {class_weight_dict[3]:.2f}") 
# Result: High weight for Class 3, Low weight for Class 1

B. The Deep Learning Model Structure

I implemented Batch Normalization after layers to stabilize learning and Dropout to prevent the model from memorizing the training data.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization

model = Sequential([
    # Input layer with 54 features (Soil types, Elevation, etc.)
    Dense(128, activation='relu', kernel_initializer='he_normal', input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.3),  # Drop 30% of neurons to prevent overfitting

    Dense(64, activation='relu', kernel_initializer='he_normal'),
    BatchNormalization(),
    Dropout(0.2),

    # Output layer: 7 Neurons for the 7 Forest Types (Softmax for probability)
    Dense(7, activation='softmax')
])

4. Challenges & Overcoming Roadblocks

• The Trap: The "Accuracy Paradox". My initial model achieved 80% accuracy immediately. However, looking at the Confusion Matrix, I realized it was predicting zero instances of "Aspen" (Class 5). It had learned to ignore the minority classes entirely.
• The Fix: I implemented Focal Loss (or strict Class Weighting) and monitored F1-Score instead of Accuracy. By penalizing the model heavily for missing an "Aspen" tree, the accuracy on the majority classes dropped slightly, but the macro-average recall improved significantly, making the tool actually useful for forestry.

5. Results & Impact

• Performance: The final DNN achieved an accuracy of ~92% on the test set, with a weighted F1-score of 0.90, successfully classifying even the rare "Cottonwood/Willow" trees.
• Scale: Processed over half a million data points efficiently using batched training.