PointNet 3D Classification

Context

3D point clouds from LiDAR, depth sensors, and CAD are ubiquitous in robotics, autonomous vehicles, and manufacturing - but traditional CNNs expect ordered grid inputs. Voxelizing point clouds loses spatial resolution and is memory-intensive. PointNet solves this by learning directly from unordered point sets.

I implemented PointNet from scratch to deeply understand permutation-invariant architectures for 3D data - a foundational building block for modern 3D vision systems.

Technical Approach

The Core Insight: Permutation Invariance

A point cloud is a set of points (x1, x2, ..., xN) where order doesn't matter. Any function f that operates on this set must satisfy f(x1, x2, ..., xN) = f(x_pi(1), x_pi(2), ..., x_pi(N)) for any permutation pi.

PointNet achieves this with a symmetric function: apply shared MLPs to each point independently, then aggregate with max pooling (a symmetric operation). This is mathematically guaranteed to be permutation-invariant.

Architecture (implemented from scratch)

Input: N × 3 point cloud
    ↓
Input T-Net → 3×3 transformation matrix (learned spatial alignment)
    ↓
Shared MLP: 3 → 64 → 64
    ↓
Feature T-Net → 64×64 transformation matrix
    ↓
Shared MLP: 64 → 128 → 1024
    ↓
Global Max Pooling (symmetric aggregation over N points)
    ↓
FC: 1024 → 512 → 256 → 40 classes

The T-Nets learn to canonicalize the input - aligning point clouds to a consistent orientation before feature extraction. This makes the model robust to rigid transformations.

Results

Gap Analysis

The 10% gap vs. the original paper comes from:

• Fewer points (512 vs. 1024) - I used 512 for faster iteration on a single GPU
• Fewer epochs (8 vs. 200+) - training budget constraint
• No data augmentation - the paper uses random rotation, jitter, and scaling

Closing the gap would require: (1) scaling to 1024 points, (2) adding augmentation, (3) regularizing the feature T-Net with an orthogonality loss on the transformation matrix, and (4) training for 100+ epochs. These are engineering decisions, not architectural limitations.

Design Decisions

• Why max pooling over mean pooling? Max pooling selects the most activated feature per dimension across all points - it's more robust to outlier points and captures the "most informative" signal. Mean pooling dilutes strong signals with noise from uninformative points.

• T-Net regularization: The 64x64 feature T-Net has 4,096 parameters in its output matrix. Without regularization, it can learn degenerate transformations. An orthogonality loss constrains it to near-orthogonal transformations, preserving feature magnitudes.

Technologies

Python · PyTorch · torch-geometric · ModelNet40 · NumPy