Assignment 5
ViT-S LoRA Fine-tuning on CIFAR-100 & Adversarial Attacks with IBM ART
Overview
🎯 Q1: ViT-S + LoRA
Fine-tuned ViT-Small (pretrained on ImageNet) on CIFAR-100 with LoRA adapters. 10 experiments across rank/alpha combinations plus Optuna search.
⚔️ Q2: Adversarial Attacks
Implemented FGSM from scratch vs IBM ART on CIFAR-10 with ResNet-18. Trained ResNet-34 adversarial detectors for PGD and BIM attacks.
🔬 Optuna HPO
Used Optuna (20 trials) to find the best LoRA configuration. Best: r=13, α=16, dropout=0.061 achieving 89.35% test accuracy.
🐳 Dockerized
All experiments run inside Docker containers with CUDA support. Full reproducibility with requirements.txt and Dockerfile.
Q1: ViT-S LoRA Results
Test Results — All Configurations
| LoRA | Rank | Alpha | Dropout | Test Accuracy | Trainable Params |
|---|---|---|---|---|---|
| No | — | — | — | 78.43% | 38,500 |
| Yes | 2 | 2 | 0.1 | 87.52% | 75,364 |
| Yes | 2 | 4 | 0.1 | 88.58% | 75,364 |
| Yes | 2 | 8 | 0.1 | 88.87% | 75,364 |
| Yes | 4 | 2 | 0.1 | 88.07% | 112,228 |
| Yes | 4 | 4 | 0.1 | 88.32% | 112,228 |
| Yes | 4 | 8 | 0.1 | 88.78% | 112,228 |
| Yes | 8 | 2 | 0.1 | 88.40% | 185,956 |
| Yes | 8 | 4 | 0.1 | 88.37% | 185,956 |
| Yes | 8 | 8 | 0.1 | 88.74% | 185,956 |
| Yes | 13 | 16 | 0.061 | 89.35% (Optuna) | 278,116 |
Q2: Adversarial Attack Results
FGSM Perturbation Strength vs Accuracy
| Epsilon | Clean Accuracy | FGSM Scratch | FGSM IBM ART |
|---|---|---|---|
| 0.000 | 93.75% | 93.75% | 93.75% |
| 0.005 | 93.75% | 45.45% | 49.83% |
| 0.010 | 93.75% | 27.27% | 31.65% |
| 0.020 | 93.75% | 18.80% | 22.82% |
| 0.040 | 93.75% | 14.92% | 18.11% |
| 0.080 | 93.75% | 12.01% | 13.55% |
| 0.100 | 93.75% | 11.55% | 12.39% |
| 0.150 | 93.75% | 10.32% | 10.62% |
| 0.200 | 93.75% | 10.19% | 10.27% |
| 0.300 | 93.75% | 10.28% | 10.30% |
Adversarial Detection Comparison
| Attack | Detection Accuracy | Precision | Recall | F1-Score |
|---|---|---|---|---|
| PGD | 50.85% | 0.5909 | 0.0066 | 0.0131 |
| BIM | 98.98% | 0.9803 | 0.9990 | 0.9895 |
Resources
📊 WandB — ViT LoRA
Training curves, gradient norms, class-wise accuracy histograms.
View Dashboard →📊 WandB — Adversarial
FGSM comparison, perturbation analysis, detector metrics.
View Dashboard →🤗 HuggingFace
Best ViT-S LoRA model weights (Optuna: r=13, α=16).
View Model →📂 Source Code
Full source code, Dockerfile, and experiment scripts.
View Repository →