Benchmarks

This page summarizes reproducible experimental results that torchdistill official repository supports.

ImageNet (ILSVRC 2012)

ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)

---

Student: ResNet-18

Top-1 validation accuracy of ResNet-18 for ILSVRC 2012 (ImageNet)

Method	Original work	torchdistill	Config
Pretrained	N/A	69.76*	torchvision
KD	N/A	71.37	torchdistill/configs/official/
AT	70.70	70.90	torchdistill/configs/official/
FT	71.43**	71.56	torchdistill/configs/official/
CRD	71.17	70.93	torchdistill/configs/official/
Tf-KD	70.42	70.52	torchdistill/configs/official/
SSKD	71.62	70.09	torchdistill/configs/official/
\(L_2\)	70.90	71.08	torchdistill/configs/official/
PAD-\(L_2\)	71.71	71.71	torchdistill/configs/official/
KR	71.61	71.64	torchdistill/configs/official/
ICKD	72.19	68.34	torchdistill/configs/sample/
DIST	72.07	71.95	torchdistill/configs/sample/
SRD	71.63	71.92	torchdistill/configs/official/
KD w/ LS	71.42	71.23	torchdistill/configs/sample/

* The pretrained ResNet-34 and ResNet-18 are provided by torchvision.

** FT is assessed with ILSVRC 2015 in the original work.

Original work

• KD: “Distilling the Knowledge in a Neural Network”

• AT: “Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer”

• FT: “Paraphrasing Complex Network: Network Compression via Factor Transfer”

• CRD: “Contrastive Representation Distillation”

• Tf-KD: “Revisiting Knowledge Distillation via Label Smoothing Regularization”

• SSDK: “Knowledge Distillation Meets Self-Supervision”

• \(L_2\), PAD-\(L_2\): “Prime-Aware Adaptive Distillation”

• KR: “Distilling Knowledge via Knowledge Review”

• ICKD: “Exploring Inter-Channel Correlation for Diversity-Preserved Knowledge Distillation”

• DIST: “Knowledge Distillation from A Stronger Teacher”

• SRD: “Understanding the Role of the Projector in Knowledge Distillation”

• KD w/ LS: “Logit Standardization in Knowledge Distillation”

References

• Yoshitomo Matsubara: “torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation”

• Yoshitomo Matsubara: “torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP”