DiRaC-I: Identifying Diverse and Rare Training Classes for Zero-Shot Learning (ACM TOMM 2023)

👓 At a glance

This repository contains the official PyTorch implementation of our ACM TOMM 2023 paper : DiRaC-I: Identifying Diverse and Rare Training Classes for Zero-Shot Learning, a work done by Sandipan Sarma and Arijit Sur at Indian Institute of Technology Guwahati.

• Most existing ZSL works in image classification use a predetermined, disjoint set of seen-unseen classes [1] to evaluate their methods. These seen (training) classes might be sub-optimal for ZSL methods to appreciate the diversity and rarity of an object domain.

• In this work, we propose a framework called Diverse and Rare Class Identifier (DiRaC-I) which, given an attribute-based dataset, can intelligently yield the most suitable “seen classes” for training ZSL models. DiRaC-I has two main goals:

  • constructing a diversified set of seed classes
  • using them to initialize a visual-semantic mining algorithm for acquiring the classes capturing both diversity and rarity in the object domain adequately. These classes can then be used as “seen classes” to train ZSL models, boosting their performance for image classification.

• We simulate a real-world scenario where visual samples of novel object classes in the wild are available to neither DiRaC-I nor the ZSL models during training and conducted extensive experiments on two benchmark data sets for zero-shot image classification — CUB and SUN.

  

🏢 Creating the work environment

Our code is based on PyTorch and has been implemented using an NVIDIA A100 80 GB. Install Anaconda/Miniconda on your system and create a conda environment using the following command:

conda env create -f dirac-i-env.yml

⌛ Dataset download and preprocessing

We evaluate our performance on two benchmark ZSL datasets: CUB and SUN, with ResNet-101 visual features for each image taken from xlsa17. The data directory should have the following structure:

datasets
└───CUB
│   └───Data
│       └───001.Black_footed_Albatross
│       └───002.Laysan_Albatross
│       └───...
│
└───SUN
│   └───Data
│       └───abbey
│       └───access_road
│       └───...

Then, we separate some train-test data for testing ResNet model performance:

python3 train_test_split.py

🚄 Training DiRaC-I

cd scripts
sh run_AL_<DATASET>.sh

🔍 Evaluating ZSL models

cd scripts
sh run_ZSL_<DATASET>.sh

Inside the script file, uncomment the one you want to run. Note that the argument al_seed should be set to original if the ZSL method runs following the original seen-unseen splits [1], and set to new_seed_final if the method runs following seen-unseen splits produced by DiRaC-I.

🏆 Results

🎁 Citation

If you use our work for your research, kindly star ⭐ our repository and consider citing our work using the following BibTex:

@article{sarma2023dirac,
  title={DiRaC-I: Identifying Diverse and Rare Training Classes for Zero-Shot Learning},
  author={Sarma, Sandipan and Sur, Arijit},
  journal={ACM Transactions on Multimedia Computing, Communications and Applications},
  volume={20},
  number={1},
  pages={1--23},
  year={2023},
  publisher={ACM New York, NY}
}

📜 References

[1] Y. Xian, C. H. Lampert, B. Schiele and Z. Akata, "Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 9, pp. 2251-2265, 1 Sept. 2019, doi: 10.1109/TPAMI.2018.2857768.