Multimodal Transformer

2022 ETRI multimodal emotion classification paper competition

Dataset

ETRI KEMDy19

Quick setup

# clone project
git clone https://github.com/DimensionSTP/multimodal-transformer.git
cd multimodal-transformer

# [OPTIONAL] create conda environment
conda create -n myenv python=3.10 -y
conda activate myenv

# install requirements
pip install -r requirements.txt

.env file setting

PROJECT_DIR={PROJECT_DIR}
CONNECTED_DIR={CONNECTED_DIR}
DEVICES={DEVICES}
HF_HOME={HF_HOME}
USER_NAME={USER_NAME}

Single Modality Training

• only audio

python unimodal_main.py mode=audio dataset=audio_kemdy19_dataset

• only text

python unimodal_main.py mode=text dataset=text_kemdy19_dataset

Multi Modality Model Hyper-Parameters Tuning

• multimodal transformer(embedding vector deep fusion)

python main.py mode=tune is_tuned=untuned num_trials={num_trials}

Multi Modality Training

• multimodal transformer(embedding vector deep fusion)

python main.py mode=train is_tuned={tuned or untuned} num_trials={num_trials}

Multi Modality Testing

• multimodal transformer(embedding vector deep fusion)

python main.py mode=test is_tuned={tuned or untuned} num_trials={num_trials} epoch={ckpt epoch}

Multi Modality Prediction

• end-to-end

python main.py mode=predict is_tuned={tuned or untuned} num_trials={num_trials} epoch={ckpt epoch}

Examples of shell scipts

• train

bash scripts/train.sh

• test

bash scripts/test.sh

If you want to change main config, use --config-name={config_name}.

Also, you can use --multirun option.

You can set additional arguments through the command line.