Update README.md

README.md

@@ -1,199 +1,199 @@
----
-license: apache-2.0
----
-# Emotion2Vec-S
-
-For more information, please refer to github [C2SER](https://github.com/zxzhao0/C2SER)
-
-## Introduction
-
-This repository contains the implementation of Emotion2Vec-S, a self-supervised learning (SSL) model for speech emotion recognition, as presented in our paper "Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought". 
-
-## Requirements and Installation
-
-This project follows the fairseq installation process.
-
-### Requirements
-
-- PyTorch version >= 1.10.0
-- Python version >= 3.8
-
-### Installation
-
-To install fairseq and develop locally:
-
-```bash
-git clone https://github.com/pytorch/fairseq
-cd fairseq
-pip install --editable ./
-```
-
-### Feature Extraction
-
-You can download the pre-trained [Emotion2vec-S model](https://drive.google.com/drive/folders/1LWWi6bahzn7fJP4fCgPleOyQ30sD_BWO?usp=drive_link) and put it in the `./Emotion2Vec-S/ckpt` folder. 
-Meanwhile，we have provided the pretrained checkpoints in the huggingface model hub. You can also download ckpt file from [here](https://huggingface.co/ASLP-lab/Emotion2Vec-S). We also provide [here](https://drive.google.com/drive/folders/12AOVJT7I9GSLJnjHa-Elc-UKgog-mZR2) the feature files for the Emo-Emilia dataset extracted using Emotion2vec-S. 
-
-If you want to extract features using Emotion2Vec-S，you will also need to provide a `wav.scp` file and place it in the `./Emotion2Vec-S` directory. Here is an example of the `wav.scp` file:：
-```pgsql
-audio_name1 /path/to/audio_name1.wav
-audio_name2 /path/to/audio_name2.wav
-audio_name3 /path/to/audio_name3.wav
-```
-
-Next, you can directly run the following code to extract features：
-```python
-import torch
-import os
-import sys
-import json
-import numpy as np
-import argparse
-from tqdm import tqdm
-import torchaudio
-import torch.nn.functional as F
-import fairseq
-from dataclasses import dataclass
-
-SAMPLING_RATE=16000
-
-@dataclass
-class UserDirModule:
-    user_dir: str
-
-def extract_fairseq_feature(wav_path, model, device):
-    try:
-        wav, sr = torchaudio.load(wav_path)
-        # 合并多声道为单声道（取平均）
-        if wav.size(0) > 1:
-            wav = torch.mean(wav, dim=0, keepdim=True)
-        if sr != SAMPLING_RATE:
-            wav = torchaudio.functional.resample(wav, sr, SAMPLING_RATE)
-        wav = wav[0, :].view(1, -1)
-        wav = wav.to(device)
-        out = model.extract_features(wav)
-        return out
-    except Exception as e:
-        print(f"Error processing audio file {wav_path}: {e}")
-        return None
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--model_path', type=str, default="./Emotion2Vec-S/ckpt/checkpoint.pt")
-    parser.add_argument('--model_dir', type=str, default="./Emotion2Vec-S/examples/data2vec/")
-    parser.add_argument('--dump_dir', type=str, default="./Emotion2Vec-S/features_frm")
-    parser.add_argument('--device', type=str, default='cuda')
-    parser.add_argument('--data', type=str, default="./Emotion2Vec-S/wav.scp")
-    parser.add_argument('--level', type=str, default="frame", help="frame or utterance")
-    args = parser.parse_args()
-
-    data = {}
-    with open(args.data, 'r') as f:
-        for line in f:
-            seg_id, wav_path = line.strip().split(maxsplit=1)
-            data[seg_id] = wav_path
-
-    os.makedirs(args.dump_dir, exist_ok=True)
-
-    seg_ids = data.keys()
-    print(f'Loaded {len(seg_ids)} audio entries')
-    # load models
-    my_model_path = UserDirModule(args.model_dir)
-    fairseq.utils.import_user_module(my_model_path)
-    model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([args.model_path])
-    model = model[0].to(args.device)
-    
-    for seg_id in tqdm(seg_ids):
-
-        wav_path = data[seg_id]
-        if not os.path.exists(wav_path):
-            print(f"WARNING: {wav_path} does not exist")
-            continue 
-        try:
-            torchaudio.load(wav_path)
-        except:
-            print(f'ERROR: Failed to load {wav_path}')
-            continue         
-
-        feat = extract_fairseq_feature(wav_path, model, args.device)
-
-        if feat is not None:
-            if args.level == 'frame':
-                feat = feat['x'].cpu().detach().numpy()[0]
-            elif args.level == 'utterance':
-                feat = feat['utt_x'].cpu().detach().numpy()[0] 
-            else:
-                raise ValueError("Unknown level: {}".format(args.level))            
-
-            save_path = os.path.join(args.dump_dir, f"{seg_id}.npy")
-            os.makedirs(os.path.dirname(save_path), exist_ok=True)
-            np.save(save_path, feat)
-            print(f"Processed: {seg_id} | Shape: {feat.shape} | Saved to: {save_path}")
-        else:
-            print(f"Skipped problematic file: {seg_id}")
-
-```
-Alternatively, you can adjust the code according to your needs. The code path is `./Emotion2Vec-S/speech_feature_extraction.py`. You can also use the `./Emotion2Vec-S/extract_feature.sh` script to batch process features for multiple datasets. The script supports parallel processing and offers the following parameters:
-
-- `--model_path`: Path to the checkpoint file
-- `--model_dir`: Path to the model
-- `--dump_dir`: Directory to save extracted features
-- `--device`: Device to run the model on (e.g., 'cuda:0')
-- `--data`: Path to the dataset scp file
-- `--level`: Level of feature (frame level or utterance level)
-
-## 2. Training and testing on EmoBox using extracted features
-
-If you want to test our model on other datasets using [EmoBox](https://github.com/emo-box/EmoBox/tree/main). There is also an example provided below, which you can modify to suit your needs:
-
-Use k-fold cross-validation with learning rates (1e-3, 1e-4) and hidden sizes (128, 256):
-
-```bash
-cd examples/sb
-data=/path/to/your/data_files
-lrs=(1e-3 1e-4)               # Learning rate list
-hidden_sizes=(128 256)        # Hidden size list
-gpus=(0 1 2 3)                # GPU list
-task_id=0
-declare -A dataset_folds=(
-    ["mesd"]=1
-)
-declare -A dataset_classes=(
-    ["mesd"]=6
-)
-datasets=("mesd")
-
-for dataset in "${datasets[@]}"; do
-    folds=${dataset_folds[$dataset]}
-    n_classes=${dataset_classes[$dataset]}
-
-    for lr in "${lrs[@]}"; do
-        for hidden_size in "${hidden_sizes[@]}"; do
-            gpu=${gpus[$task_id % ${#gpus[@]}]}
-            export CUDA_VISIBLE_DEVICES=$gpu
-            task_number=$((task_id + 1))
-            for fold in $(seq 1 $folds); do
-                echo "Training fold $fold with lr=$lr, hidden_size=$hidden_size on GPU $gpu, task_number=$task_number, dataset=$dataset..."
-                python3 train.py \
-                    hparams/data2vec2-large_freeze.yaml \
-                    --output_folder /path/to/your/${dataset}-S/fold${fold}_lr${lr}_hidden${hidden_size} \
-                    --seed 1234 \
-                    --batch_size 32 \
-                    --lr $lr \
-                    --train_annotation ${data}/${dataset}/fold_${fold}/${dataset}_train_fold_${fold}.json \
-                    --test_annotation ${data}/${dataset}/fold_${fold}/${dataset}_test_fold_${fold}.json \
-                    --number_of_epochs 100 \
-                    --feat_dir /path/to/your/dump_${dataset}-S \
-                    --label_map ${data}/${dataset}/label_map.json \
-                    --device cuda \
-                    --out_n_neurons ${n_classes} \
-                    --hidden_size $hidden_size &
-            done
-            task_id=$((task_id + 1))
-        done
-    done
-done
-
-wait
-echo "All training tasks completed."
+---
+license: apache-2.0
+---
+# Emotion2Vec-S
+
+C<sup>2</sup>SER: [Paper](https://arxiv.org/abs/2502.18186) | [Code](https://github.com/zxzhao0/C2SER) | [HuggingFace](https://huggingface.co/collections/ASLP-lab/c2ser-67bc735d820403e7969fe8a0)
+
+## Introduction
+
+This repository contains the implementation of Emotion2Vec-S, a self-supervised learning (SSL) model for speech emotion recognition, as presented in our paper "Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought". 
+
+## Requirements and Installation
+
+This project follows the fairseq installation process.
+
+### Requirements
+
+- PyTorch version >= 1.10.0
+- Python version >= 3.8
+
+### Installation
+
+To install fairseq and develop locally:
+
+```bash
+git clone https://github.com/pytorch/fairseq
+cd fairseq
+pip install --editable ./
+```
+
+### Feature Extraction
+
+You can download the pre-trained [Emotion2vec-S model](https://drive.google.com/drive/folders/1LWWi6bahzn7fJP4fCgPleOyQ30sD_BWO?usp=drive_link) and put it in the `./Emotion2Vec-S/ckpt` folder. 
+Meanwhile，we have provided the pretrained checkpoints in the huggingface model hub. You can also download ckpt file from [here](https://huggingface.co/ASLP-lab/Emotion2Vec-S). We also provide [here](https://drive.google.com/drive/folders/12AOVJT7I9GSLJnjHa-Elc-UKgog-mZR2) the feature files for the Emo-Emilia dataset extracted using Emotion2vec-S. 
+
+If you want to extract features using Emotion2Vec-S，you will also need to provide a `wav.scp` file and place it in the `./Emotion2Vec-S` directory. Here is an example of the `wav.scp` file:：
+```pgsql
+audio_name1 /path/to/audio_name1.wav
+audio_name2 /path/to/audio_name2.wav
+audio_name3 /path/to/audio_name3.wav
+```
+
+Next, you can directly run the following code to extract features：
+```python
+import torch
+import os
+import sys
+import json
+import numpy as np
+import argparse
+from tqdm import tqdm
+import torchaudio
+import torch.nn.functional as F
+import fairseq
+from dataclasses import dataclass
+
+SAMPLING_RATE=16000
+
+@dataclass
+class UserDirModule:
+    user_dir: str
+
+def extract_fairseq_feature(wav_path, model, device):
+    try:
+        wav, sr = torchaudio.load(wav_path)
+        # 合并多声道为单声道（取平均）
+        if wav.size(0) > 1:
+            wav = torch.mean(wav, dim=0, keepdim=True)
+        if sr != SAMPLING_RATE:
+            wav = torchaudio.functional.resample(wav, sr, SAMPLING_RATE)
+        wav = wav[0, :].view(1, -1)
+        wav = wav.to(device)
+        out = model.extract_features(wav)
+        return out
+    except Exception as e:
+        print(f"Error processing audio file {wav_path}: {e}")
+        return None
+
+if __name__ == '__main__':
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model_path', type=str, default="./Emotion2Vec-S/ckpt/checkpoint.pt")
+    parser.add_argument('--model_dir', type=str, default="./Emotion2Vec-S/examples/data2vec/")
+    parser.add_argument('--dump_dir', type=str, default="./Emotion2Vec-S/features_frm")
+    parser.add_argument('--device', type=str, default='cuda')
+    parser.add_argument('--data', type=str, default="./Emotion2Vec-S/wav.scp")
+    parser.add_argument('--level', type=str, default="frame", help="frame or utterance")
+    args = parser.parse_args()
+
+    data = {}
+    with open(args.data, 'r') as f:
+        for line in f:
+            seg_id, wav_path = line.strip().split(maxsplit=1)
+            data[seg_id] = wav_path
+
+    os.makedirs(args.dump_dir, exist_ok=True)
+
+    seg_ids = data.keys()
+    print(f'Loaded {len(seg_ids)} audio entries')
+    # load models
+    my_model_path = UserDirModule(args.model_dir)
+    fairseq.utils.import_user_module(my_model_path)
+    model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([args.model_path])
+    model = model[0].to(args.device)
+    
+    for seg_id in tqdm(seg_ids):
+
+        wav_path = data[seg_id]
+        if not os.path.exists(wav_path):
+            print(f"WARNING: {wav_path} does not exist")
+            continue 
+        try:
+            torchaudio.load(wav_path)
+        except:
+            print(f'ERROR: Failed to load {wav_path}')
+            continue         
+
+        feat = extract_fairseq_feature(wav_path, model, args.device)
+
+        if feat is not None:
+            if args.level == 'frame':
+                feat = feat['x'].cpu().detach().numpy()[0]
+            elif args.level == 'utterance':
+                feat = feat['utt_x'].cpu().detach().numpy()[0] 
+            else:
+                raise ValueError("Unknown level: {}".format(args.level))            
+
+            save_path = os.path.join(args.dump_dir, f"{seg_id}.npy")
+            os.makedirs(os.path.dirname(save_path), exist_ok=True)
+            np.save(save_path, feat)
+            print(f"Processed: {seg_id} | Shape: {feat.shape} | Saved to: {save_path}")
+        else:
+            print(f"Skipped problematic file: {seg_id}")
+
+```
+Alternatively, you can adjust the code according to your needs. The code path is `./Emotion2Vec-S/speech_feature_extraction.py`. You can also use the `./Emotion2Vec-S/extract_feature.sh` script to batch process features for multiple datasets. The script supports parallel processing and offers the following parameters:
+
+- `--model_path`: Path to the checkpoint file
+- `--model_dir`: Path to the model
+- `--dump_dir`: Directory to save extracted features
+- `--device`: Device to run the model on (e.g., 'cuda:0')
+- `--data`: Path to the dataset scp file
+- `--level`: Level of feature (frame level or utterance level)
+
+## 2. Training and testing on EmoBox using extracted features
+
+If you want to test our model on other datasets using [EmoBox](https://github.com/emo-box/EmoBox/tree/main). There is also an example provided below, which you can modify to suit your needs:
+
+Use k-fold cross-validation with learning rates (1e-3, 1e-4) and hidden sizes (128, 256):
+
+```bash
+cd examples/sb
+data=/path/to/your/data_files
+lrs=(1e-3 1e-4)               # Learning rate list
+hidden_sizes=(128 256)        # Hidden size list
+gpus=(0 1 2 3)                # GPU list
+task_id=0
+declare -A dataset_folds=(
+    ["mesd"]=1
+)
+declare -A dataset_classes=(
+    ["mesd"]=6
+)
+datasets=("mesd")
+
+for dataset in "${datasets[@]}"; do
+    folds=${dataset_folds[$dataset]}
+    n_classes=${dataset_classes[$dataset]}
+
+    for lr in "${lrs[@]}"; do
+        for hidden_size in "${hidden_sizes[@]}"; do
+            gpu=${gpus[$task_id % ${#gpus[@]}]}
+            export CUDA_VISIBLE_DEVICES=$gpu
+            task_number=$((task_id + 1))
+            for fold in $(seq 1 $folds); do
+                echo "Training fold $fold with lr=$lr, hidden_size=$hidden_size on GPU $gpu, task_number=$task_number, dataset=$dataset..."
+                python3 train.py \
+                    hparams/data2vec2-large_freeze.yaml \
+                    --output_folder /path/to/your/${dataset}-S/fold${fold}_lr${lr}_hidden${hidden_size} \
+                    --seed 1234 \
+                    --batch_size 32 \
+                    --lr $lr \
+                    --train_annotation ${data}/${dataset}/fold_${fold}/${dataset}_train_fold_${fold}.json \
+                    --test_annotation ${data}/${dataset}/fold_${fold}/${dataset}_test_fold_${fold}.json \
+                    --number_of_epochs 100 \
+                    --feat_dir /path/to/your/dump_${dataset}-S \
+                    --label_map ${data}/${dataset}/label_map.json \
+                    --device cuda \
+                    --out_n_neurons ${n_classes} \
+                    --hidden_size $hidden_size &
+            done
+            task_id=$((task_id + 1))
+        done
+    done
+done
+
+wait
+echo "All training tasks completed."
 ```