whisper-leaderboard

#28
by Steveeeeeeen HF staff - opened
Hugging Face for Audio org
edited 20 days ago

What does this PR do?

This PR adds a dedicated Whisper Model Leaderboard tab to compare different Whisper model implementations across various backends and hardware configurations.

Key Changes:

  • Added new environment variables for Whisper-specific repository and paths
  • Extended load_all_info_from_dataset_hub() to handle both original and Whisper evaluation results
  • Created a new "🔄 Whisper Model Leaderboard" tab with:
    • Additional columns for Backend and Device information
    • Interactive filters to compare results by backend type and hardware
    • Same evaluation metrics (WER, RTFx) as the main leaderboard

Why?

This dedicated view will help users:

  • Compare performance across different Whisper implementations
  • Understand speed/accuracy tradeoffs on different hardware
  • Make informed decisions about which backend to use for their specific needs

Testing

  • Verified that both original and Whisper leaderboards load correctly
  • Tested filtering functionality with multiple backend/device combinations
  • Confirmed that all WER and RTFx metrics display properly

TODO

  • Add secrets in the Space settings:
    QUEUE_REPO_WHISPER: Path to the repository containing Whisper evaluation results
    QUEUE_PATH_WHISPER: Local path where Whisper results will be stored

Screenshots

Capture d’écran 2024-12-02 à 10.00.45.png

Steveeeeeeen changed pull request status to open
Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment