BELLE-2/Belle-whisper-large-v3-zh

Nov 23, 2024

我使用下面的命令对您的几个中文微调模型进行转换，

ct2-transformers-converter --model model/Belle-whisper-large-v3-turbo-zh --output_dir model/Belle-whisper-large-v3-turbo-zh-ct2 --copy_files tokenizer.json --quantization float16

并使用 https://github.com/MahmoudAshraf97/whisper-diarization/blob/main/diarize.py 中的代码进行测试，在Belle-whisper-large-v2-zh-ct2上运行正常，但是在所有转换后的v3模型上都会产生如下报错：

Traceback (most recent call last):
File "F:\code\python\whisper-diarization\diarize.py", line 288, in
process_audio_file(audio_file)
File "F:\code\python\whisper-diarization\diarize.py", line 146, in process_audio_file
transcript_segments, info = whisper_pipeline.transcribe(
File "D:\software\anaconda3\envs\whisperD\lib\site-packages\faster_whisper\transcribe.py", line 443, in transcribe
features=np.concatenate(
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 80 and the array at index 2 has size 128

我加载模型部分的代码如下：

whisper_model = faster_whisper.WhisperModel(
    args.model_name, device=args.device, compute_type=mtypes[args.device]
)
whisper_pipeline = faster_whisper.BatchedInferencePipeline(whisper_model)
audio_waveform = faster_whisper.decode_audio(vocal_target)
suppress_tokens = (
    find_numeral_symbol_tokens(whisper_model.hf_tokenizer)
    if args.suppress_numerals
    else [-1]
)

if args.batch_size > 0:
    transcript_segments, info = whisper_pipeline.transcribe(
        audio_waveform,
        language,
        suppress_tokens=suppress_tokens,
        batch_size=args.batch_size,
        without_timestamps=True,
        vad_filter=True, 
        vad_parameters=dict(min_silence_duration_ms=500), 
    )
else:
    transcript_segments, info = whisper_model.transcribe(
        audio_waveform,
        language,
        suppress_tokens=suppress_tokens,
        without_timestamps=True,
        vad_filter=True,
    )

希望能得到你的帮助，十分感谢！

shuaijiang

BELLE-2 Group // Be Everyone's Large Language model Engine org Nov 26, 2024

是由于faster_whisper 的原因，更新到最新的版本试试

maomaoxion

Nov 26, 2024

是由于faster_whisper 的原因，更新到最新的版本试试

测试使用的是faster-whisper 1.1.0，已经是最新的版本了

shuaijiang

BELLE-2 Group // Be Everyone's Large Language model Engine org Nov 27, 2024

whisper-diarization 也有更新： https://github.com/MahmoudAshraf97/whisper-diarization/commit/accf5c07d231046de980b39a330b270a55259c17#diff-b315f4a191ca9ba5d54e13a3a56afd0066e1d4bdad453177290b9bfc307c20caL168

提供下更详细的信息，各个模块的version，我复现下

maomaoxion

Nov 27, 2024

•

edited Nov 27, 2024

whisper-diarization 也有更新： https://github.com/MahmoudAshraf97/whisper-diarization/commit/accf5c07d231046de980b39a330b270a55259c17#diff-b315f4a191ca9ba5d54e13a3a56afd0066e1d4bdad453177290b9bfc307c20caL168

提供下更详细的信息，各个模块的version，我复现下

Name Version Build Channel
absl-py 2.1.0 pypi_0 pypi
accelerate 1.1.1 pypi_0 pypi
aiohappyeyeballs 2.4.3 pypi_0 pypi
aiohttp 3.11.7 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
async-timeout 5.0.1 pypi_0 pypi
attrs 24.2.0 pypi_0 pypi
audioread 3.0.1 pypi_0 pypi
av 13.1.0 pypi_0 pypi
blas 1.0 mkl
braceexpand 0.1.7 pypi_0 pypi
brotli-python 1.0.9 py310hd77b12b_8
bzip2 1.0.8 h2bbff1b_6
ca-certificates 2024.9.24 haa95532_0
certifi 2024.8.30 py310haa95532_0
cffi 1.17.1 pypi_0 pypi
charset-normalizer 3.3.2 pyhd3eb1b0_0
click 8.1.7 pypi_0 pypi
cloudpickle 3.1.0 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
contourpy 1.3.1 pypi_0 pypi
ctc-forced-aligner 0.2 pypi_0 pypi
ctranslate2 4.5.0 pypi_0 pypi
cuda-cccl 12.6.77 0 nvidia
cuda-cccl_win-64 12.6.77 0 nvidia
cuda-cudart 12.1.105 0 nvidia
cuda-cudart-dev 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-libraries-dev 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvrtc-dev 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.6.77 0 nvidia
cuda-opencl-dev 12.6.77 0 nvidia
cuda-profiler-api 12.6.77 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cuda-version 12.6 3 nvidia
cycler 0.12.1 pypi_0 pypi
cython 3.0.11 pypi_0 pypi
cytoolz 1.0.0 pypi_0 pypi
datasets 3.1.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
deepmultilingualpunctuation 1.0.1 pypi_0 pypi
demucs 4.1.0a3 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distance 0.1.3 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
docopt 0.6.2 pypi_0 pypi
dora-search 0.1.12 pypi_0 pypi
editdistance 0.8.1 pypi_0 pypi
einops 0.8.0 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
executing 2.1.0 pypi_0 pypi
faster-whisper 1.1.0 pypi_0 pypi
fiddle 0.3.0 pypi_0 pypi
filelock 3.13.1 py310haa95532_0
flatbuffers 24.3.25 pypi_0 pypi
fonttools 4.55.0 pypi_0 pypi
freetype 2.12.1 ha860e81_0
frozenlist 1.5.0 pypi_0 pypi
fsspec 2024.9.0 pypi_0 pypi
future 1.0.0 pypi_0 pypi
g2p-en 2.1.0 pypi_0 pypi
giflib 5.2.2 h7edc060_0
gitdb 4.0.11 pypi_0 pypi
gitpython 3.1.43 pypi_0 pypi
gmpy2 2.1.2 py310h7f96b67_0
grpcio 1.68.0 pypi_0 pypi
huggingface-hub 0.23.5 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
hydra-core 1.3.2 pypi_0 pypi
idna 3.7 py310haa95532_0
inflect 7.4.0 pypi_0 pypi
intel-openmp 2023.1.0 h59b6b97_46320
intervaltree 3.1.0 pypi_0 pypi
ipython 8.29.0 pypi_0 pypi
ipywidgets 8.1.5 pypi_0 pypi
jedi 0.19.2 pypi_0 pypi
jinja2 3.1.4 py310haa95532_1
jiwer 3.0.5 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
jpeg 9e h827c3e9_3
julius 0.2.7 pypi_0 pypi
jupyterlab-widgets 3.0.13 pypi_0 pypi
kaldi-python-io 1.2.2 pypi_0 pypi
kaldiio 2.18.0 pypi_0 pypi
kiwisolver 1.4.7 pypi_0 pypi
lameenc 1.7.0 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
lcms2 2.12 h83e58a3_0
lerc 3.0 hd77b12b_0
levenshtein 0.26.1 pypi_0 pypi
lhotse 1.28.0 pypi_0 pypi
libcst 1.5.1 pypi_0 pypi
libcublas 12.1.0.26 0 nvidia
libcublas-dev 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufft-dev 11.0.2.4 0 nvidia
libcurand 10.3.7.77 0 nvidia
libcurand-dev 10.3.7.77 0 nvidia
libcusolver 11.4.4.55 0 nvidia
libcusolver-dev 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libcusparse-dev 12.0.2.55 0 nvidia
libdeflate 1.17 h2bbff1b_1
libffi 3.4.4 hd77b12b_1
libjpeg-turbo 2.0.0 h196d8e1_0
libnpp 12.0.2.50 0 nvidia
libnpp-dev 12.0.2.50 0 nvidia
libnvjitlink 12.1.105 0 nvidia
libnvjitlink-dev 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libnvjpeg-dev 12.1.1.14 0 nvidia
libpng 1.6.39 h8cc25b3_0
librosa 0.10.2.post1 pypi_0 pypi
libtiff 4.5.1 hd77b12b_0
libuv 1.48.0 h827c3e9_0
libwebp 1.3.2 hbc33d0d_0
libwebp-base 1.3.2 h3d04722_1
lightning-utilities 0.11.9 pypi_0 pypi
lilcom 1.8.0 pypi_0 pypi
llvmlite 0.43.0 pypi_0 pypi
loguru 0.7.2 pypi_0 pypi
lz4-c 1.9.4 h2bbff1b_1
markdown 3.7 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.3 py310h2bbff1b_0
marshmallow 3.23.1 pypi_0 pypi
matplotlib 3.9.2 pypi_0 pypi
matplotlib-inline 0.1.7 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mkl 2023.1.0 h6b88ed4_46358
mkl-service 2.4.0 py310h2bbff1b_1
mkl_fft 1.3.11 py310h827c3e9_0
mkl_random 1.2.8 py310hc64d2fc_0
more-itertools 10.5.0 pypi_0 pypi
mpc 1.1.0 h7edee0f_1
mpfr 4.0.2 h62dcd97_1
mpir 3.0.0 hec2e145_1
mpmath 1.3.0 py310haa95532_0
msgpack 1.1.0 pypi_0 pypi
multidict 6.1.0 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
nemo-toolkit 2.0.0rc0 pypi_0 pypi
networkx 3.2.1 py310haa95532_0
nltk 3.9.1 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
onnx 1.17.0 pypi_0 pypi
onnxruntime 1.20.1 pypi_0 pypi
openjpeg 2.5.2 hae555c5_0
openssl 3.0.15 h827c3e9_0
openunmix 1.3.0 pypi_0 pypi
packaging 24.2 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
pillow 11.0.0 py310hb5480e2_0
pip 24.2 py310haa95532_0
plac 1.4.3 pypi_0 pypi
platformdirs 4.3.6 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
prompt-toolkit 3.0.48 pypi_0 pypi
propcache 0.2.0 pypi_0 pypi
protobuf 5.28.3 pypi_0 pypi
psutil 6.1.0 pypi_0 pypi
pure-eval 0.2.3 pypi_0 pypi
pyannote-core 5.0.0 pypi_0 pypi
pyannote-database 5.1.0 pypi_0 pypi
pyannote-metrics 3.2.1 pypi_0 pypi
pyarrow 18.0.0 pypi_0 pypi
pybind11 2.13.6 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
pyloudnorm 0.1.1 pypi_0 pypi
pyparsing 3.2.0 pypi_0 pypi
pyreadline3 3.5.4 pypi_0 pypi
pysocks 1.7.1 py310haa95532_0
python 3.10.15 h4607a30_1
python-dateutil 2.9.0.post0 pypi_0 pypi
python-graphviz 0.20.3 pypi_0 pypi
pytorch 2.5.1 py3.10_cuda12.1_cudnn9_0 pytorch
pytorch-cuda 12.1 hde6ce7c_6 pytorch
pytorch-lightning 2.4.0 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pytz 2024.2 pypi_0 pypi
pyyaml 6.0.2 py310h827c3e9_0
rapidfuzz 3.10.1 pypi_0 pypi
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 py310haa95532_1
resampy 0.4.3 pypi_0 pypi
retrying 1.3.4 pypi_0 pypi
rich 13.9.4 pypi_0 pypi
ruamel-yaml 0.18.6 pypi_0 pypi
ruamel-yaml-clib 0.2.12 pypi_0 pypi
sacremoses 0.1.1 pypi_0 pypi
safetensors 0.4.5 pypi_0 pypi
scikit-learn 1.5.2 pypi_0 pypi
scipy 1.14.1 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
sentry-sdk 2.19.0 pypi_0 pypi
setproctitle 1.3.4 pypi_0 pypi
setuptools 75.1.0 py310haa95532_0
shellingham 1.5.4 pypi_0 pypi
six 1.16.0 pypi_0 pypi
smmap 5.0.1 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
soundfile 0.12.1 pypi_0 pypi
sox 1.5.0 pypi_0 pypi
soxr 0.5.0.post1 pypi_0 pypi
sqlite 3.45.3 h2bbff1b_0
stack-data 0.6.3 pypi_0 pypi
submitit 1.5.2 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tbb 2021.8.0 h59b6b97_0
tensorboard 2.18.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
termcolor 2.5.0 pypi_0 pypi
text-unidecode 1.3 pypi_0 pypi
texterrors 0.5.1 pypi_0 pypi
threadpoolctl 3.5.0 pypi_0 pypi
tk 8.6.14 h0416ee5_0
tokenizers 0.19.1 pypi_0 pypi
toolz 1.0.0 pypi_0 pypi
torchaudio 2.5.1 pypi_0 pypi
torchmetrics 1.6.0 pypi_0 pypi
torchvision 0.20.1 pypi_0 pypi
tqdm 4.67.0 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
transformers 4.40.2 pypi_0 pypi
treetable 0.2.5 pypi_0 pypi
typeguard 4.4.1 pypi_0 pypi
typer 0.13.1 pypi_0 pypi
typing_extensions 4.11.0 py310haa95532_0
tzdata 2024.2 pypi_0 pypi
unidecode 1.3.8 pypi_0 pypi
urllib3 2.2.3 py310haa95532_0
vc 14.40 h2eaa2aa_1
vs2015_runtime 14.40.33807 h98bb1dd_1
wandb 0.18.7 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webdataset 0.2.100 pypi_0 pypi
werkzeug 3.1.3 pypi_0 pypi
wget 3.2 pypi_0 pypi
wheel 0.44.0 py310haa95532_0
widgetsnbextension 4.0.13 pypi_0 pypi
win32-setctime 1.1.0 pypi_0 pypi
win_inet_pton 1.1.0 py310haa95532_0
wrapt 1.17.0 pypi_0 pypi
xxhash 3.5.0 pypi_0 pypi
xz 5.4.6 h8cc25b3_1
yaml 0.2.5 he774522_0
yarl 1.18.0 pypi_0 pypi
zlib 1.2.13 h8cc25b3_1
zstd 1.5.6 h8880b57_0

应该不是whisper-diarization的问题，我之前尝试过只用faster_whisper.WhisperModel加载模型也不行，我等会在看看，麻烦你了

shuaijiang

BELLE-2 Group // Be Everyone's Large Language model Engine org Nov 27, 2024

ct2-transformers-converter --model model/Belle-whisper-large-v3-turbo-zh --output_dir model/Belle-whisper-large-v3-turbo-zh-ct2 --copy_files tokenizer.json preprocessor_config.json - --quantization float16
加上preprocessor_config.json

maomaoxion

Nov 27, 2024

ct2-transformers-converter --model model/Belle-whisper-large-v3-turbo-zh --output_dir model/Belle-whisper-large-v3-turbo-zh-ct2 --copy_files tokenizer.json preprocessor_config.json - --quantization float16
加上preprocessor_config.json

好的，可以运行了，非常感谢！

maomaoxion changed discussion status to closed Nov 27, 2024

BELLE-2
/

Belle-whisper-large-v3-zh

模型转换 ct2