Hi @AIHero123 ,

Can you send me the output of running pip freeze?

I couldn't reproduce your error -- In a fresh venv with pyhton3.11, running

pip install sentencepiece
pip install transformers
pip install tokenizers==0.13.3
pip install protobuf

And then

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("togethercomputer/LLaMA-2-7B-32K")
print(tokenizer("Apples and Oranges."))

returns

{'input_ids': [2401, 793, 322, 1394, 6916, 29889], 'attention_mask': [1, 1, 1, 1, 1, 1]}

Hi,

Here's a long list based on the output of pip freeze.

Regards,
Andy

absl-py==1.4.0
accelerate==0.22.0
aiohttp==3.8.5
aiosignal==1.3.1
alabaster @ file:///home/ktietz/src/ci/alabaster_1611921544520/work
altair==5.1.1
anaconda-client==1.11.0
anaconda-project @ file:///C:/Windows/TEMP/abs_91fu4tfkih/croots/recipe/anaconda-project_1660339890874/work
annotated-types==0.5.0
anyio @ file:///C:/ci/anyio_1644481921011/work/dist
appdirs==1.4.4
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///C:/ci/argon2-cffi-bindings_1644551690056/work
arrow @ file:///opt/conda/conda-bld/arrow_1649166651673/work
astroid @ file:///C:/Windows/TEMP/abs_b0dtxgpicv/croots/recipe/astroid_1659023126745/work
astropy @ file:///C:/ci/astropy_1657719656942/work
astunparse==1.6.3
async-timeout==4.0.3
atomicwrites==1.4.0
attrs @ file:///opt/conda/conda-bld/attrs_1642510447205/work
Automat @ file:///tmp/build/80754af9/automat_1600298431173/work
autopep8 @ file:///opt/conda/conda-bld/autopep8_1650463822033/work
Babel @ file:///tmp/build/80754af9/babel_1620871417480/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
backports.functools-lru-cache @ file:///tmp/build/80754af9/backports.functools_lru_cache_1618170165463/work
backports.tempfile @ file:///home/linux1/recipes/ci/backports.tempfile_1610991236607/work
backports.weakref==1.0.post1
bcrypt @ file:///C:/Windows/Temp/abs_36kl66t_aw/croots/recipe/bcrypt_1659554334050/work
beautifulsoup4 @ file:///C:/ci/beautifulsoup4_1650293025093/work
binaryornot @ file:///tmp/build/80754af9/binaryornot_1617751525010/work
bitarray @ file:///C:/ci/bitarray_1657729621682/work
bitsandbytes-windows==0.37.5
bkcharts==0.2
black @ file:///C:/ci/black_1660239974023/work
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
blinker==1.6.2
bokeh @ file:///C:/Windows/TEMP/abs_4a259bc2-ed05-4a1f-808e-ac712cc0900cddqp8sp7/croots/recipe/bokeh_1658136660686/work
boto3 @ file:///C:/Windows/TEMP/abs_4009c406-44ba-4406-8996-204d9b11202flt4kglbk/croots/recipe/boto3_1657820114895/work
botocore @ file:///C:/ci/botocore_1657735875454/work
Bottleneck @ file:///C:/Windows/Temp/abs_3198ca53-903d-42fd-87b4-03e6d03a8381yfwsuve8/croots/recipe/bottleneck_1657175565403/work
brotlipy==0.7.0
cachetools==5.3.1
certifi @ file:///C:/b/abs_36eb5mzhph/croot/certifi_1690232276943/work/certifi
cffi @ file:///C:/Windows/Temp/abs_6808y9x40v/croots/recipe/cffi_1659598653989/work
chardet @ file:///C:/ci/chardet_1607706937985/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click @ file:///C:/ci/click_1646038595831/work
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1632508026186/work
clyent==1.2.2
cmake==3.27.4.1
colorama @ file:///C:/Windows/TEMP/abs_9439aeb1-0254-449a-96f7-33ab5eb17fc8apleb4yn/croots/recipe/colorama_1657009099097/work
colorcet @ file:///C:/ci/colorcet_1651851676912/work
comtypes==1.1.10
conda-content-trust @ file:///C:/Windows/TEMP/abs_4589313d-fc62-4ccc-81c0-b801b4449e833j1ajrwu/croots/recipe/conda-content-trust_1658126379362/work
conda-pack @ file:///tmp/build/80754af9/conda-pack_1611163042455/work
conda-package-handling @ file:///C:/b/abs_b9wp3lr1gn/croot/conda-package-handling_1691008700066/work
conda-repo-cli==1.0.41
conda-verify==3.4.2
conda_package_streaming @ file:///C:/b/abs_6c28n38aaj/croot/conda-package-streaming_1690988019210/work
constantly==15.1.0
cookiecutter @ file:///opt/conda/conda-bld/cookiecutter_1649151442564/work
cryptography @ file:///C:/ci/cryptography_1652083563162/work
cssselect==1.1.0
ctransformers==0.2.27
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
Cython @ file:///C:/b/abs_0438epndxm/croots/recipe/cython_1663692771227/work
cytoolz==0.11.0
daal4py==2021.6.0
dask @ file:///C:/ci/dask-core_1658515307198/work
dataclasses-json==0.5.14
datashader @ file:///C:/Windows/TEMP/abs_62cyd2dpuf/croots/recipe/datashader_1659349034750/work
datashape==0.5.4
debugpy @ file:///C:/ci/debugpy_1637091961445/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
diff-match-patch @ file:///Users/ktietz/demo/mc3/conda-bld/diff-match-patch_1630511840874/work
dill @ file:///tmp/build/80754af9/dill_1623919422540/work
distributed @ file:///C:/ci/distributed_1658505715201/work
dlib @ file:///C:/Users/ADMIN/dlib-19.22.99-cp39-cp39-win_amd64.whl
docutils @ file:///C:/Windows/TEMP/abs_24e5e278-4d1c-47eb-97b9-f761d871f482dy2vg450/croots/recipe/docutils_1657175444608/work
einops==0.6.1
entrypoints @ file:///C:/ci/entrypoints_1649926621128/work
et-xmlfile==1.1.0
fastjsonschema @ file:///C:/Users/BUILDE1/AppData/Local/Temp/abs_ebruxzvd08/croots/recipe/python-fastjsonschema_1661376484940/work
filelock @ file:///opt/conda/conda-bld/filelock_1647002191454/work
flake8 @ file:///opt/conda/conda-bld/flake8_1648129545443/work
flash-attn==1.0.9
Flask @ file:///home/ktietz/src/ci/flask_1611932660458/work
flatbuffers==23.5.26
fonttools==4.25.0
frozenlist==1.4.0
fsspec @ file:///C:/Windows/TEMP/abs_a2mhnomvfy/croots/recipe/fsspec_1659972224540/work
future @ file:///C:/ci/future_1607568713721/work
gast==0.4.0
gensim @ file:///C:/ci/gensim_1646825438310/work
gguf==0.3.2
gitdb==4.0.10
GitPython==3.1.36
glob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/work
google-auth==2.22.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
greenlet @ file:///C:/ci/greenlet_1628888275363/work
grpcio==1.58.0
h5py @ file:///C:/ci/h5py_1659089875384/work
HeapDict @ file:///Users/ktietz/demo/mc3/conda-bld/heapdict_1630598515714/work
holoviews @ file:///C:/Windows/TEMP/abs_fb9d2988-9681-46b6-b9ab-190cfecd15d7uinyq_4z/croots/recipe/holoviews_1658171511842/work
huggingface-hub==0.16.4
hvplot @ file:///C:/Windows/TEMP/abs_02zoq6lck3/croots/recipe/hvplot_1659026502064/work
hyperlink @ file:///tmp/build/80754af9/hyperlink_1610130746837/work
idna @ file:///tmp/build/80754af9/idna_1637925883363/work
imagecodecs @ file:///C:/b/abs_948ub5byiu/croots/recipe/imagecodecs_1664562381493/work
imageio @ file:///C:/Windows/TEMP/abs_24c1b783-7540-4ca9-a1b1-0e8aa8e6ae64hb79ssux/croots/recipe/imageio_1658785038775/work
imagesize @ file:///C:/Windows/TEMP/abs_3cecd249-3fc4-4bfc-b80b-bb227b0d701en12vqzot/croots/recipe/imagesize_1657179501304/work
importlib-metadata @ file:///C:/ci/importlib-metadata_1648562621412/work
incremental @ file:///tmp/build/80754af9/incremental_1636629750599/work
inflection==0.5.1
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
intake @ file:///opt/conda/conda-bld/intake_1647436631684/work
intervaltree @ file:///Users/ktietz/demo/mc3/conda-bld/intervaltree_1630511889664/work
ipykernel @ file:///C:/b/abs_21ykzkm7y_/croots/recipe/ipykernel_1662361803478/work
ipython @ file:///C:/Windows/TEMP/abs_45b5zb1l7q/croots/recipe/ipython_1659529855872/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1634143127070/work
isort @ file:///tmp/build/80754af9/isort_1628603791788/work
itemadapter @ file:///tmp/build/80754af9/itemadapter_1626442940632/work
itemloaders @ file:///opt/conda/conda-bld/itemloaders_1646805235997/work
itsdangerous @ file:///tmp/build/80754af9/itsdangerous_1621432558163/work
jdcal @ file:///Users/ktietz/demo/mc3/conda-bld/jdcal_1630584345063/work
jedi @ file:///C:/ci/jedi_1644315428289/work
jellyfish @ file:///C:/ci/jellyfish_1647962783748/work
Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work
jinja2-time @ file:///opt/conda/conda-bld/jinja2-time_1649251842261/work
jmespath @ file:///Users/ktietz/demo/mc3/conda-bld/jmespath_1630583964805/work
joblib @ file:///tmp/build/80754af9/joblib_1635411271373/work
json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
jsonschema @ file:///C:/b/abs_59eyhnbyej/croots/recipe/jsonschema_1663375476535/work
jupyter @ file:///C:/Windows/TEMP/abs_56xfdi__li/croots/recipe/jupyter_1659349053177/work
jupyter-console @ file:///opt/conda/conda-bld/jupyter_console_1647002188872/work
jupyter-server @ file:///C:/Windows/TEMP/abs_d3c42c59-765d-4f9b-9fa3-ad5b1369485611i_yual/croots/recipe/jupyter_server_1658754493238/work
jupyter_client @ file:///C:/ci/jupyter_client_1661836943389/work
jupyter_core @ file:///C:/b/abs_a9330r1z_i/croots/recipe/jupyter_core_1664917313457/work
jupyterlab @ file:///C:/ci/jupyterlab_1658891142428/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-server @ file:///opt/conda/conda-bld/jupyterlab_server_1644500396812/work
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
keras==2.10.0
Keras-Preprocessing==1.1.2
keyring @ file:///C:/ci/keyring_1638531673471/work
kiwisolver @ file:///C:/ci/kiwisolver_1653292407425/work
langchain==0.0.285
langsmith==0.0.35
lazy-object-proxy @ file:///C:/ci/lazy-object-proxy_1616529288960/work
libarchive-c @ file:///tmp/build/80754af9/python-libarchive-c_1617780486945/work
libclang==16.0.6
llvmlite==0.38.0
locket @ file:///C:/ci/locket_1652904031364/work
lxml @ file:///C:/ci/lxml_1657527445690/work
lz4 @ file:///C:/ci/lz4_1619516674350/work
Markdown @ file:///C:/ci/markdown_1614364082838/work
markdown-it-py==3.0.0
MarkupSafe @ file:///C:/ci/markupsafe_1621528502553/work
marshmallow==3.20.1
matplotlib @ file:///C:/ci/matplotlib-suite_1660169687702/work
matplotlib-inline @ file:///C:/ci/matplotlib-inline_1661915841596/work
mccabe==0.6.1
mdurl==0.1.2
menuinst @ file:///C:/Users/BUILDE1/AppData/Local/Temp/abs_455sf5o0ct/croots/recipe/menuinst_1661805970842/work
mistune @ file:///C:/ci/mistune_1607359457024/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
mpmath==1.2.1
msgpack @ file:///C:/ci/msgpack-python_1652329316214/work
multidict==6.0.4
multipledispatch @ file:///C:/ci/multipledispatch_1607574329826/work
munkres==1.1.4
mypy-extensions==0.4.3
nbclassic @ file:///opt/conda/conda-bld/nbclassic_1644943264176/work
nbclient @ file:///C:/ci/nbclient_1650290387259/work
nbconvert @ file:///C:/ci/nbconvert_1649741016669/work
nbformat @ file:///C:/b/abs_1dw90o2uqb/croots/recipe/nbformat_1663744957967/work
nest-asyncio @ file:///C:/ci/nest-asyncio_1649829929390/work
networkx @ file:///C:/ci/networkx_1657716998256/work
ninja==1.11.1
nltk @ file:///opt/conda/conda-bld/nltk_1645628263994/work
nose @ file:///opt/conda/conda-bld/nose_1642704612149/work
notebook @ file:///C:/Windows/TEMP/abs_79abr1_60s/croots/recipe/notebook_1659083661851/work
numba @ file:///C:/ci/numba_1650394399948/work
numexpr==2.8.5
numpy==1.24.0
numpydoc @ file:///C:/Windows/TEMP/abs_30799058-86dd-4401-b621-d172137a4d87_ra3twm7/croots/recipe/numpydoc_1657529873713/work
nvidia-cublas-cu12==12.2.5.6
nvidia-cuda-runtime-cu12==12.2.140
oauthlib==3.2.2
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
openai==0.28.0
opencv-python==4.8.0.76
openpyxl==3.0.10
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
pandas @ file:///C:/b/abs_cdcgk91igc/croots/recipe/pandas_1663772960432/work
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
panel @ file:///C:/ci/panel_1657899702145/work
param @ file:///tmp/build/80754af9/param_1636647414893/work
paramiko @ file:///opt/conda/conda-bld/paramiko_1640109032755/work
parsel @ file:///C:/ci/parsel_1646740216444/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
partd @ file:///opt/conda/conda-bld/partd_1647245470509/work
pathlib @ file:///Users/ktietz/demo/mc3/conda-bld/pathlib_1629713961906/work
pathspec @ file:///C:/Windows/TEMP/abs_581d0u45mh/croots/recipe/pathspec_1659627132171/work
patsy==0.5.2
pep8==1.7.1
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==9.2.0
pkginfo @ file:///tmp/build/80754af9/pkginfo_1643162084911/work
platformdirs @ file:///C:/b/abs_73cc5cz_1u/croots/recipe/platformdirs_1662711386458/work
plotly @ file:///C:/ci/plotly_1658142442431/work
pluggy @ file:///C:/ci/pluggy_1648024580010/work
poyo @ file:///tmp/build/80754af9/poyo_1617751526755/work
prometheus-client @ file:///C:/Windows/TEMP/abs_ab9nx8qb08/croots/recipe/prometheus_client_1659455104602/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1633440160888/work
Protego @ file:///tmp/build/80754af9/protego_1598657180827/work
protobuf==3.20.1
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///opt/conda/conda-bld/py_1644396412707/work
py-cpuinfo==9.0.0
pyarrow==13.0.0
pyasn1 @ file:///Users/ktietz/demo/mc3/conda-bld/pyasn1_1629708007385/work
pyasn1-modules==0.2.8
pycodestyle @ file:///tmp/build/80754af9/pycodestyle_1636635402688/work
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pyct @ file:///C:/ci/pyct_1658488033428/work
pycurl==7.45.1
pydantic==2.3.0
pydantic_core==2.6.3
pydeck==0.8.0
PyDispatcher==2.0.5
pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1621600989141/work
pyerfa @ file:///C:/ci/pyerfa_1621560974055/work
pyflakes @ file:///tmp/build/80754af9/pyflakes_1636644436481/work
Pygments==2.16.1
PyHamcrest @ file:///tmp/build/80754af9/pyhamcrest_1615748656804/work
PyJWT @ file:///C:/ci/pyjwt_1657511236979/work
pylint @ file:///C:/Windows/TEMP/abs_518eqlbmoo/croots/recipe/pylint_1659110354241/work
pyls-spyder==0.4.0
Pympler==1.0.1
PyNaCl @ file:///C:/Windows/Temp/abs_d5c3ajcm87/croots/recipe/pynacl_1659620667490/work
pyodbc @ file:///C:/Windows/Temp/abs_61e3jz3u05/croots/recipe/pyodbc_1659513801402/work
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
pypiwin32==223
pyrsistent @ file:///C:/ci/pyrsistent_1636093225342/work
PySocks @ file:///C:/ci/pysocks_1605307512533/work
pytest==7.1.2
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
python-lsp-black @ file:///tmp/build/80754af9/python-lsp-black_1634232156041/work
python-lsp-jsonrpc==1.0.0
python-lsp-server @ file:///tmp/build/80754af9/python-lsp-server_1648176833691/work
python-slugify @ file:///tmp/build/80754af9/python-slugify_1620405669636/work
python-snappy @ file:///C:/ci/python-snappy_1610133405910/work
pytz @ file:///C:/Windows/TEMP/abs_90eacd4e-8eff-491e-b26e-f707eba2cbe1ujvbhqz1/croots/recipe/pytz_1654762631027/work
pytz-deprecation-shim==0.1.0.post0
pyuac==0.0.3
pyviz-comms @ file:///tmp/build/80754af9/pyviz_comms_1623747165329/work
PyWavelets @ file:///C:/ci/pywavelets_1648728084106/work
pywin32==302
pywin32-ctypes @ file:///C:/ci/pywin32-ctypes_1607553594546/work
pywinpty @ file:///C:/ci_310/pywinpty_1644230983541/work/target/wheels/pywinpty-2.0.2-cp39-none-win_amd64.whl
PyYAML==6.0
pyzmq @ file:///C:/ci/pyzmq_1657615952984/work
QDarkStyle @ file:///tmp/build/80754af9/qdarkstyle_1617386714626/work
qstylizer @ file:///tmp/build/80754af9/qstylizer_1617713584600/work/dist/qstylizer-0.1.10-py2.py3-none-any.whl
QtAwesome @ file:///tmp/build/80754af9/qtawesome_1637160816833/work
qtconsole @ file:///opt/conda/conda-bld/qtconsole_1643819126524/work
QtPy @ file:///C:/ci/qtpy_1662015096047/work
queuelib==1.5.0
regex @ file:///C:/ci/regex_1658258307256/work
requests @ file:///C:/ci/requests_1657735342357/work
requests-file @ file:///Users/ktietz/demo/mc3/conda-bld/requests-file_1629455781986/work
requests-oauthlib==1.3.1
rich==13.5.2
rope @ file:///opt/conda/conda-bld/rope_1643788605236/work
rotary-emb @ git+https://github.com/HazyResearch/flash-attention.git@bb9beb36454fd424c8971638c9e88939ee9fc528#subdirectory=csrc/rotary
rsa==4.9
Rtree @ file:///C:/ci/rtree_1618421015405/work
ruamel-yaml-conda @ file:///C:/ci/ruamel_yaml_1616016898638/work
s3transfer @ file:///C:/ci/s3transfer_1654512518418/work
sacremoses==0.0.43
safetensors==0.3.3
scikit-image @ file:///C:/ci/scikit-image_1648214340990/work
scikit-learn @ file:///C:/ci/scikit-learn_1642617276183/work
scikit-learn-intelex==2021.20221004.171935
scipy==1.9.1
Scrapy @ file:///C:/Windows/TEMP/abs_f50e21j997/croots/recipe/scrapy_1659598707153/work
seaborn @ file:///tmp/build/80754af9/seaborn_1629307859561/work
Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work
sentence-transformers==2.2.2
sentencepiece==0.1.98
service-identity @ file:///Users/ktietz/demo/mc3/conda-bld/service_identity_1629460757137/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1644875935023/work
smart-open @ file:///C:/ci/smart_open_1651235069716/work
smmap==5.0.0
sniffio @ file:///C:/ci/sniffio_1614030527509/work
snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1637937080595/work
sortedcollections @ file:///tmp/build/80754af9/sortedcollections_1611172717284/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1623949099177/work
soupsieve @ file:///tmp/build/80754af9/soupsieve_1636706018808/work
Sphinx @ file:///C:/ci/sphinx_1657617205740/work
sphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/work
sphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/work
sphinxcontrib-htmlhelp @ file:///tmp/build/80754af9/sphinxcontrib-htmlhelp_1623945626792/work
sphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/work
sphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/work
sphinxcontrib-serializinghtml @ file:///tmp/build/80754af9/sphinxcontrib-serializinghtml_1624451540180/work
spyder @ file:///C:/Windows/TEMP/abs_66k5aq_mkw/croots/recipe/spyder_1659599805372/work
spyder-kernels @ file:///C:/ci/spyder-kernels_1647011408404/work
SQLAlchemy @ file:///C:/Windows/Temp/abs_f8661157-660b-49bb-a790-69ab9f3b8f7c8a8s2psb/croots/recipe/sqlalchemy_1657867864564/work
statsmodels==0.13.2
streamlit==1.26.0
sympy @ file:///C:/ci/sympy_1647853873858/work
tables==3.6.1
tabulate @ file:///C:/ci/tabulate_1657619055201/work
TBB==0.2
tblib @ file:///Users/ktietz/demo/mc3/conda-bld/tblib_1629402031467/work
tee==0.0.3
tenacity==8.2.3
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.1
tensorflow-addons==0.21.0
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.3.0
terminado @ file:///C:/ci/terminado_1644322780199/work
testpath @ file:///C:/Windows/TEMP/abs_23c7fa33-cbb9-46dc-b7c5-590c38e2de3d4bmbngal/croots/recipe/testpath_1655908553202/work
text-unidecode @ file:///Users/ktietz/demo/mc3/conda-bld/text-unidecode_1629401354553/work
textdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/work
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
three-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/work
tifffile @ file:///tmp/build/80754af9/tifffile_1627275862826/work
tinycss @ file:///tmp/build/80754af9/tinycss_1617713798712/work
tldextract @ file:///opt/conda/conda-bld/tldextract_1646638314385/work
tokenizers==0.13.3
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work
tomlkit @ file:///C:/Windows/TEMP/abs_3296qo9v6b/croots/recipe/tomlkit_1658946894808/work
toolz @ file:///tmp/build/80754af9/toolz_1636545406491/work
torch==2.2.0.dev20230909+cu118
torchaudio==2.2.0.dev20230909+cu118
torchvision==0.17.0.dev20230909+cu118
tornado @ file:///C:/ci/tornado_1606924294691/work
tqdm @ file:///C:/b/abs_0axbz66qik/croots/recipe/tqdm_1664392691071/work
traitlets @ file:///tmp/build/80754af9/traitlets_1636710298902/work
transformers @ git+https://github.com/huggingface/transformers@5af2c6269672cda01c24ad48fab13f14a3ffb746
Twisted @ file:///C:/Windows/Temp/abs_ccblv2rzfa/croots/recipe/twisted_1659592764512/work
twisted-iocpsupport @ file:///C:/ci/twisted-iocpsupport_1646798932792/work
typeguard==2.13.3
typing-inspect==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
tzlocal==4.3.1
ujson @ file:///C:/ci/ujson_1657525944442/work
Unidecode @ file:///tmp/build/80754af9/unidecode_1614712377438/work
urllib3 @ file:///C:/Windows/TEMP/abs_65ynz4fdmi/croots/recipe/urllib3_1659110473919/work
validators==0.22.0
w3lib @ file:///Users/ktietz/demo/mc3/conda-bld/w3lib_1629359764703/work
watchdog @ file:///C:/ci/watchdog_1638367441841/work
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client @ file:///C:/ci/websocket-client_1614804375980/work
Werkzeug @ file:///opt/conda/conda-bld/werkzeug_1645628268370/work
widgetsnbextension @ file:///C:/ci/widgetsnbextension_1644991377168/work
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306162074/work
win-unicode-console==0.5
wincertstore==0.2
wrapt @ file:///C:/Windows/Temp/abs_7c3dd407-1390-477a-b542-fd15df6a24085_diwiza/croots/recipe/wrapt_1657814452175/work
xarray @ file:///opt/conda/conda-bld/xarray_1639166117697/work
xlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/work
XlsxWriter @ file:///opt/conda/conda-bld/xlsxwriter_1649073856329/work
xlwings @ file:///C:/b/abs_41rmpiyy58/croots/recipe/xlwings_1664932818417/work
yapf @ file:///tmp/build/80754af9/yapf_1615749224965/work
yarl==1.9.2
zict==2.1.0
zipp @ file:///C:/ci/zipp_1652273994994/work
zope.interface @ file:///C:/ci/zope.interface_1625036252485/work
zstandard==0.19.0

thanks for the list! It looks like when you import the AutoTokenizer from the transformers library, it is referencing the wrong version of tokenizers (0.13.0.dev0 instead of 0.13.3). I suspect this has to do with your installation of transformers. Is there a reason that you need the transformers version at commit 5af2c6269672cda01c24ad48fab13f14a3ffb746? If not, I would recommend to run a fresh reinstall of transformers and tokenizers.

I managed to solve the earlier error by manually deleting all references to the incorrect tokenizers.

However, I ran into another problem:
ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a tokenizers library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

can you try and reinstall sentencepiece using pip install --no-cache-dir sentencepiece? If this does not work, you can uninstall transformers and reinstall again using pip install --no-cache-dir transformers sentencepiece.