Passed argument batch_size = auto:4.0. Detecting largest batch size
Determined largest batch size: 64
Passed argument batch_size = auto:4.0. Detecting largest batch size
Determined largest batch size: 64
hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (64,64,64,64,64)
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |↑  |0.2872|±  |0.0045|
|         |       |none  |     0|acc_norm|↑  |0.3082|±  |0.0046|

2024-10-17 23:20:11.330614: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-10-17 23:20:11.352779: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-10-17 23:20:11.358651: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-17 23:20:11.372585: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-17 23:20:12.527182: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-10-17:23:20:16,572 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-17:23:20:27,362 INFO     [__main__.py:376] Selected Tasks: ['hellaswag']
2024-10-17:23:20:27,366 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-17:23:20:27,366 INFO     [evaluator.py:201] Initializing hf model, with arguments: {'pretrained': 'EleutherAI/pythia-160m', 'revision': 'step100000', 'dtype': 'float'}
2024-10-17:23:20:27,450 INFO     [huggingface.py:129] Using device 'cuda'
2024-10-17:23:20:27,706 INFO     [huggingface.py:481] Using model type 'default'
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
2024-10-17:23:20:29,024 INFO     [huggingface.py:365] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda'}
Downloading data: 100%|██████████| 47.5M/47.5M [00:00<00:00, 169MB/s]
Downloading data: 100%|██████████| 11.8M/11.8M [00:00<00:00, 143MB/s]
Downloading data: 100%|██████████| 12.2M/12.2M [00:00<00:00, 173MB/s]
Generating train split: 100%|██████████| 39905/39905 [00:04<00:00, 9713.15 examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 10216.87 examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:01<00:00, 9867.41 examples/s]
Map: 100%|██████████| 39905/39905 [00:07<00:00, 5426.38 examples/s]
Map: 100%|██████████| 10042/10042 [00:01<00:00, 5869.01 examples/s]
2024-10-17:23:21:05,908 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-17:23:21:05,909 INFO     [task.py:415] Building contexts for hellaswag on rank 0...
100%|██████████| 10042/10042 [00:05<00:00, 1945.63it/s]
2024-10-17:23:21:12,103 INFO     [evaluator.py:489] Running loglikelihood requests
Running loglikelihood requests: 100%|██████████| 40168/40168 [04:01<00:00, 166.51it/s]
fatal: not a git repository (or any of the parent directories): .git
2024-10-17:23:25:41,634 INFO     [evaluation_tracker.py:206] Saving results aggregated