Passed argument batch_size = auto:4.0. Detecting largest batch size Determined largest batch size: 64 Passed argument batch_size = auto:4.0. Detecting largest batch size Determined largest batch size: 64 hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (64,64,64,64,64)
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2872 | ± | 0.0045 |
| none | 0 | acc_norm | ↑ | 0.3082 | ± | 0.0046 |
2024-10-17 23:20:11.330614: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-10-17 23:20:11.352779: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-10-17 23:20:11.358651: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-17 23:20:11.372585: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-17 23:20:12.527182: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-10-17:23:20:16,572 INFO [main.py:279] Verbosity set to INFO
2024-10-17:23:20:27,362 INFO [main.py:376] Selected Tasks: ['hellaswag']
2024-10-17:23:20:27,366 INFO [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-17:23:20:27,366 INFO [evaluator.py:201] Initializing hf model, with arguments: {'pretrained': 'EleutherAI/pythia-160m', 'revision': 'step100000', 'dtype': 'float'}
2024-10-17:23:20:27,450 INFO [huggingface.py:129] Using device 'cuda'
2024-10-17:23:20:27,706 INFO [huggingface.py:481] Using model type 'default'
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
2024-10-17:23:20:29,024 INFO [huggingface.py:365] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda'}
Downloading data: 100%|██████████| 47.5M/47.5M [00:00<00:00, 169MB/s]
Downloading data: 100%|██████████| 11.8M/11.8M [00:00<00:00, 143MB/s]
Downloading data: 100%|██████████| 12.2M/12.2M [00:00<00:00, 173MB/s]
Generating train split: 100%|██████████| 39905/39905 [00:04<00:00, 9713.15 examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 10216.87 examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:01<00:00, 9867.41 examples/s]
Map: 100%|██████████| 39905/39905 [00:07<00:00, 5426.38 examples/s]
Map: 100%|██████████| 10042/10042 [00:01<00:00, 5869.01 examples/s]
2024-10-17:23:21:05,908 WARNING [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-17:23:21:05,909 INFO [task.py:415] Building contexts for hellaswag on rank 0...
100%|██████████| 10042/10042 [00:05<00:00, 1945.63it/s]
2024-10-17:23:21:12,103 INFO [evaluator.py:489] Running loglikelihood requests
Running loglikelihood requests: 100%|██████████| 40168/40168 [04:01<00:00, 166.51it/s]
fatal: not a git repository (or any of the parent directories): .git
2024-10-17:23:25:41,634 INFO [evaluation_tracker.py:206] Saving results aggregated