Session crashing on Colab while running the model
#1
by
shubham13596
- opened
Tried running the code as given in the model description to begin with. However my session keeps crashing. Anyone else facing the same issue? Anyone who has been able to successfully run that code on Google Colab?
Same here
I'm running it on my 4080 and 4090 and it's crashing no matter how many nodes I get it to
Can you share the stacktrace?
| Timestamp | Level | Message |
|---|---|---|
| Aug 14, 2024, 1:37:17 AM | WARNING | WARNING:root:kernel 8d9d0099-7015-4dc0-842b-e889caaa807d restarted |
| Aug 14, 2024, 1:37:17 AM | INFO | KernelRestarter: restarting kernel (1/5), keep random ports |
| Aug 14, 2024, 1:34:29 AM | WARNING | 2024-08-13 20:04:29.453216: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT |
| Aug 14, 2024, 1:34:28 AM | WARNING | To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. |
| Aug 14, 2024, 1:34:28 AM | WARNING | 2024-08-13 20:04:28.080413: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. |
| Aug 14, 2024, 1:34:27 AM | WARNING | 2024-08-13 20:04:27.886708: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered |
| Aug 14, 2024, 1:34:27 AM | WARNING | 2024-08-13 20:04:27.815225: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered |
| Aug 14, 2024, 1:34:27 AM | WARNING | 2024-08-13 20:04:27.537977: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered |
| Aug 14, 2024, 1:32:17 AM | INFO | Kernel started: 8d9d0099-7015-4dc0-842b-e889caaa807d, name: python3 |
| Aug 14, 2024, 1:30:01 AM | INFO | Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). |
| Aug 14, 2024, 1:30:01 AM | INFO | http://172\.28\.0\.12:9000/ |
| Aug 14, 2024, 1:30:01 AM | INFO | Jupyter Notebook 6.5.5 is running at: |
| Aug 14, 2024, 1:30:01 AM | INFO | Serving notebooks from local directory: / |
| Aug 14, 2024, 1:29:59 AM | INFO | Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). |
| Aug 14, 2024, 1:29:59 AM | INFO | http://172\.28\.0\.2:9000/ |
| Aug 14, 2024, 1:29:59 AM | INFO | Jupyter Notebook 6.5.5 is running at: |
| Aug 14, 2024, 1:29:59 AM | INFO | Serving notebooks from local directory: / |
| Aug 14, 2024, 1:29:56 AM | INFO | Loading IPython parallel extension |
| Aug 14, 2024, 1:29:56 AM | INFO | google.colab serverextension initialized. |
| Aug 14, 2024, 1:29:56 AM | INFO | Loading IPython parallel extension |
| Aug 14, 2024, 1:29:56 AM | INFO | Authentication of /metrics is OFF, since other authentication is disabled. |
| Aug 14, 2024, 1:29:56 AM | INFO | Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret |
| Aug 14, 2024, 1:29:56 AM | WARNING | /root/.jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /root/.local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.d/ipyparallel.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | INFO | google.colab serverextension initialized. |
| Aug 14, 2024, 1:29:56 AM | INFO | Authentication of /metrics is OFF, since other authentication is disabled. |
| Aug 14, 2024, 1:29:56 AM | INFO | Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret |
| Aug 14, 2024, 1:29:56 AM | WARNING | /root/.jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /root/.local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /usr/local/etc/jupyter/jupyter_notebook_config.d/ipyparallel.json |
| Aug 14, 2024, 1:29:56 AM | WARNING | /etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.663 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.659 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.659 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.659 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.658 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.655 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.653 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.647 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.641 NotebookApp] Looking for jupyter_config in /root/.jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.641 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.640 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.634 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.631 NotebookApp] Looking for jupyter_config in /etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.628 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.526 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.524 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.523 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.523 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.523 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.521 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.521 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.519 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.518 NotebookApp] Looking for jupyter_config in /root/.jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.518 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.518 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.513 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.511 NotebookApp] Looking for jupyter_config in /etc/jupyter |
| Aug 14, 2024, 1:29:55 AM | WARNING | [D 19:59:55.510 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files |
This was my error log
Ran it on an A1000 too. Did not work
There was an issue with the audio processor. It should be fixed now. Please try and let me know.
Also, you need at least 16G memory to run this model. So you probably cannot do it in a free colab notebook.
Whoever wants to inference it, just a few quick notes,
- The total model size is approximately:
4.94 GB + 5 GB + 4.92 GB + 2.59 GB ≈ 17.45 GB - For inference, you typically need:
1. Space for the model weights
2. Additional memory for activations, temporary computations, and input/output buffers - A general rule of thumb is to have 1.2 to 1.5 times the model size available as VRAM, i.e. in this case 20.94 GB to 26.18 GB of VRAM.
- Using Runpod, I tried with A4500(20GB of VRAM) and A5000(24 GB of VRAM), and got out of memory error, so of course the T4(16GB VRAM) on google colab won't work.
- Try it on A40, A6000 or A100. You can follow this notebook, I ran it on runpod.
https://github.com/rs545837/Shuka_Inference/blob/main/Shuka.ipynb
All in all, I want to say hats off to Sarvam's team. The model seems really robust. Tried with a lot of noise and fumbles and extremely fast speech, it's amazing. Congrats, @rahular . When is the technical blog coming out?
I was able to run on colab
https://github.com/thefirehacker/Sarvam_ShukaV1_Demo
Runs smoothly on L4 , takes around 17GB of GPU
rahular
changed discussion status to
closed