added inference tested data

app.py

@@ -41,8 +41,6 @@ with demo:
         follow the instructions and complete the form in the 🏎️ Submit tab. Models submitted to the leaderboard are evaluated 
         on the Intel Developer Cloud ☁️. The evaluation platform consists of Gaudi Accelerators and Xeon CPUs running benchmarks from
         the  [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness).""")
-    gr.Markdown("""Join  5000+ developers on the [Intel DevHub Discord](https://intel.ly/intelllmleaderboard_discord) to get support with your submission and 
-                talk about everything from GenAI, HPC, to Quantum Computing.""")
     gr.Markdown("""A special shout-out to the 🤗 [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) 
                 team for generously sharing their code and best 
                 practices, ensuring that AI Developers have a valuable and enjoyable tool at their disposal.""")
@@ -166,7 +164,9 @@ with demo:
                                      label="Model Types",
                                      elem_id="model_types",
                                      value=["pretrained","fine-tuned","chat-models","merges/moerges"])
-            
+                    inbox_text = gr.CheckboxGroup(label = """Inference Tested Column Legend: 🟨 = Gaudi, 🟦 = Xeon, 🟥 = GPU Max, 🟠 = Core Ultra, 🟢 = Arc GPU     (Please see "❓About" tab for more info)""")
+
+            # formatting model name and adding links
             color = '#2f82d4'
             def make_clickable(row):
                 return f'<a href="https://huggingface.co/{row["Model"]}" target="_blank" style="color: {color}; text-decoration: underline;">{row["Model"]}</a>'
@@ -192,10 +192,10 @@ with demo:
                                 ["pretrained","fine-tuned","chat-models","merges/moerges"])
             
             
-            gradio_df_display = gr.Dataframe(value=initial_filtered_df, headers=["Model","Average","Hardware","Model Type","Precision",
-                                                                                 "Size","Infrastructure","ARC","HellaSwag","MMLU",
-                                                                                 "TruthfulQA","Winogrande","Affiliation"],
-                                             datatype=["html","str","str","str","str","str","str","str","str","str","str","str","str"])
+            gradio_df_display = gr.Dataframe(value=initial_filtered_df, headers=["Inference Tested","Model","Average","ARC","HellaSwag","MMLU",
+                                                                                 "TruthfulQA","Winogrande","Training Hardware","Model Type","Precision",
+                                                                                 "Size","Infrastructure","Affiliation"],
+                                             datatype=["html","html","str","str","str","str","str","str","str","str","str","str","str","str"])
             
             filter_hw.change(fn=update_df, 
                              inputs=[filter_hw, filter_platform, filter_affiliation, filter_size, filter_precision, filter_type], 

info/about.py

@@ -18,7 +18,9 @@ domain-specific benchmarks in the future. We utilize the <a href="https://github
 Eleuther AI Language Model Evaluation Harness </a>, a unified framework to test generative language models on a large number of 
 different evaluation tasks.
 
-Our current benchmarks include:
+<hr> 
+
+## Our current benchmarks include:
 
 - <a href="https://arxiv.org/abs/1803.05457" target="_blank"> AI2 Reasoning Challenge (25-shot)</a> - a set of grade-school science questions.
 - <a href="https://arxiv.org/abs/1905.07830" target="_blank"> HellaSwag (10-shot)</a> - a test of commonsense inference, which is easy for humans (~95%) but challenging for state-of-the-art models.
@@ -30,6 +32,24 @@ For all these evaluations, a higher score is better. We've chosen these benchmar
 We run an adapted version of the benchmark code specifically designed to run the EleutherAI Harness benchmarks on Gaudi processors. 
 This adapted evaluation harness is built into the Hugging Face Optimum Habana Library. Review the documentation [here](https://github.com/huggingface/optimum-habana/tree/main/examples/text-generation).
 
+<hr> 
+
+## Inference Tested Leaderboad Column:
+
+This column classifies hardware platforms where a model has undergone testing, either directly or indirectly through testing of its base model. For instance, if Intel/neural-chat-7b-v3-3 is tested and works on
+Arc GPUs, then models that are fine-tuned from this model without any architectural or algorithmic changes will be classified as "Inference Tested." 
+
+### Legend and Column Interpretation
+Refer to the following emoji key to understand the "Inference Tested" column:
+
+Inference Tested Column Legend:
+🟨 = Gaudi, 🟦 = Xeon, 🟥 = GPU Max, 🟠 = Core Ultra, 🟢 = Arc GPU
+
+For example, if a model has the 🟨🟦 emojis in the "Inference Tested" column, it indicates that the model has been tested and works on both Gaudi and Xeon hardware platforms, without implying any performance claims or full optimization for the indicated platforms.
+
+
+<hr> 
+
 ## Support and Community 
 
 Join  5000+ developers on the [Intel DevHub Discord](https://discord.gg/yNYNxK2k) to get support with your submission 
@@ -41,6 +61,8 @@ This is a fun on-leaderboard LLM chat functionality designed to provide a quick
 As the leaderboard matures and users submit models, we will rotate the available models for chat. Who knows!? You might find
 your model featured here soon! ⭐
 
+<hr> 
+
 ### Chat Functionality Notice
 - All the models in this demo run on 4th Generation Intel® Xeon® (Sapphire Rapids) utilizing AMX operations and quantized inference optimizations.
 - Terms of use: By using the chat functionality, users are required to agree to the following terms: The service is a research preview intended for non-commercial 

info/submit.py

@@ -4,7 +4,9 @@ SUBMIT_TEXT = f"""
 Models added here will be queued for evaluation on the Intel Developer Cloud ☁️. Depending on the queue, your model may take up to 10 days to show up on the leaderboard. 
 We will work to create greater transperancy as our leaderboard community grows.
 
-## First steps before submitting a model
+<hr> 
+
+## Review these steps before submitting your model
 
 ### 1) Make sure you can load your model and tokenizer using AutoClasses:
 ```python
@@ -19,7 +21,7 @@ Note: Make sure your model is public!
 
 Note: If your model needs `use_remote_code=True`, we do not support this option yet, but we are working on adding it. 
 
-### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
+### 2) Consider converting your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
 It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`.
 
 ### 3) Make sure your model has an open license.
@@ -38,4 +40,5 @@ Not all models are converted properly from `float16` to `bfloat16`, and selectin
 ## In case of model failure
 If your model fails evaluation 😔, we will contact you by opening a new discussion in your model respository. Let's work together to get your model the love it deserves ❤️!
 
+<hr> 
 """

info/train_a_model.py

@@ -5,16 +5,20 @@ LLM_BENCHMARKS_TEXT = f"""
 Intel offers a variety of platforms that can be used to train LLMs including data center and consumer grade CPUs, GPUs, and ASICs. 
 Below, you can find documentation on how to access free and paid resources to train a model on Intel hardware and submit it to the Hugging Face Model Hub.
 
+<hr> 
+
 ## Intel Developer Cloud - Quick Start
 The Intel Developer Cloud is one of the best places to access free and paid compute instances for model training. Intel offers Jupyter Notebook instances supported by
 224 Core 4th Generation Xeon Bare Metal nodes with 4x GPU Max Series 1100. To access these resources please follow the instructions below:
 1. Visit the [Intel Developer Cloud](https://bit.ly/inteldevelopercloud) and sign up for the "Standard - Free" tier to get started.
 2. Navigate to the "Training" module under the "Software" section in the left panel.
-3. Under the GenAI Essentials section, select the LLM Fine-Tuning with QLoRA notebook and click "Launch".
+3. Under the GenAI Essentials section, select the [Gemma Model Fine-tuning using SFT and LoRA](https://console.idcservice.net/training/detail/99deeb99-b0c6-4d02-a1d5-a46d95344ff3) notebook and click "Launch".
 4. Follow the instructions in the notebook to train your model using Intel® Data Center GPU Max 1100.
 5. Upload your model to the Hugging Face Model Hub.
 6. Go to the "Submit" tab on this Leaderboard and follow the instructions to submit your model.
 
+<hr> 
+
 ## Training Code Samples
 Below are some resources to get you started on training models on Intel platforms:
 - Intel® Gaudi® Accelerators
@@ -23,8 +27,11 @@ Below are some resources to get you started on training models on Intel platform
     - [Distributed Training of GPT2 LLMs on AWS](https://github.com/intel/intel-cloud-optimizations-aws/tree/main/distributed-training)
     - [Fine-tuning Falcon 7B on Xeon Processors](https://medium.com/@eduand-alvarez/fine-tune-falcon-7-billion-on-xeon-cpus-with-hugging-face-and-oneapi-a25e10803a53)
 - Intel® Data Center GPU Max Series
+    - [Gemma Model Fine-tuning using SFT and LoRA](https://console.idcservice.net/training/detail/99deeb99-b0c6-4d02-a1d5-a46d95344ff3)
     - [LLM Fine-tuning with QLoRA on Max Series GPUs](https://console.idcservice.net/training/detail/159c24e4-5598-3155-a790-2qv973tlm172)
 
+<hr> 
+
 ## Submitting your Model to the Hugging Face Model Hub
 Once your model is trained, it is a straighforward process to upload and open source it on the Hugging Face Model Hub. The commands from a Jupyter notebook are given below:
 

src/processing.py

@@ -4,7 +4,7 @@ def filter_benchmarks_table(df, hw_selected, platform_selected,
                             type_selected, affiliation_selected):
  
     filtered_df = df[
-    df['Hardware'].isin(hw_selected) &
+    df['Training Hardware'].isin(hw_selected) &
     df['Infrastructure'].isin(platform_selected) &
     df['Size'].isin(size_selected) &
     df['Precision'].isin(precision_selected) &

status/leaderboard_status_041624.csv

@@ -1,15 +1,15 @@
-Model,Average,Hardware,Model Type,Precision,Size,Infrastructure,ARC,HellaSwag,MMLU,TruthfulQA,Winogrande,Affiliation
-Intel/neural-chat-7b-v3-3,71.574,Gaudi,fine-tuned,fp16,7,Intel Developer Cloud,66.89,85.26,63.07,63.01,79.64,Intel Engineering
-Intel/neural-chat-7b-v3-2,70.858,Gaudi,fine-tuned,fp16,7,Intel Developer Cloud,67.49,83.92,63.55,59.68,79.65,Intel Engineering
-Intel/neural-chat-7b-v3-1,70.002,Gaudi,fine-tuned,fp16,7,Intel Developer Cloud,66.21,83.64,62.37,59.65,78.14,Intel Engineering
-Intel/neural-chat-7b-v3,69.906,Gaudi,fine-tuned,fp16,7,Intel Developer Cloud,67.15,83.29,62.26,58.77,78.06,Intel Engineering
-Intel/neural-chat-7b-v3-1,69.89,Gaudi,fine-tuned,int8,7,Intel Developer Cloud,65.7,83.54,62.12,59.48,78.61,Intel Engineering
-Intel/neural-chat-7b-v3-1,69.972,Gaudi,fine-tuned,bf16,7,Intel Developer Cloud,66.3,83.6,62.44,59.54,77.98,Intel Engineering
-Intel/neural-chat-7b-v3-1,68.256,Gaudi,fine-tuned,int4,7,Intel Developer Cloud,64.25,82.49,60.79,56.4,77.35,Intel Engineering
-FunDialogues/dollygem-2b-LoRA,49.368,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,41.55,71.56,35.39,32.59,65.75,No Affiliation
-ThejasElandassery/dallema,49.134,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,41.64,72.01,33.67,33.08,65.27,Student Ambassador
-utkarshsingh99/confused-gemma,49.306,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,42.06,71.86,34.14,33.2,65.27,Student Ambassador
-myasaswin/Gemma2B-LORAfied,49.566,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,42.56,71.56,35.47,32.57,65.67,Student Ambassador
-Aprajita0/Gemma-2b-Lora,49.526,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,42.24,71.88,34.14,33.23,66.14,Student Ambassador
-gopalakrishnan-d/gemma-2b-dolly-ds-lora,49.378,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,42.24,71.88,34.11,33.15,65.51,Student Ambassador
-SSK0908/gemma-2b-dolly-qa,49.452,GPU Max,fine-tuned,bf16,2,Intel Developer Cloud,42.15,71.92,33.98,33.15,66.06,Student Ambassador
+Inference Tested,Model,Average,Training Hardware,ARC,HellaSwag,MMLU,TruthfulQA,Winogrande,Model Type,Precision,Size,Infrastructure,Affiliation
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-3,71.574,Gaudi,66.89,85.26,63.07,63.01,79.64,fine-tuned,fp16,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-2,70.858,Gaudi,67.49,83.92,63.55,59.68,79.65,fine-tuned,fp16,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-1,70.002,Gaudi,66.21,83.64,62.37,59.65,78.14,fine-tuned,fp16,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3,69.906,Gaudi,67.15,83.29,62.26,58.77,78.06,fine-tuned,fp16,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-1,69.89,Gaudi,65.7,83.54,62.12,59.48,78.61,fine-tuned,int8,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-1,69.972,Gaudi,66.3,83.6,62.44,59.54,77.98,fine-tuned,bf16,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Intel/neural-chat-7b-v3-1,68.256,Gaudi,64.25,82.49,60.79,56.4,77.35,fine-tuned,int4,7,Intel Developer Cloud,Intel Engineering
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,FunDialogues/dollygem-2b-LoRA,49.368,GPU Max,41.55,71.56,35.39,32.59,65.75,fine-tuned,bf16,2,Intel Developer Cloud,No Affiliation
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,ThejasElandassery/dallema,49.134,GPU Max,41.64,72.01,33.67,33.08,65.27,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,utkarshsingh99/confused-gemma,49.306,GPU Max,42.06,71.86,34.14,33.2,65.27,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,myasaswin/Gemma2B-LORAfied,49.566,GPU Max,42.56,71.56,35.47,32.57,65.67,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,Aprajita0/Gemma-2b-Lora,49.526,GPU Max,42.24,71.88,34.14,33.23,66.14,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,gopalakrishnan-d/gemma-2b-dolly-ds-lora,49.378,GPU Max,42.24,71.88,34.11,33.15,65.51,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador
+<p>&#129000 &#128998 &#128997 &#128992 &#128994</p>,SSK0908/gemma-2b-dolly-qa,49.452,GPU Max,42.15,71.92,33.98,33.15,66.06,fine-tuned,bf16,2,Intel Developer Cloud,Student Ambassador