Update README.md

README.md

@@ -57,98 +57,74 @@ User: How could this be used to design a fracture resistant material?<end_of_utt
 Assistant:
 ```
 
-If you need to manually set the chat template:
-
-```
-IDEFICS2_CHAT_TEMPLATE = "{% for message in messages %}{{message['role'].capitalize()}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}"
-```
+ 
 
 ### Sample inference code
 
 This code snippets show how to get quickly started on a GPU:
 
 ```python
-from PIL import Image 
-import requests 
+model_id='lamm-mit/Cephalo-Llama-3.2-11B-Vision-Instruct-128k'
 
-DEVICE='cuda:0'
+model = MllamaForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.bfloat16,
+                                       #_attn_implementation="flash_attention_2",
+                                       trust_remote_code=True,
+                                       ).to (DEVICE )
 
-from transformers import AutoProcessor, Idefics2ForConditionalGeneration 
-from tqdm.notebook import tqdm
- 
-model_id='lamm-mit/Cephalo-Idefics-2-vision-8b-beta'
-
-model = Idefics2ForConditionalGeneration.from_pretrained(  model_id,
-                                                           torch_dtype=torch.bfloat16, #if your GPU allows
-                                                           _attn_implementation="flash_attention_2", #make sure Flash Attention 2 is installed
-                                                           trust_remote_code=True,
-                                                        ).to (DEVICE)
-processor = AutoProcessor.from_pretrained(
-    f"{model_id}",
-    do_image_splitting=True
-)
+processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, )
 ```
-See section towards the end for more comments on model optimization, including quantization.
 
+Simple inference example:
 
-If you need to manually set the chat template:
+We are asking a question about this image, showing a material microstructure and associated stress-strain responses. 
 
-```python
-IDEFICS2_CHAT_TEMPLATE = "{% for message in messages %}{{message['role'].capitalize()}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}"
-tokenizer = AutoTokenizer.from_pretrained(base_model_id, use_fast=True)
-tokenizer.chat_template = IDEFICS2_CHAT_TEMPLATE
-processor.tokenizer = tokenizer
-```
-
-Simple inference example:
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/4JwIGSfl82hMEyHasOSU4.png)
 
 ```
-from transformers.image_utils import load_image
+import requests
+import torch
+from PIL import Image
 
-image = load_image("https://d2r55xnwy6nx47.cloudfront.net/uploads/2018/02/Ants_Lede1300.jpg")
+url = "https://huggingface.co/lamm-mit/Cephalo-Llama-3.2-11B-Vision-Instruct-128k/resolve/main/architected_stress_strain.png"
+
+image = Image.open(requests.get(url, stream=True).raw)
 
-# Create inputs
 messages = [
-    {
-        "role": "user",
-        "content": [
-            {"type": "image"},
-            {"type": "text", "text": "What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI."},
-        ]
-    },
+    {"role": "user", "content": [
+        {"type": "image"},
+        {"type": "text", "text": "Consider the stress-strain response under compression. What are the three curves shown. Based on an inspection of the plot, do they show good agreement or are they very different?"}
+    ]}
 ]
-prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
+input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(image, input_text, return_tensors="pt").to(model.device)
 
-# Get inputs using the processor
-inputs = processor(text=prompt, images=[image], return_tensors="pt")
-inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
+output = model.generate(**inputs, max_new_tokens=512)
+print(processor.decode(output[0]))
+```
 
-# Generate
-generated_ids = model.generate(**inputs, max_new_tokens=500)
-generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
+Raw output:
 
-print(generated_texts)
+```
+<|begin_of_text|><|start_header_id|>user<|end_header_id|>
+
+<|image|>Consider the stress-strain response under compression. What are the three curves shown. Based on an inspection of the plot, do they show good agreement or are they very different?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+The image shows three curves representing the stress-strain response under compression. The x-axis represents strain, which is the deformation experienced by the material relative to its original length, while the y-axis represents stress, which is the force applied per unit area. 
+
+- The blue curve is labeled "Predicted," indicating a predicted model or simulation result.
+- The orange curve is labeled "Ground truth," indicating actual experimental data or true values.
+- The green curve is labeled "Simulation result," likely representing another simulation result for comparison.
+
+The curves show an increasing trend of stress with strain, indicating that the material becomes more stressed as it deforms. The predicted and simulation results (blue and green curves) closely follow the ground truth (orange curve), suggesting good agreement among the predicted and simulated models and the actual experimental data. This implies that the models used are accurate in predicting the material's response under compression. The curves do not show significant deviations, indicating reliable modeling and simulation techniques.<|eot_id|>
 ```
 
 Next we provide a convenience function for inference. This function takes the model, processor, question, and images, along with messages and images objects for repeated chat-like interactions with the model.
 
 ```python
-def ask_about_image (model, processor, question, 
-                     images_input=[], 
-                     verbatim=False,
-                     temperature=0.1,
-                     show_image=False,
-                     system="You are a biomaterials scientist who responds accurately. ", 
-                     init_instr = "",
-                     show_conversation=True,
-                     max_new_tokens=256, 
-                     messages=[], 
-                     images=[], 
-                     use_Markdown=False,
-                    ):
+def ask_about_image (model, processor, question, images_input=[],  verbatim=False,temperature=0.1,show_image=False,
+                     system="You are a materials scientist. ", init_instr = "", show_conversation=True,
+                     max_new_tokens=256, messages=[], images=[], use_Markdown=False):
     
-   
-    query = question
     images_input=ensure_list(images_input)
     if len (images)==0:
         if len (images_input)>0:
@@ -156,75 +132,68 @@ def ask_about_image (model, processor, question,
                 if is_url(image):
                     image= load_image(image)
                 images.append (image)
-                
+                 
                 if show_image:
                     display ( image )
     if len (messages)==0:
-       
-        base_message = {
-            "role": "user",
-            "content": [
-                {"type": "text", "text": system + init_instr},
-                # Image messages will be added dynamically here
-                {"type": "text", "text": query}
-            ]
-        }
-        
-        # Ensure the images_input is a list
-        images_input = ensure_list(images_input)
-        
-        # Add image messages dynamically
-        image_messages = [{"type": "image"} for _ in images_input]
-        base_message["content"][1:1] = image_messages  # Insert image messages before the last text message
+        messages = [
+            {"role": "user", "content": [
+                {"type": "image"},
+                {"type": "text", "text": question}
+            ]}
+        ]
+        input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
+        inputs = processor(image, input_text, return_tensors="pt").to(model.device)
         
-        # Append the constructed message to messages list
-        messages.append(base_message)
-
     else:
         messages.append (
-            {
-                "role": "user",
-                "content": [
-                    {"type": "text", "text": query
-                    }
-                ]
-            }
-        )
+           {"role": "user", "content": [
+                    
+                    {"type": "text", "text": question}
+                ]}  )
+
     if verbatim:
         print (messages)
         
-    text = processor.apply_chat_template(messages, add_generation_prompt=True)
-    inputs = processor(text=[text.strip()], images=images, return_tensors="pt", padding=True).to(DEVICE)
-     
-    generated_ids = model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=temperature, do_sample=True)
-    generated_texts = processor.batch_decode(generated_ids[:, inputs["input_ids"].size(1):], skip_special_tokens=True)
-
-    messages.append (
-        {
-            "role": "assistant",
-            "content": [ {"type": "text", "text": generated_texts[0]}, ]
-        }
-    )
+    text = processor.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = processor(text=text, images=images, return_tensors="pt", ).to(DEVICE)
+
+    generation_args = { 
+                "max_new_tokens": max_new_tokens, 
+                "temperature": temperature, 
+                "do_sample": True, 
+                } 
+
+    generate_ids = model.generate(**inputs,# eos_token_id=processor.tokenizer.eos_token_id, 
+                                  **generation_args)
+
+    generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:-1]
+    generated_texts = processor.decode(generate_ids[0], clean_up_tokenization_spaces=False) 
+
+    messages.append (  {"role": "assistant", "content": [  {"type": "text", "text": generated_texts}]}  )
+
     formatted_conversation = format_conversation(messages, images)
     
-    # Display the formatted conversation, e.g. in Jupyter Notebook
+    # Display the formatted conversation in Jupyter Notebook
     if show_conversation:
-     
         if use_Markdown:
             display(Markdown(formatted_conversation))
         else:
             display(HTML(formatted_conversation))
-
+    
     return generated_texts, messages, images
 
-question = "What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI."
+question = """What is shown in this image, and what is the relevance for materials design? Include a discussion of multi-agent AI. 
+
+First brainstorm, then organize your thoughts, then respond."""
 
 url1 = "https://d2r55xnwy6nx47.cloudfront.net/uploads/2018/02/Ants_Lede1300.jpg" 
 
 response, messages,images= ask_about_image ( model, processor, question, 
                                              images_input=[url1,],
                                              temperature=0.1,
-                                             system= '', init_instr='You carefully study the image, and respond accurately, but succinctly. Think step-by-step.\n\n', 
+                                             system= '',
+                                             init_instr='You carefully study the image, and respond accurately, but succinctly. Think step-by-step.\n\n', 
                                              show_conversation=True,
                                              max_new_tokens=512, messages=[], images=[])
 ```
@@ -235,11 +204,13 @@ Sample output:
 <small>Image by [Vaishakh Manohar](https://www.quantamagazine.org/the-simple-algorithm-that-ants-use-to-build-bridges-20180226/)</small>
 
 <pre style="white-space: pre-wrap;">
-The image depicts a group of ants moving in a coordinated manner, demonstrating their ability to navigate complex environments and adapt to changing conditions. This behavior is relevant for materials design because it highlights the potential of multi-agent AI systems to mimic natural systems and develop new materials with enhanced properties.
+The image shows a group of ants working together to move a large object. This scene illustrates the concept of swarm intelligence, where individual agents (ants) collectively achieve a complex task through decentralized, self-organized behavior. 
+
+In materials design, this concept can be applied to develop new materials and structures by mimicking the behavior of swarms. For instance, researchers have used swarm intelligence algorithms to optimize the design of composite materials, such as fiber-reinforced polymers, by simulating the behavior of ants or other swarming organisms. These algorithms can help identify the optimal arrangement of fibers to maximize strength and minimize weight.
 
-Multi-agent AI refers to the use of multiple autonomous agents working together to solve complex problems. These agents can learn from each other and adapt to new situations, similar to how ants can navigate their environment and communicate with one another. By applying these principles to materials design, researchers can develop new materials that exhibit improved performance, such as enhanced strength, flexibility, and adaptability.
+Multi-agent AI, which involves the coordination of multiple autonomous agents to achieve a common goal, can also be used in materials design. This approach can be applied to simulate the behavior of complex systems, such as biological tissues or nanomaterials, and optimize their properties through machine learning algorithms. By analyzing the behavior of individual agents and their interactions, researchers can develop new materials with improved performance and functionality.
 
-The relevance of this image for materials design lies in the inspiration it provides for developing new materials that can mimic the natural efficiency and adaptability of ants. By studying the behavior of ants, researchers can gain insights into how to design materials that can respond dynamically to changes in their environment, leading to improved performance and functionality.
+In summary, the image of ants working together to move a large object serves as a metaphor for the potential of swarm intelligence and multi-agent AI in materials design. By mimicking the behavior of swarms, researchers can develop new materials and structures with improved properties and functionality.
 </pre>
 
 ## Dataset generation