anakin87 (Stefano Fiorucci)

posted an update 3 days ago

Post

1435

🌌 Creating adventures with local LLMs

What if 🤔... Homer Simpson met Spider-Man and they went on a quest for donuts? 🍩
Or if Fred Astaire and Corporal Hicks teamed up to fight xenomorphs? 👾

In the words of Karpathy, LLMs are dream machines...
they seem specially made to simulate these wild scenarios!

𝐄𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐭𝐡𝐢𝐬 𝐢𝐝𝐞𝐚 👇
Nous Research / @teknium recently released NousResearch/CharacterCodex:
a massive dataset with information on 16k characters, both fictional and real.
I couldn't wait to play it...

After a few attempts, I found that combining the information in this dataset with a good model (like meta-llama/Meta-Llama-3-8B-Instruct) opens the doors to a myriad of chat adventures.

🛠️ Stack:
🔹Haystack for orchestration 🏗️
🔹llamafile 🦙🗂️ to run our model locally.

📓 Check out the notebook: https://t.ly/y6jrZ
(includes a bonus 🕵️ Mystery Character Quiz)

posted an update 12 days ago

Post

873

🧪 RAG Evaluation with 🔥 Prometheus 2 + Haystack

📝 Blog post: https://haystack.deepset.ai/blog/rag-evaluation-with-prometheus-2
📓 Notebook: https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prometheus2_evaluation.ipynb

─── ⋆⋅☆⋅⋆ ───

When evaluating LLMs' responses, 𝐩𝐫𝐨𝐩𝐫𝐢𝐞𝐭𝐚𝐫𝐲 𝐦𝐨𝐝𝐞𝐥𝐬 like GPT-4 are commonly used due to their strong performance.
However, relying on closed models presents challenges related to data privacy 🔒, transparency, controllability, and cost 💸.

On the other hand, 𝐨𝐩𝐞𝐧 𝐦𝐨𝐝𝐞𝐥𝐬 typically do not correlate well with human judgments and lack flexibility.

🔥 Prometheus 2 is a new family of open-source models designed to address these gaps:
🔹 two variants: prometheus-eval/prometheus-7b-v2.0; prometheus-eval/prometheus-8x7b-v2.0
🔹 trained on open-source data
🔹 high correlation with human evaluations and proprietary models
🔹 highly flexible: capable of performing direct assessments and pairwise rankings, and allowing the definition of custom evaluation criteria.

See my experiments with RAG evaluation in the links above.

posted an update 23 days ago

Post

2006

⚙️ Prompt Optimization with Haystack and DSPy

Experimental notebook: 🧪📓 https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prompt_optimization_with_dspy.ipynb

When building applications with LLMs, writing effective prompts is a long process of trial and error. 🔄
Often, if you switch models, you also have to change the prompt. 😩
What if you could automate this process?

💡 That's where DSPy comes in - a framework designed to algorithmically optimize prompts for Language Models.
By applying classical machine learning concepts (training and evaluation data, metrics, optimization), DSPy generates better prompts for a given model and task.

Recently, I explored combining DSPy with the robustness of Haystack Pipelines.

Here's how it works:
▶️ Start from a Haystack RAG pipeline with a basic prompt
🎯 Define a goal (in this case, get correct and concise answers)
📊 Create a DSPy program, define data and metrics
✨ Optimize and evaluate -> improved prompt
🚀 Build a refined Haystack RAG pipeline using the optimized prompt

1 reply

·

posted an update about 1 month ago

Post

1270

Do you want to play a game against Llama 3? 🦙🦙🦙

Meet 🧑‍🏫 𝐀𝐮𝐭𝐨𝐐𝐮𝐢𝐳𝐳𝐞𝐫, a new LLM application that you can use for learning or just for fun.

Try it out on Hugging Face Spaces 🤗 deepset/autoquizzer

𝐇𝐨𝐰 𝐢𝐭 𝐰𝐨𝐫𝐤𝐬
You provide an URL -> A multiple-choice quiz is instantly generated.

🔹 You can play the quiz yourself.

🔹 You can let the LLM play in two different ways
📕 Closed book: the LLM responds only by knowing the general topic and using its parametric knowledge and reasoning abilities.
🔎🌐 Web RAG: for each question, a Google search is done and the top 3 snippets are included in the prompt for the LLM.

𝐒𝐭𝐚𝐜𝐤
🏗️ Haystack LLM framework https://haystack.deepset.ai/
🦙 Llama 3 8B Instruct
⚡ Groq

Original idea: @Tuana

1 reply

·

Stefano Fiorucci

AI & ML interests

Organizations

anakin87's activity