Spaces:
Runtime error
Runtime error
Initial commit
Browse files- .gitattributes +1 -0
- JB_test_research_tools.py +45 -0
- README.md +57 -13
- agents.yaml +35 -0
- crew.py +183 -0
- crewai-newsletter.png +3 -0
- main.py +18 -0
- newsletter_template.html +82 -0
- poetry.lock +0 -0
- pyproject.toml +24 -0
- research.py +60 -0
- tasks.yaml +81 -0
.gitattributes
CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
+
crewai-newsletter.png filter=lfs diff=lfs merge=lfs -text
|
JB_test_research_tools.py
ADDED
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# JB 13-06-2024
|
2 |
+
# C:\Users\jfhmb\EXA_CREWAI\exa-crewai-master\exa-crewai-master\src\newsletter_gen\tools\JB_test_research_tools.py
|
3 |
+
from research import SearchAndContents
|
4 |
+
from research import FindSimilar
|
5 |
+
from research import GetContents
|
6 |
+
|
7 |
+
search_and_contents = SearchAndContents()
|
8 |
+
find_similar = FindSimilar()
|
9 |
+
get_contents = GetContents()
|
10 |
+
|
11 |
+
# dir(search_and_contents)
|
12 |
+
|
13 |
+
# Search and Contents Tool
|
14 |
+
# Searches the web based on a search query for the latest results.
|
15 |
+
# Results are only from the last week.
|
16 |
+
# Uses the Exa API.
|
17 |
+
# This also returns the contents of the search results.
|
18 |
+
print(search_and_contents.run(search_query="Grokking"))
|
19 |
+
print("==============================================================")
|
20 |
+
# EXAMPLE RESULT (ONLY 1 SHOW HERE):
|
21 |
+
# URL: https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/
|
22 |
+
# ID: https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/
|
23 |
+
# Score: 0.11678461730480194
|
24 |
+
# Published Date: 2024-06-06
|
25 |
+
# Etc.
|
26 |
+
|
27 |
+
|
28 |
+
# Find Similar Tool
|
29 |
+
# Searches for similar articles to a given article using the Exa API. Takes in a URL of the article
|
30 |
+
url = "https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/"
|
31 |
+
print(find_similar.run(article_url=url))
|
32 |
+
print("==============================================================")
|
33 |
+
|
34 |
+
|
35 |
+
|
36 |
+
# Get Contents Tool
|
37 |
+
# Gets the contents of a specific article using the Exa API.
|
38 |
+
# Takes in the ID of the article in a list,
|
39 |
+
# like this: ['https://www.cnbc.com/2024/04/18/my-news-story'].
|
40 |
+
# article_ids
|
41 |
+
article_ids = ["https://www.iotworldtoday.com/industry/openai-chief-architect-predicts-huge-large-language-model-leaps",
|
42 |
+
"https://ieeexplore.ieee.org/document/6547645/"]
|
43 |
+
|
44 |
+
print(get_contents.run(article_ids=article_ids))
|
45 |
+
print("==============================================================")
|
README.md
CHANGED
@@ -1,13 +1,57 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# NewsletterGen Crew with GUI
|
2 |
+
|
3 |
+
Welcome to the NewsletterGen Crew project, powered by [crewAI](https://crewai.com). This template is designed to help you set up a multi-agent AI system with ease, leveraging the powerful and flexible framework provided by crewAI. Our goal is to enable your agents to collaborate effectively on complex tasks, maximizing their collective intelligence and capabilities.
|
4 |
+
|
5 |
+
## Installation
|
6 |
+
|
7 |
+
Ensure you have Python >=3.10 <=3.13 installed on your system. This project uses [Poetry](https://python-poetry.org/) for dependency management and package handling, offering a seamless setup and execution experience.
|
8 |
+
|
9 |
+
First, if you haven't already, install Poetry:
|
10 |
+
|
11 |
+
```bash
|
12 |
+
pip install poetry
|
13 |
+
```
|
14 |
+
|
15 |
+
Next, navigate to your project directory and install the dependencies:
|
16 |
+
|
17 |
+
1. First lock the dependencies and then install them:
|
18 |
+
```bash
|
19 |
+
poetry lock
|
20 |
+
```
|
21 |
+
```bash
|
22 |
+
poetry install
|
23 |
+
```
|
24 |
+
### Customizing
|
25 |
+
|
26 |
+
**Add your `OPENAI_API_KEY` into the `.env` file**
|
27 |
+
|
28 |
+
- Modify `src/newsletter_gen/config/agents.yaml` to define your agents
|
29 |
+
- Modify `src/newsletter_gen/config/tasks.yaml` to define your tasks
|
30 |
+
- Modify `src/newsletter_gen/crew.py` to add your own logic, tools and specific args
|
31 |
+
- Modify `src/newsletter_gen/main.py` to add custom inputs for your agents and tasks
|
32 |
+
|
33 |
+
## Running the Project
|
34 |
+
|
35 |
+
To kickstart your crew of AI agents and begin task execution, run this from the root folder of your project:
|
36 |
+
|
37 |
+
```bash
|
38 |
+
poetry run newsletter_gen
|
39 |
+
```
|
40 |
+
|
41 |
+
This command initializes the newsletter-gen Crew, assembling the agents and assigning them tasks as defined in your configuration.
|
42 |
+
|
43 |
+
This example, unmodified, will run the create a `report.md` file with the output of a research on LLMs in the root folser
|
44 |
+
|
45 |
+
## Understanding Your Crew
|
46 |
+
|
47 |
+
The newsletter-gen Crew is composed of multiple AI agents, each with unique roles, goals, and tools. These agents collaborate on a series of tasks, defined in `config/tasks.yaml`, leveraging their collective skills to achieve complex objectives. The `config/agents.yaml` file outlines the capabilities and configurations of each agent in your crew.
|
48 |
+
|
49 |
+
## Support
|
50 |
+
|
51 |
+
For support, questions, or feedback regarding the NewsletterGen Crew or crewAI.
|
52 |
+
- Visit our [documentation](https://docs.crewai.com)
|
53 |
+
- Reach out to us through our [GitHub repository](https://github.com/joaomdmoura/crewai)
|
54 |
+
- [Joing our Discord](https://discord.com/invite/X4JWnZnxPb)
|
55 |
+
- [Chat wtih our docs](https://chatg.pt/DWjSBZn)
|
56 |
+
|
57 |
+
Let's create wonders together with the power and simplicity of crewAI.
|
agents.yaml
ADDED
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
researcher:
|
2 |
+
role: >
|
3 |
+
Senior Researcher
|
4 |
+
goal: >
|
5 |
+
Uncover cutting-edge developments in {topic}.
|
6 |
+
backstory: >
|
7 |
+
You're a seasoned journalist with a nose for news. You're known for your great research skills and ability to dig up the most interesting stories. Your reports are always thorough and well-researched, making you a trusted source of information.
|
8 |
+
You always follow the rules and guidelines provided to you and you never forget to include the complete URL of the article where you found the news.
|
9 |
+
|
10 |
+
editor:
|
11 |
+
role: >
|
12 |
+
Editor-in-Chief
|
13 |
+
goal: >
|
14 |
+
Ensure the quality and accuracy of the final newsletter.
|
15 |
+
backstory: >
|
16 |
+
You are the Editor-in-Chief of a prestigious news organization. You are responsible for overseeing the production of the newsletter and ensuring that it meets the highest standards of quality, that it is accurate, well-written, and engaging.
|
17 |
+
|
18 |
+
You review the news articles provided by the researcher, add context to each article (like why the news story is relevant), and have a great sense of what will resonate with the readers. You use this sense of judgment to reorder the news articles in a way that the most important news is at the top of the list.
|
19 |
+
|
20 |
+
designer:
|
21 |
+
role: >
|
22 |
+
Newsletter Compiler
|
23 |
+
goal: >
|
24 |
+
Fill the HTML template given to you with the news articles provided.
|
25 |
+
backstory: >
|
26 |
+
You are responsible for compiling the HTML code of the newsletter, making sure that every news article is included in the final document.
|
27 |
+
You do NOT modify the content and only update the design when necessary. You use the HTML template provided to you to create the newsletter.
|
28 |
+
role: >
|
29 |
+
{topic} Reporting Analyst
|
30 |
+
goal: >
|
31 |
+
Create detailed reports based on {topic} data analysis and research findings
|
32 |
+
backstory: >
|
33 |
+
You're a meticulous analyst with a keen eye for detail. You're known for
|
34 |
+
your ability to turn complex data into clear and concise reports, making
|
35 |
+
it easy for others to understand and act on the information you provide.
|
crew.py
ADDED
@@ -0,0 +1,183 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
from crewai import Agent, Crew, Process, Task
|
2 |
+
from crewai.project import CrewBase, agent, crew, task
|
3 |
+
from newsletter_gen.tools.research import SearchAndContents, FindSimilar, GetContents
|
4 |
+
from langchain_anthropic import ChatAnthropic
|
5 |
+
from langchain_groq import ChatGroq
|
6 |
+
from datetime import datetime
|
7 |
+
import streamlit as st
|
8 |
+
from typing import Union, List, Tuple, Dict
|
9 |
+
from langchain_core.agents import AgentFinish
|
10 |
+
import json
|
11 |
+
from langchain_google_genai import ChatGoogleGenerativeAI
|
12 |
+
import os
|
13 |
+
|
14 |
+
# JB:
|
15 |
+
# https://python.langchain.com/v0.2/docs/integrations/chat/ollama/
|
16 |
+
# LangChain supports many other chat models. Here, we're using Ollama
|
17 |
+
from langchain_community.chat_models import ChatOllama
|
18 |
+
|
19 |
+
# To get rid of the telemetry error messages, try:
|
20 |
+
# Connection Timeout Error with telemetry.crewai.com #254
|
21 |
+
# https://github.com/joaomdmoura/crewAI/issues/254
|
22 |
+
# os.environ["OTEL_SDK_DISABLED"] = "true"
|
23 |
+
os.environ["OTEL_SDK_DISABLED"] = "true"
|
24 |
+
# SUCCES:
|
25 |
+
# DIT LIJKT INDERDAAD DE TELEMETRY ERROR MESSAGES IN DE VS CODE TERMINAL TE VOORKOMEN !!!!!!!!!!
|
26 |
+
# Wel in die terminal nog deze korte messages:
|
27 |
+
# 2024-06-14 02:20:17,425 - 25632 - __init__.py-__init__:1218 - WARNING: SDK is disabled.
|
28 |
+
|
29 |
+
|
30 |
+
|
31 |
+
@CrewBase
|
32 |
+
class NewsletterGenCrew:
|
33 |
+
"""NewsletterGen crew"""
|
34 |
+
|
35 |
+
agents_config = "config/agents.yaml"
|
36 |
+
tasks_config = "config/tasks.yaml"
|
37 |
+
|
38 |
+
def llm(self):
|
39 |
+
# llm = ChatAnthropic(model_name="claude-3-sonnet-20240229", max_tokens=4096) # ORIGINAL
|
40 |
+
#llm = ChatAnthropic(model_name="claude-3-sonnet-20240229",
|
41 |
+
# # max_tokens=4096,
|
42 |
+
# cache=True,
|
43 |
+
# api_key="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"
|
44 |
+
# ) # JB
|
45 |
+
# https://console.anthropic.com/dashboard
|
46 |
+
# https://console.anthropic.com/settings/keys
|
47 |
+
# jb_anthropic_key_2_13-06-2024:
|
48 |
+
# ANTHROPIC_API_KEY=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
|
49 |
+
# https://console.anthropic.com/settings/usage
|
50 |
+
|
51 |
+
# llm = ChatGroq(model="llama3-70b-8192")
|
52 |
+
# https://console.groq.com/docs/rate-limits
|
53 |
+
# llm = ChatGroq(model="mixtral-8x7b-32768") # JB 13-06-2024 - geeft af en toe rate limit errors
|
54 |
+
|
55 |
+
# llm = ChatGoogleGenerativeAI(google_api_key=os.getenv("GOOGLE_API_KEY"))
|
56 |
+
|
57 |
+
# https://python.langchain.com/v0.2/docs/integrations/chat/ollama/
|
58 |
+
# supports many more optional parameters. Hover on your `ChatOllama(...)`
|
59 |
+
# class to view the latest available supported parameters
|
60 |
+
# llm = ChatOllama(model="llama3")
|
61 |
+
llm = ChatOllama(model="mistral:latest")
|
62 |
+
# check if ollama is running and which LLMs can then be used, run this in Anaconda cmd admin window:
|
63 |
+
# ollama list
|
64 |
+
# OUTPUT EXAMPLE:
|
65 |
+
# (newsletter-gen-py3.11) (base) C:\Users\jfhmb\EXA_CREWAI\exa-crewai-master\exa-crewai-master>ollama list
|
66 |
+
# NAME ID SIZE MODIFIED
|
67 |
+
# llama3:latest 365c0bd3c000 4.7 GB 3 days ago
|
68 |
+
# nomic-embed-text:latest 0a109f422b47 274 MB 3 days ago
|
69 |
+
# crewai-llama3:latest d952d07761cd 4.7 GB 10 days ago
|
70 |
+
# llama3:8b 365c0bd3c000 4.7 GB 10 days ago
|
71 |
+
# mistral:latest 61e88e884507 4.1 GB 6 weeks ago
|
72 |
+
# mxbai-embed-large:latest 468836162de7 669 MB 6 weeks ago
|
73 |
+
#
|
74 |
+
# OLLAMA LOGS:
|
75 |
+
# C:\Users\jfhmb\AppData\Local\Ollama
|
76 |
+
|
77 |
+
|
78 |
+
print("JB: in class NewsletterGenCrew - using llm: ", llm)
|
79 |
+
|
80 |
+
return llm
|
81 |
+
|
82 |
+
def step_callback(
|
83 |
+
self,
|
84 |
+
agent_output: Union[str, List[Tuple[Dict, str]], AgentFinish],
|
85 |
+
agent_name,
|
86 |
+
*args,
|
87 |
+
):
|
88 |
+
with st.chat_message("AI"):
|
89 |
+
# Try to parse the output if it is a JSON string
|
90 |
+
if isinstance(agent_output, str):
|
91 |
+
try:
|
92 |
+
agent_output = json.loads(agent_output)
|
93 |
+
except json.JSONDecodeError:
|
94 |
+
pass
|
95 |
+
|
96 |
+
if isinstance(agent_output, list) and all(
|
97 |
+
isinstance(item, tuple) for item in agent_output
|
98 |
+
):
|
99 |
+
|
100 |
+
for action, description in agent_output:
|
101 |
+
# Print attributes based on assumed structure
|
102 |
+
st.write(f"Agent Name: {agent_name}")
|
103 |
+
st.write(f"Tool used: {getattr(action, 'tool', 'Unknown')}")
|
104 |
+
st.write(f"Tool input: {getattr(action, 'tool_input', 'Unknown')}")
|
105 |
+
st.write(f"{getattr(action, 'log', 'Unknown')}")
|
106 |
+
with st.expander("Show observation"):
|
107 |
+
st.markdown(f"Observation\n\n{description}")
|
108 |
+
|
109 |
+
# Check if the output is a dictionary as in the second case
|
110 |
+
elif isinstance(agent_output, AgentFinish):
|
111 |
+
st.write(f"Agent Name: {agent_name}")
|
112 |
+
output = agent_output.return_values
|
113 |
+
st.write(f"I finished my task:\n{output['output']}")
|
114 |
+
|
115 |
+
# Handle unexpected formats
|
116 |
+
else:
|
117 |
+
st.write(type(agent_output))
|
118 |
+
st.write(agent_output)
|
119 |
+
|
120 |
+
@agent
|
121 |
+
def researcher(self) -> Agent:
|
122 |
+
return Agent(
|
123 |
+
config=self.agents_config["researcher"],
|
124 |
+
tools=[SearchAndContents(), FindSimilar(), GetContents()],
|
125 |
+
verbose=True,
|
126 |
+
llm=self.llm(),
|
127 |
+
step_callback=lambda step: self.step_callback(step, "Research Agent"),
|
128 |
+
)
|
129 |
+
|
130 |
+
@agent
|
131 |
+
def editor(self) -> Agent:
|
132 |
+
return Agent(
|
133 |
+
config=self.agents_config["editor"],
|
134 |
+
verbose=True,
|
135 |
+
tools=[SearchAndContents(), FindSimilar(), GetContents()],
|
136 |
+
llm=self.llm(),
|
137 |
+
step_callback=lambda step: self.step_callback(step, "Chief Editor"),
|
138 |
+
)
|
139 |
+
|
140 |
+
@agent
|
141 |
+
def designer(self) -> Agent:
|
142 |
+
return Agent(
|
143 |
+
config=self.agents_config["designer"],
|
144 |
+
verbose=True,
|
145 |
+
allow_delegation=False,
|
146 |
+
llm=self.llm(),
|
147 |
+
step_callback=lambda step: self.step_callback(step, "HTML Writer"),
|
148 |
+
)
|
149 |
+
|
150 |
+
@task
|
151 |
+
def research_task(self) -> Task:
|
152 |
+
return Task(
|
153 |
+
config=self.tasks_config["research_task"],
|
154 |
+
agent=self.researcher(),
|
155 |
+
output_file=f"logs/{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}_research_task.md",
|
156 |
+
)
|
157 |
+
|
158 |
+
@task
|
159 |
+
def edit_task(self) -> Task:
|
160 |
+
return Task(
|
161 |
+
config=self.tasks_config["edit_task"],
|
162 |
+
agent=self.editor(),
|
163 |
+
output_file=f"logs/{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}_edit_task.md",
|
164 |
+
)
|
165 |
+
|
166 |
+
@task
|
167 |
+
def newsletter_task(self) -> Task:
|
168 |
+
return Task(
|
169 |
+
config=self.tasks_config["newsletter_task"],
|
170 |
+
agent=self.designer(),
|
171 |
+
output_file=f"logs/{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}_newsletter_task.html",
|
172 |
+
)
|
173 |
+
|
174 |
+
@crew
|
175 |
+
def crew(self) -> Crew:
|
176 |
+
"""Creates the NewsletterGen crew"""
|
177 |
+
return Crew(
|
178 |
+
agents=self.agents, # Automatically created by the @agent decorator
|
179 |
+
tasks=self.tasks, # Automatically created by the @task decorator
|
180 |
+
process=Process.sequential,
|
181 |
+
verbose=2,
|
182 |
+
# process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
|
183 |
+
)
|
crewai-newsletter.png
ADDED
Git LFS Details
|
main.py
ADDED
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
#!/usr/bin/env python
|
2 |
+
from newsletter_gen.crew import NewsletterGenCrew
|
3 |
+
|
4 |
+
def load_html_template():
|
5 |
+
with open('src/newsletter_gen/config/newsletter_template.html', 'r') as file:
|
6 |
+
html_template = file.read()
|
7 |
+
|
8 |
+
return html_template
|
9 |
+
|
10 |
+
|
11 |
+
def run():
|
12 |
+
# Replace with your inputs, it will automatically interpolate any tasks and agents information
|
13 |
+
inputs = {
|
14 |
+
'topic': input('Enter the topic for yout newsletter: '),
|
15 |
+
'personal_message': input('Enter a personal message for your newsletter: '),
|
16 |
+
'html_template': load_html_template()
|
17 |
+
}
|
18 |
+
NewsletterGenCrew().crew().kickoff(inputs=inputs)
|
newsletter_template.html
ADDED
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
<!DOCTYPE html>
|
2 |
+
<html>
|
3 |
+
<head>
|
4 |
+
<title>Weekly Newsletter</title>
|
5 |
+
<style>
|
6 |
+
body {
|
7 |
+
font-family: Arial, sans-serif;
|
8 |
+
margin: 0;
|
9 |
+
padding: 0;
|
10 |
+
background-color: #f0f0f0;
|
11 |
+
}
|
12 |
+
.container {
|
13 |
+
width: 80%;
|
14 |
+
margin: auto;
|
15 |
+
padding: 20px;
|
16 |
+
background-color: #fff;
|
17 |
+
border-radius: 5px;
|
18 |
+
box-shadow: 0px 0px 10px rgba(0,0,0,0.1);
|
19 |
+
}
|
20 |
+
h1, h2 {
|
21 |
+
color: #333;
|
22 |
+
}
|
23 |
+
h1 {
|
24 |
+
text-align: center;
|
25 |
+
margin-bottom: 20px;
|
26 |
+
}
|
27 |
+
h2 {
|
28 |
+
margin: 0;
|
29 |
+
}
|
30 |
+
p {
|
31 |
+
color: #666;
|
32 |
+
line-height: 1.5;
|
33 |
+
}
|
34 |
+
ul {
|
35 |
+
list-style-type: none;
|
36 |
+
padding: 0;
|
37 |
+
}
|
38 |
+
li {
|
39 |
+
margin-bottom: 20px;
|
40 |
+
}
|
41 |
+
.read-more {
|
42 |
+
margin-top: 10px;
|
43 |
+
font-size: 14px;
|
44 |
+
}
|
45 |
+
.read-more a {
|
46 |
+
color: #0066cc;
|
47 |
+
text-decoration: none;
|
48 |
+
}
|
49 |
+
.read-more a:hover {
|
50 |
+
text-decoration: underline;
|
51 |
+
}
|
52 |
+
.goodbye {
|
53 |
+
text-align: center;
|
54 |
+
margin-top: 30px;
|
55 |
+
color: #666;
|
56 |
+
}
|
57 |
+
</style>
|
58 |
+
</head>
|
59 |
+
<body>
|
60 |
+
<div class="container">
|
61 |
+
<h1><!-- TITLE OF THE DAY --></h1>
|
62 |
+
<p><!-- PERSONAL MESSAGE --></p>
|
63 |
+
<ul>
|
64 |
+
<li>
|
65 |
+
<h2><!-- TITLE OF THE STORY --></h2>
|
66 |
+
<p><!-- SUMMARY OF THE STORY --></p>
|
67 |
+
<p><!-- WHY THIS IS IMPORTANT --></p>
|
68 |
+
<div class="read-more">
|
69 |
+
<p>Read more:</p>
|
70 |
+
<ul>
|
71 |
+
<li><a href="https://www.example.com"><!-- LINK 1 --></a>
|
72 |
+
<li><a href="https://www.example2.com"><!-- LINK 2 --></a>
|
73 |
+
</ul>
|
74 |
+
</div>
|
75 |
+
</li>
|
76 |
+
|
77 |
+
<!-- ADD ALL THE OTHER NEWS ITEMS HERE USING THE SAME FORMAT AS THE EXAPLE ABOVE -->
|
78 |
+
</ul>
|
79 |
+
<p class="goodbye">Goodbye and see you tomorrow!</p>
|
80 |
+
</div>
|
81 |
+
</body>
|
82 |
+
</html>
|
poetry.lock
ADDED
The diff for this file is too large to render.
See raw diff
|
|
pyproject.toml
ADDED
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
[tool.poetry]
|
2 |
+
name = "newsletter_gen"
|
3 |
+
version = "0.1.0"
|
4 |
+
description = "newsletter-gen using crewAI"
|
5 |
+
authors = ["Your Name <you@example.com>"]
|
6 |
+
|
7 |
+
[tool.poetry.dependencies]
|
8 |
+
python = ">=3.10,<=3.13"
|
9 |
+
crewai = {extras = ["tools"], version = "^0.28.7"}
|
10 |
+
exa-py = "^1.0.9"
|
11 |
+
langchain = "0.1.17"
|
12 |
+
langchain-core = "0.1.52"
|
13 |
+
langchain-anthropic = "^0.1.11"
|
14 |
+
langchain-groq = "^0.1.3"
|
15 |
+
streamlit = "^1.34.0"
|
16 |
+
langchain-google-genai = "^1.0.3"
|
17 |
+
pillow = "^10.3.0"
|
18 |
+
|
19 |
+
[tool.poetry.scripts]
|
20 |
+
newsletter_gen = "newsletter_gen.main:run"
|
21 |
+
|
22 |
+
[build-system]
|
23 |
+
requires = ["poetry-core"]
|
24 |
+
build-backend = "poetry.core.masonry.api"
|
research.py
ADDED
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
from crewai_tools import BaseTool
|
2 |
+
from exa_py import Exa
|
3 |
+
import os
|
4 |
+
from datetime import datetime, timedelta
|
5 |
+
|
6 |
+
|
7 |
+
class SearchAndContents(BaseTool):
|
8 |
+
name: str = "Search and Contents Tool"
|
9 |
+
description: str = (
|
10 |
+
"Searches the web based on a search query for the latest results. Results are only from the last week. Uses the Exa API. This also returns the contents of the search results."
|
11 |
+
)
|
12 |
+
|
13 |
+
def _run(self, search_query: str) -> str:
|
14 |
+
|
15 |
+
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
|
16 |
+
|
17 |
+
one_week_ago = datetime.now() - timedelta(days=7)
|
18 |
+
date_cutoff = one_week_ago.strftime("%Y-%m-%d")
|
19 |
+
|
20 |
+
search_results = exa.search_and_contents(
|
21 |
+
query=search_query,
|
22 |
+
use_autoprompt=True,
|
23 |
+
start_published_date=date_cutoff,
|
24 |
+
text={"include_html_tags": False, "max_characters": 8000},
|
25 |
+
)
|
26 |
+
|
27 |
+
return search_results
|
28 |
+
|
29 |
+
|
30 |
+
class FindSimilar(BaseTool):
|
31 |
+
name: str = "Find Similar Tool"
|
32 |
+
description: str = (
|
33 |
+
"Searches for similar articles to a given article using the Exa API. Takes in a URL of the article"
|
34 |
+
)
|
35 |
+
|
36 |
+
def _run(self, article_url: str) -> str:
|
37 |
+
|
38 |
+
one_week_ago = datetime.now() - timedelta(days=7)
|
39 |
+
date_cutoff = one_week_ago.strftime("%Y-%m-%d")
|
40 |
+
|
41 |
+
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
|
42 |
+
|
43 |
+
search_results = exa.find_similar(
|
44 |
+
url=article_url, start_published_date=date_cutoff
|
45 |
+
)
|
46 |
+
|
47 |
+
return search_results
|
48 |
+
|
49 |
+
|
50 |
+
class GetContents(BaseTool):
|
51 |
+
name: str = "Get Contents Tool"
|
52 |
+
description: str = "Gets the contents of a specific article using the Exa API. Takes in the ID of the article in a list, like this: ['https://www.cnbc.com/2024/04/18/my-news-story']."
|
53 |
+
|
54 |
+
def _run(self, article_ids: str) -> str:
|
55 |
+
|
56 |
+
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
|
57 |
+
|
58 |
+
contents = exa.get_contents(article_ids)
|
59 |
+
return contents
|
60 |
+
|
tasks.yaml
ADDED
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
research_task:
|
2 |
+
description: >
|
3 |
+
Conduct a thorough research about the latest news on {topic}. Be sure to look for sources that are reliable and publish recent news. Do not include articles that are not news material or that are not directly related to {topic}.
|
4 |
+
With this research, compile a list of the most relevant news stories that you found.
|
5 |
+
|
6 |
+
Follow these rules:
|
7 |
+
- Only include articles that are especially relevant to {topic}. Do not include any news that are not directly related to {topic}.
|
8 |
+
- Do not include sources that are not a news article. If the content of the page includes a list of articles or looks like the front page of a website, do not include it in the list!
|
9 |
+
- Summarize the news in a few sentences. Make the summary as long as necessary to include all the relevant information, but not too long for a newsletter.
|
10 |
+
- Include the URL of the article where you found the news.
|
11 |
+
- Include a minimum of 7 news articles and a maximum of 10 news articles in the list.
|
12 |
+
- When using the Search Tool, your search query should be concise (for example, "latest news on {topic}").
|
13 |
+
|
14 |
+
IMPORTANT INSTRUCTIONS ABOUT USING TOOLS: When using tools, DO NOT ESCAPE the underscore character "_", EVER. If you need to use a tool and pass in a parameter called 'search_query', you should write 'search_query', not 'search\_query'. THIS IS VERY IMPORTANT, else the tool will not work.
|
15 |
+
|
16 |
+
expected_output: >
|
17 |
+
A markdown document with the most relevant news stories. Each news story should contain the following:
|
18 |
+
- Title of the news
|
19 |
+
- Summary of the news
|
20 |
+
- URL of the article where the news was found
|
21 |
+
|
22 |
+
Here is an example of the format of a news article that you could include in the document:
|
23 |
+
|
24 |
+
<EXAMPLE>
|
25 |
+
Story 1:
|
26 |
+
- Title: **Daily briefing: AI now beats humans at basic reading and maths**
|
27 |
+
- **Summary:** AI systems can now nearly match and sometimes exceed human performance in basic tasks. The report discusses the need for new benchmarks to assess AI capabilities and highlights the ethical considerations for AI models.
|
28 |
+
- **URL:** [Nature Article](https://www.nature.com/articles/d41586-024-01125-1)
|
29 |
+
</EXAMPLE>
|
30 |
+
|
31 |
+
edit_task:
|
32 |
+
description: >
|
33 |
+
Given the list of news articles that will be used in the newsletter, do the following things:
|
34 |
+
|
35 |
+
- Rewrite the title of each news article to make it more engaging and interesting for the readers of the newsletter.
|
36 |
+
- Add a paragraph to each news article that explains why this news is important and how it can impact the readers of the newsletter.
|
37 |
+
- Reorder the bullet points in a way that the most relevant news and topics are at the top of the list based on the importance of the news and topics.
|
38 |
+
- Verify that the news articles are directly related to {topic} and that they are not off-topic. If they are off-topic, remove them from the list.
|
39 |
+
- Verify that the URLs are correct and that they lead to the correct news article. They should lead to a news article and not to a list of articles or the front page of a website. If the URL is incorrect, ask the researcher to provide the correct URL.
|
40 |
+
- Do not search for additional news articles or change the content of the news articles. Only edit the existing news articles.
|
41 |
+
|
42 |
+
IMPORTANT INSTRUCTIONS ABOUT USING TOOLS: When using tools, DO NOT ESCAPE the underscore character "_", EVER. If you need to use a tool and pass in a parameter called 'search_query', you should write 'search_query', not 'search\_query'. THIS IS VERY IMPORTANT, else the tool will not work.
|
43 |
+
|
44 |
+
expected_output: >
|
45 |
+
A markdown document with all the news to be included in the newsletter of the week. The document should have a title related to the curated stories of the week and a list of news articles.
|
46 |
+
|
47 |
+
Each one should contain:
|
48 |
+
- Title:
|
49 |
+
- Summary:
|
50 |
+
- Why this is important:
|
51 |
+
- Source: [URL of the article]
|
52 |
+
|
53 |
+
Here is an example of a document that you are expected to produce:
|
54 |
+
<EXAMPLE>
|
55 |
+
Title of the day: AI is taking over the world
|
56 |
+
|
57 |
+
- **Title:** AI Surpasses Human Capabilities in Basic Reading and Maths
|
58 |
+
**Summary:** Recent advancements in AI technology have enabled systems to match and sometimes exceed human performance in fundamental tasks such as reading and arithmetic. This shift necessitates the creation of new benchmarks to appropriately evaluate AI capabilities.
|
59 |
+
**Why this is important:** This development is crucial as it not only showcases the rapid progress in AI but also prompts a reevaluation of how AI is integrated into educational and professional settings. It raises significant ethical questions about the future role of AI in society.
|
60 |
+
**Source:** "Title of the article" by ["Source"](https://www.nature.com/articles/d41586-024-01125-1)
|
61 |
+
|
62 |
+
[... more news articles ...]
|
63 |
+
|
64 |
+
</EXAMPLE>
|
65 |
+
|
66 |
+
newsletter_task:
|
67 |
+
description: >
|
68 |
+
Fill the following HTML template with exactly the same information that is given to you. Also, include the following personal message at the beginning of the newsletter in the space provided: {personal_message}
|
69 |
+
|
70 |
+
(If there is no personal message, leave the space empty.)
|
71 |
+
|
72 |
+
Follow these rules:
|
73 |
+
- Do not truncate the information or change the order of the elements.
|
74 |
+
- Do not modify the content of the news articles.
|
75 |
+
- Do not add any additional information to the newsletter.
|
76 |
+
- Add ALL the news stories provided to the newsletter. Even if the template only shows one news story, you should include all the news stories provided as context using the format provided in the template.
|
77 |
+
|
78 |
+
{html_template}
|
79 |
+
expected_output: >
|
80 |
+
Return ONLY the contents of the HTML template, without any tripple quotes like '```', '```html' or any other text.
|
81 |
+
This output will be saved as a HTML file and sent to the subscribers.
|