The publishing version of JarvisFlow is at https://huggingface.co/aiflows/JarvisFlowModule -- This one runs too, ofc
Table of Contents
- JarvisFlow
- Controller_JarvisFlow
- UpdatePlanAtomicFlow
- Planner_JarvisFlow
- run_Jarvis
- CtrlExMem_JarvisFlow
- __init__
- IntermediateAns_Jarvis
- FinalAns_Jarvis
JarvisFlow
JarvisFlow Objects
class JarvisFlow(AbstractBossFlow)
JarvisFlow is a flow module for the boss Jarvis. It inherits from AbstractBossFlow. ( https://huggingface.co/Tachi67/AbstractBossFlowModule/tree/main). Jarvis is a general purpose agent empowered by multiple large language models and tools including a code interpreter, to take task commands in natural language, and make plans, write and run code in an interactive fashion to finish the task.
The highlight of Jarvis is that it integrates 17 large language models, each of them prompted differently to achieve seamless inter-model communication and model-user interaction. The structure of Jarvis ensures that it is much more robust, flexible and memory-efficient than previous agents empowered by one single model. What's more, Jarvis integrates modules to allow for llm's memory management, ensuring persisted mid-long term memory and efficient short-term memory management, making its life duration much longer than single-modeled agents, and more powerful, in that it is able to accumulate important knowledge e.g. code library. Jarvis can also take response from the user and the environment (e.g. code execution result), and spontaneously re-plan and re-execute to make the execution more robust and reliable.
Configuration Parameters:
memory_files(dict): mem_name-memfile_path pairs. mem_name is the name of the memory (plan, logs, code_library), and memfile_path is the path to the corresponding memory file. Configure this either in the .yaml file, or override thememory_filesentry when running the flow.subflows_config(dict): configs for subflows.MemoryReading: Module used to read in memory (https://huggingface.co/Tachi67/MemoryReadingFlowModule), output interface configured so that it outputs the neeed memory.Planner: Module used to interactively write plans for Jarvis, the planner is implemented in the JarvisFlow.CtrlExMem: Module used to execute the plan in a controller-executor manner, and update the memory. It is implemented in the JarvisFlow.
The code interpreter of Jarvis (https://huggingface.co/Tachi67/InterpreterFlowModule) relies on open-interpreter (https://github.com/KillianLucas/open-interpreter) We are extracting the specific code from open-interpreter because the litellm version of open-interpreter is not compatible with that of the current version of aiflows(v.0.1.7).
Controller_JarvisFlow
Controller_JarvisFlow Objects
class Controller_JarvisFlow(ChatAtomicFlow)
This class is a controller for JarvisFlow, it takes the plan generated by the planner, logs of previous executions, depending on the initial goal or the subsequent feedback from the branching executors (and the human), to decide which executor to call next (or to exit by calling finish).
Configuration Parameters:
commands(dict): a dictionary of commands that the controller can call, each command has a name, a description, and a list of input arguments. The commands will be injected into the system message prompt template.system_message_prompt_template(str): the template for the system message prompt, there are several components needs to be injected into the template, including the commands, plan, plan_file_location, logs, and the goal. The injection of commands is done then initalizing the flow, the rest of the components are injected at the beginning of each run.previous_messages(int): a sliding window of previous messages that will be passed to the model. This is the central part of short-term memory management.
Input Interface Non Initialized:
goal(str): the initial goal of the conversation, this is the input to the model.memory_files(dict): a dictionary of file locations that contains the plan, logs.plan(str): the plan generated by the planner, the plan will change (marked as done, or re-plan) as execution preceeds.logs(str): the logs of previous executions, the logs will be appended as execution preceeds.
Input Interface Initialized:
result(str): the result of the previous execution, this is the input to the model.memory_files(dict): a dictionary of file locations that contains the plan, logs.plan(str): the plan generated by the planner, the plan will change (marked as done, or re-plan) as execution preceeds.logs(str): the logs of previous executions, the logs will be appended as execution preceeds.goal(str): the initial goal, this is kept because the goal is also injected into the system prompts so that Jarvis does not forget what the goal is, when the memory sliding window is implemented.
Output Interface:
command(str): the command to be executed by the executor.command_args(dict): the arguments of the command to be executed by the executor.
__init__
def __init__(commands: List[Command], **kwargs)
Initialize the flow, inject the commands into the system message prompt template.
instantiate_from_config
@classmethod
def instantiate_from_config(cls, config)
Setting up the flow from the config file. In particular, setting up the prompts, backend, and commands.
UpdatePlanAtomicFlow
UpdatePlanAtomicFlow Objects
class UpdatePlanAtomicFlow(AtomicFlow)
This class is used to update the plan file with the updated plan, called by the controlller, when it realizes one step of the plan is done, and provide the updated plan, it is exactly the same as the old plan, except the step that is done is marked as done.
Planner_JarvisFlow
Planner_JarvisFlow Objects
class Planner_JarvisFlow(PlanWriterFlow)
This flow inherits from PlanWriterFlow (https://huggingface.co/Tachi67/PlanWriterFlowModule), and is used to generate a plan for Jarvis. Input Interface:
goal(str): the goal of the planner, the goal comes from the user's query when calling Jarvis.memory_files(dict): a dictionary of memory files, the keys are the names of the memory files, the values are the locations of the memory files.
Output Interfaces:
plan(str): the generated plan, the plan string will be written to the plan file and returned to the flow state of the Jarvis flow.summary(str): the summary of the planner.status(str): the status of the planner, can be "finished" or "unfinished".
run_Jarvis
CtrlExMem_JarvisFlow
CtrlExMem_JarvisFlow Objects
class CtrlExMem_JarvisFlow(CtrlExMemFlow)
This class inherits from the CtrlExMemFlow class from AbstractBossFlowModule. See: https://huggingface.co/Tachi67/AbstractBossFlowModule/blob/main/CtrlExMemFlow.py
Take notice that:
- In the controller, we only keep the previous 3 messages for memory management, that will be: a. The assistant message (controller's last command) b. Manually updated new system prompt (new logs, new plans, etc.) c. The user message (result, feedback)
- Each time one executor from the branch is executed, the logs is updated, this means: a. The logs file of Jarvis is updated. b. After MemoryReading at the end of each run of the loop, the logs in the flow_state is updated. c. The next time the controller is called, the updated logs is injected into the system prompts.
- In the prompts of the controller, when the controller realizes one step of the plan is done, we ask the controller to revise what was done and mark the current step as done. This means: a. The plan file is updated. b. The plan in the flow_state is updated. c. The next time the controller is called, the updated plan is injected into the system prompts.
This is basically how the memory management works, to allow for more space for llm execution, and make sure the llm does not forget important information.
__init__
IntermediateAns_Jarvis
IntermediateAns_Jarvis Objects
class IntermediateAns_Jarvis(HumanStandardInputFlow)
This class inherits from the HumanStandardInputFlow class. It is used to give an intermediate answer to the user. The user is then able to provide feedback on the intermediate result. Depending on the user's feedback, the controller will decide to do different things (e.g. continue, re-plan, etc.)
FinalAns_Jarvis
FinalAns_Jarvis Objects
class FinalAns_Jarvis(HumanStandardInputFlow)
This class inherits from the HumanStandardInputFlow class. It is used to give the final answer to the user.