How to end dead loop of multi-turn and enter parallel responses?
#4
by
DiTy
- opened
Hey,
In the process of using your model, I had a question. How did you mean to stop multi-turn?
Is the prompt filling correct (Will the generation be stable?), after the request A, if I just want to get a response from the executed function and just get a response from AI, should I collect the next step with an empty user_input?
<s> [INST] [BEGIN OF TASK INSTRUCTION]
Based on the previous context and API request history, generate an API request or a response as an AI assistant.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE TOOLS]
[{"name": "get_weather", "description": "Get the current weather for a location", "parameters": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, New York"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit of temperature to return"}}}, {"name": "search", "description": "Search for information on the internet", "parameters": {"query": {"type": "string", "description": "The search query, e.g. 'latest news on AI'"}}}]
[END OF AVAILABLE TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
The output should be of the JSON format, which specifies a list of generated function calls. The example format is as follows, please make sure the parameter type is correct. If no function call is needed, please make
tool_calls an empty list "[]".
#```
{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}
#```
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
What's the weather like in New York in fahrenheit?
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{"step_id": 0, "thought": "I need to get the current weather for New York in fahrenheit.", "tool_calls": [{"name": "get_weather", "arguments": {"location": "New York", "unit": "fahrenheit"}}], "next_observation": {"temperature": "61F"}, "user_input": ""}]
[END OF HISTORY STEPS]
[/INST]
agent_action:
'{"thought": "AI: The current temperature in New York is 61 degrees Fahrenheit.", "tool_calls": []}'
Did I understand everything correctly?
In addition, what is the correct format for providing parallel tool responses?
Thank you in advance for your reply!
DiTy
changed discussion title from
How to end dead loop of multi-turn?
to How to end dead loop of multi-turn and enter parallel responses?
Hi @DiTy , we have provided more details examples (https://huggingface.co/Salesforce/xLAM-8x22b-r/blob/main/example/xlam_chat_template_examples_11_21.ipynb) on how to use the models. Please let us know if you have any further questions:)
This is almost exactly what I want to know :D
Thank you
@jianguozhang
, I almost found out everything I wanted.
In your example, there is no parallel response of tools for different functions, only for one type. Is stable generation possible in this case?
messages = [
{'role': "user", 'content': "Hi, tell me pls about the weather in New York and wind speed in Krasnodar!"},
{
'role': "assistant",
'content': "The query asks for the current weather in New York and the current wind speed in Krasnodar. I will use get_current_weather for New York and get_current_wind_speed for Krasnodar.",
'tool_calls': [{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 1672444800}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}]
},
{
'role': "tool_results",
"name": "get_current_weather",
"content": {
"data": {
"weather": "sunny",
"temperature": "120F"
}
}
},
{
'role': "tool_results",
"name": "get_current_wind_speed",
"content": {
"data": {
"speed": 22.3,
"unit_of_measurement": "km/h"
}
}
}
]
Full prompt with model generation for this case:
[INST]
[BEGIN OF TASK INSTRUCTION]
You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the functions can be used, point it out and refuse to answer.
If the given question lacks the parameters required by the function, also point it out.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE_TOOLS]
[{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}, 'date': {'type': 'integer', 'description': 'a specific date to get the weather for'}}, 'required': ['location', 'format', 'date']}}}, {'type': 'function', 'function': {'name': 'get_current_temperature', 'description': 'Get the current temperature at a location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The unit to return the temperature in.'}, 'longitude': {'type': 'number', 'description': 'the longitude of the location'}, 'latitude': {'type': 'number', 'description': 'the latitude of the location'}}, 'required': ['location', 'unit', 'longitude', 'latitude']}, 'return': {'type': 'number', 'description': 'The current temperature at the specified location in the specified units, as a float.'}}}, {'type': 'function', 'function': {'name': 'get_current_wind_speed', 'description': 'Get the current wind speed in km/h at a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}}, 'required': ['location']}, 'return': {'type': 'number', 'description': 'The current wind speed at the given location in km/h, as a float.'}}}]
[END OF AVAILABLE_TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
` ` `{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}` ` `
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
Hi, tell me pls about the weather in New York and wind speed in Krasnodar!
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{'thought':"The query asks for the current weather in New York and the current wind speed in Krasnodar. I will use get_current_weather for New York and get_current_wind_speed for Krasnodar.", 'tool_calls':[{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 1672444800}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}], 'step_id':1, 'next_observation':{"data": {"weather": "sunny", "temperature": "120F"}}}, 'next_observation':{"data": {"speed": 22.3, "unit_of_measurement": "km/h"}}} ]
[END OF HISTORY STEPS]
[/INST] {"thought": "The query asks for the current weather in New York and the current wind speed in Krasnodar. I will use get_current_weather for New York and get_current_wind_speed for Krasnodar.", "tool_calls": [{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 1672444800}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}]}
It is look like hallucination.
upd: BTW it is actually doesnt work for even simple case like this and I dont know why.
[INST]
[BEGIN OF TASK INSTRUCTION]
You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the functions can be used, point it out and refuse to answer.
If the given question lacks the parameters required by the function, also point it out.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE_TOOLS]
[{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}, 'date': {'type': 'integer', 'description': 'a specific date to get the weather for'}}, 'required': ['location', 'format', 'date']}}}, {'type': 'function', 'function': {'name': 'get_current_temperature', 'description': 'Get the current temperature at a location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The unit to return the temperature in.'}, 'longitude': {'type': 'number', 'description': 'the longitude of the location'}, 'latitude': {'type': 'number', 'description': 'the latitude of the location'}}, 'required': ['location', 'unit', 'longitude', 'latitude']}, 'return': {'type': 'number', 'description': 'The current temperature at the specified location in the specified units, as a float.'}}}, {'type': 'function', 'function': {'name': 'get_current_wind_speed', 'description': 'Get the current wind speed in km/h at a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}}, 'required': ['location']}, 'return': {'type': 'number', 'description': 'The current wind speed at the given location in km/h, as a float.'}}}]
[END OF AVAILABLE_TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
` ` `{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}` ` `
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
Hi, tell me pls about the weather in New York!
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{'thought':"The user is asking for the current weather in New York. I will use the get_current_weather function with the location set to 'New York, NY' and the format set to 'fahrenheit'.", 'tool_calls':[{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 0}}], 'step_id':1, 'next_observation':{"data": {"weather": "sunny", "temperature": "120F"}}} ]
[END OF HISTORY STEPS]
[/INST] {"thought": "The user is asking for the current weather in New York. I will use the get_current_weather function with the location set to 'New York, NY' and the format set to 'fahrenheit'.", "tool_calls": [{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 0}}]}
Hi @DiTy thank you for the feedback. We've included examples of parallel function call generations in above Jupyter notebook and will soon add multi-turn interactions with parallel function calls.
By the way, while I need some time to look into the example mentioned above, I did input it into GPT-4o, and it produced similar results.
Below is the corresponding prompt
<s> [INST]
[BEGIN OF TASK INSTRUCTION]
You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the functions can be used, point it out and refuse to answer.
If the given question lacks the parameters required by the function, also point it out.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE_TOOLS]
[{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'description': 'The city and state, e.g. San Francisco, CA', 'type': 'string'}, 'format': {'description': "The temperature unit to use. Infer this from the users location. (choices: ['celsius', 'fahrenheit'])", 'type': 'string'}, 'date': {'description': 'a specific date to get the weather for', 'type': 'int'}}}}}, {'type': 'function', 'function': {'name': 'get_current_wind_speed', 'description': 'Get the current wind speed in km/h at a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'description': "The location to get the temperature for, in the format 'City, Country'", 'type': 'string'}, 'required': ['location']}}}}]
[END OF AVAILABLE_TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
```{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}```
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
Hi, tell me pls about the weather in New York and wind speed in Krasnodar!
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{'thought':"The query asks for the current weather in New York and the current wind speed in Krasnodar. I will use get_current_weather for New York and get_current_wind_speed for Krasnodar.", 'tool_calls':[{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 1672444800}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}], 'step_id':1, 'next_observation':"{\"data\": {\"weather\": \"sunny\", \"temperature\": \"120F\"}}"}, 'next_observation':"{\"data\": {\"speed\": 22.3, \"unit_of_measurement\": \"km/h\"}}"} ]
[END OF HISTORY STEPS]
[/INST]
Hey
@jianguozhang
BTW I am glad of our pleasant communication :D
I think I managed to figure out the problem. Model has hallucinations when I use default SYSTEM prompt from your chat_template:
<s> [INST]
[BEGIN OF TASK INSTRUCTION]
You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the functions can be used, point it out and refuse to answer.
If the given question lacks the parameters required by the function, also point it out.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE_TOOLS]
[{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}, 'date': {'type': 'integer', 'description': 'a specific date to get the weather for'}}, 'required': ['location', 'format', 'date']}}}, {'type': 'function', 'function': {'name': 'get_current_temperature', 'description': 'Get the current temperature at a location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The unit to return the temperature in.'}, 'longitude': {'type': 'number', 'description': 'the longitude of the location'}, 'latitude': {'type': 'number', 'description': 'the latitude of the location'}}, 'required': ['location', 'unit', 'longitude', 'latitude']}, 'return': {'type': 'number', 'description': 'The current temperature at the specified location in the specified units, as a float.'}}}, {'type': 'function', 'function': {'name': 'get_current_wind_speed', 'description': 'Get the current wind speed in km/h at a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}}, 'required': ['location']}, 'return': {'type': 'number', 'description': 'The current wind speed at the given location in km/h, as a float.'}}}]
[END OF AVAILABLE_TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
` ` `{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}` ` `
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
What's the weather like in New York in fahrenheit and wind speed in Krasnodar!
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{'thought':"I need to get the current weather in New York in Fahrenheit and the current wind speed in Krasnodar.", 'tool_calls':[{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 0}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}], 'step_id':1, 'next_observation':{"data": [{"weather": "sunny", "temperature": "120"}, {"wind_speed": 20.3, "unit_of_measure": "km/h"}]}} ]
[END OF HISTORY STEPS]
[/INST]
And model's response:
' {"thought": "I need to get the current weather in New York in Fahrenheit and the current wind speed in Krasnodar.", "tool_calls": [{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 0}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}]}'
But when I use system prompt: Based on the previous context and API request history, generate an API request or a response as an AI assistant. model hasn't hallucinations:
<s> [INST]
[BEGIN OF TASK INSTRUCTION]
Based on the previous context and API request history, generate an API request or a response as an AI assistant.
[END OF TASK INSTRUCTION]
[BEGIN OF AVAILABLE_TOOLS]
[{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}, 'date': {'type': 'integer', 'description': 'a specific date to get the weather for'}}, 'required': ['location', 'format', 'date']}}}, {'type': 'function', 'function': {'name': 'get_current_temperature', 'description': 'Get the current temperature at a location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The unit to return the temperature in.'}, 'longitude': {'type': 'number', 'description': 'the longitude of the location'}, 'latitude': {'type': 'number', 'description': 'the latitude of the location'}}, 'required': ['location', 'unit', 'longitude', 'latitude']}, 'return': {'type': 'number', 'description': 'The current temperature at the specified location in the specified units, as a float.'}}}, {'type': 'function', 'function': {'name': 'get_current_wind_speed', 'description': 'Get the current wind speed in km/h at a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The location to get the temperature for, in the format "City, Country"'}}, 'required': ['location']}, 'return': {'type': 'number', 'description': 'The current wind speed at the given location in km/h, as a float.'}}}]
[END OF AVAILABLE_TOOLS]
[BEGIN OF FORMAT INSTRUCTION]
Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
` ` `{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}` ` `
[END OF FORMAT INSTRUCTION]
[BEGIN OF QUERY]
What's the weather like in New York in fahrenheit and wind speed in Krasnodar!
[END OF QUERY]
[BEGIN OF HISTORY STEPS]
[{'thought':"I need to get the current weather in New York in Fahrenheit and the current wind speed in Krasnodar.", 'tool_calls':[{"name": "get_current_weather", "arguments": {"location": "New York, NY", "format": "fahrenheit", "date": 0}}, {"name": "get_current_wind_speed", "arguments": {"location": "Krasnodar, Russia"}}], 'step_id':1, 'next_observation':{"data": [{"weather": "sunny", "temperature": "120"}, {"wind_speed": 20.3, "unit_of_measure": "km/h"}]}} ]
[END OF HISTORY STEPS]
[/INST]
We get:
' {"thought": "AI: The weather in New York is sunny with a temperature of 120 degrees Fahrenheit, and the wind speed in Krasnodar is 20.3 km/h.", "tool_calls": []}'
In this regard, I have a question, what is your GROUNDED SYSTEM prompt (I mean system prompt in training)?
Hi @DiTy , thank you for your example and insights! We currently don't have a unified system prompt; the default system prompt in the chat template is just one of our training prompts.
If you have any additional feedback or suggestions for future model iterations, please feel free to let us know! 😁