the space link is a collection, is there a space? this has some practical use cases for our research id be interesting inn testing it out on
Hi @Alignment-Lab-AI ,
First of all, thank you very much for your attention! I haven’t yet set up a Hugging Face Space for this, but I will prepare a demo in the near future as I plan to use this technique in another paper as well.
If you’d like to quickly test the empathy level scoring using the model we provided, you can try the following input:
<|start_header_id|>system<|end_header_id|>
You are a helpful assistant.
<|eot_id|><|start_header_id|>user<|end_header_id|>
This is an Empathy Level Comparison task at the conversation-turn level. You are asked to compare the perceived empathy levels of the last conversation turn of two dialogues and predict the perceived empathy levels for both. You need to focus on the last turn of the dialogue, which starts with the "Speaker1: " string at each dialogue's last line. The two dialogues are tagged as "Dia1" and "Dia2". In prediction, please use the corresponding dialogue tag to refer to each dialogue. You should first provide the comparison result, then give the perceived empathy level for each dialogue's last turn. The perceived empathy level is divided into 16 levels. All annotations are in the range between 0 to 15 and must be made using integers only.
Dia1:
Speaker2: Hello, how are you?
Speaker1: Hello! Fine. How are you?
Speaker2: I'm good, thanks for asking!
Speaker1: That's good, so I guess we are supposed to discuss the article. I wonder, if my family or I were in a place where my very life was at risk, what would I do?
Speaker2: That's a very scary thought to have honestly.
Speaker1: Yes, I think it is something most people would rather not imagine. But it is a reality.
Dia2:
Speaker2: Hey, what did you think of that article?
Speaker1: I thought it was pretty well-written, but I wonder who is supposed to receive our empathy... the children or the smokers?
Speaker2: I think it is supposed to make you choose. I can see both sides, but I really have to side with the children.
Speaker1: I agree with that. It’s good to protect the kids.
Speaker2: Yeah, their bodies are so fragile as is.
Speaker1: As they are developing, it’s important to make sure they can develop safely.
<|eot_id|><|start_header_id|>assistant<|end_header_id|>
The model's output could look like this:
The perceived empathy level of Dia1's last turn is higher than Dia2's. The perceived empathy level of Dia1's last turn is 10, and Dia2's last turn is 7.
This template is also included in the appendix section of the paper.
You can compare Dia1 with various other dialogue contents to obtain multiple empathy scores for Dia1, and use the algorithm discussed in the paper to compute the average score. Of course, you can also apply your preferred score aggregation methods, such as voting, median, or distribution estimation, etc. I will discuss the effects of different strategies in more detail in my upcoming paper.
If you have any further questions, feel free to reach out.
Additionally, many thanks to @nicolay-r for sharing this.