neofung

/

LdIR-Qwen2-reranker-1.5B

like 2

I seek your guidance on the following questions and look forward to your reply.
Question 1:

• Is the training base of neofung/LdIR-Qwen2-reranker-1.5B based on a pretrain model or a chat model?
  From the code:

messages = [
    {"role": "user",
    "content": "\n\n".join(source)}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text])

I infer that it might be based on a chat model. Could you share the model base if it's convenient for you?

Question 2: Does this model have a classification head or a regression head at the end?

• From the file https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/reranker/modeling.py (which you mentioned in the #3 discussion), I see that the cross_entropy loss is used.
  However, from the demo code model = FlagRerankerCustom(model=model, tokenizer=tokenizer, use_fp16=False), the last layer of the instantiated object seems to be a regression head.
  Please respond to my confusion; I am very grateful and look forward to your reply.

Question 1:
https://huggingface.co/Qwen/Qwen2-1.5B-Instruct

Question 2:
A regression model. During training, we constructed each training group as one positive and train_group_size -1 negatives. We applied cross_entropy in training, target_label is initialized by torch.zeros, which means position zero in training group is positive, and remains are negatives, which meets the requirement of cross_entropy