Possible Vector Collaps Issue
#10
by
Banso
- opened
Hey, amazing work!
In some of my experiments, the model seems to predict vectors in certain areas, which causes the relatively high (> 0.7) cosine similarity.
I think this might be due to vector collapse. Maybe it's worth investigating in this direction.
You are right, the produced embeddings concentrate in a certain area of the vector space.
I suspect this is a result of using contrastive loss with low temperature (0.01), but it is surely a good research question.