---
license: cc-by-sa-4.0
datasets:
- unicamp-dl/mmarco
- bclavie/mmarco-japanese-hard-negatives
language:
- ja
---



## Evaluation on [MIRACL japanese](https://huggingface.co/datasets/miracl/miracl)
These models don't train on the MIRACL training data.

| Model            | nDCG@10 | Recall@1000 | Recall@5 | Recall@30 |
|------------------|---------|-------------|----------|-----------|
| BM25             | 0.369   | 0.931       | -        | -         |
| splade-japanese  | 0.405   | 0.931       | 0.406    | 0.663     |
| splade-japanese-efficient| 0.408  | 0.954      | 0.419   | 0.718    |
| splade-japanese-v2 | 0.580   | 0.967       | 0.629    | 0.844     |
| splade-japanese-v2-doc | 0.478 | 0.930 | 0.514 | 0.759 |
| splade-japanese-v3 | **0.604** | **0.979** | **0.647** | **0.877** |


*'splade-japanese-v2-doc' model does not require query encoder during inference.


## Evaluation on [hotchpotch/JQaRA](https://huggingface.co/datasets/hotchpotch/JQaRA)

|                     |     |           | JQaRa     |           |           |
| ------------------- | --- | --------- | --------- | --------- | --------- |
|                     |     | NDCG@10   | MRR@10    | NDCG@100  | MRR@100   |
| splade-japanese-v3  |     |  0.505    | 0.772     |    0.7    |  0.775    |
| JaColBERTv2         |     |  0.585    | 0.836     | 0.753     | 0.838     |
| JaColBERT           |     | 0.549     | 0.811     | 0.730     | 0.814     |
| bge-m3+all          |     | 0.576     | 0.818     | 0.745     | 0.820     |
| bg3-m3+dense        |     | 0.539     | 0.785     | 0.721     | 0.788     |
| m-e5-large          |     | 0.554     | 0.799     | 0.731     | 0.801     |
| m-e5-base           |     | 0.471     | 0.727     | 0.673     | 0.731     |
| m-e5-small          |     | 0.492     | 0.729     | 0.689     | 0.733     |
| GLuCoSE             |     | 0.308     | 0.518     | 0.564     | 0.527     |
| sup-simcse-ja-base  |     | 0.324     | 0.541     | 0.572     | 0.550     |
| sup-simcse-ja-large |     | 0.356     | 0.575     | 0.596     | 0.583     |
| fio-base-v0.1       |     | 0.372     | 0.616     | 0.608     | 0.622     |


下のコードを実行すれば，単語拡張や重み付けの確認ができます．

If you'd like to try it out, you can see the expansion of queries or documents by running the code below.

you need to install 

```
!pip install fugashi ipadic unidic-lite
```

```python
from transformers import AutoModelForMaskedLM,AutoTokenizer
import torch
import numpy as np

model = AutoModelForMaskedLM.from_pretrained("aken12/splade-japanese-v3") 
tokenizer = AutoTokenizer.from_pretrained("aken12/splade-japanese-v3")
vocab_dict = {v: k for k, v in tokenizer.get_vocab().items()}

def encode_query(query): ##query passsage maxlen: 32,180
    query = tokenizer(query, return_tensors="pt")
    output = model(**query, return_dict=True).logits
    output, _ = torch.max(torch.log(1 + torch.relu(output)) * query['attention_mask'].unsqueeze(-1), dim=1)
    return output

with torch.no_grad():
    model_output = encode_query(query="筑波大学では何の研究が行われているか？")

reps = model_output
idx = torch.nonzero(reps[0], as_tuple=False)

dict_splade = {}
for i in idx:
    token_value = reps[0][i[0]].item()
    if token_value > 0:
        token = vocab_dict[int(i[0])]
        dict_splade[token] = float(token_value)

sorted_dict_splade = sorted(dict_splade.items(), key=lambda item: item[1], reverse=True)
for token, value in sorted_dict_splade:
    print(token, value)
```