Ưu tiên có thể diễn giải thông qua Mô hình Phần thưởng Đa mục tiêu và Hỗn hợp Chuyên gia
•
1
Tien Dung
tiendung
AI & ML interests
None yet
Recent Activity
updated
a Space
5 days ago
Symato/tomtat
liked
a dataset
13 days ago
microsoft/orca-agentinstruct-1M-v1
updated
a collection
14 days ago
RAG
Articles
Organizations
tiendung's activity
updated
a
Space
5 days ago
liked
a
dataset
13 days ago
updated
a
collection
14 days ago
liked
a
model
14 days ago
liked
a
model
20 days ago
liked
a
dataset
20 days ago
liked
2
datasets
22 days ago
upvoted
a
paper
22 days ago
liked
a
model
22 days ago
liked
a
Space
22 days ago
updated
a
collection
22 days ago
liked
a
dataset
22 days ago
upvoted
a
collection
22 days ago
upvoted
a
paper
22 days ago
Reacted to
singhsidhukuldeep's
post with 👀
22 days ago
Post
2084
Exciting Research Alert: Revolutionizing Dense Passage Retrieval with Entailment Tuning!
The good folks at HKUST have developed a novel approach that significantly improves information retrieval by leveraging natural language inference.
The entailment tuning approach consists of several key steps to enhance dense passage retrieval performance.
Data Preparation
- Convert questions into existence claims using rule-based transformations.
- Combine retrieval data with NLI data from SNLI and MNLI datasets.
- Unify the format of both data types using a consistent prompting framework.
Entailment Tuning Process
- Initialize the model using pre-trained language models like BERT or RoBERTa.
- Apply aggressive masking (β=0.8) specifically to the hypothesis components while preserving premise information.
- Train the model to predict the masked hypothesis tokens from the premise content.
- Run the training for 10 epochs using 8 GPUs, taking approximately 1.5-3.5 hours.
Training Arguments for Entailment Tuning (Yes! They Shared Them)
- Use a learning rate of 2e-5 with 100 warmup steps.
- Set batch size to 128.
- Apply weight decay of 0.01.
- Utilize the Adam optimizer with beta values (0.9, 0.999).
- Maintain maximum gradient norm at 1.0.
Deployment
- Index passages using FAISS for efficient retrieval.
- Shard vector store across multiple GPUs.
- Enable sub-millisecond retrieval of the top-100 passages per query.
Integration with Existing Systems
- Insert entailment tuning between pre-training and fine-tuning stages.
- Maintain compatibility with current dense retrieval methods.
- Preserve existing contrastive learning approaches during fine-tuning.
Simple, intuitive, and effective!
This advancement significantly improves the quality of retrieved passages for question-answering systems and retrieval-augmented generation tasks.
The good folks at HKUST have developed a novel approach that significantly improves information retrieval by leveraging natural language inference.
The entailment tuning approach consists of several key steps to enhance dense passage retrieval performance.
Data Preparation
- Convert questions into existence claims using rule-based transformations.
- Combine retrieval data with NLI data from SNLI and MNLI datasets.
- Unify the format of both data types using a consistent prompting framework.
Entailment Tuning Process
- Initialize the model using pre-trained language models like BERT or RoBERTa.
- Apply aggressive masking (β=0.8) specifically to the hypothesis components while preserving premise information.
- Train the model to predict the masked hypothesis tokens from the premise content.
- Run the training for 10 epochs using 8 GPUs, taking approximately 1.5-3.5 hours.
Training Arguments for Entailment Tuning (Yes! They Shared Them)
- Use a learning rate of 2e-5 with 100 warmup steps.
- Set batch size to 128.
- Apply weight decay of 0.01.
- Utilize the Adam optimizer with beta values (0.9, 0.999).
- Maintain maximum gradient norm at 1.0.
Deployment
- Index passages using FAISS for efficient retrieval.
- Shard vector store across multiple GPUs.
- Enable sub-millisecond retrieval of the top-100 passages per query.
Integration with Existing Systems
- Insert entailment tuning between pre-training and fine-tuning stages.
- Maintain compatibility with current dense retrieval methods.
- Preserve existing contrastive learning approaches during fine-tuning.
Simple, intuitive, and effective!
This advancement significantly improves the quality of retrieved passages for question-answering systems and retrieval-augmented generation tasks.
updated
a
collection
26 days ago
liked
a
model
26 days ago
liked
a
dataset
26 days ago