File size: 2,430 Bytes
11f924c 50dc97e 11f924c 50dc97e ad61c6f 50dc97e ad61c6f 50dc97e ad61c6f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
---
language: en
license: mit
datasets:
- covid19
---
# CoQUAD_MPNet : MPNet model for COVID-19
## Introduction
It is a state-of-the-art language model for MPNet for Covid-19 dataset with focus on post-covid.
## How to use for Deepset Haystack
###
```
%cd /content/drive/MyDrive
!sudo apt-get install git-lfs
!git lfs install
!git clone https://huggingface.co/shaina/CoQUAD_MPNet
GIT_LFS_SKIP_SMUDGE=1
from haystack.utils import clean_wiki_text, convert_files_to_dicts, fetch_archive_from_http, print_answers
from haystack.nodes import FARMReader, TransformersReader
from haystack.utils import launch_es
launch_es()
! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-7.9.2
import os
from subprocess import Popen, PIPE, STDOUT
es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
stdout=PIPE, stderr=STDOUT,
preexec_fn=lambda: os.setuid(1) # as daemon
)
! sleep 30
from haystack.document_stores import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
import pandas as pd
df=pd.read_excel('/content/covid.xlsx')
df.fillna(value="", inplace=True)
print(df.head())
from typing import List
import requests
import pandas as pd
from haystack import Document
from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import RAGenerator, DensePassageRetriever
titles = list(df["document_identifier"].values)
texts = list(df["document_text"].values)
documents: List[Document] = []
for title, text in zip(titles, texts):
documents.append(
Document(
content=text,
meta={
"name": title or ""
}
)
)
document_store.write_documents(documents)
from haystack.nodes import ElasticsearchRetriever
retriever = ElasticsearchRetriever(document_store=document_store)
reader = FARMReader(model_name_or_path="/content/drive/MyDrive/CoQUAD_MPNet", use_gpu=True)
from haystack.pipelines import ExtractiveQAPipeline
pipe = ExtractiveQAPipeline(reader, retriever)
prediction = pipe.run(
query="What is post-COVID?", params={"Retriever": {"top_k": 10}, "Reader": {"top_k": 5}}
)
from pprint import pprint
pprint(prediction)
```
## Authors
Shaina Raza
|