mdelmas/BioGPT-Large-Natural-Products-RE-Diversity-1000-synt-v1.1

BioGPT-Large-Natural-Products-RE-Diversity-1000-synt-v1.1

Natural products represent a large pool of bioactive compounds of high interest in drug-discovery. However, these relationships are sparsely distributed across organisms and a growing part of the literature remains unannotated. This volume necessitates the development of a machine assistant to boost the completion of existing resources. Framing the task as an end-to-end Relation Extraction (RE), we propose a BioGPT model fined-tuned on synthetic data. See details about this procedure in the join article.

The model is a derived from microsoft/BioGPT-Large and was trained on a synthetic dataset generated with Vicuna-13b-v1.5, derived from LLaMA2.

The dataset is available on zenodo.

Dataset:

You can use the model directly as:

Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model_hf = "microsoft/BioGPT-Large"
lora_adapters = "mdelmas/BioGPT-Large-Natural-Products-RE-Extended-synt-v1.0"

# Load model and plug adapters using peft
model = AutoModelForCausalLM.from_pretrained(model_hf)
model = PeftModel.from_pretrained(model, lora_adapters)
model = model.merge_and_unload()
tokenizer = AutoTokenizer.from_pretrained(model_hf)

# Example from PubMed article 24048364
title_text = "Producers and important dietary sources of ochratoxin A and citrinin."
abstract_text = "Ochratoxin A (OTA) is a very important mycotoxin, and its research is focused right now on the new findings of OTA, like being a complete carcinogen, information about OTA producers and new exposure sources of OTA. Citrinin (CIT) is another important mycotoxin, too, and its research turns towards nephrotoxicity. Both additive and synergistic effects have been described in combination with OTA. OTA is produced in foodstuffs by Aspergillus Section Circumdati (Aspergillus ochraceus, A. westerdijkiae, A. steynii) and Aspergillus Section Nigri (Aspergillus carbonarius, A. foetidus, A. lacticoffeatus, A. niger, A. sclerotioniger, A. tubingensis), mostly in subtropical and tropical areas. OTA is produced in foodstuffs by Penicillium verrucosum and P. nordicum, notably in temperate and colder zones. CIT is produced in foodstuffs by Monascus species (Monascus purpureus, M. ruber) and Penicillium species (Penicillium citrinum, P. expansum, P. radicicola, P. verrucosum). OTA was frequently found in foodstuffs of both plant origin (e.g., cereal products, coffee, vegetable, liquorice, raisins, wine) and animal origin (e.g., pork/poultry). CIT was also found in foodstuffs of vegetable origin (e.g., cereals, pomaceous fruits, black olive, roasted nuts, spices), food supplements based on rice fermented with red microfungi Monascus purpureus and in foodstuffs of animal origin (e.g., cheese)."
text = title_text + " " + abstract_text

# Tokenization
input_text = text + tokenizer.eos_token + tokenizer.bos_token
input_tokens = tokenizer(input_text, return_tensors='pt')

# Decoding parameters
EVAL_GENERATION_ARGS = {"max_length": 1024,
                          "do_sample": False,
                          "forced_eos_token_id": tokenizer.eos_token_id,
                          "num_beams": 3,
                          "early_stopping": "never",
                          "length_penalty": 1.5,
                          "temperature": 0}

# Generation
with torch.no_grad():
    beam_output = model.generate(**input_tokens, **EVAL_GENERATION_ARGS)
    output = tokenizer.decode(beam_output[0][len(input_tokens["input_ids"][0]):], skip_special_tokens=True)

# Parse and print
rels = output.strip().split("; ")
for rel in rels:
    print("- " + rel)

Citation

If you find this model useful, please cite: