README.md

metadata

license: mit
base_model: roberta-large
language:
  - en
metrics:
  - f1
pipeline_tag: token-classification

MetaIE (Academia)

This is a meta-model distilled from ChatGPT-4o for information extraction in the academic domain. This is an intermediate checkpoint that can be well-transferred to all kinds of downstream information extraction tasks. This model can also be tested by different label-to-span matching as shown in the following example:

from transformers import AutoModelForTokenClassification, AutoTokenizer
from nltk import word_tokenize
import torch

device = torch.device("cuda")
path = f"KomeijiForce/roberta-large-metaie-super-academia-gpt4o"
tokenizer = AutoTokenizer.from_pretrained(path)
tagger = AutoModelForTokenClassification.from_pretrained(path).to(device)

def find_sequences(lst):
    sequences = []
    i = 0
    while i < len(lst):
        if lst[i] == 0:
            start = i
            end = i
            i += 1
            while i < len(lst) and lst[i] == 1:
                end = i
                i += 1
            sequences.append((start, end+1))
        else:
            i += 1
    return sequences

passage = '''MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.'''


labels = [
    "Research Domain",
    "Tasks for MetaIE",
    "Way to Build MetaIE",
    "Method",
    "Model Performance",
]

for label in labels:
    
    text = f"{label}: " + passage
    
    inputs = tokenizer(text, return_tensors="pt").to(device)
    tag_predictions = tagger(**inputs).logits[0].argmax(-1)

    predictions = [tokenizer.decode(inputs.input_ids[0, seq[0]:seq[1]]).strip() for seq in find_sequences(tag_predictions)]

    print(label, predictions)

The output will be

Research Domain ['natural language processing']
Tasks for MetaIE ['named entity recognition', 'relation extraction']
Way to Build MetaIE ['learning to extract "important information', 'symbolic distillation from an LLM following the label']
Method ['symbolic distillation from an LLM following the label']
Model Performance ['outperform other meta-models']