Diving into a Sea of Opinions: Multi-Modal Abstractive Summarization with Comment Sensitivity

Abstract: In the modern era, the rapid expansion of social media and the proliferation of the internet community has led to a multi-fold increase in the richness and range of views and outlooks expressed by readers and viewers. To obtain valuable insights from this vast sea of opinions, we present an inventive and holistic procedure for multi-modal abstractive summarization with comment sensitivity. Our proposed model utilizes both textual and visual modalities and examines the remarks provided by the readers to produce summaries that apprehend the significant points and opinions made by them. Our model features a transformer-based encoder that seamlessly processes both news articles and comments, merging them before transmitting the amalgamated information to the decoder. Additionally, the core segment of our architecture consists of an attention-based merging technique which is trained adversarially by means of a generator and discriminator to bridge the semantic gap between comments and articles. We have used a Bi-LSTM-based branch for image pointer generation. We assess our model on the reader-aware multi-document summarization (RA-MDS) dataset which contains news articles, their summaries, and related comments. We have extended the dataset by adding images pertaining to news articles in the corpus to increase the richness and diversity of the dataset. Our comprehensive experiments reveal that our model outperforms similar pre-trained models and baselines across two of the four evaluated metrics, showcasing its superior performance.

adversarial learning
multimodal
abstractive summarization
comment sensitivity


GranCATs: Cross-Lingual Enhancement through Granularity-Specific Contrastive Adapters

Abstract: Multilingual language models (MLLMs) have demonstrated remarkable success in various cross-lingual downstream tasks, facilitating the transfer of knowledge across numerous languages, whereas this transfer is not universally effective. Our study reveals that while existing MLLMs like mBERT can capturephrase-level alignments across the language families, they struggle to effectively capturesentence-level andparagraph-level alignments. To address this limitation, we propose GranCATs, Granularity-specific Contrastive AdapTers. We collect a new dataset that observes each sample at three distinct levels of granularity and employ contrastive learning as a pre-training task to train GranCATs on this dataset. Our objective is to enhance MLLMs' adaptation to a broader range of cross-lingual tasks by equipping them with improved capabilities to capture global information at different levels of granularity. Extensive experiments show that MLLMs with GranCATs yield significant performance advancements across various language tasks with different text granularities, including entity alignment, relation extraction, sentence classification and retrieval, and question-answering. These results validate the effectiveness of our proposed GranCATs in enhancing cross-lingual alignments across various text granularities and effectively transferring this knowledge to downstream tasks.

multilingual language models
adapters
universal patterns
cross-lingual alignments


Inducing Causal Structure for Abstractive Text Summarization

Abstract: The mainstream of data-driven abstractive summarization models tends to explore the correlations rather than the causal relationships. Among such correlations, there can be spurious ones which suffer from the language prior learned from the training corpus and therefore undermine the overall effectiveness of the learned model. To tackle this issue, we introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We assume several latent causal factors and non-causal factors, representing the content and style of the document and summary. Theoretically, we prove that the latent factors in our SCM can be identified by fitting the observed training data under certain conditions. On the basis of this, we propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors, guiding us to pursue causal information for summary generation. The key idea is to reformulate the Variational Auto-encoder (VAE) to fit the joint distribution of the document and summary variables from the training corpus. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.

abstractive text summarization
vae
structural causal model


James Ate 5 Oranges = Steve Bought 5 Pencils: Structure-Aware Denoising for Paraphrasing Word Problems

Abstract: We propose SCANING, an unsupervised framework for paraphrasing via controlled noise injection. We focus on the novel task of paraphrasing algebraic word problems having practical applications in online pedagogy as a means to reduce plagiarism as well as evoke reasoning capabilities on the part of the student instead of rote memorization. This task is more complex than paraphrasing general-domain corpora due to the difficulty in preserving critical information for solution consistency of the paraphrased word problem, managing the increased length of the text and ensuring diversity in the generated paraphrase. Existing approaches fail to demonstrate adequate performance on at least one, if not all, of these facets, necessitating the need for a more comprehensive solution. To this end, we model the noising search space as a composition of contextual and syntactic aspects to sample noising functions. This allows for learning a denoising function, that operates over both aspects and produces semantically equivalent and syntactically diverse outputs through grounded noise injection. The denoising function serves as a foundation for training a paraphrasing function, which operates solely in the input-paraphrase space without carrying any direct dependency on noise. We demonstrate that SCANING improves performance in terms of producing semantically equivalent and syntactically diverse paraphrases by 35% through extensive automated and human evaluation across 4 datasets.

mwp
education
self-supervised
paraphrasing
denoising
neural networks


NOVO: Learnable and Interpretable Document Identifiers for Model-Based IR

Abstract: Model-based Information Retrieval (Model-based IR) has gained attention due to advancements in generative language models. Unlike traditional dense retrieval methods relying on dense vector representations of documents, model-based IR leverages language models to retrieve documents by generating their unique discrete identifiers (docids). This approach effectively reduces the requirements to store separate document representations in an index. Most existing model-based IR approaches utilize pre-defined static docids, i.e., these docids are fixed and are not learnable by training on the retrieval tasks. However, these docids are not specifically optimized for retrieval tasks, which makes it difficult to learn semantics and relationships between documents and achieve satisfactory retrieval performance. To address the above limitations, we propose Neural Optimized VOcabularial (NOVO) docids. NOVO docids are unique n-gram sets identifying each document. They can be generated in any order to retrieve the corresponding document and can be optimized through training to better learn semantics and relationships between documents. We propose to optimize NOVO docids through query denoising modeling and retrieval tasks, allowing for optimizing both semantic and token representations for such docids. Experiments on two datasets under the normal and zero-shot settings show that NOVO exhibits strong performance in more effective and interpretable model-based IR.

large language models
model-based ir
information retrieval


Rethinking Sentiment Analysis under Uncertainty

Abstract: Sentiment Analysis (SA) is a fundamental task in natural language processing, which is widely used in public decision-making. Recently, deep learning have demonstrated great potential to deal with this task. However, prior works have mostly treated SA as a deterministic classification problem, and meanwhile, without quantifying the predictive uncertainty. This presents a serious problem in the SA, different annotator, due to the differences in beliefs, values, and experiences, may have different perspectives on how to label the text sentiment. Such situation will lead to inevitable data uncertainty and make the deterministic classification models feel puzzle to make decision. To address this issue, we propose a new SA paradigm with the consideration of uncertainty and conduct an expensive empirical study. Specifically, we treat SA as the regression task and introduce uncertainty quantification to obtain confidence intervals for predictions, which enables the risk assessment ability of the model and can improve the credibility of SA-aids decision-making. Experiments on five datasets show that our proposed new paradigm effectively quantifies uncertainty in SA while remaining competitive performance to point estimation, in addition to being capable of Out-Of-Distribution (OOD) detection.

sentiment analysis
out-of-distribution detection
quantifying uncertainty


Sentiment-Aware Review Summarization with Personalized Multi-Task Fine-Tuning

Abstract: Personalized review summarization is a challenging task in recommender systems, which aims to generate condensed and readable summaries for product reviews. Recently, some methods propose to adopt the sentiment signals of reviews to enhance the review summarization. However, most previous works only share the semantic features of reviews via preliminary multi-task learning, while ignoring the rich personalized information of users and products, which is crucial to both sentiment identification and comprehensive review summarization. In this paper, we propose a sentiment-aware review summarization method with an elaborately designed multi-task fine-tuning framework to make full use of personalized information of users and products effectively based on Pretrained Language Models (PLMs). We first denote two types of personalized information including IDs and historical summaries to indicate their identification and semantics information respectively. Subsequently, we propose to incorporate the IDs of the user/product into the PLMs-based encoder to learn the personalized representations of input reviews and their historical summaries in a fine-tuning way. Based on this, an auxiliary context-aware review sentiment classification task and a further sentiment-guided personalized review summarization task are jointly learned. Specifically, the sentiment representation of input review is used to identify relevant historical summaries, which are then treated as additional semantic context features to enhance the summary generation process. Extensive experimental results show our approach could generate sentiment-consistent summaries and outperforms many competitive baselines on both review summarization and sentiment classification tasks.

pre-trained language models
personalized review summarization
recommender system
sentiment classification


Bias Invariant Approaches for Improving Word Embedding Fairness

Abstract: Many public pre-trained word embeddings have been shown to encode different types of biases. Embeddings are often obtained from training on large pre-existing corpora, and therefore resulting biases can be a reflection of unfair representations in the original data. Bias, in this scenario, is a challenging problem since current mitigation techniques require knowing and understanding existing biases in the embedding, which is not always possible. In this work, we propose to improve word embedding fairness by borrowing methods from the field of data privacy. The idea behind this approach is to treat bias as if it were a special type of training data leakage. This has the unique advantage of not requiring prior knowledge of potential biases in word embeddings. We investigated two types of privacy algorithms, and measured their effect on bias using four different metrics. To investigate techniques from differential privacy, we applied Gaussian perturbation to public pre-trained word embeddings. To investigate noiseless privacy, we applied vector quantization during training. Experiments show that both approaches improve fairness for commonly used embeddings, and additionally, noiseless privacy techniques reduce the size of the resulting embedding representation.


HCL4QC: Incorporating Hierarchical Category Structures Into Contrastive Learning for E-Commerce Query Classification

Abstract: Query classification plays a crucial role in e-commerce, where the goal is to assign user queries to appropriate categories within a hierarchical product category taxonomy. However, existing methods rely on a limited number of words from the category description and often neglect the hierarchical structure of the category tree, resulting in suboptimal category representations. To overcome these limitations, we propose a novel approach named hierarchical contrastive learning framework for query classification (HCL4QC), which leverages the hierarchical category tree structure to improve the performance of query classification. Specifically, HCL4QC is designed as a plugin module that consists of two innovative losses, namely local hierarchical contrastive loss (LHCL) and global hierarchical contrastive loss (GHCL). LHCL adjusts representations of categories according to their positional relationship in the hierarchical tree, while GHCL ensures the semantic consistency between the parent category and its child categories. Our proposed method can be adapted to any query classification tasks that involve a hierarchical category structure. We conduct experiments on two real-world datasets to demonstrate the superiority of our hierarchical contrastive learning. The results demonstrate significant improvements in the query classification task, particularly for long-tail categories with sparse supervised information.

query classification
hierarchical category tree
contrastive learning


MultiPLe: Multilingual Prompt Learning for Relieving Semantic Confusions in Few-Shot Event Detection

Abstract: Event detection (ED) is a challenging task in the field of information extraction. Due to the monolingual text and rampant confusing triggers, traditional ED models suffer from semantic confusions in terms of polysemy and synonym, leading to severe detection mistakes. Such semantic confusions can be further exacerbated in a practical situation where scarce labeled data cannot provide sufficient semantic clues. To mitigate such bottleneck, we propose a multilingual prompt learning (MultiPLe) framework for few-shot event detection (FSED), including three components, i.e., a multilingual prompt, a hierarchical prototype and a quadruplet contrastive learning module. In detail, to ease the polysemy confusion, the multilingual prompt module develops the in-context semantics of triggers via the multilingual disambiguation and prior knowledge in pretrained language models. Then, the hierarchical prototype module is adopted to diminish the synonym confusion by connecting the captured inmost semantics of fuzzy triggers with labels at a fine granularity. Finally, we employ the quadruplet contrastive learning module to tackle the insufficient label representation and potential noise. Experiments on two public datasets show that MultiPLe outperforms the state-of-the-art baselines in weighted F1-score, presenting a maximum improvement of 13.63% for FSED.

semantic confusions
prompt learning
few-shot event detection


On the Thresholding Strategy for Infrequent Labels in Multi-Label Classification

Abstract: In multi-label classification, the imbalance between labels is often a concern. For a label that seldom occurs, the default threshold used to generate binarized predictions of that label is usually sub-optimal. However, directly tuning the threshold to optimize F-measure has been observed to overfit easily. In this work, we explain why this overfitting occurs. Then, we analyze the FBR heuristic, a previous technique proposed to address the overfitting issue. We explain its success but also point out some problems unobserved before. Then, we first propose a variant of the FBR heuristic that not only fixes the problems but is also more justifiable. Second, we propose a new technique based on smoothing the F-measure when tuning the threshold. We theoretically prove that, with proper parameters, smoothing results in desirable properties of the tuned threshold. Based on the idea of smoothing, we then propose jointly optimizing micro-F and macro-F as a lightweight alternative free from extra hyperparameters. Our methods are empirically evaluated on text and node classification datasets. The results show that our methods consistently outperform the FBR heuristic.

multi-label classification
infrequent labels
threshold adjustion
f-measure


Relevance-Based Infilling for Natural Language Counterfactuals

Abstract: Counterfactual explanations are a natural way for humans to gain understanding and trust in the outcomes of complex machine learning algorithms. In the context of natural language processing, generating counterfactuals is particularly challenging as it requires the generated text to be fluent, grammatically correct, and meaningful. In this study, we improve the current state of the art for the generation of such counterfactual explanations for text classifiers. Our approach, named RELITC (Relevance-based Infilling for Textual Counterfactuals), builds on the idea of masking a fraction of text tokens based on their importance in a given prediction task and employs a novel strategy, based on the entropy of their associated probability distributions, to determine the infilling order of these tokens. Our method uses less time than competing methods to generate counterfactuals that require less changes, are closer to the original text and preserve its content better, while being competitive in terms of fluency. We demonstrate the effectiveness of the method on four different datasets and show the quality of its outcomes in a comparison with human generated counterfactuals.

explainability
masked language model
counterfactuals
nlp


Class-Specific Word Sense Aware Topic Modeling via Soft Orthogonalized Topics

Abstract: We propose a word sense aware topic model for document classification based on soft orthogonalized topics. An essential problem for this task is to capture word senses related to classes, i.e., class-specific word senses. Traditional models mainly introduce semantic information of knowledge libraries for word sense discovery. However, this information may not align with the classification targets, because these targets are often subjective and task-related. We aim to model the class-specific word senses in topic space. The challenge is to optimize the class separability of the senses, i.e., obtaining sense vectors with (a) high intra-class and (b) low inter-class similarities. Most existing models predefine specific topics for each class to specify the class-specific sense vectors. We call them hard orthogonalization based methods. These methods can hardly achieve both (a) and (b) since they assume the conditional independence of topics to classes and inevitably lose topic information. To this problem, we propose soft orthogonalization for topics. Specifically, we reserve all the topics and introduce a group of class-specific weights for each word to handle the importance of topic dimensions to class separability. Besides, we detect and use highly class-specific words in each document to guide sense estimation. Our experiments on two standard datasets show that our proposal outperforms other state-of-the-art models in terms of accuracy of sense estimation, document classification, and topic modeling. In addition, our joint learning experiments with the pre-trained language model BERT showcased the best complementarity of our model in most cases compared to other topic models.

document classification
document representation
word sense disambiguation
graphical model
topic model


EAGLE: Enhance Target-Oriented Dialogs by Global Planning and Topic Flow Integration

Abstract: In this study, we propose a novel model EAGLE for target-oriented dialogue generation. Without relying on any knowledge graphs, our method integrates the global planning strategy in both topic path generation and response generation given the initial and target topics. EAGLE comprises three components: a topic path sampling strategy, a topic flow generator, and a global planner. Our approach confers a number of advantages: EAGLE is robust to the target that has never appeared in the training data set and able to plan the topic flow globally. The topic path sampling strategy samples topic paths based on two predefined rules and use the sampled paths to train the topic path generator. The topic flow generator then applies a non-autoregressive method to generate intermediate topics that link the initial and target topics smoothly. In addition, the global planner is a response generator that generates a response based on the future topic sequence and conversation history, enabling it to plan how to transition to future topics smoothly. Our experimental results demonstrate that EAGLE produces more coherent responses and smoother transitions than state-of-the-art baselines, with an overall success rate improvement of approximately 25% and an average smoothness score improvement of 10% in both offline and human evaluations.

topic transition
target-oriented
conversation generation
global planning


Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning

Abstract: Prompt tuning has enhanced the performance of Pre-trained Language Models for multi-task learning in few-shot scenarios. However, existing studies fail to consider that the prompts among different layers in Transformer are different due to the diverse information learned at each layer. In general, the bottom layers in the model tend to capture low-level semantic or structural information, while the upper layers primarily acquire task-specific knowledge. Hence, we propose a novel hierarchical prompt tuning model for few-shot multi-task learning to capture this regularity. The designed model mainly consists of three types of prompts: shared prompts, auto-adaptive prompts, and task-specific prompts. Shared prompts facilitate the sharing of general information across all tasks. Auto-adaptive prompts dynamically select and integrate relevant prompt information from all tasks into the current task. Task-specific prompts concentrate on learning task-specific knowledge. To enhance the model's adaptability to diverse inputs, we introduce deep instance-aware language prompts as the foundation for constructing the above prompts. To evaluate the effectiveness of our proposed method, we conduct extensive experiments on multiple widely-used datasets. The experimental results demonstrate that the proposed method achieves state-of-the-art performance for multi-task learning in few-shot settings and outperforms ChatGPT in the full-data setting.

prompt tuning
few-shot learning
multi-task learning


Prompt Distillation for Efficient LLM-Based Recommendation

Abstract: Large language models (LLM) have manifested unparalleled modeling capability on various tasks, e.g., multi-step reasoning, but the input to these models is mostly limited to plain text, which could be very long and contain noisy information. Long text could take long time to process, and thus may not be efficient enough for recommender systems that require immediate response. In LLM-based recommendation models, user and item IDs are usually filled in a template (i.e., discrete prompt) to allow the models to understand a given task, but the models usually need extensive fine-tuning to bridge the user/item IDs and the template words and to unleash the power of LLM for recommendation. To address the problems, we propose to distill the discrete prompt for a specific task to a set of continuous prompt vectors so as to bridge IDs and words and to reduce the inference time. We also design a training strategy with an attempt to improve the efficiency of training these models. Experimental results on three real-world datasets demonstrate the effectiveness of our PrOmpt Distillation (POD) approach on both sequential recommendation and top-N recommendation tasks. Although the training efficiency can be significantly improved, the improvement of inference efficiency is limited. This finding may inspire researchers in the community to further improve the inference efficiency of LLM-based recommendation models.

recommender systems
prompt distillation
explainable recommendation
top-n recommendation
sequential recommendation
large language models
generative recommendation


Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection

Abstract: Despite considerable advances in automated fake news detection, due to the timely nature of news, it remains a critical open question how to effectively predict the veracity of news articles based on limited fact-checks. Existing approaches typically follow a "Train-from-Scratch" paradigm, which is fundamentally bounded by the availability of large-scale annotated data. While expressive pre-trained language models (PLMs) have been adapted in a "Pre-Train-and-Fine-Tune" manner, the inconsistency between pre-training and downstream objectives also requires costly task-specific supervision. In this paper, we propose "Prompt-and-Align" (P   amp;A), a novel prompt-based paradigm for few-shot fake news detection that jointly leverages the pre-trained knowledge in PLMs and the social context topology. Our approach mitigates label scarcity by wrapping the news article in a task-related textual prompt, which is then processed by the PLM to directly elicit task-specific knowledge. To supplement the PLM with social context without inducing additional training overheads, motivated by empirical observation on user veracity consistency (i.e., social users tend to consume news of the same veracity type), we further construct a news proximity graph among news articles to capture the veracity-consistent signals in shared readerships, and align the prompting predictions along the graph edges in a confidence-informed manner. Extensive experiments on three real-world benchmarks demonstrate that P   amp;A sets new states-of-the-art for few-shot fake news detection performance by significant margins.

fake news
social networks
prompt
few-shot learning


Prompting Strategies for Citation Classification

Abstract: Citation classification aims to identify the purpose of the cited article in the citing article. Previous citation classification methods rely largely on supervised approaches. The models are trained on datasets with citing sentences or citation contexts annotated for a citation's purpose or function or intent. Recent advancements in Large Language Models (LLMs) have dramatically improved the ability of NLP systems to achieve state-of-the-art performances under zero or few-shot settings. This makes LLMs particularly suitable for tasks where sufficiently large labelled datasets are not yet available, which remains to be the case for citation classification. This paper systematically investigates the effectiveness of different prompting strategies for citation classification and compares them to promptless strategies as a baseline. Specifically, we evaluate the following four strategies, two of which we introduce for the first time, which involve updating Language Model (LM) parameters while training the model: (1) Promptless fine-tuning, (2) Fixed-prompt LM tuning, (3) Dynamic Context-prompt LM tuning (proposed), (4) Prompt + LM fine-tuning (proposed). Additionally, we test the zero-shot performance of LLMs, GPT3.5, a (5) Tuning-free prompting strategy that involves no parameter updating. Our results show that prompting methods based on LM parameter updating significantly improve citation classification performances on both domain-specific and multi-disciplinary citation classifications. Moreover, our Dynamic Context-prompting method achieves top scores both for the ACL-ARC and ACT2 citation classification datasets, surpassing the highest-performing system in the 3C shared task benchmark. Interestingly, we observe zero-shot GPT3.5 to perform well on ACT2 but poorly on the ACL-ARC dataset.

citation classification
prompt training
large language models
research evaluation


WOT-Class: Weakly Supervised Open-World Text Classification

Abstract: State-of-the-art weakly supervised text classification methods, while significantly reduced the required human supervision, still requires the supervision to cover all the classes of interest. This is never easy to meet in practice when human explore new, large corpora without complete pictures. In this paper, we work on a novel yet important problem of weakly supervised open-world text classification, where supervision is only needed for a few examples from a few known classes and the machine should handle both known and unknown classes in test time. General open-world classification has been studied mostly using image classification; however, existing methods typically assume the availability of sufficient known-class supervision and strong unknown-class prior knowledge (e.g., the number and/or data distribution). We propose a novel framework øur that lifts those strong assumptions. Specifically, it follows an iterative process of (a) clustering text to new classes, (b) mining and ranking indicative words for each class, and (c) merging redundant classes by using the overlapped indicative words as a bridge. Extensive experiments on 7 popular text classification datasets demonstrate that øur outperforms strong baselines consistently with a large margin, attaining 23.33% greater average absolute macro-F1 over existing approaches across all datasets. Such competent accuracy illuminates the practical potential of further reducing human effort for text classification.

text classification
open-world learning
weak supervision


GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking

Abstract: Retrieval-enhanced text generation has shown remarkable progress on knowledge-intensive language tasks, such as open-domain question answering and knowledge-enhanced dialogue generation, by leveraging passages retrieved from a large passage corpus for delivering a proper answer given the input query. However, the retrieved passages are not ideal for guiding answer generation because of the discrepancy between retrieval and generation, i.e., the candidate passages are all treated equally during the retrieval procedure without considering their potential to generate a proper answer. This discrepancy makes a passage retriever deliver a sub-optimal collection of candidate passages to generate the answer. In this paper, we propose the GeneRative Knowledge Improved Passage Ranking (GripRank) approach, addressing the above challenge by distilling knowledge from a generative passage estimator (GPE) to a passage ranker, where the GPE is a generative language model used to measure how likely the candidate passages can generate the proper answer. We realize the distillation procedure by teaching the passage ranker learning to rank the passages ordered by the GPE. Furthermore, we improve the distillation quality by devising a curriculum knowledge distillation mechanism, which allows the knowledge provided by the GPE can be progressively distilled to the ranker through an easy-to-hard curriculum, enabling the passage ranker to correctly recognize the provenance of the answer from many plausible candidates. We conduct extensive experiments on four datasets across three knowledge-intensive language tasks. Experimental results show advantages over the state-of-the-art methods for both passage ranking and answer generation on the KILT benchmark.

passage ranking
knowledge distillation
retrieval-enhanced text generation
knowledge-intensive language tasks


Optimizing Upstream Representations for Out-of-Domain Detection with Supervised Contrastive Learning

Abstract: Out-of-Domain (OOD) text detection has attracted significant research interest. However, conventional approaches primarily employ Cross-Entropy loss during upstream encoder training and seldom focus on optimizing discriminative In-Domain (IND) and OOD representations. To fill this gap, we introduce a novel method that applies supervised contrastive learning (SCL) to IND data for upstream representation optimization. This effectively brings the embeddings of semantically similar texts together while pushing dissimilar ones further apart, leading to more compact and distinct IND representations. This optimization subsequently improves the differentiation between IND and OOD representations, thereby enhancing the detection effect in downstream tasks. To further strengthen the ability of SCL to consolidate IND embedding clusters, and to improve the generalizability of the encoder, we propose a method that generates two different variations of the same text as "views". This is achieved by applying a twice "dropped-out" on the embeddings before performing SCL. Extensive experiments indicate that our method not only outperforms state-of-the-art approaches, but also reduces the requirement for training a large 354M-parameter model down to a more efficient 110M-parameter model, highlighting its superiority in both effectiveness and computational economy.

out-of-domain detection
supervised contrastive learning


Spans, Not Tokens: A Span-Centric Model for Multi-Span Reading Comprehension

Abstract: Many questions should be answered by not a single answer but a set of multiple answers. This emerging Multi-Span Reading Comprehension (MSRC) task requires extracting multiple non-contiguous spans from a given context to answer a question. Existing methods extend conventional single-span models to predict the positions of the start and end tokens of answer spans, or predict the beginning-inside-outside tag of each token. Such token-centric paradigms can hardly capture dependencies among span-level answers which are critical to MSRC. In this paper, we propose SpanQualifier, a span-centric scheme where spans, as opposed to tokens, are directly represented and scored to qualify as answers. Explicit span representations enable their interaction which exploits their dependencies to enhance representations. Experiments on three MSRC datasets demonstrate the effectiveness of our span-centric scheme and show that SpanQualifier achieves state-of-the-art results.

multi-span reading comprehension
question answering
msrc


Topic-Aware Contrastive Learning and K-Nearest Neighbor Mechanism for Stance Detection

Abstract: The goal of stance detection is to automatically recognize the author's expressed attitude in text towards a given target. However, social media users often express themselves briefly and implicitly, which leads to a significant number of comments lacking explicit reference information to the target, posing a challenge for stance detection. To address the missing relationship between text and target, existing studies primarily focus on incorporating external knowledge, which inevitably introduces noise information. In contrast to their work, we are dedicated to mining implicit relational information within data. Typically, users tend to emphasize their attitudes towards a relevant topic or aspect of the target while concealing others when expressing opinions. Motivated by this phenomenon, we suggest that the potential correlation between text and target can be learned from instances with similar topics. Therefore, we design a pretext task to mine the topic associations between samples and model this topic association as a dynamic weight introduced into contrastive learning. In this way, we can selectively cluster samples that have similar topics and consistent stances, while enlarging the gap between samples with different stances in the feature space. Additionally, we propose a nearest-neighbor prediction mechanism for stance classification to better utilize the features we constructed. Our experiments on two datasets demonstrate the advanced and generalization ability of our method, yielding the state-of-the-art results.

topic association
nearest neighbor mechanism
contrastive learning
stance detection
pretext task


Towards Spoken Language Understanding via Multi-Level Multi-Grained Contrastive Learning

Abstract: Spoken language understanding (SLU) is a core task in task-oriented dialogue systems, which aims at understanding user's current goal through constructing semantic frames. SLU usually consists of two subtasks, including intent detection and slot filling. Although there are some SLU frameworks joint modeling the two subtasks and achieve the high performance, most of them still overlook the inherent relationships between intents and slots, and fail to achieve mutual guidance between the two subtasks. To solve the problem, we propose a multi-level multi-grained SLU framework MMCL to apply contrastive learning at three levels, including utterance level, slot level, and word level to enable intent and slot to mutually guide each other. For the utterance level, our framework implements coarse granularity contrastive learning and fine granularity contrastive learning simultaneously. Besides, we also apply the self-distillation method to improve the robustness of the model. Experimental results and further analysis demonstrate that our proposed model achieves new state-of-the-art results on two public multi-intent SLU datasets, obtaining a 2.6 overall accuracy improvement on MixATIS dataset compared to previous best models.

self-distillation
multi-grained
spoken language understanding
multi-level
contrastive learning


A Non-Factoid Question-Answering Taxonomy

Abstract: Non-factoid question answering (NFQA) is a challenging and under-researched task that requires constructing long-form answers, such as explanations or opinions, to open-ended non-factoid questions - NFQs. There is still little understanding of the categories of NFQs that people tend to ask, what form of answers they expect to see in return, and what the key research challenges of each category are. This work presents the first comprehensive taxonomy of NFQ categories and the expected structure of answers. The taxonomy was constructed with a transparent methodology and extensively evaluated via crowdsourcing. The most challenging categories were identified through an editorial user study. We also release a dataset of categorised NFQs and a question category classifier. Finally, we conduct a quantitative analysis of the distribution of question categories using major NFQA datasets, showing that the NFQ categories that are the most challenging for current NFQA systems are poorly represented in these datasets. This imbalance may lead to insufficient system performance for challenging categories. The new taxonomy, along with the category classifier, will aid research in the area, helping to create more balanced benchmarks and to focus models on addressing specific categories.

dataset analysis
question taxonomy
editorial study
non-factoid question-answering


QUASER: Question Answering with Scalable Extractive Rationalization

Abstract: Designing natural language processing (NLP) models that produce predictions by first extracting a set of relevant input sentences, i.e., rationales, is gaining importance for improving model interpretability and producing supporting evidence for users. Current unsupervised approaches are designed to extract rationales that maximize prediction accuracy, which is invariably obtained by exploiting spurious correlations in datasets, and leads to unconvincing rationales. In this paper, we introduce unsupervised generative models to extract dual-purpose rationales, which must not only be able to support a subsequent answer prediction, but also support a reproduction of the input query. We show that such models can produce more meaningful rationales, that are less influenced by dataset artifacts, and as a result, also achieve the state-of-the-art on rationale extraction metrics on four datasets from the ERASER benchmark, significantly improving upon previous unsupervised methods. Our multi-task model is scalable and enables using state-of-the-art pretrained language models to design explainable question answering systems.

explainability
robustness
question answering


PTAU: Prompt Tuning for Attributing Unanswerable Questions

Abstract: Current question answering systems are insufficient when confronting real-life scenarios, as they can hardly be aware of whether a question is answerable given its context. Hence, there is a recent pursuit of unanswerability of a question and its attribution. Attribution of unanswerability requires the system to choose an appropriate cause for an unanswerable question. As the task is sophisticated for even human beings, it is expensive to acquire labeled data, which makes it a low-data regime problem. Moreover, the causes themselves are semantically abstract and complex, and the process of attribution is heavily question- and context-dependent. Thus, a capable model has to carefully appreciate the causes, and then, judiciously contrast the question with its context, in order to cast it into the right cause. In response to the challenges, we present PTAU, which refers to and implements a high-level human reading strategy such that one reads with anticipation. In specific, PTAU leverages the recent prompt-tuning paradigm, and is further enhanced with two innovatively conceived modules: 1) a cause-oriented template module that constructs continuous templates towards certain attributing class in high dimensional vector space; and 2) a semantics-aware label module that exploits label semantics through contrastive learning to render the classes distinguishable. Extensive experiments demonstrate that the proposed design better enlightens not only the attribution model, but also current question answering models, leading to superior performance.

question answering
prompt tuning
attribution of unanswerability


DGQAN: Dual Graph Question-Answer Attention Networks for Answer Selection

Abstract: Community question answering (CQA) becomes increasingly prevalent in recent years, providing platforms for users with various backgrounds to obtain information and share knowledge. However, the redundancy and lengthiness issues of crowd-sourced answers limit the performance of answer selection, thus leading to difficulties in reading or even misunderstandings for community users. To solve these problems, we propose the dual graph question-answer attention networks (DGQAN) for answer selection task. Aims to fully understand the internal structure of the question and the corresponding answer, firstly, we construct a dual-CQA concept graph with graph convolution networks using the original question and answer text. Specifically, our CQA concept graph exploits the correlation information between question-answer pairs to construct two sub-graphs (QSubject-Answer and QBody-Answer), respectively. Further, a novel dual attention mechanism is incorporated to model both the internal and external semantic relations among questions and answers. More importantly, we conduct experiment to investigate the impact of each layer in the BERT model. The experimental results show that DGQAN model achieves state-of-the-art performance on three datasets (SemEval-2015, 2016, and 2017), outperforming all the baseline models.

answer selection
community question answering
dual graph attention


Tag-Assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

Abstract: Multimodal sentiment analysis has been studied under the assumption that all modalities are available. However, such a strong assumption does not always hold in practice, and most of multimodal fusion models may fail when partial modalities are missing. Several works have addressed the missing modality problem; but most of them only considered the single modality missing case, and ignored the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network's attention to those missing modalities. Besides, we adopt a new space projection pattern to align common vectors. Then, a Transformer encoder-decoder network is utilized to learn the missing modality features. At last, the outputs of the Transformer encoder are used for the final sentiment classification. Extensive experiments are conducted on CMU-MOSI and IEMOCAP datasets, showing that our method can achieve significant improvements compared with several baselines.

multimodal sentiment analysis
missing modality
joint representation


Mutual Disentanglement Learning for Joint Fine-Grained Sentiment Classification and Controllable Text Generation

Abstract: Fine-grained sentiment classification (FGSC) task and fine-grained controllable text generation (FGSG) task are two representative applications of sentiment analysis, two of which together can actually form an inverse task prediction, i.e., the former aims to infer the fine-grained sentiment polarities given a text piece, while the latter generates text content that describes the input fine-grained opinions. Most of the existing work solves the FGSC and the FGSG tasks in isolation, while ignoring the complementary benefits in between. This paper combines FGSC and FGSG as a joint dual learning system, encouraging them to learn the advantages from each other. Based on the dual learning framework, we further propose decoupling the feature representations in two tasks into fine-grained aspect-oriented opinion variables and content variables respectively, by performing mutual disentanglement learning upon them. We also propose to transform the difficult "data-to-text” generation fashion widely used in FGSG into an easier text-to-text generation fashion by creating surrogate natural language text as the model inputs. Experimental results on 7 sentiment analysis benchmarks including both the document-level and sentence-level datasets show that our method significantly outperforms the current strong-performing baselines on both the FGSC and FGSG tasks. Automatic and human evaluations demonstrate that our FGSG model successfully generates fluent, diverse and rich content conditioned on fine-grained sentiments.

representation disentanglement
controllable text generation
sentiment analysis
dual learning


Graph Adaptive Semantic Transfer for Cross-Domain Sentiment Classification

Abstract: Cross-domain sentiment classification (CDSC) aims to use the transferable semantics learned from the source domain to predict the sentiment of reviews in the unlabeled target domain. Existing studies in this task attach more attention to the sequence modeling of sentences while largely ignoring the rich domain-invariant semantics embedded in graph structures (i.e., the part-of-speech tags and dependency relations). As an important aspect of exploring characteristics of language comprehension, adaptive graph representations have played an essential role in recent years. To this end, in the paper, we aim to explore the possibility of learning invariant semantic features from graph-like structures in CDSC. Specifically, we present Graph Adaptive Semantic Transfer (GAST) model, an adaptive syntactic graph embedding method that is able to learn domain-invariant semantics from both word sequences and syntactic graphs. More specifically, we first raise a POS-Transformer module to extract sequential semantic features from the word sequences as well as the part-of-speech tags. Then, we design a Hybrid Graph Attention (HGAT) module to generate syntax-based semantic features by considering the transferable dependency relations. Finally, we devise an Integrated aDaptive Strategy (IDS) to guide the joint learning process of both modules. Extensive experiments on four public datasets indicate that GAST achieves comparable effectiveness to a range of state-of-the-art models.

domain adaptation
sentiment analysis
graph embedding
web content analysis
text mining


Aspect Feature Distillation and Enhancement Network for Aspect-Based Sentiment Analysis

Abstract: Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task designed to identify the polarity of a target aspect. Some works introduce various attention mechanisms to fully mine the relevant context words of different aspects, and use the traditional cross-entropy loss to fine-tune the models for the ABSA task. However, the attention mechanism paying partial attention to aspect-unrelated words inevitably introduces irrelevant noise. Moreover, the cross-entropy loss lacks discriminative learning of features, which makes it difficult to exploit the implicit information of intra-class compactness and inter-class separability. To overcome these challenges, we propose an Aspect Feature Distillation and Enhancement Network (AFDEN) for the ABSA task. We first propose a dual-feature extraction module to extract aspect-related and aspect-unrelated features through the attention mechanisms and graph convolutional networks. Then, to eliminate the interference of aspect-unrelated words, we design a novel aspect-feature distillation module containing a gradient reverse layer that learns aspect-unrelated contextual features through adversarial training, and an aspect-specific orthogonal projection layer to further project aspect-related features into the orthogonal space of aspect-unrelated features. Finally, we propose an aspect-feature enhancement module that leverages supervised contrastive learning to capture the implicit information between the same sentiment labels and between different sentiment labels. Experimental results on three public datasets demonstrate that our AFDEN model achieves state-of-the-art performance and verify the effectiveness and robustness of our model.

supervised contrastive learning
orthogonal projection
aspect-based sentiment analysis
adversarial training


IAOTP: An Interactive End-to-End Solution for Aspect-Opinion Term Pairs Extraction

Abstract: Recently, the aspect-opinion term pairs (AOTP) extraction task has gained substantial importance in the domain of aspect-based sentiment analysis. It intends to extract the potential pair of each aspect term with its corresponding opinion term present in a user review. Some existing studies heavily relied on the annotated aspect terms and/or opinion terms, or adopted external knowledge/resources to figure out the task. Therefore, in this study, we propose a novel end-to-end solution, called an Interactive AOTP (IAOTP) model, for exploring AOTP. The IAOTP model first tracks the boundary of each token in given aspect-specific and opinion-specific representations through a span-based operation. Next, it generates the candidate AOTP by formulating the dyadic relations between tokens through the Biaffine transformation. Then, it computes the positioning information to capture the significant distance relationship that each candidate pair holds. And finally, it jointly models collaborative interactions and prediction of AOTP through a 2D self-attention. Besides the IAOTP model, this study also proposes an independent aspect/opinion encoding model (a RS model) that formulates relational semantics to obtain aspect-specific and opinion-specific representations that can effectively perform the extraction of aspect and opinion terms. Detailed experiments conducted on the publicly available benchmark datasets for AOTP, aspect terms, and opinion terms extraction tasks, clearly demonstrate the significantly improved performance of our models relative to other competitive state-of-the-art baselines.

natural language processing
aspect-opinion term pairs extraction
aspect-based sentiment analysis


Sentiment analysis and opinion mining on educational data: A survey

Abstract: Sentiment analysis AKA opinion mining is one of the most widely used NLP applications to identify human intentions from their reviews. In the education sector, opinion mining is used to listen to student opinions and enhance their learning–teaching practices pedagogically. With advancements in sentiment annotation techniques and AI methodologies, student comments can be labelled with their sentiment orientation without much human intervention. In this review article, (1) we consider the role of emotional analysis in education from four levels: document level, sentence level, entity level, and aspect level, (2) sentiment annotation techniques including lexicon-based and corpus-based approaches for unsupervised annotations are explored, (3) the role of AI in sentiment analysis with methodologies like machine learning, deep learning, and transformers are discussed, (4) the impact of sentiment analysis on educational procedures to enhance pedagogy, decision-making, and evaluation are presented. Educational institutions have been widely invested to build sentiment analysis tools and process their student feedback to draw their opinions and insights. Applications built on sentiment analysis of student feedback are reviewed in this study. Challenges in sentiment analysis like multi-polarity, polysemous, negation words, and opinion spam detection are explored and their trends in the research space are discussed. The future directions of sentiment analysis in education are discussed.

sentiment analysis
opinion mining
student feedback
ai
bert
deep learning


ASPER: Attention-based approach to extract syntactic patterns denoting semantic relations in sentential context

Abstract: Semantic relationships, such as hyponym–hypernym, cause–effect, meronym–holonym etc., between a pair of entities in a sentence are usually reflected through syntactic patterns. Automatic extraction of such patterns benefits several downstream tasks, including, entity extraction, ontology building, and question answering. Unfortunately, automatic extraction of such patterns has not yet received much attention from NLP and information retrieval researchers. In this work, we propose an attention-based supervised deep learning model, ASPER, which extracts syntactic patterns between entities exhibiting a given semantic relation in the sentential context. We validate the performance of ASPER on three distinct semantic relations—hyponym–hypernym, cause–effect, and meronym–holonym on six datasets. Experimental results show that for all these semantic relations, ASPER can automatically identify a collection of syntactic patterns reflecting the existence of such a relation between a pair of entities in a sentence. In comparison to the existing methodologies of syntactic pattern extraction, ASPER’s performance is substantially superior.

syntactic pattern
syntactic pattern extraction


Development and verification of a user-friendly software for German text simplification focused on patients with cerebral palsy

Abstract: People with disabilities, especially those who have been diagnosed with cerebral palsy or other neurological diseases, are often not able to understand the text they are confronted with in their daily life. Important information, for example from the government, is manually translated by professionals into easy language suitable for these people, however, this method cannot be scaled-up to a large number of documents. We have developed a software that translates German text into a simpler version. The software, based on the easy language rules, shortens the text and replaces difficult words with synonyms. To maintain a correct grammar during the synonym replacement, the software recognizes the phrase containing the replaced word and conjugates the other words in this phrase, if needed. For the verification of the software developed, the grammatical correctness of the replacement was quantified by German native speakers and the text shortening was compared against the results of 12 healthy volunteer readers. The software provides an easy-to-use interface, which takes into account both the mental and physical difficulties of the impaired people. Therefore, one can assume that the software will be a helpful tool for the acquirement of text content for these people.

natural language processing
lexical simplification
cerebral palsy
easy language


Predicting Facebook sentiments towards research

Abstract: Social media platforms provide users with various ways of interacting with each other, such as commenting, reacting to posts, sharing content, and uploading pictures. Facebook is one of the most popular platforms, and its users frequently share and reshare posts, including research articles. Moreover, the reactions feature on Facebook allows users to express their feelings towards the content they view, providing valuable data for analysis. This study aims to predict the emotional impact of Facebook posts relating to research articles. We collected data on Facebook posts related to various scientific research domains, including Health Sciences, Social Sciences, Dentistry, Arts, and Humanities. We observed Facebook users’ reactions towards research articles and posts and found that ‘Like’ reactions were the most common. We also noticed that research articles from the Dentistry research domain received a lot of ‘Haha’ reactions. We used machine learning models to predict the sentiment of Facebook posts related to research articles. We used features such as the research article’s title sentiment, abstract sentiment, abstract length, author count, and research domain to build the models. We used five classifiers: Random Forest, Decision Tree, K-Nearest Neighbors, Logistic Regression, and Naïve Bayes. The models were evaluated using accuracy, precision, recall, and F-1 score metrics. The Random Forest classifier was the best model for two- and three-class labels, achieving accuracy measures of 86

research sentiments
applied machine learning
sentiment analysis
facebook reactions


Do cues in a video help in handling rare words in a machine translation system under a low-resource setting?

Abstract: Video modality is an emerging research area among the numerous modalities utilized for multimodal machine translation. Multimodal machine translation uses multiple modalities to improve the machine-translated target language from the source language. However, the currently available multimodal dataset is focused on a few well-studied languages. In this paper, we propose a video-guided multimodal machine translation (VMMT) model under a low-resource setting by building a synthetic multimodal dataset of the English-Hindi language pair, the first one of its kind for this language pair. The VMMT system employs spatio-temporal video context as an additional input modality along with the source text. The spatio-temporal video context is extracted using a pre-trained 3D convolutional neural network. We report how well the VMMT systems outperform the text-only neural machine translation (NMT) system using automatic evaluation metrics and human evaluation on two test datasets: one in-domain and another out-domain. Our results indicate that the use of video context as an additional input modality enhances the performance of the MT system in resolving the various MT challenges, such as handling rare words, ambiguity, etc., in both English→Hindi and Hindi→English translations. Our experimental results show a significant improvement of up to +4.2 BLEU and +0.07 chrF scores in English→Hindi and +5.4 BLEU and +0.07 chrF scores in Hindi→English with our VMMT system over unimodal NMT system. Our findings highlight the potential of visual cues as an additional modality for improving machine translation systems especially in low-resource settings and emphasize the importance of synthetic multimodal datasets in addressing the scarcity of diverse data for less-studied language pairs.

multimodal machine translation
neural machine translation
video guided
visual attention
low resource
hindi


A survey on Named Entity Recognition — datasets, tools, and methodologies

Abstract: Natural language processing (NLP) is crucial in the current processing of data because it takes into account many sources, formats, and purposes of data as well as information from various sectors of our economy, government, and private and public lives. We perform a variety of NLP operations on the text in order to complete certain tasks. One of them is NER (Named Entity Recognition). An act of recognizing and categorizing named entities that are presented in a text document is known as named entity recognition. The purpose of NER is to find references of rigid designators in the text which belong to established semantic kinds like a person, place, organization, etc. It acts as a cornerstone for many Information Extraction-related activities. In this work, we present a thorough analysis of several methodologies for NER ranging from unsupervised learning, rule-based, supervised learning, and various Deep Learning based approaches. We examine the most relevant datasets, tools, and deep learning approaches like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Bidirectional Long Short Term Memory, Transfer learning approaches, and numerous other approaches currently being used in present-day NER problem environments and their applications. Finally, we outline the difficulties NER systems encounter and future directions.

natural language processing
named entity recognition
deep learning
convolutional neural network
bidirectional long short term memory
recurrent neural networks


Detecting abusive comments at a fine-grained level in a low-resource language

Abstract: YouTube is a video-sharing and social media platform where users create profiles and share videos for their followers to view, like, and comment on. Abusive comments on videos or replies to other comments may be offensive and detrimental for the mental health of users on the platform. It is observed that often the language used in these comments is informal and does not necessarily adhere to the formal syntactic and lexical structure of the language. Therefore, creating a rule-based system for filtering out abusive comments is challenging. This article introduces four datasets of abusive comments in Tamil and code-mixed Tamil–English extracted from YouTube. Comment-level annotation has been carried out for each dataset by assigning polarities to the comments. We hope these datasets can be used to train effective machine learning-based comment filters for these languages by mitigating the challenges associated with rule-based systems. In order to establish baselines on these datasets, we have carried out experiments with various machine learning classifiers and reported the results using F1-score, precision, and recall. Furthermore, we have employed a t-test to analyze the statistical significance of the results generated by the machine learning classifiers. Furthermore, we have employed SHAP values to analyze and explain the results generated by the machine learning classifiers. The primary contribution of this paper is the construction of a publicly accessible dataset of social media messages annotated with a fine-grained abusive speech in the low-resource Tamil language. Overall, we discovered that MURIL performed well on the binary abusive comment detection task, showing the applicability of multilingual transformers for this work. Nonetheless, a fine-grained annotation for Fine-grained abusive comment detection resulted in a significantly lower number of samples per class, and classical machine learning models outperformed deep learning models, which require extensive training datasets, on this challenge. According to our knowledge, this was the first Tamil-language study on FGACD focused on diverse ethnicities. The methodology for detecting abusive messages described in this work may aid in the creation of comment filters for other under-resourced languages on social media.

abuse detection
low-resourced languages
machine learning
deep learning models
tamil


Using Bidirectional Encoder Representations from Transformers (BERT) to classify traffic crash severity types

Abstract: Traffic crashes are a critical safety concern. Many studies have attempted to improve traffic safety by performing a wide range of studies on safety topics with the application of diverse statistical and machine learning models. The data elements contained in police-reported crash narrative information are not routinely analyzed with coded and structured crash data. In the recent years, unstructured textual contents in traffic crash narratives have been investigated by many researchers. However, most of these studies are basic text mining applications and often the dataset is limited in size. This study applied an advanced language model Bidirectional Encoder Representations from Transformers (BERT) to classify traffic injury types by using a dataset of over 750,000 unique crash narrative reports. The models have an 84.2

crashes
safety
severity
text mining
bert
language model


Semantics aware intelligent framework for content-based e-learning recommendation

Abstract: E-learning accounts for the emergence of re-skilling, up-skilling, and augmenting the traditional education system by providing knowledge delivery. The meaningful learning approach is based on a constructivist process for conceptually modeling an individual’s current and past knowledge or experience towards personalization. This research proposes a novel framework in which semantic analysis of e-content is combined with deep machine learning techniques into an e-learning Intelligent Content-Based Recommendation System (ICRS), with the goal of assisting learners in selecting appropriate e-learning materials. The system tackles textual e-content to extract representative terms and their mutual semantic relationships by which a structure of the context-based graph is developed. Thus, the e-content is semantically represented according to the learner’s terms that are expanded using the ConceptNet semantic network to represent the textual knowledge meaningfully. Furthermore, a new approach is proposed wherein more contextual and semantic information among concepts with graphs are combined to infer the relative semantic relations between terms and e-learning resources in order to build the semantic matrix. This new approach is utilized to generate learners’ semantic datasets used for the classification of available resources to enrich the recommendation. Here, four machine learning (ML) models and an augmented deep learning (DL) recommender model named LSTMM, which are developed for the ICRS framework. The models have been evaluated and compared using a user sequential semantic dataset. Our results show that the LSTMM performs better than others in terms of Accuracy and F1 Score of 0.8453 and 0.7731 respectively.

personalized e-learning recommendation
semantic analysis
term expansion
deep machine learning
knowledge graph


Rare words in text summarization

Abstract: Automatic text summarization is a difficult task, which involves a good understanding of an input text to produce fluent, brief and vast summary. The usage of text summarization models can vary from legal document summarization to news summarization. The model should be able to understand where important information is located to produce a good summary. However, infrequently used or rare words might limit model’s understanding of an input text, as the model might ignore such words or put less attention on them, especially, when the trained model is tested on the dataset, where the frequency distribution of used words is different. Another issue is that the model accepts only a limited amount of tokens (words) of an input text, which might contain redundant information or not including important information as it is located further in the text. To address the problem of rare words, we have proposed a modification to the attention mechanism of the transformer model with pointer-generator layer, where attention mechanism receives frequency information for each word, which helps to boost rare words. Additionally, our proposed supervised learning model uses the hybrid approach incorporating both extractive and abstractive elements, to include more important information for the abstractive model in a news summarization task. We have designed experiments involving a combination of six different hybrid models with varying input text sizes (measured as tokens) to test our proposed model. Four well-known datasets specific to news articles were used in this work: CNN/DM, XSum, Gigaword and DUC 2004 Task 1. Our results were compared using the well-known ROUGE metric. Our best model achieved R-1 score of 38.22, R-2 score of 15.07 and R-L score of 35.79, outperforming three existing models by several ROUGE points.

abstractive summarization
extractive summarization
pointer-generator
transformer
rare words


Improving aspect-based sentiment analysis with contrastive learning

Abstract: As a fine-grained sentiment analysis task focusing on detecting the sentiment polarity of aspect(s) in a sentence, aspect-based sentiment analysis (ABSA) plays a significant role in opinion analysis and review analysis. Recently, a number of methods have emerged to leverage contrastive learning techniques to enhance the performance of ABSA by learning fine-grained sentiment representations. In this paper, we present and compare two commonly used contrastive learning approaches for enhancing ABSA performance: sentiment-based supervised contrastive learning and augmentation-based unsupervised contrastive learning. Sentiment-based supervised contrastive learning employs sentiment labels to distinguish between positive and negative samples. Augmentation-based unsupervised contrastive learning aims to utilize various data augmentation strategies to generate positive samples. Experimental results on three public ABSA datasets demonstrate that both contrastive learning methods significantly improve the performance of ABSA. Sentiment-based supervised contrastive learning outperforms augmentation-based unsupervised contrastive learning in terms of overall performance improvements. Furthermore, we conduct additional experiments to illustrate the effectiveness and generalizability of these two contrastive learning approaches. The experimental code and data are publicly available at the link: https://github.com/Linda230/ABSA-CL.

aspect-based sentiment analysis
contrastive learning
data augmentation
sentiment label


A comprehensive review of State-of-The-Art methods for Java code generation from Natural Language Text

Abstract: Java Code Generation consists in generating automatically Java code from a Natural Language Text. This NLP task helps in increasing programmers’ productivity by providing them with immediate solutions to the simplest and most repetitive tasks. Code generation is a challenging task because of the hard syntactic rules and the necessity of a deep understanding of the semantic aspect of the programming language. Many works tried to tackle this task using either RNN-based, or Transformer-based models. The latter achieved remarkable advancement in the domain and they can be divided into three groups: (1) encoder-only models, (2) decoder-only models, and (3) encoder–decoder models. In this paper, we provide a comprehensive review of the evolution and progress of deep learning models in Java code generation task. We focus on the most important methods and present their merits and limitations, as well as the objective functions used by the community. In addition, we provide a detailed description of datasets and evaluation metrics used in the literature. Finally, we discuss results of different models on CONCODE dataset, then propose some future directions.

java code generation
language models
natural language processing
recurrent neural networks
transformer neural networks


DzNER: A large Algerian Named Entity Recognition dataset

Abstract: Named Entity Recognition (NER) is a natural language processing (NLP) task that involves assigning labels like Person, Location, and Organization to words in text. While there is a good amount of annotated data available for NER in English and other European languages, this is not the case for Arabic and its dialects. The goal of the paper is to introduce DzNER, an Algerian dataset for NER that consists of more than 21,000 manually annotated sentences (over 220,000 tokens) from Algerian Facebook pages and YouTube channels, with a focus on three prominent classes. In this study, we provide a detailed analysis of the NER tag-set used in the dataset and show that it has a good balance of quantity, diversity, and coverage of different domains. For the proof of resource-effectiveness, we also demonstrate the effectiveness of the dataset by using various language models for the sequence labeling task of NER and comparing the results to existing datasets. According to our research and knowledge, currently no available dataset meets the standards of both variability and volume as well as DzNER. We hope that this dataset and the accompanying code and models will be useful for further research on NLP for Algerian dialect and fill the gap of low resources.

algerian dialect
named entity recognition
arabic
dataset
low-resource language


Friendly Conditional Text Generator

Abstract: Our goal is to control text generation with more fine-grained conditions at lower computational cost than is possible with current alternatives; these conditions are attributes (i.e., multiple codes and free-text). As large-scale pre-trained language models (PLMs) offer excellent performance in free-form text generation, we explore efficient architectures and training schemes that can best leverage PLMs. Our framework, Friendly Conditional Text Generator (FCTG), introduces a multi-view attention (MVA) mechanism and two training tasks, Masked Attribute Modeling (MAM) and Attribute Linguistic Matching (ALM), to direct various PLMs via modalities between the text and its attributes. The motivation of FCTG is to map texts and attributes into a shared space, and bridge their modality gaps, as the texts and attributes reside in different regions of semantic space. To avoid catastrophic forgetting, modality-free embedded representations are learnt, and used to direct PLMs in this space, FCTG applies MAM to learn attribute representations, maps them in the same space as text through MVA, and optimizes their alignment in this space via ALM. Experiments on publicly available datasets show that FCTG outperforms baselines over higher level conditions at lower computation cost.

transformer
conditional text generation
neural language model
neural language generation


Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts

Abstract: Instead of mining coherent topics from a given text corpus in a completely unsupervised manner, seed-guided topic discovery methods leverage user-provided seed words to extract distinctive and coherent topics so that the mined topics can better cater to the user's interest. To model the semantic correlation between words and seeds for discovering topic-indicative terms, existing seed-guided approaches utilize different types of context signals, such as document-level word co-occurrences, sliding window-based local contexts, and generic linguistic knowledge brought by pre-trained language models. In this work, we analyze and show empirically that each type of context information has its value and limitation in modeling word semantics under seed guidance, but combining three types of contexts (i.e., word embeddings learned from local contexts, pre-trained language model representations obtained from general-domain training, and topic-indicative sentences retrieved based on seed information) allows them to complement each other for discovering quality topics. We propose an iterative framework, SeedTopicMine, which jointly learns from the three types of contexts and gradually fuses their context signals via an ensemble ranking process. Under various sets of seeds and on multiple datasets, SeedTopicMine consistently yields more coherent and accurate topics than existing seed-guided topic discovery approaches.


Making Pre-Trained Language Models End-to-End Few-Shot Learners with Contrastive Prompt Tuning

Abstract: Pre-trained Language Models (PLMs) have achieved remarkable performance for various language understanding tasks in IR systems, which require the fine-tuning process based on labeled training data. For low-resource scenarios, prompt-based learning for PLMs exploits prompts as task guidance and turns downstream tasks into masked language problems for effective few-shot fine-tuning. In most existing approaches, the high performance of prompt-based learning heavily relies on handcrafted prompts and verbalizers, which may limit the application of such approaches in real-world scenarios. To solve this issue, we present CP-Tuning, an end-to-end Contrastive Prompt Tuning framework for fine-tuning PLMs without any manual engineering of task-specific prompts and verbalizers. It is integrated with the task-invariant continuous prompt encoding technique with fully trainable prompt parameters. We further propose the pair-wise cost-sensitive contrastive learning procedure to optimize the model in order to achieve verbalizer-free class mapping and enhance the task-invariance of prompts. It explicitly learns to distinguish different classes and makes the decision boundary smoother by assigning different costs to easy and hard cases. Experiments over a variety of language understanding tasks and different PLMs show that CP-Tuning outperforms state-of-the-art methods.

deep contrastive learning
few-shot learning
pre-trained language models
prompt-based fine-tuning


Visual Matching is Enough for Scene Text Retrieval

Abstract: Given a text query, the task of scene text retrieval aims at searching and localizing all the text instances that are contained in an image gallery. The state-of-the-art method learns a cross-modal similarity between the query text and the detected text regions in natural images to facilitate retrieval. However, this cross-modal approach still cannot well bridge the heterogeneity gap between the text and image modalities. In this paper, we propose a new paradigm that converts the task into a single-modality retrieval problem. Unlike previous works that rely on character recognition or embedding, we directly leverage pictorial information by rendering query text into images to learn the glyph feature of each character, which can be utilized to capture the similarity between query and scene text images. With the extracted visual features, we devise a synthetic label image guided feature alignment mechanism that is robust to different scene text styles and layouts. The modules of glyph feature learning, text instance detection, and visual matching are jointly trained in an end-to-end framework. Experimental results show that our proposed paradigm achieves the best performance in multiple benchmark datasets. As a side product, our method can also be easily generalized to support text queries with unseen characters or languages in a zero-shot manner.

scene text retrieval
visual matching
multimodal retrieval


AGREE: Aligning Cross-Modal Entities for Image-Text Retrieval Upon Vision-Language Pre-Trained Models

Abstract: Image-text retrieval is a challenging cross-modal task that arouses much attention. While the traditional methods cannot break down the barriers between different modalities, Vision-Language Pre-trained (VLP) models greatly improve image-text retrieval performance based on massive image-text pairs. Nonetheless, the VLP-based methods are still prone to produce retrieval results that cannot be cross-modal aligned with entities. Recent efforts try to fix this problem at the pre-training stage, which is not only expensive but also unpractical due to the unavailable of full datasets. In this paper, we novelly propose a lightweight and practical approach to align cross-modal entities for image-text retrieval upon VLP models only at the fine-tuning and re-ranking stages. We employ external knowledge and tools to construct extra fine-grained image-text pairs, and then emphasize cross-modal entity alignment through contrastive learning and entity-level mask modeling in fine-tuning. Besides, two re-ranking strategies are proposed, including one specially designed for zero-shot scenarios. Extensive experiments with several VLP models on multiple Chinese and English datasets show that our approach achieves state-of-the-art results in nearly all settings.

image-text retrieval
vlp
vision-language pre-training


Can Pre-Trained Language Models Understand Chinese Humor?

Abstract: Humor understanding is an important and challenging research in natural language processing. As the popularity of pre-trained language models (PLMs), some recent work makes preliminary attempts to adopt PLMs for humor recognition and generation. However, these simple attempts do not substantially answer the question: whether PLMs are capable of humor understanding? This paper is the first work that systematically investigates the humor understanding ability of PLMs. For this purpose, a comprehensive framework with three evaluation steps and four evaluation tasks is designed. We also construct a comprehensive Chinese humor dataset, which can fully meet all the data requirements of the proposed evaluation framework. Our empirical study on the Chinese humor dataset yields some valuable observations, which are of great guiding value for future optimization of PLMs in humor understanding and generation.

pre-trained language models
chinese humor dataset
humor understanding
humor evaluation framework


An Effective Framework for Enhancing Query Answering in a Heterogeneous Data Lake

Abstract: There has been a growing interest in cross-source searching to gain rich knowledge in recent years. A data lake collects massive raw and heterogeneous data with different data schemas and query interfaces. Many real-life applications require query answering over the heterogeneous data lake, such as e-commerce, bioinformatics and healthcare. In this paper, we propose LakeAns that semantically integrates heterogeneous data schemas of the lake to enhance the semantics of query answers. To this end, we propose a novel framework to efficiently and effectively perform the cross-source searching. The framework exploits a reinforcement learning method to semantically integrate the data schemas and further create a global relational schema for the heterogeneous data. It then performs a query answering algorithm based on the global schema to find answers across multiple data sources. We conduct extensive experimental evaluations using real-life data to verify that our approach outperforms existing solutions in terms of effectiveness and efficiency.

relational schema
heterogeneous data lake
query answering


BeamQA: Multi-Hop Knowledge Graph Question Answering with Sequence-to-Sequence Prediction and Beam Search

Abstract: Knowledge Graph Question Answering (KGQA) is a task that aims to answer natural language queries by extracting facts from a knowledge graph. Current state-of-the-art techniques for KGQA rely on text-based information from graph entity and relations labels, as well as external textual corpora. By reasoning over multiple edges in the graph, these can accurately rank and return the most relevant entities. However, one of the limitations of these methods is that they cannot handle the inherent incompleteness of real-world knowledge graphs and may lead to inaccurate answers due to missing edges. To address this issue, recent advances in graph representation learning have led to the development of systems that can use link prediction techniques to handle missing edges probabilistically, allowing the system to reason with incomplete information. However, existing KGQA frameworks that use such techniques often depend on learning a transformation from the query representation to the graph embedding space, which requires access to a large training dataset. We present BeamQA, an approach that overcomes these limitations by combining a sequence-to-sequence prediction model with beam search execution in the embedding space. Our model uses a pre-trained large language model and synthetic question generation. Our experiments demonstrate the effectiveness of BeamQA when compared to other KGQA methods on two knowledge graph question-answering datasets.


Leader-Generator Net: Dividing Skill and Implicitness for Conquering FairytaleQA

Abstract: Machine reading comprehension requires systems to understand the given passage and answer questions. Previous methods mainly focus on the interaction between the question and passage. However, they ignore the deep exploration of cognitive elements behind questions, such as fine-grained reading skills (this paper focuses on narrative comprehension skills) and implicitness or explicitness of the question (whether the answer can be found in the passage). Grounded in prior literature on reading comprehension, the understanding of a question is a complex process where human beings need to understand the semantics of the question, use different reading skills for different questions, and then judge the implicitness of the question. To this end, a simple but effective Leader-Generator Network is proposed to explicitly separate and extract fine-grained reading skills and the implicitness or explicitness of the question. Specifically, the proposed skill leader accurately captures the semantic representation of fine-grained reading skills with contrastive learning. And the implicitness-aware pointer-generator adaptively extracts or generates the answer based on the implicitness or explicitness of the question. Furthermore, to validate the generalizability of the methodology, we annotate a new dataset named NarrativeQA 1.1. Experiments on the FairytaleQA and NarrativeQA 1.1 show that the proposed model achieves the state-of-the-art performance (about 5% gain on Rouge-L) on the question answering task. Our annotated data and code are available at https://github.com/pengwei-iie/Leader-Generator-Net.

machine reading comprehension
implicit or explicit question
reading skills
question answering


A Lightweight Constrained Generation Alternative for Query-Focused Summarization

Abstract: Query-focused summarization (QFS) aims to provide a summary of a document that satisfies information need of a given query and is useful in various IR applications, such as abstractive snippet generation. Current QFS approaches typically involve injecting additional information, e.g. query-answer relevance or fine-grained token-level interaction between a query and document, into a finetuned large language model. However, these approaches often require extra parameters & training, and generalize poorly to new dataset distributions. To mitigate this, we propose leveraging a recently developed constrained generation model Neurological Decoding (NLD) as an alternative to current QFS regimes which rely on additional sub-architectures and training. We first construct lexical constraints by identifying important tokens from the document using a lightweight gradient attribution model, then subsequently force the generated summary to satisfy these constraints by directly manipulating the final vocabulary likelihood. This lightweight approach requires no additional parameters or finetuning as it utilizes both an off-the-shelf neural retrieval model to construct the constraints and a standard generative language model to produce the QFS. We demonstrate the efficacy of this approach on two public QFS collections achieving near parity with the state-of-the-art model with substantially reduced complexity.

constrained generation
query-focused summarization


Mixup-Based Unified Framework to Overcome Gender Bias Resurgence

Abstract: Unwanted social biases are usually encoded in pretrained language models (PLMs). Recent efforts are devoted to mitigating intrinsic bias encoded in PLMs. However, the separate fine-tuning on applications is detrimental to intrinsic debiasing. A bias resurgence issue arises when fine-tuning the debiased PLMs on downstream tasks. To eliminate undesired stereotyped associations in PLMs during fine-tuning, we present a mixup-based framework Mix-Debias from a new unified perspective, which directly combines debiasing PLMs with fine-tuning applications. The key to Mix-Debias is applying mixup-based linear interpolation on counterfactually augmented downstream datasets, with expanded pairs from external corpora. Besides, we devised an alignment regularizer to ensure original augmented pairs and gender-balanced counterparts are spatially closer. Experimental results show that Mix-Debias can reduce biases in PLMs while maintaining a promising performance in applications.

social bias
gender debiasing
fairness
natural language processing


A Simple yet Effective Framework for Few-Shot Aspect-Based Sentiment Analysis

Abstract: The pre-training and fine-tuning paradigm has become the main-stream framework in the field of Aspect-Based Sentiment Analysis (ABSA). Although it has achieved sound performance in the domains containing enough fine-grained aspect-sentiment annotations, it is still challenging to conduct few-shot ABSA in domains where manual annotations are scarce. In this work, we argue that two kinds of gaps, i.e., domain gap and objective gap, hinder the transfer of knowledge from pre-training language models (PLMs) to ABSA tasks. To address this issue, we introduce a simple yet effective framework called FS-ABSA, which involves domain-adaptive pre-training and text-infilling fine-tuning. We approach the End-to-End ABSA task as a text-infilling problem and perform domain-adaptive pre-training with the text-infilling objective, narrowing the two gaps and consequently facilitating the knowledge transfer. Experiments show that the resulting model achieves more compelling performance than baselines under the few-shot setting while driving the state-of-the-art performance to a new level across datasets under the fully-supervised setting. Moreover, we apply our framework to two non-English low-resource languages to demonstrate its generality and effectiveness.

sentiment analysis
opinion mining
few-shot learning


Adversarial Meta Prompt Tuning for Open Compound Domain Adaptive Intent Detection

Abstract: Intent detection plays an essential role in dialogue systems. This paper takes the lead to study open compound domain adaptation (OCDA) for intent detection, which brings the advantage of improved generalization to unseen domains. OCDA for intent detection is indeed a more realistic domain adaptation setting, which learns an intent classifier from labeled source domains and adapts it to unlabeled compound target domains containing different intent classes with the source domains. At inference time, we test the intent classifier in open domains that contain previously unseen intent classes. To this end, we propose an Adversarial Meta Prompt Tuning method (called AMPT) for open compound domain adaptive intent detection. Concretely, we propose a meta prompt tuning method, which utilizes language prompts to elicit rich knowledge from large-scale pre-trained language models (PLMs) and automatically finds better prompt initialization that facilitates fast adaptation via meta learning. Furthermore, we leverage a domain adversarial training technique to acquire domain-invariant representations of diverse domains. By taking advantage of the collaborative effect of meta learning, prompt tuning, and adversarial training, we can learn an intent classifier that can effectively generalize to unseen open domains. Experimental results on two benchmark datasets (i.e., HWU64 and CLINC) show that our model can learn substantially better-generalized representations for unseen domains compared with strong competitors.

meta learning
unsupervised intent classification
adversarial training
prompt tuning


BioAug: Conditional Generation Based Data Augmentation for Low-Resource Biomedical NER

Abstract: Biomedical Named Entity Recognition (BioNER) is the fundamental task of identifying named entities from biomedical text. However, BioNER suffers from severe data scarcity and lacks high-quality labeled data due to the highly specialized and expert knowledge required for annotation. Though data augmentation has shown to be highly effective for low-resource NER in general, existing data augmentation techniques fail to produce factual and diverse augmentations for BioNER. In this paper, we present BioAug, a novel data augmentation framework for low-resource BioNER. BioAug, built on BART, is trained to solve a novel text reconstruction task based on selective masking and knowledge augmentation. Post training, we perform conditional generation and generate diverse augmentations conditioning BioAug on selectively corrupted text similar to the training stage. We demonstrate the effectiveness of BioAug on 5 benchmark BioNER datasets and show that BioAug outperforms all our baselines by a significant margin (1.5%-21.5% absolute improvement) and is able to generate augmentations that are both more factual and diverse. Code: https://github.com/Sreyan88/BioAug.

named entity recognition
information extraction
biomedical


DocGraphLM: Documental Graph Language Model for Information Extraction

Abstract: Advances in Visually Rich Document Understanding (VrDU) have enabled information extraction and question answering over documents with complex layouts. Two tropes of architectures have emerged-transformer-based models inspired by LLMs, and Graph Neural Networks. In this paper, we introduce DocGraphLM, a novel framework that combines pre-trained language models with graph semantics. To achieve this, we propose 1) a joint encoder architecture to represent documents, and 2) a novel link prediction approach to reconstruct document graphs. DocGraphLM predicts both directions and distances between nodes using a convergent joint loss function that prioritizes neighborhood restoration and downweighs distant node detection. Our experiments on three SotA datasets show consistent improvement on IE and QA tasks with the adoption of graph features. Moreover, we report that adopting the graph features accelerates convergence in the learning process druing training, despite being solely constructed through link prediction.

language model
visual document understanding
graph neural network
information extraction


Limitations of Open-Domain Question Answering Benchmarks for Document-Level Reasoning

Abstract: Many recent QA models retrieve answers from passages, rather than whole documents, due to the limitations of deep learning models with limited context size. However, this approach ignores important document-level cues that can be crucial in answering questions. This paper reviews three open-domain QA benchmarks from a document-level perspective and finds that they are biased towards passage-level information. Out of 17,000 assessed questions, 82 were identified as requiring document-level reasoning and could not be answered by passage-based models. Document-level retrieval (BM25) outperformed both dense and sparse passage-level retrieval on these questions, highlighting the need for more evaluation of models' ability to understand documents, an often-overlooked challenge in open-domain QA.

document-level reasoning
long context question answering
open-domain question answering


Multiple Topics Community Detection in Attributed Networks

Abstract: Since existing methods are often not effective to detect communities with multiple topics in attributed networks, we propose a method named SSAGCN via Autoencoder-style self-supervised learning. SSAGCN firstly designs an adaptive graph convolutional network (AGCN), which is treated as the encoder for fusing topology information and attribute information automatically, and then utilizes a dual decoder to simultaneously reconstruct network topology and attributes. By further introducing the modularity maximization and the joint optimization strategies, SSAGCN can detect communities with multiple topics in an end-to-end manner. Experimental results show that SSAGCN outperforms state-of-the-art approaches, and also can be used to conduct topic analysis well.

attributed networks
community detection
graph convolutional network
self-supervised learning
topic analysis


NC2T: Novel Curriculum Learning Approaches for Cross-Prompt Trait Scoring

Abstract: Automated essay scoring (AES) is a crucial research area with potential applications in education and beyond. However, recent studies have primarily focused on AES models that evaluate essays within a specific domain or using a holistic score, leaving a gap in research and resources for more generalized models capable of assessing essays with detailed items from multiple perspectives. As evaluating and scoring essays based on complex traits is costly and time-consuming, datasets for such AES evaluations are limited. To address these issues, we developed a cross-prompt trait scoring AES model and proposed a suitable curriculum learning (CL) design. By devising difficulty scores and introducing the key curriculum method, we demonstrated its effectiveness compared to existing CL strategies in natural language understanding tasks.

curriculum learning
cross-prompt
natural language processing
automated essay scoring
trait scoring


On Answer Position Bias in Transformers for Question Answering

Abstract: Extractive Transformer-based models for question answering (QA) are trained to predict the start and end position of the answer in a candidate paragraph. However, the true answer position can bias these models when its distribution in the training data is highly skewed. That is, models trained only with the answer at the beginning of the paragraph will perform poorly on test instances with the answer at the end. Many studies have focused on countering answer position bias but have yet to deepen our understanding of how such bias manifests in the main components of the Transformer. In this paper, we analyze the self-attention and embedding generation components of five Transformer-based models with different architectures and position embedding strategies. Our analysis shows that models tend to map position bias in their attention matrices, generating embeddings that correlate the answer and its biased position, ultimately compromising model generalization.

model understanding
reading comprehension
deep learning
natural language processing
question answering