One benefit of annotating code-switched corpora is that such discrepancies between languages become noticed.

if the same structure had been annotated differently for two different languages, say MSA and English, for which the treebanks had been developed independently, it now becomes clear when the two languages are juxtaposed in CS data that the annotation principles for at least one of the languages have to change in order to achieve the goal of having a unified annotation across languages.


We show here two examples from our corpus before we answer that question: 1. factor (the first factor, literally: the.factor the.first) 2. perspective camera (the perspective camera) In phrase 1, it is apparent that the speaker uses Arabic grammar rules, even though the noun is in English.

the noun as well as the adjective that is in concord with it are both in definite state and carry the Arabic definiteness morpheme.


In our experiments, we set the threshold to 0, PMI score lower than it will result in a segmentation.

for each dataset, we use PMI to perform unsupervised segmentation and collect the segmented words from it to build the n-gram lexicon N .


Table 8 reports the results in Fscore and OOV recall, which show a similar trend as that in Table 6, where WMSEG outperforms baselines for all five genres.

for genres with large domain variance (e.g., the ones with high OOV rates such as MZ and WEB), CWS is difficult, and its relatively low F-scores in Table 8 from baseline models confirm that.


Therefore, we test PMI and DLG in our model and compare them with the previous results from AV (see Table 6).

we use our best performing BERT-based model, i.e., BERT-CRF, with the n-gram lexicons constructed by the aforementioned three measures and run it on all benchmark datasets.


As shown in the figure, the performances of using the three measures are very similar, which indicates that WMSEG is able to robustly incorporate the wordhood information from various measures, despite that those measures focus on different aspects of n-grams when determining whether the ngrams should be treated as words.

consider that the lexicons produced by the three measures are rather different in their sizes (as shown in Table 4), the results in Figure 2 strongly demonstrate the effectiveness of our proposed approach in learning with a limited number of n-grams.


Therefore, we propose to train convolutional neural networks (CNNs) to generalize knowledge from the preliminary predictions given by the embedding space.

we first pre-train CNNs with soft predictions given by the cosine similarity between document embeddings and topic embeddings, and then adopt a self-training strategy to further refine the CNNs using their highconfident predictions on unlabeled documents.


In this paper, we enable the use of supervised techniques for unsupervised summarization.

we automatically generate a synthetic training dataset from a corpus of product reviews, and use this dataset to train a more powerful neural model with supervised learning.


The first study assessed the quality of the summaries using Best-Worst Scaling (BWS; Louviere et al., 2015), a less labor-intensive alternative to paired comparisons that has been shown to produce more reliable results than rating scales (Kiritchenko and Mohammad, 2017).

participants were shown the movie/business name, some basic background information, and a gold-standard summary.


Our approach is simple and agnostic to the training objective.

we introduce an auxiliary loss function to guide the self-attention heads in each layer towards a set of pre-determined patterns (e.g.


We demonstrate the effectiveness of our attention guidance loss through several empirical studies.

we 1) report convergence results on masked language modeling, 2) evaluate trained language models on downstream tasks, and 3) analyze the learned attention representations using probes.


Motivated by recent studies (Clark et al., 2019;Lin et al., 2019; which posit that individual attention heads can encode linguistic information, we analyze attention patterns in the self-attention heads of our models.

we search for heads that can individually perform coreference resolution.


The key idea is to transfer the burden of context understanding from modeling to learning by designing several auxiliary tasks, and leverage the auxiliary tasks as regularization in model estimation.

the model we use for response generation concatenates utterances in a conversation context as a long sequence, and only exploits one-layer self-attention in encoding and one-layer context attention in decoding.


In this paper, we address the task of end-to-end speech recognition and disfluency removal.

we investigate whether it is possible to train an ASR model end-to-end to directly map disfluent speech into fluent transcripts, without an intermediate disfluency detection step.


We design our experiments to determine if training an off-the-shelf LM architecture with our ILM framework can produce effective infilling models for a variety of datasets.

we train on three datasets of different sizes and semantics (details in Appendix A): short STORIES (Mostafazadeh et al., 2016), CS paper ABSTRACTS, and song LYRICS.


At both training and test time, examples are fed from left to right; anything to the left of a green target is available to the model as context when predicting the target.

lM only considers past context, and lM-Rev only considers future.


In Table 4, we report results for a mixture of granularities.

we run the same mask function we use for training (Appendix B) on our test data and evaluate PPL on the masked spans.


We extract the first word in a chat interaction, the bag-of-words representations (binary flags and tf-idf scores) of the chat interaction and features from sentiment lexica.

we extract flags indicating whether the turn has a positive, negative or neutral word in the list by Hamilton et al. (2016), the sentiment score of the chat interaction (summation of sentiment scores per token over number of tokens), and a flag indicating whether the interaction contains a negative word from the list by Hu and Liu (2004).


Our assumption concerning the need for the sentences used for extracting BERT representations to be a good semantic fit for adjectives in a scale, has not been confirmed by our evaluation.

differences between our methods when relying on carefully vs randomly selected sentences are minor.


On the other hand, the proposed measures uncover differences between the initially recalled and later retold stories that mirror the differences found between recalled and imagined stories (Table 2).

retold stories flow significantly more linearly than their initial counterparts in a pairwise comparison (Cohen's |d| = 0.17, p < 0.001; see Figure 3).


We crawled patients' public weblogs from TOBYO,5 a Japanese initiative of PatientsLikeMe, and reported the effects of drugs described therein.

we annotated spans describing drug reactions, related drug names, corresponding ICD-10 (the 10th edition of International Statistical Classification of Diseases and Related Health Problems created by the World Health Organisation) codes of reactions, and types of effects.


We regarded these sets of labels assigned to the span as a single annotation, i.e., concatenated each type of label as a single label.

we examined the agreement rate of an exact match of all possible annotation labels.


The Precision values are not that high as numerous additional terms were annotated by the pipeline.

for processes we received twice the number of spurious terms as correct matches.


Currently, there are still a large amount of missing terms.

for biological, chemical and physical processes the ontological coverage is low.


We first apply sandwich transformers to a different domain, while retaining the other architectural aspects and hyperparameter settings from Baevski and Auli (2019).

we use the Toronto Books Corpus (Zhu et al., 2015), which has previously been used to train GPT (Radford et al., 2018) and also BERT (Devlin et al., 2019) (combined with Wikipedia).


We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy.

we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels.


While these works ask valuable questions, they embrace model-driven approaches for manipulating the attention weights and thereafter evaluate the post-hoc explainability of the generated machine attention.

they overlook the human factor in the evaluation process -which should be integral in assessing the plausibility of the generated explanations (Riedl, 2019).


Further, it is well known that human attention is itself subjective: given the same text and task, human annotators may not always agree on which words are important.

one single human's attention should rarely be regarded as the ground-truth for attention.


Under this definition, attention scores can be seen as partial transparency.

they provide a look into the inner workings of a model, in that they produce an easily-understandable weighting of hidden states (Wiegreffe and Pinter, 2019).


Giving the networks many examples of each construction with a large variety of different content wordsthat is, large amounts of highly varied evidence about the meaning of the closed-class words-we asked during the test phase how fragile this knowledge is when transferred to new open-class words.

our probes combine novel open-class words with familiar closed-class words, to test whether the closed-class words are treated systematically by the network.


Instead of only using entities as nodes, for each question we construct a hierarchical graph to capture clues from sources with different levels of granularity.

four types of graph node are introduced: questions, paragraphs, sentences and entities (see Figure 2).


For graph propagation, we use Graph Attention Network (GAT)  (Velickovic et al., 2018) to perform message passing over the hierarchical graph.

gAT takes all the nodes as input, and updates node feature h i through its neighbors N i in the graph.


If the ground-truth answer does not exist among the entity nodes, the entity loss is zero.

the entity loss will only serve as a regularization term, and the final answer prediction will only rely on the answer span extraction module as follows.


We use HotpotQA dataset (Yang et al., 2018) for evaluation, a popular benchmark for multi-hop QA.

two sub-tasks are included in this dataset: (i) Answer prediction; and (ii) Supporting facts prediction.


Instead of constructing importance scores over the input texts on which the model makes predictions, such methods rank training examples by their influence on the model's prediction for the test input (Caruana et al., 1999;Koh and Liang, 2017;Card et al., 2019).

we are interested in the use of influence functions (Koh and Liang, 2017), which are in a sense inherently 'faithful' in that they reveal the training examples most responsible for particular predictions.


Both models are implemented on top of BERT encoders (Devlin et al., 2019).

we use BERT-Base, with the first 8 of the 12 layers frozen, only fine-tuning the last 4 transformer layers and the final projection layer.


To capture and quantify relationships between targets and context words, Octa uses a selective self-attention mechanism that handles implicit or missing targets.

octa involves two layers of attention mechanisms for, respectively, selective attention between targets and context words and attention over words based on aspects.


octa obtains a 2.6% and 1.6% accuracy improvement over BERT on SemEval-15 and SemEval-16, respectively.


We also report ROUGE (ROU) (Lin, 2004) and METEOR (MET) (Denkowski and Lavie, 2011) scores.

mETEOR also uses synonyms and stemmed forms of the words in candidate and reference sentences, and thus may be better at quantifying semantic similarities.


We hypothesize that mixing the sequence x with x changes the context for all tokens and injects too much noise, hence making learning the labels for the tokens challenging.

the relative distance between x and x in the manifold mapped by neural networks is further in sequence labeling than sentence classification (demonstrated in Figure 2), which is intuitively understandable as every data point in sentence classification is the pooling over all the tokens in one sentence while every token is a single data point in sequence labeling.


For instance, for the sentence "Rare Hendrix song draft sells for almost $17,000" and its paraphrased sentence "A rare Hendrix song design is selling for just under $17,000", although some words are different, the entity Hendrix keeps unchanged, and there are no extra entities added.

both contain one and only one entity (Hendrix) of the same type (Person).


Hammerton (2003) and Collobert et al. (2011) are among the first several studies to model sequence labeling using neural networks.

Hammerton (2003) encoded the input sequence using a unidirectional LSTM (hochreiter and Schmidhuber, 1997) while (Collobert et al., 2011) instead used a CNN with character level embedding to encode sentences.


Many applications, however, require trustworthy predictions that need to be not only accurate but also well calibrated.

a wellcalibrated model should produce reliable confident estimates for both in-distribution and out-ofdistribution (OOD) data: (1) For in-distribution data, a model should produce predictive probabilities close to the true likelihood for each class, i.e., confidence ≈ true likelihood.


As shown in Figure 1, we present the calibration of a BERT-MLP model for a text classification task on the 20NG dataset.

we train a TextCNN (Kim, 2014) and a BERT-MLP using 20NG 15 (the first 15 categories of 20NG) and then evaluate them on both in-distribution and OOD data.


We propose two new regularizers using pseudo samples both on and off the data manifold to mitigate data scarcity and prevent over-confident predictions.

our method imposes two types of regularization for better calibration during fine-tuning: (1) On-manifold regularization: We first generate on-manifold samples by interpolating the training data and their corresponding labels along the direction learned from hidden feature space; training over such augmented on-manifold data introduces a smoothness constraint within the data manifold to improve the model calibration for in-distribution data.


The fixed time to produce phone labels, which must be performed before translation, becomes a greater proportion of overall training time at lower-resource settings.

the phone end-to-end model offers similar training time reduction over the baseline to , where downsampling reduces sequence lengths by up to 60%, with unreduced sequence lengths through earlier convergence; this model offers a better trade-off between time and performance.


As far as we know, we are the first to do a large-scale study on actively training Transformer (Vaswani et al., 2017) for NMT.

with a human translation budget of only 20% of the original parallel corpus, we manage to surpass Transformer trained on the entire parallel corpus in three language pairs.


In this work, we propose holistic metrics that evaluate distinctive aspects of generated dialogues.

we consider (1) context coherence of a dialogue: the meaningfulness of a response within the context of prior query, (2) language fluency of generated responses: the quality of phrasing relative to a human native speaker, (3) response diversity of a set of generated sentences: the variety in meaning and word choice of responses, and (4) logical self-consistency: the logical consistency of utterances from a dialogue agent.


The current work instead proposes to measure response diversity utilizing augmented datasets with controlled paraphrasing, which allows for measuring diversity among top-ranked responses conditional on paraphrased queries and hence avoiding the tradeoff or dependency between diversity and quality.

for a given query, we slightly tilt the corresponding element in the query-response joint space along the query dimension (achieved by paraphrasing-augmentation) and then measure the entropy of high-quality responses in the neighbourhood of the targeted query.


The automatic metric based on augmented data has a stronger relation with that based on the baseline.

the metric based on CTG augmentation aligns with human judgments the closet.


In this paper, we present an evaluation framework to analyze social bias in NRE models.

we evaluate gender bias in English language predictions of a collection of popularly used and open source NRE models 1 (Lin et al., 2016;Wu et al., 2017;Liu et al., 2017;Feng et al., 2018).


Furthermore, since some data instances that are collected using distant supervision are noisy, we annotated the correctness of the test instances using Amazon Mechanical Turk annotations to perform a fair comparison.

we asked workers to determine whether or not a given sentence expressed a given relation.


Our unsupervised HMM+IBM1 alignment model significantly outperforms (with p < 0.001) all baselines.

it gets much higher precision scores compared to all baselines.


We go on with studying bilingual word embedding by quantitative analysis of the new word pairs, which are detected by searching bilingual words that are neighbors in the word embedding space, and evaluate them using the ground-truth bilingual dictionary.

we split the Muse dictionary of Chinese-to-English into standard training set and test set as in BLI (Artetxe et al., 2018a).


The synthetic data is generated from English Wikipedia 5 , English Gigaword (Parker et al., 2011) and Newscrawl 6 as the previous work (Ge et    2018a; Zhang et al., 2019;Kiyono et al., 2019;Grundkiewicz et al., 2019) did, using back translation and sentence corruption.

we train a transformer (base) model (Vaswani et al., 2017) for back translation using the training data of the restricted track in the BEA-19 shared task.


In this paper, we address the above issue by adapting a novel architecture for cross-lingual models called XLM (Conneau and Lample, 2019) to zero-shot abusive language detection, a task that has gained increasing importance given the recent surge in abusive online behavior and the need to develop reliable and efficient methods to detect it.

we evaluate two methods to pre-train bilingual language models, one similar to the original XLM masked model, and the other based on a novel hybrid emoji-based masked model.


Our basic architecture relies on the XLM approach described in (Conneau and Lample, 2019), specifically developed to learn joint multilingual representations enabling knowledge transfer across languages.

we borrow from XLM the method developed for unsupervised machine translation, that relies on the Masked Language Model (MLM) objective (Devlin et al., 2019) applied to multiple monolingual datasets as pretraining.


With FullyQT, we achieve higher BLEU scores than all other quantization methods for the Transformer on multiple translation tasks and avoid any loss in BLEU compared to full-precision.

out of 35 experiments, 8-bit quantization performed better than full-precision in 21 cases.


(1) In low (32) dimensions, we improve over Euclidean-based models by up to 6.1% in the mean reciprocical rank (MRR) metric.

we find that hierarchical relationships, such as WordNet's hypernym and member meronym, significantly benefit from hyperbolic space; we observe a 16% to 24% relative improvement versus Euclidean baselines.


Baseline numbers in high dimensions (Table 5) are taken from the original papers, while baseline numbers in the low-dimensional setting (Table 2) are computed using open-source implementations of each model.

we run hyper-parameter searches over the same parameters as the ones in the original papers to compute baseline numbers in the lowdimensional setting.


One advantage of using relation-specific transformations is that each relation can learn the right geometric operators based on the logical properties it has to satisfy.

we observe that in both low-and high-dimensional settings, attentionbased models can recover the performance of the best transformation on all datasets (Tables 2 and 5).


Furthermore, for relations that are neither symmetric nor anti-symmetric, we find that ATTH can outperform rotations and reflections, suggesting that combining multiple operators with attention can learn more expressive operators to model mixed logical patterns.

attentionbased transformations alleviate the need to conduct experiments with multiple geometric transformations by simply allowing the model to choose which one is best for a given relation.


Clearly, topics and tokens lie in the same embedding space, making it possible to measure the relationships between document-topics and summary-tokens via attention.

we choose the top-n topics according to θ¸, and consider these n topic embeddings as extra decoder inputs to guide the generation.


As a result, with the increase of the document length, the improvement produced by TA gets more evident.

the global semantics introduced by TA is indeed helpful to Transformer-based models on the summarization task, especially for long documents.


Negation is a mechanism that transforms a positive argument into its inverse rejection (Benamara et al., 2012).

in the task of affective analysis, negation plays a critical role as negation words can affect the word or sentence polarity causing the polarity to invert in many cases.


This work is funded by Natural Sciences and Engineering Research Council of Canada (NSERC) and the Big Data Research, Analytics, and Information Network (BRAIN) Alliance established by the Ontario Research Fund Research Excellence Program (ORF-RE).

we thank Majid Taghdimi from Questrade to provide us with the computing resources and help in the parallelization algorithm.


Due to the capability of capturing sequential information in an efficient manner, Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) networks are one of the most widely used models in text classification and generation problems.

bidirectional LSTM (Bi-LSTM) have shown impressive performance by capturing sequential information from the both directions in texts.


Recently pre-trained language models like OpenAI GPT (Radford et al., 2018), BERT (Devlin et al., 2018), ULM-FiT (Howard and Ruder, 2018) have made a breakthrough in a wide range of NLP tasks.

bERT and its variant models have shown superior performance in the GLUE benchmark for Natural Language Understanding (NLU) (Wang et al., 2019).


Subtitle sentences are then temporally aligned with the video frames.

for each frame v t , we pair it with two neighboring sentences based on the subtitle timestamps.


To encode the text input, we use BERT (Devlin et al., 2019), a transformer-based language model (Vaswani et al., 2017) that achieves state-ofthe-art performance on various NLP tasks.

we first fine-tune the BERT-base model using the masked language model and next sentence pre-diction objectives on the subtitles and QA pairs from TVQA+ train set.


To have better timestamps, we asked a set of Amazon Mechanical Turk (AMT) workers to refine the original timestamps.

we take the questions that have a localized span length of more than 10 seconds (41.33% of the questions) for refinement while leaving the rest unchanged.


The tools used for corpora annotation offer to calculate some interannotator agreement statistics.

the WebAnno we are using, makes it possible to calculate Kohen's Kappa, Fleiss' Kappa and Krippendorff's Alpha.


The shared model allows utilizing sense correlations across words and therefore allows to transfer common disambiguation rules learned from disambiguating annotation-rich words and applies the rules for improving the disambiguation of annotation-lean words that share a sense alternation pattern.

we build a single neural network model for WSD that derives sense representations and meanwhile enforces congruence between a word instance and its right sense, by using both lexical resources and sense-annotated data.


Our approach for word sense disambiguation measures the appropriateness of a word sense in a word context through a unified neural network that uses the same set of parameters for all the words and senses.

as shown in Figure 1, like the previous neural network-based methods (Raganato et al., 2017a; Luo et al., 2018b; Luo et al., 2018a), we use bidirectional LSTMs (bi-LSTMs) (Hochreiter and Schmidhuber, 1997) for encoding a word context, sense definitions and a prototype example sentence for each sense.


Since recognizing and evaluating systematic polysemy is beyond the scope of this work, we use VerbNet 2 (Kipper et al., 2006;Loper and Bird, 2002) to approximately identify verbs that have correlated word senses.

we first identify verbs that have five or less training instances for each of its senses.


When using more powerful contextualized ELMo and BERT word embeddings (the third and fourth section of Table 3 respectively), we observe similar trends but with larger performance gains overall.

compared to its equivalent word expert model Word-Specific Classifier BiLST M +Gloss , the Single Classifier BiLST M +Gloss achieved clear performance gains of 2.1% and 2.0% in F1score on the combined test set, when using ELMo embeddings and BERT embeddings respectively.


A number of researchers have recently questioned the necessity of increasingly complex neural network (NN) architectures.

several recent papers have shown that simpler, properly tuned models are at least competitive across several NLP tasks.


Second, because our model is, in essence, the primary building block of previous models, our results show that some previous papers propose needless complexity, and that gains from these previous complex neural architectures are quite modest.

the state of the art is achieved by careful tuning of simple and wellengineered models, not necessarily by adding more complexity to the model, echoing the sentiments of Lipton and Steinhardt (2018).


In addition, the TF-IDF measurement allows us to assign a weight to the words that compose the questions and their contexts according to their frequency in a given document and in all the documents of the corpus.

it is a measure that takes into account their importance, the rarity and therefore the discriminative function of words within the corpus.


This table shows very similar results overall as we see  for experiments 4, 6 and 8 for which we get a f-score near 0.62.

we can see the exact distribution of the predictions of our algorithm for experiment 8 with a confusion matrix (figure 3).


Our framework directly extends previous work on learning web-based tasks from the Mini Word-of-Bits framework using multiple demonstrations and exploration of the environment (Shi et al., 2017; Liu et al., 2018).

our DSL extends the constraint language defined in Liu et al. (2018)  to explore learning from explained demonstrations instead.


We choose these local priors to correspond to distributions that are uniform over assignments that are consistent, and has zero support otherwise, similar to previous work on pragmatic reasoning (Frank and Goodman, 2012; Monroe et al., 2017).

these distributions are proportional to indicator function over valid assignments of variables in each factor.


In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process.

we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts.


We propose two automatic methods to systematically improve the breadth and quality of the prompts used to query the existence of a relation (§ 3).

as shown in Figure 1, these are mining-based methods inspired by previous relation extraction methods (Ravichandran and Hovy, 2002), and paraphrasing-based methods that take a seed prompt (either manually created or automatically mined), and paraphrase it into several other semantically similar expressions.


Our second method for generating prompts is more targeted-it aims to improve lexical diversity while remaining relatively faithful to the original prompt.

we do so by performing paraphrasing over the original prompt into other semantically similar or identical expressions.


In this paper, we are interested in building a model of the student (question asker) in this scenario.

we investigate how to enable the student to reason pragmatically about which questions to ask to efficiently acquire knowledge, given only the topic T and the conversation history H. This setting of information-seeking conversations involves many interesting and challenging problems in natural language processing: Ã¢â‚¬Â¢ Quantifying textual information.


Once this classifier is trained, we can make use of the score it assigns different candidate questions to evaluate how specific each is to the current conversation history.

we select two kinds of negative questions: frequent questions from the training set (frequency>1) and random questions other than the observed one from the same conversation.


You will be ranking these questions on three different evaluation metrics, where ties are allowed for any metric (and encouraged if there isn't a clear signal setting candidate questions apart).

you will be evaluating these questions on their Overall Quality.


This paper explores data augmentation methods for training Neural Machine Translation to make use of similar translations, in a comparable way a human translator employs fuzzy matches.

we show how we can simply feed the neural model with information on both source and target sides of the fuzzy matches, we also extend the similarity to include semantically related translations retrieved using distributed sentence representations.


All types of matching indicate their suitability showing accuracy gains.

for fuzzy matching, which seems to be the best for our task.


With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation.

the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair.


Three sentiment knowledge prediction objectives are jointly optimized during pre-training so as to embed sentiment words, polarity, aspect-sentiment pairs into the representation.

the pair prediction is converted into multi-label classification to capture the dependency between aspect and sentiment.


For every KG, we can use the average out-degree D out avg of each entity (node) to define its sparsity.

if D out avg of a KG is larger than a threshold, we can say it is a dense or normal KG, otherwise, it is a sparse KG.


Therefore, the state of the t-th hop can be defined as s t = (r q , e t , h t ), where h t is the representation of the historical path.

we use an LSTM to encode the historical path information, h t is the output of LSTM at the t-th step.


Inspired by the above phenomenon, we propose a new strategy named dynamic anticipation, which introduces the prediction information of the embedding-based models into the multi-hop reasoning models to guide the model learning.

Specifically, for a triple query (es, rq, ?), we use the pretrained KGE models to get the probability vector of all entities being the tail entity.


Moreover, we can learn that using either strategy individually will enable our model to achieve better results than the baseline model.

the model using the DC strategy alone performs better than the model using the DA strategy alone, which is predictable, since the DA strategy only allows the agent to make a correct choice, and will not substantially alleviate the sparsity of KGs.


It then defines a score function f (e s , r q , e t ) with embeddings to measure the correct probability of each triple.

Specifically, most KGE models can be divided into three categories (Wang et al., 2017): (1) Translation-based models (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015; Sun et al., 2018) formalize the relation as a translation from a head entity to a tail entity, and often use distance-based score functions derived from these translation operations.


Our goal here is not to find the best possible way to do the post-ranking, but only to show that gains are possible.

running the preranker over a larger candidate list is not enough; we find that it is better to see what edits are made for multiple candidates before making the final decision.


Following literature in open-domain QA, we evaluate our approach on three datasets, WebQuestions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).

webQuestions (wQ) and Natural Questions (NQ) consist of short, noisy questions from web queries, in line with the motivation of our work.


Although our method is generic, it is particularly well suited to this task.

we show that FLOP can dynamically learn the embedding dimensions of different word clusters, effectively extending the idea of adaptive embeddings and softmax (Grave et al., 2017;Baevski and Auli, 2019).


The choice of binary variables z can be regulated by some prior distribution and optimized given the training data.

let q j (z) be the density function of the learnable prior of z j .


We evaluate the performance of our method on language modeling and BERT fine-tuning.


We use a hidden size of 3056 and set the initial factorization dimension r of the parameter matrices to 512.

we replace each weight matrix W in SRU using an explicit factorization PQ with an inner dimension of 512.


Since a precise list of such token types is hard to define, we thus estimate the grounding ratio based on existing grounded language corpora.

we consider a token type with more than 100 occurrences in MS COCO (after removing all stop words) as visually-grounded.


In Table 1, we compare two ablated instances of ConceptBert with its complete form.

we validate the importance of incorporating the external knowledge into VQA pipelines on top of the vision and language embeddings.


We illustrate some qualitative results of Concept-Bert complete form CVL by comparing it with the VL model.

we aim at illustrating the advantage of adding (i) the external knowledge extracted from the ConceptNet knowledge graph, and (ii) concept-vision-language embedding representations.


These models fine-tune BERT (Devlin et al., 2019) for DocRED.

two-Phase BERt  is the best reported model.


This contextual model generates a representation of each word based on the other words in the sentence.

the word vector BERT outputs for a word is dependent on the surrounding context in which it occurs.


Among these four models, DNN_2++ and BERT received the best or the second best F1 scores on all POS tags.

dNN_2++ achieved the highest F1 scores on 27 POS tags.


The vast majority of previous work (Wan et al., 2007;Wan, 2008;Wan and Xiao, 2009;Wan and Zhang, 2014) creates summaries by ranking textual segments (usually sentences) according to their relationship (e.g., similarity) to other segments and their relevance to the query.

relevance and evidence estimation are subservient to estimating the centrality of a segment (e.g., with a graph-based model).


For each document cluster, LEXRANK builds a graph G = (V, E) with nodes V corresponding to sentences and (undirected) edges E whose weights are computed based on similarity.

matrix E represents edge weights where each element E i,j corresponds to the transition probability from vertex i to vertex j.


In order to fairly compare against these systems, for every bridging anaphor a, we first map all top 20 span predictions of our system BARQA to the gold/system mentions, then we choose the gold/system mention with the highest confidence score as the predicted antecedent.

we map a predicted span s to a mention m if they share the same head and s is part of m (m is created by removing all postmodifiers from m).


In this work, we focus on the task of event detection (ED) to identify event trigger words for the cybersecurity domain.

to facilitate the future research, we introduce a new dataset for this problem, characterizing the manual annotation for 30 important cybersecurity event types and a large dataset size to develop deep learning models.


In this work, we examine Information Extraction technologies (IE) in Natural Language Processing (NLP) as a promising candidate for the knowledge extraction task from cybersecurity text.

we focus on Event Detection (ED), an important task in IE that seeks to identify trigger words of specified types of events in text (Ahn, 2006;Ji and Grishman, 2008).


In addition, we find that sentences mentioning some events in CySecED often contain at least two event trigger words for the event types.

cybersecurity events in CySecED tend to cooccur with each other in the sentences, suggesting potential inter-dependencies between events.


These dependencies can be exploited to further improve the ED performance for cybersecurity domain in the future research (Li et al., 2013;Nguyen et al., 2016a;Nguyen and Nguyen, 2019).

among the sentences in CySecED, 45.4% of the sentences do not contain any event triggers, 50.0% of the sentences host at least two event triggers, and only 4.6% of the sentences involve a single event trigger.


As shown in Fig. 1, there exist shared and private information between target extraction and target classification.

the semantic and syntactic information are essential for both tasks, so they are shared information.


In this paper, we propose a novel model of Shared-Private Representation Model (SPRM) shown in Fig. 2, which encodes the shared and private information of the target extraction sub-task and the target classification sub-task effectively at a lower cost.

a Shared BERT Network is designed to encode as much shared information of both sub-tasks as possible, and two Private BiLSTMs are introduced to get the supplementary private representations for each task with fewer parameters than BERT.


Although we have corrected spelling errors that affect more than one token, these entries have been removed from the training data set.

the model has been trained exclusively on unigrams.


The goal of our table semantic segmentation model is to classify cells into categories.

instead of performing structural segmentation where one tries to distinguish between captions, headers and rows in a stream of text (Pinto et al., 2003) we focus on semantic segmentation (i.e., assigning roles to each cell) of tables.


To tackle this problem effectively we define subtasks that take us from paper to results.

we introduce the AXCELL pipeline that consists of the following subtasks: (i) table type classification, identifying whether a table in a paper has relevant results; (ii) table segmentation, segmenting and classifying table cells according to whether they hold metrics, datasets, models, etc.


We take inspiration from unsupervised grammar induction (UGI) techniques originally proposed for natural language.

we use them to investigate if the languages that emerge in the typical setup of referential games exhibit interesting syntactic structure, and to what extent this depends on the maximum message length and number of symbols that the agents are allowed to use.


Previous work that focused on the analysis of emergent languages has primarily concentrated on semantics-based analysis.

they considered whether agents transmit information about categories or objects, or instead communicate using low-level feature information (Steels, 2010;Lazaridou et al., 2017;Bouchacourt and Baroni, 2018; Mihai and Hare, 2019, i.a.).


We first qualitatively assess the extent to which CCL-BMM and DIORA-BMM are able to infer the correct grammars for the structured baseline languages described in the previous section.

we consider if the induced grammars reflect the correct word classes defined by the preterminals, and if they capture the simple hierarchy defined on top of these word-classes.


A natural question, then, is: in the absence of argumentannotated training data, is argument mining doomed to fail?

are argument-annotated training data as indispensable as they seem in enabling state-of-the-art argument mining systems to achieve their current level of performance?


First, our competitive results call for a reexamination of existing supervised approaches to argument mining.

future work should seek to understand what caused their performances to be rivaled by an unsupervised system.


Similarly, they applied their learned RI classifier to classify only the relation between two gold argument components.

they simplified both tasks by avoiding the challenging task of identifying the locations of argument components.


Further, we run additional experiments for MT systems that are known to have low correlation with automatic metrics calculated with standard references.

we investigated MT systems augmented with either back-translation or automatic post-editing (APE).


Interestingly, previous work (Bojar et al., 2016a) has shown that a higher correlation can be achieved when comparing similar systems than when comparing different types of systems, e.g. phrase-based vs neural vs rulebased.

rule-based systems can be penalized as they produce less common translations, even when such translations are fluent and adequate.


We suspect that this may at least in part be an artifact of the rating methodology.

translations whose word order matches that of the source (i.e. translationese) are easier to rate than translations that use very different sentence structures and phrasing than the source sentence.


Comparing the two paraphrased references, we see that HQ(P) shows higher correlation for chrF and Yisi when compared to WMT.p.

yisi (which is based on word embeddings) seems to benefit from the higher accuracy of the reference translation.


We run experiments with a variety of models that have been shown that their actual quality scores have low correlation with automatic metrics.

we focus on backtranslation (Sennrich et al., 2016) and Automatic Post Editing (APE, Freitag et al. (2019)) augmented systems trained on WMT 2014 English→German.


Table 4 shows that our unsupervised model is competitive with recent models using full set of Switchboard training data.

our unsupervised model even achieves slightly improvement over the supervised methods without using contextualized word embeddings, demonstrating the effectiveness of our unsupervised model.


Despite intensive research efforts to remove noise automatically (Han and Sun, 2016;Hoffmann et al., 2011;Jiang et al., 2016;Surdeanu et al., 2012;Zhou et al., 2016), this problem has yet to be solved.

some groups (Angeli et al., 2014;Liu et al., 2016;Pershina et al., 2014;Zhang et al., 2012) have attempted to boost the performance of relation classifiers through crowdsourcingbased training data collection for relation extraction and using them along with training data for DS learning.


We first present a machine learning system that processes negation in Spanish.

we focus on two tasks: i) negation cue detection and ii) scope identification.


The test set has a total of 836 negation cues.

there are 83 different negation cues, of which 15 are simple cues, 19 are continuous cues and 49 are discontinuous cues.


We use the CRF implementation in CRFsuite (Okazaki, 2007) and scikit-learn (Pedregosa et al., 2011) with the L-BFGS training algorithm (default) and Elastic Net (L1 + L2) regularization.

we train two classifiers: the first one takes as input a sentence and predicts the negation cue BIO labels, and the second one takes as input a sentence along with information about the predicted cues and predicts the scope BIO labels.


In recent years, significant progress was made in the field of argument mining, automatic identification and extraction of argumentative structures in text (Lawrence and Reed, 2020).

several works focused on topic-related argument mining from the Web or other massive corpora(Levy et al., 2017, 2018; Wachsmuth et al., 2017; Stab et al., 2018a,b; Ein-Dor et al., 2020).


This example highlights the most problematic linguistic challenge that we encountered in the development of the ACQDIV database and corpus aggregation pipeline.

the same morphological phenomena may be encoded differently per corpus.


During the cleaning procedure, we also undertake a routine for speaker unification.

we identify the same speakers across different recording sessions, so that we can populate a UNIQUESPEAKERS table in our database.


In recent years, Deep Learning methods have become very popular in Natural Language Processing (NLP), e.g., they reach high performances by relying on very simple input representations (for example, in (Kim, 2014;Goldberg, 2016;Kim et al., 2016)).

transformer-based architectures, e.g., BERt (Devlin et al., 2019), provide representations of their inputs as a result of a pre-training stage.


In this paper, we extend the BERT training with unlabeled data in a generative adversarial setting.

we enrich the BERT fine-tuning process with an SS-GAN perspective, in the so-called GAN-BERT 1 model.


In particular, we enrich the BERT fine-tuning process with an SS-GAN perspective, in the so-called GAN-BERT 1 model.

a generator produces "fake" examples resembling the data distribution, while BERT is used as a discriminator.


As suggested in (Salimans et al., 2016), G should generate data approximating the statistics of real data as much as possible.

the average example generated in a batch by G should be similar to the real prototypical one.


Both automatic evaluation and human evaluation show that MART generates more satisfying results than previous LSTM-based approaches (Xiong et al., 2018;Zhou et al., 2019;Zhang et al., 2018) and transformer-based approaches (Zhou et al., 2018;.

mART can generate more coherent (e.g., coreference and order), less redundant paragraphs without losing paragraph accuracy (visual relevance).


To resolve this issue, Transformer-XL  introduces the idea of recurrence to the transformer language model.

the modeling of a new language segment in Transformer-XL is conditioned on hidden states from previous language segments.


Compared to the two baselines, MART produces more coherent and less redundant paragraphs.

we noticed that vanilla transformer often uses incoherent pronouns/person mentions, while MART and Transformer-XL is able to use suitable pronouns/person mentions across the sentences and thus improve the coherence of the paragraph.


Experimental results on two standard datasets show that MART has better overall performance than the baseline methods.

mART can generate more coherent, less redundant paragraphs without any degradation in relevance.


Various metrics have been proposed to quantify biases in model predictions.

several of them evaluate disparity in model performance between protected groups and advantaged groups in the test corpus.


However, both K-Means and LOGAN successfully detect strong local group bias.

lOGAN identifies a local region that the model has difficulties in making correct predictions for female group.


DecOp is composed of both truthful and deceptive first-person opinions about five different domains, is compiled in two different languages and has been developed for allowing several kinds of classification experiments.

beyond the more usual withindomain classification task, DecOp allows cross-domain, author-based and cross-language classification tasks.


Furthermore, unlike prior work employing either classification (Gong and Zhang, 2016) or generation models (Wang et al., 2019a), we propose a unified framework to couple the advantages of keyphrase classification and generation.

in addition to the joint training of both modules, we further extend the copy mechanism (See et al., 2017) to explicitly aggregate classification outputs together with tokens from the source input.


These methods undergo labor-intensive feature engineering and hence lead to the growing popularity of adopting data-driven neural networks.

for social media keyphrase prediction, most efforts are based on sequence tagging style extraction (Zhang et al., 2016, 2018) or classification from a predefined candidate list (Gong and Zhang, 2016; Zhang et al., 2017), which are however unable to produce keyphrases absent in the post or the fixed list


Following Cai et al. (2019), we first train an attribute predictor based on the Resnet-152 (He et al., 2016) features on MS-COCO 2014 caption dataset (Lin et al., 2014).

we extract noun and adjective tokens from the image captions as the attribute labels.


Since there are no publicly available datasets for multi-modal keyphrase annotation, we contribute a new dataset with social media posts from Twitter.

we employ the Twitter advanced search API 4 to query English tweets that contain both images and hashtags from January to June 2019.


We promote output diversity by training f para on an aggressivelyfiltered subset of PARANMT-50M (Wieting and Gimpel, 2018), a large corpus of backtranslated text.

we apply three filters: (1) removing sentence pairs with more than 50% trigram or unigram overlap to maximize lexical diversity and discourage copying; (2) removing pairs with lower than 50% reordering of shared words, measured by Kendall's tau (Kendall, 1938), to promote syntactic diversity; and (3) removing pairs with low semantic similarity, measured by the SIM model from Wieting et al. (2019)


We focus exclusively on semantics-preserving style transfer tasks, which means that we do not evaluate on attribute transfer datasets such as sentiment, gender, and political transfer.

we use two standard benchmark datasets for Shakespeare author imitation and formality transfer to compare STRAP against prior work.


The models are finetuned using a small learning rate of 5e-5 and converge to a good solution fairly quickly as noticed by recent work (Li et al., 2020;Kaplan et al., 2020).

each experiment completed within a day of training on a single GPU, and many experiments with small datasets took a lot less time.


Evaluating with adjusted perplexity showed that a background model interpolated with a user-only model performs well when larger amounts of user-specific text are available.

neural background interpolated with either user-only model outperforms all other models when using 10k tokens or more of the user's text.


Encouraged by the results on pre-training for dialogue generation and knowledge-grounded dialogue generation, and motivated by the problems in both sides, we consider bringing the two together in this work.

we propose knowledgegrounded dialogue generation with pre-trained language models in order to endow a generative model with both rich knowledge and good generalization ability 1 .


Therefore, we propose an unsupervised approach where learning of knowledge selection and fine-tuning of response generation are jointly conducted with unlabeled dialogues.

we build the knowledge selection module on the basis of BERT, and formalize knowledge selection as a sequence prediction process, by which the model can take advantage of the pre-training techniques and dynamically determine the relevant knowledge for a given context.


We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance.

we improve the performance over the state-of-the-art by 0.5 to 1.0% on average at 50% compression ratio.


The design of residual connection can provide us with a promising way to find identity mappings for the Transformer model.

residual connection (He et al., 2016b) can be formalized as H(x) = F(x) + x, where F could be either MHA or FFN and H is the sub-layer output.


While sociolinguistics and social psychology have long pointed to how people shape their language to convey social information (Labov, 1972; Brown and Levinson, 1978; Clark and Schunk, 1980; Weber, 2008; Locher and Graham, 2010), only recently, have computational models focused on making this information explicit (Choi et al., 2012; Danescu-Niculescu-Mizil et al., 2013; Bak et al., 2014).

works on social status and power have shown how individuals use lexical cues and linguistic strategies like accommodation to express their perceived status in relation to others (Danescu-Niculescu-Mizil et al., 2013;Prabhakaran et al., 2014).


Here, we connect interpersonal pragmatics to the language of intimacy, showing how individuals perform pragmatic acts in their questions to mitigates risk as intimacy increases, much like how politeness is employed to save face (Brown et al., 1987).

we examine pragmatic choices in questions around (i) the speaker's certainty, expressed in hedges from Hyland (2005) (ii) the speaker's belief of the social distance, expressed in swearing.


The behavior of different part-of-speech classes can also explain some of the differences observed across languages.

as can be seen in Table 1, most of the languages that show a clear preference for UD -Chinese, Hebrew, Hindi, Italian and Japanese -are all characterized by a high proportion of adpositions.


To reduce the expenses for evaluating multidocument summaries, we investigate unsupervised evaluation methods, which require neither human annotations nor reference summaries.

we focus on evaluating the relevance (Peyrard, 2019) of multi-document summaries, i.e. measuring how much salient information from the source documents is covered by the summaries.


Our work thus leverages this computational efficiency of TMs for efficient and scalable fine-tuning for BERT in document classification.

our contributions(1) Complementary Fine-tuning: We present a novel framework: TopicBERT, i.e., topic-aware BERT that leverages the advantages of both neural network-based TM and Transformer-based BERT to achieve an improved document-level understanding.


We run a second annotation round where two annotators with linguistic background re-validate the instances that were flagged as special cases by translators during the first round (more concretely, the possible translation shifts).

annotators in this phase decide, for each special case, if the annotated label should be deleted or corrected.


Instead, we take a novel approach and rely on the shared space of mBERT embeddings (Devlin et al., 2019).

we compute pair-wise cosine similarity between source and target tokens and emulate word-alignments according to this measure 16 .


This is the bias of unidirectional decoders and we replace them with simpler word selectors.

we only ask the model to select original neighbors for each word that loses its local word order information.


If we adopt pre-trained encoders, as in the second and third group, the performance clearly improves for all training sizes.

in the low-resource (1k) settings, the performance drops from the 10k settings are much smaller than those in the first group.


Analyzing the surrogates, we observed that most of the OOV tokens were punctuation or uncommon ideographs, which we expected to, and confirmed to have little effect in the downstream task performance.

we attribute the negligible gains to the nature of the task itself, as it is a news article classification task.


To this end, we propose a novel Diversified Multiple Instance Learning Network (D-MILN), which is able to achieve aspect-level sentiment classification with only document-level weak supervision.

we connect aspect-level and document-level sentiment by formulating this problem as multiple instance learning, providing a way to learn aspect-level classifier from the back propagation of document-level supervision.


In our experiments, we consider all utterances from the last plot update to the current one from both interlocutors.

starting from the last plot update, the Describer's instruction and all the clarification questions and responses are concatenated and provided as the dialog history.


Our model leverages a multitasking framework to train both a summarization model on headline-article pairs, and a Denoising Autoencoder (DAE) on a style corpus.

based on the transformer architecture (Vaswani et al., 2017), we use the style-dependent layer normalization and the style-guided encoder-attention to disentangle the language style factors from the text.


We randomly sampled 50 news abstracts from the test set and asked three native-speaker annotators for evaluation to score the generated headlines.

we conduct two tasks to evaluate on four criteria: (1) relevance, (2) attractiveness, (3) language fluency, and (4) style strength.


We progressively expand TitleStylist to include all three target styles (humor, romance, and clickbait) to demonstrate the flexibility of our model.

we simultaneously trained the summarization task on the headlines data and the DAE task on the three target style corpora.


Nonetheless, words and mathematical terms are interdependent in the context of mathematical discourse.

the proposed model presents a significantly slower decline (25%), showing better scalability properties in the context of the premise selection problem.


The contribution of this work is an interactive visualisation tool, called Starmap 1 to help TTS developers navigate through thousands of audio files of utterances originating from a target domain, and select evaluation materials that are varied, and at the same time representative of the most important communicative acts, pragmatic and expressive functions the speaker needs to perform in that domain.

a significant majority of listeners preferred the TCC voice for the solo-host podcast domain over the TGD voice (p 0.001).


In this section, we present the extended version of our sensitivity study (§4.3).

we keep the estimated natural error distribution for evaluation and use a simplified synthetic error model for training.


First, news articles on news websites update very quickly.

we would like to thank Xi Chen, Virl Hill, Jesse Pannoni, Sally Salas and Ting Cai in the Microsoft News team for their great support in releasing this dataset and for their great help in preparing the data.


Another interesting finding is that the combination of LSTM and attention can achieve the best performance.

we hope MIND can serve as a benchmark dataset for news recommendation and facilitate the research in this area.


Based on these datasets, many well-known recommendation methods have been developed.

we compare 6 methods, including simple average of the representations of previously clicked news (Average), attention mechanism used in (wu et al., 2019a) (Attention), candidate-aware attention used in (wang et al., 2018) (Candidate-Att), gated recurrent unit used in (Okura et al., 2017) (GRU), long-and short-term user representation used in ) (LSTUR) and multi-head self-attention used in (wu et al., 2019c) (Self-Att).


The entity linker used in the originally proposed framework (Hulpus, et al., 2019) is not freely available

both endpoints expect a HTTP POST message with a json payload and the header 'Content-Type: application/json'.


To actively engage readers in the text they are reading and maintain their interest in the topic, texts needs to be on an adequate level for the reader in terms of its lexical, syntactic, as well as its conceptual complexity (McNamara et al., 2006).

in the current implementation, we use DBpedia Spotlight (Daiber et al., 2013) which is freely available for several languages (DBpediaSpotlight, 2019).


This architecture decreases the implementation costs significantly.

as a further extension, we are constructing a ranking system where players can compare their score against others, to arouse players' competitive motivation.


An overview of our platform is shown in Figure 2.

the dialogue act selection is obligatory, i.e. the speaker can not make their utterance without the dialogue act selection


The neural models see a 10% gain in accuracy over the template-sensitive majority, indicating that the models are learning the range of durations for different entities.

we compare this model to a model that additionally includes a fixed effect for the entailment label associated with the pair by our recasting.


We test existing models trained on MNLI on our datasets and find that a generic NLI model is not able to capture temporal reasoning.

we use three types of models: (i) neural bag of words (NBOW; Iyyer et al., 2015) (ii) InferSent (Conneau et al., 2017), and (iii) RoBERTa (Liu et al., 2019).


When we focused on each combination of the embedding and distribution models, we found that the highest and second highest scores were achieved by OffSet with GloVe and GMM with SGNS, respectively.

in this study, we seek to understand how these vectors are distributed in the pre-trained word vector space without using contextual or lexical information.


The composed lexicon contains sentiment values in the range  [−1, 1].

to create a balanced dataset that covers a large variety of topics, we randomly selected 5 pro and 5 con arguments of each debate.


By comparison, the majority vote matches 79.30% of the original stance labels.

most of the targets are found by word similarity and the fewest by the acronym.


The third set is the entire αNLI development set introduced in Bhagavatula et al. (2020).

this indicates that naturally occurring examples in ChaosNLI-α are either highly certain or uncertain among human judgements.


To be specific, the estimated collective human performance gives JSD and KL scores far below 0.1 on all three sets.

figure 6 and 7 show the screenshots for NLI and αNLI collection, respectively.


column summarizes the difference in performance for each of the classes.

in this paper we describe the semiautomatic process of building a sentiment lexicon in the financial/economic domain, which assumes the existence of a general-language sentiment lexicon.


Johnson had also met AstraZeneca and GSK, whose headquarters are in Britain, while O’Brien had been to the Paris headquarters of the French firm, SanofiAventis.

such resources can be automatically created (Lu et al., 2011; Tai and Kao, 2013) or crafted manually.


The ideal method for displaying information to the Deaf community would be through their sign language.

hamNoSys is an alphabetic system describing signs mostly on a phonetic level.


SiGML is an XML-compliant machine-readable format that enables avatars to animate HamNoSys symbols.

hence, it is not of easy access to the Deaf community.


the proposed architecture, drawing on contemporary cognitive science, aims to address key limitations of current NLU systems through mental simulation and grounded metaphoric inference.


The current deep-learning paradigm is a statistical pattern-recognition approach predominantly applied to relatively narrow task-specific prediction.

this ambiguity is challenging for non-neural (symbolic) simulationbased approaches.


Speaker A: hi , how are you doing today ?

the findings are similar to the pairwise ACUtE-Eval results in the main paper.


To show such grounding helps, we report results with and without grounding on those tasks in Table 4, reporting perplexity.

what we feel entitled to is a better world .


Specific tasks for which the corpora could be used to evaluate the impact of processing negation.

it introduced a new element for the annotation of negation, the focus.


Overall negation processing tasks for which the corpora could be used, by language.

the predicate NOt was used to annotate any explicit statements of the non-existence of a relationship.


Thus, the average number of sentences with negation in English texts is 17.94% and in Japanese 16.59%, whereas for Spanish it is 29.13%, for Italian 21.55%, and for Chinese 26.82%.

an annotation guideline explaining the process and each of the contextual properties was provided, but it is not available.


The ConanDoyle-neg (Morante and Daelemans 2012) is a corpus of Conan Doyle stories annotated with negation cues and their scopes, as well as the event or property that is negated.

it is noteworthy that in some of the corpora (BioInfer, Genia Event, Product Review, EMC Dutch, IxaMed-GS, and German negation and speculation corpus) negation cues have not been annotated, despite the fact that the cue is the element that denotes the presence of negation in a sentence and the one to which the rest of the elements (scope, event, and focus) are connected.


Several shared tasks have addressed negation processing for English: the BioNLP’09
Shared Task 3 (Kim et al. 2009), the i2b2 NLP Challenge (Uzuner et al. 2011), the *SEM
2012 Shared Task (Morante and Blanco 2012), and the ShARe/CLEF eHealth Evaluation
Lab 2014 Task 2 (Mowery et al. 2014).

the tweet corpus has 301 sentences and 59 were annotated as negated (19.60%).


Words derived (by affixation and reduplication) are listed as sub-entries under the basewords.

the main goal of our project was to produce a high quality, human-curated resource.


The passive voice forms are not regarded as variants in the Old Javanese Wordnet.

an example of this spreadsheet is shown in Table 4.


Practically, sentiments are represented into a 2D vector, whose dimensions and values correspond to polarities (positive or negative) and corresponding intensities respectively.

under the condition of processing extremely unstable inputs (big drifts), many of the values will be very close to 1, which decreases system performance.


EAE's average accuracy is similar to BERTlarge.

table 1 shows the results for all our models.


Second, we introduce two simple yet effective scheduling strategies, namely the dynamic temperature-based sampling and dynamic noising ratio strategy.

furthermore, we show the effectiveness of MTL over pre-training approaches for both NMT and cross-lingual transfer learning NLU tasks; the proposed approach outperforms massive scale models trained on single task.


Back translation (BT) (Sennrich et al., 2016), which leverages a target-to-source model to translate the target-side monolingual data into source language and generate pseudo bitext, has been one of the most effective approaches in NMT.

back translation (Sennrich et al., 2016) utilizes the target-side monolingual data.


The neighboring text tokens are embedded using a word embedding table.

to most prior work on information extraction (Sarawagi, 2008), templatic documents do not contain much prose.


Table 1 lists the fitting results for all the languages 8 in the Europarl corpus.

the third parameter provides a depiction of the rigidness of different coefficients of proportionality.


These datasets are previously used as benchmarks for multi-document summarization competitions.

importantly, and distinguishing our work from earlier literature, we make a first attempt to generate self-contained highlights, drawing on the successes of deep contextualized representations and their extraordinary ability of encoding syntactic structure (Clark et al., 2019;Hewitt and Manning, 2019).


When the class is span, the start and end positions of slot values are obtained in the dialogue context.

this hidden vector is then mapped to distribution over the vocabulary V and over the dialogue history as shown in Eq (1).


The instructional text is regularly divided into multiple steps, which should be performed in a specified sequence.

instead, we seek representations that are capable of capturing the long-term dependencies across passages of text to derive higher-level information from the entire semantic block of each step, i.e., knowing that the expansion bays are detached without requiring any tool.


As mentioned in Section 3, the expected supervision for our model is the word-level edit matrix, but existing datasets only contain rewritten utterances.

instead of generating from scratch, such a formulation introduces edit operations and shapes the problem as prediction of a word-level edit matrix.


"Ques" is short for questions, "Avg" for average, "len" for length, "Con" for context utterance, "Cur" for current utterance, and "Rew" for rewritten utterance.

the predicted Y is not guaranteed to meet this requirement, indicating the need for a standardization step.


The newly proposed flexible knowledge inference focuses on n-ary facts of arities greater than 2.

we then optimize NeuInfer via backpropagation, and Adam (Kingma and Ba, 2015) with learning rate λ is used as the optimizer.


By differentiating the primary triple from other auxiliary description(s), NeuInfer considers the validity of the primary triple and the compatibility between the primary triple and its auxiliary description(s) to model each n-ary fact more appropriately and reasonably.

They can be divided into tensor/matrix based methods, translation based methods, and neural network based ones.


Both these wordnets are very large, by virtue of the automatic processes used to gather their contents, but on the flip side they lack the thorough validation only granted by a manual process.

only 4.1% of all word senses included this specific kind of gloss.


We follow the setting in LCML (Lin and Xu, 2019) for unknown intent detection.

as shown in Figure 2, a typical generalized zeroshot classification framework can be abstracted into two layers, the encoder layer and the zero-shot classifier layer.


Events realized by nouns can correspond to nominalizations, event nouns, contextual event readings or implicit events.

Attribution is a linguistic phenomenon that consists in “ascribing the ownership of an attitude towards some linguistic material, i.e. the text itself, a portion of it or its semantic content, to an entity” (Prasad et al., 2006; Pareti, 2015).


As for adjectives in attributive position, they can be marked as events only when their occurrence implies actual occurrences, such as the events leading up to its own existence.

the focus of this task was to identify all claim-like statements.


As shown in Kim et al. (2020), such a sequential way can better simulate a multi-turn dialog and facilitate knowledge selection in later turns.

we thus devise the disentangled variant as following, where the roles of two types of information are separated, which makes it feasible to conduct ablation study.


The knowledge selection of PostKS is supervised by a BOW loss.

these works have a different task setting from ours.


This makes our query vector combine the domain-shared feature with domain-specific feature.

as shown in the Figure 7, for a specific target domain, different examples may have different gate distributions, which indicates that our framework successfully learns how to transfer knowledge between different domains.


In addition, we take inspiration from Guo et al. (2018), who successfully apply the mix-of-the-experts (MoE) mechanism in multi-sources domain and cross-lingual6352 adaption tasks.

a simple baseline to incorporate domain-shared and domain-private features is shared-private framework (Liu et al., 2017; Zhong et al., 2018; Wu et al., 2019b).


Our next question is whether this difference is due to the differing syntactic complexity among our MWEs, and between our MWEs and those in previous work (which generally are shorter and simpler).

to most previous compositionality datasets we also consider syntactically complex constructions and publish a formal specification of each expression.


As the task scope was extended from specific-to open-domain QA, several models have been proposed to select a relevant paragraph from the text to predict the answer span (Wang et al., 2018;Clark and Gardner, 2018).

detecting supporting sentences is an essential step being able to answer the question.


These methods successfully demonstrated their potential and effectiveness in understanding relational datasets, such as entity linking in heterogeneous knowledge graphs, product recommendation systems, and detecting side effects in drug (Wu et al., 2019;Fan et al., 2019;Zitnik et al., 2018).

all the model variations undergo performance degradation compared to the original topology (PS-rnn-elmo-s).


Unlike measuring embedding bias, measuring classification bias is difficult: most NLP datasets are not annotated with protected attributes, precluding the use of standard fairness measures such as equal opportunity (Hardt et al., 2016).

Equalized odds requires that both true and false positive rates be equal across groups (Hardt et al., 2016).


as shown in section 3.3, datasets currently used to estimate classification bias in NLP – such as WinoBias (Zhao et al., 2018b) and WinoGender (Rudinger et al., 2018) – are too small to conclusively identify bias except in the most egregious cases.


As the source documents are very long and messy, they are split into multiple paragraphs by line-breaks.

to their approach, we incorporate explicit graph representations into the encoding process via a graphinformed attention mechanism.


An example is shown in Figure 1, where the narrative is uploaded to retell several lines of a script in a movie.

(1) To our best knowledge, this is the first investigation on movie script generation with a narrative.


However, a narrative plays a different and more specific role than a general context.

this is consistent with our intuition: when a narrative is more detailed and better aligned with the session in wording, it is easier to choose the best responses.


Recently, deep neural networks have been used to tackle the problem (Bamman et al., 2019), and related problems such as generating coherent and cohesive text (Cho et al., 2019) and identifying relations in generated stories (Roemmele, 2019) have also been addressed.

we manually review the data and remove such non-narrative data.


Compared to the dot-product attentions in GAT, KGAT's Edge attention focuses on fewer tokens and has a smaller entropy.

the BERt sentence retrieval outperforms ESIM sentence retrieval significantly, thus also helps improve KGAt's reasoning accuracy.


The evidence propagation and per-node label prediction in KGAT are conducted by Edge Kernels, which attentively propagate information among nodes in the graph G along the edges with the kernel attention mechanism.


This is because MetaQA KG is fairly sparse, with only 135k triples for 43k entities.

performance on 2-hop and 3-hop questions suggest that EmbedKGQA is able to infer the correct relation from the neighboring edges because the KG embeddings can model composition of relations.


Their task was to choose which of the learned response plans was better suited to the realization task (CTX, PSA, Both or Neither).

We also note that there were cases where the grammar, e.g. pronoun usage, is inconsistent.


Additional examples of success and failure cases are uploaded in the Appendix.

these models do not achieve the same level of consistent performance on generative modeling tasks as opposed to language understanding tasks (Ziegler et al., 2019;Edunov et al., 2019).


Although the original tweet mentions @narendramodi PM of India and uses Hindi keywords, but the nodes selected for expansion rightfully capture about finance ministry (@arunjaitley, @finminindia), home ministry (@amitshah), economic transformation and mostly positive sentiments about it.

fourth, it also addresses under-specificity and noisy text for sentiment classification by expanding or shrinking the network representing the tweets.


Majority of the recent studies focus on the application of neural network models.

we see substantial performance gain for N2V Rw in both the datasets when augmented with any node expansion.


The second tweet is one under-specified tweet where India's Prime Minister greets soldiers.

we evaluate the proposed method over datasets in different domains.


Kazemzadeh et al. (2014), Hu et al. (2017a), and Mao et al. (2016) introduce two benchmark datasets for the real-world 2D images

in our experiments, we presented one of these scenarios (single instruction, static, and pixellevel) since it was the closest to the pre-existing Touchdown-SDR system.


We thank members of MultiComp Lab at CMU and Berg Lab at UCSD for useful discussions.

table 7: Results for instruction ablation human study.


Since we use objects near to the target location as anchor points, this is also another useful metric.

with Touchdown, in our setup instructors, followers, and learning systems observe a partial FoV of the scene, but they can change the FoV continuously to explore the scene.


Relying on uconv's built-in deromanization is not optimal.

for languages that are related to a pretraining language with a different script (groups (b) and (c)), there is an added benefit of using romanization.


The participants ranged from 16 years old to 100 years old, almost in equal numbers of men and women, and they were all native speakers of Mapudungun.

because Machi (traditional Mapuche healers) were interviewed, we asked the transcribers to delete any culturally proprietary knowledge that a Machi may have revealed during the conversation.


Out of the 4262 distinct sentences, only 188 were recorded more than once.

tense changes and some form of negation or banning also keep the vectors very similar.


"One photon torpedo and you're a space toast."

the chance of two annotators producing the same output string is quite low.


In fact, out of the 62,501 unique tokens in the dataset, 57% appear just once, and 78% appear 3 or fewer times.

to decide the boost values, we calculate the average percentage word overlap between a segment in the question and its correct answer from AskUbuntu on the train and val sets.


Rows 2 and 3 in Table 3 present the results of using attention.

the best performance is achieved by combining pre-trained embeddings from multiple language models trained on different data sources.


Also sequence-to-sequence encoder-decoder architectures have been devised, to train systems on parallel corpora with the specific aim of news translation (Hassan et al. 2018).

one major challenge in the lexical semantics field is, to date, that of dealing with as many as possible languages at the same time (e.g., BabelNet covers 284 different languages), so to enable truly multilingual and crosslingual applications.


For example, word embeddings tend to be consistent across language variations (Aldarmaki, Mohan, and Diab 2018), whereas multilingual vector spaces have more difficulty in representing individual words (such as, e.g., homographs with unrelated senses and phrasal verbs) because of their different usage distributions.

among the main areas where multilingual and crosslingual resources and approaches are solicited, there are of course machine translation (Cho et al. 2014; Luong, Pham, and Manning 2015), crosslingual document categorization (Kocisky, Hermann, and Blunsom 2014; Gouws, Bengio, and Corrado 2015), and sentiment analysis (Tang et al. 2014).


With regard to the handling of polysemy, the embeddings obtained through the described approach reflect the multiple senses assumed by the word in different contexts.

such scores are higher than the agreement among human raters, which can be thought of as an upper bound to systems' performance.


Our edit-based model achieves a higher number of exact matches and a higher SARI score compared to prior work on sentence splitting.

our inference scheme in Sec. 2.2 still needs three times more time steps than the number of edits, i.e. around 4.0 × 3 = 12 for native English (last row in Table 11).


As part of the token-based approaches, we use Okapi BM25 (Robertson et al., 2009) and a variant, BM25+ (Lv and Zhai, 2011a).

among the various types of sources, they mostly rely on online news articles, which often serve to disseminate medical findings from research studies (Medlock et al., 2015).


We show a screenshot of the claim writing interface in Figure 6, and the claim verification interface in Figure 7.

the reason why we do not use the entirety of a large research corpus like S2ORC as our fact-checking corpus is that doing so would introduce many false negative retrievals: abstracts containing evidence relevant to a given claim, but not mentioned in the claim's source citance.


Susan Collins (R-ME) remains near the middle; she is among the most moderate senators in vote-based, speechbased, and tweet-based models.

the model lays the senators out on the real line and accurately separates them by party.


The topics related to education (Topics 5 and 6) and government services (Topic 10) and are more prominent in ukWac, partly because of the relative ease of crawling the educational and government websites, which less often rely on Javascript and have fewer explicit anti-robot restrictions.

more specifically, huberm and S n are computed for the distribution of probabilities over all documents in which a word occurs.


Many pre-trained language models, such as ELMO, BERT or GPT, have been produced by taking Web resources, such as the 1B Word Benchmark for ELMO, Wikipedia with the Toronto Book Corpus for BERT or WebText for GPT.

the way of dealing with this issue is by using robust methods to detect outliers within non-zero frequency documents and to use the traditional mean of the discounted frequencies (Winsorisation).


The values of the normalising constants b = 1.48 for M AD and c = 1.19 for S n are used to match the standard deviation value when the measures are applied to normally distributed data (Rousseeuw and Croux, 1993).

to extraction of popular URLs, ukWac is a corpus produced by crawling the .uk Internet domain (Baroni et al., 2009).


Such strong performance indicates the effectiveness of our CQA framework.

nSM annotates the questions and then anchors the model to the high-reward programs by assigning them with a deterministic probability.


For movie reviews, we compare our model with the baselines provided by DeYoung et al. (2019).

in addition, there is a task-specific predictor added after the encoder for each distinct task, so they would not intervene with one another while training.


To better understand the limits and potential of our proposed method, we further study about the QA IOU F1 Token F1  6.3 13.9 Bert-To-Bert 2019  ability and comprehensiveness of our model.

for the learning rate, we found that when using 1e - 5 without warm-up, the valid loss of SQuAD is 10% higher than using 5e - 6.


The predictor structure is the same as one in the beer rating task, where a dense layer inputs e0 and outputs one value vsent to form the rating score y movie .

annotating rationales requires much effort and only few datasets contain such labeled rationales, making supervised learning for rationalization difficult.


For instance, it allows for: the rapid development of (sub-)vocabularies; accurate indexes of words and their occurrence in the context of specific works or authors; comparative studies on different literary schools, different authors or different creative periods of a given author; diachronic studies concerned with the evolution of the Portuguese language; among many others.

its unique set of characteristics, namely being composed of complete documents instead of fragments, the carefully edited text, the variety of genres and authors, and the wide time span and orthographic conventions covered, make of it an invaluable and versatile resource for multiple research purposes, both for language technology and digital humanities.


This paper introduces the BDCamoes Collection of Portuguese Literary Documents, 1 a new language resource that is suitable for the scientific study and the technological preparation of the Portuguese language.

to many existing corpora aimed at primarily supporting the development and testing of natural language processing tools and applications, BDCamoes has a number of less common, very interesting set of characteristics that together turn it into an invaluable research resource, complementing other related resources.


To create a large semantic tagging resource for English words, we apply POS-aware Prediction with k = 3 for all words in the vocabulary of the Sketch Engine POS-specific word embeddings.

brants (2000) highlighted the importance of handling unknown words in part-of-speech tagging.


Given the well-established usefulness of part-of-speech tag annotations in many syntactically oriented downstream NLP tasks, the recently proposed notion of semantic tagging (Bjerva et al., 2016) aims at tagging words with tags informed by semantic distinctions, which are likely to be useful across a range of semantic tasks.

the Universal Semantic Tag scheme explicitly considers a sizeable number of different categories of named entities, and in Table 4, we saw that the prediction quality for them (NAM) is fairly high.


In order to unify the symbols, single-document summarization is regarded as a special input case of I = 1.

we take the DUC-04 as the test set, and DUC-03 is used for tuning the model when evaluating on DUC-04 dataset.


In this paper, we propose a joint learning approach to improve neural abstractive multi-document summarization by using single-document summarization dataset.

experimental results show the efficacy of our approach, and it can substantially outperform several strong baselines.


It involves identifying important information and filtering out redundant information from input sources.

it blurs the boundaries between documents and loses the hierarchy within the document cluster.


Computational approaches have also been conducted to analyze them (Gulordava and Baroni, 2011), proposing novel approaches for understanding lexical semantic change -for an overview, we refer to the survey by (Tahmasebi et al., 2018).

the frequency of all forms of a word were summed for each year to compute the total frequency of the word for that year.


In addition, most prior studies focused only on English, whereas comparing two or more languages can shed light on how they actually co-evolved over time.

such effects are often difficult to determine, especially when the causes are less known.


Despite its inherent problems (Pechenick et al., 2015), it is one of the few corpora of this size available in both French and English.

nevertheless, to the best of our knowledge, there has been no automatic study of the frequency correlations and patterns of cognates over time across different languages, especially one that uses large datasets.


We investigate the effectiveness of recent controllable generation methods at steering away from toxicity using REALTOXICITYPROMPTS.

lastly, because OPENAI-WT does not have available metadata, and due to the imperfect coverage of our subreddit and news reliability data, we only provide lower bound estimates of toxicity in web text corpora.


Furthermore, Jin et al. (2019) showed that BERT is the language model that best performs under adversary attacks when compared to CNN and LSTM in terms of success rate and perturbation rate, preservation of semantic content, and efficiency for text classification tasks.

Swap Noise: For each word in the text, one random pair of consecutive characters is swapped (e.g. expression → exrpession).


One way to evaluate this is by decreasing the lexical similarity between premise and hypothesis.

biLSTM-based models show significant proportion of correctly classified samples without even looking at the premise (which is an undesirable behavior).


Personality modeling from language is becoming increasingly important for many social scientific  applications.

judges showed a preference towards the high-attention messages for OPE and AGR, while CON and NEU were no better than chance.


We adapt the RuleTaker dataset to FEVER by introducing a new NOTENOUGHINFO label for unprovable question-context pairs.

Clark et al. (2020) analyze the logical reasoning capabilities of transformer-based models on a variety of question-answering and reading comprehension tasks.


These experiments showed that the information credibility on Twitter will become a key component for solving social problems, for instance.

around 3:00 PM, there is a remarkable increase in credibility due to another tweet posted by the president, which led to a sudden increase in credibility, followed by a constant drop.


In order to compute credibility scores for tweets, we needed training data.

a supervised neural network model is a mathematical function whose weights are refined iteration after iteration.


We use deep biaffine (Dozat and Manning, 2016) as the attention score of the pointer networks, and this model performs both the mention detection and the coreference resolution.

our model observes consistent performance regardless of 10 different initializations.


Interestingly, in addition to verbs we expected (acknowledge, admit, concede), we also find verbs like understand, agree, realize, know.

we settle on a task design as follows: annotators are told that we are collecting their judgments of Gw stance for a series of sentences; we then show an instructions page and guide them through 6 practice trials.


In both these examples, the embedded clause (that global warming is serious) is presented as an opinion belonging to a source entity (scientists).

Our work is related to social psychology research on persuasion (Cialdini, 1993; Orji et al., 2015) and recent NLP research on argumentation, such as predicting argument convincingness (Habernal and Gurevych, 2016; Simpson and Gurevych, 2018) and studying discourse-level and non-linguistic features predictive of persuasion (Yang and Kraut, 2017; Zhang et al., 2016).


In tweets with the word loneliness, users less often describe their own experiences (e.g., “The level of loneliness I’ve reached is at an all time high”), and more often make general statements (e.g., “we don’t know how to appreciate loneliness”) and quote celebrities and literary sources (e.g., “If you are afraid of loneliness, don’t marry. -Anton Chekhov”), than in tweets with the words lonely and solitude.

we identify words that are associated with the SOLO query terms, loneliness, lonely, and solitude, i.e., words that tend to appear in tweets with these query terms more often than they do in the General Tweet Corpus.


This is consistent with the conceptual definition of solitude as a positive, voluntary state of being alone.

we also computationally identify and compare words strongly associated with each of these terms.


We observed that the proposed method outperformed all baselines except for Simple back-off on the TOEFL dataset.

tables 4 and 5 list the results for the two tasks.


In such varieties, additional factors, such as influence from the regional languages, such as influences from Hausa, Yaruba and Igbo on Nigerian English (Olaniyi, 2014Í¾ Kperogi, 2015Í¾ Isiaka, 2019, come into play.

most lines that were obtained from external resources were later edited or pruned for length and target word and phoneme coverage, while keeping the semantics and facts intact.


The third column contains the transcription of the audio files, which have the same name as the utterance identifiers.

this rather affordable and portable hardware setup forms the core part of our inventory for collecting highquality speech data for low-resource languages and dialects across the world.


It consists of 370 texts, which total roughly 2.45 million words of text or about 300 hours of speech, excluding all interviewer utterances, collected in nine regions of the United Kingdom (such as the Hebrides and Scottish Highlands).

here and below we refer to the constituent elements of transcriptions as "tokens" or "word tokens" rather than words.


e b is the reification of John's "blond-ness", i.e., it represents the fact that John is blond.

those more specific rules are then exceptions to the general rule.


In (Searle, 1995)'s terminology, C defines when something counts as something else in the domain.

this could be needed, for instance, in argumentation systems, where the weight of each argument is assigned at the beginning, then it is inferred which arguments override other ones.


In this section we investigate the impact of the size of pre-training data to check whether AMR parsing benefits more from pre-trained models that are trained on larger datasets.

each input s in the fine-tuning task is now equipped with k + 1 outputs, one for the finetuning task while the other k for the k pre-training tasks.


Since BPE-dropout produces more fine-grained segmentation, sentences segmented with BPEdropout are longer; distribution of sentence lengths are shown in Figure 4 (a) (with p = 0.1, on average about 1.25 times longer).

we choose the first option to stay in the same setting as the standard BPE.


to confirm this, we reduce dimensionality of embeddings by SVD and visualize (Figure 7).


The only difference from BPE is how a word is segmented during model training: BPE-dropout randomly drops some merges from the BPE merge table, which results in different segmentations for the same word.

even for large datasets using BPE-dropout can result in substantially better quality for practical applications where input is likely to be noisy.


Each test consists of 16 short answer questions (a total of 48 questions).

additionally, this research (Gomaa and Fahmy, 2014b) presented the first (and the only one to our knowledge) arabic dataset (the Cairo university dataset).


In 54.14%(1155 answers), the difference is at most one point.

they used Zahran Word Embedding (Zahran et al., 2015).


It is larger than the original one for the English language (Mikolov et al., 2013a) and is enhanced with specific linguistic aspects of the Greek language.

the category of antonyms includes pairs of words that their relationship as antonyms includes higher complexity.


we re-affirm the conclusion from (Yih et al., 2016) that having each worker do one task (e.g. labeling a single node in the tree) makes annotation easier for workers.

to fully utilize powerful neural network approaches, it is necessary to have large numbers of training examples.


In general, the events for the 84th T.I.F. dataset contain many more articles than the Elections one, but in both datasets the clusters are non-trivial.

Furthermore, by using multiple documents mentioning the entity, in order to describe an event, helps to clarify its type (e.g. if an employee "left" the company to go home or was fired) and what actually happened (Hong et al., 2011).


The pipeline for the creation of events from the news articles is supported by the Elasticsearch (Gormley and Tong, 2015) database.

looking at the articles that we gather we see that most of them are published between 9 AM to around 9 PM, so the separation of events by day should not be a problem.


Recently, neural sequential models (Lample et al., 2016; Akbik et al., 2018; Vaswani et al., 2017) have shown strong performance for various sequence labeling task.

we use the model from Huggingface Transformer codebase , and the repository to finetune our model for sequence labeling task.


Least Confidence (LC) Culotta and McCallum (2005) measure the uncertainty of sequence models by the most likely predicted sequence.

these neural models usually require exhaustive human efforts for generating labels for each token, and may not perform well in lowresource settings.


Another direction of explainability evaluation is to compare the agreement of salient words annotated by humans to the saliency scores assigned by explanation techniques (DeYoung et al., 2020).

when a model recognizes a highly indicative pattern of the predicted class k, the tokens involved in the pattern would have highly positive saliency scores for this class and highly negative saliency scores for the remaining classes.


Deep neural networks (DNNs) have shown great promise in Natural Language Processing (NLP), outperforming other machine learning techniques in sentiment analysis (Devlin et al., 2018), language translation (Chorowski et al., 2015), speech recognition (Jia et al., 2018) and many other tasks 1 .

the vulnerabilities of NLP models have been exposed via, for example, small character perturbations (Ebrahimi et al., 2018), syntactically controlled paraphrasing (Iyyer et al., 2018), targeted keywords attacks (Alzantot et al., 2018;Cheng et al., 2018), and exploitation of back-translation systems (Ribeiro et al., 2018).


We further leverage a k-mer-based matching procedure to provide "word"-level information, as well as a dictionary comprised of prior knowledge about sensor names, to boost the tagging performance.

anecdotally, annotating one sensor name may cost several hundred dollars, and it takes weeks to do so for one typical building with thousands of sensing and control points.


Without human input, it is difficult for our model to correctly infer the meaning of such segments.

seNsER achieves over 79% and 67% F 1 in chunking and tagging, respectively -a notable 13-point improvement in tagging over the best compared method.


we propose to leverage such meaningful k-mers to complement the representation produced by the language model (i.e., z i defined in Eqq. (1)) as “word”-level information.

here, we present a case study about the learned k-mer embedding results.


Our main assumptions for disfluency events are: (1) Repetition-like disfluency events exhibit a common underlying structure property in the frequency domain, (2) filled pauses exhibit specific acoustic correlates with a steady frequency signature (Gabrea and O'Shaughnessy, 2000), (3) these filled pauses have usually adjacent unfilled pauses/silences (Daly, 1994).

the extraction of these two tracks from continuous speech can be decomposed in several engineering tasks (see Figure 1).


Italic for the primary track and Bold for the collateral track.

for every time-step t, for a given window-scale s, we compute the similarity ψ(t, s, i) of the frequency representation xt centered on t with its i-th closest neighbours.


We categorize the 20 classes into 4 categories based on the types of sentences: question sentence (“* SQ .”), declarative sentence (“* NP VP *” etc.), clause sentence (“SBAR *” and “S *”), and others (“OTHER”).

the performance gap between the SSANs variants is much lower that that on the local reordering task (i.e., 4% vs. 15%).


In this example, the word "talk" performs attention operation over input sequence, where the words "Bush", "held" and "Sharon" are chosen as the truly-significant words.

due to their high frequency with a limited vocabulary (e.g., 150 words 9 ), content-free words, or function words generally receive a lot of attention, although they have very little substantive meaning.


This further demonstrates that the learning process of each subtask can be enhanced by the collaborative learning.

hence we can model R 2 by propagating R 1 to M ctx .