Our measure is designed to be broadly applicable, requiring no domain-specific annotations; we provide exploratory output on justice utterances from the Supreme Court's oral arguments in the appendix and release code implementing our approach at http://convokit.cornell.edu to encourage experiments in other domains.

the method's efficacy in the present setting is likely boosted by the relative uniformity of crisis counseling conversations; and future work could aim to better accomodate settings with less structure and more linguistic variability.


Most of the successful and predominant methods for Bilingual Lexicon Induction (BLI) are mapping-based, where a linear mapping function is learned with the assumption that the word embedding spaces of different languages exhibit similar geometric structures (i.e., approximately isomorphic).

several recent studies have criticized this simplified assumption showing that it does not hold in general even for closely related languages.


Almost all CLWE methods inherently assume that embedding spaces of different languages are approximately isomorphic (i.e., similar in geometric structure).

recently researchers have questioned this simplified assumption and attributed the performance degradation of existing CLWE methods to the strong mismatches in embedding spaces caused by the linguistic and domain divergences Ormazabal et al., 2019).


Both of the individual models have the same encoderdecoder structure as JUG.

the main difference is that there is no shared latent variable between the two individual NLU and NLG models.


This is supported with the loss of ROBERTA that is adapted to NEWS, calculated on a sample of REVIEWS.

rOBErTA that is adapted to rEVIEWS results in the highest loss for a NEWS sample.


This class bundles families or groups of genes/proteins, e.g. [transcription factors] protein family or group or [aquaporins] protein family or group .

it does not incorporate very general terms (lipoprotein), locations (mitochondrial genes), functions (RNA-binding proteins) or similarity-descriptions (Caspase-like proteins).


Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods.

such benchmarks are available only for a handful of languages.


The authors released a toolkit 2 for model evaluation.

they do not provide a public leaderboard to compare the results of different models.


The Slavic-BERT has higher scores than Multi-BERT on seven out of nine tasks.

without a detailed ablation study, it is difficult to infer the main reason resulting in better performance.


While these results are not strictly comparable as they are based on different subsets of the data, use different cross-validation strategies and report different performance metrics, they collectively show that supervised models can learn to identify patients with AD using data from elicited speech samples.

as is generally the case with supervised learning on small data sets, overfitting is a concern.


This model was then used to estimate perplexity of elicited speech samples in cases and controls, with significant differences between mean perplexity scores obtained from subjects with the semantic dementia variant of FTLD and controls.

the authors did not attempt to use perplexity score as a variable in a diagnostic classification of FTLD or its subtypes.


Fritsch et al. (2019) further improved performance of this approach by substituting a neural LM (a LSTM model) for the n-gram LM, and report an improved AUC of 0.92.

it is currently unclear as to whether this level of accuracy is due to dementia-specific linguistic markers, or a result of markers of other significant differences between the case and control group such as age (x = 71.4 vs. 63) and years of education (x= 12.1 vs. 14.3) (Becker et al., 1994).


While all models tend to find the increasingly perturbed transcripts more perplexing than their minimally perturbed counterparts, this perplexity decreases with increasing contributions of the dementia LM.

when only this model is used, relative perplexity of the perturbed transcripts increases.


As such, these benchmarks do not provide reliable estimations of the actual ASR accuracy.

on the truly challenging benchmarks, such as the dinner party conversations CHiME5 (Manohar et al., 2019) benchmark, modern ASRs report WERs in the ranges of 46%-73%.


An alternative to deterministic text generation is to sample directly from the softmax distribution.

since the probability mass tends to accumulate in a long tail, this procedure generates unlikely words too often, leading to degenerate text (Fan et al., 2018;Holtzman et al., 2020).


Usually, authors report the values for perplexity computed on the original probability distribution, before truncation.

this metric does not allow different sparse decoding strategies to be compared.


After learning the joint topic vectors, embedding-based predictions can be derived for any unlabeled review.

these predictions are sub-optimal for sentiment analysis where word order plays an important role.


Recent advances in self-supervised pretraining (Radford et al., 2018; Devlin et al., 2018a; Liu et al., 2019) have resulted in impressive downstream performance on several NLP tasks (Wang et al., 2018, 2019).

this has led to the development of enormous models, which often require days of training on non-commodity hardware (e.g. TPUs)


We might expect an end-to-end ASR model (without an explicit disfluency detection component) not to effectively detect disfluencies.

we show that end-to-end ASR models do learn to directly generate fluent transcripts and their performance is comparable to a baseline pipeline system (i.e. an ASR
model followed by a specialized disfluency
detection model).


While WER with respect to the full reference transcript (containing both fluent and disfluent words) is not meaningful for integrated systems intended to produce fluent output, WER with respect to the fluent subsequence is a meaningful measure of overall system, since this is the intended output of an integrated system.

since disfluencies only comprise around 6% of the total words, the WER score largely reflects how well fluent words are recognized, rather than how well the system handles disfluencies.


As k is usually small (averaging around 2 per example in our experiments), sequence lengths remain similar to those encountered for the same x during language modeling.

using LMs to directly predict x from x˜ as in Fedus et al. (2018) effectively doubles the sequence length of x


Zhu et al. (2019); Shen et al. (2020) infill multiple variable-length sequences, but these approaches require the masked context to be iteratively updated and reprocessed to fill in blanks one a time.

our approach appends infilled text to the context and does not require reprocessing the entire input sequence for each blank.


Vega-Redondo et al. (2019) annotate business relevance and sentiment on online chat interactions among aspiring entrepreneurs.

we annotate the communicative styles cooperativeness, motivational, advice and equality on chat interactions between young aspiring entrepreneurs, and develop machine learning systems to automatically predict these styles and indicators of business success for the participants.


The impact of SpanBERT over BERT is clear, showing 2.4% improvement on average.

none of the HOI models shows a clear advantage over SpanBERT which adapts no HOI.


Table 3 shows that EE and CM reduce FL errors by 4+%, suggesting that the aggregation of non-local features indeed leads to more conservative linking decisions.

adapting an advanced encoder shows higher impact on WL errors, as SpanBERT reduces almost 10% compared to BERT, implying that representation learning is still more important for semantic matching.


Contextualized word embeddings have been employed effectively across several tasks in Natural Language Processing, as they have proved to carry useful semantic information.

it is still hard to link them to structured sources of knowledge.


On the other hand, supervised models (Huang et al., 2019;) have proved to achieve state-of-the-art results on the English benchmarks by taking advantage of manually-annotated data for the task and machine learning algorithms.

supervised approaches are mostly focused on English (Navigli, 2018;Pasini, 2020) and have only recently been applied to lower-resourced languages thanks to automatically-produced datasets (Scarlini et al., 2019;Barba et al., 2020;.


As expected, conditioning on longer histories increases the predictability of a sentence.

this effect is significantly larger for imagined stories, which suggests that imagined stories flow more linearly than recalled stories.


We crawled all the weblogs tagged as lung cancer, which resulted in 472 weblogs with 117k articles.

with clinical records written by medical practitioners, patient weblogs contain a wide variety of topics; not only records of progress of their treatment and physical conditions, but also memories of their daily lives.


Patients report nearly four times more adverse effects than therapeutic effects.

they report the nonoccurrence of expected effects with similar frequency for therapeutic and adverse effects.


This has largely been successful, with over 30 wordnets uploaded to the new interface.

there were some issues along the way, which we detail below.


Figure 2 shows some examples of failed checks on the quality of the semantic hierarchy.

if the validation process did not encounter any problems, then the projects will have the option to upload it onto the OMW system, which will make it immediately available on its online interface.


On WMT En-De, with K = 10, beam search gives a mBLEU score of 66.3 but a pairwise score of 74.

the sampling baseline generates very diverse but inaccurate hypotheses, with a pairwise score of 11.8, but a mBLEU of 28.2.


We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time.

the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models.


If the span is incorrect, we expect workers would make a best effort to interpret the span (for example, if the span is one word too long or short they will probably still understand it correctly, especially since they see it in the context of the entire sentence).

for evaluation, we label these cases with a special category, 'none', indicating that the span is incorrect or attached to the incorrect predicate.


There has been much interest in the problem of systematic generalization in recent years (Bahdanau et al., 2019;Bentivogli et al., 2016;Lake et al., 2017a,b;Gershman and Tenenbaum, 2015;McCoy et al., 2019a;Veldhoen and Zuidema, 2017;Soulos et al., 2019;Prasad et al., 2019;Richardson et al., 2019;Johnson et al., 2017, inter alia).

to our approach (testing novel words in familiar combinations), many of these studies probe systematicity by testing familiar words in novel combinations.


Depending on the characteristics of the dataset, answers can be selected either from entities in the constructed entity graph (Song et al., 2018;Dhingra et al., 2018; De Cao et al., 2019; Tu et al., 2019; Ding et al., 2019), or from spans in documents by fusing entity representations back into token-level document representation (Xiao et al., 2019).

the constructed graph is mostly used for answer prediction only, while insufficient for finding supporting facts.


In the Distractor setting, there are 2 gold paragraphs and 8 distractors.

2 gold paragraphs may not be available in the Fullwiki Setting.


In the task of sentiment analysis, we observe that training examples containing the most positively salient token in the test example generally have a higher influence to the test prediction.

we do not see this trend (in fact, it is the opposite) in the task of natural language inference.


Since the appearance and rise of attentionbased models, many work naturally inspect attention scores and interpret with them.

we are aware of the recent discussion over whether attention is a kind of faithful explanation (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).


Regarding reproducibility in NLP, a cross-cutting analysis of papers presented at the EMNLP 4 2018 conference (Dodge et al., 2019) showed a widespread, positive practice of reporting the best hyperparameter settings (74%) and the data splits used (92%).

only 8% of the papers describe the hyperparameter search bounds, 14% the search strategy, and only 30% provide the source code.


Even though most semantic-role formalisms are built upon constituent syntax, and only syntactic constituents can be labeled as arguments (e.g., FrameNet and PropBank), all the recent work on syntaxaware SRL relies on dependency representations of syntax.

we show how graph convolutional networks (GCNs) can be used to encode constituent structures and inform an SRL system.


An interesting difference is that for ELMo, syntax is more helpful for arguments very far from the predicate.

for RoBERTa, syntax is helpful on arguments 4-7 tokens away from the predicate, but hurts performance on arguments farther away from the predicate.


Tweets quoting another tweet are considered like responses and are annotated in the same way.

to let the annotator know, the quoting tweets contain an extract of the quoted text, as shown in the second tweet in Figure 2.


We could argue that the addition of the ignore class is thus not very useful since increasing the number of classes is likely to decrease the inter-rater agreement.

certain annotators reported less frustration when having the possibility to use the ignore class.


We observe that for Captions, where we transfer a factual (neutral) to romantic/humorous sentence, the add- tagger provides the best accuracy with a relatively negligible drop in BLEU scores.

for Yelp, where both polarities are clearly defined, the replace-tagger gives the best performance.


For example, Lakshmi Narayan et al. (2019) applied noise injection and word dropout and obtained a performance boost, Bodapati et al. (2019) varied the capitalization of words to increase the robustness to capitalization errors, Liu et al. (2019) augmented traditional models with pretraining on external knowledge bases.

our work can be viewed as data augmentation in the continuous hidden space without external resources.


To fight against miscalibration, a natural option is to apply a calibration method such as temperature scaling (Guo et al., 2017) in a post-processing step.

temperature scaling only learns a single parameter to rescale all the logits, which is not flexible and insufficient.


A third option is to use Bayesian neural networks (Blundell et al., 2015;Louizos and Welling, 2017), which treat model parameters as probability distributions to represent model uncertainty explicitly.

these Bayesian approaches are often prohibitive, as the priors of the model parameters are difficult to specify, and exact inference is intractable, which can also lead to unreliable uncertainty estimates.


They first train multiple models with different initializations and then average their predictions.

fine-tuning multiple language models requires extremely intensive computing resources.


Kumar et al. (2018) propose a differentiable surrogate for the expected calibration error, called maximum mean calibration error (MMCE), using kernel embedding.

such a kernel embedding method is computationally expensive and not scalable to the large pre-trained language models.


In an attempt to set the ground for more comparable research in the area, a standard for dialog act annotation was developed (Bunt et al., 2012;Bunt et al., 2017).

annotating dialogs according to this standard is an exhaustive process, especially since the annotation does not consist of a single label, which in the standard nomenclature is called a communicative function, but rather of a complex structure which includes information regarding the semantic dimension of the dialog act and relations with other segments, among others.


Such methods/metrics solely optimize/reward similarity with human-generated reference questions treated as the ground truth (GT).

in many openended generation tasks where only one or a few of many possible GTs are available through human annotation, this approach directly penalizes diversity by discouraging deviation from the GT(s).


A common practice, as seen in Vaswani et al. (2017)’s setup, is to jointly learn BPE for both source and target languages, which facilitates three-way weight sharing between the encoder’s input, the decoder’s input, and the output (i.e. classifier’s class) embeddings (Press and Wolf, 2017).

to facilitate fine-grained analysis of vocabulary sizes and their effect on class imbalance, our models separately learn source and target vocabularies; weight sharing between the encoder's  combined in Figure 4.


While Sperber et al. (2019) use a beam of size 1 for the ASR component of their cascade to compare with their two-stage end-to end models, we find that using equal beam sizes of 15 for both ASR and MT improves cascaded performance with the same model by 4-8 BLEU; combining these two parameter changes makes the same cascaded model a much more competitive baseline (compare lines 3 in both Table 2 and Table 3).

widening beam size to yield an equivalent search space for end-to-end models has diminishing returns after a certain point; we did not see further benefits with a larger beam (> 15).


Their model provides considerable training and decoding time improvements due to the reduced source sequence length, and shows consistent improvements over the baseline end-to-end model using the original filterbank feature sequences which increase with the amount of training data.

their model has lower overall performance and with much smaller performance improvements over our baselines in lower-resource conditions than the phone featured models we propose here.


Many impressive progresses have been made in neural machine translation (NMT) in the past few years (Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2019).

the general training procedure requires tremendous amounts of high-quality parallel corpus to achieve a deep model's full potential.


We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples.

the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed.


It improves over exact string matching by preventing false negatives due to semantically equivalent clause reordering.

it is still considered a strict metric and creates false negatives.


By SQL style conventions, "LIKE" usually precedes a value of the form "%[name]%" and corresponds to natural language query "contains [name]" rather than "matches [name]"; it seems plausible that the model does not understand the natural language query.

if we replace the wrong value "%[name]%" with the gold value "[name]" after the "LIKE" operator, the predicate becomes semantically equivalent to "= [value]" and hence makes the query semantically correct.


Compared with the previous two approaches, a language model, trained on an enormous amount of text, can naturally capture coherence among both words and utterances.

a good evaluation metric should not only measure the quality of gen-eration, but also the diversity of generation, which is especially important for open-ended tasks like dialogue or story generation (Hashimoto et al., 2019).


This paper provides a holistic and automatic evaluation method for open-domain dialogue models.

to prior art, our means of evaluation captures not only the quality of generation, but also the diversity and logical consistency of responses.


Thus, we need articles written about entities for which we can identify the gender information.

to obtain gender information for existing annotated datasets could be costly or impossible.


Since we obtain the distantly supervised sentences for a relation from the head entity's article, this guarantees the model will not reuse sentences from an article.

it is possible that the head entity will appear as a tail entity in other relations because an entity could appear in multiple articles.


They learn a vocabulary of edit operations from training data and thus can work for any language.

their performance is inferior to their seq2seq counterpart.


Each tweet was labelled by 3 human annotators using five classes: positive, negative, neutral, mixed, and unknown.

the authors published not the full-text of the tweets, but the IDs of the annotated tweets.


Improvements have been obtained by exploiting either more sophisticated spaces (e.g., going from Euclidean to complex or hyperbolic space) or more sophisticated operations (e.g., from translations to isometries, or to learning graph neural networks).

our approach takes a step forward in both directions.


The recent QuatE model (Zhang et al., 2019) learns KG embeddings using quaternions.

a downside is that these embeddings require very highdimensional spaces, leading to high memory costs.


Therefore, topic models explore the word co-occurrence patterns, i.e., semantics.

no Transformer-based model considers these explicit semantics.


Next, the results in rows 20-23 show that the YouTube metadata features improve the performance when combined with the Twitter followers' features.

the Facebook audience features' performance is deficient and hurts the overall performance, i.e., these estimates seem not to correlate well with the political leanings of news media.


Intelligent assistants have gained great popularity in recent years since they provide a new way for people to interact with the Internet conversationally (Hoy, 2018).

it is still challenging to answer people's diverse questions effectively.


In our test set, 87.45% news is satire and the rest of them are misleading news and clickbait.

the dataset we used for the human baseline contains 23% satire news.


In addition to that, it is also found that the use of punctuations in fake news is more frequent than authentic news, and most of the time fake news is found on the least popular sites.

since character level features have shown better results so we will incorporate the character level features in neural network models such as (Kim et al., 2016;Hwang and Sung, 2017).


We ran experiments including question length and word overlap between the passage and question as calibrator features.

these features did not improve the validation performance of the calibrator.


In particular, the WebAnno we are using, makes it possible to calculate Kohen's Kappa, Fleiss' Kappa and Krippendorff's Alpha.

in order to do it properly, a tool needs to know all interdependencies among categories of tags.


In principle, this approach is similar to knowledge-based approaches for word sense disambiguation, especially the classic algorithm Lesk and its extensions (Lesk, 1986;Agirre et al., 2014;Basile et al., 2014), that apply a common strategy to disambiguate any word by referring to sense representations and measuring overlaps between a word context and sense representations.

different from the previous knowledge-based approaches that  directly use sense representations provided by lexicalsemantic resources, we leverage sense-annotated corpus as well in a data-driven manner to learn to build sense representations and learn to measure the overlap between a word context and a sense representation.


We observe that the word-specific classifier model obtains comparable performance when trained with either in-class or outof-the-class verb instances.

the single-classifier models are able to effectively transfer common syntactic (and semantic) alternation patterns across verbs in the same VerbNet class and, thus, significantly improve the performance when trained on in-class verb instance.


The main impetus lies in the pre-trained neural language models that capture long-range dependencies among words, owing to multi-head attention that is unique in the architecture.

little is known for how linguistic properties are processed, represented, and utilized for downstream tasks among hundreds of attention heads inside the pre-trained transformerbased model.


Throughout the entire experiments, we mainly used huggingface’s seven pre-trained transformers2 , implemented with Pytorch.

since the original implemented models do not return the output vectors of the internal attention heads, we developed the wrapper class that enables extracting the output vectors from the created pre-trained model objects.


In particular, we can see the exact distribution of the predictions of our algorithm for experiment 8 with a confusion matrix (figure 3).

some measures seem to differ, such as those in experiment 2, which are lower than 0.5, or those in experiment 5, which are lower than 0.6, with both experiments taking into account features related to the context of the question (vectorization in the first case and the presence of words belonging to lexicons in the second one).


This difference in the variation can be observed in the average edit distance between the prompts of each class, which is 3.27 and 2.73 for mined and paraphrased prompts respectively.

the improvement led by ensembling paraphrases is still significant over just using one prompt (Top1 vs. Opti.), raising microaveraged accuracy from 32.7% to 36.2% on BERT-base, and from 37.8% to 40.1% on BERTlarge.


Theoretical quantities like mutual information might seem appealing in this context given their strong grounding in information theory.

applying them would potentially require us to fully specify the state space the world can be in for an open-domain conversation, as well as estimating the probability distribution over potential configurations, neither of which is trivial, if feasible.


During this same year, Spandau Ballet achieved platinum status with the compilation The Singles Collection, which kept the focus on the band between studio albums and celebrated its five years of success.

the album was released by Chrysalis Records without the band's approval and the band instigated legal action against the label.


In the area of dialogue systems, a number of contributions are concerned with the generation of variety in the utterances of the dialogue system (e.g (Wen et al., 2015; Kozlowski et al., 2003; Langkilde and Knight, 1998)).

those efforts are mainly focused on word-level paraphrases and little work has been dedicated to the generation of contextual paraphrases.


BERT is intended to be pre-trained in the described manner and then fine-tuned to a specific task.

as our goal is not to solve a specific task, but rather to determine the informational content of a model, we employ the featurebased approach without fine-tuning presented in the original work.


While neither SNCS nor the machine learning based approaches need time for training if pre-trained encoders are used, the aforementioned flexibility to train a new encoder comes at the cost of additional time needed to perform this training.

during deployment the machine learning based approaches can embed sentences much faster than SNCS, as they merely perform a number of mathematical operations while SNCS needs to search for words in a Semantic Net repeatedly for each sentence pair individually.


Therefore, we exclude this model from our study of the dialogue act clustering task.

to get an idea of the performance of this model, we add a dialogue act classification task.


Therefore, existing models provide a solid foundation for research involving contextual paraphrases.

additional work needs to be done to improve the current results and further advance the handling of contextual paraphrases.


Only very recently, recommending conferences based on authors, abstracts, and keywords became a new research area (Iana et al., 2019).

the authors approach a more general setting that includes conferences from a wide variety of fields.


In order to make the event and argument classifier shared across modalities, the image and text graph should be encoded to the same space.

it is extremely costly to obtain the parallel text and image event annotation.


On Robust04, the use of smaller BERT variants always leads to decreasing effectiveness.

when using BERT-Small and BERT-Base for the final re-ranking, the corresponding BERT-QE variants always outperform BERT-Large significantly at the 0.1 level.


Fine Tuning Results so far evaluate the ability of NMT models to integrate similar sentences.

we have run our comparisons over a "generic" model built from a heterogeneous training data set while it is well known that these models do not achieve best performance on homogeneous test sets.


According to the annotation guidelines of Prop-Bank (Bonial et al., 2010), Experiencer is typically Arg0 in numbered roles.

the numbered role Arg1 is usually assigned to the Patient argument, i.e. “the argument which undergoes the change of state or is being affected by the action”.


Due to the alleviation of error propagation and lower latency, the end-to-end ST model has been a hot topic in recent years.

large paired data of source audios and target sentences are required to train such a model, which is not easy to satisfy for most language pairs.


Recently, Wang et al. (2019b) propose to stack an ASR encoder and an MT encoder as a new ST encoder, which incorporates acoustic and linguistic knowledge respectively.

the gap between these two encoders is hard to bridge by simply concatenating the encoders.


They formalize learning strategies from easier networks to more difficult network structures.

we focus on curriculum learning in pre-training and increase the difficulty of pre-training tasks.


This is because the target word prediction task is dictionary-supervised in expanded setting rather than reference-supervised as in base setting.

our method still outperforms the simple pre-training method by a large margin.


This indicates our method can make a good use of ASR corpus and learn valuable linguistic knowledge other than simple acoustic information.

when additional MT data is used, there is still a gap between the end-to-end method and the cascaded method.


We also found some examples where both the retrieve-only y 1 and the pre-ranked y were the same, and they were copied verbatim to generate the candidate output.

several of these copied retrieved outputs were too general summaries, and since the source was ignored during generation, the generated candidate output was missing some article specific information present in the target summary.


Transformers have supplanted recurrent models in a large number of NLP tasks.

the differences in their abilities to model different syntactic properties remain largely unknown.


For models with positional encoding, the lack of the ability to generalize to higher lengths could be attributed to the fact that the model has never been trained on some of the positional encodings that it receives at test time.

the model without any explicit form of positional encoding is less susceptible to such issues if it is capable of performing the task and was found to generalize well across various hyperparameter settings.


At the same time, it is easy to show that if cos(nπ), which has a period of two is used as positional encoding, the self-attention mechanism can easily achieve the task which we also observe when we empirically evaluated with such an encoding.

the same encoding would not work for a language such as (aaaa) * , which has a periodicity of four.


Prior work studied learning from human provided rationales (Lei et al., 2016; Ross et al., 2017; Bao et al., 2018; Ghaeini et al., 2019) in order to improve model explainability.

human rationales are expensive to acquire.


Comparing correlation values for H and L in Tab3, we observe that when the models are making correct and confident predictions (H), the values of correlation tend to be higher.

when the model fails to detect the correct relation (L), we see substantially lower correlation scores.


In practice, this method produces significant speedups at both training and inference time (by selecting a small subset of columns and performing matrix multiplications given much smaller matrices).

it is reported to achieve lower performance compared to unstructured pruning (Yao et al., 2019) due to more restrictive sparse patterns


Previous sections illustrate the potential external supervision by assuming the existence of vokens.

we are currently lacking the dense annotations from tokens to images.


The vision-language module represents the interactions between the image and the question.

this module alone is not able to answer questions that require insights that are neither in the image, nor in the question.


Document-level relation extraction requires integrating information within and across multiple sentences of a document and capturing complex interactions between inter-sentence entities.

effective aggregation of relevant information in the document remains a challenging research question.


There are also feature-based approaches achieving good results by optimizing sentence selection under a summary length constraint (Feigenblat et al., 2017).

to previous work, our proposal does not simultaneously perform segment selection and query matching.


It is known that for the language modeling validation perplexity measures using teacher forcing is not the best evaluation of generative capabilities, even if there is correlation between the two.

it is a commonly used metric for language modeling, and can be parallelized and computed inexpensively without the need for autoregressive sampling of output text.


Alternatively, GCC-DEC (355M) adheres more closely to the references instead of the prior conversation context, thus resulting in higher style match for longer conversations.

this over-adherance to the conversation style does seem to impact conversation quality for longer conversations.


Concurrent work has shown the value of leveraging large amounts of Reddit data to harvest naturally occurring conversations for the purposes of downstream conversational tasks (Zhang et al., 2019).

this work does not address the issue of stylistic control or the effects of scaling models to large sizes, which are central themes of our work.


Previous work has cast a few traditional NLP tasks as question answering, such as textual entailment (McCann et al., 2018), entity-relation extraction (Li et al., 2019), and coreference resolution (Wu et al., 2020).

unlike these tasks, we do not have large scale training datasets for bridging.


Looking at the results on ISNotes, we find that BARQA trained on a small number of in-domain dataset (BASHI) achieves an accuracy of 38.16% on ISNotes, which is better than the model trained on the other two large-scale datasets (SQuAD 1.1 and QuasiBridging).

when using these two datasets to pre-train the model then fine-tuning it with the small in-domain dataset (BASHI), both settings (i.e., SQuAD 1.1 + BASHI and QuasiBridging + BASHI) achieve better results compared to using BASHI as the only training dataset.


Particularly, we notice that the performance of using QuasiBridging alone is worse than the one using SQuAD 1.1 only.

combining Qua-siBridging and BASHI achieves the best result on ISNotes, with an accuracy of 47.21%.


To this end, (Satyapanich et al., 2020) recently presents the first dataset for cybersecurity ED (called CASIE) that annotates event instances with rich annotation.

this dataset involves at least three limitations that hinder future research in this area.


Prior work has applied NLP to perform several tasks for the cybersecurity domain, including privacy policy analysis (Peng et al., 2012; Pandita et al., 2013; Zhu and Dumitras, 2016), text analysis for cybersecurity with social media text (i.e., DDos attack detection, alert generation for threads and vulnerabilities using Twitter) (Mittal et al., 2016; Wang and Zhang, 2017; Sceller et al., 2017; Chambers et al., 2018; Perera et al., 2018; Alguliyev et al., 2019; Hasan et al., 2019), and report and timeline creation of cybersecurity events (Hackmageddon, 2019; PrivacyRight, 2019).

none of these work considers the detection of event trigger words from cybersecurity articles as we do.


(Hu et al., 2019) adopts a heuristic algorithm a length penalty to avoid overlong targets.

the length penalty make the model be incline to ignore target phrases.


Compared to (Zhou et al., 2019), the extraction method proposed by (Hu et al., 2019) has solved the problem of huge search space better and achieve better results.


Take the example 4 in the table as an example, the correct extracted target should be "chili signed food items", but SPANpipeline split the gold target entity to two separate targets because of its length penalty.

our algorithm can extract the target "chili signed food items" correctly since we get the original candidates with the closest indexes and then extract the targets by the extending strategy.


Modeling temporal user context, either as a bagof-tweets (Gaur et al., 2019), or sequentially (Cao et al., 2019;Matero et al., 2019) helps in identifying suicidal intent.

in Figure 1, we show that the impact of varying time intervals between tweets is crucial for an accurate assessment.


The parsing speed of the A* parser differs dramatically for the different graphbanks.

the parsing speed with the transition systems is less sensitive to the graphbank and faster overall.


To allow the Transformer to utilize the order information of source code tokens, we train an embedding matrix W Pe that learns to encode tokens' absolute positions into vectors of dimension dmodel .

we show that capturing the order of code tokens is not helpful to learn source code representations and leads to poor summarization performance (§ 3.2).


Similar to our approach, they use tables extracted from LATEX sources.

they do not extract absolute metric values but rank papers and do not appear to utilise the text content of publications.


Evert (2004) uses the dataset of 21,796 German PP-verb combinations (German_PNV_Krenn) in his experiments, manually annotated as lexical collocations or non-collocations by Brigitte Krenn (Krenn, 2000).

evert (2004) emphasizes that it is not possible to generalize the results of experiments on verbal collocations to other types of collocations, i.e. with different syntactic relations.


Consider the adjective tief 'deep': its prototypical sense is 'reaching from top to bottom'.

in the phrase tiefe Liebe 'deep love' the adjective is not used in its basic meaning and means 'intense'.


For example, the adjective holzern has only two senses: the prototypical one 'wooden' which belongs to the domain of concrete concepts and the non-prototypical abstract one 'awkward', and both annotators correctly identified one case of the non-prototypical usage in holzerner Dialog 'awkward dialog'.

all the senses of the adjective historisch 'historical' are very abstract and the interpretation of each adjective-noun pair is highly subjective which reflects in the agreement of only 66%.


In contrast, we generate a single unbiased reference translation generated by humans instead of trying to cover a wider space of possible translations.

to human paraphrasing (our instructions asked for most diverse paraphrases), automatic paraphrasing are still far from perfect (Roy and Grangier, 2019) and mostly generate local changes that do not steer away from biases as e.g. introducing different sentence structures


We acquire 3 ratings per sentence and take the average as the final sentence score.

to WMT, we do not normalize the scores, and report the average absolute ratings.


All references generated with human translations (WMT, HQ(R) and HQ(all 4)) show negative correlation with human ratings for these extreme cases and produce the wrong order.

all references that rely purely on paraphrased references do produce the correct ranking of these three systems.


Big Data research enables computer-assisted, broadbased generalizations over rich datasets, which cannot be obtained through traditional methods of observation.

these generalizations often come with partially or completely untested assumptions about the irrelevance of potential noise in the data, the nature of which may influence results in ways which remain elusive to the researcher.


Supervised approaches have made great achievements in the slot filling task (Goo et al., 2018; Zhang et al., 2019), where substantial labeled training samples are needed.

collecting large numbers of training samples is not only expensive but also time-consuming.


Because it did not outperform BERT-base in our cross-validation based development setting, we did not further experiment with BERT-large.

we found that it resulted in the best F1-score achieved on our test set.


Table 5 shows that our unsupervised model achieves consistent performance in both Switchboard and the three cross-domain datas.

to the performance on the Switchboard dev set as shown in Table 3, our unsupervised model achieves performance similar to the ELECTRA-Base fine-tuning model.


The joint models can capture long-range dependency of disfluencies as well as chunk-level information.

training a parsing-based model requires large annotated tree-banks that contain both disfluencies and syntactic structures.


In most previous works, human annotators are asked to compose new sentences for the given sentences to make examples in some previous works.

such artificial sentences lead hidden biases, because annotators unconsciously use particular words.


In reasoning annotation, we focused only on nonentailment examples to exclude false entailment.


Many studies have used the DS approach to expand target relations and reduce the cost of constructing handcrafted training data.

a statistical analysis of the DS data from Wikipedia-DBpedia collected in this study was found to contain 49% noise.


As far as we know, Amazon Mechanical Turk and Crowdflower are widely used in English.

there is not a Korean related work there, and it is not easy to gather Korean worker.


The results obtained, with an F score of 73.35, suggest that the methods that have been previously proposed for English are transferable to Spanish.

a question that remains open is whether the methodology used is optimal for Spanish.


Everything indicates that the system seems to be extending the scope to the final punctuation mark.

there are also some errors due to the system shortening the scope, as in Ex. (21), where the second element of the coordinated adjectival phrase is not included, or Ex. (22) where the complement of the noun cable is not included.


The results of the system indicate that the methods used for English are transferable to Spanish.

a qualitative error analysis has shown that the methods applied are not optimal.


Candidates in MOCHA come from existing models, so that a metric learned on this data will be most applicable to current research.

as research in generative reading comprehension models is presently limited, the strength of these models can be low.


The second cateogry of metrics eschew some of the aforementioned issues by calculating a softer similarity score using embeddings of tokens (Clark et al., 2019b; Zhang et al., 2019).

it is unclear how to tailor them to question answering, where the passage and question should be assimilated.


Machine learning models such as deep neural networks have achieved remarkable performance in many NLP tasks.

as noticed by recent studies, these models often inherit and amplify the biases in the datasets used to train the models (Zhao et al., 2017;Bolukbasi et al., 2016;Caliskan et al., 2017;Zhou et al., 2019;Manzini et al., 2019;Blodgett et al., 2020).


This gap is only marginally larger than the performance gap of 2.4% when evaluating the model on two randomly split groups.

if we evaluate the performance gap on the sentences containing the token "racist", the performance gap between these two groups is as large as 19%.


Incorporating sentence functions into conversations has shown improvements in the quality of generated responses.

the number of utterances for different types of fine-grained sentence functions is extremely imbalanced.


Considering all query and response sentence functions, we could have 20 × 20 = 400 meta tasks.

some tasks are extremely low-resource with less than 100 samples.


With fine-tuning on responses of the target sentence function, MTL+FT can capture the correct response pattern in some cases.

it is inferior to our proposed models MAML and SML, which can not only generate words related to the target sentence function but also keep the coherence and informativeness of responses.


Seq2Seq models have been used to generate agent responses without the need for intermediate dialog components such as the DST or the Natural Language Generator (Gangadharaiah et al., 2018).

there has not been much work that uses deeper knowledge of semantic representations in task-oriented dialog.


Thus, low human performances in this task would support the validity of our corpus.

this validation procedure will provide a precise baseline against which the automated classification model performances may be compared.


Prior work has combined these three scores into a single number using geometric averaging (Xu et al., 2018) or learned weights (Pang and Gimpel, 2019).

the aggregation is computed after averaging each metric independently across the test set (corpus-level aggregation), which is problematic since systems might generate sentences that optimize only a subset of metrics.


In the proposed method, the model learned with only images achieved the highest precision, but that learned with texts using BERT in addition to the images achieved the highest recall and F1.

the result of the model using LSTM was even worse than that trained with only images.


At the beginning, when the segmentation prediction network performs poorly, the model pools over only ground truth segment bounds, allowing it to learn the cleanest topic representations.

as training progresses and the segmentation accuracy begins to converge, we switch from pooling over ground truth segments to aligning predicted and ground truth segment.


Indeed, several works (Zheng et al., 2017; Luan et al., 2019; Wadden et al., 2019) use the Boundaries setting to compare to previous Strict results.

because the Strict setting is more restrictive, this leads to overestimating the benefit of the proposed model over previous SOTA.


In terms of modeling, their approaches used regular BERT as their encoders, where the argument representations are not explicitly conditioned on triggers.

our encoder is enhanced by providing more trigger-oriented information and BERT is only used as one part of it, which results in a trigger-aware sequence encoder.


Moreover, Gharouit and Nfaoui (2017) suggest to use BabelNet as knowledge base in the detection of verbose queries and then present a comparative study between different algorithms to classify queries into two classes, verbose or succinct.

both papers deal with the classification of queries submitted to search engines.


There, it has been quite hard for the annotators to distinguish between different levels of directness so that the class distribution of the directness is sub-optimal for the classification task.

comparing the results to a majority-class classifier clearly shows that there is still a lot of information encoded in the DA feature set achieving higher UAR.


For the classification of the 3-class elaborateness and the 3-class directness, the RNN classifier yields worse results than the ANN classifier.

for the estimation of the binary directness, the RNN classifier outperforms the ANN classifier, reaching an UAR of 78 % when using linguistic features encoded as word embeddings in combination with grammatical and dialogue act features (WE+DA+G).


A very few works try structured weight pruning, e.g., Wang et al. (2019) proposes a structured pruning approach based on low-rank factorization and augmented Lagrangian L0 norm regularization.

there also exist works that prune a coherent set of sub-modules in the Transformer model.


Analogously, intimacy in language can also be constructed with intentional pragmatic choices to signal the perceived intimacy between speakers.

while psychologists have deeply explored people's behavior of self-disclosure (Cozby, 1973) as one of the major components of verbal intimacy (Fitzpatrick, 1987), intimacy in language is not just conveyed by the degree of self-disclosure.


Societal views of gender roles significantly constrain the use of intimate communication, with specific expectations not only of the individual on the basis of their gender (Caltabiano and Smithson, 1983) but dyadic effects depending on the gender identities present .

individuals are less adherent to these norms as they perceive themselves to be anonymous or when interacting with an individual whom they perceive they will not interact with again (Rubin, 1983; Wynne and Wynne, 1986; Dindia et al., 1997); without the potential loss of face or social capital in such circumstances, individuals are more likely to engage in more intimate communication.


Individuals in these encounters have little likelihood of future interactions, removing the consequences for violating intimacy norms around increased disclosure (Thibaut, 2017; Wynne and Wynne, 1986).

to both friends and strangers, individuals are least intimate with casual acquaintances for which there are some expectations of potential future interaction and, therefore, longer-term consequences for norm violations.


The topic model baseline is still able to attain moderate performance, matching the intuition that some topics are more intimate (e.g., romance) while others are less (e.g., mobile phones).

as shown in Appendix J, Figure 8, many topics span the range of intimacies, demonstrating that estimating intimacy from topic alone is insufficient.


Bohm et al. (2019) train a BERT-based evaluation function with 2,500 human ratings for 500 machinegenerated summaries from the CNN/DailyMail dataset; their method correlates better with human ratings than ROUGE and BLEU.

as their method is designed for evaluating single-document summaries, it correlates poorly with the Pyramid scores for multi-document summaries (see Â§3).


Lugosch et al. (2019) propose a pretraining method for end-to-end spoken command recognition that relies on the availability of transcribed data.

while this pretraining strategy brings improvement over a system trained without transcripts, the absence of any other text-based baseline (such as a pipeline system) prevents any conclusion on the advantage of the end-to-end training when textual supervision is available.


Harwath et al. (2018) focus on speech, exploring how spoken captions in two languages can be used simultaneously to improve performance in an English-Hindi parallel subset of the Places-205 dataset (Zhou et al., 2014).

our experiments concern the setting where speech data from a low-resource language is used in conjunction with corresponding translated written captions.


For example, in the second bag {cat, on, the}, if looking only locally, we may pick "the" as the left neighbor of "cat".

if we notice that there is another determiner "a" in the first bag, then "the" will not be the only choice.


For example, BlockBERT with sequence length N = 512 and 2 blocks is trained with ten heads using permutation (1, 2) and the other two using permutation (2, 1).

there are other ways to assign twelve attention heads, e.g., seven heads for permutation (1, 2) and the other five for permutation (2, 1).


Automatic recognition of humor has become an important task in the area of figurative language processing, which can benefit various downstream NLP applications such as dialogue systems, sentiment analysis, and machine translation (Melby and Warner, 1995; Augello et al., 2008; Ghosh et al., 2015; Bertero and Fung, 2016; Blinov et al., 2019).

humor is one of the most complicated behaviors in natural language semantics and sometimes it is even difficult for humans to interpret.


The two meanings of the same expression are consistent with its context, which creates a humorous pun in both sentences when there is a clear contrast between two meanings.

heterographic puns take advantage of phonologically same or similar words.


Especially in the pun detection task, we observe that our model requires longer contexts (more than 20 words) to detect the homographic puns.

shorter contexts (less than 10 words) are adequate for heterographic pun detection, which indicates the contribution from phonological features.


Due to daily usage, participants stated they were strongly biased regarding mouse and keyboard, where "the muscle memory" helps.

many actually considered MK as very unintuitive if they imagined never having used it before, especially compared to pen and touch, or as one participant stated for reordering: "why do I have to do all of this, why is it not as simple as the pen".


We note that by combining sentence, paragraph and token index in the corpus, whole sentences can be reconstructed.

from 5,000 tokens roughly 300 sentences could be reconstructed, which are far too few to be used for training a neural model.


Different plot types have different slots.

some slots are shared across plot types.


None of the models above does consider the scores from the base parser when ranking trees.

it seems plausible to try combining the advantages from both models, base parser and reranker, to produce a better final model.


NER systems are often trained on the clean text.

they exhibit degraded performance in real-world scenarios where the transcriptions are produced by the previous upstream component, such as OCR or ASR ( §2.2), which results in a detrimental mismatch between the training and the test conditions.


Top-k in particular creates text that is easy for machines to detect but very hard for humans.

we observe the general trend: as the number of unlikely words available to be chosen is increased, humans get better at detecting fakes while automatic systems get worse.


Third, there is no explicit rating of news articles posted by users on news platforms.

in news recommendation users' interest in news is usually inferred from their click behaviors in an implicit way (Ilievski and Roy, 2013).


Third, news recommendation can be formulated as a special text matching problem, i.e., the matching between a candidate news article and a set of previously clicked news articles in some news reading interest space.

news recommendation has attracted increasing attentions from the NLP community (An et al., 2019; Wu et al., 2019c).


Other entities are found related to the surface forms of pay (Pay television) and ride (List of amusement rides).

it is expected that the activation values related to amusement rides and the company are lower than the activation values related to payments and Amazon, making the second text more conceptually complex than the first one.


Since the goal of the dialogue is given in advance for the task-oriented dialogue, we hardly find the chit-chat logs that are compatible with the goal on the Internet nor in the existing texts.

individual environments were set up according to the goal for collecting dialogue data in the past.


Minecraft provides a framework for chatting in a virtual world.

as a by-product, we obtain an environment for the situated dialogue where dialogue participants need to consider more diverse contextual information than the text-based dialogue, e.g. spatial relations, deictic reference and gesture.


Measuring the dynamics of gestures (Byrd, 1994;Mooshammer et.al., 2012) it became evident, that articulatory gestures are related to the positions (onset, vowel, coda) within a syllable.

it is concluded that the activity of the neuronal network steering the articulators is related to the structure of a syllable.


The advent of deep learning has greatly improved the performances of natural language processing (NLP) models.

the models are becoming more complex (Yang et al., 2019; Liu et al., 2019), rendering it difficult to understand the rationale behind their predictions.


As MLM is trained by masking only a part of the input sentence, replacing too many tokens can degrade its modeling performance.

we calculate the AUC until 20% of the tokens are replaced, and refer to it as AUC rep .


We selected 200 direct objects as W c from the most frequent 250 direct objects and the other 50 direct objects as w t for each verb.

the number of tasks N was 50,000, i.e., 50 tasks for each of the 1,000 verbs.


Target Identification To detect the targets of the claim (T c ) and topic (T t ), we assume that T c is semantically related to the topic, or more specifically, to T t .

we identify T c and T t simultaneously by following three strategies.


In the provided example in the table, one can see that the modifier terrorist dominates the sentiment of the positive word haven.

both terrorist haven and terrorist attack are considered generally bad.


Mainly, its good performance on the claims that refer to consequences reinforces our intuition that designing systems tailored for particular argumentation schemes might be a good alternative to topic-specific models.

we plan to complement this work with approaches for other frequently applied schemes such as arguments by expert opinion and arguments by example.


Intuitively, such disagreements among humans should be allowed because different annotators might have different subjective views of the world and might think differently when they encounter the same reasoning task.

from a descriptive perspective, evaluating the capacity of NLP models in predicting not only individual human opinions or the majority human opinion, but also the overall distribution over human judgments provides a more representative comparison between model capabilities and 'collective' human intelligence.


Thus, from a descriptive perspective, evaluating the capacity of NLP models in predicting not only individual human opinions or the majority human opinion, but also the overall distribution over human judgments provides a more representative comparison between model capabilities and 'collective' human intelligence.

we collect ChaosNLI, a large set of Collective HumAn OpinionS for examples in several existing (English) NLI datasets, and comprehensively examine the factor of human agreement (measured by the entropy of the distribution over human annotations) on the state-of-the-art model performances.


We identify several weaknesses: beam search can only find local optima, and a genuine parallel sentence cannot be recovered once it is pruned.

the method is vulnerable when parallel sentences have different word ordering.


By constraining on a target side trie during decoding, beam search can approximate pairwise comparison between source and target sentences.

overall we present an interesting way of finding parallel sentences through trie-constrained decoding.


This is because the document encoder, SpanBERT, dominates the memory usage during inference (as also observed by Xia et al., 2020).

the peak memory usage during inference is determined by the mention proposal stage rather than the mention clustering stage.


However, there is no chain between the first supporting fact and the second supporting fact due to the lack of the external knowledge that can connect "Coker", "coach" and "Miami Hurricanes football team".

the isolated reasoning chain leads to a wrong answer.


There are cases in which not all types of negation have been considered or they have only partially been taken into account.

when merging the corpora it is very important to take into consideration the types of negations (syntactic, morphological, lexical) and merge only those corpora completely annotated with the same types to avoid the system being trained with false negatives.


The overall size of the released data (600 training questions) is in line with real-world scenarios, where the high cost of domain expert time limits the amount of quality data that can reasonably be collected.

the dataset is meant to stimulate research in domain adaptation, in addition to developing algorithms for longer questions and answers than the current leaderboards.


This means that all SOTA Sup and UnSup approaches can be considered as a module that operates on the feed-in parameter Q.

we propose two different strategies for semi-supervision that emphasize the two-way interaction between the supervised signal and unsupervised alignment based on the message passing mechanisms, see Figure 1.


Several pre-trained visually grounded models have been proposed recently, and they are conceptually similar yet vary in design details, making evaluating them complicated and difficult.

for simplicity, we propose a simple and performant baseline, VisualBERT (see Figure 2), and base our analysis on this model.


We would like to adopt a similar training procedure as BERT but VisualBERT must learn to accommodate both language and visual input.

we reach to a resource of paired data: COCO (Chen et al., 2015) that contains images each paired with 5 independent captions.


As such, drift detection is treated as sequence segmentation task, and the better segmentations, the higher the overall accuracy.

experiment results of our model and all baselines are compared with the same metric, i.e., overall accuracy.


There is no guarantee that these questions need to aggregate heterogeneous information to find the answer.

designing hybrid question answering systems would probably yield marginal benefits over the non-hybrid ones, which greatly hinders the research development in building hybrid question answering systems.


However, directly maximizing the marginal likelihood is unnecessarily complicated as the marginalization leads to huge computation cost.

we propose to train the three models independently and then combine them to do inference.


Note that both common functional words and rare specific concepts in Chinese are commonly composed of multiple characters.

the contrast between common and rare characters is not so obvious, which leads to small α (no overwhelmingly functional words in syntax) and huge  β + γnorm (extremely analytic in morphology).


As described in section 2, GTM is an extension to W-LDA with the main difference that GTM models topics in a larger context and incorporates more global information with the graph encoder.

the improvements of GTM over W-LDA indicate the effectiveness of such information for topic modeling.


Wilkes-Gibbs and Clark (1992) indicated that when people collaborate on referring expressions, they issue full noun phrases initially, after which they begin to shorten the phrases.

in the process of repair, one could say "Give me the T8 Torx screwdriver", and then "Give me the screwdriver", or just "next one".


Here, however, we are interested in knowing the tools that are needed for each step of the task.

we utilize the toolbox provided by the instructors as a gazetteer 3 and search for the complete and/or partial mention of tools in the text.


Information extraction from instructional text is often performed by driving a dense representation of meanings from every sentence of the instructions.

it is assumed that the steps are described with single actions in imperative sentences.


However, the predicted Y is not guaranteed to meet this requirement, indicating the need for a standardization step.

the overall procedure of generation is divided into two stages: first the algorithm delimits standard edit regions via searching minimal covering rectangles for each connected region; then it manipulates the incomplete utterance based on these standard edit regions to produce the rewritten utterance.


As mentioned in Section 3, the expected supervision for our model is the word-level edit matrix, but existing datasets only contain rewritten utterances.

we use a procedure to automatically derive (noisy) word-level edit matrices (i.e.


However, none of automatic metrics reflects the utterance fluency or the improvement on downstream tasks.

human evaluations are included to evaluate the fluency of rewritten utterances and their boost on the downstream task.


The studies on n-ary facts are relatively scarcer, although they are also ubiquitous in the real world.

this paper addresses knowledge inference on n-ary facts.


Then, the values of their interaction vector, measuring the compatibility in many different views, are all encouraged to be large.

for each dimension, the minimum over it of all the interaction vectors is not allowed to be too small.


By differentiating the primary triple from other auxiliary description(s), NeuInfer considers the validity of the primary triple and the compatibility between the primary triple and its auxiliary description(s) to model each n-ary fact more appropriately and reasonably.

it is not surprising that NeuInfer beats the baselines.


It suggests that the validity evaluation component plays a pivotal role in our method.

each component of our method is necessary.


For each n-ary fact, NeuInfer distinguishes the primary triple from other auxiliary description(s) and models them properly.

neuInfer well handles various types of entity and relation inference concerning the primary triple coupled with any number of its auxiliary description(s).


Generally, the main consideration of job seekers is the skills they need to master, such as Python, English, and Go Language.

although S2SA generates some right words, like "preferred", it does not increase the quality of the generated text because it generates inaccurate skills.


Moreover, PostKS++ selects a quite different knowledge sen-tence at the 3 rd turn from those at previous turns, which is about the topic History of Australia but not Georgia (U.S. state).

postKS++ generates a response which is not coherent to the previous context at the 3 rd turn.


We think "answer-span" supervision allows the model to track the supporting sentences from simple word matching.

our investigations are focused on evaluating and analyzing the effectiveness of the proposed graph neural network-based model for classifying supporting sentences compared to the well-known answer-selection QA models.


In many real-world applications, there are multiple dialog types in human-bot conversations (called multi-type dialogs), such as chit-chat, task oriented dialogs, recommendation dialogs, and even question answering (Ram et al., 2018;Wang et al., 2014;Zhou et al., 2018b).

it is crucial to study how to proactively and naturally make conversational recommendation by the bots in the context of multi-type human-bot communication.


Finally we use some rules to generate a description for each goal (e.g., which side, the seeker or the recommender, to start the dialog, how to complete the goal).

we have task templates for guidance of data annotation.


Additionally, it is not trivial to train a model for a task in a particular language (e.g., English) and apply it directly to another language where only limited training data is available (i.e., low-resource languages).

it is essential to investigate strategies that allow one to use the large amount of training data available for English for the benefit of other languages.


For example, some uploaded paragraphs express one's subjective opinions about the movie, the actors, or simply copy the script.

we manually review the data and remove such non-narrative data.


The information that has been expressed is decayed.

the updated narrative focuses more on the remaining information.


This second approach turns out to perform better.

we only report the results with this latter method5 .


Here, instructions are given from the perspective of a partial field of view (FoV) of the scene, and these FoVs can dynamically be changed.

the correct interpretation of the sequence of instructions will require reasoning about what is currently visible in the FoV (e.g., grounding of objects) but also what is not visible yet.


Second, unlike other datasets, the target locations in Refer360° are randomly distributed and thus may occur anywhere – not just on predetermined objects.

target locations are less prone to bias (Devlin et al., 2015; Agrawal et al., 2016; Jabri et al., 2016; Goyal et al., 2016; Cirik et al., 2018b).


Further, real scenes are 3D, but the FoV is represented in 2D in our task.

interpreting some instructions will require inferences about depth.


To be more specific, when annotators give instructions, they use supporting objects as anchor points to help guide the attention of the follower.

the availability of a rich set of objects is essential for describing the target location.


First, previous datasets use the entire scene as a single field of view.

there is reduced need to describe how to find the target location sequentially


Instead of letting annotators decide where to place targets in the scene, we randomly picked a target location in the scene and asked them to describe how to find that loca-tion.

our instruction sequences are complex as we show next.


For each instruction sequence that was accurately found by another annotator, we paid a bonus of $0.10 to both the annotator who found the location and the annotator who wrote the instruction sequence.

for both the finding and describing task annotators have an interest in performing the task accurately.


Romanization is not necessarily reversible with simple rules due to information loss.

previous work on romanized machine translation has focused on source-side romanization only (Du and Way, 2017;Wang et al., 2018;Aqlan et al., 2019;Briakou and Carpuat, 2019;Gheini and May, 2019).


In practice, for a neural translation model in OpenNMT (Klein et al., 2017), the word embedding parameters accout for 80% of the total parameters.

it is significant to reduce the parameters of the embedding layer for deploying NLP models to memory-constrained devices.


Different KGs are usually extracted from separate data sources or contributed by people with different expertise.

it is natural for these KGs to constitute complementary knowledge of the world that can be expressed in different languages, structures and levels of specificity (Lehmann et al., 2015;Speer et al., 2017).


However, the characters in these tokens are probably very informative (for example "http" in a token would signal that the token is a URL).

fastText, which uses n-grams from a token to compute embeddings, would emit more meaningful representations.


In this setting, multilingual access is governed by the mapping of terms onto their underlying sense descriptions, such that all vectors co-exist in the same semantic space.

for each term we have thus the "blended" terminological vector along with those describing all senses associated to that term.


For example, word embeddings tend to be consistent across language variations (Aldarmaki, Mohan, and Diab 2018), whereas multilingual vector spaces have more difficulty in representing individual words (such as, e.g., homographs with unrelated senses and phrasal verbs) because of their different usage distributions.

using such words in the alignment dictionary may undermine the mapping (Aldarmaki and Diab 2019).


As mentioned by Amsili and Seminck (2017), association with arbitrary proper names (e.g., Jane) is irrelevant.

to make our baseline stronger, we sometimes represented a given answer by a word that is not part of this answer but that is syntactically or semantically associated with it in the text.


Our data collection process also reveals that even in a domain as critically important as medical news, only a small fraction of news articles (24.6%) include a complete citation and a link to the original research.

the presented task has utility in medical factchecking, identifying health-related misinformation, and assessing some empirically verifiable aspects of health news reporting.


In spite of recent advances, the transformerbased models are large, and using them to compare each query with each document is computationally expensive even for a small corpus.

the two-stage approach remains a prudent choice.


This challenge is especially acute during public health crises like the current COVID-19 pandemic, due to the extremely fast rate at which new findings are reported and the risks associated with making decisions based on outdated or incomplete information.

there is a need for automated tools to assist researchers and the public in evaluating the veracity of scientific claims.


We examined the rationales that the SCIFACT-trained model identified but the FEVER-trained model missed, and found that they generally contain sciencespecific vocabulary.

training on additional out-of-domain data provides little benefit.


Vote-based ideal points, which only model binary values, cannot capture this nuance in his opinion.

sanders' vote-based ideal point is pulled to the right.


It is known that traditional frequency measures, such as the mean (an estimator of location) and standard deviation (an estimator of scale) are not robust to outliers: a single frequency burst can move them out of bounds.

the field of robust statistic has introduced several robust estimators (Rousseeuw and Croux, 1993).


Unlike the topical variation, which can be observed from the keywords using an unsupervised approach, non-topical variation in genres is expressed via stylistic features, which are harder to interpret (Biber, 1995).

this requires a supervised approach to determine the genre categories.


On the contrary, we consider the different task where the questions are single-turn and have no context.

a novel challenge arises in retrieving accurate support sets without conversationbased context information.


The accumulated general knowledge acquired during meta-learning enables the model to generalize over varied tasks instead of fitting the distribution of data points from a single task.

the tasks generated from tiny (less than 1%) portion of the training data are sufficient for meta learner to acquire the general knowledge.


We thus learn a model that can subsequently adapt the initial parameters to each new given question and specialize the adapted parameters to the particular domain of the new questions.

with the help of the adapted parameters, MRL-CQA can answer each new question more precisely than PG.


For example, only providing the prediction to medical tasks may not be enough, and providing the associated reasons is more crucial for the practical applications.

there has been increasing attempts that focus on interpretability or explainability of the machine-learned models.


This paper focuses on zero-shot rationalization by transferring knowledge from question answering.

three datasets are used in the experiments, where a QA dataset, SQuAD 2.0, is utilized for the transfer purpose and two benchmark rationalization datasets, BeerReview and MovieReview, are used for evaluating the performance of zero-shot rationalization for the proposed method.


Even if LMs satisfy the criteria described in 3.3, there is no exact guarantee that LM scores will reflect the effectiveness of human processing of specific constructions in general.

there seems to be a danger of confusing LM artifacts with language facts.


However, the overwhelming number of newly-sprung news makes it difficult for users to find their interested content (Wu et al., 2019c).

personalized news recommendation becomes an important technology to online reading experience (IJntema et al., 2010).


In this way, we directly derive scores of long answers from its paragraph-level representations and obtain scores of short answers from the start and end positions on the token-level representations.

the long and short answer selection tasks can be trained jointly to promote each other.


Generally speaking, a document can be decomposed to a list of paragraphs, which can be further decomposed to lists of sentences and lists of tokens.

it is straightforward to treat the document structure as a tree, which has four types of nodes, namely token nodes, sentence nodes, paragraph nodes, and a document node.


The reason is that, for instances with no answer or no apparent answers, fine-grained information is more crucial.

using the score of short answer spans might be more accurate than the long answer score from paragraph nodes, which are coarse-grained.


To create a large semantic tagging resource for English words, we apply POS-aware Prediction with k = 3 for all words in the vocabulary of the Sketch Engine POS-specific word embeddings.

we obtain semantic tag vectors for 6,143,073 word forms from their English vocabulary.


It is natural to think that in real scenarios, misspelled words will appear at a much lower rate than in this test.

this result can be seen as a kind of lower-bound estimator for performance on Natural Noise in real scenarios.


In all the experiments, each model was trained/fine-tuned on the original SQuAD v1.1 training set, and tested on each one of the generated adversarial datasets.

we see that all models are affected by these adversarial samples, but also found that some adversaries are model-specific because they do not affect all models as much as they affect the model they are targeting.


However, while the sentence-level biLSTM augments sentence representations with the local context, they may be still unaware of the main topic.

we compute element-wise products and differences between the document encoding and a sentence encoding to measure their correlations, and further concatenate the products and differences with the sentence encoding to obtain the final sentence representation that is used for predicting its sentence type.


In addition, news articles are known to follow inverted pyramid (Bell, 1998) or other commonly used styles where the output labels are not independent.

we also use a linear chain CRF (Lafferty et al., 2001) layer on the output scores of the local classifier to model dependence among discourse labels.


Loneliness and solitude have also been shown to play a role in the adaptive fitness of our species (Hawkley and Capitanio, 2015;Larson, 1990).

loneliness and solitude are starting to receive substantial amounts of attention from the medical and psychological research.


A tweet may include the term loneliness, lonely, or solitude and yet may not be relevant to the state of being alone.

we manually examined a small sample of SOLO to determine the percentage of relevant tweets.


We have shown that by using the search terms loneliness, lonely, and solitude we can collect voluminous corpora of tweets highly related to the state of being alone.

this search strategy over the Twitter stream can be used to monitor the positive and negative aspects of being alone and their relation to well-being over the entire population across time, geographical regions, and demographic groups.


A register sample of the dataset available for public download is shown in Figure 2.

to form the database, the third section was downloaded from January 1998 to January 2018 into a PDF format and converted to a text format.


Words extracted by the methods described above can contain undesired words whose meanings are far from the target OOV word.

we calculate more accurate similarity scores for the extracted words through a neural network surface encoder.


A FLEECE merger refers to a process in which what used to be different vowels in words such as meat, meet, piece, peace converged over time, putting all of them in one lexical set, FLEECE, in most varieties of English.

meet and meat will be homophones in the accents that have gone through the FLEECE merger, and a minimal pair in the ac cents that have not.


PrOnto has been built via standard minimization principles from the ontology engineering literature (Brank et al., 2005;Bandeira et al., 2016).

PrOnto does not truly fit for legal reasoning: PrOnto only defines the main concepts involved in the GDPR and their inter-relations, but it lacks another component necessary for legal reasoning: defeasibility (cf. (Casini et al., 2015))


For the most part, legal interpretations will be figured out later; in such cases, it is generally the role of jurisprudence to rule out the correct application of the provisions (case law), but in some cases legislation will be amended in order to account for new exceptions.

it seems there is no need to introduce more complex defeasible schema for modeling legislation.


Variables in AMR graphs are only necessary to indicate co-referring nodes and they do not carry any semantic information by themselves.

aMR graphs are first converted into aMR trees by removing variables and duplicating the co-referring nodes.


There is, however, a drawback of BPE in its deterministic nature: it splits words into unique subword sequences, which means that for each word a model observes only one segmentation.

a model is likely not to reach its full potential in exploiting morphology, learning the compositionality of words and being robust to segmentation errors.


Since BPE-dropout produces more fine-grained segmentation, sentences segmented with BPEdropout are longer; distribution of sentence lengths are shown in Figure 4 (a) (with p = 0.1, on average about 1.25 times longer).

there is a potential danger that models trained with BPEdropout may tend to use more fine-grained segmentation in inference and hence to slow inference down.


Taking advantage of the availability of Zahran-WE (Zahran et al., 2015), we evaluated word distribution in the semantic space vs. WE distribution.

in our basic system, we replaced the semantic space vector words by Zahran-WE (from the CBOW and SkipGram models) and calculated the correlation on the AR-ASAG Dataset.


The comparison between 3 different score thresholds demonstrates the lower the perplexity, the better the generation quality.

the final F1 score becomes higher with the better generated tokens.


For models with diverse architectures, we expect rationales to be diverse as well and to cause low consistency.

we focus on a set of models with the same architecture, trained from different random seeds as well as the same architecture, but with randomly initialized weights.


Our method has the advantage of being able to account for non-linearities in the decision boundary that a local approach such as LIME cannot handle, albeit at a cost of higher computational complexity (a similar point was made in (Blaas et al., 2020) for Gaussian processes).

we are able to discover words that our framework views as important, but LIME does not, and vice versa.


Unfortunately, sensor metadata in different buildings often follows distinct naming conventions.

learning a tagger currently requires extensive annotations on a per building basis.


As there usually exist buildings that are already tagged, leveraging this information could potentially expedite the tagging process in a new building.

in this paper, we seek to answer the following question: Can we learn a sensor metadata tagger for a new building based on its raw metadata and some existing fully annotated building(s)?


We observe that some k-mers (i.e., substrings of length-k) (Compeau et al., 2011) express the same meaning regardless of buildings, e.g., "T", "tmp", or "temp" almost always appear in sensor names related to temperature.

we propose to leverage such meaningful k-mers to complement the representation produced by the language model (i.e., z i defined in Eqq. (1)) as “word”-level information.


There are only 4 classes in building C that appear in either building A or B, so there is not much difference between chunking and tagging.

we only evaluate chunking when training models based on building A and B and testing them on building C. We evaluate the performance of SeNsER with regard to chunking and tagging using the precision, recall, and F 1 scores, similar to NER tasks.


Moreover, regex for tagging patterns cannot transfer across buildings, which is our goal in this work.

regex is neither an economical nor scalable solution.


The higher the value of uit−1 the more likely there was a recent mention of the entity being tracked by the cell.

uit−1 provides an alternative to distance-based features commonly used in pairwise scores for spans (Lee et al., 2017).


Second, the discourse element of a specific sentence depends on context.

considering individual sentences only would have difficulties in identifying discourse elements.


Self-attention ignores word order in a sentence.

position representations are developed to cooperate with self-attention.


This is consistent with the experimental results on the Chinese dataset.

the effectiveness of the SPE and ISA can be verified on both the Chinese and the English datasets.


From a natural language processing (NLP) perspective, treatments of hate speech focus mainly on the problem of identification (Fortuna and Nunes, 2018).

given a span of text, the task is to identify whether it is an instance of hate speech or not.


A word sequence can be regarded as a thin graph.

it can be used to replace the word lattice as the input of GMN.


Compared with the tiny graph, the overall lattice has more noisy nodes (i.e. invalid words in the corresponding sentence).

we think it is reasonable that the performance of tiny lattice (PKU+JIEBA) is comparable to the performance of the overall lattice (Lattice).


For subtasks AE and OE, the key features for determining the existence of aspect and opinion terms are the representations of the original and adjacent words.

we construct two encoders to extract local AE-oriented features X A and OE-oriented features X O : For subtask SC, the process of feature generation is different from that in AE/OE.


Although SC needs to assign a polarity to every word, we know the ground truth aspect terms in AE during the training process.

we directly use the ground truth tag sequence Ya of AE to refine the labeling process in SC.


Although fixing the random seed may help replicating the exact same values, it hides an important aspect of most machine learning systems: the variation due to random initialization and different training-test splits.

we repeat each of their experiments 10 times, and allow the random seed to differ in each run.


An explanation that was given by the authors of why the classifier predictions were worse for the German texts than for the Italian and Czech was that for German it was a five-class problem, whereas for Italian and Czech only a three-class problem.

in our experiment, three experiment settings are carried out.


Since all texts are written by students taking the Cambridge First exam, the proficiency levels of the authors are less heterogeneous than in the MERLIN corpus, where texts of all CEFR levels are included.

the models and features may not be able to predict the fine-grained scores of a more homogeneous dataset with regard to the levels of the writers.


However, the performance of the dependency context turned out to be always in the middle between smaller and larger linear windows, and we found nothing notable.

the following analysis only focuses on the results of the linear context window.


Specifically, we first empirically study the functionalities of the encoder and the decoder in NAT models, and find that the encoder takes a more important role than the decoder regarding the translation quality.

we propose to train the encoder more rigorously by masking the encoder input while training.


The graph learning mechanism will be performed separately for claim-based and evidencebased graph.

we denote H c and H e as the representations of all nodes in claim-based graph and evidence-based graphs, respectively.


Prior to the advent of neural approaches, generation systems typically separated content selection (what to say) from surface realization (how to say it) (Reiter and Dale, 1997)

many generation datasets only focused on the latter stage (Wen et al., 2015;Gardent et al., 2017b).


Due to the complexity of the task, s decontext may still have grammatical errors, even if annotators were instructed to fix grammar.

a second set of annotators were asked to further correct the sentence and were shown the table with highlighted cells as additional context.


This task is different from what the annotators perform, since they are provided a starting sentence requiring revision.

the task is more challenging, as the model must generate a new sentence instead of revising an existing one.


The original BERT model is pre-trained with both Wikipedia and the Books corpus (Zhu et al., 2015), the former of which contains our (unrevised) test targets.

we also pre-train a version of BERT on the Books corpus only, which we consider a more correct baseline.


All speeches in an assembly were transcribed into a local assembly minutes based on the local autonomy law.

the local assembly minutes form an extremely large amount of text data.


All speeches in an assembly are transcribed into a local assembly minutes based on the local autonomy law.

the local assembly minutes result in an extremely large amount of text data.


Generally speaking we mark locations that are named, nominal, or pronominal.

there are six principal types and the implicit "O" type, which indicates unmarked (or Other).


Our proposed optimization methods introduce a trust-region-type regularization (Conn et al., 2000) at each iteration, and then update the model only within a small neighborhood of the previous iterate.

they can effectively prevent aggressive updating and stabilize the finetuning process.


Compared to CGMH, our method can not only use the gradient norm to locate the editing position more accurately but also our method can use the gradient to fine-tune the inserted or replaced words.

our gradientguided method can achieve a better result than CGMH.


We recover the dropped pronoun as one of 17 types pronouns pre-defined in (Yang et al., 2015), which include five types of abstract pronouns corresponding to the non-anaphoric pronouns.

traditional rulebased pronoun resolution methods are not suitable for recovering dropped pronouns.


These results also indicate that decreasing the number of words to be arranged does not guarantee higher agreement, and being presented with a semantically clusterable bigger set of words (like the ones produced in Phase 1) may be preferable to imposing an arbitrary size limit on the classes.

of the difficulty of some verb sets, the inter-annotator agreement scores for some classes show low positive correlation.


As a consequence of the difficulty of some verb sets, the inter-annotator agreement scores for some classes show low positive correlation.

evaluation of representation models would best be focused on classes with higher inter-annotator agreement and consequently clearer semantics.


Moreover, the manual process also poses problems related to the subjective notions of what constitutes hate-speech and offensive language, which might result in misuse of this process to silence minorities and to suppress criticism of official policies, political opposition, or religious beliefs.

an automated system to detect hate speech and offensive language is inevitable.


They also highlighted the fact that different features are important for each hate-speech label, all of which cannot be captured by a single model.

ensemble methods have been used by studies such as Park and Fung (2017), who have used a hybrid-Convolutional Neural Network (CNN) which combines word and character level CNN, Pitsilis et al. (2018), who have used an ensemble of Long Short-term Memory (LSTM) classifiers with majority voting and confidence based aggregation, and Mahata et al. (2019), who used an ensemble of CNN and LSTM based classifiers to capture both salient local information and long term contexts


However, ensembles are carefully selected task-specific combinations which might not generalize well and are computationally expensive.

a single model with a greater robustness and generalization is desirable.


The highest performance of RomUrEm can be attributed to the fact that it is trained on tweets having both random and hateful content, while the other embeddings are trained on common texts.

to make a fair comparison, we perform fine-tuning for all embeddings on RUHSOLD dataset in order to perform transfer learning from one domain to the other.


However, it is relatively an easier task as compared to fine-grained classification.

more concrete conclusions can be drawn by analyzing the results on fine-grained classification task.


Previous work has shown that non-AAVE speakers often fail to demonstrate comprehension of AAVE speech, and we acknowledge that such misunderstandings may influence the intent-equivalence of our dataset (Jones et al., 2019).

to determine the semantic validity of the translations, we asked annotators who selfidentified as native AAVE speakers and/or codeswitchers to verify whether translated SAE phrases preserved the meaning of original AAVE phrases.


Pretrained sentence encoders have become particularly popular due to the success of training models for downstream tasks on top of their learned representations, greatly improving the results compared to training from scratch, especially in low-data regimes (see Table 1).

we also probe the usefulness of ConveRT encodings for transfer learning in the intent classification task: the model must classify the user's utterance into one of several predefined classes, that is, intents (e.g., within e-banking intents can be card lost or replace card).


Only 5.9% of the words in the Switchboard are disfluent, and BERT is only trained on fluent texts such as books and Wikipedia, so the baseline parser may be starved of disfluent training examples.

self-training on a corpus of conversational speech may compensate for the scarcity of disfluent gold data.


However, these methods are usually designed for written text, so they do not consider the properties of spoken language.

this paper aims at generalizing the idea of language model pre-training to lattices generated by recognition systems.


This task also demands BERT to distinguish the characteristic of source and target domain for better knowledge transfer.

we propose BERT posttraining which takes BERT's pre-trained weights as the initialization for basic language understanding and adapts BERT by novel self-supervised pretrained tasks: domain-distinguish task and target domain masked language model.


This ability will enhance the domain discriminator, which will help to find more complicated domain specific features and get abandoned by adversarial training.

we further decrease the divergence between the domains, which is quantitatively measured by A-distance in Sec 4.6.


With the advent of NMT models, we have seen an increase in the quality of translation systems.

a random sample of translations might have few examples with lowquality scores.


An Unsupervised Alternative A heuristic-based, albeit lower-coverage, alternative to manual alignment is to use questions' mentions of column names.

we use automatically-generated exact-match features in our baseline models for comparison in our experiments.


Our oracle studies also show that there is large unrealized further potential for our annotations.

it remains an exciting challenge for future research to use our lexical alignment annotations more effectively.


One problem with this is that the reference text often contains extra information that is not presented in the table because human beings have external knowledge beyond the input table when writing the text, or it even misses some important information in the table (Dhingra et al., 2019) due to the noise from the dataset collection process.

unconstrained training with such mis-matching information usually leads to hallucinated words or phrases in generated texts, making them unfaithful to the table and thus harmful in practical uses.


The focus of our work is to build a powerful encoder to incorporate the information from other modality.

we will first begin with an introduction to the incorpo-ration method.


Paragraphlevel models select multiple answer spans, one for each paragraph, to form a possible outcome.

multiple A-consistent answer spans can occur in a single outcome, as long as they are in different paragraphs.


We attribute this to the fact that correct answers often have multiple relevant mentions for TriviaQA (also see §5.6), whereas for Narra-tiveQA this is rarely the case.

inference with Sum in NarrativeQA could potentially boost the probability of irrelevant frequent strings.


Very few researches on copyable generative models pay attention to handle external knowledge, while in knowledge-driven conversation, the description words from knowledge are usually an important component of dialog response.

we leverage a knowledge-aware pointer network upon recurrent knowledge interactive decoder, which integrates the Seq2seq model and pointer networks containing two pointers that refer to utterance attention distribution and knowledge attention distribution.


Using the constrained inference net with the unconstrained decoder will bias the decoder to use the anchor words in the aligned sentences – the model is not required to do this, but variational learning will pull the inference network and true model posterior towards each other (i.e. the ELBO objective pressures them to agree).

if the inference net is constrained, but the decoder is not, learning will try to find a weakly constrained decoder to match the approximate posterior.


On analyzing the argmax outputs from the inference network of the trained LAP-CINF-UDEC model, we find that 42% of the predicted anchor words are nouns, 39% of them are verbs, and 11% are adjectives, compared to 58%, 33% and 6% respectively for the RAKE extracted keywords for the ROC data.

the inference network learned along-with the LAP-CINF-UDEC model has higher preference for verbs and adjectives compared to the RAKE algorithm.


The intuition is based on the observation that RNN classifier tends to outperform CNN classifier on longer sequences, while CNN classifier has the ability to capture local context information within a shorter sequence.

for longer sequences we will assign higher weight to the predictions from RNN.


Human languages evolve to communicate about real-world events.

understanding events plays a critical role in natural language understanding (NLU).