Notice that most emotions are relatively independent, which means the way we express certain emotion is pretty different from other emotions.

for an unseen class we can hardly find a similar class in the seen class set.


The design conditions of Marvin and Linzen (2018) and our cross-linguistic replications rule out the possibility of memorizing the training data or relying on statistical correlations/token collocations.

our findings indicate that LSTM language models can distinguish grammatical from ungrammatical subject-verb agreement dependencies with considerable overall accuracy across languages, but their accuracy declines on some constructions (in particular, center-embedded clauses).


Reason (2) is that many of the nouns and verbs in the direct translation of the evaluation sets do not appear in the language models' vocabularies.

some nouns or focus verbs would effectively be <unk>s if left in, rendering that particular example unusable.


In our model, legislator embeddings interact with embedding representations of Donald Trump himself, constructed from a neural network using the text (and timing) of his tweets during the same time frame.

our model leverages the text feature extraction capabilities of neural networks and incorporates the legislator sentiment in tweets about Trump as well as the strategic decision about whether and when to tweet about him.


A key limitation of initial methods for estimating ideal points was the inability to perform out-ofsample predictions.

they could not be used to predict votes on new legislation.


In order to use our model to generate diverse disfluent sentences, we experimented with different variants of PG and found PG-NC-AD-ID produced better performances.

we finally chose PG-NC-AD-ID and the heuristic Planner applying the position choosing heuristic described in Section 3.2.1, since the model-based Planner always chose certain most probable positions, and generated less diverse disfluent sentences, which did not work well as augmented data GPT2 was used to replace the LSTM decoder and trained on a partial pretraining dataset to alleviate the domain gap.


To effectively perform CWS and POS tagging, combining them into a joint task is proved to have better performance than separately conducting the two tasks in a sequence (Ng and Low, 2004).

many studies were proposed in the past decade for joint CWS and POS tagging (Jiang et al., 2008, 2009; Sun, 2011; Zeng et al., 2013; Zheng et al., 2013; Kurita et al., 2017; Shao et al., 2017; Zhang et al., 2018).


The reason could be straightforward: the output of kvMN is built upon value (knowledge) embeddings and therefore information from key (context feature) embeddings does not directly contribute to it other than providing weights for the value.

kvMN acts in a similar yet inferior 12 way of setting (3) where only knowledge is used.


Note that the original PTB does not contain dependency annotations.

we convert them into Universal Dependencies using Stanford CoreNLP.


Since LIGHT-LS cannot rank high-frequency words at the top, it often outputs the target words as is.

17 LIGHT-LS has a low changed proportion.


As discussed in section 4.2, SOC eliminates the compositional effects of a given word or phrase.

regularizing SOC explanations does not prohibit the model from utilizing contextual information related to group identifiers.


In line with Dacrema et al. (2019) we only consider experimental results reproducible if the available source code requires only minimal modifications to work correctly.

we do not replicate the ablation study that is reported on in Tables 5 to 8.


An important feature of our method is that while the model explicitly takes advantage of n-gram information, the model only outputs character-level encodings that is consistent with BERT.


Note that ZEN employs a different method to incorporate word (n-gram) information, which could be complementary to some other previous approaches.

it is potentially beneficial to combine it with other character-encoding approaches.


We observe that the gold justification contains some sentences that are not relevant to the fact check, and the extractive summary is fooled to select explanation sentences that are close to the gold summary.

the explanation does not provide enough information about the chosen veracity label.


We use the LIAR dataset to train such models, which contains fact checked single-sentence claims that already contain professional justifications.

we make an initial step towards automating the generation of professional fact checking justifications.


Additionally, rather than creating new data from scratch, contrast sets augment existing test examples to fill in systematic gaps.

contrast sets often require less effort to create, and they remain grounded in the original data distribution of some training set.


Intuitively, fine-tuning RoBERTa on an intermediate task can cause the model to forget some of its ability to perform the MLM task.

a future direction for potential improvement for intermediate-task training may be integrating the MLM objective into intermediate-task training or bounding network parameter changes to reduce catastrophic forgetting (Kirkpatrick et al., 2016; Chen et al., 2019).


Retrieving at-least one helpful sentence is crucial to obtain contextual information for the To-Do item.

we evaluate our approaches based on the proportion of emails where at-least one helpful sentence is present in the top K retrieved sentences.


In this work, we demonstrate that it is possible to achieve SoTA results by cluster ranking alone, i.e. by linking mentions directly to the entities.

our model is less complex than the existing entitylevel models (Lee et al., 2018;Kantor and Globerson, 2019) using similar mention representations.


First of all, the current SoTA mention-ranking systems tend to be hybrids, using entity-level features alongside mention-pair features.

such models are usually more complex than pure mention ranking models, and substantially increase the number of trainable parameters.


CEFR levels are also used in classroom language teaching to differentiate between different learner groups.

one can have a Swedish class for B1 learners, which presupposes that learners taking the class have mastered all or most of the skills of the lower levels A1 and A2 and should have mastered all or most skills introduced at B1 after having finished the class.


To this aim, we start from one language pair, for example French/Swedish, and for each entry we look up possible translations in the third language, English in this case, from the other two resources.

if we start with the French/Swedish list, we retrieve English translations from the aligned English/Swedish and English/French lists.


This is due to the fact that we only perform a single-step translation look-up, and that some translations might only be reachable under certain circumstances.

in a second step, we remove partial entries which are covered by more complete entries.


Since each hypothesis starts with a different assumption and makes a (mathematically) different claim, it can only be tested with a certain set of statistical methods.

nLP practitioners ought to define their target hypothesis before choosing an assessment method.


A large p-value implies that the data could easily have been observed under the null-hypothesis.

a lower p-value is used as evidence towards rejecting the null-hypothesis.


The more answers to a question there are, the more likely it is that any other answer is equal to one of the expected answers.

the higher number of answers in SQuAD1.1 contributes to the higher human performance compared to FQuAD1.1 regarding the exact match metric.


We argue that product attributes and values are highly correlated, e.g., it will be easier to extract the values on condition that the product attributes are given.

we jointly model the attribute prediction and value extraction tasks from multiple aspects towards the interactions between attributes and values.


The model proposed along with the MAE dataset takes textual product descriptions, visual information, and product attributes as input and treats the attribute value extraction task  as predicting the value for a given product attribute.

we compare our M-JAVE model with the MAE-model only on the value extraction task.


A pre-trained language model may not handle this issue well, as it cannot identify the TCS dimensions in temporal mentions and effectively learn from them.

it cannot generalize well to similar events without explicit temporal mentions.


Such temporal arguments are usually composed of a duration phrase and a numerical head (e.g., "four times per") indicating the frequency within the duration (e.g., "week").

we check for multiple keywords that indicate the start of a frequency expression, including "every," "per," "once," .


Moreover, all these reward approaches are fixed and do not change over training, and all the metrics may not be important over every stage of the training.

it might be useful to consider using a dynamic combination of metrics, which rewards to use early vs. later, or which rewards might be useful to come back later in training, and consider the context of the full history of rewards, as well as the models current state and the nature of the metric.


Past work has reported great variation in how human evaluations are done (van der Lee et al., 2019).

we begin with a meta-analysis of a subset of human evaluation experiments from EMNLP 2019, which we then use as the basis for claims about the power of human evaluations in NLP more generally.


For real speech inter speaker comparison (last row), we calculated DTW for sentences uttered by all speakers (4 common sentences, common).

for each speaker we compared the time series with all time series of the other speakers uttering the same sentence once, or more than once, and calculated DTW for each pair of time series.


This variety is driven by many factors such as topics, individuals, settings, and even differences between spoken and written text.

it is difficult to scale traditional word-level debiasing approaches (which involve bias-attribute words such as man, woman) (Bolukbasi et al., 2016) to sentences.


On the other hand, the potential number of sentences are unbounded which makes it harder to precisely characterize the sentences in which bias is present or absent.

it is not trivial to directly convert these words to sentences to obtain a representation from pretrained sentence encoders.


Note that even if the magnitudes of sentence representations are not normalized, the debiased representations are still pointing in directions orthogonal to the bias subspace.

skipping the equalize step still results in debiased sentence representations as measured by our definition of bias.


For BERT fine-tuned on QNLI, using a larger number of domains reduces the variance in effect size and improves stability of the algorithm.

it is important to use a large variety of templates across different domains.


Occasionally new words emerge, and every now and then some words fall out of favour so much that they may be considered extinct.

it is not surprising that prominent thinkers of the past, including Charles Darwin, have argued that just as there is a natural selection in the plant and animal species over time, there is a natural selection of words over time (Darwin, 1968;Darwin, 2003).


In this analogy, word type maps to a species, and word frequency maps to the species population.

the creation and use of a new word is akin to the birth of a new species, the wide-spread use of a word is akin to the thriving of a species, and the complete lack of use of a word is akin to the extinction of a species.


To smooth frequencies, we aggregate frequencies across ten-year spans.

the entry for a word w and 1809, includes the total number of times w occurred in books from 1800 up to and including 1809; the entry for 1810, includes the total number of times w occurred in books from 1801 up to and including 1810; and so on.


However, while accuracy in emulated low-resource scenarios is over 50% for all languages, for the truly lowresource languages Popoluca and Tepehua, our best model only obtains 37.4% and 28.4% accuracy, respectively.

we conclude that canonical segmentation is still a challenging task for low-resource languages.


We use BPE (Sennrich et al., 2016) for subword segmentation because it generates the merge operations in a deterministic order.

a vocabulary based on a smaller number of merges is a subset of a vocabulary based on more merges estimated from the same training data.


BLI accuracy assumes that all test words are equally important, but the importance of a word depends on the downstream task; e.g., "the" is irrelevant in document classification but important in dependency parsing.

future work should focus on downstream tasks instead of BLI.


In hybrid clustering, we use Exchange for the clustering part and Brown for building the hierarchical structure.

for a hybrid clustering, one defines a set number of classes c, runs Exchange over the corpus to compute the c classes, and then runs Brown over the flat clustering.


The vast majority of the final AMI is achieved within three or four iterations.

stopping Exchange after three iterations results in a three-fold speedup compared to suggestions in the literature (Uszkoreit and Brants, 2008;Martin et al., 1998).


R-MeN considers each triple as a sequence of 3 input vectors that recurrently interact with a memory using a transformer self-attention mechanism.

r-MeN encodes new information from interactions between the memory and each input vector to return a corresponding vector.


Thus R-MeN encodes new information from interactions between the memory and each input vector to return a corresponding vector.

r-MeN feeds these 3 returned vectors to a convolutional neural network-based decoder to produce a scalar score for the triple.


Knowledge graphs (KGs) -representing the genuine relationships among entities in the form of triples (subject, relation, object) denoted as (s, r, o) -are often insufficient for knowledge presentation due to the lack of many valid triples (West et al., 2014).

research work has been focusing on inferring whether a new triple missed in KGs is likely valid or not (Bordes et al., 2011(Bordes et al., , 2013Socher et al., 2013).


Such a growing prevalence of cyberbullying on social media has detrimental societal effects, such as victims may experience lower self-esteem, increased suicidal ideation, and a variety of negative emotional responses (Hinduja and Patchin, 2014).

it has become critically important to be able to detect and prevent cyberbullying text, image, video on social media.


It is also crucial to understand why a media session is detected as cyberbullying.

we study the novel problem of explainable cyberbullying detection that aims at improving detection performance and highlighting explainable comments.


However, these projects are usually not based on open source software and do not comply with a standardized workflow.

their frameworks are neither reusable, nor are their data interoperable.


Laughter has been shown to be very important to human affiliation (Provine, 1996) and solidarity (Hay, 2000).

we detect the number of occurrences of strings indicating laughter (e.g. ‘ha’, ‘lol’) in the user’s response, and use this as a reward.


While it would be ideal if our chatbots could intelligently adapt their conversation style to a new user, in reality most baseline dialog models struggle to maintain topic coherence, even over a few utterances (for an analysis of this effect, see ).

we reward semantic similarity between the user's input and the bot's response, to encourage the bot to stay on topic and produce reasonable answers.


At the same time, it must be noted that out of respect for competition not all companies are disclosing the details of deals or exact amounts.

the true investment amounts (especially in Asia) might be noticeably larger.


There are a greater number of European search companies offering enterprise services and specific languages.

their market presence is more fragmented resulting in a weak position.


Although Chinese is the second largest language in terms of number of users, the total number of European Internet users exceeds it.

"top ten language" as criteria used to identify advantages for data must be carefully evaluated.


With PropBank, our aim is to provide this indispensable contextual information through annotating the argument structure of each verb.

it is evident that PropBank's function is indispensable for processing and properly interpreting Turkish.


To sum up, TRropBank provides unprecedented data on the overall tendencies of Turkish verbs within the framework of transitivity and the portion of idiomatic expressions.

we can infer that TRropBank helps us unveil the properties of argument structure of Turkish verbs in regards to theoretical linguistics in addition to being a valuable asset for NLP solutions.


Many verbs seem to be ambiguous when it comes to this categorization, and they seem to change category depending on context.

once again, intuition had to be relied upon for the classification of these verbs.


The three physical concepts at play in the simulations -friction, collision, and gravity are either a cause or an effect of some collision.

collisions were the most common physical event in the simulations (average = 54 per task) and so we decided to only record collisions.


Although most tasks in NLP require reasoning, the LegalAI tasks are somehow different, as legal reasoning must strictly follow the rules well-defined in law.

combining predefined rules and AI technology is essential to legal reasoning.


The task LJP mainly concerns how to predict the judgment results from both the fact description of a case and the contents of the statutory articles in the Civil Law system.

lJP is an essential and representative task in countries with Civil law system like France, Germany, Japan, and China.


In those countries with the Common Law system like the United States, Canada, and India, judicial decisions are made according to similar and representative cases in the past.

how to identify the most similar case is the primary concern in the judgment of the Common Law system.


This task is slightly different from previous Machine Reading Comprehension (MRC) since the document which contains the correct answer is not directly provided.

how to collect the domain knowledge from massive QA data becomes extremely important.


However, as depicted in Figure 2(b), in MAT-INF, our jointly labeled fashion can guarantee that X remains the same as in a single task and only Y is added.

mATINF provides a fair and ideal stage for exploring multi-task learning, especially auxiliary and multi-task supervision under a single dataset.


We have worked with a limited number of eight journalists, as it is cumbersome and expensive to obtain domain expert annotations, but the crowd-sourcing platforms are generally low-cost and employ a very high number of annotators for their tasks (in our case eighty).

the non-experts have annotated generally more articles and also more articles in common with each other -the latter could make their agreement scores more robust.


Surprisingly, the expert community and the crowd-workers appear to agree on what is biased and what is not at approximately 70% of the time.

in the majority of the articles the individuals in E and NE recognize the evidence in the text to mark it as biased or unbiased.


Interestingly, the STDEV in E vs. All is much lower than in NE vs. All, which can be an indicator of the consistency and reliability of the journalists.

given the lower variance, one might not need as many expert annotators as crowd workers to obtain a high quality media bias detection dataset.


Since the annotator agreement results for the Figure Eight dataset were not satisfactory, we use only the MTurk dataset for the rest of our experiments.

when we refer to the non-expert annotations, only the dataset from MTurk is considered.


However, the huge volume of online conversations produced daily hinders people's capability of finding the information they are interested in.

there is pressing demand for developing a conversation recommendation engine that tracks ongoing conversations and recommends suitable ones to users.


Even if we filter to only keep people who have watched a movie, it is hard to guarantee they can recall all the scenes in the movie.

the factuality check is only done by the person who performed this dialogue.


Intuitively, Chinese NER is correlated with word segmentation, and named entity boundaries are also word boundaries.

the potential boundary information presented by the additional segmentation input can provide better guidance to label each character, which is consistent with the conclusion in (Zhang and Yang, 2018).


With this vetting strategy, the most valuable data is always selected first.

vetting budget is the only factor controlling the vetting procedure.


For the purpose of word alignment, this translation Transformer is used as-is to extract representations of the source and the target sequences, and our alignment technique does not change the parameters of the Transformer.

improvements to the translation system can be expected to directly carry over to alignment quality, and the alignment component does not affect translation output in any way.


Using BPE units instead of words also improved results for GIZA++ (e.g., 20.9% vs. 18.9% for Germanâ†’English in a single direction).

we use the exact same input data for GIZA++ and all our neural approaches.


Most of the language pairs do not contain an adequately sized development set for word alignment experiments.

rather than early stopping, we used a fixed number of updates for each training stage across all languages pairs: 90k for training the translation model, 10k for the alignment layer and 10k for guided alignment training (batch-size: 36k words).


This performance improvement finding resultant of high-order features or structure learning suggests that the same benefits might be observed in SRL.

this paper intends to explore the integration and effect of high-order structures learning in the neural SRL model.


In the current GPU memory conditions, second-order J = 2 is the upper limit that can be explored in practice if without pruning.

we enumerate all three second-order structures as objects of study in SRL, as shown in the left part of Figure 1, namely sibling (sib), co-parents (cop), and grandparent (gp).


Mean field variational inference approximates a true posterior distribution with a factorized variational distribution and tries to iteratively minimize its KL divergence.

we use mean field variational inference approximates to obtain the final arc distribution.


The number of human descriptions per image varies in DIDEC and as we also removed some captions during preprocessing, images do not have an equal number of captions.

we report the average number of captions per image for each split, as well as their range, in Table 5.


Unlike images, actions in videos are also inherently linked to social aspects such as intentions (why the action is taking place), effects (what changes due to the action), and attributes that describe the agent.

for video understanding, such as when captioning videos or when answering questions about videos, one must have an understanding of these commonsense aspects.


However, we want to make sure that these descriptions are also relevant to the video.

we utilize human workers from Amazon Mechanical Turk (AMT) for selecting the most relevant commonsense descriptions.


Moreover, since the video is not used for retrieving these, the commonsense annotations may be out-of-context.

we bring in human annotators to watch the video, read the caption, and then use the set of descriptions from ATOMIC to select the relevant once and to discard the irrelevant or out of context descriptions.


We have 21 types of templates with each template having numerous possibilities for combinations of the slots in the template.

we get 21 types of questions (7 each for intention, effect, and attribute) as shown in Table 6.


A non-biologist may expect 'bleaching' to refer to cleaning, sterilizing, or whitening (Nagel, 2014).

terminology from the engineering domain does not always provide a good starting point for the identification of relevant biological information (Fayemi et al., 2015;Kruiper et al., 2017).


A major issue is that engineers know little biology or few details of plants or animals.

engineers have trouble identifying, selecting and understanding relevant biological information (Vattam and Goel, 2011; Vattam and Goel, 2013b).


However, when analyzing the top predictions, the models are forced to make one decision.

even if the model assigns high scores to both labels of "woman cooking" and "man cooking", it has to pick one as the prediction.


Moreover, the vast majority of these papers do not engage with the relevant literature outside of NLP to ground normative concerns when proposing quantitative techniques for measuring or mitigating "bias."

we fnd that many of these techniques are poorly matched to their motivations, and are not comparable to one another.


Similarly, "controlling images," such as stereotypes of Black women, which are linguistically and visually transmitted through literature, news media, television, and so forth, provide "ideological justifcation" for their continued oppression (Collins, 2000, Chapter 4).

many groups have sought to bring about social changes through changes in language, disrupting patterns of oppression and marginalization via so-called "gender-fair" language (Sczesny et al., 2016;Menegatti and Rubini, 2017), language that is more inclusive to people with disabilities (ADA, 2018), and language that is less dehumanizing (e.g., abandoning the use of the term "illegal" in everyday discourse on immigration in the U.S. (Rosa, 2019)).


In an ideal world, shared task papers would engage with "bias" more critically, but given the nature of shared tasks it is understandable that they do not.

we excluded them from our counts for techniques as well.


Different model configurations may influence model performance greatly.

we conduct an ablation study to validate the effectiveness of each model component used in this work.


In Figure 3, we see that in general, the closer two words occur in sequence, the more they influence each other, leading to correspondingly high DI.

we stratify by the sequential distance of words when we investigate syntactic distance.


In the familiar-scaffold setting, we randomly distribute 1000 occurrences of each scaffold throughout the corpus outside of the rule patterns.

each scaffold is seen often enough to be memorized (see Appendix B).


Empirically, LSTMs encode the most recent noun as the subject of a verb by default, but they are still capable of learning to encode grammatical inflection from the first word in a sequence rather than the most recent (Ravfogel et al., 2019).

while inductive biases inherent to the model play a critical role in the ability of an LSTM to learn effectively, they are neither necessary nor sufficient in determining what the model can learn.


The speed of acquisition of the dependency rule in a familiar-scaffold training environment therefore has an explanation other than hierarchical composition.

in order to confirm our proposed compositional bias, we observe the interactions between scaffold and superstructure (long distance dependency) using DI.


After model training, learned G and E will build a two-way projection between document-topic distribution and document-word distribution.

g and E could be used for topic generation and cluster inference.


They used Word2Vec vectors (Mikolov et al., 2013) of positive words and random negative words as input to the learning objective.

their entity embeddings are aligned with the Word2Vec word embeddings.


The contained data should be as natural as possible, but we gave priority to collecting clear audio-visual dialog data.

we separated the participants into different sound-proof chambers connected by audio-visual communication to record the video clips from the second-person perspective and speech in individual channels.


The purpose of this project is to construct a multimodal dialog corpus that is useful for developing the dialog systems.

we collected clear audio signals and video clips that capture speakers' behavior from the second-person perspective.


While EHRs contain medical billing codes that aim to represent the conditions and treatments patients may have, much of the information is only present in the patient notes.

it is critical to develop robust algorithms to infer patients' conditions and treatments from their written notes.


Their perception is based on extensive conversational experience with real people, rather than using an explicit reference that helps to determine what is correct or wrong.

the main contribution of this effort is to investigate whether dialogue modeling approaches used for dialogue systems can detect anomalous conversations in contrast to normal ones.


The sequence-to-sequence approach is also on par with the others, which is noteworthy because unlike the others, it cannot capture long-term dependencies in dialogues.

long-term context appears to be not necessary for the scoring of these dialogues by the annotators.


This can be very costly in terms of resources.

the research and development of these systems could benefit significantly from an automated approach that can evaluate conversations.


Intuitively, initial scenes of an episode tend to have high similarity with all other scenes in the screenplay, and on their own are not very informative (e.g., the crime, victim, and suspects are introduced but the perpetrator is not yet known).

the undirected version of TEXTRANK tends to favor the first part of the story and the resulting summary consists mainly of initial scenes.


Moreover, models are usually agnostic to the noises or artifacts of the training data, such as reference divergence, making them vulnerable to hallucinations (Kryscinski et al., 2019a;Wiseman et al., 2017;Dhingra et al., 2019).

models can generate texts that are not consistent with the input, yet would likely have reasonable model log-likelihood.


The pretraining prepares BertS2S to be more aware of the domain of the document and less prone to language model vulnerabilities.

bertS2S is more confident in predicting tokens from the document than TranS2S, hence, improving faithfulness.


In fact, for extrinsic hallucinations, even though PTGEN hallucinates less than BERTS2S (63.3% vs. 64.1%), 6.6% of BERTS2S hallucinations were factual, compared to 2.2% of PTGEN.

if we consider factual hallucinations to be valid, this means that even for extrinsic cases, BERTS2S hallucinates the least.


In the first stage, a text representation model ViBERT is trained in the bi-modal sequence-tosequence approach for scene layout generation on COCO.

visual commonsense knowledge like spatial relations will be encoded in ViBERT by the supervision of caption and image layout.


However, the background knowledge required in the commonsense reasoning task, such as spatial relations, causes and effects, scientific facts and social conventions, are usually not explicitly provided by the text.

it is difficult to capture such knowledge solely from the raw texts.


In contrast, the crowd workers tended to select more specific frames instead of comprehensive frames, such as the frame Intentionally act.

the crowdsourcing approach better captured the meaning of the ICSI English frames both cross-culturally and cross-linguistically than those obtained via the transferring approach.


However, in the framing task, the frame candidates for a given target word should be given for crowd workers, with exemplar sentences for each frame.

the crowdsourcing approach is more of a method of extending the existing FrameNet resources than a method of building the initial data.


For many lowresource languages -in particular those ones which are on the brink of extinction -this means they will effectively disappear from research when existing data is of low quality and new data can no longer be obtained.

making existing data ready for re-use -"lifting" it into current research environments -is vital for any cross-linguistic research.


We demonstrate the training strategy in Figure 2.

the model is trained to produce consistent results despite seeing partial views of the input -thereby improving underlying representations.


As in real life, to succeed a betrayal must be a surprise to the victim.

players pride themselves on being able to lie and detect lies.


Diplomacy games are different in that they can last a month. . . and people already play the game for free

we do not want compensation to interfere with what these players already do well: lying.


Even the obituary of the game's inventor explains Diplomacy rewards all manner of mendacity: spying, lying, bribery, rumor mongering, psychological manipulation, outright intimidation, betrayal, vengeance and backstabbing (the use of actual cutlery is discouraged)" (Fox, 2013).

our goal is to have compensation mechanisms that get people to play this game as they normally would, finish their games, and put up with our (slightly) cumbersome interface.


A high compression rate for large-scale language models is difficult to achieve on downstream NLP tasks.

there are few works in exploring and optimizing hardware-friendly model compression techniques for state-of-the-art Transformer-based pre-trained language models, to reduce the weight storage and computation on computer system while maintaining prediction accuracy.


There are two reasons: i) Information compressed in image features can be partially retrieved from neighboring pixels since spatially they share similar and uniform characteristics, whereas syntax and semantics information in deep Transformer in language/text domain are not uniformly characterized; ii) Intuitively, the high-level semantic, syntax, and language understanding capability might be broken when we prune zero or near-zero weights in the latent space.

a high compression rate for large-scale language models is difficult to achieve on downstream NLP tasks.


The results show that the embedding layers and linear weights in transformer layers are sensitive on CoLA and MRPC datasets.

we set the compression ratios of corresponding weights zero to ensure the final accuracy.


(2) Lexical entrainment: Speakers tend to reuse words that were effective in previous mentions (Garrod and Anderson, 1987; Brennan and Clark, 1996) possibly due to priming effects (Pickering and Garrod, 2004).

besides being a challenging problem intriguing from a linguistic and psycholinguistic point of view, computationally modelling the generation of subsequent references can contribute to better user adaptation in dialogue systems and to more natural humancomputer interaction.


The top-scoring utterance in the game round is selected as a referring utterance for i and used as an additional caption for extracting subsequent references in the following game rounds.

of this procedure, for a given dialogue and an image i, we obtain a reference chain made up of the referring utterances-maximum one per round-that refer to i in the dialogue.


In particular, we consider three measures based on n-gram matching: BLEU-2 (Papineni et al., 2002), 8 ROUGE (Lin, 2004), and observed.

some of the candidate images have multimodal representations (if they were already mentioned in the dialogue), while others do not.


Instead, the target values of the subtasks are based on "change scores" which represent only the existence or degree of LSC.

of this simplification, the evaluation methods used in the Unsupervised LSC are incompatible with the WSI and DWSI tasks.


The intuition behind this measure is that a perfect system should predict probability one for the gold sense and zero for any other sense.

the further the predicted probability deviates from one, the higher the error.


In theory the true answer is the emergence year, but in a classification setting it is reasonable to allow some margin of error.

the predictions of an emergence is counted as correct if it falls within the bounds of a 5 year window centered on the true emergence year.


While our proposed model is able to reason very effectively achieving state-of-the-art on the recent datasets, it is able to do so with an assumption of perfect OCR.

it will be pertinent to have better OCR models for chart images.


With the help of those techniques, it is observed that NAR models can match the accuracy of AR models for some tasks (Ren et al., 2019), but the gap still exists for some other tasks (Gu et al., 2017; Chen et al., 2019).

several questions come out naturally: (1) Why the gap still exists for some tasks?


It is non-trivial to directly measure and compare the target token dependency in different modalities (i.e., speech or text) and different conditional source modalities (i.e., speech or text).

we have several considerations in the design of CoMMA: 1) We use masked language modeling in BERT (Devlin et al., 2018) with source condition to train CoMMA, which can help measure the dependency on target context when predicting the current masked token.


Such annotation is difficult to obtain at scale.

recent work has focused on training such models with less supervision: one line of work assumes that only the order of actions happening in the video is given and use this weak supervision to perform action segmentation (Bojanowski et al., 2014;Huang et al., 2016;Ding and Xu, 2018;Chang et al., 2019).


When using EPA to identify the polarity of sentiment, the E, P, and A weights need to be projected to one value before integrating it into a deep learning model.

the EPA values are transformed into one single weight WEPA which is regarded as an affective influence value.


All the results congruently suggest that affective terms with informative sentiment representation can effectively and consistently contribute to model enhancement across different datasets, including both short sentences and long paragraphs, of which the later is more significant.

highlighting the affective terms relevant to sentiment could further improve the attention mechanism.


One significant observation is EPA together serves as the best representation which indicates the orthogonality of each component could be supplementary to each other.

we use EPA with equal weights for our model comparison in Section 4.2..


In these approaches, the overall mutual effect between texts and prices is distributed over the neural network, which makes it difficult to extract this effect and apply it to tasks other than price prediction.

we take a new approach by explicitly describing this mutual effect in terms of a vector.


Precisely, our stock embedding is trained through a binary classification problem, namely, whether a stock price goes up or down in comparison with the previous day's price.

an acquired stock embedding captures the relation between a stock name and a news article even when the article has no direct mention of the stock.


Furthermore, to examine the effect of the data size, we tested different dataset portions: 1 year, 3 years, and the whole dataset.

the experimental variants involved five methods (four comparison + our proposal) and three data sizes, or a total of 15 experiments.


Overall, our model successfully achieved 68.8% accuracy for the R&B dataset, which was significantly better than any of the other four variants.

far, the evaluation on classification has shown the capability of our framework in understanding news articles.


In our set of interventions, we only modify the flow of information within the network, versus the number of trainable parameters.

we do not have confounding factors of varying network capacity.


KonText also supports automatic text selection based on custom proportions of individual attributes (typically text registers).

the created subcorpus can keep the user-specified proportion of the main registers without the need to specify individual texts.


Data sets often contain data from specific domains such as sports-, celebrities-or music-related topics.

annotation methods can only be evaluated on a small part of the KG concerning a specific domain.


Furthermore, annotation of texts based on the abovementioned KGs shows whether KORE 50 adequately evaluates entity linking systems on varying KGs or just covers specific domains.

kORE 50 DYWC is not limited to evaluating NERD systems, but can also be used to analyse kGs in terms of advantages and disadvantages with regard to varying tasks -potentially limited to (or enhanced by) domain-specific knowledge.


In NERD, empty data sets will lead to an increased false positive rate and thus a lower precision of the entity linking-system (Waitelonis et al., 2016).

documents that cannot be annotated, as is often the case with Crunchbase, should be excluded for a more realistic evaluation.


Those untruthful answers may attribute to multiple factors such as misunderstandings of the question, improper expressions during writing, and even intentionally malicious attacks from the competitors (Carmel et al., 2018).

automatically verifying the answer veracity is becoming a demanding need, which can offer a more reliable online shopping environment, for example, by triggering a double-check on the detected doubtful answers.


To predict the veracity of an answer in the QA settings, one can notice that it is insufficient to consider the answer alone since the question text also carries important semantic information for the prediction.

we need to appropriately leverage the question text into the verification process.


The sof tmax() function on the other hand will normalize the score for each evidence sentence.

more important evidence can have larger weight and attach more importance in the subsequent prediction process.


We conjecture that DeClarE treats each claim-evidence pair as one training instance without considering the relations between evidence sentences.

the model can be misled by conflicting evidence sentences and makes random predictions.


Enterprises are the subject in economic activities.

the entities corresponding to enterprises are candidates of the sentiment targets.


Furthermore, the conjunctions always play an important role in distinguishing sentiment polarity conflict situations because people normally emphasizes the part behind the adversative conjunction.

we determinate the sentiment polarity according to the emphasized part.


Note that the probability p(d t |b t ) is omitted as database result d t is deterministically obtained given b t .

the system response can be generated as a three-step process: first predict the belief state b t , then use b t to query the database and obtain d t , finally generate the system response r t based on all the conditions.


Task Subjectivity In some tasks, aggregated annotations always correspond to the correct answer (Brew et al., 2010).

to fully utilize the crowd's knowledge, different approaches have been proposed to aggregate labels, from simply applying majority voting to more sophisticated strategies to assess annotators' reliability (Yang et al., 2018;Srinivasan and Chander, 2019;Rodrigues et al., 2014).


We decided to treat the first, second, and third choices differently as they represent the workers' priorities.

we give the highest weight to the first choices (1.0) and lower weights (0.6) and (0.3) to the second and third choices, respectively.


Font Recall (FR) Less popular fonts could be underrepresented by the models.

we need an evaluation metric that measures the performance of models in learning individual labels.


That is, we wish to make it impossible to predict, for example, a person's gender, from their embedding.

predictions made using these embeddings (such as about profession) will also be independent of these attributes.


Although these times are benchmarked on FB15K only, the relative differences in spe stays constant for the larger datasets, as time to read from/write to memory is negligible.

we are able to train our full model on the Wikidata knowledge graph for 10 epochs in a time of around 10 hours on a system with 64 cpus.


For FB3M and Wikidata, we found these values to be too small.

we increased the values for FB3M to 100.0 for gender and 500.0 for the other three attributes.


In the worst case scenario, outliers introduce a high correlation when there is no association between metric and human scores for the rest of the systems.

future evaluations should also measure correlations after removing outlier systems.


Our system produces de-tokenized cased output after BPE decoding, whereas previous systems produce traditional tokenized lower-cased output.

we lowercase and tokenize our system outputs to have fair comparisons with previous systems.


One possible reason could be the intrinsic difference of linguistic patterns between human conversations and writing text, resulting in a large gap of data distributions (Bao et al., 2019).

pre-training dialogue language models using chit-chat corpora from social media, such as Twitter or Reddit, has been recently investigated, especially for dialogue response generation (Zhang et al., 2019) and retrieval (Henderson et al., 2019b).


We care the most in this paper whether TOD-BERT, a pre-trained language model using aggregated taskoriented corpora, can show any advantage over BERT.

we avoid adding too many additional components on top of its architecture when fine-tuning on each downstream task.


The source URLs, together with the other meta-data were formatted and stored in JSON format making corpus 1 easily reusable and reproducible.

this makes it easy for any interested researcher to process the dataset and use it in modeling epidemiological event extraction or any other related NLP tasks.


Conscious or subconscious gesturing is part of non-verbal communication (Argyle, 2010).

gestures, such as hand gestures, head movements, facial expressions or body posture, which are connected to speech (Kendon, 2004;McNeill, 2015), are called co-speech gestures.


Since the stated data artificially overrepresents positive examples, the model trained on stated data is initially miscalibrated-the points it is uncertain about are actually almost all negative points.

uncertainty sampling initially collects very few additional positive examples.


Existing models are optimized by maximizing the conditional word probabilities of a reference sentence, a common signal for training language models.

these models can learn to produce fluent sentences, but some crucial input concepts and relations may be messed up or even dropped.


For more detail, we remove any abbreviations from a KG node (such as "New York (NY)" is changed to "New York"), before finding the first phrase in the sentence that matches the longest prefix of the node.


This is quite intuitive, because edge labels carry important relational knowledge between the two connected nodes.

discarding these labels will cause loss of significant semantic information.


Also, context vectors are used to predict slot gates, which is essential to be able to trigger the state generator.

using self-supervision to align contextual slot vectors may help get better attention distributions and better slot gate prediction.


Also, each perturbed sample may generate slot values that have different number of words, and maintaining consistency of sequential distributions could be challenging.

we use slot gate distribution and attention distribution as intermediate targets since the former is the first stage for the whole prediction process, and the latter directly influences the copy mechanism.


Since code completion is an important scenario, we would like to test model's ability in predicting the correct token merely based on preceding PL contexts.

we add an additional setting for PL side, where the input includes the complete NL documentation and preceding PL codes.


TC is predominantly used in Taiwan, Hong Kong, and Macau, whereas SC is mainly adopted in mainland China and SC characters are simplified versions of TC characters in terms of strokes and parts.

chinese NLP practitioners apply script converters 1   dataset into their desired language.


Although Hong Kong and Taiwan both use Traditional Chinese, they are stylistically different as the dominant spoken language in HK is Cantonese and in TW is Taiwanese Mandarin.

it is quite essential to test the performance of our algorithms on these two styles.


Orientation compares the extent to which an utterance aims to advance the conversation forwards with the extent to which it looks backwards.

we must somehow quantify how the utterance relates to the subsequent and preceding interaction.


In all previous work, the summary length is weakly controlled by length embeddings or a soft length penalty (Zhou and Rush, 2019;Wang and Lee, 2018;Fevry and Phang, 2018;Baziotis et al., 2019).

the generated summaries by different systems vary considerably in average length, for example, ranging from 9 to 15 on a headline corpus (Section 4.1).


We show that ROUGE F1 is unfortunately sensitive to summary output length, in general favoring models that produce longer summaries.

we argue that controlling the output length should be an integral part of the summarization task and that a fair system comparison can only be conducted between summaries in the same length bracket.


Our model generates a candidate summary in a non-autoregressive fashion, in contrast to the beam search in Zhou and Rush (2019).

we are able to simultaneously consider forward and backward language models, using the geometric average of their perplexities.


Second, our personal observation is that the Wikipedia conventions have gained some popularity among the online community since their first publication.

by opting for the Wikipedia conventions we remain close to other online content.


Yet, existing corpora differ greatly in size, thematic focus, annotation quality, granularity of the underlying conceptual entity representation and the way individual entity classes are defined-even when different corpora cover the same entity classes.

merging all available annotations into one large corpus and regard it as a coherent source of gene/protein annotations (Wang et al., 2009;Wang et al., 2010;Galea et al., 2018) might not be advisable.


Since the original test sets for those domains are scarce (50 reviews each) we decided to use the original out-of-domain training set of 900 reviews for testing purposes and create the new split of development and test sets.

the task consists of 1000 reviews, which is comparable in size to the in-domain test dataset of 1400 reviews.


To increase the difficulty and diversity of the task, we filter out multiple abstracts from the same article.

there is at most one negative pair created from each pair of articles.


Unlike most written languages in the world, the Chinese writing system does not use explicit delimiters (e.g., white space) to separate words in written text.

chinese word segmentation (cWS) conventionally serves as the first step in chinese language processing, especially for many downstream tasks such as text classification (Zeng et al., 2018), question answering (Liu et al., 2018), machine translation (Yang et al., 2018), etc.


Although some studies sidestepped the idea by incorporating contextual n-grams (Pei et al., 2014;Zhou et al., 2017) or word attention (Higashiyama et al., 2019) into the sequence labeling process, they are limited in either concatenating word and character embeddings or requiring a well-defined word lexicon.

it has not been fully explored what would be the best way of representing contextual information such as wordhood features in neural CWS models.


WMSEG provides a general way of integrating wordhood information for CWS, we expect other wordhood measures to play the same role in it.

we test PMI and DLG in our model and compare them with the previous results from AV (see Table 6).


However, while existing abstractive summarization models are optimized to generate summaries that highly overlap with human references (Paulus et al., 2018), this does not guarantee factually correct summaries, as shown in Figure 1.

maintaining factual correctness of the generated summaries remains a critical yet unsolved problem.


Despite these efforts, none of the existing work has focused explicitly on optimizing an abstractive summarization system with a correctness objective.

even state-of-the-art systems trained with ample data still produce summaries with a substantial number of factual errors (Goodrich et al., 2019; Kryscinski et al., 2019a).


Due to the absence of ground truth summaries, the model is not trained to reconstruct the aggregate encoding of reviews, but rather it only learns to reconstruct the encoding of individual reviews.

it may not be able to generate meaningful text when the number of reviews is large.


Different from the existing work that enhances the representation capability of models through neural architecture engineering, we turn to an orthogonal direction that we keep the generation model simple, and optimize the simple structure by learning with auxiliary tasks that encode context understanding.

our model can provide high-quality responses at a low cost.


The challenge requires a model to connect semantically related words and utterances.

we design masked content recovery tasks on both a word level and an utterance level to enhance the self-attention module in terms of awareness of the semantic connections.


The generation model of SNN is just a simple RNN sequence-to-sequence with one layer encoder and one layer decoder.

our model is comparable with SSN in terms of complexity and speed.


In many cases, utterances are just formed by copying a proportion of words from the profiles.

recognizing the semantic connections and the relationship among words in contexts is more critical for the data.


The problem that arises
here is that since many disfluent words are copies
of fluent words, if the same cost is used to align
fluent and disfluent words, the alignment will be
ambiguous (i.e. there will be multiple alignments with the same cost).

to force the aligner to
prefer aligning null (i.e. deletions) for disfluent
words and copy for fluent words, we modify the
alignment weights so the intuitively correct alignment scores better, and so will be chosen by the
alignment algorithm


For each approach we compute a score, which we define as the percentage of examples where the annotator did not correctly identify the machinegenerated sentence.

a higher score implies a better (more natural, human-like) model.


However, we observe that the lower left corner formed by the classes Family, Characteristics and Biographical Sketch is almost symmetric in its confusions, which led us to inspect and classify the types of errors.

we investigated all errors manually and classified them in three main types of errors: errors due to Ambiguity (39%), errors due to wrong Annotation (18%) and errors tagged as Other (42%) where the errors are more difficult to explain (see last column in Table 6).


This work addresses the question of how to automatically structure obituaries.

we acquire a new corpus consisting of 20058 obituaries of which 1008 are annotated.


To avoid overlapping between f a and f c , we set f c as 0 if the cluster is the initial cluster (L6).

f c becomes the consultation such that when f c > 0, the span g x is likely to match the cluster C y , and vice versa.


We note that the sentences comprised within the same cluster define similar contexts for the target word, hence implying that l is very likely to be used with the same meaning across sentences.

we associate each cluster with one of l's meanings and a disambiguation score.


Furthermore, adverse effects may occur due to a combination of drugs.

our guideline instructs to label a span describing effects with all possible drugs, so that correspondences between drugs and effects are many-to-many.


For these agreed sets of labels, we took the longest span attached to the exactly matching sets in order to complement the ambiguity in span identification.

we obtained 677 labelled spans of drug reactions on 169 weblog articles as summarised in Table 5.


As discussed in Sec. 4.2., our annotation dataset consists of challenging examples that (a) have many-to-many correspondences between drugs and drug reactions, (b) require to distinguish drugs related to a specific reaction from other drugs prescribed together, and (c) require to label not only phrases but also sentences or even longer spans.

our dataset is also useful as an advanced dataset for existing aspect-based sentiment analysis and sequential labelling problems.


They are particularly painful for patients who need long-term treatment, such as cancer patients.

information on adverse drug reactions is essential for medical practitioners to prescribe drugs that maximise the therapeutic effects while minimising the adverse drug reactions as much as possible.


However, this did not scale well, as sometimes we received notice of a new wordnet, and it took us months to find time to write a converter.

we decided to try to shift this burden, as far as possible, to the individual wordnet projects.


Some examples were: wordnets with no synsets, senses and synsets having different parts of speech, synsets with no part of speech, examples or definitions consisting of empty strings, different projects interpreting the directions of relations differently, definitions in unknown languages, projects including the PWN definitions, among others.

the new OMW system keeps adding an increasing number of extra checks on the content of uploaded wordnets, which warn the uploader of any problems before any upload is possible.


An example is that traditional Chinese includes about 17k distinct tokens.

it could be expected to suffer from underfitting not only word embedding but also character embedding.


A three-layer transformer network, for example, would be denoted s fs f s f, with the flow of computation moving from input on the left to output on the right.

any string in the regular language (s|f) * defines a valid network that uses the same building blocks as the original transformer.


For two attention mechanisms to be similar, they must put attention on the same parts of the text.

we first define a metric for quantifying the overlap in the words selected by human annotators and by deep learning models.


However, 2 gold paragraphs may not be available in the Fullwiki Setting.

the Fullwiki setting is more challenge which requires to search the entire Wikipedia to find relevant documents.


While Koh and Liang (2017) show that influence functions can be a good approximation even when the convexity assumption is not satisfied (in their case, a CNN for image classification), it is still not obvious that the influence function would work for BERT.

we conduct a sanity check: for each instance in our test set, we by turns remove the most positively influential 10%, the most negatively influential 10%, the least influential (where influence scores are near zero) 10%, and a random 10% of training examples.


When removing a token with the most positive saliency score, we expect the model to be less confident about its current prediction; it could possibly make a different prediction.

we expect to see a most different influence distribution from the original influence result compared to removing the token with median or the most negative saliency score.


We observe that targets and aspects interrelate in subtle ways, often yielding conflicting sentiments.

a naive aggregation of sentiments from aspects and targets treated separately, as in existing sentiment analysis models, impairs performance.


A context word can be associated with one or more targets.

capturing Target-Context relationships is pivotal.


However, the publicly available datasets of MOOC are limited in size with few types of data, which hinders advanced models and novel attempts in related topics.

we present MOOCCube, a large-scale data repository of over 700 MOOC courses, 100k concepts, 8 million student behaviors with an external resource.


Moreover, these datasets only contain a small size of specific entities or relation instances, e.g., prerequisite relation of TutorialBank (Fabbri et al., 2018) only has 794 cases, making it insufficient for advanced models (such as graph neural networks).

we present MOOCCube, a data repository that integrates courses, concepts, student behaviors, relationships, and external resources.


However, existing taxonomies like ConceptNet (Liu and Singh, 2004) or Wiki Taxonomy (Ponzetto and Strube, 2007) cannot be directly applied to course concepts because course concepts are mostly academic terms and the non-academic categories greatly interfere with the quality of taxonomy.

we select a crosslingual term taxonomy from CNCTST 4 as a basis and lead manual annotation to build a serviceable course concept taxonomy for MOOCCube.


However, labeling all possible pairs is infeasible, for 100K concepts may generate over 500 billion candidate pairs.

we lead a distantly supervised annotation in three stages.


Second, we base our experiments on a dataset derived from the Enron corpus (Klimt and Yang, 2004) which consists of email exchanges in an American corporation.

we restrict our attention to the notion of politeness as widely accepted by the speakers of North American English in a formal setting.


In each round, 1.67% of the entire parallel corpus is selected and added into the training corpus.

we ensure the token budget is 20% of the entire parallel corpus in the final round.


For instance, humans have been found to use their knowledge about typical colours of an object when perceiving an instance of that object, in order to compensate for, e.g., perceptually challenging illumination conditions and achieve colour constancy (Mitterer and de Ruiter, 2008;Witzel and Gegenfurtner, 2018).

the visual perception of object colours can be thought of as leveraging top-down knowledge for bottom-up processing of sensory input, in accordance with traditional approaches in psychology (e.g.


We also tried visual features extracted with a neural object recognizer (Simonyan and Zisserman, 2014) which only give a small improvement over colour histograms.

in Section 4, we report results only for RGB histograms, as they are more transparent as representations and do not include any conceptual information on objects.


This component relies only on conceptual information about the object which consists of assignments of objects to object types and colour distributions for object types reflected in the data.

this classifier predicts colour terms given only the object type, which is supposed to mimic the memory colour effect discussed in Section 2.


For CBOs, there is still a clear improvement over BOTTOM-UP, whereas for CNOs the model mostly relies on BOTTOM-UP.

even though the fusion is hard-coded, it achieves the desired flexible pattern for combining the components.


Both context coherence and response fluency (quality metrics) can naturally be captured by metrics based on strong language models like GPT-2 (Radford et al., 2019).

we propose to recruit and fine-tune GPT-2 as a basis of our quality metrics.


We define gender bias in NRE as a difference in model performance when predicting on sentences from male versus female articles.

we need articles written about entities for which we can identify the gender information.


However, to obtain gender information for existing annotated datasets could be costly or impossible.

we elected to create WikiGenderBias with this gender information to be able to detect scenarios like that in Figure  1.


Many of these entities have gender information and their corresponding articles are readily available.

we create our dataset based on sentences extracted from Wikipedia.


Besides, Wikipedia includes more articles written about males than about females.

there are more male instances than female instances in WikiGenderBias as well.


Equality of Opportunity (EoO) was originally proposed to measure and address allocative biases (Hardt et al., 2016).

we examine this metric in the context of relation extraction to better understand how allocative biases can begin to emerge at this stage.


A solution to reduce the computational burden of these networks is to lower numerical precision.

numerical values can be represented using fewer bits (Tang and Kwan, 1993;Marchesi et al., 1993).


A major advantage to delaying quantization is to perform more training steps in the same given amount of time.

when training time is a constraint, a possible strategy is to train a model without quantization, perform more training steps and finally post-quantize the model.


Figures 1b and 1d show, that the accuracy of the negative class drops from 0.95 to 0.88 and that the number of negative samples classified as positive raises from 5% to 12%.

we recommend training BERT models on the balanced data set.


The resulting models achieved both lower accuracy on the negative class.

we recommend training these models on the balanced data set.


Further, due to the limited position index during pre-training, most Transformer-based models have a maximum capacity of input tokens.

they often truncate the length of a document to satisfy the length limitation of the encoder, which may lose some important semantics, especially for long documents.


Topic embedding with masked attention (TEMA): Since a topic is a distribution over tokens in the vocabulary, we use the mixture of token embeddings to represent the corresponding topic embedding.

topics with large proportions for a document can be considered as extra input tokens of the decoder.


Secondly, the Transformer models are getting bigger and bigger (Sanh et al., 2019), resulting in a fact that it is almost impossible to pre-train such a big model on a personal computer.

models with plug-andplay property are attractive to Transformer-based models (Dathathri et al., 2019).


Compared with BertSUM that ignores the subsequent document-tokens, BertSUM+TA is able to reserve these information in some degree, since the topic model extracts global semantics from all tokens in the document.

with the increase of the document length, the improvement produced by TA gets more evident.


Clearly, TA introduces a few parameters, less than 10%.

tA has a friendly memory footprint to the transformer models.


It can be seen that a small number of topics are inadequate to express all the semantics, while too many topics are redundant and introduce more learnable parameters.


Ideally, if we can learn the expressions representing for a certain action or domain and how they compose an utterance for existing intents, then we can learn how to compose utterances for few-shot intents naturally.

we define an intent as a combination of a domain and an action.


We find that MaxProb gives good confidence estimates on in-domain data, but is overconfident on OOD data.

maxProb performs poorly in mixed settings: it does not abstain enough on OOD examples, relative to in-domain examples.


In this paper we want to show, analysing a particular procedure applied for lexico-semantic annotation of BCPM, that in practice such a single choice may consist of a chain of interdependent decisions, and annotators can agree or disagree at every step.

we have to value each such decision separately and then combine the result.


The PCC (Ogrodniczuk et al., 2015), in turn, is randomly selected from the whole NKJP corpus.


Typical MWEs may be compositional (e.g. dawka śmiertelna ‘deadly dose’ is a ‘dose’) or not (e.g. there is not a meaning for centrum ‘centre’ for centrum handlowe ‘shopping centre’, ‘mall’

we decided to allow linguists to optionally annotate elements of MWEs.


Most existing data-driven approaches for Word sense disambiguation (WSD) build word-specific classifiers to predict the right sense of a word instance, which lack the capability to generalize across words and therefore require sufficient sense-annotated data for every word (de Lacalle and Agirre, 2015) in order to disambiguate them well.

the model's performance decreases significantly when there is a lack of training data for a word or some of its senses.


In addition, the sense inventory WordNet (Miller et al., 1990) contains an example for many word senses and the well-composed example (Table 2) can precisely illustrate essential semantic or syntactic constraints in adopting a word sense.

we use the example of a word sense, if available, as a prototype in a regularization component of the shared model for guiding the WSD system to concentrate on most relevant segments of a word context.


The example provided by the sense inventory WordNet (Miller et al., 1990) for each word sense, if available, clearly illustrates essential semantic or syntactic constraints for adopting a word sense in a real sentence.

the shared model measures correspondence between a sense prototype and a word context as well, with both the prototype and context encodings attended by the gloss encoding.


Note that the metric for the STS-B task is Pearson and Spearman scores.

we measured the validation Pearson score instead of validation accuracy for choosing influential attention heads for the STS-B task.


They are what the sender suggests to the receiver of the message and that must be decoded and understood : it is a non-literal or implicit message.

of the several steps that led to our reference corpus, we were able to identify three categories reflecting the intention expressed by the speaker by producing a statement and more precisely a question : opinion, will and doubt.


However, we found that directly optimizing prompts guided by gradients was unstable and often yielded prompts in unnatural English in our preliminary experiments.

we instead resorted to a more straightforward hillclimbing method that starts with an initial prompt, then masks out one token at a time and replaces it with the most probable token conditioned on the other tokens, inspired by the mask-predict decoding algorithm used in non-autoregressive machine translation (Ghazvininejad et al., 2019): 9 where w i is the i-th token in the prompt and t r \ i is the prompt with the i-th token masked out.


Further analysis reveals that about 12.6% of dev questions and 15.7% test ones are considered unanswerable by crowd workers, which is a byproduct of the information-asymmetric setting adopted when the data was collected.

many reference questions could be considered uninformative by our definition, since they might cause the QA model to abstain from answering.


As each sentence pair generates a unique encoding, common optimisations that allow for faster computations cannot be employed.

we exclude this model from our study of the dialogue act clustering task.


As can be seen, BERT-Large significantly outperforms all non-BERT baselines by a big margin, regardless of whether query expansion is used.

only significance tests relative to BERT-Large are shown.


The importance of these knowledge has been verified by tasks at different level, for instance, sentence-level sentiment classification (Taboada et al., 2011;Shin et al., 2017;Lei et al., 2018), aspect-level sentiment classification (Vo and Zhang, 2015;Zeng et al., 2019), opinion extraction (Li and Lam, 2017;Gui et al., 2017;Fan et al., 2019) and so on.

we assume that, by integrating these knowledge into the pre-training process, the learned representation would be more sentimentspecific and appropriate for sentiment analysis.


An aspectsentiment pair refers to the mention of an aspect and its corresponding sentiment word.

a sentiment word with its nearest noun will be considered as an aspect-sentiment pair.


Aspect sentiment pairs reveal more information than sentiment words do.

in order to capture the dependency between aspect and sentiment, an aspectsentiment pair objective is proposed.


The reason why we employ the conventional named semantic roles is that the named semantic roles are intuitively more understandable for humans compared to the numbered roles.

the named roles are expected to be used in queries when language learners or linguists want to extract example sentences that, e.g., contain "Experiencer" arguments in the emotional sense, or "Source" arguments in the moving sense.


However, the composed meaning of the light verb construction, i.e., the chat frame and the verb 'oshaberi-sur-u' (chat-do-PRS) are assigned to the light verb 'shi-ma-su'.

the role sets of the chat frame are applied to the arguments and adjuncts except for 'oshaberi o' (ArgM PRX).


The recipient 'imooto-ni' (sister DAT) must be part of a construction (Goldberg, 1995).

the recipient can be considered as an argument because the recipient appears depending on the create frame.


In the Contacting frame, the recipient role is defined as the Addressee role that indicates a core role (i.e., an essential argument).

both English language resources offer an analysis that is the same as our analysis for the Japanese data.


The results show that the accuracies of semantic role annotation systems are almost the same as the current quality of named semantic roles, for results near to the achievements of previous work.

we can estimate that the quality of the annotated semantic roles are highly promising.


Although these explorations have achieved promising results, they are still limited to those specific KGs whose entities have additional text information.

reasoning over sparse KGs is still an important but not fully resolved problem, and requires a more generalized approach to this problem.


In this task, we give the model more explicit and strong cross-lingual training signals.

the encoder has the ability to perform simple word translation, and the burden on the ST decoder is largely reduced.


The model achieves the best performance when we choose N = 8.

we use this strategy in our main experiments.


The original datasets do not have all mention boundary labels annotated.

in order to evaluate both mention detection and entity disambiguation, we extend previous labels and create new end-to-end question entity-linking datasets, WebQSP EL and GraphQ EL .


Moreover, these results concern the expressive power of Transformers and do not apply to learning and generalization abilities.

transformers' ability to model formal languages requires further investigation.


Instead of binding our vokenizer to a specific pre-training framework (e.g., BERT), we want to enable its extensibility to other frameworks (e.g., RoBERTa).

we introduce a "revokenization" technique to address this limitation.


The vokenization runs at a speed of 100K tokens / second with 4 Titan V100 GPU.

the vokenization of the full Wikipedia is finished in 8 hours.


It also provides information on how the answer can be derived from the question.

the complexity of the questions can be increased based on the supporting knowledge base.


Our model is different from the previous methods since we use pre-trained image and language features and fuse them with KG embeddings to incorporate the external knowledge into the VQA task.

our model does not need additional knowledge annotations or search queries and reduces computational costs.


The answers to these type of questions often are entities out of the main entities in the question and the visual features in the image.

the information extracted from the knowledge graph plays an important role in determining the answer.


The translation workload was divided into 10 batches for each language, which were submitted separately to Gengo.

different parts of the dataset might have been translated by different translators.


The idea of face acts appears quite attractive from a computational standpoint for their potential role in understanding what is "meant" from what is "said" (Grice et al., 1975; Brown et al., 1987; Leech, 2016).

politeness has been widely researched in various domains of language technologies (Walker et al., 1997; Gupta et al., 2007; Wang et al., 2012; Abdul-Mageed and Diab, 2012; Danescu-Niculescu-Mizil et al., 2013) in addition to foundational work in pragmatics and sociolinguistics (Brown et al., 1987; Grice et al., 1975; Leech, 2016).


However, much prior work modeling politeness reduces the problem to a rating task or binary prediction task, separating polite and impolite behavior.

what the models end up learning is mainly overt markers of politeness or rudeness, rather than the underlying indirect strategies for achieving politeness or rudeness through raising or attacking face, even in the indirect case where no overt markers of rudeness or politeness might be explicitly displayed.


Different from the SQuAD-style question answering where there is no specific requirement for the position of the predicted span, in bridging anaphora resolution, an anaphor must appear after its antecedent.

in the inference stage, for each bridging anaphor a, we first identify the position of a in its context c a , then we only predict text spans which appear before a.


This gives us a quasi-bridging example with two adjacent sentences (i.e., s y and s x ) and a bridging link (i.e., justicethe obstruction).

we obtain a large amount of "quasibridging" training data (i.e., around 2.8 million bridging pairs) by applying the method described above to the NYT19 section of the automatically parsed Gigaword corpus.


Last but not least, CASIE has not been comprehensively evaluated with the state-of-the-art ED systems, making it challenging to accurately estimate the difficulty/complexity of the dataset.

in this work, we introduce a novel dataset for cybersecurity ED (called CySecED) that is manually annotated for 30 event types to better characterize the important cyber attacks and vulnerabilities reported in texts.


For target sentiment analysis, both shared information of both sub-tasks and private information of each sub-tasks should be considered.

a shared network is designed to encode shared information between the two sub-tasks, such as semantic and syntactic information of the input sentence.


In addition, shared features generally portray common information between the two sub-tasks, like semantic and syntactic information, which also exist in other NLP tasks.

the prevalent model of Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018), which is a pre-trained bidirectional Transformer encoder that achieves state-of-the-art performances across a variety of NLP tasks, is chosen as the shared network.


Secondly, SPRM achieves 0.66%, 4.25%, and 1.76% absolute gains on three datasets compared to the best SPAN method SPAN-pipeline, indicating the efficacy of the Shared BERT.

sPRM can get better performance with fewer parameters compared to sPAN-pipeline, which employs two separate BERT encoding network for target extraction and target classification, respectively.


Besides, SPRM with BERT-Base outperforms SPAN-pipeline with BERT-Large, which uses almost 5 times the trainable parameters of SPRM with BERT-Base.

the introduction of Shared BERT can not only connect the task of target extraction and target classification to some extent but also reduce the parameter number.


Collocations are numerous in every language, and due to their idiosyncratic nature, they pose considerable problems for non-native speakers.

there is a strong need to include them in dictionaries, and they have received considerable attention in lexicography.


In other words, they simplified both tasks by avoiding the challenging task of identifying the locations of argument components.

their approach cannot be applied in a realistic setting where the input is an unannotated text.


This is largely because argumentative relationships can occur between ACs in non-adjacent sentences.

a deeper semantic understanding of ACs is necessary in order to develop features or heuristics for detecting argumentative relationships between them.


The paraphrased references had only minor changes and consequently only minor impact on the automatic metrics.

we changed the instructions and asked linguists to paraphrase the sentence as much as possible while also suggesting using synonyms and different sentence structures.


We generated our paraphrased reference translation with the instructions to modify the translations as much as possible.

the non-translationese, perhaps more natural, nature of the paraphrased translations make it more demanding to assign an accurate rating.


As mentioned above, relevant entity mentions and their types are only annotated for sentences containing experiment information and neighboring sentences.

we here compute agreement on the detection of entity mention and type assignment on the subset of 90 sentences that both annotators considered as containing experimental information.


Spatial domain queries have several unique properties making them be more challenging for language understanding than common conversational queries, including lexicalsimilar but diverse intents and highly ambiguous words.

a special tailored LU framework for spatial domain queries is necessary.


Considering that entities are quite diverse and ambiguous, and entities with the same name may belong to different tags.

we will make full use of the tag information of each mention and candidate entities to overcome the ambiguity of entities.


While some pre-trained models trained with web texts or Wikipedia are public, our preliminary results indicate that fine-tuning of a pre-trained model trained with the same corpus has a better performance.


When a sentence describing a complex situation is used in a hypothesis, it is difficult to find the entailing premise sentences.

we decided to use only short token sentences as hypothesis sentences.


For example, in our experience, we found that errors in entity linking have an adverse effect on learning relation extraction model.

when creating crowdsourcing data, we decided that using a common source text rather than using different source texts for each task would be helpful for the overall analysis of information extraction.


Second, even the same entity may tagged with different grade and types of entities depending on context and worker.

the final data is created by merging the results of two people's work.


However, there is not a Korean related work there, and it is not easy to gather Korean worker.

we use Crowdworks 1 platform, which is the representative of Korean crowdsourcing companies.


We found that the most useful features were lemmas and part-of-speech tags.

we discarded the rest of features and conducted experiments to find out the optimal window for which lemma and PoS tags features should be added.


During the feature tuning process, we discovered that the least informative features were dep rel (44, 45), is ancestor (48, 49), heads second order (46, 47) and path length (53).

we conducted experiments removing all these features and the two least informative 8 (45, 49), but the results did not improve the initial experiment.


Therefore, we conducted experiments removing all these features and the two least informative 8 (45, 49), but the results did not improve the initial experiment.

we decided to select the initial set (24 features in total) as features for reporting results with the test set.


(iii) In the gold data (dev+train), a majority of scopes begin in the negation cue (69.50%).

the system tends to take the negation cue as the start of the scope.


A relatively small decrease in the weights of high-frequency tokens will prevent the generation probabilities of ground-truth tokens from ascending continually, which may result in an obvious degradation of the overall performance.

we ensure that all the token weights are equal to or bigger than 1 considering the training stability as well as designing convenience.


In this case, we don't consider the factor of noisy tokens so that the weight increases monotonically as the frequency decreases.

this weighting function is suitable for cleaner training data where the extremely low-frequency tokens only take up a small proportion.


While the aforementioned vanilla transformer is a powerful method, it is less suitable for video paragraph captioning due to its inability to utilize video segments and sentences history information.

given the unified encoder-decoder transformer, we augment it with an external memory module, which helps it to utilize video segments and the corresponding caption history to generate the next sentence.


Current metrics, such as BLEU, ROUGE and METEOR, are agnostic to the end-task while LERC is trained with the passage and question as input.

lERC assigns a score that better reflects human judgement.


This raises an issue with the evaluation; a metric can look strong when evaluated on current model outputs, but may in-fact struggle in the future when QA systems produce better answers.

using only these candidates for evaluation could lead to overconfidence in a learned metric's capabilities.


One explanation is that current generative QA models may not generate many candidates that would require the metric to use the passage.

even the complete version of LERC may have learned to ignore the passage.


Besides a small number of high-resource sentence functions, a large portion of sentence functions is infrequent.

dialogue generation conditioned on these infrequent sentence functions suffers from data deficiency.


In meta-learning, the goal of the trained model is to quickly learn a new task from a small amount of data.

the model should be able to learn transferable knowledge on a large number of different tasks.


For example, in the recently released response generation dataset with manually annotated sentence functions STC-SeFun (Bi et al., 2019), more than 40% of utterances are Positive Declarative while utterances annotated with Declarative with Interrogative words account for less than 1% of the entire dataset.

dialogue generation models suffer from data deficiency for these infrequent sentence functions.


Although we can not compare intervals at ordinal scale, we know, e.g., that "neutral" is closer to "positive" than "negative".

we need another property to verify monotonicity with respect to category closeness.


This is a metric that integrates aspects from the previous three metric families, including two parameters β1 and β2 to combine different components.

this metric can capture the different quality aspects involved in the OC process.


Moreover, there are proofs that these poor performances are not influenced by factors such as age, sex, confidence, and experience and that professionals such as psychologists, detectives and judges are no more accurate than students and other citizens in this task (Aamodt and Custer, 2006).

the possible dangers associated with the mentioned phenomena made necessary the development of new approaches for unmasking deceit not relying on human judgement.


When applied to classification tasks like language recognition, RNNs are typically combined with a "decoder": additional layer(s) that map their hidden states to a prediction.

despite differences in state expressiveness, rational RNNs might be able to achieve comparable empirical performance to non-rational RNNs on NLP tasks.


However, we also construct a language that an LSTM can recognize without a decoder, but a QRNN cannot recognize with any decoder.

no decoder can fully compensate for the weakness of the QRNN compared to the LSTM.


Finally, we conduct experiments on formal languages, justifying that our theorems correctly predict which languages unsaturated recognizers trained by gradient descent can learn.

we view our hierarchy as a useful formal tool for understanding the relative capabilities of different RNN architectures.


On the other hand, the value of the RNN and GRU cell is bounded by 1, and QRNN and LSTM cells can only grow linearly in time.


We then use the decoder as an oracle to decide any RE language.

an RR-complete encoder with an RE decoder is Turing-complete.


Note that we define decoders as functions from the final state to the output.

adding an additional QRNN layer does not count as a "decoder" (as it reads multiple states).


We subsidiarily annotated two types of event attributes with each bounding box: "doing-the-action," or "done-the-action".

of the annotation, we got 2,300 bounding boxes in 272 flow graph recipes.


And some bounding boxes may be linked to the same r-NE because the object can be multiple or separated in the image.

the relationship from bounding boxes to r-NEs is so-called one-to-many.


As the recipes for our r-FG-BB dataset, we selected, from the r-FG corpus, those having an image to each step.


In the computer vision community, researchers build their datasets with bounding boxes that have IoUs larger than 0.5 (Su et al., 2012; Papadopoulos et al., 2017).

we can say that many annotated bounding boxes are suitable for the dataset.


In our r-FG-BB dataset, some bounding boxes are connected through the flow graph.

we can try an interesting novel task: bounding box linking.


Procedural text generation from an image sequence is a prominent area because it requires a model to consider the context and the output coherency (Nishimura et al., 2019; Chandu et al., 2019).

these studies focused on generating grounded captions: incorporating a structure into the model implicitly (Chandu et al., 2019) and preferentially decoding important terms (Nishimura et al., 2019).


The results showed that the proposed simple models could ground bounding boxes with nodes in a flow graph, which gives a linking between the bounding boxes over images while classifying event attributes correctly.

our dataset is useful for contextual visual grounding.


Chr-to-En is also highly meaningful in helping spread Cherokee history and culture.

in this paper, we contribute our effort to Cherokee revitalization by constructing a clean Cherokee-English parallel dataset, ChrEn, which results in 14,151 pairs of sentences with around 313K English tokens and 206K Cherokee tokens.


If only one of the two language models' vocabularies contains a given type, then the language model that is missing the type gives a probability of 0.

the interpolated probability for that type would be (p+0)/2, where p is the probability from the language model that contains that type.


While some recent papers on knowledge-grounded dialogues have paid attention to the problem of knowledge selection (Lian et al., 2019; Kim et al., 2020; Ren et al., 2019), the knowledge selection module is either deeply coupled with the specially configured models (Lian et al., 2019; Ren et al., 2019) and thus is incompatible with the pre-trained language models, or it is learned with human annotations (Dinan et al., 2019; Kim et al., 2018) which are difficult to obtain in practice (e.g., the dataset in (Zhou et al., 2018b) does not contain annotations for knowledge selection).

we propose an unsupervised approach where learning of knowledge selection and fine-tuning of response generation are jointly conducted with unlabeled dialogues.


The learning algorithm starts from training with pseudo ground-truth that is constructed by making full use of responses as an alternation of human annotations, and then alternatively updates the knowledge selection model and the response generation model through a reinforcement learning approach and a curriculum learning approach respectively.

knowledge selection is further optimized with the feedback from response generation, and the knowledge used for fine-tuning the response generation model gradually moves from the pseudo ground-truth to the prediction of the knowledge selection module.


We choose BERT (Devlin et al., 2018) as the backbone of the encoder.

the encoder can take advantage of pre-training, and the multi-layer bidirectional attention mechanism in BERT allows a dialogue context and the associated knowledge to sufficiently interact with each other, resulting in context-aware knowledge representations.


However, many of the existing pruning methods rely on classic regularizers that act on the individual weights by penalizing them to zero (Guo et al., 2019;Sanh et al., 2020).

the pruned model tends to maintain the same architecture as the original model despite the reduced parameter count, which does not practically lead to an improvement in inference latency (Wen et al., 2016).


While intimacy generally refers to the closeness and interdependence of partners, the extent of self-disclosure, and the warmth or affection experienced within the relationship (Perlman and Fehr, 1987), the concept of intimacy is not restricted to the closeness or interactions between people in intimate relationships, as even people who are not in intimate relationships can have intimate interactions in a certain space and time (Wynne and Wynne, 1986).

a widely-accepted conceptualization of intimacy is to distinguish between intimate interactions and intimate relationships (Hinde, 1981).


However, individuals are less adherent to these norms as they perceive themselves to be anonymous or when interacting with an individual whom they perceive they will not interact with again (Rubin, 1983; Wynne and Wynne, 1986; Dindia et al., 1997); without the potential loss of face or social capital in such circumstances, individuals are more likely to engage in more intimate communication.

the norms of a social context and expectations around the loss of social capital for violations of these norms act as primary drivers of selecting the degree of intimacy expressed in a given context.


Second, each sentence in the document are mapped to the node it is most related to.

2 each node will have its own sentence sets.


The distributional hypothesis suggests that similar (key)words appear in similar contexts (Firth, 1957).

the co-occurrence rate between two keywords reveals helpful clues for their relatedness.


Our model offers an elegant way to learn features directly from large corpora, bypassing the dependence on mature NLP pipelines.

it is not limited to resource-rich languages and can be used by any applications that operate on text.


Table 5 clearly shows that ZAP fails to transfer many predicates, perhaps because it has unreliable (or no) word-alignment probabilities for infrequent predicates and it is not fine-tuned for this domain (it was trained on Europarl).

also the argument scores are very low, since for each predicate it misses, the system cannot recover any arguments.


In some way, an explicit reordering model can be regarded as a LM which constrains candidate words to come from the input sentence.

it may suffer from the same problem as unidirectional LMs: at one time, contexts from only one direction can be utilized instead of from both directions.


As the attention operation is quadratic to the sequence length, this fundamentally limits the maximum length of the input sequence, and thus restricts the model capacity in terms of capturing long-distance dependencies.

downstream tasks have to either truncate their sequences to leading tokens (Nogueira and Cho, 2019) or split their sequences with a sliding window (Joshi et al., 2019a,b).


All the models follow the BERT-Base setting, i.e., L = 12, H = 768, A = 12, and are trained on the same corpus -BooksCorpus and English Wikipedia with uncased word piece tokens.

all models use the same vocabulary as Google BERT (uncased version) with vocabulary size 30,522.


Due to the large size of model parameters and deep architectures, modern neural networks training requires significant amounts of computing resources.

there is an increasing interest in training neural networks with low memory (Sohoni et al., 2019).


However, due to the insufficient support for sparse tensors from the current deep learning platforms, some of them have to represent a sparse matrix using a dense matrix with a binary mask or rely on customized CUDA kernels (Gray et al., 2017).

the speed-up or reduction in memory consumption is sometimes limited in practice.


In addition, some puns are created by replacing a word with another word with the same or similar pronunciation as examples shown in Table 1.

to recognize puns, it is essential to model the association between words in the sentence and the pronunciation of words.


Note that we use the average pooling as an alternative when we remove the phonological attention module.

we can see the drop after removing each of the three features.


Note that when we extract emotion-cause pairs, we obtain the emotions and causes for each text simultaneously.

we also evaluate the performance of emotion extraction and cause extraction.


For speech recognition, we stream the audio recorded by the headset to IBM Watson servers to receive a transcription, which is then analyzed in a command-based fashion.

our speech module not only handles dictations as in Teixeira et al. (2019), but can correct mistakes in place.


As with other methods, once a surrogate is assigned, it is held out.

less frequent words are assigned to the next most locally frequent surrogate.


The aspects are regarded as instances and their sentiment distributions are predicted by an attention-based classifier, while the document is regarded as a bag and its sentiment distribution is computed as a combination of the aspect-level sentiment distributions.

we provide a framework for learning aspectlevel classifier by optimizing the document-level predictions.


Aiming at easily detect texts and the respective seed URLs, without introducing relevant query biases, we decided to avoid using the category names as query keywords (SchÃƒÂ¤fer and Bildhauer, 2013).

we associated to each category a set of TA keywords belonging to the basic Tunisian vocabulary.


We also tried to learn the writing style from the click-baity headlines since they have shown superior attraction to readers.

we used The Examiner -SpamClickBait News dataset, denoted as the Clickbait dataset.


We analyze the differences in performance on the three languages and show that the reasons for this failure are due to the composition and quality of the k-best lists.

we show that the gold tree ratio in the English k-best list is much higher than for German and Czech, and that the trees in the English k-best list show a higher variety, thus making it easier for the reranker to distinguish between high-and low-quality trees.