Compositional embedding models build a representation ( or embedding ) for a linguistic structure based on its component word embeddings .

We propose a Feature-rich Compositional Embedding Model ( FCM ) for relation extraction that is expressive , generalizes to new domains , and is easy-to-implement .

The key idea is to combine both ( unlexicalized ) handcrafted features with learned word embeddings .

The model is able to directly tackle the difficulties met by traditional compositional embeddings models , such as handling arbitrary types of sentence annotations and utilizing global information for composition .

We test the proposed model on two relation extraction tasks , and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task , and the SemEval 2010 relation classification task .

The combination of our model and a loglinear classifier with hand-crafted features gives state-of-the-art results .

We made our implementation available for general use 1 .

Two common NLP feature types are lexical properties of words and unlexicalized linguistic / structural interactions between words .

Prior work on relation extraction has extensively studied how to design such features by combining discrete lexical properties ( e.g. the identity of a word , ⇤ ⇤ Gormley and Yu contributed equally .

1 https://github.com / mgormley / pacaya its lemma , its morphological features ) with aspects of a word 's linguistic context ( e.g. whether it lies between two entities or on a dependency path between them ) .

While these help learning , they make generalization to unseen words difficult .

An alternative approach to capturing lexical information relies on continuous word embeddings2 as representative of words but generalizable to new words .

Embedding features have improved many tasks , including NER , chunking , dependency parsing , semantic role labeling , and relation extraction ( Miller et al . , 2004 ; Turian et al . , 2010 ; Koo et al . , 2008 ; Roth and Woodsend , 2014 ; Sun et al . , 2011 ; Plank and Moschitti , 2013 ; Nguyen and Grishman , 2014 ) .

Embeddings can capture lexical information , but alone they are insufficient : in state-of-the-art systems , they are used alongside features of the broader linguistic context .

In this paper , we introduce a compositional model that combines unlexicalized linguistic context and word embeddings for relation extraction , a task in which contextual feature construction plays a major role in generalizing to unseen data .

Our model allows for the composition of embeddings with arbitrary linguistic structure , as expressed by hand crafted features .

In the following sections , we begin with a precise construction of compositional embeddings using word embeddings in conjunction with unlexicalized features .

Various feature sets used in prior work ( Turian et al . , 2010 ; Nguyen and Grishman , 2014 ; Hermann et al . , 2014 ; Roth and Woodsend , 2014 )

A feature that depends on the embedding for this context word could generalize to other lexical indicators of the same relation ( e.g. " operating " ) that do n't appear with ART during training .

But lexical information alone is insufficient ; relation extraction requires the identification of lexical roles : where a word appears structurally in the sentence .

In ( 2 ) , the word " of " between " suburbs " and " Baghdad " suggests that the first entity is part of the second , yet the earlier occurrence after " direction " is of no significance to the relation .

Even finer information can be expressed by a word 's role on the dependency path between entities .

In ( 3 ) we can distinguish the word " died " from other irrelevant words that do n't appear between the entities .

tured as special cases of this construction .

Adding these compositional embeddings directly to a standard log-linear model yields a special case of our full model .

We then treat the word embeddings as parameters giving rise to our powerful , efficient , and easy-to-implement log-bilinear model .

The model capitalizes on arbitrary types of linguistic annotations by better utilizing features associated with substructures of those annotations , including global information .

We choose features to promote different properties and to distinguish different functions of the input words .

First , it decomposes the annotated sentence into substructures ( i.e. a word and associated annotations ) .

Second , it extracts features for each substructure ( word ) , and combines them with the word 's embedding to form a substructure embedding .

Third , we sum over substructure embeddings to form a composed annotated sentence embedding , which is used by a final softmax layer to predict the output label ( relation ) .

The result is a state-of-the-art relation extractor for unseen domains from ACE 2005 ( Walker et al . , 2006 ) and the relation classification dataset from SemEval-2010 Task 8 ( Hendrickx et al . , 2010 ) .

Contributions This paper makes several contributions , including :

We introduce the Yu and Dredze ( 2015 ) and Yu et al . ( 2015 ) .

Additionally , we have extended FCM to incorporate a low-rank embedding of the features ( Yu et al . , 2015 ) , which focuses on fine-grained relation extraction for ACE and ERE .

This paper obtains better results than the low-rank extension on ACE coarse-grained relation extraction .

We have presented FCM , a new compositional model for deriving sentence-level and substructure embeddings from word embeddings .

Compared to existing compositional models , FCM can easily handle arbitrary types of input and handle global information for composition , while remaining easy to implement .

We have demonstrated that FCM alone attains near state-of-the-art performances on several relation extraction tasks , and in combination with traditional feature based loglinear models it obtains state-of-the-art results .

Our next steps in improving FCM focus on enhancements based on task-specific embeddings or loss functions as in Hashimoto et al . ( 2015 ; dos Santos et al . ( 2015 ) .

Moreover , as the model provides a general idea for representing both sentences and sub-structures in language , it has the potential to contribute useful components to various tasks , such as dependency parsing , SRL and paraphrasing .

Also as kindly pointed out by one anonymous reviewer , our FCM can be applied to the TAC-KBP ( Ji et al . , 2010 ) tasks , by replacing the training objective to a multi-instance multilabel one ( e.g. Surdeanu et al . ( 2012 ) ) .

We plan to explore the above applications of FCM in the future .


Chinese short text matching usually employs word sequences rather than character sequences to get better performance .

However , Chinese word segmentation can be erroneous , ambiguous or inconsistent , which consequently hurts the final matching performance .

To address this problem , we propose neural graph matching networks , a novel sentence matching framework capable of dealing with multi-granular input information .

Instead of a character sequence or a single word sequence , paired word lattices formed from multiple word segmentation hypotheses are used as input and the model learns a graph representation according to an attentive graph matching mechanism .

Experiments on two Chinese datasets show that our models outperform the state-of-the-art short text matching models .

Short text matching ( STM ) is a fundamental task of natural language processing ( NLP ) .

It is usually recognized as a paraphrase identification task or a sentence semantic matching task .

Given a pair of sentences , a matching model is to predict their semantic similarity .

It is widely used in question answer systems and dialogue systems ( Gao et al . , 2019 ; Yu et al . , 2014 ) .

The recent years have seen advances in deep learning methods for text matching ( Mueller and Thyagarajan , 2016 ; Gong et al . , 2017 ; Chen et al . , 2017 ; Lan and Xu , 2018 ) .

However , almost all of these models are initially proposed for English text matching .

Applying them for Chinese text matching , we have two choices .

One is to take Chinese characters as the input of models .

Another is first to segment each sentence into words , and then to take these words as input tokens .

Although character-based models can overcome the * Kai Yu is the corresponding author .

problem of data sparsity to some degree ( Li et al . , 2019 ) , the main drawback of these models is that explicit word information is not fully exploited , which can be potentially useful for semantic matching .

However , word-based models often suffer some potential issues caused by word segmentation .

As shown in Figure 1 , the character sequence " 南 京 市 长 江 大 桥(South

Capital City Long River Big Bridge ) " has two different meanings with different word segmentation .

The first one refers to a bridge ( Segment-1 , Segment-2 ) , and the other refers to a person ( Segment-3 ) .

The ambiguity may be eliminated with more context .

Additionally , the segmentation granularity of different tools is different .

For example , " 长江大 桥(Yangtze River Bridge ) " in Segment-1 is divided into two words " 长江(Yangtze River ) " and " 大 桥(Bridge ) " in Segment-2 .

It has been shown that multi-granularity information is important for text matching ( Lai et al . , 2019 ) .

Here we propose a neural graph matching method ( GMN ) for Chinese short text matching .

Instead of segmenting each sentence into a word sequence , we keep all possible segmentation paths to form a word lattice graph , as shown in Figure 1 .

GMN takes a pair of word lattice graphs as input and updates the representations of nodes according to the graph matching attention mechanism .

Also , GMN can be combined with pre-trained language models , e.g. BERT ( Devlin et al . , 2019 ) .

It can be regarded as a method to integrate word information in these pre-trained language models during the fine-tuning phase .

The experiments on two Chinese Datasets show that our model outperforms not only previous state-of-the-art models but also the pre-trained model BERT as well as some variants of BERT .

In this paper , we propose a neural graph matching model for Chinese short text matching .

It takes a pair of word lattices as input instead of word or character sequences .

The utilization of word lattice can provide more multi-granularity information and avoid the error propagation issue of word segmentation .

Additionally , our model and the pre-training model are complementary .

It can be regarded as a flexible method to introduce word information into BERT during the fine-tuning phase .

The experimental results show that our model outperforms the state-of-the-art text matching models as well as some BERT-based models .


Chinese Pinyin input method is very important for Chinese language information processing .

Users may make errors when they are typing in Chinese words .

In this paper , we are concerned with the reasons that cause the errors .

Inspired by the observation that pressing backspace is one of the most common user behaviors to modify the errors , we collect 54 , 309 , 334 error-correction pairs from a realworld data set that contains 2 , 277 , 786 users via backspace operations .

In addition , we present a comparative analysis of the data to achieve a better understanding of users ' input behaviors .

Comparisons with English typos suggest that some language-specific properties result in a part of Chinese input errors .

Unlike western languages , Chinese is unique due to its logographic writing system .

Chinese users can not directly type in Chinese words using a QW-ERTY keyboard .

Pinyin is the official system to transcribe Chinese characters into the Latin alphabet .

Based on this transcription system , Pinyin input methods have been proposed to assist users to type in Chinese words ( Chen , 1997 ) .

The typical way to type in Chinese words is in a sequential manner ( Wang et al . , 2001 ) .

Assume users want to type in the Chinese word " 什 么(what ) " .

First , they mentally generate and type in corresponding Pinyin " shenme " .

Then , a Chinese Pinyin input method displays a list of Chinese words which share that Pinyin , as shown in Fig . 1 .

Users visually search the target word from candidates and select numeric key " 1 " to get the result .

The last two steps do not exist in typing process of English words , which indicates that it is more complicated for Chinese users to type in Chinese words .

Chinese users may make errors when they are typing in Chinese words .

As shown in Fig . 2 , a user may mistype " shenme " as " shenem " .

Typical Chinese Pinyin input method can not return the right word .

Users may not realize that an error occurs and select the first candidate word " 什恶魔 " ( a meaningless word ) as the result .

This greatly limits user experience since users have to identify errors and modify them , or can not get the right word .

In this paper , we analyze the reasons that cause errors in Chinese Pinyin input method .

This analysis is helpful in enhancing the user experience and the performance of Chinese Pinyin input method .

In practice , users press backspace on the keyboard to modify the errors , they delete the mistyped word and re-type in the correct word .

Motivated by this ob-servation , we can extract error-correction pairs from backspace operations .

These error-correction pairs are of great importance in Chinese spelling correction task which generally relies on sets of confusing words .

We extract 54 , 309 , 334 error-correction pairs from user input behaviors and further study them .

Our comparative analysis of Chinese and English typos suggests that some language-specific properties of Chinese lead to a part of input errors .

To the best of our knowledge , this paper is the first one which analyzes user input behaviors in Chinese Pinyin input method .

The rest of this paper is organized as follows .

Section 3 introduces how we collect errors in Chinese Pinyin input method .

In Section 4 , we investigate the reasons that result in these errors .

Section 5 concludes the whole paper and discusses future work .

In this paper , we study user input behaviors in Chinese Pinyin input method from backspace operations .

We aim at analyzing the reasons that cause these errors .

Users signal that they are very likely to make errors if they press backspace on the keyboard .

Then they modify the errors and type in the correct words they want .

Different from the previous research , we extract abundant Pinyin-correction and Chinese word-correction pairs from backspace operations .

Compared with English typos , we observe some language-specific properties in Chinese have impact on errors .

All in all , user behaviors ( Zheng et al . , 2009 ; Zheng et al . , 2010 ; Zheng et al . , 2011b ) in Chinese Pinyin input method provide novel perspectives for natural language processing tasks .

Below we sketch three possible directions for the future work : ( 1 ) we should consider position features in analyzing Pinyin errors .

For example , it is less likely that users make errors in the first letter of an input Pinyin .

( 2 ) we aim at designing a selfadaptive input method that provide error-tolerant features ( Chen and Lee , 2000 ; Zheng et al . , 2011a ) .

( 3 ) we want to build a Chinese spelling correction system based on extracted error-correction pairs .


We present an inexact search algorithm for the problem of predicting a two-layered dependency graph .

The algorithm is based on a k-best version of the standard cubictime search algorithm for projective dependency parsing , which is used as the backbone of a beam search procedure .

This allows us to handle the complex nonlocal feature dependencies occurring in bistratal parsing if we model the interdependency between the two layers .

We apply the algorithm to the syntacticsemantic dependency parsing task of the CoNLL-2008 Shared Task , and we obtain a competitive result equal to the highest published for a system that jointly learns syntactic and semantic structure .

Numerous linguistic theories assume a multistratal model of linguistic structure , such as a layer of surface syntax , deep syntax , and shallow semantics .

Examples include Meaning-Text Theory ( Mel'čuk , 1988 ) , Discontinuous Grammar ( Buch-Kromann , 2006 ) , Extensible Dependency Grammar ( Debusmann et al . , 2004 ) , and the Functional Generative Description ( Sgall et al . , 1986 ) which forms the theoretical foundation of the Prague Dependency Treebank ( Hajič , 1998 ) .

In the statistical NLP community , the most widely used grammatical resource is the Penn Treebank ( Marcus et al . , 1993 ) .

This is a purely syntactic resource , but we can also include this treebank in the category of multistratal resources since the PropBank ( Palmer et al . , 2005 ) and NomBank ( Meyers et al . , 2004 ) projects have annotated shallow semantic structures on top of it .

Dependency-converted versions of the Penn Treebank , PropBank and NomBank were used in the CoNLL-2008 Shared Task ( Surdeanu et al . , 2008 ) , in which the task of the participants was to produce a bistratal dependency structure consisting of surface syntax and shallow semantics .

Producing a consistent multistratal structure is a conceptually and computationally complex task , and most previous methods have employed a purely pipeline-based decomposition of the task .

This includes the majority of work on shallow semantic analysis ( Gildea and Jurafsky , 2002 , inter alia ) .

Nevertheless , since it is obvious that syntax and semantics are highly interdependent , it has repeatedly been suggested that the problems of syntactic and semantic analysis should be carried out simultaneously rather than in a pipeline , and that modeling the interdependency between syntax and semantics would improve the quality of all the substructures .

The purpose of the CoNLL-2008 Shared Task was to study the feasibility of a joint analysis of syntax and semantics , and while most participating systems used a pipeline-based approach to the problem , there were a number of contributions that attempted to take the interdependence between syntax and semantics into account .

The top-performing system in the task ( Johansson and Nugues , 2008 ) applied a very simple reranking scheme by means of a k-best syntactic output , similar to previous attempts ( Gildea and Jurafsky , 2002 ; Toutanova et al . , 2005 ) to improve semantic role labeling performance by using mul-tiple parses .

The system by Henderson et al . ( 2008 ) extended previous stack-based algorithms for dependency parsing by using two separate stacks to build the syntactic and semantic graphs .

Lluís and Màrquez ( 2008 ) proposed a model that simultaneously predicts syntactic and semantic links , but since its search algorithm could not take the syntactic-semantic interdependencies into account , a pre-parsing step was still needed .

In addition , before the CoNLL-2008 shared task there have been a few attempts to jointly learn syntactic and semantic structure ; for instance , Merlo and Musillo ( 2008 ) appended semantic role labels to the phrase tags in a constituent treebank and applied a conventional constituent parser to predict constituent structure and semantic roles .

In this paper , we propose a new approximate search method for bistratal dependency analysis .

The search method is based on a beam search procedure that extends a k-best version of the standard cubic-time search algorithm for projective dependency parsing .

This is similar to the search method for constituent parsing used by Huang ( 2008 ) , who referred to it as cube pruning , inspired by an idea from machine translation decoding ( Chiang , 2007 ) .

The cube pruning approach , which is normally used to solve the arg max problem , was also recently extended to summing problems , which is needed in some learning algorithms ( Gimpel and Smith , 2009 ) .

We apply the algorithm on the CoNLL-2008 Shared Task data , and obtain the same evaluation score as the best previously published system that simultaneously learns syntactic and semantic structure ( Titov et al . , 2009 ) .

In this paper , we have presented a new approximate search method to solve the problem of jointly predicting the two layers in a bistratal dependency graph .

The algorithm shows competitive performance on the treebank used in the CoNLL-2008 Shared Task , a bistratal treebank consisting of a surface-syntactic and a shallow semantic layer .

In addition to the syntactic-semantic task that we have described in this paper , we believe that our method can be used in other types of multistratal syntactic frameworks , such as a representation of surface and deep syntax as in Meaning-Text Theory ( Mel'čuk , 1988 ) .

The optimization problem that we set out to solve is intractable , but we have shown that reasonable performance can be achieved with an inexact , beam search-based search method .

This is not obvious : it has previously been shown that using an inexact search procedure when the learning algorithm assumes that the search is exact may lead to slow convergence or even divergence ( Kulesza and Pereira , 2008 ) , but this does not seem to be a problem in our case .

While we used a beam search method as the method of approximation , other methods are certainly possible .

An interesting example is the recent system by Smith and Eisner ( 2008 ) , which used loopy belief propagation in a dependency parser using highly complex features , while still maintaining cubic-time search complexity .

An obvious drawback of our approach compared to traditional pipeline-based semantic role labeling methods is that the speed of the algorithm is highly dependent on the size of the interdependency feature representation Φ i .

Also , extracting these features is fairly complex , and it is of critical importance to implement the feature extraction procedure efficiently since it is one of the bottlenecks of the algorithm .

It is plausible that our performance suffers from the absence of other frequently used syntax-based features such as dependent-of-dependent and voice .

It is thus highly dubious that a joint modeling of syntactic and semantic structure is worth the additional implementational effort .

So far , no system using tightly integrated syntactic and semantic processing has been competitive with the best systems , which have been either completely pipelinebased ( Che et al . , 2008 ; Ciaramita et al . , 2008 ) or employed only a loose syntactic-semantic coupling ( Johansson and Nugues , 2008 ) .

It has been conjectured that modeling the semantics of the sentence would also help in syntactic disambiguation ; however , it is likely that this is already implicitly taken into account by the lexical features present in virtually all modern parsers .

In addition , a problem that our beam search method has in common with the constituent parsing method by Huang ( 2008 ) is that highly nonlocal features must be computed late .

In our case , this means that if there is a long distance between a predicate and an argument , the secondary link between them will be unlikely to influence the final search result .


This work describes an answer ranking engine for non-factoid questions built using a large online community-generated question-answer collection ( Yahoo ! Answers ) .

We show how such collections may be used to effectively set up large supervised learning experiments .

Furthermore we investigate a wide range of feature types , some exploiting NLP processors , and demonstrate that using them in combination leads to considerable improvements in accuracy .

The problem of Question Answering ( QA ) has received considerable attention in the past few years .

Nevertheless , most of the work has focused on the task of factoid QA , where questions match short answers , usually in the form of named or numerical entities .

Thanks to international evaluations organized by conferences such as the Text REtrieval Conference ( TREC)1 or the Cross Language Evaluation Forum ( CLEF ) Workshop2 , annotated corpora of questions and answers have become available for several languages , which has facilitated the development of robust machine learning models for the task .

The situation is different once one moves beyond the task of factoid QA .

Comparatively little research has focused on QA models for non-factoid questions such as causation , manner , or reason questions .

Because virtually no training data is available for this problem , most automated systems train either on small hand-annotated corpora built in house ( Higashinaka and Isozaki , 2008 ) or on question-answer pairs harvested from Frequently Asked Questions ( FAQ ) lists or similar resources ( Soricut and Brill , 2006 ) .

None of these situations is ideal : the cost of building the training corpus in the former setup is high ; in the latter scenario the data tends to be domain-specific , hence unsuitable for the learning of open-domain models .

On the other hand , recent years have seen an explosion of user-generated content ( or social media ) .

Of particular interest in our context are communitydriven question-answering sites , such as Yahoo !

Answers 3 , where users answer questions posed by other users and best answers are selected manually either by the asker or by all the participants in the thread .

The data generated by these sites has significant advantages over other web resources : ( a ) it has a high growth rate and it is already abundant ; ( b ) it covers a large number of topics , hence it offers a better approximation of open-domain content ; and ( c ) it is available for many languages .

Community QA sites , similar to FAQs , provide large number of questionanswer pairs .

Nevertheless , this data has a significant drawback : it has high variance of quality , i.e. , answers range from very informative to completely irrelevant or even abusive .

Table 1 shows some examples of both high and low quality content .

In this paper we address the problem of answer ranking for non-factoid questions from social media content .

Our research objectives focus on answering the following two questions :

1 . Is it possible to learn an answer ranking model for complex questions from such noisy data ?

This is an interesting question because a positive answer indicates that a plethora of training data is readily available to QA researchers and system developers .

2 . Which features are most useful in this scenario ?

Are similarity models as effective as models that learn question-to-answer transformations ?

Does syntactic and semantic information help ?

For generality , we focus only on textual features extracted from the answer text and we ignore all meta data information that is not generally available .

Notice that we concentrate on one component of a possible social-media QA system .

In addition to answer ranking , a complete system would have to search for similar questions already answered ( Jeon et al . , 2005 ) , and rank content quality using " social " features such as the authority of users ( Jeon et al . , 2006 ; Agichtein et al . , 2008 ) .

This is not the focus of our work : here we investigate the problem of learning an answer ranking model capable of dealing with complex questions , using a large number of , possible noisy , question-answer pairs .

By focusing exclusively on textual content we increase the portability of our approach to other collections where " social " features might not available , e.g. , Web search .

We describe our approach , including all the features explored for answer modeling , in Section 2 .

We introduce the corpus used in our empirical analysis in Section 3 .

We detail our experiments and analyze the results in Section 4 .

We overview related work in Section 5 and conclude the paper in Section 6 .

In this work we described an answer ranking engine for non-factoid questions built using a large community-generated question-answer collection .

On one hand , this study shows that we can effectively exploit large amounts of available Web data to do research on NLP for non-factoid QA systems , without any annotation or evaluation cost .

This provides an excellent framework for large-scale experimentation with various models that otherwise might be hard to understand or evaluate .

On the other hand , we expect the outcome of this process to help several applications , such as open-domain QA on the Web and retrieval from social media .

For example , on the Web our ranking system could be combined with a passage retrieval system to form a QA system for complex questions .

On social media , our system should be combined with a component that searches for similar questions already answered ; this output can possibly be filtered further by a content-quality module that explores " social " features such as the authority of users , etc .

We show that the best ranking performance is obtained when several strategies are combined into a single model .

We obtain the best results when similarity models are aggregated with features that model question-to-answer transformations , frequency and density of content , and correlation of QA pairs with external collections .

While the features that model question-to-answer transformations provide most benefits , we show that the combination is crucial for improvement .

Lastly , we show that syntactic dependency parsing and coarse semantic disambiguation yield a small , yet statistically significant performance increase on top of the traditional bag-of-words and n-gram representation .

We obtain these results using only off-the-shelf NLP processors that were not adapted in any way for our task .


Disfluencies in spontaneous speech are known to be associated with prosodic disruptions .

However , most algorithms for disfluency detection use only word transcripts .

Integrating prosodic cues has proved difficult because of the many sources of variability affecting the acoustic correlates .

This paper introduces a new approach to extracting acoustic-prosodic cues using text-based distributional prediction of acoustic cues to derive vector z-score features ( innovations ) .

We explore both early and late fusion techniques for integrating text and prosody , showing gains over a high-accuracy text-only model .

Speech disfluencies are frequent events in spontaneous speech .

The rate of disfluencies varies with the speaker and context ; one study observed disfluencies once in every 20 words , affecting up to one third of utterances ( Shriberg , 1994 ) .

Disfluencies are important to account for , both because of the challenge that the disrupted grammatical flow poses for natural language processing of spoken transcripts and because of the information that they provide about the speaker .

Most work on disfluency detection builds on the framework that annotates a disfluency in terms of a reparandum followed by an interruption point ( + ) , an optional interregnum ( { } ) , and then the repair , if any .

A few simple examples are given below : Based on the similarity / differences between the reparandum and the repair , disfluencies are often categorized into three types : repetition ( the first example ) , rephrase ( the next example ) , and restart ( the last example ) .

The interruption point is associated with a disruption in the realization of a prosodic phrase , which could involve cutting words off or elongation associated with hesitation , followed by a prosodic reset at the start of the repair .

There may also be emphasis in the repair to highlight the correction .

Researchers have been working on automatic disfluency detection for many years ( Lickley , 1994 ; Shriberg et al . , 1997 ; Charniak and Johnson , 2001 ; Johnson and Charniak , 2004 ; Lease et al . , 2006 ;

Qian and Liu , 2013 ; Zayats et al . , 2016 ) , motivated in part by early work on parsing speech that assumed reliable detection of the interruption point ( Nakatani and Hirschberg , 1994 ; Shriberg and Stolcke , 1997 ; Liu et al . , 2006 ) .

The first efforts to integrate prosody with word cues for disfluency detection ( Baron et al . , 2002 ; Snover et al . , 2004 ) found gains from using prosody , but word cues played the primary role .

In subsequent work ( Qian and Liu , 2013 ; Honnibal and Johnson , 2014 ; Wang et al . , 2017 ) , more effective models of word transcripts have been the main source of performance gains .

The success of recent neural network systems raises the question of what the role is for prosody in future work .

In the next section , we hypothesize where prosody might help and look at the relative frequency of these cases and the performance of a high accuracy disfluency detection algorithm in these contexts .

With the premise that there is a potential for prosody to benefit disfluency detection , we then propose a new approach to extracting prosodic features .

A major challenge for all efforts to incorporate prosodic cues in spoken language understanding is the substantial variability in the acoustic correlates of prosody .

For example , duration cues are expected to be useful -disfluencies are often associated with duration lengthening related to hesitation .

However , duration varies with phonetic context , word function , prosodic phrase structure , speaking rate , etc .

To account for some of this variability , various feature normalization techniques are used , but typically these account for only limited contexts , e.g. phonetic context for duration or speaker pitch range for fundamental frequency .

In our work , we introduce a mechanism for normalization using the full sentence context .

We train a sequential neural prediction model to estimate distributions of acoustic features for each word , given the word sequence of a sentence .

Then , the actual observed acoustic feature is used to find the prediction error , normalized by the estimated variance .

We refer to the resulting features as innovations , which can be thought of as a non-linear version of the innovations in a Kalman filter .

The innovations will be large when the acoustic cues do not reflect the expected prosodic structure , such as during hesitations , disfluencies , and contrastive or emphatic stress .

The idea is to provide prosodic cues that are less redundant with the textual cues .

We assess the new prosodic features in experiments on disfluency detection using the Switchboard corpus , exploring both early and late fusion techniques to integrate innovations with text features .

Our analysis shows that prosody does help with detecting some of the more difficult types of disfluencies .

This paper has three main contributions .

First , our analysis of a high performance disfluency detection algorithm confirms hypotheses about contexts where text-only models have high error rates .

Second , we introduce a novel representation of prosodic cues , i.e. the innovation vector resulting from predicting prosodic cues given the whole sentence context .

Analyses of the innovation distributions show expected patterns of prosodic cues at interruption points .

Finally , we demonstrate improved disfluency detection performance on Switchboard by integrating prosody and textbased features in a neural network architecture , while comparing early and late fusion approaches .

In this paper , we introduce a novel approach to extracting acoustic-prosodic cues with the goal of improving disfluency detection , but also with the intention of impacting spoken language processing more generally .

Our initial analysis of a textonly disfluency detection system shows that despite high performance of such models , there exists a big gap in the performance of text-based approaches for some types of disfluencies , such as restarts and non-trivial or long rephrases .

Thus , prosody cues , which can be indicative of interruption points , have a potential to contribute towards detection of more difficult types of disfluencies .

Since the acoustic-prosodic cues carry information related to multiple phenomena , it can be difficult to isolate the cues that are relevant to specific events , such as interruption points .

In this work , we introduce a novel approach where we extract relevant acoustic-prosodic information using textbased distributional prediction of acoustic cues to derive vector z-score features , or innovations .

The innovations point to irregularities in prosody flow that are not predicted by the text , helping to better isolate signals relevant to disfluency detection that are not simply redundant with textual cues .

We explore both early and late fusion approaches to combine innovations with text-based features .

Our experiments show that innovation features are better predictors of disfluencies compared to the original acoustic cues .

Our analysis of the errors and of the innovation features point to a limitation of the current work , which is in the modeling of F0 features .

The current model obtains word-based F0 ( and energy ) features by simply averaging the values over the duration of the word , which loses any distinctions between rising and falling F0 .

By leveraging polynomial contour models , we expect to improve both intonation and energy features , which we hope will reduce some of the false detections associated with emphasis and unexpected fluent phrase boundaries .

An important next step is to test the system using ASR rather than hand transcripts .

It is possible that errors in the transcripts could hurt the residual prediction , but if prosody is used to refine the recognition hypothesis , this could actually lead to improved recognition .

Finally , we expect that the innovation model of prosody can benefit other NLP tasks , such as sarcasm and intent detection , as well as detecting paralinguist information .

However , most algorithms for disfluency detection use only word transcripts .

Speech disfluencies are frequent events in spontaneous speech .


In this work , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall .

We present a generic phrase training algorithm which is parameterized with feature functions and can be optimized jointly with the translation engine to directly maximize the end-to-end system performance .

Multiple data-driven feature functions are proposed to capture the quality and confidence of phrases and phrase pairs .

Experimental results demonstrate consistent and significant improvement over the widely used method that is based on word alignment matrix only .

Phrase has become the standard basic translation unit in Statistical Machine Translation ( SMT ) since it naturally captures context dependency and models internal word reordering .

In a phrase-based SMT system , the phrase translation table is the defining component which specifies alternative translations and their probabilities for a given source phrase .

In learning such a table from parallel corpus , two related issues need to be addressed ( either separately or jointly ): which pairs are considered valid translations and how to assign weights , such as probabilities , to them .

The first problem is referred to as phrase pair extraction , which identifies phrase pairs that are supposed to be translations of each other .

Methods have been proposed , based on syntax , that take advantage of linguistic constraints and alignment of grammatical structure , such as in Yamada and Knight ( 2001 ) and Wu ( 1995 ) .

The most widely used approach derives phrase pairs from word alignment matrix ( Och and Ney , 2003 ; Koehn et al . , 2003 ) .

Other methods do not depend on word alignments only , such as directly modeling phrase alignment in a joint generative way ( Marcu and Wong , 2002 ) , pursuing information extraction perspective ( Venugopal et al . , 2003 ) , or augmenting with modelbased phrase pair posterior ( Deng and Byrne , 2005 ) .

Using relative frequency as translation probability is a common practice to measure goodness of a phrase pair .

Since most phrases appear only a few times in training data , a phrase pair translation is also evaluated by lexical weights ( Koehn et al . , 2003 ) or term weighting ( Zhao et al . , 2004 ) as additional features to avoid overestimation .

The translation probability can also be discriminatively trained such as in Tillmann and Zhang ( 2006 ) .

The focus of this paper is the phrase pair extraction problem .

As in information retrieval , precision and recall issues need to be addressed with a right balance for building a phrase translation table .

High precision requires that identified translation candidates are accurate , while high recall wants as much valid phrase pairs as possible to be extracted , which is important and necessary for online translation that requires coverage .

In the word-alignment derived phrase extraction approach , precision can be improved by filtering out most of the entries by using a statistical significance test ( Johnson et al . , 2007 ) .

On the other hand , there are valid translation pairs in the training corpus that are not learned due to word alignment errors as shown in Deng and Byrne ( 2005 ) .

We would like to improve phrase translation accuracy and at the same time extract as many as possible valid phrase pairs that are missed due to incorrect word alignments .

One approach is to leverage underlying word alignment quality such as in Ayan and Dorr ( 2006 ) .

In this work , we present a generic discriminative phrase pair extraction framework that can integrate multiple features aiming to identify correct phrase translation candidates .

A significant deviation from most other approaches is that the framework is parameterized and can be optimized jointly with the decoder to maximize translation performance on a development set .

Within the general framework , the main work is on investigating useful metrics .

We employ features based on word alignment models and alignment matrix .

We also propose information metrics that are derived from both bilingual and monolingual perspectives .

All these features are data-driven and independent of languages .

The proposed phrase extraction framework is general to apply linguistic features such as semantic , POS tags and syntactic dependency .

In this paper , the problem of extracting phrase translation is formulated as an information retrieval process implemented with a log-linear model aiming for a balanced precision and recall .

We have presented a generic phrase translation extraction procedure which is parameterized with feature functions .

It can be optimized jointly with the translation engine to directly maximize the end-to-end translation performance .

Multiple feature functions were investigated .

Our experimental results on IWSLT Chinese-English corpus have demonstrated consistent and significant improvement over the widely used word alignment matrix based extraction method .


Embedding methods which enforce a partial order or lattice structure over the concept space , such as Order Embeddings ( OE ) ( Vendrov et al . , 2016 ) , are a natural way to model transitive relational data ( e.g. entailment graphs ) .

However , OE learns a deterministic knowledge base , limiting expressiveness of queries and the ability to use uncertainty for both prediction and learning ( e.g. learning from expectations ) .

Probabilistic extensions of OE ( Lai and Hockenmaier , 2017 ) have provided the ability to somewhat calibrate these denotational probabilities while retaining the consistency and inductive bias of ordered models , but lack the ability to model the negative correlations found in real-world knowledge .

In this work we show that a broad class of models that assign probability measures to OE can never capture negative correlation , which motivates our construction of a novel box lattice and accompanying probability measure to capture anticorrelation and even disjoint concepts , while still providing the benefits of probabilistic modeling , such as the ability to perform rich joint and conditional queries over arbitrary sets of concepts , and both learning from and predicting calibrated uncertainty .

We show improvements over previous approaches in modeling the Flickr and WordNet entailment graphs , and investigate the power of the model .

Structured embeddings based on regions , densities , and orderings have gained popularity in recent years for their inductive bias towards the essential asymmetries inherent in problems such as image captioning ( Vendrov et al . , 2016 ) , lexical and textual entailment ( Erk , 2009 ; Vilnis and McCallum , 2015 ; Lai and Hockenmaier , 2017 ; Athiwaratkun and Wilson , 2018 ) , and knowledge graph completion and reasoning ( He et al . , 2015 ; Nickel and Kiela , 2017 ; Li et al . , 2017 ) .

Models that easily encode asymmetry , and related properties such as transitivity ( the two components of commonplace relations such as partially ordered sets and lattices ) , have great utility in these applications , leaving less to be learned from the data than arbitrary relational models .

At their best , they resemble a hybrid between embedding models and structured prediction .

As noted by Vendrov et al . ( 2016 ) and Li et al . ( 2017 ) , while the models learn sets of embeddings , these parameters obey rich structural constraints .

The entire set can be thought of as one , sometimes provably consistent , structured prediction , such as an ontology in the form of a single directed acyclic graph .

While the structured prediction analogy applies best to Order Embeddings ( OE ) , which embeds consistent partial orders , other region-and density-based representations have been proposed for the express purpose of inducing a bias towards asymmetric relationships .

For example , the Gaussian Embedding ( GE ) model ( Vilnis and Mc-Callum , 2015 ) aims to represent the asymmetry and uncertainty in an object 's relations and attributes by means of uncertainty in the representation .

However , while the space of representations is a manifold of probability distributions , the model is not truly probabilistic in that it does not model asymmetries and relations in terms of prob-abilities , but in terms of asymmetric comparison functions such as the originally proposed KL divergence and the recently proposed thresholded divergences ( Athiwaratkun and Wilson , 2018 ) .

Probabilistic models are especially compelling for modeling ontologies , entailment graphs , and knowledge graphs .

Their desirable properties include an ability to remain consistent in the presence of noisy data , suitability towards semisupervised training using the expectations and uncertain labels present in these large-scale applications , the naturality of representing the inherent uncertainty of knowledge they store , and the ability to answer complex queries involving more than 2 variables .

Note that the final one requires a true joint probabilistic model with a tractable inference procedure , not something provided by e.g. matrix factorization .

We take the dual approach to density-based embeddings and model uncertainty about relationships and attributes as explicitly probabilistic , while basing the probability on a latent space of geometric objects that obey natural structural biases for modeling transitive , asymmetric relations .

The most similar work are the probabilistic order embeddings ( POE ) of Lai ( Lai and Hockenmaier , 2017 ) , which apply a probability measure to each order embedding 's forward cone ( the set of points greater than the embedding in each dimension ) , assigning a finite and normalized volume to the unbounded space .

However , POE suffers severe limitations as a probabilistic model , including an inability to model negative correlations between concepts , which motivates the construction of our box lattice model .

Our model represents objects , concepts , and events as high-dimensional products-of-intervals ( hyperrectangles or boxes ) , with an event 's unary probability coming from the box volume and joint probabilities coming from overlaps .

This contrasts with POE 's approach of defining events as the forward cones of vectors , extending to infinity , integrated under a probability measure that assigns them finite volume .

One desirable property of a structured representation for ordered data , originally noted in ( Vendrov et al . , 2016 ) is a " slackness " shared by OE , POE , and our model : when the model predicts an " edge " or lack thereof ( i.e. P ( a|b )

= 0 or 1 , or a zero constraint violation in the case of OE ) , being exposed to that fact again will not update the model .

Moreover , there are large degrees of freedom in parameter space that exhibit this slackness , giving the model the ability to embed complex structure with 0 loss when compared to models based on symmetric inner products or distances between embeddings , e.g. bilinear GLMs ( Collins et al . , 2002 ) , Trans-E ( Bordes et al . , 2013 ) , and other embedding models which must always be pushing and pulling parameters towards and away from each other .

Our experiments demonstrate the power of our approach to probabilistic ordering-biased relational modeling .

First , we investigate an instructive 2-dimensional toy dataset that both demonstrates the way the model self organizes its box event space , and enables sensible answers to queries involving arbitrary numbers of variables , despite being trained on only pairwise data .

We achieve a new state of the art in denotational probability modeling on the Flickr entailment dataset ( Lai and Hockenmaier , 2017 ) , and a matching state-of-the-art on WordNet hypernymy ( Vendrov et al . , 2016 ; Miller , 1995 ) with the concurrent work on thresholded Gaussian embedding of Athiwaratkun and Wilson ( 2018 ) , achieving our best results by training on additional co-occurrence expectations aggregated from leaf types .

We find that the strong empirical performance of probabilistic ordering models , and our box lattice model in particular , and their endowment of new forms of training and querying , make them a promising avenue for future research in representing structured knowledge .

We have only scratched the surface of possible applications .

An exciting direction is the incorporation of multi-relational data for general knowledge representation and inference .

Secondly , more complex representations , such as 2n-dimensional products of 2-dimensional convex polyhedra , would offer greater flexibility in tiling event space .

Improved inference of the latent boxes , either through better optimization or through Bayesian approaches is another natural extension .

Our greatest interest is in the application of this powerful new tool to the many areas where other structured embeddings have shown promise .


State-of-the-art statistical machine translation ( MT ) systems have made significant progress towards producing user-acceptable translation output .

However , there is still no efficient way for MT systems to inform users which words are likely translated correctly and how confident it is about the whole sentence .

We propose a novel framework to predict wordlevel and sentence-level MT errors with a large number of novel features .

Experimental results show that the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score .

The Pearson correlation between the proposed confidence measure and the human-targeted translation edit rate ( HTER ) is 0.6 .

Improvements between 0.4 and 0.9 TER reduction are obtained with the n-best list reranking task using the proposed confidence measure .

Also , we present a visualization prototype of MT errors at the word and sentence levels with the objective to improve post-editor productivity .

State-of-the-art Machine Translation ( MT ) systems are making progress to generate more usable translation outputs .

In particular , statistical machine translation systems ( Koehn et al . , 2007 ; Bach et al . , 2007 ; Shen et al . , 2008 ) have advanced to a state that the translation quality for certain language pairs ( e.g. Spanish-English , French-English , Iraqi-English ) in certain domains ( e.g. broadcasting news , force-protection , travel ) is acceptable to users .

However , a remaining open question is how to predict confidence scores for machine translated words and sentences .

An MT system typically returns the best translation candidate from its search space , but still has no reliable way to inform users which word is likely to be correctly translated and how confident it is about the whole sentence .

Such information is vital to realize the utility of machine translation in many areas .

For example , a post-editor would like to quickly identify which sentences might be incorrectly translated and in need of correction .

Other areas , such as cross-lingual question-answering , information extraction and retrieval , can also benefit from the confidence scores of MT output .

Finally , even MT systems can leverage such information to do n-best list reranking , discriminative phrase table and rule filtering , and constraint decoding ( Hildebrand and Vogel , 2008 ) .

Numerous attempts have been made to tackle the confidence estimation problem .

The work of Blatz et al . ( 2004 ) is perhaps the best known study of sentence and word level features and their impact on translation error prediction .

Along this line of research , improvements can be obtained by incorporating more features as shown in ( Quirk , 2004 ; Sanchis et al . , 2007 ; Raybaud et al . , 2009 ; Specia et al . , 2009 ) .

Soricut and Echihabi ( 2010 ) developed regression models which are used to predict the expected BLEU score of a given translation hypothesis .

Improvement also can be obtained by using target part-of-speech and null dependency link in a MaxEnt classifier ( Xiong et al . , 2010 ) .

Ueffing and Ney ( 2007 ) introduced word posterior probabilities ( WPP ) features and applied them in the n-best list reranking .

From the usability point of view , back-translation is a tool to help users to assess the accuracy level of MT output ( Bach et al . , 2007 ) .

Literally , it translates backward the MT output into the source language to see whether the output of backward translation matches the original source sentence .

However , previous studies had a few shortcomings .

First , source-side features were not extensively investigated .

Blatz et al.(2004 ) only investigated source ngram frequency statistics and source language model features , while other work mainly focused on target side features .

Second , previous work attempted to incorporate more features but faced scalability issues , i.e. , to train many features we need many training examples and to train discriminatively we need to search through all possible translations of each training example .

Another issue of previous work was that they are all trained with BLEU / TER score computing against the translation references which is different from predicting the human-targeted translation edit rate ( HTER ) which is crucial in post-editing applications ( Snover et al . , 2006 ; Papineni et al . , 2002 ) .

Finally , the backtranslation approach faces a serious issue when forward and backward translation models are symmetric .

In this case , back-translation will not be very informative to indicate forward translation quality .

In this paper , we predict error types of each word in the MT output with a confidence score , extend it to the sentence level , then apply it to n-best list reranking task to improve MT quality , and finally design a visualization prototype .

We try to answer the following questions :

• Can we use a rich feature set such as sourceside information , alignment context , and dependency structures to improve error prediction performance ?

• Can we predict more translation error types i.e substitution , insertion , deletion and shift ?

• How good do our prediction methods correlate with human correction ?

• Do confidence measures help the MT system to select a better translation ?

• How confidence score can be presented to improve end-user perception ?

In Section 2 , we describe the models and training method for the classifier .

We describe novel features including source-side , alignment context , and dependency structures in Section 3 .

Experimental results and analysis are reported in Section 4 .

Section 5 and 6 present applications of confidence scores .

In this paper we proposed a method to predict confidence scores for machine translated words and sentences based on a feature-rich classifier using linguistic and context features .

Our major contributions are three novel feature sets including source side information , alignment context , and dependency structures .

Experimental results show that by combining the source side information , alignment context , and dependency structure features with word posterior probability and target POS context ( Ueffing & Ney 2007 ; Xiong et al . , 2010 ) , the MT error prediction accuracy is increased from 69.1 to 72.2 in F-score .

Our framework is able to predict error types namely insertion , substitution and shift .

The Pearson correlation with human judgement increases from 0.52 to 0.6 .

Furthermore , we show that the proposed confidence scores can help the MT system to select better translations and as a result improvements between 0.4 and 0.9 TER reduction are obtained .

Finally , we demonstrate a prototype to visualize translation errors .

This work can be expanded in several directions .

First , we plan to apply confidence estimation to perform a second-pass constraint decoding .

After the first pass decoding , our confidence estimation model can label which word is likely to be correctly translated .

The second-pass decoding utilizes the confidence informa-tion to constrain the search space and hopefully can find a better hypothesis than in the first pass .

This idea is very similar to the multi-pass decoding strategy employed by speech recognition engines .

Moreover , we also intend to perform a user study on our visualization prototype to see if it increases the productivity of post-editors .


Figurative and metaphorical language are commonplace in discourse , and figurative expressions play an important role in communication and cognition .

However , figurative language has been a relatively under-studied area in NLP , and it remains an open question to what extent modern language models can interpret nonliteral phrases .

To address this question , we introduce Fig-QA , a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings .

We evaluate the performance of several state-of-the-art language models on this task , and find that although language models achieve performance significantly over chance , they still fall short of human performance , particularly in zero-or few-shot settings .

This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models .

All our words are but crumbs that fall down from the feast of the mind ( Gibran , 1926 ) .

When humans read such a metaphorical phrase , how do they interpret it ?

Conceptual metaphors structure our everyday language and are used to map everyday physical experiences and emotions onto abstract concepts ( Lakoff and Johnson , 1981 ) .

They allow us to communicate complex ideas , to emphasize emotions , and to make humorous statements ( Fussell and Moss , 2008 ) .

However , despite relating words in a way that differs from their accepted definition , these phrases are readily interpreted by human listeners , and are common in discourse ( Shutova , 2011 ) , occurring on average every three sentences ( Mio and Katz , 1996 ; Fussell and Moss , 2008 )

The ability to interpret figurative language has been viewed as a bottleneck in natural language un-derstanding , but it has not been studied as widely as literal language ( Shutova , 2011 ; Tong et al . , 2021 ) .

Figurative language often relies on shared commonsense or cultural knowledge , and in some cases may be difficult to solve using language statistics .

This presents a challenge to language models ( LMs ) , as strong LMs trained only on text may not be able to make sense of the physical world , nor the social or cultural knowledge that language is grounded in ( Bender and Koller , 2020 ; Bisk et al . , 2020 ) .

Most previous work on figurative language focuses on metaphor detection , where a model is trained to identify the existence of metaphors in text ( Tsvetkov et al . , 2014 ; Stowe and Palmer , 2018 ; Leong et al . , 2020 ) , with datasets consisting mostly of conventionalized metaphors and idioms in wide use .

However , identifying these common metaphors that already appear often in language may be an easy task for LMs , and not fully test their ability to interpret figurative language .

The little work that exists on metaphor interpretation frames it as a task linking metaphorical phrases to literal rewordings , either through paraphrase detection ( Bizzoni and Lappin , 2018 ) or paraphrase generation ( Shutova , 2010 ; Su et al . , 2017 ; Mao et al . , 2018 ) ( details in § 7 ) Another line of work probes for metaphorical understanding in LMs , but this is similar to the metaphor detection task , in that the LM is not actually asked to choose an interpretation for the metaphor ( Pedinotti et al . , 2021 ; Aghazadeh et al . , 2022 ) .

While interesting , this work does not take into account the fact that metaphors are rich with different implications that may vary depending on the context .

In this work , we ask whether or not LMs can correctly make inferences regarding creative , relatively novel metaphors generated by humans .

This task is harder for two reasons : ( 1 ) inference is harder than identification or paraphrasing , as it requires understanding the underlying semantics , and ( 2 ) the metaphors in our dataset are novel cre-ations , and many may not appear even once in the LMs ' training data .

We propose a minimal task inspired by the Winograd schema ( Levesque et al . , 2012 ) , where LMs are tasked with choosing the entailed phrase from two opposite metaphorical phrases .

An example of a paired sentence is " Her commitment is as sturdy as ( plywood / oak ) " .

The correct answer would be either " She was ( committed / uncommitted ) " .

This can also be seen as an entailment task , where input x is the premise , and the output y is the hypothesis .

2We crowdsource a benchmark Fig-QA , consisting of 10,256 such metaphors and implications ( § 2 ) , which can be used to evaluate the nonliteral reasoning abilities of LMs or for more broad studies of figurative language in general ( we provide preliminary analyses in § 3 ) .

Through extensive experiments over strong pre-trained LMs ( § 4 ) , we find that although they can be fine-tuned to do reasonably well , their few-shot performance falls significantly short of human performance ( § 5 ) .

An in-depth analysis ( § 6 ) uncovers several insights :

( 1 ) LMs do not make use of the metaphorical context well , instead relying on the predicted probability of interpretations alone , ( 2 ) the task of associating a metaphor with an interpretation is more difficult than the reverse , ( 3 ) even strong models such as GPT-3 make inexplicable errors that are not well-aligned with human ones , indicating that further work is needed to properly model nonliteral language .

We present a Winograd-like benchmark task to test the ability of LMs to reason about figurative language , based on large-scale collection of creative metaphors written by humans .

We find a large gap between LM zero-shot and human performance on this dataset , but show that models can be fine-tuned to perform well on this particular task .

We hope that this work will encourage further study of nonliteral reasoning in LMs , especially in few-shot settings .

Given that metaphorical reasoning may play a role in problem-solving and linguistic creativity , the development of models , training methods , or datasets that enable metaphorical reasoning may improve models ' abilities to reason creatively and draw analogies between situations that may appear to be different on the surface .

One avenue we hope to investigate is multimodal metaphors , as this dataset currently includes only text-based metaphors .

Nonliteral expressions also remain understudied cross-linguistically , but further work on identifying and interpreting metaphors in other languages may also improve the abilities of multilingual models .


In this paper , we propose a method for learning a classifier which combines outputs of more than one Japanese named entity extractors .

The proposed combination method belongs to the family of stacked generalizers , which is in principle a technique of combining outputs of several classifiers at the first stage by learning a second stage classifier to combine those outputs at the first stage .

Individual models to be combined are based on maximum entropy models , one of which always considers surrounding contexts of a fixed length , while the other considers those of variable lengths according to the number of constituent morphemes of named entities .

As an algorithm for learning the second stage classifier , we employ a decision list learning method .

Experimental evaluation shows that the proposed method achieves improvement over the best known results with Japanese named entity extractors based on maximum entropy models .

In the recent corpus-based NLP research , system combination techniques have been successfully applied to several tasks such as parts-of-speech tagging ( van Halteren et al . , 1998 ) , base noun phrase chunking ( Tjong Kim Sang , 2000 ) , and parsing ( Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) .

The aim of system combination is to combine portions of the individual systems ' outputs which are partial but can be regarded as highly accurate .

The process of system combination can be decomposed into the following two sub-processes :

Collect systems which behave as differently as possible : it would help a lot if at least the collected systems tend to make errors of different types , because simple voting technique can identify correct outputs .

Previously studied techniques for collecting such systems include : i ) using several existing real systems ( van Halteren et al . , 1998 ; Brill and Wu , 1998 ; Henderson and Brill , 1999 ; Tjong Kim Sang , 2000 ) , ii ) bagging / boosting techniques ( Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) , and iii ) switching the data expression and obtaining several models ( Tjong Kim Sang , 2000 ) .

2 . Combine the outputs of the several systems : previously studied techniques include : i ) voting techniques ( van Halteren et al . , 1998 ; Tjong Kim Sang , 2000 ; Henderson and Brill , 1999 ; Henderson and Brill , 2000 ) , ii ) switching among several systems according to confidence values they provide ( Henderson and Brill , 1999 ) , iii ) stacking techniques ( Wolpert , 1992 ) which train a second stage classifier for combining outputs of classifiers at the first stage ( van Halteren et al . , 1998 ; Brill and Wu , 1998 ; Tjong Kim Sang , 2000 ) .

In this paper , we propose a method for combining outputs of ( Japanese ) named entity chunkers , which belongs to the family of stacking techniques .

In the sub-process 1 , we focus on models which differ in the lengths of preceding / subsequent contexts to be incorporated in the models .

As the base model for supervised learning of Japanese named entity chunking , we employ a model based on the maximum entropy model ( Uchimoto et al . , 2000 ) , which performed the best in IREX ( Information Retrieval and Extraction Exercise ) Workshop ( IREX Committee , 1999 ) among those based on machine learning techniques .

Uchimoto et al . ( 2000 ) reported that the optimal number of preceding / subsequent contexts to be incorporated in the model is two morphemes to both left

In this paper , we train several maximum entropy models which differ in the lengths of preceding / subsequent contexts , and then combine their outputs .

As the sub-process 2 , we propose to apply a stacking technique which learns a classifier for combining outputs of several named entity chunkers .

This second stage classifier learns rules for accepting / rejecting outputs of several individual named entity chunkers .

The proposed method can be applied to the cases where the number of constituent systems is quite small ( e.g. , two ) .

Actually , in the experimental evaluation , we show that the results of combining the best performing model of Uchimoto et al . ( 2000 ) with the one which performs poorly but extracts named entities quite different from those of the best performing model can help improve the performance of the best model .

This paper proposed a method for learning a classifier to combine outputs of more than one Japanese named entity chunkers .

Experimental evaluation showed that the proposed method achieved improvement in F-measure over the best known results with an ME model ( Uchimoto et al . , 2000 ) , when a complementary model extracted named entities quite differently from the best performing model .


In this paper , we propose a joint architecture that captures language , rhyme and meter for sonnet modelling .

We assess the quality of generated poems using crowd and expert judgements .

The stress and rhyme models perform very well , as generated poems are largely indistinguishable from human-written poems .

Expert evaluation , however , reveals that a vanilla language model captures meter implicitly , and that machine-generated poems still underperform in terms of readability and emotion .

Our research shows the importance expert evaluation for poetry generation , and that future research should look beyond rhyme / meter and focus on poetic language .

With the recent surge of interest in deep learning , one question that is being asked across a number of fronts is : can deep learning techniques be harnessed for creative purposes ?

Creative applications where such research exists include the composition of music ( Humphrey et al . , 2013 ; Sturm et al . , 2016 ; Choi et al . , 2016 ) , the design of sculptures ( Lehman et al . , 2016 ) , and automatic choreography ( Crnkovic-Friis and Crnkovic-Friis , 2016 ) .

In this paper , we focus on a creative textual task : automatic poetry composition .

A distinguishing feature of poetry is its aesthetic forms , e.g. rhyme and rhythm / meter .

In this work , we treat the task of poem generation as a constrained language modelling task , such that lines of a given poem rhyme , and each line follows a canonical meter and has a fixed number 1 Noting that there are many notable divergences from this in the work of particular poets ( e.g. Walt Whitman ) and poetry types ( such as free verse or haiku ) .

Shall I compare thee to a summer 's day ?

Thou art more lovely and more temperate :

Rough winds do shake the darling buds of May , And summer 's lease hath all too short a date : of stresses .

Specifically , we focus on sonnets and generate quatrains in iambic pentameter ( e.g. see Figure 1 ) , based on an unsupervised model of language , rhyme and meter trained on a novel corpus of sonnets .

• our proposed stress and rhyme models work very well , generating sonnet quatrains with stress and rhyme patterns that are indistinguishable from human-written poems and rated highly by an expert ; • a vanilla language model trained over our sonnet corpus , surprisingly , captures meter implicitly at human-level performance ; • while crowd workers rate the poems generated by our best model as nearly indistinguishable from published poems by humans , an expert annotator found the machine-generated poems to lack readability and emotion , and our best model to be only comparable to a vanilla language model on these dimensions ; • most work on poetry generation focuses on meter ( Greene et al . , 2010 ; Ghazvininejad et al . , 2016 ; Hopkins and Kiela , 2017 ) ; our results suggest that future research should look beyond meter and focus on improving readability .

In this , we develop a new annotation framework for the evaluation of machine-generated poems , and release both a novel data of sonnets and the full source code associated with this research.2

We propose a joint model of language , meter and rhyme that captures language and form for modelling sonnets .

We provide quantitative analyses for each component , and assess the quality of generated poems using judgements from crowdworkers and a literature expert .

Our research reveals that vanilla LSTM language model captures meter implicitly , and our proposed rhyme model performs exceptionally well .

Machine-generated generated poems , however , still underperform in terms of readability and emotion .


This paper presents a deep semantic similarity model ( DSSM ) , a special type of deep neural networks designed for text analysis , for recommending target documents to be of interest to a user based on a source document that she is reading .

We observe , identify , and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents , which we collect from commercial Web browser logs .

The DSSM is trained on millions of Web transitions , and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized .

The effectiveness of the DSSM is demonstrated using two interestingness tasks : automatic highlighting and contextual entity search .

The results on large-scale , real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks , outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models .

Tasks of predicting what interests a user based on the document she is reading are fundamental to many online recommendation systems .

A recent survey is due to Ricci et al . ( 2011 ) .

In this paper , we exploit the use of a deep semantic model for two such interestingness tasks in which document semantics play a crucial role : automatic highlighting and contextual entity search .

In this task we want a recommendation system to automatically discover the entities ( e.g. , a person , location , organi-zation etc . ) that interest a user when reading a document and to highlight the corresponding text spans , referred to as keywords afterwards .

We show in this study that document semantics are among the most important factors that influence what is perceived as interesting to the user .

For example , we observe in Web browsing logs that when a user reads an article about a movie , she is more likely to browse to an article about an actor or character than to another movie or the director .

After identifying the keywords that represent the entities of interest to the user , we also want the system to recommend new , interesting documents by searching the Web for supplementary information about these entities .

The task is challenging because the same keywords often refer to different entities , and interesting supplementary information to the highlighted entity is highly sensitive to the semantic context .

For example , " Paul Simon " can refer to many people , such as the singer and the senator .

Consider an article about the music of Paul Simon and another about his life .

Related content about his upcoming concert tour is much more interesting in the first context , while an article about his family is more interesting in the second .

At the heart of these two tasks is the notion of interestingness .

In this paper , we model and make use of this notion of interestingness with a deep semantic similarity model ( DSSM ) .

The model , extending from the deep neural networks shown recently to be highly effective for speech recognition ( Hinton et al . , 2012 ; Deng et al . , 2013 ) and computer vision ( Krizhevsky et al . , 2012 ; Markoff , 2014 ) , is semantic because it maps documents to feature vectors in a latent semantic space , also known as semantic representations .

The model is deep because it employs a neural network with several hidden layers including a special convolutional-pooling structure to identify keywords and extract hidden semantic features at different levels of abstractions , layer by layer .

The semantic representation is computed through a deep neural network after its training by backpropagation with respect to an objective tailored to the respective interestingness tasks .

We obtain naturally occurring " interest " signals by observing Web browser transitions , from a source document to a target document , in Web usage logs of a commercial browser .

Our training data is sampled from these transitions .

The use of the DSSM to model interestingness is motivated by the recent success of applying related deep neural networks to computer vision ( Krizhevshy et al . 2012 ; Markoff , 2014 ) , speech recognition ( Hinton et al . 2012 ) , text processing ( Collobert et al . 2011 ) , and Web search ( Huang et al . 2013 ) .

Among them , ( Huang et al . 2013 ) is most relevant to our work .

They also use a deep neural network to map documents to feature vectors in a latent semantic space .

However , their model is designed to represent the relevance between queries and documents , which differs from the notion of interestingness between documents studied in this paper .

It is often the case that a user is interested in a document because it provides supplementary information about the entities or concepts she encounters when reading another document although the overall contents of the second documents is not highly relevant .

For example , a user may be interested in knowing more about the history of University of Washington after reading the news about President Obama 's visit to Seattle .

To better model interestingness , we extend the model of Huang et al . ( 2013 ) in two significant aspects .

First , while Huang et al . treat a document as a bag of words for semantic mapping , the DSSM treats a document as a sequence of words and tries to discover prominent keywords .

These keywords represent the entities or concepts that might interest users , via the convolutional and max-pooling layers which are related to the deep models used for computer vision ( Krizhevsky et al . , 2013 ) and speech recognition ( Deng et al . , 2013a ) but are not used in Huang et al . 's model .

The DSSM then forms the high-level semantic representation of the whole document based on these keywords .

Second , instead of directly computing the document relevance score using cosine similarity in the learned semantic space , as in Huang et al . ( 2013 ) , we feed the features derived from the semantic representations of documents to a ranker which is trained in a supervised manner .

As a result , a document that is not highly relevant to another document a user is reading ( i.e. , the distance between their derived feature 1 We stress here that , although the click signal is available to form a dataset and a gold standard ranker ( to be described in vectors is big ) may still have a high score of interestingness because the former provides useful information about an entity mentioned in the latter .

Such information and entity are encoded , respectively , by ( some subsets of ) the semantic features in their corresponding documents .

In Sections 4 and 5 , we empirically demonstrate that the aforementioned two extensions lead to significant quality improvements for the two interestingness tasks presented in this paper .

Before giving a formal description of the DSSM in Section 3 , we formally define the interestingness function , and then introduce our data set of naturally occurring interest signals .

Modeling interestingness is fundamental to many online recommendation systems .

We obtain naturally occurring interest signals by observing Web browsing transitions where users click from one webpage to another .

We propose to model this " interestingness " with a deep semantic similarity model ( DSSM ) , based on deep neural networks with special convolutional-pooling structure , mapping source-target document pairs to feature vectors in a latent semantic space .

We train the DSSM using browsing transitions between documents .

Finally , we demonstrate the effectiveness of our model on two interestingness tasks : automatic highlighting and contextual entity search .

Our results on large-scale , real-world datasets show that the semantics of documents computed by the DSSM are important for modeling interestingness and that the new model leads to significant improvements on both tasks .

DSSM is shown to outperform not only the classic document models that do not use ( latent ) semantics but also stateof-the-art topic models that do not have the deep and convolutional architecture characterizing the DSSM .

One area of future work is to extend our method to model interestingness given an entire user session , which consists of a sequence of browsing events .

We believe that the prior browsing and interaction history recorded in the session provides additional signals for predicting interestingness .

To capture such signals , our model needs to be extended to adequately represent time series ( e.g. , causal relations and consequences of actions ) .

One potentially effective model for such a purpose is based on the architecture of recurrent neural networks ( e.g. , Mikolov et al . 2010 ; Chen and Deng , 2014 ) , which can be incorporated into the deep semantic model proposed in this paper .


Current measures for evaluating text simplification systems focus on evaluating lexical text aspects , neglecting its structural aspects .

In this paper we propose the first measure to address structural aspects of text simplification , called SAMSA .

It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output .

SAMSA provides a reference-less automatic evaluation procedure , avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence .

Our human evaluation experiments show both SAMSA 's substantial correlation with human judgments , as well as the deficiency of existing reference-based measures in evaluating structural simplification .

Text simplification ( TS ) addresses the translation of an input sentence into one or more simpler sentences .

It is a useful preprocessing step for several NLP tasks , such as machine translation ( Chandrasekar et al . , 1996 ; Mishra et al . , 2014 ) and relation extraction ( Niklaus et al . , 2016 ) , and has also been shown useful in the development of reading aids , e.g. , for people with dyslexia ( Rello et al . , 2013 ) or non-native speakers ( Siddharthan , 2002 ) .

The task has attracted much attention in the past decade ( Zhu et al . , 2010 ; Woodsend and Lapata , 2011 ; Wubben et al . , 2012 ; Siddharthan and Angrosh , 2014 ; Narayan and Gardent , 2014 ) , but has yet to converge on an evaluation protocol that yields comparable results across different methods and strongly correlates with human judgments .

This is in part due to the difficulty to combine the effects of different simplification operations ( e.g. , deletion , splitting and substitution ) .

al . ( 2016 ) has recently made considerable progress towards that goal , and proposed to tackle it both by using an improved reference-based measure , named SARI , and by increasing the number of references .

However , their research focused on lexical , rather than structural simplification , which provides a complementary view of TS quality as this paper will show .

This paper focuses on the evaluation of the structural aspects of the task .

We introduce the semantic measure SAMSA ( Simplification Automatic evaluation Measure through Semantic Annotation ) , the first structure-aware measure for TS in general , and the first to use semantic structure in this context in particular .

SAMSA stipulates that an optimal split of the input is one where each predicate-argument structure is assigned its own sentence , and measures to what extent this assertion holds for the input-output pair in question , by using semantic structure .

SAMSA focuses on the core semantic components of the sentence , and is tolerant towards the deletion of other units .

2For example , SAMSA will assign a high score to the output split " John got home .

for the input sentence " John got home and gave Mary a call .

" , as it splits each of its predicate-argument structures to a different sentence .

Splits that alter predicate-argument relations such as " John got home and gave .

Mary called . " are penalized by SAMSA .

SAMSA 's use of semantic structures for TS evaluation has several motivations .

First , it provides means to measure the extent to which the meaning of the source is preserved in the output .

Second , it provides means for measuring whether the input sentence was split to semantic units of the right granularity .

Third , defining a semantic measure that does not require references avoids the difficulties incurred by their non-uniqueness , and the difficulty in collecting high quality references , as reported by Xu et al . ( 2015 ) and by Narayan and Gardent ( 2014 ) with respect to the Parallel Wikipedia Corpus ( PWKP ; Zhu et al . , 2010 ) .

SAMSA is further motivated by its use of semantic annotation only on the source side , which allows to evaluate multiple systems using same source-side annotation , and avoids the need to parse system outputs , which can be garbled .

In this paper we use the UCCA scheme for defining semantic structure ( Abend and Rappoport , 2013 ) .

UCCA has been shown to be preserved remarkably well across translations ( Sulem et al . , 2015 ) and has also been successfully used for machine translation evaluation ( Birch et al . , 2016 )

We note , however , that SAMSA can be adapted to work with any semantic scheme that captures predicate-argument relations , such as AMR ( Banarescu et al . , 2013 ) or Discourse Representation Structures ( Kamp , 1981 ) , as used by Narayan and Gardent ( 2014 ) .

We experiment with SAMSA both where semantic annotation is carried out manually , and where it is carried out by a parser .

We conduct human rating experiments and compare the resulting system rankings with those predicted by SAMSA .

We find that SAMSA 's rankings obtain high correlations with human rankings , and compare favorably to existing referencebased measures for TS .

Moreover , our results show that existing measures , which mainly target lexical simplification , are ill-suited to predict human judgments where structural simplification is involved .

Finally , we apply SAMSA to the dataset of the QATS shared task on simplification evaluation ( Štajner et al . , 2016 ) .

We find that SAMSA obtains comparative correlation with human judgments on the task , despite operating in a more restricted setting , as it does not use human ratings as training data and focuses only on structural aspects of simplicity .

Section 5 details the collection of human judgments .

Our experimental setup for comparing our human and automatic rankings is given in Section 6 , and results are given in Section 7 , showing superior results for SAMSA .

A discussion on the results is presented in Section 8 .

Section 9 presents experiments with SAMSA on the QATS evaluation benchmark .

We presented the first structure-aware metric for text simplification , SAMSA , and the first evaluation experiments that directly target the structural simplification component , separately from the lexical component .

We argue that the structural and lexical dimensions of simplification are loosely related , and that TS evaluation protocols should assess both .

We empirically demonstrate that strong measures that assess lexical simplification quality ( notably SARI ) , fail to correlate with human judgments when structural simplification is performed by the evaluated systems .

Our experiments show that SAMSA correlates well with human judgments in such settings , which demonstrates its usefulness for evaluating and tuning statistical simplification systems , and shows that structural evaluation provides a complementary perspective on simplification quality .


Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , highdimensional feature representations .

A small subset of such features is often selected manually .

This is problematic when features lack clear linguistic meaning as in embeddings or when the information is blended across features .

In this paper , we use tensors to map high-dimensional feature vectors into low dimensional representations .

We explicitly maintain the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

Our parser consistently outperforms the Turbo and MST parsers across 14 different languages .

We also obtain the best published UAS results on 5 languages .

Finding an expressive representation of input sentences is crucial for accurate parsing .

Syntactic relations manifest themselves in a broad range of surface indicators , ranging from morphological to lexical , including positional and part-of-speech ( POS ) tagging features .

Traditionally , parsing research has focused on modeling the direct connection between the features and the predicted syntactic relations such as head-modifier ( arc ) relations in dependency parsing .

Even in the case of firstorder parsers , this results in a high-dimensional vector representation of each arc .

Discrete features , and their cross products , can be further complemented with auxiliary information about words participating in an arc , such as continuous vector representations of words .

The exploding dimensionality of rich feature vectors must then be balanced with the difficulty of effectively learning the associated parameters from limited training data .

A predominant way to counter the high dimensionality of features is to manually design or select a meaningful set of feature templates , which are used to generate different types of features ( Mc-Donald et al . , 2005a ; Koo and Collins , 2010 ; Martins et al . , 2013 ) .

Direct manual selection may be problematic for two reasons .

First , features may lack clear linguistic interpretation as in distributional features or continuous vector embeddings of words .

Second , designing a small subset of templates ( and features ) is challenging when the relevant linguistic information is distributed across the features .

For instance , morphological properties are closely tied to part-of-speech tags , which in turn relate to positional features .

Therefore , we may suffer a performance loss if we select only a small subset of the features .

On the other hand , by including all the rich features , we face over-fitting problems .

We depart from this view and leverage highdimensional feature vectors by mapping them into low dimensional representations .

We begin by representing high-dimensional feature vectors as multi-way cross-products of smaller feature vectors that represent words and their syntactic relations ( arcs ) .

The associated parameters are viewed as a tensor ( multi-way array ) of low rank , and optimized for parsing performance .

By explicitly representing the tensor in a low-rank form , we have direct control over the effective dimensionality of the set of parameters .

We obtain role-dependent low-dimensional representations for words ( head , modifier ) that are specifically tailored for parsing accuracy , and use standard online algorithms for optimizing the low-rank tensor components .

The overall approach has clear linguistic and computational advantages :

• Our low dimensional embeddings are tailored to the syntactic context of words ( head , modifier ) .

This low dimensional syntactic abstraction can be thought of as a proxy to manually constructed POS tags .

• By automatically selecting a small number of dimensions useful for parsing , we can leverage a wide array of ( correlated ) features .

Unlike parsers such as MST , we can easily benefit from auxiliary information ( e.g. , word vectors ) appended as features .

We implement the low-rank factorization model in the context of first-and third-order dependency parsing .

The model was evaluated on 14 languages , using dependency data from CoNLL 2008 and CoNLL 2006 .

We compare our results against the MST ( McDonald et al . , 2005a ) and Turbo ( Martins et al . , 2013 ) parsers .

The low-rank parser achieves average performance of 89.08 % across 14 languages , compared to 88.73 % for the Turbo parser , and 87.19 % for MST .

The power of the low-rank model becomes evident in the absence of any part-of-speech tags .

For instance , on the English dataset , the low-rank model trained without POS tags achieves 90.49 % on first-order parsing , while the baseline gets 86.70 % if trained under the same conditions , and 90.58 % if trained with 12 core POS tags .

Finally , we demonstrate that the model can successfully leverage word vector representations , in contrast to the baselines .

Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , high-dimensional feature representations .

We introduce a low-rank factorization method that enables to map high dimensional feature vectors into low dimensional representations .

Our method maintains the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

We implement the approach on first-order to third-order dependency parsing .

Our parser outperforms the Turbo and MST parsers across 14 languages .

Future work involves extending the tensor component to capture higher-order structures .

In particular , we would consider second-order structures such as grandparent-head-modifier by increasing the dimensionality of the tensor .

This tensor will accordingly be a four or five-way array .

The online update algorithm remains applicable since each dimension is optimized in an alternating fashion .

A small subset of such features is often selected manually .

Direct manual selection may be problematic for two reasons .