We address relation extraction as an analogy problem by proposing a novel approach to learn representations of relations expressed by their textual mentions .

In our assumption , if two pairs of entities belong to the same relation , then those two pairs are analogous .

Following this idea , we collect a large set of analogous pairs by matching triples in knowledge bases with web-scale corpora through distant supervision .

We leverage this dataset to train a hierarchical siamese network in order to learn entity-entity embeddings which encode relational information through the different linguistic paraphrasing expressing the same relation .

We evaluate our model in a one-shot learning task by showing a promising generalization capability in order to classify unseen relation types , which makes this approach suitable to perform automatic knowledge base population with minimal supervision .

Moreover , the model can be used to generate pretrained embeddings which provide a valuable signal when integrated into an existing neuralbased model by outperforming the state-ofthe-art methods on a downstream relation extraction task .

The task of identifying semantic relationships between entities in unstructured textual corpora , namely Relation Extraction ( RE ) , is often a prerequisite for many other natural language understanding tasks , e.g. automatic knowledge base population , question answering , etc . RE is commonly addressed as a classification task ( Bunescu et al . , 2005 ) , where a model is trained to classify relation mentions in text among a predefined set of relation types .

For instance , given the sentence " Robert Plant is the singer of the band Led Zeppelin " , an effective RE system might extract the triple memberOf(ROBERT PLANT , LED ZEP-PELIN ) , where memberOf is a relation label expressed by the linguistic context " is the singer of the band " .

Since a given relation can be expressed using different textual patterns surrounding entities , the state-of-the-art RE models which follow this approach need a considerable amount of examples for each relation to reach satisfactory performance .

Distant supervision ( Mintz et al . , 2009 ) instead uses training examples from a knowledge base , guaranteeing a large amount of ( popular ) relation examples without human intervention , which can be used effectively by neural networks ( Lin et al . , 2016 ; Glass et al . , 2018 ) .

However , even with this technique , approaching RE as a classification task presents several limitations :

( 1 ) distant supervision models are not accurate in extracting relations with a long-tailed distribution , because they typically have a small set of instances in knowledge bases ; ( 2 ) in most domains , relation types are very specific and only a few examples of each relation are available ; ( 3 ) these models can not be applied to recognize new relation types not observed during training .

In this paper , we address RE from a different perspective by reducing it to an analogy problem .

Our assumption states that if two pairs of entities , ( A , B ) and ( C , D ) , have at least one relation in common r , then those two pairs are analogous .

Viceversa , solving proportional analogies , such as A : B = C : D , consists of identifying the implicit relations shared between two pairs of entities .

For example , ROME : ITALY = PARIS : FRANCE is a valid analogy because capitalOf is a relation in common .

Based on this idea , we propose an end-to-end neural model able to measure the degree of analogical similarity between two entity pairs , instead of predicting a confidence score for each relation type .

An entity pair is represented through its mentions in a textual corpus , sequences of sentences where entities in the pair co-occur .

If a mention represents a specific relation type , then this relationship is expressed by the linguistic context surrounding the two entities .

E.g. , " Rome is the capital of Italy " or " The capital of France is Paris " referring to the example above .

Thus , given two analogous entity pairs represented by their textual mentions sets as input , the model is trained to minimize the difference between the representations of relations having the same linguistic patterns .

In other words , the model learns the different paraphrases expressing the same relation .

In our research hypothesis , a model trained in such way is able to recognize analogies between unseen entity pairs belonging to new unseen relation types by : ( 1 ) generalizing over the sequence of words in the mentions ; ( 2 ) projecting the sequence of words in the mentions into a vector space representing relational semantics .

This approach poses several research questions : ( RQ1 ) How to collect and organize a dataset for training ?

( RQ2 ) What kinds of models are effective for this task ?

( RQ3 ) How should the model be evaluated ?

Knowledge bases , such as Wikidata or DBpedia , consist of large relational data sources organized in the form of triples , predicate(SUBJECT , OBJECT ) .

We exploit this information to build a reliable set of analogous facts used as ground truth .

Then , we adopt distant supervision to retrieve relation mentions in web-scale textual corpora by matching the subject-object entities which co-occur in the same sentences ( Riedel et al . , 2010 ; ElSahar et al . , 2018 ; Glass and Gliozzo , 2018a ) .

Through this technique we can train our model on millions of analogy examples without human supervision .

Since our goal is to train a model able to compute the relational similarity given two sets of textual mentions , we use siamese networks to learn discriminative features between those two instances ( Hadsell et al . , 2006 ) .

This kind of neural network has been used in both computer vision ( Koch et al . , 2015 ) and natural language processing ( Mueller and Thyagarajan , 2016 ; Neculoiu et al . , 2016 ) in order to map two similar instances close in a feature space .

However , in our setting each instance consists of a set of mentions , therefore it is inherently a multi-instance learning task 1 .

We propose a hierarchical siamese network 1 Due to the weak supervision , the whole set of mentions with an attention mechanism at both word level ( Yang et al . , 2016 ) and at the set level ( Ilse et al . , 2018 ) in order to select the textual mention which better describes the relation .

To the best of our knowledge , this is the first application of a siamese network by pairing sets of instances , so it can be considered a novelty of this work .

We evaluate the generalization capability of our model in recognizing unseen relation types through an one-shot relational classification task introduced in this paper .

We train the parameters of the model on a subset of most frequent relations of one of three different distantly supervised datasets used in our experiments .

Then , we evaluate it on the long-tailed relations of each dataset .

During the test phase , only a single example for each unseen relation is provided .

This example is not used to update the parameters of the model as in a classification task , but rather to produce the vector representation of the relation itself .

Entity pairs having mention sets close to this representation are more likely to be analogous .

The experiments show promising results of our approach on this task , compared with the recent deep models commonly used for encoding textual representations ( Conneau et al . , 2017 ) .

However , when the number of the unseen relation types increases , the performance of our model become far from the results obtained in the one-shot image classification ( Koch et al . , 2015 ) , opening an interesting challenge for future work .

Finally , our model shows a transfer capability in other tasks through the use of its pre-trained vectors .

Indeed , a branch of the hierarchical siamese network can be used to generate entity-entity representations given sets of mentions as input , that we call analogy embeddings .

In our experiments , we integrate those representations into an existing end-to-end model based on convolutional networks ( Glass and Gliozzo , 2018b ) , outperfoming the state-of-the-art systems on two shared datasets commonly used for distantly supervised relation extraction .

In this paper , we proposed a novel approach to learn representations of relations in text .

Alignments between knowledge bases and textual corpora are used as ground truth in order to collect a set of analogies between entity pairs .

We designed a hierarchical siamese network trained to recognize those analogies .

The experiments showed the two main advantages of our approach .

First , the model can generalize on new unseen relation types , obtaining promising results in one-shot learning compared with the state-of-the-art sentence encoders .

Second , the model can generate low-rank representations can help existing neuralbased models designed for other tasks .

As future work , we plan to continue our investigation by extending the method with other ideas .

For instance , the use of positional embeddings , as well as the use of placeholders replacing the entities in the textual mentions are promising future directions .

Finally , we plan also to explore the use of analogy embeddings in other tasks , such as question answering and knowledge base population .

In this paper , we proposed a novel approach to learn representations of relations in text .

We designed a hierarchical siamese network trained to recognize those analogies .


Biomedical Concept Normalization ( BCN ) is widely used in biomedical text processing as a fundamental module .

Owing to numerous surface variants of biomedical concepts , BCN still remains challenging and unsolved .

In this paper , we exploit biomedical concept hypernyms to facilitate BCN .

We propose Biomedical Concept Normalizer with Hypernyms ( BCNH ) , a novel framework that adopts list-wise training to make use of both hypernyms and synonyms , and also employs norm constraint on the representation of hypernym-hyponym entity pairs .

The experimental results show that BCNH outperform the previous state-of-the-art model on the NCBI dataset .

Code will be available at https://github.com/ yan-cheng / BCNH .

Biomedical Concept Normalization ( BCN ) plays an important and prerequisite role in biomedical text processing .

The goal of BCN is to link the entity mention in the context to its normalized CUI ( Unique Concept Identifier ) in the biomedical dictionaries such as UMLS ( Bodenreider , 2004 ) , SNOMED-CT ( Spackman et al . , 1997 ) and MedDRA ( Brown et al . , 1999 ) .

Figure 1 is an example of BCN from NCBI dataset ( Dogan et al . , 2014 ) , the mention B-cell non-Hodgkins lymphomas should be linked to D016393 Lymphoma , B-Cell in the MEDIC ( Davis et al . , 2012 ) dictionary .

Recent works on BCN usually adopt encoders like CNN ( Li et al . , 2017 ) , LSTM ( Phan et al . , 2019 ) , ELMo ( Peters et al . , 2018 ; Schumacher et al . , 2020 ) or BioBERT ( Lee et al . , 2020 ; Fakhraei et al . , 2019 ; Ji et al . , 2020 ) to embed both the mention and the concept 's name entities , and then feed the representations to the following classifier or ranking network to determine the corresponding concept in the biomedical dictionary .

However , biomedical dictionaries are generally sparse in nature : a concept is usually provided with only CUI , referred name ( recommended concept name string ) , synonyms ( acceptable name variants , synonyms ) , and related concepts ( mainly hypernym concepts ) .

Therefore , effectively using the limited information in the biomedical dictionary where the candidate entities came from is paramount for the BCN task .

For concept 's synonym entities , recent BNE ( Phan et al . , 2019 ) and BIOSYN ( Sung et al . , 2020 ) tries to make full use of them by synonym marginalization to enhance biomedical entity representation and achieved consistent performance improvement .

Unfortunately , previous works generally ignore concept hypernym hierarchy structure , which is exactly the initial motivation of biomedical dictionary : organization of thousands of concepts under a unified and multi-level hierarchical classification schema .

We believe that leveraging hypernym information in the biomedical dictionary can improve the BCN performance based on two intuitions .

First , hard negative sampling ( Fakhraei et al . , 2019 ; Phan et al . , 2019 ) is vital for the BCN model 's discriminating ability and a hypernym is a hard negative example for its hyponym naturally .

Second , injecting the hypernym hierarchy information during the training process is beneficial for encoders , since currently used encoders like BioBERT only encodes the context semantics in biomedical corpora instead of the biomedical concept structural information .

To this end , we propose Biomedical Concept Normalizer with Hypernyms ( BCNH ) , a novel framework combining the list-wise cross entropy loss with norm constraint on hypernym-hyponym entity pairs .

Concretely , we reformulate the candidate target list as a three-level relevance list to consider both synonyms and hypernyms , and apply the list-wise cross entropy loss .

On the one hand , synonyms help to encode surface name variants , on the other hand , hypernyms help encode hierarchical structural information .

We also apply the norm constraint on the embedding of hypernym-hyponym entity pairs to further preserve the principal hypernym relation .

Specifically , for a hypernym-hyponym entity pair ( e hyper , e hypo ) , we constraint that the norm of hypernym entity e hyper is larger than that of e hyper in a multi-task manner .

We conduct experiments on the NCBI dataset and outperforms the previous state-of-the-art model .

To sum up , the contributions of this paper are as follows .

First , for the first time , we reformulate the candidate target list as a three-level relevance list and apply the list-wise loss to attend all candidate entities .

Second , we innovatively use norm constraint to model the hypernym-hyponym relation , preserving the hierarchy structure information inside the entity representation .

The proposed BCNH outperforms the previous state-of-the-art model on the NCBI dataset , leading to an improvement of 0.73 % on top1 accuracy .

In this paper , we propose BCNH to leverage hypernyms in the biomedical concept normalization task .

We adopts both list-wise training and norm constraint with the help of hypernym information .

The experimental results on the NCBI dataset show that BCNH outperforms previous state-of-the-art models .


The sentiment classification performance relies on high-quality sentiment resources .

However , these resources are imbalanced in different languages .

Cross-language sentiment classification ( CLSC ) can leverage the rich resources in one language ( source language ) for sentiment classification in a resource-scarce language ( target language ) .

Bilingual embeddings could eliminate the semantic gap between two languages for CLSC , but ignore the sentiment information of text .

This paper proposes an approach to learning bilingual sentiment word embeddings ( BSWE ) for English-Chinese CLSC .

The proposed B-SWE incorporate sentiment information of text into bilingual embeddings .

Furthermore , we can learn high-quality BSWE by simply employing labeled corpora and their translations , without relying on largescale parallel corpora .

Experiments on NLP&CC 2013 CLSC dataset show that our approach outperforms the state-of-theart systems .

Sentiment classification is a task of predicting sentiment polarity of text , which has attracted considerable interest in the NLP field .

To date , a number of corpus-based approaches ( Pang et al . , 2002 ; Pang and Lee , 2004 ; Kennedy and Inkpen , 2006 ) have been developed for sentiment classification .

The approaches heavily rely on quality and quantity of the labeled corpora , which are considered as the most valuable resources in sentiment classification task .

However , such sentiment resources are imbalanced in different languages .

To leverage resources in the source language to improve the sentiment classification performance in the target language , cross-language sentiment classification ( CLSC ) approaches have been investigated .

The traditional CLSC approaches employ machine translation ( MT ) systems to translate corpora in the source language into the target language , and train the sentiment classifiers in the target language ( Banea et al . , 2008 ) .

Directly employing the translated resources for sentiment classification in the target language is simple and could get acceptable results .

However , the gap between the source language and target language inevitably impacts the performance of sentiment classification .

To improve the classification accuracy , multiview approaches have been proposed .

In these approaches , the resources in the source language and their translations in the target language are both used to train sentiment classifiers in two independent views ( Wan , 2009 ;

Gui et al . , 2013 ; Zhou et al . , 2014a ) .

The final results are determined by ensemble classifiers in these two views to overcome the weakness of monolingual classifiers .

However , learning language-specific classifiers in each view fails to capture the common sentiment information of two languages during training process .

With the revival of interest in deep learning ( Hinton and Salakhutdinov , 2006 ) , shared deep representations ( or embeddings ) ( Bengio et al . , 2013 ) are employed for CLSC ( Chandar A P et al . , 2013 ) .

Usually , paired sentences from parallel corpora are used to learn word embeddings across languages ( Chandar A P et al . , 2013 ; Chandar A P et al . , 2014 ) , eliminating the need of MT systems .

The learned bilingual embeddings could easily project the training data and test data into a common space , where training and testing are performed .

However , high-quality bilingual embeddings rely on the large-scale task-related parallel corpora , which are not always readily available .

Meanwhile , though semantic similarities across languages are captured during bilingual embedding learning process , sentiment information of text is ignored .

That is , bilingual embeddings learned from unlabeled parallel corpora are not effective enough for CLSC because of a lack of explicit sentiment information .

Tang and Wan ( 2014 ) first proposed a bilingual sentiment embedding model using the original training data and the corresponding translations through a linear mapping rather than deep learning technique .

This paper proposes a denoising autoencoder based approach to learning bilingual sentiment word embeddings ( BSWE ) for CLSC , which incorporates sentiment polarities of text into the bilingual embeddings .

The proposed approach learns BSWE with the original labeled documents and their translations instead of parallel corpora .

The BSWE learning process consists of two phases : the unsupervised phase of semantic learning and the supervised phase of sentiment learning .

In the unsupervised phase , sentiment words and their negation features are extracted from the source training data and their translations to represent paired documents .

These features are used as inputs for a denoising autoencoder to learn the bilingual embeddings .

In the supervised phase , sentiment polarity labels of documents are used to guide BSWE learning for incorporating sentiment information into the bilingual embeddings .

The learned BSWE are applied to project English training data and Chinese test data into a common space .

In this space , a linear support vector machine ( SVM ) is used to perform training and testing .

The experiments are carried on NLP&CC 2013 CLSC dataset , including book , DVD and music categories .

Experimental results show that our approach achieves 80.68 % average accuracy , which outperforms the state-of-the-art systems on this dataset .

Although the BSWE are only evaluated on English-Chinese CLSC here , it can be popularized to many other languages .

The major contributions of this work can be summarized as follows :

• We propose bilingual sentiment word embeddings ( BSWE ) for CLSC based on deep learning technique .

Experimental results show that the proposed BSWE significantly outperform the bilingual embeddings by incorporating sentiment information .

• Instead of large-scale parallel corpora , only the labeled English corpora and Englishto-Chinese translations are required for B-SWE learning .

It is proved that in spite of the small-scale of training set , our approach outperforms the state-of-the-art systems in NLP&CC 2013 CLSC share task .

• We employ sentiment words and their negation features rather than all words in documents to learn sentiment-specific embeddings , which significantly reduces the dimension of input vectors as well as improves sentiment classification performance .

This paper proposes an approach to learning B-SWE by incorporating sentiment information into the bilingual embeddings for CLSC .

The proposed approach learns BSWE with the labeled documents and their translations rather than parallel corpora .

In addition , BDR is proposed to enhance the sentiment expression ability which combines English and Chinese representations .

Experiments on the NLP&CC 2013 CLSC dataset show that our approach outperforms the previous stateof-the-art systems as well as traditional bilingual embedding systems .

The proposed BSWE are only evaluated on English-Chinese CLSC in this paper , but it can be popularized to other languages .

Both semantic and sentiment information play an important role in sentiment classification .

In the following work , we will further investigate the relationship between semantic and sentiment information for CLSC , and balance their functions to optimize their combination for CLSC .

The proposed B-SWE incorporate sentiment information of text into bilingual embeddings .


During real-life interactions , people are naturally gesturing and modulating their voice to emphasize specific points or to express their emotions .

With the recent growth of social websites such as YouTube , Facebook , and Amazon , video reviews are emerging as a new source of multimodal and natural opinions that has been left almost untapped by automatic opinion analysis techniques .

This paper presents a method for multimodal sentiment classification , which can identify the sentiment expressed in utterance-level visual datastreams .

Using a new multimodal dataset consisting of sentiment annotated utterances extracted from video reviews , we show that multimodal sentiment analysis can be effectively performed , and that the joint use of visual , acoustic , and linguistic modalities can lead to error rate reductions of up to 10.5 % as compared to the best performing individual modality .

Video reviews represent a growing source of consumer information that gained increasing interest from companies , researchers , and consumers .

Popular web platforms such as YouTube , Amazon , Facebook , and ExpoTV have reported a significant increase in the number of consumer reviews in video format over the past five years .

Compared to traditional text reviews , video reviews provide a more natural experience as they allow the viewer to better sense the reviewer 's emotions , beliefs , and intentions through richer channels such as intonations , facial expressions , and body language .

Much of the work to date on opinion analysis has focused on textual data , and a number of resources have been created including lexicons ( Wiebe and Riloff , 2005 ; Esuli and Sebastiani , 2006 ) or large annotated datasets ( Maas et al . , 2011 ) .

Given the accelerated growth of other media on the Web and elsewhere , which includes massive collections of videos ( e.g. , YouTube , Vimeo , VideoLectures ) , images ( e.g. , Flickr , Picasa ) , audio clips ( e.g. , podcasts ) , the ability to address the identification of opinions in the presence of diverse modalities is becoming increasingly important .

This has motivated researchers to start exploring multimodal clues for the detection of sentiment and emotions in video content ( Morency et al . , 2011 ; Wagner et al . , 2011 ) .

In this paper , we explore the addition of speech and visual modalities to text analysis in order to identify the sentiment expressed in video reviews .

Given the non homogeneous nature of full-video reviews , which typically include a mixture of positive , negative , and neutral statements , we decided to perform our experiments and analyses at the utterance level .

This is in line with earlier work on text-based sentiment analysis , where it has been observed that full-document reviews often contain both positive and negative comments , which led to a number of methods addressing opinion analysis at sentence level .

Our results show that relying on the joint use of linguistic , acoustic , and visual modalities allows us to better sense the sentiment being expressed as compared to the use of only one modality at a time .

Another important aspect of this paper is the introduction of a new multimodal opinion database annotated at the utterance level which is , to our knowledge , the first of its kind .

In our work , this dataset enabled a wide range of multimodal sentiment analysis experiments , addressing the relative importance of modalities and individual features .

The following section presents related work in text-based sentiment analysis and audio-visual emotion recognition .

Section 3 describes our new multimodal datasets with utterance-level sentiment annotations .

Section 4 presents our multimodal sen-timent analysis approach , including details about our linguistic , acoustic , and visual features .

Our experiments and results on multimodal sentiment classification are presented in Section 5 , with a detailed discussion and analysis in Section 6 .

In this paper , we presented a multimodal approach for utterance-level sentiment classification .

We introduced a new multimodal dataset consisting AU6 AU12 AU45 AUs 1,1 + 4 of sentiment annotated utterances extracted from video reviews , where each utterance is associated with a video , acoustic , and linguistic datastream .

Our experiments show that sentiment annotation of utterance-level visual datastreams can be effectively performed , and that the use of multiple modalities can lead to error rate reductions of up to 10.5 % as compared to the use of one modality at a time .

In future work , we plan to explore alternative multimodal fusion methods , such as decision-level and meta-level fusion , to improve the integration of the visual , acoustic , and linguistic modalities .


In modern machine translation practice , a statistical phrasal or hierarchical translation system usually relies on a huge set of translation rules extracted from bi-lingual training data .

This approach not only results in space and efficiency issues , but also suffers from the sparse data problem .

In this paper , we propose to use factorized grammars , an idea widely accepted in the field of linguistic grammar construction , to generalize translation rules , so as to solve these two problems .

We designed a method to take advantage of the XTAG English Grammar to facilitate the extraction of factorized rules .

We experimented on various setups of low-resource language translation , and showed consistent significant improvement in BLEU over state-ofthe-art string-to-dependency baseline systems with 200 K words of bi-lingual training data .

A statistical phrasal ( Koehn et al . , 2003 ; Och and Ney , 2004 ) or hierarchical ( Chiang , 2005 ; Marcu et al . , 2006 ) machine translation system usually relies on a very large set of translation rules extracted from bi-lingual training data with heuristic methods on word alignment results .

According to our own experience , we obtain about 200 GB of rules from training data of about 50 M words on each side .

This immediately becomes an engineering challenge on space and search efficiency .

A common practice to circumvent this problem is to filter the rules based on development sets in the step of rule extraction or before the decoding phrase , instead of building a real distributed system .

However , this strategy only works for research systems , for which the segments for translation are always fixed .

However , do we really need such a large rule set to represent information from the training data of much smaller size ?

Linguists in the grammar construction field already showed us a perfect solution to a similar problem .

The answer is to use a factorized grammar .

Linguists decompose lexicalized linguistic structures into two parts , ( unlexicalized ) templates and lexical items .

Templates are further organized into families .

Each family is associated with a set of lexical items which can be used to lexicalize all the templates in this family .

For example , the XTAG English Grammar ( XTAG-Group , 2001 ) , a hand-crafted grammar based on the Tree Adjoining Grammar ( TAG ) ( Joshi and Schabes , 1997 ) formalism , is a grammar of this kind , which employs factorization with LTAG e-tree templates and lexical items .

Factorized grammars not only relieve the burden on space and search , but also alleviate the sparse data problem , especially for low-resource language translation with few training data .

With a factored model , we do not need to observe exact " template -lexical item " occurrences in training .

New rules can be generated from template families and lexical items either offline or on the fly , explicitly or implicitly .

In fact , the factorization approach has been successfully applied on the morphological level in previous study on MT ( Koehn and Hoang , 2007 ) .

In this work , we will go further to investigate factorization of rule structures by exploiting the rich XTAG English Grammar .

We evaluate the effect of using factorized translation grammars on various setups of low-resource language translation , since low-resource MT suffers greatly on poor generalization capability of trans-lation rules .

With the help of high-level linguistic knowledge for generalization , factorized grammars provide consistent significant improvement in BLEU ( Papineni et al . , 2001 ) over string-todependency baseline systems with 200 K words of bi-lingual training data .

This work also closes the gap between compact hand-crafted translation rules and large-scale unorganized automatic rules .

This may lead to a more effective and efficient statistical translation model that could better leverage generic linguistic knowledge in MT .

In the rest of this paper , we will first provide a short description of our baseline system in Section 2 .

Then , we will introduce factorized translation grammars in Section 3 .

We will illustrate the use of the XTAG English Grammar to facilitate the extraction of factorized rules in Section 4 .

Implementation details are provided in Section 5 .

Experimental results are reported in Section 6 .

In this paper , we proposed a novel statistical machine translation model using a factorized structurebased translation grammar .

This model not only alleviates the sparse data problem but only relieves the burden on space and search , both of which are imminent issues for the popular phrasal and/or hierarchical MT systems .

We took low-resource language translation , especially X-to-English translation tasks , for case study .

We designed a method to exploit family information in the XTAG English Grammar to facilitate the extraction of factorized rules .

We tested the new model on low-resource translation , and the use of factorized models showed significant improvement in BLEU on systems with 200 K words of bi-lingual training data of various language pairs and genres .

The factorized translation grammar proposed here shows an interesting way of using richer syntactic resources , with high potential for future research .

In future , we will explore various learning methods for better estimation of families , templates and lexical items .

The target linguistic knowledge that we used in this paper will provide a nice starting point for unsupervised learning algorithms .

We will also try to further exploit the factorized representation with discriminative learning .

Features defined on templates and families will have good generalization capability .


Recent developments in statistical modeling of various linguistic phenomena have shown that additional features give consistent performance improvements .

Quite often , improvements are limited by the number of features a system is able to explore .

This paper describes a novel progressive training algorithm that selects features from virtually unlimited feature spaces for conditional maximum entropy ( CME ) modeling .

Experimental results in edit region identification demonstrate the benefits of the progressive feature selection ( PFS ) algorithm : the PFS algorithm maintains the same accuracy performance as previous CME feature selection algorithms ( e.g. , Zhou et al . , 2003 ) when the same feature spaces are used .

When additional features and their combinations are used , the PFS gives 17.66 % relative improvement over the previously reported best result in edit region identification on Switchboard corpus ( Kahn et al . , 2005 ) , which leads to a 20 % relative error reduction in parsing the Switchboard corpus when gold edits are used as the upper bound .

Conditional Maximum Entropy ( CME ) modeling has received a great amount of attention within natural language processing community for the past decade ( e.g. , Berger et al . , 1996 ; Reynar and Ratnaparkhi , 1997 ; Koeling , 2000 ; Malouf , 2002 ;

Zhou et al . , 2003 ; Riezler and Vasserman , 2004 ) .

One of the main advantages of CME modeling is the ability to incorporate a variety of features in a uniform framework with a sound mathematical foundation .

Recent improvements on the original incremental feature selection ( IFS ) algorithm , such as Malouf ( 2002 ) and Zhou et al . ( 2003 ) , greatly speed up the feature selection process .

However , like many other statistical modeling algorithms , such as boosting ( Schapire and Singer , 1999 ) and support vector machine ( Vapnik 1995 ) , the algorithm is limited by the size of the defined feature space .

Past results show that larger feature spaces tend to give better results .

However , finding a way to include an unlimited amount of features is still an open research problem .

In this paper , we propose a novel progressive feature selection ( PFS ) algorithm that addresses the feature space size limitation .

The algorithm is implemented on top of the Selective Gain Computation ( SGC )

algorithm ( Zhou et al . , 2003 ) , which offers fast training and high quality models .

Theoretically , the new algorithm is able to explore an unlimited amount of features .

Because of the improved capability of the CME algorithm , we are able to consider many new features and feature combinations during model construction .

To demonstrate the effectiveness of our new algorithm , we conducted a number of experiments on the task of identifying edit regions , a practical task in spoken language processing .

Based on the convention from Shriberg ( 1994 ) and Charniak and Johnson ( 2001 ) , a disfluent spoken utterance is divided into three parts : the reparandum , the part that is repaired ; the inter-regnum , which can be filler words or empty ; and the repair / repeat , the part that replaces or repeats the reparandum .

The first two parts combined are called an edit or edit region .

interregnum It is , you know , this is a tough problem .

This paper presents our progressive feature selection algorithm that greatly extends the feature space for conditional maximum entropy modeling .

The new algorithm is able to select features from feature space in the order of tens of millions in practice , i.e. , 8 times the maximal size previous algorithms are able to process , and unlimited space size in theory .

Experiments on edit region identification task have shown that the increased feature space leads to 17.66 % relative improvement ( or 3.85 % absolute ) over the best result reported by Kahn et al . ( 2005 ) , and 10.65 % relative improvement ( or 2.14 % absolute ) over the new baseline SGC algorithm with all the variables from Zhang and Weng ( 2005 ) .

We also show that symbolic prosody labels together with confidence scores are useful in edit region identification task .

In addition , the improvements in the edit identification lead to a relative 20 % error reduction in parsing disfluent sentences when gold edits are used as the upper bound .


This study tackles the problem of paraphrase acquisition : achieving high coverage as well as accuracy .

Our method first induces paraphrase patterns from given seed paraphrases , exploiting the generality of paraphrases exhibited by pairs of lexical variants , e.g. , " amendment " and " amending , " in a fully empirical way .

It then searches monolingual corpora for new paraphrases that match the patterns .

This can extract paraphrases comprising words that are completely different from those of the given seeds .

In experiments , our method expanded seed sets by factors of 42 to 206 , gaining 84 % to 208 % more coverage than a previous method that generalizes only identical word forms .

Human evaluation through a paraphrase substitution test demonstrated that the newly acquired paraphrases retained reasonable quality , given substantially high-quality seeds .

One of the characteristics of human languages is that the same semantic content can be expressed using several different linguistic expressions , i.e. , paraphrases .

Dealing with paraphrases is an important issue in a broad range of natural language processing ( NLP ) tasks ( Madnani and Dorr , 2010 ; Androutsopoulos and Malakasiotis , 2010 ) .

To adequately and robustly deal with paraphrases , a large-scale knowledge base containing words and phrases having approximately the same meaning is indispensable .

Thus , the task of automatically creating such large-scale paraphrase lexicons has been drawing the attention of many researchers ( see Section 2 for details ) .

The challenge is to en-sure substantial coverage along with high accuracy despite the natural tension between these factors .

Among the different types of language resources , monolingual corpora1 offer the largest coverage , but the quality of the extracted candidates is generally rather low .

The difficulty lies in the manner of distinguishing paraphrases from expressions that stand in different semantic relations , e.g. , antonyms and sibling words , using only the statistics estimated from such corpora .

In contrast , highly accurate paraphrases can be extracted from parallel or comparable corpora , but their coverage is limited owing to the limited availability of such corpora for most languages .

This study aims to improve coverage while maintaining accuracy .

To that end , we propose a method that exploits the generality exhibited by pairs of lexical variants .

Given a seed set of paraphrase pairs , our method first induces paraphrase patterns by generalizing not only identical word forms ( Fujita et al . , 2012 ) but also pairs of lexical variants .

For instance , from a seed pair ( 1a ) , a pattern ( 1b ) is acquired , where the pair of lexical variants ( " amendment " , " amending " ) and the shared word form " regulation " are generalized .

( 1 ) a. amendment of regulation ⇔ amending regulation b. X : ment of Y : ϕ ⇔ X : ing Y : ϕ

With such patterns , new paraphrase pairs that would have been missed using only the surface forms are extracted from a monolingual corpus .

Obtainable pairs can include those comprising words that are completely different from those of the seed paraphrases , e.g. , ( 2a ) and ( 2b ) .

( 2 ) a. investment of resources ⇔ investing resources b. recruitment of engineers ⇔ recruiting engineers While the generality underlying paraphrases has been exploited either by handcrafted rules ( Harris , 1957 ; Mel'čuk and Polguère , 1987 ; Jacquemin , 1999 ; Fujita et al . , 2007 ) or by data-driven techniques ( Ganitkevitch et al . , 2011 ; Fujita et al . , 2012 ) , we still lack a robust and accurate way of identifying various types of lexical variants .

Our method tackles this issue using affix patterns that are also acquired from high-quality seed paraphrases in a fully empirical way .

Consequently , our method has the potential to apply to many languages .

We proposed a method for expanding given paraphrase lexicons by first inducing paraphrase patterns and then searching monolingual corpora with these patterns for new paraphrase pairs .

To the best of our knowledge , this is the first attempt to exploit various types of lexical variants for acquiring paraphrases in a completely empirical way .

Our method requires minimal language-dependent resources , i.e. , stoplists and tokenizers , other than raw corpora .

We demonstrated the quantitative impact of our method and confirmed the potential quality of the expanded paraphrase lexicon .

( i ) Paraphrase lexicons created by different methods and sources have different properties .

Designing an overall model to harmonize such heterogeneous lexicons is an important issue .

( ii ) We aim to investigate an extensive collection of corpora : there are far more corpora than those we used in this experiment .

We are also interested in expanding paraphrase lexicons created by a method other than bilingual pivoting ; for instance , those extracted from a Web-harvested monolingual comparable corpus ( Hashimoto et al . , 2011 ; Yan et al . , 2013 ) .

( iii ) We will apply our method to various languages for demonstrating its applicability , extending it for a wider range of lexical variants depending on the targeted language .

( iv ) Paraphrases are the fundamental linguistic phenomena that affect a wide range of NLP tasks .

We are therefore interested in determining to what extent our paraphrase lexicons can improve the performance of application tasks such as machine translation , text summarization , and text simplification .


Social media is a rich source of rumours and corresponding community reactions .

Rumours reflect different characteristics , some shared and some individual .

We formulate the problem of classifying tweet level judgements of rumours as a supervised learning task .

Both supervised and unsupervised domain adaptation are considered , in which tweets from a rumour are classified on the basis of other annotated rumours .

We demonstrate how multi-task learning helps achieve good results on rumours from the 2011 England riots .

There is an increasing need to interpret and act upon rumours spreading quickly through social media , especially in circumstances where their veracity is hard to establish .

For instance , during an earthquake in Chile rumours spread through Twitter that a volcano had become active and that there was a tsunami warning in Valparaiso ( Mendoza et al . , 2010 ) .

Other examples , from the riots in England in 2011 , were that rioters were going to attack Birmingham 's children hospital and that animals had escaped from the zoo ( Procter et al . , 2013 ) .

Social scientists ( Procter et al . , 2013 ) analysed manually a sample of tweets expressing different judgements towards rumours and categorised them manually in supporting , denying or questioning .

The goal here is to carry out tweet-level judgement classification automatically , in order to assist in ( near ) real-time rumour monitoring by journalists and authorities ( Procter et al . , 2013 ) .

In addition , information about tweet-level judgements has been used as a first step for early rumour detection by ( Zhao et al . , 2015 ) .

The focus here is on tweet- ( Qazvinian et al . , 2011 ) or proposed regular expressions as a solution ( Zhao et al . , 2015 ) .

We expect posts expressing similar opinions to exhibit many similar characteristics across different rumours .

Based on the assumption of a common underlying linguistic signal , we build a transfer learning system that labels newly emerging rumours for which we have little or no annotated data .

Results demonstrate that Gaussian Processbased multi task learning allows for significantly improved performance .

The novel contributions of this paper are : 1 .

Formulating the problem of classifying judgements of rumours in both supervised and unsupervised domain adaptation settings .

2 . Showing how a multi-task learning approach outperforms singletask methods .

This paper investigated the problem of classifying judgements expressed in tweets about rumours .

First , we considered a setting where no training data from target rumour is available ( LOO ) .

Without access to annotated examples of the target rumour the learning problem becomes very difficult .

We showed that in the supervised domain adaptation setting ( LPO ) even annotating a small number of tweets helps to achieve better results .

Moreover , we demonstrated the benefits of a multi task learning approach , as well as that Brown cluster features are more useful for the task than simple bag of words .

Judgement estimation is undoubtedly of great value e.g. for marketing , politics and journalism , helping to target widely believed topics .

Although the focus here is on classifying community reactions , Castillo et al . ( 2013 ) showed that community reaction is correlated with actual rumour veracity .

Consequently our classification methods may prove useful in the broader and more challenging task of annotating veracity .

An interesting direction for future work would be adding non-textual features .

For example , the rumour diffusion pattern ( Lukasik et al . , 2015 ) may be a useful cue for judgement classification .


In neural machine translation , a source sequence of words is encoded into a vector from which a target sequence is generated in the decoding phase .

Differently from statistical machine translation , the associations between source words and their possible target counterparts are not explicitly stored .

Source and target words are at the two ends of a long information processing procedure , mediated by hidden states at both the source encoding and the target decoding phases .

This makes it possible that a source word is incorrectly translated into a target word that is not any of its admissible equivalent counterparts in the target language .

Neural machine translation ( NMT ) is an endto-end approach to machine translation that has achieved competitive results vis-a-vis statistical machine translation ( SMT ) on various language pairs ( Bahdanau et al . , 2015 ; Cho et al . , 2014 ; Sutskever et al . , 2014 ; Luong and Manning , 2015 ) .

In NMT , the sequence-to-sequence ( seq2seq ) model learns word embeddings for both source and target words synchronously .

However , as illustrated in Figure 1 , source and target word embeddings are at the two ends of a long information processing procedure .

The individual associations between them will gradually become loose due to the separation of source-side hidden states ( represented by h 1 , . .

, h T in Fig . 1 ) and a target- side hidden state ( represented by s t in Fig . 1 ) .

As a result , in the absence of a more tight interaction between source and target word pairs , the seq2seq model in NMT produces tentative translations that contain incorrect alignments of source words with target counterparts that are non-admissible equivalents in any possible translation context .

Differently from SMT , in NMT an attention model is adopted to help align output with input words .

The attention model is based on the estimation of a probability distribution over all input words for each target word .

Word alignments with attention weights can then be easily deduced from such distributions and support the translation .

Nevertheless , sometimes one finds translations by NMT that contain surprisingly wrong word alignments , that would unlikely occur in SMT .

For instance , Figure 2 shows two Chineseto-English translation examples by NMT .

In the top example , the NMT seq2seq model incorrectly aligns the target side end of sentence mark eos to 下旬 / late with a high attention weight ( 0.80 in this example ) due to the failure of appropriately capturing the similarity , or the lack of it , between the source word 下旬 / late and the target eos .

It is also worth noting that , as 本 / this and 月 / month end up not being translated in this example , inappropriate alignment of target side eos is likely the responsible factor for under translation in NMT as the decoding process ends once a target eos is generated .

Statistics on our development data show that as much as 50 % of target side eos do not properly align to source side eos .

The second example in Figure 2 shows another case where source words are translated into target items that are not their possible translations in that or in any other context .

In particular , 冬奥 会 / winter olympics is incorrectly translated into a target comma " , " and 载誉 / honors into have .

In this paper , to address the problem illustrated above , we seek to shorten the distance within the seq2seq NMT information processing procedure between source and target word embeddings .

This is a method we term as bridging , and can be conceived as strengthening the focus of the attention mechanism into more translation-plausible source and target word alignments .

In doing so , we hope that the seq2seq model is able to learn more appropriate word alignments between source and target words .

We propose three simple yet effective strategies to bridge between word embeddings .

The inspiring insight in all these three models is to move source word embeddings closer to target word embeddings along the seq2seq NMT information processing procedure .

We categorize these strategies in terms of how close the source and target word embeddings are along that procedure , schematically depicted in Fig . 1 .

( 1 ) Source-side bridging model : Our first strategy for bridging , which we call source-side bridging , is to move source word embeddings just one step closer to the target end .

Each source word embedding is concatenated with the respective source hidden state at the same position so that the attention model can more closely benefit from source word embeddings to produce word alignments .

( 2 ) Target-side bridging model : In a second more bold strategy , we seek to incorporate relevant source word embeddings more closely into the prediction of the next target hidden state .

In particular , the most appropriate source words are selected according to their attention weights and they are made to more closely interact with target hidden states .

( 3 ) Direct bridging model : The third model consists of directly bridging between source and target word embeddings .

The training objective is optimized towards minimizing the distance between target word embeddings and their most relevant source word embeddings , selected according to the attention model .

Experiments on Chinese-English translation with extensive analysis demonstrate that directly bridging word embeddings at the two ends can produce better word alignments and thus achieve better translation .

We have presented three models to bridge source and target word embeddings for NMT .

The three models seek to shorten the distance between source and target word embeddings along the extensive information procedure in the encoderdecoder neural network .

Experiments on Chinese to English translation shows that the proposed models can significantly improve the translation quality .

Further in-depth analysis demonstrate that our models are able ( 1 ) to learn better word alignments than the baseline NMT , ( 2 ) to alleviate the notorious problems of over and under translation in NMT , and ( 3 ) to learn direct mappings between source and target words .

In future work , we will explore further strategies to bridge the source and target side for sequence-to-sequence and tree-based NMT .

Additionally , we also intend to apply these methods to other sequence-to-sequence tasks , including natural language conversation .


Reordering model is important for the statistical machine translation ( SMT ) .

Current phrase-based SMT technologies are good at capturing local reordering but not global reordering .

This paper introduces syntactic knowledge to improve global reordering capability of SMT system .

Syntactic knowledge such as boundary words , POS information and dependencies is used to guide phrase reordering .

Not only constraints in syntax tree are proposed to avoid the reordering errors , but also the modification of syntax tree is made to strengthen the capability of capturing phrase reordering .

Furthermore , the combination of parse trees can compensate for the reordering errors caused by single parse tree .

Finally , experimental results show that the performance of our system is superior to that of the state-of-the-art phrase-based SMT system .

In the last decade , statistical machine translation ( SMT ) has been widely studied and achieved good translation results .

Two kinds of SMT system have been developed , one is phrase-based SMT and the other is syntax-based SMT .

In phrase-based SMT systems ( Koehn et al . , 2003 ; Koehn , 2004 ) , foreign sentences are firstly segmented into phrases which consists of adjacent words .

Then source phrases are translated into target phrases respectively according to knowledge usually learned from bilingual parallel corpus .

Fi-nally the most likely target sentence based on a certain statistical model is inferred by combining and reordering the target phrases with the aid of search algorithm .

On the other hand , syntax-based SMT systems ( Liu et al . , 2006 ; Yamada et al . , 2001 ) mainly depend on parse trees to complete the translation of source sentence .

As studied in previous SMT projects , language model , translation model and reordering model are the three major components in current SMT systems .

Due to the difference between the source and target languages , the order of target phrases in the target sentence may differ from the order of source phrases in the source sentence .

To make the translation results be closer to the target language style , a mathematic model based on the statistic theory is constructed to reorder the target phrases .

This statistic model is called as reordering model .

As shown in Figure 1 , the order of the translations of " 欧元 " and " 的 " is changed .

The order of the translation of " 欧元 / 的 " and " 大幅 / 升值 " is altered as well .

The former reordering case with the smaller distance is usually referred as local reordering and the latter with the longer distance reordering as global reordering .

Phrase-based SMT system can effectively capture the local word reordering information which is common enough to be observed in training data .

But it is hard to model global phrase reordering .

Although syntactic knowledge used in syntax-based SMT systems can help reorder phrases , the resulting model is usually much more complicated than a phrase-based system .

There have been considerable amount of efforts to improve the reordering model in SMT systems , ranging from the fundamental distance-based distortion model ( Och and Ney , 2004 ; Koehn et al . , 2003 ) , flat reordering model ( Wu , 1996 ; Zens et al . , 2004 ; Kumar et al . , 2005 ) , to lexicalized reordering model ( Tillmann , 2004 ; Kumar et al . , 2005 ; Koehn et al . , 2005 ) , hierarchical phrase-based model ( Chiang , 2005 ) , and maximum entropy-based phrase reordering model ( Xiong et al . , 2006 ) .

Due to the absence of syntactic knowledge in these systems , the ability to capture global reordering knowledge is not powerful .

Although syntax-based SMT systems ( Yamada et al . , 2001 ; Quirk et al . , 2005 ; Liu et al . , 2006 ) are good at modeling global reordering , their performance is subject to parsing errors to a large extent .

In this paper , we propose a new method to improve reordering model by introducing syntactic information .

Syntactic knowledge such as boundary of sub-trees , part-of-speech ( POS ) and dependency relation is incorporated into the SMT system to strengthen the ability to handle global phrase reordering .

Our method is different from previous syntax-based SMT systems in which the translation process was modeled based on specific syntactic structures , either phrase structures or dependency relations .

In our system , syntactic knowledge is used just to decide where we should combine adjacent phrases and what their reordering probability is .

For example , according to the syntactic information in Figure 1 , the phrase translation combination should take place between " 大幅 " and " 升值 " rather than between " 的 " and " 大幅 " .

Moreover , the non-monotone phrase reordering should occur between " 欧元 / 的 " and " 大幅 / 升值 " rather than between " 欧元 / 的 " and " 大幅 " .

We train a maxi-mum entropy model , which is able to integrate rich syntactic knowledge , to estimate phrase reordering probabilities .

To enhance the performance of phrase reordering model , some modification on the syntax trees are also made to relax the phrase reordering constraints .

Additionally , the combination of other kinds of syntax trees is introduced to overcome the deficiency of single parse tree .

The experimental results show that the performance of our system is superior to that of the state-of-art phrasebased SMT system .

The roadmap of this paper is : Section 2 gives the related work .

Section 4 explains the generalization of reordering knowledge .

The procedures of training and decoding are described in Section 5 and Section 6 respectively .

The experimental results are shown in Section 7 .

In this paper , syntactic knowledge is introduced to capture global reordering of SMT system .

This method can not only inherit the advantage of local reordering ability of standard phrase-based SMT system , but also capture the global reordering as the syntax-based SMT system .

The experimental results showed the effectiveness of our method .

In the future work , we plan to improve the reordering model by introducing N-best syntax trees and exploiting richer syntactic knowledge .


We introduce the task of incremental semantic role labeling ( iSRL ) , in which semantic roles are assigned to incomplete input ( sentence prefixes ) .

iSRL is the semantic equivalent of incremental parsing , and is useful for language modeling , sentence completion , machine translation , and psycholinguistic modeling .

We propose an iSRL system that combines an incremental TAG parser with a semantically enriched lexicon , a role propagation algorithm , and a cascade of classifiers .

Our approach achieves an SRL Fscore of 78.38 % on the standard CoNLL 2009 dataset .

It substantially outperforms a strong baseline that combines gold-standard syntactic dependencies with heuristic role assignment , as well as a baseline based on Nivre 's incremental dependency parser .

Humans are able to assign semantic roles such as agent , patient , and theme to an incoming sentence before it is complete , i.e. , they incrementally build up a partial semantic representation of a sentence prefix .

The athlete realized [ her goals ] PATIENT / THEME were out of reach .

When reaching the noun phrase her goals , the human language processor is faced with a semantic role ambiguity : her goals can either be the PA-TIENT of the verb realize , or it can be the THEME of a subsequent verb that has not been encountered yet .

Experimental evidence shows that the human language processor initially prefers the PA-TIENT role , but switches its preference to the theme role when it reaches the subordinate verb were .

Such semantic garden paths occur because human language processing occurs word-by-word , and are well attested in the psycholinguistic literature ( e.g. , Pickering et al . , 2000 ) .

Computational systems for performing semantic role labeling ( SRL ) , on the other hand , proceed non-incrementally .

They require the whole sentence ( typically together with its complete syntactic structure ) as input and assign all semantic roles at once .

The reason for this is that most features used by current SRL systems are defined globally , and can not be computed on sentence prefixes .

In this paper , we propose incremental SRL ( iSRL ) as a new computational task that mimics human semantic role assignment .

The aim of an iSRL system is to determine semantic roles while the input unfolds : given a sentence prefix and its partial syntactic structure ( typically generated by an incremental parser ) , we need to ( a ) identify which words in the input participate in the semantic roles as arguments and predicates ( the task of role identification ) , and ( b ) assign correct semantic labels to these predicate / argument pairs ( the task of role labeling ) .

Performing these two tasks incrementally is substantially harder than doing it non-incrementally , as the processor needs to commit to a role assignment on the basis of incomplete syntactic and semantic information .

As an example , take ( 1 ): on reaching athlete , the processor should assign this word the AGENT role , even though it has not seen the corresponding predicate yet .

Similarly , upon reaching realized , the processor can complete the AGENT role , but it should also predict that this verb also has a PATIENT role , even though it has not yet encountered the argument that fills this role .

A system that performs SRL in a fully incremental fashion therefore needs to be able to assign incomplete semantic roles , unlike existing full-sentence SRL models .

The uses of incremental SRL mirror the applications of incremental parsing : iSRL models can be used in language modeling to assign better string probabilities , in sentence completion systems to provide semantically informed completions , in any real time application systems , such as dialog processing , and to incrementalize applications such as machine translation ( e.g. , in speech-tospeech MT ) .

Crucially , any comprehensive model of human language understanding needs to combine an incremental parser with an incremental semantic processor ( Padó et al . , 2009 ; Keller , 2010 ) .

The present work takes inspiration from the psycholinguistic modeling literature by proposing an iSRL system that is built on top of a cognitively motivated incremental parser , viz . , the Psycholinguistically Motivated Tree Adjoining Grammar parser of Demberg et al . ( 2013 ) .

This parser includes a predictive component , i.e. , it predicts syntactic structure for upcoming input during incremental processing .

This makes PLTAG particularly suitable for iSRL , allowing it to predict incomplete semantic roles as the input string unfolds .

Competing approaches , such as iSRL based on an incremental dependency parser , do not share this advantage , as we will discuss in Section 4.3 .

In this paper , we introduced the new task of incremental semantic role labeling and proposed a system that solves this task by combining an incremental TAG parser with a semantically enriched lexicon , a role propagation algorithm , and a cascade of classifiers .

This system achieved a fullsentence SRL F-score of 78.38 % on the standard CoNLL dataset .

Not only is the full-sentence score considerably higher than the Majority-Baseline ( which is a strong baseline , as it uses gold-standard syntactic dependencies ) , but we also observe that our iSRL system performs well incrementally , i.e. , it predicts both complete and incomplete semantic role triples correctly early on in the sentence .

We attributed this to the fact that our TAG-based architecture makes it possible to predict upcoming syntactic structure together with the corresponding semantic roles .


Generating fluent natural language responses from structured semantic representations is a critical step in task-oriented conversational systems .

Avenues like the E2E NLG Challenge have encouraged the development of neural approaches , particularly sequence-tosequence ( Seq2Seq ) models for this problem .

The semantic representations used , however , are often underspecified , which places a higher burden on the generation model for sentence planning , and also limits the extent to which generated responses can be controlled in a live system .

In this paper , we ( 1 ) propose using tree-structured semantic representations , like those used in traditional rule-based NLG systems , for better discourse-level structuring and sentence-level planning ; ( 2 ) introduce a challenging dataset using this representation for the weather domain ; ( 3 ) introduce a constrained decoding approach for Seq2Seq models that leverages this representation to improve semantic correctness ; and ( 4 ) demonstrate promising results on our dataset and the E2E dataset .

With their end-to-end trainability , neural approaches to natural language generation ( NNLG ) , particularly sequence-to-sequence ( Seq2Seq ) models , have been promoted with great fanfare in recent years ( Wen et al . , 2015 ( Wen et al . , , 2016 ; ; Mei et al . , 2016 ; Kiddon et al . , 2016 ; Dušek and Jurcicek , 2016 ) , and avenues like the recent E2E NLG challenge ( Dušek et al . , 2018 ( Dušek et al . , , 2019 ) ) have made available large datasets to promote the development of these models .

Nevertheless , current NNLG models arguably remain inadequate for most real-world task-oriented dialogue systems , given their inability to ( i ) reliably perform common sentence planning and discourse structuring operations ( Reed et al . , 2018 ) , ( ii ) generalize to complex inputs ( Wiseman et al . , 2017 ) , and ( 3 ) avoid generating texts with semantic errors including hallucinated content ( Dušek et al . , 2018 ( Dušek et al . , , 2019 ) ) .

1In this paper , we explore the extent to which these issues can be addressed by incorporating lessons from pre-neural NLG systems into a neural framework .

We begin by arguing in favor of enriching the input to neural generators to include discourse relations -long taken to be central in traditional NLG -and underscore the importance of exerting control over these relations when generating text , particularly when using user models to structure responses .

In a closely related work , Reed et al . ( 2018 ) , the authors add control tokens ( to indicate contrast and sentence structure ) to a flat input MR , and show that these can be effectively used to control structure .

However , their methods are only able to control the presence or absence of these relations , without more fine-grained control over their structure .

We thus go beyond their approach and propose using full tree structures as inputs , and generating treestructured outputs as well .

This allows us to define a novel method of constrained decoding for standard sequence-to-sequence models for generation , which helps ensure that the generated text contains all and only the specified content , as in classic approaches to surface realization .

On the E2E dataset , our experiments demonstrate much better control over CONTRAST relations than using Reed et al . 's method , and also show improved diversity and expressiveness over standard baselines .

We also release a new dataset of responses in the weather domain , which includes the JUSTIFY , JOIN and CONTRAST rela- tions , and where discourse-level structures come into play .

On both E2E and weather datasets , we show that constrained decoding over our enriched inputs results in higher semantic correctness as well as better generalizability and data efficiency .

The rest of this paper is organized as follows : Section 2 describes the motivation for using compositional inputs organized around discourse relations .

Section 3 explains our data collection approach and dataset .

2 Section 4 shows how to incorporate compositional inputs into NNLG and describes our constrained decoding algorithm .

Section 5 presents our experimental setup and results .

2 Towards More Expressive Meaning Representations

We show that using rich tree-structured meaning representations can improve expressiveness and semantic correctness in generation .

We also propose a constrained decoding technique that leverages tree-structured MRs to exert precise control over the discourse structure and semantic correctness of the generated text .

We release a challenging new dataset for the weather domain and an enriched E2E dataset that include tree-structured MRs .

Our experiments show that constrained decoding , together with tree-structured MRs , can greatly improve semantic correctness as well as enhance data efficiency and generalizability .


Large language models ( LM ) based on Transformers allow to generate plausible long texts .

In this paper , we explore how this generation can be further controlled at decoding time to satisfy certain constraints ( e.g. being nontoxic , conveying certain emotions , using a specific writing style , etc . ) without fine-tuning the LM .

Precisely , we formalize constrained generation as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint .

This approach , in addition to being easier and cheaper to train than fine-tuning the LM , allows to apply the constraint more finely and dynamically .

We propose several original methods to search this generation tree , notably the Monte Carlo Tree Search ( MCTS ) which provides theoretical guarantees on the search efficiency , but also simpler methods based on reranking a pool of diverse sequences using the discriminator scores .

These methods are evaluated , with automatic and human-based metrics , on two types of constraints and languages : review polarity and emotion control in French and English .

We show that discriminatorguided MCTS decoding achieves state-of-theart results without having to tune the language model , in both tasks and languages .

We also demonstrate that other proposed decoding methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged .

Generative language models exist for a long time , but with advent of the transformer architecture ( Vaswani et al . , 2017 ) and increasing computing capabilities , they are now able to generate well written and long texts .

In particular , large models , such as the well known GPT-2 ( Radford et al . , 2019 ) and GPT-3 ( Brown et al . , 2020 ) , have been used successfully for various applications : assisting writers , summarizing , augmentating data for subsequent NLP tasks , generating fake news ( Kumar et al . , 2020 ; Papanikolaou and Pierleoni , 2020 ; Zellers et al . , 2019 ) .

Yet , beside the prompt used to initiate the generation process , there are few options to have control on the generation process .

Being able to add some constraints on the generated texts is useful for various situations .

For example , it allows to create texts that follow a certain writing style , convey a certain emotion or polarity or to ensure that a generated summary contains correct information .

More critically , it can be used to prevent the inherent toxicity of language models trained on the internet , or to not reproduce gender or race stereotypes .

So far , most methods necessitate to fine-tune the LM , so that it specifically learns to model this constraint , i.e. the constraint is -hopefully-incorporated in the LM .

This finetuning approach has several drawbacks .

It implies to train multiple specific LMs ( one per constraint ) , which is costly , when even possible given the size of current state-of-the-art LM , and results in several models .

In this paper , we propose new approaches to add such additional constraints on the texts but at decoding time .

We exploit a discriminator that is trained to determine if a text follows a given constraint or not ; its output provides information to guide the generation toward texts that satisfy this expected constraint .

In order to make the most of the discriminator information , we propose an original method based on the Monte Carlo Tree Search ( MCTS ) algorithm ( Coulom , 2006 ) , namely Plug and Play Language -Monte

We also propose simpler methods based on re-ranking to fulfil this goal .

Both approaches do not require to fine-tune the LM ; adding a new constraint can thus simply be done by providing a discriminator verifying if a text complies with what is expected .

More precisely , our main contributions are the following ones :

we propose to use MCTS as a decoding strat-egy to implement constrained generation and we show , on 3 datasets and 2 languages , that it yields state-of-the-art results while offering more flexibility ; 2 . we also explore simpler generation methods based on re-ranking and show that this kind of approach , with low computational costs , can also be competitive if the diversity within propositions to re-rank is encouraged ;

3 . we provide a fully functional code implementing a batched textual MCTS 1 working with the popular HuggingFace 's Transformers library ( Wolf et al . , 2020 ) 2 Related work

The goal of constrained textual generation is to find the sequence of tokens x 1 : T which maximises p(x 1 : T | c ) , given a constraint c.

Few methods address the constrained textual generation .

Classconditional language models ( CC-LMs ) , as the Conditional Transformer Language ( CTRL ) model ( Keskar et al . , 2019 ) , train or fine-tune the weights θ of a single neural model directly for controllable generation , by appending a control code in the beginning of a training sequence .

The control code indicates the constraint to verify and is related to a class containing texts that satisfy the constraint .

For the sake of simplicity , we will denote without distinction the class , the constraint verified by its texts and the associated control code by c.

Trained with different control codes , the model learns p θ ( x 1 : T | c )

The constraint can then be applied during generation by appending the corresponding control code to the prompt .

While this method gives some kind of control over the generation , the control codes need to be defined upfront and the LM still needs to be trained specifically for each set of control codes .

This is an important limitation since the current trend in text generation is the use of large pre-trained models which can hardly be fine-tuned ( for instance , the last version of GPT , GPT-3 , can not be fine-tuned without access to very large hardware resources ) .

Discriminator-based methods The general idea of discriminator-guided generation is to combine 1 https://github.com / NohTow / PPL-MCTS a disciminator D with a generative LM .

The discriminator explicitly models the constraint by calculating the probability p D ( c | x 1 : T ) of the sequence x 1 : T to satisfy the constraint c.

This probability is directly related to p(x 1 : T | c ) through Bayes ' rule :

Discriminator-based methods alleviate the training cost problem , as discriminators are easier to train than a LM .

Moreover , any additional constraint can be defined a posteriori without tuning the LM , only by training another discriminator .

The discriminators have been used in different ways to explore the search space .

In the work of ( Holtzman et al . , 2018 ; Scialom et al . , 2020 ) , the space is first searched using beam search to generate a pool of proposals with a high likelihood p θ ( x 1 : T ) , and then the discriminator is used to re-rank them .

However , in addition that beam search can miss sequences with high likelihood , it is biased towards the likelihood , while the best sequence might only have an average likelihood , but satisfies the constraint perfectly .

Hence , it might be more suitable to take the discriminator probability into account during decoding rather than after generating a whole sequence .

In this case , the discriminator is used at each generation step t to get the probability p D ( c | x 1 : t ) for each token of the vocabulary V , and merge it to the likelihood p θ ( x 1 : t ) to choose which token to emit .

In order to reduce the cost of using a discriminator on every possible continuation , GeDi ( Krause et al . , 2020 ) proposes to use CC-LMs as generative discriminators .

The method relies on the fact that the CC-LM computes p θ ( x t | x 1 : t-1 , c ) for all tokens of the vocabulary which can be used to get p θ ( c | x 1 : t ) for all tokens using Bayes ' equation .

This approach is thus at the intersection of tuning the LM and using a discriminator : it tunes a small LM ( the CC-LM ) to guide a bigger one .

In Plug And Play Language Model ( PPLM ) ( Dathathri et al . , 2020 ) , the discriminator is used to shift the hidden states of the pre-trained transformer-based LM towards the desired class at every generation step .

PPLM can be used on any LM and with any discriminator .

However , PPLM needs to access the LM to modify its hidden states , while our approach only requires the output logits .

As some LM can only be used through access to logits ( e.g. GPT-3 API ) , this makes our approach more plug and play than PPLM .

A common drawback of all these approaches is their lack of a long-term vision of the generation .

Indeed , the discriminator probabilities become necessarily more meaningful as the sequence grows and might only be trustable to guide the search when the sequence is ( nearly ) finished .

When used in a myopic decoding strategy , classification errors will cause the generation process to deviate further and further .

Trying to optimize a score defined in the long horizon by making short term decisions is very similar to common game setups such as chess , where the Monte Carlo Tree Search ( MCTS ) has proven to be really effective ( Silver et al . , 2018 ) , which motivated our approach .

In this paper , we show that it is possible to control generation with the help of a discriminator that implements some expected constraints on the text during decoding .

This flexible approach is very useful when using very large language models , such as GPT-3 , whose fine-tuning computational costs are prohibitive .

In contrast , training a discriminator is easier and cheaper .

Our proposed methods , that mix the discriminator constraint and the generation , yield performance that is equivalent to the best approaches based on LM tuning at lower training cost .

On the other hand , such approaches have an additional cost during inference because of the cost of the discriminator being applied to candidate generations .

A study on this additional cost depending on the type of discriminator used can be found in ( Chaffin et al . , 2022 ) .

PPL-MCTS offers a solution for cases where training is too costly for the downstream application or the language model is not directly accessible .

Seeing text generation as a tree exploration process , an existing approach such as GeDi indeed lowers the cost of width exploration but the depth exploration is still an issue .

Using GeDi for constrained generation is thus very similar to a standard maximum likelihood search which still lacks of an optimal search method .

On the other hand , Monte Carlo Tree Search provides an efficient way to explore the tree by determining the best local choice in the long run , lowering the cost of depth exploration .

Thus , these two methods solve different facets of constrained generation , and the combination of the two is a promising perspective .

Moreover , MCTS allows to precisely define the best compromise between cost and quality through the number of iterations and the roll-out size , while ensuring the efficiency of the search theoretically .

For reproducibility purposes , our implementation is made available at https : //github.com / NohTow / PPL-MCTS .

Several research avenues are opened by this work .

For methods yielding high perplexity , it would be interesting to explore how to set the α parameter in order to reach the best compromise between accuracy and perplexity .

Similarly , the size ( number of tokens considered ) of the rollout in MCTS offers some ways to control the cost / performance compromise .

An adaptive rollout size , for example rolling-out until the score of the discriminator is above or below a threshold as in ( Cotarelo et al . , 2021 ) , would seem particularly suited for texts .

It should also be noted that finetuning a model and controlling the generation with a discriminator can be used jointly .

For instance , one can use PPL-MCTS on a tuned LM , which will most likely result in even better performances because sequences considered during the search will have an overall higher quality for the considered task .

Finally , not only can PPL-MCTS be applied to any property that a discriminator can identify , but it can also work using other scoring methods ( human evaluation , regular expressions , heuristic based evaluation , ... ) as long as the score reflects compliance with the expected property .

Yet , beside the prompt used to initiate the generation process , there are few options to have control on the generation process .

In this paper , we propose new approaches to add such additional constraints on the texts but at decoding time .

In contrast , training a discriminator is easier and cheaper .


Unsupervised style transfer in text has previously been explored through the sentiment transfer task .

The task entails inverting the overall sentiment polarity in a given input sentence , while preserving its content .

From the Aspect-Based Sentiment Analysis ( ABSA ) task , we know that multiple sentiment polarities can often be present together in a sentence with multiple aspects .

In this paper , the task of aspect-level sentiment controllable style transfer is introduced , where each of the aspect-level sentiments can individually be controlled at the output .

To achieve this goal , a BERT-based encoder-decoder architecture with saliency weighted polarity injection is proposed , with unsupervised training strategies , such as ABSA masked-languagemodelling .

Through both automatic and manual evaluation , we show that the system is successful in controlling aspect-level sentiments .

With a rapid increase in the quality of generated text , due to the rise of neural text generation models ( Kalchbrenner and Blunsom , 2013 ; Cho et al . , 2014 ; Sutskever et al . , 2014 ; Vaswani et al . , 2017 ) , controllable text generation is quickly becoming the next frontier in the field of text generation .

Controllable text generation is the task of generating realistic sentences whose attributes can be controlled .

The attributes to control can be : ( i ) .

Stylistic : Like politeness , sentiment , formality etc , ( ii ) .

Content : Like information , entities , keywords etc . or ( iii ) .

Ordering : Like ordering of information , events , plots etc .

Controlling sentence level polarity has been well explored as a style transfer task .

Zhang et al . ( 2018 ) used unsupervised machine translation techniques for polarity transfer in sentences .

Yang et al . ( 2018 ) § equal contribution

The service was speedy and the salads were great , but the chicken was bland and stale .

The service was slow , but the salads were great and the chicken was tasty and fresh .

used language models as discriminators to achieve style ( polarity ) transfer in sentences .

Li et al . ( 2018a ) proposed a simpler method where they deleted the attribute markers and devise a method to replace or generate the target attribute-key phrases in the sentence .

In this paper we explore a more fine-grained style transfer task , where each aspect 's polarities can be changed individually .

Recent interest in Aspect-Based Sentiment Analysis ( ABSA ) ( Pontiki et al . , 2014 ) has shown that sentiment information can vary within a sentence , with differing sentiments expressed towards different aspect terms of target entities ( e.g. ' food ' , ' service ' in a restaurant domain ) .

We introduce the task of aspect-level sentiment transfer -the task of rewriting sentences to transfer them from a given set of aspect-term polarities ( such as ' positive sentiment ' towards the service of a restaurant and a ' positive sentiment ' towards the taste of the food ) to a different set of aspect-term polarities ( such as ' negative sentiment ' towards the service of a restaurant and a ' positive ' sentiment towards the taste of the food ) .

This is a more challenging task than regular style transfer as the style attributes here are not the overall attributes for the whole sentence , but are localized to specific parts of the sentence , and multiple opposing at-tributes could be present within the same sentence .

The target of the transformation made needs to be localized and the other content expressed in the rest of the sentence need to be preserved at the output .

An example of the task is shown in Figure 1 .

For successful manipulation of the generated sentences , a few challenges need to be addressed : ( i ) .

The model should learn to associate the right polarities with the right aspects .

The model needs to be able to correctly process the aspectpolarity query and accordingly delete , replace and generate text sequence to satisfy the query .

The polarities of the aspects not in the query should not be affected .

The non-attribute content and fluency of the text should be preserved .

We explore this task in an unsupervised setting ( as is common with most style-transfer tasks due to the lack of an aligned parallel corpus ) using only monolingual unaligned corpora .

In this work , a novel encoder-decoder architecture is proposed to perform unsupervised aspect-level sentiment transfer .

A BERT ( Devlin et al . , 2019 ) based encoder is used that is trained to understand aspect-specific polarity information .

We also propose using a ' polarity injection ' method , where saliency-weighted aspect-specific polarity information is added to the hidden representations from the encoder to complete the query for the decoder .

In this paper , the task of aspect-level sentiment style transfer has been introduced , where stylistic attributes can be localized to different parts of a sentence .

We have proposed a BERT-based encoder-decoder architecture with saliency-based polarity injection and show that it can be successful at the task when trained in an unsupervised setting .

The experiments have been conducted on an aspect level polarity tagged benchmark dataset related to the restaurant domain .

This work is hopefully an important initial step in developing a fine-grained controllable style transfer system .

In the future , we would like to explore the ability to transfer such systems to data-sparse domains , and explore injecting attributes such as emotions to targets attributes in larger pieces of text .


We consider the problem of predicting measurable responses to scientific articles based primarily on their text content .

Specifically , we consider papers in two fields ( economics and computational linguistics ) and make predictions about downloads and within-community citations .

Our approach is based on generalized linear models , allowing interpretability ; a novel extension that captures first-order temporal effects is also presented .

We demonstrate that text features significantly improve accuracy of predictions over metadata features like authors , topical categories , and publication venues .

Written communication is an essential component of the complex social phenomenon of science .

As such , natural language processing is well-positioned to provide tools for understanding the scientific process , by analyzing the textual artifacts ( papers , proceedings , etc . ) that it produces .

This paper is about modeling collections of scientific documents to understand how their textual content relates to how a scientific community responds to them .

While past work has often focused on citation structure ( Borner et al . , 2003 ; Qazvinian and Radev , 2008 ) , our emphasis is on the text content , following Ramage et al . ( 2010 ) and Gerrish and Blei ( 2010 ) .

Instead of task-independent exploratory data analysis ( e.g. , topic modeling ) or multi-document sum-marization , we consider supervised models of the collective response of a scientific community to a published article .

There are many measures of impact of a scientific paper ; ours come from direct measurements of the number of downloads ( from an established website where prominent economists post papers before formal publication ) and citations ( within a fixed scientific community ) .

We adopt a discriminative approach based on generalized linear models that can make use of any text or metadata features , and show that simple lexical features offer substantial power in modeling out-ofsample response and in forecasting response for future articles .

Realistic forecasting evaluations require methodological care beyond the usual best practices of train / test separation , and we elucidate these issues .

In addition , we introduce a new regularization technique that leverages the intuition that the relationship between observable features and response should evolve smoothly over time .

This regularizer allows the learner to rely more strongly on more recent evidence , while taking into account a long history of training data .

Our time series-inspired regularizer is computationally efficient in learning and is a significant advance over earlier text-driven forecasting models that ignore the time variable altogether ( Kogan et al . , 2009 ; Joshi et al . , 2010 ) .

We evaluate our approaches in two novel experimental settings : predicting downloads of economics articles and predicting citation of papers at ACL conferences .

Our approaches substantially outper-594 form text-ignorant baselines on ground-truth predictions .

Our time series models permit flexibility in features and offer a novel and perhaps more interpretable view of the data than summary statistics .

We presented a statistical approach to predicting a scientific community 's response to an article , based on its textual content .

To improve the interpretability of the linear model , we developed a novel time series regularizer that encourages gradual changes across time steps .

Our experiments showed that text features significantly improve accuracy of predictions over baseline models , and we found that the feature weights learned with the time series regularizer reflect important trends in the literature .


We present an algorithm for generating referring expressions in open domains .

Existing algorithms work at the semantic level and assume the availability of a classification for attributes , which is only feasible for restricted domains .

Our alternative works at the realisation level , relies on Word-Net synonym and antonym sets , and gives equivalent results on the examples cited in the literature and improved results for examples that prior approaches can not handle .

We believe that ours is also the first algorithm that allows for the incremental incorporation of relations .

We present a novel corpus-evaluation using referring expressions from the Penn Wall Street Journal Treebank .

Referring expression generation has historically been treated as a part of the wider issue of generating text from an underlying semantic representation .

The task has therefore traditionally been approached at the semantic level .

Entities in the real world are logically represented ; for example ( ignoring quantifiers ) , a big brown dog might be represented as big1(x ) ∧ brown1(x ) ∧ dog1(x ) , where the predicates big1 , brown1 and dog1 represent different attributes of the variable ( entity ) x.

The task of referring expression generation has traditionally been framed as the identification of the shortest logical description for the referent entity that differentiates it from all other entities in the discourse domain .

For example , if there were a small brown dog ( small1(x ) ∧ brown1(x ) ∧ dog1(x ) ) in context , the minimal description for the big brown dog would be big1(x ) ∧ dog1(x)1 .

This semantic framework makes it difficult to apply existing referring expression generation algorithms to the many regeneration tasks that are important today ; for example , summarisation , openended question answering and text simplification .

Unlike in traditional generation , the starting point in these tasks is unrestricted text , rather than a semantic representation of a small domain .

It is difficult to extract the required semantics from unrestricted text ( this task would require sense disambiguation , among other issues ) and even harder to construct a classification for the extracted predicates in the manner that existing approaches require ( cf . , § 2 ) .

In this paper , we present an algorithm for generating referring expressions in open domains .

We discuss the literature and detail the problems in applying existing approaches to reference generation to open domains in § 2 .

We then present our approach in § 3 , contrasting it with existing approaches .

We extend our approach to handle relations in § 3.3 and present a novel corpus-based evaluation on the Penn WSJ Treebank in § 4 .

We have described an algorithm for generating referring expressions that can be used in any domain .

Our algorithm selects attributes and relations that are distinctive in context .

It does not rely on the availability of an adjective classification scheme and uses WordNet antonym and synonym lists instead .

It is also , as far as we know , the first algorithm that allows for the incremental incorporation of relations and the first that handles nominals .

In a novel evaluation , our algorithm successfully generates identical referring expressions to those in the Penn WSJ Treebank in over 80 % of cases .

In future work , we plan to use this algorithm as part of a system for generation from a database of user opinions on products which has been automatically extracted from newsgroups and similar text .

This is midway between regeneration and the classical task of generating from a knowledge base because , while the database itself provides structure , many of the field values are strings corresponding to phrases used in the original text .

Thus , our lexicalised approach is directly applicable to this task .


This work investigates design choices in modeling a discourse scheme for improving opinion polarity classification .

For this , two diverse global inference paradigms are used : a supervised collective classification framework and an unsupervised optimization framework .

Both approaches perform substantially better than baseline approaches , establishing the efficacy of the methods and the underlying discourse scheme .

We also present quantitative and qualitative analyses showing how the improvements are achieved .

The importance of discourse in opinion analysis is being increasingly recognized ( Polanyi and Zaenen , 2006 ) .

Motivated by the need to enable discourse-based opinion analysis , previous research ( Asher et al . , 2008 ; Somasundaran et al . , 2008 ) developed discourse schemes and created manually annotated corpora .

However , it was not known whether and how well these linguistic ideas and schemes can be translated into effective computational implementations .

In this paper , we first investigate ways in which an opinion discourse scheme can be computationally modeled , and then how it can be utilized to improve polarity classification .

Specifically , the discourse scheme we use is from Somasundaran et al . ( 2008 ) , which was developed to support a global , interdependent polarity interpretation .

To achieve discourse-based global inference , we explore two different frameworks .

The first is a supervised framework that learns interdependent opinion interpretations from training data .

The second is an unsupervised optimization framework which uses constraints to express the ideas of coherent opinion interpretation embodied in the scheme .

For the supervised framework , we use Iterative Collective Classification ( ICA ) , which facilitates machine learning using relational information .

The unsupervised optimization is implemented as an Integer Linear Programming ( ILP ) problem .

Via our implementations , we aim to empirically test if discourse-based approaches to opinion analysis are useful .

Our results show that both of our implementations achieve significantly better accuracies in polarity classification than classifiers using local information alone .

This confirms the hypothesis that the discourse-based scheme is useful , and also shows that both of our design choices are effective .

We also find that there is a difference in the way ICA and ILP achieve improvements , and a simple hybrid approach , which incorporates the strengths of both , is able to achieve significant overall improvements over both .

Our analyses show that even when our discourse-based methods bootstrap from noisy classifications , they can achieve good improvements .

The rest of this paper is organized as follows : we discuss related work in Section 2 and the discourse scheme in Section 3 .

We present our discourse-based implementations in Section 4 , experiments in Section 5 , discussions in Section 6 and conclusions in Section 7 .

This work focuses on the first step to ascertain whether discourse relations are useful for improving opinion polarity classification , whether they can be modeled and what modeling choices can be used .

To this end , we explored two distinct paradigms : the supervised ICA and the unsupervised ILP .

We showed that both of our approaches are effective in exploiting discourse relations to significantly improve polarity classification .

We found that there is a difference in how ICA and ILP achieve improvements , and that combining the two in a hybrid approach can lead to further overall improvement .

Quantitatively , we showed that our approach is able to achieve a large increase in recall of the polar categories without harming the precision , which results in the performance improvements .

Qualitatively , we illustrated how , even if the bootstrapping process is noisy , the optimization and discourse constraints effectively rectify the misclassifications .

The improvements of our diverse global inference approaches indicate that discourse information can be adapted in different ways to augment and improve existing opinion analysis techniques .

The automation of the discourse-relation recognition is the next step in this research .

The behavior of ICA and ILP can change , depending on the automation of discourse level recognition .

The implementation and comparison of the two methods under full automation is the focus of our future work .