Spain

A multi-hop question answering (QA) dataset aims to test reasoning and inference skills by requiring a model to read multiple paragraphs to answer a given question. However, current datasets do not provide a complete explanation for the reasoning process from the question to the answer. Further, previous studies revealed that many examples in existing multi-hop datasets do not require multi-hop reasoning to answer a question. In this study, we present a new multi-hop QA dataset, called 2WikiMultiHopQA, which uses structured and unstructured data. In our dataset, we introduce the evidence information containing a reasoning path for multi-hop questions. The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model. We carefully design a pipeline and a set of templates when generating a question-answer pair that guarantees the multi-hop steps and the quality of the questions. We also exploit the structured format in Wikidata and use logical rules to create questions that are natural but still require multi-hop reasoning. Through experiments, we demonstrate that our dataset is challenging for multi-hop models and it ensures that multi-hop reasoning is required.

COLING 2020

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

comprehensive evaluation

multi-hop dataset

explanation dataset

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

This Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) was to be held in Bangkok, Thailand, during August 1-6, 2021 and is now virtual on the Underline platform.

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

ACL 2021

poster

Neural representation learning can be extended to meta-textual information that is readily available for many information retrieval tasks, for example, patent classes in prior-art retrieval, topical information in Wikipedia articles, or product categories in e-commerce data. We present a framework that learns embeddings for meta-textual categories, and optimizes a pairwise ranking objective for improved matching based on combined embeddings of textual and meta-textual information.

Embedding Meta-Textual Information for Improved Learning to Rank

We propose a graph-based model with Dual-tier Heterogeneous Graph (DHG) for document-level relation extraction, It is composed of a structure modeling layer followed by a relation reasoning layer, and is capable of not only capturing both the sequential and structural information of documents but also mixing them together to benefit for multi-hop reasoning and final decision-making.

Document-level Relation Extraction with Dual-tier Heterogeneous Graph

Many automatic evaluation metrics have been proposed to score the overall quality of a response in open-domain dialogue. Generally, the overall quality is comprised of various aspects, such as relevancy, specificity, and empathy, and the importance of each aspect differs according to the task. For instance, specificity is mandatory in a food-ordering dialogue task, whereas fluency is preferred in a language-teaching dialogue system. However, existing metrics are not designed to cope with such flexibility. For example, BLEU score fundamentally relies only on word overlapping, whereas BERTScore relies on semantic similarity between reference and candidate response. Thus, they are not guaranteed to capture the required aspects, i.e., specificity. To design a metric that is flexible to a task, we first propose making these qualities manageable by grouping them into three groups: understandability, sensibleness, and likability, where likability is a combination of qualities that are essential for a task. We also propose a simple method to composite metrics of each aspect to obtain a single metric called USL-H, which stands for Understandability, Sensibleness, and Likability in Hierarchy. We demonstrated that USL-H score achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics.

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems

With growing applications of Machine Learning in daily lives, Natural Language Processing (NLP) has emerged as a heavily researched area. Finding its applications in tasks ranging from simple Q/A chatbots to fully fledged conversational AI, NLP models are vital. Word and Sen- tence embeddings are one of the most common starting points of any NLP task. A word em- bedding represents a given word in a predefined vector-space while maintaining vector relations with similar or dis-similar entities. As such, different pretrained embedding such as Word2Vec, GloVe, FastText have been developed. These embeddings generated on millions of words are however very large in terms of size. Having embeddings with floating point precision also makes the downstream evaluation slow. In this paper we present a novel method to convert continuous embedding to its binary representation, thus reducing the overall size of the embedding while keeping the semantic and relational knowledge intact. This will facilitate an option of porting such big embedding onto devices where space is limited. We also present different approaches suitable for different downstream tasks based on the requirement of contextual and semantic in- formation. Experiments have shown comparable result in downstream tasks with 7 to 15 times reduction in file size and about 5% change in evaluation parameters. 


Word Embedding Binarization with Semantic Information Preservation

Conversational recommender systems focus on the task of suggesting products to users based on the conversation flow. Recently, the use of external knowledge in the form of knowledge graphs has shown to improve the performance in recommendation and dialogue systems. Information from knowledge graphs aids in enriching those systems by providing additional information such as closely related products and textual descriptions of the items. However, knowledge graphs are incomplete since they do not contain all factual information present on the web. Furthermore, when working on a specific domain, knowledge graphs in its entirety contribute towards extraneous information and noise. In this work, we study several subgraph construction methods and compare their performance across the recommendation task. We incorporate pre-trained embeddings from the subgraphs along with positional embeddings in our models. Extensive experiments show that our method has a relative improvement of at least 5.62\% compared to the state-of-the-art on multiple metrics on the recommendation task.

Suggest me a movie for tonight: Leveraging Knowledge Graphs for Conversational Recommendation

This poster presents the data and evaluation of a writing assistance system that helps students improve their academic writing by incorporating vocabulary that is more typical in the academic setting. Given an input text, the system suggests word substitutions according to an academic word list, and ranks them with a masked language model. As there is no existing dataset for academic lexical substitution (LS), we constructed our evaluation data by compiling an academic word list and an informal word list, and transforming an LS dataset with these lists. Our approach makes use of BERT (Devlin et al., 2019), a masked language model that has been shown to attain good performance in lexical substitution (Zhou et al., 2019).

Automatic Assistance for Academic Word Usage

The dominance of machine learning-based NLP has led to overemphasizing model accuracies rather than studying the reasons behind their errors. In this paper, we explore and compare the error patterns of some common sentiment analysis models in the finance domain. We discover six linguistic patterns that are pervasive among the errors.

Financial Sentiment Analysis: An Investigation into Common Mistakes and Silver Bullets

Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data which may have a different distribution, e.g. a different domain. Although many methods have been proposed to solve this problem, we cannot get to know what causes this phenomenon yet. Under the background of domain adaptation, we investigate the cause of catastrophic forgetting from the perspectives of modules and parameters (neurons). The investigation on the modules of the NMT model shows that some modules have tight relation with the general-domain knowledge while some other modules are more essential in the domain adaptation. And the investigation on the parameters shows that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general-domain. We conduct experiments across different language pairs and domains to ensure the validity and reliability of our findings.

Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

Informational bias is bias conveyed through sentences or
clauses that provide tangential, speculative or background information
that can sway readers' opinions towards entities. By nature,
informational bias is context-dependent, but previous work on
informational bias detection has not explored the role of context beyond
the sentence. We present experiments with four kinds of context for
informational bias in English news articles: neighboring sentences, the
full article, articles on the same event from other news publishers, and
articles from the same domain (but potentially different events). We
find that integrating event context improves classification performance
over a very strong baseline. In addition, we perform the first error
analysis of models on this task. We find that the best-performing
context-inclusive model outperforms the baseline on
longer sentences, and sentences from politically centrist articles.

Context in Informational Bias Detection

Motion recognition is one of the basic cognitive capabilities of many life forms, however, detecting and understanding motion in text is not a trivial task. In addition, identifying motion entities in natural language is not only challenging but also beneficial for a better natural language understanding. In this paper, we present a Motion Entity Tagging (MET) model to identify entities in motion in a text using the Literal-Motion-in-Text (LiMiT) dataset for training and evaluating the model. Then we propose a new method to split clauses and phrases from complex and long motion sentences to improve the performance of our MET model. We also present results showing that motion features, in particular, entity in motion benefits the Named-Entity Recognition (NER) task. Finally, we present an analysis for the special co-occurrence relation between the person category in NER and animate entities in motion, which significantly improves the classification performance for the person category in NER.

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

Downloads

Next from COLING 2020

Embedding Meta-Textual Information for Improved Learning to Rank

Similar lecture

Explanations for CommonsenseQA: New Dataset and Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Embedding Meta-Textual Information for Improved Learning to Rank

Similar lecture

Explanations for CommonsenseQA: New Dataset and Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads