Thailand

In this report, we describe our Transformers for text classification baseline (TTCB) submissions to a shared task on implicit and underspecified language 2021. We cast the task of predicting revision requirements in collaboratively edited instructions as text classification. We considered Transformer-based models which are the current state-of-the-art methods for text classification. We explored different training schemes, loss functions, and data augmentations. Our best result of 68.45% test
accuracy (68.84% validation accuracy), however, consists of an XLNet model with a linear annealing scheduler and a cross-entropy loss. We do not observe any significant gain on any validation metric based on our various design choices except the MiniLM which has a higher validation F1 score and is faster to train by a half but also a lower validation accuracy score.

IJCNLP-AACL 2021

TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021

revision prediction

text classification

transformers

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

workshop paper

Current dialogue summarization systems usually encode the text with a number of general semantic features (e.g., keywords and topics) to gain more powerful dialogue modeling capabilities. However, these features are obtained via open-domain toolkits that are dialog-agnostic or heavily relied on human annotations. In this paper, we show how DialoGPT, a pre-trained model for conversational response generation, can be developed as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAMSum and AMI, and employ pre-trained and non pre-trained models as our summarizers. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the-art performance on the SAMSum dataset.

Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

The growth of online consumer health questions has led to the necessity for reliable and
accurate question answering systems. A recent study showed that manual summarization of consumer health questions brings significant improvement in retrieving relevant answers. However, the automatic summarization
of long questions is a challenging task due to
the lack of training data and the complexity of
the related subtasks, such as the question focus and type recognition. In this paper, we introduce a reinforcement learning-based framework for abstractive question summarization.
We propose two novel rewards obtained from
the downstream tasks of (i) question-type identification and (ii) question-focus recognition
to regularize the question generation model.
These rewards ensure the generation of semantically valid questions and encourage the
inclusion of key medical entities/foci in the
question summary. We evaluated our proposed method on two benchmark datasets and
achieved higher performance over state-of-the-art models. The manual evaluation of the
summaries reveals that the generated questions are more diverse and have fewer factual
inconsistencies than the baseline summaries.
The source code is available here: https://github.com/shwetanlp/CHQ-Summ.

Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards

Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. 
We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. 
Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. 
However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation.
Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development. Our code and annotated data is publicly available at https://github.com/AkariAsai/unanswerable_qa.

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Relation linking is a crucial component of Knowledge Base Question Answering systems. Existing systems use a wide variety of heuristics, or ensembles of multiple systems, heavily relying on the surface question text. However, the explicit semantic parse of the question is a rich source of relation information that is not taken advantage of. We propose a simple transformer-based neural model for relation linking that leverages the AMR semantic parse of a sentence. Our system significantly outperforms the state-of-the-art on 4 popular benchmark datasets. These are based on either DBpedia or Wikidata, demonstrating that our approach is effective across KGs.

A Semantics-aware Transformer Model of Relation Linking for Knowledge Base Question Answering

Users of medical question answering systems often submit long and detailed questions, making it hard to achieve high recall in answer retrieval. To alleviate this problem, we propose a novel Multi-Task Learning (MTL) method with data augmentation for medical question understanding. We first establish an equivalence between the tasks of question summarization and Recognizing Question Entailment (RQE) using their definitions in the medical domain. Based on this equivalence, we propose a data augmentation algorithm to use just one dataset to optimize for both tasks, with a weighted MTL loss. We introduce gradually soft parameter-sharing: a constraint for decoder parameters to be close, that is gradually loosened as we move to the highest layer. We show through ablation studies that our proposed novelties improve performance. Our method outperforms existing MTL methods across 4 datasets of medical question pairs, in ROUGE scores, RQE accuracy and human evaluation. Finally, we show that our method fares better than single-task learning under 4 low-resource settings.

A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding

Early fusion models with cross-attention have shown better-than-human performance on some question answer benchmarks, while it is a poor fit for retrieval since it prevents pre-computation of the answer representations. We present a supervised data mining method using an accurate early fusion model to improve the training of an efficient late fusion retrieval model. We first train an accurate classification model with cross-attention between questions and answers. The cross-attention model is then used to annotate additional passages in order to generate weighted training examples for a neural retrieval model. The resulting retrieval model with additional data significantly outperforms retrieval models directly trained with gold annotations on Precision at N~(P@N) and Mean Reciprocal Rank~(MRR).

Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation

We conclude that the implicit UCCA parser does not address NFHs consistently, which could result either from inconsistent annotation, insufﬁcient training data or a modelling limitation. We consider this phenomenon important, as it is pervasive in text and critical for correct inference. Careful design and ﬁne-grained annotation of NFHs in Meaning Representations would beneﬁt downstream tasks such as machine translation, natural language inference and question answering, particularly when they require numeric reasoning, as recovering and categorizing them. We are investigating the treatment of this phenomenon by other meaning representations, such as AMR. We encourage researchers in meaning representations, and computational linguistics in general, to address this phenomenon in future research.

Meaning Representation of Numeric Fused-Heads in UCCA

Transformer-based language models (LMs) have shown strong performance on an array of natural language understanding tasks. However, the question of how these models react to implicit meaning has been largely unexplored. We investigate this using the complement coercion phenomenon, which involves sentences like "The student finished the book about sailing" where the action "read" is implicit. We compare LMs' surprisal estimates at various critical sentence regions in sentences with and without implicit meaning. Effects associated with recovering implicit meaning were found at a critical region other than where sentences minimally differ. We then use follow-up experiments to factor out potential confounds, revealing different perspectives that offer a richer and more accurate picture.

Transformer-based language models and complement coercion: Experimental studies

Noun phrases (NPs) in text are linked via discourse relations. One such well-known relation is bridging , in which an NP indirectly refers to an entity mentioned elsewhere in the text. 
Our aim is to construct a high-quality large-scale manually annotated corpus, that covers bridging relations. To this end, we generalize and simplify the definition of bridging relations, to make them amenable to crowdsourcing. In our updated definition NPs are connected to co-reference clusters via natural-language prepositions.
The result is an annotation scheme in which each NP is potentially linked to other NPs via coreference links and/or via a preposition. While the collection process is ongoing, we already have 2790 documents annotated (each including 3 paragraphs), covering 504,250 bridging links and 14,958 coreference clusters (not including singletons).



Large Scale Crowdsourcing of Noun-Phrase Links

Data-to-text generation systems are trained on large datasets, such as WebNLG, RotoWire, E2E or DART. Beyond traditional token-overlap evaluation metrics (BLEU or METEOR), a key concern faced by recent generators is to control 
the factuality of the generated text with respect to the input data specification. We report on our experience when developing an automatic factuality evaluation system for data-to-text generation that we are testing on WebNLG and E2E data. We aim to prepare gold data annotated manually to identify cases where the text communicates more information than is warranted based on the input data (extra) or fails to communicate data that is part of the input (missing). While analyzing reference (data, text) samples, we encountered a range of systematic uncertainties that are related to cases on implicit phenomena in text, and the nature of non-linguistic knowledge we expect to be involved when assessing factuality. 
We derive from our experience a set of evaluation guidelines to reach high inter-annotator agreement on such cases

TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021

Downloads

Next from IJCNLP-AACL 2021

Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

Similar lecture

The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES