Thailand

Human-like syntactic generalization abilities of neural language models (LMs) have received wide attention, but discourse-level knowledge in these models has been less explored. In this study, we analyze the LMs with respect to a particular aspect of discourse, topic-comment structure, a crucial part of text production. To separately analyze the knowledge of LMs on topic-comment structure, we chose the Japanese language, a topic-prominent language, for designing probing tasks. Experimental results suggest that LMs have different generalizations from humans in terms of the topic-comment structure. These results imply that LMs do not perform human-like discourse processing in text generation.

IJCNLP-AACL 2021

Topicalization in Language Models: A Case Study on Japanese

topicalization

probing

language model

discourse

[![](https://assets.underline.io/uploads/markdown_image/1/image/81a3c0317d24f663b49b996875024d45.png)](https://aclanthology.org/volumes/2022.coling-1/)



### THE CONFERENCE WE KNOW AND WE WANT.
**COLING**, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes but also in industrial research departments including tech-startups. COLING provides opportunities for all these communities to showcase their exciting discovery.

In fall of 2022, COLING will be held in Gyeongju in a hybrid format. All participants can either present at the venue site or join virtually. As more people get vaccinated, we are happy to provide safer environments for our colleagues. We believe that COLING 2022 will be one of the conferences, free from the pandemic. The hybrid format gives presenters and sponsors a valuable opportunity to promote their companies in both an online and in-person venue. For the first time in a long time, customers can interact with their sponsor's products first-hand. The online venue, too, gives sponsors the chance to network with those unable to attend the in-person session.

The hybrid format ultimately lends itself to greater exposure for our sponsors: COLING2022 will let you reach more potential partners and customers than ever before!<br><br><br>

To gain access to this event page you are required to register.<br> Find more information and pay the registration fee on event the organizer’s website **[https://coling2022.org/reg](https://coling2022.org/reg)**

COLING 2022

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

workshop paper

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefix of the target sequence. While many attempts to understand the internal workings of NMT models have been made, none of them explicitly evaluates relative source and target contributions to a generation decision. We argue that this relative contribution can be evaluated by adopting a variant of Layerwise Relevance Propagation (LRP). Its underlying 'conservation principle' makes relevance propagation unique: differently from other methods, it evaluates not an abstract quantity reflecting token importance, but the proportion of each token's influence. We extend LRP to the Transformer and conduct an analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes, when varying the training objective or the amount of training data, and during the training process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.

Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation

Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.

Comparing Test Sets with Item Response Theory

A growing body of literature has focused on detailing the linguistic knowledge embedded in large, pretrained language models. Existing work has shown that non-linguistic biases in models can drive model behavior away from linguistic generalizations. We hypothesized that competing linguistic processes within a language, rather than just non-linguistic model biases, could obscure underlying linguistic knowledge. We tested this claim by exploring a single phenomenon in four languages: English, Chinese, Spanish, and Italian. While human behavior has been found to be similar across languages, we find cross-linguistic variation in model behavior. We show that competing processes in a language act as constraints on model behavior and demonstrate that targeted fine-tuning can re-weight the learned constraints, uncovering otherwise dormant linguistic knowledge in models. Our results suggest that models need to learn both the linguistic constraints in a language and their relative ranking, with mismatches in either producing non-human-like behavior.

Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning

Interpretability is an important aspect of the trustworthiness of a model's predictions. Transformer's predictions are widely explained by the attention weights, i.e., a probability distribution generated at its self-attention unit (head). Current empirical studies provide shreds of evidence that attention weights are not explanations by proving that they are not unique. A recent study showed theoretical justifications to this observation by proving the non-identifiability of attention weights. For a given input to a head and its output, if the attention weights generated in it are unique, we call the weights identifiable. In this work, we provide deeper theoretical analysis and empirical observations on the identifiability of attention weights. Ignored in the previous works, we find the attention weights are more identifiable than we currently perceive by uncovering the hidden role of the key vector. However, the weights are still prone to be non-unique attentions that make them unfit for interpretation. To tackle this issue, we provide a variant of the encoder layer that decouples the relationship between key and value vector and provides identifiable weights up to the desired length of the input. We prove the applicability of such variations by providing empirical justifications on varied text classification tasks. The implementations are available at https://github.com/declare-lab/identifiable-transformers.

More Identifiable yet Equally Performant Transformers for Text Classification

Natural Language Generation (NLG) is a key component in a task-oriented dialogue system, which converts the structured meaning representation (MR) to the natural language.
For large-scale conversational systems, where it is common to have over hundreds of intents and thousands of slots, neither template-based approaches nor model-based approaches are scalable.
Recently, neural NLGs started leveraging transfer learning and showed promising results in few-shot settings.
This paper proposes AugNLG, a novel data augmentation approach that combines a self-trained neural retrieval model with a few-shot learned NLU model, to automatically create MR-to-Text data from open-domain texts.
The proposed system mostly outperforms the state-of-the-art methods on the FewshotWOZ data in both BLEU and Slot Error Rate. 
We further confirm improved results on the FewshotSGD data and provide comprehensive analysis results on key components of our system. Our code and data are available at https://github.com/XinnuoXu/AugNLG.

AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation

Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT~\citep{Devlin2019BERTPO}, which is based on Chinese characters (chars). Second, the masked language model pre-training is based on a single vocab, limiting its downstream task performances. In this work, we first experimentally demonstrate that building a vocab via Chinese word segmentation (CWS) guided sub-word tokenization (SGT) can improve the performances of Chinese PLMs. Then we propose two versions of multi-vocab pre-training (MVP), Hi-MVP and AL-MVP, to improve the models' expressiveness. Experiments show that: (a) MVP training strategies improve PLMs' downstream performances, especially it can improve the PLM's performances on span-level tasks; (b) our AL-MVP outperforms the recent AMBERT \citep{Zhang2020AMBERTAP} after large-scale pre-training, and it is more robust against adversarial attacks.

MVP-BERT: Multi-Vocab Pre-training for Chinese BERT

The internet has actually come to be an essential resource of health knowledge for individuals around the world in the present situation of the coronavirus condition pandemic(COVID- 19). During pandemic situations, myths, sensationalism, rumours and misinformation, generated intentionally or unintentionally, spread rapidly through social networks. Twitter is one of these popular social networks people use to share COVID-19 related news, information, and thoughts that reflect their perception and opinion about the pandemic. Evaluation of tweets for recognizing misinformation can create beneficial understanding to review the top quality and also the readability of online information concerning the COVID-19. This paper presents a multilingual COVID-19 related tweet analysis method, CMTA, that uses BERT, a deep learning model for multilingual tweet misinformation detection and classification. CMTA extracts features from multilingual textual data, which is then categorized into specific information classes. Classification is done by a Dense-CNN model trained on tweets manually annotated into information classes (i.e., ’false’, ’partly false’, ’misleading’). The paper presents an analysis of multilingual tweets from February to June, showing the distribution type of information spread across different languages. To access the performance of the CMTA multilingual model, we performed a comparative analysis of 8 monolingual model and CMTA for the misinformation detection task. The results show that our proposed CMTA model has surpassed various monolingual models which consolidated the fact that through transfer learning a multilingual framework could be developed

CMTA: COVID-19 Misinformation Multilingual Analysis on Twitter

Individuals with autism spectrum disorder (ASD) experience difficulties in social aspects of communication, but the linguistic characteristics associated with deficits in discourse and pragmatic expression are often difficult to precisely identify and quantify. We are currently collecting a corpus of transcribed natural conversations produced in an experimental setting in which participants with and without ASD complete a number of collaborative tasks with their neurotypical peers. Using this dyadic conversational data, we investigate three pragmatic features – politeness, uncertainty, and informativeness – and present a dataset of utterances annotated for each of these features on a three-point scale. We then introduce ongoing work in developing and training neural models to automatically predict these features, with the goal of identifying the same between-groups differences that are observed using manual annotations. We find the best performing model for all three features is a feedforward neural network trained with BERT embeddings. Our models yield higher acc racy than ones used in previous approaches for deriving these features, with F1 exceeding 0.82 for all three pragmatic features.

Predicting pragmatic discourse features in the language of adults with autism spectrum disorder

We present the first Chinese adversarial natural language inference (NLI) evaluation set (CANLI) to probe the limits of pretrained language models (PLMs). CANLI is based on the linguistic phenomenon of causative- passive homonymy (CPH). The disambiguation of CPH is easy for humans, but difficult for machines, as the causative and the passive are not distinguished by the sentences’ syntactic structure. We created the CANLI in two related settings: (i) CA: a syntactic heuristic setting and (ii) CB: a semantic role labeling setting. We find that state-of-the-art PLMs that are fine-tuned on existing large-scale Chinese NLI benchmark datasets perform poorly on CANLI. Further, we notice that the PLM’s performance on CA and CB is independent of each other, suggesting that the PLM has not learned the shared underlying linguistic knowledge CPH, but instead the respective spurious patterns. We conclude that there is considerable room for improvement in the NLI system.

Adversarial Datasets for NLI Tasks: the Case of the Chinese Causative-Passive Homonymy

**Read paper:**
https://aclanthology.org/2021.acl-srw.30/

Most earlier work on text summarization is carried out on news article datasets. The summary in these datasets is naturally located at the beginning of the text. Hence, a model can spuriously utilize this correlation for summary gen- eration instead of truly learning to summarize. To address this issue, we constructed a new dataset, SUMPUBMED, using scientific articles from the PubMed archive. We conducted a human analysis of summary coverage, redundancy, readability, coherence, and informativeness on SUMPUBMED. SUMPUBMED is challenging because (a) the summary is distributed throughout the text (not-localized on top), and (b) it contains rare domain-specific scientific terms. We observe that seq2seq models that adequately summarize news articles struggle to summarize SUMPUBMED. Thus, SUMPUBMED opens new avenues for the future improvement of models as well as the development of new evaluation metrics.

Topicalization in Language Models: A Case Study on Japanese

Downloads

Next from IJCNLP-AACL 2021

Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation

Similar lecture

Topicalization in Language Models: A Case Study on Japanese

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Topicalization in Language Models: A Case Study on Japanese

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2021

Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation

Similar lecture

Topicalization in Language Models: A Case Study on Japanese

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads