Taiwan

Transformer-based language models are often trained on structured text where non-lexical markers of sentence and discourse structure (e.g., punctuation and casing) are present and used consistently. Transformers encode these markers and arguably benefit from the information they convey. Yet, a systematic evaluations of the contribution of non-lexical markers to model performance, and of whether models&#39; behavior changes significantly in their absence, is currently lacking. This knowledge is both relevant from a theoretical standpoint, but also important to understand how well pretrained models may perform in common application scenarios where casing and punctuation are absent or inconsistent. 
Here, we analyze GPT-2&#39;s language modeling behavior in parallel corpora that differ in the presence vs. absence of consistent punctuation and casing. We compute GPT-2&#39;s precision and uncertainty in next-token prediction for multiple context sizes, and compare the resulting performance distributions across corpora. We find that absence of non-lexical markers, especially punctuation, increases model uncertainty, and it affects (but does not catastrophically disrupt) GPT-2&#39;s precision in next-token prediction. Interestingly, the absence of non-lexical markers prevents the model from benefiting from larger contexts in order to reduce the uncertainty of its predictions. Future work will extend this paradigm to a wider range of models and systematically investigate how features of training text affect both language modeling and downstream predictive performance.

IJCNLP-AACL 2022

Evaluating the role of non-lexical markers in GPT-2’s language modeling behavior

non-lexical markers

uncertainty

language modeling

generalization

evaluation

nlp applications

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

ACL 2022

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

Transformer-based language models are often trained on structured text where non-lexical markers of sentence and discourse structure (e.g., punctuation and casing) are present and used consistently. Transformers encode these markers and arguably benefit from the information they convey. Yet, a systematic evaluations of the contribution of non-lexical markers to model performance, and of whether models' behavior changes significantly in their absence, is currently lacking. This knowledge is both relevant from a theoretical standpoint, but also important to understand how well pretrained models may perform in common application scenarios where casing and punctuation are absent or inconsistent. 
Here, we analyze GPT-2's language modeling behavior in parallel corpora that differ in the presence vs. absence of consistent punctuation and casing. We compute GPT-2's precision and uncertainty in next-token prediction for multiple context sizes, and compare the resulting performance distributions across corpora. We find that absence of non-lexical markers, especially punctuation, increases model uncertainty, and it affects (but does not catastrophically disrupt) GPT-2's precision in next-token prediction. Interestingly, the absence of non-lexical markers prevents the model from benefiting from larger contexts in order to reduce the uncertainty of its predictions. Future work will extend this paradigm to a wider range of models and systematically investigate how features of training text affect both language modeling and downstream predictive performance.

workshop paper

The AACL, Asia-Pacific Chapter of the Association for Computational Linguistics, and the AFNLP, Asian Federation of Natural Language Processing, invite you to participate in the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) and the 10th International Joint Conference on Natural Language Processing (IJCNLP).

A registration is required to access this event. Access to the event site will be granted to all registered attendees on **November 16th**. If you have not already done so, please register using the link below:
 
 https://conference.iis.sinica.edu.tw/servlet/Register?ConferenceID=411

Registration Is Required

The AACL, Asia-Pacific Chapter of the Association for Computational Linguistics, and the AFNLP, Asian Federation of Natural Language Processing, invite you to participate in the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) and the 10th International Joint Conference on Natural Language Processing (IJCNLP). 

The distinction between mass nouns and count nouns has a long history in formal semantics, and linguists have been trying to identify the semantic properties defining the two classes. However, they also recognized that both can undergo meaning shifts and be used in contexts of a different type, via nominal coercion.
In this study, we present an approach to measure the meaning shift in count-mass coercion in English that makes use of static and contextualized word embedding distance. Our results show that the coercion shifts are detected only by a small subset of the traditional word embedding models and that the shifts detected by the contextualized embedding of BERT are more pronounced for mass nouns.

Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings.

In this presentation, we introduce a new approach that simulates the construction of phrasal meaning as an analogical process implemented with deep learning techniques. We propose a distributional representation of constructional phrases (namely Relational Embeddings) and a neural network model for generating new embeddings analogically rather than applying traditional compositional operations, which we call ANNE (Analogical Neural Network for Embeddings). 
We experimented with our analogical neural network to understand how it can generalize and be extendable to different scenarios. We argued that the proposed methodology could open the doors to new analyses in distributional semantics as well as in computational models of language processing.

Compositionality as an Analogical Process: Introducing ANNE.


From COMET to COMES – Can Summary Evaluation Benefit from Translation Evaluation?

Random Text Perturbations Work, but not Always

A Comparative Analysis of Stance Detection Approaches and Datasets

Assessing Neural Referential Form Selectors on a Realistic Multilingual Dataset

Assessing Resource-Performance Trade-off of Natural Language Models using Data Envelopment Analysis

Large pre-trained language models have brought remarkable progress in NLP. Pre-training and Fine-tuning have given state-of- art performance across tasks in text processing. Data Augmentation techniques have also helped build state-of-art models on low or zero resource tasks. Many works in the past have attempted at learning a single massively- multilingual machine translation model for zero-shot translation. Although those translation models are producing correct translations, the main challenge is those models are producing the wrong languages for zero-shot translation. This work and its results indicate that prompt conditioned large models do not suffer from off-target language errors i.e. errors arising due to translation to wrong languages. We empirically demonstrate the effectiveness of self-supervised pre-training and data augmentation for zero-shot multi-lingual machine translation.

MALM: Mixing Augmented Language Modeling for Zero-Shot Machine Translation

This corpus contains more than 30,000 classical Chinese poems in their tokenized form. The poems are from the Complete Tang Poems (全唐詩,CTP) and the Complete Song Poems (全宋詩,CSP). Two groups of annotators from two universities carried out the task. More than 5000 of these poems were annotated repeatedly for the inter-rater agreement analysis. The annotators did not agree with each other perfectly, as many readers may have expected, but the degrees and types of their agreements are worthy of discussion. The annotations support some comments, of the Ming dynasty, about the formats of the poems. We applied machine learning methods to train and test their usability for the tokenization task. We could identify 87% of the words, 77% of the whole lines, and 18% of the whole poems in the CTP correctly. We have expanded the scale of our annotated poems, have extended our exploration with deep learning methods, and we wish to report these extensions in a formal report.

Introducing a Large Corpus of Tokenized Classical Chinese Poems of Tang and Song Dynasties

Recent studies have reported that neural reasoning improves its accuracy when generating the intermediate reasoning steps. However, it is yet unclear what factors lead to such benefits.
We investigate and factorize the benefit of generating intermediate steps for symbolic reasoning.
Specifically, we decomposed the reasoning strategy regarding step granularity and chaining strategy. 
With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the performance, with the gap becoming even larger as the extrapolation length becomes longer.
Surprisingly, we also found that certain configurations lead to nearly perfect performance, even in the case of length extrapolation.
Our results indicate the importance of further exploring effective strategies for neural reasoning models.

Evaluating the role of non-lexical markers in GPT-2’s language modeling behavior

Downloads

Next from IJCNLP-AACL 2022

Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings.

Similar lecture

The Grammar-Learning Trajectories of Neural Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Evaluating the role of non-lexical markers in GPT-2’s language modeling behavior

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2022

Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings.

Similar lecture

The Grammar-Learning Trajectories of Neural Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads