Korea, Republic of

We release a pretrained Japanese masked lan-
guage model for an academic domain. Pre-
trained masked language models have recently
improved the performance of various natural
language processing applications. In domains
such as medical and academic, which include
a lot of technical terms, domain-specific pre-
training is effective. While domain-specific
masked language models for medical and SNS
domains are widely used in Japanese, along
with domain-independent ones, pretrained mod-
els specific to the academic domain are not
publicly available. In this study, we pretrained
a RoBERTa-based Japanese masked language
model on paper abstracts from the academic
database CiNii Articles. Experimental results
on Japanese text classification in the academic
domain revealed the effectiveness of the pro-
posed model over existing pretrained models.

COLING 2022

A Japanese Masked Language Model for Academic Domain

academic domain

japanese

pretrain

[![](https://assets.underline.io/uploads/markdown_image/1/image/81a3c0317d24f663b49b996875024d45.png)](https://aclanthology.org/volumes/2022.coling-1/)



### THE CONFERENCE WE KNOW AND WE WANT.
**COLING**, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes but also in industrial research departments including tech-startups. COLING provides opportunities for all these communities to showcase their exciting discovery.

In fall of 2022, COLING will be held in Gyeongju in a hybrid format. All participants can either present at the venue site or join virtually. As more people get vaccinated, we are happy to provide safer environments for our colleagues. We believe that COLING 2022 will be one of the conferences, free from the pandemic. The hybrid format gives presenters and sponsors a valuable opportunity to promote their companies in both an online and in-person venue. For the first time in a long time, customers can interact with their sponsor's products first-hand. The online venue, too, gives sponsors the chance to network with those unable to attend the in-person session.

The hybrid format ultimately lends itself to greater exposure for our sponsors: COLING2022 will let you reach more potential partners and customers than ever before!<br><br><br>

To gain access to this event page you are required to register.<br> Find more information and pay the registration fee on event the organizer’s website **[https://coling2022.org/reg](https://coling2022.org/reg)**

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

workshop paper

[Mask]ing named entities during pretraining helps BART to preserve named entities in a generated summary:
We address the named entity omission - the drawback of many current abstractive text summarizers. We suggest a custom pretraining objective to enhance the model’s attention on the named entities in a text.

Named Entity Inclusion in Abstractive Text Summarization

A scientific paper is traditionally prefaced by an abstract that summarizes the paper. Recently, research highlights that focus on the main findings of the paper have emerged as a complementary summary in addition to an abstract. However, highlights are not yet as common as abstracts, and are absent in many papers.
In this paper, we aim to automatically generate research highlights using different sections of a research paper as input. We investigate whether the use of named entity recognition on the input improves the quality of the generated highlights. In particular, we have used two deep learning-based models: the first is a pointer-generator network, and the second augments the first model with coverage mechanism. We then augment each of the above models with named entity recognition features. 
The proposed method can be used to produce highlights for papers with missing highlights. Our experiments show that adding named entity information improves the performance of the deep learning-based summarizers in terms of ROUGE, METEOR and BERTScore measures.

Named Entity Recognition Based Automatic Generation of Research Highlights

We address automatic citation sentence generation, which reduces the burden on writing scientific papers.
For highly accurate citation senetence generation, appropriate language must be learned using information such as the relationship between the cited source and the cited paper as well as the context in which the paper cited.
Although the abstracts of papers have been used for the generation in the past, they often contain extra information in the citation sentence, which might negatively impact the generation of citation sentences.
Therefore, this study attempts to learn a highly accurate citation sentence generation model using sentences from cited articles that resemble the previous sentence to the cited location, thereby utilizing information that is more useful for citation sentence generation.

Citation Sentence Generation Leveraging the Content of Cited Papers

The paper describes neural models developed for the DAGPap22 shared task hosted at the Third Workshop on Scholarly Document Processing. This shared task targets the automatic detection of generated scientific papers. Our work focuses on comparing different transformer-based models as well as using additional datasets and techniques to deal with imbalanced classes. As a final submission, we utilized an ensemble of SciBERT, RoBERTa, and DeBERTa fine-tuned using random oversampling technique. Our model achieved 99.24% in terms of F1-score. The official evaluation results have put our system at the third place.

Detecting generated scientific papers using an ensemble of transformer models

We present a new gold-standard dataset and a benchmark for the \textit{Research Theme Identification} task, a sub-task of the Scholarly Knowledge Graph Generation shared task, at the 3rd Workshop on Scholarly Document Processing. The objective of the shared task was to label given research papers with research themes from a total of 36 themes. The benchmark was compiled using data drawn from the
largest overall assessment of university research
output ever undertaken globally (the Research Excellence Framework - 2014).

We provide a performance comparison of a transformer-based ensemble, which obtains multiple predictions for a research paper, given its multiple textual fields (e.g. title, abstract, reference), with traditional machine learning models. The ensemble involves enriching the initial data with additional information from open-access digital libraries and Argumentative Zoning techniques. It uses a weighted sum aggregation for the multiple predictions to obtain a final single prediction for the given research paper.

Both data and the ensemble are publicly available on https://www.kaggle.com/competitions/sdp2022-scholarly-knowledge-graph-generation/data?select=task1_test_no_label.csv and https://github.com/ProjectDoSSIER/sdp2022, respectively.

Benchmark for Research Theme Classification of Scholarly Documents

This paper is a description of our participation in the Multi-document Summarization for Literature Review (MSLR) Shared Task, in which we explore summarization models to create an automatic review of scientific results. Rather than maximizing the metrics using expensive computational models, we placed ourselves in a situation of scarce computational resources and investigate the limits of a base sequence to sequence models (thus with a limited input length) to the task. Although we explore methods to feed the abstractive model with salient sentences only (using a first extractive step), we find that the results still need some improvements.

Exploring the limits of a base BART for multi-document summarization in the medical domain

The MuP-2022 shared task focuses on multi-perspective scientific document summarization. Given a scientific document, with multiple reference summaries, our goal was to develop a model that can produce a generic summary covering as many aspects of the document as covered by all of its reference summaries. This paper describes our best official model, a fine-tuned BART large, along with a discussion on
the challenges of this task and some of our unofficial models including SOTA generation models. Our submitted model out performed the given, MuP 2022 shared task baselines on ROUGE-2, ROUGE-L and average ROUGE F1-scores. Code of our submission can be accessed here.

Multi-Perspective Scientific Document Summarization Using Pre-trained Generation Models

Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications

We are presenting an approach to the SV-Ident Shared Task which requires the detection and disambiguation of survey variables in sentences taken from social science publications. It deals with both subtasks as problems of semantic textual similarity (STS) and relies on the use of sentence transformers. Sentences and
variables are examined for semantic similarity for both detecting sentences containing variables and disambiguating the respective variables. The focus is placed on analyzing the effects of including different parts of the variables and observing the differences between English and German instances.

Variable Detection and Disambiguation Based on Semantic Similarity

Political competitions are complex settings where candidates use campaigns to promote their chances to be elected. One choice focuses on conducting a positive campaign that highlights the candidate's achievements, leadership skills, and future programs. The alternative is to focus on a negative campaign that emphasizes the \textit{negative aspects of the competing person} and is aimed at\textit{ offending opponents or the opponent’s supporters}. In this proposal, we concentrate on negative campaigns in Israeli elections. 
This work introduces an empirical case study on automatic detection of negative campaigns, using machine learning and natural language processing approaches, applied to the Hebrew-language data from Israeli municipal elections. 
Our contribution is multi-fold: (1) We provide TONIC---daTaset fOr Negative polItical Campaign in Hebrew---which consists of annotated posts from Facebook related to Israeli municipal elections; (2) We introduce results of a case study, that explored several research questions. RQ1: Which classifier and representation perform best for this task? We employed several traditional classifiers which are known for their excellent performance in IR tasks and two pre-trained models based on BERT architecture; several standard representations were employed with traditional ML models. RQ2: Does a negative campaign always contain offensive language? Can a model, trained to detect offensive language, also detect negative campaigns? We are trying to answer this question by reporting results for the transfer learning from a dataset annotated with offensive language to our dataset. 
RQ3: Does a negative campaign necessarily express negative sentiment? Can sentiment analysis help to detect negative campaigns? We experiment with sentiment labels to enrich data representation and report our findings. 

Our dataset and pre-trained models will be freely available for researchers.

Premium content

A Japanese Masked Language Model for Academic Domain

Downloads

Next from COLING 2022

Named Entity Inclusion in Abstractive Text Summarization

Similar lecture

Topicalization in Language Models: A Case Study on Japanese

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES