Thailand

This paper presents the first large-scale meta-evaluation of machine translation (MT). We annotated MT evaluations conducted in 769 research papers published from 2010 to 2020. Our study shows that practices for automatic MT evaluation have dramatically changed during the past decade and follow concerning trends. An increasing number of MT evaluations exclusively rely on differences between BLEU scores to draw conclusions, without performing any kind of statistical significance testing nor human evaluation, while at least 108 metrics claiming to be better than BLEU have been proposed. MT evaluations in recent papers tend to copy and compare automatic metric scores from previous work to claim the superiority of a method or an algorithm without confirming neither exactly the same training, validating, and testing data have been used nor the metric scores are comparable. Furthermore, tools for reporting standardized metric scores are still far from being widely adopted by the MT community. After showing how the accumulation of these pitfalls leads to dubious evaluation, we propose a guideline to encourage better automatic MT evaluation along with a simple meta-evaluation scoring method to assess its credibility.

ACL 2021

Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

evaluation guideline

meta-evaluation

machine translation evaluation

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

This Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) was to be held in Bangkok, Thailand, during August 1-6, 2021 and is now virtual on the Underline platform.

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

technical paper

An Empirical Study on Adversarial Attack on NMT: Languages and Positions Matter

Few-NERD: A Few-shot Named Entity Recognition Dataset

MultiMET: A Multimodal Dataset for Metaphor Understanding

Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech

Dynamic and Multi-Channel Graph Convolutional Networks for Aspect-Based Sentiment Analysis

Automatic Text Simplification for Social Good: Progress and Challenges

A Neural Edge-Editing Approach for Document-Level Relation Graph Extraction

Dialogue-oriented Pre-training

GrantRel: Grant Information Extraction via Joint Entity and Relation Extraction

Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model

https://d3smihljt9218e.cloudfront.net/transcript/17871/subtitle/477d223b176f7ecfb16ec5c8e2295a58.vtt

English (automatic)

Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

Downloads

Next from ACL 2021

An Empirical Study on Adversarial Attack on NMT: Languages and Positions Matter

Similar lecture

Robustifying Multi-hop QA through Pseudo-Evidentiality Training

VIDEO LIBRARY

EVENTS

COMPANY

RESOURCES