United States

Peer assessment is an effective and efficient pedagogical strategy for delivering feedback to learners. Asking students to provide quality feedback, which contains suggestions and mentions problems, can promote metacognition by reviewers and better assist reviewees in revising their work. Thus, various supervised machine learning algorithms have been proposed to detect quality feedback. However, all these powerful algorithms have the same Achilles&#39; heel: the reliance on sufficient historical data. In other words, collecting adequate peer feedback for training a supervised algorithm can take several semesters before the model can be deployed to a new class. In this paper, we present a new paradigm, called incremental zero-shot learning (IZSL), to tackle the problem of lacking sufficient historical data. Our results show that the method can achieve acceptable &quot;cold-start&quot; performance without needing any domain data, and it outperforms BERT when trained on the same data collected incrementally.

NAACL 2022

Starting from &quot;Zero&quot;: An Incremental Zero-shot Learning Approach for Assessing Peer Feedback Comments

peer assessment zero-shot learning cold-start natural language inference

Welcome to the **largest national, multi-specialty research** event featuring research from nearly 700 medical students, residents, fellows, and international medical graduates.

Use the buttons below or navigation menu to access the event.
 


[![](https://assets.underline.io/uploads/markdown_image/1/image/3e33a8c754cc49610b30950617d813e7.png)](https://underline.io/events/434/sessions?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/551404d429c4e274f9a54cbdd7081918.png)](https://underline.io/events/434/posters?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/677d92f566b880cebd36905fc738fc10.png)](https://underline.io/events/434/lounge)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/c7392833c7261c3874c5b43fa2f4df19.png)](https://underline.io/events/434/help-desk)
 

#### Event Details

The 2023 American Medical Association Research Challenge includes a poster symposium hall and a semifinals hall with the top scored research. **Log in at any time throughout the event to:**

* **View posters and presentations** on a variety of topics and specialties
* **Score posters in the semifinals** to help decide the top five finalists (AMA member exclusive opportunity)
* **Connect with participants** and provide feedback to help advance their research
* **Attend education sessions** with physician experts on key research topics

At the conclusion of this event, the AMA will announce the top five finalists who will advance to the AMA Research Challenge finals, present their research to an elite panel of judges, and compete to win a $10,000 grand prize sponsored by Laurel Road. 

### Grand Prize Sponsor

AMA would like to thank Laurel Road for sponsoring a $10,000 grand prize for the winner of the AMA Research Challenge. Laurel Road is not involved in the judging or scoring process to select the winner. The winner will be announced on February 4, 2024 during the AMA Research Challenge finals. Payment of the grand prize will be directly from Laurel Road. See the AMA Research Challenge Official Rules for terms and conditions.

![](https://assets.underline.io/uploads/markdown_image/1/image/611e331d4a0eba0920f240cfc04b4d77.png)

**There is no registration fee for this event! **

Register to explore research in a variety of topics. AMA members get exclusive access to score posters in the semifinals.

**Please use this link [here](https://www.ama-assn.org/about/events/ama-research-challenge) to register for free.**

Registration Is Required

AMA Research Challenge 2023

Join the AMA Research Challenge poster symposium and semifinals to explore top research posters, score posters in the semifinals (an AMA member exclusive opportunity) and attend education sessions with experienced physicians to learn tips to advance your research and career. 

Peer assessment is an effective and efficient pedagogical strategy for delivering feedback to learners. Asking students to provide quality feedback, which contains suggestions and mentions problems, can promote metacognition by reviewers and better assist reviewees in revising their work. Thus, various supervised machine learning algorithms have been proposed to detect quality feedback. However, all these powerful algorithms have the same Achilles' heel: the reliance on sufficient historical data. In other words, collecting adequate peer feedback for training a supervised algorithm can take several semesters before the model can be deployed to a new class. In this paper, we present a new paradigm, called incremental zero-shot learning (IZSL), to tackle the problem of lacking sufficient historical data. Our results show that the method can achieve acceptable "cold-start" performance without needing any domain data, and it outperforms BERT when trained on the same data collected incrementally.

Starting from "Zero": An Incremental Zero-shot Learning Approach for Assessing Peer Feedback Comments

workshop paper

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

Spoken grammatical error correction (SGEC) is an important process 
to provide feedback for second language learning. Due to a lack of 
end-to-end training data, SGEC is often implemented as a cascaded, 
modular, system: speech recognition; disfluency removal; and grammatical 
error correction (GEC). This cascaded structure enables efficient use of 
training data for each module. It is, however, difficult to compare 
and evaluate the system performance as upstreams modules 
may introduce errors. For example, the GEC module input depends on the 
output of non-native speech recognition and disfluency detection, both 
challenging tasks for learner data. This work focuses on the assessment and de- velopment of SGEC systems. To mitigate error propagation in cascaded SGEC systems, semi-supervised finetuning and self-distillation approaches are investigated. Lastly, confidence based filtering is adopted to improve the feedback quality.

On Assessing and Developing Spoken ’Grammatical Error Correction’ Systems

In field of teaching, true/false questioning is an important educational method for assessing students' general understanding of learning materials. Manually creating such questions requires extensive human effort and expert knowledge. Question Generation (QG) technique offers the possibility to automatically generate a large number of questions. However, there is limited work on automatic true/false question generation due to the lack of training data and difficulty finding question-worthy content. In this paper, we propose an unsupervised True/False Question Generation approach (TF-QG) that automatically generates true/false questions from a given passage for reading comprehension test. TF-QG consists of a template-based framework that aims to test the specific knowledge in the passage by leveraging various NLP techniques, and a generative framework to generate more flexible and complicated questions by using a novel masking-and-infilling strategy. Human evaluation shows that our approach can generate high-quality and valuable true/false questions. In addition, simulated testing on the generated questions challenges the state-of-the-art inference models from NLI, QA, and fact verification tasks.

Automatic True/False Question Generation for Educational Purpose

“Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers' ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similar transformer-based classification models.

Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms

The growing demand for learning English as a second language has led to an increasing interest in automatic approaches for assessing spoken language proficiency. One of the most significant challenges in this field is the lack of publicly available annotated spoken data. Another common issue is the lack of consistency and coherence in human assessment. To tackle both problems, in this paper we address the task of automatically predicting the scores of spoken test responses of Englishas- a-second-language learners by training neural models on written data and using the presence of grammatical errors as a feature, as they can be considered consistent indicators of proficiency through their distribution and frequency. Specifically, we train a feature extractor on EFCAMDAT, a large written corpus containing error annotations and proficiency levels assigned by human experts, in order to extract information related to grammatical errors and, in turn, we use the resulting model for inference on the CLC-FCE corpus, on the ICNALE corpus, and on the spoken section of the TLT-school corpus, a collection of proficiency tests taken by Italian students. The work investigates the impact of the feature extractor on spoken proficiency assessment as well as the written-to-spoken approach. We find that our error-based approach can be beneficial for assessing spoken proficiency. The results obtained on the considered datasets are discussed and evaluated with appropriate metrics.

Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English

The dominating paradigm for content scoring is to learn an instance-based model, i.e. to use lexical features derived from the learner answers themselves. An alternative approach that receives much less attention is however to learn a similarity-based model. We introduce an architecture that efficiently learns a similarity model and find that results on the standard ASAP dataset are on par with a BERT-based classification approach.

Similarity-Based Content Scoring – How to Make S-BERT Keep Up With BERT

In this paper, we explore the role of topic information in student essays from an argument mining perspective. We cluster a recently released corpus through topic modeling into prompts and train argument identification models on various data settings. Results show that, given the same amount of training data, prompt-specific training performs better than cross-prompt training. However, the advantage can be overcome by introducing large amounts of cross-prompt training data.

Don’t Drop the Topic – The Role of the Prompt in Argument Identification in Student Writing

When listening comprehension is tested as a free-text production task, a challenge for scoring the answers is the resulting wide range of spelling variants. When judging whether a variant is acceptable or not, human raters perform a complex holistic decision. In this paper, we present a corpus study in which we analyze human acceptability decisions in a high stakes test for German. We show that for human experts, spelling variants are harder to score consistently than other answer variants. Furthermore, we examine how the decision can be operationalized using features that could be applied by an automatic scoring system. We show that simple measures like edit distance and phonetic similarity between a given answer and the target answer can model the human acceptability decisions with the same inter-annotator agreement as humans, and discuss implications of the remaining inconsistencies.

‘Meet Me at the Ribary’ – Acceptability of Spelling Variants in Free-Text Answers to Listening Comprehension Prompts

Automatic grouping of textual answers has the potential of allowing batch grading, but is challenging because the answers, especially longer essays, have many claims. To explore the feasibility of grouping together answers based on their semantic meaning, this paper investigates the grouping of short textual answers, proxies of single claims. This is approached as a paraphrase identification task, where neural and non-neural sentence embeddings and a paraphrase identification model are tested. These methods are evaluated on a dataset consisting of over 4000 short textual answers from various disciplines. The results map out the suitable question types for the paraphrase identification model and those for the neural and non-neural methods.

Towards Automatic Short Answer Assessment for Finnish as a Paraphrase Retrieval Task

Discovering individuals' suicidality on social media has become increasingly important. Many researchers have studied to detect suicidality by using a suicide dictionary. However, while prior work focused on matching a word in a post with a suicide dictionary without considering contexts, little attention has been paid to how the word can be associated with the suicide-related context. To address this problem, we propose a suicidality detection model based on a graph neural network to grasp the dynamic semantic information of the suicide vocabulary by learning the relations between a given post and words. The extensive evaluation demonstrates that the proposed model achieves higher performance than the state-of-the-art methods. We believe the proposed model has great utility in identifying the suicidality of individuals and hence preventing individuals from potential suicide risks at an early stage.

Detecting Suicidality with a Contextual Graph Neural Network

There is growing evidence that mobile text message exchanges between patients and therapists can augment traditional cognitive behavioral therapy. The automatic characterization of patient thinking patterns in this asynchronous text communication may guide treatment and assist in therapist training. In this work, we automatically identify distorted thinking in text-based patient-therapist exchanges, investigating the role of conversation history (context) in distortion prediction. We identify six unique types of cognitive distortions and utilize BERT-based architectures to represent text messages within the context of the conversation. We propose two approaches for leveraging dynamic conversation context in model training. By representing the text messages within the context of the broader patient-therapist conversation, the models better emulate the therapist's task of recognizing distorted thoughts. This multi-turn classification approach also leverages the clustering of distorted thinking in the conversation timeline. We demonstrate that including conversation context, including the proposed dynamic context methods, improves distortion prediction performance. The proposed architectures and conversation encoding approaches achieve performance comparable to inter-rater agreement. The presence of any distorted thinking is identified with relatively high performance at 0.73 F1, significantly outperforming the best context-agnostic models (0.68 F1).

Downloads

Next from NAACL 2022

On Assessing and Developing Spoken ’Grammatical Error Correction’ Systems

Similar lecture

"Let's Unpack My Feedback!" - An Approach to Enhance Medical Students' Peer Feedback

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES