Malta

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs&#39;&#39; sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

EACL 2024

Can Large Language Models Reason About Goal-Oriented Tasks?

sequential reasoning

in context learning

large language models

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs'' sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

workshop paper

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

No two authors write alike. Personal flourishes invoked in written narratives,
from lexicon to rhetorical devices, imply a particular author—what literary theorists
label the implied or virtual author; distinct from the real author or narrator
of a text. Early large language models trained on unfiltered training sets drawn
from a variety of discordant sources yielded incoherent personalities, problematic
for conversational tasks but proving useful for sampling literature from multiple
perspectives. Successes in alignment research in recent years have allowed researchers
to impose subjectively consistent personae on language models via instruction
tuning and reinforcement learning from human feedback (RLHF), but whether aligned
models retain the ability to model an arbitrary virtual author has received little
scrutiny. By studying 4,374 stories sampled from three OpenAI language models,
we show successive versions of GPT-3 suffer from increasing degrees of “mode collapse”
whereby overfitting the model during alignment constrains it from generalizing
over authorship: models suffering from mode collapse become unable to assume a
multiplicity of perspectives. Our method and results are significant for researchers
seeking to employ language models in sociological simulations.

Detecting Mode Collapse in Language Models via Narration

The extraction of structured data from websites is critical for numerous Artificial Intelligence applications, but modern web design increasingly stores information visually in images rather than in text. This shift calls into question the optimal technique, as language-only models fail without textual cues while new multimodal models like GPT-4 promise image understanding abilities. We conduct the first rigorous comparison between text-based and vision-based models for extracting event metadata harvested from comic convention websites. Surprisingly, our results between GPT-4 Vision and GPT-4 Text uncover a significant accuracy advantage for vision-based methods in an applies-to-apples setting, indicating that vision models may be outpacing language-alone techniques in the task of information extraction from websites. We release our dataset and provide a qualitative analysis to guide further research in multi-modal models for web information extraction.

The Future of Web Data Mining: Insights from Multimodal and Code-based Extraction Methods

Freedom of Information Act (FOIA) legislation grants citizens the right to request information from various levels of the government, and aims to promote the transparency of governmental agencies. However, the processing of these requests is often met with delays, due to the inherent complexity of gathering the required documents. To obtain accurate estimates of the processing times of requests, and to identify bottlenecks in the process, this research proposes a pipeline to automatically extract these timelines from decision letters of Dutch FOIA requests. These decision letters are responses to requests, and contain an overview of the process, including when the request was received, and possible communication between the requester and the relevant agency. The proposed pipeline can extract dates with an accuracy of .94, extract event phrases with a mean ROUGE- L F1 score of .80 and can classify events with a macro F1 score of .79. Out of the 50 decision letters used for testing (each letter containing one timeline), the model correctly classified 10 of the timelines completely correct, with an average of 3.1 mistakes per decision letter.

Timeline Extraction from Decision Letters Using ChatGPT

We describe a new weakly supervised method for sentence-level event detection, based exclusively on linear prototype patterns like "people got sick" or "a roadside bomb killed people". We propose a new BERT based algorithm for approximate pattern matching to identify event phrases, semantically similar to these prototypes. To the best of our knowledge, a similar approach has not been used in the context of event detection. We experimented with two event corpora in the area of disease outbreaks and terrorism and we achieved promising results in sentence level event identification: 0.78 F1 score for new disease cases detection and 0.68 F1 in detecting terrorist attacks. Results were in line with some state-of-the-art systems.

Leveraging Approximate Pattern Matching with BERT for Event Detection

There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text. The survey presented in this paper aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest ‚Äì both from the NLP and political science communities ‚Äì offering a unified view of the work done across different disciplines.

Socio-political Events of Conflict and Unrest: A Survey of Available Datasets

ChatGPT, developed by OpenAI, has made a significant impact on the world, mainly on how people interact with technology. In this study, we evaluate ChatGPT's ability to detect hate speech in Turkish tweets and measure its strength using zero- and few-shot paradigms and compare the results to the supervised fine-tuning BERT model. On evaluations with the SIU2023-NST dataset, ChatGPT achieved 65.81 accuracy in detecting hate speech for the few-shot setting, while BERT with supervised fine-tuning achieved 82.22 accuracy. This results supports previous findings that show that, despite its much smaller size, BERT is more suitable for natural language classifications tasks such as hate speech detection.

Evaluating ChatGPT's Ability to Detect Hate Speech in Turkish Tweets

This paper introduces a zero-shot hate detection experiment using a multimodal large model. Although the implemented model comprises an unsupervised method, results demonstrate that its performance is comparable to previous supervised methods. Furthemore, this study proposed experiments with various prompts and demonstrated that simpler prompts, as opposed to the commonly used detailed prompts in large language models, led to better performance for multimodal hate speech event detection tasks. While supervised methods offer high performance, they require significant computational resources for training, and the approach proposed here can mitigate this issue. The code is publicly available at https://github.com/yamagishi0824/zeroshot-hate-detect.

YYama@Multimodal Hate Speech Event Detection 2024: Simpler Prompts, Better Results - Enhancing Zero-shot Detection with a Large Multimodal Model

This paper describes the system that participated in the Climate Activism Stance and Hate Event Detection shared task organized at The 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024). The system tackles the important task of hate speech detection by combining large language model predictions with manually designed features, while trying to explain where the LLM approach fails to predict the correct results.

RACAI at ClimateActivism 2024: Improving Detection of Hate Speech by Extending LLM Predictions with Handcrafted Features

In the context of the proliferation of multimodal hate speech related to the Russia-Ukraine conflict, we introduce a unified multimodal fusion system for detecting hate speech and its targets in text-embedded images. Our approach leverages the Twitter-based RoBERTa and Swin Transformer V2 models to encode textual and visual modalities, and employs the Multilayer Perceptron (MLP) fusion mechanism for classification. Our system achieved macro F1 scores of 87.27 for hate speech detection and 80.05 for hate speech target detection in the Multimodal Hate Speech Event Detection Challenge 2024, securing the 1st rank in both subtasks. We open-source the trained models at https://huggingface.co/Yestin-Wang

CLTL@Multimodal Hate Speech Event Detection 2024: The Winning Approach to Detecting Multimodal Hate Speech and Its Targets

CASE in EACL 2024 proposes the shared task on Hate Speech and Stance Detection during Climate Activism. In our participation in the stance detection task, we have tested different approaches using LLMs for this classification task. We have tested a generative model using the classical seq2seq structure. Subsequently, we have considerably improved the results by replacing the last layer of these LLMs with a classifier layer. We have also studied how the performance is affected by the amount of data used in training. For this purpose, a partition of the dataset has been used and external data from posture detection tasks has been added.

Premium content

Can Large Language Models Reason About Goal-Oriented Tasks?

Downloads

Next from EACL 2024

Detecting Mode Collapse in Language Models via Narration

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES