Austria

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark — a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

ACL 2025

Transfer of Structural Knowledge from Synthetic Languages

workshop paper

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Automatic metaphor detection has often been inspired by linguistic procedures for manual metaphor identification. In this work, we test how closely the steps required by the Metaphor Identification Procedure VU Amsterdam (MIPVU) can be translated into prompts for generative Large Language Models (LLMs) and how well three commonly used LLMs are able to perform these steps. We find that while the procedure itself can be modeled with only a few compromises, neither language model is able to match the performance of supervised, fine-tuned methods for metaphor detection. All models failed to sufficiently filter out literal examples, where no contrast between the contextual and a more basic or concrete meaning was present. Both versions of LLaMa however signaled interesting potentials in detecting similarities between literal and metaphoric meanings that may be exploited in further work.

Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection

In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders.

Language Models are Universal Embedders

Speech Event Extraction (SpeechEE) is a challenging task that lies at the intersection of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), requiring the identification of structured event information from spoken language. In this work, we present a modular, pipeline-based SpeechEE framework that integrates high-performance ASR with semantic search-enhanced prompting of Large Language Models (LLMs). Our system first classifies speech segments likely to contain events using a hybrid filtering mechanism including rule-based, BERT-based, and LLM-based models. It then employs few-shot LLM prompting, dynamically enriched via semantic similarity retrieval, to identify event triggers and extract corresponding arguments. We evaluate the pipeline using multiple LLMs—Llama3-8B, GPT-4o-mini, and o1-mini—highlighting significant performance gains with o1-mini, which achieves 63.3% F1 on trigger classification and 27.8% F1 on argument classification, outperforming prior benchmarks. Our results demonstrate that pipeline approaches, when empowered by retrieval-augmented LLMs, can rival or exceed end-to-end systems while maintaining interpretability and modularity. This work provides practical insights into LLM-driven event extraction and opens pathways for future hybrid models combining textual and acoustic features.

SpeechEE@XLLM25: Retrieval-Enhanced Few-Shot Prompting for Speech Event Extraction

Large language models (LLMs) have demon-
strated impressive reasoning capabilities
through Chain-of-Thought (CoT) prompting.
However, their reasoning processes remain
inexplicable and uncontrollable. In this paper,
we introduce a Step-Wise Reasoning and
Verification (SWRV) framework, designed
as a two-stage Parser–Verifier pipeline, that
decomposes the reasoning process into discrete
inference steps and rigorously validates each
one. Our Parser extracts problem constraints
and the sequence of reasoning steps from
the LLM’s output, and our Verifier, either
LM-based or powered by a symbolic solver,
checks the logical correctness of every step. To
ensure robust parsing, we fine-tune a compact
LM on a small, high-quality annotation set
generated by a more capable LM. Experiments
on the LLMSR dataset built atop LogiQA show
significant gains over baselines, illustrating
the effectiveness of our method for step-wise
analysis of LLM reasoning.

LLMSR@XLLM25: SWRV: Empowering Self-Verification of Small
Language Models through Step-wise Reasoning and Verification

Large, high-quality annotated corpora remain
scarce in document-level entity and relation ex-
traction in zero-shot or few-shot settings. In
this paper, we present a fully automatic, LLM-
based pipeline for synthetic data generation and
in-context learning for document-level entity
and relation extraction. In contrast to exist-
ing approaches that rely on manually annotated
demonstrations or direct zero-shot inference,
our method combines synthetic data generation
with retrieval-based in-context learning, using
a reasoning-optimized language model. This
allows us to build a high-quality demonstra-
tion database without manual annotation and to
dynamically retrieve relevant examples at infer-
ence time. Based on our approach we produce
a synthetic dataset of over 5k Wikipedia ab-
stracts with approximately 59k entities and 30k
relation triples. Finally, we evaluate in-context
learning performance on the DocIE shared task,
extracting entities and relations from long doc-
uments in a zero-shot setting. We find that
in-context joint entity and relation extraction
at document-level remains a challenging task,
even for state-of-the-art large language models.

DocIE@XLLM25: In-Context Learning for Information Extraction using
Fully Synthetic Demonstrations

This paper illustrates our team system approach
in XLLM-ACL 2025 Task-III: LLM for Struc-
tural Reasoning (LLM-SR), aiming to solve
both Task: Question parsing and CoT parsing.
The process of extracting statements and evi-
dence is similar to Discourse Parsing. Correct
extraction of statements or evidence from the
COT is crucial at the outset. Next, the pair-
wise relationship between a specific statement
and its corresponding evidence is assessed (a
statement should be followed by its related evi-
dence from the CoT). Both semantic and lexical
similarity are used to evaluate the accuracy of
statements and evidence predictions. Finally,
once a statement-evidence pair is correctly ex-
tracted, it is evaluated to determine whether the
evidence can logically deduce the statement. To
tackle Question Parsing and CoT Parsing, we
implement and investigate various solutions, in-
cluding (1) applying different structural prompt
formats like JSON, Markdown, or XML. (2)
utilising various prompt techniques: Few-shot,
Chain of thought, and Multi-hop prompting.
(3) Taking advantage of Natural Language In-
ference (NLI) model for the Statement Verifi-
cation step. Our best official result is a 243.047
mean score for test phases A and B, and finally,
we rank 7th on the final leaderboard.

LLMSR@XLLM25: Integrating Reasoning Prompt Strategies with
Structural Prompt Formats for Enhanced Logical Inference

We introduce UIEPrompter, a unified, training-
free framework that secures 1st place in
the ACL 2025 shared competition on uni-
versal document-level information extraction.
UIEPrompter effectively addresses both named
entity recognition and relation extraction with-
out the need for annotated data. Leveraging
large language models, UIEPrompter estab-
lishes a zero-shot baseline through role-specific
prompts, which are then refined via few-shot
guidance and constrained output generation
prompt to align with competition schemas. Ad-
ditionally, by integrating outputs from several
large language models, we reduce individual
model biases, thereby improving overall perfor-
mance. Evaluated on the competition evalua-
tion dataset, UIEPrompter showcases outstand-
ing performance in document-level information
extraction, ultimately securing first place. The
implementation code is available on GitHub.

DocIE@XLLM25: UIEPrompter: A Unified Training-Free Framework for Universal Document-Level Information Extraction via Structured Prompt

The LLMSR@XLLM25 formulates a low-
resource structural reasoning task that chal-
lenges LLMs to generate interpretable, step-by-
step rationales with minimal labeled data. We
present Less is More, the third-place winning
approach in the LLMSR@XLLM25, which
focuses on structured reasoning from only
24 labeled examples. Our approach lever-
ages a multi-agent framework with reverse-
prompt induction, retrieval-augmented rea-
soning synthesis via GPT-4o, and dual-stage
reward-guided filtering to distill high-quality
supervision across three subtasks: question
parsing, CoT parsing, and step-level veri-
fication. All modules are fine-tuned from
Meta-Llama-3-8B-Instruct under a unified
LoRA+ setup. By combining structure valida-
tion with reward filtering across few-shot and
zero-shot prompts, our pipeline consistently
improves structure reasoning quality. These re-
sults underscore the value of controllable data
distillation in enhancing structured inference
under low-resource constraints. Our code is
available at https://github.com/JhCircle/
Less-is-More.

LLMSR@XLLM25: Less is More: Enhancing Structured Multi-AgentReasoning via Quality-Guided Distillation

We present Team asdfo123’s submission to
the LLMSR@XLLM25 shared task, which evalu-
ates large language models on producing fine-
grained, controllable, and interpretable rea-
soning processes. Systems must extract all
problem conditions, decompose a chain of
thought into statement–evidence pairs, and ver-
ify the logical validity of each pair. Lever-
aging only the off-the-shelf Meta-Llama-3-
8B-Instruct, we craft a concise few-shots,
multi-turn prompt that first enumerates all con-
ditions and then guides the model to label,
cite, and adjudicate every reasoning step. A
lightweight post-processor based on regular
expressions normalises spans and enforces the
official JSON schema. Without fine-tuning,
external retrieval, or ensembling, our method
ranks 5th overall, achieving macro-F1 scores
on par with substantially more complex and
resource-consuming pipelines. We conclude
by analysing the strengths and limitations
of our approach and outlining directions for
future research in structural reasoning with
LLMs. Our code is available at https://
github.com/asdfo123/LLMSR-asdfo123.

LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning

The schematization of knowledge, including
the extraction of entities and relations from
documents, poses significant challenges to tra-
ditional approaches because of the document’s
ambiguity, heterogeneity, and high cost domain-
specific training. Although Large Language
Models (LLMs) allow for extraction without
prior training on the dataset, the requirement
of fine-tuning along with low precision, espe-
cially in relation extraction, serves as an ob-
stacle. In absence of domain-specific training,
we present a new zero-shot ensemble approach
using DeepSeek-R1-Distill-Llama-70B, Llama-
3.3-70B, and Qwen-2.5-32B. Our key innova-
tion is a two-stage pipeline that first consoli-
dates high-confidence entities through ensem-
ble techniques, then leverages Qwen-2.5-32B
with engineered prompts to generate precise
semantic triples. This approach effectively re-
solves the low precision problem typically en-
countered in relation extraction. Experiments
demonstrate significant gains in both accuracy
and efficiency across diverse domains, with
our method ranking in the top 2 on the of-
ficial leaderboard in Shared Task-IV of The
1st Joint Workshop on Large Language Mod-
els and Structure Modeling. This competitive
performance validates our approach as a com-
pelling solution for practitioners seeking robust
document-level information extraction without
the burden of task-specific fine-tuning. Our
code can be found at https://github.com/
dinhthienan33/ZeroSemble.

Downloads

Next from ACL 2025

Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES