Thailand

Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt &quot;post processes&#39;&#39; paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search &amp; refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. 
The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. 
Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://github.com/forangel2014/ItD.

ACL 2024

ItD: Large Language Models Can Teach Themselves Induction through Deduction

induction

inductive reasoning

large language models

The 46th Annual Meeting of the Cognitive Science Society was an in-person meeting held in Rotterdam, The Netherlands at the Postillion Hotel & Conference Centre.

**ON-DEMAND PROGRAM ACCESS AFTER THE CONFERENCE** 
Recordings of the invited program are now available. You can access them by clicking on the 'Schedule' icon on the left. Select view by week and navigate to the day and time slot of the recording you wish to view. Click on the time slot to access the recording.

Recordings available (click on each recording to view):

**Thursday, July 25** 
0900: [Keynote Speaker Morgan Barense on Enhancing real-world event memory](https://underline.io/events/465/sessions/17997/lecture/99084-dynamics-between-minds-and-the-environment) 
1000: [Gleitman Award Winner Isabelle Dautriche on Language Foundations: Insights from acquisition, communication, cognition, and more](https://underline.io/events/465/sessions/18000/lecture/99085-gleitman-talk) 
1415: [Invited Symposium: Dynamics between minds and the environment](https://underline.io/events/465/sessions?eventSessionId=18016&searchGroup=lecture) 
1700: [Rumelhart Prize Presentation Speaker Alison Gopnik on Exploit, explore, empower: Three ages and three intelligences](https://underline.io/events/465/sessions/18028/lecture/99161-rp1-rumelhart-prize-presentation) 

**Friday, July 26** 
0900: [C.L. de Carvalho-Heineken Prize Keynote Speaker Kia Nobre on Focusing in memory](https://underline.io/events/465/sessions/18035/lecture/99162-hp1-c-l-de-carvalho-heineken-prize-keynote-address) 
1030: [Rumelhart Symposium: Childhood as exploration](https://underline.io/events/465/sessions/18038/lecture/100346-childhood-as-exploration) 
1415: [Elman Prize Symposium](https://underline.io/events/465/sessions?eventSessionId=18053&searchGroup=lecture) 
1600: [Invited Symposium: Dynamics between minds](https://underline.io/events/465/sessions?eventSessionId=18065&searchGroup=lecture) 
1745: [Keynote Speaker Andrea E. Martin on Neural dynamics encode the structure and statistics of language](https://underline.io/events/465/sessions/18077/lecture/99271-neural-dynamics-encode-the-structure-and-statistics-of-language) 

**Saturday, July 27** 
0900: [Keynote Speaker Gregor Schöner on How higher cognition emerges from the dynamics of strongly interacting neural populations](https://underline.io/events/465/sessions/18083/lecture/99273-dynamics-within-the-mind) 
1000: [Glushko Talks](https://underline.io/events/465/sessions?eventSessionId=18086&searchGroup=lecture) 
1500: [Invited Symposium: Dynamics within minds](https://underline.io/events/465/sessions?eventSessionId=18100&searchGroup=lecture) 


#### **CogSci 2024 Program Book**
<div style="position:relative;padding-top:0;width:900px;height:500px;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/cble/" seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


**The Program Book can be downloaded [here](https://drive.google.com/file/d/1yRVp1cBqRnAbdYi4GXWV_pZeyWNsRquv/view?usp=sharing).**

#### **Letter from the President**

Welcome to the 2024 Meeting of the Cognitive Science Society in Rotterdam! CogSci 2024 brings together a large community of cognitive scientists who have traveled here from around the world, as well as a smaller group of remote presenters. I would like to extend an especially warm welcome to our first-time CogSci meeting attendees, and hope that 
their experience inspires them to join our future meetings for many years to come.

I want to thank this year’s conference co-chairs, Larissa K Samuelson, Stefan Frank, Mariya Toneva, Allyson Mackey, and Eliot Hazeltine, for putting together an amazing series of invited talks and symposia around the theme of Dynamics of Cognition. The co-chairs also deserve credit for handling a record-breaking number of submissions and producing a truly exciting conference program 

Beyond the invited talks and symposia dedicated to this year’s conference theme, I would like to highlight the presentations and events honoring the 
Rumelhart, Elman, Gleitman and C .L . de Carvalho Heineken prize winners, including the Rumelhart reception on Thursday. As you may know, we are recording all of the invited talks and symposia, and all prize-winners’ talks, and will be making this content available after the conference. 

The Cognitive Science Society relies on the volunteer efforts of our Governing Board members who work on matters related to our membership; conference policies; diversity and inclusion, international, and outreach initiatives; prizes and much more. We welcome interest from Society members who would like to become more involved in what we do. The 
Society is deeply grateful to Rick Dale and Andrea Bender, who serve as editors of the two Society journals, Cognitive Science and Topics in Cognitive Science respectively. We especially acknowledge our editors’ generosity in donating their stipends each year to D&I and outreach efforts. 

This year’s Annual Meeting depends critically on the support of Marischal De Armond and his team at Podium Conference Specialists: Sharon Zwack, Cendrine De Vis, Sarah-Kate Burke, and Rachel Waller worked tirelessly to secure the venue and handled a multitude of logistical and organizational issues, from arranging coffee breaks and receptions to making sure that a huge number of posters fits comfortably into the available space. Finally, and importantly, I would like to acknowledge the Cognitive Science Society Executive Officer, Erica Wojcik, for managing the complex and ever expanding sphere of our Society’s activities.

I hope you enjoy this year’s meeting and the many different opportunities to engage with the vibrant community of cognitive scientists gathered here. I also hope you get a chance to explore the great city of Rotterdam with its many attractions, museums, restaurants and scenic views. If you’d like to talk more about our Society, or simply want to say hello, please stop by!

![](https://assets.underline.io/markdown_image/1/image/3c66d0e21fff30e204df1138ffd15173.jpeg)

**Anna Papafragou** 
President, Cognitive Science 
Society 2023-2024

**CODE OF CONDUCT** 
By attending the CogSci 2024 Conference, you are required to adhere to the society’s [Code of Conduct](https://drive.google.com/file/d/16-6KkptF0Gn3ZYGJDlpqpwTdImw45Ng0/view?usp=drive_link).

**ABOUT THE COGNITIVE SCIENCE SOCIETY** 
The Cognitive Science Society brings together researchers from around the world who hold a common goal: understanding the nature of the human mind. The mission of the Society is to promote Cognitive Science as a discipline, and to foster scientific interchange among researchers in various areas of study, including Artificial Intelligence, Linguistics, Anthropology, Psychology, Neuroscience, Philosophy, and Education.

The Society is a non-profit professional organization and its activities include sponsoring an annual conference and publishing the journals Cognitive Science and TopiCS.

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

It looks like you are not registered for this event. 

To access the site please register [**here**](https://cognitivesciencesociety.org/registration/). 

Please register!

CogSci 2024

The 46th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt "post processes'' paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. 
The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. 
Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://github.com/forangel2014/ItD.

poster

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

Large language models (LLMs) have exhibited great potential in mathematical reasoning. However, there remains a performance gap in this area between existing open-source models and closed-source models such as GPT-4. In this paper, we introduce MathGenie, a novel method for generating diverse and reliable math problems by leveraging the ground-truth solutions of the seed data. We augment these ground-truth solutions and use a specially finetuned model to translate these augmented solutions back into new questions. Subsequently, we generate code-integrated solutions for these questions. To ensure the correctness of the code-integrated solutions, we employ rationale-based verification for filtering. Then, we finetune various pretrained models, ranging from 7B to 70B, on the newly curated data, resulting in a family of models known as MathGenie. These models consistently outperform previous open-source models across five representative mathematical reasoning datasets, achieving state-of-the-art performance. In particular, MathGenie-InternLM2 achieves an accuracy of 87.7% on GSM8K and 55.7% on MATH, securing the best overall score.

MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs

Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) with retrieved documents related to user queries. However, RAG only focuses on improving the response quality of LLMs via enhancing queries indiscriminately with retrieved information, paying little attention to what type of knowledge LLMs really need to answer original queries more accurately. In this paper, we suggest that long-tail knowledge is crucial for RAG as LLMs have already remembered common world knowledge during large-scale pre-training. Based on our observation, we propose a simple but effective long-tail knowledge detection method for LLMs. Specifically, the novel Generative Expected Calibration Error (GECE) metric is derived to measure the ``long-tailness'' of knowledge based on both statistics and semantics. Hence, we retrieve relevant documents and infuse them into the model for patching knowledge loopholes only when the input query relates to long-tail knowledge. Experiments show that, compared to existing RAG pipelines, our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.

On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models

While recent Large Language Models (LLMs) have proven useful in answering user queries, they are prone to hallucination, and their responses often lack credibility due to missing references to reliable sources. An intuitive solution to these issues would be to include in-text citations referring to external documents as evidence. While previous works have directly prompted LLMs to generate in-text citations, their performances are far from satisfactory, especially when it comes to smaller LLMs. In this work, we propose an effective training framework using fine-grained rewards to teach LLMs to generate highly supportive and relevant citations, while ensuring the correctness of their responses. We also conduct a systematic analysis of applying these fine-grained rewards to common LLM training strategies, demonstrating its advantage over conventional practices. We conduct extensive experiments on Question Answering (QA) datasets taken from the ALCE benchmark and validate the model's generalizability using EXPERTQA. On LLaMA-2-7B, the incorporation of fine-grained rewards achieves the best performance among the baselines, even surpassing that of GPT-3.5-turbo.

Training Language Models to Generate Text with Citations via Fine-grained Rewards

Assembly code search is vital for reducing the burden on reverse engineers, allowing them to quickly identify specific functions using natural language within vast binary programs.
Despite its significance, this critical task is impeded by the complexities involved in building high-quality datasets. This paper explores training a Large Language Model (LLM) to emulate a general compiler. By leveraging Ubuntu packages to compile a dataset of 20 billion tokens, we further continue pre-train CodeLlama as a Virtual Compiler (ViC), capable of compiling any source code to assembly code. This approach allows for "virtual" compilation across a wide range of programming languages without the need for a real compiler, preserving semantic equivalency and expanding the possibilities for assembly code dataset construction. Furthermore, we use ViC to construct a sufficiently large dataset for assembly code search. 
Employing this extensive dataset, we achieve a substantial improvement in assembly code search performance, with our model surpassing the leading baseline by 26%.

Virtual Compiler Is All You Need For Assembly Code Search

Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them; that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods primarily focus on further training them. However, the extra training of LLMs is usually expensive in terms of GPU computing; even worse, some LLMs are not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective---Black-Box Prompt Optimization (BPO)---to perform alignments. The idea is to optimize user prompts to suit LLMs' input understanding, so as to best realize users' intents without updating LLMs' parameters. BPO leverages human preferences to optimize prompts, thus making it superior to LLM (e.g., ChatGPT) as a prompt engineer. Moreover, BPO is model-agnostic, and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version and 10% for GPT-4. Notably, the BPO-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining BPO with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training

In infilling tasks, sub-tokens, representing instances where a complete token is segmented into two parts, often emerge at the boundaries of prefixes, middles, and suffixes. Traditional methods focused on training models at the token level, leading to sub-optimal performance in character-level infilling tasks during the inference stage. Alternately, some approaches considered character-level infilling, but they relied on predicting sub-tokens in inference, yet this strategy diminished ability in character-level infilling tasks due to the large perplexity of the model on sub-tokens. In this paper, we introduce FIM-SE, which stands for Fill-In-the-Middle with both Starting and Ending character constraints. The proposed method addresses character-level infilling tasks by utilizing a line-level format to avoid predicting any sub-token in inference. In addition, we incorporate two special tokens to signify the rest of the incomplete lines, thereby enhancing generation guidance. Extensive experiments demonstrate that our proposed approach surpasses previous methods, offering a significant advantage. Code is available at https://github.com/SenseLLM/FIM-SE.

Empowering Character-level Text Infilling by Eliminating Sub-Tokens

Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named ``Attribute First, then Generate``, breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments (``select first``) and then conditioning the generation process on them (``then generate``), we ensure these segments also act as the output's fine-grained attributions (``select`` becomes ``attribute``). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains - and in some cases enhances - both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors.

Attribute First, then Generate: Locally-attributable Grounded Text Generation

Ocean science, which delves into the oceans that are reservoirs of life and biodiversity, is of great significance given that oceans cover over 70% of our planet’s surface. Recently, advances in Large Language Models (LLMs) have transformed the paradigm in science. Despite the success in other domains, current LLMs often fall short in catering to the needs of domain experts like oceanographers, and the potential of LLMs for ocean science is under-explored. The intrinsic reason may be the immense and intricate nature of ocean data as well as the necessity for higher granularity and richness in knowledge. To alleviate these issues, we introduce OceanGPT, the first-ever LLM in the ocean domain, which is expert in various ocean science tasks. We propose DoInstruct, a novel framework to automatically obtain a large volume of ocean domain instruction data, which generates instructions based on multi-agent collaboration. Additionally, we construct the first oceanography benchmark, OceanBench, to evaluate the capabilities of LLMs in the ocean domain. Though comprehensive experiments, OceanGPT not only shows a higher level of knowledge expertise for oceans science tasks but also gains preliminary embodied intelligence capabilities in ocean technology.

OceanGPT: A Large Language Model for Ocean Science Tasks

Social media bot detection has always been an arms race between advancements in machine learning bot detectors and adversarial bot strategies to evade detection. In this work, we bring the arms race to the next level by investigating the opportunities and risks of state-of-the-art large language models (LLMs) in social bot detection. To investigate the opportunities, we design novel LLM-based bot detectors by proposing a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities. To illuminate the risks, we explore the possibility of LLM-guided manipulation of user textual and structured information to evade detection. Extensive experiments with three LLMs on two datasets demonstrate that instruction tuning on merely 1,000 annotated examples produces specialized LLMs that outperform state-of-the-art baselines by up to 9.1% on both datasets, while LLM-guided manipulation strategies could significantly bring down the performance of existing bot detectors by up to 29.6% and harm the calibration and reliability of bot detection systems.

Downloads

Next from ACL 2024

MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs

Similar lecture

Causal induction in time

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES