arxiv:2409.04057

Self-Harmonized Chain of Thought

Published on Sep 6

· Submitted by

Xalphinions on Sep 12

Upvote

Authors:

Ziqi Jin ,

Abstract

Chain-of-Thought (CoT) prompting reveals that large language models are capable of performing complex reasoning via intermediate steps. CoT prompting is primarily categorized into three approaches. The first approach utilizes straightforward prompts like ``Let's think step by step'' to generate a sequential thought process before yielding an answer. The second approach makes use of human-crafted, step-by-step demonstrations to guide the model's reasoning process. The third automates the generation of reasoned demonstrations with the 'Let's think step by step'.This approach sometimes leads to reasoning errors, highlighting the need to diversify demonstrations to mitigate its misleading effects. However, diverse demonstrations pose challenges for effective representations. In this work, we propose ECHO, a self-harmonized chain-of-thought prompting method. It consolidates diverse solution paths into a uniform and effective solution pattern.ECHO demonstrates the best overall performance across three reasoning domains.

View arXiv page View PDF Add to collection

Community

Xalphinions

Paper author Paper submitter Sep 12

Is it possible for model to generate its own demonstration, which could potentially shifting its own performance from "zero-shot" to "few-shot"? Auto-CoT have shown its impact to replace human annotation. In this paper, we propose ECHO, a novel method that refine its generated demonstration using mutual prompting - refine one demonstration using others. Our method unifies the demonstrations and shows performance improvement in a wide range of tasks.

librarian-bot

Sep 13

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Xalphinions

Paper author Paper submitter Sep 16

Hi,
I have just updated the code to the GitHub for the lastest OPENAI standard: https://github.com/Xalp/ECHO

MaziyarPanahi

Sep 17

Thank you for including the Github repo! I didn't know there was any code! Based on the paper, I implemented a demo and it worked very well! https://x.com/MaziyarPanahi/status/1836083810930213098

I will go through the github to be sure if I got everything right, thanks again! (super interesting paper!)

Xalphinions

Paper author Sep 18

Amazing demonstrations you have done. Thank you!

onealeph0cc

Sep 18

Thank you for your work and publication. The source code you've provided is very helpful. While I believe we all intuitively feel this approach should work, I'd like to raise a question:

Beyond intuition, what specific ideas or principles led you to believe that this iterative process would consistently improve rationale quality?

Xalphinions

Paper author Sep 19

Hi, there are multiple intuition which I'd like to share after finishing this paper:
(1) The unified demonstrations will better match the case in the pre-training data of the model, where context from the same corpus are mutual relevant and consistent.
(2) Cognitive Load Theory (by John Sweller): learning is most effective when the cognitive load on working memory is minimized. If all demonstrations are coherent, it is easier to learn the pattern and follow both for human and model.
(3) You can also explain this with the Entropy Theory: unified demos reduces the information entropy (disorder and uncertainty), thus increasing the predictability.

These points are my personal opinions, which may not be able to prove. But I hope some of these ideas help you better understand why the method work.