[Wei et al., 2022] introduced chain-of-thought prompting as a technique to prompt LLMs to respond in a way that facilitates coherent and step-by-step reasoning processes. The main contribution lies in the proposal and exploring the thought chain prompt, demonstrating its effectiveness in eliciting more structured and thoughtful responses from LLMs compared to traditional prompts.
Principle of the chain of thought
Introduced by Wei et al. (2022), the method Chain of Thought (CoT) helps develop complex reasoning skills through intermediate reasoning steps. You can combine it with a few-step response method to get better results. results on more complex tasks that require reasoning before responding.
"The odd numbers in this group add up to an even number: 4, 8, 9, 15, 12, 2, 1.
A: Adding all the odd numbers (9, 15, 1) gives 25. The answer is False.
The odd numbers in this group add up to an even number: 17, 10, 19, 4, 8, 12, 24.
A: Adding all the odd numbers (17, 19) gives 36. The answer is True.
The odd numbers in this group add up to an even number: 16, 11, 14, 4, 8, 13, 24.
A: Adding all the odd numbers (11, 13) gives 24. The answer is True.
The odd numbers in this group add up to an even number: 17, 9, 10, 12, 13, 4, 2.
A: Adding all the odd numbers (17, 9, 13) gives 39. The answer is False.
The odd numbers in this group add up to an even number: 15, 32, 5, 13, 82, 7, 1.
HAS : "
The prompt generates the following response:
"Adding all the odd numbers (15, 5, 13, 7, 1) gives 41. The answer is False."
In itself, a single example of few-shot prompting is sufficient here.
Zero Shot CoT
The concept of “Zero-Shot Chain of Thought” (Zero-shot-CoT) prompting is an advanced iteration of the CoT prompting mechanism, where the “zero-shot” aspect implies that the model is able to perform some reasoning without having seen examples of the task during training. Augmenting queries with the phrase “Let’s think step by step” made it easier for LLMs to generate a sequential reasoning chain. This reasoning chain was then found to be instrumental in obtaining more accurate answers. This technique is based on the idea that the model, just like a human, can benefit from more detailed and logical steps to process the prompt and generate an answer.
Automatic CoT
When applying thought chain prompts with demonstrations, the process involves manually creating effective and diverse examples. This manual effort could lead to suboptimal solutions. Zhang et al. (2022) propose an approach to eliminate manual efforts by leveraging LLMs with the “Let’s think step by step” prompt to generate reasoning chains for demonstrations one by one. This automatic process can still result in errors in the generated chains. To mitigate the effects of errors, the diversity of demonstrations matters. This work proposes Auto-CoT, which samples questions with diversity and generates chains of reasoning to construct the demonstrations.
Auto-CoT consists of two main steps:
- Question clustering: partitioning questions from a given dataset into a few clusters
- Demonstration sampling: select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics
Simple heuristics could be the length of the questions (e.g., 60 tokens) and the number of steps in the justification (e.g., 5 reasoning steps). This encourages the model to use simple and precise demonstrations.
Golden chain-of-thought
The Golden Chain of Thought provides an innovative approach to generating query responses based on instructions. This methodology leverages a set of “ground truth chain of thought” solutions embedded in the prompt, greatly simplifying the model’s task as it avoids the need for independent CoT generation. In parallel, a new benchmark including detective puzzles was designed, to evaluate the abductive reasoning abilities of LLMs, which is also considered as an evaluation of the Golden CoT. GPT-4 shows commendable performance, with a puzzle solving rate of 83 %, in contrast to the standard CoT’s solving rate of 38 %.
Next, we always add the mystery name, suspect list, and mystery content (body) to the prompt. When we want to invoke chain reasoning, we also add the following:
Full answer:
Let's think step by step.
When we want to provide a golden chain of thought, we add the following prompt:
Solution: {solution}
Finally, we always ask for the final answer with
Final answer:
EN 
FR