By systematically selecting and annotating the questions the most uncertain, active prompting not only helps refine the understanding of the model, but also leverages human expertise more effectively. The process begins with generating multiple predictions for each question, followed by calculating the uncertainty (uncertainty estimate) using various measures such as disagreement, entropy, and variance. This strategic selection process ensures that the most informative questions are prioritized for annotation.
The human in the loop
Active Prompt1 is a technique to improve the performance of Chain of Thought (CoT) prompting by selectively annotating examples where the model has the most uncertainty. This approach maximizes the efficiency of human annotation efforts by focusing only on the most challenging questions for the model.
The Active Invite consists of four main steps:
Uncertainty estimation:
It prompts the model multiple times (k) with unlabeled questions using the Chain of Thought Prompt with some human-written thought strings or the Zero-Shot (CoT) Chain of Thought Prompt without human-written thought strings to generate possible answers with intermediate steps for a set of unlabeled questions. It calculates the uncertainty u based on k answers via a selected uncertainty measure.
Selection: questions with the greatest uncertainty
are selected for human annotation.
Annotation: Questions selected in step 2 are manually annotated by humans to provide more accurate answers.
Inference: Newly annotated examples are used to improve the model's performance in answering future questions.
Active prompting saves significant human resources by reducing the need to annotate all training data. It outperforms other techniques such as automatic thought chain prompting, random thought chain prompting, and self-consistency on a range of reasoning tasks. research on the active prompt is the first to show the benefits of selectively annotating questions in the CoT prompt for solving complex reasoning tasks.
Example of active prompting
Step 1. Estimation of uncertainty
First, you ask the model multiple times (k) for each unlabeled question using:
A number of annotated examples if you wish to use the CoT option
“Think step by step” if you want to use the CoT Zero-Shot option
Let's say you choose the CoT option. You provide examples (Q1 and Q2), then ask your pool question (Q3). Repeat this process k times for each question.
Q1: Josh and Anna were both born on August 17th, but in different years. To consolidate celebrations they also got married on August 17 when Josh turned 22. If today they're celebrating 30 years of marriage and their combined age is exactly 5 times what Josh's age was when they married, how old was Anna when they got married? A1: Let's think step by step. To calculate how old was Anna when they got married, we have to know their combined age, Josh's age after 30 years, and Anna's age after 30 years from their marriage. Since their combined age is 5 times Josh's age when he got married, their combined age is 5 * 22 = 110 years. Josh must be 30 years older than his age when they got married, so he is 22 + 30 = 52 years old now. Therefore, Anna's current age will be 110 - 52 = 58 years. If they married 30 years ago, Anna must have been 58 - 30 = 28 years old when they married The answer is 28. Q2: John buys a chair. He then buys a table that is 3 times the price of the chair. Then, he buys a couch that is 5 times the price of the table. If John paid $380 for all these items, what is the price of the couch? A2: Let's think step by step. To calculate the price of the couch, we need to know the price of the chair, the price of the table, and the relationship between the chair, table, couch, and total money paid. Let x be the price of the chair, 3 * x be the price of the table, and 5 * (3 * x) = 15 * x be the price of the couch. The relationship between the chair, table, couch, and the total price paid is x + 3 * x + 15 * x = $380, which is 19 * x = 380, and x=20. The price of the couch is 15 * x, which is 15 * 20 = $300. The answer is 300. Q3: John has 5 apples, and he gives 2 to Mary. How many apples does John have left?
You will thus obtain k answers for each of your n questions.
Next, you need to measure the uncertainty of the model for each question based on the k answers it generates for a given question.
To do this, you select the uncertainty measure. An example measure might be disagreement:
You use the disagreement between the k answers generated for a given question in the pool. The disagreement calculates the unique answers in the predictions.
You count the number of unique answers generated for a question, h
Calculate the disagreement by dividing the uncertainty = h/k.
Step 2. Selection
Then you select the questions with the highest uncertainty based on the metric. For simplicity, let's just look at an example question from the set of most uncertain questions.
Imagine that you consider disagreement as a metric of usage uncertainty. You ask the model the question k times and find that the question below consistently produces results Different LLMs, meaning the model is uncertain in the response:
John has 5 apples, and he gives 2 to Mary.
How many apples does John have left?
Step 3. Annotation
You manually annotate the selected question to provide a clear and correct answer:
Q: John has 5 apples, and he gives 2 to Mary. How many apples does John have left?
A: John starts with 5 apples. He gives away 2 apples. Therefore, he has 5 - 2 = 3 apples left.
Step 4. Inference
This annotated question becomes an example for the model.
Using Active Prompting
The Active Prompting code is open source and available for further research and implementation on GitHub.
EN 
FR