Enforcing 3 Demos in MIPROv2 Meta Prompt #1964
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previous Behavior:
In some cases, only 1 demo was included in the meta-prompt. This occurred when few-shot examples were generated with a size of 1 (since the size is randomly chosen between 1: max_bootstrap_fewshot), and when max_bootstrap_fewshot is set to 1.
New behavior:
We wanted for the # of few-shot examples in the prompt to always equal 3 to avoid potential overfitting of the generated instruction to a single example.
The fix:
We will essentially did the following:
prompt_demos = (my_demos + all_other_demos)[:min(3, len(my_demos + all_other_demos))]
Which will give us 3 demos for all prompts, unless there are less than 3 demos total across all bootstrapped few-shot examples. Note that we keep the demos empty for a single instruction generation, just in case demos end up being counter productive in the meta prompt.
A few other minor additions: