Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforcing 3 Demos in MIPROv2 Meta Prompt #1964

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

klopsahlong
Copy link
Collaborator

Previous Behavior:

In some cases, only 1 demo was included in the meta-prompt. This occurred when few-shot examples were generated with a size of 1 (since the size is randomly chosen between 1: max_bootstrap_fewshot), and when max_bootstrap_fewshot is set to 1.

New behavior:

We wanted for the # of few-shot examples in the prompt to always equal 3 to avoid potential overfitting of the generated instruction to a single example.

The fix:

We will essentially did the following:
prompt_demos = (my_demos + all_other_demos)[:min(3, len(my_demos + all_other_demos))]

Which will give us 3 demos for all prompts, unless there are less than 3 demos total across all bootstrapped few-shot examples. Note that we keep the demos empty for a single instruction generation, just in case demos end up being counter productive in the meta prompt.

A few other minor additions:

  • Renamed some variables for legibility
  • Added back in the generated module description to the meta-prompt, which summarizes what the module we're generating an instruction for is responsible for

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants