Machine translation metrics

[mahmoudaymo]

Machine translation metrics such as Bleu or PED are usually computed at corpus level meaning that to evaluate the program we need to feed it all the Examples or Devset instead of one example at a time. How can we use these metrics in DSPY

[chenmoneygithub]

I don't fully understand the issue, computing BLEU in DSPy shouldn't be different from other cases. You can define any custom metric that takes in the train/dev/test data, and the output from DSPy program: https://dspy.ai/cheatsheet/#dspy-metrics