raises error when dataset is an empty list in NanoBEIREvaluator #3122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Copilot reviewed 1 out of 1 changed files in this pull request and generated no suggestions.
tomaarsen
reviewed
Dec 9, 2024
| @@ -420,6 +420,8 @@ def _load_dataset(self, dataset_name: DatasetNameType, **ir_evaluator_kwargs) -> | |||
| ) | |||
|
|
|||
| def _validate_dataset_names(self): | |||
| if self.dataset_names == []: | |||
There was a problem hiding this comment.
Suggested change
| if self.dataset_names == []: | |
| if len(self.dataset_names) == 0: |
This is a personal preference, but I prefer checking lengths like this :)
There was a problem hiding this comment.
on second thought, theres also this case when datasets = [''], an empty string,
currently it recognizes as an invalid dataset
ValueError: Dataset(s) [''] not found in the NanoBEIR collection.Valid dataset names are: ['climatefever', 'dbpedia', 'fever', 'fiqa2018', 'hotpotqa', 'msmarco', 'nfcorpus', 'nq', 'quoraretrieval', 'scidocs', 'arguana', 'scifact', 'touche2020']
would it be nicer to consider it as an empty dataset and use this check instead?
There was a problem hiding this comment.
I don't think we have to worry about this case, I don't think many people would use [""] and if they do, the error is already pretty descriptive.
|
Thanks again!
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Raises an error when dataset names are a empty list in NanoBEIREvaluator.
Fixes #3103
Who can review?
@tomaarsen