Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CASEHold scenario #3164

Merged
merged 4 commits into from
Nov 18, 2024
Merged

Add CASEHold scenario #3164

merged 4 commits into from
Nov 18, 2024

Conversation

yifanmai
Copy link
Collaborator

CaseHOLD (Case Holdings On Legal Decisions) is a multiple choice question answering scenario where the task is to identify the relevant holding of a cited case (Zheng et al, 2021).

@yifanmai yifanmai requested review from mtake and ryokawajp November 15, 2024 19:39
@yifanmai
Copy link
Collaborator Author

Hi @ryokawajp and @mtake, please review this pull request.

By approving, you acknowledge that:

  • You will be listed as a co-author of the merge commit
  • You agree to license your work under Apache 2.0

I have made the following changes have been made on top of your original commit:

  • Used the test split instead of the validation split (despite the comment, the test split does have gold labels)
  • Removed the unneeded call to list_datasets()
  • Removed writing of JSONL files
  • Simplified the dataset loading logic
  • Edited the description and metadata
  • Changed all group and scenario names to casehold for consistency - removed QA because existing references in the literature call this CaseHOLD rather than CaseHOLDQA

Copy link
Collaborator

@ryokawajp ryokawajp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Thanks.

@yifanmai yifanmai merged commit 56f2454 into main Nov 18, 2024
8 checks passed
@yifanmai yifanmai deleted the yifanmai/fix-casehold branch November 18, 2024 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants