List Splitting/Clipping #565
Labels
effort:medium
More complex, might take several days to a week
enhancement
New feature or request
performance
Efficiency improvements
wip
Describe the solution you'd like
There is a bunch of prior work on splitting postings lists; this allows the high impact list to be traversed (more likely) than the low impact list, and since the low impact lists are typically long, accelerates retrieval. Recent work in EMNLP findings showed that this is highly beneficial to learned sparse retrieval mechanisms. I would like to see if this can be pulled into the PISA engine with perhaps a more elegant design (I currently have a private prototype used in the EMNLP experiments but it's a bit rough for release still). This would then supersede the (currently empty) repo here: https://github.com/JMMackenzie/postings-clipping
Paper: https://aclanthology.org/2022.findings-emnlp.205/
Additional context
TBA - This is a draft issue for now.
The text was updated successfully, but these errors were encountered: