Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize ODFE config to index 1M vectors #11

Open
DmitryKey opened this issue Apr 10, 2021 · 0 comments
Open

Optimize ODFE config to index 1M vectors #11

DmitryKey opened this issue Apr 10, 2021 · 0 comments
Assignees

Comments

@DmitryKey
Copy link
Owner

As discussed in https://dmitry-kan.medium.com/speeding-up-bert-search-in-elasticsearch-750f1f34f455 the current configuration of ODFE allows indexing maximum 200k vectors.

The goal is to index 1M vectors to compare with all other KNN implementations.

@DmitryKey DmitryKey self-assigned this Apr 10, 2021
DmitryKey added a commit that referenced this issue Aug 28, 2024
* #9 fix for method typo

* #3 docs for ODFE index configuration and hyper-parameters

* #9 reqs freeze

* #11 optimized params to reach 700k vectors indexed (still not 1M)

* #16 indexer for hnswlib, stores randomly generated vectors into binary index on disk

* #16 I/O for binary vector format from Yandex (image dataset)

* #16 hnswlib indexer for big-ann

* #16 vector data visualizer (tensorboard)

* #17 NSW graph visualization

* #17 NSW graph implementation

* #17 pca and t-sne

* #17 viz code (fbin->tsv)

* #17 sharding

* #17 sharding algorithm, first two steps

* #17 sharding algorithm, first two steps

* added toml config

* added IDE path

---------

Co-authored-by: dmitry.kan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant