feat: create embeddings store (WIP)

[mathisobadia]

So this is a bigger PR than what I would have like initially and would like some feedback before finishing

Context:

#16 the main reason for this change was to remove the embeddings class dependency on the fs module. fs is only available in node, and not in other runtimes such as Cloudflare workers, Vercel edge runtime etc...

The idea is to make the embeddings class more abstract and to create a new class called embeddingsStore that represents the way embeddings are stored. I also implemented 3 different stores to make sure that it was the right level of abstraction and that it would work with both files and vector databases. So there is 3 new files that implement different types of stores:

• file store: works the same as the embeddings class was initially working, saves a json to the file system using fs
• memory store: the most simple store possible, nothing is saved everything is just kept in memory
• pinecone store: uses the pinecone client to query embeddings in a pinecone index

Notes

I updated the examples but not the docs, I want to make sure that this is the right path before doing that

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
docs-promptable	❌ Failed (Inspect)			Feb 16, 2023 at 1:59PM (UTC)

[cfortuner]

this is awesome 👏 🔥 @mathisobadia

It looks really good! I'm going to dig into this more today and share my thoughts.

[rschwabco]

@mathisobadia Pinecone 🥑 here! Would love to pitch in to replace the unofficial Pinecone client with the official one 😃 . I can open a PR against your branch if that works.

[mathisobadia]

@rschwabco oh yeah ! the fact that there is an official pinecone client is news to me! But what I liked about this specific client is that it’s based on fetch, so It can run in all modern runtimes like vercel edge runtime / cloudflare workers etc… it looks like the the official client is based on axios which does not work on the edge runtime (last time I checked). This is also an issue with the openai client so it’s not like it’s the only blocker but I guess now is a good time to give feedback on the official pinecone client 😅
Here is some additional info on why supporting those runtime is important for this kind of apps
https://vercel.com/blog/gpt-3-app-next-js-vercel-edge-functions#edge-functions-vs.-serverless-functions
and additional info on the edge runtime
https://edge-runtime.vercel.sh/features/available-apis
The requirements are very similar in cloudflare workers and I think deno / netlify edge functions so this is not a vercel only thing.
Anyway all this to say that it would be great if the official pinecone client supported a fetch based solution ! Let me know if that is something you can consider !

[rschwabco]

Hey @mathisobadia - while I know Axios doesn't support the Edge runtime, there are a couple of things that may be helpful here:

1. You can still use a node runtime in Vercel.
2. Our client is currently in early preview, I'll make sure that future releases do not rely on Axios.

[kacperlukawski]

@mathisobadia Great! It would be great to include some other stores - I can help implement Qdrant!

[mathisobadia]

@kacperlukawski Yeah definitely! But just so you know this PR might not get merged in it's current state. We have been discussing with @cfortuner the best way to include embeddings / document stores and we are still actively trying things out to see what is the best API to do that so it's probably better to wait until something gets merged before we start implementing other vector stores.

[rschwabco]

@mathisobadia - just FYI, we've moved @pinecone-database/pinecone to use fetch instead of axios. I hope that helps. If you want, we can re-open this PR and update the versions, should be pretty seamless.