Does "nlp.pipe(docs)" groups requests to an API ?

[JeanMarieGRALL]

Hello,

I am using spacy-llm to perform joint few-shot NER and few-shot Relation Extraction using OpenAI API (and GPT 3.5 turbo).
I have read the following sentence on the spacy-llm Readme:
"

Note that for efficient usage of resources, typically you would use nlp.pipe(docs) with a batch, instead of calling nlp(doc) with a single document.

"
I would like to know nlp.pipe(docs) groups the request to the API ? i.e. does it prompt the context once for all sentences in docs or is the context repeated for each sentence ?
Otherwise, is it possible to do so ?

Thank you in advance :)

[rmitsch]

Hi @JeanMarieGRALL!

I would like to know nlp.pipe(docs) groups the request to the API ?

In the case of GPT 3.5 no, because unfortunately the OpenAI API doesn't allow batching different prompts (it allows sending a conversation, but not disparate texts). Because of that we are forced to send individual requests. This applies to all models using the /chat/completions endpoint. All models targeting the /completions endpoint, which OpenAI considers legacy at this point, use batching and are therefore significantly more efficient.

See https://platform.openai.com/docs/models/model-endpoint-compatibility to check which models use which endpoint.

i.e. does it prompt the context once for all sentences in docs or is the context repeated for each sentence ?

This is always the case. Each doc is rendered into one prompt - how this is done exactly depends on the corresponding prompt template. The point of batching in this context is to include multiple prompts/docs into one API request, as a single API request has always a certain amount of overhead.