adding new tokens to tokenizer without disturbing the base models embedding weight metrics of tokens

[riyajatar37003]

Hi,
Lets assume i have few completely new tokens which was never seen by tokenizer/model, now i just want to add these new tokens to tokenizer and update the weights of tokens that are newly added without touching other tokens weight matrics. How can i achieve this any guidance or sources for this with the help of sentence-transformers .train method?

thanks
@tomaarsen

[tomaarsen]

Hello!

I believe @kacperlukawski wrote on this exact topic in his blogpost about Word Injection: https://www.kacperlukawski.com/posts/word-injection/
I think it should cover exactly what you're looking for.

• Tom Aarsen

[riyajatar37003]

bge reranker is not direclt supported in sentence transformer .
i tried loading with cross encoder but the scores are different compare to auto class of hf.
@tomaarsen

[tomaarsen]

See #3117 (comment)