History

浪子 48fea5d844 增加AI摘要		2024-05-09 16:35:20 +08:00
..
dist	增加AI摘要	2024-05-09 16:35:20 +08:00
README.md	增加AI摘要	2024-05-09 16:35:20 +08:00
index.d.ts	增加AI摘要	2024-05-09 16:35:20 +08:00
index.js	增加AI摘要	2024-05-09 16:35:20 +08:00
lite.d.ts	增加AI摘要	2024-05-09 16:35:20 +08:00
lite.js	增加AI摘要	2024-05-09 16:35:20 +08:00
package.json	增加AI摘要	2024-05-09 16:35:20 +08:00

README.md

⏳ js-tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models. This is a pure JS port of the original tiktoken library.

Install the library from NPM:

npm install js-tiktoken

Usage

Basic usage follows, which includes all the OpenAI encoders and ranks:

import assert from "node:assert";
import { getEncoding, encodingForModel } from "js-tiktoken";

const enc = getEncoding("gpt2");
assert(enc.decode(enc.encode("hello world")) === "hello world");