Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Add token counting #47

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

Feat: Add token counting #47

wants to merge 19 commits into from

Conversation

Angelchev
Copy link
Member

@Angelchev Angelchev commented Jan 3, 2024

Closes: #41

Overview

  • Add command to count tokens from a visual selection.
  • Add python script to count tokens with OpenAI's tiktoken.
  • Add utils.vim to launch python scripts more easily.
  • Add automatic venv setup for Neovim with lazy.nvim package manager via build.lua.
    • Create a python virtual environment automatically for Linux or Windows (Untested).
    • Install dependencies from requirements.txt automatically.
  • Add Neovim animated notification integration with nvim-notify.

Todo

  • Add python test for new script
  • Check for tiktoken dependency in Python script, and throw error if not present.
  • Investigate automatically setting up python venv and installing requirements for vim 8.
  • Update README.md to document optionally setting up python venv.
  • Update vimdocs:
    • New command and availability dependant on tiktoken
    • Integration of nvim-notify

Testing

  • Test automatic setup for Neovim with lazy.nvim.
    • Linux
    • Mac OSX
    • Windows

Screenshots

Neovim Animated Notifications via nvim-notify

Screenshot_20240103_233935

Automatic venv setup for Neovim with lazy.nvim

Screenshot_20240103_234057

@Angelchev Angelchev added documentation Improvements or additions to documentation enhancement New feature or request labels Jan 3, 2024
@Angelchev Angelchev self-assigned this Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feat: Add tokens counter
1 participant