openai/tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
18377 stars
Python
1 awesome list
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- Stars
- 10,768
- Forks
- 1,107
- Commits
- 1991
- Language
- Rust
- Awesome lists
- 3
Similar repositories
pytorch/text
Models, data loaders and abstractions for language processing, powered by PyTorch
3560 stars
Python
1 awesome list
huggingface/sentence-transformers
State-of-the-Art Embeddings, Retrieval, and Reranking
18723 stars
Python
1 awesome list
botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
171 stars
TypeScript
1 awesome list
huggingface/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
160951 stars
Python
5 awesome lists
huggingface/peft
🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
21184 stars
Python
1 awesome list
Tracked growth
1 capture since 2026-05-25
Latest capture 2026-05-25 21:00
Stars history
Total starsCommits history
Default branch commitsMetadata
- Created: 2019-11-01
- First commit: —
- Last pushed: 2026-05-22
- Website: https://huggingface.co/docs/tokenizers
- Archived: no
- Stack detected: —
- License: Apache-2.0
AI development signals
No AI development config files detected.