Repository profile
馃挜 Fast State-of-the-Art Tokenizers optimized for Research and Production
Repository updates
Get generated huggingface/tokenizers development summaries by email, or follow the weekly and monthly RSS feeds.
Sign in to subscribe by email. RSS feeds are public.
Sign in to subscribeTracked growth, recent movement, and commit velocity from stored repository snapshots.
Latest capture 2026-07-15 03:13
5 captures since 2026-05-25
Stars from baseline +121
All tracked data
Frameworks, package managers, ecosystems, and dependency manifests found during catalog scans.
Scanned 2026-07-15 03:13
tokenizers/Cargo.toml
rust ecosystem,
33 dependencies
bindings/node/Cargo.toml
rust ecosystem,
6 dependencies
bindings/node/package.json
javascript ecosystem,
21 dependencies
bindings/python/Cargo.toml
rust ecosystem,
17 dependencies
bindings/python/pyproject.toml
python ecosystem,
13 dependencies
bindings/python/setup.cfg
python ecosystem,
0 dependencies
bindings/node/yarn.lock
javascript ecosystem,
688 dependencies
tokenizers/examples/unstable_wasm/Cargo.toml
rust ecosystem,
5 dependencies
Searchable topics, generated tags, and stack labels that explain where this repository fits.
Agent instructions and tool configuration paths found in the repository tree.
Nearest indexed repositories by embedding similarity.
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Models, data loaders and abstractions for language processing, powered by PyTorch
State-of-the-Art Embeddings, Retrieval, and Reranking
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
馃 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
馃 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
