T
tokenizers
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
★11.0kstars
Rust
Apache-2.0
Updated: Today
📋 Project at a Glance
Tap to expand
What's this?A bert/gpt tool in the Data & Infrastructure category, built with Rust, open-source
Who made it?Maintained by huggingface team, 11K⭐ on GitHub, #172 out of 3133 in Data & Infrastructure
Why does it exist?With the growing demand for bert and gpt in Data & Infrastructure, huggingface created tokenizers as a streamlined solution.
What can it do?Key use cases: language-model, natural-language-processing, natural-language-understanding
How to install with AI?Use an AI coding assistant to run cargo build and handle the compilation setup from the README.
🔗 github.com/huggingface/tokenizers | 官网 https://huggingface.co/docs/tokenizers
🔗 github.com/huggingface/tokenizers | 官网 https://huggingface.co/docs/tokenizers
Topics
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers