S
sentencepiece
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
★12.0kstars
C++
Apache-2.0
Updated: Today
📋 Project at a Glance
Tap to expand
What's this?A natural-language-processing/neural-machine-translation tool in the Data & Infrastructure category, built with C++, open-source
Who made it?Maintained by google team, 12K⭐ on GitHub, #155 out of 3133 in Data & Infrastructure
Why does it exist?As the Data & Infrastructure landscape evolved, the google team identified the need for better natural-language-processing solutions. sentencepiece was created to simplify neural-machine-translation workflows.
What can it do?Key use cases: word-segmentation
How to install with AI?Use an AI coding assistant to follow the README and automatically handle the install and environment setup.
🔗 github.com/google/sentencepiece
🔗 github.com/google/sentencepiece
Topics
natural-language-processingneural-machine-translationword-segmentation