C
Curator
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
★1.7kstars
Python
Apache-2.0
Updated: Today
📋 Project at a Glance
Tap to expand
What's this?A data/data-curation tool in the Data & Infrastructure category, built with Python, open-source
Who made it?Maintained by NVIDIA-NeMo team, 1.7K⭐ on GitHub, #816 out of 3133 in Data & Infrastructure
Why does it exist?The NVIDIA-NeMo team recognized that existing data tools in Data & Infrastructure were hard to use. Curator was designed to make data-curation more accessible.
What can it do?Key use cases: data-prep, data-preparation, data-processing
How to install with AI?Use an AI coding assistant (Claude Code, Cursor, Copilot) to automatically set up pip dependencies and virtual env. Follow the README — the AI handles the rest.
🔗 github.com/NVIDIA-NeMo/Curator
🔗 github.com/NVIDIA-NeMo/Curator
Topics
datadata-curationdata-prepdata-preparationdata-processingdata-processing-pipelinesdata-qualitydatacurationdatarecipesdeduplicationfast-data-processingfine-tuninglarge-language-modelslarge-scale-data-processingllmllm-data-qualityllmappspythonsemantic-deduplication