2 results found Sort:

NVIDIA
NeMo-Curator NVIDIA
83
615
apache-2.0
15
Scalable data pre processing and curation toolkit for LLMs
llm data python llmapps data-prep datarecipes fine-tuning data-quality datacuration data-curation deduplication data-processing data-preparation llm-data-quality fast-data-processing large-language-models semantic-deduplication data-processing-pipelines large-scale-data-processing
Created 2024-03-14
202 commits to main branch, last one a day ago
IBM
data-prep-kit IBM
132
302
apache-2.0
16
Open source project for data preparation of LLM application builders
llm ray data spark python llmapps malware data-prep finetuning datarecipes code-quality datacuration deduplication data-preparation data-preprocessing large-language-models large-scale-data-processing data-preprocessing-pipelines
Created 2024-04-08
3,862 commits to dev branch, last one 2 days ago