2 results found Sort:

NVIDIA
NeMo-Curator NVIDIA
102
769
apache-2.0
14
Scalable data pre processing and curation toolkit for LLMs
llm data python llmapps data-prep datarecipes fine-tuning data-quality datacuration data-curation deduplication data-processing data-preparation llm-data-quality fast-data-processing large-language-models semantic-deduplication data-processing-pipelines large-scale-data-processing
Created 2024-03-14
255 commits to main branch, last one 2 days ago
MinishLab
semhash MinishLab
20
481
mit
3
Fast Semantic Text Deduplication
datasets vicinity model2vec deduplication preprocessing semantic-deduplication
Created 2024-10-13
39 commits to main branch, last one 7 days ago