NLPOptimize / flash-tokenizer

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

Date Created 2025-03-09 (26 days ago)

Commits 79 (last one a day ago)

Stargazers 189 (147 this week)

Watchers 2 (0 this week)

Forks 1

License unknown

Ranking

RepositoryStats indexes 635,117 repositories, of these NLPOptimize/flash-tokenizer is ranked #196,498 (69th percentile) for total stargazers, and #483,325 for total watchers. Github reports the primary language for this repository as C++, for repositories using this language it is ranked #11,089/33,893.

NLPOptimize/flash-tokenizer is also tagged with popular topics, for these it's ranked: python (#9,247/23506), deep-learning (#3,755/8831), cpp (#1,471/3642), nlp (#1,052/2529), bert (#280/579), huggingface (#152/418)

Other Information

There have been 3 releases, the latest one was published on 2025-03-10 (25 days ago) with the name dataset.

Homepage URL: https://github.com/NLPOptimize/flash-tokenizer

All Topics

cpp nlp bert trie cpp17 flash python pybind11 tokenizer wordpiece huggingface berttokenizer deep-learning wordpiece-tokenization

Star History

Github stargazers over time

Watcher History

Github watchers over time, collection started in '23

Recent Commit History

79 commits on the default branch (main) since jan '22

Yearly Commits

Commits to the default branch (main) per year

Issue History

No issues have been posted

Languages

The primary language is C++ but there's also others...

updated: 2025-04-03 @ 04:43pm, id: 945231616 / R_kgDOOFcXAA