2 results found Sort:

alasdairforsythe
tokenmonster alasdairforsythe
21
551
mit
10
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator
Created 2023-05-12
196 commits to main branch, last one about a year ago
bzick
tokenizer bzick
6
100
mit
2
Tokenizer (lexer) for golang
lexer parse golang parser tokenizer tokenizing
Created 2021-10-19
44 commits to master branch, last one 15 days ago