11 results found Sort:

rsennrich
subword-nmt rsennrich
465
2.2k
mit
54
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
bpe nmt segmentation subword-units machine-translation neural-machine-translation
Created 2015-09-01
144 commits to master branch, last one about a month ago
VKCOM
YouTokenToMe VKCOM
101
953
mit
26
Unsupervised text tokenizer focused on computational efficiency
bpe nlp tokenization word-segmentation natural-language-processing
This repository has been archived (exclude archived)
Created 2019-06-06
85 commits to master branch, last one about a year ago
niieani
gpt-tokenizer niieani
35
433
mit
4
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 gpt-4 gpt-4o openai decoder encoder tokenizer machine-learning
Created 2023-03-22
93 commits to main branch, last one 7 days ago
OpenNMT
Tokenizer OpenNMT
70
277
mit
19
Fast and customizable text tokenization library with BPE and SentencePiece support
bpe cpp icu python unicode tokenizer tokenization sentencepiece machine-translation natural-language-processing
Created 2017-02-14
601 commits to master branch, last one about a year ago
zurawiki
tiktoken-rs zurawiki
46
241
mit
5
Ready-made tokenizer library for working with GPT and tiktoken
bpe rust openai tokenizer
Created 2023-02-02
140 commits to main branch, last one 4 months ago
soaxelbrooke
python-bpe soaxelbrooke
38
223
mit
6
Byte Pair Encoding for Python!
bpe nlp python
Created 2017-09-22
89 commits to master branch, last one 2 years ago
akretion
nfelib akretion
58
144
mit
26
nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e
bpe cte nfe mdfe nfse sped brasil python nota-fiscal-eletronica
Created 2017-09-18
242 commits to master branch, last one 2 months ago
gautierdag
bpeasy gautierdag
2
138
mit
2
Fast bare-bones BPE for modern tokenizer training
bpe tokenizer tokenization
Created 2023-11-09
51 commits to main branch, last one about a month ago
samber
go-gpt-3-encoder samber
20
78
mit
3
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
go bpe codex gpt-2 gpt-3 token openai decoder encoder tokenizer transformer byte-pair-encoding
Created 2022-12-21
13 commits to main branch, last one 11 months ago
aespinilla
GPT3-Tokenizer aespinilla
4
30
mit
2
GPT3 encoder & decoder tool written in Swift
bpe gpt3 apple swift openai chatgpt decoder encoder chatgpt3 tokenizer openai-api chatgpt-api encoder-decoder swift-package-manager
Created 2023-02-08
13 commits to master branch, last one about a year ago
zouharvi
tokenization-scorer zouharvi
3
28
mit
1
Simple-to-use scoring function for arbitrarily tokenized texts.
bpe subword segmentation tokenization
Created 2023-04-29
21 commits to main branch, last one 17 days ago