11 results found Sort:

rsennrich
subword-nmt rsennrich
464
2.2k
mit
55
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
bpe nmt segmentation subword-units machine-translation neural-machine-translation
Created 2015-09-01
144 commits to master branch, last one 4 months ago
VKCOM
YouTokenToMe VKCOM
103
960
mit
26
Unsupervised text tokenizer focused on computational efficiency
bpe nlp tokenization word-segmentation natural-language-processing
This repository has been archived (exclude archived)
Created 2019-06-06
85 commits to master branch, last one about a year ago
niieani
gpt-tokenizer niieani
38
479
mit
5
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 gpt-4 gpt-4o gpt-o1 openai decoder encoder tokenizer machine-learning
Created 2023-03-22
115 commits to main branch, last one 12 days ago
OpenNMT
Tokenizer OpenNMT
70
291
mit
19
Fast and customizable text tokenization library with BPE and SentencePiece support
bpe cpp icu python unicode tokenizer tokenization sentencepiece machine-translation natural-language-processing
Created 2017-02-14
601 commits to master branch, last one about a year ago
zurawiki
tiktoken-rs zurawiki
49
268
mit
5
Ready-made tokenizer library for working with GPT and tiktoken
bpe rust openai tokenizer
Created 2023-02-02
156 commits to main branch, last one about a month ago
soaxelbrooke
python-bpe soaxelbrooke
38
224
mit
6
Byte Pair Encoding for Python!
bpe nlp python
Created 2017-09-22
89 commits to master branch, last one 2 years ago
akretion
nfelib akretion
59
148
mit
26
nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e
bpe cte nfe mdfe nfse sped brasil python nota-fiscal-eletronica
Created 2017-09-18
242 commits to master branch, last one 5 months ago
gautierdag
bpeasy gautierdag
3
142
mit
2
Fast bare-bones BPE for modern tokenizer training
bpe tokenizer tokenization
Created 2023-11-09
52 commits to main branch, last one 2 months ago
samber
go-gpt-3-encoder samber
20
79
mit
3
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
go bpe codex gpt-2 gpt-3 token openai decoder encoder tokenizer transformer byte-pair-encoding
Created 2022-12-21
14 commits to main branch, last one 19 days ago
zouharvi
tokenization-scorer zouharvi
4
33
mit
1
Simple-to-use scoring function for arbitrarily tokenized texts.
bpe subword segmentation tokenization
Created 2023-04-29
26 commits to main branch, last one 29 days ago
aespinilla
GPT3-Tokenizer aespinilla
4
30
mit
2
GPT3 encoder & decoder tool written in Swift
bpe gpt3 apple swift openai chatgpt decoder encoder chatgpt3 tokenizer openai-api chatgpt-api encoder-decoder swift-package-manager
Created 2023-02-08
13 commits to master branch, last one about a year ago