111 results found Sort:

theseer
tokenizer theseer
21
5.2k
other
9
A small library for converting tokenized PHP source code into XML (and potentially other formats)
php xml tokenizer
Created 2017-04-05
75 commits to master branch, last one 7 months ago
Chevrotain
chevrotain Chevrotain
204
2.5k
apache-2.0
30
Parser Building Toolkit for JavaScript
lexer parsing grammars tokenizer javascript typescript open-source parser-library
Created 2015-04-15
2,340 commits to master branch, last one 18 days ago
natasha
natasha natasha
110
1.2k
mit
58
Solves basic Russian NLP tasks, API for lower level Natasha projects
ner nlp python syntax russian tokenizer embeddings morphology visualization sentence-segmentation
Created 2016-08-03
413 commits to master branch, last one 20 days ago
roshan-research
hazm roshan-research
179
1.2k
mit
25
Persian NLP Toolkit
nlp farsi python persian tokenizer embeddings persian-nlp pos-tagging lemmatization normalization text-processing dependency-parser natural-language-processing
Created 2013-10-29
1,411 commits to master branch, last one 4 months ago
lovit
soynlp lovit
182
943
other
44
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
nlp tokenizer korean-nlp postagging word-extraction korean-text-processing
Created 2017-05-13
766 commits to master branch, last one 3 years ago
no-context
moo no-context
66
830
bsd-3-clause
12
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
lexer regexp tokenizer javascript
Created 2017-03-10
363 commits to main branch, last one about a year ago
ikawaha
kagome ikawaha
54
825
mit
23
Self-contained Japanese Morphological Analyzer written in pure Go
korean japanese tokenizer nlp-library pos-tagging segmentation hacktoberfest japanese-language morphological-analysis
Created 2014-06-26
806 commits to v2 branch, last one 2 months ago
dqbd
tiktokenizer dqbd
87
702
mit
6
Online playground for OpenAPI tokenizers
nextjs openai chatgpt t3-stack tiktoken tokenizer
Created 2023-03-02
54 commits to master branch, last one 5 months ago
BLKSerene
Wordless BLKSerene
92
695
gpl-3.0
28
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus tagger stopword tokenizer lemmatizer literature linguistics translation corpus-tools corpus-search corpus-analysis corpus-processing corpus-statistics dependency-parser corpus-linguistics
Created 2018-08-23
1,450 commits to main branch, last one a day ago
cbaziotis
ekphrasis cbaziotis
90
661
mit
18
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...
nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction
Created 2017-02-07
77 commits to master branch, last one 2 years ago
open-korean-text
open-korean-text open-korean-text
97
613
apache-2.0
53
Open Korean Text Processor - An Open-source Korean Text Processor
korean tokenizer text-processing korean-tokenizer korean-text-processing natural-language-processing
Created 2017-01-24
799 commits to master branch, last one 7 months ago
wangfenjin
simple wangfenjin
83
600
mit
6
支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
fts fts5 cpp14 pinyin sqlite chinese sqlite3 tokenizer sqlite3-fts5
Created 2020-03-03
157 commits to master branch, last one 20 days ago
jflex-de
jflex jflex-de
114
586
other
23
The fast scanner generator for Java™ with full Unicode support
cup dfa nfa flex java yacc lexer regexp grammar parsing scanner tokenizer bazel-rules maven-plugin lexer-generator dfa-minimization lexical-analyzer scanner-generator
Created 2015-02-15
2,109 commits to master branch, last one about a year ago
alasdairforsythe
tokenmonster alasdairforsythe
20
550
mit
10
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator
Created 2023-05-12
196 commits to main branch, last one 11 months ago
glayzzle
php-parser glayzzle
71
533
bsd-3-clause
19
:herb: NodeJS PHP Parser - extract AST or tokens
ast php lexer parser php-ast tokenizer javascript php-parser development static-code-analysis
Created 2014-12-07
1,835 commits to main branch, last one about a year ago
lydell
js-tokens lydell
31
503
mit
7
Tiny JavaScript tokenizer.
regex tokenizer ecmascript javascript
Created 2014-03-08
148 commits to main branch, last one about a month ago
hplt-project
sacremoses hplt-project
57
487
mit
12
Python port of Moses tokenizer, truecaser and normalizer
nlp tokenizer machine-translation
Created 2018-04-20
374 commits to master branch, last one about a year ago
lionsoul2014
friso lionsoul2014
93
482
apache-2.0
33
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other...
c tokenizer cjk-tokenizer php-tokenizer full-text-search korean-tokenizer chinese-tokenizer japanese-tokenizer chinese-word-segmentation
Created 2014-03-31
148 commits to master branch, last one about a year ago
CogComp
cogcomp-nlp CogComp
142
471
other
62
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...
ner nlp pos cogcomp big-data tokenizer lemmatizer similarity data-mining pos-tagging lemmatization transliteration dependency-parsing relation-extraction parts-of-speech-tagging named-entity-recognition natural-language-processing natural-language-understanding
Created 2015-10-18
2,621 commits to master branch, last one 2 years ago
leodevbro
vscode-blockman leodevbro
17
465
mit
9
VSCode extension to highlight nested code blocks
ast parser tokenizer vscode-api indentation vscode-blockman highlight-blocks vscode-extension abstract-syntax-tree
Created 2021-05-12
201 commits to main branch, last one about a month ago
niieani
gpt-tokenizer niieani
37
449
mit
5
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 gpt-4 gpt-4o openai decoder encoder tokenizer machine-learning
Created 2023-03-22
102 commits to main branch, last one 2 days ago
neurosnap
sentences neurosnap
38
437
mit
14
A multilingual command line sentence tokenizer in Golang
cli sentences tokenizer sentence-tokenizer
Created 2015-08-07
223 commits to main branch, last one 8 months ago
timtadh
lexmachine timtadh
28
406
other
13
Lex machinary for go.
go dfa lex nfa lexer tokenizer regular-expression lexical-analysis-engines lexical-analysis-framework
Created 2014-04-28
157 commits to master branch, last one 2 years ago
polm
fugashi polm
33
398
mit
8
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
nlp mecab japanese tokenizer cython-wrapper
Created 2019-10-14
282 commits to master branch, last one 6 days ago
taishi-i
nagisa taishi-i
22
386
mit
12
A Japanese tokenizer based on recurrent neural networks
nlp dynet japanese tokenizer nlp-library pos-tagging sequence-labeling word-segmentation natural-language-processing
Created 2018-02-14
194 commits to master branch, last one 4 months ago
lindera
lindera lindera
37
385
mit
6
A multilingual morphological analysis library.
library analyzer tokenizer multilingual morphological
Created 2020-01-22
496 commits to main branch, last one 2 days ago
ku-nlp
jumanpp ku-nlp
44
379
apache-2.0
31
Juman++ (a Morphological Analyzer Toolkit)
cjk nlp juman japanese tokenizer pos-tagger pos-tagging word-segmentation part-of-speech-tagger morphological-analyser morphological-analysis
Created 2016-10-11
1,093 commits to master branch, last one about a year ago
belladoreai
llama-tokenizer-js belladoreai
22
336
mit
4
JS tokenizer for LLaMA 1 and 2
llm llama tokenizer javascript
Created 2023-06-11
38 commits to master branch, last one 4 months ago
daac-tools
vibrato daac-tools
14
333
apache-2.0
7
🎤 vibrato: Viterbi-based accelerated tokenizer
nlp rust japanese tokenizer segmentation tokenization morphological-analysis
Created 2022-07-06
172 commits to main branch, last one about a month ago
guillaume-be
rust-tokenizers guillaume-be
26
295
apache-2.0
8
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
rust-lang tokenizer transformer deep-learning
Created 2019-11-09
442 commits to main branch, last one about a year ago