97 results found Sort:

theseer
tokenizer theseer
23
5.1k
other
9
A small library for converting tokenized PHP source code into XML (and potentially other formats)
php xml tokenizer
Created 2017-04-05
75 commits to master branch, last one 2 months ago
Chevrotain
chevrotain Chevrotain
200
2.4k
apache-2.0
31
Parser Building Toolkit for JavaScript
lexer parsing grammars tokenizer javascript typescript open-source parser-library
Created 2015-04-15
2,320 commits to master branch, last one 9 months ago
natasha
natasha natasha
108
1.2k
mit
59
Solves basic Russian NLP tasks, API for lower level Natasha projects
ner nlp python syntax russian tokenizer embeddings morphology visualization sentence-segmentation
Created 2016-08-03
409 commits to master branch, last one 2 months ago
roshan-research
hazm roshan-research
179
1.1k
mit
23
Persian NLP Toolkit
nlp farsi python persian tokenizer embeddings persian-nlp pos-tagging lemmatization normalization text-processing dependency-parser natural-language-processing
Created 2013-10-29
1,406 commits to master branch, last one 29 days ago
lovit
soynlp lovit
181
907
other
44
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
nlp tokenizer korean-nlp postagging word-extraction korean-text-processing
Created 2017-05-13
766 commits to master branch, last one 3 years ago
no-context
moo no-context
64
807
bsd-3-clause
11
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
lexer regexp tokenizer javascript
Created 2017-03-10
363 commits to main branch, last one about a year ago
ikawaha
kagome ikawaha
53
792
mit
23
Self-contained Japanese Morphological Analyzer written in pure Go
korean japanese tokenizer nlp-library pos-tagging segmentation hacktoberfest japanese-language morphological-analysis
Created 2014-06-26
785 commits to v2 branch, last one 24 days ago
BLKSerene
Wordless BLKSerene
88
673
gpl-3.0
29
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus tagger stopword tokenizer lemmatizer literature linguistics translation corpus-tools corpus-search corpus-analysis corpus-processing corpus-statistics dependency-parser corpus-linguistics
Created 2018-08-23
1,397 commits to main branch, last one 7 days ago
cbaziotis
ekphrasis cbaziotis
92
660
mit
18
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...
nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction
Created 2017-02-07
77 commits to master branch, last one about a year ago
open-korean-text
open-korean-text open-korean-text
94
600
apache-2.0
53
Open Korean Text Processor - An Open-source Korean Text Processor
korean tokenizer text-processing korean-tokenizer korean-text-processing natural-language-processing
Created 2017-01-24
799 commits to master branch, last one 2 months ago
dqbd
tiktokenizer dqbd
69
591
mit
4
Online playground for OpenAPI tokenizers
nextjs openai chatgpt t3-stack tiktoken tokenizer
Created 2023-03-02
54 commits to master branch, last one 18 days ago
jflex-de
jflex jflex-de
111
576
other
23
The fast scanner generator for Java™ with full Unicode support
cup dfa nfa flex java yacc lexer regexp grammar parsing scanner tokenizer bazel-rules maven-plugin lexer-generator dfa-minimization lexical-analyzer scanner-generator
Created 2015-02-15
2,109 commits to master branch, last one about a year ago
glayzzle
php-parser glayzzle
68
519
bsd-3-clause
19
:herb: NodeJS PHP Parser - extract AST or tokens
ast php lexer parser php-ast tokenizer javascript php-parser development static-code-analysis
Created 2014-12-07
1,835 commits to main branch, last one 8 months ago
alasdairforsythe
tokenmonster alasdairforsythe
20
515
mit
10
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator
Created 2023-05-12
196 commits to main branch, last one 6 months ago
wangfenjin
simple wangfenjin
71
503
mit
6
支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
fts fts5 cpp14 pinyin sqlite chinese sqlite3 tokenizer sqlite3-fts5
Created 2020-03-03
155 commits to master branch, last one about a month ago
lydell
js-tokens lydell
30
481
mit
7
Tiny JavaScript tokenizer.
regex tokenizer ecmascript javascript
Created 2014-03-08
143 commits to main branch, last one 18 days ago
hplt-project
sacremoses hplt-project
59
479
mit
13
Python port of Moses tokenizer, truecaser and normalizer
nlp tokenizer machine-translation
Created 2018-04-20
374 commits to master branch, last one 7 months ago
lionsoul2014
friso lionsoul2014
94
473
apache-2.0
34
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other...
c tokenizer cjk-tokenizer php-tokenizer full-text-search korean-tokenizer chinese-tokenizer japanese-tokenizer chinese-word-segmentation
Created 2014-03-31
148 commits to master branch, last one 8 months ago
CogComp
cogcomp-nlp CogComp
144
469
other
63
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...
ner nlp pos cogcomp big-data tokenizer lemmatizer similarity data-mining pos-tagging lemmatization transliteration dependency-parsing relation-extraction parts-of-speech-tagging named-entity-recognition natural-language-processing natural-language-understanding
Created 2015-10-18
2,621 commits to master branch, last one about a year ago
neurosnap
sentences neurosnap
38
424
mit
15
A multilingual command line sentence tokenizer in Golang
cli sentences tokenizer sentence-tokenizer
Created 2015-08-07
223 commits to main branch, last one 3 months ago
timtadh
lexmachine timtadh
31
402
other
13
Lex machinary for go.
go dfa lex nfa lexer tokenizer regular-expression lexical-analysis-engines lexical-analysis-framework
Created 2014-04-28
157 commits to master branch, last one about a year ago
niieani
gpt-tokenizer niieani
31
394
mit
5
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 gpt-4 openai decoder encoder tokenizer machine-learning
Created 2023-03-22
67 commits to main branch, last one 7 months ago
taishi-i
nagisa taishi-i
22
376
mit
12
A Japanese tokenizer based on recurrent neural networks
nlp dynet japanese tokenizer nlp-library pos-tagging sequence-labeling word-segmentation natural-language-processing
Created 2018-02-14
192 commits to master branch, last one 4 months ago
polm
fugashi polm
31
372
mit
7
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
nlp mecab japanese tokenizer cython-wrapper
Created 2019-10-14
274 commits to master branch, last one about a month ago
ku-nlp
jumanpp ku-nlp
44
369
apache-2.0
31
Juman++ (a Morphological Analyzer Toolkit)
cjk nlp juman japanese tokenizer pos-tagger pos-tagging word-segmentation part-of-speech-tagger morphological-analyser morphological-analysis
Created 2016-10-11
1,093 commits to master branch, last one about a year ago
lindera-morphology
lindera lindera-morphology
36
359
mit
6
A multilingual morphological analysis library.
library analyzer tokenizer multilingual morphological
Created 2020-01-22
441 commits to main branch, last one 4 days ago
leodevbro
vscode-blockman leodevbro
16
344
mit
6
VSCode extension to highlight nested code blocks
ast parser tokenizer vscode-api indentation vscode-blockman highlight-blocks vscode-extension abstract-syntax-tree
Created 2021-05-12
200 commits to main branch, last one about a month ago
belladoreai
llama-tokenizer-js belladoreai
21
313
mit
3
JS tokenizer for LLaMA 1 and 2
llm llama tokenizer javascript
Created 2023-06-11
36 commits to master branch, last one about a month ago
daac-tools
vibrato daac-tools
14
303
apache-2.0
7
🎤 vibrato: Viterbi-based accelerated tokenizer
nlp rust japanese tokenizer segmentation tokenization morphological-analysis
Created 2022-07-06
171 commits to main branch, last one a day ago
bitextor
bitextor bitextor
43
283
gpl-3.0
30
Bitextor generates translation memories from multilingual websites
tmx warc wget crawler apertium bitextor hunalign bicleaner bleualign tokenizer corpus-tools dictionaries corpus-generator document-aligner parallel-corpora corpus-processing machine-translation sentence-segmentation neural-machine-translation statistical-machine-translation
Created 2018-04-16
3,994 commits to master branch, last one 8 months ago