123 results found Sort:

theseer
tokenizer theseer
22
5.2k
other
8
A small library for converting tokenized PHP source code into XML (and potentially other formats)
php xml tokenizer
Created 2017-04-05
75 commits to master branch, last one about a year ago
Chevrotain
chevrotain Chevrotain
211
2.6k
apache-2.0
29
Parser Building Toolkit for JavaScript
lexer parsing grammars tokenizer javascript typescript open-source parser-library
Created 2015-04-15
2,345 commits to master branch, last one 11 days ago
roshan-research
hazm roshan-research
184
1.3k
mit
23
Persian NLP Toolkit
nlp farsi python persian tokenizer embeddings persian-nlp pos-tagging lemmatization normalization text-processing dependency-parser natural-language-processing
Created 2013-10-29
1,411 commits to master branch, last one 9 months ago
natasha
natasha natasha
109
1.2k
mit
57
Solves basic Russian NLP tasks, API for lower level Natasha projects
ner nlp python syntax russian tokenizer embeddings morphology visualization sentence-segmentation
Created 2016-08-03
413 commits to master branch, last one 5 months ago
dqbd
tiktokenizer dqbd
125
1.1k
mit
10
Online playground for OpenAPI tokenizers
nextjs openai chatgpt t3-stack tiktoken tokenizer
Created 2023-03-02
57 commits to master branch, last one about a month ago
lovit
soynlp lovit
185
958
other
41
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
nlp tokenizer korean-nlp postagging word-extraction korean-text-processing
Created 2017-05-13
766 commits to master branch, last one 4 years ago
ikawaha
kagome ikawaha
56
853
mit
22
Self-contained Japanese Morphological Analyzer written in pure Go
korean japanese tokenizer nlp-library pos-tagging segmentation hacktoberfest japanese-language morphological-analysis
Created 2014-06-26
816 commits to v2 branch, last one a day ago
no-context
moo no-context
68
843
bsd-3-clause
11
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
lexer regexp tokenizer javascript
Created 2017-03-10
363 commits to main branch, last one about a year ago
BLKSerene
Wordless BLKSerene
93
714
gpl-3.0
26
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus tagger stopword tokenizer lemmatizer literature linguistics translation corpus-tools corpus-search corpus-analysis corpus-processing corpus-statistics dependency-parser corpus-linguistics
Created 2018-08-23
1,499 commits to main branch, last one 19 hours ago
cbaziotis
ekphrasis cbaziotis
92
666
mit
18
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...
nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction
Created 2017-02-07
77 commits to master branch, last one 2 years ago
wangfenjin
simple wangfenjin
95
664
other
6
支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
fts fts5 cpp14 pinyin sqlite chinese sqlite3 tokenizer sqlite3-fts5
Created 2020-03-03
165 commits to master branch, last one about a month ago
risesoft-y9
Data-Labeling risesoft-y9
93
664
gpl-3.0
68
数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。
java vue3 nacos docker chinese tokenizer springboot2 elasticsearch data-annotations tokenizer-parser data-annotation-tools
Created 2024-08-26
30 commits to main branch, last one 2 months ago
open-korean-text
open-korean-text open-korean-text
98
624
apache-2.0
48
Open Korean Text Processor - An Open-source Korean Text Processor
korean tokenizer text-processing korean-tokenizer korean-text-processing natural-language-processing
Created 2017-01-24
799 commits to master branch, last one about a year ago
jflex-de
jflex jflex-de
117
602
other
22
The fast scanner generator for Java™ with full Unicode support
cup dfa nfa flex java yacc lexer regexp grammar parsing scanner tokenizer bazel-rules maven-plugin lexer-generator dfa-minimization lexical-analyzer scanner-generator
Created 2015-02-15
2,110 commits to master branch, last one 2 months ago
alasdairforsythe
tokenmonster alasdairforsythe
20
575
mit
11
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator
Created 2023-05-12
196 commits to main branch, last one about a year ago
therealoliver
Deepdive-llama3-from-scratch therealoliver
42
559
mit
4
Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.
gpt rms llms mask rope llama swiglu kv-cache rms-norm attention inference residuals tokenizer transformer language-model llm-configuration attention-mechanism positional-encoding multi-head-attention rotary-position-encoding
Created 2025-02-19
9 commits to main branch, last one about a month ago
glayzzle
php-parser glayzzle
72
542
bsd-3-clause
18
:herb: NodeJS PHP Parser - extract AST or tokens
ast php lexer parser php-ast tokenizer javascript php-parser development static-code-analysis
Created 2014-12-07
1,844 commits to main branch, last one 2 months ago
niieani
gpt-tokenizer niieani
38
541
mit
4
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 gpt-4 gpt-4o gpt-o1 openai decoder encoder tokenizer machine-learning
Created 2023-03-22
118 commits to main branch, last one 23 days ago
lydell
js-tokens lydell
34
517
mit
6
Tiny JavaScript tokenizer.
regex tokenizer ecmascript javascript
Created 2014-03-08
153 commits to main branch, last one about a month ago
lionsoul2014
friso lionsoul2014
91
496
apache-2.0
32
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other...
c tokenizer cjk-tokenizer php-tokenizer full-text-search korean-tokenizer chinese-tokenizer japanese-tokenizer chinese-word-segmentation
Created 2014-03-31
148 commits to master branch, last one about a year ago
hplt-project
sacremoses hplt-project
59
492
mit
11
Python port of Moses tokenizer, truecaser and normalizer
nlp tokenizer machine-translation
Created 2018-04-20
374 commits to master branch, last one about a year ago
leodevbro
vscode-blockman leodevbro
17
475
mit
7
VSCode extension to highlight nested code blocks
ast parser tokenizer vscode-api indentation vscode-blockman highlight-blocks vscode-extension abstract-syntax-tree
Created 2021-05-12
201 commits to main branch, last one 5 months ago
CogComp
cogcomp-nlp CogComp
144
475
other
61
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...
ner nlp pos cogcomp big-data tokenizer lemmatizer similarity data-mining pos-tagging lemmatization transliteration dependency-parsing relation-extraction parts-of-speech-tagging named-entity-recognition natural-language-processing natural-language-understanding
Created 2015-10-18
2,621 commits to master branch, last one 2 years ago
neurosnap
sentences neurosnap
38
447
mit
13
A multilingual command line sentence tokenizer in Golang
cli sentences tokenizer sentence-tokenizer
Created 2015-08-07
223 commits to main branch, last one about a year ago
lindera
lindera lindera
41
441
mit
6
A multilingual morphological analysis library.
library analyzer tokenizer multilingual morphological
Created 2020-01-22
527 commits to main branch, last one a day ago
polm
fugashi polm
36
439
mit
7
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
nlp mecab japanese tokenizer cython-wrapper
Created 2019-10-14
284 commits to main branch, last one 2 months ago
timtadh
lexmachine timtadh
28
410
other
11
Lex machinary for go.
go dfa lex nfa lexer tokenizer regular-expression lexical-analysis-engines lexical-analysis-framework
Created 2014-04-28
157 commits to master branch, last one 2 years ago
taishi-i
nagisa taishi-i
23
396
mit
10
A Japanese tokenizer based on recurrent neural networks
nlp dynet japanese tokenizer nlp-library pos-tagging sequence-labeling word-segmentation natural-language-processing
Created 2018-02-14
194 commits to master branch, last one 9 months ago
ku-nlp
jumanpp ku-nlp
44
385
apache-2.0
31
Juman++ (a Morphological Analyzer Toolkit)
cjk nlp juman japanese tokenizer pos-tagger pos-tagging word-segmentation part-of-speech-tagger morphological-analyser morphological-analysis
Created 2016-10-11
1,093 commits to master branch, last one 2 years ago
daac-tools
vibrato daac-tools
15
353
apache-2.0
7
🎤 vibrato: Viterbi-based accelerated tokenizer
nlp rust japanese tokenizer segmentation tokenization morphological-analysis
Created 2022-07-06
176 commits to main branch, last one 9 days ago