Search Results - RepositoryStats

tokenizer theseer

21

5.2k

other

9

A small library for converting tokenized PHP source code into XML (and potentially other formats)

php xml tokenizer

Created 2017-04-05

75 commits to master branch, last one 8 months ago

chevrotain Chevrotain

205

2.5k

apache-2.0

30

Parser Building Toolkit for JavaScript

lexer parsing grammars tokenizer javascript typescript open-source parser-library

Created 2015-04-15

2,340 commits to master branch, last one about a month ago

hazm roshan-research

179

1.2k

mit

25

Persian NLP Toolkit

nlp farsi python persian tokenizer embeddings persian-nlp pos-tagging lemmatization normalization text-processing dependency-parser natural-language-processing

Created 2013-10-29

1,411 commits to master branch, last one 5 months ago

natasha natasha

109

1.2k

mit

59

Solves basic Russian NLP tasks, API for lower level Natasha projects

ner nlp python syntax russian tokenizer embeddings morphology visualization sentence-segmentation

Created 2016-08-03

413 commits to master branch, last one about a month ago

soynlp lovit

183

944

other

44

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

nlp tokenizer korean-nlp postagging word-extraction korean-text-processing

Created 2017-05-13

766 commits to master branch, last one 3 years ago

moo no-context

66

832

bsd-3-clause

12

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

lexer regexp tokenizer javascript

Created 2017-03-10

363 commits to main branch, last one about a year ago

kagome ikawaha

54

827

mit

23

Self-contained Japanese Morphological Analyzer written in pure Go

korean japanese tokenizer nlp-library pos-tagging segmentation hacktoberfest japanese-language morphological-analysis

Created 2014-06-26

806 commits to v2 branch, last one 3 months ago

tiktokenizer dqbd

88

709

mit

6

Online playground for OpenAPI tokenizers

nextjs openai chatgpt t3-stack tiktoken tokenizer

Created 2023-03-02

54 commits to master branch, last one 6 months ago

Wordless BLKSerene

92

700

gpl-3.0

28

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

corpus tagger stopword tokenizer lemmatizer literature linguistics translation corpus-tools corpus-search corpus-analysis corpus-processing corpus-statistics dependency-parser corpus-linguistics

Created 2018-08-23

1,451 commits to main branch, last one 12 days ago

ekphrasis cbaziotis

90

661

mit

18

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction

Created 2017-02-07

77 commits to master branch, last one 2 years ago

open-korean-text open-korean-text

97

614

apache-2.0

52

Open Korean Text Processor - An Open-source Korean Text Processor

korean tokenizer text-processing korean-tokenizer korean-text-processing natural-language-processing

Created 2017-01-24

799 commits to master branch, last one 8 months ago

simple wangfenjin

86

610

mit

6

支持中文和拼音的 SQLite fts5 全文搜索扩展｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin

fts fts5 cpp14 pinyin sqlite chinese sqlite3 tokenizer sqlite3-fts5

Created 2020-03-03

157 commits to master branch, last one about a month ago

jflex jflex-de

114

587

other

23

The fast scanner generator for Java™ with full Unicode support

cup dfa nfa flex java yacc lexer regexp grammar parsing scanner tokenizer bazel-rules maven-plugin lexer-generator dfa-minimization lexical-analyzer scanner-generator

Created 2015-02-15

2,109 commits to master branch, last one about a year ago

tokenmonster alasdairforsythe

21

551

mit

10

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator

Created 2023-05-12

196 commits to main branch, last one about a year ago

php-parser glayzzle

71

534

bsd-3-clause

19

:herb: NodeJS PHP Parser - extract AST or tokens

ast php lexer parser php-ast tokenizer javascript php-parser development static-code-analysis

Created 2014-12-07

1,840 commits to main branch, last one 10 days ago

js-tokens lydell

31

504

mit

7

Tiny JavaScript tokenizer.

regex tokenizer ecmascript javascript

Created 2014-03-08

148 commits to main branch, last one about a month ago

sacremoses hplt-project

57

488

mit

12

Python port of Moses tokenizer, truecaser and normalizer

nlp tokenizer machine-translation

Created 2018-04-20

374 commits to master branch, last one about a year ago

friso lionsoul2014

93

483

apache-2.0

33

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other...

c tokenizer cjk-tokenizer php-tokenizer full-text-search korean-tokenizer chinese-tokenizer japanese-tokenizer chinese-word-segmentation

Created 2014-03-31

148 commits to master branch, last one about a year ago

cogcomp-nlp CogComp

142

473

other

62

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...

ner nlp pos cogcomp big-data tokenizer lemmatizer similarity data-mining pos-tagging lemmatization transliteration dependency-parsing relation-extraction parts-of-speech-tagging named-entity-recognition natural-language-processing natural-language-understanding

Created 2015-10-18

2,621 commits to master branch, last one 2 years ago

vscode-blockman leodevbro

17

466

mit

9

VSCode extension to highlight nested code blocks

ast parser tokenizer vscode-api indentation vscode-blockman highlight-blocks vscode-extension abstract-syntax-tree

Created 2021-05-12

201 commits to main branch, last one about a month ago

gpt-tokenizer niieani

38

455

mit

5

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.

bpe gpt-2 gpt-3 gpt-4 gpt-4o openai decoder encoder tokenizer machine-learning

Created 2023-03-22

109 commits to main branch, last one 8 days ago

sentences neurosnap

39

439

mit

14

A multilingual command line sentence tokenizer in Golang

cli sentences tokenizer sentence-tokenizer

Created 2015-08-07

223 commits to main branch, last one 8 months ago

lexmachine timtadh

28

406

other

13

Lex machinary for go.

go dfa lex nfa lexer tokenizer regular-expression lexical-analysis-engines lexical-analysis-framework

Created 2014-04-28

157 commits to master branch, last one 2 years ago

fugashi polm

33

402

mit

8

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

nlp mecab japanese tokenizer cython-wrapper

Created 2019-10-14

283 commits to master branch, last one 10 days ago

nagisa taishi-i

22

389

mit

12

A Japanese tokenizer based on recurrent neural networks

nlp dynet japanese tokenizer nlp-library pos-tagging sequence-labeling word-segmentation natural-language-processing

Created 2018-02-14

194 commits to master branch, last one 5 months ago

lindera lindera

37

388

mit

6

A multilingual morphological analysis library.

library analyzer tokenizer multilingual morphological

Created 2020-01-22

501 commits to main branch, last one 8 days ago

jumanpp ku-nlp

44

380

apache-2.0

31

Juman++ (a Morphological Analyzer Toolkit)

cjk nlp juman japanese tokenizer pos-tagger pos-tagging word-segmentation part-of-speech-tagger morphological-analyser morphological-analysis

Created 2016-10-11

1,093 commits to master branch, last one about a year ago

llama-tokenizer-js belladoreai

22

343

mit

4

JS tokenizer for LLaMA 1 and 2

llm llama tokenizer javascript

Created 2023-06-11

38 commits to master branch, last one 4 months ago

vibrato daac-tools

14

338

apache-2.0

7

🎤 vibrato: Viterbi-based accelerated tokenizer

nlp rust japanese tokenizer segmentation tokenization morphological-analysis

Created 2022-07-06

172 commits to main branch, last one about a month ago

Data-Labeling risesoft-y9

49

310

gpl-3.0

38

数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构筑基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注是一个完全开源的项目，无商业版，但是需要依赖开源的数字底座进行人员岗位管控。各类词库结果会定期在本平台公开。

java vue3 nacos docker chinese tokenizer springboot2 elasticsearch data-annotations tokenizer-parser data-annotation-tools

Created 2024-08-26

21 commits to main branch, last one 8 days ago