Search Results - RepositoryStats

tokenizer theseer

22

5.2k

other

9

A small library for converting tokenized PHP source code into XML (and potentially other formats)

php xml tokenizer

Created 2017-04-05

75 commits to master branch, last one 9 months ago

chevrotain Chevrotain

207

2.5k

apache-2.0

30

Parser Building Toolkit for JavaScript

lexer parsing grammars tokenizer javascript typescript open-source parser-library

Created 2015-04-15

2,340 commits to master branch, last one 2 months ago

hazm roshan-research

179

1.2k

mit

25

Persian NLP Toolkit

nlp farsi python persian tokenizer embeddings persian-nlp pos-tagging lemmatization normalization text-processing dependency-parser natural-language-processing

Created 2013-10-29

1,411 commits to master branch, last one 6 months ago

natasha natasha

110

1.2k

mit

59

Solves basic Russian NLP tasks, API for lower level Natasha projects

ner nlp python syntax russian tokenizer embeddings morphology visualization sentence-segmentation

Created 2016-08-03

413 commits to master branch, last one 2 months ago

soynlp lovit

183

946

other

44

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

nlp tokenizer korean-nlp postagging word-extraction korean-text-processing

Created 2017-05-13

766 commits to master branch, last one 3 years ago

kagome ikawaha

55

835

mit

23

Self-contained Japanese Morphological Analyzer written in pure Go

korean japanese tokenizer nlp-library pos-tagging segmentation hacktoberfest japanese-language morphological-analysis

Created 2014-06-26

806 commits to v2 branch, last one 4 months ago

moo no-context

67

834

bsd-3-clause

12

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

lexer regexp tokenizer javascript

Created 2017-03-10

363 commits to main branch, last one about a year ago

tiktokenizer dqbd

92

720

mit

6

Online playground for OpenAPI tokenizers

nextjs openai chatgpt t3-stack tiktoken tokenizer

Created 2023-03-02

55 commits to master branch, last one 29 days ago

Wordless BLKSerene

92

701

gpl-3.0

27

An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation

corpus tagger stopword tokenizer lemmatizer literature linguistics translation corpus-tools corpus-search corpus-analysis corpus-processing corpus-statistics dependency-parser corpus-linguistics

Created 2018-08-23

1,453 commits to main branch, last one 17 days ago

ekphrasis cbaziotis

90

663

mit

18

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction

Created 2017-02-07

77 commits to master branch, last one 2 years ago

simple wangfenjin

89

624

mit

6

支持中文和拼音的 SQLite fts5 全文搜索扩展｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin

fts fts5 cpp14 pinyin sqlite chinese sqlite3 tokenizer sqlite3-fts5

Created 2020-03-03

160 commits to master branch, last one a day ago

open-korean-text open-korean-text

98

615

apache-2.0

49

Open Korean Text Processor - An Open-source Korean Text Processor

korean tokenizer text-processing korean-tokenizer korean-text-processing natural-language-processing

Created 2017-01-24

799 commits to master branch, last one 9 months ago

jflex jflex-de

115

589

other

23

The fast scanner generator for Java™ with full Unicode support

cup dfa nfa flex java yacc lexer regexp grammar parsing scanner tokenizer bazel-rules maven-plugin lexer-generator dfa-minimization lexical-analyzer scanner-generator

Created 2015-02-15

2,109 commits to master branch, last one about a year ago

tokenmonster alasdairforsythe

20

557

mit

11

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator

Created 2023-05-12

196 commits to main branch, last one about a year ago

php-parser glayzzle

71

540

bsd-3-clause

19

:herb: NodeJS PHP Parser - extract AST or tokens

ast php lexer parser php-ast tokenizer javascript php-parser development static-code-analysis

Created 2014-12-07

1,843 commits to main branch, last one 16 days ago

js-tokens lydell

33

505

mit

7

Tiny JavaScript tokenizer.

regex tokenizer ecmascript javascript

Created 2014-03-08

150 commits to main branch, last one 28 days ago

sacremoses hplt-project

59

489

mit

12

Python port of Moses tokenizer, truecaser and normalizer

nlp tokenizer machine-translation

Created 2018-04-20

374 commits to master branch, last one about a year ago

friso lionsoul2014

92

485

apache-2.0

33

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other...

c tokenizer cjk-tokenizer php-tokenizer full-text-search korean-tokenizer chinese-tokenizer japanese-tokenizer chinese-word-segmentation

Created 2014-03-31

148 commits to master branch, last one about a year ago

gpt-tokenizer niieani

38

479

mit

5

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

bpe gpt-2 gpt-3 gpt-4 gpt-4o gpt-o1 openai decoder encoder tokenizer machine-learning

Created 2023-03-22

115 commits to main branch, last one 12 days ago

cogcomp-nlp CogComp

142

473

other

62

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...

ner nlp pos cogcomp big-data tokenizer lemmatizer similarity data-mining pos-tagging lemmatization transliteration dependency-parsing relation-extraction parts-of-speech-tagging named-entity-recognition natural-language-processing natural-language-understanding

Created 2015-10-18

2,621 commits to master branch, last one 2 years ago

vscode-blockman leodevbro

17

470

mit

9

VSCode extension to highlight nested code blocks

ast parser tokenizer vscode-api indentation vscode-blockman highlight-blocks vscode-extension abstract-syntax-tree

Created 2021-05-12

201 commits to main branch, last one 2 months ago

Data-Labeling risesoft-y9

74

446

gpl-3.0

52

数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构筑基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注是一个完全开源的项目，无商业版，但是需要依赖开源的数字底座进行人员岗位管控。各类词库结果会定期在本平台公开。