30 results found Sort:

google
sentencepiece google
1.2k
10.4k
apache-2.0
126
Unsupervised text tokenizer for Neural Network-based text generation.
word-segmentation neural-machine-translation natural-language-processing
Created 2017-03-07
986 commits to master branch, last one 4 months ago
baidu
lac baidu
595
3.9k
apache-2.0
105
百度NLP：分词，词性标注，命名实体识别，词重要性
java python chinese-nlp lexical-analysis word-segmentation part-of-speech-tagger named-entity-recognition chinese-word-segmentation
Created 2018-07-02
155 commits to master branch, last one 3 years ago
wolfgarbe
SymSpell wolfgarbe
298
3.2k
mit
71
SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
spelling symspell spellcheck levenshtein spell-check fuzzy-search edit-distance fuzzy-matching text-segmentation word-segmentation damerau-levenshtein spelling-correction levenshtein-distance chinese-text-segmentation chinese-word-segmentation approximate-string-matching
Created 2014-03-25
488 commits to master branch, last one 2 months ago
PyThaiNLP
pythainlp PyThaiNLP
273
994
apache-2.0
46
Thai natural language processing in Python
thai python soundex thai-nlp nlp-library thai-soundex hacktoberfest thai-language text-processing thai-nlp-library word-segmentation computational-linguistics natural-language-processing
Created 2016-06-23
4,847 commits to dev branch, last one a day ago
VKCOM
YouTokenToMe VKCOM
103
964
mit
26
Unsupervised text tokenizer focused on computational efficiency
bpe nlp tokenization word-segmentation natural-language-processing
This repository has been archived (exclude archived)
Created 2019-06-06
85 commits to master branch, last one about a year ago
mammothb
symspellpy mammothb
121
809
mit
16
Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
python spelling symspell spellcheck levenshtein spell-check fuzzy-search edit-distance fuzzy-matching text-segmentation word-segmentation damerau-levenshtein spelling-correction levenshtein-distance chinese-text-segmentation chinese-word-segmentation approximate-string-matching
Created 2018-08-13
294 commits to master branch, last one 5 days ago
ckiplab
ckip-transformers ckiplab
74
702
gpl-3.0
15
CKIP Transformers
ckip transformers language-model word-segmentation part-of-speech-tagging named-entity-recognition
Created 2020-08-17
116 commits to master branch, last one about a year ago
cbaziotis
ekphrasis cbaziotis
90
663
mit
18
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...
nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction
Created 2017-02-07
77 commits to master branch, last one 2 years ago
vncorenlp
VnCoreNLP vncorenlp
146
600
other
31
A Vietnamese natural language processing toolkit (NAACL 2018)
ner nlp java parsing python3 vnmarmot vncorenlp pos-tagger vietnamese pos-tagging rdrsegmenter vietnamese-nlp word-segmenter word-segmentation dependency-parsing vietnamese-tokenizer sentence-segmentation named-entity-recognition natural-language-processing
Created 2017-12-30
46 commits to master branch, last one about a year ago
JayYip
m3tl JayYip
125
547
apache-2.0
19
BERT for Multitask Learning
cws ner nlp bert transformer part-of-speech encoder-decoder pretrained-models word-segmentation multitask-learning multi-task-learning text-classification named-entity-recognition
Created 2018-11-29
535 commits to master branch, last one about a year ago
bab2min
Kiwi bab2min
50
545
other
17
Kiwi(지능형 한국어 형태소 분석기)
cpp nlp korean korean-nlp morphology korean-tokenizer word-segmentation korean-text-processing morphological-analysis
Created 2017-02-21
1,124 commits to main branch, last one 9 days ago
modelscope
AdaSeq modelscope
38
427
apache-2.0
13
AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models
crf ner nlp bert pytorch chinese-nlp entity-typing multi-modal-ner sequence-labeling word-segmentation relation-extraction token-classification information-extraction named-entity-recognition natural-language-processing natural-language-understanding
Created 2022-12-01
246 commits to master branch, last one about a year ago
taishi-i
nagisa taishi-i
23
391
mit
12
A Japanese tokenizer based on recurrent neural networks
nlp dynet japanese tokenizer nlp-library pos-tagging sequence-labeling word-segmentation natural-language-processing
Created 2018-02-14
194 commits to master branch, last one 6 months ago
ku-nlp
jumanpp ku-nlp
44
382
apache-2.0
31
Juman++ (a Morphological Analyzer Toolkit)
cjk nlp juman japanese tokenizer pos-tagger pos-tagging word-segmentation part-of-speech-tagger morphological-analyser morphological-analysis
Created 2016-10-11
1,093 commits to master branch, last one about a year ago
jacksonllee
pycantonese jacksonllee
38
364
mit
20
Cantonese Linguistics and NLP
nlp python jyutping cantonese stop-words linguistics pycantonese word-segmentation part-of-speech-tagging computational-linguistics natural-language-processing
Created 2014-12-13
328 commits to main branch, last one 7 months ago
yongzhuo
Pytorch-NLU yongzhuo
51
342
apache-2.0
4
Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词、抽取式文本摘要等序列标注任务。 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label c...
bert python3 pytorch pos-tagging transformers pretrained-models sequence-labeling word-segmentation text-classification named-entity-recognition chinese-text-segmentation chinese-text-classification
Created 2021-08-29
13 commits to main branch, last one 5 months ago
bab2min
kiwipiepy bab2min
29
294
other
7
Python API for Kiwi
nlp korean korean-nlp python-library korean-tokenizer word-segmentation morphological-analysis
Created 2019-09-06
677 commits to main branch, last one 9 days ago
jidasheng
bi-lstm-crf jidasheng
48
246
mit
9
A PyTorch implementation of the BI-LSTM-CRF model.
crf ner nlp bilstm pytorch lstm-crf crf-model bilstm-crf bi-lstm-crf pos-tagging sequence-tagging sequence-labeling word-segmentation
Created 2019-12-03
6 commits to master branch, last one about a month ago
monpa-team
monpa monpa-team
26
245
other
23
MONPA 罔拍是一個提供正體中文斷詞、詞性標註以及命名實體辨識的多任務模型
ner nlp pos bert albert pos-tagging word-segmentation named-entity-recognition chinese-word-segmentation
Created 2019-07-23
54 commits to master branch, last one 2 years ago
fastcws
fastcws fastcws
8
199
bsd-2-clause
3
轻量级高性能中文分词项目
chinese wordseg wordbreak word-break nlp-chinese word-segment word-segmenter wordsegmentation word-segmentation hidden-markov-model frequency-dictionary
Created 2023-01-27
7 commits to devel branch, last one about a year ago
taishi-i
toiro taishi-i
9
119
apache-2.0
7
A comparison tool of Japanese tokenizers
nlp bert japanese nlp-library word-segmentation natural-language-processing
Created 2020-08-13
74 commits to master branch, last one about a year ago
ckiplab
ckipnlp ckiplab
15
115
gpl-3.0
9
CKIP CoreNLP Toolkits
nlp ckip sentence-parsing word-segmentation sentence-segmentation coreference-resolution part-of-speech-tagging named-entity-recognition
Created 2019-04-01
259 commits to master branch, last one about a year ago
Ailln
nlp-roadmap Ailln
12
107
mit
4
🗺️ 一个自然语言处理的学习路线图
nlp roadmap word-embedding sequence-labeling word-segmentation natural-language-processing
Created 2019-04-17
42 commits to master branch, last one about a year ago
peterolson
hanzi-tools peterolson
17
104
unlicense
8
Converts from Chinese characters to pinyin, between simplified and traditional, and does word segmentation.
pinyin word-segmentation chinese-characters simplified-chinese traditional-chinese
Created 2016-11-23
75 commits to master branch, last one 3 years ago
wolfgarbe
WordSegmentationTM wolfgarbe
14
80
mit
5
Fast Word Segmentation with Triangular Matrix
symspell spellcheck spell-check spellchecker spell-checker spell-corrector spelling-checker text-segmentation word-segmentation spelling-corrector spelling-correction
Created 2018-04-21
33 commits to master branch, last one 3 years ago
datquocnguyen
RDRsegmenter datquocnguyen
10
79
other
8
A Fast and Accurate Vietnamese Word Segmenter (LREC 2018)
vietnamese vietnamese-nlp word-segmentation vietnamese-tokenizer
Created 2017-09-19
29 commits to master branch, last one 2 years ago
ruanchaves
hashformers ruanchaves
4
70
mit
6
Hashformers is a framework for hashtag segmentation with Transformers and Large Language Models (LLMs).
nlp bert llms paper twitter transformer segmentation transformers deep-learning tweet-analysis hashtag-segmentor transformers-gpt2 word-segmentation sentiment-analysis sentiment-polarity large-language-models tweets-classification sentiment-classification twitter-sentiment-analysis natural-language-processing
Created 2020-05-21
509 commits to master branch, last one 4 months ago
phongnt570
UETsegmenter phongnt570
13
69
unknown
10
A toolkit for Vietnamese word segmentation
java vietnamese word-segmentation natural-language-processing
Created 2016-04-11
26 commits to master branch, last one 2 years ago
ye-kyaw-thu
sylbreak ye-kyaw-thu
19
56
apache-2.0
7
Syllable segmentation tool for Myanmar language (Burmese) by Ye.
burmese myanmar syllable word-segmentation regular-expressions
Created 2017-04-01
189 commits to master branch, last one 11 months ago
giganticode
codeprep giganticode
11
46
unknown
5
A toolkit for pre-processing large source code corpora
language-modeling word-segmentation source-code-analysis natural-language-processing mining-software-repositories
Created 2019-04-05
319 commits to master branch, last one 3 years ago