Search Results - RepositoryStats

4.5k

31.1k

mit

562

💫 Industrial-strength Natural Language Processing (NLP) in Python

ai nlp spacy cython python nlp-library data-science tokenization deep-learning entity-linking neural-network neural-networks machine-learning text-classification artificial-intelligence named-entity-recognition natural-language-processing

Created 2014-07-03

16,225 commits to master branch, last one about a month ago

tokencost AgentOps-AI

73

1.6k

mit

10

Easy token price estimates for 400+ LLMs. TokenOps.

llm price token claude openai analytics tokenization observability price-tracker large-language-models

Created 2023-12-03

257 commits to main branch, last one 11 days ago

Cosmos-Tokenizer NVIDIA

73

1.6k

apache-2.0

24

A suite of image and video neural tokenizers

diffusion tokenization transformers

This repository has been archived (exclude archived)

Created 2024-10-30

51 commits to main branch, last one about a month ago

lunasec lunasec-io

168

1.4k

other

29

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTra...

Created 2021-03-16

3,454 commits to master branch, last one 10 months ago

databunker securitybunker

81

1.3k

mit

33

Secure Vault for Customer PII/PHI/PCI/KYC Records

Created 2019-12-08

1,134 commits to master branch, last one a day ago

Ravencoin RavenProject

688

1.1k

mit

216

Ravencoin Core integration/staging tree

asset raven token bitcoin ravencoin blockchain tokenization

Created 2017-07-09

16,643 commits to master branch, last one 10 months ago

YouTokenToMe VKCOM

103

965

mit

25

Unsupervised text tokenizer focused on computational efficiency

bpe nlp tokenization word-segmentation natural-language-processing

This repository has been archived (exclude archived)

Created 2019-06-06

85 commits to master branch, last one about a year ago

spacy-streamlit explosion

117

828

mit

18

👑 spaCy building blocks and visualizers for Streamlit apps

ner nlp spacy streamlit visualizer visualizers tokenization word-vectors machine-learning dependency-parsing text-classification part-of-speech-tagging named-entity-recognition natural-language-processing

Created 2020-06-23

85 commits to master branch, last one about a year ago

datacamp-python-data-science-track AmoDinho

522

826

mit

19

All the slides, accompanying code and exercises all stored in this repo. 🎈

nlp bokeh pandas python datacamp datascience data-science scikit-learn tokenization neural-network datacamp-course datacamp-python machinelearning neural-networks datacamp-projects datacamp-exercises datacamp-machine-learning datacamp-solutions-python natural-language-processing

Created 2018-02-07

280 commits to master branch, last one about a year ago

trankit nlp-uoregon

103

747

apache-2.0

24

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

nlp pytorch adapters xlm-roberta deeplearning multilingual tokenization lemmatization language-model machine-learning dependency-parsing morphological-tagging sentence-segmentation part-of-speech-tagging universal-dependencies artificial-intelligence natural-language-processing

Created 2021-01-08

114 commits to master branch, last one 5 months ago

ekphrasis cbaziotis

92

666

mit

18

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction

Created 2017-02-07

77 commits to master branch, last one 2 years ago

tokenmonster alasdairforsythe

19

572

mit

11

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator

Created 2023-05-12

196 commits to main branch, last one about a year ago

NLP-Cube adobe

94

557

apache-2.0

30

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse nlp-cube embeddings tokenization lemmatization dependency-parser language-pipeline dependency-parsing sentence-splitting machine-translation part-of-speech-tagger information-extraction universal-dependencies

Created 2018-03-13

157 commits to master branch, last one about a year ago

php-text-analysis yooper

89

529

mit

40

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php php-language tokenization text-analysis php-text-analysis

Created 2012-05-21

245 commits to master branch, last one 2 months ago

ClangKit macmade

45

363

unknown

15

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c code llvm clang source parsing c-plus-plus diagnostics objective-c tokenization static-analysis syntax-highlighting

Created 2012-08-12

4 commits to main branch, last one 7 years ago

vibrato daac-tools

15

352

apache-2.0

7

🎤 vibrato: Viterbi-based accelerated tokenizer

nlp rust japanese tokenizer segmentation tokenization morphological-analysis

Created 2022-07-06

174 commits to main branch, last one 22 days ago

sudachi.rs WorksApplications

37

341

apache-2.0

6

Sudachi in Rust 🦀 and new generation of SudachiPy

rust python sudachi nlp-libary pos-tagging segmentation tokenization morphological-analysis

Created 2019-11-23

482 commits to develop branch, last one 2 months ago

Tokenizer OpenNMT

73

302

mit

19

Fast and customizable text tokenization library with BPE and SentencePiece support

bpe cpp icu python unicode tokenizer tokenization sentencepiece machine-translation natural-language-processing

Created 2017-02-14

601 commits to master branch, last one about a year ago

TOTEM SaberaTalukder

38

287

unknown

11

The official code 👩‍💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis

time-series tokenization foundation-models time-series-analysis time-series-imputation representation-learning time-series-forecasting time-series-foundation-model time-series-anomaly-detection

Created 2024-02-27

4 commits to master branch, last one 23 days ago

OmniTokenizer FoundationVision

7

286

mit

5

[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.

vae vqvae tokenization image-generation video-generation auto-regressive-model

Created 2024-06-13

15 commits to main branch, last one 8 months ago

razdel natasha

32

260

mit

14

Rule-based token, sentence segmentation for Russian language

nlp python russian tokenization sentence-segmentation sentence-boundary-detection

Created 2018-11-10

98 commits to master branch, last one about a year ago

codechain CodeChain-io

51

257

agpl-3.0

25

CodeChain's official implementation in Rust.

rust asset blockchain tokenization digital-securities

Created 2018-01-23

3,775 commits to master branch, last one 4 years ago

TokenScript SmartTokenLabs

68

243

mit

17

TokenScript schema, specs and paper

xml web3 mobile tokens security blockchain cryptography tokenisation tokenization

Created 2019-02-06

563 commits to main branch, last one 8 months ago

MyGO zjukg

4

240

unknown

4

[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation

mygo multi-modal tokenization multi-modal-fusion contrastive-learning knowledge-graph-completion multi-modal-knowledge-graph

Created 2024-04-15

11 commits to main branch, last one 2 months ago

vaporetto daac-tools

10

235

apache-2.0

2

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

nlp rust analyzer japanese tokenizer segmentation tokenization morphological-analysis

Created 2021-08-18

263 commits to main branch, last one 22 days ago

nlp-cheat-sheet-python janlukasschroeder

70

232

unknown

8

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

nlp nltk spacy spans lexnlp python spacy-nlp cheat-sheet pos-tagging starter-kit introduction tokenization lemmatization machine-learning dependency-parsing sentence-similarity named-entity-recognition

Created 2019-09-07

38 commits to master branch, last one 2 years ago

Python_Natural_Language_Processing milaan9

173

198

mit

1

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand...

nlp tf-idf stemming stopwords bag-of-words tokenization lemmatization termfrequency tutor-milaan9 ipython-notebook python4everybody python4datascience partofspeech-tagger vocabulary-matching sentence-segmentation inversedocumentfrequency named-entity-recognition

Created 2021-11-08

19 commits to main branch, last one 2 years ago

simplemma adbar

12

154

mit

5

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

nlp wordlist tokenizer lemmatiser lemmatizer corpus-tools tokenization lemmatization low-resource-nlp language-detection morphological-analysis language-identification

Created 2021-01-18

233 commits to main branch, last one 3 months ago