Search Results - RepositoryStats

4.4k

30.6k

mit

562

💫 Industrial-strength Natural Language Processing (NLP) in Python

ai nlp spacy cython python nlp-library data-science tokenization deep-learning entity-linking neural-network neural-networks machine-learning text-classification artificial-intelligence named-entity-recognition natural-language-processing

Created 2014-07-03

16,218 commits to master branch, last one about a month ago

tokencost AgentOps-AI

64

1.5k

mit

10

Easy token price estimates for 400+ LLMs. TokenOps.

llm price token claude openai analytics tokenization observability price-tracker large-language-models

Created 2023-12-03

245 commits to main branch, last one about a month ago

lunasec lunasec-io

168

1.4k

other

29

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTra...

Created 2021-03-16

3,454 commits to master branch, last one 8 months ago

Cosmos-Tokenizer NVIDIA

56

1.4k

apache-2.0

24

A suite of image and video neural tokenizers

diffusion tokenization transformers

Created 2024-10-30

31 commits to main branch, last one 5 days ago

databunker securitybunker

77

1.3k

mit

34

Secure Vault for Customer PII/PHI/PCI/KYC Records

Created 2019-12-08

1,104 commits to master branch, last one 14 days ago

Ravencoin RavenProject

680

1.1k

mit

216

Ravencoin Core integration/staging tree

asset raven token bitcoin ravencoin blockchain tokenization

Created 2017-07-09

16,643 commits to master branch, last one 8 months ago

YouTokenToMe VKCOM

102

965

mit

26

Unsupervised text tokenizer focused on computational efficiency

bpe nlp tokenization word-segmentation natural-language-processing

This repository has been archived (exclude archived)

Created 2019-06-06

85 commits to master branch, last one about a year ago

spacy-streamlit explosion

115

819

mit

19

👑 spaCy building blocks and visualizers for Streamlit apps

ner nlp spacy streamlit visualizer visualizers tokenization word-vectors machine-learning dependency-parsing text-classification part-of-speech-tagging named-entity-recognition natural-language-processing

Created 2020-06-23

85 commits to master branch, last one about a year ago

datacamp-python-data-science-track AmoDinho

526

807

mit

20

All the slides, accompanying code and exercises all stored in this repo. 🎈

nlp bokeh pandas python datacamp datascience data-science scikit-learn tokenization neural-network datacamp-course datacamp-python machinelearning neural-networks datacamp-projects datacamp-exercises datacamp-machine-learning datacamp-solutions-python natural-language-processing

Created 2018-02-07

280 commits to master branch, last one about a year ago

trankit nlp-uoregon

102

743

apache-2.0

24

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

nlp pytorch adapters xlm-roberta deeplearning multilingual tokenization lemmatization language-model machine-learning dependency-parsing morphological-tagging sentence-segmentation part-of-speech-tagging universal-dependencies artificial-intelligence natural-language-processing

Created 2021-01-08

114 commits to master branch, last one 3 months ago

ekphrasis cbaziotis

90

664

mit

19

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

nlp semeval tokenizer nlp-library tokenization spell-corrector text-processing text-segmentation word-segmentation word-normalization spelling-correction

Created 2017-02-07

77 commits to master branch, last one 2 years ago

tokenmonster alasdairforsythe

20

560

mit

11

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

tokenize tokenizer tokenizing vocabulary tokenisation tokenization text-tokenization vocabulary-builder vocabulary-generator

Created 2023-05-12

196 commits to main branch, last one about a year ago

NLP-Cube adobe

93

557

apache-2.0

32

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse nlp-cube embeddings tokenization lemmatization dependency-parser language-pipeline dependency-parsing sentence-splitting machine-translation part-of-speech-tagger information-extraction universal-dependencies

Created 2018-03-13

157 commits to master branch, last one about a year ago

php-text-analysis yooper

87

527

mit

42

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php php-language tokenization text-analysis php-text-analysis

Created 2012-05-21

245 commits to master branch, last one 15 days ago

ClangKit macmade

46

361

unknown

17

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c code llvm clang source parsing c-plus-plus diagnostics objective-c tokenization static-analysis syntax-highlighting

Created 2012-08-12

4 commits to main branch, last one 7 years ago

vibrato daac-tools

14

342

apache-2.0

7

🎤 vibrato: Viterbi-based accelerated tokenizer

nlp rust japanese tokenizer segmentation tokenization morphological-analysis

Created 2022-07-06

173 commits to main branch, last one 14 days ago

sudachi.rs WorksApplications

35

330

apache-2.0

7

Sudachi in Rust 🦀 and new generation of SudachiPy

rust python sudachi nlp-libary pos-tagging segmentation tokenization morphological-analysis

Created 2019-11-23

482 commits to develop branch, last one 2 days ago

Tokenizer OpenNMT

70

293

mit

19

Fast and customizable text tokenization library with BPE and SentencePiece support

bpe cpp icu python unicode tokenizer tokenization sentencepiece machine-translation natural-language-processing

Created 2017-02-14

601 commits to master branch, last one about a year ago

OmniTokenizer FoundationVision

7

282

mit

5

[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.

vae vqvae tokenization image-generation video-generation auto-regressive-model

Created 2024-06-13

15 commits to main branch, last one 6 months ago

codechain CodeChain-io

51

258

agpl-3.0

26

CodeChain's official implementation in Rust.

rust asset blockchain tokenization digital-securities

Created 2018-01-23

3,775 commits to master branch, last one 4 years ago

razdel natasha

31

251

mit

15

Rule-based token, sentence segmentation for Russian language

nlp python russian tokenization sentence-segmentation sentence-boundary-detection

Created 2018-11-10

98 commits to master branch, last one about a year ago

TokenScript SmartTokenLabs

71

243

mit

18

TokenScript schema, specs and paper

xml web3 mobile tokens security blockchain cryptography tokenisation tokenization

Created 2019-02-06

563 commits to main branch, last one 6 months ago

vaporetto daac-tools

9

231

apache-2.0

3

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

nlp rust analyzer japanese tokenizer segmentation tokenization morphological-analysis

Created 2021-08-18

261 commits to main branch, last one about a month ago

MyGO zjukg

4

231

unknown

4

[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation

mygo multi-modal tokenization multi-modal-fusion contrastive-learning knowledge-graph-completion multi-modal-knowledge-graph

Created 2024-04-15

11 commits to main branch, last one 24 days ago

nlp-cheat-sheet-python janlukasschroeder

67

225

unknown

7

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

nlp nltk spacy spans lexnlp python spacy-nlp cheat-sheet pos-tagging starter-kit introduction tokenization lemmatization machine-learning dependency-parsing sentence-similarity named-entity-recognition

Created 2019-09-07

38 commits to master branch, last one about a year ago

Python_Natural_Language_Processing milaan9

173

197

mit

2

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand...

nlp tf-idf stemming stopwords bag-of-words tokenization lemmatization termfrequency tutor-milaan9 ipython-notebook python4everybody python4datascience partofspeech-tagger vocabulary-matching sentence-segmentation inversedocumentfrequency named-entity-recognition

Created 2021-11-08

19 commits to main branch, last one 2 years ago

l8w8jwt GlitchedPolygons

43

152

apache-2.0

5

Minimal, OpenSSL-less and super lightweight JWT library written in C.

c ecc jwt auth token performance cryptography tokenization authorization authentication token-based-authentication

Created 2020-01-15

309 commits to master branch, last one 2 months ago

icetk THUDM

17

151

unknown

10

A unified tokenization tool for Images, Chinese and English.

transformer tokenization

Created 2021-12-22

14 commits to main branch, last one about a year ago

vtext rth

11

150

apache-2.0

5

Simple NLP in Rust with Python bindings

nlp tf-idf bag-of-words tokenization information-retrieval

Created 2018-11-05

143 commits to main branch, last one 4 years ago

simplemma adbar

12

149

mit

7

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

nlp wordlist tokenizer lemmatiser lemmatizer corpus-tools tokenization lemmatization low-resource-nlp language-detection morphological-analysis language-identification

Created 2021-01-18

233 commits to main branch, last one about a month ago