10 results found Sort:

jimmc414
onefilellm jimmc414
72
751
mit
9
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
doi llm pdf pmid text arxiv ipynb github papers sci-hub tiktoken repository pull-request youtube-transcript-api
Created 2023-03-17
169 commits to main branch, last one 8 days ago
dqbd
tiktokenizer dqbd
92
738
mit
6
Online playground for OpenAPI tokenizers
nextjs openai chatgpt t3-stack tiktoken tokenizer
Created 2023-03-02
55 commits to master branch, last one 2 months ago
pkoukk
tiktoken-go pkoukk
79
704
mit
6
go version of tiktoken
go gpt-4 golang openai chatgpt tiktoken gpt-35-turbo
Created 2023-03-08
64 commits to main branch, last one 8 months ago
CNSeniorious000
free-chat CNSeniorious000
54
179
mit
3
An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia
aibot astro openai chatgpt solidjs tiktoken openai-api
Created 2023-03-30
398 commits to promplate-demo branch, last one 22 hours ago
saschaschramm
chatgpt saschaschramm
15
148
unknown
9
Analysis of OpenAI's ChatGPT
chat gpt-4 openai chatgpt plugins analysis tiktoken cl100kbase gpt-4-0613 gpt-35-turbo gpt-4-32k-0613 language-model gpt-35-turbo-0301 gpt-35-turbo-1106 chat-markup-language gpt-35-turbo-16k-0613 text-davinci-002-render text-davinci-002-plugins text-davinci-002-render-sha
Created 2022-12-03
64 commits to main branch, last one about a year ago
openshieldai
openshield openshieldai
4
70
apache-2.0
2
OpenShield is a new generation security layer for AI models
ai llm llama owasp golang models openai python firewall guardian security tiktoken probllama tokenizer openai-api security-tools artificial-intelligence
Created 2024-06-05
798 commits to main branch, last one 3 days ago
tryAGI
Tiktoken tryAGI
4
63
mit
5
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
ai gpt4 csharp openai tokens chatgpt encoding p50kbase tiktoken langchain cl100kbase gpt35turbo tiktoken-sharp langchain-dotnet
Created 2023-06-09
153 commits to main branch, last one 3 days ago
ElmiraGhorbani
chatgpt-long-term-memory ElmiraGhorbani
3
55
unknown
2
The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.
gpt-3 redis memory openai python vector chatbot chatgpt context tiktoken datastore embeddings chatgpt-api llama-index gpt-35-turbo text-retrieval long-term-memory similarity-search text-summarization embedding-similarity
Created 2023-07-19
53 commits to main branch, last one about a year ago
cahya-wirawan
rwkv-tokenizer cahya-wirawan
2
40
apache-2.0
2
A fast RWKV Tokenizer written in Rust
llm rwkv trie tiktoken tokenizer huggingface
Created 2024-05-29
86 commits to main branch, last one 4 months ago
chonkie-ai
autotiktokenizer chonkie-ai
3
35
mit
2
🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨
nlp tiktoken tokenizers transformers machine-learning
Created 2024-11-04
49 commits to main branch, last one 27 days ago