Search Results - RepositoryStats

679

7.4k

other

28

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

Created 2023-05-18

3,257 commits to main branch, last one 22 hours ago

promptfoo promptfoo

405

5.0k

mit

21

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command ...

ci llm rag cicd ci-cd llmops prompts testing llm-eval evaluation pentesting red-teaming llm-evaluation prompt-testing prompt-engineering evaluation-framework vulnerability-scanners llm-evaluation-framework

Created 2023-04-28

3,165 commits to main branch, last one 12 hours ago

giskard Giskard-AI

277

4.1k

apache-2.0

33

🐢 Open-Source Evaluation & Testing for AI & LLM systems

Created 2022-03-06

10,170 commits to main branch, last one 2 days ago

deepeval confident-ai

330

4.1k

apache-2.0

23

The LLM Evaluation Framework

llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework

Created 2023-08-10

3,948 commits to main branch, last one a day ago

garak NVIDIA

265

3.1k

apache-2.0

30

the LLM vulnerability scanner

ai llm-security llm-evaluation security-scanners vulnerability-assessment

Created 2023-05-10

1,492 commits to main branch, last one a day ago

AutoRAG Marker-Inc-Korea

228

3.0k

apache-2.0

23

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

qa llm ops rag automl python llm-ops analysis pipeline embeddings evaluation open-source benchmarking optimization llm-evaluation rag-evaluation document-parser retrieval-augmented-generation

Created 2024-01-10

837 commits to main branch, last one 2 days ago

helicone Helicone

270

2.5k

apache-2.0

14

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

gpt llm llmops openai llm-cost analytics langchain evaluation monitoring playground llama-index open-source ycombinator llm-evaluation agent-monitoring llm-observability prompt-management prompt-engineering large-language-models

Created 2023-01-31

3,088 commits to main branch, last one 17 hours ago

LLM-Engineers-Handbook PacktPublishing

353

2.1k

mit

32

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

aws llm rag genai mlops llmops llm-evaluation fine-tuning-llm ml-system-design

Created 2024-04-09

170 commits to main branch, last one 21 days ago

agenta Agenta-AI

215

1.6k

mit

22

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

llm-tools llm-platform llm-framework llm-as-a-judge llm-evaluation llm-monitoring llm-playground rag-evaluation llmops-platform llm-observability prompt-management prompt-engineering

Created 2023-04-26

10,700 commits to main branch, last one 2 days ago

lmnr lmnr-ai

70

1.4k

apache-2.0

5

Laminar - open-source all-in-one platform for engineering AI products. Crate data flywheel for you AI app. Traces, Evals, Datasets, Labels. YC S24.

ai rag aiops evals agents llmops analytics rust-lang evaluation monitoring open-source self-hosted llm-workflow observability llm-evaluation developer-tools ai-observability pipeline-builder llm-observability

Created 2024-08-29

257 commits to main branch, last one 11 hours ago

prompty microsoft

52

613

mit

12

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandab...

llms generative-ai llm-evaluation promptengineering

Created 2024-04-22

297 commits to main branch, last one 7 days ago

continuous-eval relari-ai

31

456

apache-2.0

4

Data-Driven Evaluation for LLM-Powered Applications

rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation

Created 2023-12-08

96 commits to main branch, last one 3 months ago

Awesome-LLM-Eval onejune2018

43

446

mit

9

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

llm nlp rag bert gpt3 qwen llama openai chatglm chatgpt dataset benchmark evaluation awsome-list leaderboard awsome-lists llm-evaluation machine-learning large-language-model

Created 2023-04-26

263 commits to main branch, last one about a month ago

awesome-azure-openai-llm kimtth

47

336

unknown

10

a curated list of 🌌 Azure OpenAI, 🦙Large Language Models, and references with notes.

gpt llm rag agent openai awesome chatgpt copilot langchain llm-agent cheatsheet llama-index awesome-list azure-openai llm-evaluation semantic-kernel prompt-engineering

Created 2023-04-13

175 commits to main branch, last one 3 days ago

palico-ai palico-ai

26

335

mit

3

Build, Improve Performance, and Productionize your LLM Application with an Integrated Framework

ai llm rag docker nodejs openai autogen portkey anthropic langchain llm-agent full-stack javascript llamaindex typescript langchain-js llm-framework llm-evaluation llm-observability

Created 2024-03-12

419 commits to main branch, last one 24 days ago

Awesome-LLM-in-Social-Science Value4AI

24

327

mit

8

Awesome papers involving LLMs in Social Science.

llms policy alignment economics llm-agent psychology llm-evaluation social-network social-science large-language-models simulation-environment

Created 2023-10-15

132 commits to main branch, last one a day ago

athina-evals athina-ai

14

242

unknown

5

Python SDK for running evaluations on LLM generated responses

llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit

Created 2023-11-22

639 commits to main branch, last one 3 days ago

MixEval Psycoy

37

229

unknown

1

The official evaluation suite and dynamic data release for MixEval.

mixeval benchmark evaluation llm-inference llm-evaluation benchmark-mixture foundation-models benchmarking-suite evaluation-framework large-language-model large-language-models benchmarking-framework large-multimodal-models llm-evaluation-framework

Created 2024-06-01

120 commits to main branch, last one about a month ago

WebCanvas iMeanAI

12

224

mit

4

Connect agents to live web environments evaluation.

agent llm-agent llm-evaluation benchmark-framework

Created 2024-06-06

507 commits to main branch, last one 6 days ago

llms-tools PetroIvaniuk

24

157

apache-2.0

2

A list of LLMs Tools & Projects

ai llm chatgpt chat-bot chatbots data-science llm-evaluation open-source-llm machine-learning

Created 2023-05-09

63 commits to main branch, last one 26 days ago

superpipe villagecomputing

2

108

unknown

2

Superpipe - optimized LLM pipelines for structured data

llm data-labeling classification llm-evaluation data-extraction structured-data llm-optimization

Created 2024-02-07

99 commits to main branch, last one 6 months ago

hallucination-index rungalileo

7

104

unknown

7

Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.

llm rag openai hallucinations llm-evaluation large-language-models retrieval-augmented-generation

Created 2023-11-15

7 commits to main branch, last one 3 months ago

autoarena kolenaIO

7

101

apache-2.0

7

Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation

ai llm rag testing evaluation hacktoberfest llm-evaluation

Created 2024-08-28

137 commits to trunk branch, last one 2 months ago

raga-llm-hub raga-ai-hub

10

100

other

2

Framework for LLM evaluation, guardrails and security

llmops guardrails llm-security llm-evaluation

Created 2024-03-02

9 commits to main branch, last one 3 months ago

CommonGen-Eval allenai

3

87

apache-2.0

6

Evaluating LLMs with CommonGen-Lite

llm llama2 chatgpt evaluation gpt-evaluation llm-evaluation text-generation

Created 2024-01-04

37 commits to main branch, last one 11 months ago

dart-math hkust-nlp

3

86

mit

1

[NeurIPS'24] Official code for *🎯DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving*

llm nlp mathematics llm-training deep-learning llm-inference llm-evaluation

Created 2024-05-29

78 commits to main branch, last one 11 days ago

LeanEuclid loganrjmurphy

5

80

mit

7

LeanEuclid is a benchmark for autoformalization in the domain of Euclidean geometry, targeting the proof assistant Lean.

lean4 formalization llm-evaluation theorem-proving autoformalization euclidean-geometry

Created 2024-05-27

14 commits to master branch, last one 6 months ago

just-eval Re-Align

6

76

mit

4

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

llm gpt4 llm-eval evaluation llm-evaluation llm-evaluation-toolkit

Created 2023-11-19

40 commits to main branch, last one 10 months ago

parea-sdk-py parea-ai

6

75

apache-2.0

2

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

llm llmops metrics llm-eval llm-tools generative-ai llm-evaluation good-first-issue llms-benchmarking prompt-engineering llm-evaluation-toolkit llm-evaluation-framework

Created 2023-07-24

1,067 commits to main branch, last one 3 months ago

LLMEvaluation alopatenko

4

74

unknown

6

A comprehensive guide to LLM evaluation methods designed to assist in identifying the most suitable evaluation techniques for various use cases, promote the adoption of best practices in LLM assessmen...

llm evaluation llm-evaluation llm-benchmarking generative-ai-benchmarking

Created 2024-04-02

247 commits to main branch, last one 7 days ago