23 results found Sort:

langfuse
langfuse langfuse
396
4.4k
other
18
🪢 Open source LLM engineering platform: Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
gpt llm evals llmops openai analytics langchain evaluation monitoring playground llama-index open-source self-hosted ycombinator observability llm-evaluation prompt-management prompt-engineering large-language-models
Created 2023-05-18
2,062 commits to main branch, last one 12 hours ago
Giskard-AI
giskard Giskard-AI
228
3.7k
apache-2.0
29
🐢 Open-Source Evaluation & Testing for LLMs and ML models
llm mlops llmops llm-eval ai-safety ml-safety ai-testing ml-testing ai-red-team ai-security fairness-ai llm-security ml-validation llm-evaluation rag-evaluation red-team-tools responsible-ai trustworthy-ai model-monitoring ethical-artificial-intelligence
Created 2022-03-06
9,731 commits to main branch, last one a day ago
promptfoo
promptfoo promptfoo
230
3.4k
mit
18
Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with comman...
ci llm rag cicd ci-cd llmops prompts testing llm-eval evaluation llm-evaluation prompt-testing prompt-engineering evaluation-framework llm-evaluation-framework
Created 2023-04-28
1,361 commits to main branch, last one 20 hours ago
confident-ai
deepeval confident-ai
159
2.3k
apache-2.0
16
The LLM Evaluation Framework
llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework
Created 2023-08-10
3,019 commits to main branch, last one a day ago
Agenta-AI
agenta Agenta-AI
161
968
mit
20
The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
llm rag llms llmops langchain llm-tools llama-index llm-framework llm-evaluation prompt-toolkit rag-evaluation human-annotation prompt-management prompt-engineering large-language-models
Created 2023-04-26
7,686 commits to main branch, last one 22 hours ago
relari-ai
continuous-eval relari-ai
20
362
apache-2.0
4
Open-Source Evaluation for GenAI Application Pipelines
rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation
Created 2023-12-08
89 commits to main branch, last one 8 days ago
onejune2018
Awesome-LLM-Eval onejune2018
38
336
mit
8
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.
llm nlp rag bert gpt3 qwen llama openai chatglm chatgpt dataset benchmark evaluation awsome-list leaderboard awsome-lists llm-evaluation machine-learning large-language-model
Created 2023-04-26
237 commits to main branch, last one 7 days ago
Value4AI
Awesome-LLM-in-Social-Science Value4AI
8
173
mit
5
Awesome papers involving LLMs in Social Science.
llms policy alignment economics llm-agent psychology llm-evaluation social-network social-science large-language-models simulation-environment
Created 2023-10-15
81 commits to main branch, last one 10 days ago
Psycoy
MixEval Psycoy
19
152
unknown
1
The official evaluation suite and dynamic data release for MixEval.
mixeval benchmark evaluation llm-inference llm-evaluation benchmark-mixture foundation-models benchmarking-suite evaluation-framework large-language-model large-language-models benchmarking-framework large-multimodal-models llm-evaluation-framework
Created 2024-06-01
35 commits to main branch, last one 7 days ago
athina-ai
athina-evals athina-ai
11
151
unknown
5
Python SDK for running evaluations on LLM generated responses
llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit
Created 2023-11-22
455 commits to main branch, last one a day ago
microsoft
prompty microsoft
7
131
mit
4
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandab...
llms generative-ai llm-evaluation promptengineering
Created 2024-04-22
93 commits to main branch, last one 2 days ago
villagecomputing
superpipe villagecomputing
2
102
unknown
2
Superpipe - optimized LLM pipelines for structured data
llm data-labeling classification llm-evaluation data-extraction structured-data llm-optimization
Created 2024-02-07
99 commits to main branch, last one 8 days ago
raga-ai-hub
raga-llm-hub raga-ai-hub
7
83
other
2
Framework for LLM evaluation, guardrails and security
llmops guardrails llm-security llm-evaluation
Created 2024-03-02
5 commits to main branch, last one 27 days ago
allenai
CommonGen-Eval allenai
3
83
apache-2.0
6
Evaluating LLMs with CommonGen-Lite
llm llama2 chatgpt evaluation gpt-evaluation llm-evaluation text-generation
Created 2024-01-04
37 commits to main branch, last one 5 months ago
PetroIvaniuk
llms-tools PetroIvaniuk
13
80
apache-2.0
2
A list of LLMs Tools & Projects
ai llm chatgpt chat-bot chatbots data-science llm-evaluation open-source-llm machine-learning
Created 2023-05-09
49 commits to main branch, last one a day ago
Re-Align
just-eval Re-Align
5
67
mit
3
A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.
llm gpt4 llm-eval evaluation llm-evaluation llm-evaluation-toolkit
Created 2023-11-19
40 commits to main branch, last one 4 months ago
loganrjmurphy
LeanEuclid loganrjmurphy
2
54
mit
4
LeanEuclid is a benchmark for autoformalization in the domain of Euclidean geometry, targeting the proof assistant Lean.
lean4 formalization llm-evaluation theorem-proving autoformalization euclidean-geometry
Created 2024-05-27
14 commits to master branch, last one 26 days ago
rungalileo
hallucination-index rungalileo
4
51
unknown
5
Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.
llm rag openai hallucinations llm-evaluation large-language-models retrieval-augmented-generation
Created 2023-11-15
3 commits to main branch, last one 7 months ago
deshwalmahesh
PHUDGE deshwalmahesh
6
44
unknown
1
Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute, relative and much more. It contains a list of all the availab...
ai ml llm nlp sota judge phi-3 pytorch evaluation finetuning hallucination custom-dataset llm-evaluation feedback-collection hallucination-detection
Created 2024-05-11
29 commits to main branch, last one 26 days ago
parea-ai
parea-sdk-py parea-ai
4
43
apache-2.0
2
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm llmops metrics llm-eval llm-tools generative-ai llm-evaluation good-first-issue llms-benchmarking prompt-engineering llm-evaluation-toolkit llm-evaluation-framework
Created 2023-07-24
925 commits to main branch, last one 6 days ago
iMeanAI
WebCanvas iMeanAI
2
39
mit
2
This repository has no description...
agent llm-agent llm-evaluation benchmark-framework
Created 2024-06-06
422 commits to main branch, last one 2 days ago
AntonioGr7
pratical-llms AntonioGr7
8
32
unknown
3
A collection of hand on notebook for LLMs practitioner
llm genai llm-serving llm-training quantization llm-inference llm-evaluation
Created 2024-04-04
51 commits to main branch, last one 27 days ago
ChanLiang
CONNER ChanLiang
1
28
unknown
2
The implementation for EMNLP 2023 paper ”Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators“
llama chatgpt emnlp2023 factuality hallucinations llm-evaluation nlg-evaluation large-language-models
Created 2023-10-12
12 commits to main branch, last one 5 months ago