46 results found Sort:

EleutherAI
lm-evaluation-harness EleutherAI
2.0k
7.6k
mit
39
A framework for few-shot evaluation of language models.
transformer language-model evaluation-framework
Created 2020-08-28
3,624 commits to main branch, last one a day ago
promptfoo
promptfoo promptfoo
433
5.3k
mit
23
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command ...
ci llm rag cicd ci-cd llmops prompts testing llm-eval evaluation pentesting red-teaming llm-evaluation prompt-testing prompt-engineering evaluation-framework vulnerability-scanners llm-evaluation-framework
Created 2023-04-28
3,610 commits to main branch, last one 7 hours ago
confident-ai
deepeval confident-ai
383
4.6k
apache-2.0
27
The LLM Evaluation Framework
llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework
Created 2023-08-10
4,223 commits to main branch, last one 10 hours ago
Giskard-AI
giskard Giskard-AI
291
4.3k
apache-2.0
33
🐢 Open-Source Evaluation & Testing for AI & LLM systems
llm mlops llmops llm-eval ai-safety ml-safety ai-testing ml-testing ai-red-team ai-security fairness-ai llm-security ml-validation llm-evaluation rag-evaluation red-team-tools responsible-ai trustworthy-ai evaluation-framework ethical-artificial-intelligence
Created 2022-03-06
10,216 commits to main branch, last one 10 hours ago
huggingface
lighteval huggingface
130
1.0k
mit
28
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
evaluation huggingface evaluation-metrics evaluation-framework
Created 2024-01-26
304 commits to main branch, last one 5 hours ago
MaurizioFD
RecSys2019_DeepLearning_Evaluation MaurizioFD
249
985
agpl-3.0
39
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
bpr knn bprmf bprslim funksvd deep-learning neural-network slimelasticnet hyperparameters reproducibility matrix-completion recommender-system evaluation-framework matrix-factorization recommendation-system reproducible-research collaborative-filtering hybrid-recommender-system recommendation-algorithms content-based-recommendation
Created 2019-04-02
63 commits to master branch, last one 3 years ago
relari-ai
continuous-eval relari-ai
32
466
apache-2.0
4
Data-Driven Evaluation for LLM-Powered Applications
rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation
Created 2023-12-08
106 commits to main branch, last one 7 days ago
TonicAI
tonic_validate TonicAI
29
279
mit
14
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
llm rag llms llmops evaluation-metrics evaluation-framework large-language-models retrieval-augmented-generation
Created 2023-10-23
343 commits to main branch, last one 2 months ago
athina-ai
athina-evals athina-ai
16
258
unknown
6
Python SDK for running evaluations on LLM generated responses
llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit
Created 2023-11-22
706 commits to main branch, last one 3 days ago
JinjieNi
MixEval JinjieNi
39
233
unknown
1
The official evaluation suite and dynamic data release for MixEval.
mixeval benchmark evaluation llm-inference llm-evaluation benchmark-mixture foundation-models benchmarking-suite evaluation-framework large-language-model large-language-models benchmarking-framework large-multimodal-models llm-evaluation-framework
Created 2024-06-01
120 commits to main branch, last one 2 months ago
diningphil
PyDGN diningphil
13
220
bsd-3-clause
6
A research library for automating experiments on Deep Graph Networks
deep-graph-networks evaluation-framework deep-learning-for-graphs
Created 2020-03-21
461 commits to main branch, last one 4 months ago
zeno-ml
zeno zeno-ml
11
215
mit
8
AI Data Management & Evaluation Platform
ai python evaluation data-science machine-learning evaluation-framework
This repository has been archived (exclude archived)
Created 2022-02-03
1,044 commits to main branch, last one about a year ago
ServiceNow
AgentLab ServiceNow
52
211
other
5
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
lab llm agent agents benchmark prompting llm-agents web-agents evaluation-framework
Created 2024-05-21
519 commits to main branch, last one 2 days ago
aiverify-foundation
moonshot aiverify-foundation
42
205
apache-2.0
8
Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
llm red-teaming benchmarking trustworthy-ai evaluation-framework
Created 2023-12-14
2,030 commits to main branch, last one 6 days ago
lartpang
PySODEvalToolkit lartpang
20
168
mit
3
PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection
mae latex metrics python3 fm-curve pr-curve saliency e-measure evaluator f-measure s-measure evaluation co-saliency evaluation-metrics saliency-detection evaluation-framework metrics-visualization salient-object-detection co-salient-object-detection camouflaged-object-detection
Created 2020-07-08
66 commits to master branch, last one 4 months ago
bijington
expressive bijington
26
165
mit
10
Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.
parsing xamarin evaluation netstandard hacktoberfest cross-platform expression-parser evaluation-framework expression-evaluator
Created 2016-06-13
291 commits to main branch, last one 4 months ago
empirical-run
empirical empirical-run
13
153
mit
6
Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application
llm llmops testing llm-inference test-automation testing-framework evaluation-framework
Created 2024-03-14
218 commits to main branch, last one 8 months ago
symflower
eval-dev-quality symflower
7
143
mit
4
DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
llms evaluation software-quality evaluation-framework software-development
Created 2024-03-28
766 commits to main branch, last one 3 days ago
AI21Labs
lm-evaluation AI21Labs
15
124
apache-2.0
5
Evaluation suite for large-scale language models.
language-model evaluation-framework
Created 2021-08-05
7 commits to main branch, last one 3 years ago
nlp-uoregon
mlmm-evaluation nlp-uoregon
17
115
apache-2.0
4
Multilingual Large Language Models Evaluation Benchmark
nlp datasets evaluation multilingual language-model evaluation-datasets evaluation-framework large-language-models natural-language-processing
Created 2023-08-07
18 commits to main branch, last one about a year ago
tsenst
CrowdFlow tsenst
22
113
gpl-3.0
7
Optical Flow Dataset and Benchmark for Visual Crowd Analysis
dataset tracking optical-flow trajectories crowd-analysis crowd-counting benchmark-suite computer-vision video-analytics synthetic-images video-processing motion-estimation video-surveillance evaluation-framework multi-object-tracking tracking-by-detection tub-crowdflow-dataset
Created 2018-09-10
40 commits to master branch, last one about a year ago
microsoft
eureka-ml-insights microsoft
15
106
apache-2.0
16
A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.
ai llm mllm machine-learning evaluation-framework artificial-intelligence
Created 2024-07-18
58 commits to main branch, last one 20 hours ago
haeyeoni
lidar_slam_evaluator haeyeoni
16
91
unknown
2
LiDAR SLAM comparison and evaluation framework
slam lidar-slam evaluation-framework
Created 2021-07-26
17 commits to main branch, last one 3 years ago
codefuse-ai
codefuse-evaluation codefuse-ai
13
85
other
3
Industrial-level evaluation benchmarks for Coding LLMs in the full life-cycle of AI native software developing.企业级代码大模型评测体系,持续开放中
lcc codefuse codetranseval code-evaluation codecommenteval repository-eval evaluation-framework
Created 2023-09-28
27 commits to master branch, last one about a year ago
kaiko-ai
eva kaiko-ai
7
79
apache-2.0
6
Evaluation framework for oncology foundation models (FMs)
oncology machine-learning foundation-models evaluation-framework
Created 2024-01-16
324 commits to main branch, last one 2 days ago
BMW-InnovationLab
SORDI-AI-Evaluation-GUI BMW-InnovationLab
3
77
apache-2.0
2
This repository allows you to evaluate a trained computer vision model and get general information and evaluation metrics with little configuration.
ai bmw sordi docker python dataset no-code rest-api evaluation tensorflow deeplearning synthetic-data computer-vision evaluation-framework
Created 2022-10-28
10 commits to main branch, last one about a year ago
pyrddlgym-project
pyRDDLGym pyrddlgym-project
20
75
mit
7
A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.
gym rddl planner planners planning gymnasium simulator simulation visualizer model-based benchmarking rddl-domains visualisation visualization gym-environments benchmark-framework evaluation-framework benchmarking-framework reinforcement-learning planning-domain-definition-language
Created 2022-07-10
2,103 commits to main branch, last one 15 days ago
nouhadziri
DialogEntailment nouhadziri
5
74
mit
7
The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"
bert dialogue-evaluation evaluation-framework natural-language-inference
Created 2019-04-13
10 commits to master branch, last one 4 months ago
pentoai
vectory pentoai
0
71
mit
5
Vectory provides a collection of tools to track and compare embedding versions.
deep-learning embedding-python embedding-vectors deep-neural-networks evaluation-framework embeddings-similarity
Created 2022-09-30
45 commits to main branch, last one 2 years ago
OPTML-Group
Diffusion-MU-Attack OPTML-Group
3
68
mit
0
The official implementation of ECCV'24 paper "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now". This work introduces one fast and effe...
robustness unlearning stable-diffusion adversarial-attacks evaluation-framework attack-unlearned-diffusion-model
Created 2023-10-17
68 commits to main branch, last one 2 months ago