45 results found Sort:

EleutherAI
lm-evaluation-harness EleutherAI
2.0k
7.3k
mit
39
A framework for few-shot evaluation of language models.
transformer language-model evaluation-framework
Created 2020-08-28
3,593 commits to main branch, last one 19 hours ago
promptfoo
promptfoo promptfoo
405
5.0k
mit
21
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command ...
ci llm rag cicd ci-cd llmops prompts testing llm-eval evaluation pentesting red-teaming llm-evaluation prompt-testing prompt-engineering evaluation-framework vulnerability-scanners llm-evaluation-framework
Created 2023-04-28
3,165 commits to main branch, last one 12 hours ago
Giskard-AI
giskard Giskard-AI
277
4.1k
apache-2.0
33
🐢 Open-Source Evaluation & Testing for AI & LLM systems
llm mlops llmops llm-eval ai-safety ml-safety ai-testing ml-testing ai-red-team ai-security fairness-ai llm-security ml-validation llm-evaluation rag-evaluation red-team-tools responsible-ai trustworthy-ai evaluation-framework ethical-artificial-intelligence
Created 2022-03-06
10,170 commits to main branch, last one 2 days ago
confident-ai
deepeval confident-ai
330
4.1k
apache-2.0
23
The LLM Evaluation Framework
llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework
Created 2023-08-10
3,948 commits to main branch, last one a day ago
MaurizioFD
RecSys2019_DeepLearning_Evaluation MaurizioFD
249
985
agpl-3.0
39
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
bpr knn bprmf bprslim funksvd deep-learning neural-network slimelasticnet hyperparameters reproducibility matrix-completion recommender-system evaluation-framework matrix-factorization recommendation-system reproducible-research collaborative-filtering hybrid-recommender-system recommendation-algorithms content-based-recommendation
Created 2019-04-02
63 commits to master branch, last one 3 years ago
huggingface
lighteval huggingface
109
889
mit
29
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
evaluation huggingface evaluation-metrics evaluation-framework
Created 2024-01-26
273 commits to main branch, last one 10 hours ago
relari-ai
continuous-eval relari-ai
31
456
apache-2.0
4
Data-Driven Evaluation for LLM-Powered Applications
rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation
Created 2023-12-08
96 commits to main branch, last one 3 months ago
TonicAI
tonic_validate TonicAI
27
272
mit
14
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
llm rag llms llmops evaluation-metrics evaluation-framework large-language-models retrieval-augmented-generation
Created 2023-10-23
343 commits to main branch, last one about a month ago
athina-ai
athina-evals athina-ai
14
242
unknown
5
Python SDK for running evaluations on LLM generated responses
llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit
Created 2023-11-22
639 commits to main branch, last one 3 days ago
Psycoy
MixEval Psycoy
37
229
unknown
1
The official evaluation suite and dynamic data release for MixEval.
mixeval benchmark evaluation llm-inference llm-evaluation benchmark-mixture foundation-models benchmarking-suite evaluation-framework large-language-model large-language-models benchmarking-framework large-multimodal-models llm-evaluation-framework
Created 2024-06-01
120 commits to main branch, last one about a month ago
diningphil
PyDGN diningphil
13
219
bsd-3-clause
7
A research library for automating experiments on Deep Graph Networks
deep-graph-networks evaluation-framework deep-learning-for-graphs
Created 2020-03-21
461 commits to main branch, last one 3 months ago
zeno-ml
zeno zeno-ml
11
215
mit
8
AI Data Management & Evaluation Platform
ai python evaluation data-science machine-learning evaluation-framework
This repository has been archived (exclude archived)
Created 2022-02-03
1,044 commits to main branch, last one about a year ago
aiverify-foundation
moonshot aiverify-foundation
39
191
apache-2.0
7
Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
llm red-teaming benchmarking trustworthy-ai evaluation-framework
Created 2023-12-14
1,935 commits to main branch, last one a day ago
lartpang
PySODEvalToolkit lartpang
20
168
mit
3
PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection
mae latex metrics python3 fm-curve pr-curve saliency e-measure evaluator f-measure s-measure evaluation co-saliency evaluation-metrics saliency-detection evaluation-framework metrics-visualization salient-object-detection co-salient-object-detection camouflaged-object-detection
Created 2020-07-08
66 commits to master branch, last one 2 months ago
bijington
expressive bijington
26
165
mit
10
Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.
parsing xamarin evaluation netstandard hacktoberfest cross-platform expression-parser evaluation-framework expression-evaluator
Created 2016-06-13
291 commits to main branch, last one 2 months ago
ServiceNow
AgentLab ServiceNow
32
160
other
6
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
llm agents benchmark prompting llm-agents web-agents evaluation-framework
Created 2024-05-21
500 commits to main branch, last one 20 hours ago
empirical-run
empirical empirical-run
12
151
mit
6
Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application
llm llmops testing llm-inference test-automation testing-framework evaluation-framework
Created 2024-03-14
218 commits to main branch, last one 7 months ago
symflower
eval-dev-quality symflower
5
139
mit
4
DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
llms evaluation software-quality evaluation-framework software-development
Created 2024-03-28
714 commits to main branch, last one 2 days ago
AI21Labs
lm-evaluation AI21Labs
14
124
apache-2.0
5
Evaluation suite for large-scale language models.
language-model evaluation-framework
Created 2021-08-05
7 commits to main branch, last one 3 years ago
nlp-uoregon
mlmm-evaluation nlp-uoregon
17
115
apache-2.0
4
Multilingual Large Language Models Evaluation Benchmark
nlp datasets evaluation multilingual language-model evaluation-datasets evaluation-framework large-language-models natural-language-processing
Created 2023-08-07
18 commits to main branch, last one about a year ago
tsenst
CrowdFlow tsenst
22
110
gpl-3.0
7
Optical Flow Dataset and Benchmark for Visual Crowd Analysis
dataset tracking optical-flow trajectories crowd-analysis crowd-counting benchmark-suite computer-vision video-analytics synthetic-images video-processing motion-estimation video-surveillance evaluation-framework multi-object-tracking tracking-by-detection tub-crowdflow-dataset
Created 2018-09-10
40 commits to master branch, last one about a year ago
microsoft
eureka-ml-insights microsoft
14
104
apache-2.0
16
A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.
ai llm mllm machine-learning evaluation-framework artificial-intelligence
Created 2024-07-18
47 commits to main branch, last one 2 days ago
haeyeoni
lidar_slam_evaluator haeyeoni
16
91
unknown
2
LiDAR SLAM comparison and evaluation framework
slam lidar-slam evaluation-framework
Created 2021-07-26
17 commits to main branch, last one 3 years ago
codefuse-ai
codefuse-evaluation codefuse-ai
12
81
other
3
Industrial-level evaluation benchmarks for Coding LLMs in the full life-cycle of AI native software developing.企业级代码大模型评测体系,持续开放中
lcc codefuse codetranseval code-evaluation codecommenteval repository-eval evaluation-framework
Created 2023-09-28
27 commits to master branch, last one 11 months ago
BMW-InnovationLab
SORDI-AI-Evaluation-GUI BMW-InnovationLab
3
76
apache-2.0
3
This repository allows you to evaluate a trained computer vision model and get general information and evaluation metrics with little configuration.
ai bmw sordi docker python dataset no-code rest-api evaluation tensorflow deeplearning synthetic-data computer-vision evaluation-framework
Created 2022-10-28
10 commits to main branch, last one about a year ago
nouhadziri
DialogEntailment nouhadziri
5
74
mit
7
The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"
bert dialogue-evaluation evaluation-framework natural-language-inference
Created 2019-04-13
10 commits to master branch, last one 3 months ago
pyrddlgym-project
pyRDDLGym pyrddlgym-project
17
73
mit
8
A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.
gym rddl planner planners planning gymnasium simulator simulation visualizer model-based benchmarking rddl-domains visualisation visualization gym-environments benchmark-framework evaluation-framework benchmarking-framework reinforcement-learning planning-domain-definition-language
Created 2022-07-10
2,065 commits to main branch, last one 24 days ago
kaiko-ai
eva kaiko-ai
4
72
apache-2.0
6
Evaluation framework for oncology foundation models (FMs)
oncology machine-learning foundation-models evaluation-framework
Created 2024-01-16
322 commits to main branch, last one a day ago
pentoai
vectory pentoai
0
71
mit
5
Vectory provides a collection of tools to track and compare embedding versions.
deep-learning embedding-python embedding-vectors deep-neural-networks evaluation-framework embeddings-similarity
Created 2022-09-30
45 commits to main branch, last one 2 years ago
jinzhuoran
RWKU jinzhuoran
5
63
unknown
3
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024
benchmark forgetting unlearning privacy-protection adversarial-attacks evaluation-framework large-language-models right-to-be-forgotten membership-inference-attack natural-language-processing
Created 2024-05-28
13 commits to main branch, last one 2 months ago