36 results found Sort:

EleutherAI
lm-evaluation-harness EleutherAI
1.5k
5.6k
mit
36
A framework for few-shot evaluation of language models.
transformer language-model evaluation-framework
Created 2020-08-28
3,415 commits to main branch, last one 23 hours ago
promptfoo
promptfoo promptfoo
223
3.3k
mit
18
Test your prompts, agents, and RAGs. Use LLM evals to improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with comman...
ci llm rag cicd ci-cd llmops prompts testing llm-eval evaluation llm-evaluation prompt-testing prompt-engineering evaluation-framework llm-evaluation-framework
Created 2023-04-28
1,331 commits to main branch, last one 9 hours ago
confident-ai
deepeval confident-ai
153
2.2k
apache-2.0
17
The LLM Evaluation Framework
llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework
Created 2023-08-10
2,995 commits to main branch, last one 4 hours ago
MaurizioFD
RecSys2019_DeepLearning_Evaluation MaurizioFD
250
981
agpl-3.0
39
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
bpr knn bprmf bprslim funksvd deep-learning neural-network slimelasticnet hyperparameters reproducibility matrix-completion recommender-system evaluation-framework matrix-factorization recommendation-system reproducible-research collaborative-filtering hybrid-recommender-system recommendation-algorithms content-based-recommendation
Created 2019-04-02
63 commits to master branch, last one 2 years ago
huggingface
lighteval huggingface
53
450
mit
32
LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.
evaluation huggingface evaluation-metrics evaluation-framework
Created 2024-01-26
126 commits to main branch, last one about a month ago
relari-ai
continuous-eval relari-ai
19
359
apache-2.0
4
Open-Source Evaluation for GenAI Application Pipelines
rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation
Created 2023-12-08
90 commits to main branch, last one a day ago
TonicAI
tonic_validate TonicAI
25
221
mit
12
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
llm rag llms llmops evaluation-metrics evaluation-framework large-language-models retrieval-augmented-generation
Created 2023-10-23
340 commits to main branch, last one 2 days ago
diningphil
PyDGN diningphil
12
213
bsd-3-clause
7
A research library for automating experiments on Deep Graph Networks
deep-graph-networks evaluation-framework deep-learning-for-graphs
Created 2020-03-21
447 commits to main branch, last one a day ago
zeno-ml
zeno zeno-ml
9
212
mit
8
AI Data Management & Evaluation Platform
ai python evaluation data-science machine-learning evaluation-framework
This repository has been archived (exclude archived)
Created 2022-02-03
1,044 commits to main branch, last one 8 months ago
bijington
expressive bijington
25
159
mit
9
Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.
parsing xamarin evaluation netstandard hacktoberfest cross-platform expression-parser evaluation-framework expression-evaluator
Created 2016-06-13
288 commits to main branch, last one 27 days ago
lartpang
PySODEvalToolkit lartpang
20
151
mit
3
PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection
mae latex metrics python3 fm-curve pr-curve saliency e-measure evaluator f-measure s-measure evaluation co-saliency evaluation-metrics saliency-detection evaluation-framework metrics-visualization salient-object-detection co-salient-object-detection camouflaged-object-detection
Created 2020-07-08
63 commits to master branch, last one about a month ago
athina-ai
athina-evals athina-ai
11
147
unknown
5
Python SDK for running evaluations on LLM generated responses
llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit
Created 2023-11-22
445 commits to main branch, last one 2 days ago
Psycoy
MixEval Psycoy
17
143
unknown
1
The official evaluation suite and dynamic data release for MixEval.
mixeval benchmark evaluation llm-inference llm-evaluation benchmark-mixture foundation-models benchmarking-suite evaluation-framework large-language-model large-language-models benchmarking-framework large-multimodal-models llm-evaluation-framework
Created 2024-06-01
35 commits to main branch, last one 10 hours ago
empirical-run
empirical empirical-run
10
140
mit
5
Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application
llm llmops testing llm-inference test-automation testing-framework evaluation-framework
Created 2024-03-14
218 commits to main branch, last one 29 days ago
AI21Labs
lm-evaluation AI21Labs
13
122
apache-2.0
5
Evaluation suite for large-scale language models.
language-model evaluation-framework
Created 2021-08-05
7 commits to main branch, last one 2 years ago
tsenst
CrowdFlow tsenst
22
104
gpl-3.0
7
Optical Flow Dataset and Benchmark for Visual Crowd Analysis
dataset tracking optical-flow trajectories crowd-analysis crowd-counting benchmark-suite computer-vision video-analytics synthetic-images video-processing motion-estimation video-surveillance evaluation-framework multi-object-tracking tracking-by-detection tub-crowdflow-dataset
Created 2018-09-10
40 commits to master branch, last one 10 months ago
aiverify-foundation
moonshot aiverify-foundation
15
88
apache-2.0
4
Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
llm red-teaming benchmarking trustworthy-ai evaluation-framework
Created 2023-12-14
987 commits to main branch, last one 10 days ago
haeyeoni
lidar_slam_evaluator haeyeoni
15
83
unknown
2
LiDAR SLAM comparison and evaluation framework
slam lidar-slam evaluation-framework
Created 2021-07-26
17 commits to main branch, last one 2 years ago
nouhadziri
DialogEntailment nouhadziri
5
75
mit
7
The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"
bert dialogue-evaluation evaluation-framework natural-language-inference
Created 2019-04-13
9 commits to master branch, last one 2 years ago
BMW-InnovationLab
SORDI-AI-Evaluation-GUI BMW-InnovationLab
3
74
apache-2.0
2
This repository allows you to evaluate a trained computer vision model and get general information and evaluation metrics with little configuration.
ai bmw sordi docker python dataset no-code rest-api evaluation tensorflow deeplearning synthetic-data computer-vision evaluation-framework
Created 2022-10-28
10 commits to main branch, last one 7 months ago
nlp-uoregon
mlmm-evaluation nlp-uoregon
13
66
apache-2.0
3
Multilingual Large Language Models Evaluation Benchmark
nlp datasets evaluation multilingual language-model evaluation-datasets evaluation-framework large-language-models natural-language-processing
Created 2023-08-07
18 commits to main branch, last one 10 months ago
pentoai
vectory pentoai
0
65
mit
5
Vectory provides a collection of tools to track and compare embedding versions.
deep-learning embedding-python embedding-vectors deep-neural-networks evaluation-framework embeddings-similarity
Created 2022-09-30
45 commits to main branch, last one about a year ago
codefuse-ai
codefuse-evaluation codefuse-ai
9
64
other
3
Industrial-level evaluation benchmarks for Coding LLMs in the full life-cycle of AI native software developing.企业级代码大模型评测体系,持续开放中
lcc codefuse codetranseval code-evaluation codecommenteval repository-eval evaluation-framework
Created 2023-09-28
27 commits to master branch, last one 5 months ago
ashafaei
OD-test ashafaei
11
61
mit
5
OD-test: A Less Biased Evaluation of Out-of-Distribution (Outlier) Detectors (PyTorch)
od-test pytorch deep-learning in-distribution anomaly-detection outlier-detection evaluation-framework
Created 2018-09-12
40 commits to master branch, last one 4 years ago
powerflows
powerflows-dmn powerflows
6
51
apache-2.0
9
Power Flows DMN - Powerful decisions and rules engine
dmn xml feel java mvel yaml rules groovy kotlin dmn-model dmn-engine evaluation javascript kotlin-dsl rule-engine rules-engine decision-engine decision-tables evaluation-framework
Created 2018-08-03
253 commits to master branch, last one 5 years ago
symflower
eval-dev-quality symflower
2
50
mit
3
DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
llms evaluation software-quality evaluation-framework software-development
Created 2024-03-28
375 commits to main branch, last one 2 days ago
SpikeInterface
spiketoolkit SpikeInterface
31
49
mit
9
Python-based tools for pre-, post-processing, validating, and curating spike sorting datasets.
neuroscience spike-sorting electrophysiology evaluation-framework
This repository has been archived (exclude archived)
Created 2018-09-28
1,509 commits to master branch, last one 2 years ago
kaiko-ai
eva kaiko-ai
2
48
apache-2.0
5
Evaluation framework for oncology foundation models (FMs)
oncology machine-learning foundation-models evaluation-framework
Created 2024-01-16
228 commits to main branch, last one 2 days ago
sb-ai-lab
Sim4Rec sb-ai-lab
1
44
apache-2.0
6
Simulator for training and evaluation of Recommender Systems
rl-training user-modeling recommendation synthetic-data recommender-system evaluation-framework
Created 2022-11-28
29 commits to main branch, last one 8 months ago
yupidevs
pactus yupidevs
0
43
mit
1
Framework to evaluate Trajectory Classification Algorithms
python trajectory transformers classification trajectory-analysis evaluation-framework classification-models
Created 2022-10-01
158 commits to main branch, last one 9 months ago