Search Results - RepositoryStats

deepeval confident-ai

327

4.0k

apache-2.0

23

The LLM Evaluation Framework

llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-metrics llm-evaluation-framework

Created 2023-08-10

3,938 commits to main branch, last one 20 hours ago

agentops AgentOps-AI

232

2.4k

mit

27

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen

ai llm groq agent evals crewai ollama openai autogen mistral agentops anthropic langchain cost-estimation evaluation-metrics

Created 2023-08-15

554 commits to main branch, last one a day ago

AB3DMOT xinshuoweng

403

1.7k

other

49

(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"

kitti 3d-mot 3d-multi kitti-3d robotics tracking real-time evaluation 3d-tracking computer-vision machine-learning 2d-mot-evaluation evaluation-metrics multi-object-tracking 3d-multi-object-tracking

Created 2019-06-19

221 commits to master branch, last one about a year ago

evaluation-guidebook huggingface

55

903

other

8

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

llm tutorial guidebook evaluation machine-learning evaluation-metrics large-language-models

Created 2024-10-09

72 commits to main branch, last one 14 days ago

lighteval huggingface

106

881

mit

29

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

evaluation huggingface evaluation-metrics evaluation-framework

Created 2024-01-26

266 commits to main branch, last one 21 hours ago

rliable google-research

48

785

apache-2.0

11

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

rl google benchmarking machine-learning evaluation-metrics reinforcement-learning

Created 2021-08-20

72 commits to master branch, last one 4 months ago

OCTIS MIND-Lab

106

737

mit

15

OCTIS: Comparing Topic Models is Simple! A python package to optimize and evaluate topic models (accepted at EACL2021 demo track)

nlp nlproc nlp-library topic-models topic-modeling evaluation-metrics neural-topic-models bayesian-optimization hyperparameter-search hyperparameter-tuning latent-semantic-analysis hyperparameter-optimization latent-dirichlet-allocation natural-language-processing non-negative-matrix-factorization

Created 2020-03-13

1,150 commits to master branch, last one 4 months ago

jiwer jitsi

98

653

apache-2.0

15

Evaluate your speech-to-text system with similarity measures such as word error rate (WER)

wer python3 speech-to-text word-error-rate evaluation-metrics automatic-speech-recognition

Created 2018-06-19

90 commits to master branch, last one about a month ago

image-similarity-measures nekhtiari

68

591

mit

13

:chart_with_upwards_trend: Implementation of eight evaluation metrics to access the similarity between two images. The eight metrics are as follows: RMSE, PSNR, SSIM, ISSM, FSIM, SRE, SAM, and UIQ.

p1 image metrics processing machine-learning evaluation-metrics

Created 2020-04-06

98 commits to master branch, last one 4 months ago

COMET Unbabel

82

515

apache-2.0

20

A Neural Framework for MT Evaluation

nlp machine-learning evaluation-metrics machine-translation artificial-intelligence natural-language-processing

Created 2020-05-28

554 commits to master branch, last one 12 days ago

ranx AmenRa

27

496

mit

11

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

numba python comparison evaluation metasearch data-fusion rank-fusion score-fusion ranking-metrics evaluation-metrics recommender-systems information-retrieval information-retrieval-metrics information-retrieval-evaluation

Created 2020-06-02

284 commits to master branch, last one 5 months ago

pynlpl proycon

68

479

gpl-3.0

31

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks su...

nlp folia python library linguistics nlp-library text-processing machine-learning search-algorithms evaluation-metrics language-modelling computational-linguistics natural-language-processing

Created 2010-07-06

2,160 commits to master branch, last one about a year ago

continuous-eval relari-ai

31

455

apache-2.0

4

Data-Driven Evaluation for LLM-Powered Applications

rag llmops llm-evaluation evaluation-metrics evaluation-framework information-retrieval retrieval-augmented-generation

Created 2023-12-08

96 commits to main branch, last one 3 months ago

SpecVQGAN v-iashin

39

353

mit

8

Source code for "Taming Visually Guided Sound Generation" (Oral at the BMVC 2021)

gan vas bmvc audio video vqvae melgan pytorch vggsound multi-modal transformer video-features audio-generation evaluation-metrics video-understanding

Created 2021-10-17

24 commits to main branch, last one 5 months ago

factCC salesforce

31

289

bsd-3-clause

10

Resources for the "Evaluating the Factual Consistency of Abstractive Text Summarization" paper

evaluation-metrics text-summarization

Created 2019-10-23

5 commits to master branch, last one 3 years ago

tonic_validate TonicAI

27

271

mit

14

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

llm rag llms llmops evaluation-metrics evaluation-framework large-language-models retrieval-augmented-generation

Created 2023-10-23

343 commits to main branch, last one about a month ago

LRV-Instruction FuxiaoLiu

13

262

bsd-3-clause

11

[ICLR'24] Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

gpt vqa iclr gpt-4 llama llava vicuna vision chatgpt iclr2024 evaluation multimodal hallucination object-detection foundation-models evaluation-metrics prompt-engineering vision-and-language

Created 2023-06-15

366 commits to main branch, last one 9 months ago

pyrouge bheinzerling

71

251

mit

4

A Python wrapper for the ROUGE summarization evaluation package

nlp rouge summarization evaluation-metrics

Created 2014-01-14

38 commits to master branch, last one 5 years ago

generative-evaluation-prdc clovaai

28

249

mit

9

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

icml recall fidelity icml2020 diversity icml-2020 precision evaluation deep-learning generative-model machine-learning evaluation-metrics generative-adversarial-network

Created 2020-02-21

6 commits to master branch, last one 4 years ago

athina-evals athina-ai

14

239

unknown

5

Python SDK for running evaluations on LLM generated responses

llmops llm-ops llm-eval evaluation llm-evaluation evaluation-metrics evaluation-framework llm-evaluation-toolkit

Created 2023-11-22

635 commits to main branch, last one a day ago

Twitter-Sentiment-Analysis sharmaroshan

120

234

gpl-3.0

4

It is a Natural Language Processing Problem where Sentiment Analysis is done by Classifying the Positive tweets from negative tweets by machine learning models for classification, text mining, text a...

eda nlp hashtags wordcloud bag-of-words datacleaning data-analysis classification count-vectorizer cross-validation machine-learning data-visualization evaluation-metrics sentiment-analysis

Created 2019-03-31

10 commits to master branch, last one 5 years ago

NER-Evaluation davidsbatista

48

217

mit

11

An implementation of a full named-entity evaluation metrics based on SemEval'13 Task 9 - not at tag/token level but considering all the tokens that are part of the named-entity

ner semeval crfsuite semeval-2013 ner-evaluation notebook-jupyter evaluation-metrics named-entity-recognition

Created 2018-05-17

65 commits to master branch, last one 5 months ago

foundation-model-benchmarking-tool aws-samples

32

211

mit-0

8

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.

g5 g6 p5 g6e p4d llama2 llama3 bedrock trainium benchmark sagemaker inferentia benchmarking generative-ai foundation-models evaluation-metrics

Created 2024-01-09

1,315 commits to main branch, last one 9 hours ago

Awesome-Evaluation-of-Visual-Generation ziqihuangg

12

210

unknown

7

A list of works on evaluation of visual generation models, including evaluation metrics, models, and systems

awesome benchmark evaluation image-generation video-generation evaluation-system generative-models evaluation-metrics

Created 2024-03-16

80 commits to master branch, last one 3 months ago

CLEval clovaai

28

184

mit

10

CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks

end-to-end-ocr text-detection text-recognition evaluation-metrics text-detection-recognition

Created 2020-05-27

15 commits to master branch, last one about a year ago

awesome-diffusion-v2v wenhao728

6

168

mit

6

Awesome diffusion Video-to-Video (V2V). A collection of paper on diffusion model-based video editing, aka. video-to-video (V2V) translation. And a video editing benchmark code.

survey benchmark video-editing video-to-video diffusion-models evaluation-metrics

Created 2024-06-15

20 commits to main branch, last one 20 days ago