Hi there, I am building a lot of small proof of concepts and ideas for some time now. Some of them are getting attention in my company and now I am fully working on improving them. Most are RAG based systems or agentic systems to automate tasks. As I am more and more updating and trying different thing I am now in a situation where I would like to get Figures… Metrics to tell if it is better or not than before in the specific goal of the tool (data retrieval, content creation) So I would like to build specific benchmarks on top of the tools. Can someone point me a framework, technique, course or any resources to build such benchmarks? Thank you so much …

It’s a field I know nothing about, but it seems that Ragas is famous. [image] Use RAGAS with huggingface LLM Intermediate Hello everybody, I want to use the RAGAS lib to evaluate my RAG pipeline. The evaluation model should be a huggingface model like Llama-2, Mistral, Gemma and more. How can I implement it with the named library or is there another solution? The examples by the team Examples by RAGAS team aren’t helpful for me, because they doesn’t show, how to use specific Huggingface model. Also a specifc langchain pipeline doesn’t work, because it isn’t the required class. Best regards Christian

seems promising indeed ! I didn’t know about that, thanks a lot for the idea!!

Thanks a lot, it is exactly what I need… You are awesome @John6666

A fluke, but I’ll still be happy if I hit it!:laughing:

Custom BenchMark creation

Intermediate

John6666 January 31, 2025, 5:01am 3

It’s a field I know nothing about, but it seems that Ragas is famous.

Topic		Replies	Views
Use RAGAS with huggingface LLM Intermediate	17	11501	March 17, 2025
Evaluating RAG only with open-source Intermediate	1	704	May 24, 2024
We built RapidFire AI RAG: 16–24x faster RAG experimentation + live evals (try it in Colab) Show and Tell	0	116	December 23, 2025
How does RAGAS work? Beginners	1	792	March 4, 2024
Looking for simple ways to evaluate an AI agent Beginners	3	1013	October 6, 2026

Custom BenchMark creation

Related topics