Skip to main content

Evaluate Java AI Agents
with Confidence

JUnit 5-native · Local-first · Framework-agnostic
25+ metrics · LLM-as-judge · Zero SaaS required

Get Started →View Metrics

Write your first evaluation in minutes

Add the dependency, annotate your test, and AgentEval handles the rest. Scores are reported in the JUnit output you already know.

Full Quickstart Guide →
RefundAgentTest.java
@ExtendWith(AgentEvalExtension.class)
class RefundAgentTest {

    @Test
    @AgentTest
    @Metric(value = AnswerRelevancy.class, threshold = 0.7)
    @Metric(value = Faithfulness.class, threshold = 0.8)
    void shouldAnswerRefundQuestions() {
        var testCase = AgentTestCase.builder()
            .input("What is our refund policy?")
            .actualOutput(agent.run("What is our refund policy?"))
            .retrievalContext(List.of(
                "Customers may request a full refund within 30 days of purchase."
            ))
            .build();

        AgentAssertions.assertThat(testCase)
            .meetsMetric(new AnswerRelevancy(0.7))
            .meetsMetric(new Faithfulness(0.8));
    }
}

Everything you need to evaluate AI agents

📊

25+ Built-in Metrics

Response quality, RAG evaluation, agent tool use, multi-turn conversations. Every metric returns a 0–1 score with a human-readable reason.

🧪

JUnit 5 Native

@AgentTest, @Metric, @DatasetSource annotations. Evaluations run as ordinary test methods — no new tooling to learn.

⚖️

LLM-as-Judge

OpenAI, Anthropic, Google, Ollama, Azure, Bedrock — plug in any provider. Research-backed G-Eval prompts, fully inspectable.

🔌

Framework Agnostic

Evaluates any Java AI agent. Optional auto-capture for Spring AI, LangChain4j, LangGraph4j, and MCP.

🔒

Local-First

No cloud, no SaaS, no data leaves your machine. Pair with Ollama for fully private evaluations.

🚀

CI/CD Ready

Standard JUnit XML output works with Jenkins, GitHub Actions, and GitLab CI. Maven and Gradle plugin included.

Install

Maven

<dependency>
  <groupId>org.byteveda.agenteval</groupId>
  <artifactId>agenteval-junit5</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

Gradle (Kotlin DSL)

testImplementation("org.byteveda.agenteval:agenteval-junit5:0.1.0")