Metrics Overview

AgentEval provides 27 built-in metrics organized into five categories. All metrics return an EvalScore (0.0–1.0) with a human-readable reason.

Response Quality

Evaluate how well the agent's response answers the user's question.

Metric	Default Threshold	Requires LLM	Available Since
AnswerRelevancy	0.7	Yes	P0
Faithfulness	0.7	Yes	P0
Hallucination	0.5	Yes	P0
Toxicity	0.5	Yes	P0
Correctness (G-Eval)	0.5	Yes	P1
SemanticSimilarity	0.7	No (embeddings)	P1
BiasDetection	0.5	Yes	P1
Conciseness	0.5	Yes	P1
Coherence	0.7	Yes	P1

RAG Metrics

Evaluate the quality of Retrieval-Augmented Generation pipelines.

Metric	Default Threshold	Requires LLM	Available Since
ContextualPrecision	0.7	Yes	P1
ContextualRecall	0.7	Yes	P1
ContextualRelevancy	0.7	Yes	P1
RetrievalCompleteness	0.8	No	P1

Agent Metrics

Evaluate tool use, planning, and execution trajectories.

Metric	Default Threshold	Requires LLM	Available Since
ToolSelectionAccuracy	0.8	No	P0
TaskCompletion	0.5	Yes	P0
ToolArgumentCorrectness	0.8	No	P1
ToolResultUtilization	0.7	Yes	P1
PlanQuality	0.7	Yes	P1
PlanAdherence	0.7	Yes	P1
TrajectoryOptimality	0.5	Yes	P1
StepLevelErrorLocalization	0.5	Yes	P1

Conversation Metrics

Evaluate multi-turn conversation quality and coherence.

Metric	Default Threshold	Requires LLM	Available Since
ConversationCoherence	0.7	Yes	P1
ContextRetention	0.7	Yes	P1
TopicDriftDetection	0.5	Yes	P1
ConversationResolution	0.5	Yes	P1

Custom Metrics

Build your own metrics using the G-Eval framework or pure Java logic.

Type	When to Use
G-Eval Custom Metric	Any evaluation criteria expressible in natural language
Deterministic Metric	Rule-based checks — regex, JSON schema, keyword matching
Composite Metric	Combine multiple metrics with weighted scoring

Using Multiple Metrics

@Test
@AgentTest
@Metric(value = AnswerRelevancy.class, threshold = 0.7)
@Metric(value = Faithfulness.class, threshold = 0.8)
@Metric(value = ToolSelectionAccuracy.class, threshold = 0.9)
void comprehensiveEval() {
    var testCase = AgentTestCase.builder()
        .input("What is the status of order #12345?")
        .actualOutput(agent.run("What is the status of order #12345?"))
        .retrievalContext(retrievedDocs)
        .toolCalls(agent.getLastToolCalls())
        .expectedToolCalls(List.of(ToolCall.of("GetOrderStatus", Map.of("orderId", "12345"))))
        .build();

    AgentAssertions.assertThat(testCase)
        .meetsMetric(new AnswerRelevancy(0.7))
        .meetsMetric(new Faithfulness(0.8))
        .meetsMetric(new ToolSelectionAccuracy(0.9));
}

Programmatic Batch Evaluation

var metrics = List.of(
    new AnswerRelevancy(0.7),
    new Faithfulness(0.8),
    new ToolSelectionAccuracy(0.9)
);

EvalResults results = AgentEval.evaluate(dataset, metrics);
results.summary();       // prints console report
results.passRate();      // e.g. 0.94
results.averageScore();  // e.g. 0.87