Evaluations - Overview

Middleware LLM Evaluations measures LLM and agent outputs against defined criteria, including accuracy, safety, relevance, helpfulness, tool usage, and goal completion.

Evaluation results attach to the corresponding trace span, linking a quality score with the input, output, and operation being evaluated. This helps investigate individual failures and compare quality after changes to prompts, models, or application logic.

A typical evaluation workflow is:

LLM / Agent Response ↓ Evaluation ↓ Score + Verdict + Reasoning ↓ Trace Investigation ↓ Metrics and Alerts ↓ Improve Application

Why Use Evaluations#

Traditional application metrics can tell you whether an LLM request completed, how long it took, and how many tokens it consumed.

They cannot tell you whether the response was actually useful or correct.

Evaluations add a measurable quality signal to LLM output. You can define criteria such as:

• Is the response correct?
• Is the response relevant to the user's question?
• Is the generated content toxic?
• Did the agent complete the requested task?
• Did the agent select the correct tool?
• Is the answer grounded in the provided context?

This makes response quality measurable alongside latency, tokens, cost, and other production signals.

How Evaluations Work#

Every evaluation creates a result associated with the span being evaluated.

An evaluation contains:

• Name: Identifies what is being evaluated, such as toxicity or goal_completeness.
• Value: Boolean, numeric score, or categorical label.
• Verdict: Optional pass or fail result based on the configured acceptance criteria.
• Reasoning: Optional explanation of why the evaluator produced the result.

The result is associated with the original span, you can review the evaluation next to the input and output that produced it.

Evaluation Types#

Middleware supports both LLM-as-a-Judge evaluations and deterministic evaluations.

LLM-as-a-Judge#

LLM-as-a-Judge uses a model to assess an LLM or agent response against a defined rubric. The score reflects that rubric and the evaluation input; it is a quality signal to investigate alongside the underlying response.

Use it when evaluation requires judgment rather than an exact comparison.

Common use cases include:

• Toxicity detection
• Prompt-injection detection
• Topic relevance
• Sentiment
• Response quality
• Goal completion
• Tool selection
• Tool argument correctness
• Groundedness

You define the evaluation criteria in natural language, and the judge model applies those criteria to matching spans.

Deterministic Evaluations#

Deterministic evaluations use application logic instead of another LLM.

Examples include:

• Regex matching
• JSON validation
• Response length checks
• Exact-value comparison
• Fuzzy comparison against an expected result

Use deterministic checks when success can be determined objectively.

Run Evaluations#

Middleware supports two ways to evaluate your LLM application:

Both approaches attach evaluation results to spans using the same result format.

Server-Side Evaluations#

Server-side evaluations run against matching trace spans in Middleware.

Use them when you want to continuously evaluate production traffic without adding evaluation logic to your application.

Go to:LLM Observability → Evaluation → Add Evaluator

You can:

• Start with a predefined evaluator template.
• Create a custom LLM-as-a-Judge evaluator.

Built-In Evaluation Templates#

Middleware provides predefined evaluators for common LLM and agent quality checks.

Each template provides a predefined judge prompt, output type, and acceptance criteria. You can select the judge model and modify the prompt or criteria before publishing the evaluator.

Create a Custom Evaluator#

Select LLM-as-a-Judge when you need evaluation criteria specific to your application.

Configure the evaluator using the following sections.

Details#

Define:

• Name: Unique name for the evaluator. This is also used as the evaluation score name.
• Judge Model: Model used to evaluate matching spans.

Scope the Evaluator#

Use Scope to control which production spans are evaluated.

You can configure:

• Service Name: Limit evaluation to one or more services.
• Sampling Rate: Percentage of matching spans to evaluate.
• Evaluate On: Span attribute filters that determine which spans qualify.

For example: gen_ai.operation.name = chat

Use Add Rule to combine multiple conditions.

If no Evaluate On rules are configured, every span within the configured scope qualifies for evaluation.

Evaluate Root Spans#

To evaluate only root spans at the trace level, add an Evaluate On rule for: parent_span_id and leave its value empty. This limits the evaluator to spans without a parent.

Configure the Evaluation Prompt#

Define the instructions that the judge model uses to evaluate each span.

The prompt contains:

• System Prompt: Defines the judge's role and evaluation criteria.
• User Prompt: Provides the data that should be evaluated.

You can reference data from the evaluated span using template variables:

For example:

1Span Input: {{span.input}}
2Span Output: {{span.output}}

You can also map prompt variables directly to other span attributes by their attribute key.

Configure the Evaluation Output#

Use Structured Output to define the result the judge should return.

Middleware supports three output types:

Category#

Returns one value from a predefined set.

For example: ON_TOPIC, OFF_TOPIC ,NEUTRAL

Define which categories should be treated as passing results.

Boolean#

Returns: true or: false

Configure whether true or false represents a passing result.

Score#

Returns a numeric score within a defined range.

For example: 1–5

Configure a minimum or maximum threshold to determine whether the evaluation passes.

Include a reasoning field when you want the judge's explanation stored alongside the evaluation result.

Publish the Evaluator#

After configuring the evaluator, select Save and Publish.

Once published, Middleware starts evaluating new spans that match its scope and sampling configuration.

You can also save an evaluator as a draft and publish it later.

View Evaluation Results#

Once an evaluator starts processing spans, open a trace in LLM Traces.

Select the evaluated span and open the Evaluation tab.

You can review:

• Evaluation name
• Value or score
• Pass/fail verdict
• Judge reasoning

This lets you inspect the evaluation result alongside the original LLM or agent operation.

Client-Side Evaluations#

Use client-side evaluations when evaluation logic needs to run inside your application.

This is useful for:

• Deterministic checks
• Custom business rules
• RAG-aware evaluations
• Offline evaluation
• CI evaluation
• Evaluations that depend on application-specific data

Client-side evaluations can submit a value calculated by your application or use your own LLM judge.

Results appear in Middleware using the same evaluation format as server-side evaluations, so both types can be investigated and monitored together.

Monitor Evaluation Results#

Evaluation results are exported as both correlated records and metrics.

This lets you investigate individual failures while also monitoring quality trends across production traffic.

Evaluation Metrics#

Middleware provides evaluation metrics including:

• gen_ai.evaluations.count — number of evaluation runs.
• gen_ai.evaluations.score — evaluation score.
• gen_ai.evaluations.outcome — evaluation outcome such as pass, fail, or error.
• gen_ai.evaluations.cost.usd — estimated evaluation cost when available.

Use these metrics to:

• Monitor pass rate.
• Track average evaluation scores.
• Compare quality across services or models.
• Detect quality regressions.
• Monitor evaluation cost.
• Create alerts when quality drops.

Common Evaluation Workflows#

Different LLM applications can evaluate different stages of their workflow.

LLM Applications#

Evaluate whether the generated response is accurate, relevant, safe, or useful.

Agents#

Evaluate whether the agent selected the correct tool, provided valid tool arguments, completed the requested goal, and produced an appropriate final response.

RAG Pipelines#

Evaluate routing, retrieval relevance, and generation separately to determine which stage caused a low-quality response.

Because evaluations attach to individual spans, you can evaluate the specific operation responsible for each part of the workflow.

Troubleshooting#

Evaluation Results Are Not Appearing#

Verify that:

• The evaluator is Published, not saved as a draft.
• The configured service name matches your trace spans.
• Evaluate On rules match the intended spans.
• The sampling rate is high enough for matching spans to be selected.

When sampling is below 100%, not every matching span receives an evaluation result.

Judge Returns an Error or No Score#

Verify that the configured judge model is supported.

Also check the available evaluation token allowance. Evaluations stop running if the included token allowance has been exhausted.

Input or Output Is Empty#

Variables such as:

1{{span.input}}
2{{span.output}}

read their values from the span's GenAI attributes.

If the span does not contain the corresponding GenAI input or output data, the evaluator will not have that content available for evaluation.