LLM Observability - Overview

LLM Observability makes model calls first-class telemetry. One SDK setup, Middleware's own or OpenLIT and Traceloop over OpenTelemetry, and every call carries its model, prompt, completion, tokens, latency and cost into the same trace as the surrounding request. LLM monitoring charts that per model and per service, LLM evaluation scores the answers with LLM-as-judge and custom evaluators, and GPU monitoring shows the hardware underneath, so a slow response can be traced to a prompt, a provider or a saturated card. The Cookbooks get you from first call to a traced RAG pipeline without reading the spec.

Features

ConfigureAdd the Middleware, OpenLIT or Traceloop SDK to your appCookbooksWorked examples from one model call to a RAG pipelineLLM monitoringTokens, latency, cost and errors per model and per serviceLLM evaluationScore responses with LLM-as-judge and your own evaluatorsGPU monitoringUtilization, memory and temperature of your GPUs

Related Features

APMTraces, services, profiles and endpointsInfrastructureHosts, Containers, Kubernetes etc.DashboardsVisualise telemetry dataAlertsCreate alert rules and service level objectives