LLM Runtime Monitoring: OpenTelemetry GenAI and Production Debugging | AI

Loading…

LLM Runtime Monitoring: OpenTelemetry GenAI and Production Debugging | AI | graphwiz.ai

LLM applications introduce new failure modes. A 200 OK response can hide a hallucination, a degraded prompt template can silently reduce quality, and a model provider can silently change behavior. Runtime monitoring — tracking token throughput, latency, cache hit ratios, and error rates — tells you whether your LLM stack is healthy, not just whether it's responding.

This article complements LLM Observability and Eval-Driven Development by focusing on runtime instrumentation, streaming metrics, LLM gateway patterns, and production debugging.

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry added standardized semantic conventions for GenAI in v1.37+. These conventions define a common set of attributes for LLM calls, making traces from different tools and providers comparable.

The GenAI Trace Model

Every LLM call produces a span with standardized attributes:

Attribute (OTel GenAI)	Description	Example
`gen_ai.operation.name`	Operation type (e.g., "chat", "completion")	"chat"
`gen_ai.request.model`	Model identifier	"gpt-4o"
`gen_ai.response.model`	Actual model used (may differ from request)	"gpt-4o-2024-11-20"
`gen_ai.request.temperature`	Temperature parameter	"0.7"
`gen_ai.request.max_tokens`	Max tokens requested	"2048"
`gen_ai.completion_count`	Number of completions	"1"
`gen_ai.prompt.tokens`	Prompt token count	"150"
`gen_ai.completion.tokens`	Completion token count	"420"
`gen_ai.usage.total_tokens`	Total tokens	"570"
`gen_ai.response.finish_reason`	Completion reason ("stop", "length", "content_filter")	"stop"
`gen_ai.response.id`	Provider response ID	"chatcmpl-abc123"
`gen_ai.system`	System (provider)	"openai"
`gen_ai.token_type`	Token pricing tier	"prompt" / "completion"

Instrumenting with OpenTelemetry GenAI

from openllmetry.sdk import OpenLLMetery

# Initialize OTel with GenAI instrumentation
otel = OpenLLMetery(
    service_name="llm-runtime-monitoring",
    environment="production",
    endpoint="https://otel.yourdomain.com:4317",
)

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

@otel.instrument_llm_calls()
def call_openai_chat(messages: list, temperature: float = 0.7):
    """This is automatically traced with GenAI attributes."""
    from openai import OpenAI
    client = OpenAI()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=temperature,
    )

    return response.choices[0].message.content

The @otel.instrument_llm_calls() decorator adds GenAI attributes to the span automatically — no manual instrumentation required.

Streaming Metrics with OpenTelemetry

Streaming adds time-to-first-token (TTFT) and inter-token latency metrics. These are critical for user experience.

Attribute	Description	Target (70th percentile)
`gen_ai.streaming.first_token_latency_ms`	Time from request start to first token	< 500ms
`gen_ai.streaming.inter_token_latency_ms`	Time between consecutive tokens	< 50ms
`gen_ai.streaming.token_count`	Total tokens in stream	varies
`gen_ai.streaming.end_reason`	Stream end reason ("stop", "length", "error")	"stop"

from openllmetry.sdk import StreamingCallback

def on_token(token: str, latency_ms: float):
    # Track inter-token latency
    metrics.record("gen_ai.streaming.inter_token_latency_ms", latency_ms)

    # Track cumulative tokens
    metrics.increment("gen_ai.streaming.token_count")

@otel.instrument_llm_calls(streaming_callback=on_token)
async def stream_chat(messages: list):
    from openai import AsyncOpenAI
    client = AsyncOpenAI()

    stream = await client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        stream=True,
    )

    async for chunk in stream:
        yield chunk.choices[0].delta.content

Storage Backend: Loki vs Prometheus

Use Case	Recommended Backend	Reason
High-cardinality traces (LLM prompts)	Loki	Efficient for log-like data with labels
Low-cardinality metrics (latency, cost)	Prometheus	Time-series optimized, alerting built-in
Both traces + metrics	Loki + Prometheus + Grafana	Best of both worlds

# loki-config.yaml
limits_config:
  # Allow high cardinality for LLM traces
  max_streams_per_user: 0
  max_global_streams_per_user: 0
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h # 7 days

schema_config:
  configs:
    - from: "2026-07-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  tsdb:
    # Local disk cache for recent traces
    path: /loki/tsdb
    retention:
      enabled: true
      range: 336h # 14 days

LLM Gateway Instrumentation

LLM gateways (LiteLLM, Portkey, Katara, Gateway.ai) add a routing and cost control layer on top of direct provider calls. Instrumenting the gateway gives you visibility into routing decisions, cache hits, and multi-provider latency.

LiteLLM Instrumentation

LiteLLM exposes Prometheus metrics and OpenTelemetry traces.

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_base: https://api.openai.com/v1
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      api_base: https://api.anthropic.com/v1

litellm_settings:
  # Enable Prometheus metrics at /metrics
  drop_params: true  # Don't log prompts to metrics
  set_verbose: false

  # Routing: cost-aware
  fallback: [{ "model": "claude-3-5-sonnet" }]

  # Caching
  cache:
    type: simple  # In-memory cache (production: Redis)
    ttl: 3600

Deploy:

# docker-compose.litellm.yml
services:
  litellm:
    image: ghcr.io/berriai/litellm:main
    command: ["--config", "/litellm_config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/litellm_config.yaml
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      REDIS_HOST: redis
      REDIS_PORT: 6379
    depends_on:
      - redis

Metrics from LiteLLM Gateway

LiteLLM exposes Prometheus metrics at /metrics:

# Key LiteLLM metrics
litellm_requests_total{model="gpt-4o", status="success"}
litellm_requests_total{model="gpt-4o", status="error"}

litellm_cache_hits_total{model="gpt-4o"}
litellm_cache_misses_total{model="gpt-4o"}

litellm_latency_seconds{model="gpt-4o", quantile="0.5"}
litellm_latency_seconds{model="gpt-4o", quantile="0.95"}
litellm_latency_seconds{model="gpt-4o", quantile="0.99"}

litellm_tokens_total{model="gpt-4o", token_type="prompt"}
litellm_tokens_total{model="gpt-4o", token_type="completion"}

Grafana dashboard query for cache hit rate:

# Cache hit rate per model
sum(rate(litellm_cache_hits_total[5m])) by (model)
/
sum(rate(litellm_cache_hits_total[5m])) by (model)
  + sum(rate(litellm_cache_misses_total[5m])) by (model)

Portkey Gateway Instrumentation

Portkey adds A/B testing, load balancing, and semantic caching.

// Portkey TypeScript SDK
import Portkey from "portkey";

const portkey = new Portkey({
  apiKey: process.env.PORTKEY_API_KEY,
  config: {
    mode: "single", // A/B testing vs single model
    targets: [
      {
        provider: "openai",
        api_key: process.env.OPENAI_API_KEY,
        fallback: [{ provider: "anthropic", api_key: process.env.ANTHROPIC_API_KEY }],
      },
    ],
    cache: {
      mode: "semantic", // Semantic caching (vs exact)
      max_age: 3600,
      semantic: {
        similarity_threshold: 0.85,
        embedding_model: "text-embedding-3-small",
      },
    },
  },
});

// This call is automatically traced with Portkey metrics
const response = await portkey.chat.completions.create({
  messages: [{ role: "user", content: "Hello" }],
  model: "gpt-4o",
});

Portkey exposes metrics at /v1/metrics:

portkey_requests_total (status, provider, cache_status)
portkey_latency_seconds (provider, quantile)
portkey_tokens_total (provider, token_type)
portkey_cache_hit_ratio (cache_type)

Multi-Tenant Cost Control

In multi-tenant SaaS applications, you need to enforce per-tenant budget limits to prevent cost blowouts. The pattern: middleware that checks budgets before routing requests.

Budget Enforcement Architecture

User Request
  ↓
Tenant Middleware (check budget)
  ↓
Budget OK → LLM Gateway (routing)
  ↓
Provider (OpenAI / Anthropic / etc)
  ↓
Response + Cost → Update Budget
  ↓
User Response

Budget Store Design

-- budget_allocations table
CREATE TABLE budget_allocations (
  tenant_id UUID PRIMARY KEY,
  monthly_limit_usd DECIMAL(10, 2) NOT NULL,
  monthly_spend_usd DECIMAL(10, 2) NOT NULL DEFAULT 0,
  alert_threshold_percent INTEGER NOT NULL DEFAULT 80,
  hard_limit_enabled BOOLEAN NOT NULL DEFAULT true,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
);

-- spending_history table (for reconciliation)
CREATE TABLE spending_history (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL REFERENCES budget_allocations(tenant_id),
  spent_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  model TEXT NOT NULL,
  prompt_tokens INTEGER NOT NULL,
  completion_tokens INTEGER NOT NULL,
  cost_usd DECIMAL(10, 4) NOT NULL,
  metadata JSONB,
  INDEX idx_spending_tenant_date (tenant_id, spent_at),
);

-- Create a materialized view for daily rollup
CREATE MATERIALIZED VIEW daily_spending AS
SELECT
  tenant_id,
  DATE(spent_at) AS spend_date,
  SUM(cost_usd) AS total_cost_usd,
  SUM(prompt_tokens) AS total_prompt_tokens,
  SUM(completion_tokens) AS total_completion_tokens
FROM spending_history
GROUP BY tenant_id, DATE(spent_at);

CREATE UNIQUE INDEX ON daily_spending (tenant_id, spend_date);

Budget Middleware

# budget_middleware.py
from fastapi import Request, HTTPException
import httpx
from decimal import Decimal

class BudgetMiddleware:
    def __init__(self, db_pool, litellm_base_url: str = "http://litellm:4000"):
        self.db = db_pool
        self.litellm_client = httpx.AsyncClient(base_url=litellm_base_url)

    async def __call__(self, request: Request, call_next):
        tenant_id = request.headers.get("X-Tenant-ID")
        if not tenant_id:
            raise HTTPException(status_code=400, detail="Missing X-Tenant-ID header")

        # Check budget before allowing request
        budget = await self._check_budget(tenant_id)
        if budget["monthly_spend_usd"] >= budget["monthly_limit_usd"]:
            if budget["hard_limit_enabled"]:
                raise HTTPException(
                    status_code=403,
                    detail=f"Tenant {tenant_id} exceeded monthly budget of ${budget['monthly_limit_usd']}",
                )

        # Forward request to LLM gateway
        response = await call_next(request)

        # Record spending after request completes
        cost = self._calculate_cost(response)
        await self._record_spend(tenant_id, cost)

        return response

    async def _check_budget(self, tenant_id: str) -> dict:
        async with self.db.acquire() as conn:
            row = await conn.fetchrow(
                """
                SELECT
                  monthly_limit_usd,
                  monthly_spend_usd,
                  hard_limit_enabled,
                  alert_threshold_percent
                FROM budget_allocations
                WHERE tenant_id = $1
                """,
                tenant_id,
            )
        return dict(row)

    def _calculate_cost(self, response: dict) -> dict:
        """Calculate cost based on LiteLLM response headers."""
        prompt_tokens = int(response.headers.get("X-Prompt-Tokens", 0))
        completion_tokens = int(response.headers.get("X-Completion-Tokens", 0))

        # Pricing (example for GPT-4o)
        prompt_cost = prompt_tokens * 0.0000025  # $2.50 per 1M tokens
        completion_cost = completion_tokens * 0.00001  # $10.00 per 1M tokens
        total_cost = prompt_cost + completion_cost

        return {
            "prompt_tokens": prompt_tokens,
            "completion_tokens": completion_tokens,
            "cost_usd": Decimal(str(total_cost)),
        }

    async def _record_spend(self, tenant_id: str, cost: dict):
        async with self.db.acquire() as conn:
            await conn.execute(
                """
                INSERT INTO spending_history
                  (tenant_id, model, prompt_tokens, completion_tokens, cost_usd)
                VALUES ($1, $2, $3, $4, $5)
                """,
                tenant_id,
                "gpt-4o",
                cost["prompt_tokens"],
                cost["completion_tokens"],
                cost["cost_usd"],
            )

            # Update monthly spend
            await conn.execute(
                """
                UPDATE budget_allocations
                SET
                  monthly_spend_usd = monthly_spend_usd + $1,
                  updated_at = NOW()
                WHERE tenant_id = $2
                """,
                cost["cost_usd"],
                tenant_id,
            )

Budget Reconciliation

Billing systems and LLM provider billing sometimes disagree. Run a reconciliation job:

# reconcile_billing.py
import httpx
import asyncpg

async def reconcile_billing(tenant_id: str):
    """Compare internal tracking vs provider billing."""

    # 1. Get internal spending
    internal_spending = await get_internal_spend(tenant_id)

    # 2. Get provider spending (via API)
    provider_spending = await get_provider_billing(tenant_id)

    # 3. Compare and alert on drift
    drift_pct = (provider_spending - internal_spending) / internal_spending * 100

    if abs(drift_pct) > 10:  # More than 10% drift
        await alert(
            severity="warning",
            message=f"Billing drift for tenant {tenant_id}: {drift_pct:.2f}%. "
                    f"Internal: ${internal_spending:.2f}, Provider: ${provider_spending:.2f}",
        )

Production Debugging

When an LLM application behaves unexpectedly in production, you need a systematic approach to debugging.

Debug Flowchart

User reports issue
  ↓
Check: Is it a latency spike?
  → Yes: Check provider status, gateway routing, cache hit ratio
  → No: Check quality / hallucinations
    ↓
Check: Is it a quality regression?
  → Yes: Run eval suite against production traces
    → If eval fails: Prompt drift or model drift
      → Roll back prompt
    → If eval passes: User expectation changed
  → No: Check error rate
    ↓
Check: Is it an error spike?
  → Yes: Check provider status, rate limits, context length
  → No: Check cost anomaly
    ↓
Check: Is it a cost anomaly?
  → Yes: Check per-tenant spend, cache hit ratio, model routing
  → No: Deep dive into traces

Debugging Latency Spikes

-- Find slow requests (>5s)
SELECT
  gen_ai.request.model,
  AVG(gen_ai.streaming.first_token_latency_ms) AS avg_ttft_ms,
  AVG(gen_ai.streaming.inter_token_latency_ms) AS avg_inter_token_ms,
  COUNT(*) AS request_count
FROM traces
WHERE gen_ai.streaming.first_token_latency_ms > 5000
  AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY gen_ai.request.model
ORDER BY avg_ttft_ms DESC;

If latency spikes correlate with a specific model:

Check provider status page (OpenAI / Anthropic / etc)
Check gateway routing (is traffic being rerouted to slower fallback?)
Check cache hit ratio (are caches missing, causing more provider calls?)

Debugging Quality Regressions

-- Compare eval scores over time
SELECT
  DATE(timestamp) AS date,
  AVG(eval_score) AS avg_score,
  COUNT(*) AS eval_count
FROM eval_results
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY DATE(timestamp)
ORDER BY date DESC;

If eval scores dropped:

Check if prompt version changed (via prompt-index.json)
Check if model version changed (provider may have silently updated)
Run a smoke test against both prompt versions

Debugging Cost Anomalies

-- Find high-cost tenants
SELECT
  tenant_id,
  SUM(cost_usd) AS total_cost_usd,
  SUM(prompt_tokens) AS total_prompt_tokens,
  SUM(completion_tokens) AS total_completion_tokens,
  COUNT(*) AS request_count
FROM spending_history
WHERE spent_at > NOW() - INTERVAL '24 hours'
GROUP BY tenant_id
ORDER BY total_cost_usd DESC
LIMIT 10;

If a tenant's cost spiked:

Check if they changed usage pattern (e.g., switched to longer prompts)
Check if gateway is routing to expensive models (vs cheaper ones)
Check cache hit ratio (maybe cache is failing, causing more provider calls)

Debugging Error Spikes

-- Find error types
SELECT
  error_type,
  COUNT(*) AS error_count,
  AVG(latency_ms) AS avg_latency_ms
FROM traces
WHERE status = "error"
  AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY error_type
ORDER BY error_count DESC;

Common error types and debugging steps:

Error Type	Likely Cause	Debugging Step
`rate_limit_exceeded`	Hit provider rate limit	Check token quota, implement backoff
`context_length_exceeded`	Prompt + completion > max tokens	Check prompt length, add truncation
`content_filter_triggered`	Safety filter blocked response	Review content moderation settings
`timeout`	Provider response timeout	Check gateway routing, increase timeout
`invalid_api_key`	API key invalid or rotated	Check credentials store

Tool Comparison

Tool	Self-Hosted?	OTel Support	LLM Gateway?	Best For
Langfuse	✅ Yes	✅ Yes	❌ No	Full tracing + eval (focus on eval)
Arize Phoenix	✅ Yes	✅ Yes	❌ No	Notebook-first, experiment tracking
Helicone	✅ Yes	✅ Yes	❌ No	Simple proxy-based observability
Braintrust	❌ Cloud	✅ Yes	❌ No	Enterprise, built-in eval
LangSmith	❌ Cloud	❌ No	❌ No	LangChain integration, prompt engineering
LiteLLM	✅ Yes	✅ Yes	✅ Yes	LLM gateway with routing + caching
Portkey	❌ Cloud	✅ Yes	✅ Yes	A/B testing, semantic caching
Katara	✅ Yes	✅ Yes	✅ Yes	Open-source LLM gateway with Redis cache
OpenLLMetry	✅ Yes	✅ Yes	❌ No	OTel instrumentation library

For most self-hosted setups, LiteLLM (gateway + routing) + Arize Phoenix (traces + eval) + Prometheus (metrics) is a comprehensive stack.

Cross-Links

LLM Observability and Eval-Driven Development — eval suites, prompt versioning, drift detection
vLLM Self-Hosted Inference Guide — production inference with Prometheus metrics built-in
Self-Hosted AI Stack — LiteLLM gateway in production
Production Observability Stack — Prometheus + Grafana + Loki for metrics and traces
Disaster Recovery for Self-Hosted — backup strategies for your eval baselines and budget store
n8n Automation on GB10 — workflow automation tools that can integrate with observability

Summary

Capability	Tool / Pattern
Standardized tracing	OpenTelemetry GenAI semantic conventions
Runtime metrics	Prometheus + Grafana (low-cardinality)
Streaming metrics	Loki (high-cardinality traces)
LLM gateway instrumentation	LiteLLM / Portkey / Katara
Multi-tenant budget enforcement	PostgreSQL + middleware + reconcile job
Latency debugging	Compare TTFT and inter-token latency
Quality debugging	Run eval suite against production traces
Cost debugging	Per-tenant spend analysis + cache hit ratio

Runtime monitoring fills the gap between "did the request succeed?" and "was the output good?". Without it, you cannot debug cost anomalies, latency spikes, or provider issues — and you cannot enforce per-tenant budgets in production.