Traces & observability

The trace / span model

A span is one unit of work with a start and end. A trace is the set of spans that share a traceId; each span points at its parentSpanId, so the trace reconstructs into a tree. The root span (no parent) is the request itself; children are the LLM calls, tool runs, and retrievals nested under it. This is the canonical span shape the whole system reads and writes:

traceId        string          groups spans into one trace
spanId         string          unique per span
parentSpanId?  string          links to the parent → builds the tree
name           string          operation name (e.g. "openai.chat")
startTime      number (ms)     epoch millis
endTime        number (ms)
duration       number (ms)     endTime - startTime
status         "ok" | "error" | "unset"
statusMessage? string          error detail when status = error
attributes     Record<string, string | number | boolean>
events         ParsedEvent[]   { name, timestamp, attributes }
projectId?     string          tenant stamp (server-set, never client)

There is no hard-coded "span type" enum — a span's kind is derived from its attributes and name at read time. The model comes from the standard OTel GenAI attributes gen_ai.request.model / llm.model; the service from the service.name resource attribute. The explorer colors nodes by inferred kind:

Where traces live & retention

Ingested spans go two places at once. A process-local in-memory store (a bounded, TTL-swept ring — 20k spans, 30-minute TTL in the web tier) serves real-time reads with zero query latency. Every span is also persisted to the durable trace_spans Postgres table, which is the long-term authority behind search, aggregation, and export. Reads try memory first, then fall through to Postgres.

Every read that crosses a tenant boundary is fail-closed on projectId: the in-memory store is a process-wide singleton, so a span whose server-stamped projectIdis missing or doesn't match the caller's validated project is dropped — a known trace ID from another tenant resolves to zero spans. Postgres reads are RLS-scoped to the org's projects on top of that.

The older step-based agent-trace format (agent_traces /agent_steps) is still accepted and readable; new instrumentation should emit spans.

Sending traces

Two supported paths, one storage backend. Use whichever fits your stack — both land in the same trace_spans table and the same in-memory store.

1. First-party SDK — wrap any function with traceable() (or trace an inline block with traced()). It captures name, args, return value, duration, and errors, then batches spans to POST /api/v1/traces/ingest. Optional sessionId / userId / conversationId attach to the span and are inherited by every child span in the same async context.

import { traceable, traced } from "@evalguard/sdk";

// Config via env: EVALGUARD_API_KEY, EVALGUARD_PROJECT_ID,
//                 EVALGUARD_BASE_URL (default https://evalguard.ai/api)

// Wrap a function — spans are auto-created + batched to /traces/ingest
const askLLM = traceable(
  async (prompt: string) => openai.chat(prompt),
  { name: "ask-llm", sessionId: "sess_123" },
);

// Inline span with custom metadata
const rows = await traced("load-context", async (span) => {
  const data = await db.query(...);
  span.metadata.rows = data.length;
  return data;
});

2. OpenTelemetry (OTLP/HTTP) — point any OTel exporter at the ingest endpoint. Because every span goes through the same ParsedSpan shape, OTel spans and SDK spans are indistinguishable once stored. Full wiring, the Collector config, and the one-line migration from LangSmith / Langfuse / Phoenix live in /docs/opentelemetry.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://evalguard.ai/api/v1/ingest/otlp/traces
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20${EVALGUARD_API_KEY}

Ingestion is quota-gated (a 429 means you hit your plan's monthly trace cap) and, for the SDK path, verifies optional Ed25519 span signatures before anything is stored.

Querying traces

All read endpoints live under /api/v1/traces and require a projectId (or orgId for aggregate) so every query is pinned to one tenant.

The SDK wraps the read endpoints too:

await client.listTraces(projectId);              // GET /traces
await client.getTrace(traceId);                  // GET /traces/{id}
await client.searchTraces(projectId, "errors from gpt-4");
await client.aggregateTraces({ orgId, projectId, model }); // analytics
await client.analyzeTrace({ traceId, projectId, callLLM: false });
await client.exportTraces({ projectId, format: "otlp-json" });

/traces/aggregate transparently serves from a ClickHouse rollup at high volume when configured, and falls back to Postgres otherwise — same response shape either way. See /docs/clickhouse for that opt-in scale path.

The trace explorer

Open any trace at /dashboard/traces/<id> and switch between three views of the same spans:

• Waterfall — spans stacked by depth and relative start; the go-to for "where is the latency."
• Messages — a conversation-shaped replay of LLM + tool spans for debugging chat threading and tool calls.
• Graph — a node-and-edge DAG of the run's topology, colored by span kind.

The graph view has its own deep-dive — /docs/agent-graph — covering the layout algorithm, node colors, and keyboard/screen-reader accessibility.

Beyond viewing

Traces aren't a dead end. A recorded trace can be scanned for failure modes or adversarial red-team findings post-hoc, curated into an eval dataset (/traces/to-dataset, with near-duplicate removal), exported to any OTLP-JSON backend, or imported from Helicone / Langfuse / Portkey / HuggingFace. Observability here is the front door to evaluation, not a separate silo.