Trace Model

What Is a Trace

A trace is a structured record of a single LLM-powered execution. It captures everything that happened from the moment a request was initiated to the moment a response was returned — including every tool call, retry, latency measurement, and error along the way.

Each trace is persisted as a MongoDB document and exposed through the dashboard, CLI, and API. Traces are immutable once created — they serve as a permanent audit log of every AI operation in your system.

TraceSchema

The TraceSchema Pydantic model defines the shape of every trace document. It is validated on write and read, ensuring data integrity at both persistence boundaries.

StepSchema

Each trace contains an ordered list of steps. A step represents a single atomic operation — a tool call, an LLM invocation, a retry attempt, a retrieval query — that occurred during the execution.

Validation Rules

Both schema models enforce constraints at the Pydantic level:

# StepSchema validators
@field_validator("duration")
def validate_duration(cls, value: float) -> float:
    if value < 0:
        raise ValueError("duration must be >= 0")
    return value

# TraceSchema validators
@field_validator("latency")
def validate_latency(cls, value: float) -> float:
    if value < 0:
        raise ValueError("latency must be >= 0")
    return value

@field_validator("token_count", "retry_count")
def validate_non_negative(cls, value: int) -> int:
    if value < 0:
        raise ValueError("value must be >= 0")
    return value

status is typed as a Literal, so only the values "success", "warning", and "failed" are accepted. Any invalid value is rejected at the API or persistence layer.

Status Inference

When a trace is persisted, the status is inferred through a fallback chain:

1. If the payload explicitly sets status to a valid value, use it
2. If any step has success=False, status becomes "failed"
3. If failure_reason is set or retry_count > 0, status becomes "warning"
4. Otherwise, status is "success"

Full Trace Example

Here is a complete trace document as it appears in MongoDB, after normalization and validation:

{
  "trace_id": "tr_2kf9q3m1",
  "prompt": "Explain transformers",
  "response": "Transformers are neural architectures built around self-attention...",
  "latency": 3420.0,
  "token_count": 1247,
  "model_name": "gpt-4.1-mini",
  "project_id": "default",
  "project_name": null,
  "api_key": null,
  "environment": "development",
  "status": "success",
  "retry_count": 1,
  "slow_request": true,
  "failure_reason": null,
  "created_at": "2026-05-31T14:22:10",
  "updated_at": "2026-05-31T14:22:14",
  "steps": [
    {
      "step_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "tool_name": "query_embedding",
      "input": {
        "session_id": "abc12345",
        "query": "Explain transformers",
        "embedding_model": "text-embedding-3-large"
      },
      "output": {
        "vector_dimensions": 1536,
        "embedding_norm": 0.9983,
        "checksum": "0xa1b2c3"
      },
      "duration": 180.42,
      "success": true,
      "timestamp": "2026-05-31T14:22:10"
    },
    {
      "step_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "tool_name": "vector_retrieval",
      "input": {
        "session_id": "abc12345",
        "query": "Explain transformers",
        "index": "research_embeddings_v2",
        "top_k": 8
      },
      "output": {
        "documents_found": 18,
        "candidate_chunks": 8,
        "selected_ids": [
          "attention_is_all_you_need",
          "rag_failure_playbook"
        ]
      },
      "duration": 340.15,
      "success": true,
      "timestamp": "2026-05-31T14:22:10"
    }
  ]
}

How Traces Are Stored

Traces flow through a multi-stage pipeline from creation to persistence:

1. Payload construction — The build_trace_payload()function assembles the raw trace dict from the decorator's captured data, coerced result, and context variable state.
2. Normalization — normalize_trace_document() validates and transforms the payload against TraceSchema and StepSchema, filling in defaults and inferring any missing fields (status, retry count, slow_request flag).
3. Persistence — The normalized document is inserted into the traces MongoDB collection via Motor (async driver). The CLI bridges the sync/async boundary with a persistent event loop.
4. Broadcast — A trace.created event is pushed to all connected WebSocket clients, including the dashboard and the monitor CLI.