Documentation

tracellm monitor

Watch live trace events from the terminal.

Overview

The tracellm monitor command opens a full-screen live dashboard — like htop for AI systems. It connects to the backend WebSocket for real-time trace events and falls back to polling MongoDB when the server is unavailable, with automatic reconnection.

Syntax

terminalCopy

bash

tracellm monitor [OPTIONS]

Option	Type	Default	Description
--refresh, -r	float	2.0	Polling interval in seconds
--limit, -l	int	10	Number of recent traces to display
--ws-host	str	env or 127.0.0.1	Backend WebSocket host
--ws-port	int	env or 8000	Backend WebSocket port

What Happens Internally

Starts a WebSocket listener in a background daemon thread that auto-discovers the endpoint by probing common ports (8000, 8001, 8080, 3000).
Connects to the WebSocket at ws://<host>:<port>/ws and listens for trace.created events. Implements exponential backoff reconnection from 1s to 30s.
Renders a Rich Live dashboard with two panels:
- Overview — total traces, completed, errors, avg latency, P95 latency, total tokens, WebSocket status
- Recent Traces — time, status, model, latency, tokens, steps, and prompt preview
Switches data sources — uses live WebSocket events when connected, polls MongoDB every --refresh seconds otherwise.
Handles shutdown on Ctrl+C — sets the stop event, terminates the WebSocket thread, and prints "Monitor stopped."

Example Output

The monitor renders a full-screen dashboard that updates in real time:

Monitor dashboardCopy

text

╭── TraceLLM Monitor ─────────────────────── Ctrl+C to stop ──╮
│                                                                  │
│  Monitor active                                                   │
│                                                                  │
│  ╭── Overview ───────────────────────────────────────────────╮   │
│  │                                                            │   │
│  │  Total Traces    12                                       │   │
│  │  Completed       10                                       │   │
│  │  Errors          2                                        │   │
│  │  Avg Latency     2,340 ms                                 │   │
│  │  P95 Latency     4,120 ms                                 │   │
│  │  Total Tokens    14,823                                   │   │
│  │  WebSocket       ● Connected                              │   │
│  │                                                            │   │
│  ╰────────────────────────────────────────────────────────────╯   │
│                                                                  │
│  ╭── Recent Traces (12 unique) ─────────────────────────────╮   │
│  │                                                            │   │
│  │  Time      Status   Model          Latency  Tokens  Steps  │   │
│  │  14:22:10  SUCCESS  gpt-4.1-mini   3420ms  1,247   9      │   │
│  │  14:20:05  SUCCESS  gpt-4.1-mini   1890ms    892   7      │   │
│  │  14:18:30  FAILED   gpt-4.1-mini   4500ms    312   3      │   │
│  │  14:15:44  SUCCESS  gpt-4.1-mini   2760ms  1,024   8      │   │
│  │  14:12:10  SUCCESS  gpt-4.1-mini   3120ms  1,156   9      │   │
│  │                                                            │   │
│  ╰────────────────────────────────────────────────────────────╯   │
╰──────────────────────────────────────────────────────────────────╯

Common Errors

Error	Cause	Fix
WebSocket Unavailable	Backend is not running or port is wrong	Start the stack with tracellm start first, or specify --ws-port
DB poll error	MongoDB is unreachable	Check MongoDB connection and MONGO_URL
Dashboard is empty	No traces have been recorded yet	Run tracellm trace to generate traces

Troubleshooting

Info

When the WebSocket is unreachable, the monitor falls back to polling MongoDB automatically. You will see a yellow "Unavailable" status and the polling interval indicator in the subtitle.

Tip

Set TRACELLM_WS_HOST and TRACELLM_WS_PORT environment variables to configure the WebSocket endpoint without passing CLI flags.