Paybond middleware trace shows what happens at the tool boundary: policy authorization, handler execution, evidence submission, and sandbox settlement. It does not show model reasoning or chain-of-thought — Paybond guards side-effecting tools, not LLM inference.
Use trace to answer: Did Harbor authorize this spend? Did my handler run? Was evidence submitted? What settlement state did sandbox return?
Quick start
Terminal 1 — scaffold policy, validate, and smoke:
paybond dev loop --offline # or paybond dev loop after paybond login
Terminal 2 — open the local dashboard (same project directory):
paybond dev trace
The UI listens at http://127.0.0.1:9477. Select a run from the sidebar or open the trace_url printed by smoke/loop (for example http://127.0.0.1:9477/runs/<run-id>).
Hosted replay (no local server): /demo/agent-trace.
What the timeline shows
Each guarded tool call produces a vertical timeline with phase-colored steps:
| Phase | Badge color | Typical step |
|---|---|---|
| Tool | Teal | Tool call resolved (tool_selected) |
| Authorize | Amber | Spend approved, denied, or held for approval |
| Evidence | Purple | Auto-evidence submitted after a successful handler |
| Result | Green | Handler duration, spend finalized, sandbox settlement status |
Typical happy path:
Tool call: travel.book_hotel → Paybond approved $187.00 → Tool executed (42ms) → Evidence submitted → Settlement: released
Denied or approval-held calls include spend_denied or approval_required steps under Authorize so you can debug policy failures without Harbor logs.
Observability surfaces
| Surface | Command / URL | Best for |
|---|---|---|
| Local dashboard | paybond dev trace → :9477 | First-time adoption, smoke debugging, live refresh while iterating |
| Terminal trace | paybond agent run trace --run-id <id> --format table | CI, headless environments, multi-step bind → execute flows |
| Hosted replay | /demo/agent-trace?intent=<uuid> | Sharing a visual walkthrough without running the local server |
| SDK sink | traceSink on PaybondAgentRun.bind() | Custom telemetry, tests, in-app debug panels |
| Console dossier | /console/operations/intents/<intentId> | Production operators — intent-centric audit (requires auth) |
| Console agent run | /console/operations/agent-runs/<runId> | Tenant-scoped run timeline (when Gateway persistence is enabled) |
All local and CLI surfaces share the same PaybondTraceEvent vocabulary. The hosted replay animates smoke JSON with the same phase colors as the local dashboard.
Dev, sandbox, and production
Sandbox (Paybond environment) and dev (local tooling on your machine) are not the same thing. Production binds can emit trace events too, but nothing is recorded unless you wire a sink.
Local dashboard vs trace events
Local dashboard (paybond dev trace) | PaybondTraceEvent stream | |
|---|---|---|
| What it is | HTTP UI on 127.0.0.1:9477 reading .paybond/dev-trace.jsonl | Structured interceptor events (tool_selected, spend_authorized, …) |
| Runs in production deploys? | No — local dev tool only | Can — if you pass traceSink or use Gateway reporting |
| On by default? | Only after paybond dev smoke / dev loop | No in production app binds |
Do not run paybond dev trace in production. It is for adoption and local debugging on your laptop.
What turns trace on
On PaybondAgentRun.bind(), Kit resolves the sink as:
traceSink ?? onTrace ?? resolveDevTraceSink()
The only automatic sink (without you passing traceSink) is the in-process dev collector, active only after activateDevTraceCollector — that is, during paybond dev smoke or paybond dev loop. A typical production instrument().bind() does not attach a sink unless you set one.
By environment
| Capability | Dev (local) | Sandbox | Production |
|---|---|---|---|
paybond dev trace dashboard | Yes | Yes (same local UI after smoke) | No |
.paybond/dev-trace.jsonl | Yes | Yes (when dev collector ran) | No |
.paybond/runs/<id>.trace.json | Yes | Yes | Yes (CLI bind + execute) |
Auto traceSink on app bind | Only if dev collector active | Same | No — opt in |
paybond agent run trace (CLI) | Yes | Yes | Yes |
Hosted /demo/agent-trace replay | Yes | Yes (smoke JSON) | N/A (smoke replay only) |
| Console intent dossier | When logged in | Yes | Yes |
| Console agent-run timeline | When Gateway reporter used | Yes | Yes — opt in |
Sandbox smoke with paybond login still writes trace files when the dev collector or CLI trace sinks are active. That is real Harbor authorization against the sandbox Gateway — not an offline mock — but the dashboard remains a process on your machine.
Production observability
Default production middleware binds do not enable trace. Operators use the intent dossier (/console/operations/intents/<intentId>) for audit-grade investigation.
To record middleware trace in production:
- Custom telemetry — pass
traceSinkon bind and forward events to your observability stack (TypeScript and Python). - Console agent-run view — wire
paybond.harbor.createAgentRunTraceReporter(runId)(TypeScript) orpaybond.harbor.create_agent_run_trace_reporter(run_id)(Python) so events persist to Gateway and appear at/console/operations/agent-runs/<runId>. - CLI multi-step bind —
paybond agent run bind --productionwrites.paybond/runs/<run_id>.trace.jsonand forwards events to Gateway whenpaybond logincredentials are available (TypeScript and Python CLI).
Gateway trace reporting is fire-and-forget — failures are swallowed so middleware execution is never blocked on telemetry.
Production examples
CLI — production attach with trace
paybond agent run bind attaches trace sinks automatically: a local .paybond/runs/<run_id>.trace.json file on every platform, plus Gateway forwarding when the CLI has paybond login credentials (TypeScript and Python).
paybond login paybond agent run bind --production \ --attach-intent-id "$PAYBOND_INTENT_ID" \ --capability-token "$PAYBOND_CAPABILITY_TOKEN" \ --payee-did "$APP_PAYEE_DID" \ --payee-signing-seed-hex "$APP_PAYEE_SEED_HEX" \ --agent-recognition-key-id "$APP_AGENT_RECOGNITION_KEY_ID" \ --agent-recognition-signing-seed-hex "$APP_AGENT_RECOGNITION_SEED_HEX" \ --policy-file paybond.policy.yaml \ --format json paybond agent tool execute --run-id <run-id> \ --operation paid-tool \ --tool-call-id call-1 \ --result-body '{"status":"ok","cost_cents":100}' \ --format json # Local timeline (no browser) paybond agent run trace --run-id <run-id> --format table # Console (authenticated session + real run id): /console/operations/agent-runs/<run-id>
Signing credential flags and env fallbacks are documented in Production attach. There is no paybond dev trace step in production — that dashboard is local dev only.
SDK — custom telemetry (TypeScript and Python)
Forward events to your own stack with traceSink / trace_sink on a production attach bind:
TypeScript:
import type { PaybondTraceEvent } from "@paybond/kit/agent"; import { Paybond, seed32FromHex } from "@paybond/kit"; const paybond = await Paybond.open({ apiKey: process.env.PAYBOND_API_KEY!, expectedEnvironment: "production", }); const traceSink = (event: PaybondTraceEvent) => { // Forward to OpenTelemetry, Datadog, structured logs, etc. console.log(JSON.stringify(event)); }; const run = await paybond.agentRun.bind({ attach: { intentId: process.env.PAYBOND_INTENT_ID!, capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!, productionEvidence: { payeeDid: process.env.APP_PAYEE_DID!, payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!), agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!, agentRecognitionSigningSeed: seed32FromHex( process.env.APP_AGENT_RECOGNITION_SEED_HEX!, ), }, }, registry, traceSink, }); await run.interceptor.wrapExecute({ /* tool call */ });
Python:
import os from paybond_kit import Paybond paybond = await Paybond.open( api_key=os.environ["PAYBOND_API_KEY"], expected_environment="production", ) def trace_sink(event: dict[str, object]) -> None: # Forward to your observability stack print(event) run = await paybond.agent_run.bind( { "attach": { "intent_id": os.environ["PAYBOND_INTENT_ID"], "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"], "production_evidence": { "payee_did": os.environ["APP_PAYEE_DID"], "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]), "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"], "agent_recognition_signing_seed": seed32_from_hex( os.environ["APP_AGENT_RECOGNITION_SEED_HEX"] ), }, }, "registry": registry, "trace_sink": trace_sink, } ) await run.interceptor.wrap_execute({...})
SDK — console agent-run timeline (TypeScript and Python)
To populate /console/operations/agent-runs/<runId> from application code, register the run and pass the Gateway reporter sink on bind.
TypeScript:
import { Paybond, seed32FromHex } from "@paybond/kit"; const paybond = await Paybond.open({ apiKey: process.env.PAYBOND_API_KEY!, expectedEnvironment: "production", }); const runId = "run-prod-1"; const reporter = paybond.harbor.createAgentRunTraceReporter(runId); const traceSink = reporter.createSink({ intentId: process.env.PAYBOND_INTENT_ID!, operation: "paid-tool", sandbox: false, allowedTools: ["paid-tool"], }); const run = await paybond.agentRun.bind({ runId, attach: { intentId: process.env.PAYBOND_INTENT_ID!, capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!, productionEvidence: { payeeDid: process.env.APP_PAYEE_DID!, payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!), agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!, agentRecognitionSigningSeed: seed32FromHex( process.env.APP_AGENT_RECOGNITION_SEED_HEX!, ), }, }, registry, traceSink, }); await run.interceptor.wrapExecute({ /* tool call */ }); // Optional: await in-flight Gateway writes before process exit await reporter.flush();
Python:
import os from paybond_kit import Paybond from paybond_kit.agent.gateway_trace_reporter import AgentRunUpsertInput paybond = await Paybond.open( api_key=os.environ["PAYBOND_API_KEY"], expected_environment="production", ) run_id = "run-prod-1" reporter = paybond.harbor.create_agent_run_trace_reporter(run_id) trace_sink = reporter.create_sink( AgentRunUpsertInput( intent_id=os.environ["PAYBOND_INTENT_ID"], operation="paid-tool", sandbox=False, allowed_tools=["paid-tool"], ) ) run = await paybond.agent_run.bind( { "run_id": run_id, "attach": { "intent_id": os.environ["PAYBOND_INTENT_ID"], "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"], "production_evidence": { "payee_did": os.environ["APP_PAYEE_DID"], "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]), "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"], "agent_recognition_signing_seed": seed32_from_hex( os.environ["APP_AGENT_RECOGNITION_SEED_HEX"] ), }, }, "registry": registry, "trace_sink": trace_sink, } ) await run.interceptor.wrap_execute(...) await reporter.flush()
Python trace events are normalized to camelCase wire fields before Gateway POST so the console timeline renders tool names, amounts, and evidence ids correctly.
Data flow
flowchart LR
subgraph kit [Kit interceptor]
Interceptor[PaybondToolInterceptor]
Sink[traceSink / dev collector]
end
subgraph persist [Project files]
Jsonl[".paybond/dev-trace.jsonl"]
RunFile[".paybond/runs/run_id.trace.json"]
Audit[".paybond/dev-audit.jsonl"]
end
subgraph ui [Trace UI]
Server["paybond dev trace :9477"]
API["GET /api/events"]
Dashboard[Vertical timeline HTML]
end
Interceptor --> Sink
Sink --> Jsonl
Sink --> RunFile
Smoke[dev smoke / dev loop] --> Audit
Server --> API
API --> Jsonl
API --> DashboardPaybondToolInterceptoremits structured events as each side-effecting tool moves through authorize → execute → finalize → evidence.paybond dev smokeandpaybond dev loopactivate an in-processDevTraceCollectorduring the smoke bind/execute cycle. On completion, the collector writes one summary row to.paybond/dev-trace.jsonl(and a summary line to.paybond/dev-audit.jsonl).paybond dev tracestarts a local HTTP server that merges events from disk and an in-memory ring buffer, then serves a self-contained HTML dashboard.- The dashboard polls
/api/eventsevery two seconds and re-renders the selected run's timeline. - Multi-step
paybond agent run bind+tool executewrites.paybond/runs/<run_id>.trace.jsonforpaybond agent run trace.
Run paybond dev trace from the same working directory as smoke/loop so .paybond/dev-trace.jsonl is found. The file keeps the last 100 runs.
Event model
Structured middleware events (PaybondTraceEvent) are the source of truth. The dashboard derives human-readable steps[] from them via devTraceStepsFromEvents.
| Event type | When emitted | Key fields |
|---|---|---|
tool_selected | Side-effecting tool resolved | toolName, toolCallId, operation |
spend_authorized | Harbor allows the spend | decisionId, amountCents, auditId |
spend_denied | Hard denial | message, code, auditId |
approval_required | Approval hold | message, code, auditId |
tool_executed | Handler finished | durationMs |
spend_finalized | Decision consumed or released | status: consumed | released |
evidence_submitted | Auto-evidence succeeded | evidenceId, presetId, sandboxLifecycleStatus |
Each event includes runId, operation, and recordedAt for correlation.
Summary row (one line in .paybond/dev-trace.jsonl per smoke run):
{ "id": "run-abc", "run_id": "run-abc", "preset": "travel", "operation": "travel.book_hotel", "intent_id": "00000000-0000-4000-8000-000000000001", "requested_spend_cents": 18700, "authorized": true, "evidence_submitted": true, "sandbox_lifecycle_status": "released", "steps": [ "..." ], "trace_events": [ "..." ] }
Expand Raw trace JSON in the dashboard or use --format json on CLI commands for the full payload.
Local dashboard HTTP API
paybond dev trace binds to 127.0.0.1:9477 by default (--port to override).
| Route | Response |
|---|---|
/ | Dashboard HTML — recent runs sidebar + timeline |
/runs/<run-id> | Same HTML, focused on the matching run |
/api/events | JSON { "events": DevTraceEvent[], "has_credentials": boolean } |
has_credentials is true when PAYBOND_API_KEY is set in the trace server's environment (informational banner only).
The dashboard is bundled with @paybond/kit (kit/ts/dev/trace-ui/dashboard.html) — no separate npm package.
Deep links from smoke
paybond agent sandbox smoke, paybond dev smoke, and paybond dev loop print checklist lines when table output is enabled:
✓ Trace → http://127.0.0.1:9477/runs/<run-id> ✓ Console → https://…/console/operations/intents/<intent-id> ✓ Replay → https://…/demo/agent-trace?intent=<intent-id>
JSON smoke output includes trace_url, console_url, and agent_trace_url in the data object. Console and replay links require a Harbor UUID intent_id from a real sandbox bind.
SDK integration (traceSink) — sandbox
For local or sandbox debugging, pass traceSink on bind (or trace_sink in Python). Production attach examples are in Production examples above.
TypeScript:
import type { PaybondTraceEvent } from "@paybond/kit/agent"; const events: PaybondTraceEvent[] = []; const run = await paybond.agentRun.bind({ bootstrap: { kind: "sandbox", operation: "travel.book_hotel", requestedSpendCents: 20_000 }, registry, traceSink: (event) => events.push(event), });
Python:
trace_events: list[dict[str, object]] = [] run = await paybond.agent_run.bind( { "bootstrap": { "kind": "sandbox", "operation": "travel.book_hotel", "requested_spend_cents": 20_000, }, "registry": registry, "trace_sink": trace_events.append, } )
onTrace / on_trace are deprecated aliases for traceSink / trace_sink.
Terminal trace (no browser)
After a multi-step bind and execute:
paybond agent run bind --sandbox \ --operation travel.book_hotel \ --requested-spend-cents 20000 \ --completion-preset cost_and_completion paybond agent tool execute --run-id <id> \ --operation travel.book_hotel \ --tool-call-id call-1 \ --result-body '{"status":"completed","cost_cents":18700}' paybond agent run trace --run-id <id> --format table
Reads .paybond/runs/<run_id>.trace.json. Use --format json for trace_events[] and steps[] in automation.
Hosted replay
/demo/agent-trace replays sandbox smoke output as an animated timeline using the same phase colors and step ordering as the local dashboard. Deep-link a real intent:
/demo/agent-trace?intent=<harbor-intent-uuid>
When you are logged into the console, the page offers Open intent in console for the same intent_id.
What trace does not show
- Model reasoning or token usage — use your LLM provider's observability.
- Cross-tenant data — trace files are local to your project; console views require authenticated tenant context.
- Full Harbor audit logs — trace is middleware-shaped debugging; use the intent dossier for operator-grade investigation.
Label timeline steps as Tool call, not "Agent reasoning", to avoid implying visibility into the model.
Related
- Agent quickstart —
paybond dev loopentry path - Production attach — production bind credentials and CLI flags
- Agent middleware — Trace events — interceptor API reference
- CLI contract —
paybond dev— flags, JSON fields, exit codes - Coding-agent setup — MCP and scaffold workflows with trace
- Middleware trace guide — public guide mirror
- Demo hub · Middleware trace replay