Paybond middleware trace shows what happens at the tool boundary: policy authorization, handler execution, evidence submission, and sandbox settlement. It does not show model reasoning or chain-of-thought — Paybond guards side-effecting tools, not LLM inference.
Use trace to answer: Was this spend authorized? Did my handler run? Was evidence submitted? What settlement state did sandbox return?
Quick start
Terminal 1 — scaffold policy, validate, and smoke:
paybond dev loop --offline # or paybond dev loop after paybond login
Terminal 2 — open the local dashboard (same project directory):
paybond dev trace
The UI listens at http://127.0.0.1:9477. Select a run from the sidebar or open the trace_url printed by smoke/loop (for example http://127.0.0.1:9477/runs/<run-id>).
paybond dev loop and paybond dev smoke record trace events to .paybond/dev-trace.jsonl but do not start the HTTP dashboard. The startup banner line ✓ Trace dashboard → http://127.0.0.1:9477 is the URL to open after you run paybond dev trace in a second terminal — until then, that URL will refuse connections.
Hosted replay (no local server): /demo/agent-trace. For the interactive proof aha (edit evidence → release vs refund), open the Kit Proof Lab.
What the timeline shows
Each guarded tool call produces a vertical timeline with phase-colored steps:
| Phase | Badge color | Typical step |
|---|---|---|
| Tool | Teal | Tool call resolved (tool_selected) |
| Authorize | Amber | Spend ceiling authorized, denied, or held for approval |
| Evidence | Purple | Auto-evidence submitted after a successful handler |
| Result | Green | Handler duration, spend finalized, sandbox settlement status |
Typical happy path:
Tool call: travel.book_hotel → Paybond authorized up to $200.00 (20,000 cents) → Tool executed (42ms) → Evidence submitted (reported cost $187.00 (18,700 cents); predicate evaluated) → Settlement: released — captured $187.00 (18,700 cents); unused $13.00 (1,300 cents) released
requested_spend_cents is the maximum authorized amount in integer cents: 20000 means $200.00. cost_cents is the cost reported later in tool-result evidence: 18700 means $187.00. The cost_and_completion preset checks that the reported cost is within the authorized ceiling; it does not add the values together. For this variable-cost preset the validated reported cost resizes settlement: Paybond captures $187.00 and releases the unused $13.00 authorization. Fixed-price templates still capture the full authorized amount. The settlement amount is derived only from Paybond's stored, validated, signed evidence — callers cannot supply or override it.
Denied or approval-held calls include spend_denied or approval_required steps under Authorize so you can debug policy failures without upstream service logs.
Observability surfaces
| Surface | Command / URL | Best for |
|---|---|---|
| Local dashboard | paybond dev trace → :9477 | First-time adoption, smoke debugging, live refresh while iterating |
| Terminal trace | paybond agent run trace --run-id <id> --format table | CI, headless environments, multi-step bind → execute flows |
| Hosted replay | /demo/agent-trace?intent=<uuid> | Sharing a visual walkthrough without running the local server |
| SDK sink | traceSink on PaybondAgentRun.bind() | Custom telemetry, tests, in-app debug panels |
| Console dossier | /console/operations/intents/<intentId> | Production operators — intent-centric audit (requires auth) |
| Console agent run | /console/operations/agent-runs/<runId> | Tenant-scoped run timeline (when Gateway persistence is enabled) |
All local and CLI surfaces share the same PaybondTraceEvent vocabulary. The hosted replay animates smoke JSON with the same phase colors as the local dashboard.
Dev, sandbox, and production
Sandbox (Paybond environment) and dev (local tooling on your machine) are not the same thing. Production binds can emit trace events too, but nothing is recorded unless you wire a sink.
Local dashboard vs trace events
Local dashboard (paybond dev trace) | PaybondTraceEvent stream | |
|---|---|---|
| What it is | HTTP UI on 127.0.0.1:9477 reading .paybond/dev-trace.jsonl | Structured interceptor events (tool_selected, spend_authorized, …) |
| Runs in production deploys? | No — local dev tool only | Can — if you pass traceSink or use Gateway reporting |
| On by default? | Only after paybond dev smoke / dev loop | No in production app binds |
Do not run paybond dev trace in production. It is for adoption and local debugging on your laptop.
What turns trace on
On PaybondAgentRun.bind(), Kit resolves the sink as:
traceSink ?? onTrace ?? resolveDevTraceSink()
The only automatic sink (without you passing traceSink) is the in-process dev collector, active only after activateDevTraceCollector — that is, during paybond dev smoke or paybond dev loop. A typical production instrument().bind() does not attach a sink unless you set one.
By environment
| Capability | Dev (local) | Sandbox | Production |
|---|---|---|---|
paybond dev trace dashboard | Yes | Yes (same local UI after smoke) | No |
.paybond/dev-trace.jsonl | Yes | Yes (when dev collector ran) | No |
.paybond/runs/<id>.trace.json | Yes | Yes | Yes (CLI bind and execute) |
Auto traceSink on app bind | Only if dev collector active | Same | No — opt in |
paybond agent run trace (CLI) | Yes | Yes | Yes |
Hosted /demo/agent-trace replay | Yes | Yes (smoke JSON) | N/A (smoke replay only) |
| Console intent dossier | When logged in | Yes | Yes |
| Console agent-run timeline | When Gateway reporter used | Yes | Yes — opt in |
Sandbox smoke with paybond login still writes trace files when the dev collector or CLI trace sinks are active. That is real Harbor authorization against the sandbox Gateway — not an offline mock — but the dashboard remains a process on your machine.
Production observability
Default production middleware binds do not enable trace. Operators use the intent dossier (/console/operations/intents/<intentId>) for audit-grade investigation.
To record middleware trace in production:
- Custom telemetry — pass
traceSinkon bind and forward events to your observability stack (TypeScript and Python). - Console agent-run view — wire
paybond.harbor.createAgentRunTraceReporter(runId)(TypeScript) orpaybond.harbor.create_agent_run_trace_reporter(run_id)(Python) so events persist to Gateway and appear at/console/operations/agent-runs/<runId>. - CLI multi-step bind —
paybond agent run bind --productionwrites.paybond/runs/<run_id>.trace.jsonand forwards events to Gateway whenpaybond logincredentials are available (TypeScript and Python CLI).
Gateway trace reporting is fire-and-forget — failures are swallowed so middleware execution is never blocked on telemetry.
Production examples
CLI — production attach with trace
paybond agent run bind attaches trace sinks automatically: a local .paybond/runs/<run_id>.trace.json file on every platform, plus Gateway forwarding when the CLI has paybond login credentials (TypeScript and Python).
paybond login paybond agent run bind --production \ --attach-intent-id "$PAYBOND_INTENT_ID" \ --capability-token "$PAYBOND_CAPABILITY_TOKEN" \ --payee-did "$APP_PAYEE_DID" \ --payee-signing-seed-hex "$APP_PAYEE_SEED_HEX" \ --agent-recognition-key-id "$APP_AGENT_RECOGNITION_KEY_ID" \ --agent-recognition-signing-seed-hex "$APP_AGENT_RECOGNITION_SEED_HEX" \ --policy-file paybond.policy.yaml \ --format json paybond agent tool execute --run-id <run-id> \ --operation paid-tool \ --tool-call-id call-1 \ --result-body '{"status":"ok","cost_cents":100}' \ --format json # Local timeline (no browser) paybond agent run trace --run-id <run-id> --format table # Console (authenticated session and real run id): /console/operations/agent-runs/<run-id>
Signing credential flags and env fallbacks are documented in Production attach. There is no paybond dev trace step in production — that dashboard is local dev only.
SDK — custom telemetry (TypeScript and Python)
Forward events to your own stack with traceSink / trace_sink on a production attach bind:
TypeScript:
import type { PaybondTraceEvent } from "@paybond/kit/agent"; import { Paybond, seed32FromHex } from "@paybond/kit"; const paybond = await Paybond.open({ apiKey: process.env.PAYBOND_API_KEY!, expectedEnvironment: "production", }); const traceSink = (event: PaybondTraceEvent) => { // Forward to OpenTelemetry, Datadog, structured logs, etc. console.log(JSON.stringify(event)); }; const run = await paybond.agentRun.bind({ attach: { intentId: process.env.PAYBOND_INTENT_ID!, capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!, productionEvidence: { payeeDid: process.env.APP_PAYEE_DID!, payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!), agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!, agentRecognitionSigningSeed: seed32FromHex( process.env.APP_AGENT_RECOGNITION_SEED_HEX!, ), }, }, registry, traceSink, }); await run.interceptor.wrapExecute({ /* tool call */ });
Python:
import os from paybond_kit import Paybond paybond = await Paybond.open( api_key=os.environ["PAYBOND_API_KEY"], expected_environment="production", ) def trace_sink(event: dict[str, object]) -> None: # Forward to your observability stack print(event) run = await paybond.agent_run.bind( { "attach": { "intent_id": os.environ["PAYBOND_INTENT_ID"], "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"], "production_evidence": { "payee_did": os.environ["APP_PAYEE_DID"], "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]), "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"], "agent_recognition_signing_seed": seed32_from_hex( os.environ["APP_AGENT_RECOGNITION_SEED_HEX"] ), }, }, "registry": registry, "trace_sink": trace_sink, } ) await run.interceptor.wrap_execute({...})
SDK — console agent-run timeline (TypeScript and Python)
To populate /console/operations/agent-runs/<runId> from application code, register the run and pass the Gateway reporter sink on bind.
TypeScript:
import { Paybond, seed32FromHex } from "@paybond/kit"; const paybond = await Paybond.open({ apiKey: process.env.PAYBOND_API_KEY!, expectedEnvironment: "production", }); const runId = "run-prod-1"; const reporter = paybond.harbor.createAgentRunTraceReporter(runId); const traceSink = reporter.createSink({ intentId: process.env.PAYBOND_INTENT_ID!, operation: "paid-tool", sandbox: false, allowedTools: ["paid-tool"], }); const run = await paybond.agentRun.bind({ runId, attach: { intentId: process.env.PAYBOND_INTENT_ID!, capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!, productionEvidence: { payeeDid: process.env.APP_PAYEE_DID!, payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!), agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!, agentRecognitionSigningSeed: seed32FromHex( process.env.APP_AGENT_RECOGNITION_SEED_HEX!, ), }, }, registry, traceSink, }); await run.interceptor.wrapExecute({ /* tool call */ }); // Optional: await in-flight Gateway writes before process exit await reporter.flush();
Python:
import os from paybond_kit import Paybond from paybond_kit.agent.gateway_trace_reporter import AgentRunUpsertInput paybond = await Paybond.open( api_key=os.environ["PAYBOND_API_KEY"], expected_environment="production", ) run_id = "run-prod-1" reporter = paybond.harbor.create_agent_run_trace_reporter(run_id) trace_sink = reporter.create_sink( AgentRunUpsertInput( intent_id=os.environ["PAYBOND_INTENT_ID"], operation="paid-tool", sandbox=False, allowed_tools=["paid-tool"], ) ) run = await paybond.agent_run.bind( { "run_id": run_id, "attach": { "intent_id": os.environ["PAYBOND_INTENT_ID"], "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"], "production_evidence": { "payee_did": os.environ["APP_PAYEE_DID"], "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]), "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"], "agent_recognition_signing_seed": seed32_from_hex( os.environ["APP_AGENT_RECOGNITION_SEED_HEX"] ), }, }, "registry": registry, "trace_sink": trace_sink, } ) await run.interceptor.wrap_execute(...) await reporter.flush()
Python trace events are normalized to camelCase wire fields before Gateway POST so the console timeline renders tool names, amounts, and evidence ids correctly.
Data flow
flowchart LR
subgraph kit [Kit interceptor]
Interceptor[PaybondToolInterceptor]
Sink[traceSink / dev collector]
end
subgraph persist [Project files]
Jsonl[".paybond/dev-trace.jsonl"]
RunFile[".paybond/runs/run_id.trace.json"]
Audit[".paybond/dev-audit.jsonl"]
end
subgraph ui [Trace UI]
Server["paybond dev trace :9477"]
API["GET /api/events"]
Dashboard[Vertical timeline HTML]
end
Interceptor --> Sink
Sink --> Jsonl
Sink --> RunFile
Smoke[dev smoke / dev loop] --> Audit
Server --> API
API --> Jsonl
API --> DashboardPaybondToolInterceptoremits structured events as each side-effecting tool moves through authorize → execute → finalize → evidence.paybond dev smokeandpaybond dev loopactivate an in-processDevTraceCollectorduring the smoke bind/execute cycle. On completion, the collector writes one summary row to.paybond/dev-trace.jsonl(and a summary line to.paybond/dev-audit.jsonl).paybond dev tracestarts a local HTTP server that merges events from disk and an in-memory ring buffer, then serves a self-contained HTML dashboard.- The dashboard polls
/api/eventsevery two seconds and re-renders the selected run's timeline. - Multi-step
paybond agent run bindandtool executewrites.paybond/runs/<run_id>.trace.jsonforpaybond agent run trace.
Run paybond dev trace from the same working directory as smoke/loop so .paybond/dev-trace.jsonl is found. The file keeps the last 100 runs.
Event model
Structured middleware events (PaybondTraceEvent) are the source of truth. The dashboard derives human-readable steps[] from them via devTraceStepsFromEvents.
| Event type | When emitted | Key fields |
|---|---|---|
tool_selected | Side-effecting tool resolved | toolName, toolCallId, operation |
spend_authorized | Paybond allows the spend | decisionId, amountCents, auditId |
spend_denied | Hard denial | message, code, auditId |
approval_required | Approval hold | message, code, auditId |
tool_executed | Handler finished | durationMs |
spend_finalized | Decision consumed or released | status: consumed | released |
evidence_submitted | Auto-evidence succeeded | evidenceId, presetId, sandboxLifecycleStatus |
Each event includes runId, operation, and recordedAt for correlation.
Summary row (one line in .paybond/dev-trace.jsonl per smoke run):
{ "id": "run-abc", "run_id": "run-abc", "preset": "travel", "operation": "travel.book_hotel", "intent_id": "00000000-0000-4000-8000-000000000001", "requested_spend_cents": 18700, "authorized": true, "evidence_submitted": true, "sandbox_lifecycle_status": "released", "steps": [ "..." ], "trace_events": [ "..." ] }
Expand Raw trace JSON in the dashboard or use --format json on CLI commands for the full payload.
Local dashboard HTTP API
paybond dev trace binds to 127.0.0.1:9477 by default (--port to override).
| Route | Response |
|---|---|
/ | Dashboard HTML — recent runs sidebar and timeline |
/runs/<run-id> | Same HTML, focused on the matching run |
/api/events | JSON { "events": DevTraceEvent[], "has_credentials": boolean } |
has_credentials is true when the trace server process can resolve a sandbox PAYBOND_API_KEY — from the process environment, from the env file (default .env.local, or PAYBOND_ENV_FILE / --env-file), or from a CLI profile's env_file. It controls an informational banner only; local trace events are recorded regardless.
If you see Sandbox credentials not detected after paybond login, the key is usually in .env.local but was not exported into the shell that runs paybond dev trace. Either run trace from the same project directory (Kit reads .env.local by default) or point at your file explicitly:
paybond dev trace --env-file ~/.env.local # or export PAYBOND_ENV_FILE=~/.env.local paybond dev trace
Runs from paybond dev loop after login still hit the real sandbox; the banner does not mean those runs failed.
The dashboard is bundled with @paybond/kit (kit/ts/dev/trace-ui/dashboard.html) — no separate npm package.
Deep links from smoke
paybond agent sandbox smoke, paybond dev smoke, and paybond dev loop print checklist lines when table output is enabled:
✓ Trace → http://127.0.0.1:9477/runs/<run-id> ✓ Console → https://…/console/operations/intents/<intent-id> ✓ Replay → https://…/demo/agent-trace?intent=<intent-id>
JSON smoke output includes trace_url, console_url, and agent_trace_url in the data object. Console and replay links require a sandbox UUID intent_id from a real sandbox bind.
SDK integration (traceSink) — sandbox
For local or sandbox debugging, pass traceSink on bind (or trace_sink in Python). Production attach examples are in Production examples above.
TypeScript:
import type { PaybondTraceEvent } from "@paybond/kit/agent"; const events: PaybondTraceEvent[] = []; const run = await paybond.agentRun.bind({ bootstrap: { kind: "sandbox", operation: "travel.book_hotel", requestedSpendCents: 20_000 }, registry, traceSink: (event) => events.push(event), });
Python:
trace_events: list[dict[str, object]] = [] run = await paybond.agent_run.bind( { "bootstrap": { "kind": "sandbox", "operation": "travel.book_hotel", "requested_spend_cents": 20_000, }, "registry": registry, "trace_sink": trace_events.append, } )
onTrace / on_trace are deprecated aliases for traceSink / trace_sink.
Terminal trace (no browser)
After a multi-step bind and execute:
paybond agent run bind --sandbox \ --operation travel.book_hotel \ --requested-spend-cents 20000 \ --completion-preset cost_and_completion paybond agent tool execute --run-id <id> \ --operation travel.book_hotel \ --tool-call-id call-1 \ --result-body '{"status":"completed","cost_cents":18700}' paybond agent run trace --run-id <id> --format table
Evidence payload shape: --result-body on paybond agent tool execute is submitted as auto-evidence as-is. It must satisfy the completion_preset from bind (for cost_and_completion, use top-level status and cost_cents). Nested vendor JSON without an SDK evidenceMapper fails Harbor predicate checks and may appear as cli.agent.evidence_failed / HTTP 502 from the Gateway. For travel vendor-pack fields, bind with --completion-preset ach_travel_booking or x402_travel_booking instead. See Agent middleware — Per-run CLI trace.
Reads .paybond/runs/<run_id>.trace.json. Use --format json for trace_events[] and steps[] in automation.
Hosted replay
/demo/agent-trace replays sandbox smoke output as an animated timeline using the same phase colors and step ordering as the local dashboard. Deep-link a real intent:
/demo/agent-trace?intent=<harbor-intent-uuid>
When you are logged into the console, the page offers Open intent in console for the same intent_id.
What trace does not show
- Model reasoning or token usage — use your LLM provider's observability.
- Cross-tenant data — trace files are local to your project; console views require authenticated tenant context.
- Full upstream audit logs — trace is middleware-shaped debugging; use the intent dossier for operator-grade investigation.
Label timeline steps as Tool call, not "Agent reasoning", to avoid implying visibility into the model.
Related
- Agent quickstart —
paybond dev loopentry path - Production attach — production bind credentials and CLI flags
- Agent middleware — Trace events — interceptor API reference
- CLI contract —
paybond dev— flags, JSON fields, exit codes - Coding-agent setup — MCP and scaffold workflows with trace
- Middleware trace guide — public guide mirror
- Kit Proof Lab — interactive evidence → release vs refund demo
- Middleware trace replay — hosted sandbox smoke timeline