paybondpaybond
Sign in

Middleware trace

How Paybond visual trace works — local dev dashboard, event model, timeline UI, CLI trace, and hosted replay.

Paybond middleware trace shows what happens at the tool boundary: policy authorization, handler execution, evidence submission, and sandbox settlement. It does not show model reasoning or chain-of-thought — Paybond guards side-effecting tools, not LLM inference.

Use trace to answer: Did Harbor authorize this spend? Did my handler run? Was evidence submitted? What settlement state did sandbox return?

Quick start

Terminal 1 — scaffold policy, validate, and smoke:

paybond dev loop --offline   # or paybond dev loop after paybond login

Terminal 2 — open the local dashboard (same project directory):

paybond dev trace

The UI listens at http://127.0.0.1:9477. Select a run from the sidebar or open the trace_url printed by smoke/loop (for example http://127.0.0.1:9477/runs/<run-id>).

Hosted replay (no local server): /demo/agent-trace.

What the timeline shows

Each guarded tool call produces a vertical timeline with phase-colored steps:

PhaseBadge colorTypical step
ToolTealTool call resolved (tool_selected)
AuthorizeAmberSpend approved, denied, or held for approval
EvidencePurpleAuto-evidence submitted after a successful handler
ResultGreenHandler duration, spend finalized, sandbox settlement status

Typical happy path:

Tool call: travel.book_hotel
  → Paybond approved $187.00
  → Tool executed (42ms)
  → Evidence submitted
  → Settlement: released

Denied or approval-held calls include spend_denied or approval_required steps under Authorize so you can debug policy failures without Harbor logs.

Observability surfaces

SurfaceCommand / URLBest for
Local dashboardpaybond dev trace:9477First-time adoption, smoke debugging, live refresh while iterating
Terminal tracepaybond agent run trace --run-id <id> --format tableCI, headless environments, multi-step bind → execute flows
Hosted replay/demo/agent-trace?intent=<uuid>Sharing a visual walkthrough without running the local server
SDK sinktraceSink on PaybondAgentRun.bind()Custom telemetry, tests, in-app debug panels
Console dossier/console/operations/intents/<intentId>Production operators — intent-centric audit (requires auth)
Console agent run/console/operations/agent-runs/<runId>Tenant-scoped run timeline (when Gateway persistence is enabled)

All local and CLI surfaces share the same PaybondTraceEvent vocabulary. The hosted replay animates smoke JSON with the same phase colors as the local dashboard.

Dev, sandbox, and production

Sandbox (Paybond environment) and dev (local tooling on your machine) are not the same thing. Production binds can emit trace events too, but nothing is recorded unless you wire a sink.

Local dashboard vs trace events

Local dashboard (paybond dev trace)PaybondTraceEvent stream
What it isHTTP UI on 127.0.0.1:9477 reading .paybond/dev-trace.jsonlStructured interceptor events (tool_selected, spend_authorized, …)
Runs in production deploys?No — local dev tool onlyCan — if you pass traceSink or use Gateway reporting
On by default?Only after paybond dev smoke / dev loopNo in production app binds

Do not run paybond dev trace in production. It is for adoption and local debugging on your laptop.

What turns trace on

On PaybondAgentRun.bind(), Kit resolves the sink as:

traceSink ?? onTrace ?? resolveDevTraceSink()

The only automatic sink (without you passing traceSink) is the in-process dev collector, active only after activateDevTraceCollector — that is, during paybond dev smoke or paybond dev loop. A typical production instrument().bind() does not attach a sink unless you set one.

By environment

CapabilityDev (local)SandboxProduction
paybond dev trace dashboardYesYes (same local UI after smoke)No
.paybond/dev-trace.jsonlYesYes (when dev collector ran)No
.paybond/runs/<id>.trace.jsonYesYesYes (CLI bind + execute)
Auto traceSink on app bindOnly if dev collector activeSameNo — opt in
paybond agent run trace (CLI)YesYesYes
Hosted /demo/agent-trace replayYesYes (smoke JSON)N/A (smoke replay only)
Console intent dossierWhen logged inYesYes
Console agent-run timelineWhen Gateway reporter usedYesYes — opt in

Sandbox smoke with paybond login still writes trace files when the dev collector or CLI trace sinks are active. That is real Harbor authorization against the sandbox Gateway — not an offline mock — but the dashboard remains a process on your machine.

Production observability

Default production middleware binds do not enable trace. Operators use the intent dossier (/console/operations/intents/<intentId>) for audit-grade investigation.

To record middleware trace in production:

  1. Custom telemetry — pass traceSink on bind and forward events to your observability stack (TypeScript and Python).
  2. Console agent-run view — wire paybond.harbor.createAgentRunTraceReporter(runId) (TypeScript) or paybond.harbor.create_agent_run_trace_reporter(run_id) (Python) so events persist to Gateway and appear at /console/operations/agent-runs/<runId>.
  3. CLI multi-step bindpaybond agent run bind --production writes .paybond/runs/<run_id>.trace.json and forwards events to Gateway when paybond login credentials are available (TypeScript and Python CLI).

Gateway trace reporting is fire-and-forget — failures are swallowed so middleware execution is never blocked on telemetry.

Production examples

CLI — production attach with trace

paybond agent run bind attaches trace sinks automatically: a local .paybond/runs/<run_id>.trace.json file on every platform, plus Gateway forwarding when the CLI has paybond login credentials (TypeScript and Python).

paybond login

paybond agent run bind --production \
  --attach-intent-id "$PAYBOND_INTENT_ID" \
  --capability-token "$PAYBOND_CAPABILITY_TOKEN" \
  --payee-did "$APP_PAYEE_DID" \
  --payee-signing-seed-hex "$APP_PAYEE_SEED_HEX" \
  --agent-recognition-key-id "$APP_AGENT_RECOGNITION_KEY_ID" \
  --agent-recognition-signing-seed-hex "$APP_AGENT_RECOGNITION_SEED_HEX" \
  --policy-file paybond.policy.yaml \
  --format json

paybond agent tool execute --run-id <run-id> \
  --operation paid-tool \
  --tool-call-id call-1 \
  --result-body '{"status":"ok","cost_cents":100}' \
  --format json

# Local timeline (no browser)
paybond agent run trace --run-id <run-id> --format table

# Console (authenticated session + real run id): /console/operations/agent-runs/<run-id>

Signing credential flags and env fallbacks are documented in Production attach. There is no paybond dev trace step in production — that dashboard is local dev only.

SDK — custom telemetry (TypeScript and Python)

Forward events to your own stack with traceSink / trace_sink on a production attach bind:

TypeScript:

import type { PaybondTraceEvent } from "@paybond/kit/agent";
import { Paybond, seed32FromHex } from "@paybond/kit";

const paybond = await Paybond.open({
  apiKey: process.env.PAYBOND_API_KEY!,
  expectedEnvironment: "production",
});

const traceSink = (event: PaybondTraceEvent) => {
  // Forward to OpenTelemetry, Datadog, structured logs, etc.
  console.log(JSON.stringify(event));
};

const run = await paybond.agentRun.bind({
  attach: {
    intentId: process.env.PAYBOND_INTENT_ID!,
    capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!,
    productionEvidence: {
      payeeDid: process.env.APP_PAYEE_DID!,
      payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!),
      agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!,
      agentRecognitionSigningSeed: seed32FromHex(
        process.env.APP_AGENT_RECOGNITION_SEED_HEX!,
      ),
    },
  },
  registry,
  traceSink,
});

await run.interceptor.wrapExecute({ /* tool call */ });

Python:

import os

from paybond_kit import Paybond

paybond = await Paybond.open(
    api_key=os.environ["PAYBOND_API_KEY"],
    expected_environment="production",
)

def trace_sink(event: dict[str, object]) -> None:
    # Forward to your observability stack
    print(event)

run = await paybond.agent_run.bind(
    {
        "attach": {
            "intent_id": os.environ["PAYBOND_INTENT_ID"],
            "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"],
            "production_evidence": {
                "payee_did": os.environ["APP_PAYEE_DID"],
                "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]),
                "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"],
                "agent_recognition_signing_seed": seed32_from_hex(
                    os.environ["APP_AGENT_RECOGNITION_SEED_HEX"]
                ),
            },
        },
        "registry": registry,
        "trace_sink": trace_sink,
    }
)

await run.interceptor.wrap_execute({...})

SDK — console agent-run timeline (TypeScript and Python)

To populate /console/operations/agent-runs/<runId> from application code, register the run and pass the Gateway reporter sink on bind.

TypeScript:

import { Paybond, seed32FromHex } from "@paybond/kit";

const paybond = await Paybond.open({
  apiKey: process.env.PAYBOND_API_KEY!,
  expectedEnvironment: "production",
});

const runId = "run-prod-1";
const reporter = paybond.harbor.createAgentRunTraceReporter(runId);
const traceSink = reporter.createSink({
  intentId: process.env.PAYBOND_INTENT_ID!,
  operation: "paid-tool",
  sandbox: false,
  allowedTools: ["paid-tool"],
});

const run = await paybond.agentRun.bind({
  runId,
  attach: {
    intentId: process.env.PAYBOND_INTENT_ID!,
    capabilityToken: process.env.PAYBOND_CAPABILITY_TOKEN!,
    productionEvidence: {
      payeeDid: process.env.APP_PAYEE_DID!,
      payeeSigningSeed: seed32FromHex(process.env.APP_PAYEE_SEED_HEX!),
      agentRecognitionKeyId: process.env.APP_AGENT_RECOGNITION_KEY_ID!,
      agentRecognitionSigningSeed: seed32FromHex(
        process.env.APP_AGENT_RECOGNITION_SEED_HEX!,
      ),
    },
  },
  registry,
  traceSink,
});

await run.interceptor.wrapExecute({ /* tool call */ });

// Optional: await in-flight Gateway writes before process exit
await reporter.flush();

Python:

import os

from paybond_kit import Paybond
from paybond_kit.agent.gateway_trace_reporter import AgentRunUpsertInput

paybond = await Paybond.open(
    api_key=os.environ["PAYBOND_API_KEY"],
    expected_environment="production",
)

run_id = "run-prod-1"
reporter = paybond.harbor.create_agent_run_trace_reporter(run_id)
trace_sink = reporter.create_sink(
    AgentRunUpsertInput(
        intent_id=os.environ["PAYBOND_INTENT_ID"],
        operation="paid-tool",
        sandbox=False,
        allowed_tools=["paid-tool"],
    )
)

run = await paybond.agent_run.bind(
    {
        "run_id": run_id,
        "attach": {
            "intent_id": os.environ["PAYBOND_INTENT_ID"],
            "capability_token": os.environ["PAYBOND_CAPABILITY_TOKEN"],
            "production_evidence": {
                "payee_did": os.environ["APP_PAYEE_DID"],
                "payee_signing_seed": seed32_from_hex(os.environ["APP_PAYEE_SEED_HEX"]),
                "agent_recognition_key_id": os.environ["APP_AGENT_RECOGNITION_KEY_ID"],
                "agent_recognition_signing_seed": seed32_from_hex(
                    os.environ["APP_AGENT_RECOGNITION_SEED_HEX"]
                ),
            },
        },
        "registry": registry,
        "trace_sink": trace_sink,
    }
)

await run.interceptor.wrap_execute(...)

await reporter.flush()

Python trace events are normalized to camelCase wire fields before Gateway POST so the console timeline renders tool names, amounts, and evidence ids correctly.

Data flow

flowchart LR
  subgraph kit [Kit interceptor]
    Interceptor[PaybondToolInterceptor]
    Sink[traceSink / dev collector]
  end
  subgraph persist [Project files]
    Jsonl[".paybond/dev-trace.jsonl"]
    RunFile[".paybond/runs/run_id.trace.json"]
    Audit[".paybond/dev-audit.jsonl"]
  end
  subgraph ui [Trace UI]
    Server["paybond dev trace :9477"]
    API["GET /api/events"]
    Dashboard[Vertical timeline HTML]
  end
  Interceptor --> Sink
  Sink --> Jsonl
  Sink --> RunFile
  Smoke[dev smoke / dev loop] --> Audit
  Server --> API
  API --> Jsonl
  API --> Dashboard
  1. PaybondToolInterceptor emits structured events as each side-effecting tool moves through authorize → execute → finalize → evidence.
  2. paybond dev smoke and paybond dev loop activate an in-process DevTraceCollector during the smoke bind/execute cycle. On completion, the collector writes one summary row to .paybond/dev-trace.jsonl (and a summary line to .paybond/dev-audit.jsonl).
  3. paybond dev trace starts a local HTTP server that merges events from disk and an in-memory ring buffer, then serves a self-contained HTML dashboard.
  4. The dashboard polls /api/events every two seconds and re-renders the selected run's timeline.
  5. Multi-step paybond agent run bind + tool execute writes .paybond/runs/<run_id>.trace.json for paybond agent run trace.

Run paybond dev trace from the same working directory as smoke/loop so .paybond/dev-trace.jsonl is found. The file keeps the last 100 runs.

Event model

Structured middleware events (PaybondTraceEvent) are the source of truth. The dashboard derives human-readable steps[] from them via devTraceStepsFromEvents.

Event typeWhen emittedKey fields
tool_selectedSide-effecting tool resolvedtoolName, toolCallId, operation
spend_authorizedHarbor allows the spenddecisionId, amountCents, auditId
spend_deniedHard denialmessage, code, auditId
approval_requiredApproval holdmessage, code, auditId
tool_executedHandler finisheddurationMs
spend_finalizedDecision consumed or releasedstatus: consumed | released
evidence_submittedAuto-evidence succeededevidenceId, presetId, sandboxLifecycleStatus

Each event includes runId, operation, and recordedAt for correlation.

Summary row (one line in .paybond/dev-trace.jsonl per smoke run):

{
  "id": "run-abc",
  "run_id": "run-abc",
  "preset": "travel",
  "operation": "travel.book_hotel",
  "intent_id": "00000000-0000-4000-8000-000000000001",
  "requested_spend_cents": 18700,
  "authorized": true,
  "evidence_submitted": true,
  "sandbox_lifecycle_status": "released",
  "steps": [ "..." ],
  "trace_events": [ "..." ]
}

Expand Raw trace JSON in the dashboard or use --format json on CLI commands for the full payload.

Local dashboard HTTP API

paybond dev trace binds to 127.0.0.1:9477 by default (--port to override).

RouteResponse
/Dashboard HTML — recent runs sidebar + timeline
/runs/<run-id>Same HTML, focused on the matching run
/api/eventsJSON { "events": DevTraceEvent[], "has_credentials": boolean }

has_credentials is true when PAYBOND_API_KEY is set in the trace server's environment (informational banner only).

The dashboard is bundled with @paybond/kit (kit/ts/dev/trace-ui/dashboard.html) — no separate npm package.

paybond agent sandbox smoke, paybond dev smoke, and paybond dev loop print checklist lines when table output is enabled:

✓ Trace → http://127.0.0.1:9477/runs/<run-id>
✓ Console → https://…/console/operations/intents/<intent-id>
✓ Replay → https://…/demo/agent-trace?intent=<intent-id>

JSON smoke output includes trace_url, console_url, and agent_trace_url in the data object. Console and replay links require a Harbor UUID intent_id from a real sandbox bind.

SDK integration (traceSink) — sandbox

For local or sandbox debugging, pass traceSink on bind (or trace_sink in Python). Production attach examples are in Production examples above.

TypeScript:

import type { PaybondTraceEvent } from "@paybond/kit/agent";

const events: PaybondTraceEvent[] = [];

const run = await paybond.agentRun.bind({
  bootstrap: { kind: "sandbox", operation: "travel.book_hotel", requestedSpendCents: 20_000 },
  registry,
  traceSink: (event) => events.push(event),
});

Python:

trace_events: list[dict[str, object]] = []

run = await paybond.agent_run.bind(
    {
        "bootstrap": {
            "kind": "sandbox",
            "operation": "travel.book_hotel",
            "requested_spend_cents": 20_000,
        },
        "registry": registry,
        "trace_sink": trace_events.append,
    }
)

onTrace / on_trace are deprecated aliases for traceSink / trace_sink.

Terminal trace (no browser)

After a multi-step bind and execute:

paybond agent run bind --sandbox \
  --operation travel.book_hotel \
  --requested-spend-cents 20000 \
  --completion-preset cost_and_completion

paybond agent tool execute --run-id <id> \
  --operation travel.book_hotel \
  --tool-call-id call-1 \
  --result-body '{"status":"completed","cost_cents":18700}'

paybond agent run trace --run-id <id> --format table

Reads .paybond/runs/<run_id>.trace.json. Use --format json for trace_events[] and steps[] in automation.

Hosted replay

/demo/agent-trace replays sandbox smoke output as an animated timeline using the same phase colors and step ordering as the local dashboard. Deep-link a real intent:

/demo/agent-trace?intent=<harbor-intent-uuid>

When you are logged into the console, the page offers Open intent in console for the same intent_id.

What trace does not show

  • Model reasoning or token usage — use your LLM provider's observability.
  • Cross-tenant data — trace files are local to your project; console views require authenticated tenant context.
  • Full Harbor audit logs — trace is middleware-shaped debugging; use the intent dossier for operator-grade investigation.

Label timeline steps as Tool call, not "Agent reasoning", to avoid implying visibility into the model.