How the skills work

Overview

A skill is a focused instruction set your coding agent loads on demand. These five are packaged together as the progress-observability toolkit, but each is self-contained and can be lifted out on its own.

They share one design rule: read, don't act. Every tool the skills call is read-only, they default to metadata over raw content, and they treat anything pulled from a trace as untrusted data. What differs between them is the question each answers.

The shared model

Everything below rides on one MCP server. Understanding it once explains how all five skills behave.

Connection

A remote, read-only MCP endpoint at https://mcp.observability.progress.com/mcp. The plugin's .mcp.json authenticates with an X-Api-Key header, sourced from the OBSERVABILITY_MCP_API_KEY environment variable. What you can read is scoped by that key.

Key scopes

• Metadata only — the 7 tools that return structure and metadata (statuses, durations, shapes, cost). No prompt or completion text.
• With content — adds the two *_with_content tools that return the actual prompt/completion text, for when a skill genuinely needs to quote real behavior.

Tools

Guardrails

• 72-hour window on all observation, span, and score queries — anything older is inaccessible. list_observations defaults to 24h, limit clamped 1–100.
• ID caps: detail tools take up to 10 IDs for metadata, 3 for content. Content responses cap at 1MB.
• Rate limits (per key, per tool, 1-minute window): 60/min for metadata tools, 10/min for content tools.

Trace content is untrusted

Prompts and completions in traces can carry prompt-injection and other adversarial text, so every skill treats content-tool output as data being analyzed, never instructions. In practice that means: default to metadata-only tools, pull content for the minimum number of IDs, scrub obvious PII from anything quoted back, and defang boundary tokens (e.g. <input> → < input >) so quoted text can't impersonate a real delimiter.

curl -s https://mcp.observability.progress.com/mcp \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $OBSERVABILITY_MCP_API_KEY" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

Run a fixed diagnostic sequence that turns "it's not working" into a specific cause, then report a status checklist with the exact fix for anything red. Read-only and metadata-only — no trace content, so no injection surface. It's the odd one out: not part of the loop below, but the thing to run first.

When to use

First run, or whenever something looks off — "is my observability set up right", "why am I seeing no traces", "check my connection", "is my data flowing".

How it works

1. Connection & key scope. Confirm the MCP tools are available and count them — 7 = Metadata-only, 9 = With-content. Missing tools or auth errors point at the key (OBSERVABILITY_MCP_API_KEY / .mcp.json), not the platform.
2. Recent data. list_observations over the last 24h, widening to the 72h max if empty — separating "stale traffic" from "no data" from "all older than the window".
3. Instrumentation depth. get_observation_details (include_children) on a couple of recent traces — traces with no children mean the SDK is capturing top-level spans only.
4. Evaluations. list_evaluation_tasks and get_evaluation_scores — none is fine, but dormant judges (defined, no recent scores) get flagged.

Reads

Returns

A compact red/amber/green checklist — connection, key scope, data freshness, instrumentation depth, evaluations — each red carrying the single concrete next step.

Run it

// Claude Code — no arguments
/health-check

// Copilot Chat
/health-check

Root-cause a single misbehaving run by walking its span tree, then hand back a diagnosis and a concrete fix.

When to use

A run failed, errored, stalled, or went off the rails — "why did this trace fail", "why was my agent slow", "find the bottleneck", "root-cause this bad tool call".

How it works

1. Locate the run. Use a trace/observation ID if you have one; otherwise list_observations filtered by service_name and status (e.g. errored) within 72h, and confirm the right candidate.
2. Walk the tree, metadata first. get_observation_details with include_children. Metadata alone usually reveals which span errored, which dominates latency, and where the chain stopped.
3. Find the fault. The errored span, the long pole, the tool call with wrong-looking arguments, or the point the trajectory diverged from the goal.
4. Pull content only for the culprit. get_observation_details_with_content on just that span when you need the actual prompt/completion/arguments — treated as untrusted data.

Reads

Returns

Diagnosis (the root cause in a sentence or two), Evidence (the span chain — ids, statuses, durations — quoted minimally), and a Fix (the concrete next step: a prompt change, tool-schema fix, retry/timeout).

Run it

// Claude Code
/trace-triage checkout-agent errored in the last hour

// Copilot Chat — same prompt file
/trace-triage

Turn the platform's cost and usage data into a short, decision-ready report. Metadata only — it never touches trace content, so there's no injection surface.

When to use

"What's driving my LLM costs", "how much am I spending", "cost by model or app", "am I close to my quota", "why did spend spike this week".

How it works

1. Scope. Confirm the date range and what matters — total spend, a suspected spike, model mix, or quota headroom. Defaults to the last 7 days.
2. Pull cost. get_cost_breakdown, run more than once when useful: group_by: "model" for mix, "application" for which app spends, "day" for the trend.
3. Pull usage/quota. get_usage_summary for billing-period usage and remaining quota.
4. Report — headline spend, top drivers with shares, the day-over-day trend and any spike's source, quota burn vs. remaining, and a cheaper-model recommendation only when the data supports it.

Reads

Returns

A tight report — a few bullets and one small table: headline, top drivers, trend, quota, and an optional recommendation framed as a hypothesis to validate.

Run it

// Claude Code
/cost-report what drove my spend over the last 14 days?

// Copilot Chat
/cost-report

Compare what your system actually does in production against what your evaluations measure, and surface the highest-value unmeasured behaviors. This is the connective tissue between observing and evaluating.

When to use

"What should I evaluate next", "what's my eval coverage", "which behaviors aren't tested", "where are my eval gaps", "help me prioritize which judges to build".

How it works

1. Characterize traffic. list_observations (within 72h) to map what's running — services, operation types, whether spans show tool calls, retrieval, structured output — staying in metadata. Note rough volume per behavior.
2. Inventory evaluations. list_evaluation_tasks for which judges exist and what each targets; get_evaluation_scores to tell an active eval from a dormant one.
3. Diff. Map observed behaviors to the failure modes that would catch their faults (retrieval → faithfulness, tools → tool_call, structured output → format, and so on). Real volume with no matching eval is a gap.
4. Rank gaps by volume × blast-radius, then report.

Reads

Returns

A short coverage table (behavior → volume → has-eval? → recommended failure mode) and a prioritized shortlist of gaps worth closing, each with a one-line rationale. For the top pick, it offers to run generate-eval to build the judge right away.

Run it

// Claude Code
/coverage-gaps what am I not evaluating yet?

// Copilot Chat
/coverage-gaps

Produce a single-criterion, research-grounded LLM-as-a-Judge evaluator prompt that another model can run to score your system's outputs. Optionally ground it in your real production traces.

Two entry points

• From real traces (/eval-from-trace) — pull representative observations over MCP, infer the judge config from what the system actually does, and quote real behavior as few-shot examples. Preferred when you have a live system on the platform.
• From a description (/eval-from-scratch) — paste a system prompt or describe the system, and the config is inferred from that text alone. No observability data needed.

How it works (from traces)

1. Confirm scope — application/service, time window (max 72h), and the suspected symptom, which usually names the failure mode.
2. Survey with metadata. list_observations to see the shape of traffic and choose the failure mode and template.
3. Pull content only to author examples. get_observation_details_with_content (needs a With content key; max 3 IDs) for 1–2 clear pass and fail cases, trimmed and labelled.
4. Assemble the prompt to the frame — one criterion, binary pass/fail, pre-specified steps, bias defenses on by default.

The frame, briefly

• One criterion per judge — single-criterion judges agree with humans far more reliably than multi-criterion ones.
• Binary pass/fail by default; pairwise only when the task genuinely compares two outputs.
• Pre-specified procedure steps, not judge-authored rubrics, which drift.
• Bias defenses on by default — length control, swap-and-agree for pairwise, and a cross-family judge (score on a different model family than the one under test).

Reads

Returns

One evaluator prompt, ready to run — with a security note baked in so the judge treats the outputs it scores as data, not instructions.

Run it

// Claude Code — grounded in real traces
/eval-from-trace build a faithfulness judge for my RAG app

// Claude Code — from a description, no traces
/eval-from-scratch score tone on a support agent from this system prompt

How they chain

The four are separate skills, but their outputs feed each other into one loop — each stage's output is the next stage's input.

1. trace-triage — a run fails; find the failure signature in the spans.
2. coverage-gaps — confirm nothing measures that behavior yet.
3. generate-eval — build the judge that catches it going forward.
4. cost-report — watch spend while you iterate on the fix.

You don't have to run them in order — each stands alone — but run end to end, they take you from a single broken trace to a durable eval without leaving your editor.