How the skills work

Overview

A skill is a focused instruction set your coding agent loads on demand. These five are packaged together as the progress-observability toolkit, but each is self-contained and can be lifted out on its own.

They share one design rule: read, don't act. Every tool the skills call is read-only, they default to metadata over raw content, and they treat anything pulled from a trace as untrusted data. What differs between them is the question each answers.

The shared model

Everything below rides on one MCP server. Understanding it once explains how the platform-reading skills behave - scaffold-agent and instrument-agent are the exceptions: they write code and make no MCP calls at all, leaving trace confirmation to health-check.

Connection

A remote, read-only MCP endpoint at https://mcp.observability.progress.com/mcp. The plugin's .mcp.json authenticates with an X-Api-Key header, sourced from the OBSERVABILITY_MCP_API_KEY environment variable. What you can read is scoped by that key.

Key scopes

• Metadata only - structure and metadata (statuses, durations, shapes, cost). No prompt or completion text. Detail tools: get_observation_details, get_evaluation_task.
• With content - adds get_observation_details_with_content and get_evaluation_task_with_content, which return the actual prompt/completion text. The metadata tools remain available too.

MCP API keys come from API Keys and are not available on the free tier - without one, none of the platform-reading skills (health-check onward) can run. A Metadata-only key exposes 7 tools; a With-content key exposes all 9 (verified against the live server). Scope is detected by presence of the *_with_content tools. Calling a content tool may require an interactive approval (elicitation) - on denial, skills continue metadata-only.

Tools

Guardrails

• 72-hour window on all observation, span, and score queries - anything older is inaccessible. list_observations defaults to 24h, limit clamped 1–100.
• ID caps: detail tools take up to 10 IDs for metadata, 3 for content. Content responses cap at 1MB.
• Rate limiting - the server may throttle bursts; skills back off and retry once before reporting.

Trace content is untrusted

Prompts and completions in traces can carry prompt-injection and other adversarial text, so every skill treats content-tool output as data being analyzed, never instructions. In practice that means: default to metadata-only tools, pull content for the minimum number of IDs, scrub obvious PII from anything quoted back, and defang boundary tokens (e.g. <input> → < input >) so quoted text can't impersonate a real delimiter.

curl -s https://mcp.observability.progress.com/mcp \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $OBSERVABILITY_MCP_API_KEY" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

Retrofit Progress Observability onto an agent you already have - Python, TypeScript/JavaScript, or .NET - with the smallest possible diff, then run it once so spans start flowing. The mirror image of scaffold-agent.

This skill writes code - the instrumentation edits and nothing else. It makes no MCP calls: confirming the traces arrived is left to health-check.

When to use

An existing project that isn't on the platform yet - "instrument my agent", "add observability to this repo", "connect this to Progress Observability" - or when traces should be flowing and aren't.

How it works

1. Detect - language, LLM SDKs and frameworks in use, any existing telemetry, and how the app takes config. It reports the planned diff before editing anything.
2. Wire - per a verified language reference: progress-observability (PyPI), @progress/observability (npm), or Progress.Observability.Instrumentation (NuGet). Init at process start, before any LLM client exists; Integration key from config, never hardcoded; flush on exit.
3. Run once - with a keyless path available: decorated or wrapped functions emit real spans with no LLM credentials, so the pipeline can be proven before model keys exist.
4. Hand off - no platform read: it reports the wiring, then tells you to run your agent and confirm the traces are flowing in at observability.progress.com for your app_name.

Auto-instrumentation covers 30+ integrations - OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex, Mistral, Groq, Ollama, Cohere, WatsonX and more; LangChain, LlamaIndex, CrewAI, Haystack, OpenAI Agents, MCP; the major vector stores - plus any OpenAI-compatible endpoint (OpenRouter, LiteLLM, vLLM, gateways), which needs nothing but a base URL. The full list ships in the skill's references, extracted from the SDKs themselves. Anything genuinely uncovered gets explicit structure via decorators or a span wrapper, and the skill says plainly which of the two it's doing rather than implying coverage it lacks.

Tested against real fixtures

Three fixture repos - Python, TypeScript, and .NET - hold deliberately uninstrumented agents, the expected result after the skill runs, and a harness that verifies over MCP that spans actually landed. Every Monday, CI runs seven end-to-end jobs against the live platform and an automated session re-applies the skill to every fixture and grades the result - so the instructions can't silently rot.

Returns

A minimal diff (dependency, init, config wiring, flush), a run that emits at least one trace, and a hand-off telling you where to see the traces and how to confirm them with /health-check - plus, if a run misbehaves, the usual causes to check: init ordering, key type, or a missing flush.

Run it

// Claude Code
/instrument-agent add observability to this repo

// Copilot Chat
/instrument-agent

Create a new .NET agent project for your domain with the observability wiring already in place, starting from the dotnet-agent-starter template.

This skill writes files. It creates a project on disk and makes no MCP calls at all. Its sibling instrument-agent covers the other case: observability for an agent that already exists.

When to use

Starting something new - "build me an agent that triages support tickets", "scaffold a .NET agent with observability", "new agent project from the template". Not for instrumenting an existing app.

How it works

1. Gather the domain - what the agent does, the 2–4 actions it needs, whether it answers from documents, and a short name.
2. Fetch the template at its newest release tag, then drop the .git folder.
3. Fill the marked slots - app name, system prompt, tools, corpus. Nothing else is touched.
4. Verify with dotnet build before reporting success.

Generated tools are working stubs over plausible in-memory data, each with a [Description] the model selects on, and at least one honest not-found path - so the new agent emits real tool-call spans immediately and trace-triage has something to walk.

Never changes

Returns

A project that compiles, plus the handoff: the dotnet user-secrets commands for your Azure OpenAI settings and the Progress Integration key, dotnet run, and then /health-check - after which the platform-reading skills work against your own agent.

Run it

// Claude Code
/scaffold-agent an agent that triages support tickets against our KB

// Copilot Chat
/scaffold-agent

Run a fixed diagnostic sequence that turns "it's not working" into a specific cause, then report a status checklist with the exact fix for anything red. Read-only and metadata-only - no trace content, so no injection surface. It's the odd one out: not part of the loop below, but the thing to run first.

When to use

First run, or whenever something looks off - "is my observability set up right", "why am I seeing no traces", "check my connection", "is my data flowing".

How it works

1. Connection & key scope. Confirm the MCP tools are available, then read the scope from presence of the content tools - get_observation_details_with_content present means With-content (9 tools); absent means Metadata-only (7). Missing tools or auth errors point at the key (OBSERVABILITY_MCP_API_KEY / .mcp.json), not the platform.
2. Recent data. list_observations over the last 24h, widening to the 72h max if empty - separating "stale traffic" from "no data" from "all older than the window".
3. Instrumentation depth. get_observation_details (include_children) on a couple of recent traces - traces with no children mean the SDK is capturing top-level spans only.
4. Evaluations. list_evaluation_tasks and get_evaluation_scores - none is fine, but dormant judges (defined, no recent scores) get flagged.

Reads

Returns

A compact red/amber/green checklist - connection, key scope, data freshness, instrumentation depth, evaluations - each red carrying the single concrete next step.

Run it

// Claude Code - no arguments
/health-check

// Copilot Chat
/health-check

Root-cause a single misbehaving run by walking its span tree, then hand back a diagnosis and a concrete fix.

When to use

A run failed, errored, stalled, or went off the rails - "why did this trace fail", "why was my agent slow", "find the bottleneck", "root-cause this bad tool call".

How it works

1. Locate the run. Use a trace/observation ID if you have one; otherwise list_observations filtered by service_name and status (e.g. errored) within 72h, and confirm the right candidate.
2. Walk the tree, metadata first. get_observation_details with include_children. Metadata alone usually reveals which span errored, which dominates latency, and where the chain stopped.
3. Find the fault. The errored span, the long pole, the tool call with wrong-looking arguments, or the point the trajectory diverged from the goal.
4. Pull content only for the culprit. get_observation_details_with_content on just that span when you need the actual prompt/completion/arguments - treated as untrusted data.

Reads

Returns

Diagnosis (the root cause in a sentence or two), Evidence (the span chain - ids, statuses, durations - quoted minimally), and a Fix (the concrete next step: a prompt change, tool-schema fix, retry/timeout).

Run it

// Claude Code
/trace-triage checkout-agent errored in the last hour

// Copilot Chat - same prompt file
/trace-triage

Turn the platform's cost and usage data into a short, decision-ready report. Metadata only - it never touches trace content, so there's no injection surface.

When to use

"What's driving my LLM costs", "how much am I spending", "cost by model or app", "am I close to my quota", "why did spend spike this week".

How it works

1. Scope. Confirm the date range and what matters - total spend, a suspected spike, model mix, or quota headroom. Defaults to the last 7 days.
2. Pull cost. get_cost_breakdown, run more than once when useful: group_by: "model" for mix, "application" for which app spends, "day" for the trend.
3. Pull usage/quota. get_usage_summary for billing-period usage and remaining quota.
4. Report - headline spend, top drivers with shares, the day-over-day trend and any spike's source, quota burn vs. remaining, and a cheaper-model recommendation only when the data supports it.

Reads

Returns

A tight report - a few bullets and one small table: headline, top drivers, trend, quota, and an optional recommendation framed as a hypothesis to validate.

Run it

// Claude Code
/cost-report what drove my spend over the last 14 days?

// Copilot Chat
/cost-report

Compare what your system actually does in production against what your evaluations measure, and surface the highest-value unmeasured behaviors. This is the connective tissue between observing and evaluating.

When to use

"What should I evaluate next", "what's my eval coverage", "which behaviors aren't tested", "where are my eval gaps", "help me prioritize which judges to build".

How it works

1. Characterize traffic. list_observations (within 72h) to map what's running - services, operation types, whether spans show tool calls, retrieval, structured output - staying in metadata. Note rough volume per behavior.
2. Inventory evaluations. list_evaluation_tasks for which judges exist and what each targets; get_evaluation_scores to tell an active eval from a dormant one.
3. Diff. Map observed behaviors to the failure modes that would catch their faults (retrieval → faithfulness, tools → tool_call, structured output → format, and so on). Real volume with no matching eval is a gap.
4. Rank gaps by volume × blast-radius, then report.

Reads

Returns

A short coverage table (behavior → volume → has-eval? → recommended failure mode) and a prioritized shortlist of gaps worth closing, each with a one-line rationale. For the top pick, it offers to run generate-eval to build the judge right away.

Run it

// Claude Code
/coverage-gaps what am I not evaluating yet?

// Copilot Chat
/coverage-gaps

Produce a single-criterion, research-grounded LLM-as-a-Judge evaluator prompt that another model can run to score your system's outputs. Optionally ground it in your real production traces.

Want to try the frame without installing anything? The free eval builder renders the same judge prompts in your browser.

Two entry points

• From real traces (/eval-from-trace) - pull representative observations over MCP, infer the judge config from what the system actually does, and quote real behavior as few-shot examples. Preferred when you have a live system on the platform.
• From a description (/eval-from-scratch) - paste a system prompt or describe the system, and the config is inferred from that text alone. No observability data needed.

How it works (from traces)

1. Confirm scope - application/service, time window (max 72h), and the suspected symptom, which usually names the failure mode.
2. Survey with metadata. list_observations to see the shape of traffic and choose the failure mode and template.
3. Pull content only to author examples. get_observation_details_with_content (needs a With content key; max 3 IDs) for 1–2 clear pass and fail cases, trimmed and labelled.
4. Assemble the prompt to the frame - one criterion, binary pass/fail, pre-specified steps, bias defenses on by default.

The frame, briefly

• One criterion per judge - single-criterion judges agree with humans far more reliably than multi-criterion ones.
• Binary pass/fail by default; pairwise only when the task genuinely compares two outputs.
• Pre-specified procedure steps, not judge-authored rubrics, which drift.
• Bias defenses on by default - length control, swap-and-agree for pairwise, and a cross-family judge (score on a different model family than the one under test).

Reads

Returns

One evaluator prompt, ready to run - with a security note baked in so the judge treats the outputs it scores as data, not instructions.

Run it

// Claude Code - grounded in real traces
/eval-from-trace build a faithfulness judge for my RAG app

// Claude Code - from a description, no traces
/eval-from-scratch score tone on a support agent from this system prompt

How they chain

The four are separate skills, but their outputs feed each other into one loop - each stage's output is the next stage's input.

1. trace-triage - a run fails; find the failure signature in the spans.
2. coverage-gaps - confirm nothing measures that behavior yet.
3. generate-eval - build the judge that catches it going forward.
4. cost-report - watch spend while you iterate on the fix.

You don't have to run them in order - each stands alone - but run end to end, they take you from a single broken trace to a durable eval without leaving your editor.