observability-oss skills & plugins for progress-observability
Open source · for teams shipping AI agents

Find out why your agent failed - without leaving your editor.

Plugins and skills that connect Claude Code and GitHub Copilot to the Progress Observability Platform - root-cause failures, track cost, and build evals from your real traces, over MCP.

Install in 2 minutes Try the eval builder - free

No Progress account? The eval builder runs entirely in your browser - no signup, no key.

claude · your-agent-repo
> /trace-triage why did checkout-agent stall in the last hour?
 
● Scanning the last hour… 3 failed runs, 1 pattern
● Walking the span tree for run 9f2c… (41 spans)
● Fault isolated: refund_lookup → timeout after 3 retries
  ↳ called with order_id=null on every failing run
  ↳ null traces back to the extract_order step
 
Diagnosis: extract_order drops the order ID when the user pastes a full email thread - long inputs push it out of the prompt.
Fix: pin order_id to a structured output field; retry the 3 runs.
Next: /coverage-gaps - nothing measures this behavior yet.

What /trace-triage does with a failing run - diagnosis grounded in your real spans.

Overview

The toolkit

One MCP connection to the Progress Observability Platform, seven skills on top of it. Shipped from a single source as a Claude Code plugin and a VS Code / Copilot setup - the same skills run in either.

progress-observability Claude Code Copilot

instrument-agent retrofits observability onto the agent you already have - Python, TypeScript, or .NET; scaffold-agent creates a new instrumented .NET agent from the starter template. Once traces are flowing, health-check confirms the wiring and four read-only workflows run over the Progress Observability MCP server: trace triage, cost and usage reports, eval coverage gaps, and LLM-as-a-Judge eval generation grounded in your real traces.

/instrument-agent /scaffold-agent /health-check /trace-triage /cost-report /coverage-gaps /eval-from-trace

Run it three ways - a Claude Code plugin, a VS Code / Copilot setup, or any agent via npx skills. Install steps below.

Repository Docs Install
evsec

Security & compliance eval packs written from your agent's real traces - injection, PII, tool-abuse judges plus attack test-sets, mapped to OWASP LLM Top-10 and the EU AI Act. Try it on a bundled sample trace, no key needed.

Early access
Seven skills

Each does one job well

Use them through the plugin, or lift a single skill into any agent. All treat trace content as untrusted data; only the two setup skills write files.

/instrument-agent

Instrument the agent you already have

Detects the language and framework - Python, TypeScript, or .NET - wires the Progress SDK with the smallest possible diff, then runs it once so spans start flowing and hands off to health-check to confirm them. The mirror image of scaffold-agent.

Setup · start here
/scaffold-agent

Start a new agent, already traced

Clones the dotnet-agent-starter template and fills in your domain - instructions, tools, knowledge base - leaving the observability wiring intact. Writes a new project; makes no MCP calls.

Setup · new agent
/health-check

Check your setup

Read-only diagnostic - connection, key scope, whether traces are flowing, and how deep your instrumentation goes. The four below chain into a loop; this one you run first.

Setup · run first
/trace-triage

Root-cause a run

Walk a failed or slow run's span tree, find the fault, hand back a diagnosis and a fix.

 /cost-report

Watch the spend

Cost by model, app, and day; quota burn; and what drove a spike - decision-ready.

 /coverage-gaps

Find the blind spots

Which production behaviors have no eval measuring them, ranked by what to build first.

 /eval-from-trace

Build the judge

Research-grounded LLM-as-a-Judge evaluator prompts, grounded in your real traces.

Docs - how each skill works, what it reads, and how to run it → · Skill source on GitHub → · Free eval builder - try the judge frame in your browser →

How it works

Triage a failure, find the gap, build the eval, track the cost

The four skills chain into one loop - each stage's output is the next stage's input.

01
trace-triage

A run fails. Find the failure signature in the spans.

02
coverage-gaps

Confirm nothing measures that behavior yet.

03
eval-from-trace

Build the judge that catches it going forward.

04
cost-report

Watch spend while you iterate on the fix.

Templates

Getting-started templates

Starter repos pre-wired for observability - clone, add your keys, and you have a traced agent to build on. First one is live: dotnet-agent-starter - a .NET agent on Microsoft Agent Framework and Azure OpenAI, instrumented with the Progress SDK. Clone it directly, or let /scaffold-agent tailor it to your domain. Already have an agent? /instrument-agent retrofits the same wiring onto existing Python, TypeScript, or .NET code.

Quickstart

Running in three steps

What you need: a Progress Observability account on a paid plan, and two keys from API Keys - an MCP API key (acm_…) the skills read traces with, and an Integration API key (ac_p_…) your agent sends them with. MCP keys aren't on the free tier.

No account yet? Start free: the eval builder runs in your browser, and /instrument-agent and /scaffold-agent write code without any MCP key.

1 - set your keys · Progress Observability → API Keys · every setup
export OBSERVABILITY_MCP_API_KEY="acm_…"   # MCP key - skills read traces
export OBSERVABILITY_API_KEY="ac_p_…"       # Integration key - your agent sends them
2 - add the marketplace and install
/plugin marketplace add observability-oss/progress-observability-plugin
/plugin install progress-observability@progress-observability
3 - connect your agent, verify, then ask in plain language
/instrument-agent
/health-check
"why did checkout-agent stall in the last hour?"
"what's driving my spend this week?"
"what should I evaluate next?"
FAQ

Common questions

Do I need a paid Progress Observability plan?
For the skills that read the platform - health-check, trace-triage, cost-report, coverage-gaps, eval-from-trace - yes: they authenticate with an MCP API key, which isn't available on the free tier. But three things are free with no account at all: the eval builder (runs in your browser), /instrument-agent, and /scaffold-agent (both write code locally and make no MCP calls).
Which API keys do I need, and what does each one do?
Two, both from API Keys in the platform. The MCP API key (acm_…) is what the skills read traces with - Metadata-only scope covers everything except quoting raw trace text. The Integration API key (ac_p_…) is what your instrumented agent sends traces with. They're separate on purpose: reading and writing have different blast radii.
Is this official Progress software?
No - it's an independent, MIT-licensed community project, not affiliated with or endorsed by Progress Software. It builds on the public MCP server of the Progress Observability Platform.
What do the skills read and write?
Every platform-facing tool the skills call is read-only, they default to metadata over raw content, and anything pulled from a trace is treated as untrusted data - never as instructions. Only the two setup skills (instrument-agent, scaffold-agent) write files, and only in your project. Details in the docs.
Which coding agents does this work with?
Claude Code and GitHub Copilot (CLI and VS Code) get the full plugin. Any other agent - Cursor, Codex, Cline, Amp, and ~70 more - can install the individual skills via npx skills add. The MCP connection is standard, so anything that speaks MCP can use it.
Early access

Want a say in what gets built next?

This toolkit is early and moving fast, and the roadmap is set by the people using it. If you're shipping agents on Progress Observability, become a design partner - you get direct support from the maintainer, and your use cases decide what ships next.

Reach out on LinkedIn Open a GitHub issue

Maintained by Lyubomir Atanasov · also on Discord under the same name.