Your AI agent takes first watch: automated incident response for an internal OMS

Give your team an on-call agent that acknowledges the page, reads your runbook, investigates on your own servers with your own credentials, and writes the evidence back onto the incident — before anyone's phone rings at 2am. Here's how an IT team sets it up.

The system we're protecting

Every business that moves product runs some flavor of this: an internal Order Management System. Ours is four private services on a VLAN — nothing public-facing, nothing an external uptime checker can even reach:

• oms-api-01 — the order-intake API (orders in, from the storefront and CSV imports)
• oms-db-01 — Postgres: orders, inventory, the source of truth
• oms-worker-01 / -02 — fulfillment workers that drain the queue and cut shipping labels
• oms-queue-01 — the job broker the workers pull from

Because none of these are reachable from the outside, we watch them from the inside: a pylon-beacon agent on each box pushes CPU, memory, disk, and custom health metrics out to PylonMon over one outbound HTTPS connection. If a box goes quiet, that silence is itself the alert. (If you haven't set beacons up yet, the getting-started guide covers it — this post picks up where a team already has monitoring in place.)

The problem we're solving isn't detecting the outage. It's the 2am gap between "something broke" and "a human with context is looking at it." That gap is where orders pile up and revenue leaks. So we're going to put an AI agent in that gap.

1. Stand up the team workspace

On Enterprise, a workspace is a real team, not a login. In Settings → Team & roles we invite the people who run the OMS and give each a role — read-only for the folks who just watch dashboards, write for the engineers, admin for the leads:

Roles are real: read/write/admin/owner. A read-only night-desk account can see everything and touch nothing.

2. Set up the on-call rotation

Now the pager. Under Alerts → On-call we build a weekly rotation and — the part teams love — each person sets how they personally want to be paged. Maria takes SMS, Sam wants a phone call, Priya routes to her Slack DM. One rotation, per-member paging preference:

Monitors, escalation ladders, and incident rules can all page "whoever's on-call" — and each person is reached the way they asked. But here's the thing: we'd rather not page any of them for something an agent can handle. That's next.

3. Add the AI agent as a teammate

The agent joins the team like a person — through a channel. Under Alerts → + → 🤖 AI agent we point a channel at our agent runtime's HTTPS endpoint. Every alert and recovery POSTs a structured JSON envelope (no prose to parse), including write-back URLs and the incident context:

PylonMon never holds a credential to your infrastructure and never runs a model. It just hands your agent the alert with full context; the agent works on your systems, with your access.

The agent's write-backs (ack, note, resolve) authenticate with its own key — and this is the important safety part. We create an agent-scoped key: it can read monitors, incidents and logs, and ack/note/resolve incidents, and nothing else. It can't create, edit, pause, or delete a monitor or channel, can't declare incidents, can't touch the status page. A misbehaving agent can annotate an incident; it can never dismantle your monitoring. That's the leash:

Agent scope = read + incident write-back only. Notes it leaves are attributed 🤖 <key name> on the timeline, so humans always know what the agent did versus what a person did.

4. Write the runbook — it onboards your humans AND your agent

Here's the move that makes the whole thing reliable instead of a party trick. We write a short runbook for the OMS database and attach it to oms-db-01's monitor. The same document does double duty: it's what a new hire reads on their first on-call shift, and it's what the agent reads before it touches anything. Write your procedure once; it trains both.

# oms-db-01 disk pressure

## Likely cause
Postgres WAL or the order_events audit table. WAL
bloats when a replica falls behind; order_events grows
unbounded if the nightly prune job failed.

## Safe remediation (in order)
1. Check df -h /var/lib/postgresql and pg_wal size.
2. If WAL: confirm replica lag, then checkpoint.
3. If order_events: run /opt/oms/prune_events.sh (safe,
   idempotent, keeps 90 days).
## Do NOT
- Never DROP or TRUNCATE by hand. Never touch pg_wal
  files directly. If unsure, escalate to on-call.

Notice the runbook has a Do NOT section. That's not just for humans — it's the guardrail that keeps the agent doing the safe, documented thing instead of improvising a DROP TABLE at 2am.

5. Point your agent at PylonMon (the Claude example)

Because PylonMon is a standard MCP server, an MCP-native agent needs almost no glue. For Claude, it's a one-file skill and a key kept outside any repo. On the box your agent runs on:

ops@agent-runner:~$ mkdir -p ~/.pylonmon && nano ~/.pylonmon/key   # paste the agent key
ops@agent-runner:~$ curl -fsSL https://pylonmon.com/agent-skill > ~/.claude/skills/pylonmon/SKILL.md
✓ MCP endpoint https://pylonmon.com/mcp reachable · 10 tools
   list_monitors · get_monitor · list_incidents · get_incident · ack_incident
   add_incident_note · resolve_incident · search_logs · list_runbooks · get_runbook

Any MCP agent works the same way — the tools and the discipline are baked into the server. The full copy-paste skill is in the docs.

6. The incident, start to finish

It's 02:47. The nightly prune job on oms-db-01 silently failed two days ago, and the order_events table just crossed 90% disk. Watch what happens — and notice that no human is involved until the very end, and only because the agent chose to loop one in.

Two minutes, no phone rang, orders never stopped flowing for more than the blip. The agent followed your runbook — including the guardrails — and knew the difference between "fixed the symptom" and "here's the real bug a human should look at."

And when the agent can't safely resolve something — an ambiguous failure, anything the runbook says to escalate, or a monitor that's still down after remediation — it doesn't pretend. It leaves its findings on the timeline and lets the escalation ladder page the human on-call, who wakes up to a half-solved problem with the evidence already gathered. That's the whole value: your night shift is an agent that never sleeps, and your humans get paged less and arrive better-informed.

The morning after

Maria opens the dashboard with her coffee. There's one resolved incident overnight, a clean 🤖 timeline showing exactly what happened and what was done, and a flag that the prune cron needs a real fix. Nobody was woken. Sam files a one-line ticket for the cron bug. That's a good night.

Honest note on the tier

The AI-agent channel and agent-scoped keys are an Enterprise feature ($32/mo — flat, not per-seat, up to 50 members). If you're a solo operator, the free and Pro tiers still do real monitoring and human alerting; the autonomous-agent layer is the piece that earns the Enterprise line, and it's priced for a team, not an enterprise procurement department. The MCP endpoint itself works with any agent runtime — you're never locked to one vendor, and PylonMon never becomes a place your infrastructure credentials live.

If your team runs anything that pages a human at 2am, give the first watch to an agent instead — start a workspace → or read the BYO-AI docs.