Devil Watchdog

API Docs

Overview

Devil Watchdog is a centralized event monitoring hub. Use these APIs to send events, report server health metrics, and send heartbeat pings from your applications, servers, and cron jobs.

All API endpoints accept and return JSON. Use Content-Type: application/json for POST requests.

Base URL: https://<your-deployment-domain>

Authentication

API requests are authenticated with an API key sent in the X-API-Key header. Each project has its own API key (prefixed with dw_). Get your API key from the project settings in the dashboard.

Header

X-API-Key: dw_your_api_key_here

Send Events

POST/api/eventsAuth: X-API-Key

Send a single event or a batch of up to 100 events. Events appear in the real-time feed and can trigger notifications.

Request Body

Send a single object or an array of objects (max 100):

channelstringrequired

Grouping category (1-100 chars). Examples: deploy, auth, payments, cron

namestringrequired

Event name (1-200 chars). Examples: deploy.success, user.signup, payment.failed

descriptionstring

Human-readable detail (max 2000 chars)

iconstring

Emoji icon (max 50 chars)

severityenumdefault: "info"

One of: info warning error critical

tagsstring[]

Up to 20 tags for filtering (each max 50 chars)

metadataobject

Arbitrary key-value data stored with the event

notifybooleandefault: false

Trigger a notification (Telegram, etc.) if project notifications are configured for this severity level

resolvesstring | string[]

Event names this event fixes. Open warning+ events with those names are marked resolved and leave the dashboard — e.g. tests.passed sending "tests.failed". With a matched hostname, only that server's events are resolved.

Example — Single Event

curl

curl -X POST https://your-domain.com/api/events \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dw_your_api_key" \
  -d '{
    "channel": "deploy",
    "name": "deploy.success",
    "description": "Deployed v2.4.1 to production",
    "icon": "🚀",
    "severity": "info",
    "tags": ["production", "v2.4.1"],
    "metadata": {
      "commit": "a1b2c3d",
      "duration_seconds": 42
    },
    "notify": true
  }'

Response

201 Created

{
  "success": true,
  "event": { "id": "uuid-here" }
}

Example — Batch

curl

curl -X POST https://your-domain.com/api/events \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dw_your_api_key" \
  -d '[
    { "channel": "cron", "name": "backup.started", "severity": "info" },
    { "channel": "cron", "name": "cleanup.done", "severity": "info" }
  ]'

201 Created

{
  "success": true,
  "events": [{ "id": "uuid-1" }, { "id": "uuid-2" }],
  "count": 2
}

Server Health Report

POST/api/servers/reportAuth: X-API-Key

Report server health metrics (disk, memory, CPU, services). The server must be registered in the dashboard first with a matching hostname. Thresholds are evaluated automatically and alerts are generated when metrics exceed configured limits.

Request Body

hostnamestringrequired

Must match the hostname registered in the dashboard (1-200 chars)

diskarray

Array of disk mount objects

memoryobject

Memory usage metrics

cpuobject

CPU usage metrics

servicesarray

Array of service status objects (max 50)

processesobject

Process-state counts (used for zombie-process alerting)

temperatureobject

Hardware temperatures in Celsius (used for overheating alerts)

customobject

Arbitrary key-value data

Object Schemas

disk[] items:

mountstringrequired

Mount point path (e.g. /, /home)

totalnumberrequired

Total space in bytes

usednumberrequired

Used space in bytes

percentnumberrequired

Usage percentage (0-100)

memory:

totalnumberrequired

Total memory in bytes

usednumberrequired

Used memory in bytes

availablenumberrequired

Available memory in bytes

swap_totalnumberdefault: 0

Total swap in bytes

swap_usednumberdefault: 0

Used swap in bytes

cpu:

usage_percentnumberrequired

CPU usage percentage (0-100)

load_1mnumber

1-minute load average

load_5mnumber

5-minute load average

load_15mnumber

15-minute load average

coresnumber

Number of CPU cores (used to normalize load thresholds)

services[] items:

namestringrequired

Service name (e.g. nginx, postgres)

runningbooleanrequired

Whether the service is currently running

processes:

totalnumberrequired

Total number of processes on the host

zombienumberrequired

Zombie (defunct) process count — triggers the zombie_warning / zombie_critical thresholds when exceeded. Used to catch child-reaper bugs and broken supervisor configurations.

temperature:

max_cnumberrequired

Hottest sensor reading, in Celsius

cpu_cnumber

CPU package sensor, where the host labels one

zonesarray

Every reading as { name, value_c } (max 40) — charted, and useful for telling a hot disk from a hot CPU after the fact

Alerts fire on cpu_c, falling back to max_c when the host labels no CPU sensor, against the temp_warning / temp_critical thresholds. Omit the whole object on a host with no sensors (any VPS) and nothing is evaluated.

Example

curl

curl -X POST https://your-domain.com/api/servers/report \
  -H "Content-Type: application/json" \
  -H "X-API-Key: dw_your_api_key" \
  -d '{
    "hostname": "web-prod-01",
    "disk": [
      { "mount": "/", "total": 107374182400, "used": 64424509440, "percent": 60 },
      { "mount": "/data", "total": 536870912000, "used": 322122547200, "percent": 60 }
    ],
    "memory": {
      "total": 17179869184,
      "used": 12884901888,
      "available": 4294967296
    },
    "cpu": {
      "usage_percent": 35.2,
      "load_1m": 1.5,
      "load_5m": 1.2,
      "load_15m": 0.9,
      "cores": 4
    },
    "services": [
      { "name": "nginx", "running": true },
      { "name": "postgres", "running": true },
      { "name": "redis", "running": true }
    ],
    "processes": {
      "total": 448,
      "zombie": 3
    },
    "temperature": {
      "cpu_c": 56.0,
      "max_c": 56.9,
      "zones": [
        { "name": "coretemp/Package id 0", "value_c": 56.0 },
        { "name": "nvme/Composite", "value_c": 48.9 }
      ]
    },
    "custom": {
      "uptime_hours": 720,
      "app_version": "2.4.1"
    }
  }'

200 OK

{
  "ok": true,
  "server": { "id": "uuid", "status": "healthy" },
  "alerts": []
}

When thresholds are breached, the alerts array contains details:

200 OK — with alerts

{
  "ok": true,
  "server": { "id": "uuid", "status": "warning" },
  "alerts": [
    {
      "metric": "disk",
      "label": "Disk /data",
      "value": 87,
      "threshold": 80,
      "severity": "warning"
    }
  ]
}

Default Thresholds

Disk warning: 80%

Disk critical: 90%

Memory warning: 80%

Memory critical: 95%

CPU warning: 80%

CPU critical: 95%

Zombie warning: 20

Zombie critical: 50

Zombie thresholds are counts, not percentages.

Heartbeat Ping

GET / POST / HEAD/api/ping/:slugAuth: None

Send a heartbeat ping from your application or cron job. The slug is a unique identifier generated when you create a heartbeat in the dashboard. No authentication is required — the slug itself acts as the token.

If the heartbeat doesn't receive a ping within its configured interval + grace period, it will be marked as down and a notification will be sent.

Example

curl

# Simple GET request — perfect for cron jobs
curl https://your-domain.com/api/ping/abc123xyz

# Or use POST
curl -X POST https://your-domain.com/api/ping/abc123xyz

# Add to end of a cron job
0 * * * * /usr/local/bin/backup.sh && curl -s https://your-domain.com/api/ping/abc123xyz

200 OK

{ "ok": true }

Rate Limits

All endpoints are rate-limited to 120 requests/minute per API key (or per slug for heartbeat pings). The rate limiter uses a sliding window.

When rate-limited, you'll receive a 429 response with these headers:

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Remaining: 0

{
  "error": "Rate limit exceeded",
  "retryAfterMs": 12000
}

Error Handling

All errors return JSON with an error field. Validation errors include a details object with per-field errors.

400Validation error — check the details field

401Missing or invalid API key

403Resource is disabled (e.g. paused server)

404Resource not found (e.g. unregistered hostname)

429Rate limit exceeded — wait and retry

500Server error — retry with backoff

Validation Error Example

400 Bad Request

{
  "error": "Validation failed",
  "details": {
    "channel": ["String must contain at least 1 character(s)"],
    "name": ["Required"]
  }
}

Full Examples

Bash — Server Monitoring Agent

Collect and report server metrics via cron. Schedule this to run every 5 minutes.

watchdog-agent.sh

#!/bin/bash
API_KEY="dw_your_api_key"
BASE_URL="https://your-domain.com"
HOSTNAME=$(hostname)

# Collect disk metrics
DISK_JSON=$(df -B1 --output=target,size,used,pcent -x tmpfs -x devtmpfs 2>/dev/null | \
  tail -n +2 | awk '{
    gsub(/%/,"",$4);
    printf "{\"mount\":\"%s\",\"total\":%s,\"used\":%s,\"percent\":%s}\n",$1,$2,$3,$4
  }' | paste -sd, | sed 's/^/[/;s/$/]/')

# Collect memory metrics
read -r MEM_TOTAL MEM_USED MEM_AVAIL <<< $(free -b | awk '/Mem:/{print $2,$3,$7}')
read -r SWAP_TOTAL SWAP_USED <<< $(free -b | awk '/Swap:/{print $2,$3}')

# Collect CPU metrics
CPU_USAGE=$(top -bn1 | awk '/^%Cpu/{printf "%.1f", 100-$8}')
read -r LOAD1 LOAD5 LOAD15 <<< $(cat /proc/loadavg | awk '{print $1,$2,$3}')
CORES=$(nproc)

# Collect service status
services=("nginx" "postgres" "redis")
SVC_JSON="["
for svc in "${services[@]}"; do
  running=$(systemctl is-active "$svc" 2>/dev/null)
  [ "$running" = "active" ] && r="true" || r="false"
  SVC_JSON+="\"{\\\"name\\\":\\\"$svc\\\",\\\"running\\\":$r}\","
done
SVC_JSON="${SVC_JSON%,}]"

# Collect process counts (catches zombie accumulation)
PROC_TOTAL=$(ps -e --no-headers | wc -l)
PROC_ZOMBIE=$(ps -eo stat --no-headers | awk '/^Z/' | wc -l)

curl -s -X POST "$BASE_URL/api/servers/report" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $API_KEY" \
  -d "{
    \"hostname\": \"$HOSTNAME\",
    \"disk\": $DISK_JSON,
    \"memory\": {
      \"total\": $MEM_TOTAL, \"used\": $MEM_USED, \"available\": $MEM_AVAIL,
      \"swap_total\": ${SWAP_TOTAL:-0}, \"swap_used\": ${SWAP_USED:-0}
    },
    \"cpu\": {
      \"usage_percent\": $CPU_USAGE,
      \"load_1m\": $LOAD1, \"load_5m\": $LOAD5, \"load_15m\": $LOAD15,
      \"cores\": $CORES
    },
    \"services\": $SVC_JSON,
    \"processes\": { \"total\": $PROC_TOTAL, \"zombie\": $PROC_ZOMBIE }
  }"

Python — Send Events

watchdog.py

import requests

API_KEY = "dw_your_api_key"
BASE_URL = "https://your-domain.com"

def send_event(channel, name, description=None, severity="info", notify=False, metadata=None):
    """Send an event to Devil Watchdog."""
    payload = {
        "channel": channel,
        "name": name,
        "severity": severity,
        "notify": notify,
    }
    if description:
        payload["description"] = description
    if metadata:
        payload["metadata"] = metadata

    response = requests.post(
        f"{BASE_URL}/api/events",
        json=payload,
        headers={"X-API-Key": API_KEY},
    )
    response.raise_for_status()
    return response.json()

# Examples
send_event("deploy", "deploy.started", "Deploying v2.4.1")

send_event(
    "payments",
    "payment.failed",
    description="Stripe charge failed for order #1234",
    severity="error",
    notify=True,
    metadata={"order_id": "1234", "amount": 99.99, "error": "card_declined"},
)

Node.js — Send Events

watchdog.ts

const API_KEY = "dw_your_api_key";
const BASE_URL = "https://your-domain.com";

async function sendEvent(event: {
  channel: string;
  name: string;
  description?: string;
  severity?: "info" | "warning" | "error" | "critical";
  notify?: boolean;
  tags?: string[];
  metadata?: Record<string, unknown>;
}) {
  const res = await fetch(`${BASE_URL}/api/events`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "X-API-Key": API_KEY,
    },
    body: JSON.stringify({ severity: "info", ...event }),
  });

  if (!res.ok) throw new Error(`Watchdog error: ${res.status}`);
  return res.json();
}

// Example: notify on critical error
await sendEvent({
  channel: "auth",
  name: "auth.bruteforce",
  description: "50 failed login attempts from 192.168.1.100",
  severity: "critical",
  notify: true,
  metadata: { ip: "192.168.1.100", attempts: 50 },
});

Cron Job — Heartbeat Ping

crontab

# Ping every 5 minutes to confirm the backup cron is alive
*/5 * * * * curl -fsS https://your-domain.com/api/ping/your-slug > /dev/null

# Ping after a specific job completes
0 2 * * * /usr/local/bin/nightly-backup.sh && curl -fsS https://your-domain.com/api/ping/backup-slug

Devil Watchdog API Documentation