Maksim Totmin 727fd1195a feat: add route_fallbacks metric to Prometheus and Zabbix monitoring
- Prometheus: new pulse_route_fallbacks_total counter
- Zabbix: extend response with route_requests_total, route_fallbacks_total, uptime_seconds
- Docs: update Prometheus metrics table and Zabbix response example
2026-06-25 23:27:12 +07:00

9.5 KiB
Raw Permalink Blame History

API Reference

Authentication

Method Path Auth Description
POST /api/auth/login Login: {"username":"admin","password":"admin"}access_token + refresh_token
POST /api/auth/refresh Refresh access token using refresh_token

All endpoints except /api/route, /api/health/*, /api/monitoring/* require Authorization: Bearer <jwt>.

Roles: admin (full access), viewer (read-only nodes/metrics).

Core

Method Path Auth Description
GET /api/route Route a call. Query: caller_id, dest, ingress_trunk
PUT /api/route/mode admin Set balancing mode: {"mode":"best|weighted_random"} — см. Routing
GET /api/health Full health status with connections and stats
GET /api/health/live Liveness probe — always 200 {"status":"alive"}
GET /api/health/ready Readiness — 200 {"status":"ready"} или 503 {"reason":"nats_disconnected"}

Health Response

{
  "status": "ok",
  "version": "1.0.0",
  "connections": {
    "nats": "connected|disconnected",
    "esl": {
      "status": "disabled|connected|disconnected",
      "host": "127.0.0.1",
      "uptime_seconds": 3600,
      "reconnects": 0,
      "gateway_ops": 42,
      "events_recv": 15000
    }
  },
  "stats": {
    "total_nodes": 3,
    "healthy_nodes": 2,
    "route_requests": 5000,
    "route_fallbacks": 10,
    "uptime_seconds": 7200
  }
}

Route Response (success)

{ "node_id": "pbx-03", "score": 88, "sip_gateway": "sip:pbx03.lan:5060", "matched_rule": "none" }

matched_rule — имя сработавшего routing rule, либо "none" если применился глобальный scoring.

Route Mode Response

PUT /api/route/mode возвращает текущий режим:

{ "mode": "best" }

Route Response (fallback — all nodes unhealthy)

{
  "fallback": true,
  "sip_gateway": "sip:operator.lan:5060",
  "reason": "all_nodes_unhealthy",
  "matched_rule": "my-pool-rule",
  "nodes": [
    { "id": "pbx-01", "score": -100, "reason": "disabled", "status": "disabled" },
    { "id": "pbx-02", "score": -1, "reason": "stale", "status": "stale" },
    { "id": "pbx-03", "score": 70, "reason": "", "status": "healthy" }
  ]
}

Route Response (no nodes registered)

{ "error": "no_nodes_registered", "matched_rule": "none" }

Nodes

Method Path Role Description
GET /api/nodes admin/viewer List all nodes with scores and state
GET /api/nodes/{id}/metrics admin/viewer Metrics history (ring buffer, ~30 min)
PUT /api/nodes/{id}/toggle admin {"disabled":true, "reason":"..."} — returns updated NodeInfo

NodeInfo (GET /api/nodes)

{
  "node_id": "pbx-01",
  "ts": 1719000000,
  "status": "ok",
  "active_calls": 42,
  "max_calls": 250,
  "idle_cpu": 35.0,
  "load_avg": 0.85,
  "call_failure_rate": 0.5,
  "disabled": false,
  "disabled_reason": "",
  "score": 88.0,
  "lethal_reason": "",
  "is_stale": false,
  "seconds_ago": 3.0
}

Node Metrics (GET /api/nodes/{id}/metrics)

История метрик ноды из ring buffer (~30 мин, шаг 5 с):

{
  "node_id": "pbx-01",
  "snapshots": [
    { "active_calls": 42, "load_avg": 0.85, "score": 88.0, ... }
  ],
  "count": 360
}

Trunks

Method Path Role Description
GET /api/trunks admin All trunks. Query: type=ingress|balance|fallback
POST /api/trunks admin Create trunk
PUT /api/trunks/{id} admin Update trunk
DELETE /api/trunks/{id} admin Delete trunk

Trunk model:

{
  "id": "trk-001",
  "name": "pbx-03 balance",
  "type": "ingress|balance|fallback",
  "node_id": "pbx-03",
  "gateway": "sip:pbx03.lan:5060",
  "codecs": ["PCMU", "PCMA"],
  "context": "default",
  "enabled": true,
  "description": "",
  "created_at": "...",
  "updated_at": "..."
}

Trunk types:

Type Purpose Quantity
ingress Incoming trunk (where calls come from) 1..N
balance Destination trunk (bound to a Node) 1..N
fallback Operator fallback (when all balance score < 0) exactly 1

Routing Rules

Method Path Role Description
GET /api/routing/rules admin/viewer Список правил с счётчиками hits
POST /api/routing/rules admin Создать правило
PUT /api/routing/rules/{id} admin Обновить правило
DELETE /api/routing/rules/{id} admin Удалить правило
PUT /api/routing/rules/{id}/toggle admin Вкл/выкл: {"enabled":true}
PUT /api/routing/rules/reorder admin Новый порядок: ["rr-abc", "rr-def", ...]

Поля правила — см. docs/routing.md. Ответы содержат полную модель правила.

Users (admin only)

Method Path Description
GET /api/users List users (без password_hash)
POST /api/users {"username":"...","password":"...","role":"admin|viewer"}
PUT /api/users/{id} Partial update: {"username":"...","role":"viewer"}
DELETE /api/users/{id} Удалить (204 No Content)

Защиты:

  • user-01 нельзя удалить или понизить до viewer
  • Нельзя удалить последнего администратора
  • GET-ответ исключает поле password_hash

Monitoring (API-key auth)

Use X-API-Key header (not JWT).

Method Path Description
GET /api/monitoring/zabbix JSON для Zabbix LLD
GET /api/monitoring/prometheus text/plain метрики для Prometheus

Zabbix Response

{
  "data": [
    {
      "{#NODE_ID}": "pbx-01",
      "{#STATUS}": "ok",
      "{#ACTIVE_CALLS}": 42,
      "{#MAX_CALLS}": 250,
      "{#IDLE_CPU}": 35.0,
      "{#LOAD_AVG}": 0.85,
      "{#CALL_FAILURE_RATE}": 0.5,
      "{#SCORE}": 88.0,
      "{#DISABLED}": false,
      "{#IS_STALE}": false,
      "{#SECONDS_AGO}": 3.0
    }
  ],
  "route_requests_total": 5000,
  "route_fallbacks_total": 42,
  "uptime_seconds": 7200
}

Prometheus Metrics

Метрика Type Labels
pulse_route_requests_total counter
pulse_route_fallbacks_total counter
pulse_nodes_total gauge
pulse_nodes_healthy gauge
pulse_uptime_seconds gauge
pulse_node_active_calls gauge node
pulse_node_max_calls gauge node
pulse_node_idle_cpu gauge node
pulse_node_load_avg gauge node
pulse_node_call_failure_rate gauge node
pulse_node_score gauge node
pulse_node_seconds_ago gauge node
pulse_node_disabled gauge node (0/1)
pulse_node_stale gauge node (0/1)
pulse_node_status_ok gauge node (0/1)

WebSocket

Method Path Description
WS /ws/metrics Real-time metrics (JWT в query ?token=)

Типы сообщений (поле type):

Тип Payload Описание
nodes_update []NodeInfo Обновление метрик всех нод (каждые 5 с)
balancer_update BalancerInfo Состояние балансировщика (uptime, mode, ESL, NATS)
gateway_update {"gateway","trunk_id","status"} Статус FS-gateway (up/down)

Examples

# Route (no auth)
curl 'http://localhost:8080/api/route?caller_id=74951234567&dest=123&ingress_trunk=trk-001'
# → {"node_id":"pbx-03","score":88,"sip_gateway":"sip:pbx03.lan:5060"}

# Login
TOKEN=$(curl -s -X POST http://localhost:8080/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"admin"}' | jq -r '.access_token')

# Nodes with JWT
curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/nodes

# Toggle node (admin only)
curl -X PUT http://localhost:8080/api/nodes/pbx-01/toggle \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"disabled":true,"reason":"maintenance"}'

# Route mode (admin)
curl -X PUT http://localhost:8080/api/route/mode \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"mode":"weighted_random"}'
# → {"mode":"weighted_random"}

# Routing rules (admin)
curl -X POST http://localhost:8080/api/routing/rules \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name":"VIP pool",
    "priority":10,
    "match_caller_id":"^7495",
    "action_type":"pool",
    "action_pool_ids":["pbx-01","pbx-02"]
  }'

# Toggle rule (admin)
curl -X PUT http://localhost:8080/api/routing/rules/rr-abc123/toggle \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"enabled":false}'

# Reorder rules (admin)
curl -X PUT http://localhost:8080/api/routing/rules/reorder \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '["rr-def456", "rr-abc123"]'

# Prometheus metrics (API-key)
curl -H "X-API-Key: $(jq -r '.monitoring_api_key' data/config.json)" \
  http://localhost:8080/api/monitoring/prometheus