docs: restructure documentation into docs/ directory
- Create docs/ with 7 focused files: architecture, api, scoring, agent, deployment, freeswitch, development - Rewrite README.md as concise overview with links to docs/ - Remove AGENTS.md (content merged into docs/ and README)
This commit is contained in:
parent
110289a093
commit
f20d5aa0e2
282
AGENTS.md
282
AGENTS.md
@ -1,282 +0,0 @@
|
||||
# pulse-lets-go — Telephone Load Balancer
|
||||
|
||||
Балансировщик телефонной нагрузки на базе FreeSWITCH + NATS.
|
||||
Принимает метрики от нижестоящих АТС, вычисляет score готовности,
|
||||
возвращает лучшую ноду для следующего вызова через `/api/route`.
|
||||
|
||||
## Стек
|
||||
|
||||
| Компонент | Технология |
|
||||
|-----------|-----------|
|
||||
| Backend | Go (stdlib net/http или Chi, nats.go, golang-jwt, bcrypt) |
|
||||
| Frontend | SvelteKit + Tailwind + shadcn-svelte |
|
||||
| Auth | JWT access/refresh, bcrypt, роли: admin / viewer |
|
||||
| Шина | NATS — `pulse.metrics.<node_id>` |
|
||||
| Хранилище | JSON (`data/config.json`, `trunks.json`, `users.json`) + ASCII-лог метрик |
|
||||
| Деплой | systemd + единый бинарник, без Docker |
|
||||
|
||||
## Модели данных
|
||||
|
||||
### NATS message (node → balancer, каждые 5 сек)
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "pbx-01",
|
||||
"ts": 1719000000,
|
||||
"status": "ok",
|
||||
"active_calls": 42,
|
||||
"max_calls": 250,
|
||||
"idle_cpu": 35,
|
||||
"load_avg": 0.85,
|
||||
"call_failure_rate": 0.5,
|
||||
"sip_gateway": "sip:pbx-01.lan:5060"
|
||||
}
|
||||
```
|
||||
|
||||
`call_failure_rate`: 0.0..100.0 (проценты).
|
||||
|
||||
### In-memory node state (engine)
|
||||
|
||||
Все поля метрики + дополнительные:
|
||||
- `disabled` (bool) — ручное исключение админом
|
||||
- `disabled_reason` (string)
|
||||
|
||||
## Scoring Engine
|
||||
|
||||
### Lethal (score → -100, нода исключается)
|
||||
|
||||
```
|
||||
status != "ok"
|
||||
stale > stale_threshold_sec (по умолчанию 20 сек)
|
||||
active_calls >= max_calls
|
||||
idle_cpu < idle_cpu_min (по умолчанию 5)
|
||||
disabled == true
|
||||
call_failure_rate > call_failure_rate_lethal (по умолчанию 15.0)
|
||||
```
|
||||
|
||||
### Weighted score
|
||||
|
||||
```
|
||||
call_score = min(100, max(0, 100 - (active_calls / max_calls * 100)))
|
||||
load_score = min(100, max(0, 100 - (load_avg * 50)))
|
||||
idle_score = min(100, max(0, idle_cpu))
|
||||
fail_score = min(100, max(0, 100 - call_failure_rate))
|
||||
|
||||
score = call_score * 0.40 + load_score * 0.30 + idle_score * 0.20 + fail_score * 0.10
|
||||
```
|
||||
|
||||
Веса и пороги конфигурируются в `config.json`.
|
||||
|
||||
### Оптимизация
|
||||
|
||||
`best_node_id` + `best_score` + `fallback_active` кэшируются, обновляются при каждой метрике.
|
||||
`/api/route` читает их за RLock, O(1).
|
||||
|
||||
## HTTP API
|
||||
|
||||
### Auth
|
||||
- `POST /api/auth/login` → JWT
|
||||
- Все эндпоинты (кроме login и `/api/monitoring/*`) требуют `Authorization: Bearer <jwt>`
|
||||
- `/api/monitoring/*` — API-key в заголовке `X-API-Key`
|
||||
|
||||
### Core
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/route?caller_id=X&dest=Y&ingress_trunk=Z` | – | Выбор ноды для звонка (без JWT) |
|
||||
|
||||
Response (успех):
|
||||
```json
|
||||
{ "node_id": "pbx-03", "score": 88, "sip_gateway": "sip:pbx03.lan:5060" }
|
||||
```
|
||||
|
||||
Response (fallback — все ноды unhealthy):
|
||||
```json
|
||||
{
|
||||
"fallback": true,
|
||||
"sip_gateway": "sip:operator.lan:5060",
|
||||
"reason": "all_nodes_unhealthy",
|
||||
"nodes": [...]
|
||||
}
|
||||
```
|
||||
|
||||
Response (нет нод вообще):
|
||||
```json
|
||||
{ "error": "no_nodes_registered" }
|
||||
```
|
||||
|
||||
### Nodes
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/nodes` | admin/viewer | Список нод со score |
|
||||
| `GET` | `/api/nodes/:id/metrics` | admin/viewer | История метрик (ring buffer) |
|
||||
| `PUT` | `/api/nodes/:id/toggle` | admin | `{"disabled": true/false, "reason": "..."}` |
|
||||
|
||||
### Trunks
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/trunks` | admin | Все транки |
|
||||
| `GET` | `/api/trunks?type=ingress\|balance\|fallback` | admin | Фильтр по типу |
|
||||
| `POST` | `/api/trunks` | admin | Создать |
|
||||
| `PUT` | `/api/trunks/:id` | admin | Обновить |
|
||||
| `DELETE` | `/api/trunks/:id` | admin | Удалить |
|
||||
|
||||
### Users (admin only)
|
||||
|
||||
| Метод | Путь | Описание |
|
||||
|-------|------|----------|
|
||||
| `GET` | `/api/users` | Список |
|
||||
| `POST` | `/api/users` | Создать |
|
||||
| `PUT` | `/api/users/:id` | Обновить |
|
||||
| `DELETE` | `/api/users/:id` | Удалить |
|
||||
|
||||
### Monitoring (API-key, не JWT)
|
||||
|
||||
| Метод | Путь | Описание |
|
||||
|-------|------|----------|
|
||||
| `GET` | `/api/monitoring/zabbix` | JSON для Zabbix LLD + items |
|
||||
| `GET` | `/api/monitoring/prometheus` | text/plain метрики |
|
||||
|
||||
### WebSocket
|
||||
|
||||
| Метод | Путь | Описание |
|
||||
|-------|------|----------|
|
||||
| `WS` | `/ws/metrics` | Real-time метрики на фронт (JWT в query param) |
|
||||
|
||||
## Терминология транков
|
||||
|
||||
| Тип | Назначение | Сколько |
|
||||
|-----|-----------|---------|
|
||||
| `ingress` | Входящий транк (откуда звонки приходят) | 1..N |
|
||||
| `balance` | Транк назначения (куда балансятся, привязан к Node) | 1..N |
|
||||
| `fallback` | Резервный транк оператора (когда все balance score < 0) | ровно 1 |
|
||||
|
||||
Модель транка:
|
||||
```json
|
||||
{
|
||||
"id": "trk-001",
|
||||
"name": "pbx-03 balance",
|
||||
"type": "ingress|balance|fallback",
|
||||
"node_id": "pbx-03",
|
||||
"gateway": "sip:pbx03.lan:5060",
|
||||
"codecs": ["PCMU", "PCMA"],
|
||||
"context": "default",
|
||||
"enabled": true,
|
||||
"description": "",
|
||||
"created_at": "...",
|
||||
"updated_at": "..."
|
||||
}
|
||||
```
|
||||
|
||||
## FreeSWITCH интеграция
|
||||
|
||||
FS использует Lua скрипт в dialplan → `mod_curl` дёргает `/api/route` → получает JSON → парсит `sip_gateway` → bridge.
|
||||
|
||||
```
|
||||
<extension name="pulse_route">
|
||||
<condition field="destination_number" expression="^.*$">
|
||||
<action application="lua" data="route.lua"/>
|
||||
</condition>
|
||||
</extension>
|
||||
```
|
||||
|
||||
route.lua: GET `/api/route?...` → если `fallback: true` идёт на fallback gateway, иначе на `sip_gateway`.
|
||||
|
||||
## config.json
|
||||
|
||||
```json
|
||||
{
|
||||
"nats_url": "nats://localhost:4222",
|
||||
"nats_user": "",
|
||||
"nats_password": "",
|
||||
"listen_addr": ":8080",
|
||||
"jwt_secret": "auto-generated-on-first-run",
|
||||
"monitoring_api_key": "auto-generated-on-first-run",
|
||||
"stale_threshold_sec": 20,
|
||||
"scoring": {
|
||||
"idle_cpu_min": 5,
|
||||
"call_failure_rate_lethal": 15.0,
|
||||
"weights": {
|
||||
"call_score": 0.40,
|
||||
"load_score": 0.30,
|
||||
"idle_score": 0.20,
|
||||
"fail_score": 0.10
|
||||
}
|
||||
},
|
||||
"tls": {
|
||||
"enabled": false,
|
||||
"cert_file": "",
|
||||
"key_file": ""
|
||||
},
|
||||
"rate_limit": {
|
||||
"enabled": true,
|
||||
"route_per_sec": 5000,
|
||||
"api_per_sec": 100
|
||||
},
|
||||
"log": {
|
||||
"level": "info",
|
||||
"format": "text"
|
||||
},
|
||||
"esl": {
|
||||
"host": "",
|
||||
"port": 8021,
|
||||
"password": "",
|
||||
"password_env": "",
|
||||
"sofia_profile": "external",
|
||||
"gateway_prefix": "pulse"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Структура проекта
|
||||
|
||||
```
|
||||
pulse-lets-go/
|
||||
├── cmd/pulse-lets-go/main.go
|
||||
├── cmd/pulse-lets-go-agent/
|
||||
├── cmd/emulator/
|
||||
├── cmd/siptest/
|
||||
├── internal/
|
||||
│ ├── api/ — handlers: auth, nodes, trunks, users, monitoring, ws
|
||||
│ ├── engine/ — scorer + router (sync.RWMutex, best_node кэш)
|
||||
│ ├── nats/ — subscriber, in-memory node store
|
||||
│ ├── esl/ — FreeSWITCH ESL client (raw TCP)
|
||||
│ ├── ami/ — Asterisk AMI client (raw TCP)
|
||||
│ ├── config/ — manager: чтение/атомарная запись JSON
|
||||
│ ├── models/ — все общие типы
|
||||
│ └── log/ — ASCII-metrics логгер + ротация
|
||||
├── contrib/ — route.lua, agent конфиги
|
||||
├── data/ — JSON конфиги (gitignored)
|
||||
├── web/ — SvelteKit + shadcn-svelte + Tailwind
|
||||
│ ├── src/routes/
|
||||
│ │ ├── +layout.svelte
|
||||
│ │ ├── +page.svelte # дашборд
|
||||
│ │ ├── login/+page.svelte
|
||||
│ │ ├── trunks/+page.svelte
|
||||
│ │ ├── nodes/+page.svelte
|
||||
│ │ └── users/+page.svelte
|
||||
│ ├── src/lib/
|
||||
│ │ ├── components/
|
||||
│ │ └── api.ts
|
||||
│ ├── svelte.config.js
|
||||
│ ├── tailwind.config.js
|
||||
│ └── package.json
|
||||
├── go.mod
|
||||
├── Makefile
|
||||
└── AGENTS.md
|
||||
```
|
||||
|
||||
## Порядок реализации
|
||||
|
||||
1. `cmd/pulse-lets-go/main.go` + Go module + пустая структура internal
|
||||
2. `internal/models/` — все типы данных
|
||||
3. `internal/config/` — чтение/запись JSON, атомарное сохранение
|
||||
4. `internal/nats/` — подписка, in-memory store
|
||||
5. `internal/engine/` — scorer + router
|
||||
6. `internal/api/` — все хендлеры
|
||||
7. `internal/log/` — ASCII-логгер + ring buffer
|
||||
8. Web-фронт (SvelteKit)
|
||||
9. Makefile + systemd unit
|
||||
10. Тесты + graceful shutdown
|
||||
777
README.md
777
README.md
@ -13,86 +13,28 @@
|
||||
|
||||
## Features
|
||||
|
||||
- **pulse-lets-go-agent** — агент сбора метрик для FreeSWITCH (ESL) и Asterisk (AMI). Работает на каждой PBX-ноде.
|
||||
- **Scoring engine** — lethal checks (disabled, stale, overload, CPU, failure rate) + weighted score (call, load, idle, fail) с конфигурируемыми весами.
|
||||
- **NATS шина** — асинхронный сбор метрик через `pulse.metrics.<node_id>`.
|
||||
- **Auto-создание транков** — при получении первой метрики от новой ноды с `sip_gateway` создаётся balance-транк автоматически.
|
||||
- **Gateway health WS** — статус FS-gateway (registered/unregistered) транслируется на фронт через WebSocket.
|
||||
- **REST API** — JWT (access + refresh), роли admin/viewer, rate limiting, CORS, health endpoints.
|
||||
- **WebSocket** — real-time трансляция метрик на SvelteKit дашборд.
|
||||
- **FreeSWITCH ESL** — управление Sofia-шлюзами через raw TCP ESL.
|
||||
- **Monitoring** — Prometheus и Zabbix эндпоинты с отдельным API-key.
|
||||
- **SvelteKit UI** — дашборд, таблицы нод, транков, пользователей.
|
||||
- **Защита пользователей** — первичный администратор (user-01) не может быть удалён или понижен до viewer.
|
||||
- **Graceful shutdown** — корректное завершение с таймаутом 10 с.
|
||||
- **systemd ready** — unit с изоляцией (NoNewPrivileges, ProtectSystem, PrivateTmp).
|
||||
|
||||
---
|
||||
|
||||
## Архитектура
|
||||
|
||||
```
|
||||
┌─────────────────────┐ ESL/AMI ┌──────────────────────┐
|
||||
│ PBX-01 │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ collect │ (on each PBX node) │
|
||||
├─────────────────────┤ ├──────────────────────┤
|
||||
│ PBX-02 │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ ├──────────────────────┤
|
||||
├─────────────────────┤ ├──────────────────────┤
|
||||
│ PBX-N │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ └─────────┬────────────┘
|
||||
└─────────────────────┘ │ pulse.metrics.<id>
|
||||
│ every 5s
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ NATS Server │
|
||||
│ nats://:4222 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ pulse-lets-go │
|
||||
│ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ NATS Sub │ │
|
||||
│ │ (pulse.metrics)│ │
|
||||
│ └────────┬───────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ Engine │ │
|
||||
│ │ (scoring) │──│──▶ /api/route
|
||||
│ └────────┬───────┘ │ ↓
|
||||
│ ▼ │ FreeSWITCH bridge
|
||||
│ ┌────────────────┐ │ sip:pbx-03:5060
|
||||
│ │ REST API │──│──▶ /api/nodes
|
||||
│ │ (JWT auth) │ │ /api/trunks
|
||||
│ └────────┬───────┘ │ /api/users
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ WebSocket │──│──▶ SvelteKit UI
|
||||
│ └────────────────┘ │ (ws://:8080/ws/metrics)
|
||||
│ ┌────────────────┐ │
|
||||
│ │ ESL Client │──│──▶ FreeSWITCH ESL
|
||||
│ │ (gateways) │ │ :8021
|
||||
│ └────────────────┘ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ Monitoring │──│──▶ /api/monitoring/prometheus
|
||||
│ │ (API-key) │ │ /api/monitoring/zabbix
|
||||
│ └────────────────┘ │
|
||||
└──────────────────────┘
|
||||
```
|
||||
- **pulse-lets-go-agent** — агент сбора метрик для FreeSWITCH (ESL) и Asterisk (AMI)
|
||||
- **Scoring engine** — lethal checks + weighted score (call, load, idle, fail) с конфигурируемыми весами
|
||||
- **NATS шина** — асинхронный сбор метрик через `pulse.metrics.<node_id>`
|
||||
- **REST API** — JWT (access + refresh), роли admin/viewer, rate limiting, CORS
|
||||
- **WebSocket** — real-time трансляция метрик на SvelteKit дашборд
|
||||
- **FreeSWITCH ESL** — управление Sofia-шлюзами через raw TCP ESL
|
||||
- **Monitoring** — Prometheus и Zabbix эндпоинты с отдельным API-key
|
||||
- **SvelteKit UI** — дашборд, таблицы нод, транков, пользователей
|
||||
- **Graceful shutdown** с таймаутом 10 с
|
||||
- **systemd ready** с изоляцией (NoNewPrivileges, ProtectSystem, PrivateTmp)
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Требования
|
||||
### Requirements
|
||||
|
||||
- Go 1.26+
|
||||
- Node.js 22+
|
||||
- NATS Server (автоматически качается `make nats`)
|
||||
- NATS Server (`make nats` скачивает автоматически)
|
||||
|
||||
### 2. Сборка
|
||||
### Build & Run
|
||||
|
||||
```bash
|
||||
git clone git@git.totmin.ru:en2zmax/pulse-lets-go.git
|
||||
@ -100,23 +42,21 @@ cd pulse-lets-go
|
||||
make build
|
||||
```
|
||||
|
||||
### 3. Запуск
|
||||
|
||||
```bash
|
||||
# Терминал 1: NATS
|
||||
# Terminal 1: NATS
|
||||
make nats
|
||||
|
||||
# Терминал 2: балансировщик
|
||||
# Terminal 2: balancer
|
||||
./bin/pulse-lets-go
|
||||
|
||||
# Терминал 3: эмулятор метрик (3 здоровые ноды)
|
||||
# Terminal 3: emulator (3 healthy nodes)
|
||||
make emulate-normal
|
||||
```
|
||||
|
||||
### 4. Проверка
|
||||
### Check
|
||||
|
||||
```bash
|
||||
# Route (без авторизации)
|
||||
# Route (no auth)
|
||||
curl 'http://localhost:8080/api/route?caller_id=123&dest=456&ingress_trunk=trk-001'
|
||||
|
||||
# Login
|
||||
@ -127,9 +67,25 @@ curl -X POST http://localhost:8080/api/auth/login \
|
||||
|
||||
---
|
||||
|
||||
## Конфигурация
|
||||
## Documentation
|
||||
|
||||
При первом запуске `config.json` создаётся автоматически в директории `data/` (или `$PULSE_DATA_DIR`). JWT-секрет и API-key генерируются случайно.
|
||||
| Document | Content |
|
||||
|----------|---------|
|
||||
| [docs/architecture.md](docs/architecture.md) | Архитектура, диаграмма потоков данных, стек, структура проекта |
|
||||
| [docs/api.md](docs/api.md) | Полная справка REST API + WebSocket: auth, route, nodes, trunks, users, monitoring, примеры |
|
||||
| [docs/scoring.md](docs/scoring.md) | Scoring engine: lethal-условия, weighted score, пример расчёта, кэширование |
|
||||
| [docs/agent.md](docs/agent.md) | pulse-lets-go-agent: collectors, конфигурация, деплой, troubleshooting |
|
||||
| [docs/deployment.md](docs/deployment.md) | Production деплой (8 шагов), systemd unit, security, rollback |
|
||||
| [docs/freeswitch.md](docs/freeswitch.md) | Интеграция с FreeSWITCH: route.lua, dialplan, ESL, предварительные проверки |
|
||||
| [docs/development.md](docs/development.md) | Команды Makefile, code style, добавление эндпоинта, PR process |
|
||||
| [CHANGELOG.md](CHANGELOG.md) | История изменений (Keep a Changelog) |
|
||||
| [CONTRIBUTING.md](CONTRIBUTING.md) | Code style, conventional commits, процесс PR |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
При первом запуске `config.json` создаётся автоматически в `data/` (или `$PULSE_DATA_DIR`). JWT-секрет и API-key генерируются случайно.
|
||||
|
||||
### config.json
|
||||
|
||||
@ -139,8 +95,8 @@ curl -X POST http://localhost:8080/api/auth/login \
|
||||
"nats_user": "",
|
||||
"nats_password": "",
|
||||
"listen_addr": ":8080",
|
||||
"jwt_secret": "aabbccdd...32-байта-hex",
|
||||
"monitoring_api_key": "11223344...32-байта-hex",
|
||||
"jwt_secret": "auto-generated-32-byte-hex",
|
||||
"monitoring_api_key": "auto-generated-32-byte-hex",
|
||||
"stale_threshold_sec": 20,
|
||||
"scoring": {
|
||||
"idle_cpu_min": 5,
|
||||
@ -177,647 +133,22 @@ curl -X POST http://localhost:8080/api/auth/login \
|
||||
}
|
||||
```
|
||||
|
||||
| Поле | Дефолт | Описание |
|
||||
|------|--------|----------|
|
||||
| `nats_url` | `nats://localhost:4222` | URL NATS сервера |
|
||||
| `nats_user` | `""` | Имя пользователя NATS (если нужна аутентификация) |
|
||||
| `nats_password` | `""` | Пароль NATS |
|
||||
| `listen_addr` | `:8080` | Адрес HTTP-сервера |
|
||||
| `jwt_secret` | auto | Секрет для подписи JWT (32 байта hex) |
|
||||
| `monitoring_api_key` | auto | API-ключ для `/api/monitoring/*` |
|
||||
| `stale_threshold_sec` | `20` | Через сколько секунд без метрик нода считается stale |
|
||||
| `scoring.idle_cpu_min` | `5` | Минимальный idle CPU. Ниже — lethal |
|
||||
| `scoring.call_failure_rate_lethal` | `15.0` | Максимальный failure rate. Выше — lethal |
|
||||
| `scoring.weights.*` | 0.40/0.30/0.20/0.10 | Веса компонентов скора (сумма = 1.0) |
|
||||
| `tls.enabled` | `false` | Включить HTTPS |
|
||||
| `rate_limit.enabled` | `true` | Включить rate limiting |
|
||||
| `rate_limit.route_per_sec` | `5000` | Лимит `/api/route` на IP |
|
||||
| `rate_limit.api_per_sec` | `100` | Лимит остальных `/api/*` на IP |
|
||||
| `log.level` | `info` | Уровень лога: `debug`, `info`, `warn`, `error` |
|
||||
| `log.format` | `text` | Формат: `text` или `json` |
|
||||
| `esl.host` | `""` | Хост FreeSWITCH ESL. Пусто — ESL отключён |
|
||||
| `esl.port` | `8021` | Порт ESL |
|
||||
| `esl.password` | `""` | Пароль ESL (ClueCon) |
|
||||
| `esl.password_env` | `""` | Или имя переменной окружения с паролем |
|
||||
### Key Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `nats_url` | `nats://localhost:4222` | NATS server URL |
|
||||
| `listen_addr` | `:8080` | HTTP server address |
|
||||
| `stale_threshold_sec` | `20` | Seconds without metrics before node is stale |
|
||||
| `scoring.idle_cpu_min` | `5` | Minimum idle CPU. Below — lethal |
|
||||
| `scoring.call_failure_rate_lethal` | `15.0` | Maximum failure rate. Above — lethal |
|
||||
| `rate_limit.route_per_sec` | `5000` | `/api/route` per-IP limit |
|
||||
| `rate_limit.api_per_sec` | `100` | `/api/*` per-IP limit |
|
||||
| `log.level` | `info` | `debug`, `info`, `warn`, `error` |
|
||||
| `esl.host` | `""` | FS ESL host. Empty — ESL disabled |
|
||||
|
||||
---
|
||||
## Прочее
|
||||
|
||||
## Scoring Engine
|
||||
|
||||
### Lethal-условия (score = -100)
|
||||
|
||||
Если хотя бы одно истинно — нода исключается из роутинга:
|
||||
|
||||
| Условие | Порог |
|
||||
|---------|-------|
|
||||
| `status != "ok"` | Любой статус кроме ok |
|
||||
| Stale | > `stale_threshold_sec` (20 с) |
|
||||
| `active_calls >= max_calls` | 100% заполнение |
|
||||
| `idle_cpu < idle_cpu_min` | < 5% |
|
||||
| `call_failure_rate > call_failure_rate_lethal` | > 15% |
|
||||
| `disabled == true` | Ручное отключение админом |
|
||||
|
||||
### Weighted score (0..100)
|
||||
|
||||
```
|
||||
call_score = clamp(100 - (active_calls / max_calls * 100), 0, 100)
|
||||
load_score = clamp(100 - (load_avg * 50), 0, 100)
|
||||
idle_score = clamp(idle_cpu, 0, 100)
|
||||
fail_score = clamp(100 - call_failure_rate, 0, 100)
|
||||
|
||||
score = call_score × 0.40 +
|
||||
load_score × 0.30 +
|
||||
idle_score × 0.20 +
|
||||
fail_score × 0.10
|
||||
```
|
||||
|
||||
### Пример
|
||||
|
||||
Нода `pbx-01`: `active_calls=42`, `max_calls=250`, `load_avg=0.85`, `idle_cpu=35`, `fail_rate=0.5`
|
||||
|
||||
```
|
||||
call_score = clamp(100 - (42/250)*100, 0, 100) = 83.2
|
||||
load_score = clamp(100 - 0.85*50, 0, 100) = 57.5
|
||||
idle_score = clamp(35, 0, 100) = 35.0
|
||||
fail_score = clamp(100 - 0.5, 0, 100) = 99.5
|
||||
|
||||
score = 83.2×0.40 + 57.5×0.30 + 35.0×0.20 + 99.5×0.10 = 70.1
|
||||
```
|
||||
|
||||
### Кэширование
|
||||
|
||||
`best_node_id` + `best_score` + `fallback_active` пересчитываются при каждой метрике.
|
||||
`/api/route` читает их за RLock — O(1).
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Auth
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `POST` | `/api/auth/login` | – | Вход: `{"username":"admin","password":"admin"}` → access_token + refresh_token |
|
||||
| `POST` | `/api/auth/refresh` | – | Обновление access-токена через refresh_token |
|
||||
|
||||
Все остальные эндпоинты (кроме `/api/route`, `/api/health/*`, `/api/monitoring/*`) требуют `Authorization: Bearer <jwt>`.
|
||||
|
||||
### Core
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/route` | – | Выбор ноды для звонка. Query: `caller_id`, `dest`, `ingress_trunk` |
|
||||
| `GET` | `/api/health` | – | Статистика: всего нод, здоровых, запросов, uptime |
|
||||
| `GET` | `/api/health/live` | – | Liveness probe (всегда 200) |
|
||||
| `GET` | `/api/health/ready` | – | Readiness probe (NATS + метрики) |
|
||||
|
||||
### Nodes
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/nodes` | admin/viewer | Список всех нод со score, состоянием |
|
||||
| `GET` | `/api/nodes/{id}/metrics` | admin/viewer | История метрик (ring buffer, ~30 мин) |
|
||||
| `PUT` | `/api/nodes/{id}/toggle` | admin | `{"disabled":true, "reason":"..."}` |
|
||||
|
||||
### Trunks
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/trunks` | admin | Все транки. Query: `type=ingress\|balance\|fallback` |
|
||||
| `POST` | `/api/trunks` | admin | Создать транк |
|
||||
| `PUT` | `/api/trunks/{id}` | admin | Обновить транк |
|
||||
| `DELETE` | `/api/trunks/{id}` | admin | Удалить транк |
|
||||
|
||||
### Users
|
||||
|
||||
| Метод | Путь | Роль | Описание |
|
||||
|-------|------|------|----------|
|
||||
| `GET` | `/api/users` | admin | Список пользователей |
|
||||
| `POST` | `/api/users` | admin | Создать пользователя |
|
||||
| `PUT` | `/api/users/{id}` | admin | Обновить пользователя |
|
||||
| `DELETE` | `/api/users/{id}` | admin | Удалить пользователя |
|
||||
|
||||
### Monitoring (API-key, не JWT)
|
||||
|
||||
| Метод | Путь | Описание |
|
||||
|-------|------|----------|
|
||||
| `GET` | `/api/monitoring/zabbix` | JSON для Zabbix LLD |
|
||||
| `GET` | `/api/monitoring/prometheus` | `text/plain` метрики для Prometheus |
|
||||
|
||||
### WebSocket
|
||||
|
||||
| Метод | Путь | Описание |
|
||||
|-------|------|----------|
|
||||
| `WS` | `/ws/metrics` | Real-time метрики на фронт (JWT в query `?token=`) |
|
||||
|
||||
### Примеры запросов
|
||||
|
||||
```bash
|
||||
# Route (no auth)
|
||||
curl 'http://localhost:8080/api/route?caller_id=74951234567&dest=123&ingress_trunk=trk-001'
|
||||
# → {"node_id":"pbx-03","score":88,"sip_gateway":"sip:pbx03.lan:5060"}
|
||||
|
||||
# Fallback (все ноды unhealthy)
|
||||
# → {"fallback":true,"sip_gateway":"sip:operator.lan:5060","reason":"all_nodes_unhealthy","nodes":[...]}
|
||||
|
||||
# Login
|
||||
TOKEN=$(curl -s -X POST http://localhost:8080/api/auth/login \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"username":"admin","password":"admin"}' | jq -r '.access_token')
|
||||
|
||||
# Nodes with JWT
|
||||
curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/nodes
|
||||
|
||||
# Toggle node (admin only)
|
||||
curl -X PUT http://localhost:8080/api/nodes/pbx-01/toggle \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"disabled":true,"reason":"maintenance"}'
|
||||
|
||||
# Prometheus metrics (API-key)
|
||||
curl -H "X-API-Key: $(jq -r '.monitoring_api_key' data/config.json)" \
|
||||
http://localhost:8080/api/monitoring/prometheus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FreeSWITCH Интеграция
|
||||
|
||||
### Dialplan (route.lua)
|
||||
|
||||
FreeSWITCH использует Lua-скрипт в dialplan. Скрипт дёргает `/api/route` через `mod_curl` и бриджится на полученный gateway.
|
||||
|
||||
```lua
|
||||
-- contrib/route.lua (сокращённо)
|
||||
api = freeswitch.API()
|
||||
caller_id = session:getVariable("caller_id_number") or ""
|
||||
dest = session:getVariable("destination_number") or ""
|
||||
url = "http://localhost:8080/api/route?caller_id=" .. caller_id .. "&dest=" .. dest
|
||||
|
||||
raw = api:execute("curl", url)
|
||||
body = raw:match("\r?\n\r?\n(.+)") or raw
|
||||
|
||||
local ok, route = pcall(cjson.decode, body)
|
||||
if not ok then session:hangup("NORMAL_TEMPORARY_FAILURE") return end
|
||||
|
||||
if route.fallback then
|
||||
session:bridge(route.sip_gateway) -- оператор
|
||||
elseif route.sip_gateway then
|
||||
session:bridge(route.sip_gateway) -- лучшая нода
|
||||
else
|
||||
session:hangup("NORMAL_TEMPORARY_FAILURE")
|
||||
end
|
||||
```
|
||||
|
||||
Конфигурация в dialplan:
|
||||
|
||||
```xml
|
||||
<extension name="route_call">
|
||||
<condition field="destination_number" expression="^.*$">
|
||||
<action application="set" data="balancer_url=http://balancer.lan:8080"/>
|
||||
<action application="set" data="ingress_trunk=trk-001"/>
|
||||
<action application="lua" data="route.lua"/>
|
||||
</condition>
|
||||
</extension>
|
||||
```
|
||||
|
||||
### ESL (автоматическое управление шлюзами)
|
||||
|
||||
При включении `esl.host` в конфиге балансировщик:
|
||||
- Подключается к FreeSWITCH ESL (`:8021`)
|
||||
- Синхронизирует ingress-транки как Sofia-шлюзы
|
||||
- Слушает события регистрации шлюзов
|
||||
- Передаёт статус gateway на WebSocket фронта
|
||||
|
||||
---
|
||||
|
||||
## Модели данных
|
||||
|
||||
### NATS-сообщение (PBX → Balancer, каждые 5 с)
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "pbx-01",
|
||||
"ts": 1719000000,
|
||||
"status": "ok",
|
||||
"active_calls": 42,
|
||||
"max_calls": 250,
|
||||
"idle_cpu": 35,
|
||||
"load_avg": 0.85,
|
||||
"call_failure_rate": 0.5
|
||||
}
|
||||
```
|
||||
|
||||
### Транки
|
||||
|
||||
| Тип | Назначение |
|
||||
|-----|-----------|
|
||||
| `ingress` | Входящий транк (откуда звонки приходят) |
|
||||
| `balance` | Транк назначения (привязан к Node) |
|
||||
| `fallback` | Резервный транк (когда все balance score < 0), ровно 1 |
|
||||
|
||||
---
|
||||
|
||||
## Деплой
|
||||
|
||||
### systemd
|
||||
|
||||
```bash
|
||||
make deploy # полный деплой в /opt/pulse-lets-go
|
||||
make install-systemd # устанавливает systemd unit
|
||||
```
|
||||
|
||||
```ini
|
||||
# deploy/pulse-lets-go.service
|
||||
[Unit]
|
||||
Description=Pulse Lets Go — Telephone Load Balancer
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=pulse
|
||||
Group=pulse
|
||||
ExecStart=/opt/pulse-lets-go/bin/pulse-lets-go
|
||||
WorkingDirectory=/opt/pulse-lets-go
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/pulse-lets-go/data /opt/pulse-lets-go/log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Структура директорий
|
||||
|
||||
```
|
||||
/opt/pulse-lets-go/
|
||||
├── bin/pulse-lets-go # Бинарник
|
||||
├── data/
|
||||
│ ├── config.json # Конфигурация
|
||||
│ ├── trunks.json # Транки
|
||||
│ └── users.json # Пользователи
|
||||
├── web/ # SvelteKit статика
|
||||
└── log/
|
||||
└── metrics.YYYY-MM-DD.log # ASCII-лог метрик
|
||||
```
|
||||
|
||||
### Security
|
||||
|
||||
- `NoNewPrivileges=true` — запрет повышения привилегий
|
||||
- `ProtectSystem=strict` — read-only системные директории
|
||||
- `ProtectHome=yes` — изоляция home
|
||||
- `PrivateTmp=true` — изолированный /tmp
|
||||
- JWT-секрет и API-key генерируются случайно при первом запуске
|
||||
|
||||
---
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Предусловия (перед деплоем pulse-lets-go)
|
||||
|
||||
| № | Проверка | Команда | Ожидание |
|
||||
|:-:|----------|---------|----------|
|
||||
| 1 | **ESL запущен** | `nc -z 127.0.0.1 8021 && echo OK` | `OK` |
|
||||
| 2 | **ESL пароль** | `echo "auth ClueCon" \| nc 127.0.0.1 8021` | `+OK accepted` |
|
||||
| 3 | **mod_curl загружен** | `fs_cli -x "show modules" \| grep mod_curl` | `<load module="mod_curl"/>` |
|
||||
| 4 | **mod_lua загружен** | `fs_cli -x "show modules" \| grep mod_lua` | `<load module="mod_lua"/>` |
|
||||
| 5 | **NATS установлен** | `which nats-server` | путь к бинарнику |
|
||||
| 6 | **Права на ESL** | ACL в `event_socket.conf.xml` разрешает 127.0.0.1 | `localhost` или `127.0.0.1` в ACL |
|
||||
|
||||
### Загрузка mod_curl без рестарта FreeSWITCH
|
||||
|
||||
Если `mod_curl` не загружен — звонки работать не будут. Загружается **без остановки FS**:
|
||||
|
||||
```bash
|
||||
fs_cli -x "load mod_curl"
|
||||
# Добавить в modules.conf.xml для персистентности:
|
||||
# <load module="mod_curl"/>
|
||||
```
|
||||
|
||||
### Порядок деплоя (8 шагов)
|
||||
|
||||
**Шаг 1 — Деплой на сервер**
|
||||
```bash
|
||||
make deploy
|
||||
scp -r /opt/pulse-lets-go devadmin@10.101.60.81:/opt/
|
||||
scp contrib/route.lua devadmin@10.101.60.81:/tmp/
|
||||
```
|
||||
|
||||
**Шаг 2 — NATS сервер** (если не установлен)
|
||||
```bash
|
||||
ssh devadmin@10.101.60.81
|
||||
sudo mkdir -p /opt/nats
|
||||
# Установить nats-server из репозитория или скопировать бинарник
|
||||
# Запуск: nats-server -p 4222 -D &
|
||||
```
|
||||
|
||||
**Шаг 3 — Деплой agent на PBX-ноды** (повторить для каждой UC/Asterisk ноды)
|
||||
```bash
|
||||
# Скопировать бинарник и конфиг на каждую PBX:
|
||||
scp bin/pulse-lets-go-agent devadmin@PBX_IP:/usr/local/bin/
|
||||
scp contrib/agent-uc.json devadmin@PBX_IP:/etc/pulse-lets-go-agent/agent.json
|
||||
|
||||
# Запустить:
|
||||
ssh devadmin@PBX_IP "systemctl start pulse-lets-go-agent"
|
||||
|
||||
# Проверить лог:
|
||||
# journalctl -u pulse-lets-go-agent -f
|
||||
# → [ami] подключён к 127.0.0.1:6154
|
||||
# → [agent] опубликована метрика: calls=0/150 load=0.28 cpu=99% status=ok
|
||||
```
|
||||
|
||||
**Шаг 4 — route.lua в FS scripts**
|
||||
```bash
|
||||
sudo mkdir -p /etc/freeswitch/scripts
|
||||
sudo cp /tmp/route.lua /etc/freeswitch/scripts/
|
||||
```
|
||||
|
||||
**Шаг 5 — Dialplan** (добавить `pulse_route` extension)
|
||||
```xml
|
||||
<!-- В /etc/freeswitch/dialplan/default.xml перед остальными extension -->
|
||||
<extension name="pulse_route">
|
||||
<condition field="destination_number" expression="^(.*)$">
|
||||
<action application="set" data="balancer_url=http://127.0.0.1:8080"/>
|
||||
<action application="lua" data="route.lua"/>
|
||||
</condition>
|
||||
</extension>
|
||||
```
|
||||
```bash
|
||||
sudo fs_cli -x "reloadxml"
|
||||
```
|
||||
|
||||
**Шаг 6 — Создание balance-транков для каждой PBX-ноды**
|
||||
```bash
|
||||
# Получить список gateway из FS (или список нод из конфига):
|
||||
echo "api sofia profile external gwlist" | nc 127.0.0.1 8021
|
||||
|
||||
# Создать balance-транк для каждой UC-ноды:
|
||||
curl -X POST .../api/trunks
|
||||
-d '{"name":"uc06","type":"balance","node_id":"uc06","gateway":"sip:10.101.60.115:5060","enabled":true}'
|
||||
```
|
||||
Транки также создаются автоматически при получении первой метрики от агента (если в метрике передан `sip_gateway`).
|
||||
|
||||
**Шаг 7 — Запуск pulse-lets-go**
|
||||
```bash
|
||||
# config.json создастся автоматически при первом запуске в /opt/pulse-lets-go/data/
|
||||
# Добавить esl блок в config.json:
|
||||
# "esl": {"host": "127.0.0.1", "port": 8021, "password": "ClueCon"}
|
||||
/opt/pulse-lets-go/bin/pulse-lets-go
|
||||
```
|
||||
|
||||
**Шаг 8 — Проверка**
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/api/health | jq '.connections'
|
||||
# → {"nats":"connected","esl":"connected"}
|
||||
curl 'http://127.0.0.1:8080/api/route?caller_id=123&dest=456'
|
||||
# → {"node_id":"pbx-03","score":88,"sip_gateway":...}
|
||||
```
|
||||
|
||||
### Rollback (если route.lua сломал звонки)
|
||||
|
||||
```bash
|
||||
# 1. Убрать pulse_route из dialplan
|
||||
sudo sed -i '/pulse_route/,/<\/extension>/d' /etc/freeswitch/dialplan/default.xml
|
||||
sudo fs_cli -x "reloadxml"
|
||||
|
||||
# 2. Остановить pulse-lets-go
|
||||
sudo systemctl stop pulse-lets-go
|
||||
|
||||
# Звонки продолжают идти по старому dialplan без балансировщика.
|
||||
```
|
||||
|
||||
### Права доступа (production)
|
||||
|
||||
| Операция | Требуемые права |
|
||||
|----------|----------------|
|
||||
| Чтение FS конфигов | root или freeswitch |
|
||||
| `fs_cli -x` команды | root или freeswitch |
|
||||
| ESL (порт 8021) | ACL в `event_socket.conf.xml` |
|
||||
| Установка ПО | root |
|
||||
| Запись в `/etc/freeswitch/scripts/` | root |
|
||||
| `reloadxml` | root или freeswitch |
|
||||
|
||||
### Версии (протестировано)
|
||||
|
||||
| Компонент | Версия |
|
||||
|-----------|--------|
|
||||
| FreeSWITCH | 1.10.12+ |
|
||||
| NATS Server | 2.10+ |
|
||||
| Go | 1.26 |
|
||||
| ОС | Linux (CentOS 7+, AlmaLinux, Arch) |
|
||||
|
||||
---
|
||||
|
||||
## Metrics Agent (pulse-lets-go-agent)
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────┐ pulse.metrics.<id> ┌─────────────────────┐
|
||||
│ pulse-lets-go-agent│──────────────────────────▶ │ NATS Server │
|
||||
│ (on each PBX node) │ every 5 seconds │ (central) │
|
||||
├─────────────────────┤ └────────┬────────────┘
|
||||
│ FreeSWITCH (ESL) │ │
|
||||
│ Asterisk (AMI) │ ▼
|
||||
│ System (/proc) │ ┌─────────────────┐
|
||||
└─────────────────────┘ │ pulse-lets-go │
|
||||
│ (scoring engine) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
The agent runs on each PBX node (FreeSWITCH or Asterisk), collects metrics, and publishes them to NATS at a regular interval (default 5 seconds). The pulse-lets-go balancer subscribes to all metrics and uses them for scoring and routing decisions.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
make build-agent
|
||||
# → bin/pulse-lets-go-agent (static binary, ~9 MB)
|
||||
```
|
||||
|
||||
### Configuration (agent.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "uc06",
|
||||
"type": "asterisk",
|
||||
"nats_url": "nats://balancer.host:4222",
|
||||
"nats_user": "",
|
||||
"nats_password": "",
|
||||
"interval_sec": 5,
|
||||
"max_calls": 150,
|
||||
"sip_gateway": "sip:10.101.60.115:5060",
|
||||
"sip_gateway_auto": false,
|
||||
"failure_window": 1000,
|
||||
"esl": { "host": "127.0.0.1", "port": 8021, "password": "ClueCon" },
|
||||
"ami": { "host": "127.0.0.1", "port": 6154, "username": "ctt", "password": "cttpass" }
|
||||
}
|
||||
```
|
||||
|
||||
| Поле | Тип | Default | Описание |
|
||||
|------|-----|---------|----------|
|
||||
| `node_id` | string | — | Уникальный ID ноды (uc06, ses-sip) |
|
||||
| `type` | string | — | `freeswitch` или `asterisk` |
|
||||
| `nats_url` | string | — | URL NATS-сервера |
|
||||
| `nats_user` | string | `""` | Пользователь NATS |
|
||||
| `nats_password` | string | `""` | Пароль NATS |
|
||||
| `interval_sec` | int | `5` | Интервал публикации метрик |
|
||||
| `max_calls` | int | `250` | Ёмкость ноды (fallback) |
|
||||
| `sip_gateway` | string | `""` | SIP-адрес для auto-создания транка |
|
||||
| `sip_gateway_auto` | bool | `false` | Авто-определение SIP из PBX |
|
||||
| `failure_window` | int | `1000` | Размер окна для call_failure_rate |
|
||||
| `esl.*` | object | — | FreeSWITCH ESL настройки (`host`, `port`, `password`) |
|
||||
| `ami.*` | object | — | Asterisk AMI настройки (`host`, `port`, `username`, `password`) |
|
||||
|
||||
### Collectors
|
||||
|
||||
| Collector | Источник | Метрики | PBX |
|
||||
|-----------|----------|---------|-----|
|
||||
| `freeswitch` | ESL: `show channels count`, `json status`, `eval $${idle_cpu}` | active_calls, max_calls, idle_cpu | ✅ |
|
||||
| `asterisk` | AMI: `CoreShowChannels`, `CoreSettings`, Hangup events | active_calls, max_calls, call_failure_rate | ✅ |
|
||||
| `system` | `/proc/loadavg`, `/proc/stat` | load_avg, idle_cpu (fallback) | ✅ |
|
||||
|
||||
**Call Failure Rate:** скользящее окно последних N (1000) завершённых звонков.
|
||||
|
||||
| Исход | FS (ESL) | Asterisk (AMI) |
|
||||
|-------|----------|---------------|
|
||||
| Успех | `CHANNEL_HANGUP` + `Hangup-Cause: NORMAL_CLEARING` | `Hangup` + `Cause: 16` |
|
||||
| Отказ | Любой другой `Hangup-Cause` | Любой другой `Cause` |
|
||||
|
||||
### Collector Interface (расширяемость)
|
||||
|
||||
```go
|
||||
type Collector interface {
|
||||
Type() string // "freeswitch" | "asterisk"
|
||||
Connect() error // ESL auth / AMI login
|
||||
Collect() (*PBXResult, error) // Сбор метрик
|
||||
ListenHangup(onHangup func(bool)) // Подписка на hangup события
|
||||
Close() // Закрытие соединения
|
||||
}
|
||||
```
|
||||
|
||||
Новый тип ноды (omnichannel, другой PBX) = новый файл `collector_<type>.go`, реализующий 5 методов интерфейса. Ничего в существующем коде менять не нужно.
|
||||
|
||||
### Template Configs
|
||||
|
||||
| Файл | Назначение | Тип |
|
||||
|------|-----------|-----|
|
||||
| `contrib/agent-uc.json` | Asterisk UC-ноды (05, 06, 66-69) | asterisk |
|
||||
| `contrib/agent-sessip.json` | FreeSWITCH ses-sip (10.3.0.44) | freeswitch |
|
||||
|
||||
### Deployment
|
||||
|
||||
```bash
|
||||
# 1. Собрать
|
||||
make build-agent
|
||||
|
||||
# 2. Скопировать на ноду
|
||||
scp bin/pulse-lets-go-agent devadmin@NODE_IP:/usr/local/bin/
|
||||
|
||||
# 3. Создать конфиг
|
||||
scp contrib/agent-uc.json devadmin@NODE_IP:/etc/pulse-lets-go-agent/agent.json
|
||||
|
||||
# 4. Запустить
|
||||
ssh devadmin@NODE_IP "pulse-lets-go-agent -config /etc/pulse-lets-go-agent/agent.json"
|
||||
```
|
||||
|
||||
### Systemd Unit
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/pulse-lets-go-agent.service
|
||||
[Unit]
|
||||
Description=Pulse Lets Go Agent — PBX metrics collector
|
||||
After=network.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
ExecStart=/usr/local/bin/pulse-lets-go-agent -config /etc/pulse-lets-go-agent/agent.json
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
# На PBX-ноде — проверить лог агента:
|
||||
journalctl -u pulse-lets-go-agent -f
|
||||
# Ожидаем:
|
||||
# [publisher] NATS подключён к nats://host:4222
|
||||
# [ami] подключён к 127.0.0.1:6154 (для Asterisk)
|
||||
# [agent] опубликована метрика: calls=0/150 load=0.28 cpu=99% status=ok
|
||||
|
||||
# На балансировщике — проверить что нода появилась:
|
||||
curl http://localhost:8080/api/nodes | jq '.[] | {node_id, score, active_calls}'
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
| Проблема | Причина | Решение |
|
||||
|----------|---------|---------|
|
||||
| `nats: no servers available` | NATS не запущен или недоступен | `systemctl start nats-server` |
|
||||
| `ami auth rejected` | Неверный пароль или доступ | Проверить `/etc/asterisk/manager.conf` |
|
||||
| `esl connect timeout` | ESL не слушает на порту | `fs_cli -x "load mod_event_socket"` |
|
||||
| `calls=0/0` | max_calls не удалось получить | Указать `max_calls` в `agent.json` |
|
||||
|
||||
---
|
||||
|
||||
## Разработка
|
||||
|
||||
### Команды
|
||||
|
||||
| Команда | Описание |
|
||||
|---------|----------|
|
||||
| `make build` | Полная сборка (Go + SvelteKit) |
|
||||
| `make dev` | Go backend + SvelteKit dev server (hot-reload) |
|
||||
| `make test` | Go тесты с coverage |
|
||||
| `make test-race` | Тесты с race detector |
|
||||
| `make lint` | `go vet ./...` |
|
||||
| `make fmt` | `go fmt ./...` |
|
||||
| `make build-agent` | Сборка pulse-lets-go-agent |
|
||||
| `make build-siptest` | Сборка SIP-тестера |
|
||||
| `make emulate-*` | Эмуляция метрик (normal, overload, chaos, stale, ...) |
|
||||
| `make siptest-*` | SIP-тестирование (uas, uac, full) |
|
||||
|
||||
### Стек
|
||||
|
||||
| Компонент | Технология |
|
||||
|-----------|-----------|
|
||||
| Backend | Go 1.26, stdlib `net/http` + `http.ServeMux` |
|
||||
| Frontend | SvelteKit 2 + Tailwind CSS 4 + Lucide |
|
||||
| Auth | JWT (golang-jwt/v5, HS256) + bcrypt |
|
||||
| Message bus | NATS (nats.go) |
|
||||
| Storage | JSON-файлы (data/) |
|
||||
| ESL | Raw TCP (без внешних библиотек) |
|
||||
|
||||
### Структура проекта
|
||||
|
||||
```
|
||||
pulse-lets-go/
|
||||
├── cmd/
|
||||
│ ├── pulse-lets-go/ # Точка входа (main.go)
|
||||
│ ├── pulse-lets-go-agent/ # Агент сбора метрик (FS / Asterisk)
|
||||
│ ├── emulator/ # Эмулятор метрик для тестирования
|
||||
│ └── siptest/ # SIP-тестер для e2e проверок
|
||||
├── internal/
|
||||
│ ├── api/ # HTTP-хендлеры (auth, nodes, trunks, users, monitoring, ws, route)
|
||||
│ ├── config/ # JSON-конфиги (atomic save, thread-safe)
|
||||
│ ├── engine/ # Scoring engine + router (sync.RWMutex, best-node cache)
|
||||
│ ├── esl/ # FreeSWITCH ESL client (raw TCP)
|
||||
│ ├── ami/ # Asterisk AMI client (raw TCP)
|
||||
│ ├── log/ # ASCII-логгер метрик с ротацией
|
||||
│ ├── models/ # Все типы данных (NodeMetric, NodeState, Trunk, User, ...)
|
||||
│ └── nats/ # NATS subscriber
|
||||
├── contrib/ # route.lua, agent-*.json (FreeSWITCH dialplan + агент)
|
||||
├── deploy/ # systemd unit
|
||||
├── web/ # SvelteKit фронтенд
|
||||
└── data/ # JSON-файлы времени выполнения (gitignored)
|
||||
```
|
||||
AGENTS.md содержит только этот файл — структура и связь рабочих элементов.
|
||||
see docs/development.md — план реализации и стек.
|
||||
|
||||
178
docs/agent.md
Normal file
178
docs/agent.md
Normal file
@ -0,0 +1,178 @@
|
||||
# Metrics Agent (pulse-lets-go-agent)
|
||||
|
||||
The agent runs on each PBX node (FreeSWITCH or Asterisk), collects metrics, and publishes them to NATS every 5 seconds.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────┐ pulse.metrics.<id> ┌─────────────────────┐
|
||||
│ pulse-lets-go-agent│──────────────────────────▶ │ NATS Server │
|
||||
│ (on each PBX node) │ every 5 seconds │ (central) │
|
||||
├─────────────────────┤ └────────┬────────────┘
|
||||
│ FreeSWITCH (ESL) │ │
|
||||
│ Asterisk (AMI) │ ▼
|
||||
│ System (/proc) │ ┌─────────────────┐
|
||||
└─────────────────────┘ │ pulse-lets-go │
|
||||
│ (scoring engine) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## NATS Message Format
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "pbx-01",
|
||||
"ts": 1719000000,
|
||||
"status": "ok",
|
||||
"active_calls": 42,
|
||||
"max_calls": 250,
|
||||
"idle_cpu": 35,
|
||||
"load_avg": 0.85,
|
||||
"call_failure_rate": 0.5,
|
||||
"sip_gateway": "sip:pbx-01.lan:5060"
|
||||
}
|
||||
```
|
||||
|
||||
- `status`: `"ok"` or any error string (triggers lethal)
|
||||
- `call_failure_rate`: 0.0..100.0 (percentage)
|
||||
- `sip_gateway`: optional, triggers auto-trunk creation on first metric
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
make build-agent
|
||||
# → bin/pulse-lets-go-agent (static binary, ~9 MB)
|
||||
```
|
||||
|
||||
## Configuration (agent.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "uc06",
|
||||
"type": "asterisk",
|
||||
"nats_url": "nats://balancer.host:4222",
|
||||
"nats_user": "",
|
||||
"nats_password": "",
|
||||
"interval_sec": 5,
|
||||
"max_calls": 150,
|
||||
"sip_gateway": "sip:10.101.60.115:5060",
|
||||
"sip_gateway_auto": false,
|
||||
"failure_window": 1000,
|
||||
"esl": { "host": "127.0.0.1", "port": 8021, "password": "ClueCon" },
|
||||
"ami": { "host": "127.0.0.1", "port": 6154, "username": "ctt", "password": "cttpass" }
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `node_id` | string | — | Unique node ID (uc06, ses-sip) |
|
||||
| `type` | string | — | `freeswitch` or `asterisk` |
|
||||
| `nats_url` | string | — | NATS server URL |
|
||||
| `nats_user` | string | `""` | NATS username |
|
||||
| `nats_password` | string | `""` | NATS password |
|
||||
| `interval_sec` | int | `5` | Metric publish interval |
|
||||
| `max_calls` | int | `250` | Node capacity (fallback) |
|
||||
| `sip_gateway` | string | `""` | SIP address for auto-trunk creation |
|
||||
| `sip_gateway_auto` | bool | `false` | Auto-detect SIP from PBX |
|
||||
| `failure_window` | int | `1000` | Window size for call_failure_rate |
|
||||
| `esl.*` | object | — | FreeSWITCH ESL settings (`host`, `port`, `password`) |
|
||||
| `ami.*` | object | — | Asterisk AMI settings (`host`, `port`, `username`, `password`) |
|
||||
|
||||
## Collectors
|
||||
|
||||
| Collector | Source | Metrics | PBX |
|
||||
|-----------|--------|---------|-----|
|
||||
| `freeswitch` | ESL: `show channels count`, `json status`, `eval $${idle_cpu}` | active_calls, max_calls, idle_cpu | ✅ |
|
||||
| `asterisk` | AMI: `CoreShowChannels`, `CoreSettings`, Hangup events | active_calls, max_calls, call_failure_rate | ✅ |
|
||||
| `system` | `/proc/loadavg`, `/proc/stat` | load_avg, idle_cpu (fallback) | ✅ |
|
||||
|
||||
### Call Failure Rate
|
||||
|
||||
Sliding window of last N (default 1000) completed calls.
|
||||
|
||||
| Outcome | FS (ESL) | Asterisk (AMI) |
|
||||
|---------|----------|---------------|
|
||||
| Success | `CHANNEL_HANGUP` + `Hangup-Cause: NORMAL_CLEARING` | `Hangup` + `Cause: 16` |
|
||||
| Failure | Any other `Hangup-Cause` | Any other `Cause` |
|
||||
|
||||
## Collector Interface (extensibility)
|
||||
|
||||
```go
|
||||
type Collector interface {
|
||||
Type() string // "freeswitch" | "asterisk"
|
||||
Connect() error // ESL auth / AMI login
|
||||
Collect() (*PBXResult, error) // Collect metrics
|
||||
ListenHangup(onHangup func(bool)) // Subscribe to hangup events
|
||||
Close() // Close connection
|
||||
}
|
||||
```
|
||||
|
||||
New PBX type = new `collector_<type>.go` implementing 5 interface methods. No existing code changes needed.
|
||||
|
||||
## Template Configs
|
||||
|
||||
| File | Target | Type |
|
||||
|------|--------|------|
|
||||
| `contrib/agent-uc.json` | Asterisk UC nodes (05, 06, 66-69) | asterisk |
|
||||
| `contrib/agent-sessip.json` | FreeSWITCH ses-sip (10.3.0.44) | freeswitch |
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
# 1. Build
|
||||
make build-agent
|
||||
|
||||
# 2. Copy to node
|
||||
scp bin/pulse-lets-go-agent devadmin@NODE_IP:/usr/local/bin/
|
||||
|
||||
# 3. Create config
|
||||
scp contrib/agent-uc.json devadmin@NODE_IP:/etc/pulse-lets-go-agent/agent.json
|
||||
|
||||
# 4. Start
|
||||
ssh devadmin@NODE_IP "pulse-lets-go-agent -config /etc/pulse-lets-go-agent/agent.json"
|
||||
```
|
||||
|
||||
## Systemd Unit
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/pulse-lets-go-agent.service
|
||||
[Unit]
|
||||
Description=Pulse Lets Go Agent — PBX metrics collector
|
||||
After=network.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
ExecStart=/usr/local/bin/pulse-lets-go-agent -config /etc/pulse-lets-go-agent/agent.json
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# On PBX node — check agent log:
|
||||
journalctl -u pulse-lets-go-agent -f
|
||||
# Expected:
|
||||
# [publisher] NATS connected to nats://host:4222
|
||||
# [ami] connected to 127.0.0.1:6154 (for Asterisk)
|
||||
# [agent] published metric: calls=0/150 load=0.28 cpu=99% status=ok
|
||||
|
||||
# On balancer — check node appeared:
|
||||
curl http://localhost:8080/api/nodes | jq '.[] | {node_id, score, active_calls}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| `nats: no servers available` | NATS not running or unreachable | `systemctl start nats-server` |
|
||||
| `ami auth rejected` | Wrong password or access | Check `/etc/asterisk/manager.conf` |
|
||||
| `esl connect timeout` | ESL not listening on port | `fs_cli -x "load mod_event_socket"` |
|
||||
| `calls=0/0` | max_calls not obtained | Set `max_calls` in `agent.json` |
|
||||
138
docs/api.md
Normal file
138
docs/api.md
Normal file
@ -0,0 +1,138 @@
|
||||
# API Reference
|
||||
|
||||
## Authentication
|
||||
|
||||
| Method | Path | Auth | Description |
|
||||
|--------|------|------|-------------|
|
||||
| `POST` | `/api/auth/login` | – | Login: `{"username":"admin","password":"admin"}` → `access_token` + `refresh_token` |
|
||||
| `POST` | `/api/auth/refresh` | – | Refresh access token using `refresh_token` |
|
||||
|
||||
All endpoints except `/api/route`, `/api/health/*`, `/api/monitoring/*` require `Authorization: Bearer <jwt>`.
|
||||
|
||||
Roles: `admin` (full access), `viewer` (read-only nodes/metrics).
|
||||
|
||||
## Core
|
||||
|
||||
| Method | Path | Auth | Description |
|
||||
|--------|------|------|-------------|
|
||||
| `GET` | `/api/route` | – | Route a call. Query: `caller_id`, `dest`, `ingress_trunk` |
|
||||
| `GET` | `/api/health` | – | Stats: total nodes, healthy, requests, uptime |
|
||||
| `GET` | `/api/health/live` | – | Liveness probe (always 200) |
|
||||
| `GET` | `/api/health/ready` | – | Readiness probe (NATS + metrics) |
|
||||
|
||||
### Route Response (success)
|
||||
|
||||
```json
|
||||
{ "node_id": "pbx-03", "score": 88, "sip_gateway": "sip:pbx03.lan:5060" }
|
||||
```
|
||||
|
||||
### Route Response (fallback — all nodes unhealthy)
|
||||
|
||||
```json
|
||||
{
|
||||
"fallback": true,
|
||||
"sip_gateway": "sip:operator.lan:5060",
|
||||
"reason": "all_nodes_unhealthy",
|
||||
"nodes": [...]
|
||||
}
|
||||
```
|
||||
|
||||
### Route Response (no nodes registered)
|
||||
|
||||
```json
|
||||
{ "error": "no_nodes_registered" }
|
||||
```
|
||||
|
||||
## Nodes
|
||||
|
||||
| Method | Path | Role | Description |
|
||||
|--------|------|------|-------------|
|
||||
| `GET` | `/api/nodes` | admin/viewer | List all nodes with scores and state |
|
||||
| `GET` | `/api/nodes/{id}/metrics` | admin/viewer | Metrics history (ring buffer, ~30 min) |
|
||||
| `PUT` | `/api/nodes/{id}/toggle` | admin | `{"disabled":true, "reason":"..."}` |
|
||||
|
||||
## Trunks
|
||||
|
||||
| Method | Path | Role | Description |
|
||||
|--------|------|------|-------------|
|
||||
| `GET` | `/api/trunks` | admin | All trunks. Query: `type=ingress\|balance\|fallback` |
|
||||
| `POST` | `/api/trunks` | admin | Create trunk |
|
||||
| `PUT` | `/api/trunks/{id}` | admin | Update trunk |
|
||||
| `DELETE` | `/api/trunks/{id}` | admin | Delete trunk |
|
||||
|
||||
Trunk model:
|
||||
```json
|
||||
{
|
||||
"id": "trk-001",
|
||||
"name": "pbx-03 balance",
|
||||
"type": "ingress|balance|fallback",
|
||||
"node_id": "pbx-03",
|
||||
"gateway": "sip:pbx03.lan:5060",
|
||||
"codecs": ["PCMU", "PCMA"],
|
||||
"context": "default",
|
||||
"enabled": true,
|
||||
"description": "",
|
||||
"created_at": "...",
|
||||
"updated_at": "..."
|
||||
}
|
||||
```
|
||||
|
||||
Trunk types:
|
||||
|
||||
| Type | Purpose | Quantity |
|
||||
|------|---------|----------|
|
||||
| `ingress` | Incoming trunk (where calls come from) | 1..N |
|
||||
| `balance` | Destination trunk (bound to a Node) | 1..N |
|
||||
| `fallback` | Operator fallback (when all balance score < 0) | exactly 1 |
|
||||
|
||||
## Users (admin only)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/users` | List users |
|
||||
| `POST` | `/api/users` | Create user |
|
||||
| `PUT` | `/api/users/{id}` | Update user |
|
||||
| `DELETE` | `/api/users/{id}` | Delete user |
|
||||
|
||||
Note: `user-01` (primary admin) cannot be deleted or demoted to viewer.
|
||||
|
||||
## Monitoring (API-key auth)
|
||||
|
||||
Use `X-API-Key` header (not JWT).
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/monitoring/zabbix` | JSON for Zabbix LLD + items |
|
||||
| `GET` | `/api/monitoring/prometheus` | `text/plain` metrics for Prometheus |
|
||||
|
||||
## WebSocket
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `WS` | `/ws/metrics` | Real-time metrics stream for frontend (JWT in query `?token=`) |
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Route (no auth)
|
||||
curl 'http://localhost:8080/api/route?caller_id=74951234567&dest=123&ingress_trunk=trk-001'
|
||||
# → {"node_id":"pbx-03","score":88,"sip_gateway":"sip:pbx03.lan:5060"}
|
||||
|
||||
# Login
|
||||
TOKEN=$(curl -s -X POST http://localhost:8080/api/auth/login \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"username":"admin","password":"admin"}' | jq -r '.access_token')
|
||||
|
||||
# Nodes with JWT
|
||||
curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/nodes
|
||||
|
||||
# Toggle node (admin only)
|
||||
curl -X PUT http://localhost:8080/api/nodes/pbx-01/toggle \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"disabled":true,"reason":"maintenance"}'
|
||||
|
||||
# Prometheus metrics (API-key)
|
||||
curl -H "X-API-Key: $(jq -r '.monitoring_api_key' data/config.json)" \
|
||||
http://localhost:8080/api/monitoring/prometheus
|
||||
```
|
||||
117
docs/architecture.md
Normal file
117
docs/architecture.md
Normal file
@ -0,0 +1,117 @@
|
||||
# Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
pulse-lets-go — балансировщик телефонной нагрузки на базе FreeSWITCH + NATS.
|
||||
|
||||
Агенты PBX отправляют метрики в NATS каждые 5 секунд. Балансировщик вычисляет weighted score готовности каждой ноды, кэширует лучшую и отдаёт её FreeSWITCH через `/api/route` для следующего звонка.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
┌─────────────────────┐ ESL/AMI ┌──────────────────────┐
|
||||
│ PBX-01 │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ collect │ (on each PBX node) │
|
||||
├─────────────────────┤ ├──────────────────────┤
|
||||
│ PBX-02 │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ ├──────────────────────┤
|
||||
├─────────────────────┤ ├──────────────────────┤
|
||||
│ PBX-N │──────────▶│ pulse-lets-go-agent │
|
||||
│ (FS or Asterisk) │ └─────────┬────────────┘
|
||||
└─────────────────────┘ │ pulse.metrics.<id>
|
||||
│ every 5s
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ NATS Server │
|
||||
│ nats://:4222 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ pulse-lets-go │
|
||||
│ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ NATS Sub │ │
|
||||
│ │ (pulse.metrics)│ │
|
||||
│ └────────┬───────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ Engine │ │
|
||||
│ │ (scoring) │──│──▶ /api/route
|
||||
│ └────────┬───────┘ │ ↓
|
||||
│ ▼ │ FreeSWITCH bridge
|
||||
│ ┌────────────────┐ │ sip:pbx-03:5060
|
||||
│ │ REST API │──│──▶ /api/nodes
|
||||
│ │ (JWT auth) │ │ /api/trunks
|
||||
│ └────────┬───────┘ │ /api/users
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ WebSocket │──│──▶ SvelteKit UI
|
||||
│ └────────────────┘ │ (ws://:8080/ws/metrics)
|
||||
│ ┌────────────────┐ │
|
||||
│ │ ESL Client │──│──▶ FreeSWITCH ESL
|
||||
│ │ (gateways) │ │ :8021
|
||||
│ └────────────────┘ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ Monitoring │──│──▶ /api/monitoring/prometheus
|
||||
│ │ (API-key) │ │ /api/monitoring/zabbix
|
||||
│ └────────────────┘ │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Role |
|
||||
|-----------|------|
|
||||
| **pulse-lets-go** | Central balancer: NATS subscriber, scoring engine, REST API, WebSocket hub, ESL client |
|
||||
| **pulse-lets-go-agent** | Per-PBX metrics collector (FreeSWITCH ESL / Asterisk AMI) |
|
||||
| **NATS** | Message bus — `pulse.metrics.<node_id>` subject |
|
||||
| **FreeSWITCH** | Softswitch — uses `route.lua` + `mod_curl` to query `/api/route` |
|
||||
| **SvelteKit UI** | Dashboard with real-time metrics via WebSocket |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Component | Technology |
|
||||
|-----------|-----------|
|
||||
| Backend | Go 1.26, stdlib `net/http` + `http.ServeMux` |
|
||||
| Frontend | SvelteKit 2 + Tailwind CSS 4 + Lucide |
|
||||
| Auth | JWT (golang-jwt/v5, HS256) + bcrypt |
|
||||
| Message bus | NATS (nats.go) |
|
||||
| Storage | JSON files (`data/`) |
|
||||
| ESL | Raw TCP (no external libraries) |
|
||||
| Deployment | Single binary + systemd |
|
||||
|
||||
## Data Flow Summary
|
||||
|
||||
1. **Agent** collects metrics from PBX (ESL/AMI + /proc) every 5s
|
||||
2. **Agent** publishes `pulse.metrics.<node_id>` to NATS
|
||||
3. **Balancer** NATS subscriber receives metrics, updates in-memory node state
|
||||
4. **Scoring engine** recalculates scores, caches best node
|
||||
5. **FreeSWITCH** dialplan calls `/api/route` via `mod_curl`
|
||||
6. **Balancer** returns best node's `sip_gateway` (O(1) from cache)
|
||||
7. **FreeSWITCH** bridges the call to the returned gateway
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
pulse-lets-go/
|
||||
├── cmd/
|
||||
│ ├── pulse-lets-go/ # Main entry point
|
||||
│ ├── pulse-lets-go-agent/ # Metrics collector agent (FS / Asterisk)
|
||||
│ ├── emulator/ # Metrics emulator for testing
|
||||
│ └── siptest/ # SIP tester for e2e checks
|
||||
├── internal/
|
||||
│ ├── api/ # HTTP handlers (auth, nodes, trunks, users, monitoring, ws, route)
|
||||
│ ├── config/ # JSON config manager (atomic save, thread-safe)
|
||||
│ ├── engine/ # Scoring engine + router (sync.RWMutex, best-node cache)
|
||||
│ ├── esl/ # FreeSWITCH ESL client (raw TCP)
|
||||
│ ├── ami/ # Asterisk AMI client (raw TCP)
|
||||
│ ├── log/ # ASCII metrics logger with rotation
|
||||
│ ├── models/ # Data types (NodeMetric, NodeState, Trunk, User, ...)
|
||||
│ └── nats/ # NATS subscriber
|
||||
├── contrib/ # route.lua, agent configs
|
||||
├── deploy/ # systemd unit
|
||||
├── web/ # SvelteKit frontend
|
||||
├── data/ # Runtime JSON files (gitignored)
|
||||
└── docs/ # Documentation
|
||||
```
|
||||
176
docs/deployment.md
Normal file
176
docs/deployment.md
Normal file
@ -0,0 +1,176 @@
|
||||
# Deployment
|
||||
|
||||
## Quick Deploy
|
||||
|
||||
```bash
|
||||
make deploy # full deploy to /opt/pulse-lets-go
|
||||
make install-systemd # install systemd unit
|
||||
```
|
||||
|
||||
## Directory Structure (after deploy)
|
||||
|
||||
```
|
||||
/opt/pulse-lets-go/
|
||||
├── bin/pulse-lets-go # Binary
|
||||
├── data/
|
||||
│ ├── config.json # Configuration
|
||||
│ ├── trunks.json # Trunks
|
||||
│ └── users.json # Users
|
||||
├── web/ # SvelteKit static files
|
||||
└── log/
|
||||
└── metrics.YYYY-MM-DD.log # ASCII metrics log
|
||||
```
|
||||
|
||||
## systemd Unit
|
||||
|
||||
```ini
|
||||
# deploy/pulse-lets-go.service
|
||||
[Unit]
|
||||
Description=Pulse Lets Go — Telephone Load Balancer
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=pulse
|
||||
Group=pulse
|
||||
ExecStart=/opt/pulse-lets-go/bin/pulse-lets-go
|
||||
WorkingDirectory=/opt/pulse-lets-go
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/pulse-lets-go/data /opt/pulse-lets-go/log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- `NoNewPrivileges=true` — privilege escalation blocked
|
||||
- `ProtectSystem=strict` — system directories read-only
|
||||
- `ProtectHome=yes` — home directory isolated
|
||||
- `PrivateTmp=true` — isolated /tmp
|
||||
- JWT secret and API-key auto-generated on first run
|
||||
|
||||
## Production Deployment (8 Steps)
|
||||
|
||||
### Step 1 — Deploy to Server
|
||||
|
||||
```bash
|
||||
make deploy
|
||||
scp -r /opt/pulse-lets-go devadmin@10.101.60.81:/opt/
|
||||
scp contrib/route.lua devadmin@10.101.60.81:/tmp/
|
||||
```
|
||||
|
||||
### Step 2 — NATS Server (if not installed)
|
||||
|
||||
```bash
|
||||
ssh devadmin@10.101.60.81
|
||||
sudo mkdir -p /opt/nats
|
||||
# Install nats-server from repository or copy binary
|
||||
# Run: nats-server -p 4222 -D &
|
||||
```
|
||||
|
||||
### Step 3 — Deploy Agent on PBX Nodes
|
||||
|
||||
Repeat for each UC/Asterisk node:
|
||||
|
||||
```bash
|
||||
# Copy binary and config to each PBX:
|
||||
scp bin/pulse-lets-go-agent devadmin@PBX_IP:/usr/local/bin/
|
||||
scp contrib/agent-uc.json devadmin@PBX_IP:/etc/pulse-lets-go-agent/agent.json
|
||||
|
||||
# Start:
|
||||
ssh devadmin@PBX_IP "systemctl start pulse-lets-go-agent"
|
||||
|
||||
# Check log:
|
||||
# journalctl -u pulse-lets-go-agent -f
|
||||
# → [ami] connected to 127.0.0.1:6154
|
||||
# → [agent] published metric: calls=0/150 load=0.28 cpu=99% status=ok
|
||||
```
|
||||
|
||||
### Step 4 — route.lua in FS Scripts
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /etc/freeswitch/scripts
|
||||
sudo cp /tmp/route.lua /etc/freeswitch/scripts/
|
||||
```
|
||||
|
||||
### Step 5 — Dialplan
|
||||
|
||||
Add `pulse_route` extension in `/etc/freeswitch/dialplan/default.xml` before other extensions:
|
||||
|
||||
```xml
|
||||
<extension name="pulse_route">
|
||||
<condition field="destination_number" expression="^(.*)$">
|
||||
<action application="set" data="balancer_url=http://127.0.0.1:8080"/>
|
||||
<action application="lua" data="route.lua"/>
|
||||
</condition>
|
||||
</extension>
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo fs_cli -x "reloadxml"
|
||||
```
|
||||
|
||||
### Step 6 — Create Balance Trunks
|
||||
|
||||
Trunks are created automatically when an agent sends its first metric with `sip_gateway`. Manual creation:
|
||||
|
||||
```bash
|
||||
curl -X POST .../api/trunks \
|
||||
-d '{"name":"uc06","type":"balance","node_id":"uc06","gateway":"sip:10.101.60.115:5060","enabled":true}'
|
||||
```
|
||||
|
||||
### Step 7 — Start pulse-lets-go
|
||||
|
||||
```bash
|
||||
# config.json auto-creates on first run in /opt/pulse-lets-go/data/
|
||||
# Add esl block to config.json:
|
||||
# "esl": {"host": "127.0.0.1", "port": 8021, "password": "ClueCon"}
|
||||
/opt/pulse-lets-go/bin/pulse-lets-go
|
||||
```
|
||||
|
||||
### Step 8 — Verify
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/api/health | jq '.connections'
|
||||
# → {"nats":"connected","esl":"connected"}
|
||||
curl 'http://127.0.0.1:8080/api/route?caller_id=123&dest=456'
|
||||
# → {"node_id":"pbx-03","score":88,"sip_gateway":...}
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
# 1. Remove pulse_route from dialplan
|
||||
sudo sed -i '/pulse_route/,/<\/extension>/d' /etc/freeswitch/dialplan/default.xml
|
||||
sudo fs_cli -x "reloadxml"
|
||||
|
||||
# 2. Stop pulse-lets-go
|
||||
sudo systemctl stop pulse-lets-go
|
||||
|
||||
# Calls continue through original dialplan without balancer.
|
||||
```
|
||||
|
||||
## Access Permissions (production)
|
||||
|
||||
| Operation | Required Permissions |
|
||||
|-----------|---------------------|
|
||||
| Read FS configs | root or freeswitch |
|
||||
| `fs_cli -x` commands | root or freeswitch |
|
||||
| ESL (port 8021) | ACL in `event_socket.conf.xml` |
|
||||
| Install software | root |
|
||||
| Write to `/etc/freeswitch/scripts/` | root |
|
||||
| `reloadxml` | root or freeswitch |
|
||||
|
||||
## Tested Versions
|
||||
|
||||
| Component | Version |
|
||||
|-----------|---------|
|
||||
| FreeSWITCH | 1.10.12+ |
|
||||
| NATS Server | 2.10+ |
|
||||
| Go | 1.26 |
|
||||
| OS | Linux (CentOS 7+, AlmaLinux, Arch) |
|
||||
119
docs/development.md
Normal file
119
docs/development.md
Normal file
@ -0,0 +1,119 @@
|
||||
# Development
|
||||
|
||||
## Requirements
|
||||
|
||||
- Go 1.26+
|
||||
- Node.js 22+
|
||||
- NATS Server (auto-downloaded via `make nats`)
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
git clone git@git.totmin.ru:en2zmax/pulse-lets-go.git
|
||||
cd pulse-lets-go
|
||||
make build
|
||||
```
|
||||
|
||||
## Makefile Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `make build` | Full build (Go + SvelteKit) |
|
||||
| `make dev` | Go backend + SvelteKit dev server (hot-reload) |
|
||||
| `make test` | Go tests with coverage |
|
||||
| `make test-race` | Tests with race detector |
|
||||
| `make lint` | `go vet ./...` |
|
||||
| `make fmt` | `go fmt ./...` |
|
||||
| `make build-agent` | Build pulse-lets-go-agent |
|
||||
| `make build-siptest` | Build SIP tester |
|
||||
| `make emulate-*` | Metrics emulation (normal, overload, chaos, stale, ...) |
|
||||
| `make siptest-*` | SIP testing (uas, uac, full) |
|
||||
| `make nats` | Start local NATS server |
|
||||
|
||||
## Running Locally
|
||||
|
||||
```bash
|
||||
# Terminal 1: NATS
|
||||
make nats
|
||||
|
||||
# Terminal 2: balancer
|
||||
./bin/pulse-lets-go
|
||||
|
||||
# Terminal 3: metrics emulator (3 healthy nodes)
|
||||
make emulate-normal
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
Before committing:
|
||||
|
||||
```bash
|
||||
make fmt
|
||||
make lint
|
||||
make test
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
pulse-lets-go/
|
||||
├── cmd/
|
||||
│ ├── pulse-lets-go/ # Main entry point
|
||||
│ ├── pulse-lets-go-agent/ # Metrics collector (FS / Asterisk)
|
||||
│ ├── emulator/ # Metrics emulator for testing
|
||||
│ └── siptest/ # SIP tester for e2e checks
|
||||
├── internal/
|
||||
│ ├── api/ # HTTP handlers (auth, nodes, trunks, users, monitoring, ws, route)
|
||||
│ ├── config/ # JSON config manager (atomic save, thread-safe)
|
||||
│ ├── engine/ # Scoring engine + router (sync.RWMutex, best-node cache)
|
||||
│ ├── esl/ # FreeSWITCH ESL client (raw TCP)
|
||||
│ ├── ami/ # Asterisk AMI client (raw TCP)
|
||||
│ ├── log/ # ASCII metrics logger with rotation
|
||||
│ ├── models/ # Data types (NodeMetric, NodeState, Trunk, User, ...)
|
||||
│ └── nats/ # NATS subscriber
|
||||
├── contrib/ # route.lua, agent configs
|
||||
├── deploy/ # systemd unit
|
||||
├── web/ # SvelteKit frontend
|
||||
└── data/ # Runtime JSON files (gitignored)
|
||||
```
|
||||
|
||||
## Adding a New Endpoint
|
||||
|
||||
1. Define model in `internal/models/types.go`
|
||||
2. Implement handler in `internal/api/<name>.go`
|
||||
3. Register route in `internal/api/router.go` with required middleware (auth, admin)
|
||||
4. Add tests in `internal/engine/engine_test.go` if scoring is affected
|
||||
|
||||
## Commit Messages
|
||||
|
||||
[Conventional Commits](https://www.conventionalcommits.org/) format:
|
||||
|
||||
```
|
||||
feat: add support for custom scoring weights
|
||||
fix: handle nil ESL client in route handler
|
||||
docs: add API reference table to README
|
||||
refactor: extract scoring logic into separate method
|
||||
test: add lethal condition tests for stale nodes
|
||||
chore: update Go dependencies
|
||||
```
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. `cmd/pulse-lets-go/main.go` + Go module + пустая структура `internal/`
|
||||
2. `internal/models/` — все типы данных
|
||||
3. `internal/config/` — чтение/запись JSON, атомарное сохранение
|
||||
4. `internal/nats/` — подписка, in-memory store
|
||||
5. `internal/engine/` — scorer + router
|
||||
6. `internal/api/` — все хендлеры
|
||||
7. `internal/log/` — ASCII-логгер + ring buffer
|
||||
8. Web-фронт (SvelteKit)
|
||||
9. Makefile + systemd unit
|
||||
10. Тесты + graceful shutdown
|
||||
|
||||
## PR Process
|
||||
|
||||
1. Branch from `main`, named `feature/`, `fix/`, or `docs/`
|
||||
2. One PR — one logical change
|
||||
3. All tests pass (`make test`)
|
||||
4. Code passes `make lint`
|
||||
5. If API changes — update docs
|
||||
104
docs/freeswitch.md
Normal file
104
docs/freeswitch.md
Normal file
@ -0,0 +1,104 @@
|
||||
# FreeSWITCH Integration
|
||||
|
||||
pulse-lets-go routes calls through FreeSWITCH via `mod_curl` → Lua dialplan → `/api/route`.
|
||||
|
||||
## Dialplan (route.lua)
|
||||
|
||||
FreeSWITCH uses a Lua script in its dialplan. The script calls `/api/route` via `mod_curl` and bridges to the returned gateway.
|
||||
|
||||
```lua
|
||||
-- contrib/route.lua (simplified)
|
||||
api = freeswitch.API()
|
||||
caller_id = session:getVariable("caller_id_number") or ""
|
||||
dest = session:getVariable("destination_number") or ""
|
||||
url = "http://localhost:8080/api/route?caller_id=" .. caller_id .. "&dest=" .. dest
|
||||
|
||||
raw = api:execute("curl", url)
|
||||
body = raw:match("\r?\n\r?\n(.+)") or raw
|
||||
|
||||
local ok, route = pcall(cjson.decode, body)
|
||||
if not ok then session:hangup("NORMAL_TEMPORARY_FAILURE") return end
|
||||
|
||||
if route.fallback then
|
||||
session:bridge(route.sip_gateway) -- operator
|
||||
elseif route.sip_gateway then
|
||||
session:bridge(route.sip_gateway) -- best node
|
||||
else
|
||||
session:hangup("NORMAL_TEMPORARY_FAILURE")
|
||||
end
|
||||
```
|
||||
|
||||
## Dialplan Configuration
|
||||
|
||||
```xml
|
||||
<extension name="route_call">
|
||||
<condition field="destination_number" expression="^.*$">
|
||||
<action application="set" data="balancer_url=http://balancer.lan:8080"/>
|
||||
<action application="set" data="ingress_trunk=trk-001"/>
|
||||
<action application="lua" data="route.lua"/>
|
||||
</condition>
|
||||
</extension>
|
||||
```
|
||||
|
||||
Reload dialplan:
|
||||
```bash
|
||||
sudo fs_cli -x "reloadxml"
|
||||
```
|
||||
|
||||
## ESL (Gateway Management)
|
||||
|
||||
When `esl.host` is configured in `config.json`, the balancer:
|
||||
- Connects to FreeSWITCH ESL (`:8021`)
|
||||
- Synchronizes ingress trunks as Sofia gateways
|
||||
- Listens for gateway registration events
|
||||
- Streams gateway status to WebSocket frontend
|
||||
|
||||
### Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"esl": {
|
||||
"host": "127.0.0.1",
|
||||
"port": 8021,
|
||||
"password": "ClueCon",
|
||||
"password_env": "",
|
||||
"sofia_profile": "external",
|
||||
"gateway_prefix": "pulse"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `password_env` — alternative: name of environment variable containing password
|
||||
- `sofia_profile` — Sofia profile to manage (default: `external`)
|
||||
- `gateway_prefix` — prefix for auto-created gateways (default: `pulse`)
|
||||
|
||||
## Prerequisites (before deployment)
|
||||
|
||||
| # | Check | Command | Expected |
|
||||
|:-:|-------|---------|----------|
|
||||
| 1 | **ESL running** | `nc -z 127.0.0.1 8021 && echo OK` | `OK` |
|
||||
| 2 | **ESL password** | `echo "auth ClueCon" \| nc 127.0.0.1 8021` | `+OK accepted` |
|
||||
| 3 | **mod_curl loaded** | `fs_cli -x "show modules" \| grep mod_curl` | `<load module="mod_curl"/>` |
|
||||
| 4 | **mod_lua loaded** | `fs_cli -x "show modules" \| grep mod_lua` | `<load module="mod_lua"/>` |
|
||||
| 5 | **ESL ACL** | ACL in `event_socket.conf.xml` allows 127.0.0.1 | `localhost` or `127.0.0.1` in ACL |
|
||||
|
||||
## Loading mod_curl Without Restart
|
||||
|
||||
```bash
|
||||
fs_cli -x "load mod_curl"
|
||||
# Add to modules.conf.xml for persistence:
|
||||
# <load module="mod_curl"/>
|
||||
```
|
||||
|
||||
## Rollback (if route.lua breaks calls)
|
||||
|
||||
```bash
|
||||
# 1. Remove pulse_route from dialplan
|
||||
sudo sed -i '/pulse_route/,/<\/extension>/d' /etc/freeswitch/dialplan/default.xml
|
||||
sudo fs_cli -x "reloadxml"
|
||||
|
||||
# 2. Stop pulse-lets-go
|
||||
sudo systemctl stop pulse-lets-go
|
||||
|
||||
# Calls continue through original dialplan without balancer.
|
||||
```
|
||||
71
docs/scoring.md
Normal file
71
docs/scoring.md
Normal file
@ -0,0 +1,71 @@
|
||||
# Scoring Engine
|
||||
|
||||
Вычисляет score (0..100) для каждой ноды. Ноды со score = -100 исключаются из роутинга.
|
||||
|
||||
## Lethal Conditions (score → -100)
|
||||
|
||||
Если хотя бы одно условие истинно — нода исключается:
|
||||
If at least one is true — node is excluded from routing.
|
||||
|
||||
| Condition | Threshold |
|
||||
|-----------|-----------|
|
||||
| `status != "ok"` | Any status other than `ok` |
|
||||
| Stale | > `stale_threshold_sec` (default 20s) |
|
||||
| `active_calls >= max_calls` | 100% capacity used |
|
||||
| `idle_cpu < idle_cpu_min` | < 5% |
|
||||
| `call_failure_rate > call_failure_rate_lethal` | > 15% |
|
||||
| `disabled == true` | Manual admin disable |
|
||||
|
||||
## Weighted Score (0..100)
|
||||
|
||||
```
|
||||
call_score = clamp(100 - (active_calls / max_calls * 100), 0, 100)
|
||||
load_score = clamp(100 - (load_avg * 50), 0, 100)
|
||||
idle_score = clamp(idle_cpu, 0, 100)
|
||||
fail_score = clamp(100 - call_failure_rate, 0, 100)
|
||||
|
||||
score = call_score × 0.40 +
|
||||
load_score × 0.30 +
|
||||
idle_score × 0.20 +
|
||||
fail_score × 0.10
|
||||
```
|
||||
|
||||
Weight defaults (configurable in `config.json`): call 40%, load 30%, idle 20%, fail 10%.
|
||||
|
||||
## Example
|
||||
|
||||
Node `pbx-01`: `active_calls=42`, `max_calls=250`, `load_avg=0.85`, `idle_cpu=35`, `fail_rate=0.5`
|
||||
|
||||
```
|
||||
call_score = clamp(100 - (42/250)*100, 0, 100) = 83.2
|
||||
load_score = clamp(100 - 0.85*50, 0, 100) = 57.5
|
||||
idle_score = clamp(35, 0, 100) = 35.0
|
||||
fail_score = clamp(100 - 0.5, 0, 100) = 99.5
|
||||
|
||||
score = 83.2×0.40 + 57.5×0.30 + 35.0×0.20 + 99.5×0.10 = 70.1
|
||||
```
|
||||
|
||||
## Caching
|
||||
|
||||
`best_node_id`, `best_score`, and `fallback_active` are recalculated on every metric update.
|
||||
`/api/route` reads them under RLock — O(1) constant time.
|
||||
|
||||
## Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"stale_threshold_sec": 20,
|
||||
"scoring": {
|
||||
"idle_cpu_min": 5,
|
||||
"call_failure_rate_lethal": 15.0,
|
||||
"weights": {
|
||||
"call_score": 0.40,
|
||||
"load_score": 0.30,
|
||||
"idle_score": 0.20,
|
||||
"fail_score": 0.10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Weights must sum to 1.0. All thresholds configurable in `config.json`.
|
||||
Loading…
x
Reference in New Issue
Block a user