Compare commits
3
Commits
a0b4cb2b14
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9b85df9f6c | ||
|
|
e5e0ffedc0 | ||
|
|
17ff5c591f |
@@ -1,7 +1,7 @@
|
|||||||
GO ?= go
|
GO ?= go
|
||||||
BIN_DIR ?= bin
|
BIN_DIR ?= bin
|
||||||
|
|
||||||
.PHONY: all build test vet fmt tidy run clean seed
|
.PHONY: all build test vet fmt tidy run clean seed check e2e
|
||||||
|
|
||||||
all: build
|
all: build
|
||||||
|
|
||||||
@@ -18,6 +18,24 @@ test:
|
|||||||
vet:
|
vet:
|
||||||
$(GO) vet ./...
|
$(GO) vet ./...
|
||||||
|
|
||||||
|
## check: vet + race tests + architecture guard (the address core must not
|
||||||
|
## depend on any events/outages adapter)
|
||||||
|
check: vet test
|
||||||
|
@if $(GO) list -deps ./cmd/addrkitd | grep -Eq 'domain/outages|adapters/arcgis'; then \
|
||||||
|
echo "addrkitd must not depend on events/outages packages" >&2; \
|
||||||
|
exit 1; \
|
||||||
|
fi
|
||||||
|
|
||||||
|
## e2e: process-level tests against a real registry snapshot.
|
||||||
|
## Set ADDRKIT_E2E_REGISTRY to a Bulk NDJSON dump (and optionally
|
||||||
|
## ADDRKIT_E2E_LOG to an EДДС address log). Data is never copied into the repo.
|
||||||
|
e2e:
|
||||||
|
@test -n "$(ADDRKIT_E2E_REGISTRY)" || { \
|
||||||
|
echo "set ADDRKIT_E2E_REGISTRY=/path/to/address_registry.ndjson" >&2; \
|
||||||
|
exit 1; \
|
||||||
|
}
|
||||||
|
$(GO) test -tags e2e -count=1 -v ./test/e2e
|
||||||
|
|
||||||
## fmt: format all Go sources
|
## fmt: format all Go sources
|
||||||
fmt:
|
fmt:
|
||||||
gofmt -l -w .
|
gofmt -l -w .
|
||||||
@@ -30,11 +48,14 @@ tidy:
|
|||||||
run: build
|
run: build
|
||||||
$(BIN_DIR)/addrkitd -config config.json
|
$(BIN_DIR)/addrkitd -config config.json
|
||||||
|
|
||||||
## seed: build registry.db from a registry dump (override IN=/OUT=)
|
## seed: build registry.db from a registry dump (override IN=/OUT=/CITY=/CITIES=)
|
||||||
IN ?= registry.ndjson
|
IN ?= registry.ndjson
|
||||||
OUT ?= registry.db
|
OUT ?= registry.db
|
||||||
|
CITY ?=
|
||||||
|
CITIES ?=
|
||||||
seed:
|
seed:
|
||||||
$(BIN_DIR)/addrkit seed -in $(IN) -out $(OUT)
|
$(BIN_DIR)/addrkit seed -in $(IN) -out $(OUT) \
|
||||||
|
$(if $(CITY),-city "$(CITY)") $(if $(CITIES),-cities "$(CITIES)")
|
||||||
|
|
||||||
## clean: remove build artifacts
|
## clean: remove build artifacts
|
||||||
clean:
|
clean:
|
||||||
|
|||||||
@@ -1,36 +1,32 @@
|
|||||||
# addrkit
|
# addrkit
|
||||||
|
|
||||||
**Универсальный детерминированный сервис нормализации и поиска адресов.**
|
**Детерминированный многогородской сервис нормализации и поиска адресов.**
|
||||||
Разбирает вольные реплики (дроби, литеры, корпуса, строения, опечатки),
|
Разбирает вольные реплики (дроби, литеры, корпуса, строения, опечатки),
|
||||||
сопоставляет их с каноническим реестром и отдаёт ответ **только из данных** —
|
сопоставляет их с каноническим реестром и отдаёт ответ **только из данных** —
|
||||||
LLM не участвует в пути корректности.
|
LLM не участвует в пути корректности.
|
||||||
|
|
||||||
Ядро продукта — **только адрес**: оно не привязано ни к ЕДДС, ни к отключениям.
|
Сервис делает одну вещь: превращает свободный адрес в канонический. Он не
|
||||||
Опциональный модуль «**события по дому**» (отключения, аварии, плановые работы —
|
знает ни про отключения, ни про аварии, ни про другие события. Если такие
|
||||||
данные, привязанные к конкретному дому) подключается отдельно; ЕДДС — лишь один
|
данные нужны, их источник — отдельный сервис, который использует addrkit как
|
||||||
из его потребителей.
|
нормализатор.
|
||||||
|
|
||||||
Проект на Go, живёт офлайн (один статический бинарь + SQLite), без MCP в
|
## Что умеет
|
||||||
рантайме.
|
|
||||||
|
|
||||||
## Позиционирование
|
- **Города.** Город — часть модели и ключа улицы. «Ленина 5» в Новосибирске и
|
||||||
|
в Бердске — разные адреса; без указания города сервис честно вернёт
|
||||||
- **Ядро (универсальное):** нормализация, структурирование и поиск адреса по
|
`ambiguous`, а не выберет наугад.
|
||||||
реестру, статусы `found/partial/ambiguous/not_found`, версия данных.
|
- **Алиасы городов.** `нск`, `новосиб`, `новосибирская область`, `нсо` — всё
|
||||||
Годится для колл-центра, доставки, CRM, аналитики, любого робота.
|
это настраивается, а не зашито в код.
|
||||||
- **Опциональный модуль «события по дому»:** получает канонический адрес из
|
- **Строгий дом.** Дробь, литера, корпус и строение должны совпасть. Запрос
|
||||||
ядра и отдаёт события, **привязанные к этому дому** (совпадение дома, дроби,
|
«1 стр. 2» не может вернуть «1».
|
||||||
литеры и корпуса; строение игнорируется). Источник — plug-in
|
- **Безопасные статусы.** `found`, `partial`, `ambiguous`, `not_found`.
|
||||||
(`outages.Source`): сегодня ArcGIS-портал ЕДДС, завтра любой фид (ES, SQL,
|
Сервис лучше скажет «уточните», чем подставит чужой дом.
|
||||||
API). Ядро про события ничего не знает.
|
- **Опечатки.** Нечёткое сопоставление названия улицы с порогом и явной
|
||||||
|
зоной неоднозначности.
|
||||||
## Зачем
|
- **Опциональный LLM-rewriter.** Может предложить варианты написания, но
|
||||||
|
ответ всегда берётся из реестра, а вариант принимается только если меняет
|
||||||
Существующая логика поиска адреса в диалоговом сценарии страдала от
|
*написание* улицы, не трогая город, порядковый номер и дом.
|
||||||
нечёткого поиска по всей строке (`fuzziness: AUTO`, `size:1`), из-за чего
|
- **Офлайн.** Один статический бинарь + SQLite. В пути резолва нет сети.
|
||||||
дробь/корпус/литера могли дать неверный дом. `addrkit` разделяет задачу на
|
|
||||||
компоненты, сопоставляет дом строго и вместо угадывания возвращает явный
|
|
||||||
статус (`partial`/`ambiguous`).
|
|
||||||
|
|
||||||
## Архитектура
|
## Архитектура
|
||||||
|
|
||||||
@@ -40,24 +36,44 @@ LLM не участвует в пути корректности.
|
|||||||
cmd/addrkitd сервис (HTTP)
|
cmd/addrkitd сервис (HTTP)
|
||||||
cmd/addrkit CLI: seed | verify | resolve | eval | selfcheck
|
cmd/addrkit CLI: seed | verify | resolve | eval | selfcheck
|
||||||
internal/
|
internal/
|
||||||
domain/address нормализация и разбор адреса (чистые функции)
|
domain/address нормализация, словарь городов, разбор адреса
|
||||||
domain/match сопоставление с индексом, статусы ← ядро
|
domain/match сопоставление с индексом, статусы ← ядро
|
||||||
domain/outages «события по дому»: фильтр по дому ← опциональный модуль
|
|
||||||
adapters/registrystore SQLite (modernc, без CGO)
|
adapters/registrystore SQLite (modernc, без CGO)
|
||||||
adapters/arcgis клиент источника событий (timeout/retry) ← пример plug-in
|
|
||||||
adapters/llm опциональный rewriter (OpenAI-совместимый)
|
adapters/llm опциональный rewriter (OpenAI-совместимый)
|
||||||
adapters/httpapi версионированный JSON API
|
adapters/httpapi версионированный JSON API
|
||||||
config конфиг + ${VAR}
|
config конфиг + ${VAR} + словарь городов
|
||||||
seed сборка индекса из дампа реестра
|
seed сборка индекса из дампа реестра
|
||||||
|
eval offline-оценка качества
|
||||||
|
pkg/addrkit публичный фасад для использования как библиотеки
|
||||||
```
|
```
|
||||||
|
|
||||||
Поток запроса:
|
Поток запроса:
|
||||||
|
|
||||||
```
|
```
|
||||||
"2-я Станционная улица 1/1 стр."
|
"нск, 2-я Станционная улица 1/1 стр."
|
||||||
-> address.Parse -> {ordinal:2, name:станционная, house:1, frac:1}
|
-> address.Parser -> {city: Новосибирск, ordinal: 2, name: станционная, house: 1, frac: 1}
|
||||||
-> match.Resolve -> exact street -> exact house -> FOUND
|
-> match.Resolve -> exact city -> exact street -> exact house -> FOUND
|
||||||
-> (опц. модуль «события по дому») -> фильтр по дому -> ответ
|
```
|
||||||
|
|
||||||
|
`match.Index` неизменяемый, резолвер не хранит состояния. Индекс собирается
|
||||||
|
один раз при старте и безопасен для конкурентных чтений.
|
||||||
|
|
||||||
|
Если нормализатор нужен другому проекту как библиотека, используйте
|
||||||
|
`pkg/addrkit`: он не делает I/O и не требует HTTP или SQLite.
|
||||||
|
|
||||||
|
```go
|
||||||
|
parser := addrkit.NewParser(
|
||||||
|
addrkit.NewDictionary([]addrkit.City{{Name: "Новосибирск", Aliases: []string{"нск"}}}),
|
||||||
|
"",
|
||||||
|
)
|
||||||
|
parsed := parser.Parse("нск, Ленина 5")
|
||||||
|
|
||||||
|
resolver := addrkit.NewResolverWithOptions(
|
||||||
|
addrkit.NewIndex(version, entries, addrkit.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
addrkit.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
result := resolver.Resolve(ctx, "нск, Ленина 5")
|
||||||
```
|
```
|
||||||
|
|
||||||
## Быстрый старт
|
## Быстрый старт
|
||||||
@@ -65,72 +81,127 @@ internal/
|
|||||||
```bash
|
```bash
|
||||||
make build
|
make build
|
||||||
|
|
||||||
# 1. Собрать индекс из дампа реестра (Bulk NDJSON или JSONL).
|
# 1. Собрать индекс из дампа реестра.
|
||||||
./bin/addrkit seed -in registry.ndjson -out registry.db -version 2026-10-01
|
# -city задаёт город для строк без поля city;
|
||||||
|
# -cities — необязательный JSON-справочник алиасов.
|
||||||
|
./bin/addrkit seed -in registry.ndjson -out registry.db \
|
||||||
|
-version 2026-10-01 -city "Новосибирск"
|
||||||
|
|
||||||
# 2. Запустить сервис.
|
# 2. Запустить сервис.
|
||||||
cp config.example.json config.json
|
cp config.example.json config.json
|
||||||
ADDRKIT_PORTAL_URL='https://map.example/.../101/query' ./bin/addrkitd -config config.json
|
./bin/addrkitd -config config.json
|
||||||
|
|
||||||
# 3. Проверить.
|
# 3. Проверить.
|
||||||
curl -s localhost:6171/v1/resolve -d '{"address":"2-я Станционная улица 1/1 стр."}'
|
curl -s localhost:6171/v1/resolve -d '{"address":"нск, 2-я Станционная 1/1 стр."}'
|
||||||
curl -s localhost:6171/v1/outages -d '{"address":"станционная 30а к9/1"}'
|
curl -s localhost:6171/v1/cities
|
||||||
```
|
```
|
||||||
|
|
||||||
CLI без сервиса:
|
CLI без сервиса:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./bin/addrkit resolve -db registry.db "кошурникова 11/1"
|
./bin/addrkit resolve -db registry.db -city "Новосибирск" "кошурникова 11/1"
|
||||||
./bin/addrkit verify -db registry.db
|
./bin/addrkit verify -db registry.db
|
||||||
./bin/addrkit selfcheck -db registry.db
|
./bin/addrkit selfcheck -db registry.db
|
||||||
|
./bin/addrkit shadow -db registry.db -es https://es:9200 -sample 300
|
||||||
```
|
```
|
||||||
|
|
||||||
## API v1
|
## API v1
|
||||||
|
|
||||||
| Метод | Путь | Назначение |
|
| Метод | Путь | Назначение |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| POST | `/v1/resolve` | **[ядро]** адрес → `{status, address, district, confidence, candidates[]}` |
|
| POST | `/v1/resolve` | адрес → `{status, address, city, district, confidence, candidates[], data_version}` |
|
||||||
| POST | `/v1/outages` | **[опц. модуль]** адрес → канонический адрес + события/отключения по дому |
|
| POST | `/v1/city` | только город: `нск` → `Новосибирск` |
|
||||||
|
| GET | `/v1/cities` | канонические города, присутствующие в индексе |
|
||||||
| GET | `/v1/healthz` | живость |
|
| GET | `/v1/healthz` | живость |
|
||||||
| GET | `/v1/readyz` | готовность (индекс загружен) |
|
| GET | `/v1/readyz` | готовность (индекс загружен) |
|
||||||
| GET | `/v1/version` | версия данных |
|
| GET | `/v1/version` | версия данных |
|
||||||
|
|
||||||
Статусы: `found`, `partial`, `ambiguous`, `not_found`, `portal_timeout`,
|
Запрос:
|
||||||
`infra_error`.
|
|
||||||
|
|
||||||
## Конфигурация
|
```json
|
||||||
|
{"address": "Ленина 5", "city": "Бердск"}
|
||||||
|
```
|
||||||
|
|
||||||
|
`city` необязателен и переопределяет город, найденный в тексте. Лимит адреса —
|
||||||
|
512 символов, ответ на слишком длинный ввод — `400`.
|
||||||
|
|
||||||
|
Только город, без улицы:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:6171/v1/city -d '{"query":"нск"}'
|
||||||
|
# {"status":"found","city":"Новосибирск"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Это же доступно как библиотека: `addrkit.NewDictionary(...).MatchCity("нск")`.
|
||||||
|
|
||||||
|
Статусы: `found`, `partial`, `ambiguous`, `not_found`.
|
||||||
|
|
||||||
|
## Города и конфигурация
|
||||||
|
|
||||||
`config.json` (см. `config.example.json`). Значения поддерживают `${VAR}`;
|
`config.json` (см. `config.example.json`). Значения поддерживают `${VAR}`;
|
||||||
незаданная переменная = ошибка запуска (fail closed). Секреты — только из
|
незаданная переменная = ошибка запуска (fail closed).
|
||||||
окружения.
|
|
||||||
|
|
||||||
Промпт LLM-rewriter **не зашит в код**: встроенный дефолт лежит в
|
```json
|
||||||
`internal/adapters/llm/default_prompt.txt` и компилируется в бинарь (`go:embed`).
|
{
|
||||||
Переопределить можно в конфиге: `llm.system_prompt` (инлайн) или
|
"addr": ":6171",
|
||||||
`llm.system_prompt_file` (путь к файлу; приоритетнее инлайна).
|
"data_path": "registry.db",
|
||||||
|
"cities": [
|
||||||
|
{"name": "Новосибирск", "aliases": ["нск", "новосиб", "новосибирская область", "нсо"]},
|
||||||
|
{"name": "Бердск"}
|
||||||
|
],
|
||||||
|
"default_city": "Новосибирск",
|
||||||
|
"resolve": {"fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `cities` — словарь канонических названий и алиасов. Один и тот же алиас не
|
||||||
|
может принадлежать двум городам: конфиг не загрузится.
|
||||||
|
- `default_city` — удобство для одногородского деплоя: запрос «Ленина 5» без
|
||||||
|
города будет отнесён к нему. В многогородском деплое оставьте поле пустым:
|
||||||
|
угадывание города — это ровно та ошибка, которую сервис не должен делать.
|
||||||
|
- `resolve.ambiguity_margin` — насколько близкие улицы считаются
|
||||||
|
неоднозначными. Значение по умолчанию одно для сервиса, CLI и eval.
|
||||||
|
|
||||||
|
Справочник городов можно держать отдельным файлом (`addrkit seed -cities
|
||||||
|
cities.json`); формат — JSON-массив тех же объектов.
|
||||||
|
|
||||||
|
## Обновление данных и совместимость
|
||||||
|
|
||||||
|
`registry.db` содержит `schema_version`. Формат ключа улицы включает город,
|
||||||
|
поэтому база, собранная старой версией, **не откроется** новым бинарём: в
|
||||||
|
ошибке будет сказано «re-run addrkit seed». Это сделано намеренно — молча
|
||||||
|
резолвить против полупустых полей хуже, чем не стартовать.
|
||||||
|
|
||||||
|
Порядок обновления:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make build
|
||||||
|
./bin/addrkit seed -in registry.ndjson -out registry.db.new -city "Новосибирск"
|
||||||
|
mv registry.db.new registry.db
|
||||||
|
systemctl restart addrkitd
|
||||||
|
```
|
||||||
|
|
||||||
## Принятые решения
|
## Принятые решения
|
||||||
|
|
||||||
- **Детерминизм.** Один вход — один выход, порядок кандидатов стабилен.
|
- **Детерминизм.** Один вход — один выход, порядок кандидатов стабилен:
|
||||||
`resolve` не зависит от сети.
|
индекс сортирует записи при сборке, SQL-запрос тоже имеет `ORDER BY`.
|
||||||
- **Дом сравнивается строго.** Литера/дробь/корпус должны совпасть; иначе
|
- **Дом и строение сравниваются строго.** Литера/дробь/корпус/строение должны
|
||||||
`partial`, а не «похожий» дом.
|
совпасть; иначе `partial` или `ambiguous`, но не «похожий» дом.
|
||||||
- **LLM — опция, выключена по умолчанию.** `adapters/llm` лишь предлагает
|
- **Город — часть ключа.** Одинаковые улицы разных городов не смешиваются.
|
||||||
варианты записи адреса; результат всегда берётся из реестра, поэтому модель
|
- **LLM — опция, выключена по умолчанию.** Модель не может «выдумать» адрес;
|
||||||
не может «выдумать» адрес. Вариант принимается только если он исправляет
|
ошибки rewriter'а логируются, но не меняют детерминированный ответ.
|
||||||
*написание* улицы, не меняя слова, порядковый и дом (`compatibleVariant`).
|
|
||||||
- **Ядро универсально, события по дому — опциональный модуль.** Ядро
|
|
||||||
(`domain/address`, `domain/match`) не знает ни про ЕДДС, ни про отключения;
|
|
||||||
источник событий — plug-in (`outages.Source`), поэтому модуль переиспользуем
|
|
||||||
для любых данных, привязанных к дому.
|
|
||||||
- **Данные.** Сервис владеет своим индексом, построенным из дампа реестра.
|
- **Данные.** Сервис владеет своим индексом, построенным из дампа реестра.
|
||||||
Источник истины и регламент обновления — предмет отдельного согласования.
|
Источник истины и регламент обновления — предмет отдельного согласования.
|
||||||
- **Часовой пояс.** Даты отключений трактуются в `Asia/Novosibirsk` (UTC+7).
|
- **Границы ввода.** Адрес ограничен 512 символами, токен нечёткого
|
||||||
|
сопоставления — 64; это защищает CPU от патологического ввода.
|
||||||
|
- **Сервис не делает событий.** Ни `/v1/outages`, ни portal-конфига, ни
|
||||||
|
часового пояса в ядре.
|
||||||
|
|
||||||
## Тесты
|
## Тесты
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make test # go test -race ./...
|
make test # go test -race ./...
|
||||||
|
make check # vet + race tests + проверка, что ядро не тянет events-адаптеры
|
||||||
```
|
```
|
||||||
|
|
||||||
Проверка парсера на реальном реестре (вне репозитория):
|
Проверка парсера на реальном реестре (вне репозитория):
|
||||||
@@ -140,8 +211,72 @@ ADDRKIT_REGISTRY=/path/to/address_registry.ndjson \
|
|||||||
go test ./internal/seed -run TestRealRegistryParseRate -v
|
go test ./internal/seed -run TestRealRegistryParseRate -v
|
||||||
```
|
```
|
||||||
|
|
||||||
На дампе из 73 936 строк парсер распознаёт **99.98 %** (14 строк — заведомо
|
### E2E на реальном снимке ЕДДС
|
||||||
битые форматы; они не угадываются, а попадают в отчёт `stats.unparsed_sample`).
|
|
||||||
|
Процессные тесты собирают `addrkit` и `addrkitd`, сеют временную БД из
|
||||||
|
настоящего Bulk NDJSON-дампа, поднимают HTTP-сервис и проверяют весь реестр
|
||||||
|
(`selfcheck`), выборку канонических адресов по HTTP, реальные запросы из
|
||||||
|
`edds_address_log`, алиасы города, строгость дома/строения, лимиты и нагрузку
|
||||||
|
(10 000 запросов, p99, RSS, 0 неверных адресов).
|
||||||
|
|
||||||
|
Данные содержат PII и **не копируются в репозиторий**: путь передаётся
|
||||||
|
переменной окружения, временные файлы живут в `t.TempDir()`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ADDRKIT_E2E_REGISTRY=/path/to/address_registry.ndjson \
|
||||||
|
ADDRKIT_E2E_LOG=/path/to/edds_address_log.ndjson \
|
||||||
|
make e2e
|
||||||
|
```
|
||||||
|
|
||||||
|
Опционально: `ADDRKIT_E2E_CITY` (по умолчанию `Новосибирск`),
|
||||||
|
`ADDRKIT_E2E_SAMPLE` (каноническая HTTP-выборка, по умолчанию 2000),
|
||||||
|
`ADDRKIT_E2E_LOAD` (нагрузочные запросы, по умолчанию 10 000).
|
||||||
|
|
||||||
|
Тест проверяет **канонический ввод** — то, что старый конвейер ЕДДС передавал в
|
||||||
|
Elasticsearch после SEE. Сырые реплики с числительными словами
|
||||||
|
(`выставочная шестнадцать`, `сто дробь один`) остаются зоной SEE/LLM: для них
|
||||||
|
тест измеряет долю `found` и не требует 100 %.
|
||||||
|
|
||||||
|
### Shadow: старый ES-путь против нового addrkit
|
||||||
|
|
||||||
|
`addrkit shadow` прогоняет один и тот же запрос через **старый ES-поиск**
|
||||||
|
(ровно тот запрос, который делает робот ЕДДС: `multi_match`, `fuzziness: AUTO`,
|
||||||
|
`size: 1`) и через новый локальный резолвер. В ES уходят **только `_search`**:
|
||||||
|
ничего не индексируется, не обновляется и не удаляется.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export ADDRKIT_ES_URL=https://es:9200
|
||||||
|
export ADDRKIT_ES_USER=elastic
|
||||||
|
export ADDRKIT_ES_PASSWORD=...
|
||||||
|
|
||||||
|
# Сгенерированные реалистичные искажения реестра
|
||||||
|
./bin/addrkit shadow -db registry.db -city "Новосибирск" \
|
||||||
|
-es "$ADDRKIT_ES_URL" -sample 1000 -workers 8 -out shadow-generated.json
|
||||||
|
|
||||||
|
# Реальные реплики из прод-лога (read-only, последние N документов)
|
||||||
|
./bin/addrkit shadow -db registry.db -city "Новосибирск" \
|
||||||
|
-es "$ADDRKIT_ES_URL" -log-es-index edds_address_log -log-limit 3000 \
|
||||||
|
-workers 8 -out shadow-log.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Старый путь получает ту же предобработку, что и в роботе (`-old-preprocess`,
|
||||||
|
по умолчанию включено): нижний регистр и удаление административных слов
|
||||||
|
(`город`, `новосибирск`, районы, `дом`, `номер`). Это делает сравнение честным.
|
||||||
|
|
||||||
|
Ключевые поля отчёта:
|
||||||
|
|
||||||
|
- `new.wrong` — новый путь вернул другой дом; **должно быть 0**, команда
|
||||||
|
завершается с кодом 1;
|
||||||
|
- `compare.old_wrong_new_safe` — старый путь угадал дом, новый отказался
|
||||||
|
(`partial`/`ambiguous`/`not_found`);
|
||||||
|
- `compare.old_wrong_new_correct` — старый ошибся, новый ответил верно;
|
||||||
|
- `compare.old_correct_new_not_exact` — цена строгости: старый угадал верно,
|
||||||
|
новый отказался или потерял информацию; эти случаи уходят в fallback
|
||||||
|
(оператор/LLM), их нужно смотреть глазами.
|
||||||
|
|
||||||
|
`-log` принимает локальный bulk NDJSON, `-log-es-index` — читает лог прямо из
|
||||||
|
ES. Для сырых реплик SEE-преобразование числительных не воспроизводится,
|
||||||
|
поэтому лог-режим показывает нижнюю границу, а не боевую точность пайплайна.
|
||||||
|
|
||||||
## Оценка качества (`addrkit eval`)
|
## Оценка качества (`addrkit eval`)
|
||||||
|
|
||||||
@@ -152,53 +287,26 @@ ADDRKIT_REGISTRY=/path/to/address_registry.ndjson \
|
|||||||
./bin/addrkit eval -db registry.db -sample 800 -seed 1
|
./bin/addrkit eval -db registry.db -sample 800 -seed 1
|
||||||
```
|
```
|
||||||
|
|
||||||
Категории и результат на реальном реестре (по 800 адресам на seed):
|
Категории: регистр, `г. Город`, лишние `дом`/пунктуация, `ё`, сокращения и
|
||||||
|
перестановка типа улицы, синоним «корпус», опечатка в названии, потеря
|
||||||
|
порядкового/литеры/дроби/строения.
|
||||||
|
|
||||||
| Категория | Пример искажения | Точность |
|
Ключевая инварианта: **`found` всегда указывает на запись из индекса**. Если
|
||||||
|---|---|---|
|
информации не хватает (потерян номер дома, литера или строение), результат
|
||||||
| baseline | канонический адрес как есть | 100 % |
|
уходит в `partial`/`ambiguous`/`not_found`, но не превращается в чужой адрес.
|
||||||
| case_lower / case_upper | регистр | 100 % |
|
|
||||||
| city_prefix | `г. Новосибирск, …` | 100 % |
|
|
||||||
| filler_dom | `… дом 5` | 100 % |
|
|
||||||
| punctuation | лишние `,`/пробелы | 100 % |
|
|
||||||
| yo | `ё` вместо `е` | 100 % |
|
|
||||||
| type_abbrev | `ул.`/`пер.`/`пр-т` | 100 % |
|
|
||||||
| type_removed | улица без слова «улица» | 100 % |
|
|
||||||
| type_front | `улица Станционная 1` | 100 % |
|
|
||||||
| corpus_synonym | `корпус` вместо `к` | 100 % |
|
|
||||||
| structure_drop | без `стр.` | 100 % |
|
|
||||||
| **typo** | опечатка в названии улицы | ~78–82 % (остальное — `ambiguous`) |
|
|
||||||
| drop_ordinal / drop_letter / drop_fraction | потеря `2-я` / литеры / дроби | потери (lossy) |
|
|
||||||
|
|
||||||
Ключевой результат: **0 случаев, когда сервис подставил неверный дом**
|
|
||||||
(«safety violations») на десятках тысяч сгенерированных кейсов. Опечатки либо
|
|
||||||
распознаются, либо честно уходят в `ambiguous`/`not_found`, но не превращаются
|
|
||||||
в чужой адрес.
|
|
||||||
|
|
||||||
С подключённым LLM-rewriter'ом (опция, `llm.enabled`) точность на опечатках
|
|
||||||
поднимается до **~90 %** при том же нуле неверных адресов (вариант модели
|
|
||||||
принимается только если исправляет лишь *написание* улицы — см. «Принятые
|
|
||||||
решения»).
|
|
||||||
|
|
||||||
## Самопроверка (`addrkit selfcheck`)
|
## Самопроверка (`addrkit selfcheck`)
|
||||||
|
|
||||||
Прогоняет **весь** реестр на самого себя (не выборку):
|
Прогоняет весь реестр на самого себя и показывает долю точных совпадений и
|
||||||
|
примеры не-`found`. Это самый быстрый способ заметить деградацию парсера
|
||||||
```bash
|
после изменения данных.
|
||||||
./bin/addrkit selfcheck -db registry.db
|
|
||||||
```
|
|
||||||
|
|
||||||
На реестре 72 825 записей: **72 811 (99.98 %) резолвятся точно**, 14 — `partial`
|
|
||||||
(битые форматы дома: `27с1`, `32 к 1/4-1/6`, `112с729`), 0 `not_found`,
|
|
||||||
0 неверных.
|
|
||||||
|
|
||||||
## Локальный dev
|
## Локальный dev
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make build
|
make build
|
||||||
./bin/addrkit seed -in registry.ndjson -out dev/registry.db -version 2026-10-01
|
./bin/addrkit seed -in registry.ndjson -out dev/registry.db -city "Новосибирск"
|
||||||
cp config.example.json dev/config.json # укажите data_path и portal.url
|
cp config.example.json dev/config.json # укажите data_path
|
||||||
./bin/addrkitd -config dev/config.json
|
./bin/addrkitd -config dev/config.json
|
||||||
curl -s localhost:6171/v1/resolve -d '{"address":"2-я Станционная улица 1/1 стр."}'
|
curl -s localhost:6171/v1/resolve -d '{"address":"нск, 2-я Станционная 1/1 стр."}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+114
-15
@@ -38,6 +38,8 @@ func main() {
|
|||||||
os.Exit(runEval(os.Args[2:]))
|
os.Exit(runEval(os.Args[2:]))
|
||||||
case "selfcheck":
|
case "selfcheck":
|
||||||
os.Exit(runSelfcheck(os.Args[2:]))
|
os.Exit(runSelfcheck(os.Args[2:]))
|
||||||
|
case "shadow":
|
||||||
|
os.Exit(runShadow(os.Args[2:]))
|
||||||
case "help", "-h", "--help":
|
case "help", "-h", "--help":
|
||||||
usage()
|
usage()
|
||||||
os.Exit(0)
|
os.Exit(0)
|
||||||
@@ -49,14 +51,15 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func usage() {
|
func usage() {
|
||||||
fmt.Fprint(os.Stderr, `addrkit — address registry toolkit
|
fmt.Fprint(os.Stderr, `addrkit — multi-city address registry toolkit
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
addrkit seed -in registry.ndjson -out registry.db [-version 2026-10-09]
|
addrkit seed -in registry.ndjson -out registry.db [-version 2026-10-09] [-city "Новосибирск"] [-cities cities.json]
|
||||||
addrkit verify -db registry.db
|
addrkit verify -db registry.db
|
||||||
addrkit resolve -db registry.db "2-я Станционная улица 1/1 стр."
|
addrkit resolve -db registry.db [-city "Новосибирск"] [-cities cities.json] "2-я Станционная улица 1/1 стр."
|
||||||
addrkit eval -db registry.db [-sample 500] [-seed 1]
|
addrkit eval -db registry.db [-sample 500] [-seed 1] [-cities cities.json]
|
||||||
addrkit selfcheck -db registry.db
|
addrkit selfcheck -db registry.db
|
||||||
|
addrkit shadow -db registry.db -es https://es:9200 [-sample 300] [-log file.ndjson]
|
||||||
`)
|
`)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -65,6 +68,8 @@ func runSeed(args []string) int {
|
|||||||
in := fs.String("in", "", "input registry dump (Bulk NDJSON or JSONL)")
|
in := fs.String("in", "", "input registry dump (Bulk NDJSON or JSONL)")
|
||||||
out := fs.String("out", "", "output SQLite database")
|
out := fs.String("out", "", "output SQLite database")
|
||||||
version := fs.String("version", "", "data version (defaults to input file mtime)")
|
version := fs.String("version", "", "data version (defaults to input file mtime)")
|
||||||
|
city := fs.String("city", "", "default city for rows without a city field")
|
||||||
|
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
|
||||||
_ = fs.Parse(args)
|
_ = fs.Parse(args)
|
||||||
|
|
||||||
if *in == "" || *out == "" {
|
if *in == "" || *out == "" {
|
||||||
@@ -77,6 +82,12 @@ func runSeed(args []string) int {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
cities, err := loadCities(*citiesPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "seed: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
f, err := os.Open(*in)
|
f, err := os.Open(*in)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(os.Stderr, "seed: open input: %v\n", err)
|
fmt.Fprintf(os.Stderr, "seed: open input: %v\n", err)
|
||||||
@@ -89,7 +100,11 @@ func runSeed(args []string) int {
|
|||||||
fmt.Fprintf(os.Stderr, "seed: read: %v\n", err)
|
fmt.Fprintf(os.Stderr, "seed: read: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
entries, stats := seed.Build(sources)
|
parser := address.NewParser(address.NewDictionary(cities), *city)
|
||||||
|
entries, stats := seed.Build(sources, seed.BuildOptions{
|
||||||
|
Parser: parser,
|
||||||
|
DefaultCity: *city,
|
||||||
|
})
|
||||||
if stats.Parsed == 0 {
|
if stats.Parsed == 0 {
|
||||||
fmt.Fprintln(os.Stderr, "seed: no parseable entries; refusing to write empty index")
|
fmt.Fprintln(os.Stderr, "seed: no parseable entries; refusing to write empty index")
|
||||||
return 1
|
return 1
|
||||||
@@ -109,6 +124,15 @@ func runSeed(args []string) int {
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// loadCities reads an optional city dictionary file. An empty path is valid and
|
||||||
|
// means "no city names beyond those in the address".
|
||||||
|
func loadCities(path string) ([]address.City, error) {
|
||||||
|
if path == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return config.LoadCitiesFile(path)
|
||||||
|
}
|
||||||
|
|
||||||
func runVerify(args []string) int {
|
func runVerify(args []string) int {
|
||||||
fs := flag.NewFlagSet("verify", flag.ExitOnError)
|
fs := flag.NewFlagSet("verify", flag.ExitOnError)
|
||||||
db := fs.String("db", "", "SQLite database")
|
db := fs.String("db", "", "SQLite database")
|
||||||
@@ -123,8 +147,10 @@ func runVerify(args []string) int {
|
|||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
streets := make(map[string]struct{})
|
streets := make(map[string]struct{})
|
||||||
|
cities := make(map[string]int)
|
||||||
for _, e := range entries {
|
for _, e := range entries {
|
||||||
streets[e.StreetKey] = struct{}{}
|
streets[e.StreetKey] = struct{}{}
|
||||||
|
cities[e.City]++
|
||||||
}
|
}
|
||||||
enc := json.NewEncoder(os.Stdout)
|
enc := json.NewEncoder(os.Stdout)
|
||||||
enc.SetIndent("", " ")
|
enc.SetIndent("", " ")
|
||||||
@@ -132,6 +158,7 @@ func runVerify(args []string) int {
|
|||||||
"version": version,
|
"version": version,
|
||||||
"entries": len(entries),
|
"entries": len(entries),
|
||||||
"distinct_streets": len(streets),
|
"distinct_streets": len(streets),
|
||||||
|
"cities": cities,
|
||||||
})
|
})
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
@@ -139,6 +166,8 @@ func runVerify(args []string) int {
|
|||||||
func runResolve(args []string) int {
|
func runResolve(args []string) int {
|
||||||
fs := flag.NewFlagSet("resolve", flag.ExitOnError)
|
fs := flag.NewFlagSet("resolve", flag.ExitOnError)
|
||||||
db := fs.String("db", "", "SQLite database")
|
db := fs.String("db", "", "SQLite database")
|
||||||
|
city := fs.String("city", "", "optional city hint")
|
||||||
|
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
|
||||||
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
|
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
|
||||||
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
|
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
|
||||||
llmToken := fs.String("llm-token", "", "LLM auth token")
|
llmToken := fs.String("llm-token", "", "LLM auth token")
|
||||||
@@ -148,6 +177,11 @@ func runResolve(args []string) int {
|
|||||||
fmt.Fprintln(os.Stderr, `resolve: -db and an address are required`)
|
fmt.Fprintln(os.Stderr, `resolve: -db and an address are required`)
|
||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
|
cities, err := loadCities(*citiesPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "resolve: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
version, entries, err := registrystore.Load(*db)
|
version, entries, err := registrystore.Load(*db)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(os.Stderr, "resolve: %v\n", err)
|
fmt.Fprintf(os.Stderr, "resolve: %v\n", err)
|
||||||
@@ -160,8 +194,13 @@ func runResolve(args []string) int {
|
|||||||
TimeoutSeconds: 5, MaxTokens: 512, DisableThinking: true,
|
TimeoutSeconds: 5, MaxTokens: 512, DisableThinking: true,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
resolver := match.NewResolver(match.NewIndex(version, entries, match.DefaultOptions()), rewriter)
|
parser := address.NewParser(address.NewDictionary(cities), "")
|
||||||
res := resolver.Resolve(context.Background(), rest[0])
|
resolver := match.NewResolverWithOptions(
|
||||||
|
match.NewIndex(version, entries, match.DefaultOptions()),
|
||||||
|
rewriter,
|
||||||
|
match.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
res := resolver.ResolveInCity(context.Background(), rest[0], *city)
|
||||||
enc := json.NewEncoder(os.Stdout)
|
enc := json.NewEncoder(os.Stdout)
|
||||||
enc.SetIndent("", " ")
|
enc.SetIndent("", " ")
|
||||||
_ = enc.Encode(res)
|
_ = enc.Encode(res)
|
||||||
@@ -177,6 +216,7 @@ func runEval(args []string) int {
|
|||||||
db := fs.String("db", "", "SQLite database")
|
db := fs.String("db", "", "SQLite database")
|
||||||
sample := fs.Int("sample", 500, "number of registry entries to sample")
|
sample := fs.Int("sample", 500, "number of registry entries to sample")
|
||||||
seedVal := fs.Int64("seed", 1, "random seed")
|
seedVal := fs.Int64("seed", 1, "random seed")
|
||||||
|
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
|
||||||
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
|
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
|
||||||
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
|
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
|
||||||
llmToken := fs.String("llm-token", "", "LLM auth token")
|
llmToken := fs.String("llm-token", "", "LLM auth token")
|
||||||
@@ -186,11 +226,19 @@ func runEval(args []string) int {
|
|||||||
fmt.Fprintln(os.Stderr, "eval: -db is required")
|
fmt.Fprintln(os.Stderr, "eval: -db is required")
|
||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
|
cities, err := loadCities(*citiesPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "eval: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
_, entries, err := registrystore.Load(*db)
|
_, entries, err := registrystore.Load(*db)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(os.Stderr, "eval: %v\n", err)
|
fmt.Fprintf(os.Stderr, "eval: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
cities = mergeCities(cities, entries)
|
||||||
|
defaultCity := soleCity(entries)
|
||||||
|
parser := address.NewParser(address.NewDictionary(cities), defaultCity)
|
||||||
ix := match.NewIndex("eval", entries, match.DefaultOptions())
|
ix := match.NewIndex("eval", entries, match.DefaultOptions())
|
||||||
|
|
||||||
var rewriter match.Rewriter
|
var rewriter match.Rewriter
|
||||||
@@ -208,7 +256,7 @@ func runEval(args []string) int {
|
|||||||
rewriter = counter
|
rewriter = counter
|
||||||
fmt.Fprintf(os.Stderr, "llm rewriter enabled: %s (%s)\n", *llmURL, *llmModel)
|
fmt.Fprintf(os.Stderr, "llm rewriter enabled: %s (%s)\n", *llmURL, *llmModel)
|
||||||
}
|
}
|
||||||
resolver := match.NewResolver(ix, rewriter)
|
resolver := match.NewResolverWithOptions(ix, rewriter, match.ResolverOptions{Parser: parser})
|
||||||
|
|
||||||
rng := rand.New(rand.NewSource(*seedVal))
|
rng := rand.New(rand.NewSource(*seedVal))
|
||||||
cases := eval.Generate(entries, *sample, rng)
|
cases := eval.Generate(entries, *sample, rng)
|
||||||
@@ -286,14 +334,22 @@ func runSelfcheck(args []string) int {
|
|||||||
fmt.Fprintf(os.Stderr, "selfcheck: %v\n", err)
|
fmt.Fprintf(os.Stderr, "selfcheck: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
resolver := match.NewResolver(match.NewIndex("selfcheck", entries, match.DefaultOptions()), nil)
|
parser := address.NewParser(
|
||||||
|
address.NewDictionary(mergeCities(nil, entries)),
|
||||||
|
soleCity(entries),
|
||||||
|
)
|
||||||
|
resolver := match.NewResolverWithOptions(
|
||||||
|
match.NewIndex("selfcheck", entries, match.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
match.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
exact := 0
|
exact := 0
|
||||||
byStatus := map[string]int{}
|
byStatus := map[string]int{}
|
||||||
var failures []string
|
var failures []string
|
||||||
for _, e := range entries {
|
for _, e := range entries {
|
||||||
res := resolver.Resolve(context.Background(), e.Address)
|
res := resolver.ResolveInCity(context.Background(), e.Address, e.City)
|
||||||
if res.Status == address.StatusFound && sameEntry(res.Address, e) {
|
if res.Status == address.StatusFound && res.City == e.City && sameEntry(res.Address, e, parser) {
|
||||||
exact++
|
exact++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -316,10 +372,53 @@ func runSelfcheck(args []string) int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// sameEntry reports whether a resolved address is the same street and house
|
// sameEntry reports whether a resolved address is the same street and house
|
||||||
// unit as the registry entry it came from.
|
// unit as the registry entry it came from. City is checked separately against
|
||||||
func sameEntry(resolved string, e address.Entry) bool {
|
// Result.City by the caller; forcing it here only keeps the parse city-aware.
|
||||||
p := address.Parse(resolved)
|
func sameEntry(resolved string, e address.Entry, parser *address.Parser) bool {
|
||||||
|
p := parser.ParseInCity(resolved, e.City)
|
||||||
return p.StreetKey == e.StreetKey &&
|
return p.StreetKey == e.StreetKey &&
|
||||||
p.House == e.House && p.HouseFrac == e.HouseFrac && p.HouseLetter == e.HouseLetter &&
|
p.House == e.House && p.HouseFrac == e.HouseFrac && p.HouseLetter == e.HouseLetter &&
|
||||||
p.Building == e.Building && p.BuildingFrac == e.BuildingFrac && p.BuildingLetter == e.BuildingLetter
|
p.Building == e.Building && p.BuildingFrac == e.BuildingFrac && p.BuildingLetter == e.BuildingLetter &&
|
||||||
|
p.Structure == e.Structure
|
||||||
|
}
|
||||||
|
|
||||||
|
// mergeCities augments an explicit city dictionary with the canonical names
|
||||||
|
// found in the index. The names are enough for parsing addresses that include
|
||||||
|
// the city; aliases still come from the dictionary file.
|
||||||
|
func mergeCities(base []address.City, entries []address.Entry) []address.City {
|
||||||
|
seen := make(map[string]struct{}, len(base))
|
||||||
|
for _, c := range base {
|
||||||
|
seen[address.NormalizeText(c.Name)] = struct{}{}
|
||||||
|
}
|
||||||
|
out := append([]address.City(nil), base...)
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.City == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := address.NormalizeText(e.City)
|
||||||
|
if _, ok := seen[key]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
out = append(out, address.City{Name: e.City})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// soleCity returns the only city in the index, or "" when the index is
|
||||||
|
// city-less or multi-city.
|
||||||
|
func soleCity(entries []address.Entry) string {
|
||||||
|
names := make(map[string]string)
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.City != "" {
|
||||||
|
names[e.CityKey] = e.City
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(names) != 1 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
for _, name := range names {
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
return ""
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,685 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
// Shadow mode: run the old EДДС path (Elasticsearch fuzzy search) and addrkit
|
||||||
|
// on the same queries, without changing any production answer.
|
||||||
|
//
|
||||||
|
// The Elasticsearch side is strictly read-only: this file only issues
|
||||||
|
// `_search` requests. Nothing is indexed, updated or deleted.
|
||||||
|
//
|
||||||
|
// Two datasets are supported:
|
||||||
|
// - generated: realistic perturbations of real registry rows (eval harness);
|
||||||
|
// - log: real user utterances from the EДДС address log (file or ES index).
|
||||||
|
//
|
||||||
|
// The report compares the old top-1 guess with the new deterministic verdict.
|
||||||
|
// The key number is "old_wrong_new_safe": cases where the old path returned a
|
||||||
|
// different house and the new one refused to guess.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"encoding/json"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math/rand"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/eval"
|
||||||
|
)
|
||||||
|
|
||||||
|
type shadowOptions struct {
|
||||||
|
db string
|
||||||
|
city string
|
||||||
|
citiesPath string
|
||||||
|
esURL string
|
||||||
|
esUser string
|
||||||
|
esPass string
|
||||||
|
esIndex string
|
||||||
|
insecure bool
|
||||||
|
timeout time.Duration
|
||||||
|
workers int
|
||||||
|
sample int
|
||||||
|
seed int64
|
||||||
|
logPath string
|
||||||
|
logField string
|
||||||
|
logESIndex string
|
||||||
|
logLimit int
|
||||||
|
maxExamples int
|
||||||
|
oldPreproc bool
|
||||||
|
out string
|
||||||
|
}
|
||||||
|
|
||||||
|
func runShadow(args []string) int {
|
||||||
|
fs := flag.NewFlagSet("shadow", flag.ExitOnError)
|
||||||
|
db := fs.String("db", "", "local registry.db (new path)")
|
||||||
|
city := fs.String("city", "", "default city for the new path")
|
||||||
|
citiesPath := fs.String("cities", "", "optional JSON city dictionary")
|
||||||
|
esURL := fs.String("es", envOr("ADDRKIT_ES_URL", ""), "Elasticsearch base URL (old path, read-only)")
|
||||||
|
esUser := fs.String("es-user", envOr("ADDRKIT_ES_USER", ""), "Elasticsearch user")
|
||||||
|
esPass := fs.String("es-pass", envOr("ADDRKIT_ES_PASSWORD", ""), "Elasticsearch password")
|
||||||
|
esIndex := fs.String("es-index", "address_registry", "Elasticsearch address index")
|
||||||
|
insecure := fs.Bool("insecure", false, "skip TLS verification for Elasticsearch")
|
||||||
|
timeout := fs.Duration("timeout", 10*time.Second, "per-request timeout")
|
||||||
|
workers := fs.Int("workers", 8, "parallel queries")
|
||||||
|
sample := fs.Int("sample", 300, "registry entries to perturb (each yields several queries)")
|
||||||
|
seed := fs.Int64("seed", 1, "random seed for generated queries")
|
||||||
|
logPath := fs.String("log", "", "address log bulk NDJSON (real utterances)")
|
||||||
|
logField := fs.String("log-field", "address", "log field to replay: address|request")
|
||||||
|
logESIndex := fs.String("log-es-index", "", "fetch real utterances from this ES index instead of a file")
|
||||||
|
logLimit := fs.Int("log-limit", 500, "documents to fetch from -log-es-index")
|
||||||
|
maxExamples := fs.Int("max-examples", 20, "examples per category in the report")
|
||||||
|
oldPreproc := fs.Bool("old-preprocess", true, "apply the robot's administrative-word stripping to the old ES query")
|
||||||
|
out := fs.String("out", "", "write the JSON report to this file (default stdout)")
|
||||||
|
_ = fs.Parse(args)
|
||||||
|
|
||||||
|
if *db == "" || *esURL == "" {
|
||||||
|
fmt.Fprintln(os.Stderr, "shadow: -db and -es (or ADDRKIT_ES_URL) are required")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
if *workers < 1 {
|
||||||
|
*workers = 1
|
||||||
|
}
|
||||||
|
cities, err := loadCities(*citiesPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shadow: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
_, entries, err := registrystore.Load(*db)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shadow: load registry: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
parser := address.NewParser(address.NewDictionary(cities), *city)
|
||||||
|
resolver := match.NewResolverWithOptions(
|
||||||
|
match.NewIndex("shadow", entries, match.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
match.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
|
opts := shadowOptions{
|
||||||
|
db: *db, city: *city, citiesPath: *citiesPath,
|
||||||
|
esURL: *esURL, esUser: *esUser, esPass: *esPass, esIndex: *esIndex,
|
||||||
|
insecure: *insecure, timeout: *timeout, workers: *workers,
|
||||||
|
sample: *sample, seed: *seed, logPath: *logPath, logField: *logField,
|
||||||
|
logESIndex: *logESIndex, logLimit: *logLimit, maxExamples: *maxExamples,
|
||||||
|
oldPreproc: *oldPreproc, out: *out,
|
||||||
|
}
|
||||||
|
|
||||||
|
cases, mode, note, err := buildShadowCases(opts, entries, parser)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shadow: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
if len(cases) == 0 {
|
||||||
|
fmt.Fprintln(os.Stderr, "shadow: no queries to replay")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
client := newESClient(opts)
|
||||||
|
report := runShadowQueries(opts, cases, mode, note, parser, resolver, client)
|
||||||
|
|
||||||
|
data, err := json.MarshalIndent(report, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shadow: encode report: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
if opts.out != "" {
|
||||||
|
if err := os.WriteFile(opts.out, data, 0o600); err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shadow: write report: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Println(string(data))
|
||||||
|
if report.New.Wrong > 0 {
|
||||||
|
return 1 // a wrong new answer is a blocker, make it visible to CI
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func envOr(name, fallback string) string {
|
||||||
|
if v := strings.TrimSpace(os.Getenv(name)); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return fallback
|
||||||
|
}
|
||||||
|
|
||||||
|
type shadowCase struct {
|
||||||
|
Query string
|
||||||
|
Expect string
|
||||||
|
ExpectCity string
|
||||||
|
RequireExact bool
|
||||||
|
Category string
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildShadowCases(opts shadowOptions, entries []address.Entry, parser *address.Parser) ([]shadowCase, string, string, error) {
|
||||||
|
switch {
|
||||||
|
case opts.logESIndex != "":
|
||||||
|
rows, err := fetchLogCases(opts)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", "", fmt.Errorf("fetch address log from ES: %w", err)
|
||||||
|
}
|
||||||
|
return logCases(rows, opts.logField), "log", "raw utterances are compared without the SEE conversion step", nil
|
||||||
|
case opts.logPath != "":
|
||||||
|
rows, err := readLogCases(opts.logPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", "", fmt.Errorf("read address log: %w", err)
|
||||||
|
}
|
||||||
|
return logCases(rows, opts.logField), "log", "raw utterances are compared without the SEE conversion step", nil
|
||||||
|
default:
|
||||||
|
rng := rand.New(rand.NewSource(opts.seed))
|
||||||
|
src := eval.Generate(entries, opts.sample, rng)
|
||||||
|
out := make([]shadowCase, 0, len(src))
|
||||||
|
for _, c := range src {
|
||||||
|
out = append(out, shadowCase{
|
||||||
|
Query: c.Query, Expect: c.Expect, ExpectCity: c.ExpectCity,
|
||||||
|
RequireExact: c.RequireExact, Category: c.Category,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, "generated", "real registry rows perturbed by the eval harness", nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type logRow struct {
|
||||||
|
Utterance string
|
||||||
|
Request string
|
||||||
|
}
|
||||||
|
|
||||||
|
func logCases(rows []logRow, field string) []shadowCase {
|
||||||
|
out := make([]shadowCase, 0, len(rows))
|
||||||
|
seen := map[string]struct{}{}
|
||||||
|
for _, r := range rows {
|
||||||
|
q := r.Utterance
|
||||||
|
if field == "request" {
|
||||||
|
q = r.Request
|
||||||
|
}
|
||||||
|
q = strings.TrimSpace(q)
|
||||||
|
if q == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := seen[q]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[q] = struct{}{}
|
||||||
|
out = append(out, shadowCase{Query: q, Category: "log"})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func readLogCases(path string) ([]logRow, error) {
|
||||||
|
f, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
dec := json.NewDecoder(f)
|
||||||
|
var rows []logRow
|
||||||
|
for {
|
||||||
|
var raw json.RawMessage
|
||||||
|
if err := dec.Decode(&raw); err != nil {
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var probe map[string]json.RawMessage
|
||||||
|
if json.Unmarshal(raw, &probe) != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := probe["index"]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var rec struct {
|
||||||
|
Address string `json:"address"`
|
||||||
|
Request string `json:"request"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal(raw, &rec)
|
||||||
|
var req struct {
|
||||||
|
Address string `json:"address"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal([]byte(rec.Request), &req)
|
||||||
|
rows = append(rows, logRow{Utterance: rec.Address, Request: req.Address})
|
||||||
|
}
|
||||||
|
return rows, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func fetchLogCases(opts shadowOptions) ([]logRow, error) {
|
||||||
|
body := map[string]any{
|
||||||
|
"size": opts.logLimit,
|
||||||
|
"sort": []any{map[string]any{"start_date": map[string]string{"order": "desc"}}},
|
||||||
|
"_source": []string{"address", "request"},
|
||||||
|
}
|
||||||
|
data, err := esRequest(context.Background(), newESClient(opts), opts.esURL, opts.logESIndex, "_search", body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var resp struct {
|
||||||
|
Hits struct {
|
||||||
|
Hits []struct {
|
||||||
|
Source struct {
|
||||||
|
Address string `json:"address"`
|
||||||
|
Request string `json:"request"`
|
||||||
|
} `json:"_source"`
|
||||||
|
} `json:"hits"`
|
||||||
|
} `json:"hits"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &resp); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
rows := make([]logRow, 0, len(resp.Hits.Hits))
|
||||||
|
for _, h := range resp.Hits.Hits {
|
||||||
|
var req struct {
|
||||||
|
Address string `json:"address"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal([]byte(h.Source.Request), &req)
|
||||||
|
rows = append(rows, logRow{Utterance: h.Source.Address, Request: req.Address})
|
||||||
|
}
|
||||||
|
return rows, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type esClient struct {
|
||||||
|
http *http.Client
|
||||||
|
user string
|
||||||
|
pass string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newESClient(opts shadowOptions) *esClient {
|
||||||
|
tr := &http.Transport{}
|
||||||
|
if opts.insecure {
|
||||||
|
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} // #nosec G402 -- self-signed on-prem ES
|
||||||
|
}
|
||||||
|
return &esClient{
|
||||||
|
http: &http.Client{Timeout: opts.timeout, Transport: tr},
|
||||||
|
user: opts.esUser,
|
||||||
|
pass: opts.esPass,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// esRequest performs a read-only search. It only builds `_search` URLs.
|
||||||
|
func esRequest(ctx context.Context, c *esClient, base, index, path string, body any) ([]byte, error) {
|
||||||
|
raw, err := json.Marshal(body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
url := strings.TrimRight(base, "/") + "/" + index + "/" + path
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, bytes.NewReader(raw))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
if c.user != "" {
|
||||||
|
req.SetBasicAuth(c.user, c.pass)
|
||||||
|
}
|
||||||
|
resp, err := c.http.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
data, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return nil, fmt.Errorf("ES %s: HTTP %d: %s", path, resp.StatusCode, truncate(string(data), 200))
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncate(s string, n int) string {
|
||||||
|
if len(s) <= n {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
return s[:n]
|
||||||
|
}
|
||||||
|
|
||||||
|
// oldDeleteWords mirrors the administrative words the EДДС robot removes
|
||||||
|
// before calling Elasticsearch (robot script: DELETE_WORDS). Matching is
|
||||||
|
// token-based here, so "новосибирская" is not mangled by removing the
|
||||||
|
// substring "новосибирск".
|
||||||
|
var oldDeleteWords = map[string]struct{}{
|
||||||
|
"дзержинский": {}, "железнодорожный": {}, "заельцовский": {},
|
||||||
|
"калининский": {}, "кировский": {}, "ленинский": {}, "октябрьский": {},
|
||||||
|
"первомайский": {}, "советский": {}, "центральный": {},
|
||||||
|
"район": {}, "город": {}, "номер": {}, "новосибирск": {}, "дом": {},
|
||||||
|
"г": {}, "обл": {}, "область": {},
|
||||||
|
}
|
||||||
|
|
||||||
|
func preprocessOld(q string) string {
|
||||||
|
fields := strings.Fields(strings.ToLower(q))
|
||||||
|
out := fields[:0]
|
||||||
|
for _, tok := range fields {
|
||||||
|
tok = strings.Trim(tok, ".,;:()")
|
||||||
|
if tok == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := oldDeleteWords[tok]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, tok)
|
||||||
|
}
|
||||||
|
return strings.Join(out, " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// oldQuery mirrors the EДДС robot's normalize_address query exactly.
|
||||||
|
func oldQuery(q string) map[string]any {
|
||||||
|
return map[string]any{
|
||||||
|
"query": map[string]any{
|
||||||
|
"bool": map[string]any{
|
||||||
|
"must": []any{map[string]any{
|
||||||
|
"multi_match": map[string]any{
|
||||||
|
"query": q, "fields": []string{"address"},
|
||||||
|
"fuzziness": "AUTO", "operator": "and", "prefix_length": 2,
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
"should": []any{map[string]any{
|
||||||
|
"match_phrase": map[string]any{
|
||||||
|
"address": map[string]any{"query": q, "slop": 3, "boost": 2},
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"size": 1,
|
||||||
|
"_source": []string{"address", "district_name"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type esHit struct {
|
||||||
|
Address string `json:"address"`
|
||||||
|
District string `json:"district_name"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func oldSearch(ctx context.Context, c *esClient, opts shadowOptions, q string) (esHit, error) {
|
||||||
|
data, err := esRequest(ctx, c, opts.esURL, opts.esIndex, "_search", oldQuery(q))
|
||||||
|
if err != nil {
|
||||||
|
return esHit{}, err
|
||||||
|
}
|
||||||
|
var resp struct {
|
||||||
|
Hits struct {
|
||||||
|
Hits []struct {
|
||||||
|
Source esHit `json:"_source"`
|
||||||
|
} `json:"hits"`
|
||||||
|
} `json:"hits"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &resp); err != nil {
|
||||||
|
return esHit{}, err
|
||||||
|
}
|
||||||
|
if len(resp.Hits.Hits) == 0 {
|
||||||
|
return esHit{}, nil
|
||||||
|
}
|
||||||
|
return resp.Hits.Hits[0].Source, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type shadowReport struct {
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
Note string `json:"note"`
|
||||||
|
Queries int `json:"queries"`
|
||||||
|
Workers int `json:"workers"`
|
||||||
|
ESIndex string `json:"es_index"`
|
||||||
|
OldPreprocess bool `json:"old_preprocess"`
|
||||||
|
Old sideReport `json:"old"`
|
||||||
|
New sideReport `json:"new"`
|
||||||
|
Compare compareReport `json:"compare"`
|
||||||
|
Examples []shadowExample `json:"examples,omitempty"`
|
||||||
|
Errors int `json:"errors"`
|
||||||
|
ElapsedMS int64 `json:"elapsed_ms"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type sideReport struct {
|
||||||
|
Found int `json:"found"`
|
||||||
|
Exact int `json:"exact"`
|
||||||
|
Wrong int `json:"wrong"`
|
||||||
|
Lossy int `json:"lossy"`
|
||||||
|
Partial int `json:"partial,omitempty"`
|
||||||
|
Ambiguous int `json:"ambiguous,omitempty"`
|
||||||
|
NotFound int `json:"not_found,omitempty"`
|
||||||
|
Error int `json:"error,omitempty"`
|
||||||
|
P50MS float64 `json:"p50_ms"`
|
||||||
|
P95MS float64 `json:"p95_ms"`
|
||||||
|
P99MS float64 `json:"p99_ms"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type compareReport struct {
|
||||||
|
BothCorrect int `json:"both_correct"`
|
||||||
|
OldCorrectNewNotExact int `json:"old_correct_new_not_exact"`
|
||||||
|
OldWrongNewSafe int `json:"old_wrong_new_safe"`
|
||||||
|
OldWrongNewCorrect int `json:"old_wrong_new_correct"`
|
||||||
|
OldMissNewCorrect int `json:"old_miss_new_correct,omitempty"`
|
||||||
|
OldWrongNewWrong int `json:"old_wrong_new_wrong"`
|
||||||
|
OldLossyNewSafe int `json:"old_lossy_new_safe,omitempty"`
|
||||||
|
OldLossyNewCorrect int `json:"old_lossy_new_correct,omitempty"`
|
||||||
|
BothNotExact int `json:"both_not_exact"`
|
||||||
|
Same int `json:"same,omitempty"`
|
||||||
|
OldOnly int `json:"old_only,omitempty"`
|
||||||
|
NewOnly int `json:"new_only,omitempty"`
|
||||||
|
Differ int `json:"differ,omitempty"`
|
||||||
|
BothMiss int `json:"both_miss,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type shadowExample struct {
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
Query string `json:"query"`
|
||||||
|
Expected string `json:"expected,omitempty"`
|
||||||
|
Old string `json:"old,omitempty"`
|
||||||
|
New string `json:"new,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type shadowOutcome struct {
|
||||||
|
hit esHit
|
||||||
|
parsed address.ParsedAddress
|
||||||
|
has bool
|
||||||
|
err error
|
||||||
|
latency time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
func runShadowQueries(opts shadowOptions, cases []shadowCase, mode, note string, parser *address.Parser, resolver *match.Resolver, client *esClient) shadowReport {
|
||||||
|
ctx := context.Background()
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
type job struct{ c shadowCase }
|
||||||
|
type result struct {
|
||||||
|
c shadowCase
|
||||||
|
old shadowOutcome
|
||||||
|
newRes match.Result
|
||||||
|
newLat time.Duration
|
||||||
|
newErr error
|
||||||
|
}
|
||||||
|
|
||||||
|
jobs := make(chan job)
|
||||||
|
results := make(chan result)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < opts.workers; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for j := range jobs {
|
||||||
|
oldStart := time.Now()
|
||||||
|
oldQ := j.c.Query
|
||||||
|
if opts.oldPreproc {
|
||||||
|
oldQ = preprocessOld(oldQ)
|
||||||
|
}
|
||||||
|
hit, err := oldSearch(ctx, client, opts, oldQ)
|
||||||
|
old := shadowOutcome{hit: hit, err: err, latency: time.Since(oldStart)}
|
||||||
|
if err == nil {
|
||||||
|
old.parsed = parser.ParseInCity(hit.Address, j.c.ExpectCity)
|
||||||
|
old.has = hit.Address != ""
|
||||||
|
if j.c.ExpectCity == "" && opts.city != "" {
|
||||||
|
old.parsed = parser.ParseInCity(hit.Address, opts.city)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
newStart := time.Now()
|
||||||
|
res := resolver.Resolve(ctx, j.c.Query)
|
||||||
|
results <- result{c: j.c, old: old, newRes: res, newLat: time.Since(newStart)}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
for _, c := range cases {
|
||||||
|
jobs <- job{c: c}
|
||||||
|
}
|
||||||
|
close(jobs)
|
||||||
|
wg.Wait()
|
||||||
|
close(results)
|
||||||
|
}()
|
||||||
|
|
||||||
|
var oldLat, newLat []time.Duration
|
||||||
|
rep := shadowReport{
|
||||||
|
Mode: mode, Note: note, Workers: opts.workers, ESIndex: opts.esIndex,
|
||||||
|
OldPreprocess: opts.oldPreproc, Queries: len(cases), Examples: []shadowExample{},
|
||||||
|
}
|
||||||
|
for r := range results {
|
||||||
|
oldLat = append(oldLat, r.old.latency)
|
||||||
|
newLat = append(newLat, r.newLat)
|
||||||
|
if r.old.err != nil {
|
||||||
|
rep.Old.Error++
|
||||||
|
rep.Errors++
|
||||||
|
}
|
||||||
|
if r.newErr != nil {
|
||||||
|
rep.New.Error++
|
||||||
|
rep.Errors++
|
||||||
|
}
|
||||||
|
|
||||||
|
if mode == "generated" {
|
||||||
|
classifyGenerated(&rep, opts, parser, r.c, r.old, r.newRes)
|
||||||
|
} else {
|
||||||
|
classifyLog(&rep, opts, parser, r.c, r.old, r.newRes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rep.Old.P50MS, rep.Old.P95MS, rep.Old.P99MS = percentiles(oldLat)
|
||||||
|
rep.New.P50MS, rep.New.P95MS, rep.New.P99MS = percentiles(newLat)
|
||||||
|
rep.ElapsedMS = time.Since(start).Milliseconds()
|
||||||
|
return rep
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyGenerated(rep *shadowReport, opts shadowOptions, parser *address.Parser, c shadowCase, old shadowOutcome, newRes match.Result) {
|
||||||
|
want := parser.ParseInCity(c.Expect, c.ExpectCity)
|
||||||
|
oldExact := old.has && sameAddressUnit(old.parsed, want)
|
||||||
|
oldWrong := old.has && !oldExact && c.RequireExact
|
||||||
|
oldLossy := old.has && !oldExact && !c.RequireExact
|
||||||
|
newExact := newRes.Status == address.StatusFound && sameAddressUnit(parser.ParseInCity(newRes.Address, c.ExpectCity), want)
|
||||||
|
newWrong := newRes.Status == address.StatusFound && !newExact && c.RequireExact
|
||||||
|
newSafe := newRes.Status == address.StatusPartial ||
|
||||||
|
newRes.Status == address.StatusAmbiguous ||
|
||||||
|
newRes.Status == address.StatusNotFound
|
||||||
|
|
||||||
|
if old.has {
|
||||||
|
rep.Old.Found++
|
||||||
|
}
|
||||||
|
if oldExact {
|
||||||
|
rep.Old.Exact++
|
||||||
|
}
|
||||||
|
if oldWrong {
|
||||||
|
rep.Old.Wrong++
|
||||||
|
}
|
||||||
|
if oldLossy {
|
||||||
|
rep.Old.Lossy++
|
||||||
|
}
|
||||||
|
if newRes.Status == address.StatusFound {
|
||||||
|
rep.New.Found++
|
||||||
|
}
|
||||||
|
if newExact {
|
||||||
|
rep.New.Exact++
|
||||||
|
}
|
||||||
|
if newWrong {
|
||||||
|
rep.New.Wrong++
|
||||||
|
}
|
||||||
|
if newRes.Status == address.StatusFound && !newExact && !c.RequireExact {
|
||||||
|
rep.New.Lossy++
|
||||||
|
}
|
||||||
|
switch newRes.Status {
|
||||||
|
case address.StatusPartial:
|
||||||
|
rep.New.Partial++
|
||||||
|
case address.StatusAmbiguous:
|
||||||
|
rep.New.Ambiguous++
|
||||||
|
case address.StatusNotFound:
|
||||||
|
rep.New.NotFound++
|
||||||
|
}
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case oldExact && newExact:
|
||||||
|
rep.Compare.BothCorrect++
|
||||||
|
case oldExact:
|
||||||
|
rep.Compare.OldCorrectNewNotExact++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_correct_new_not_exact", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case newExact && oldWrong:
|
||||||
|
rep.Compare.OldWrongNewCorrect++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case newExact && oldLossy:
|
||||||
|
rep.Compare.OldLossyNewCorrect++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_lossy_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case newExact:
|
||||||
|
rep.Compare.OldMissNewCorrect++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_miss_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case oldWrong && newWrong:
|
||||||
|
rep.Compare.OldWrongNewWrong++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_wrong", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case oldWrong && newSafe:
|
||||||
|
rep.Compare.OldWrongNewSafe++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_safe", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case oldLossy && newSafe:
|
||||||
|
rep.Compare.OldLossyNewSafe++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_lossy_new_safe", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
default:
|
||||||
|
rep.Compare.BothNotExact++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyLog(rep *shadowReport, opts shadowOptions, parser *address.Parser, c shadowCase, old shadowOutcome, newRes match.Result) {
|
||||||
|
if old.has {
|
||||||
|
rep.Old.Found++
|
||||||
|
}
|
||||||
|
if newRes.Status == address.StatusFound {
|
||||||
|
rep.New.Found++
|
||||||
|
}
|
||||||
|
oldOK := old.has && old.parsed.HasHouse
|
||||||
|
newOK := newRes.Status == address.StatusFound
|
||||||
|
switch {
|
||||||
|
case oldOK && newOK && sameAddressUnit(old.parsed, parser.ParseInCity(newRes.Address, opts.city)):
|
||||||
|
rep.Compare.Same++
|
||||||
|
case oldOK && !newOK:
|
||||||
|
rep.Compare.OldOnly++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "old_only", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case !oldOK && newOK:
|
||||||
|
rep.Compare.NewOnly++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "new_only", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
case oldOK && newOK:
|
||||||
|
rep.Compare.Differ++
|
||||||
|
addExample(rep, opts, shadowExample{Kind: "differ", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
|
||||||
|
default:
|
||||||
|
rep.Compare.BothMiss++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func addExample(rep *shadowReport, opts shadowOptions, ex shadowExample) {
|
||||||
|
if len(rep.Examples) >= opts.maxExamples {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
rep.Examples = append(rep.Examples, ex)
|
||||||
|
}
|
||||||
|
|
||||||
|
func sameAddressUnit(a, b address.ParsedAddress) bool {
|
||||||
|
return a.CityKey == b.CityKey &&
|
||||||
|
a.StreetKey == b.StreetKey &&
|
||||||
|
a.House == b.House && a.HouseFrac == b.HouseFrac && a.HouseLetter == b.HouseLetter &&
|
||||||
|
a.Building == b.Building && a.BuildingFrac == b.BuildingFrac && a.BuildingLetter == b.BuildingLetter &&
|
||||||
|
a.Structure == b.Structure
|
||||||
|
}
|
||||||
|
|
||||||
|
func percentiles(values []time.Duration) (p50, p95, p99 float64) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return 0, 0, 0
|
||||||
|
}
|
||||||
|
sort.Slice(values, func(i, j int) bool { return values[i] < values[j] })
|
||||||
|
at := func(q float64) float64 {
|
||||||
|
i := int(float64(len(values)-1) * q)
|
||||||
|
return float64(values[i].Microseconds()) / 1000
|
||||||
|
}
|
||||||
|
return at(0.50), at(0.95), at(0.99)
|
||||||
|
}
|
||||||
+32
-32
@@ -1,9 +1,9 @@
|
|||||||
// Command addrkitd runs the address resolution service.
|
// Command addrkitd runs the address resolution service.
|
||||||
//
|
//
|
||||||
// It loads the canonical registry into memory, wires the optional adapters
|
// It loads the canonical registry into memory, wires the optional LLM rewriter
|
||||||
// (ArcGIS portal, LLM rewriter) and serves the versioned JSON API. The process
|
// and serves the versioned JSON API. The process fails closed: if the registry
|
||||||
// fails closed: if the registry cannot be loaded it exits rather than serving
|
// cannot be loaded or is empty it exits rather than serving misleading
|
||||||
// an empty index.
|
// "not found" answers.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -17,13 +17,12 @@ import (
|
|||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/adapters/arcgis"
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/adapters/httpapi"
|
"git.totmin.ru/en2zmax/addrkit/internal/adapters/httpapi"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/adapters/llm"
|
"git.totmin.ru/en2zmax/addrkit/internal/adapters/llm"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore"
|
"git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/config"
|
"git.totmin.ru/en2zmax/addrkit/internal/config"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
@@ -40,10 +39,14 @@ func main() {
|
|||||||
|
|
||||||
version, entries, err := registrystore.Load(cfg.DataPath)
|
version, entries, err := registrystore.Load(cfg.DataPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// Fail closed: no registry, no service.
|
// Fail closed: no usable registry, no service.
|
||||||
logger.Error("load registry", "error", err, "path", cfg.DataPath)
|
logger.Error("load registry", "error", err, "path", cfg.DataPath)
|
||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
|
if len(entries) == 0 {
|
||||||
|
logger.Error("registry is empty; refusing to serve", "path", cfg.DataPath)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
logger.Info("registry loaded", "version", version, "entries", len(entries))
|
logger.Info("registry loaded", "version", version, "entries", len(entries))
|
||||||
|
|
||||||
index := match.NewIndex(version, entries, match.Options{
|
index := match.NewIndex(version, entries, match.Options{
|
||||||
@@ -52,40 +55,47 @@ func main() {
|
|||||||
MaxCandidates: cfg.Resolve.MaxCandidates,
|
MaxCandidates: cfg.Resolve.MaxCandidates,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
parser := address.NewParser(address.NewDictionary(cfg.Cities), cfg.DefaultCity)
|
||||||
|
|
||||||
var rewriter match.Rewriter
|
var rewriter match.Rewriter
|
||||||
if cfg.LLM.Enabled {
|
if cfg.LLM.Enabled {
|
||||||
rewriter = llm.New(cfg.LLM)
|
rewriter = llm.New(cfg.LLM)
|
||||||
logger.Info("llm rewriter enabled", "url", cfg.LLM.URL, "model", cfg.LLM.Model)
|
logger.Info("llm rewriter enabled", "url", cfg.LLM.URL, "model", cfg.LLM.Model)
|
||||||
}
|
}
|
||||||
resolver := match.NewResolver(index, rewriter)
|
resolver := match.NewResolverWithOptions(index, rewriter, match.ResolverOptions{
|
||||||
|
Parser: parser,
|
||||||
|
OnRewriteError: func(err error) {
|
||||||
|
logger.Warn("llm rewriter failed; using deterministic result", "error", err)
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
var portal outages.Source
|
cities := resolver.Cities()
|
||||||
if cfg.Portal.URL != "" {
|
logger.Info("cities loaded", "count", len(cities), "cities", cities, "default", cfg.DefaultCity)
|
||||||
portal = arcgis.New(
|
if cfg.DefaultCity != "" {
|
||||||
cfg.Portal.URL,
|
if len(cities) > 1 {
|
||||||
time.Duration(cfg.Portal.TimeoutSeconds)*time.Second,
|
logger.Warn("default_city is set for a multi-city index; city-less queries will always use it", "default", cfg.DefaultCity)
|
||||||
cfg.Portal.Retries,
|
} else if len(cities) == 1 && cities[0] != cfg.DefaultCity {
|
||||||
)
|
logger.Warn("default_city is not present in the index", "default", cfg.DefaultCity, "index_city", cities[0])
|
||||||
logger.Info("outages module enabled", "url", cfg.Portal.URL)
|
}
|
||||||
} else {
|
|
||||||
// portal.url is optional: without it only /v1/resolve is served.
|
|
||||||
logger.Info("outages module disabled (portal.url not set)")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
loc := operationalLocation()
|
srv := httpapi.New(resolver, logger)
|
||||||
srv := httpapi.New(resolver, portal, logger, loc)
|
|
||||||
|
|
||||||
httpServer := &http.Server{
|
httpServer := &http.Server{
|
||||||
Addr: cfg.Addr,
|
Addr: cfg.Addr,
|
||||||
Handler: srv.Handler(),
|
Handler: srv.Handler(),
|
||||||
ReadHeaderTimeout: 5 * time.Second,
|
ReadHeaderTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 15 * time.Second,
|
||||||
|
WriteTimeout: 20 * time.Second,
|
||||||
|
IdleTimeout: 60 * time.Second,
|
||||||
|
MaxHeaderBytes: 64 << 10,
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
logger.Info("listening", "addr", cfg.Addr, "tz", loc.String())
|
logger.Info("listening", "addr", cfg.Addr)
|
||||||
if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||||
logger.Error("serve", "error", err)
|
logger.Error("serve", "error", err)
|
||||||
stop()
|
stop()
|
||||||
@@ -116,13 +126,3 @@ func newLogger(level string) *slog.Logger {
|
|||||||
}
|
}
|
||||||
return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl}))
|
return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl}))
|
||||||
}
|
}
|
||||||
|
|
||||||
// operationalLocation returns Asia/Novosibirsk when the tz database is
|
|
||||||
// available and falls back to a fixed UTC+7 offset otherwise (containers
|
|
||||||
// without tzdata).
|
|
||||||
func operationalLocation() *time.Location {
|
|
||||||
if loc, err := time.LoadLocation("Asia/Novosibirsk"); err == nil {
|
|
||||||
return loc
|
|
||||||
}
|
|
||||||
return time.FixedZone("Asia/Novosibirsk", 7*3600)
|
|
||||||
}
|
|
||||||
|
|||||||
+7
-5
@@ -2,16 +2,18 @@
|
|||||||
"addr": ":6171",
|
"addr": ":6171",
|
||||||
"data_path": "registry.db",
|
"data_path": "registry.db",
|
||||||
"log_level": "info",
|
"log_level": "info",
|
||||||
|
"cities": [
|
||||||
|
{
|
||||||
|
"name": "Новосибирск",
|
||||||
|
"aliases": ["нск", "новосиб", "новосибирская область", "нсо"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default_city": "Новосибирск",
|
||||||
"resolve": {
|
"resolve": {
|
||||||
"fuzzy_threshold": 0.82,
|
"fuzzy_threshold": 0.82,
|
||||||
"ambiguity_margin": 0.06,
|
"ambiguity_margin": 0.06,
|
||||||
"max_candidates": 5
|
"max_candidates": 5
|
||||||
},
|
},
|
||||||
"portal": {
|
|
||||||
"url": "${ADDRKIT_PORTAL_URL}",
|
|
||||||
"timeout_seconds": 2,
|
|
||||||
"retries": 1
|
|
||||||
},
|
|
||||||
"llm": {
|
"llm": {
|
||||||
"enabled": false,
|
"enabled": false,
|
||||||
"url": "${ADDRKIT_LLM_URL}",
|
"url": "${ADDRKIT_LLM_URL}",
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=addrkit address resolution service
|
Description=addrkit multi-city address resolution service
|
||||||
Documentation=file:///opt/addrkit/README.md
|
Documentation=file:///opt/addrkit/README.md
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|||||||
+7
-1
@@ -2,7 +2,13 @@
|
|||||||
"addr": "127.0.0.1:6172",
|
"addr": "127.0.0.1:6172",
|
||||||
"data_path": "dev/registry.db",
|
"data_path": "dev/registry.db",
|
||||||
"log_level": "warn",
|
"log_level": "warn",
|
||||||
|
"cities": [
|
||||||
|
{
|
||||||
|
"name": "Новосибирск",
|
||||||
|
"aliases": ["нск", "новосиб", "новосибирская область", "нсо"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"default_city": "Новосибирск",
|
||||||
"resolve": { "fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5 },
|
"resolve": { "fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5 },
|
||||||
"portal": { "url": "https://map.novo-sibirsk.ru/elitegis/rest/services/maps/disconnections/MapServer/101/query", "timeout_seconds": 2, "retries": 0 },
|
|
||||||
"llm": { "enabled": true, "url": "http://127.0.0.1:6193", "token": "EMPTY", "model": "qwen3-14b-awq", "timeout_seconds": 5, "max_tokens": 512, "disable_thinking": true }
|
"llm": { "enabled": true, "url": "http://127.0.0.1:6193", "token": "EMPTY", "model": "qwen3-14b-awq", "timeout_seconds": 5, "max_tokens": 512, "disable_thinking": true }
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,256 +0,0 @@
|
|||||||
// Package arcgis is the adapter for the external outages portal exposed as an
|
|
||||||
// ArcGIS MapServer /query endpoint.
|
|
||||||
//
|
|
||||||
// The portal only supports substring (LIKE) filtering, so precision is
|
|
||||||
// guaranteed locally: whatever the portal returns is re-filtered by the
|
|
||||||
// outages domain using the parsed house unit. The client adds the reliability
|
|
||||||
// the portal lacks: a hard timeout, bounded retries with backoff and context
|
|
||||||
// cancellation.
|
|
||||||
package arcgis
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net/http"
|
|
||||||
"net/url"
|
|
||||||
"regexp"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
|
|
||||||
)
|
|
||||||
|
|
||||||
// reDistrictSuffix matches the administrative suffix the portal appends to its
|
|
||||||
// display address, e.g. "Медкадры улица, 1 (Заельцовский район)". It must be
|
|
||||||
// removed before the address is parsed, otherwise the trailing district text
|
|
||||||
// hides the house number. The district is available separately as a field.
|
|
||||||
var reDistrictSuffix = regexp.MustCompile(`\s*\([^)]*\)\s*$`)
|
|
||||||
|
|
||||||
// cleanAddress strips the portal's district suffix and surrounding spaces.
|
|
||||||
func cleanAddress(s string) string {
|
|
||||||
return strings.TrimSpace(reDistrictSuffix.ReplaceAllString(s, ""))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Client is a bounded HTTP client for the portal.
|
|
||||||
type Client struct {
|
|
||||||
baseURL string
|
|
||||||
http *http.Client
|
|
||||||
retries int
|
|
||||||
}
|
|
||||||
|
|
||||||
// New builds a client. timeout applies to a single attempt; retries is the
|
|
||||||
// number of *additional* attempts after the first.
|
|
||||||
func New(baseURL string, timeout time.Duration, retries int) *Client {
|
|
||||||
return &Client{
|
|
||||||
baseURL: baseURL,
|
|
||||||
http: &http.Client{Timeout: timeout},
|
|
||||||
retries: retries,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Query asks the portal for outages matching the canonical address. The
|
|
||||||
// returned records are raw portal rows; callers must still filter by house.
|
|
||||||
func (c *Client) Query(ctx context.Context, canonical string) ([]outages.Outage, error) {
|
|
||||||
params := url.Values{}
|
|
||||||
params.Set("where", "address like '"+likePattern(canonical)+"'")
|
|
||||||
params.Set("outFields", "")
|
|
||||||
params.Set("returnGeometry", "false")
|
|
||||||
params.Set("f", "json")
|
|
||||||
|
|
||||||
var lastErr error
|
|
||||||
for attempt := 0; attempt <= c.retries; attempt++ {
|
|
||||||
if attempt > 0 {
|
|
||||||
backoff := time.Duration(attempt) * 150 * time.Millisecond
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return nil, ctx.Err()
|
|
||||||
case <-time.After(backoff):
|
|
||||||
}
|
|
||||||
}
|
|
||||||
rows, err := c.do(ctx, params)
|
|
||||||
if err == nil {
|
|
||||||
return rows, nil
|
|
||||||
}
|
|
||||||
lastErr = err
|
|
||||||
}
|
|
||||||
return nil, fmt.Errorf("portal query failed after %d attempts: %w", c.retries+1, lastErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *Client) do(ctx context.Context, params url.Values) ([]outages.Outage, error) {
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"?"+params.Encode(), nil)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("build request: %w", err)
|
|
||||||
}
|
|
||||||
resp, err := c.http.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
defer resp.Body.Close()
|
|
||||||
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
return nil, fmt.Errorf("portal status %d", resp.StatusCode)
|
|
||||||
}
|
|
||||||
// Bound the response size to protect the service from a misbehaving portal.
|
|
||||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return parseResponse(body)
|
|
||||||
}
|
|
||||||
|
|
||||||
// portalResponse mirrors the subset of the ArcGIS /query envelope we use.
|
|
||||||
type portalResponse struct {
|
|
||||||
Features []struct {
|
|
||||||
Value string `json:"value"`
|
|
||||||
Attributes map[string]any `json:"attributes"`
|
|
||||||
} `json:"features"`
|
|
||||||
Fields []struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
Domain struct {
|
|
||||||
CodedValues []struct {
|
|
||||||
Code any `json:"code"`
|
|
||||||
Name string `json:"name"`
|
|
||||||
} `json:"codedValues"`
|
|
||||||
} `json:"domain"`
|
|
||||||
} `json:"fields"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseResponse(body []byte) ([]outages.Outage, error) {
|
|
||||||
var pr portalResponse
|
|
||||||
if err := json.Unmarshal(body, &pr); err != nil {
|
|
||||||
return nil, fmt.Errorf("decode portal response: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build code -> name maps from the coded-value domains.
|
|
||||||
maps := map[string]map[string]string{}
|
|
||||||
for _, f := range pr.Fields {
|
|
||||||
if len(f.Domain.CodedValues) == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
m := make(map[string]string, len(f.Domain.CodedValues))
|
|
||||||
for _, cv := range f.Domain.CodedValues {
|
|
||||||
m[codeKey(cv.Code)] = cv.Name
|
|
||||||
}
|
|
||||||
maps[f.Name] = m
|
|
||||||
}
|
|
||||||
|
|
||||||
out := make([]outages.Outage, 0, len(pr.Features))
|
|
||||||
for _, feat := range pr.Features {
|
|
||||||
addr := cleanAddress(addressOf(feat.Attributes, feat.Value))
|
|
||||||
o := outages.Outage{
|
|
||||||
Address: addr,
|
|
||||||
Type: lookup(maps, "type_id", feat.Attributes),
|
|
||||||
System: lookup(maps, "system_id", feat.Attributes),
|
|
||||||
Time: lookup(maps, "time_category", feat.Attributes),
|
|
||||||
StartDate: stringOf(feat.Attributes["start_date"]),
|
|
||||||
EndDate: stringOf(feat.Attributes["end_date"]),
|
|
||||||
}
|
|
||||||
o.Action = actionFor(o.System)
|
|
||||||
// Parse the address at the boundary so the domain matches on structure
|
|
||||||
// and never depends on this portal's text format.
|
|
||||||
o.Unit = outages.HouseUnitOf(address.Parse(addr))
|
|
||||||
out = append(out, o)
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func addressOf(attrs map[string]any, fallback string) string {
|
|
||||||
if v, ok := attrs["address"]; ok {
|
|
||||||
if s := stringOf(v); s != "" {
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return fallback
|
|
||||||
}
|
|
||||||
|
|
||||||
func lookup(maps map[string]map[string]string, field string, attrs map[string]any) string {
|
|
||||||
m, ok := maps[field]
|
|
||||||
if !ok {
|
|
||||||
return stringOf(attrs[field])
|
|
||||||
}
|
|
||||||
return m[codeKey(attrs[field])]
|
|
||||||
}
|
|
||||||
|
|
||||||
// codeKey normalises a coded value coming from JSON (float64) or a string.
|
|
||||||
func codeKey(v any) string {
|
|
||||||
switch t := v.(type) {
|
|
||||||
case nil:
|
|
||||||
return ""
|
|
||||||
case string:
|
|
||||||
return t
|
|
||||||
case float64:
|
|
||||||
return strconv.FormatInt(int64(t), 10)
|
|
||||||
case json.Number:
|
|
||||||
return t.String()
|
|
||||||
default:
|
|
||||||
return fmt.Sprintf("%v", v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// stringOf renders an attribute value as a string. The important case is
|
|
||||||
// numeric dates: fmt's %v turns a float64 into scientific notation ("1.79e+12")
|
|
||||||
// which the date parser cannot read, so integers are formatted explicitly.
|
|
||||||
func stringOf(v any) string {
|
|
||||||
switch t := v.(type) {
|
|
||||||
case nil:
|
|
||||||
return ""
|
|
||||||
case string:
|
|
||||||
return t
|
|
||||||
case float64:
|
|
||||||
return strconv.FormatInt(int64(t), 10)
|
|
||||||
case json.Number:
|
|
||||||
return t.String()
|
|
||||||
default:
|
|
||||||
return fmt.Sprintf("%v", v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// actionFor maps a portal system name to the canonical action code used by the
|
|
||||||
// scenario (kept for parity with the existing integration).
|
|
||||||
func actionFor(system string) string {
|
|
||||||
switch system {
|
|
||||||
case "Горячее водоснабжение":
|
|
||||||
return "waterWarm"
|
|
||||||
case "Холодное водоснабжение и канализация", "Холодное водоснабжение":
|
|
||||||
return "waterCold"
|
|
||||||
case "Электроснабжение":
|
|
||||||
return "electricitySupply"
|
|
||||||
case "Газоснабжение":
|
|
||||||
return "gasSupply"
|
|
||||||
case "Теплоснабжение":
|
|
||||||
return "heating"
|
|
||||||
default:
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// likePattern builds the substring pattern the portal supports. Only pure
|
|
||||||
// numeric tokens (house numbers) are padded with spaces, so "1" does not
|
|
||||||
// incidentally match "10". Ordinals ("3-я") and houses with a letter ("24а")
|
|
||||||
// are left as-is: padding an ordinal breaks the match when it starts the
|
|
||||||
// string, and precision is guaranteed by the local house filter anyway.
|
|
||||||
func likePattern(canonical string) string {
|
|
||||||
parts := strings.Fields(canonical)
|
|
||||||
for i, part := range parts {
|
|
||||||
if isPureDigits(part) {
|
|
||||||
parts[i] = " " + part + " "
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "%" + strings.Join(parts, "%") + "%"
|
|
||||||
}
|
|
||||||
|
|
||||||
func isPureDigits(s string) bool {
|
|
||||||
if s == "" {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
for _, r := range s {
|
|
||||||
if r < '0' || r > '9' {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
package arcgis
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
const envelope = `{
|
|
||||||
"features": [
|
|
||||||
{"value": "Станционная улица 1/1", "attributes": {
|
|
||||||
"address": "Станционная улица 1/1",
|
|
||||||
"type_id": 1, "system_id": 5, "time_category": 2,
|
|
||||||
"start_date": "2026-10-01T00:00:00", "end_date": "2026-10-31T00:00:00"
|
|
||||||
}}
|
|
||||||
],
|
|
||||||
"fields": [
|
|
||||||
{"name": "type_id", "domain": {"codedValues": [{"code": 1, "name": "Плановое"}]}},
|
|
||||||
{"name": "system_id", "domain": {"codedValues": [{"code": 5, "name": "Теплоснабжение"}]}},
|
|
||||||
{"name": "time_category", "domain": {"codedValues": [{"code": 2, "name": "Текущие"}]}}
|
|
||||||
]
|
|
||||||
}`
|
|
||||||
|
|
||||||
func TestQueryParsesEnvelope(t *testing.T) {
|
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
if got := r.URL.Query().Get("f"); got != "json" {
|
|
||||||
t.Errorf("missing f=json, got %q", got)
|
|
||||||
}
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_, _ = w.Write([]byte(envelope))
|
|
||||||
}))
|
|
||||||
defer ts.Close()
|
|
||||||
|
|
||||||
c := New(ts.URL, time.Second, 0)
|
|
||||||
rows, err := c.Query(context.Background(), "Станционная улица 1/1")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Query: %v", err)
|
|
||||||
}
|
|
||||||
if len(rows) != 1 {
|
|
||||||
t.Fatalf("rows = %d, want 1", len(rows))
|
|
||||||
}
|
|
||||||
got := rows[0]
|
|
||||||
if got.Type != "Плановое" || got.System != "Теплоснабжение" || got.Time != "Текущие" {
|
|
||||||
t.Errorf("coded values not resolved: %+v", got)
|
|
||||||
}
|
|
||||||
if got.Action != "heating" {
|
|
||||||
t.Errorf("Action = %q, want heating", got.Action)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestQueryRetriesThenFails(t *testing.T) {
|
|
||||||
var calls int
|
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
calls++
|
|
||||||
http.Error(w, "boom", http.StatusInternalServerError)
|
|
||||||
}))
|
|
||||||
defer ts.Close()
|
|
||||||
|
|
||||||
c := New(ts.URL, time.Second, 2)
|
|
||||||
if _, err := c.Query(context.Background(), "Станционная улица 1/1"); err == nil {
|
|
||||||
t.Fatal("expected error")
|
|
||||||
}
|
|
||||||
if calls != 3 {
|
|
||||||
t.Errorf("calls = %d, want 3 (1 + 2 retries)", calls)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestQueryStripsDistrictSuffix is a regression test: the portal appends the
|
|
||||||
// district in parentheses, which used to hide the house number from the parser.
|
|
||||||
func TestQueryStripsDistrictSuffix(t *testing.T) {
|
|
||||||
const env = `{"features":[{"value":"Медкадры улица, 1 (Заельцовский район)","attributes":{"address":"Медкадры улица, 1 (Заельцовский район)"}}]}`
|
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_, _ = w.Write([]byte(env))
|
|
||||||
}))
|
|
||||||
defer ts.Close()
|
|
||||||
|
|
||||||
c := New(ts.URL, time.Second, 0)
|
|
||||||
rows, err := c.Query(context.Background(), "Медкадры улица 1")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Query: %v", err)
|
|
||||||
}
|
|
||||||
if len(rows) != 1 {
|
|
||||||
t.Fatalf("rows = %d, want 1", len(rows))
|
|
||||||
}
|
|
||||||
if rows[0].Address != "Медкадры улица, 1" {
|
|
||||||
t.Errorf("district suffix not stripped: %q", rows[0].Address)
|
|
||||||
}
|
|
||||||
if rows[0].Unit.House != "1" {
|
|
||||||
t.Errorf("house not parsed at the boundary: %+v", rows[0].Unit)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestQueryFormatsNumericDates guards against scientific notation in dates,
|
|
||||||
// which broke the "current outage" filter.
|
|
||||||
func TestQueryFormatsNumericDates(t *testing.T) {
|
|
||||||
const env = `{"features":[{"value":"Ленина улица, 1","attributes":{"address":"Ленина улица, 1","start_date":1790820000000,"end_date":1791799200000}}]}`
|
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
_, _ = w.Write([]byte(env))
|
|
||||||
}))
|
|
||||||
defer ts.Close()
|
|
||||||
|
|
||||||
rows, err := New(ts.URL, time.Second, 0).Query(context.Background(), "Ленина улица 1")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Query: %v", err)
|
|
||||||
}
|
|
||||||
if rows[0].StartDate != "1790820000000" || rows[0].EndDate != "1791799200000" {
|
|
||||||
t.Errorf("numeric dates must be integer-formatted: %q..%q", rows[0].StartDate, rows[0].EndDate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLikePattern guards the ordinal/house padding rule: padding an ordinal
|
|
||||||
// that starts the string breaks the portal match.
|
|
||||||
func TestLikePattern(t *testing.T) {
|
|
||||||
got := likePattern("3-я Механическая улица 24а")
|
|
||||||
if strings.Contains(got, " 3-я ") {
|
|
||||||
t.Errorf("ordinal must not be space-padded: %q", got)
|
|
||||||
}
|
|
||||||
if strings.Contains(got, " 24а ") {
|
|
||||||
t.Errorf("house with a letter must not be space-padded: %q", got)
|
|
||||||
}
|
|
||||||
if !strings.Contains(likePattern("Ленина улица 1"), "% 1 %") {
|
|
||||||
t.Errorf("pure numeric house must be space-padded: %q", likePattern("Ленина улица 1"))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -14,10 +14,13 @@ type ctxKey int
|
|||||||
const requestIDKey ctxKey = 0
|
const requestIDKey ctxKey = 0
|
||||||
|
|
||||||
// requestIDMW assigns a request id (for correlation in logs) and echoes it.
|
// requestIDMW assigns a request id (for correlation in logs) and echoes it.
|
||||||
|
// A client-supplied id is accepted only when it is short and well-formed; the
|
||||||
|
// header ends up in logs, so accepting arbitrary bytes would be an injection
|
||||||
|
// and log-flooding vector.
|
||||||
func requestIDMW(next http.Handler) http.Handler {
|
func requestIDMW(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
id := r.Header.Get("X-Request-Id")
|
id := r.Header.Get("X-Request-Id")
|
||||||
if id == "" {
|
if !validRequestID(id) {
|
||||||
id = newRequestID()
|
id = newRequestID()
|
||||||
}
|
}
|
||||||
w.Header().Set("X-Request-Id", id)
|
w.Header().Set("X-Request-Id", id)
|
||||||
@@ -26,6 +29,23 @@ func requestIDMW(next http.Handler) http.Handler {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const maxRequestIDLen = 128
|
||||||
|
|
||||||
|
func validRequestID(id string) bool {
|
||||||
|
if id == "" || len(id) > maxRequestIDLen {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, r := range id {
|
||||||
|
switch {
|
||||||
|
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
|
||||||
|
case r == '-', r == '_', r == '.':
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// loggingMW logs one line per request with method, path, status and duration.
|
// loggingMW logs one line per request with method, path, status and duration.
|
||||||
// It records no addresses (PII) — only routing metadata.
|
// It records no addresses (PII) — only routing metadata.
|
||||||
func loggingMW(log *slog.Logger) func(http.Handler) http.Handler {
|
func loggingMW(log *slog.Logger) func(http.Handler) http.Handler {
|
||||||
@@ -45,8 +65,10 @@ func loggingMW(log *slog.Logger) func(http.Handler) http.Handler {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// timeoutMW bounds the whole request, so a slow portal cannot tie up a
|
// timeoutMW bounds the whole request by setting a context deadline. Handlers
|
||||||
// goroutine indefinitely.
|
// and adapters that talk to the network honour it; purely CPU-bound handlers
|
||||||
|
// are additionally bounded by input limits and the HTTP server's read/write
|
||||||
|
// timeouts in cmd/addrkitd.
|
||||||
func timeoutMW(d time.Duration) func(http.Handler) http.Handler {
|
func timeoutMW(d time.Duration) func(http.Handler) http.Handler {
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
|||||||
@@ -1,52 +1,54 @@
|
|||||||
// Package httpapi exposes the addrkit domain over a small, versioned JSON API.
|
// Package httpapi exposes the addrkit domain over a small, versioned JSON API.
|
||||||
//
|
//
|
||||||
// It contains no business logic: handlers decode requests, call the resolver
|
// It contains no business logic: handlers decode requests, call the resolver
|
||||||
// and the outages source, and encode responses. Cross-cutting concerns
|
// and encode responses. Cross-cutting concerns (timeouts, recovery, request
|
||||||
// (timeouts, recovery, request ids, logging) live in middleware.
|
// ids, logging) live in middleware.
|
||||||
package httpapi
|
package httpapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Resolver is the read side of the domain (implemented by *match.Resolver).
|
// Resolver is the read side of the domain (implemented by *match.Resolver).
|
||||||
type Resolver interface {
|
type Resolver interface {
|
||||||
Resolve(ctx context.Context, raw string) match.Result
|
Resolve(ctx context.Context, raw string) match.Result
|
||||||
|
ResolveInCity(ctx context.Context, raw, city string) match.Result
|
||||||
|
MatchCity(query string) (string, bool)
|
||||||
Version() string
|
Version() string
|
||||||
Ready() bool
|
Ready() bool
|
||||||
|
Cities() []string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// maxAddressRunes bounds user input. A real address is far shorter; the limit
|
||||||
|
// keeps a pathological string from turning into a CPU-bound fuzzy scan.
|
||||||
|
const maxAddressRunes = 512
|
||||||
|
|
||||||
|
// maxCityQueryRunes bounds the separate city-only endpoint.
|
||||||
|
const maxCityQueryRunes = 128
|
||||||
|
|
||||||
// Server wires the domain to HTTP.
|
// Server wires the domain to HTTP.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
resolver Resolver
|
resolver Resolver
|
||||||
outages outages.Source
|
|
||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
now func() time.Time
|
|
||||||
loc *time.Location
|
|
||||||
timeout time.Duration
|
timeout time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// New builds a server. loc controls how outage dates are interpreted; it
|
// New builds a server.
|
||||||
// should be the operational timezone (Asia/Novosibirsk).
|
func New(resolver Resolver, log *slog.Logger) *Server {
|
||||||
func New(resolver Resolver, src outages.Source, log *slog.Logger, loc *time.Location) *Server {
|
if log == nil {
|
||||||
if loc == nil {
|
log = slog.Default()
|
||||||
loc = time.UTC
|
|
||||||
}
|
}
|
||||||
return &Server{
|
return &Server{
|
||||||
resolver: resolver,
|
resolver: resolver,
|
||||||
outages: src,
|
|
||||||
log: log,
|
log: log,
|
||||||
now: time.Now,
|
|
||||||
loc: loc,
|
|
||||||
timeout: 10 * time.Second,
|
timeout: 10 * time.Second,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -55,16 +57,21 @@ func New(resolver Resolver, src outages.Source, log *slog.Logger, loc *time.Loca
|
|||||||
func (s *Server) Handler() http.Handler {
|
func (s *Server) Handler() http.Handler {
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.HandleFunc("POST /v1/resolve", s.handleResolve)
|
mux.HandleFunc("POST /v1/resolve", s.handleResolve)
|
||||||
mux.HandleFunc("POST /v1/outages", s.handleOutages)
|
mux.HandleFunc("POST /v1/city", s.handleCity)
|
||||||
|
mux.HandleFunc("GET /v1/cities", s.handleCities)
|
||||||
mux.HandleFunc("GET /v1/healthz", s.handleHealthz)
|
mux.HandleFunc("GET /v1/healthz", s.handleHealthz)
|
||||||
mux.HandleFunc("GET /v1/readyz", s.handleReadyz)
|
mux.HandleFunc("GET /v1/readyz", s.handleReadyz)
|
||||||
mux.HandleFunc("GET /v1/version", s.handleVersion)
|
mux.HandleFunc("GET /v1/version", s.handleVersion)
|
||||||
|
|
||||||
return recoverMW(s.log)(requestIDMW(loggingMW(s.log)(timeoutMW(s.timeout)(mux))))
|
// Logging sits outside recovery so a panic is still recorded as a 500.
|
||||||
|
return requestIDMW(loggingMW(s.log)(recoverMW(s.log)(timeoutMW(s.timeout)(mux))))
|
||||||
}
|
}
|
||||||
|
|
||||||
type resolveRequest struct {
|
type resolveRequest struct {
|
||||||
Address string `json:"address"`
|
Address string `json:"address"`
|
||||||
|
// City is an optional hint for clients that keep the city separate from
|
||||||
|
// the street line. It overrides a city found in the address text.
|
||||||
|
City string `json:"city"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -72,64 +79,49 @@ func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
writeJSON(w, http.StatusOK, s.resolver.Resolve(r.Context(), req.Address))
|
writeJSON(w, http.StatusOK, s.resolver.ResolveInCity(r.Context(), req.Address, req.City))
|
||||||
}
|
}
|
||||||
|
|
||||||
type outagesResponse struct {
|
type cityRequest struct {
|
||||||
Status address.Status `json:"status"`
|
Query string `json:"query"`
|
||||||
Address string `json:"address,omitempty"`
|
|
||||||
District string `json:"district,omitempty"`
|
|
||||||
Outages []outages.Outage `json:"outages"`
|
|
||||||
AsOf string `json:"as_of,omitempty"`
|
|
||||||
DataVersion string `json:"data_version,omitempty"`
|
|
||||||
Detail string `json:"detail,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleOutages(w http.ResponseWriter, r *http.Request) {
|
type cityResponse struct {
|
||||||
req, ok := s.decode(w, r)
|
Status string `json:"status"`
|
||||||
if !ok {
|
City string `json:"city,omitempty"`
|
||||||
return
|
}
|
||||||
}
|
|
||||||
ctx := r.Context()
|
|
||||||
res := s.resolver.Resolve(ctx, req.Address)
|
|
||||||
|
|
||||||
out := outagesResponse{
|
// handleCity normalizes a city-only query ("нск", "г. Новосибирск",
|
||||||
Status: res.Status,
|
// "Новосибирская область"). It is the small, separate use case for projects
|
||||||
Address: res.Address,
|
// that need city names but not streets.
|
||||||
District: res.District,
|
func (s *Server) handleCity(w http.ResponseWriter, r *http.Request) {
|
||||||
DataVersion: res.DataVersion,
|
var req cityRequest
|
||||||
Outages: []outages.Outage{},
|
r.Body = http.MaxBytesReader(w, r.Body, 16*1024)
|
||||||
}
|
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||||
if res.Status != address.StatusFound {
|
writeError(w, http.StatusBadRequest, "invalid json body")
|
||||||
// No canonical address => nothing to query. Return the resolver's
|
|
||||||
// verdict so the caller can ask for clarification.
|
|
||||||
writeJSON(w, http.StatusOK, out)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if s.outages == nil {
|
if req.Query == "" {
|
||||||
// The events module is optional and off when portal.url is empty.
|
writeError(w, http.StatusBadRequest, "query is required")
|
||||||
out.Status = address.StatusInfra
|
|
||||||
out.Detail = "outages source is not configured"
|
|
||||||
writeJSON(w, http.StatusOK, out)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if utf8.RuneCountInString(req.Query) > maxCityQueryRunes {
|
||||||
|
writeError(w, http.StatusBadRequest, "query is too long")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if city, ok := s.resolver.MatchCity(req.Query); ok {
|
||||||
|
writeJSON(w, http.StatusOK, cityResponse{Status: "found", City: city})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, cityResponse{Status: "not_found"})
|
||||||
|
}
|
||||||
|
|
||||||
raw, err := s.outages.Query(ctx, res.Address)
|
func (s *Server) handleCities(w http.ResponseWriter, _ *http.Request) {
|
||||||
if err != nil {
|
cities := s.resolver.Cities()
|
||||||
if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
|
if cities == nil {
|
||||||
out.Status = address.StatusPortalTimeout
|
cities = []string{}
|
||||||
} else {
|
|
||||||
out.Status = address.StatusInfra
|
|
||||||
}
|
|
||||||
writeJSON(w, http.StatusOK, out)
|
|
||||||
return
|
|
||||||
}
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string][]string{"cities": cities})
|
||||||
parsed := address.Parse(res.Address)
|
|
||||||
current := outages.Current(raw, s.now(), s.loc)
|
|
||||||
out.Outages = outages.MatchingHouse(current, parsed)
|
|
||||||
out.AsOf = s.now().In(s.loc).Format(time.RFC3339)
|
|
||||||
writeJSON(w, http.StatusOK, out)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
||||||
@@ -160,6 +152,14 @@ func (s *Server) decode(w http.ResponseWriter, r *http.Request) (resolveRequest,
|
|||||||
writeError(w, http.StatusBadRequest, "address is required")
|
writeError(w, http.StatusBadRequest, "address is required")
|
||||||
return resolveRequest{}, false
|
return resolveRequest{}, false
|
||||||
}
|
}
|
||||||
|
if utf8.RuneCountInString(req.Address) > maxAddressRunes {
|
||||||
|
writeError(w, http.StatusBadRequest, "address is too long")
|
||||||
|
return resolveRequest{}, false
|
||||||
|
}
|
||||||
|
if utf8.RuneCountInString(req.City) > 128 {
|
||||||
|
writeError(w, http.StatusBadRequest, "city is too long")
|
||||||
|
return resolveRequest{}, false
|
||||||
|
}
|
||||||
return req, true
|
return req, true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,46 +2,51 @@ package httpapi
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func entry(t *testing.T, city, raw string) address.Entry {
|
||||||
|
t.Helper()
|
||||||
|
p := address.NewParser(address.NewDictionary(cityDictionary()), "").ParseInCity(raw, city)
|
||||||
|
return address.Entry{
|
||||||
|
ID: raw, Address: raw, District: "Тестовый",
|
||||||
|
City: p.City, CityKey: p.CityKey,
|
||||||
|
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
|
||||||
|
StreetType: p.StreetType, StreetKey: p.StreetKey,
|
||||||
|
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
|
||||||
|
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
|
||||||
|
Structure: p.Structure,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cityDictionary() []address.City {
|
||||||
|
return []address.City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск", "новосиб", "новосибирская область"}},
|
||||||
|
{Name: "Бердск"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func testResolver(t *testing.T) *match.Resolver {
|
func testResolver(t *testing.T) *match.Resolver {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
var entries []address.Entry
|
var entries []address.Entry
|
||||||
for _, a := range []string{"Станционная улица 1/1", "2-я Станционная улица 1/1", "Искитимская улица 5"} {
|
for _, a := range []string{"Станционная улица 1/1", "2-я Станционная улица 1/1", "Искитимская улица 5"} {
|
||||||
p := address.Parse(a)
|
entries = append(entries, entry(t, "", a))
|
||||||
entries = append(entries, address.Entry{
|
|
||||||
ID: a, Address: a, District: "Тестовый",
|
|
||||||
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey,
|
|
||||||
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
return match.NewResolver(match.NewIndex("test-version", entries, match.DefaultOptions()), nil)
|
return match.NewResolver(match.NewIndex("test-version", entries, match.DefaultOptions()), nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeSource struct {
|
func newServer(t *testing.T, resolver Resolver) *httptest.Server {
|
||||||
list []outages.Outage
|
|
||||||
err error
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f fakeSource) Query(context.Context, string) ([]outages.Outage, error) {
|
|
||||||
return f.list, f.err
|
|
||||||
}
|
|
||||||
|
|
||||||
func newServer(t *testing.T, src fakeSource) *httptest.Server {
|
|
||||||
t.Helper()
|
t.Helper()
|
||||||
srv := New(testResolver(t), src, slog.New(slog.NewTextHandler(io.Discard, nil)), time.UTC)
|
srv := New(resolver, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
ts := httptest.NewServer(srv.Handler())
|
ts := httptest.NewServer(srv.Handler())
|
||||||
t.Cleanup(ts.Close)
|
t.Cleanup(ts.Close)
|
||||||
return ts
|
return ts
|
||||||
@@ -57,7 +62,7 @@ func postJSON(t *testing.T, url, body string) *http.Response {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveEndpoint(t *testing.T) {
|
func TestResolveEndpoint(t *testing.T) {
|
||||||
ts := newServer(t, fakeSource{})
|
ts := newServer(t, testResolver(t))
|
||||||
resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"2-я Станционная 1/1"}`)
|
resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"2-я Станционная 1/1"}`)
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
||||||
@@ -73,38 +78,120 @@ func TestResolveEndpoint(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestOutagesEndpointFiltersHouse(t *testing.T) {
|
func TestResolveValidation(t *testing.T) {
|
||||||
src := fakeSource{list: []outages.Outage{
|
ts := newServer(t, testResolver(t))
|
||||||
{Address: "Станционная улица 1/1", System: "Теплоснабжение"},
|
for _, body := range []string{`{}`, `{"address":""}`, `not json`} {
|
||||||
{Address: "Станционная улица 1/2", System: "Теплоснабжение"},
|
resp := postJSON(t, ts.URL+"/v1/resolve", body)
|
||||||
}}
|
resp.Body.Close()
|
||||||
ts := newServer(t, src)
|
if resp.StatusCode != http.StatusBadRequest {
|
||||||
resp := postJSON(t, ts.URL+"/v1/outages", `{"address":"станционная 1/1"}`)
|
t.Errorf("body %q: status = %d, want 400", body, resp.StatusCode)
|
||||||
defer resp.Body.Close()
|
}
|
||||||
|
|
||||||
var out outagesResponse
|
|
||||||
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
|
|
||||||
t.Fatalf("decode: %v", err)
|
|
||||||
}
|
|
||||||
if out.Status != address.StatusFound {
|
|
||||||
t.Fatalf("status = %q, want found", out.Status)
|
|
||||||
}
|
|
||||||
if len(out.Outages) != 1 || out.Outages[0].Address != "Станционная улица 1/1" {
|
|
||||||
t.Fatalf("house filter failed: %+v", out.Outages)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveValidation(t *testing.T) {
|
func TestResolveRejectsOversizedAddress(t *testing.T) {
|
||||||
ts := newServer(t, fakeSource{})
|
ts := newServer(t, testResolver(t))
|
||||||
resp := postJSON(t, ts.URL+"/v1/resolve", `{}`)
|
body, _ := json.Marshal(map[string]string{"address": strings.Repeat("а", maxAddressRunes+1)})
|
||||||
defer resp.Body.Close()
|
resp := postJSON(t, ts.URL+"/v1/resolve", string(body))
|
||||||
|
resp.Body.Close()
|
||||||
if resp.StatusCode != http.StatusBadRequest {
|
if resp.StatusCode != http.StatusBadRequest {
|
||||||
t.Fatalf("status = %d, want 400", resp.StatusCode)
|
t.Fatalf("status = %d, want 400", resp.StatusCode)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolveCityHint(t *testing.T) {
|
||||||
|
entries := []address.Entry{
|
||||||
|
entry(t, "Новосибирск", "Ленина улица 5"),
|
||||||
|
entry(t, "Бердск", "Ленина улица 5"),
|
||||||
|
}
|
||||||
|
resolver := match.NewResolver(
|
||||||
|
match.NewIndex("test", entries, match.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
)
|
||||||
|
ts := newServer(t, resolver)
|
||||||
|
|
||||||
|
resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"Ленина 5","city":"Бердск"}`)
|
||||||
|
defer resp.Body.Close()
|
||||||
|
var res match.Result
|
||||||
|
if err := json.NewDecoder(resp.Body).Decode(&res); err != nil {
|
||||||
|
t.Fatalf("decode: %v", err)
|
||||||
|
}
|
||||||
|
if res.Status != address.StatusFound || res.City != "Бердск" || res.Address != "Ленина улица 5" {
|
||||||
|
t.Fatalf("unexpected result: %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCitiesEndpoint(t *testing.T) {
|
||||||
|
entries := []address.Entry{
|
||||||
|
entry(t, "Новосибирск", "Ленина улица 5"),
|
||||||
|
entry(t, "Бердск", "Ленина улица 5"),
|
||||||
|
}
|
||||||
|
resolver := match.NewResolver(match.NewIndex("test", entries, match.DefaultOptions()), nil)
|
||||||
|
ts := newServer(t, resolver)
|
||||||
|
|
||||||
|
resp, err := http.Get(ts.URL + "/v1/cities")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("get: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
var body struct {
|
||||||
|
Cities []string `json:"cities"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
|
||||||
|
t.Fatalf("decode: %v", err)
|
||||||
|
}
|
||||||
|
if len(body.Cities) != 2 || body.Cities[0] != "Бердск" || body.Cities[1] != "Новосибирск" {
|
||||||
|
t.Fatalf("cities = %v", body.Cities)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCityEndpoint(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary(cityDictionary()), "")
|
||||||
|
resolver := match.NewResolverWithOptions(
|
||||||
|
match.NewIndex("test", nil, match.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
match.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
ts := newServer(t, resolver)
|
||||||
|
|
||||||
|
for _, tt := range []struct {
|
||||||
|
query string
|
||||||
|
city string
|
||||||
|
}{
|
||||||
|
{"нск", "Новосибирск"},
|
||||||
|
{"г. Новосибирск", "Новосибирск"},
|
||||||
|
{"Новосибирская область", "Новосибирск"},
|
||||||
|
{"Бердск", "Бердск"},
|
||||||
|
{"Томск", ""},
|
||||||
|
} {
|
||||||
|
body, _ := json.Marshal(map[string]string{"query": tt.query})
|
||||||
|
resp := postJSON(t, ts.URL+"/v1/city", string(body))
|
||||||
|
var got cityResponse
|
||||||
|
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||||
|
resp.Body.Close()
|
||||||
|
t.Fatalf("%q: decode: %v", tt.query, err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
if tt.city == "" {
|
||||||
|
if got.Status != "not_found" {
|
||||||
|
t.Errorf("%q: status = %q, want not_found", tt.query, got.Status)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if got.Status != "found" || got.City != tt.city {
|
||||||
|
t.Errorf("%q: %+v", tt.query, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resp := postJSON(t, ts.URL+"/v1/city", `{}`)
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusBadRequest {
|
||||||
|
t.Errorf("empty query status = %d, want 400", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestReadyzAndVersion(t *testing.T) {
|
func TestReadyzAndVersion(t *testing.T) {
|
||||||
ts := newServer(t, fakeSource{})
|
ts := newServer(t, testResolver(t))
|
||||||
|
|
||||||
resp, err := http.Get(ts.URL + "/v1/readyz")
|
resp, err := http.Get(ts.URL + "/v1/readyz")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -120,9 +207,11 @@ func TestReadyzAndVersion(t *testing.T) {
|
|||||||
t.Fatalf("version: %v", err)
|
t.Fatalf("version: %v", err)
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
var v map[string]string
|
var body map[string]string
|
||||||
_ = json.NewDecoder(resp.Body).Decode(&v)
|
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
|
||||||
if v["data_version"] != "test-version" {
|
t.Fatalf("decode: %v", err)
|
||||||
t.Fatalf("version = %q", v["data_version"])
|
}
|
||||||
|
if body["data_version"] != "test-version" {
|
||||||
|
t.Fatalf("data_version = %q", body["data_version"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,10 +14,12 @@ import (
|
|||||||
_ "embed"
|
_ "embed"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/config"
|
"git.totmin.ru/en2zmax/addrkit/internal/config"
|
||||||
)
|
)
|
||||||
@@ -128,7 +130,9 @@ func (r *Rewriter) Rewrite(ctx context.Context, raw string) ([]string, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var cr chatResponse
|
var cr chatResponse
|
||||||
if err := json.NewDecoder(resp.Body).Decode(&cr); err != nil {
|
// Bound the reply: a broken or hostile model must not be able to stream an
|
||||||
|
// unbounded JSON body into the service.
|
||||||
|
if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&cr); err != nil {
|
||||||
return nil, fmt.Errorf("decode llm response: %w", err)
|
return nil, fmt.Errorf("decode llm response: %w", err)
|
||||||
}
|
}
|
||||||
if len(cr.Choices) == 0 {
|
if len(cr.Choices) == 0 {
|
||||||
@@ -154,8 +158,15 @@ func parseVariants(content string) []string {
|
|||||||
}
|
}
|
||||||
out := make([]string, 0, len(parsed.Variants))
|
out := make([]string, 0, len(parsed.Variants))
|
||||||
for _, v := range parsed.Variants {
|
for _, v := range parsed.Variants {
|
||||||
if s := strings.TrimSpace(v); s != "" {
|
s := strings.TrimSpace(v)
|
||||||
out = append(out, s)
|
// A variant is a short address; the cap also keeps a hostile model
|
||||||
|
// from making the resolver scan a megabyte-long token.
|
||||||
|
if s == "" || utf8.RuneCountInString(s) > 512 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, s)
|
||||||
|
if len(out) == 3 {
|
||||||
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
|
|||||||
@@ -86,3 +86,17 @@ func TestParseVariants(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestParseVariantsCapsCountAndLength(t *testing.T) {
|
||||||
|
long := strings.Repeat("x", 600)
|
||||||
|
content := `{"variants":["` + long + `","a","b","c","d"]}`
|
||||||
|
out := parseVariants(content)
|
||||||
|
if len(out) != 3 {
|
||||||
|
t.Fatalf("variants = %v, want 3", out)
|
||||||
|
}
|
||||||
|
for _, v := range out {
|
||||||
|
if len([]rune(v)) > 512 {
|
||||||
|
t.Fatalf("oversized variant accepted: %d runes", len([]rune(v)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -5,37 +5,49 @@ package registrystore
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"database/sql"
|
"database/sql"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
|
||||||
_ "modernc.org/sqlite" // registers the "sqlite" driver
|
_ "modernc.org/sqlite" // registers the "sqlite" driver
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// SchemaVersion identifies the on-disk format. It is independent of the data
|
||||||
|
// version (the registry snapshot date) and must be bumped whenever the key
|
||||||
|
// format or columns change. A service refuses to open a database it does not
|
||||||
|
// understand instead of silently resolving against half-empty fields.
|
||||||
|
const SchemaVersion = "2"
|
||||||
|
|
||||||
const schema = `
|
const schema = `
|
||||||
CREATE TABLE IF NOT EXISTS meta (
|
CREATE TABLE IF NOT EXISTS meta (
|
||||||
key TEXT PRIMARY KEY,
|
key TEXT PRIMARY KEY,
|
||||||
value TEXT NOT NULL
|
value TEXT NOT NULL
|
||||||
);
|
);
|
||||||
CREATE TABLE IF NOT EXISTS address (
|
CREATE TABLE IF NOT EXISTS address (
|
||||||
id TEXT PRIMARY KEY,
|
id TEXT PRIMARY KEY,
|
||||||
address TEXT NOT NULL,
|
address TEXT NOT NULL,
|
||||||
district TEXT NOT NULL DEFAULT '',
|
district TEXT NOT NULL DEFAULT '',
|
||||||
ordinal TEXT NOT NULL DEFAULT '',
|
city TEXT NOT NULL DEFAULT '',
|
||||||
name_key TEXT NOT NULL DEFAULT '',
|
city_key TEXT NOT NULL DEFAULT '',
|
||||||
name_display TEXT NOT NULL DEFAULT '',
|
ordinal TEXT NOT NULL DEFAULT '',
|
||||||
street_type TEXT NOT NULL DEFAULT '',
|
name_key TEXT NOT NULL DEFAULT '',
|
||||||
street_key TEXT NOT NULL,
|
name_display TEXT NOT NULL DEFAULT '',
|
||||||
house TEXT NOT NULL DEFAULT '',
|
street_type TEXT NOT NULL DEFAULT '',
|
||||||
house_frac TEXT NOT NULL DEFAULT '',
|
street_key TEXT NOT NULL,
|
||||||
house_letter TEXT NOT NULL DEFAULT '',
|
house TEXT NOT NULL DEFAULT '',
|
||||||
building TEXT NOT NULL DEFAULT '',
|
house_frac TEXT NOT NULL DEFAULT '',
|
||||||
building_frac TEXT NOT NULL DEFAULT '',
|
house_letter TEXT NOT NULL DEFAULT '',
|
||||||
|
building TEXT NOT NULL DEFAULT '',
|
||||||
|
building_frac TEXT NOT NULL DEFAULT '',
|
||||||
building_letter TEXT NOT NULL DEFAULT '',
|
building_letter TEXT NOT NULL DEFAULT '',
|
||||||
structure TEXT NOT NULL DEFAULT ''
|
structure TEXT NOT NULL DEFAULT ''
|
||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS idx_address_street ON address(street_key);
|
CREATE INDEX IF NOT EXISTS idx_address_street ON address(street_key);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_address_city ON address(city_key);
|
||||||
`
|
`
|
||||||
|
|
||||||
// Save writes entries to path atomically: it builds a temporary file and
|
// Save writes entries to path atomically: it builds a temporary file and
|
||||||
@@ -44,13 +56,21 @@ CREATE INDEX IF NOT EXISTS idx_address_street ON address(street_key);
|
|||||||
// The build uses the rollback journal (not WAL) on purpose, so that the result
|
// The build uses the rollback journal (not WAL) on purpose, so that the result
|
||||||
// is a single file with no companion -wal/-shm that a rename would orphan.
|
// is a single file with no companion -wal/-shm that a rename would orphan.
|
||||||
func Save(path, version string, entries []address.Entry) error {
|
func Save(path, version string, entries []address.Entry) error {
|
||||||
tmp := path + ".tmp"
|
if len(entries) == 0 {
|
||||||
removeDBFiles(tmp)
|
return errors.New("refusing to publish an empty index")
|
||||||
|
}
|
||||||
|
|
||||||
db, err := sql.Open("sqlite", "file:"+tmp+
|
tmp := path + ".tmp"
|
||||||
"?_pragma=journal_mode(DELETE)"+
|
// A previous failed build may have left a complete temp database with the
|
||||||
"&_pragma=synchronous(FULL)"+
|
// same meta rows; reusing it would hit a primary-key conflict.
|
||||||
"&_pragma=busy_timeout(5000)")
|
_ = os.Remove(tmp)
|
||||||
|
removeSidecars(tmp)
|
||||||
|
|
||||||
|
uri, err := dsn(tmp, false)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
db, err := sql.Open("sqlite", uri)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("open temp db: %w", err)
|
return fmt.Errorf("open temp db: %w", err)
|
||||||
}
|
}
|
||||||
@@ -68,12 +88,46 @@ func Save(path, version string, entries []address.Entry) error {
|
|||||||
if err := os.Rename(tmp, path); err != nil {
|
if err := os.Rename(tmp, path); err != nil {
|
||||||
return fmt.Errorf("publish db: %w", err)
|
return fmt.Errorf("publish db: %w", err)
|
||||||
}
|
}
|
||||||
// Drop only the temp file's journal siblings; the published base file must
|
// The rename is atomic, but a crash immediately after it may lose the
|
||||||
// survive.
|
// directory entry. Syncing the parent makes the publish durable.
|
||||||
|
syncDir(filepath.Dir(path))
|
||||||
removeSidecars(tmp)
|
removeSidecars(tmp)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dsn builds a SQLite URI. Concatenating "file:" + path breaks on paths that
|
||||||
|
// contain "?", "#" or "%", so let net/url escape the path for us.
|
||||||
|
func dsn(path string, readOnly bool) (string, error) {
|
||||||
|
abs, err := filepath.Abs(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("resolve db path: %w", err)
|
||||||
|
}
|
||||||
|
u := url.URL{Scheme: "file", Path: abs}
|
||||||
|
q := u.Query()
|
||||||
|
q.Add("_pragma", "busy_timeout(5000)")
|
||||||
|
if readOnly {
|
||||||
|
q.Set("mode", "ro")
|
||||||
|
q.Add("_pragma", "query_only(1)")
|
||||||
|
} else {
|
||||||
|
q.Add("_pragma", "journal_mode(DELETE)")
|
||||||
|
q.Add("_pragma", "synchronous(FULL)")
|
||||||
|
}
|
||||||
|
u.RawQuery = q.Encode()
|
||||||
|
return u.String(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// syncDir fsyncs a directory so a freshly renamed file survives a crash. It is
|
||||||
|
// best effort: some filesystems do not support directory fsync, and failing to
|
||||||
|
// sync here is no worse than not trying.
|
||||||
|
func syncDir(dir string) {
|
||||||
|
f, err := os.Open(dir)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
_ = f.Sync()
|
||||||
|
}
|
||||||
|
|
||||||
func insertAll(db *sql.DB, version string, entries []address.Entry) error {
|
func insertAll(db *sql.DB, version string, entries []address.Entry) error {
|
||||||
tx, err := db.Begin()
|
tx, err := db.Begin()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -84,11 +138,15 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
|
|||||||
if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('version', ?)`, version); err != nil {
|
if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('version', ?)`, version); err != nil {
|
||||||
return fmt.Errorf("insert meta: %w", err)
|
return fmt.Errorf("insert meta: %w", err)
|
||||||
}
|
}
|
||||||
|
if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('schema_version', ?)`, SchemaVersion); err != nil {
|
||||||
|
return fmt.Errorf("insert schema version: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
stmt, err := tx.Prepare(`INSERT INTO address (
|
stmt, err := tx.Prepare(`INSERT INTO address (
|
||||||
id, address, district, ordinal, name_key, name_display, street_type, street_key,
|
id, address, district, city, city_key, ordinal, name_key, name_display,
|
||||||
house, house_frac, house_letter, building, building_frac, building_letter, structure
|
street_type, street_key, house, house_frac, house_letter, building,
|
||||||
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`)
|
building_frac, building_letter, structure
|
||||||
|
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare insert: %w", err)
|
return fmt.Errorf("prepare insert: %w", err)
|
||||||
}
|
}
|
||||||
@@ -101,8 +159,9 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
|
|||||||
id = fmt.Sprintf("row-%d", i)
|
id = fmt.Sprintf("row-%d", i)
|
||||||
}
|
}
|
||||||
if _, err := stmt.Exec(
|
if _, err := stmt.Exec(
|
||||||
id, e.Address, e.District, e.Ordinal, e.NameKey, e.NameDisplay, e.StreetType, e.StreetKey,
|
id, e.Address, e.District, e.City, e.CityKey, e.Ordinal, e.NameKey, e.NameDisplay,
|
||||||
e.House, e.HouseFrac, e.HouseLetter, e.Building, e.BuildingFrac, e.BuildingLetter, e.Structure,
|
e.StreetType, e.StreetKey, e.House, e.HouseFrac, e.HouseLetter, e.Building,
|
||||||
|
e.BuildingFrac, e.BuildingLetter, e.Structure,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return fmt.Errorf("insert %q: %w", e.Address, err)
|
return fmt.Errorf("insert %q: %w", e.Address, err)
|
||||||
}
|
}
|
||||||
@@ -114,26 +173,41 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Load opens path read-only and returns the data version and all entries.
|
// Load opens path read-only and returns the data version and all entries.
|
||||||
|
// It fails closed when the file was produced by an incompatible schema.
|
||||||
func Load(path string) (version string, entries []address.Entry, err error) {
|
func Load(path string) (version string, entries []address.Entry, err error) {
|
||||||
if _, err := os.Stat(path); err != nil {
|
if _, err := os.Stat(path); err != nil {
|
||||||
return "", nil, fmt.Errorf("stat db: %w", err)
|
return "", nil, fmt.Errorf("stat db: %w", err)
|
||||||
}
|
}
|
||||||
db, err := sql.Open("sqlite", "file:"+path+
|
uri, err := dsn(path, true)
|
||||||
"?mode=ro"+
|
if err != nil {
|
||||||
"&_pragma=query_only(1)"+
|
return "", nil, err
|
||||||
"&_pragma=busy_timeout(5000)")
|
}
|
||||||
|
db, err := sql.Open("sqlite", uri)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("open db: %w", err)
|
return "", nil, fmt.Errorf("open db: %w", err)
|
||||||
}
|
}
|
||||||
defer db.Close()
|
defer db.Close()
|
||||||
|
|
||||||
|
var schemaVersion string
|
||||||
|
if err := db.QueryRow(`SELECT value FROM meta WHERE key='schema_version'`).Scan(&schemaVersion); err != nil {
|
||||||
|
return "", nil, fmt.Errorf("read schema version (re-run addrkit seed): %w", err)
|
||||||
|
}
|
||||||
|
if schemaVersion != SchemaVersion {
|
||||||
|
return "", nil, fmt.Errorf(
|
||||||
|
"unsupported index schema %q (want %q); re-run addrkit seed",
|
||||||
|
schemaVersion, SchemaVersion,
|
||||||
|
)
|
||||||
|
}
|
||||||
if err := db.QueryRow(`SELECT value FROM meta WHERE key='version'`).Scan(&version); err != nil {
|
if err := db.QueryRow(`SELECT value FROM meta WHERE key='version'`).Scan(&version); err != nil {
|
||||||
return "", nil, fmt.Errorf("read version: %w", err)
|
return "", nil, fmt.Errorf("read version: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
rows, err := db.Query(`SELECT id, address, district, ordinal, name_key, name_display,
|
rows, err := db.Query(`SELECT id, address, district, city, city_key, ordinal,
|
||||||
street_type, street_key, house, house_frac, house_letter, building, building_frac,
|
name_key, name_display, street_type, street_key, house, house_frac,
|
||||||
building_letter, structure FROM address`)
|
house_letter, building, building_frac, building_letter, structure
|
||||||
|
FROM address
|
||||||
|
ORDER BY city_key, street_key, house, house_frac, house_letter,
|
||||||
|
building, building_frac, building_letter, structure, address, id`)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("query: %w", err)
|
return "", nil, fmt.Errorf("query: %w", err)
|
||||||
}
|
}
|
||||||
@@ -142,9 +216,10 @@ func Load(path string) (version string, entries []address.Entry, err error) {
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var e address.Entry
|
var e address.Entry
|
||||||
if err := rows.Scan(
|
if err := rows.Scan(
|
||||||
&e.ID, &e.Address, &e.District, &e.Ordinal, &e.NameKey, &e.NameDisplay,
|
&e.ID, &e.Address, &e.District, &e.City, &e.CityKey, &e.Ordinal,
|
||||||
&e.StreetType, &e.StreetKey, &e.House, &e.HouseFrac, &e.HouseLetter,
|
&e.NameKey, &e.NameDisplay, &e.StreetType, &e.StreetKey, &e.House,
|
||||||
&e.Building, &e.BuildingFrac, &e.BuildingLetter, &e.Structure,
|
&e.HouseFrac, &e.HouseLetter, &e.Building, &e.BuildingFrac,
|
||||||
|
&e.BuildingLetter, &e.Structure,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return "", nil, fmt.Errorf("scan: %w", err)
|
return "", nil, fmt.Errorf("scan: %w", err)
|
||||||
}
|
}
|
||||||
@@ -156,12 +231,6 @@ func Load(path string) (version string, entries []address.Entry, err error) {
|
|||||||
return version, entries, nil
|
return version, entries, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// removeDBFiles deletes a database file and its journal siblings.
|
|
||||||
func removeDBFiles(base string) {
|
|
||||||
_ = os.Remove(base)
|
|
||||||
removeSidecars(base)
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeSidecars deletes only the -wal/-shm companions of a database file.
|
// removeSidecars deletes only the -wal/-shm companions of a database file.
|
||||||
func removeSidecars(base string) {
|
func removeSidecars(base string) {
|
||||||
_ = os.Remove(base + "-wal")
|
_ = os.Remove(base + "-wal")
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
package registrystore
|
package registrystore
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"database/sql"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
@@ -11,11 +13,13 @@ func TestSaveLoadRoundTrip(t *testing.T) {
|
|||||||
path := filepath.Join(t.TempDir(), "registry.db")
|
path := filepath.Join(t.TempDir(), "registry.db")
|
||||||
|
|
||||||
addr := "Станционная улица 1/1"
|
addr := "Станционная улица 1/1"
|
||||||
p := address.Parse(addr)
|
parser := address.NewParser(address.NewDictionary([]address.City{{Name: "Новосибирск"}}), "")
|
||||||
|
p := parser.ParseInCity(addr, "Новосибирск")
|
||||||
entries := []address.Entry{{
|
entries := []address.Entry{{
|
||||||
ID: "a1", Address: addr, District: "Ленинский",
|
ID: "a1", Address: addr, District: "Ленинский",
|
||||||
|
City: p.City, CityKey: p.CityKey,
|
||||||
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey,
|
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey,
|
||||||
House: p.House, HouseFrac: p.HouseFrac,
|
House: p.House, HouseFrac: p.HouseFrac, Structure: "2",
|
||||||
}}
|
}}
|
||||||
|
|
||||||
if err := Save(path, "2026-10-09", entries); err != nil {
|
if err := Save(path, "2026-10-09", entries); err != nil {
|
||||||
@@ -35,6 +39,58 @@ func TestSaveLoadRoundTrip(t *testing.T) {
|
|||||||
if got[0].Address != addr || got[0].District != "Ленинский" || got[0].StreetKey != p.StreetKey {
|
if got[0].Address != addr || got[0].District != "Ленинский" || got[0].StreetKey != p.StreetKey {
|
||||||
t.Errorf("round-tripped entry mismatch: %+v", got[0])
|
t.Errorf("round-tripped entry mismatch: %+v", got[0])
|
||||||
}
|
}
|
||||||
|
if got[0].City != "Новосибирск" || got[0].CityKey != "новосибирск" || got[0].Structure != "2" {
|
||||||
|
t.Errorf("city/structure not round-tripped: %+v", got[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaveRejectsEmptyIndex(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "registry.db")
|
||||||
|
if err := Save(path, "v1", nil); err == nil {
|
||||||
|
t.Fatal("expected error for empty index")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadRejectsIncompatibleSchema(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "old.db")
|
||||||
|
uri, err := dsn(path, false)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("dsn: %v", err)
|
||||||
|
}
|
||||||
|
db, err := sql.Open("sqlite", uri)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("open: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)`); err != nil {
|
||||||
|
t.Fatalf("create meta: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`INSERT INTO meta(key, value) VALUES ('schema_version', '1')`); err != nil {
|
||||||
|
t.Fatalf("insert schema version: %v", err)
|
||||||
|
}
|
||||||
|
db.Close()
|
||||||
|
|
||||||
|
_, _, err = Load(path)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "unsupported index schema") {
|
||||||
|
t.Fatalf("Load error = %v, want unsupported schema", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPathWithSpecialCharacters guards the SQLite URI construction: a naive
|
||||||
|
// "file:" + path breaks on ?, # and % in the path.
|
||||||
|
func TestPathWithSpecialCharacters(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "registry ? # 100%.db")
|
||||||
|
entries := []address.Entry{{ID: "x", Address: "Первая улица 1"}}
|
||||||
|
|
||||||
|
if err := Save(path, "v1", entries); err != nil {
|
||||||
|
t.Fatalf("Save: %v", err)
|
||||||
|
}
|
||||||
|
version, got, err := Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load: %v", err)
|
||||||
|
}
|
||||||
|
if version != "v1" || len(got) != 1 {
|
||||||
|
t.Fatalf("version=%q entries=%+v", version, got)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestSaveIsAtomicOverwrite verifies that re-seeding replaces the previous data
|
// TestSaveIsAtomicOverwrite verifies that re-seeding replaces the previous data
|
||||||
|
|||||||
+87
-20
@@ -4,11 +4,15 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Config is the whole service configuration.
|
// Config is the whole service configuration.
|
||||||
@@ -21,25 +25,24 @@ type Config struct {
|
|||||||
LogLevel string `json:"log_level"`
|
LogLevel string `json:"log_level"`
|
||||||
|
|
||||||
Resolve ResolveConfig `json:"resolve"`
|
Resolve ResolveConfig `json:"resolve"`
|
||||||
Portal PortalConfig `json:"portal"`
|
|
||||||
LLM LLMConfig `json:"llm"`
|
// Cities is the dictionary of cities this deployment understands. A
|
||||||
|
// single-city deployment lists one city and may set DefaultCity so that
|
||||||
|
// queries like "ленина 5" work without naming it.
|
||||||
|
Cities []address.City `json:"cities"`
|
||||||
|
DefaultCity string `json:"default_city"`
|
||||||
|
|
||||||
|
LLM LLMConfig `json:"llm"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveConfig tunes the matching engine.
|
// ResolveConfig tunes the matching engine. Defaults come from
|
||||||
|
// match.DefaultOptions so there is exactly one source of truth.
|
||||||
type ResolveConfig struct {
|
type ResolveConfig struct {
|
||||||
FuzzyThreshold float64 `json:"fuzzy_threshold"`
|
FuzzyThreshold float64 `json:"fuzzy_threshold"`
|
||||||
AmbiguityMargin float64 `json:"ambiguity_margin"`
|
AmbiguityMargin float64 `json:"ambiguity_margin"`
|
||||||
MaxCandidates int `json:"max_candidates"`
|
MaxCandidates int `json:"max_candidates"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// PortalConfig configures the external outages portal (ArcGIS). When URL is
|
|
||||||
// empty the "events by house" module is disabled and only /v1/resolve works.
|
|
||||||
type PortalConfig struct {
|
|
||||||
URL string `json:"url"`
|
|
||||||
TimeoutSeconds int `json:"timeout_seconds"`
|
|
||||||
Retries int `json:"retries"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// LLMConfig configures the optional, disabled-by-default query rewriter.
|
// LLMConfig configures the optional, disabled-by-default query rewriter.
|
||||||
type LLMConfig struct {
|
type LLMConfig struct {
|
||||||
Enabled bool `json:"enabled"`
|
Enabled bool `json:"enabled"`
|
||||||
@@ -63,17 +66,14 @@ type LLMConfig struct {
|
|||||||
|
|
||||||
// Default returns a config with sane defaults for the non-required fields.
|
// Default returns a config with sane defaults for the non-required fields.
|
||||||
func Default() Config {
|
func Default() Config {
|
||||||
|
resolve := match.DefaultOptions()
|
||||||
return Config{
|
return Config{
|
||||||
Addr: ":6171",
|
Addr: ":6171",
|
||||||
LogLevel: "info",
|
LogLevel: "info",
|
||||||
Resolve: ResolveConfig{
|
Resolve: ResolveConfig{
|
||||||
FuzzyThreshold: 0.82,
|
FuzzyThreshold: resolve.FuzzyThreshold,
|
||||||
AmbiguityMargin: 0.03,
|
AmbiguityMargin: resolve.AmbiguityMargin,
|
||||||
MaxCandidates: 5,
|
MaxCandidates: resolve.MaxCandidates,
|
||||||
},
|
|
||||||
Portal: PortalConfig{
|
|
||||||
TimeoutSeconds: 2,
|
|
||||||
Retries: 1,
|
|
||||||
},
|
},
|
||||||
LLM: LLMConfig{
|
LLM: LLMConfig{
|
||||||
TimeoutSeconds: 3,
|
TimeoutSeconds: 3,
|
||||||
@@ -91,6 +91,9 @@ func Load(path string) (Config, error) {
|
|||||||
}
|
}
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
dec := json.NewDecoder(strings.NewReader(expandEnv(string(raw))))
|
dec := json.NewDecoder(strings.NewReader(expandEnv(string(raw))))
|
||||||
|
// A typo in a key (fuzzy_treshold, data_pth, portal, ...) must not silently
|
||||||
|
// fall back to a default. Fail closed and let the operator fix the config.
|
||||||
|
dec.DisallowUnknownFields()
|
||||||
if err := dec.Decode(&cfg); err != nil {
|
if err := dec.Decode(&cfg); err != nil {
|
||||||
return Config{}, fmt.Errorf("parse config: %w", err)
|
return Config{}, fmt.Errorf("parse config: %w", err)
|
||||||
}
|
}
|
||||||
@@ -107,6 +110,25 @@ func Load(path string) (Config, error) {
|
|||||||
return cfg, nil
|
return cfg, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LoadCitiesFile reads a standalone JSON array of cities. It is used by the
|
||||||
|
// seed CLI so that seeding and serving share the same city vocabulary.
|
||||||
|
func LoadCitiesFile(path string) ([]address.City, error) {
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read cities: %w", err)
|
||||||
|
}
|
||||||
|
var cities []address.City
|
||||||
|
dec := json.NewDecoder(bytes.NewReader(raw))
|
||||||
|
dec.DisallowUnknownFields()
|
||||||
|
if err := dec.Decode(&cities); err != nil {
|
||||||
|
return nil, fmt.Errorf("parse cities: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateCities(cities, ""); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return cities, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (c Config) validate() error {
|
func (c Config) validate() error {
|
||||||
if c.Addr == "" {
|
if c.Addr == "" {
|
||||||
return fmt.Errorf("config: addr is required")
|
return fmt.Errorf("config: addr is required")
|
||||||
@@ -114,11 +136,56 @@ func (c Config) validate() error {
|
|||||||
if c.DataPath == "" {
|
if c.DataPath == "" {
|
||||||
return fmt.Errorf("config: data_path is required")
|
return fmt.Errorf("config: data_path is required")
|
||||||
}
|
}
|
||||||
|
if c.Resolve.FuzzyThreshold <= 0 || c.Resolve.FuzzyThreshold > 1 {
|
||||||
|
return fmt.Errorf("config: resolve.fuzzy_threshold must be in (0, 1]")
|
||||||
|
}
|
||||||
|
if c.Resolve.AmbiguityMargin < 0 || c.Resolve.AmbiguityMargin > 1 {
|
||||||
|
return fmt.Errorf("config: resolve.ambiguity_margin must be in [0, 1]")
|
||||||
|
}
|
||||||
|
if c.Resolve.MaxCandidates < 1 {
|
||||||
|
return fmt.Errorf("config: resolve.max_candidates must be at least 1")
|
||||||
|
}
|
||||||
if c.LLM.Enabled && c.LLM.URL == "" {
|
if c.LLM.Enabled && c.LLM.URL == "" {
|
||||||
return fmt.Errorf("config: llm.url is required when llm.enabled is true")
|
return fmt.Errorf("config: llm.url is required when llm.enabled is true")
|
||||||
}
|
}
|
||||||
// portal.url is optional: without it the "events by house" module is
|
return validateCities(c.Cities, c.DefaultCity)
|
||||||
// disabled and only /v1/resolve is served.
|
}
|
||||||
|
|
||||||
|
// validateCities checks the city dictionary for the mistakes that would make
|
||||||
|
// matching silently ambiguous: empty names, duplicate cities and an alias
|
||||||
|
// claimed by two different cities.
|
||||||
|
func validateCities(cities []address.City, defaultCity string) error {
|
||||||
|
seenNames := make(map[string]string)
|
||||||
|
seenAliases := make(map[string]string)
|
||||||
|
|
||||||
|
for _, city := range cities {
|
||||||
|
name := strings.TrimSpace(city.Name)
|
||||||
|
key := address.NormalizeText(name)
|
||||||
|
if key == "" {
|
||||||
|
return fmt.Errorf("config: cities[].name must not be empty")
|
||||||
|
}
|
||||||
|
if prev, ok := seenNames[key]; ok {
|
||||||
|
return fmt.Errorf("config: duplicate city %q and %q", prev, name)
|
||||||
|
}
|
||||||
|
seenNames[key] = name
|
||||||
|
|
||||||
|
aliases := append([]string{name}, city.Aliases...)
|
||||||
|
for _, alias := range aliases {
|
||||||
|
norm := address.NormalizeText(alias)
|
||||||
|
if norm == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if owner, ok := seenAliases[norm]; ok && owner != key {
|
||||||
|
return fmt.Errorf("config: alias %q is claimed by both %q and %q", alias, owner, key)
|
||||||
|
}
|
||||||
|
seenAliases[norm] = key
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if defaultCity = strings.TrimSpace(defaultCity); defaultCity != "" && len(cities) > 0 {
|
||||||
|
if _, ok := seenNames[address.NormalizeText(defaultCity)]; !ok {
|
||||||
|
return fmt.Errorf("config: default_city %q is not listed in cities", defaultCity)
|
||||||
|
}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
)
|
)
|
||||||
|
|
||||||
func writeFile(t *testing.T, dir, name, content string) string {
|
func writeFile(t *testing.T, dir, name, content string) string {
|
||||||
@@ -19,7 +21,6 @@ func baseConfig(promptFields string) string {
|
|||||||
return `{
|
return `{
|
||||||
"addr": ":6171",
|
"addr": ":6171",
|
||||||
"data_path": "registry.db",
|
"data_path": "registry.db",
|
||||||
"portal": { "url": "http://portal" },
|
|
||||||
"llm": { "enabled": true, "url": "http://llm"` + promptFields + ` }
|
"llm": { "enabled": true, "url": "http://llm"` + promptFields + ` }
|
||||||
}`
|
}`
|
||||||
}
|
}
|
||||||
@@ -60,11 +61,90 @@ func TestLoadMissingSystemPromptFileFailsClosed(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPortalURLOptional(t *testing.T) {
|
func TestDefaultsMatchMatcher(t *testing.T) {
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
// No portal.url: the events module is disabled, the core still loads.
|
|
||||||
path := writeFile(t, dir, "config.json", `{"addr": ":6171", "data_path": "x.db"}`)
|
path := writeFile(t, dir, "config.json", `{"addr": ":6171", "data_path": "x.db"}`)
|
||||||
if _, err := Load(path); err != nil {
|
|
||||||
t.Fatalf("portal.url must be optional: %v", err)
|
cfg, err := Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load: %v", err)
|
||||||
|
}
|
||||||
|
want := match.DefaultOptions()
|
||||||
|
if cfg.Resolve.FuzzyThreshold != want.FuzzyThreshold ||
|
||||||
|
cfg.Resolve.AmbiguityMargin != want.AmbiguityMargin ||
|
||||||
|
cfg.Resolve.MaxCandidates != want.MaxCandidates {
|
||||||
|
t.Fatalf("config defaults %+v do not match matcher defaults %+v", cfg.Resolve, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectsBadResolveRanges(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
cases := map[string]string{
|
||||||
|
"threshold": `{"addr":":6171","data_path":"x.db","resolve":{"fuzzy_threshold":0}}`,
|
||||||
|
"margin": `{"addr":":6171","data_path":"x.db","resolve":{"ambiguity_margin":2}}`,
|
||||||
|
"candidates": `{"addr":":6171","data_path":"x.db","resolve":{"max_candidates":0}}`,
|
||||||
|
}
|
||||||
|
for name, body := range cases {
|
||||||
|
path := writeFile(t, dir, name+".json", body)
|
||||||
|
if _, err := Load(path); err == nil {
|
||||||
|
t.Errorf("%s: expected validation error", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCitiesAndDefaultCity(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := writeFile(t, dir, "config.json", `{
|
||||||
|
"addr": ":6171", "data_path": "x.db",
|
||||||
|
"cities": [
|
||||||
|
{"name": "Новосибирск", "aliases": ["нск", "новосибирская область"]},
|
||||||
|
{"name": "Бердск"}
|
||||||
|
],
|
||||||
|
"default_city": "Новосибирск"
|
||||||
|
}`)
|
||||||
|
cfg, err := Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load: %v", err)
|
||||||
|
}
|
||||||
|
if len(cfg.Cities) != 2 || cfg.DefaultCity != "Новосибирск" {
|
||||||
|
t.Fatalf("cities = %+v, default = %q", cfg.Cities, cfg.DefaultCity)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectsUnknownDefaultCity(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := writeFile(t, dir, "config.json", `{
|
||||||
|
"addr": ":6171", "data_path": "x.db",
|
||||||
|
"cities": [{"name": "Бердск"}],
|
||||||
|
"default_city": "Новосибирск"
|
||||||
|
}`)
|
||||||
|
if _, err := Load(path); err == nil {
|
||||||
|
t.Fatal("expected error for default_city absent from cities")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCitiesFile(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := writeFile(t, dir, "cities.json", `[
|
||||||
|
{"name": "Новосибирск", "aliases": ["нск"]},
|
||||||
|
{"name": "Бердск"}
|
||||||
|
]`)
|
||||||
|
cities, err := LoadCitiesFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadCitiesFile: %v", err)
|
||||||
|
}
|
||||||
|
if len(cities) != 2 || cities[0].Name != "Новосибирск" {
|
||||||
|
t.Fatalf("cities = %+v", cities)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectsUnknownConfigFields(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
// "portal" was removed with the events module; silently ignoring it would
|
||||||
|
// hide a stale deployment config.
|
||||||
|
path := writeFile(t, dir, "config.json",
|
||||||
|
`{"addr":":6171","data_path":"x.db","portal":{"url":"http://example"}}`)
|
||||||
|
if _, err := Load(path); err == nil {
|
||||||
|
t.Fatal("expected error for unknown config field")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,141 @@
|
|||||||
|
package address
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func testDictionary() *Dictionary {
|
||||||
|
return NewDictionary([]City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск", "новосиб", "новосибирская область", "нсо"}},
|
||||||
|
{Name: "Бердск"},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseCityFromAliases(t *testing.T) {
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
raw string
|
||||||
|
city string
|
||||||
|
nameKey string
|
||||||
|
}{
|
||||||
|
{"нск, 2-я Станционная улица 1", "Новосибирск", "станционная"},
|
||||||
|
{"НСК Ленина 5", "Новосибирск", "ленина"},
|
||||||
|
{"Новосибирская область, Станционная 1", "Новосибирск", "станционная"},
|
||||||
|
{"Бердск, Ленина 5", "Бердск", "ленина"},
|
||||||
|
{"г. Новосибирск, ул. Ленина 5", "Новосибирск", "ленина"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
got := parser.Parse(tt.raw)
|
||||||
|
if got.City != tt.city {
|
||||||
|
t.Errorf("%q: City = %q, want %q", tt.raw, got.City, tt.city)
|
||||||
|
}
|
||||||
|
if got.NameKey != tt.nameKey {
|
||||||
|
t.Errorf("%q: NameKey = %q, want %q", tt.raw, got.NameKey, tt.nameKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseCityIsPartOfStreetKey(t *testing.T) {
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
|
||||||
|
nsk := parser.Parse("Новосибирск, Ленина 5")
|
||||||
|
berdsk := parser.Parse("Бердск, Ленина 5")
|
||||||
|
if nsk.StreetKey == berdsk.StreetKey {
|
||||||
|
t.Fatalf("same street key for different cities: %q", nsk.StreetKey)
|
||||||
|
}
|
||||||
|
if nsk.CityKey != "новосибирск" || berdsk.CityKey != "бердск" {
|
||||||
|
t.Fatalf("city keys = %q / %q", nsk.CityKey, berdsk.CityKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseDefaultCity(t *testing.T) {
|
||||||
|
parser := NewParser(testDictionary(), "Новосибирск")
|
||||||
|
got := parser.Parse("Ленина 5")
|
||||||
|
if got.City != "Новосибирск" || got.CityKey != "новосибирск" {
|
||||||
|
t.Fatalf("City = %q / %q", got.City, got.CityKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The default city is also recognised when the user spells it out, even
|
||||||
|
// if the dictionary contains only aliases.
|
||||||
|
explicit := NewParser(DefaultDictionary(), "Новосибирск").Parse("г. Новосибирск, Ленина 5")
|
||||||
|
if explicit.City != "Новосибирск" || explicit.NameKey != "ленина" {
|
||||||
|
t.Fatalf("explicit default city: %+v", explicit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseInCityOverridesDetectedCity(t *testing.T) {
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
got := parser.ParseInCity("Бердск, Ленина 5", "Новосибирск")
|
||||||
|
if got.City != "Новосибирск" {
|
||||||
|
t.Fatalf("City = %q, want forced Новосибирск", got.City)
|
||||||
|
}
|
||||||
|
if got.NameKey != "ленина" {
|
||||||
|
t.Fatalf("NameKey = %q", got.NameKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseWithoutCityKeepsCityKeyEmpty(t *testing.T) {
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
got := parser.Parse("Ленина 5")
|
||||||
|
if got.City != "" || got.CityKey != "" {
|
||||||
|
t.Fatalf("City = %q / %q, want empty", got.City, got.CityKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeTextStillLeavesUnknownCityTokens(t *testing.T) {
|
||||||
|
// A city that is not in the dictionary must not be silently dropped: the
|
||||||
|
// resolver should fail rather than guess another city.
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
got := parser.Parse("Томск, Ленина 5")
|
||||||
|
if got.City != "" {
|
||||||
|
t.Fatalf("City = %q, want empty for unknown city", got.City)
|
||||||
|
}
|
||||||
|
if got.NameKey == "ленина" {
|
||||||
|
t.Fatalf("unknown city token disappeared from the street name")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMatchCity(t *testing.T) {
|
||||||
|
dict := testDictionary()
|
||||||
|
cases := []struct {
|
||||||
|
query string
|
||||||
|
want string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{"нск", "Новосибирск", true},
|
||||||
|
{"г. Новосибирск", "Новосибирск", true},
|
||||||
|
{"Новосибирская область", "Новосибирск", true},
|
||||||
|
{"Бердск", "Бердск", true},
|
||||||
|
{"Бердск, Ленина 5", "", false},
|
||||||
|
{"Томск", "", false},
|
||||||
|
}
|
||||||
|
for _, tt := range cases {
|
||||||
|
got, ok := dict.MatchCity(tt.query)
|
||||||
|
if ok != tt.ok || got.Name != tt.want {
|
||||||
|
t.Errorf("MatchCity(%q) = %+v, %v; want %q, %v", tt.query, got, ok, tt.want, tt.ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func FuzzParse(f *testing.F) {
|
||||||
|
for _, seed := range []string{
|
||||||
|
"",
|
||||||
|
"2-я Станционная улица 1/1 стр.",
|
||||||
|
"нск, Ленина 5",
|
||||||
|
"Бердск, Ленина 5 к2",
|
||||||
|
"г. Новосибирск, ул. Кошурникова, д. 11/1 кв. 5",
|
||||||
|
"\x00\xff",
|
||||||
|
} {
|
||||||
|
f.Add(seed)
|
||||||
|
}
|
||||||
|
parser := NewParser(testDictionary(), "")
|
||||||
|
f.Fuzz(func(t *testing.T, raw string) {
|
||||||
|
first := parser.Parse(raw)
|
||||||
|
second := parser.Parse(raw)
|
||||||
|
if first.StreetKey != second.StreetKey {
|
||||||
|
t.Fatalf("non-deterministic street key for %q: %q vs %q", raw, first.StreetKey, second.StreetKey)
|
||||||
|
}
|
||||||
|
if NormalizeText(raw) != NormalizeText(NormalizeText(raw)) {
|
||||||
|
t.Fatalf("NormalizeText is not idempotent for %q", raw)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -0,0 +1,284 @@
|
|||||||
|
package address
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// City is one entry of the configurable city dictionary.
|
||||||
|
//
|
||||||
|
// Aliases are additional surface forms that refer to the city: common
|
||||||
|
// abbreviations ("нск"), the region name ("новосибирская область") and so on.
|
||||||
|
// The canonical Name is always accepted as an alias too. Matching is
|
||||||
|
// token-based, case-insensitive and "ё"-insensitive, so aliases should be
|
||||||
|
// written naturally.
|
||||||
|
type City struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Aliases []string `json:"aliases,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dictionary is the immutable, deployment-specific vocabulary of the parser:
|
||||||
|
// cities, filler words, street types and ordinals.
|
||||||
|
//
|
||||||
|
// Keeping this data in a value instead of package-level maps makes the parser
|
||||||
|
// reusable across deployments: a service in Novosibirsk and a service in Tomsk
|
||||||
|
// can run the same code with different city lists, without recompilation.
|
||||||
|
type Dictionary struct {
|
||||||
|
cities []cityAlias
|
||||||
|
|
||||||
|
filler map[string]struct{}
|
||||||
|
streetTypes map[string]string
|
||||||
|
ordinalWords map[string]string
|
||||||
|
ordinalSuffixes map[string]struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// cityAlias is one token sequence that resolves to a city.
|
||||||
|
type cityAlias struct {
|
||||||
|
tokens []string
|
||||||
|
name string
|
||||||
|
key string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewDictionary builds a dictionary with the built-in federal vocabulary and
|
||||||
|
// the supplied cities. The input slice is copied; mutating it later does not
|
||||||
|
// affect the dictionary.
|
||||||
|
func NewDictionary(cities []City) *Dictionary {
|
||||||
|
d := &Dictionary{
|
||||||
|
filler: cloneSet(defaultFillerWords),
|
||||||
|
streetTypes: cloneMap(defaultStreetTypes),
|
||||||
|
ordinalWords: cloneMap(defaultOrdinalWords),
|
||||||
|
ordinalSuffixes: cloneSet(defaultOrdinalSuffixes),
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := make(map[string]struct{})
|
||||||
|
for _, c := range cities {
|
||||||
|
name := strings.TrimSpace(c.Name)
|
||||||
|
cityKey := NormalizeText(name)
|
||||||
|
if name == "" || cityKey == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
aliases := append([]string{name}, c.Aliases...)
|
||||||
|
for _, alias := range aliases {
|
||||||
|
tokens := strings.Fields(NormalizeText(alias))
|
||||||
|
if len(tokens) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
dedup := cityKey + "|" + strings.Join(tokens, " ")
|
||||||
|
if _, ok := seen[dedup]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[dedup] = struct{}{}
|
||||||
|
d.cities = append(d.cities, cityAlias{
|
||||||
|
tokens: tokens,
|
||||||
|
name: name,
|
||||||
|
key: cityKey,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Longest aliases win, so "новосибирская область" is preferred over a
|
||||||
|
// hypothetical shorter "новосибирская". The key is the tie-breaker that
|
||||||
|
// keeps the order deterministic when two cities share an alias.
|
||||||
|
d.sortCities()
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
|
||||||
|
// sortCities orders aliases by specificity: more tokens first, then longer
|
||||||
|
// text, then key. The order decides which city wins when aliases overlap.
|
||||||
|
func (d *Dictionary) sortCities() {
|
||||||
|
sort.SliceStable(d.cities, func(i, j int) bool {
|
||||||
|
a, b := d.cities[i], d.cities[j]
|
||||||
|
if len(a.tokens) != len(b.tokens) {
|
||||||
|
return len(a.tokens) > len(b.tokens)
|
||||||
|
}
|
||||||
|
as, bs := strings.Join(a.tokens, " "), strings.Join(b.tokens, " ")
|
||||||
|
if len(as) != len(bs) {
|
||||||
|
return len(as) > len(bs)
|
||||||
|
}
|
||||||
|
if a.key != b.key {
|
||||||
|
return a.key < b.key
|
||||||
|
}
|
||||||
|
return as < bs
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// withDefaultCity returns a dictionary that also recognises name as a city.
|
||||||
|
// It lets a single-city deployment use "-city Новосибирск" (or default_city)
|
||||||
|
// without duplicating the canonical name in the aliases list.
|
||||||
|
func (d *Dictionary) withDefaultCity(name string) *Dictionary {
|
||||||
|
key := NormalizeText(name)
|
||||||
|
if key == "" {
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
for _, a := range d.cities {
|
||||||
|
if a.key == key {
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out := *d
|
||||||
|
out.cities = append(append([]cityAlias(nil), d.cities...), cityAlias{
|
||||||
|
tokens: strings.Fields(key),
|
||||||
|
name: name,
|
||||||
|
key: key,
|
||||||
|
})
|
||||||
|
out.sortCities()
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultDictionary returns the built-in vocabulary without cities. It is the
|
||||||
|
// parser used by address.Parse for callers that do not need city awareness.
|
||||||
|
func DefaultDictionary() *Dictionary {
|
||||||
|
return NewDictionary(nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// MatchCity reports whether query consists only of a city name/alias (plus
|
||||||
|
// generic filler such as "г.") and returns the canonical city. It is the
|
||||||
|
// city-only counterpart of Parse, useful for projects that normalize city
|
||||||
|
// names without a street.
|
||||||
|
func (d *Dictionary) MatchCity(query string) (City, bool) {
|
||||||
|
name, _, rest, ok := d.matchCity(strings.Fields(NormalizeText(query)))
|
||||||
|
if !ok {
|
||||||
|
return City{}, false
|
||||||
|
}
|
||||||
|
for _, tok := range rest {
|
||||||
|
if !d.isFiller(tok) {
|
||||||
|
return City{}, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return City{Name: name}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// matchCity finds the longest city alias in tokens and returns the city plus
|
||||||
|
// the tokens with the alias removed. Matching is contiguous on token
|
||||||
|
// boundaries, so "нск" never matches inside another word.
|
||||||
|
func (d *Dictionary) matchCity(tokens []string) (name, key string, rest []string, ok bool) {
|
||||||
|
for _, alias := range d.cities {
|
||||||
|
if idx := indexTokens(tokens, alias.tokens); idx >= 0 {
|
||||||
|
rest = make([]string, 0, len(tokens)-len(alias.tokens))
|
||||||
|
rest = append(rest, tokens[:idx]...)
|
||||||
|
rest = append(rest, tokens[idx+len(alias.tokens):]...)
|
||||||
|
return alias.name, alias.key, rest, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", "", tokens, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// indexTokens returns the start of the first contiguous occurrence of want in
|
||||||
|
// tokens, or -1.
|
||||||
|
func indexTokens(tokens, want []string) int {
|
||||||
|
if len(want) == 0 || len(want) > len(tokens) {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
for i := 0; i+len(want) <= len(tokens); i++ {
|
||||||
|
matched := true
|
||||||
|
for j := range want {
|
||||||
|
if tokens[i+j] != want[j] {
|
||||||
|
matched = false
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matched {
|
||||||
|
return i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *Dictionary) isFiller(tok string) bool {
|
||||||
|
_, ok := d.filler[tok]
|
||||||
|
return ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *Dictionary) streetType(tok string) (string, bool) {
|
||||||
|
t, ok := d.streetTypes[tok]
|
||||||
|
return t, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *Dictionary) ordinalWord(tok string) (string, bool) {
|
||||||
|
o, ok := d.ordinalWords[tok]
|
||||||
|
return o, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *Dictionary) isOrdinalSuffix(tok string) bool {
|
||||||
|
_, ok := d.ordinalSuffixes[tok]
|
||||||
|
return ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneSet(src map[string]bool) map[string]struct{} {
|
||||||
|
dst := make(map[string]struct{}, len(src))
|
||||||
|
for k, v := range src {
|
||||||
|
if v {
|
||||||
|
dst[k] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return dst
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneMap(src map[string]string) map[string]string {
|
||||||
|
dst := make(map[string]string, len(src))
|
||||||
|
for k, v := range src {
|
||||||
|
dst[k] = v
|
||||||
|
}
|
||||||
|
return dst
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultFillerWords are tokens that carry no street-name meaning and that
|
||||||
|
// people routinely include ("г. Новосибирск", "улица ... дом 5", "... область").
|
||||||
|
//
|
||||||
|
// City names are deliberately NOT listed here: they come from Dictionary, so
|
||||||
|
// the same code can serve any city (or many cities) without edits.
|
||||||
|
var defaultFillerWords = map[string]bool{
|
||||||
|
"город": true, "г": true,
|
||||||
|
"область": true, "обл": true,
|
||||||
|
"район": true, "дом": true, "д": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultStreetTypes maps a surface form (including common abbreviations) to a
|
||||||
|
// single canonical street type. The type is informational for matching (we
|
||||||
|
// strip it from the key) but useful for disambiguation and display.
|
||||||
|
var defaultStreetTypes = map[string]string{
|
||||||
|
"ул": "улица", "улица": "улица",
|
||||||
|
"пер": "переулок", "пер-к": "переулок", "переулок": "переулок",
|
||||||
|
"пр": "проезд", "проезд": "проезд",
|
||||||
|
"пр-т": "проспект", "просп": "проспект", "проспект": "проспект",
|
||||||
|
"ш": "шоссе", "шоссе": "шоссе",
|
||||||
|
"мкр": "микрорайон", "микрорайон": "микрорайон",
|
||||||
|
"туп": "тупик", "тупик": "тупик",
|
||||||
|
"спуск": "спуск",
|
||||||
|
"б-р": "бульвар", "бул": "бульвар", "бульвар": "бульвар",
|
||||||
|
"пл": "площадь", "площадь": "площадь",
|
||||||
|
"пос": "поселок", "поселок": "поселок",
|
||||||
|
"тракт": "тракт",
|
||||||
|
"аллея": "аллея",
|
||||||
|
"наб": "набережная", "набережная": "набережная",
|
||||||
|
"кольцо": "кольцо",
|
||||||
|
"просека": "просека",
|
||||||
|
"квартал": "квартал",
|
||||||
|
"территория": "территория",
|
||||||
|
"дорога": "дорога",
|
||||||
|
"разъезд": "разъезд",
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultOrdinalWords maps spelled-out ordinals to their digit form. Only the
|
||||||
|
// forms that actually occur in practice are listed; unknown words are treated
|
||||||
|
// as part of the street name.
|
||||||
|
var defaultOrdinalWords = map[string]string{
|
||||||
|
"первый": "1", "первая": "1", "первое": "1",
|
||||||
|
"второй": "2", "вторая": "2", "второе": "2",
|
||||||
|
"третий": "3", "третья": "3", "третье": "3",
|
||||||
|
"четвертый": "4", "четвертая": "4", "четвертое": "4",
|
||||||
|
"пятый": "5", "пятая": "5", "пятое": "5",
|
||||||
|
"шестой": "6", "шестая": "6", "шестое": "6",
|
||||||
|
"седьмой": "7", "седьмая": "7", "седьмое": "7",
|
||||||
|
"восьмой": "8", "восьмая": "8", "восьмое": "8",
|
||||||
|
"девятый": "9", "девятая": "9", "девятое": "9",
|
||||||
|
"десятый": "10", "десятая": "10", "десятое": "10",
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultOrdinalSuffixes are the gender/number endings that may follow an
|
||||||
|
// ordinal digit when it is written as a separate token ("2 я", "10 й").
|
||||||
|
var defaultOrdinalSuffixes = map[string]bool{
|
||||||
|
"я": true, "й": true, "е": true,
|
||||||
|
"ая": true, "ый": true, "ой": true, "ое": true,
|
||||||
|
"ий": true, "ья": true, "ье": true, "ьи": true,
|
||||||
|
}
|
||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
var fold = cases.Fold()
|
var fold = cases.Fold()
|
||||||
|
|
||||||
// reTrailingParenthetical matches a trailing "(...)" group, e.g. the district
|
// reTrailingParenthetical matches a trailing "(...)" group, e.g. the district
|
||||||
// suffix that the outages portal appends: "Медкадры улица, 1 (Заельцовский
|
// suffix that source systems append: "Медкадры улица, 1 (Заельцовский
|
||||||
// район)". Such administrative noise must not be treated as part of the house.
|
// район)". Such administrative noise must not be treated as part of the house.
|
||||||
var reTrailingParenthetical = regexp.MustCompile(`\s*\([^)]*\)\s*$`)
|
var reTrailingParenthetical = regexp.MustCompile(`\s*\([^)]*\)\s*$`)
|
||||||
|
|
||||||
|
|||||||
@@ -24,42 +24,121 @@ var (
|
|||||||
// Trailing apartment part: " кв 5", " квартира 12", " кв. 7а". Removed
|
// Trailing apartment part: " кв 5", " квартира 12", " кв. 7а". Removed
|
||||||
// before house parsing so it cannot leak into the street name.
|
// before house parsing so it cannot leak into the street name.
|
||||||
reApartment = regexp.MustCompile(`(?:^|\s)(?:квартира|кв)\.?(?:\s+\d+[а-я]?)?\s*$`)
|
reApartment = regexp.MustCompile(`(?:^|\s)(?:квартира|кв)\.?(?:\s+\d+[а-я]?)?\s*$`)
|
||||||
|
|
||||||
|
// Spoken fractions: "18 дробь 1" is the same house unit as "18/1".
|
||||||
|
reFractionWord = regexp.MustCompile(`\s+(?:дробь|дробью)\s+`)
|
||||||
)
|
)
|
||||||
|
|
||||||
// Parse decomposes a free-form address into its structured components. It is
|
// Parser turns free-form address strings into structured addresses using a
|
||||||
// total: unparseable input still yields a non-nil ParsedAddress whose street
|
// deployment-specific Dictionary. It is immutable and safe for concurrent use.
|
||||||
// part is best-effort and whose HasHouse is false.
|
type Parser struct {
|
||||||
|
dict *Dictionary
|
||||||
|
defaultCityName string
|
||||||
|
defaultCityKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewParser builds a parser. defaultCity is used when the query does not name a
|
||||||
|
// city and the dictionary has no match; pass an empty string for deployments
|
||||||
|
// where the city must be present in every query.
|
||||||
|
func NewParser(dict *Dictionary, defaultCity string) *Parser {
|
||||||
|
if dict == nil {
|
||||||
|
dict = DefaultDictionary()
|
||||||
|
}
|
||||||
|
p := &Parser{}
|
||||||
|
if name := strings.TrimSpace(defaultCity); name != "" {
|
||||||
|
dict = dict.withDefaultCity(name)
|
||||||
|
p.defaultCityName = name
|
||||||
|
p.defaultCityKey = NormalizeText(name)
|
||||||
|
}
|
||||||
|
p.dict = dict
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultParser is used by the package-level Parse convenience function. It has
|
||||||
|
// no cities and is therefore fully backward compatible: callers that need city
|
||||||
|
// awareness construct their own Parser.
|
||||||
|
var defaultParser = NewParser(DefaultDictionary(), "")
|
||||||
|
|
||||||
|
// DefaultParser returns the shared city-less parser.
|
||||||
|
func DefaultParser() *Parser {
|
||||||
|
return defaultParser
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse decomposes a free-form address with the default (city-less) parser.
|
||||||
func Parse(raw string) ParsedAddress {
|
func Parse(raw string) ParsedAddress {
|
||||||
|
return defaultParser.Parse(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse decomposes a free-form address into its structured components. It is
|
||||||
|
// total: unparseable input still yields a usable ParsedAddress whose street
|
||||||
|
// part is best-effort and whose HasHouse is false.
|
||||||
|
func (p *Parser) Parse(raw string) ParsedAddress {
|
||||||
|
return p.parse(raw, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseInCity parses raw and forces cityName as the result's city. It is used
|
||||||
|
// by the seed pipeline, where the city is an explicit field of the source row
|
||||||
|
// and must win over whatever the address text contains.
|
||||||
|
func (p *Parser) ParseInCity(raw, cityName string) ParsedAddress {
|
||||||
|
return p.parse(raw, cityName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// MatchCity reports whether query is a city name/alias and returns its
|
||||||
|
// canonical form. It does not guess a city from an arbitrary address.
|
||||||
|
func (p *Parser) MatchCity(query string) (City, bool) {
|
||||||
|
return p.dict.MatchCity(query)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Parser) parse(raw, forcedCity string) ParsedAddress {
|
||||||
normalized := NormalizeText(raw)
|
normalized := NormalizeText(raw)
|
||||||
// A trailing apartment ("кв 5") is not part of the address identity.
|
// A trailing apartment ("кв 5") is not part of the address identity.
|
||||||
normalized = strings.TrimSpace(reApartment.ReplaceAllString(normalized, ""))
|
normalized = strings.TrimSpace(reApartment.ReplaceAllString(normalized, ""))
|
||||||
p := ParsedAddress{Raw: raw, Normalized: normalized}
|
// Collapse the spoken fraction marker to the canonical separator before
|
||||||
|
// the right-to-left house parser runs: "18 дробь 1" -> "18/1".
|
||||||
|
normalized = reFractionWord.ReplaceAllString(normalized, "/")
|
||||||
|
parsed := ParsedAddress{Raw: raw, Normalized: normalized}
|
||||||
|
|
||||||
rest := normalized
|
rest := normalized
|
||||||
|
if cityName, cityKey, restTokens, ok := p.dict.matchCity(strings.Fields(rest)); ok {
|
||||||
|
parsed.City = cityName
|
||||||
|
parsed.CityKey = cityKey
|
||||||
|
rest = strings.Join(restTokens, " ")
|
||||||
|
}
|
||||||
|
if forcedCity = strings.TrimSpace(forcedCity); forcedCity != "" {
|
||||||
|
parsed.City = forcedCity
|
||||||
|
parsed.CityKey = NormalizeText(forcedCity)
|
||||||
|
} else if parsed.City == "" && p.defaultCityName != "" {
|
||||||
|
// A single-city deployment can omit the city in queries. In a
|
||||||
|
// multi-city deployment default_city should stay empty: guessing a
|
||||||
|
// city is exactly the kind of silent error this service avoids.
|
||||||
|
parsed.City = p.defaultCityName
|
||||||
|
parsed.CityKey = p.defaultCityKey
|
||||||
|
}
|
||||||
|
|
||||||
if g, start, ok := matchSuffix(reStructure, rest); ok {
|
if g, start, ok := matchSuffix(reStructure, rest); ok {
|
||||||
p.Structure = trimLeadingZeros(g[1])
|
parsed.Structure = trimLeadingZeros(g[1])
|
||||||
rest = strings.TrimSpace(rest[:start])
|
rest = strings.TrimSpace(rest[:start])
|
||||||
}
|
}
|
||||||
if g, start, ok := matchSuffix(reBuilding, rest); ok {
|
if g, start, ok := matchSuffix(reBuilding, rest); ok {
|
||||||
// The regex also matches a bare "к"; only accept it when a digit or a
|
// The regex also matches a bare "к"; only accept it when a digit or a
|
||||||
// letter actually follows, otherwise it is not a corpus marker.
|
// letter actually follows, otherwise it is not a corpus marker.
|
||||||
if g[1] != "" || g[2] != "" {
|
if g[1] != "" || g[2] != "" {
|
||||||
p.Building = trimLeadingZeros(g[1])
|
parsed.Building = trimLeadingZeros(g[1])
|
||||||
p.BuildingLetter = firstNonEmpty(g[2], g[4])
|
parsed.BuildingLetter = firstNonEmpty(g[2], g[4])
|
||||||
p.BuildingFrac = trimLeadingZeros(g[3])
|
parsed.BuildingFrac = trimLeadingZeros(g[3])
|
||||||
rest = strings.TrimSpace(rest[:start])
|
rest = strings.TrimSpace(rest[:start])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if g, start, ok := matchSuffix(reHouse, rest); ok {
|
if g, start, ok := matchSuffix(reHouse, rest); ok {
|
||||||
p.House = trimLeadingZeros(g[1])
|
parsed.House = trimLeadingZeros(g[1])
|
||||||
p.HouseLetter = firstNonEmpty(g[2], g[4])
|
parsed.HouseLetter = firstNonEmpty(g[2], g[4])
|
||||||
p.HouseFrac = trimLeadingZeros(g[3])
|
parsed.HouseFrac = trimLeadingZeros(g[3])
|
||||||
p.HasHouse = true
|
parsed.HasHouse = true
|
||||||
rest = strings.TrimSpace(rest[:start])
|
rest = strings.TrimSpace(rest[:start])
|
||||||
}
|
}
|
||||||
|
|
||||||
parseStreet(&p, rest)
|
p.parseStreet(&parsed, rest)
|
||||||
return p
|
return parsed
|
||||||
}
|
}
|
||||||
|
|
||||||
// firstNonEmpty returns the first non-empty argument. House/building letters
|
// firstNonEmpty returns the first non-empty argument. House/building letters
|
||||||
@@ -73,8 +152,8 @@ func firstNonEmpty(values ...string) string {
|
|||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseStreet fills the street-related fields of p from the leftover string
|
// parseStreet fills the street-related fields of out from the leftover string
|
||||||
// (the address with its house block removed).
|
// (the address with its city and house block removed).
|
||||||
//
|
//
|
||||||
// All recognised type tokens are removed from the name. This matters for
|
// All recognised type tokens are removed from the name. This matters for
|
||||||
// registry rows such as "Гусинобродский Тракт улица 21", where both "Тракт"
|
// registry rows such as "Гусинобродский Тракт улица 21", where both "Тракт"
|
||||||
@@ -82,7 +161,7 @@ func firstNonEmpty(values ...string) string {
|
|||||||
// says "Гусинобродский 21". The first type token becomes the canonical type.
|
// says "Гусинобродский 21". The first type token becomes the canonical type.
|
||||||
// If removing them would leave the name empty (a street literally called
|
// If removing them would leave the name empty (a street literally called
|
||||||
// "Набережная"), the type token is kept as the name.
|
// "Набережная"), the type token is kept as the name.
|
||||||
func parseStreet(p *ParsedAddress, s string) {
|
func (p *Parser) parseStreet(out *ParsedAddress, s string) {
|
||||||
tokens := strings.Fields(s)
|
tokens := strings.Fields(s)
|
||||||
name := make([]string, 0, len(tokens))
|
name := make([]string, 0, len(tokens))
|
||||||
streetType := ""
|
streetType := ""
|
||||||
@@ -90,10 +169,10 @@ func parseStreet(p *ParsedAddress, s string) {
|
|||||||
for _, tok := range tokens {
|
for _, tok := range tokens {
|
||||||
// Filler tokens ("г", "город", "дом", "область", ...) never belong to
|
// Filler tokens ("г", "город", "дом", "область", ...) never belong to
|
||||||
// the street name.
|
// the street name.
|
||||||
if fillerWords[tok] {
|
if p.dict.isFiller(tok) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if t, ok := streetTypes[tok]; ok {
|
if t, ok := p.dict.streetType(tok); ok {
|
||||||
if streetType == "" {
|
if streetType == "" {
|
||||||
streetType = t
|
streetType = t
|
||||||
}
|
}
|
||||||
@@ -102,55 +181,55 @@ func parseStreet(p *ParsedAddress, s string) {
|
|||||||
name = append(name, tok)
|
name = append(name, tok)
|
||||||
}
|
}
|
||||||
|
|
||||||
p.StreetType = streetType
|
out.StreetType = streetType
|
||||||
if len(name) == 0 && streetType != "" {
|
if len(name) == 0 && streetType != "" {
|
||||||
name = []string{streetType}
|
name = []string{streetType}
|
||||||
}
|
}
|
||||||
if len(name) == 0 {
|
if len(name) == 0 {
|
||||||
p.StreetKey = streetKey("", "")
|
out.StreetKey = streetKey(out.CityKey, "", "")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
name = extractOrdinal(p, name)
|
name = p.extractOrdinal(out, name)
|
||||||
p.NameDisplay = strings.Join(name, " ")
|
out.NameDisplay = strings.Join(name, " ")
|
||||||
|
|
||||||
sorted := append([]string(nil), name...)
|
sorted := append([]string(nil), name...)
|
||||||
sort.Strings(sorted)
|
sort.Strings(sorted)
|
||||||
p.NameKey = strings.Join(sorted, " ")
|
out.NameKey = strings.Join(sorted, " ")
|
||||||
p.StreetKey = streetKey(p.Ordinal, p.NameKey)
|
out.StreetKey = streetKey(out.CityKey, out.Ordinal, out.NameKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
// extractOrdinal recognises an ordinal and removes it from the name tokens.
|
// extractOrdinal recognises an ordinal and removes it from the name tokens.
|
||||||
// It handles both the usual leading form ("2-я Сухарная") and an ordinal that
|
// It handles both the usual leading form ("2-я Сухарная") and an ordinal that
|
||||||
// appears after the name ("Сухарная 2-я улица"), which both people and models
|
// appears after the name ("Сухарная 2-я улица"), which both people and models
|
||||||
// produce.
|
// produce.
|
||||||
func extractOrdinal(p *ParsedAddress, tokens []string) []string {
|
func (p *Parser) extractOrdinal(out *ParsedAddress, tokens []string) []string {
|
||||||
if len(tokens) == 0 {
|
if len(tokens) == 0 {
|
||||||
return tokens
|
return tokens
|
||||||
}
|
}
|
||||||
|
|
||||||
// Leading forms first: they are by far the most common.
|
// Leading forms first: they are by far the most common.
|
||||||
if m := reOrdinalToken.FindStringSubmatch(tokens[0]); m != nil {
|
if m := reOrdinalToken.FindStringSubmatch(tokens[0]); m != nil {
|
||||||
p.Ordinal = normalizeOrdinal(m[1])
|
out.Ordinal = normalizeOrdinal(m[1])
|
||||||
return tokens[1:]
|
return tokens[1:]
|
||||||
}
|
}
|
||||||
if isDigits(tokens[0]) && len(tokens) >= 2 && ordinalSuffixes[tokens[1]] {
|
if isDigits(tokens[0]) && len(tokens) >= 2 && p.dict.isOrdinalSuffix(tokens[1]) {
|
||||||
p.Ordinal = normalizeOrdinal(tokens[0])
|
out.Ordinal = normalizeOrdinal(tokens[0])
|
||||||
return tokens[2:]
|
return tokens[2:]
|
||||||
}
|
}
|
||||||
if d, ok := ordinalWords[tokens[0]]; ok {
|
if d, ok := p.dict.ordinalWord(tokens[0]); ok {
|
||||||
p.Ordinal = normalizeOrdinal(d)
|
out.Ordinal = normalizeOrdinal(d)
|
||||||
return tokens[1:]
|
return tokens[1:]
|
||||||
}
|
}
|
||||||
|
|
||||||
// Otherwise look for an ordinal anywhere among the name tokens.
|
// Otherwise look for an ordinal anywhere among the name tokens.
|
||||||
for i, tok := range tokens {
|
for i, tok := range tokens {
|
||||||
if m := reOrdinalToken.FindStringSubmatch(tok); m != nil {
|
if m := reOrdinalToken.FindStringSubmatch(tok); m != nil {
|
||||||
p.Ordinal = normalizeOrdinal(m[1])
|
out.Ordinal = normalizeOrdinal(m[1])
|
||||||
return removeRange(tokens, i, i+1)
|
return removeRange(tokens, i, i+1)
|
||||||
}
|
}
|
||||||
if isDigits(tok) && i+1 < len(tokens) && ordinalSuffixes[tokens[i+1]] {
|
if isDigits(tok) && i+1 < len(tokens) && p.dict.isOrdinalSuffix(tokens[i+1]) {
|
||||||
p.Ordinal = normalizeOrdinal(tok)
|
out.Ordinal = normalizeOrdinal(tok)
|
||||||
return removeRange(tokens, i, i+2)
|
return removeRange(tokens, i, i+2)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -71,6 +71,12 @@ func TestParseHouseBlock(t *testing.T) {
|
|||||||
wantName: "выборная", wantType: "улица",
|
wantName: "выборная", wantType: "улица",
|
||||||
wantHouse: "45", wantLetter: "а", wantFrac: "4", wantHas: true,
|
wantHouse: "45", wantLetter: "а", wantFrac: "4", wantHas: true,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "spoken fraction word",
|
||||||
|
raw: "Горская улица 18 дробь 1",
|
||||||
|
wantName: "горская", wantType: "улица",
|
||||||
|
wantHouse: "18", wantFrac: "1", wantHas: true,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "corpus letter without digit",
|
name: "corpus letter without digit",
|
||||||
raw: "1-я Ельцовка улица 1 кД",
|
raw: "1-я Ельцовка улица 1 кД",
|
||||||
@@ -96,7 +102,7 @@ func TestParseHouseBlock(t *testing.T) {
|
|||||||
wantHouse: "11", wantFrac: "1", wantHas: true,
|
wantHouse: "11", wantFrac: "1", wantHas: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "portal district suffix in parentheses",
|
name: "district suffix in parentheses",
|
||||||
raw: "Медкадры улица, 1 (Заельцовский район)",
|
raw: "Медкадры улица, 1 (Заельцовский район)",
|
||||||
wantName: "медкадры", wantType: "улица",
|
wantName: "медкадры", wantType: "улица",
|
||||||
wantHouse: "1", wantHas: true,
|
wantHouse: "1", wantHas: true,
|
||||||
|
|||||||
@@ -2,73 +2,19 @@ package address
|
|||||||
|
|
||||||
import "strings"
|
import "strings"
|
||||||
|
|
||||||
// fillerWords are tokens that carry no street-name meaning and that people
|
// streetKey builds the canonical, city-aware, type-independent street key:
|
||||||
// routinely include ("г. Новосибирск", "улица ... дом 5", "... область").
|
//
|
||||||
// Dropping them makes the service tolerant of natural phrasing. District
|
// "<cityKey>|<ordinal>|<sorted name tokens>"
|
||||||
// adjectives are deliberately NOT dropped: they can legitimately be part of a
|
//
|
||||||
// street name.
|
// The city is part of the key so that "Ленина" in Novosibirsk and "Ленина" in
|
||||||
var fillerWords = map[string]bool{
|
// Berdsk are different streets. The type word is dropped so that a user who
|
||||||
"новосибирск": true, "нск": true, "нсо": true,
|
// omits it ("станционная 1" instead of "Станционная улица 1") still matches;
|
||||||
"город": true, "г": true,
|
// type confusion is surfaced later as ambiguity.
|
||||||
"область": true, "обл": true,
|
//
|
||||||
"район": true, "дом": true, "д": true,
|
// None of the components may contain "|": NormalizeText folds all punctuation
|
||||||
}
|
// that could appear in a name into spaces, and ordinals are digits only.
|
||||||
|
func streetKey(cityKey, ordinal, nameKey string) string {
|
||||||
// streetTypes maps a surface form (including common abbreviations) to a single
|
return cityKey + "|" + ordinal + "|" + nameKey
|
||||||
// canonical street type. The type is informational for matching (we strip it
|
|
||||||
// from the key) but useful for disambiguation and display.
|
|
||||||
var streetTypes = map[string]string{
|
|
||||||
"ул": "улица", "улица": "улица",
|
|
||||||
"пер": "переулок", "пер-к": "переулок", "переулок": "переулок",
|
|
||||||
"пр": "проезд", "проезд": "проезд",
|
|
||||||
"пр-т": "проспект", "просп": "проспект", "проспект": "проспект",
|
|
||||||
"ш": "шоссе", "шоссе": "шоссе",
|
|
||||||
"мкр": "микрорайон", "микрорайон": "микрорайон",
|
|
||||||
"туп": "тупик", "тупик": "тупик",
|
|
||||||
"спуск": "спуск",
|
|
||||||
"б-р": "бульвар", "бул": "бульвар", "бульвар": "бульвар",
|
|
||||||
"пл": "площадь", "площадь": "площадь",
|
|
||||||
"пос": "поселок", "поселок": "поселок",
|
|
||||||
"тракт": "тракт",
|
|
||||||
"аллея": "аллея",
|
|
||||||
"наб": "набережная", "набережная": "набережная",
|
|
||||||
"кольцо": "кольцо",
|
|
||||||
"просека": "просека",
|
|
||||||
"квартал": "квартал",
|
|
||||||
"территория": "территория",
|
|
||||||
"дорога": "дорога",
|
|
||||||
"разъезд": "разъезд",
|
|
||||||
}
|
|
||||||
|
|
||||||
// ordinalWords maps spelled-out ordinals to their digit form. Only the forms
|
|
||||||
// that actually occur in practice are listed; unknown words are treated as
|
|
||||||
// part of the street name.
|
|
||||||
var ordinalWords = map[string]string{"первый": "1", "первая": "1", "первое": "1",
|
|
||||||
"второй": "2", "вторая": "2", "второе": "2",
|
|
||||||
"третий": "3", "третья": "3", "третье": "3",
|
|
||||||
"четвертый": "4", "четвертая": "4", "четвертое": "4",
|
|
||||||
"пятый": "5", "пятая": "5", "пятое": "5",
|
|
||||||
"шестой": "6", "шестая": "6", "шестое": "6",
|
|
||||||
"седьмой": "7", "седьмая": "7", "седьмое": "7",
|
|
||||||
"восьмой": "8", "восьмая": "8", "восьмое": "8",
|
|
||||||
"девятый": "9", "девятая": "9", "девятое": "9",
|
|
||||||
"десятый": "10", "десятая": "10", "десятое": "10",
|
|
||||||
}
|
|
||||||
|
|
||||||
// ordinalSuffixes are the gender/number endings that may follow an ordinal
|
|
||||||
// digit when it is written as a separate token ("2 я", "10 й").
|
|
||||||
var ordinalSuffixes = map[string]bool{
|
|
||||||
"я": true, "й": true, "е": true,
|
|
||||||
"ая": true, "ый": true, "ой": true, "ое": true,
|
|
||||||
"ий": true, "ья": true, "ье": true, "ьи": true,
|
|
||||||
}
|
|
||||||
|
|
||||||
// streetKey builds the canonical, type-independent street key:
|
|
||||||
// "<ordinal>|<sorted name tokens>". The type word is dropped so that a user
|
|
||||||
// who omits it ("станционная 1" instead of "Станционная улица 1") still
|
|
||||||
// matches; type confusion is surfaced later as ambiguity.
|
|
||||||
func streetKey(ordinal, nameKey string) string {
|
|
||||||
return ordinal + "|" + nameKey
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// normalizeOrdinal trims leading zeros so "02-я" and "2-я" agree.
|
// normalizeOrdinal trims leading zeros so "02-я" and "2-я" agree.
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ package address
|
|||||||
// Status is the machine-readable outcome of an address resolution.
|
// Status is the machine-readable outcome of an address resolution.
|
||||||
//
|
//
|
||||||
// It is intentionally explicit: callers must never have to guess whether an
|
// It is intentionally explicit: callers must never have to guess whether an
|
||||||
// empty result means "not found" or "the upstream failed".
|
// empty result means "not found" or "the resolver refused to choose".
|
||||||
type Status string
|
type Status string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -25,11 +25,6 @@ const (
|
|||||||
StatusAmbiguous Status = "ambiguous"
|
StatusAmbiguous Status = "ambiguous"
|
||||||
// StatusNotFound means nothing in the registry matched.
|
// StatusNotFound means nothing in the registry matched.
|
||||||
StatusNotFound Status = "not_found"
|
StatusNotFound Status = "not_found"
|
||||||
// StatusPortalTimeout means the external outages portal did not answer in
|
|
||||||
// time. It is an infrastructure outcome, not "no outages".
|
|
||||||
StatusPortalTimeout Status = "portal_timeout"
|
|
||||||
// StatusInfra is a generic terminal infrastructure failure.
|
|
||||||
StatusInfra Status = "infra_error"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// ParsedAddress is the structured form of a free-form address string.
|
// ParsedAddress is the structured form of a free-form address string.
|
||||||
@@ -41,6 +36,13 @@ type ParsedAddress struct {
|
|||||||
Raw string `json:"raw"`
|
Raw string `json:"raw"`
|
||||||
Normalized string `json:"normalized"`
|
Normalized string `json:"normalized"`
|
||||||
|
|
||||||
|
// City is the canonical city name when the query names one (or when the
|
||||||
|
// parser is configured with a default city). CityKey is its normalized
|
||||||
|
// form. The city is part of StreetKey, so identical street names in
|
||||||
|
// different cities never collide.
|
||||||
|
City string `json:"city,omitempty"`
|
||||||
|
CityKey string `json:"city_key,omitempty"`
|
||||||
|
|
||||||
// Street components.
|
// Street components.
|
||||||
//
|
//
|
||||||
// NameKey is the order-insensitive canonical name (tokens sorted), so
|
// NameKey is the order-insensitive canonical name (tokens sorted), so
|
||||||
@@ -73,6 +75,9 @@ type Entry struct {
|
|||||||
Address string `json:"address"`
|
Address string `json:"address"`
|
||||||
District string `json:"district,omitempty"`
|
District string `json:"district,omitempty"`
|
||||||
|
|
||||||
|
City string `json:"city,omitempty"`
|
||||||
|
CityKey string `json:"city_key,omitempty"`
|
||||||
|
|
||||||
Ordinal string `json:"ordinal,omitempty"`
|
Ordinal string `json:"ordinal,omitempty"`
|
||||||
NameKey string `json:"name_key,omitempty"`
|
NameKey string `json:"name_key,omitempty"`
|
||||||
NameDisplay string `json:"name_display,omitempty"`
|
NameDisplay string `json:"name_display,omitempty"`
|
||||||
|
|||||||
@@ -0,0 +1,198 @@
|
|||||||
|
package match
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
|
)
|
||||||
|
|
||||||
|
// staticRewriter returns a fixed list of variants, emulating the LLM adapter.
|
||||||
|
type staticRewriter []string
|
||||||
|
|
||||||
|
func (s staticRewriter) Rewrite(context.Context, string) ([]string, error) { return s, nil }
|
||||||
|
|
||||||
|
// TestRewriteInheritsDetectedCity guards the multi-city LLM path: the embedded
|
||||||
|
// prompt tells the model to strip administrative words, so a correct variant
|
||||||
|
// usually has no city. It must inherit the city detected in the original query,
|
||||||
|
// while a variant naming a different city must still be rejected.
|
||||||
|
func TestRewriteInheritsDetectedCity(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
|
||||||
|
entries := cityEntries(t, parser,
|
||||||
|
[2]string{"Новосибирск", "Кошурникова улица 5"},
|
||||||
|
[2]string{"Бердск", "Кошурникова улица 5"},
|
||||||
|
)
|
||||||
|
// A threshold above any typo score forces the deterministic path to miss.
|
||||||
|
ix := NewIndex("test", entries, Options{FuzzyThreshold: 0.999, MaxCandidates: 5})
|
||||||
|
|
||||||
|
cityless := NewResolverWithOptions(ix, staticRewriter{"Кошурникова 5"}, ResolverOptions{Parser: parser})
|
||||||
|
res := cityless.ResolveInCity(context.Background(), "нск кошурникиова 5", "")
|
||||||
|
if res.Status != address.StatusFound || res.City != "Новосибирск" {
|
||||||
|
t.Fatalf("cityless variant: %+v", res)
|
||||||
|
}
|
||||||
|
|
||||||
|
conflict := NewResolverWithOptions(ix, staticRewriter{"Бердск, Кошурникова 5"}, ResolverOptions{Parser: parser})
|
||||||
|
res = conflict.ResolveInCity(context.Background(), "нск кошурникиова 5", "")
|
||||||
|
if res.Status == address.StatusFound {
|
||||||
|
t.Fatalf("conflicting city variant was accepted: %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func matchCityDictionary() []address.City {
|
||||||
|
return []address.City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск"}},
|
||||||
|
{Name: "Бердск"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cityEntries(t *testing.T, parser *address.Parser, rows ...[2]string) []address.Entry {
|
||||||
|
t.Helper()
|
||||||
|
entries := make([]address.Entry, 0, len(rows))
|
||||||
|
for _, row := range rows {
|
||||||
|
p := parser.ParseInCity(row[1], row[0])
|
||||||
|
entries = append(entries, address.Entry{
|
||||||
|
ID: row[0] + "/" + row[1], Address: row[1],
|
||||||
|
City: p.City, CityKey: p.CityKey,
|
||||||
|
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
|
||||||
|
StreetType: p.StreetType, StreetKey: p.StreetKey,
|
||||||
|
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
|
||||||
|
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
|
||||||
|
Structure: p.Structure,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return entries
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIndexKeepsCitiesApart(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
|
||||||
|
entries := cityEntries(t, parser,
|
||||||
|
[2]string{"Новосибирск", "Ленина улица 5"},
|
||||||
|
[2]string{"Бердск", "Ленина улица 5"},
|
||||||
|
)
|
||||||
|
resolver := NewResolverWithOptions(
|
||||||
|
NewIndex("test", entries, DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
|
nsk := resolver.ResolveInCity(t.Context(), "Ленина 5", "Новосибирск")
|
||||||
|
if nsk.Status != address.StatusFound || nsk.City != "Новосибирск" {
|
||||||
|
t.Fatalf("Новосибирск: %+v", nsk)
|
||||||
|
}
|
||||||
|
berdsk := resolver.ResolveInCity(t.Context(), "Ленина 5", "Бердск")
|
||||||
|
if berdsk.Status != address.StatusFound || berdsk.City != "Бердск" {
|
||||||
|
t.Fatalf("Бердск: %+v", berdsk)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Without a city the resolver must not pick one: both streets are exact
|
||||||
|
// matches in different cities.
|
||||||
|
res := resolver.Resolve(t.Context(), "Ленина 5")
|
||||||
|
if res.Status != address.StatusAmbiguous {
|
||||||
|
t.Fatalf("cityless status = %q, want ambiguous (candidates: %+v)", res.Status, res.Candidates)
|
||||||
|
}
|
||||||
|
if len(res.Candidates) != 2 {
|
||||||
|
t.Fatalf("candidates = %+v, want one per city", res.Candidates)
|
||||||
|
}
|
||||||
|
if res.City != "" {
|
||||||
|
t.Fatalf("ambiguous cityless result must not report a city, got %q", res.City)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCitylessQueryReportsTheOnlyCity(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
|
||||||
|
entries := cityEntries(t, parser, [2]string{"Новосибирск", "Ленина улица 5"})
|
||||||
|
resolver := NewResolverWithOptions(
|
||||||
|
NewIndex("test", entries, DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
|
res := resolver.Resolve(t.Context(), "Ленина 5")
|
||||||
|
if res.Status != address.StatusFound || res.City != "Новосибирск" {
|
||||||
|
t.Fatalf("result = %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStructureIsPartOfHouseIdentity(t *testing.T) {
|
||||||
|
ix := buildIndex(t,
|
||||||
|
"Станционная улица 1",
|
||||||
|
"Станционная улица 1 стр. 2",
|
||||||
|
)
|
||||||
|
|
||||||
|
found := ix.Resolve(address.Parse("станционная 1 стр. 2"))
|
||||||
|
if found.Status != address.StatusFound || found.Address != "Станционная улица 1 стр. 2" {
|
||||||
|
t.Fatalf("exact structure: %+v", found)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A structure that does not exist must never resolve to another building.
|
||||||
|
wrong := ix.Resolve(address.Parse("станционная 1 стр. 3"))
|
||||||
|
if wrong.Status == address.StatusFound {
|
||||||
|
t.Fatalf("wrong structure resolved to %q", wrong.Address)
|
||||||
|
}
|
||||||
|
if wrong.Status != address.StatusAmbiguous {
|
||||||
|
t.Fatalf("wrong structure status = %q, want ambiguous", wrong.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
base := ix.Resolve(address.Parse("станционная 1"))
|
||||||
|
if base.Status != address.StatusFound || base.Address != "Станционная улица 1" {
|
||||||
|
t.Fatalf("base house: %+v", base)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIndexOrderIsDeterministic(t *testing.T) {
|
||||||
|
addresses := []string{
|
||||||
|
"Станционная улица 1/1",
|
||||||
|
"Станционная улица 2/1",
|
||||||
|
"Станционная улица 3/1 к1",
|
||||||
|
"Станционная улица 3/1 к2",
|
||||||
|
"Искитимская улица 5",
|
||||||
|
}
|
||||||
|
forward := buildEntries(t, addresses...)
|
||||||
|
backward := buildEntries(t, reverse(addresses)...)
|
||||||
|
|
||||||
|
a := NewIndex("v", forward, DefaultOptions())
|
||||||
|
b := NewIndex("v", backward, DefaultOptions())
|
||||||
|
|
||||||
|
for _, q := range []string{"станционная 3/1", "станционная 1/2", "искитимсая 5"} {
|
||||||
|
ra := a.Resolve(address.Parse(q))
|
||||||
|
rb := b.Resolve(address.Parse(q))
|
||||||
|
if !reflect.DeepEqual(ra, rb) {
|
||||||
|
t.Fatalf("%q: results differ\n a=%+v\n b=%+v", q, ra, rb)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLongTokenDoesNotMatch(t *testing.T) {
|
||||||
|
ix := buildIndex(t, "Станционная улица 1")
|
||||||
|
res := ix.Resolve(address.Parse(strings.Repeat("а", 4000)))
|
||||||
|
if res.Status != address.StatusNotFound {
|
||||||
|
t.Fatalf("status = %q, want not_found", res.Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildEntries(t *testing.T, addresses ...string) []address.Entry {
|
||||||
|
t.Helper()
|
||||||
|
entries := make([]address.Entry, 0, len(addresses))
|
||||||
|
for _, a := range addresses {
|
||||||
|
p := address.Parse(a)
|
||||||
|
entries = append(entries, address.Entry{
|
||||||
|
ID: a, Address: a,
|
||||||
|
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
|
||||||
|
StreetType: p.StreetType, StreetKey: p.StreetKey,
|
||||||
|
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
|
||||||
|
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
|
||||||
|
Structure: p.Structure,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return entries
|
||||||
|
}
|
||||||
|
|
||||||
|
func reverse(in []string) []string {
|
||||||
|
out := make([]string, len(in))
|
||||||
|
for i, s := range in {
|
||||||
|
out[len(in)-1-i] = s
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
+137
-32
@@ -9,6 +9,7 @@ package match
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"sort"
|
"sort"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
)
|
)
|
||||||
@@ -16,16 +17,19 @@ import (
|
|||||||
// Candidate is one plausible match, returned alongside a non-"found" result so
|
// Candidate is one plausible match, returned alongside a non-"found" result so
|
||||||
// the caller (or an operator) can see what the resolver was choosing between.
|
// the caller (or an operator) can see what the resolver was choosing between.
|
||||||
type Candidate struct {
|
type Candidate struct {
|
||||||
Address string `json:"address"`
|
Address string `json:"address"`
|
||||||
District string `json:"district,omitempty"`
|
City string `json:"city,omitempty"`
|
||||||
House string `json:"house,omitempty"`
|
District string `json:"district,omitempty"`
|
||||||
Score float64 `json:"score"`
|
House string `json:"house,omitempty"`
|
||||||
|
Structure string `json:"structure,omitempty"`
|
||||||
|
Score float64 `json:"score"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Result is the outcome of a resolution.
|
// Result is the outcome of a resolution.
|
||||||
type Result struct {
|
type Result struct {
|
||||||
Status address.Status `json:"status"`
|
Status address.Status `json:"status"`
|
||||||
Address string `json:"address,omitempty"`
|
Address string `json:"address,omitempty"`
|
||||||
|
City string `json:"city,omitempty"`
|
||||||
District string `json:"district,omitempty"`
|
District string `json:"district,omitempty"`
|
||||||
Confidence float64 `json:"confidence"`
|
Confidence float64 `json:"confidence"`
|
||||||
Candidates []Candidate `json:"candidates,omitempty"`
|
Candidates []Candidate `json:"candidates,omitempty"`
|
||||||
@@ -59,10 +63,10 @@ func DefaultOptions() Options {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type streetRef struct {
|
type streetRef struct {
|
||||||
key string
|
key string
|
||||||
ordinal string
|
cityKey string
|
||||||
nameKey string
|
ordinal string
|
||||||
streetType string
|
nameKey string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Index is an immutable, in-memory registry index. It is safe for concurrent
|
// Index is an immutable, in-memory registry index. It is safe for concurrent
|
||||||
@@ -72,26 +76,46 @@ type Index struct {
|
|||||||
opts Options
|
opts Options
|
||||||
byKey map[string][]address.Entry
|
byKey map[string][]address.Entry
|
||||||
streets []streetRef
|
streets []streetRef
|
||||||
|
cities map[string]string // cityKey -> canonical display name
|
||||||
|
size int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// maxFuzzyNameRunes bounds the fuzzy scan. No real street name is longer than
|
||||||
|
// this; without the bound a single huge token would make every Levenshtein
|
||||||
|
// comparison walk a very long string and turn a request into a CPU sink.
|
||||||
|
const maxFuzzyNameRunes = 96
|
||||||
|
|
||||||
// NewIndex builds an index from canonical entries. Entries sharing a StreetKey
|
// NewIndex builds an index from canonical entries. Entries sharing a StreetKey
|
||||||
// are grouped; distinct streets are collected for typo-tolerant matching.
|
// are grouped; distinct streets are collected for typo-tolerant matching.
|
||||||
|
//
|
||||||
|
// Entries are sorted first, so candidate order is deterministic regardless of
|
||||||
|
// the order in which the storage adapter returned them.
|
||||||
func NewIndex(version string, entries []address.Entry, opts Options) *Index {
|
func NewIndex(version string, entries []address.Entry, opts Options) *Index {
|
||||||
|
sorted := append([]address.Entry(nil), entries...)
|
||||||
|
sort.SliceStable(sorted, func(i, j int) bool { return entryLess(sorted[i], sorted[j]) })
|
||||||
|
|
||||||
ix := &Index{
|
ix := &Index{
|
||||||
version: version,
|
version: version,
|
||||||
opts: opts,
|
opts: opts,
|
||||||
byKey: make(map[string][]address.Entry, len(entries)),
|
byKey: make(map[string][]address.Entry, len(sorted)),
|
||||||
|
cities: make(map[string]string),
|
||||||
|
size: len(sorted),
|
||||||
}
|
}
|
||||||
seen := make(map[string]struct{}, len(entries))
|
seen := make(map[string]struct{}, len(sorted))
|
||||||
for _, e := range entries {
|
for _, e := range sorted {
|
||||||
ix.byKey[e.StreetKey] = append(ix.byKey[e.StreetKey], e)
|
ix.byKey[e.StreetKey] = append(ix.byKey[e.StreetKey], e)
|
||||||
|
if e.CityKey != "" {
|
||||||
|
if _, ok := ix.cities[e.CityKey]; !ok {
|
||||||
|
ix.cities[e.CityKey] = e.City
|
||||||
|
}
|
||||||
|
}
|
||||||
if _, ok := seen[e.StreetKey]; !ok {
|
if _, ok := seen[e.StreetKey]; !ok {
|
||||||
seen[e.StreetKey] = struct{}{}
|
seen[e.StreetKey] = struct{}{}
|
||||||
ix.streets = append(ix.streets, streetRef{
|
ix.streets = append(ix.streets, streetRef{
|
||||||
key: e.StreetKey,
|
key: e.StreetKey,
|
||||||
ordinal: e.Ordinal,
|
cityKey: e.CityKey,
|
||||||
nameKey: e.NameKey,
|
ordinal: e.Ordinal,
|
||||||
streetType: e.StreetType,
|
nameKey: e.NameKey,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -99,22 +123,57 @@ func NewIndex(version string, entries []address.Entry, opts Options) *Index {
|
|||||||
return ix
|
return ix
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// entryLess defines the canonical order of registry rows. Address strings are
|
||||||
|
// the last meaningful tie-breaker; ID makes the order total even for duplicated
|
||||||
|
// rows.
|
||||||
|
func entryLess(a, b address.Entry) bool {
|
||||||
|
switch {
|
||||||
|
case a.CityKey != b.CityKey:
|
||||||
|
return a.CityKey < b.CityKey
|
||||||
|
case a.StreetKey != b.StreetKey:
|
||||||
|
return a.StreetKey < b.StreetKey
|
||||||
|
case a.House != b.House:
|
||||||
|
return a.House < b.House
|
||||||
|
case a.HouseFrac != b.HouseFrac:
|
||||||
|
return a.HouseFrac < b.HouseFrac
|
||||||
|
case a.HouseLetter != b.HouseLetter:
|
||||||
|
return a.HouseLetter < b.HouseLetter
|
||||||
|
case a.Building != b.Building:
|
||||||
|
return a.Building < b.Building
|
||||||
|
case a.BuildingFrac != b.BuildingFrac:
|
||||||
|
return a.BuildingFrac < b.BuildingFrac
|
||||||
|
case a.BuildingLetter != b.BuildingLetter:
|
||||||
|
return a.BuildingLetter < b.BuildingLetter
|
||||||
|
case a.Structure != b.Structure:
|
||||||
|
return a.Structure < b.Structure
|
||||||
|
case a.Address != b.Address:
|
||||||
|
return a.Address < b.Address
|
||||||
|
default:
|
||||||
|
return a.ID < b.ID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Version returns the data version the index was built from.
|
// Version returns the data version the index was built from.
|
||||||
func (ix *Index) Version() string { return ix.version }
|
func (ix *Index) Version() string { return ix.version }
|
||||||
|
|
||||||
// Size returns the number of indexed entries.
|
// Size returns the number of indexed entries.
|
||||||
func (ix *Index) Size() int {
|
func (ix *Index) Size() int { return ix.size }
|
||||||
n := 0
|
|
||||||
for _, v := range ix.byKey {
|
// Cities returns the distinct canonical city names present in the index, in
|
||||||
n += len(v)
|
// deterministic order. A city-less index returns an empty slice.
|
||||||
|
func (ix *Index) Cities() []string {
|
||||||
|
out := make([]string, 0, len(ix.cities))
|
||||||
|
for _, name := range ix.cities {
|
||||||
|
out = append(out, name)
|
||||||
}
|
}
|
||||||
return n
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// Resolve resolves a parsed address. It never returns an address that is not
|
// Resolve resolves a parsed address. It never returns an address that is not
|
||||||
// present in the index.
|
// present in the index.
|
||||||
func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
||||||
res := Result{DataVersion: ix.version}
|
res := Result{DataVersion: ix.version, City: p.City}
|
||||||
|
|
||||||
entries := ix.byKey[p.StreetKey]
|
entries := ix.byKey[p.StreetKey]
|
||||||
similarity := 1.0
|
similarity := 1.0
|
||||||
@@ -133,6 +192,11 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
|||||||
res.Status = address.StatusNotFound
|
res.Status = address.StatusNotFound
|
||||||
return res
|
return res
|
||||||
}
|
}
|
||||||
|
if res.City == "" {
|
||||||
|
// A city-less query may still match exactly one city; report it so the
|
||||||
|
// caller does not have to re-derive the city from the candidates.
|
||||||
|
res.City = commonCity(entries)
|
||||||
|
}
|
||||||
|
|
||||||
if !p.HasHouse {
|
if !p.HasHouse {
|
||||||
// Street is known but the caller did not supply a house: safe but
|
// Street is known but the caller did not supply a house: safe but
|
||||||
@@ -148,6 +212,7 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
|||||||
case 1:
|
case 1:
|
||||||
res.Status = address.StatusFound
|
res.Status = address.StatusFound
|
||||||
res.Address = matched[0].Address
|
res.Address = matched[0].Address
|
||||||
|
res.City = matched[0].City
|
||||||
res.District = matched[0].District
|
res.District = matched[0].District
|
||||||
res.Confidence = clamp01(similarity)
|
res.Confidence = clamp01(similarity)
|
||||||
return res
|
return res
|
||||||
@@ -162,7 +227,14 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
|||||||
}
|
}
|
||||||
res.Status = address.StatusPartial
|
res.Status = address.StatusPartial
|
||||||
res.Confidence = clamp01(similarity * 0.9)
|
res.Confidence = clamp01(similarity * 0.9)
|
||||||
res.Candidates = candidates(entries, similarity, ix.opts.MaxCandidates)
|
// Prefer houses that share the number/fraction/letter: they are the
|
||||||
|
// plausible corrections (a different corpus or structure), while the
|
||||||
|
// rest of the street is noise.
|
||||||
|
related := matchHouseUnitAll(entries, p)
|
||||||
|
if len(related) == 0 {
|
||||||
|
related = entries
|
||||||
|
}
|
||||||
|
res.Candidates = candidates(related, similarity, ix.opts.MaxCandidates)
|
||||||
return res
|
return res
|
||||||
default:
|
default:
|
||||||
res.Status = address.StatusAmbiguous
|
res.Status = address.StatusAmbiguous
|
||||||
@@ -176,12 +248,16 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
|
|||||||
// the best street when the match is unambiguous, or the list of competing
|
// the best street when the match is unambiguous, or the list of competing
|
||||||
// street keys when several are within the ambiguity margin.
|
// street keys when several are within the ambiguity margin.
|
||||||
func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry, similarity float64, ambiguousKeys []string) {
|
func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry, similarity float64, ambiguousKeys []string) {
|
||||||
if p.NameKey == "" {
|
if p.NameKey == "" || utf8.RuneCountInString(p.NameKey) > maxFuzzyNameRunes {
|
||||||
return nil, 0, nil
|
return nil, 0, nil
|
||||||
}
|
}
|
||||||
best := 0.0
|
best := 0.0
|
||||||
bestKey := ""
|
bestKey := ""
|
||||||
for _, s := range ix.streets {
|
for _, s := range ix.streets {
|
||||||
|
// A query that names a city may only match streets in that city.
|
||||||
|
if p.CityKey != "" && s.cityKey != p.CityKey {
|
||||||
|
continue
|
||||||
|
}
|
||||||
// If the query carries an ordinal, the candidate's ordinal must match:
|
// If the query carries an ordinal, the candidate's ordinal must match:
|
||||||
// a user who says "1-й Северный" does not mean the plain "Северная".
|
// a user who says "1-й Северный" does not mean the plain "Северная".
|
||||||
if !ordinalCompatible(p.Ordinal, s.ordinal) {
|
if !ordinalCompatible(p.Ordinal, s.ordinal) {
|
||||||
@@ -197,6 +273,9 @@ func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry,
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, s := range ix.streets {
|
for _, s := range ix.streets {
|
||||||
|
if p.CityKey != "" && s.cityKey != p.CityKey {
|
||||||
|
continue
|
||||||
|
}
|
||||||
if !ordinalCompatible(p.Ordinal, s.ordinal) {
|
if !ordinalCompatible(p.Ordinal, s.ordinal) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -232,10 +311,12 @@ func (ix *Index) representatives(keys []string, score float64) []Candidate {
|
|||||||
}
|
}
|
||||||
e := es[0]
|
e := es[0]
|
||||||
out = append(out, Candidate{
|
out = append(out, Candidate{
|
||||||
Address: e.Address,
|
Address: e.Address,
|
||||||
District: e.District,
|
City: e.City,
|
||||||
House: e.House,
|
District: e.District,
|
||||||
Score: clamp01(score),
|
House: e.House,
|
||||||
|
Structure: e.Structure,
|
||||||
|
Score: clamp01(score),
|
||||||
})
|
})
|
||||||
if len(out) == ix.opts.MaxCandidates {
|
if len(out) == ix.opts.MaxCandidates {
|
||||||
break
|
break
|
||||||
@@ -275,13 +356,16 @@ func sameHouseSeveral(entries []address.Entry, p address.ParsedAddress) bool {
|
|||||||
|
|
||||||
// houseUnitEqual compares the parts of a house unit. Unspecified (empty) parts
|
// houseUnitEqual compares the parts of a house unit. Unspecified (empty) parts
|
||||||
// must also be empty on the entry side, so "60/1" never matches "60/1 к148".
|
// must also be empty on the entry side, so "60/1" never matches "60/1 к148".
|
||||||
|
// The structure ("стр. 2") is part of the identity: a caller who explicitly
|
||||||
|
// asked for a different structure must not be handed another building.
|
||||||
func houseUnitEqual(e address.Entry, p address.ParsedAddress) bool {
|
func houseUnitEqual(e address.Entry, p address.ParsedAddress) bool {
|
||||||
return e.House == p.House &&
|
return e.House == p.House &&
|
||||||
e.HouseFrac == p.HouseFrac &&
|
e.HouseFrac == p.HouseFrac &&
|
||||||
e.HouseLetter == p.HouseLetter &&
|
e.HouseLetter == p.HouseLetter &&
|
||||||
e.Building == p.Building &&
|
e.Building == p.Building &&
|
||||||
e.BuildingFrac == p.BuildingFrac &&
|
e.BuildingFrac == p.BuildingFrac &&
|
||||||
e.BuildingLetter == p.BuildingLetter
|
e.BuildingLetter == p.BuildingLetter &&
|
||||||
|
e.Structure == p.Structure
|
||||||
}
|
}
|
||||||
|
|
||||||
func candidates(entries []address.Entry, score float64, limit int) []Candidate {
|
func candidates(entries []address.Entry, score float64, limit int) []Candidate {
|
||||||
@@ -291,10 +375,12 @@ func candidates(entries []address.Entry, score float64, limit int) []Candidate {
|
|||||||
out := make([]Candidate, 0, len(entries))
|
out := make([]Candidate, 0, len(entries))
|
||||||
for _, e := range entries {
|
for _, e := range entries {
|
||||||
out = append(out, Candidate{
|
out = append(out, Candidate{
|
||||||
Address: e.Address,
|
Address: e.Address,
|
||||||
District: e.District,
|
City: e.City,
|
||||||
House: e.House,
|
District: e.District,
|
||||||
Score: clamp01(score),
|
House: e.House,
|
||||||
|
Structure: e.Structure,
|
||||||
|
Score: clamp01(score),
|
||||||
})
|
})
|
||||||
if len(out) == limit {
|
if len(out) == limit {
|
||||||
break
|
break
|
||||||
@@ -312,3 +398,22 @@ func clamp01(v float64) float64 {
|
|||||||
}
|
}
|
||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// commonCity returns the city shared by all entries, or "" when they span
|
||||||
|
// cities (or carry no city at all).
|
||||||
|
func commonCity(entries []address.Entry) string {
|
||||||
|
key, name := "", ""
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.CityKey == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if key == "" {
|
||||||
|
key, name = e.CityKey, e.City
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if e.CityKey != key {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,6 +1,9 @@
|
|||||||
package match
|
package match
|
||||||
|
|
||||||
import "strings"
|
import (
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
// levenshtein returns the edit distance between two strings, measured in
|
// levenshtein returns the edit distance between two strings, measured in
|
||||||
// runes. It uses the rolling-row formulation: O(min(n,m)) memory.
|
// runes. It uses the rolling-row formulation: O(min(n,m)) memory.
|
||||||
@@ -102,18 +105,30 @@ func min3(a, b, c int) int {
|
|||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// maxTokenRunes is the longest token the fuzzy matcher will compare. Real
|
||||||
|
// street-name tokens are far shorter; anything longer is noise or abuse.
|
||||||
|
const maxTokenRunes = 64
|
||||||
|
|
||||||
// tokenScore scores two name tokens. A single edit is tolerated for tokens of
|
// tokenScore scores two name tokens. A single edit is tolerated for tokens of
|
||||||
// length >= 4 and two edits for length >= 8; otherwise it falls back to the
|
// length >= 4 and two edits for length >= 8; otherwise it falls back to the
|
||||||
// plain ratio. Short names (<= 3 chars) are intentionally not fuzzy-matched:
|
// plain ratio. Short names (<= 3 chars) are intentionally not fuzzy-matched:
|
||||||
// one edit is too large a fraction of the word to be safe.
|
// one edit is too large a fraction of the word to be safe.
|
||||||
|
//
|
||||||
|
// Tokens longer than maxTokenRunes are never similar to a real street name.
|
||||||
|
// Rejecting them up front keeps a pathological input from turning one request
|
||||||
|
// into millions of Levenshtein cell updates.
|
||||||
func tokenScore(a, b string) float64 {
|
func tokenScore(a, b string) float64 {
|
||||||
if a == b {
|
if a == b {
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
ra, rb := utf8.RuneCountInString(a), utf8.RuneCountInString(b)
|
||||||
|
if ra == 0 || rb == 0 || ra > maxTokenRunes || rb > maxTokenRunes {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
d := levenshtein(a, b)
|
d := levenshtein(a, b)
|
||||||
minLen := len([]rune(a))
|
minLen := ra
|
||||||
if lb := len([]rune(b)); lb < minLen {
|
if rb < minLen {
|
||||||
minLen = lb
|
minLen = rb
|
||||||
}
|
}
|
||||||
switch {
|
switch {
|
||||||
case d == 1 && minLen >= 4:
|
case d == 1 && minLen >= 4:
|
||||||
|
|||||||
@@ -20,6 +20,25 @@ type Rewriter interface {
|
|||||||
Rewrite(ctx context.Context, raw string) ([]string, error)
|
Rewrite(ctx context.Context, raw string) ([]string, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Parser is the address parser used by the resolver. It is defined here
|
||||||
|
// (consumer side) so the resolver can be tested with a stub and so a
|
||||||
|
// deployment can plug in its own dictionary.
|
||||||
|
type Parser interface {
|
||||||
|
Parse(raw string) address.ParsedAddress
|
||||||
|
ParseInCity(raw, city string) address.ParsedAddress
|
||||||
|
MatchCity(query string) (address.City, bool)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolverOptions tunes optional resolver behaviour.
|
||||||
|
type ResolverOptions struct {
|
||||||
|
// Parser defaults to address.DefaultParser().
|
||||||
|
Parser Parser
|
||||||
|
// OnRewriteError, when set, observes failures of the optional rewriter.
|
||||||
|
// The resolver always keeps the deterministic result, but operators still
|
||||||
|
// need to know that the fallback is broken.
|
||||||
|
OnRewriteError func(error)
|
||||||
|
}
|
||||||
|
|
||||||
// Resolver wraps an Index with an optional Rewriter.
|
// Resolver wraps an Index with an optional Rewriter.
|
||||||
//
|
//
|
||||||
// The deterministic path is always tried first. The rewriter is consulted only
|
// The deterministic path is always tried first. The rewriter is consulted only
|
||||||
@@ -27,13 +46,30 @@ type Rewriter interface {
|
|||||||
// suggestions must pass compatibleVariant before being considered, so the
|
// suggestions must pass compatibleVariant before being considered, so the
|
||||||
// model cannot substitute a different street or corrupt the house unit.
|
// model cannot substitute a different street or corrupt the house unit.
|
||||||
type Resolver struct {
|
type Resolver struct {
|
||||||
index *Index
|
index *Index
|
||||||
rewriter Rewriter
|
rewriter Rewriter
|
||||||
|
parser Parser
|
||||||
|
onRewriteError func(error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewResolver builds a resolver. rewriter may be nil to disable rewriting.
|
// NewResolver builds a resolver with the default parser and no rewriter
|
||||||
|
// diagnostics.
|
||||||
func NewResolver(index *Index, rewriter Rewriter) *Resolver {
|
func NewResolver(index *Index, rewriter Rewriter) *Resolver {
|
||||||
return &Resolver{index: index, rewriter: rewriter}
|
return NewResolverWithOptions(index, rewriter, ResolverOptions{})
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewResolverWithOptions builds a resolver with explicit options.
|
||||||
|
func NewResolverWithOptions(index *Index, rewriter Rewriter, opts ResolverOptions) *Resolver {
|
||||||
|
parser := opts.Parser
|
||||||
|
if parser == nil {
|
||||||
|
parser = address.DefaultParser()
|
||||||
|
}
|
||||||
|
return &Resolver{
|
||||||
|
index: index,
|
||||||
|
rewriter: rewriter,
|
||||||
|
parser: parser,
|
||||||
|
onRewriteError: opts.OnRewriteError,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Version returns the current data version.
|
// Version returns the current data version.
|
||||||
@@ -42,9 +78,35 @@ func (r *Resolver) Version() string { return r.index.Version() }
|
|||||||
// Ready reports whether the resolver has a usable index.
|
// Ready reports whether the resolver has a usable index.
|
||||||
func (r *Resolver) Ready() bool { return r.index != nil && r.index.Size() > 0 }
|
func (r *Resolver) Ready() bool { return r.index != nil && r.index.Size() > 0 }
|
||||||
|
|
||||||
|
// Cities returns the canonical city names present in the index.
|
||||||
|
func (r *Resolver) Cities() []string {
|
||||||
|
if r.index == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return r.index.Cities()
|
||||||
|
}
|
||||||
|
|
||||||
|
// MatchCity reports whether query is a city name/alias and returns the
|
||||||
|
// canonical name. It uses the resolver's parser dictionary, not the index, so
|
||||||
|
// it also recognizes aliases for cities that have no rows yet.
|
||||||
|
func (r *Resolver) MatchCity(query string) (string, bool) {
|
||||||
|
city, ok := r.parser.MatchCity(query)
|
||||||
|
if !ok {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return city.Name, true
|
||||||
|
}
|
||||||
|
|
||||||
// Resolve parses raw and resolves it, consulting the rewriter when enabled.
|
// Resolve parses raw and resolves it, consulting the rewriter when enabled.
|
||||||
func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
|
func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
|
||||||
parsed := address.Parse(raw)
|
return r.ResolveInCity(ctx, raw, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveInCity is Resolve with an explicit city hint. The hint overrides a
|
||||||
|
// city found in the address text and lets clients pass the city as a separate
|
||||||
|
// field.
|
||||||
|
func (r *Resolver) ResolveInCity(ctx context.Context, raw, city string) Result {
|
||||||
|
parsed := r.parse(raw, city)
|
||||||
res := r.index.Resolve(parsed)
|
res := r.index.Resolve(parsed)
|
||||||
|
|
||||||
if res.Status == address.StatusFound || r.rewriter == nil {
|
if res.Status == address.StatusFound || r.rewriter == nil {
|
||||||
@@ -55,13 +117,27 @@ func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
|
|||||||
}
|
}
|
||||||
|
|
||||||
alternatives, err := r.rewriter.Rewrite(ctx, raw)
|
alternatives, err := r.rewriter.Rewrite(ctx, raw)
|
||||||
if err != nil || len(alternatives) == 0 {
|
if err != nil {
|
||||||
|
if r.onRewriteError != nil {
|
||||||
|
r.onRewriteError(err)
|
||||||
|
}
|
||||||
|
return res
|
||||||
|
}
|
||||||
|
if len(alternatives) == 0 {
|
||||||
return res
|
return res
|
||||||
}
|
}
|
||||||
|
|
||||||
best := res
|
best := res
|
||||||
for _, alt := range alternatives {
|
for _, alt := range alternatives {
|
||||||
variant := address.Parse(alt)
|
variant := r.parse(alt, city)
|
||||||
|
// The embedded prompt asks the model to strip administrative words
|
||||||
|
// (including the city), so a correct variant often has no city. It
|
||||||
|
// inherits the original's city; only a conflicting explicit city is a
|
||||||
|
// disqualification. Without this, the rewriter silently contributes
|
||||||
|
// nothing in multi-city deployments (where default_city is empty).
|
||||||
|
if variant.CityKey == "" && parsed.CityKey != "" {
|
||||||
|
variant = r.parser.ParseInCity(alt, parsed.City)
|
||||||
|
}
|
||||||
// The model may fix the spelling of the street, nothing else.
|
// The model may fix the spelling of the street, nothing else.
|
||||||
if !compatibleVariant(parsed, variant) {
|
if !compatibleVariant(parsed, variant) {
|
||||||
continue
|
continue
|
||||||
@@ -74,6 +150,13 @@ func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
|
|||||||
return best
|
return best
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *Resolver) parse(raw, city string) address.ParsedAddress {
|
||||||
|
if strings.TrimSpace(city) != "" {
|
||||||
|
return r.parser.ParseInCity(raw, city)
|
||||||
|
}
|
||||||
|
return r.parser.Parse(raw)
|
||||||
|
}
|
||||||
|
|
||||||
// compatibleVariant reports whether an LLM rewrite may be considered. A rewrite
|
// compatibleVariant reports whether an LLM rewrite may be considered. A rewrite
|
||||||
// is acceptable only when it:
|
// is acceptable only when it:
|
||||||
// - keeps the exact house unit (number, fraction, letter, corpus, structure
|
// - keeps the exact house unit (number, fraction, letter, corpus, structure
|
||||||
@@ -82,6 +165,9 @@ func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
|
|||||||
// - has the same words in the street name, each within a small edit distance
|
// - has the same words in the street name, each within a small edit distance
|
||||||
// — so a different street can never be substituted.
|
// — so a different street can never be substituted.
|
||||||
func compatibleVariant(orig, variant address.ParsedAddress) bool {
|
func compatibleVariant(orig, variant address.ParsedAddress) bool {
|
||||||
|
if orig.CityKey != variant.CityKey {
|
||||||
|
return false
|
||||||
|
}
|
||||||
if !sameHouseUnit(orig, variant) {
|
if !sameHouseUnit(orig, variant) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
@@ -108,9 +194,13 @@ func nearToken(a, b string) bool {
|
|||||||
if a == b {
|
if a == b {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
minLen := len([]rune(a))
|
ra, rb := len([]rune(a)), len([]rune(b))
|
||||||
if lb := len([]rune(b)); lb < minLen {
|
if ra == 0 || rb == 0 || ra > maxTokenRunes || rb > maxTokenRunes {
|
||||||
minLen = lb
|
return false
|
||||||
|
}
|
||||||
|
minLen := ra
|
||||||
|
if rb < minLen {
|
||||||
|
minLen = rb
|
||||||
}
|
}
|
||||||
d := damerau(a, b)
|
d := damerau(a, b)
|
||||||
switch {
|
switch {
|
||||||
@@ -129,7 +219,8 @@ func sameHouseUnit(a, b address.ParsedAddress) bool {
|
|||||||
a.HouseLetter == b.HouseLetter &&
|
a.HouseLetter == b.HouseLetter &&
|
||||||
a.Building == b.Building &&
|
a.Building == b.Building &&
|
||||||
a.BuildingFrac == b.BuildingFrac &&
|
a.BuildingFrac == b.BuildingFrac &&
|
||||||
a.BuildingLetter == b.BuildingLetter
|
a.BuildingLetter == b.BuildingLetter &&
|
||||||
|
a.Structure == b.Structure
|
||||||
}
|
}
|
||||||
|
|
||||||
// better ranks results: a stronger status wins, then higher confidence.
|
// better ranks results: a stronger status wins, then higher confidence.
|
||||||
|
|||||||
@@ -42,6 +42,26 @@ func TestCompatibleVariantProtectsApartment(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCompatibleVariantProtectsCityAndStructure makes sure a rewrite cannot
|
||||||
|
// move the address to another city or another building.
|
||||||
|
func TestCompatibleVariantProtectsCityAndStructure(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary([]address.City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск"}},
|
||||||
|
{Name: "Бердск"},
|
||||||
|
}), "")
|
||||||
|
orig := parser.Parse("нск, Сухарная улица 95 стр. 1")
|
||||||
|
|
||||||
|
if !compatibleVariant(orig, parser.Parse("Новосибирск, Сухраная улица 95 стр. 1")) {
|
||||||
|
t.Error("same-city spelling fix must be accepted")
|
||||||
|
}
|
||||||
|
if compatibleVariant(orig, parser.Parse("Бердск, Сухарная улица 95 стр. 1")) {
|
||||||
|
t.Error("city change must be rejected")
|
||||||
|
}
|
||||||
|
if compatibleVariant(orig, parser.Parse("нск, Сухарная улица 95 стр. 2")) {
|
||||||
|
t.Error("structure change must be rejected")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDamerau(t *testing.T) {
|
func TestDamerau(t *testing.T) {
|
||||||
if got := damerau("гоглоя", "гоголя"); got != 1 {
|
if got := damerau("гоглоя", "гоголя"); got != 1 {
|
||||||
t.Errorf("transposition distance = %d, want 1", got)
|
t.Errorf("transposition distance = %d, want 1", got)
|
||||||
|
|||||||
@@ -1,132 +0,0 @@
|
|||||||
// Package outages contains the deterministic, pure logic for selecting outages
|
|
||||||
// that belong to a concrete house unit. It does not talk to the network: the
|
|
||||||
// portal client lives in an adapter and feeds this package.
|
|
||||||
package outages
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Outage is a single outage record as returned by the external portal.
|
|
||||||
type Outage struct {
|
|
||||||
Type string `json:"type,omitempty"`
|
|
||||||
System string `json:"system,omitempty"`
|
|
||||||
Address string `json:"address,omitempty"`
|
|
||||||
StartDate string `json:"start_date,omitempty"`
|
|
||||||
EndDate string `json:"end_date,omitempty"`
|
|
||||||
Time string `json:"time,omitempty"`
|
|
||||||
Action string `json:"action,omitempty"`
|
|
||||||
|
|
||||||
// Unit is the structured house unit, filled by the source adapter so the
|
|
||||||
// domain never has to parse the raw external address format. Not exported
|
|
||||||
// to JSON: it is an internal matching key.
|
|
||||||
Unit HouseUnit `json:"-"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// HouseUnit identifies a concrete house within a street: number, optional
|
|
||||||
// fraction and letter, and the corpus (building) with its own fraction and
|
|
||||||
// letter. "Outages by house" matches on exactly this.
|
|
||||||
type HouseUnit struct {
|
|
||||||
House string
|
|
||||||
HouseFrac string
|
|
||||||
HouseLetter string
|
|
||||||
Building string
|
|
||||||
BuildingFrac string
|
|
||||||
BuildingLetter string
|
|
||||||
}
|
|
||||||
|
|
||||||
// HouseUnitOf extracts the unit from a parsed address.
|
|
||||||
func HouseUnitOf(p address.ParsedAddress) HouseUnit {
|
|
||||||
return HouseUnit{
|
|
||||||
House: p.House,
|
|
||||||
HouseFrac: p.HouseFrac,
|
|
||||||
HouseLetter: p.HouseLetter,
|
|
||||||
Building: p.Building,
|
|
||||||
BuildingFrac: p.BuildingFrac,
|
|
||||||
BuildingLetter: p.BuildingLetter,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Source is anything that can answer "what outages are there for this
|
|
||||||
// canonical address". It is implemented by the ArcGIS portal adapter and by
|
|
||||||
// fakes in tests.
|
|
||||||
type Source interface {
|
|
||||||
Query(ctx context.Context, canonical string) ([]Outage, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// MatchingHouse keeps only the outages that address the same house unit as the
|
|
||||||
// target. This is the fix for the classic bug where a substring match returned
|
|
||||||
// a neighbouring house's outage: matching is done on the parsed structure, not
|
|
||||||
// on the raw text.
|
|
||||||
func MatchingHouse(list []Outage, target address.ParsedAddress) []Outage {
|
|
||||||
want := HouseUnitOf(target)
|
|
||||||
out := make([]Outage, 0, len(list))
|
|
||||||
for _, o := range list {
|
|
||||||
unit := o.Unit
|
|
||||||
if unit == (HouseUnit{}) {
|
|
||||||
// The source did not pre-parse the address (older/alternative
|
|
||||||
// adapters); fall back to parsing it here. Adapters should set
|
|
||||||
// Unit so the domain stays independent of external formats.
|
|
||||||
unit = HouseUnitOf(address.Parse(o.Address))
|
|
||||||
}
|
|
||||||
if unit == want {
|
|
||||||
out = append(out, o)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Current keeps outages whose [start, end] range covers now. The portal
|
|
||||||
// already filters by date, but we re-check locally because the portal's notion
|
|
||||||
// of "today" and ours must agree (Asia/Novosibirsk, UTC+7).
|
|
||||||
func Current(list []Outage, now time.Time, loc *time.Location) []Outage {
|
|
||||||
out := make([]Outage, 0, len(list))
|
|
||||||
for _, o := range list {
|
|
||||||
start, okStart := parsePortalTime(o.StartDate, loc)
|
|
||||||
end, okEnd := parsePortalTime(o.EndDate, loc)
|
|
||||||
if !okStart || !okEnd {
|
|
||||||
// Undated records are kept: better to report a possibly-current
|
|
||||||
// outage than to silently drop it.
|
|
||||||
out = append(out, o)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if (now.Equal(start) || now.After(start)) && (now.Equal(end) || now.Before(end)) {
|
|
||||||
out = append(out, o)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// parsePortalTime accepts the shapes the portal may produce: epoch
|
|
||||||
// milliseconds, epoch seconds, RFC3339 and a few common date formats. All
|
|
||||||
// results are interpreted in loc.
|
|
||||||
func parsePortalTime(s string, loc *time.Location) (time.Time, bool) {
|
|
||||||
s = strings.TrimSpace(s)
|
|
||||||
if s == "" {
|
|
||||||
return time.Time{}, false
|
|
||||||
}
|
|
||||||
if n, err := strconv.ParseInt(s, 10, 64); err == nil {
|
|
||||||
// Heuristic: 13+ digits is milliseconds, otherwise seconds.
|
|
||||||
if n > 1e12 {
|
|
||||||
return time.UnixMilli(n).In(loc), true
|
|
||||||
}
|
|
||||||
return time.Unix(n, 0).In(loc), true
|
|
||||||
}
|
|
||||||
layouts := []string{
|
|
||||||
time.RFC3339,
|
|
||||||
"2006-01-02T15:04:05",
|
|
||||||
"2006-01-02 15:04:05",
|
|
||||||
"2006-01-02",
|
|
||||||
}
|
|
||||||
for _, layout := range layouts {
|
|
||||||
if t, err := time.ParseInLocation(layout, s, loc); err == nil {
|
|
||||||
return t, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return time.Time{}, false
|
|
||||||
}
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
package outages
|
|
||||||
|
|
||||||
import (
|
|
||||||
"strconv"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestMatchingHouse(t *testing.T) {
|
|
||||||
list := []Outage{
|
|
||||||
{Address: "Станционная улица 1/1", System: "Теплоснабжение"},
|
|
||||||
{Address: "Станционная улица 1/2", System: "Теплоснабжение"},
|
|
||||||
{Address: "Станционная улица 1/1 к1", System: "Теплоснабжение"},
|
|
||||||
{Address: "Станционная улица 10", System: "Теплоснабжение"},
|
|
||||||
}
|
|
||||||
target := address.Parse("2-я Станционная улица 1/1")
|
|
||||||
|
|
||||||
got := MatchingHouse(list, target)
|
|
||||||
if len(got) != 1 {
|
|
||||||
t.Fatalf("expected exactly 1 matching outage, got %d: %+v", len(got), got)
|
|
||||||
}
|
|
||||||
if got[0].Address != "Станционная улица 1/1" {
|
|
||||||
t.Errorf("matched wrong outage: %q", got[0].Address)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMatchingHouseIgnoresStructure(t *testing.T) {
|
|
||||||
list := []Outage{{Address: "Станционная улица 1/1 стр.2"}}
|
|
||||||
target := address.Parse("станционная 1/1 стр.3")
|
|
||||||
if got := MatchingHouse(list, target); len(got) != 1 {
|
|
||||||
t.Fatalf("structure must not affect house matching; got %d", len(got))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMatchingHouseDistrictSuffix covers the fallback path (Unit not set by the
|
|
||||||
// adapter): the parser must itself drop the portal's "(<district> район)".
|
|
||||||
func TestMatchingHouseDistrictSuffix(t *testing.T) {
|
|
||||||
list := []Outage{{Address: "Медкадры улица, 1 (Заельцовский район)"}}
|
|
||||||
target := address.Parse("Медкадры улица 1")
|
|
||||||
if got := MatchingHouse(list, target); len(got) != 1 {
|
|
||||||
t.Fatalf("district suffix must not break house matching; got %d", len(got))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMatchingHouseUsesStructuredUnit verifies the domain matches on the unit
|
|
||||||
// the adapter parsed, not on the raw address text.
|
|
||||||
func TestMatchingHouseUsesStructuredUnit(t *testing.T) {
|
|
||||||
list := []Outage{{Address: "неважно", Unit: HouseUnit{House: "5", HouseFrac: "1"}}}
|
|
||||||
if got := MatchingHouse(list, address.Parse("что угодно 5/1")); len(got) != 1 {
|
|
||||||
t.Fatalf("structured unit not used for matching; got %d", len(got))
|
|
||||||
}
|
|
||||||
if got := MatchingHouse(list, address.Parse("что угодно 5/2")); len(got) != 0 {
|
|
||||||
t.Fatalf("different unit must not match; got %d", len(got))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCurrent(t *testing.T) {
|
|
||||||
loc := time.FixedZone("NSK", 7*3600)
|
|
||||||
now := time.Date(2026, 10, 9, 12, 0, 0, 0, loc)
|
|
||||||
within := now.Add(-time.Hour).UnixMilli()
|
|
||||||
after := now.Add(time.Hour).UnixMilli()
|
|
||||||
before := now.Add(-48 * time.Hour).UnixMilli()
|
|
||||||
|
|
||||||
list := []Outage{
|
|
||||||
{Address: "A", StartDate: itoa(within), EndDate: itoa(after)}, // current
|
|
||||||
{Address: "B", StartDate: itoa(before), EndDate: itoa(before)}, // past
|
|
||||||
{Address: "C", StartDate: "", EndDate: ""}, // undated -> kept
|
|
||||||
}
|
|
||||||
got := Current(list, now, loc)
|
|
||||||
if len(got) != 2 {
|
|
||||||
t.Fatalf("expected 2 current outages, got %d: %+v", len(got), got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func itoa(v int64) string {
|
|
||||||
return strconv.FormatInt(v, 10)
|
|
||||||
}
|
|
||||||
+36
-10
@@ -26,8 +26,12 @@ import (
|
|||||||
// information (ordinal, house letter, fraction) do not require an exact match;
|
// information (ordinal, house letter, fraction) do not require an exact match;
|
||||||
// for them a different, real address is "lossy" rather than "wrong".
|
// for them a different, real address is "lossy" rather than "wrong".
|
||||||
type Case struct {
|
type Case struct {
|
||||||
Query string
|
Query string
|
||||||
Expect string
|
Expect string
|
||||||
|
// ExpectCity is the canonical city of the source entry. Empty for a
|
||||||
|
// city-less index; otherwise a "found" result in another city is a safety
|
||||||
|
// violation even when the street and house text look identical.
|
||||||
|
ExpectCity string
|
||||||
Category string
|
Category string
|
||||||
RequireExact bool
|
RequireExact bool
|
||||||
}
|
}
|
||||||
@@ -41,6 +45,7 @@ type CategoryStat struct {
|
|||||||
Ambiguous int `json:"ambiguous"`
|
Ambiguous int `json:"ambiguous"`
|
||||||
NotFound int `json:"not_found"`
|
NotFound int `json:"not_found"`
|
||||||
WrongHouse int `json:"wrong_house"`
|
WrongHouse int `json:"wrong_house"`
|
||||||
|
WrongCity int `json:"wrong_city"`
|
||||||
WrongStreet int `json:"wrong_street"`
|
WrongStreet int `json:"wrong_street"`
|
||||||
Lossy int `json:"lossy"`
|
Lossy int `json:"lossy"`
|
||||||
}
|
}
|
||||||
@@ -72,11 +77,19 @@ func Run(resolver *match.Resolver, cases []Case) Report {
|
|||||||
case address.StatusFound:
|
case address.StatusFound:
|
||||||
rp := address.Parse(res.Address)
|
rp := address.Parse(res.Address)
|
||||||
equivalent := rp.StreetKey == exp.StreetKey && sameUnit(rp, exp)
|
equivalent := rp.StreetKey == exp.StreetKey && sameUnit(rp, exp)
|
||||||
|
sameCity := c.ExpectCity == "" || res.City == c.ExpectCity
|
||||||
switch {
|
switch {
|
||||||
case equivalent:
|
case equivalent && sameCity:
|
||||||
st.Exact++
|
st.Exact++
|
||||||
rep.Exact++
|
rep.Exact++
|
||||||
exact = true
|
exact = true
|
||||||
|
case !sameCity:
|
||||||
|
// A wrong city is always a safety violation, even when the
|
||||||
|
// case is allowed to be lossy: the address text can be
|
||||||
|
// identical across cities.
|
||||||
|
st.WrongCity++
|
||||||
|
rep.Wrong++
|
||||||
|
rep.WrongExamples = appendExample(rep.WrongExamples, c, res.Address)
|
||||||
case !c.RequireExact:
|
case !c.RequireExact:
|
||||||
// Missing information resolved to a different real address:
|
// Missing information resolved to a different real address:
|
||||||
// expected, not a safety violation.
|
// expected, not a safety violation.
|
||||||
@@ -111,14 +124,17 @@ func Run(resolver *match.Resolver, cases []Case) Report {
|
|||||||
return rep
|
return rep
|
||||||
}
|
}
|
||||||
|
|
||||||
// sameUnit compares house units, ignoring the structure component.
|
// sameUnit compares house units, including the structure: two buildings that
|
||||||
|
// differ only by "стр." are different addresses.
|
||||||
func sameUnit(a, b address.ParsedAddress) bool {
|
func sameUnit(a, b address.ParsedAddress) bool {
|
||||||
return a.House == b.House &&
|
return a.CityKey == b.CityKey &&
|
||||||
|
a.House == b.House &&
|
||||||
a.HouseFrac == b.HouseFrac &&
|
a.HouseFrac == b.HouseFrac &&
|
||||||
a.HouseLetter == b.HouseLetter &&
|
a.HouseLetter == b.HouseLetter &&
|
||||||
a.Building == b.Building &&
|
a.Building == b.Building &&
|
||||||
a.BuildingFrac == b.BuildingFrac &&
|
a.BuildingFrac == b.BuildingFrac &&
|
||||||
a.BuildingLetter == b.BuildingLetter
|
a.BuildingLetter == b.BuildingLetter &&
|
||||||
|
a.Structure == b.Structure
|
||||||
}
|
}
|
||||||
|
|
||||||
func appendExample(examples []string, c Case, got string) []string {
|
func appendExample(examples []string, c Case, got string) []string {
|
||||||
@@ -185,16 +201,24 @@ func perturbations(e address.Entry, rng *rand.Rand) []Case {
|
|||||||
var out []Case
|
var out []Case
|
||||||
add := func(query, category string, requireExact bool) {
|
add := func(query, category string, requireExact bool) {
|
||||||
if query != "" && query != base {
|
if query != "" && query != base {
|
||||||
out = append(out, Case{Query: query, Expect: base, Category: category, RequireExact: requireExact})
|
out = append(out, Case{
|
||||||
|
Query: query, Expect: base, ExpectCity: e.City,
|
||||||
|
Category: category, RequireExact: requireExact,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Baseline: the canonical address itself must always resolve exactly.
|
// Baseline: the canonical address itself must always resolve exactly.
|
||||||
out = append(out, Case{Query: base, Expect: base, Category: "baseline", RequireExact: true})
|
out = append(out, Case{
|
||||||
|
Query: base, Expect: base, ExpectCity: e.City,
|
||||||
|
Category: "baseline", RequireExact: true,
|
||||||
|
})
|
||||||
|
|
||||||
add(strings.ToLower(base), "case_lower", true)
|
add(strings.ToLower(base), "case_lower", true)
|
||||||
add(strings.ToUpper(base), "case_upper", true)
|
add(strings.ToUpper(base), "case_upper", true)
|
||||||
add("г. Новосибирск, "+base, "city_prefix", true)
|
if e.City != "" {
|
||||||
|
add("г. "+e.City+", "+base, "city_prefix", true)
|
||||||
|
}
|
||||||
add(insertDom(base, e.House), "filler_dom", true)
|
add(insertDom(base, e.House), "filler_dom", true)
|
||||||
add(corruptPunctuation(base), "punctuation", true)
|
add(corruptPunctuation(base), "punctuation", true)
|
||||||
add(addYo(base), "yo", true)
|
add(addYo(base), "yo", true)
|
||||||
@@ -212,7 +236,9 @@ func perturbations(e address.Entry, rng *rand.Rand) []Case {
|
|||||||
add(strings.Replace(base, "к"+e.Building, "корпус "+e.Building, 1), "corpus_synonym", true)
|
add(strings.Replace(base, "к"+e.Building, "корпус "+e.Building, 1), "corpus_synonym", true)
|
||||||
}
|
}
|
||||||
if hasStructure(base) {
|
if hasStructure(base) {
|
||||||
add(dropStructure(base), "structure_drop", true)
|
// Dropping "стр. N" discards information: the base house may or may
|
||||||
|
// not exist on its own, so a different real address is acceptable.
|
||||||
|
add(dropStructure(base), "structure_drop", false)
|
||||||
}
|
}
|
||||||
if q := mutateLongestToken(base, rng); q != "" {
|
if q := mutateLongestToken(base, rng); q != "" {
|
||||||
add(q, "typo", true)
|
add(q, "typo", true)
|
||||||
|
|||||||
@@ -4,6 +4,9 @@ import (
|
|||||||
"math/rand"
|
"math/rand"
|
||||||
"testing"
|
"testing"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestMoveTypeFront guards against the slice-aliasing regression that made the
|
// TestMoveTypeFront guards against the slice-aliasing regression that made the
|
||||||
@@ -47,3 +50,37 @@ func TestMutateOnlyNameTokens(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRunDetectsWrongCity guards the multi-city safety net: canonical address
|
||||||
|
// text is identical across cities, so a found result must be checked against
|
||||||
|
// the expected city, not only against street/house text.
|
||||||
|
func TestRunDetectsWrongCity(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary([]address.City{{Name: "Бердск"}}), "")
|
||||||
|
p := parser.ParseInCity("Ленина улица 5", "Бердск")
|
||||||
|
entries := []address.Entry{{
|
||||||
|
ID: "b1", Address: "Ленина улица 5",
|
||||||
|
City: p.City, CityKey: p.CityKey,
|
||||||
|
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetKey: p.StreetKey, House: p.House,
|
||||||
|
}}
|
||||||
|
resolver := match.NewResolverWithOptions(
|
||||||
|
match.NewIndex("test", entries, match.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
match.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
|
cases := []Case{
|
||||||
|
{Query: "Ленина 5", Expect: "Ленина улица 5", ExpectCity: "Новосибирск", Category: "wrong_city", RequireExact: true},
|
||||||
|
{Query: "Ленина 5", Expect: "Ленина улица 5", ExpectCity: "Бердск", Category: "right_city", RequireExact: true},
|
||||||
|
}
|
||||||
|
rep := Run(resolver, cases)
|
||||||
|
|
||||||
|
if got := rep.Categories["wrong_city"]; got.WrongCity != 1 || got.Exact != 0 {
|
||||||
|
t.Fatalf("wrong_city = %+v, want WrongCity=1 Exact=0", got)
|
||||||
|
}
|
||||||
|
if got := rep.Categories["right_city"]; got.Exact != 1 || got.WrongCity != 0 {
|
||||||
|
t.Fatalf("right_city = %+v, want Exact=1 WrongCity=0", got)
|
||||||
|
}
|
||||||
|
if rep.Wrong != 1 {
|
||||||
|
t.Fatalf("report Wrong = %d, want 1", rep.Wrong)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ func TestRealRegistryParseRate(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("read registry: %v", err)
|
t.Fatalf("read registry: %v", err)
|
||||||
}
|
}
|
||||||
entries, stats := Build(sources)
|
entries, stats := Build(sources, BuildOptions{})
|
||||||
|
|
||||||
rate := 0.0
|
rate := 0.0
|
||||||
if stats.Total > 0 {
|
if stats.Total > 0 {
|
||||||
|
|||||||
+75
-19
@@ -20,24 +20,38 @@ type Source struct {
|
|||||||
ID string
|
ID string
|
||||||
Address string
|
Address string
|
||||||
District string
|
District string
|
||||||
|
// City is optional: when the dump is already split per city it may be
|
||||||
|
// omitted and supplied as the default city to Build.
|
||||||
|
City string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stats summarises a build and, crucially, surfaces rows the parser could not
|
// Stats summarises a build and, crucially, surfaces rows the parser could not
|
||||||
// structure. A non-empty UnparsedCount is a data-quality signal worth alerting
|
// structure. A non-empty UnparsedCount is a data-quality signal worth alerting
|
||||||
// on, not an error.
|
// on, not an error.
|
||||||
type Stats struct {
|
type Stats struct {
|
||||||
Total int `json:"total"`
|
Total int `json:"total"`
|
||||||
Parsed int `json:"parsed"`
|
Parsed int `json:"parsed"`
|
||||||
Unparsed int `json:"unparsed"`
|
Unparsed int `json:"unparsed"`
|
||||||
Duplicates int `json:"duplicates"`
|
Duplicates int `json:"duplicates"`
|
||||||
DistinctStreets int `json:"distinct_streets"`
|
DistinctStreets int `json:"distinct_streets"`
|
||||||
UnparsedSample []string `json:"unparsed_sample,omitempty"`
|
Cities map[string]int `json:"cities,omitempty"`
|
||||||
|
UnparsedSample []string `json:"unparsed_sample,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildOptions controls how source rows become canonical entries.
|
||||||
|
type BuildOptions struct {
|
||||||
|
// Parser defaults to address.DefaultParser().
|
||||||
|
Parser *address.Parser
|
||||||
|
// DefaultCity is used for rows that neither carry a city field nor name a
|
||||||
|
// city in the address. Leave it empty for multi-city dumps where every row
|
||||||
|
// is expected to identify its city.
|
||||||
|
DefaultCity string
|
||||||
}
|
}
|
||||||
|
|
||||||
// ReadBulkNDJSON reads an Elasticsearch Bulk NDJSON dump: pairs of an action
|
// ReadBulkNDJSON reads an Elasticsearch Bulk NDJSON dump: pairs of an action
|
||||||
// line ({"index":{"_id":...}}) followed by a source line
|
// line ({"index":{"_id":...}}) followed by a source line
|
||||||
// ({"address":...,"district_name":...}). Bare source lines (no action line)
|
// ({"address":...,"district_name":...,"city":...}). Bare source lines (no
|
||||||
// are also accepted, which makes plain JSONL usable too.
|
// action line) are also accepted, which makes plain JSONL usable too.
|
||||||
func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
|
func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
|
||||||
sc := bufio.NewScanner(r)
|
sc := bufio.NewScanner(r)
|
||||||
// 1 MiB per line is far beyond any realistic address row.
|
// 1 MiB per line is far beyond any realistic address row.
|
||||||
@@ -75,6 +89,7 @@ func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
|
|||||||
if d, ok := obj["district_name"]; ok {
|
if d, ok := obj["district_name"]; ok {
|
||||||
_ = json.Unmarshal(d, &src.District)
|
_ = json.Unmarshal(d, &src.District)
|
||||||
}
|
}
|
||||||
|
src.City = firstString(obj, "city", "city_name")
|
||||||
sources = append(sources, src)
|
sources = append(sources, src)
|
||||||
pendingID = ""
|
pendingID = ""
|
||||||
}
|
}
|
||||||
@@ -85,17 +100,40 @@ func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
|
|||||||
return sources, nil
|
return sources, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// firstString returns the first non-empty string field among names.
|
||||||
|
func firstString(obj map[string]json.RawMessage, names ...string) string {
|
||||||
|
for _, name := range names {
|
||||||
|
raw, ok := obj[name]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var s string
|
||||||
|
if err := json.Unmarshal(raw, &s); err == nil && strings.TrimSpace(s) != "" {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
// Build parses every source row into a canonical Entry and reports statistics.
|
// Build parses every source row into a canonical Entry and reports statistics.
|
||||||
// Rows without a house number are still indexed for street lookups but counted
|
// Rows without a house number are still indexed for street lookups but counted
|
||||||
// as unparsed, since a registry row without a house is suspicious.
|
// as unparsed, since a registry row without a house is suspicious.
|
||||||
func Build(sources []Source) ([]address.Entry, Stats) {
|
func Build(sources []Source, opts BuildOptions) ([]address.Entry, Stats) {
|
||||||
|
parser := opts.Parser
|
||||||
|
if parser == nil {
|
||||||
|
parser = address.NewParser(address.DefaultDictionary(), opts.DefaultCity)
|
||||||
|
}
|
||||||
entries := make([]address.Entry, 0, len(sources))
|
entries := make([]address.Entry, 0, len(sources))
|
||||||
streets := make(map[string]struct{})
|
streets := make(map[string]struct{})
|
||||||
seen := make(map[string]struct{})
|
seen := make(map[string]struct{})
|
||||||
stats := Stats{Total: len(sources)}
|
stats := Stats{Total: len(sources), Cities: make(map[string]int)}
|
||||||
|
|
||||||
for _, src := range sources {
|
for _, src := range sources {
|
||||||
p := address.Parse(src.Address)
|
// Precedence: an explicit source field wins over a city found in the
|
||||||
|
// address text; the parser's default city is the last resort. Passing
|
||||||
|
// an empty hint is what lets the parser apply that precedence.
|
||||||
|
cityHint := strings.TrimSpace(src.City)
|
||||||
|
p := parser.ParseInCity(src.Address, cityHint)
|
||||||
if p.NameKey == "" {
|
if p.NameKey == "" {
|
||||||
stats.Unparsed++
|
stats.Unparsed++
|
||||||
appendSample(&stats, src.Address)
|
appendSample(&stats, src.Address)
|
||||||
@@ -110,9 +148,11 @@ func Build(sources []Source) ([]address.Entry, Stats) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
entry := address.Entry{
|
entry := address.Entry{
|
||||||
ID: src.ID,
|
ID: namespacedID(src.ID, p.CityKey),
|
||||||
Address: src.Address,
|
Address: src.Address,
|
||||||
District: src.District,
|
District: src.District,
|
||||||
|
City: p.City,
|
||||||
|
CityKey: p.CityKey,
|
||||||
Ordinal: p.Ordinal,
|
Ordinal: p.Ordinal,
|
||||||
NameKey: p.NameKey,
|
NameKey: p.NameKey,
|
||||||
NameDisplay: p.NameDisplay,
|
NameDisplay: p.NameDisplay,
|
||||||
@@ -128,9 +168,8 @@ func Build(sources []Source) ([]address.Entry, Stats) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The source registry contains rows that differ only by "стр."
|
// The source registry contains rows that differ only by "стр."
|
||||||
// (structure). Since structure does not affect which house an outage
|
// (structure). They are distinct buildings, so structure is part of
|
||||||
// belongs to, drop such duplicates rather than surfacing them as
|
// the de-duplication key and an exact structure query can be honoured.
|
||||||
// spurious ambiguity at query time.
|
|
||||||
if key := dedupKey(entry); key != "" {
|
if key := dedupKey(entry); key != "" {
|
||||||
if _, ok := seen[key]; ok {
|
if _, ok := seen[key]; ok {
|
||||||
stats.Duplicates++
|
stats.Duplicates++
|
||||||
@@ -141,24 +180,41 @@ func Build(sources []Source) ([]address.Entry, Stats) {
|
|||||||
|
|
||||||
entries = append(entries, entry)
|
entries = append(entries, entry)
|
||||||
streets[p.StreetKey] = struct{}{}
|
streets[p.StreetKey] = struct{}{}
|
||||||
|
stats.Cities[cityLabel(p.City)]++
|
||||||
}
|
}
|
||||||
stats.DistinctStreets = len(streets)
|
stats.DistinctStreets = len(streets)
|
||||||
sort.Strings(stats.UnparsedSample)
|
sort.Strings(stats.UnparsedSample)
|
||||||
return entries, stats
|
return entries, stats
|
||||||
}
|
}
|
||||||
|
|
||||||
// dedupKey identifies the same house unit within the same street, ignoring the
|
// namespacedID keeps source ids unique when several city registries are merged
|
||||||
// structure component.
|
// into one database. Source systems often number houses independently.
|
||||||
|
func namespacedID(id, cityKey string) string {
|
||||||
|
if id == "" || cityKey == "" {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
return cityKey + "/" + id
|
||||||
|
}
|
||||||
|
|
||||||
|
// dedupKey identifies the same house unit within the same city and street,
|
||||||
|
// including its structure and building components.
|
||||||
func dedupKey(e address.Entry) string {
|
func dedupKey(e address.Entry) string {
|
||||||
if e.House == "" {
|
if e.House == "" {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
return strings.Join([]string{
|
return strings.Join([]string{
|
||||||
e.StreetKey, e.House, e.HouseFrac, e.HouseLetter,
|
e.CityKey, e.StreetKey, e.House, e.HouseFrac, e.HouseLetter,
|
||||||
e.Building, e.BuildingFrac, e.BuildingLetter,
|
e.Building, e.BuildingFrac, e.BuildingLetter, e.Structure,
|
||||||
}, "|")
|
}, "|")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cityLabel(city string) string {
|
||||||
|
if city == "" {
|
||||||
|
return "(none)"
|
||||||
|
}
|
||||||
|
return city
|
||||||
|
}
|
||||||
|
|
||||||
func appendSample(s *Stats, address string) {
|
func appendSample(s *Stats, address string) {
|
||||||
const maxSamples = 20
|
const maxSamples = 20
|
||||||
if len(s.UnparsedSample) < maxSamples {
|
if len(s.UnparsedSample) < maxSamples {
|
||||||
|
|||||||
@@ -3,11 +3,13 @@ package seed
|
|||||||
import (
|
import (
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestReadBulkNDJSON(t *testing.T) {
|
func TestReadBulkNDJSON(t *testing.T) {
|
||||||
input := `{"index": {"_index": "address_registry", "_id": "a1"}}
|
input := `{"index": {"_index": "address_registry", "_id": "a1"}}
|
||||||
{"address": "Станционная улица 1/1", "district_name": "Ленинский"}
|
{"address": "Станционная улица 1/1", "district_name": "Ленинский", "city": "Новосибирск"}
|
||||||
{"index": {"_id": "a2"}}
|
{"index": {"_id": "a2"}}
|
||||||
{"address": "Искитимская улица 5", "district_name": "Кировский"}
|
{"address": "Искитимская улица 5", "district_name": "Кировский"}
|
||||||
|
|
||||||
@@ -20,7 +22,7 @@ func TestReadBulkNDJSON(t *testing.T) {
|
|||||||
if len(src) != 3 {
|
if len(src) != 3 {
|
||||||
t.Fatalf("got %d sources, want 3", len(src))
|
t.Fatalf("got %d sources, want 3", len(src))
|
||||||
}
|
}
|
||||||
if src[0].ID != "a1" || src[0].District != "Ленинский" {
|
if src[0].ID != "a1" || src[0].District != "Ленинский" || src[0].City != "Новосибирск" {
|
||||||
t.Errorf("first source = %+v", src[0])
|
t.Errorf("first source = %+v", src[0])
|
||||||
}
|
}
|
||||||
if src[2].Address != "Третья улица 7" {
|
if src[2].Address != "Третья улица 7" {
|
||||||
@@ -34,7 +36,7 @@ func TestBuild(t *testing.T) {
|
|||||||
{ID: "2", Address: "Искитимская улица 5"},
|
{ID: "2", Address: "Искитимская улица 5"},
|
||||||
{ID: "3", Address: "Станционная улица"}, // no house -> unparsed
|
{ID: "3", Address: "Станционная улица"}, // no house -> unparsed
|
||||||
}
|
}
|
||||||
entries, stats := Build(sources)
|
entries, stats := Build(sources, BuildOptions{})
|
||||||
|
|
||||||
if stats.Total != 3 {
|
if stats.Total != 3 {
|
||||||
t.Errorf("Total = %d, want 3", stats.Total)
|
t.Errorf("Total = %d, want 3", stats.Total)
|
||||||
@@ -53,3 +55,71 @@ func TestBuild(t *testing.T) {
|
|||||||
t.Errorf("DistinctStreets = %d, want 2", stats.DistinctStreets)
|
t.Errorf("DistinctStreets = %d, want 2", stats.DistinctStreets)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildKeepsCitiesApart(t *testing.T) {
|
||||||
|
sources := []Source{
|
||||||
|
{ID: "1", Address: "Ленина улица 5", City: "Новосибирск"},
|
||||||
|
{ID: "1", Address: "Ленина улица 5", City: "Бердск"},
|
||||||
|
{ID: "2", Address: "Ленина улица 7", City: "Новосибирск"},
|
||||||
|
}
|
||||||
|
entries, stats := Build(sources, BuildOptions{})
|
||||||
|
|
||||||
|
if len(entries) != 3 {
|
||||||
|
t.Fatalf("entries = %d, want 3: %+v", len(entries), entries)
|
||||||
|
}
|
||||||
|
if stats.DistinctStreets != 2 {
|
||||||
|
t.Errorf("DistinctStreets = %d, want 2 (same street in two cities)", stats.DistinctStreets)
|
||||||
|
}
|
||||||
|
if entries[0].StreetKey == entries[1].StreetKey {
|
||||||
|
t.Errorf("streets from different cities share a key: %q", entries[0].StreetKey)
|
||||||
|
}
|
||||||
|
if entries[0].ID == entries[1].ID {
|
||||||
|
t.Errorf("source ids collided across cities: %q", entries[0].ID)
|
||||||
|
}
|
||||||
|
if stats.Cities["Новосибирск"] != 2 || stats.Cities["Бердск"] != 1 {
|
||||||
|
t.Errorf("city stats = %v", stats.Cities)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildAppliesDefaultCity(t *testing.T) {
|
||||||
|
sources := []Source{{ID: "1", Address: "Станционная улица 1/1"}}
|
||||||
|
entries, _ := Build(sources, BuildOptions{DefaultCity: "Новосибирск"})
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Fatalf("entries = %d", len(entries))
|
||||||
|
}
|
||||||
|
if entries[0].City != "Новосибирск" || entries[0].CityKey != "новосибирск" {
|
||||||
|
t.Errorf("city = %q / %q", entries[0].City, entries[0].CityKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildKeepsStructuresDistinct(t *testing.T) {
|
||||||
|
sources := []Source{
|
||||||
|
{ID: "1", Address: "Станционная улица 1"},
|
||||||
|
{ID: "2", Address: "Станционная улица 1 стр. 2"},
|
||||||
|
}
|
||||||
|
entries, stats := Build(sources, BuildOptions{})
|
||||||
|
|
||||||
|
if len(entries) != 2 {
|
||||||
|
t.Fatalf("entries = %d, want 2 (structure is part of identity)", len(entries))
|
||||||
|
}
|
||||||
|
if stats.Duplicates != 0 {
|
||||||
|
t.Errorf("Duplicates = %d, want 0", stats.Duplicates)
|
||||||
|
}
|
||||||
|
if entries[0].Structure == entries[1].Structure {
|
||||||
|
t.Errorf("structures collapsed: %+v", entries)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildParsesCityFromAddress(t *testing.T) {
|
||||||
|
parser := address.NewParser(address.NewDictionary([]address.City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск", "новосибирская область"}},
|
||||||
|
}), "")
|
||||||
|
sources := []Source{{ID: "1", Address: "нск, Станционная улица 1/1"}}
|
||||||
|
entries, _ := Build(sources, BuildOptions{Parser: parser})
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Fatalf("entries = %d", len(entries))
|
||||||
|
}
|
||||||
|
if entries[0].City != "Новосибирск" || entries[0].NameKey != "станционная" {
|
||||||
|
t.Errorf("entry = %+v", entries[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
// Package addrkit is the public library facade of the address normalizer.
|
||||||
|
//
|
||||||
|
// The service (cmd/addrkitd) and the CLI use the same domain packages through
|
||||||
|
// this API. Another project in a different repository can import this package
|
||||||
|
// directly instead of calling the HTTP service; nothing here does I/O, opens a
|
||||||
|
// database or depends on the service configuration.
|
||||||
|
//
|
||||||
|
// Example:
|
||||||
|
//
|
||||||
|
// parser := addrkit.NewParser(
|
||||||
|
// addrkit.NewDictionary([]addrkit.City{
|
||||||
|
// {Name: "Новосибирск", Aliases: []string{"нск"}},
|
||||||
|
// }),
|
||||||
|
// "",
|
||||||
|
// )
|
||||||
|
// parsed := parser.Parse("нск, Ленина 5")
|
||||||
|
//
|
||||||
|
// index := addrkit.NewIndex("2026-10-01", entries, addrkit.DefaultOptions())
|
||||||
|
// resolver := addrkit.NewResolver(index, nil)
|
||||||
|
// result := resolver.Resolve(context.Background(), "нск, Ленина 5")
|
||||||
|
package addrkit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Core value types.
|
||||||
|
type (
|
||||||
|
// City is one entry of the configurable city dictionary.
|
||||||
|
City = address.City
|
||||||
|
// ParsedAddress is the structured form of a free-form address.
|
||||||
|
ParsedAddress = address.ParsedAddress
|
||||||
|
// Entry is one canonical registry row.
|
||||||
|
Entry = address.Entry
|
||||||
|
// Status is the machine-readable outcome of a resolution.
|
||||||
|
Status = address.Status
|
||||||
|
)
|
||||||
|
|
||||||
|
// Resolution statuses.
|
||||||
|
const (
|
||||||
|
StatusFound = address.StatusFound
|
||||||
|
StatusPartial = address.StatusPartial
|
||||||
|
StatusAmbiguous = address.StatusAmbiguous
|
||||||
|
StatusNotFound = address.StatusNotFound
|
||||||
|
)
|
||||||
|
|
||||||
|
// Dictionary is the immutable vocabulary used by the parser.
|
||||||
|
type Dictionary = address.Dictionary
|
||||||
|
|
||||||
|
// NewDictionary builds a dictionary with the built-in vocabulary and cities.
|
||||||
|
func NewDictionary(cities []City) *Dictionary {
|
||||||
|
return address.NewDictionary(cities)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parser turns free-form address strings into structured addresses.
|
||||||
|
type Parser = address.Parser
|
||||||
|
|
||||||
|
// NewParser builds a parser. defaultCity may be empty.
|
||||||
|
func NewParser(dict *Dictionary, defaultCity string) *Parser {
|
||||||
|
return address.NewParser(dict, defaultCity)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse uses the shared city-less parser.
|
||||||
|
func Parse(raw string) ParsedAddress {
|
||||||
|
return address.Parse(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Matching and resolution.
|
||||||
|
type (
|
||||||
|
// Options tunes the index.
|
||||||
|
Options = match.Options
|
||||||
|
// Candidate is one plausible match.
|
||||||
|
Candidate = match.Candidate
|
||||||
|
// Result is the outcome of a resolution.
|
||||||
|
Result = match.Result
|
||||||
|
// Rewriter is the optional query rewriter (for example an LLM).
|
||||||
|
Rewriter = match.Rewriter
|
||||||
|
// Index is an immutable in-memory registry index.
|
||||||
|
Index = match.Index
|
||||||
|
// Resolver resolves parsed addresses against an index.
|
||||||
|
Resolver = match.Resolver
|
||||||
|
)
|
||||||
|
|
||||||
|
// ResolverOptions tunes optional resolver behaviour.
|
||||||
|
type ResolverOptions struct {
|
||||||
|
// Parser defaults to a city-less parser.
|
||||||
|
Parser *Parser
|
||||||
|
// OnRewriteError observes failures of the optional rewriter. The resolver
|
||||||
|
// always keeps the deterministic result.
|
||||||
|
OnRewriteError func(error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultOptions returns the conservative matcher defaults.
|
||||||
|
func DefaultOptions() Options {
|
||||||
|
return match.DefaultOptions()
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewIndex builds an index from canonical entries.
|
||||||
|
func NewIndex(version string, entries []Entry, opts Options) *Index {
|
||||||
|
return match.NewIndex(version, entries, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewResolver builds a resolver with the default parser.
|
||||||
|
func NewResolver(index *Index, rewriter Rewriter) *Resolver {
|
||||||
|
return match.NewResolver(index, rewriter)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewResolverWithOptions builds a resolver with explicit options.
|
||||||
|
func NewResolverWithOptions(index *Index, rewriter Rewriter, opts ResolverOptions) *Resolver {
|
||||||
|
return match.NewResolverWithOptions(index, rewriter, match.ResolverOptions{
|
||||||
|
Parser: opts.Parser,
|
||||||
|
OnRewriteError: opts.OnRewriteError,
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
package addrkit_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.totmin.ru/en2zmax/addrkit/pkg/addrkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestPublicAPI exercises the facade exactly the way another repository would:
|
||||||
|
// parse a city-aware query and resolve it against an in-memory index.
|
||||||
|
func TestPublicAPI(t *testing.T) {
|
||||||
|
dict := addrkit.NewDictionary([]addrkit.City{
|
||||||
|
{Name: "Новосибирск", Aliases: []string{"нск"}},
|
||||||
|
{Name: "Бердск"},
|
||||||
|
})
|
||||||
|
parser := addrkit.NewParser(dict, "")
|
||||||
|
|
||||||
|
entries := make([]addrkit.Entry, 0, 2)
|
||||||
|
for _, city := range []string{"Новосибирск", "Бердск"} {
|
||||||
|
p := parser.ParseInCity("Ленина улица 5", city)
|
||||||
|
entries = append(entries, addrkit.Entry{
|
||||||
|
ID: city, Address: "Ленина улица 5",
|
||||||
|
City: p.City, CityKey: p.CityKey,
|
||||||
|
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
|
||||||
|
StreetType: p.StreetType, StreetKey: p.StreetKey,
|
||||||
|
House: p.House,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
resolver := addrkit.NewResolverWithOptions(
|
||||||
|
addrkit.NewIndex("2026-10-01", entries, addrkit.DefaultOptions()),
|
||||||
|
nil,
|
||||||
|
addrkit.ResolverOptions{Parser: parser},
|
||||||
|
)
|
||||||
|
|
||||||
|
res := resolver.ResolveInCity(context.Background(), "нск, Ленина 5", "")
|
||||||
|
if res.Status != addrkit.StatusFound || res.City != "Новосибирск" {
|
||||||
|
t.Fatalf("result = %+v", res)
|
||||||
|
}
|
||||||
|
|
||||||
|
ambiguous := resolver.Resolve(context.Background(), "Ленина 5")
|
||||||
|
if ambiguous.Status != addrkit.StatusAmbiguous {
|
||||||
|
t.Fatalf("cityless result = %+v", ambiguous)
|
||||||
|
}
|
||||||
|
}
|
||||||
+53
-31
@@ -1,10 +1,12 @@
|
|||||||
openapi: 3.1.0
|
openapi: 3.1.0
|
||||||
info:
|
info:
|
||||||
title: addrkit
|
title: addrkit
|
||||||
version: "1.0.0"
|
version: "2.0.0"
|
||||||
description: >
|
description: >
|
||||||
Детерминированный сервис нормализации и поиска адресов. Ответ всегда
|
Детерминированный многогородской сервис нормализации и поиска адресов.
|
||||||
берётся из реестра; LLM не участвует в пути корректности.
|
Ответ всегда берётся из реестра; LLM не участвует в пути корректности.
|
||||||
|
Сервис не знает ни про отключения, ни про другие события: это отдельная
|
||||||
|
задача, которая может использовать addrkit как нормализатор.
|
||||||
servers:
|
servers:
|
||||||
- url: http://localhost:6171
|
- url: http://localhost:6171
|
||||||
paths:
|
paths:
|
||||||
@@ -21,7 +23,15 @@ paths:
|
|||||||
properties:
|
properties:
|
||||||
address:
|
address:
|
||||||
type: string
|
type: string
|
||||||
example: "2-я Станционная улица 1/1 стр."
|
maxLength: 512
|
||||||
|
example: "нск, 2-я Станционная улица 1/1 стр."
|
||||||
|
city:
|
||||||
|
type: string
|
||||||
|
maxLength: 128
|
||||||
|
description: >
|
||||||
|
Необязательная подсказка. Переопределяет город, найденный
|
||||||
|
в тексте адреса.
|
||||||
|
example: "Новосибирск"
|
||||||
responses:
|
responses:
|
||||||
"200":
|
"200":
|
||||||
description: Результат сопоставления
|
description: Результат сопоставления
|
||||||
@@ -31,26 +41,54 @@ paths:
|
|||||||
$ref: "#/components/schemas/ResolveResult"
|
$ref: "#/components/schemas/ResolveResult"
|
||||||
"400":
|
"400":
|
||||||
description: Некорректный запрос
|
description: Некорректный запрос
|
||||||
/v1/outages:
|
/v1/city:
|
||||||
post:
|
post:
|
||||||
summary: Отключения по адресу (после строгого сопоставления дома)
|
summary: Нормализовать только название города
|
||||||
|
description: >
|
||||||
|
Отдельный сценарий для проектов, которым нужны названия городов, а не
|
||||||
|
улицы. Возвращает канонический город по имени или алиасу.
|
||||||
requestBody:
|
requestBody:
|
||||||
required: true
|
required: true
|
||||||
content:
|
content:
|
||||||
application/json:
|
application/json:
|
||||||
schema:
|
schema:
|
||||||
type: object
|
type: object
|
||||||
required: [address]
|
required: [query]
|
||||||
properties:
|
properties:
|
||||||
address:
|
query:
|
||||||
type: string
|
type: string
|
||||||
|
maxLength: 128
|
||||||
|
example: "нск"
|
||||||
responses:
|
responses:
|
||||||
"200":
|
"200":
|
||||||
description: Канонический адрес и отключения
|
description: Результат
|
||||||
content:
|
content:
|
||||||
application/json:
|
application/json:
|
||||||
schema:
|
schema:
|
||||||
$ref: "#/components/schemas/OutagesResult"
|
type: object
|
||||||
|
required: [status]
|
||||||
|
properties:
|
||||||
|
status:
|
||||||
|
type: string
|
||||||
|
enum: [found, not_found]
|
||||||
|
city: { type: string }
|
||||||
|
"400":
|
||||||
|
description: Некорректный запрос
|
||||||
|
/v1/cities:
|
||||||
|
get:
|
||||||
|
summary: Канонические города, присутствующие в индексе
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: Список городов
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
type: object
|
||||||
|
required: [cities]
|
||||||
|
properties:
|
||||||
|
cities:
|
||||||
|
type: array
|
||||||
|
items: { type: string }
|
||||||
/v1/healthz:
|
/v1/healthz:
|
||||||
get:
|
get:
|
||||||
summary: Живость
|
summary: Живость
|
||||||
@@ -78,13 +116,16 @@ components:
|
|||||||
schemas:
|
schemas:
|
||||||
Status:
|
Status:
|
||||||
type: string
|
type: string
|
||||||
enum: [found, partial, ambiguous, not_found, portal_timeout, infra_error]
|
enum: [found, partial, ambiguous, not_found]
|
||||||
Candidate:
|
Candidate:
|
||||||
type: object
|
type: object
|
||||||
|
required: [address, score]
|
||||||
properties:
|
properties:
|
||||||
address: { type: string }
|
address: { type: string }
|
||||||
|
city: { type: string }
|
||||||
district: { type: string }
|
district: { type: string }
|
||||||
house: { type: string }
|
house: { type: string }
|
||||||
|
structure: { type: string }
|
||||||
score: { type: number, format: double }
|
score: { type: number, format: double }
|
||||||
ResolveResult:
|
ResolveResult:
|
||||||
type: object
|
type: object
|
||||||
@@ -92,29 +133,10 @@ components:
|
|||||||
properties:
|
properties:
|
||||||
status: { $ref: "#/components/schemas/Status" }
|
status: { $ref: "#/components/schemas/Status" }
|
||||||
address: { type: string }
|
address: { type: string }
|
||||||
|
city: { type: string }
|
||||||
district: { type: string }
|
district: { type: string }
|
||||||
confidence: { type: number, format: double }
|
confidence: { type: number, format: double }
|
||||||
candidates:
|
candidates:
|
||||||
type: array
|
type: array
|
||||||
items: { $ref: "#/components/schemas/Candidate" }
|
items: { $ref: "#/components/schemas/Candidate" }
|
||||||
data_version: { type: string }
|
data_version: { type: string }
|
||||||
Outage:
|
|
||||||
type: object
|
|
||||||
properties:
|
|
||||||
type: { type: string }
|
|
||||||
system: { type: string }
|
|
||||||
address: { type: string }
|
|
||||||
start_date: { type: string }
|
|
||||||
end_date: { type: string }
|
|
||||||
action: { type: string }
|
|
||||||
OutagesResult:
|
|
||||||
type: object
|
|
||||||
properties:
|
|
||||||
status: { $ref: "#/components/schemas/Status" }
|
|
||||||
address: { type: string }
|
|
||||||
district: { type: string }
|
|
||||||
outages:
|
|
||||||
type: array
|
|
||||||
items: { $ref: "#/components/schemas/Outage" }
|
|
||||||
as_of: { type: string }
|
|
||||||
data_version: { type: string }
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user