Compare commits

..
5 Commits
Author SHA1 Message Date
addrkit 66f4cef90d docs: position addrkit as a universal address service
- core = address normalization/resolution (universal, no EDDS coupling)
- "events by house" (outages) framed as an optional plugin module
- README: positioning, API table marks /v1/outages optional, selfcheck section,
  updated typo numbers and LLM variant-guard note
2026-10-09 20:57:20 +07:00
addrkit 6a3e1cc872 addrkit: 'selfcheck' command — resolve the whole registry against itself
Exhaustive check of all 72,825 indexed addresses: 72,811 exact (99.98%),
14 partial (malformed house units, e.g. '27с1', '32 к 1/4-1/6'), 0 not_found,
0 ambiguous.
2026-10-09 20:41:35 +07:00
addrkit 78691f3a22 addrkit: strict ordinal matching, LLM-variant guard, externalized prompt
- match: require candidate ordinal to equal the query ordinal in fuzzy search
  (deterministic typo recall 73% -> 80%, no wrong houses)
- resolver: compatibleVariant guard — LLM may only fix street spelling, keeping
  the same words, ordinal and house unit; damerau distance for transpositions
- llm: prompt moved out of code (go:embed default_prompt.txt) with
  llm.system_prompt / llm.system_prompt_file overrides; guided JSON parsing
  tolerant of thinking blocks
- address: recognize ordinals appearing after the street name
- tests: first coverage for the LLM adapter and config prompt loading; README

Measured (sample 200): deterministic 94.7-95.8% exact / 0 wrong across 4 seeds;
LLM+guard typo 90.7-89.8% / 0 wrong. LLM fallback latency ~1.2-1.9s.
2026-10-09 20:40:20 +07:00
addrkit 5cc3556ac5 addrkit: quality evaluation harness + parser hardening
- add internal/eval: deterministic citizen-style perturbation harness
  (typos, case, word order, abbreviations, filler words, dropped units)
- parser: strip all street-type tokens, filler words (город/дом/д/область),
  trailing apartment (кв N); handle letter-before-fraction and corpus letters
- matching: token-aware typo-tolerant street similarity; wider ambiguity
  margin to surface near-collisions instead of guessing
- seed: correct parse-rate accounting, structure-aware de-duplication
- CLI: 'addrkit eval' command; README with measured results

Measured on the real 73,936-row registry: 99.98% parsed, 0 wrong-house
safety violations across 34k+ perturbed cases.
2026-10-09 20:13:50 +07:00
addrkit 3be331bb65 addrkit: deterministic address resolution service (Go)
- domain: address normalization/parsing, matching with explicit statuses
- adapters: SQLite registry store, ArcGIS outages client, optional LLM rewriter, HTTP API
- seed pipeline with duplicate handling and unparsed reporting
- CLI (seed/verify/resolve), systemd unit, OpenAPI contract, tests
2026-10-09 20:05:32 +07:00
42 changed files with 1013 additions and 4258 deletions
+3 -24
View File
@@ -1,7 +1,7 @@
GO ?= go GO ?= go
BIN_DIR ?= bin BIN_DIR ?= bin
.PHONY: all build test vet fmt tidy run clean seed check e2e .PHONY: all build test vet fmt tidy run clean seed
all: build all: build
@@ -18,24 +18,6 @@ test:
vet: vet:
$(GO) vet ./... $(GO) vet ./...
## check: vet + race tests + architecture guard (the address core must not
## depend on any events/outages adapter)
check: vet test
@if $(GO) list -deps ./cmd/addrkitd | grep -Eq 'domain/outages|adapters/arcgis'; then \
echo "addrkitd must not depend on events/outages packages" >&2; \
exit 1; \
fi
## e2e: process-level tests against a real registry snapshot.
## Set ADDRKIT_E2E_REGISTRY to a Bulk NDJSON dump (and optionally
## ADDRKIT_E2E_LOG to an EДДС address log). Data is never copied into the repo.
e2e:
@test -n "$(ADDRKIT_E2E_REGISTRY)" || { \
echo "set ADDRKIT_E2E_REGISTRY=/path/to/address_registry.ndjson" >&2; \
exit 1; \
}
$(GO) test -tags e2e -count=1 -v ./test/e2e
## fmt: format all Go sources ## fmt: format all Go sources
fmt: fmt:
gofmt -l -w . gofmt -l -w .
@@ -48,14 +30,11 @@ tidy:
run: build run: build
$(BIN_DIR)/addrkitd -config config.json $(BIN_DIR)/addrkitd -config config.json
## seed: build registry.db from a registry dump (override IN=/OUT=/CITY=/CITIES=) ## seed: build registry.db from a registry dump (override IN=/OUT=)
IN ?= registry.ndjson IN ?= registry.ndjson
OUT ?= registry.db OUT ?= registry.db
CITY ?=
CITIES ?=
seed: seed:
$(BIN_DIR)/addrkit seed -in $(IN) -out $(OUT) \ $(BIN_DIR)/addrkit seed -in $(IN) -out $(OUT)
$(if $(CITY),-city "$(CITY)") $(if $(CITIES),-cities "$(CITIES)")
## clean: remove build artifacts ## clean: remove build artifacts
clean: clean:
+104 -212
View File
@@ -1,32 +1,36 @@
# addrkit # addrkit
**Детерминированный многогородской сервис нормализации и поиска адресов.** **Универсальный детерминированный сервис нормализации и поиска адресов.**
Разбирает вольные реплики (дроби, литеры, корпуса, строения, опечатки), Разбирает вольные реплики (дроби, литеры, корпуса, строения, опечатки),
сопоставляет их с каноническим реестром и отдаёт ответ **только из данных** — сопоставляет их с каноническим реестром и отдаёт ответ **только из данных** —
LLM не участвует в пути корректности. LLM не участвует в пути корректности.
Сервис делает одну вещь: превращает свободный адрес в канонический. Он не Ядро продукта — **только адрес**: оно не привязано ни к ЕДДС, ни к отключениям.
знает ни про отключения, ни про аварии, ни про другие события. Если такие Опциональный модуль «**события по дому**» (отключения, аварии, плановые работы —
данные нужны, их источник — отдельный сервис, который использует addrkit как данные, привязанные к конкретному дому) подключается отдельно; ЕДДС — лишь один
нормализатор. из его потребителей.
## Что умеет Проект на Go, живёт офлайн (один статический бинарь + SQLite), без MCP в
рантайме.
- **Города.** Город — часть модели и ключа улицы. «Ленина 5» в Новосибирске и ## Позиционирование
в Бердске — разные адреса; без указания города сервис честно вернёт
`ambiguous`, а не выберет наугад. - **Ядро (универсальное):** нормализация, структурирование и поиск адреса по
- **Алиасы городов.** `нск`, `новосиб`, `новосибирская область`, `нсо` — всё реестру, статусы `found/partial/ambiguous/not_found`, версия данных.
это настраивается, а не зашито в код. Годится для колл-центра, доставки, CRM, аналитики, любого робота.
- **Строгий дом.** Дробь, литера, корпус и строение должны совпасть. Запрос - **Опциональный модуль «события по дому»:** получает канонический адрес из
«1 стр. 2» не может вернуть «1». ядра и отдаёт события, **привязанные к этому дому** (совпадение дома, дроби,
- **Безопасные статусы.** `found`, `partial`, `ambiguous`, `not_found`. литеры и корпуса; строение игнорируется). Источник — plug-in
Сервис лучше скажет «уточните», чем подставит чужой дом. (`outages.Source`): сегодня ArcGIS-портал ЕДДС, завтра любой фид (ES, SQL,
- **Опечатки.** Нечёткое сопоставление названия улицы с порогом и явной API). Ядро про события ничего не знает.
зоной неоднозначности.
- **Опциональный LLM-rewriter.** Может предложить варианты написания, но ## Зачем
ответ всегда берётся из реестра, а вариант принимается только если меняет
*написание* улицы, не трогая город, порядковый номер и дом. Существующая логика поиска адреса в диалоговом сценарии страдала от
- **Офлайн.** Один статический бинарь + SQLite. В пути резолва нет сети. нечёткого поиска по всей строке (`fuzziness: AUTO`, `size:1`), из-за чего
дробь/корпус/литера могли дать неверный дом. `addrkit` разделяет задачу на
компоненты, сопоставляет дом строго и вместо угадывания возвращает явный
статус (`partial`/`ambiguous`).
## Архитектура ## Архитектура
@@ -36,44 +40,24 @@ LLM не участвует в пути корректности.
cmd/addrkitd сервис (HTTP) cmd/addrkitd сервис (HTTP)
cmd/addrkit CLI: seed | verify | resolve | eval | selfcheck cmd/addrkit CLI: seed | verify | resolve | eval | selfcheck
internal/ internal/
domain/address нормализация, словарь городов, разбор адреса domain/address нормализация и разбор адреса (чистые функции)
domain/match сопоставление с индексом, статусы ← ядро domain/match сопоставление с индексом, статусы ← ядро
domain/outages «события по дому»: фильтр по дому ← опциональный модуль
adapters/registrystore SQLite (modernc, без CGO) adapters/registrystore SQLite (modernc, без CGO)
adapters/arcgis клиент источника событий (timeout/retry) ← пример plug-in
adapters/llm опциональный rewriter (OpenAI-совместимый) adapters/llm опциональный rewriter (OpenAI-совместимый)
adapters/httpapi версионированный JSON API adapters/httpapi версионированный JSON API
config конфиг + ${VAR} + словарь городов config конфиг + ${VAR}
seed сборка индекса из дампа реестра seed сборка индекса из дампа реестра
eval offline-оценка качества
pkg/addrkit публичный фасад для использования как библиотеки
``` ```
Поток запроса: Поток запроса:
``` ```
"нск, 2-я Станционная улица 1/1 стр." "2-я Станционная улица 1/1 стр."
-> address.Parser -> {city: Новосибирск, ordinal: 2, name: станционная, house: 1, frac: 1} -> address.Parse -> {ordinal:2, name:станционная, house:1, frac:1}
-> match.Resolve -> exact city -> exact street -> exact house -> FOUND -> match.Resolve -> exact street -> exact house -> FOUND
``` -> (опц. модуль «события по дому») -> фильтр по дому -> ответ
`match.Index` неизменяемый, резолвер не хранит состояния. Индекс собирается
один раз при старте и безопасен для конкурентных чтений.
Если нормализатор нужен другому проекту как библиотека, используйте
`pkg/addrkit`: он не делает I/O и не требует HTTP или SQLite.
```go
parser := addrkit.NewParser(
addrkit.NewDictionary([]addrkit.City{{Name: "Новосибирск", Aliases: []string{"нск"}}}),
"",
)
parsed := parser.Parse("нск, Ленина 5")
resolver := addrkit.NewResolverWithOptions(
addrkit.NewIndex(version, entries, addrkit.DefaultOptions()),
nil,
addrkit.ResolverOptions{Parser: parser},
)
result := resolver.Resolve(ctx, "нск, Ленина 5")
``` ```
## Быстрый старт ## Быстрый старт
@@ -81,127 +65,72 @@ result := resolver.Resolve(ctx, "нск, Ленина 5")
```bash ```bash
make build make build
# 1. Собрать индекс из дампа реестра. # 1. Собрать индекс из дампа реестра (Bulk NDJSON или JSONL).
# -city задаёт город для строк без поля city; ./bin/addrkit seed -in registry.ndjson -out registry.db -version 2026-10-01
# -cities — необязательный JSON-справочник алиасов.
./bin/addrkit seed -in registry.ndjson -out registry.db \
-version 2026-10-01 -city "Новосибирск"
# 2. Запустить сервис. # 2. Запустить сервис.
cp config.example.json config.json cp config.example.json config.json
./bin/addrkitd -config config.json ADDRKIT_PORTAL_URL='https://map.example/.../101/query' ./bin/addrkitd -config config.json
# 3. Проверить. # 3. Проверить.
curl -s localhost:6171/v1/resolve -d '{"address":"нск, 2-я Станционная 1/1 стр."}' curl -s localhost:6171/v1/resolve -d '{"address":"2-я Станционная улица 1/1 стр."}'
curl -s localhost:6171/v1/cities curl -s localhost:6171/v1/outages -d '{"address":"станционная 30а к9/1"}'
``` ```
CLI без сервиса: CLI без сервиса:
```bash ```bash
./bin/addrkit resolve -db registry.db -city "Новосибирск" "кошурникова 11/1" ./bin/addrkit resolve -db registry.db "кошурникова 11/1"
./bin/addrkit verify -db registry.db ./bin/addrkit verify -db registry.db
./bin/addrkit selfcheck -db registry.db ./bin/addrkit selfcheck -db registry.db
./bin/addrkit shadow -db registry.db -es https://es:9200 -sample 300
``` ```
## API v1 ## API v1
| Метод | Путь | Назначение | | Метод | Путь | Назначение |
|---|---|---| |---|---|---|
| POST | `/v1/resolve` | адрес → `{status, address, city, district, confidence, candidates[], data_version}` | | POST | `/v1/resolve` | **[ядро]** адрес → `{status, address, district, confidence, candidates[]}` |
| POST | `/v1/city` | только город: `нск` → `Новосибирск` | | POST | `/v1/outages` | **[опц. модуль]** адрес → канонический адрес + события/отключения по дому |
| GET | `/v1/cities` | канонические города, присутствующие в индексе |
| GET | `/v1/healthz` | живость | | GET | `/v1/healthz` | живость |
| GET | `/v1/readyz` | готовность (индекс загружен) | | GET | `/v1/readyz` | готовность (индекс загружен) |
| GET | `/v1/version` | версия данных | | GET | `/v1/version` | версия данных |
Запрос: Статусы: `found`, `partial`, `ambiguous`, `not_found`, `portal_timeout`,
`infra_error`.
```json ## Конфигурация
{"address": "Ленина 5", "city": "Бердск"}
```
`city` необязателен и переопределяет город, найденный в тексте. Лимит адреса —
512 символов, ответ на слишком длинный ввод — `400`.
Только город, без улицы:
```bash
curl -s localhost:6171/v1/city -d '{"query":"нск"}'
# {"status":"found","city":"Новосибирск"}
```
Это же доступно как библиотека: `addrkit.NewDictionary(...).MatchCity("нск")`.
Статусы: `found`, `partial`, `ambiguous`, `not_found`.
## Города и конфигурация
`config.json` (см. `config.example.json`). Значения поддерживают `${VAR}`; `config.json` (см. `config.example.json`). Значения поддерживают `${VAR}`;
незаданная переменная = ошибка запуска (fail closed). незаданная переменная = ошибка запуска (fail closed). Секреты — только из
окружения.
```json Промпт LLM-rewriter **не зашит в код**: встроенный дефолт лежит в
{ `internal/adapters/llm/default_prompt.txt` и компилируется в бинарь (`go:embed`).
"addr": ":6171", Переопределить можно в конфиге: `llm.system_prompt` (инлайн) или
"data_path": "registry.db", `llm.system_prompt_file` (путь к файлу; приоритетнее инлайна).
"cities": [
{"name": "Новосибирск", "aliases": ["нск", "новосиб", "новосибирская область", "нсо"]},
{"name": "Бердск"}
],
"default_city": "Новосибирск",
"resolve": {"fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5}
}
```
- `cities` — словарь канонических названий и алиасов. Один и тот же алиас не
может принадлежать двум городам: конфиг не загрузится.
- `default_city` — удобство для одногородского деплоя: запрос «Ленина 5» без
города будет отнесён к нему. В многогородском деплое оставьте поле пустым:
угадывание города — это ровно та ошибка, которую сервис не должен делать.
- `resolve.ambiguity_margin` — насколько близкие улицы считаются
неоднозначными. Значение по умолчанию одно для сервиса, CLI и eval.
Справочник городов можно держать отдельным файлом (`addrkit seed -cities
cities.json`); формат — JSON-массив тех же объектов.
## Обновление данных и совместимость
`registry.db` содержит `schema_version`. Формат ключа улицы включает город,
поэтому база, собранная старой версией, **не откроется** новым бинарём: в
ошибке будет сказано «re-run addrkit seed». Это сделано намеренно — молча
резолвить против полупустых полей хуже, чем не стартовать.
Порядок обновления:
```bash
make build
./bin/addrkit seed -in registry.ndjson -out registry.db.new -city "Новосибирск"
mv registry.db.new registry.db
systemctl restart addrkitd
```
## Принятые решения ## Принятые решения
- **Детерминизм.** Один вход — один выход, порядок кандидатов стабилен: - **Детерминизм.** Один вход — один выход, порядок кандидатов стабилен.
индекс сортирует записи при сборке, SQL-запрос тоже имеет `ORDER BY`. `resolve` не зависит от сети.
- **Дом и строение сравниваются строго.** Литера/дробь/корпус/строение должны - **Дом сравнивается строго.** Литера/дробь/корпус должны совпасть; иначе
совпасть; иначе `partial` или `ambiguous`, но не «похожий» дом. `partial`, а не «похожий» дом.
- **Город — часть ключа.** Одинаковые улицы разных городов не смешиваются. - **LLM — опция, выключена по умолчанию.** `adapters/llm` лишь предлагает
- **LLM — опция, выключена по умолчанию.** Модель не может «выдумать» адрес; варианты записи адреса; результат всегда берётся из реестра, поэтому модель
ошибки rewriter'а логируются, но не меняют детерминированный ответ. не может «выдумать» адрес. Вариант принимается только если он исправляет
*написание* улицы, не меняя слова, порядковый и дом (`compatibleVariant`).
- **Ядро универсально, события по дому — опциональный модуль.** Ядро
(`domain/address`, `domain/match`) не знает ни про ЕДДС, ни про отключения;
источник событий — plug-in (`outages.Source`), поэтому модуль переиспользуем
для любых данных, привязанных к дому.
- **Данные.** Сервис владеет своим индексом, построенным из дампа реестра. - **Данные.** Сервис владеет своим индексом, построенным из дампа реестра.
Источник истины и регламент обновления — предмет отдельного согласования. Источник истины и регламент обновления — предмет отдельного согласования.
- **Границы ввода.** Адрес ограничен 512 символами, токен нечёткого - **Часовой пояс.** Даты отключений трактуются в `Asia/Novosibirsk` (UTC+7).
сопоставления — 64; это защищает CPU от патологического ввода.
- **Сервис не делает событий.** Ни `/v1/outages`, ни portal-конфига, ни
часового пояса в ядре.
## Тесты ## Тесты
```bash ```bash
make test # go test -race ./... make test # go test -race ./...
make check # vet + race tests + проверка, что ядро не тянет events-адаптеры
``` ```
Проверка парсера на реальном реестре (вне репозитория): Проверка парсера на реальном реестре (вне репозитория):
@@ -211,72 +140,8 @@ ADDRKIT_REGISTRY=/path/to/address_registry.ndjson \
go test ./internal/seed -run TestRealRegistryParseRate -v go test ./internal/seed -run TestRealRegistryParseRate -v
``` ```
### E2E на реальном снимке ЕДДС На дампе из 73 936 строк парсер распознаёт **99.98 %** (14 строк — заведомо
битые форматы; они не угадываются, а попадают в отчёт `stats.unparsed_sample`).
Процессные тесты собирают `addrkit` и `addrkitd`, сеют временную БД из
настоящего Bulk NDJSON-дампа, поднимают HTTP-сервис и проверяют весь реестр
(`selfcheck`), выборку канонических адресов по HTTP, реальные запросы из
`edds_address_log`, алиасы города, строгость дома/строения, лимиты и нагрузку
(10 000 запросов, p99, RSS, 0 неверных адресов).
Данные содержат PII и **не копируются в репозиторий**: путь передаётся
переменной окружения, временные файлы живут в `t.TempDir()`.
```bash
ADDRKIT_E2E_REGISTRY=/path/to/address_registry.ndjson \
ADDRKIT_E2E_LOG=/path/to/edds_address_log.ndjson \
make e2e
```
Опционально: `ADDRKIT_E2E_CITY` (по умолчанию `Новосибирск`),
`ADDRKIT_E2E_SAMPLE` (каноническая HTTP-выборка, по умолчанию 2000),
`ADDRKIT_E2E_LOAD` (нагрузочные запросы, по умолчанию 10 000).
Тест проверяет **канонический ввод** — то, что старый конвейер ЕДДС передавал в
Elasticsearch после SEE. Сырые реплики с числительными словами
(`выставочная шестнадцать`, `сто дробь один`) остаются зоной SEE/LLM: для них
тест измеряет долю `found` и не требует 100 %.
### Shadow: старый ES-путь против нового addrkit
`addrkit shadow` прогоняет один и тот же запрос через **старый ES-поиск**
(ровно тот запрос, который делает робот ЕДДС: `multi_match`, `fuzziness: AUTO`,
`size: 1`) и через новый локальный резолвер. В ES уходят **только `_search`**:
ничего не индексируется, не обновляется и не удаляется.
```bash
export ADDRKIT_ES_URL=https://es:9200
export ADDRKIT_ES_USER=elastic
export ADDRKIT_ES_PASSWORD=...
# Сгенерированные реалистичные искажения реестра
./bin/addrkit shadow -db registry.db -city "Новосибирск" \
-es "$ADDRKIT_ES_URL" -sample 1000 -workers 8 -out shadow-generated.json
# Реальные реплики из прод-лога (read-only, последние N документов)
./bin/addrkit shadow -db registry.db -city "Новосибирск" \
-es "$ADDRKIT_ES_URL" -log-es-index edds_address_log -log-limit 3000 \
-workers 8 -out shadow-log.json
```
Старый путь получает ту же предобработку, что и в роботе (`-old-preprocess`,
по умолчанию включено): нижний регистр и удаление административных слов
(`город`, `новосибирск`, районы, `дом`, `номер`). Это делает сравнение честным.
Ключевые поля отчёта:
- `new.wrong` — новый путь вернул другой дом; **должно быть 0**, команда
завершается с кодом 1;
- `compare.old_wrong_new_safe` — старый путь угадал дом, новый отказался
(`partial`/`ambiguous`/`not_found`);
- `compare.old_wrong_new_correct` — старый ошибся, новый ответил верно;
- `compare.old_correct_new_not_exact` — цена строгости: старый угадал верно,
новый отказался или потерял информацию; эти случаи уходят в fallback
(оператор/LLM), их нужно смотреть глазами.
`-log` принимает локальный bulk NDJSON, `-log-es-index` — читает лог прямо из
ES. Для сырых реплик SEE-преобразование числительных не воспроизводится,
поэтому лог-режим показывает нижнюю границу, а не боевую точность пайплайна.
## Оценка качества (`addrkit eval`) ## Оценка качества (`addrkit eval`)
@@ -287,26 +152,53 @@ ES. Для сырых реплик SEE-преобразование числит
./bin/addrkit eval -db registry.db -sample 800 -seed 1 ./bin/addrkit eval -db registry.db -sample 800 -seed 1
``` ```
Категории: регистр, `г. Город`, лишние `дом`/пунктуация, `ё`, сокращения и Категории и результат на реальном реестре (по 800 адресам на seed):
перестановка типа улицы, синоним «корпус», опечатка в названии, потеря
порядкового/литеры/дроби/строения.
Ключевая инварианта: **`found` всегда указывает на запись из индекса**. Если | Категория | Пример искажения | Точность |
информации не хватает (потерян номер дома, литера или строение), результат |---|---|---|
уходит в `partial`/`ambiguous`/`not_found`, но не превращается в чужой адрес. | baseline | канонический адрес как есть | 100 % |
| case_lower / case_upper | регистр | 100 % |
| city_prefix | `г. Новосибирск, …` | 100 % |
| filler_dom | `… дом 5` | 100 % |
| punctuation | лишние `,`/пробелы | 100 % |
| yo | `ё` вместо `е` | 100 % |
| type_abbrev | `ул.`/`пер.`/`пр-т` | 100 % |
| type_removed | улица без слова «улица» | 100 % |
| type_front | `улица Станционная 1` | 100 % |
| corpus_synonym | `корпус` вместо `к` | 100 % |
| structure_drop | без `стр.` | 100 % |
| **typo** | опечатка в названии улицы | ~78–82 % (остальное — `ambiguous`) |
| drop_ordinal / drop_letter / drop_fraction | потеря `2-я` / литеры / дроби | потери (lossy) |
Ключевой результат: **0 случаев, когда сервис подставил неверный дом**
(«safety violations») на десятках тысяч сгенерированных кейсов. Опечатки либо
распознаются, либо честно уходят в `ambiguous`/`not_found`, но не превращаются
в чужой адрес.
С подключённым LLM-rewriter'ом (опция, `llm.enabled`) точность на опечатках
поднимается до **~90 %** при том же нуле неверных адресов (вариант модели
принимается только если исправляет лишь *написание* улицы — см. «Принятые
решения»).
## Самопроверка (`addrkit selfcheck`) ## Самопроверка (`addrkit selfcheck`)
Прогоняет весь реестр на самого себя и показывает долю точных совпадений и Прогоняет **весь** реестр на самого себя (не выборку):
примеры не-`found`. Это самый быстрый способ заметить деградацию парсера
после изменения данных. ```bash
./bin/addrkit selfcheck -db registry.db
```
На реестре 72 825 записей: **72 811 (99.98 %) резолвятся точно**, 14 — `partial`
(битые форматы дома: `27с1`, `32 к 1/4-1/6`, `112с729`), 0 `not_found`,
0 неверных.
## Локальный dev ## Локальный dev
```bash ```bash
make build make build
./bin/addrkit seed -in registry.ndjson -out dev/registry.db -city "Новосибирск" ./bin/addrkit seed -in registry.ndjson -out dev/registry.db -version 2026-10-01
cp config.example.json dev/config.json # укажите data_path cp config.example.json dev/config.json # укажите data_path и portal.url
./bin/addrkitd -config dev/config.json ./bin/addrkitd -config dev/config.json
curl -s localhost:6171/v1/resolve -d '{"address":"нск, 2-я Станционная 1/1 стр."}' curl -s localhost:6171/v1/resolve -d '{"address":"2-я Станционная улица 1/1 стр."}'
``` ```
+15 -114
View File
@@ -38,8 +38,6 @@ func main() {
os.Exit(runEval(os.Args[2:])) os.Exit(runEval(os.Args[2:]))
case "selfcheck": case "selfcheck":
os.Exit(runSelfcheck(os.Args[2:])) os.Exit(runSelfcheck(os.Args[2:]))
case "shadow":
os.Exit(runShadow(os.Args[2:]))
case "help", "-h", "--help": case "help", "-h", "--help":
usage() usage()
os.Exit(0) os.Exit(0)
@@ -51,15 +49,14 @@ func main() {
} }
func usage() { func usage() {
fmt.Fprint(os.Stderr, `addrkit — multi-city address registry toolkit fmt.Fprint(os.Stderr, `addrkit — address registry toolkit
Usage: Usage:
addrkit seed -in registry.ndjson -out registry.db [-version 2026-10-09] [-city "Новосибирск"] [-cities cities.json] addrkit seed -in registry.ndjson -out registry.db [-version 2026-10-09]
addrkit verify -db registry.db addrkit verify -db registry.db
addrkit resolve -db registry.db [-city "Новосибирск"] [-cities cities.json] "2-я Станционная улица 1/1 стр." addrkit resolve -db registry.db "2-я Станционная улица 1/1 стр."
addrkit eval -db registry.db [-sample 500] [-seed 1] [-cities cities.json] addrkit eval -db registry.db [-sample 500] [-seed 1]
addrkit selfcheck -db registry.db addrkit selfcheck -db registry.db
addrkit shadow -db registry.db -es https://es:9200 [-sample 300] [-log file.ndjson]
`) `)
} }
@@ -68,8 +65,6 @@ func runSeed(args []string) int {
in := fs.String("in", "", "input registry dump (Bulk NDJSON or JSONL)") in := fs.String("in", "", "input registry dump (Bulk NDJSON or JSONL)")
out := fs.String("out", "", "output SQLite database") out := fs.String("out", "", "output SQLite database")
version := fs.String("version", "", "data version (defaults to input file mtime)") version := fs.String("version", "", "data version (defaults to input file mtime)")
city := fs.String("city", "", "default city for rows without a city field")
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
_ = fs.Parse(args) _ = fs.Parse(args)
if *in == "" || *out == "" { if *in == "" || *out == "" {
@@ -82,12 +77,6 @@ func runSeed(args []string) int {
} }
} }
cities, err := loadCities(*citiesPath)
if err != nil {
fmt.Fprintf(os.Stderr, "seed: %v\n", err)
return 1
}
f, err := os.Open(*in) f, err := os.Open(*in)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "seed: open input: %v\n", err) fmt.Fprintf(os.Stderr, "seed: open input: %v\n", err)
@@ -100,11 +89,7 @@ func runSeed(args []string) int {
fmt.Fprintf(os.Stderr, "seed: read: %v\n", err) fmt.Fprintf(os.Stderr, "seed: read: %v\n", err)
return 1 return 1
} }
parser := address.NewParser(address.NewDictionary(cities), *city) entries, stats := seed.Build(sources)
entries, stats := seed.Build(sources, seed.BuildOptions{
Parser: parser,
DefaultCity: *city,
})
if stats.Parsed == 0 { if stats.Parsed == 0 {
fmt.Fprintln(os.Stderr, "seed: no parseable entries; refusing to write empty index") fmt.Fprintln(os.Stderr, "seed: no parseable entries; refusing to write empty index")
return 1 return 1
@@ -124,15 +109,6 @@ func runSeed(args []string) int {
return 0 return 0
} }
// loadCities reads an optional city dictionary file. An empty path is valid and
// means "no city names beyond those in the address".
func loadCities(path string) ([]address.City, error) {
if path == "" {
return nil, nil
}
return config.LoadCitiesFile(path)
}
func runVerify(args []string) int { func runVerify(args []string) int {
fs := flag.NewFlagSet("verify", flag.ExitOnError) fs := flag.NewFlagSet("verify", flag.ExitOnError)
db := fs.String("db", "", "SQLite database") db := fs.String("db", "", "SQLite database")
@@ -147,10 +123,8 @@ func runVerify(args []string) int {
return 1 return 1
} }
streets := make(map[string]struct{}) streets := make(map[string]struct{})
cities := make(map[string]int)
for _, e := range entries { for _, e := range entries {
streets[e.StreetKey] = struct{}{} streets[e.StreetKey] = struct{}{}
cities[e.City]++
} }
enc := json.NewEncoder(os.Stdout) enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ") enc.SetIndent("", " ")
@@ -158,7 +132,6 @@ func runVerify(args []string) int {
"version": version, "version": version,
"entries": len(entries), "entries": len(entries),
"distinct_streets": len(streets), "distinct_streets": len(streets),
"cities": cities,
}) })
return 0 return 0
} }
@@ -166,8 +139,6 @@ func runVerify(args []string) int {
func runResolve(args []string) int { func runResolve(args []string) int {
fs := flag.NewFlagSet("resolve", flag.ExitOnError) fs := flag.NewFlagSet("resolve", flag.ExitOnError)
db := fs.String("db", "", "SQLite database") db := fs.String("db", "", "SQLite database")
city := fs.String("city", "", "optional city hint")
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL") llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name") llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
llmToken := fs.String("llm-token", "", "LLM auth token") llmToken := fs.String("llm-token", "", "LLM auth token")
@@ -177,11 +148,6 @@ func runResolve(args []string) int {
fmt.Fprintln(os.Stderr, `resolve: -db and an address are required`) fmt.Fprintln(os.Stderr, `resolve: -db and an address are required`)
return 2 return 2
} }
cities, err := loadCities(*citiesPath)
if err != nil {
fmt.Fprintf(os.Stderr, "resolve: %v\n", err)
return 1
}
version, entries, err := registrystore.Load(*db) version, entries, err := registrystore.Load(*db)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "resolve: %v\n", err) fmt.Fprintf(os.Stderr, "resolve: %v\n", err)
@@ -194,13 +160,8 @@ func runResolve(args []string) int {
TimeoutSeconds: 5, MaxTokens: 512, DisableThinking: true, TimeoutSeconds: 5, MaxTokens: 512, DisableThinking: true,
}) })
} }
parser := address.NewParser(address.NewDictionary(cities), "") resolver := match.NewResolver(match.NewIndex(version, entries, match.DefaultOptions()), rewriter)
resolver := match.NewResolverWithOptions( res := resolver.Resolve(context.Background(), rest[0])
match.NewIndex(version, entries, match.DefaultOptions()),
rewriter,
match.ResolverOptions{Parser: parser},
)
res := resolver.ResolveInCity(context.Background(), rest[0], *city)
enc := json.NewEncoder(os.Stdout) enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ") enc.SetIndent("", " ")
_ = enc.Encode(res) _ = enc.Encode(res)
@@ -216,7 +177,6 @@ func runEval(args []string) int {
db := fs.String("db", "", "SQLite database") db := fs.String("db", "", "SQLite database")
sample := fs.Int("sample", 500, "number of registry entries to sample") sample := fs.Int("sample", 500, "number of registry entries to sample")
seedVal := fs.Int64("seed", 1, "random seed") seedVal := fs.Int64("seed", 1, "random seed")
citiesPath := fs.String("cities", "", "optional JSON file with the city dictionary")
llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL") llmURL := fs.String("llm-url", "", "enable the LLM rewriter with this base URL")
llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name") llmModel := fs.String("llm-model", "qwen3-14b-awq", "LLM model name")
llmToken := fs.String("llm-token", "", "LLM auth token") llmToken := fs.String("llm-token", "", "LLM auth token")
@@ -226,19 +186,11 @@ func runEval(args []string) int {
fmt.Fprintln(os.Stderr, "eval: -db is required") fmt.Fprintln(os.Stderr, "eval: -db is required")
return 2 return 2
} }
cities, err := loadCities(*citiesPath)
if err != nil {
fmt.Fprintf(os.Stderr, "eval: %v\n", err)
return 1
}
_, entries, err := registrystore.Load(*db) _, entries, err := registrystore.Load(*db)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "eval: %v\n", err) fmt.Fprintf(os.Stderr, "eval: %v\n", err)
return 1 return 1
} }
cities = mergeCities(cities, entries)
defaultCity := soleCity(entries)
parser := address.NewParser(address.NewDictionary(cities), defaultCity)
ix := match.NewIndex("eval", entries, match.DefaultOptions()) ix := match.NewIndex("eval", entries, match.DefaultOptions())
var rewriter match.Rewriter var rewriter match.Rewriter
@@ -256,7 +208,7 @@ func runEval(args []string) int {
rewriter = counter rewriter = counter
fmt.Fprintf(os.Stderr, "llm rewriter enabled: %s (%s)\n", *llmURL, *llmModel) fmt.Fprintf(os.Stderr, "llm rewriter enabled: %s (%s)\n", *llmURL, *llmModel)
} }
resolver := match.NewResolverWithOptions(ix, rewriter, match.ResolverOptions{Parser: parser}) resolver := match.NewResolver(ix, rewriter)
rng := rand.New(rand.NewSource(*seedVal)) rng := rand.New(rand.NewSource(*seedVal))
cases := eval.Generate(entries, *sample, rng) cases := eval.Generate(entries, *sample, rng)
@@ -334,22 +286,14 @@ func runSelfcheck(args []string) int {
fmt.Fprintf(os.Stderr, "selfcheck: %v\n", err) fmt.Fprintf(os.Stderr, "selfcheck: %v\n", err)
return 1 return 1
} }
parser := address.NewParser( resolver := match.NewResolver(match.NewIndex("selfcheck", entries, match.DefaultOptions()), nil)
address.NewDictionary(mergeCities(nil, entries)),
soleCity(entries),
)
resolver := match.NewResolverWithOptions(
match.NewIndex("selfcheck", entries, match.DefaultOptions()),
nil,
match.ResolverOptions{Parser: parser},
)
exact := 0 exact := 0
byStatus := map[string]int{} byStatus := map[string]int{}
var failures []string var failures []string
for _, e := range entries { for _, e := range entries {
res := resolver.ResolveInCity(context.Background(), e.Address, e.City) res := resolver.Resolve(context.Background(), e.Address)
if res.Status == address.StatusFound && res.City == e.City && sameEntry(res.Address, e, parser) { if res.Status == address.StatusFound && sameEntry(res.Address, e) {
exact++ exact++
continue continue
} }
@@ -372,53 +316,10 @@ func runSelfcheck(args []string) int {
} }
// sameEntry reports whether a resolved address is the same street and house // sameEntry reports whether a resolved address is the same street and house
// unit as the registry entry it came from. City is checked separately against // unit as the registry entry it came from.
// Result.City by the caller; forcing it here only keeps the parse city-aware. func sameEntry(resolved string, e address.Entry) bool {
func sameEntry(resolved string, e address.Entry, parser *address.Parser) bool { p := address.Parse(resolved)
p := parser.ParseInCity(resolved, e.City)
return p.StreetKey == e.StreetKey && return p.StreetKey == e.StreetKey &&
p.House == e.House && p.HouseFrac == e.HouseFrac && p.HouseLetter == e.HouseLetter && p.House == e.House && p.HouseFrac == e.HouseFrac && p.HouseLetter == e.HouseLetter &&
p.Building == e.Building && p.BuildingFrac == e.BuildingFrac && p.BuildingLetter == e.BuildingLetter && p.Building == e.Building && p.BuildingFrac == e.BuildingFrac && p.BuildingLetter == e.BuildingLetter
p.Structure == e.Structure
}
// mergeCities augments an explicit city dictionary with the canonical names
// found in the index. The names are enough for parsing addresses that include
// the city; aliases still come from the dictionary file.
func mergeCities(base []address.City, entries []address.Entry) []address.City {
seen := make(map[string]struct{}, len(base))
for _, c := range base {
seen[address.NormalizeText(c.Name)] = struct{}{}
}
out := append([]address.City(nil), base...)
for _, e := range entries {
if e.City == "" {
continue
}
key := address.NormalizeText(e.City)
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
out = append(out, address.City{Name: e.City})
}
return out
}
// soleCity returns the only city in the index, or "" when the index is
// city-less or multi-city.
func soleCity(entries []address.Entry) string {
names := make(map[string]string)
for _, e := range entries {
if e.City != "" {
names[e.CityKey] = e.City
}
}
if len(names) != 1 {
return ""
}
for _, name := range names {
return name
}
return ""
} }
-685
View File
@@ -1,685 +0,0 @@
package main
// Shadow mode: run the old EДДС path (Elasticsearch fuzzy search) and addrkit
// on the same queries, without changing any production answer.
//
// The Elasticsearch side is strictly read-only: this file only issues
// `_search` requests. Nothing is indexed, updated or deleted.
//
// Two datasets are supported:
// - generated: realistic perturbations of real registry rows (eval harness);
// - log: real user utterances from the EДДС address log (file or ES index).
//
// The report compares the old top-1 guess with the new deterministic verdict.
// The key number is "old_wrong_new_safe": cases where the old path returned a
// different house and the new one refused to guess.
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"sort"
"strings"
"sync"
"time"
"git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
"git.totmin.ru/en2zmax/addrkit/internal/eval"
)
type shadowOptions struct {
db string
city string
citiesPath string
esURL string
esUser string
esPass string
esIndex string
insecure bool
timeout time.Duration
workers int
sample int
seed int64
logPath string
logField string
logESIndex string
logLimit int
maxExamples int
oldPreproc bool
out string
}
func runShadow(args []string) int {
fs := flag.NewFlagSet("shadow", flag.ExitOnError)
db := fs.String("db", "", "local registry.db (new path)")
city := fs.String("city", "", "default city for the new path")
citiesPath := fs.String("cities", "", "optional JSON city dictionary")
esURL := fs.String("es", envOr("ADDRKIT_ES_URL", ""), "Elasticsearch base URL (old path, read-only)")
esUser := fs.String("es-user", envOr("ADDRKIT_ES_USER", ""), "Elasticsearch user")
esPass := fs.String("es-pass", envOr("ADDRKIT_ES_PASSWORD", ""), "Elasticsearch password")
esIndex := fs.String("es-index", "address_registry", "Elasticsearch address index")
insecure := fs.Bool("insecure", false, "skip TLS verification for Elasticsearch")
timeout := fs.Duration("timeout", 10*time.Second, "per-request timeout")
workers := fs.Int("workers", 8, "parallel queries")
sample := fs.Int("sample", 300, "registry entries to perturb (each yields several queries)")
seed := fs.Int64("seed", 1, "random seed for generated queries")
logPath := fs.String("log", "", "address log bulk NDJSON (real utterances)")
logField := fs.String("log-field", "address", "log field to replay: address|request")
logESIndex := fs.String("log-es-index", "", "fetch real utterances from this ES index instead of a file")
logLimit := fs.Int("log-limit", 500, "documents to fetch from -log-es-index")
maxExamples := fs.Int("max-examples", 20, "examples per category in the report")
oldPreproc := fs.Bool("old-preprocess", true, "apply the robot's administrative-word stripping to the old ES query")
out := fs.String("out", "", "write the JSON report to this file (default stdout)")
_ = fs.Parse(args)
if *db == "" || *esURL == "" {
fmt.Fprintln(os.Stderr, "shadow: -db and -es (or ADDRKIT_ES_URL) are required")
return 2
}
if *workers < 1 {
*workers = 1
}
cities, err := loadCities(*citiesPath)
if err != nil {
fmt.Fprintf(os.Stderr, "shadow: %v\n", err)
return 1
}
_, entries, err := registrystore.Load(*db)
if err != nil {
fmt.Fprintf(os.Stderr, "shadow: load registry: %v\n", err)
return 1
}
parser := address.NewParser(address.NewDictionary(cities), *city)
resolver := match.NewResolverWithOptions(
match.NewIndex("shadow", entries, match.DefaultOptions()),
nil,
match.ResolverOptions{Parser: parser},
)
opts := shadowOptions{
db: *db, city: *city, citiesPath: *citiesPath,
esURL: *esURL, esUser: *esUser, esPass: *esPass, esIndex: *esIndex,
insecure: *insecure, timeout: *timeout, workers: *workers,
sample: *sample, seed: *seed, logPath: *logPath, logField: *logField,
logESIndex: *logESIndex, logLimit: *logLimit, maxExamples: *maxExamples,
oldPreproc: *oldPreproc, out: *out,
}
cases, mode, note, err := buildShadowCases(opts, entries, parser)
if err != nil {
fmt.Fprintf(os.Stderr, "shadow: %v\n", err)
return 1
}
if len(cases) == 0 {
fmt.Fprintln(os.Stderr, "shadow: no queries to replay")
return 1
}
client := newESClient(opts)
report := runShadowQueries(opts, cases, mode, note, parser, resolver, client)
data, err := json.MarshalIndent(report, "", " ")
if err != nil {
fmt.Fprintf(os.Stderr, "shadow: encode report: %v\n", err)
return 1
}
if opts.out != "" {
if err := os.WriteFile(opts.out, data, 0o600); err != nil {
fmt.Fprintf(os.Stderr, "shadow: write report: %v\n", err)
return 1
}
}
fmt.Println(string(data))
if report.New.Wrong > 0 {
return 1 // a wrong new answer is a blocker, make it visible to CI
}
return 0
}
func envOr(name, fallback string) string {
if v := strings.TrimSpace(os.Getenv(name)); v != "" {
return v
}
return fallback
}
type shadowCase struct {
Query string
Expect string
ExpectCity string
RequireExact bool
Category string
}
func buildShadowCases(opts shadowOptions, entries []address.Entry, parser *address.Parser) ([]shadowCase, string, string, error) {
switch {
case opts.logESIndex != "":
rows, err := fetchLogCases(opts)
if err != nil {
return nil, "", "", fmt.Errorf("fetch address log from ES: %w", err)
}
return logCases(rows, opts.logField), "log", "raw utterances are compared without the SEE conversion step", nil
case opts.logPath != "":
rows, err := readLogCases(opts.logPath)
if err != nil {
return nil, "", "", fmt.Errorf("read address log: %w", err)
}
return logCases(rows, opts.logField), "log", "raw utterances are compared without the SEE conversion step", nil
default:
rng := rand.New(rand.NewSource(opts.seed))
src := eval.Generate(entries, opts.sample, rng)
out := make([]shadowCase, 0, len(src))
for _, c := range src {
out = append(out, shadowCase{
Query: c.Query, Expect: c.Expect, ExpectCity: c.ExpectCity,
RequireExact: c.RequireExact, Category: c.Category,
})
}
return out, "generated", "real registry rows perturbed by the eval harness", nil
}
}
type logRow struct {
Utterance string
Request string
}
func logCases(rows []logRow, field string) []shadowCase {
out := make([]shadowCase, 0, len(rows))
seen := map[string]struct{}{}
for _, r := range rows {
q := r.Utterance
if field == "request" {
q = r.Request
}
q = strings.TrimSpace(q)
if q == "" {
continue
}
if _, ok := seen[q]; ok {
continue
}
seen[q] = struct{}{}
out = append(out, shadowCase{Query: q, Category: "log"})
}
return out
}
func readLogCases(path string) ([]logRow, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
dec := json.NewDecoder(f)
var rows []logRow
for {
var raw json.RawMessage
if err := dec.Decode(&raw); err != nil {
if err == io.EOF {
break
}
return nil, err
}
var probe map[string]json.RawMessage
if json.Unmarshal(raw, &probe) != nil {
continue
}
if _, ok := probe["index"]; ok {
continue
}
var rec struct {
Address string `json:"address"`
Request string `json:"request"`
}
_ = json.Unmarshal(raw, &rec)
var req struct {
Address string `json:"address"`
}
_ = json.Unmarshal([]byte(rec.Request), &req)
rows = append(rows, logRow{Utterance: rec.Address, Request: req.Address})
}
return rows, nil
}
func fetchLogCases(opts shadowOptions) ([]logRow, error) {
body := map[string]any{
"size": opts.logLimit,
"sort": []any{map[string]any{"start_date": map[string]string{"order": "desc"}}},
"_source": []string{"address", "request"},
}
data, err := esRequest(context.Background(), newESClient(opts), opts.esURL, opts.logESIndex, "_search", body)
if err != nil {
return nil, err
}
var resp struct {
Hits struct {
Hits []struct {
Source struct {
Address string `json:"address"`
Request string `json:"request"`
} `json:"_source"`
} `json:"hits"`
} `json:"hits"`
}
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
rows := make([]logRow, 0, len(resp.Hits.Hits))
for _, h := range resp.Hits.Hits {
var req struct {
Address string `json:"address"`
}
_ = json.Unmarshal([]byte(h.Source.Request), &req)
rows = append(rows, logRow{Utterance: h.Source.Address, Request: req.Address})
}
return rows, nil
}
type esClient struct {
http *http.Client
user string
pass string
}
func newESClient(opts shadowOptions) *esClient {
tr := &http.Transport{}
if opts.insecure {
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} // #nosec G402 -- self-signed on-prem ES
}
return &esClient{
http: &http.Client{Timeout: opts.timeout, Transport: tr},
user: opts.esUser,
pass: opts.esPass,
}
}
// esRequest performs a read-only search. It only builds `_search` URLs.
func esRequest(ctx context.Context, c *esClient, base, index, path string, body any) ([]byte, error) {
raw, err := json.Marshal(body)
if err != nil {
return nil, err
}
url := strings.TrimRight(base, "/") + "/" + index + "/" + path
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, bytes.NewReader(raw))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
if c.user != "" {
req.SetBasicAuth(c.user, c.pass)
}
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
data, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("ES %s: HTTP %d: %s", path, resp.StatusCode, truncate(string(data), 200))
}
return data, nil
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// oldDeleteWords mirrors the administrative words the EДДС robot removes
// before calling Elasticsearch (robot script: DELETE_WORDS). Matching is
// token-based here, so "новосибирская" is not mangled by removing the
// substring "новосибирск".
var oldDeleteWords = map[string]struct{}{
"дзержинский": {}, "железнодорожный": {}, "заельцовский": {},
"калининский": {}, "кировский": {}, "ленинский": {}, "октябрьский": {},
"первомайский": {}, "советский": {}, "центральный": {},
"район": {}, "город": {}, "номер": {}, "новосибирск": {}, "дом": {},
"г": {}, "обл": {}, "область": {},
}
func preprocessOld(q string) string {
fields := strings.Fields(strings.ToLower(q))
out := fields[:0]
for _, tok := range fields {
tok = strings.Trim(tok, ".,;:()")
if tok == "" {
continue
}
if _, ok := oldDeleteWords[tok]; ok {
continue
}
out = append(out, tok)
}
return strings.Join(out, " ")
}
// oldQuery mirrors the EДДС robot's normalize_address query exactly.
func oldQuery(q string) map[string]any {
return map[string]any{
"query": map[string]any{
"bool": map[string]any{
"must": []any{map[string]any{
"multi_match": map[string]any{
"query": q, "fields": []string{"address"},
"fuzziness": "AUTO", "operator": "and", "prefix_length": 2,
},
}},
"should": []any{map[string]any{
"match_phrase": map[string]any{
"address": map[string]any{"query": q, "slop": 3, "boost": 2},
},
}},
},
},
"size": 1,
"_source": []string{"address", "district_name"},
}
}
type esHit struct {
Address string `json:"address"`
District string `json:"district_name"`
}
func oldSearch(ctx context.Context, c *esClient, opts shadowOptions, q string) (esHit, error) {
data, err := esRequest(ctx, c, opts.esURL, opts.esIndex, "_search", oldQuery(q))
if err != nil {
return esHit{}, err
}
var resp struct {
Hits struct {
Hits []struct {
Source esHit `json:"_source"`
} `json:"hits"`
} `json:"hits"`
}
if err := json.Unmarshal(data, &resp); err != nil {
return esHit{}, err
}
if len(resp.Hits.Hits) == 0 {
return esHit{}, nil
}
return resp.Hits.Hits[0].Source, nil
}
type shadowReport struct {
Mode string `json:"mode"`
Note string `json:"note"`
Queries int `json:"queries"`
Workers int `json:"workers"`
ESIndex string `json:"es_index"`
OldPreprocess bool `json:"old_preprocess"`
Old sideReport `json:"old"`
New sideReport `json:"new"`
Compare compareReport `json:"compare"`
Examples []shadowExample `json:"examples,omitempty"`
Errors int `json:"errors"`
ElapsedMS int64 `json:"elapsed_ms"`
}
type sideReport struct {
Found int `json:"found"`
Exact int `json:"exact"`
Wrong int `json:"wrong"`
Lossy int `json:"lossy"`
Partial int `json:"partial,omitempty"`
Ambiguous int `json:"ambiguous,omitempty"`
NotFound int `json:"not_found,omitempty"`
Error int `json:"error,omitempty"`
P50MS float64 `json:"p50_ms"`
P95MS float64 `json:"p95_ms"`
P99MS float64 `json:"p99_ms"`
}
type compareReport struct {
BothCorrect int `json:"both_correct"`
OldCorrectNewNotExact int `json:"old_correct_new_not_exact"`
OldWrongNewSafe int `json:"old_wrong_new_safe"`
OldWrongNewCorrect int `json:"old_wrong_new_correct"`
OldMissNewCorrect int `json:"old_miss_new_correct,omitempty"`
OldWrongNewWrong int `json:"old_wrong_new_wrong"`
OldLossyNewSafe int `json:"old_lossy_new_safe,omitempty"`
OldLossyNewCorrect int `json:"old_lossy_new_correct,omitempty"`
BothNotExact int `json:"both_not_exact"`
Same int `json:"same,omitempty"`
OldOnly int `json:"old_only,omitempty"`
NewOnly int `json:"new_only,omitempty"`
Differ int `json:"differ,omitempty"`
BothMiss int `json:"both_miss,omitempty"`
}
type shadowExample struct {
Kind string `json:"kind"`
Query string `json:"query"`
Expected string `json:"expected,omitempty"`
Old string `json:"old,omitempty"`
New string `json:"new,omitempty"`
}
type shadowOutcome struct {
hit esHit
parsed address.ParsedAddress
has bool
err error
latency time.Duration
}
func runShadowQueries(opts shadowOptions, cases []shadowCase, mode, note string, parser *address.Parser, resolver *match.Resolver, client *esClient) shadowReport {
ctx := context.Background()
start := time.Now()
type job struct{ c shadowCase }
type result struct {
c shadowCase
old shadowOutcome
newRes match.Result
newLat time.Duration
newErr error
}
jobs := make(chan job)
results := make(chan result)
var wg sync.WaitGroup
for i := 0; i < opts.workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := range jobs {
oldStart := time.Now()
oldQ := j.c.Query
if opts.oldPreproc {
oldQ = preprocessOld(oldQ)
}
hit, err := oldSearch(ctx, client, opts, oldQ)
old := shadowOutcome{hit: hit, err: err, latency: time.Since(oldStart)}
if err == nil {
old.parsed = parser.ParseInCity(hit.Address, j.c.ExpectCity)
old.has = hit.Address != ""
if j.c.ExpectCity == "" && opts.city != "" {
old.parsed = parser.ParseInCity(hit.Address, opts.city)
}
}
newStart := time.Now()
res := resolver.Resolve(ctx, j.c.Query)
results <- result{c: j.c, old: old, newRes: res, newLat: time.Since(newStart)}
}
}()
}
go func() {
for _, c := range cases {
jobs <- job{c: c}
}
close(jobs)
wg.Wait()
close(results)
}()
var oldLat, newLat []time.Duration
rep := shadowReport{
Mode: mode, Note: note, Workers: opts.workers, ESIndex: opts.esIndex,
OldPreprocess: opts.oldPreproc, Queries: len(cases), Examples: []shadowExample{},
}
for r := range results {
oldLat = append(oldLat, r.old.latency)
newLat = append(newLat, r.newLat)
if r.old.err != nil {
rep.Old.Error++
rep.Errors++
}
if r.newErr != nil {
rep.New.Error++
rep.Errors++
}
if mode == "generated" {
classifyGenerated(&rep, opts, parser, r.c, r.old, r.newRes)
} else {
classifyLog(&rep, opts, parser, r.c, r.old, r.newRes)
}
}
rep.Old.P50MS, rep.Old.P95MS, rep.Old.P99MS = percentiles(oldLat)
rep.New.P50MS, rep.New.P95MS, rep.New.P99MS = percentiles(newLat)
rep.ElapsedMS = time.Since(start).Milliseconds()
return rep
}
func classifyGenerated(rep *shadowReport, opts shadowOptions, parser *address.Parser, c shadowCase, old shadowOutcome, newRes match.Result) {
want := parser.ParseInCity(c.Expect, c.ExpectCity)
oldExact := old.has && sameAddressUnit(old.parsed, want)
oldWrong := old.has && !oldExact && c.RequireExact
oldLossy := old.has && !oldExact && !c.RequireExact
newExact := newRes.Status == address.StatusFound && sameAddressUnit(parser.ParseInCity(newRes.Address, c.ExpectCity), want)
newWrong := newRes.Status == address.StatusFound && !newExact && c.RequireExact
newSafe := newRes.Status == address.StatusPartial ||
newRes.Status == address.StatusAmbiguous ||
newRes.Status == address.StatusNotFound
if old.has {
rep.Old.Found++
}
if oldExact {
rep.Old.Exact++
}
if oldWrong {
rep.Old.Wrong++
}
if oldLossy {
rep.Old.Lossy++
}
if newRes.Status == address.StatusFound {
rep.New.Found++
}
if newExact {
rep.New.Exact++
}
if newWrong {
rep.New.Wrong++
}
if newRes.Status == address.StatusFound && !newExact && !c.RequireExact {
rep.New.Lossy++
}
switch newRes.Status {
case address.StatusPartial:
rep.New.Partial++
case address.StatusAmbiguous:
rep.New.Ambiguous++
case address.StatusNotFound:
rep.New.NotFound++
}
switch {
case oldExact && newExact:
rep.Compare.BothCorrect++
case oldExact:
rep.Compare.OldCorrectNewNotExact++
addExample(rep, opts, shadowExample{Kind: "old_correct_new_not_exact", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case newExact && oldWrong:
rep.Compare.OldWrongNewCorrect++
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case newExact && oldLossy:
rep.Compare.OldLossyNewCorrect++
addExample(rep, opts, shadowExample{Kind: "old_lossy_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case newExact:
rep.Compare.OldMissNewCorrect++
addExample(rep, opts, shadowExample{Kind: "old_miss_new_correct", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case oldWrong && newWrong:
rep.Compare.OldWrongNewWrong++
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_wrong", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case oldWrong && newSafe:
rep.Compare.OldWrongNewSafe++
addExample(rep, opts, shadowExample{Kind: "old_wrong_new_safe", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
case oldLossy && newSafe:
rep.Compare.OldLossyNewSafe++
addExample(rep, opts, shadowExample{Kind: "old_lossy_new_safe", Query: c.Query, Expected: c.Expect, Old: old.hit.Address, New: newRes.Address})
default:
rep.Compare.BothNotExact++
}
}
func classifyLog(rep *shadowReport, opts shadowOptions, parser *address.Parser, c shadowCase, old shadowOutcome, newRes match.Result) {
if old.has {
rep.Old.Found++
}
if newRes.Status == address.StatusFound {
rep.New.Found++
}
oldOK := old.has && old.parsed.HasHouse
newOK := newRes.Status == address.StatusFound
switch {
case oldOK && newOK && sameAddressUnit(old.parsed, parser.ParseInCity(newRes.Address, opts.city)):
rep.Compare.Same++
case oldOK && !newOK:
rep.Compare.OldOnly++
addExample(rep, opts, shadowExample{Kind: "old_only", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
case !oldOK && newOK:
rep.Compare.NewOnly++
addExample(rep, opts, shadowExample{Kind: "new_only", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
case oldOK && newOK:
rep.Compare.Differ++
addExample(rep, opts, shadowExample{Kind: "differ", Query: c.Query, Old: old.hit.Address, New: newRes.Address})
default:
rep.Compare.BothMiss++
}
}
func addExample(rep *shadowReport, opts shadowOptions, ex shadowExample) {
if len(rep.Examples) >= opts.maxExamples {
return
}
rep.Examples = append(rep.Examples, ex)
}
func sameAddressUnit(a, b address.ParsedAddress) bool {
return a.CityKey == b.CityKey &&
a.StreetKey == b.StreetKey &&
a.House == b.House && a.HouseFrac == b.HouseFrac && a.HouseLetter == b.HouseLetter &&
a.Building == b.Building && a.BuildingFrac == b.BuildingFrac && a.BuildingLetter == b.BuildingLetter &&
a.Structure == b.Structure
}
func percentiles(values []time.Duration) (p50, p95, p99 float64) {
if len(values) == 0 {
return 0, 0, 0
}
sort.Slice(values, func(i, j int) bool { return values[i] < values[j] })
at := func(q float64) float64 {
i := int(float64(len(values)-1) * q)
return float64(values[i].Microseconds()) / 1000
}
return at(0.50), at(0.95), at(0.99)
}
+25 -33
View File
@@ -1,9 +1,9 @@
// Command addrkitd runs the address resolution service. // Command addrkitd runs the address resolution service.
// //
// It loads the canonical registry into memory, wires the optional LLM rewriter // It loads the canonical registry into memory, wires the optional adapters
// and serves the versioned JSON API. The process fails closed: if the registry // (ArcGIS portal, LLM rewriter) and serves the versioned JSON API. The process
// cannot be loaded or is empty it exits rather than serving misleading // fails closed: if the registry cannot be loaded it exits rather than serving
// "not found" answers. // an empty index.
package main package main
import ( import (
@@ -17,11 +17,11 @@ import (
"syscall" "syscall"
"time" "time"
"git.totmin.ru/en2zmax/addrkit/internal/adapters/arcgis"
"git.totmin.ru/en2zmax/addrkit/internal/adapters/httpapi" "git.totmin.ru/en2zmax/addrkit/internal/adapters/httpapi"
"git.totmin.ru/en2zmax/addrkit/internal/adapters/llm" "git.totmin.ru/en2zmax/addrkit/internal/adapters/llm"
"git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore" "git.totmin.ru/en2zmax/addrkit/internal/adapters/registrystore"
"git.totmin.ru/en2zmax/addrkit/internal/config" "git.totmin.ru/en2zmax/addrkit/internal/config"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match" "git.totmin.ru/en2zmax/addrkit/internal/domain/match"
) )
@@ -39,14 +39,10 @@ func main() {
version, entries, err := registrystore.Load(cfg.DataPath) version, entries, err := registrystore.Load(cfg.DataPath)
if err != nil { if err != nil {
// Fail closed: no usable registry, no service. // Fail closed: no registry, no service.
logger.Error("load registry", "error", err, "path", cfg.DataPath) logger.Error("load registry", "error", err, "path", cfg.DataPath)
os.Exit(1) os.Exit(1)
} }
if len(entries) == 0 {
logger.Error("registry is empty; refusing to serve", "path", cfg.DataPath)
os.Exit(1)
}
logger.Info("registry loaded", "version", version, "entries", len(entries)) logger.Info("registry loaded", "version", version, "entries", len(entries))
index := match.NewIndex(version, entries, match.Options{ index := match.NewIndex(version, entries, match.Options{
@@ -55,47 +51,33 @@ func main() {
MaxCandidates: cfg.Resolve.MaxCandidates, MaxCandidates: cfg.Resolve.MaxCandidates,
}) })
parser := address.NewParser(address.NewDictionary(cfg.Cities), cfg.DefaultCity)
var rewriter match.Rewriter var rewriter match.Rewriter
if cfg.LLM.Enabled { if cfg.LLM.Enabled {
rewriter = llm.New(cfg.LLM) rewriter = llm.New(cfg.LLM)
logger.Info("llm rewriter enabled", "url", cfg.LLM.URL, "model", cfg.LLM.Model) logger.Info("llm rewriter enabled", "url", cfg.LLM.URL, "model", cfg.LLM.Model)
} }
resolver := match.NewResolverWithOptions(index, rewriter, match.ResolverOptions{ resolver := match.NewResolver(index, rewriter)
Parser: parser,
OnRewriteError: func(err error) {
logger.Warn("llm rewriter failed; using deterministic result", "error", err)
},
})
cities := resolver.Cities() portal := arcgis.New(
logger.Info("cities loaded", "count", len(cities), "cities", cities, "default", cfg.DefaultCity) cfg.Portal.URL,
if cfg.DefaultCity != "" { time.Duration(cfg.Portal.TimeoutSeconds)*time.Second,
if len(cities) > 1 { cfg.Portal.Retries,
logger.Warn("default_city is set for a multi-city index; city-less queries will always use it", "default", cfg.DefaultCity) )
} else if len(cities) == 1 && cities[0] != cfg.DefaultCity {
logger.Warn("default_city is not present in the index", "default", cfg.DefaultCity, "index_city", cities[0])
}
}
srv := httpapi.New(resolver, logger) loc := operationalLocation()
srv := httpapi.New(resolver, portal, logger, loc)
httpServer := &http.Server{ httpServer := &http.Server{
Addr: cfg.Addr, Addr: cfg.Addr,
Handler: srv.Handler(), Handler: srv.Handler(),
ReadHeaderTimeout: 5 * time.Second, ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 15 * time.Second,
WriteTimeout: 20 * time.Second,
IdleTimeout: 60 * time.Second,
MaxHeaderBytes: 64 << 10,
} }
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop() defer stop()
go func() { go func() {
logger.Info("listening", "addr", cfg.Addr) logger.Info("listening", "addr", cfg.Addr, "tz", loc.String())
if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { if err := httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.Error("serve", "error", err) logger.Error("serve", "error", err)
stop() stop()
@@ -126,3 +108,13 @@ func newLogger(level string) *slog.Logger {
} }
return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl})) return slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl}))
} }
// operationalLocation returns Asia/Novosibirsk when the tz database is
// available and falls back to a fixed UTC+7 offset otherwise (containers
// without tzdata).
func operationalLocation() *time.Location {
if loc, err := time.LoadLocation("Asia/Novosibirsk"); err == nil {
return loc
}
return time.FixedZone("Asia/Novosibirsk", 7*3600)
}
+5 -7
View File
@@ -2,18 +2,16 @@
"addr": ":6171", "addr": ":6171",
"data_path": "registry.db", "data_path": "registry.db",
"log_level": "info", "log_level": "info",
"cities": [
{
"name": "Новосибирск",
"aliases": ["нск", "новосиб", "новосибирская область", "нсо"]
}
],
"default_city": "Новосибирск",
"resolve": { "resolve": {
"fuzzy_threshold": 0.82, "fuzzy_threshold": 0.82,
"ambiguity_margin": 0.06, "ambiguity_margin": 0.06,
"max_candidates": 5 "max_candidates": 5
}, },
"portal": {
"url": "${ADDRKIT_PORTAL_URL}",
"timeout_seconds": 2,
"retries": 1
},
"llm": { "llm": {
"enabled": false, "enabled": false,
"url": "${ADDRKIT_LLM_URL}", "url": "${ADDRKIT_LLM_URL}",
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit] [Unit]
Description=addrkit multi-city address resolution service Description=addrkit address resolution service
Documentation=file:///opt/addrkit/README.md Documentation=file:///opt/addrkit/README.md
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
+1 -7
View File
@@ -2,13 +2,7 @@
"addr": "127.0.0.1:6172", "addr": "127.0.0.1:6172",
"data_path": "dev/registry.db", "data_path": "dev/registry.db",
"log_level": "warn", "log_level": "warn",
"cities": [
{
"name": "Новосибирск",
"aliases": ["нск", "новосиб", "новосибирская область", "нсо"]
}
],
"default_city": "Новосибирск",
"resolve": { "fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5 }, "resolve": { "fuzzy_threshold": 0.82, "ambiguity_margin": 0.06, "max_candidates": 5 },
"portal": { "url": "https://map.novo-sibirsk.ru/elitegis/rest/services/maps/disconnections/MapServer/101/query", "timeout_seconds": 2, "retries": 0 },
"llm": { "enabled": true, "url": "http://127.0.0.1:6193", "token": "EMPTY", "model": "qwen3-14b-awq", "timeout_seconds": 5, "max_tokens": 512, "disable_thinking": true } "llm": { "enabled": true, "url": "http://127.0.0.1:6193", "token": "EMPTY", "model": "qwen3-14b-awq", "timeout_seconds": 5, "max_tokens": 512, "disable_thinking": true }
} }
+226
View File
@@ -0,0 +1,226 @@
// Package arcgis is the adapter for the external outages portal exposed as an
// ArcGIS MapServer /query endpoint.
//
// The portal only supports substring (LIKE) filtering, so precision is
// guaranteed locally: whatever the portal returns is re-filtered by the
// outages domain using the parsed house unit. The client adds the reliability
// the portal lacks: a hard timeout, bounded retries with backoff and context
// cancellation.
package arcgis
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
)
// Client is a bounded HTTP client for the portal.
type Client struct {
baseURL string
http *http.Client
retries int
}
// New builds a client. timeout applies to a single attempt; retries is the
// number of *additional* attempts after the first.
func New(baseURL string, timeout time.Duration, retries int) *Client {
return &Client{
baseURL: baseURL,
http: &http.Client{Timeout: timeout},
retries: retries,
}
}
// Query asks the portal for outages matching the canonical address. The
// returned records are raw portal rows; callers must still filter by house.
func (c *Client) Query(ctx context.Context, canonical string) ([]outages.Outage, error) {
params := url.Values{}
params.Set("where", "address like '"+likePattern(canonical)+"'")
params.Set("outFields", "")
params.Set("returnGeometry", "false")
params.Set("f", "json")
var lastErr error
for attempt := 0; attempt <= c.retries; attempt++ {
if attempt > 0 {
backoff := time.Duration(attempt) * 150 * time.Millisecond
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(backoff):
}
}
rows, err := c.do(ctx, params)
if err == nil {
return rows, nil
}
lastErr = err
}
return nil, fmt.Errorf("portal query failed after %d attempts: %w", c.retries+1, lastErr)
}
func (c *Client) do(ctx context.Context, params url.Values) ([]outages.Outage, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"?"+params.Encode(), nil)
if err != nil {
return nil, fmt.Errorf("build request: %w", err)
}
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("portal status %d", resp.StatusCode)
}
// Bound the response size to protect the service from a misbehaving portal.
body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, err
}
return parseResponse(body)
}
// portalResponse mirrors the subset of the ArcGIS /query envelope we use.
type portalResponse struct {
Features []struct {
Value string `json:"value"`
Attributes map[string]any `json:"attributes"`
} `json:"features"`
Fields []struct {
Name string `json:"name"`
Domain struct {
CodedValues []struct {
Code any `json:"code"`
Name string `json:"name"`
} `json:"codedValues"`
} `json:"domain"`
} `json:"fields"`
}
func parseResponse(body []byte) ([]outages.Outage, error) {
var pr portalResponse
if err := json.Unmarshal(body, &pr); err != nil {
return nil, fmt.Errorf("decode portal response: %w", err)
}
// Build code -> name maps from the coded-value domains.
maps := map[string]map[string]string{}
for _, f := range pr.Fields {
if len(f.Domain.CodedValues) == 0 {
continue
}
m := make(map[string]string, len(f.Domain.CodedValues))
for _, cv := range f.Domain.CodedValues {
m[codeKey(cv.Code)] = cv.Name
}
maps[f.Name] = m
}
out := make([]outages.Outage, 0, len(pr.Features))
for _, feat := range pr.Features {
o := outages.Outage{
Address: addressOf(feat.Attributes, feat.Value),
Type: lookup(maps, "type_id", feat.Attributes),
System: lookup(maps, "system_id", feat.Attributes),
Time: lookup(maps, "time_category", feat.Attributes),
StartDate: stringOf(feat.Attributes["start_date"]),
EndDate: stringOf(feat.Attributes["end_date"]),
}
o.Action = actionFor(o.System)
out = append(out, o)
}
return out, nil
}
func addressOf(attrs map[string]any, fallback string) string {
if v, ok := attrs["address"]; ok {
if s := stringOf(v); s != "" {
return s
}
}
return fallback
}
func lookup(maps map[string]map[string]string, field string, attrs map[string]any) string {
m, ok := maps[field]
if !ok {
return stringOf(attrs[field])
}
return m[codeKey(attrs[field])]
}
// codeKey normalises a coded value coming from JSON (float64) or a string.
func codeKey(v any) string {
switch t := v.(type) {
case nil:
return ""
case string:
return t
case float64:
return strconv.FormatInt(int64(t), 10)
case json.Number:
return t.String()
default:
return fmt.Sprintf("%v", v)
}
}
func stringOf(v any) string {
if s, ok := v.(string); ok {
return s
}
if v == nil {
return ""
}
return fmt.Sprintf("%v", v)
}
// actionFor maps a portal system name to the canonical action code used by the
// scenario (kept for parity with the existing integration).
func actionFor(system string) string {
switch system {
case "Горячее водоснабжение":
return "waterWarm"
case "Холодное водоснабжение и канализация", "Холодное водоснабжение":
return "waterCold"
case "Электроснабжение":
return "electricitySupply"
case "Газоснабжение":
return "gasSupply"
case "Теплоснабжение":
return "heating"
default:
return ""
}
}
// likePattern builds the substring pattern the portal supports. Numeric tokens
// are padded with spaces, matching the historical integration, so that "1" does
// not incidentally match "10".
func likePattern(canonical string) string {
parts := strings.Fields(canonical)
for i, part := range parts {
if containsDigit(part) {
parts[i] = " " + part + " "
}
}
return "%" + strings.Join(parts, "%") + "%"
}
func containsDigit(s string) bool {
for _, r := range s {
if r >= '0' && r <= '9' {
return true
}
}
return false
}
+68
View File
@@ -0,0 +1,68 @@
package arcgis
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"time"
)
const envelope = `{
"features": [
{"value": "Станционная улица 1/1", "attributes": {
"address": "Станционная улица 1/1",
"type_id": 1, "system_id": 5, "time_category": 2,
"start_date": "2026-10-01T00:00:00", "end_date": "2026-10-31T00:00:00"
}}
],
"fields": [
{"name": "type_id", "domain": {"codedValues": [{"code": 1, "name": "Плановое"}]}},
{"name": "system_id", "domain": {"codedValues": [{"code": 5, "name": "Теплоснабжение"}]}},
{"name": "time_category", "domain": {"codedValues": [{"code": 2, "name": "Текущие"}]}}
]
}`
func TestQueryParsesEnvelope(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.URL.Query().Get("f"); got != "json" {
t.Errorf("missing f=json, got %q", got)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(envelope))
}))
defer ts.Close()
c := New(ts.URL, time.Second, 0)
rows, err := c.Query(context.Background(), "Станционная улица 1/1")
if err != nil {
t.Fatalf("Query: %v", err)
}
if len(rows) != 1 {
t.Fatalf("rows = %d, want 1", len(rows))
}
got := rows[0]
if got.Type != "Плановое" || got.System != "Теплоснабжение" || got.Time != "Текущие" {
t.Errorf("coded values not resolved: %+v", got)
}
if got.Action != "heating" {
t.Errorf("Action = %q, want heating", got.Action)
}
}
func TestQueryRetriesThenFails(t *testing.T) {
var calls int
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
http.Error(w, "boom", http.StatusInternalServerError)
}))
defer ts.Close()
c := New(ts.URL, time.Second, 2)
if _, err := c.Query(context.Background(), "Станционная улица 1/1"); err == nil {
t.Fatal("expected error")
}
if calls != 3 {
t.Errorf("calls = %d, want 3 (1 + 2 retries)", calls)
}
}
+3 -25
View File
@@ -14,13 +14,10 @@ type ctxKey int
const requestIDKey ctxKey = 0 const requestIDKey ctxKey = 0
// requestIDMW assigns a request id (for correlation in logs) and echoes it. // requestIDMW assigns a request id (for correlation in logs) and echoes it.
// A client-supplied id is accepted only when it is short and well-formed; the
// header ends up in logs, so accepting arbitrary bytes would be an injection
// and log-flooding vector.
func requestIDMW(next http.Handler) http.Handler { func requestIDMW(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id := r.Header.Get("X-Request-Id") id := r.Header.Get("X-Request-Id")
if !validRequestID(id) { if id == "" {
id = newRequestID() id = newRequestID()
} }
w.Header().Set("X-Request-Id", id) w.Header().Set("X-Request-Id", id)
@@ -29,23 +26,6 @@ func requestIDMW(next http.Handler) http.Handler {
}) })
} }
const maxRequestIDLen = 128
func validRequestID(id string) bool {
if id == "" || len(id) > maxRequestIDLen {
return false
}
for _, r := range id {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
case r == '-', r == '_', r == '.':
default:
return false
}
}
return true
}
// loggingMW logs one line per request with method, path, status and duration. // loggingMW logs one line per request with method, path, status and duration.
// It records no addresses (PII) — only routing metadata. // It records no addresses (PII) — only routing metadata.
func loggingMW(log *slog.Logger) func(http.Handler) http.Handler { func loggingMW(log *slog.Logger) func(http.Handler) http.Handler {
@@ -65,10 +45,8 @@ func loggingMW(log *slog.Logger) func(http.Handler) http.Handler {
} }
} }
// timeoutMW bounds the whole request by setting a context deadline. Handlers // timeoutMW bounds the whole request, so a slow portal cannot tie up a
// and adapters that talk to the network honour it; purely CPU-bound handlers // goroutine indefinitely.
// are additionally bounded by input limits and the HTTP server's read/write
// timeouts in cmd/addrkitd.
func timeoutMW(d time.Duration) func(http.Handler) http.Handler { func timeoutMW(d time.Duration) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+59 -68
View File
@@ -1,54 +1,52 @@
// Package httpapi exposes the addrkit domain over a small, versioned JSON API. // Package httpapi exposes the addrkit domain over a small, versioned JSON API.
// //
// It contains no business logic: handlers decode requests, call the resolver // It contains no business logic: handlers decode requests, call the resolver
// and encode responses. Cross-cutting concerns (timeouts, recovery, request // and the outages source, and encode responses. Cross-cutting concerns
// ids, logging) live in middleware. // (timeouts, recovery, request ids, logging) live in middleware.
package httpapi package httpapi
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"log/slog"
"net/http" "net/http"
"time" "time"
"unicode/utf8"
"log/slog"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match" "git.totmin.ru/en2zmax/addrkit/internal/domain/match"
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
) )
// Resolver is the read side of the domain (implemented by *match.Resolver). // Resolver is the read side of the domain (implemented by *match.Resolver).
type Resolver interface { type Resolver interface {
Resolve(ctx context.Context, raw string) match.Result Resolve(ctx context.Context, raw string) match.Result
ResolveInCity(ctx context.Context, raw, city string) match.Result
MatchCity(query string) (string, bool)
Version() string Version() string
Ready() bool Ready() bool
Cities() []string
} }
// maxAddressRunes bounds user input. A real address is far shorter; the limit
// keeps a pathological string from turning into a CPU-bound fuzzy scan.
const maxAddressRunes = 512
// maxCityQueryRunes bounds the separate city-only endpoint.
const maxCityQueryRunes = 128
// Server wires the domain to HTTP. // Server wires the domain to HTTP.
type Server struct { type Server struct {
resolver Resolver resolver Resolver
outages outages.Source
log *slog.Logger log *slog.Logger
now func() time.Time
loc *time.Location
timeout time.Duration timeout time.Duration
} }
// New builds a server. // New builds a server. loc controls how outage dates are interpreted; it
func New(resolver Resolver, log *slog.Logger) *Server { // should be the operational timezone (Asia/Novosibirsk).
if log == nil { func New(resolver Resolver, src outages.Source, log *slog.Logger, loc *time.Location) *Server {
log = slog.Default() if loc == nil {
loc = time.UTC
} }
return &Server{ return &Server{
resolver: resolver, resolver: resolver,
outages: src,
log: log, log: log,
now: time.Now,
loc: loc,
timeout: 10 * time.Second, timeout: 10 * time.Second,
} }
} }
@@ -57,21 +55,16 @@ func New(resolver Resolver, log *slog.Logger) *Server {
func (s *Server) Handler() http.Handler { func (s *Server) Handler() http.Handler {
mux := http.NewServeMux() mux := http.NewServeMux()
mux.HandleFunc("POST /v1/resolve", s.handleResolve) mux.HandleFunc("POST /v1/resolve", s.handleResolve)
mux.HandleFunc("POST /v1/city", s.handleCity) mux.HandleFunc("POST /v1/outages", s.handleOutages)
mux.HandleFunc("GET /v1/cities", s.handleCities)
mux.HandleFunc("GET /v1/healthz", s.handleHealthz) mux.HandleFunc("GET /v1/healthz", s.handleHealthz)
mux.HandleFunc("GET /v1/readyz", s.handleReadyz) mux.HandleFunc("GET /v1/readyz", s.handleReadyz)
mux.HandleFunc("GET /v1/version", s.handleVersion) mux.HandleFunc("GET /v1/version", s.handleVersion)
// Logging sits outside recovery so a panic is still recorded as a 500. return recoverMW(s.log)(requestIDMW(loggingMW(s.log)(timeoutMW(s.timeout)(mux))))
return requestIDMW(loggingMW(s.log)(recoverMW(s.log)(timeoutMW(s.timeout)(mux))))
} }
type resolveRequest struct { type resolveRequest struct {
Address string `json:"address"` Address string `json:"address"`
// City is an optional hint for clients that keep the city separate from
// the street line. It overrides a city found in the address text.
City string `json:"city"`
} }
func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) { func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) {
@@ -79,49 +72,55 @@ func (s *Server) handleResolve(w http.ResponseWriter, r *http.Request) {
if !ok { if !ok {
return return
} }
writeJSON(w, http.StatusOK, s.resolver.ResolveInCity(r.Context(), req.Address, req.City)) writeJSON(w, http.StatusOK, s.resolver.Resolve(r.Context(), req.Address))
} }
type cityRequest struct { type outagesResponse struct {
Query string `json:"query"` Status address.Status `json:"status"`
Address string `json:"address,omitempty"`
District string `json:"district,omitempty"`
Outages []outages.Outage `json:"outages,omitempty"`
AsOf string `json:"as_of,omitempty"`
DataVersion string `json:"data_version,omitempty"`
} }
type cityResponse struct { func (s *Server) handleOutages(w http.ResponseWriter, r *http.Request) {
Status string `json:"status"` req, ok := s.decode(w, r)
City string `json:"city,omitempty"` if !ok {
} return
}
ctx := r.Context()
res := s.resolver.Resolve(ctx, req.Address)
// handleCity normalizes a city-only query ("нск", "г. Новосибирск", out := outagesResponse{
// "Новосибирская область"). It is the small, separate use case for projects Status: res.Status,
// that need city names but not streets. Address: res.Address,
func (s *Server) handleCity(w http.ResponseWriter, r *http.Request) { District: res.District,
var req cityRequest DataVersion: res.DataVersion,
r.Body = http.MaxBytesReader(w, r.Body, 16*1024) }
if err := json.NewDecoder(r.Body).Decode(&req); err != nil { if res.Status != address.StatusFound {
writeError(w, http.StatusBadRequest, "invalid json body") // No canonical address => nothing to query. Return the resolver's
// verdict so the caller can ask for clarification.
writeJSON(w, http.StatusOK, out)
return return
} }
if req.Query == "" {
writeError(w, http.StatusBadRequest, "query is required")
return
}
if utf8.RuneCountInString(req.Query) > maxCityQueryRunes {
writeError(w, http.StatusBadRequest, "query is too long")
return
}
if city, ok := s.resolver.MatchCity(req.Query); ok {
writeJSON(w, http.StatusOK, cityResponse{Status: "found", City: city})
return
}
writeJSON(w, http.StatusOK, cityResponse{Status: "not_found"})
}
func (s *Server) handleCities(w http.ResponseWriter, _ *http.Request) { raw, err := s.outages.Query(ctx, res.Address)
cities := s.resolver.Cities() if err != nil {
if cities == nil { if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
cities = []string{} out.Status = address.StatusPortalTimeout
} else {
out.Status = address.StatusInfra
} }
writeJSON(w, http.StatusOK, map[string][]string{"cities": cities}) writeJSON(w, http.StatusOK, out)
return
}
parsed := address.Parse(res.Address)
current := outages.Current(raw, s.now(), s.loc)
out.Outages = outages.MatchingHouse(current, parsed)
out.AsOf = s.now().In(s.loc).Format(time.RFC3339)
writeJSON(w, http.StatusOK, out)
} }
func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) { func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) {
@@ -152,14 +151,6 @@ func (s *Server) decode(w http.ResponseWriter, r *http.Request) (resolveRequest,
writeError(w, http.StatusBadRequest, "address is required") writeError(w, http.StatusBadRequest, "address is required")
return resolveRequest{}, false return resolveRequest{}, false
} }
if utf8.RuneCountInString(req.Address) > maxAddressRunes {
writeError(w, http.StatusBadRequest, "address is too long")
return resolveRequest{}, false
}
if utf8.RuneCountInString(req.City) > 128 {
writeError(w, http.StatusBadRequest, "city is too long")
return resolveRequest{}, false
}
return req, true return req, true
} }
+47 -136
View File
@@ -2,51 +2,46 @@ package httpapi
import ( import (
"bytes" "bytes"
"context"
"encoding/json" "encoding/json"
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings"
"testing" "testing"
"time"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address" "git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match" "git.totmin.ru/en2zmax/addrkit/internal/domain/match"
"git.totmin.ru/en2zmax/addrkit/internal/domain/outages"
) )
func entry(t *testing.T, city, raw string) address.Entry {
t.Helper()
p := address.NewParser(address.NewDictionary(cityDictionary()), "").ParseInCity(raw, city)
return address.Entry{
ID: raw, Address: raw, District: "Тестовый",
City: p.City, CityKey: p.CityKey,
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
Structure: p.Structure,
}
}
func cityDictionary() []address.City {
return []address.City{
{Name: "Новосибирск", Aliases: []string{"нск", "новосиб", "новосибирская область"}},
{Name: "Бердск"},
}
}
func testResolver(t *testing.T) *match.Resolver { func testResolver(t *testing.T) *match.Resolver {
t.Helper() t.Helper()
var entries []address.Entry var entries []address.Entry
for _, a := range []string{"Станционная улица 1/1", "2-я Станционная улица 1/1", "Искитимская улица 5"} { for _, a := range []string{"Станционная улица 1/1", "2-я Станционная улица 1/1", "Искитимская улица 5"} {
entries = append(entries, entry(t, "", a)) p := address.Parse(a)
entries = append(entries, address.Entry{
ID: a, Address: a, District: "Тестовый",
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
})
} }
return match.NewResolver(match.NewIndex("test-version", entries, match.DefaultOptions()), nil) return match.NewResolver(match.NewIndex("test-version", entries, match.DefaultOptions()), nil)
} }
func newServer(t *testing.T, resolver Resolver) *httptest.Server { type fakeSource struct {
list []outages.Outage
err error
}
func (f fakeSource) Query(context.Context, string) ([]outages.Outage, error) {
return f.list, f.err
}
func newServer(t *testing.T, src fakeSource) *httptest.Server {
t.Helper() t.Helper()
srv := New(resolver, slog.New(slog.NewTextHandler(io.Discard, nil))) srv := New(testResolver(t), src, slog.New(slog.NewTextHandler(io.Discard, nil)), time.UTC)
ts := httptest.NewServer(srv.Handler()) ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close) t.Cleanup(ts.Close)
return ts return ts
@@ -62,7 +57,7 @@ func postJSON(t *testing.T, url, body string) *http.Response {
} }
func TestResolveEndpoint(t *testing.T) { func TestResolveEndpoint(t *testing.T) {
ts := newServer(t, testResolver(t)) ts := newServer(t, fakeSource{})
resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"2-я Станционная 1/1"}`) resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"2-я Станционная 1/1"}`)
defer resp.Body.Close() defer resp.Body.Close()
@@ -78,120 +73,38 @@ func TestResolveEndpoint(t *testing.T) {
} }
} }
func TestResolveValidation(t *testing.T) { func TestOutagesEndpointFiltersHouse(t *testing.T) {
ts := newServer(t, testResolver(t)) src := fakeSource{list: []outages.Outage{
for _, body := range []string{`{}`, `{"address":""}`, `not json`} { {Address: "Станционная улица 1/1", System: "Теплоснабжение"},
resp := postJSON(t, ts.URL+"/v1/resolve", body) {Address: "Станционная улица 1/2", System: "Теплоснабжение"},
resp.Body.Close() }}
if resp.StatusCode != http.StatusBadRequest { ts := newServer(t, src)
t.Errorf("body %q: status = %d, want 400", body, resp.StatusCode) resp := postJSON(t, ts.URL+"/v1/outages", `{"address":"станционная 1/1"}`)
defer resp.Body.Close()
var out outagesResponse
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
t.Fatalf("decode: %v", err)
} }
if out.Status != address.StatusFound {
t.Fatalf("status = %q, want found", out.Status)
}
if len(out.Outages) != 1 || out.Outages[0].Address != "Станционная улица 1/1" {
t.Fatalf("house filter failed: %+v", out.Outages)
} }
} }
func TestResolveRejectsOversizedAddress(t *testing.T) { func TestResolveValidation(t *testing.T) {
ts := newServer(t, testResolver(t)) ts := newServer(t, fakeSource{})
body, _ := json.Marshal(map[string]string{"address": strings.Repeat("а", maxAddressRunes+1)}) resp := postJSON(t, ts.URL+"/v1/resolve", `{}`)
resp := postJSON(t, ts.URL+"/v1/resolve", string(body)) defer resp.Body.Close()
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest { if resp.StatusCode != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", resp.StatusCode) t.Fatalf("status = %d, want 400", resp.StatusCode)
} }
} }
func TestResolveCityHint(t *testing.T) {
entries := []address.Entry{
entry(t, "Новосибирск", "Ленина улица 5"),
entry(t, "Бердск", "Ленина улица 5"),
}
resolver := match.NewResolver(
match.NewIndex("test", entries, match.DefaultOptions()),
nil,
)
ts := newServer(t, resolver)
resp := postJSON(t, ts.URL+"/v1/resolve", `{"address":"Ленина 5","city":"Бердск"}`)
defer resp.Body.Close()
var res match.Result
if err := json.NewDecoder(resp.Body).Decode(&res); err != nil {
t.Fatalf("decode: %v", err)
}
if res.Status != address.StatusFound || res.City != "Бердск" || res.Address != "Ленина улица 5" {
t.Fatalf("unexpected result: %+v", res)
}
}
func TestCitiesEndpoint(t *testing.T) {
entries := []address.Entry{
entry(t, "Новосибирск", "Ленина улица 5"),
entry(t, "Бердск", "Ленина улица 5"),
}
resolver := match.NewResolver(match.NewIndex("test", entries, match.DefaultOptions()), nil)
ts := newServer(t, resolver)
resp, err := http.Get(ts.URL + "/v1/cities")
if err != nil {
t.Fatalf("get: %v", err)
}
defer resp.Body.Close()
var body struct {
Cities []string `json:"cities"`
}
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
t.Fatalf("decode: %v", err)
}
if len(body.Cities) != 2 || body.Cities[0] != "Бердск" || body.Cities[1] != "Новосибирск" {
t.Fatalf("cities = %v", body.Cities)
}
}
func TestCityEndpoint(t *testing.T) {
parser := address.NewParser(address.NewDictionary(cityDictionary()), "")
resolver := match.NewResolverWithOptions(
match.NewIndex("test", nil, match.DefaultOptions()),
nil,
match.ResolverOptions{Parser: parser},
)
ts := newServer(t, resolver)
for _, tt := range []struct {
query string
city string
}{
{"нск", "Новосибирск"},
{"г. Новосибирск", "Новосибирск"},
{"Новосибирская область", "Новосибирск"},
{"Бердск", "Бердск"},
{"Томск", ""},
} {
body, _ := json.Marshal(map[string]string{"query": tt.query})
resp := postJSON(t, ts.URL+"/v1/city", string(body))
var got cityResponse
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
resp.Body.Close()
t.Fatalf("%q: decode: %v", tt.query, err)
}
resp.Body.Close()
if tt.city == "" {
if got.Status != "not_found" {
t.Errorf("%q: status = %q, want not_found", tt.query, got.Status)
}
continue
}
if got.Status != "found" || got.City != tt.city {
t.Errorf("%q: %+v", tt.query, got)
}
}
resp := postJSON(t, ts.URL+"/v1/city", `{}`)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("empty query status = %d, want 400", resp.StatusCode)
}
}
func TestReadyzAndVersion(t *testing.T) { func TestReadyzAndVersion(t *testing.T) {
ts := newServer(t, testResolver(t)) ts := newServer(t, fakeSource{})
resp, err := http.Get(ts.URL + "/v1/readyz") resp, err := http.Get(ts.URL + "/v1/readyz")
if err != nil { if err != nil {
@@ -207,11 +120,9 @@ func TestReadyzAndVersion(t *testing.T) {
t.Fatalf("version: %v", err) t.Fatalf("version: %v", err)
} }
defer resp.Body.Close() defer resp.Body.Close()
var body map[string]string var v map[string]string
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { _ = json.NewDecoder(resp.Body).Decode(&v)
t.Fatalf("decode: %v", err) if v["data_version"] != "test-version" {
} t.Fatalf("version = %q", v["data_version"])
if body["data_version"] != "test-version" {
t.Fatalf("data_version = %q", body["data_version"])
} }
} }
+2 -13
View File
@@ -14,12 +14,10 @@ import (
_ "embed" _ "embed"
"encoding/json" "encoding/json"
"fmt" "fmt"
"io"
"net/http" "net/http"
"regexp" "regexp"
"strings" "strings"
"time" "time"
"unicode/utf8"
"git.totmin.ru/en2zmax/addrkit/internal/config" "git.totmin.ru/en2zmax/addrkit/internal/config"
) )
@@ -130,9 +128,7 @@ func (r *Rewriter) Rewrite(ctx context.Context, raw string) ([]string, error) {
} }
var cr chatResponse var cr chatResponse
// Bound the reply: a broken or hostile model must not be able to stream an if err := json.NewDecoder(resp.Body).Decode(&cr); err != nil {
// unbounded JSON body into the service.
if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&cr); err != nil {
return nil, fmt.Errorf("decode llm response: %w", err) return nil, fmt.Errorf("decode llm response: %w", err)
} }
if len(cr.Choices) == 0 { if len(cr.Choices) == 0 {
@@ -158,15 +154,8 @@ func parseVariants(content string) []string {
} }
out := make([]string, 0, len(parsed.Variants)) out := make([]string, 0, len(parsed.Variants))
for _, v := range parsed.Variants { for _, v := range parsed.Variants {
s := strings.TrimSpace(v) if s := strings.TrimSpace(v); s != "" {
// A variant is a short address; the cap also keeps a hostile model
// from making the resolver scan a megabyte-long token.
if s == "" || utf8.RuneCountInString(s) > 512 {
continue
}
out = append(out, s) out = append(out, s)
if len(out) == 3 {
break
} }
} }
return out return out
-14
View File
@@ -86,17 +86,3 @@ func TestParseVariants(t *testing.T) {
}) })
} }
} }
func TestParseVariantsCapsCountAndLength(t *testing.T) {
long := strings.Repeat("x", 600)
content := `{"variants":["` + long + `","a","b","c","d"]}`
out := parseVariants(content)
if len(out) != 3 {
t.Fatalf("variants = %v, want 3", out)
}
for _, v := range out {
if len([]rune(v)) > 512 {
t.Fatalf("oversized variant accepted: %d runes", len([]rune(v)))
}
}
}
+28 -97
View File
@@ -5,23 +5,14 @@ package registrystore
import ( import (
"database/sql" "database/sql"
"errors"
"fmt" "fmt"
"net/url"
"os" "os"
"path/filepath"
_ "modernc.org/sqlite" // registers the "sqlite" driver _ "modernc.org/sqlite" // registers the "sqlite" driver
"git.totmin.ru/en2zmax/addrkit/internal/domain/address" "git.totmin.ru/en2zmax/addrkit/internal/domain/address"
) )
// SchemaVersion identifies the on-disk format. It is independent of the data
// version (the registry snapshot date) and must be bumped whenever the key
// format or columns change. A service refuses to open a database it does not
// understand instead of silently resolving against half-empty fields.
const SchemaVersion = "2"
const schema = ` const schema = `
CREATE TABLE IF NOT EXISTS meta ( CREATE TABLE IF NOT EXISTS meta (
key TEXT PRIMARY KEY, key TEXT PRIMARY KEY,
@@ -31,8 +22,6 @@ CREATE TABLE IF NOT EXISTS address (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
address TEXT NOT NULL, address TEXT NOT NULL,
district TEXT NOT NULL DEFAULT '', district TEXT NOT NULL DEFAULT '',
city TEXT NOT NULL DEFAULT '',
city_key TEXT NOT NULL DEFAULT '',
ordinal TEXT NOT NULL DEFAULT '', ordinal TEXT NOT NULL DEFAULT '',
name_key TEXT NOT NULL DEFAULT '', name_key TEXT NOT NULL DEFAULT '',
name_display TEXT NOT NULL DEFAULT '', name_display TEXT NOT NULL DEFAULT '',
@@ -47,7 +36,6 @@ CREATE TABLE IF NOT EXISTS address (
structure TEXT NOT NULL DEFAULT '' structure TEXT NOT NULL DEFAULT ''
); );
CREATE INDEX IF NOT EXISTS idx_address_street ON address(street_key); CREATE INDEX IF NOT EXISTS idx_address_street ON address(street_key);
CREATE INDEX IF NOT EXISTS idx_address_city ON address(city_key);
` `
// Save writes entries to path atomically: it builds a temporary file and // Save writes entries to path atomically: it builds a temporary file and
@@ -56,21 +44,13 @@ CREATE INDEX IF NOT EXISTS idx_address_city ON address(city_key);
// The build uses the rollback journal (not WAL) on purpose, so that the result // The build uses the rollback journal (not WAL) on purpose, so that the result
// is a single file with no companion -wal/-shm that a rename would orphan. // is a single file with no companion -wal/-shm that a rename would orphan.
func Save(path, version string, entries []address.Entry) error { func Save(path, version string, entries []address.Entry) error {
if len(entries) == 0 {
return errors.New("refusing to publish an empty index")
}
tmp := path + ".tmp" tmp := path + ".tmp"
// A previous failed build may have left a complete temp database with the removeDBFiles(tmp)
// same meta rows; reusing it would hit a primary-key conflict.
_ = os.Remove(tmp)
removeSidecars(tmp)
uri, err := dsn(tmp, false) db, err := sql.Open("sqlite", "file:"+tmp+
if err != nil { "?_pragma=journal_mode(DELETE)"+
return err "&_pragma=synchronous(FULL)"+
} "&_pragma=busy_timeout(5000)")
db, err := sql.Open("sqlite", uri)
if err != nil { if err != nil {
return fmt.Errorf("open temp db: %w", err) return fmt.Errorf("open temp db: %w", err)
} }
@@ -88,46 +68,12 @@ func Save(path, version string, entries []address.Entry) error {
if err := os.Rename(tmp, path); err != nil { if err := os.Rename(tmp, path); err != nil {
return fmt.Errorf("publish db: %w", err) return fmt.Errorf("publish db: %w", err)
} }
// The rename is atomic, but a crash immediately after it may lose the // Drop only the temp file's journal siblings; the published base file must
// directory entry. Syncing the parent makes the publish durable. // survive.
syncDir(filepath.Dir(path))
removeSidecars(tmp) removeSidecars(tmp)
return nil return nil
} }
// dsn builds a SQLite URI. Concatenating "file:" + path breaks on paths that
// contain "?", "#" or "%", so let net/url escape the path for us.
func dsn(path string, readOnly bool) (string, error) {
abs, err := filepath.Abs(path)
if err != nil {
return "", fmt.Errorf("resolve db path: %w", err)
}
u := url.URL{Scheme: "file", Path: abs}
q := u.Query()
q.Add("_pragma", "busy_timeout(5000)")
if readOnly {
q.Set("mode", "ro")
q.Add("_pragma", "query_only(1)")
} else {
q.Add("_pragma", "journal_mode(DELETE)")
q.Add("_pragma", "synchronous(FULL)")
}
u.RawQuery = q.Encode()
return u.String(), nil
}
// syncDir fsyncs a directory so a freshly renamed file survives a crash. It is
// best effort: some filesystems do not support directory fsync, and failing to
// sync here is no worse than not trying.
func syncDir(dir string) {
f, err := os.Open(dir)
if err != nil {
return
}
defer f.Close()
_ = f.Sync()
}
func insertAll(db *sql.DB, version string, entries []address.Entry) error { func insertAll(db *sql.DB, version string, entries []address.Entry) error {
tx, err := db.Begin() tx, err := db.Begin()
if err != nil { if err != nil {
@@ -138,15 +84,11 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('version', ?)`, version); err != nil { if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('version', ?)`, version); err != nil {
return fmt.Errorf("insert meta: %w", err) return fmt.Errorf("insert meta: %w", err)
} }
if _, err := tx.Exec(`INSERT INTO meta(key, value) VALUES ('schema_version', ?)`, SchemaVersion); err != nil {
return fmt.Errorf("insert schema version: %w", err)
}
stmt, err := tx.Prepare(`INSERT INTO address ( stmt, err := tx.Prepare(`INSERT INTO address (
id, address, district, city, city_key, ordinal, name_key, name_display, id, address, district, ordinal, name_key, name_display, street_type, street_key,
street_type, street_key, house, house_frac, house_letter, building, house, house_frac, house_letter, building, building_frac, building_letter, structure
building_frac, building_letter, structure ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`)
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`)
if err != nil { if err != nil {
return fmt.Errorf("prepare insert: %w", err) return fmt.Errorf("prepare insert: %w", err)
} }
@@ -159,9 +101,8 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
id = fmt.Sprintf("row-%d", i) id = fmt.Sprintf("row-%d", i)
} }
if _, err := stmt.Exec( if _, err := stmt.Exec(
id, e.Address, e.District, e.City, e.CityKey, e.Ordinal, e.NameKey, e.NameDisplay, id, e.Address, e.District, e.Ordinal, e.NameKey, e.NameDisplay, e.StreetType, e.StreetKey,
e.StreetType, e.StreetKey, e.House, e.HouseFrac, e.HouseLetter, e.Building, e.House, e.HouseFrac, e.HouseLetter, e.Building, e.BuildingFrac, e.BuildingLetter, e.Structure,
e.BuildingFrac, e.BuildingLetter, e.Structure,
); err != nil { ); err != nil {
return fmt.Errorf("insert %q: %w", e.Address, err) return fmt.Errorf("insert %q: %w", e.Address, err)
} }
@@ -173,41 +114,26 @@ func insertAll(db *sql.DB, version string, entries []address.Entry) error {
} }
// Load opens path read-only and returns the data version and all entries. // Load opens path read-only and returns the data version and all entries.
// It fails closed when the file was produced by an incompatible schema.
func Load(path string) (version string, entries []address.Entry, err error) { func Load(path string) (version string, entries []address.Entry, err error) {
if _, err := os.Stat(path); err != nil { if _, err := os.Stat(path); err != nil {
return "", nil, fmt.Errorf("stat db: %w", err) return "", nil, fmt.Errorf("stat db: %w", err)
} }
uri, err := dsn(path, true) db, err := sql.Open("sqlite", "file:"+path+
if err != nil { "?mode=ro"+
return "", nil, err "&_pragma=query_only(1)"+
} "&_pragma=busy_timeout(5000)")
db, err := sql.Open("sqlite", uri)
if err != nil { if err != nil {
return "", nil, fmt.Errorf("open db: %w", err) return "", nil, fmt.Errorf("open db: %w", err)
} }
defer db.Close() defer db.Close()
var schemaVersion string
if err := db.QueryRow(`SELECT value FROM meta WHERE key='schema_version'`).Scan(&schemaVersion); err != nil {
return "", nil, fmt.Errorf("read schema version (re-run addrkit seed): %w", err)
}
if schemaVersion != SchemaVersion {
return "", nil, fmt.Errorf(
"unsupported index schema %q (want %q); re-run addrkit seed",
schemaVersion, SchemaVersion,
)
}
if err := db.QueryRow(`SELECT value FROM meta WHERE key='version'`).Scan(&version); err != nil { if err := db.QueryRow(`SELECT value FROM meta WHERE key='version'`).Scan(&version); err != nil {
return "", nil, fmt.Errorf("read version: %w", err) return "", nil, fmt.Errorf("read version: %w", err)
} }
rows, err := db.Query(`SELECT id, address, district, city, city_key, ordinal, rows, err := db.Query(`SELECT id, address, district, ordinal, name_key, name_display,
name_key, name_display, street_type, street_key, house, house_frac, street_type, street_key, house, house_frac, house_letter, building, building_frac,
house_letter, building, building_frac, building_letter, structure building_letter, structure FROM address`)
FROM address
ORDER BY city_key, street_key, house, house_frac, house_letter,
building, building_frac, building_letter, structure, address, id`)
if err != nil { if err != nil {
return "", nil, fmt.Errorf("query: %w", err) return "", nil, fmt.Errorf("query: %w", err)
} }
@@ -216,10 +142,9 @@ func Load(path string) (version string, entries []address.Entry, err error) {
for rows.Next() { for rows.Next() {
var e address.Entry var e address.Entry
if err := rows.Scan( if err := rows.Scan(
&e.ID, &e.Address, &e.District, &e.City, &e.CityKey, &e.Ordinal, &e.ID, &e.Address, &e.District, &e.Ordinal, &e.NameKey, &e.NameDisplay,
&e.NameKey, &e.NameDisplay, &e.StreetType, &e.StreetKey, &e.House, &e.StreetType, &e.StreetKey, &e.House, &e.HouseFrac, &e.HouseLetter,
&e.HouseFrac, &e.HouseLetter, &e.Building, &e.BuildingFrac, &e.Building, &e.BuildingFrac, &e.BuildingLetter, &e.Structure,
&e.BuildingLetter, &e.Structure,
); err != nil { ); err != nil {
return "", nil, fmt.Errorf("scan: %w", err) return "", nil, fmt.Errorf("scan: %w", err)
} }
@@ -231,6 +156,12 @@ func Load(path string) (version string, entries []address.Entry, err error) {
return version, entries, nil return version, entries, nil
} }
// removeDBFiles deletes a database file and its journal siblings.
func removeDBFiles(base string) {
_ = os.Remove(base)
removeSidecars(base)
}
// removeSidecars deletes only the -wal/-shm companions of a database file. // removeSidecars deletes only the -wal/-shm companions of a database file.
func removeSidecars(base string) { func removeSidecars(base string) {
_ = os.Remove(base + "-wal") _ = os.Remove(base + "-wal")
+2 -58
View File
@@ -1,9 +1,7 @@
package registrystore package registrystore
import ( import (
"database/sql"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address" "git.totmin.ru/en2zmax/addrkit/internal/domain/address"
@@ -13,13 +11,11 @@ func TestSaveLoadRoundTrip(t *testing.T) {
path := filepath.Join(t.TempDir(), "registry.db") path := filepath.Join(t.TempDir(), "registry.db")
addr := "Станционная улица 1/1" addr := "Станционная улица 1/1"
parser := address.NewParser(address.NewDictionary([]address.City{{Name: "Новосибирск"}}), "") p := address.Parse(addr)
p := parser.ParseInCity(addr, "Новосибирск")
entries := []address.Entry{{ entries := []address.Entry{{
ID: "a1", Address: addr, District: "Ленинский", ID: "a1", Address: addr, District: "Ленинский",
City: p.City, CityKey: p.CityKey,
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey, Ordinal: p.Ordinal, NameKey: p.NameKey, StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House, HouseFrac: p.HouseFrac, Structure: "2", House: p.House, HouseFrac: p.HouseFrac,
}} }}
if err := Save(path, "2026-10-09", entries); err != nil { if err := Save(path, "2026-10-09", entries); err != nil {
@@ -39,58 +35,6 @@ func TestSaveLoadRoundTrip(t *testing.T) {
if got[0].Address != addr || got[0].District != "Ленинский" || got[0].StreetKey != p.StreetKey { if got[0].Address != addr || got[0].District != "Ленинский" || got[0].StreetKey != p.StreetKey {
t.Errorf("round-tripped entry mismatch: %+v", got[0]) t.Errorf("round-tripped entry mismatch: %+v", got[0])
} }
if got[0].City != "Новосибирск" || got[0].CityKey != "новосибирск" || got[0].Structure != "2" {
t.Errorf("city/structure not round-tripped: %+v", got[0])
}
}
func TestSaveRejectsEmptyIndex(t *testing.T) {
path := filepath.Join(t.TempDir(), "registry.db")
if err := Save(path, "v1", nil); err == nil {
t.Fatal("expected error for empty index")
}
}
func TestLoadRejectsIncompatibleSchema(t *testing.T) {
path := filepath.Join(t.TempDir(), "old.db")
uri, err := dsn(path, false)
if err != nil {
t.Fatalf("dsn: %v", err)
}
db, err := sql.Open("sqlite", uri)
if err != nil {
t.Fatalf("open: %v", err)
}
if _, err := db.Exec(`CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)`); err != nil {
t.Fatalf("create meta: %v", err)
}
if _, err := db.Exec(`INSERT INTO meta(key, value) VALUES ('schema_version', '1')`); err != nil {
t.Fatalf("insert schema version: %v", err)
}
db.Close()
_, _, err = Load(path)
if err == nil || !strings.Contains(err.Error(), "unsupported index schema") {
t.Fatalf("Load error = %v, want unsupported schema", err)
}
}
// TestPathWithSpecialCharacters guards the SQLite URI construction: a naive
// "file:" + path breaks on ?, # and % in the path.
func TestPathWithSpecialCharacters(t *testing.T) {
path := filepath.Join(t.TempDir(), "registry ? # 100%.db")
entries := []address.Entry{{ID: "x", Address: "Первая улица 1"}}
if err := Save(path, "v1", entries); err != nil {
t.Fatalf("Save: %v", err)
}
version, got, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
if version != "v1" || len(got) != 1 {
t.Fatalf("version=%q entries=%+v", version, got)
}
} }
// TestSaveIsAtomicOverwrite verifies that re-seeding replaces the previous data // TestSaveIsAtomicOverwrite verifies that re-seeding replaces the previous data
+18 -85
View File
@@ -4,15 +4,11 @@
package config package config
import ( import (
"bytes"
"encoding/json" "encoding/json"
"fmt" "fmt"
"os" "os"
"regexp" "regexp"
"strings" "strings"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
) )
// Config is the whole service configuration. // Config is the whole service configuration.
@@ -25,24 +21,24 @@ type Config struct {
LogLevel string `json:"log_level"` LogLevel string `json:"log_level"`
Resolve ResolveConfig `json:"resolve"` Resolve ResolveConfig `json:"resolve"`
Portal PortalConfig `json:"portal"`
// Cities is the dictionary of cities this deployment understands. A
// single-city deployment lists one city and may set DefaultCity so that
// queries like "ленина 5" work without naming it.
Cities []address.City `json:"cities"`
DefaultCity string `json:"default_city"`
LLM LLMConfig `json:"llm"` LLM LLMConfig `json:"llm"`
} }
// ResolveConfig tunes the matching engine. Defaults come from // ResolveConfig tunes the matching engine.
// match.DefaultOptions so there is exactly one source of truth.
type ResolveConfig struct { type ResolveConfig struct {
FuzzyThreshold float64 `json:"fuzzy_threshold"` FuzzyThreshold float64 `json:"fuzzy_threshold"`
AmbiguityMargin float64 `json:"ambiguity_margin"` AmbiguityMargin float64 `json:"ambiguity_margin"`
MaxCandidates int `json:"max_candidates"` MaxCandidates int `json:"max_candidates"`
} }
// PortalConfig configures the external outages portal (ArcGIS).
type PortalConfig struct {
URL string `json:"url"`
TimeoutSeconds int `json:"timeout_seconds"`
Retries int `json:"retries"`
}
// LLMConfig configures the optional, disabled-by-default query rewriter. // LLMConfig configures the optional, disabled-by-default query rewriter.
type LLMConfig struct { type LLMConfig struct {
Enabled bool `json:"enabled"` Enabled bool `json:"enabled"`
@@ -66,14 +62,17 @@ type LLMConfig struct {
// Default returns a config with sane defaults for the non-required fields. // Default returns a config with sane defaults for the non-required fields.
func Default() Config { func Default() Config {
resolve := match.DefaultOptions()
return Config{ return Config{
Addr: ":6171", Addr: ":6171",
LogLevel: "info", LogLevel: "info",
Resolve: ResolveConfig{ Resolve: ResolveConfig{
FuzzyThreshold: resolve.FuzzyThreshold, FuzzyThreshold: 0.82,
AmbiguityMargin: resolve.AmbiguityMargin, AmbiguityMargin: 0.03,
MaxCandidates: resolve.MaxCandidates, MaxCandidates: 5,
},
Portal: PortalConfig{
TimeoutSeconds: 2,
Retries: 1,
}, },
LLM: LLMConfig{ LLM: LLMConfig{
TimeoutSeconds: 3, TimeoutSeconds: 3,
@@ -91,9 +90,6 @@ func Load(path string) (Config, error) {
} }
cfg := Default() cfg := Default()
dec := json.NewDecoder(strings.NewReader(expandEnv(string(raw)))) dec := json.NewDecoder(strings.NewReader(expandEnv(string(raw))))
// A typo in a key (fuzzy_treshold, data_pth, portal, ...) must not silently
// fall back to a default. Fail closed and let the operator fix the config.
dec.DisallowUnknownFields()
if err := dec.Decode(&cfg); err != nil { if err := dec.Decode(&cfg); err != nil {
return Config{}, fmt.Errorf("parse config: %w", err) return Config{}, fmt.Errorf("parse config: %w", err)
} }
@@ -110,25 +106,6 @@ func Load(path string) (Config, error) {
return cfg, nil return cfg, nil
} }
// LoadCitiesFile reads a standalone JSON array of cities. It is used by the
// seed CLI so that seeding and serving share the same city vocabulary.
func LoadCitiesFile(path string) ([]address.City, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read cities: %w", err)
}
var cities []address.City
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&cities); err != nil {
return nil, fmt.Errorf("parse cities: %w", err)
}
if err := validateCities(cities, ""); err != nil {
return nil, err
}
return cities, nil
}
func (c Config) validate() error { func (c Config) validate() error {
if c.Addr == "" { if c.Addr == "" {
return fmt.Errorf("config: addr is required") return fmt.Errorf("config: addr is required")
@@ -136,55 +113,11 @@ func (c Config) validate() error {
if c.DataPath == "" { if c.DataPath == "" {
return fmt.Errorf("config: data_path is required") return fmt.Errorf("config: data_path is required")
} }
if c.Resolve.FuzzyThreshold <= 0 || c.Resolve.FuzzyThreshold > 1 {
return fmt.Errorf("config: resolve.fuzzy_threshold must be in (0, 1]")
}
if c.Resolve.AmbiguityMargin < 0 || c.Resolve.AmbiguityMargin > 1 {
return fmt.Errorf("config: resolve.ambiguity_margin must be in [0, 1]")
}
if c.Resolve.MaxCandidates < 1 {
return fmt.Errorf("config: resolve.max_candidates must be at least 1")
}
if c.LLM.Enabled && c.LLM.URL == "" { if c.LLM.Enabled && c.LLM.URL == "" {
return fmt.Errorf("config: llm.url is required when llm.enabled is true") return fmt.Errorf("config: llm.url is required when llm.enabled is true")
} }
return validateCities(c.Cities, c.DefaultCity) if c.Portal.URL == "" {
} return fmt.Errorf("config: portal.url is required")
// validateCities checks the city dictionary for the mistakes that would make
// matching silently ambiguous: empty names, duplicate cities and an alias
// claimed by two different cities.
func validateCities(cities []address.City, defaultCity string) error {
seenNames := make(map[string]string)
seenAliases := make(map[string]string)
for _, city := range cities {
name := strings.TrimSpace(city.Name)
key := address.NormalizeText(name)
if key == "" {
return fmt.Errorf("config: cities[].name must not be empty")
}
if prev, ok := seenNames[key]; ok {
return fmt.Errorf("config: duplicate city %q and %q", prev, name)
}
seenNames[key] = name
aliases := append([]string{name}, city.Aliases...)
for _, alias := range aliases {
norm := address.NormalizeText(alias)
if norm == "" {
continue
}
if owner, ok := seenAliases[norm]; ok && owner != key {
return fmt.Errorf("config: alias %q is claimed by both %q and %q", alias, owner, key)
}
seenAliases[norm] = key
}
}
if defaultCity = strings.TrimSpace(defaultCity); defaultCity != "" && len(cities) > 0 {
if _, ok := seenNames[address.NormalizeText(defaultCity)]; !ok {
return fmt.Errorf("config: default_city %q is not listed in cities", defaultCity)
}
} }
return nil return nil
} }
+3 -84
View File
@@ -4,8 +4,6 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
) )
func writeFile(t *testing.T, dir, name, content string) string { func writeFile(t *testing.T, dir, name, content string) string {
@@ -21,6 +19,7 @@ func baseConfig(promptFields string) string {
return `{ return `{
"addr": ":6171", "addr": ":6171",
"data_path": "registry.db", "data_path": "registry.db",
"portal": { "url": "http://portal" },
"llm": { "enabled": true, "url": "http://llm"` + promptFields + ` } "llm": { "enabled": true, "url": "http://llm"` + promptFields + ` }
}` }`
} }
@@ -61,90 +60,10 @@ func TestLoadMissingSystemPromptFileFailsClosed(t *testing.T) {
} }
} }
func TestDefaultsMatchMatcher(t *testing.T) { func TestLoadRequiresPortalURL(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
path := writeFile(t, dir, "config.json", `{"addr": ":6171", "data_path": "x.db"}`) path := writeFile(t, dir, "config.json", `{"addr": ":6171", "data_path": "x.db"}`)
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
want := match.DefaultOptions()
if cfg.Resolve.FuzzyThreshold != want.FuzzyThreshold ||
cfg.Resolve.AmbiguityMargin != want.AmbiguityMargin ||
cfg.Resolve.MaxCandidates != want.MaxCandidates {
t.Fatalf("config defaults %+v do not match matcher defaults %+v", cfg.Resolve, want)
}
}
func TestRejectsBadResolveRanges(t *testing.T) {
dir := t.TempDir()
cases := map[string]string{
"threshold": `{"addr":":6171","data_path":"x.db","resolve":{"fuzzy_threshold":0}}`,
"margin": `{"addr":":6171","data_path":"x.db","resolve":{"ambiguity_margin":2}}`,
"candidates": `{"addr":":6171","data_path":"x.db","resolve":{"max_candidates":0}}`,
}
for name, body := range cases {
path := writeFile(t, dir, name+".json", body)
if _, err := Load(path); err == nil { if _, err := Load(path); err == nil {
t.Errorf("%s: expected validation error", name) t.Fatal("expected error for missing portal.url")
}
}
}
func TestCitiesAndDefaultCity(t *testing.T) {
dir := t.TempDir()
path := writeFile(t, dir, "config.json", `{
"addr": ":6171", "data_path": "x.db",
"cities": [
{"name": "Новосибирск", "aliases": ["нск", "новосибирская область"]},
{"name": "Бердск"}
],
"default_city": "Новосибирск"
}`)
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
if len(cfg.Cities) != 2 || cfg.DefaultCity != "Новосибирск" {
t.Fatalf("cities = %+v, default = %q", cfg.Cities, cfg.DefaultCity)
}
}
func TestRejectsUnknownDefaultCity(t *testing.T) {
dir := t.TempDir()
path := writeFile(t, dir, "config.json", `{
"addr": ":6171", "data_path": "x.db",
"cities": [{"name": "Бердск"}],
"default_city": "Новосибирск"
}`)
if _, err := Load(path); err == nil {
t.Fatal("expected error for default_city absent from cities")
}
}
func TestLoadCitiesFile(t *testing.T) {
dir := t.TempDir()
path := writeFile(t, dir, "cities.json", `[
{"name": "Новосибирск", "aliases": ["нск"]},
{"name": "Бердск"}
]`)
cities, err := LoadCitiesFile(path)
if err != nil {
t.Fatalf("LoadCitiesFile: %v", err)
}
if len(cities) != 2 || cities[0].Name != "Новосибирск" {
t.Fatalf("cities = %+v", cities)
}
}
func TestRejectsUnknownConfigFields(t *testing.T) {
dir := t.TempDir()
// "portal" was removed with the events module; silently ignoring it would
// hide a stale deployment config.
path := writeFile(t, dir, "config.json",
`{"addr":":6171","data_path":"x.db","portal":{"url":"http://example"}}`)
if _, err := Load(path); err == nil {
t.Fatal("expected error for unknown config field")
} }
} }
-141
View File
@@ -1,141 +0,0 @@
package address
import "testing"
func testDictionary() *Dictionary {
return NewDictionary([]City{
{Name: "Новосибирск", Aliases: []string{"нск", "новосиб", "новосибирская область", "нсо"}},
{Name: "Бердск"},
})
}
func TestParseCityFromAliases(t *testing.T) {
parser := NewParser(testDictionary(), "")
tests := []struct {
raw string
city string
nameKey string
}{
{"нск, 2-я Станционная улица 1", "Новосибирск", "станционная"},
{"НСК Ленина 5", "Новосибирск", "ленина"},
{"Новосибирская область, Станционная 1", "Новосибирск", "станционная"},
{"Бердск, Ленина 5", "Бердск", "ленина"},
{"г. Новосибирск, ул. Ленина 5", "Новосибирск", "ленина"},
}
for _, tt := range tests {
got := parser.Parse(tt.raw)
if got.City != tt.city {
t.Errorf("%q: City = %q, want %q", tt.raw, got.City, tt.city)
}
if got.NameKey != tt.nameKey {
t.Errorf("%q: NameKey = %q, want %q", tt.raw, got.NameKey, tt.nameKey)
}
}
}
func TestParseCityIsPartOfStreetKey(t *testing.T) {
parser := NewParser(testDictionary(), "")
nsk := parser.Parse("Новосибирск, Ленина 5")
berdsk := parser.Parse("Бердск, Ленина 5")
if nsk.StreetKey == berdsk.StreetKey {
t.Fatalf("same street key for different cities: %q", nsk.StreetKey)
}
if nsk.CityKey != "новосибирск" || berdsk.CityKey != "бердск" {
t.Fatalf("city keys = %q / %q", nsk.CityKey, berdsk.CityKey)
}
}
func TestParseDefaultCity(t *testing.T) {
parser := NewParser(testDictionary(), "Новосибирск")
got := parser.Parse("Ленина 5")
if got.City != "Новосибирск" || got.CityKey != "новосибирск" {
t.Fatalf("City = %q / %q", got.City, got.CityKey)
}
// The default city is also recognised when the user spells it out, even
// if the dictionary contains only aliases.
explicit := NewParser(DefaultDictionary(), "Новосибирск").Parse("г. Новосибирск, Ленина 5")
if explicit.City != "Новосибирск" || explicit.NameKey != "ленина" {
t.Fatalf("explicit default city: %+v", explicit)
}
}
func TestParseInCityOverridesDetectedCity(t *testing.T) {
parser := NewParser(testDictionary(), "")
got := parser.ParseInCity("Бердск, Ленина 5", "Новосибирск")
if got.City != "Новосибирск" {
t.Fatalf("City = %q, want forced Новосибирск", got.City)
}
if got.NameKey != "ленина" {
t.Fatalf("NameKey = %q", got.NameKey)
}
}
func TestParseWithoutCityKeepsCityKeyEmpty(t *testing.T) {
parser := NewParser(testDictionary(), "")
got := parser.Parse("Ленина 5")
if got.City != "" || got.CityKey != "" {
t.Fatalf("City = %q / %q, want empty", got.City, got.CityKey)
}
}
func TestNormalizeTextStillLeavesUnknownCityTokens(t *testing.T) {
// A city that is not in the dictionary must not be silently dropped: the
// resolver should fail rather than guess another city.
parser := NewParser(testDictionary(), "")
got := parser.Parse("Томск, Ленина 5")
if got.City != "" {
t.Fatalf("City = %q, want empty for unknown city", got.City)
}
if got.NameKey == "ленина" {
t.Fatalf("unknown city token disappeared from the street name")
}
}
func TestMatchCity(t *testing.T) {
dict := testDictionary()
cases := []struct {
query string
want string
ok bool
}{
{"нск", "Новосибирск", true},
{"г. Новосибирск", "Новосибирск", true},
{"Новосибирская область", "Новосибирск", true},
{"Бердск", "Бердск", true},
{"Бердск, Ленина 5", "", false},
{"Томск", "", false},
}
for _, tt := range cases {
got, ok := dict.MatchCity(tt.query)
if ok != tt.ok || got.Name != tt.want {
t.Errorf("MatchCity(%q) = %+v, %v; want %q, %v", tt.query, got, ok, tt.want, tt.ok)
}
}
}
func FuzzParse(f *testing.F) {
for _, seed := range []string{
"",
"2-я Станционная улица 1/1 стр.",
"нск, Ленина 5",
"Бердск, Ленина 5 к2",
"г. Новосибирск, ул. Кошурникова, д. 11/1 кв. 5",
"\x00\xff",
} {
f.Add(seed)
}
parser := NewParser(testDictionary(), "")
f.Fuzz(func(t *testing.T, raw string) {
first := parser.Parse(raw)
second := parser.Parse(raw)
if first.StreetKey != second.StreetKey {
t.Fatalf("non-deterministic street key for %q: %q vs %q", raw, first.StreetKey, second.StreetKey)
}
if NormalizeText(raw) != NormalizeText(NormalizeText(raw)) {
t.Fatalf("NormalizeText is not idempotent for %q", raw)
}
})
}
-284
View File
@@ -1,284 +0,0 @@
package address
import (
"sort"
"strings"
)
// City is one entry of the configurable city dictionary.
//
// Aliases are additional surface forms that refer to the city: common
// abbreviations ("нск"), the region name ("новосибирская область") and so on.
// The canonical Name is always accepted as an alias too. Matching is
// token-based, case-insensitive and "ё"-insensitive, so aliases should be
// written naturally.
type City struct {
Name string `json:"name"`
Aliases []string `json:"aliases,omitempty"`
}
// Dictionary is the immutable, deployment-specific vocabulary of the parser:
// cities, filler words, street types and ordinals.
//
// Keeping this data in a value instead of package-level maps makes the parser
// reusable across deployments: a service in Novosibirsk and a service in Tomsk
// can run the same code with different city lists, without recompilation.
type Dictionary struct {
cities []cityAlias
filler map[string]struct{}
streetTypes map[string]string
ordinalWords map[string]string
ordinalSuffixes map[string]struct{}
}
// cityAlias is one token sequence that resolves to a city.
type cityAlias struct {
tokens []string
name string
key string
}
// NewDictionary builds a dictionary with the built-in federal vocabulary and
// the supplied cities. The input slice is copied; mutating it later does not
// affect the dictionary.
func NewDictionary(cities []City) *Dictionary {
d := &Dictionary{
filler: cloneSet(defaultFillerWords),
streetTypes: cloneMap(defaultStreetTypes),
ordinalWords: cloneMap(defaultOrdinalWords),
ordinalSuffixes: cloneSet(defaultOrdinalSuffixes),
}
seen := make(map[string]struct{})
for _, c := range cities {
name := strings.TrimSpace(c.Name)
cityKey := NormalizeText(name)
if name == "" || cityKey == "" {
continue
}
aliases := append([]string{name}, c.Aliases...)
for _, alias := range aliases {
tokens := strings.Fields(NormalizeText(alias))
if len(tokens) == 0 {
continue
}
dedup := cityKey + "|" + strings.Join(tokens, " ")
if _, ok := seen[dedup]; ok {
continue
}
seen[dedup] = struct{}{}
d.cities = append(d.cities, cityAlias{
tokens: tokens,
name: name,
key: cityKey,
})
}
}
// Longest aliases win, so "новосибирская область" is preferred over a
// hypothetical shorter "новосибирская". The key is the tie-breaker that
// keeps the order deterministic when two cities share an alias.
d.sortCities()
return d
}
// sortCities orders aliases by specificity: more tokens first, then longer
// text, then key. The order decides which city wins when aliases overlap.
func (d *Dictionary) sortCities() {
sort.SliceStable(d.cities, func(i, j int) bool {
a, b := d.cities[i], d.cities[j]
if len(a.tokens) != len(b.tokens) {
return len(a.tokens) > len(b.tokens)
}
as, bs := strings.Join(a.tokens, " "), strings.Join(b.tokens, " ")
if len(as) != len(bs) {
return len(as) > len(bs)
}
if a.key != b.key {
return a.key < b.key
}
return as < bs
})
}
// withDefaultCity returns a dictionary that also recognises name as a city.
// It lets a single-city deployment use "-city Новосибирск" (or default_city)
// without duplicating the canonical name in the aliases list.
func (d *Dictionary) withDefaultCity(name string) *Dictionary {
key := NormalizeText(name)
if key == "" {
return d
}
for _, a := range d.cities {
if a.key == key {
return d
}
}
out := *d
out.cities = append(append([]cityAlias(nil), d.cities...), cityAlias{
tokens: strings.Fields(key),
name: name,
key: key,
})
out.sortCities()
return &out
}
// DefaultDictionary returns the built-in vocabulary without cities. It is the
// parser used by address.Parse for callers that do not need city awareness.
func DefaultDictionary() *Dictionary {
return NewDictionary(nil)
}
// MatchCity reports whether query consists only of a city name/alias (plus
// generic filler such as "г.") and returns the canonical city. It is the
// city-only counterpart of Parse, useful for projects that normalize city
// names without a street.
func (d *Dictionary) MatchCity(query string) (City, bool) {
name, _, rest, ok := d.matchCity(strings.Fields(NormalizeText(query)))
if !ok {
return City{}, false
}
for _, tok := range rest {
if !d.isFiller(tok) {
return City{}, false
}
}
return City{Name: name}, true
}
// matchCity finds the longest city alias in tokens and returns the city plus
// the tokens with the alias removed. Matching is contiguous on token
// boundaries, so "нск" never matches inside another word.
func (d *Dictionary) matchCity(tokens []string) (name, key string, rest []string, ok bool) {
for _, alias := range d.cities {
if idx := indexTokens(tokens, alias.tokens); idx >= 0 {
rest = make([]string, 0, len(tokens)-len(alias.tokens))
rest = append(rest, tokens[:idx]...)
rest = append(rest, tokens[idx+len(alias.tokens):]...)
return alias.name, alias.key, rest, true
}
}
return "", "", tokens, false
}
// indexTokens returns the start of the first contiguous occurrence of want in
// tokens, or -1.
func indexTokens(tokens, want []string) int {
if len(want) == 0 || len(want) > len(tokens) {
return -1
}
for i := 0; i+len(want) <= len(tokens); i++ {
matched := true
for j := range want {
if tokens[i+j] != want[j] {
matched = false
break
}
}
if matched {
return i
}
}
return -1
}
func (d *Dictionary) isFiller(tok string) bool {
_, ok := d.filler[tok]
return ok
}
func (d *Dictionary) streetType(tok string) (string, bool) {
t, ok := d.streetTypes[tok]
return t, ok
}
func (d *Dictionary) ordinalWord(tok string) (string, bool) {
o, ok := d.ordinalWords[tok]
return o, ok
}
func (d *Dictionary) isOrdinalSuffix(tok string) bool {
_, ok := d.ordinalSuffixes[tok]
return ok
}
func cloneSet(src map[string]bool) map[string]struct{} {
dst := make(map[string]struct{}, len(src))
for k, v := range src {
if v {
dst[k] = struct{}{}
}
}
return dst
}
func cloneMap(src map[string]string) map[string]string {
dst := make(map[string]string, len(src))
for k, v := range src {
dst[k] = v
}
return dst
}
// defaultFillerWords are tokens that carry no street-name meaning and that
// people routinely include ("г. Новосибирск", "улица ... дом 5", "... область").
//
// City names are deliberately NOT listed here: they come from Dictionary, so
// the same code can serve any city (or many cities) without edits.
var defaultFillerWords = map[string]bool{
"город": true, "г": true,
"область": true, "обл": true,
"район": true, "дом": true, "д": true,
}
// defaultStreetTypes maps a surface form (including common abbreviations) to a
// single canonical street type. The type is informational for matching (we
// strip it from the key) but useful for disambiguation and display.
var defaultStreetTypes = map[string]string{
"ул": "улица", "улица": "улица",
"пер": "переулок", "пер-к": "переулок", "переулок": "переулок",
"пр": "проезд", "проезд": "проезд",
"пр-т": "проспект", "просп": "проспект", "проспект": "проспект",
"ш": "шоссе", "шоссе": "шоссе",
"мкр": "микрорайон", "микрорайон": "микрорайон",
"туп": "тупик", "тупик": "тупик",
"спуск": "спуск",
"б-р": "бульвар", "бул": "бульвар", "бульвар": "бульвар",
"пл": "площадь", "площадь": "площадь",
"пос": "поселок", "поселок": "поселок",
"тракт": "тракт",
"аллея": "аллея",
"наб": "набережная", "набережная": "набережная",
"кольцо": "кольцо",
"просека": "просека",
"квартал": "квартал",
"территория": "территория",
"дорога": "дорога",
"разъезд": "разъезд",
}
// defaultOrdinalWords maps spelled-out ordinals to their digit form. Only the
// forms that actually occur in practice are listed; unknown words are treated
// as part of the street name.
var defaultOrdinalWords = map[string]string{
"первый": "1", "первая": "1", "первое": "1",
"второй": "2", "вторая": "2", "второе": "2",
"третий": "3", "третья": "3", "третье": "3",
"четвертый": "4", "четвертая": "4", "четвертое": "4",
"пятый": "5", "пятая": "5", "пятое": "5",
"шестой": "6", "шестая": "6", "шестое": "6",
"седьмой": "7", "седьмая": "7", "седьмое": "7",
"восьмой": "8", "восьмая": "8", "восьмое": "8",
"девятый": "9", "девятая": "9", "девятое": "9",
"десятый": "10", "десятая": "10", "десятое": "10",
}
// defaultOrdinalSuffixes are the gender/number endings that may follow an
// ordinal digit when it is written as a separate token ("2 я", "10 й").
var defaultOrdinalSuffixes = map[string]bool{
"я": true, "й": true, "е": true,
"ая": true, "ый": true, "ой": true, "ое": true,
"ий": true, "ья": true, "ье": true, "ьи": true,
}
-14
View File
@@ -1,7 +1,6 @@
package address package address
import ( import (
"regexp"
"strings" "strings"
"unicode" "unicode"
@@ -13,11 +12,6 @@ import (
// alike, which is what we want for matching. // alike, which is what we want for matching.
var fold = cases.Fold() var fold = cases.Fold()
// reTrailingParenthetical matches a trailing "(...)" group, e.g. the district
// suffix that source systems append: "Медкадры улица, 1 (Заельцовский
// район)". Such administrative noise must not be treated as part of the house.
var reTrailingParenthetical = regexp.MustCompile(`\s*\([^)]*\)\s*$`)
// NormalizeText folds a free-form string into a canonical form suitable for // NormalizeText folds a free-form string into a canonical form suitable for
// parsing and matching. // parsing and matching.
// //
@@ -34,19 +28,11 @@ func NormalizeText(s string) string {
s = norm.NFKC.String(s) s = norm.NFKC.String(s)
s = fold.String(s) s = fold.String(s)
s = strings.ReplaceAll(s, "ё", "е") s = strings.ReplaceAll(s, "ё", "е")
s = stripTrailingParenthetical(s)
s = mapSeparators(s) s = mapSeparators(s)
s = strings.Join(strings.Fields(s), " ") s = strings.Join(strings.Fields(s), " ")
return strings.TrimSpace(s) return strings.TrimSpace(s)
} }
// stripTrailingParenthetical removes a single trailing "(...)" group. It runs
// before separator mapping because that step turns parentheses into spaces and
// the group would no longer be recognisable.
func stripTrailingParenthetical(s string) string {
return strings.TrimSpace(reTrailingParenthetical.ReplaceAllString(s, ""))
}
// mapSeparators unifies characters that carry no semantic meaning for address // mapSeparators unifies characters that carry no semantic meaning for address
// matching. It never removes digits, letters or the '/' house separator. // matching. It never removes digits, letters or the '/' house separator.
func mapSeparators(s string) string { func mapSeparators(s string) string {
+33 -112
View File
@@ -24,121 +24,42 @@ var (
// Trailing apartment part: " кв 5", " квартира 12", " кв. 7а". Removed // Trailing apartment part: " кв 5", " квартира 12", " кв. 7а". Removed
// before house parsing so it cannot leak into the street name. // before house parsing so it cannot leak into the street name.
reApartment = regexp.MustCompile(`(?:^|\s)(?:квартира|кв)\.?(?:\s+\d+[а-я]?)?\s*$`) reApartment = regexp.MustCompile(`(?:^|\s)(?:квартира|кв)\.?(?:\s+\d+[а-я]?)?\s*$`)
// Spoken fractions: "18 дробь 1" is the same house unit as "18/1".
reFractionWord = regexp.MustCompile(`\s+(?:дробь|дробью)\s+`)
) )
// Parser turns free-form address strings into structured addresses using a
// deployment-specific Dictionary. It is immutable and safe for concurrent use.
type Parser struct {
dict *Dictionary
defaultCityName string
defaultCityKey string
}
// NewParser builds a parser. defaultCity is used when the query does not name a
// city and the dictionary has no match; pass an empty string for deployments
// where the city must be present in every query.
func NewParser(dict *Dictionary, defaultCity string) *Parser {
if dict == nil {
dict = DefaultDictionary()
}
p := &Parser{}
if name := strings.TrimSpace(defaultCity); name != "" {
dict = dict.withDefaultCity(name)
p.defaultCityName = name
p.defaultCityKey = NormalizeText(name)
}
p.dict = dict
return p
}
// defaultParser is used by the package-level Parse convenience function. It has
// no cities and is therefore fully backward compatible: callers that need city
// awareness construct their own Parser.
var defaultParser = NewParser(DefaultDictionary(), "")
// DefaultParser returns the shared city-less parser.
func DefaultParser() *Parser {
return defaultParser
}
// Parse decomposes a free-form address with the default (city-less) parser.
func Parse(raw string) ParsedAddress {
return defaultParser.Parse(raw)
}
// Parse decomposes a free-form address into its structured components. It is // Parse decomposes a free-form address into its structured components. It is
// total: unparseable input still yields a usable ParsedAddress whose street // total: unparseable input still yields a non-nil ParsedAddress whose street
// part is best-effort and whose HasHouse is false. // part is best-effort and whose HasHouse is false.
func (p *Parser) Parse(raw string) ParsedAddress { func Parse(raw string) ParsedAddress {
return p.parse(raw, "")
}
// ParseInCity parses raw and forces cityName as the result's city. It is used
// by the seed pipeline, where the city is an explicit field of the source row
// and must win over whatever the address text contains.
func (p *Parser) ParseInCity(raw, cityName string) ParsedAddress {
return p.parse(raw, cityName)
}
// MatchCity reports whether query is a city name/alias and returns its
// canonical form. It does not guess a city from an arbitrary address.
func (p *Parser) MatchCity(query string) (City, bool) {
return p.dict.MatchCity(query)
}
func (p *Parser) parse(raw, forcedCity string) ParsedAddress {
normalized := NormalizeText(raw) normalized := NormalizeText(raw)
// A trailing apartment ("кв 5") is not part of the address identity. // A trailing apartment ("кв 5") is not part of the address identity.
normalized = strings.TrimSpace(reApartment.ReplaceAllString(normalized, "")) normalized = strings.TrimSpace(reApartment.ReplaceAllString(normalized, ""))
// Collapse the spoken fraction marker to the canonical separator before p := ParsedAddress{Raw: raw, Normalized: normalized}
// the right-to-left house parser runs: "18 дробь 1" -> "18/1".
normalized = reFractionWord.ReplaceAllString(normalized, "/")
parsed := ParsedAddress{Raw: raw, Normalized: normalized}
rest := normalized rest := normalized
if cityName, cityKey, restTokens, ok := p.dict.matchCity(strings.Fields(rest)); ok {
parsed.City = cityName
parsed.CityKey = cityKey
rest = strings.Join(restTokens, " ")
}
if forcedCity = strings.TrimSpace(forcedCity); forcedCity != "" {
parsed.City = forcedCity
parsed.CityKey = NormalizeText(forcedCity)
} else if parsed.City == "" && p.defaultCityName != "" {
// A single-city deployment can omit the city in queries. In a
// multi-city deployment default_city should stay empty: guessing a
// city is exactly the kind of silent error this service avoids.
parsed.City = p.defaultCityName
parsed.CityKey = p.defaultCityKey
}
if g, start, ok := matchSuffix(reStructure, rest); ok { if g, start, ok := matchSuffix(reStructure, rest); ok {
parsed.Structure = trimLeadingZeros(g[1]) p.Structure = trimLeadingZeros(g[1])
rest = strings.TrimSpace(rest[:start]) rest = strings.TrimSpace(rest[:start])
} }
if g, start, ok := matchSuffix(reBuilding, rest); ok { if g, start, ok := matchSuffix(reBuilding, rest); ok {
// The regex also matches a bare "к"; only accept it when a digit or a // The regex also matches a bare "к"; only accept it when a digit or a
// letter actually follows, otherwise it is not a corpus marker. // letter actually follows, otherwise it is not a corpus marker.
if g[1] != "" || g[2] != "" { if g[1] != "" || g[2] != "" {
parsed.Building = trimLeadingZeros(g[1]) p.Building = trimLeadingZeros(g[1])
parsed.BuildingLetter = firstNonEmpty(g[2], g[4]) p.BuildingLetter = firstNonEmpty(g[2], g[4])
parsed.BuildingFrac = trimLeadingZeros(g[3]) p.BuildingFrac = trimLeadingZeros(g[3])
rest = strings.TrimSpace(rest[:start]) rest = strings.TrimSpace(rest[:start])
} }
} }
if g, start, ok := matchSuffix(reHouse, rest); ok { if g, start, ok := matchSuffix(reHouse, rest); ok {
parsed.House = trimLeadingZeros(g[1]) p.House = trimLeadingZeros(g[1])
parsed.HouseLetter = firstNonEmpty(g[2], g[4]) p.HouseLetter = firstNonEmpty(g[2], g[4])
parsed.HouseFrac = trimLeadingZeros(g[3]) p.HouseFrac = trimLeadingZeros(g[3])
parsed.HasHouse = true p.HasHouse = true
rest = strings.TrimSpace(rest[:start]) rest = strings.TrimSpace(rest[:start])
} }
p.parseStreet(&parsed, rest) parseStreet(&p, rest)
return parsed return p
} }
// firstNonEmpty returns the first non-empty argument. House/building letters // firstNonEmpty returns the first non-empty argument. House/building letters
@@ -152,8 +73,8 @@ func firstNonEmpty(values ...string) string {
return "" return ""
} }
// parseStreet fills the street-related fields of out from the leftover string // parseStreet fills the street-related fields of p from the leftover string
// (the address with its city and house block removed). // (the address with its house block removed).
// //
// All recognised type tokens are removed from the name. This matters for // All recognised type tokens are removed from the name. This matters for
// registry rows such as "Гусинобродский Тракт улица 21", where both "Тракт" // registry rows such as "Гусинобродский Тракт улица 21", where both "Тракт"
@@ -161,7 +82,7 @@ func firstNonEmpty(values ...string) string {
// says "Гусинобродский 21". The first type token becomes the canonical type. // says "Гусинобродский 21". The first type token becomes the canonical type.
// If removing them would leave the name empty (a street literally called // If removing them would leave the name empty (a street literally called
// "Набережная"), the type token is kept as the name. // "Набережная"), the type token is kept as the name.
func (p *Parser) parseStreet(out *ParsedAddress, s string) { func parseStreet(p *ParsedAddress, s string) {
tokens := strings.Fields(s) tokens := strings.Fields(s)
name := make([]string, 0, len(tokens)) name := make([]string, 0, len(tokens))
streetType := "" streetType := ""
@@ -169,10 +90,10 @@ func (p *Parser) parseStreet(out *ParsedAddress, s string) {
for _, tok := range tokens { for _, tok := range tokens {
// Filler tokens ("г", "город", "дом", "область", ...) never belong to // Filler tokens ("г", "город", "дом", "область", ...) never belong to
// the street name. // the street name.
if p.dict.isFiller(tok) { if fillerWords[tok] {
continue continue
} }
if t, ok := p.dict.streetType(tok); ok { if t, ok := streetTypes[tok]; ok {
if streetType == "" { if streetType == "" {
streetType = t streetType = t
} }
@@ -181,55 +102,55 @@ func (p *Parser) parseStreet(out *ParsedAddress, s string) {
name = append(name, tok) name = append(name, tok)
} }
out.StreetType = streetType p.StreetType = streetType
if len(name) == 0 && streetType != "" { if len(name) == 0 && streetType != "" {
name = []string{streetType} name = []string{streetType}
} }
if len(name) == 0 { if len(name) == 0 {
out.StreetKey = streetKey(out.CityKey, "", "") p.StreetKey = streetKey("", "")
return return
} }
name = p.extractOrdinal(out, name) name = extractOrdinal(p, name)
out.NameDisplay = strings.Join(name, " ") p.NameDisplay = strings.Join(name, " ")
sorted := append([]string(nil), name...) sorted := append([]string(nil), name...)
sort.Strings(sorted) sort.Strings(sorted)
out.NameKey = strings.Join(sorted, " ") p.NameKey = strings.Join(sorted, " ")
out.StreetKey = streetKey(out.CityKey, out.Ordinal, out.NameKey) p.StreetKey = streetKey(p.Ordinal, p.NameKey)
} }
// extractOrdinal recognises an ordinal and removes it from the name tokens. // extractOrdinal recognises an ordinal and removes it from the name tokens.
// It handles both the usual leading form ("2-я Сухарная") and an ordinal that // It handles both the usual leading form ("2-я Сухарная") and an ordinal that
// appears after the name ("Сухарная 2-я улица"), which both people and models // appears after the name ("Сухарная 2-я улица"), which both people and models
// produce. // produce.
func (p *Parser) extractOrdinal(out *ParsedAddress, tokens []string) []string { func extractOrdinal(p *ParsedAddress, tokens []string) []string {
if len(tokens) == 0 { if len(tokens) == 0 {
return tokens return tokens
} }
// Leading forms first: they are by far the most common. // Leading forms first: they are by far the most common.
if m := reOrdinalToken.FindStringSubmatch(tokens[0]); m != nil { if m := reOrdinalToken.FindStringSubmatch(tokens[0]); m != nil {
out.Ordinal = normalizeOrdinal(m[1]) p.Ordinal = normalizeOrdinal(m[1])
return tokens[1:] return tokens[1:]
} }
if isDigits(tokens[0]) && len(tokens) >= 2 && p.dict.isOrdinalSuffix(tokens[1]) { if isDigits(tokens[0]) && len(tokens) >= 2 && ordinalSuffixes[tokens[1]] {
out.Ordinal = normalizeOrdinal(tokens[0]) p.Ordinal = normalizeOrdinal(tokens[0])
return tokens[2:] return tokens[2:]
} }
if d, ok := p.dict.ordinalWord(tokens[0]); ok { if d, ok := ordinalWords[tokens[0]]; ok {
out.Ordinal = normalizeOrdinal(d) p.Ordinal = normalizeOrdinal(d)
return tokens[1:] return tokens[1:]
} }
// Otherwise look for an ordinal anywhere among the name tokens. // Otherwise look for an ordinal anywhere among the name tokens.
for i, tok := range tokens { for i, tok := range tokens {
if m := reOrdinalToken.FindStringSubmatch(tok); m != nil { if m := reOrdinalToken.FindStringSubmatch(tok); m != nil {
out.Ordinal = normalizeOrdinal(m[1]) p.Ordinal = normalizeOrdinal(m[1])
return removeRange(tokens, i, i+1) return removeRange(tokens, i, i+1)
} }
if isDigits(tok) && i+1 < len(tokens) && p.dict.isOrdinalSuffix(tokens[i+1]) { if isDigits(tok) && i+1 < len(tokens) && ordinalSuffixes[tokens[i+1]] {
out.Ordinal = normalizeOrdinal(tok) p.Ordinal = normalizeOrdinal(tok)
return removeRange(tokens, i, i+2) return removeRange(tokens, i, i+2)
} }
} }
-13
View File
@@ -71,12 +71,6 @@ func TestParseHouseBlock(t *testing.T) {
wantName: "выборная", wantType: "улица", wantName: "выборная", wantType: "улица",
wantHouse: "45", wantLetter: "а", wantFrac: "4", wantHas: true, wantHouse: "45", wantLetter: "а", wantFrac: "4", wantHas: true,
}, },
{
name: "spoken fraction word",
raw: "Горская улица 18 дробь 1",
wantName: "горская", wantType: "улица",
wantHouse: "18", wantFrac: "1", wantHas: true,
},
{ {
name: "corpus letter without digit", name: "corpus letter without digit",
raw: "1-я Ельцовка улица 1 кД", raw: "1-я Ельцовка улица 1 кД",
@@ -101,12 +95,6 @@ func TestParseHouseBlock(t *testing.T) {
wantName: "кошурникова", wantType: "улица", wantName: "кошурникова", wantType: "улица",
wantHouse: "11", wantFrac: "1", wantHas: true, wantHouse: "11", wantFrac: "1", wantHas: true,
}, },
{
name: "district suffix in parentheses",
raw: "Медкадры улица, 1 (Заельцовский район)",
wantName: "медкадры", wantType: "улица",
wantHouse: "1", wantHas: true,
},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -166,7 +154,6 @@ func TestNormalizeIdempotent(t *testing.T) {
" 2-Я Станционная\u00a0улица 1/1 стр. ", " 2-Я Станционная\u00a0улица 1/1 стр. ",
"ЁЛКИ д. 5, кв. 3", "ЁЛКИ д. 5, кв. 3",
"СТАНИЦИОННАЯ — улица, 30а", "СТАНИЦИОННАЯ — улица, 30а",
"Медкадры улица, 1 (Заельцовский район)",
"", "",
} }
for _, in := range inputs { for _, in := range inputs {
+67 -13
View File
@@ -2,19 +2,73 @@ package address
import "strings" import "strings"
// streetKey builds the canonical, city-aware, type-independent street key: // fillerWords are tokens that carry no street-name meaning and that people
// // routinely include ("г. Новосибирск", "улица ... дом 5", "... область").
// "<cityKey>|<ordinal>|<sorted name tokens>" // Dropping them makes the service tolerant of natural phrasing. District
// // adjectives are deliberately NOT dropped: they can legitimately be part of a
// The city is part of the key so that "Ленина" in Novosibirsk and "Ленина" in // street name.
// Berdsk are different streets. The type word is dropped so that a user who var fillerWords = map[string]bool{
// omits it ("станционная 1" instead of "Станционная улица 1") still matches; "новосибирск": true, "нск": true, "нсо": true,
// type confusion is surfaced later as ambiguity. "город": true, "г": true,
// "область": true, "обл": true,
// None of the components may contain "|": NormalizeText folds all punctuation "район": true, "дом": true, "д": true,
// that could appear in a name into spaces, and ordinals are digits only. }
func streetKey(cityKey, ordinal, nameKey string) string {
return cityKey + "|" + ordinal + "|" + nameKey // streetTypes maps a surface form (including common abbreviations) to a single
// canonical street type. The type is informational for matching (we strip it
// from the key) but useful for disambiguation and display.
var streetTypes = map[string]string{
"ул": "улица", "улица": "улица",
"пер": "переулок", "пер-к": "переулок", "переулок": "переулок",
"пр": "проезд", "проезд": "проезд",
"пр-т": "проспект", "просп": "проспект", "проспект": "проспект",
"ш": "шоссе", "шоссе": "шоссе",
"мкр": "микрорайон", "микрорайон": "микрорайон",
"туп": "тупик", "тупик": "тупик",
"спуск": "спуск",
"б-р": "бульвар", "бул": "бульвар", "бульвар": "бульвар",
"пл": "площадь", "площадь": "площадь",
"пос": "поселок", "поселок": "поселок",
"тракт": "тракт",
"аллея": "аллея",
"наб": "набережная", "набережная": "набережная",
"кольцо": "кольцо",
"просека": "просека",
"квартал": "квартал",
"территория": "территория",
"дорога": "дорога",
"разъезд": "разъезд",
}
// ordinalWords maps spelled-out ordinals to their digit form. Only the forms
// that actually occur in practice are listed; unknown words are treated as
// part of the street name.
var ordinalWords = map[string]string{"первый": "1", "первая": "1", "первое": "1",
"второй": "2", "вторая": "2", "второе": "2",
"третий": "3", "третья": "3", "третье": "3",
"четвертый": "4", "четвертая": "4", "четвертое": "4",
"пятый": "5", "пятая": "5", "пятое": "5",
"шестой": "6", "шестая": "6", "шестое": "6",
"седьмой": "7", "седьмая": "7", "седьмое": "7",
"восьмой": "8", "восьмая": "8", "восьмое": "8",
"девятый": "9", "девятая": "9", "девятое": "9",
"десятый": "10", "десятая": "10", "десятое": "10",
}
// ordinalSuffixes are the gender/number endings that may follow an ordinal
// digit when it is written as a separate token ("2 я", "10 й").
var ordinalSuffixes = map[string]bool{
"я": true, "й": true, "е": true,
"ая": true, "ый": true, "ой": true, "ое": true,
"ий": true, "ья": true, "ье": true, "ьи": true,
}
// streetKey builds the canonical, type-independent street key:
// "<ordinal>|<sorted name tokens>". The type word is dropped so that a user
// who omits it ("станционная 1" instead of "Станционная улица 1") still
// matches; type confusion is surfaced later as ambiguity.
func streetKey(ordinal, nameKey string) string {
return ordinal + "|" + nameKey
} }
// normalizeOrdinal trims leading zeros so "02-я" and "2-я" agree. // normalizeOrdinal trims leading zeros so "02-я" and "2-я" agree.
+6 -11
View File
@@ -10,7 +10,7 @@ package address
// Status is the machine-readable outcome of an address resolution. // Status is the machine-readable outcome of an address resolution.
// //
// It is intentionally explicit: callers must never have to guess whether an // It is intentionally explicit: callers must never have to guess whether an
// empty result means "not found" or "the resolver refused to choose". // empty result means "not found" or "the upstream failed".
type Status string type Status string
const ( const (
@@ -25,6 +25,11 @@ const (
StatusAmbiguous Status = "ambiguous" StatusAmbiguous Status = "ambiguous"
// StatusNotFound means nothing in the registry matched. // StatusNotFound means nothing in the registry matched.
StatusNotFound Status = "not_found" StatusNotFound Status = "not_found"
// StatusPortalTimeout means the external outages portal did not answer in
// time. It is an infrastructure outcome, not "no outages".
StatusPortalTimeout Status = "portal_timeout"
// StatusInfra is a generic terminal infrastructure failure.
StatusInfra Status = "infra_error"
) )
// ParsedAddress is the structured form of a free-form address string. // ParsedAddress is the structured form of a free-form address string.
@@ -36,13 +41,6 @@ type ParsedAddress struct {
Raw string `json:"raw"` Raw string `json:"raw"`
Normalized string `json:"normalized"` Normalized string `json:"normalized"`
// City is the canonical city name when the query names one (or when the
// parser is configured with a default city). CityKey is its normalized
// form. The city is part of StreetKey, so identical street names in
// different cities never collide.
City string `json:"city,omitempty"`
CityKey string `json:"city_key,omitempty"`
// Street components. // Street components.
// //
// NameKey is the order-insensitive canonical name (tokens sorted), so // NameKey is the order-insensitive canonical name (tokens sorted), so
@@ -75,9 +73,6 @@ type Entry struct {
Address string `json:"address"` Address string `json:"address"`
District string `json:"district,omitempty"` District string `json:"district,omitempty"`
City string `json:"city,omitempty"`
CityKey string `json:"city_key,omitempty"`
Ordinal string `json:"ordinal,omitempty"` Ordinal string `json:"ordinal,omitempty"`
NameKey string `json:"name_key,omitempty"` NameKey string `json:"name_key,omitempty"`
NameDisplay string `json:"name_display,omitempty"` NameDisplay string `json:"name_display,omitempty"`
-198
View File
@@ -1,198 +0,0 @@
package match
import (
"context"
"reflect"
"strings"
"testing"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
)
// staticRewriter returns a fixed list of variants, emulating the LLM adapter.
type staticRewriter []string
func (s staticRewriter) Rewrite(context.Context, string) ([]string, error) { return s, nil }
// TestRewriteInheritsDetectedCity guards the multi-city LLM path: the embedded
// prompt tells the model to strip administrative words, so a correct variant
// usually has no city. It must inherit the city detected in the original query,
// while a variant naming a different city must still be rejected.
func TestRewriteInheritsDetectedCity(t *testing.T) {
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
entries := cityEntries(t, parser,
[2]string{"Новосибирск", "Кошурникова улица 5"},
[2]string{"Бердск", "Кошурникова улица 5"},
)
// A threshold above any typo score forces the deterministic path to miss.
ix := NewIndex("test", entries, Options{FuzzyThreshold: 0.999, MaxCandidates: 5})
cityless := NewResolverWithOptions(ix, staticRewriter{"Кошурникова 5"}, ResolverOptions{Parser: parser})
res := cityless.ResolveInCity(context.Background(), "нск кошурникиова 5", "")
if res.Status != address.StatusFound || res.City != "Новосибирск" {
t.Fatalf("cityless variant: %+v", res)
}
conflict := NewResolverWithOptions(ix, staticRewriter{"Бердск, Кошурникова 5"}, ResolverOptions{Parser: parser})
res = conflict.ResolveInCity(context.Background(), "нск кошурникиова 5", "")
if res.Status == address.StatusFound {
t.Fatalf("conflicting city variant was accepted: %+v", res)
}
}
func matchCityDictionary() []address.City {
return []address.City{
{Name: "Новосибирск", Aliases: []string{"нск"}},
{Name: "Бердск"},
}
}
func cityEntries(t *testing.T, parser *address.Parser, rows ...[2]string) []address.Entry {
t.Helper()
entries := make([]address.Entry, 0, len(rows))
for _, row := range rows {
p := parser.ParseInCity(row[1], row[0])
entries = append(entries, address.Entry{
ID: row[0] + "/" + row[1], Address: row[1],
City: p.City, CityKey: p.CityKey,
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
Structure: p.Structure,
})
}
return entries
}
func TestIndexKeepsCitiesApart(t *testing.T) {
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
entries := cityEntries(t, parser,
[2]string{"Новосибирск", "Ленина улица 5"},
[2]string{"Бердск", "Ленина улица 5"},
)
resolver := NewResolverWithOptions(
NewIndex("test", entries, DefaultOptions()),
nil,
ResolverOptions{Parser: parser},
)
nsk := resolver.ResolveInCity(t.Context(), "Ленина 5", "Новосибирск")
if nsk.Status != address.StatusFound || nsk.City != "Новосибирск" {
t.Fatalf("Новосибирск: %+v", nsk)
}
berdsk := resolver.ResolveInCity(t.Context(), "Ленина 5", "Бердск")
if berdsk.Status != address.StatusFound || berdsk.City != "Бердск" {
t.Fatalf("Бердск: %+v", berdsk)
}
// Without a city the resolver must not pick one: both streets are exact
// matches in different cities.
res := resolver.Resolve(t.Context(), "Ленина 5")
if res.Status != address.StatusAmbiguous {
t.Fatalf("cityless status = %q, want ambiguous (candidates: %+v)", res.Status, res.Candidates)
}
if len(res.Candidates) != 2 {
t.Fatalf("candidates = %+v, want one per city", res.Candidates)
}
if res.City != "" {
t.Fatalf("ambiguous cityless result must not report a city, got %q", res.City)
}
}
func TestCitylessQueryReportsTheOnlyCity(t *testing.T) {
parser := address.NewParser(address.NewDictionary(matchCityDictionary()), "")
entries := cityEntries(t, parser, [2]string{"Новосибирск", "Ленина улица 5"})
resolver := NewResolverWithOptions(
NewIndex("test", entries, DefaultOptions()),
nil,
ResolverOptions{Parser: parser},
)
res := resolver.Resolve(t.Context(), "Ленина 5")
if res.Status != address.StatusFound || res.City != "Новосибирск" {
t.Fatalf("result = %+v", res)
}
}
func TestStructureIsPartOfHouseIdentity(t *testing.T) {
ix := buildIndex(t,
"Станционная улица 1",
"Станционная улица 1 стр. 2",
)
found := ix.Resolve(address.Parse("станционная 1 стр. 2"))
if found.Status != address.StatusFound || found.Address != "Станционная улица 1 стр. 2" {
t.Fatalf("exact structure: %+v", found)
}
// A structure that does not exist must never resolve to another building.
wrong := ix.Resolve(address.Parse("станционная 1 стр. 3"))
if wrong.Status == address.StatusFound {
t.Fatalf("wrong structure resolved to %q", wrong.Address)
}
if wrong.Status != address.StatusAmbiguous {
t.Fatalf("wrong structure status = %q, want ambiguous", wrong.Status)
}
base := ix.Resolve(address.Parse("станционная 1"))
if base.Status != address.StatusFound || base.Address != "Станционная улица 1" {
t.Fatalf("base house: %+v", base)
}
}
func TestIndexOrderIsDeterministic(t *testing.T) {
addresses := []string{
"Станционная улица 1/1",
"Станционная улица 2/1",
"Станционная улица 3/1 к1",
"Станционная улица 3/1 к2",
"Искитимская улица 5",
}
forward := buildEntries(t, addresses...)
backward := buildEntries(t, reverse(addresses)...)
a := NewIndex("v", forward, DefaultOptions())
b := NewIndex("v", backward, DefaultOptions())
for _, q := range []string{"станционная 3/1", "станционная 1/2", "искитимсая 5"} {
ra := a.Resolve(address.Parse(q))
rb := b.Resolve(address.Parse(q))
if !reflect.DeepEqual(ra, rb) {
t.Fatalf("%q: results differ\n a=%+v\n b=%+v", q, ra, rb)
}
}
}
func TestLongTokenDoesNotMatch(t *testing.T) {
ix := buildIndex(t, "Станционная улица 1")
res := ix.Resolve(address.Parse(strings.Repeat("а", 4000)))
if res.Status != address.StatusNotFound {
t.Fatalf("status = %q, want not_found", res.Status)
}
}
func buildEntries(t *testing.T, addresses ...string) []address.Entry {
t.Helper()
entries := make([]address.Entry, 0, len(addresses))
for _, a := range addresses {
p := address.Parse(a)
entries = append(entries, address.Entry{
ID: a, Address: a,
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House, HouseFrac: p.HouseFrac, HouseLetter: p.HouseLetter,
Building: p.Building, BuildingFrac: p.BuildingFrac, BuildingLetter: p.BuildingLetter,
Structure: p.Structure,
})
}
return entries
}
func reverse(in []string) []string {
out := make([]string, len(in))
for i, s := range in {
out[len(in)-1-i] = s
}
return out
}
+14 -119
View File
@@ -9,7 +9,6 @@ package match
import ( import (
"sort" "sort"
"unicode/utf8"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address" "git.totmin.ru/en2zmax/addrkit/internal/domain/address"
) )
@@ -18,10 +17,8 @@ import (
// the caller (or an operator) can see what the resolver was choosing between. // the caller (or an operator) can see what the resolver was choosing between.
type Candidate struct { type Candidate struct {
Address string `json:"address"` Address string `json:"address"`
City string `json:"city,omitempty"`
District string `json:"district,omitempty"` District string `json:"district,omitempty"`
House string `json:"house,omitempty"` House string `json:"house,omitempty"`
Structure string `json:"structure,omitempty"`
Score float64 `json:"score"` Score float64 `json:"score"`
} }
@@ -29,7 +26,6 @@ type Candidate struct {
type Result struct { type Result struct {
Status address.Status `json:"status"` Status address.Status `json:"status"`
Address string `json:"address,omitempty"` Address string `json:"address,omitempty"`
City string `json:"city,omitempty"`
District string `json:"district,omitempty"` District string `json:"district,omitempty"`
Confidence float64 `json:"confidence"` Confidence float64 `json:"confidence"`
Candidates []Candidate `json:"candidates,omitempty"` Candidates []Candidate `json:"candidates,omitempty"`
@@ -64,9 +60,9 @@ func DefaultOptions() Options {
type streetRef struct { type streetRef struct {
key string key string
cityKey string
ordinal string ordinal string
nameKey string nameKey string
streetType string
} }
// Index is an immutable, in-memory registry index. It is safe for concurrent // Index is an immutable, in-memory registry index. It is safe for concurrent
@@ -76,46 +72,26 @@ type Index struct {
opts Options opts Options
byKey map[string][]address.Entry byKey map[string][]address.Entry
streets []streetRef streets []streetRef
cities map[string]string // cityKey -> canonical display name
size int
} }
// maxFuzzyNameRunes bounds the fuzzy scan. No real street name is longer than
// this; without the bound a single huge token would make every Levenshtein
// comparison walk a very long string and turn a request into a CPU sink.
const maxFuzzyNameRunes = 96
// NewIndex builds an index from canonical entries. Entries sharing a StreetKey // NewIndex builds an index from canonical entries. Entries sharing a StreetKey
// are grouped; distinct streets are collected for typo-tolerant matching. // are grouped; distinct streets are collected for typo-tolerant matching.
//
// Entries are sorted first, so candidate order is deterministic regardless of
// the order in which the storage adapter returned them.
func NewIndex(version string, entries []address.Entry, opts Options) *Index { func NewIndex(version string, entries []address.Entry, opts Options) *Index {
sorted := append([]address.Entry(nil), entries...)
sort.SliceStable(sorted, func(i, j int) bool { return entryLess(sorted[i], sorted[j]) })
ix := &Index{ ix := &Index{
version: version, version: version,
opts: opts, opts: opts,
byKey: make(map[string][]address.Entry, len(sorted)), byKey: make(map[string][]address.Entry, len(entries)),
cities: make(map[string]string),
size: len(sorted),
} }
seen := make(map[string]struct{}, len(sorted)) seen := make(map[string]struct{}, len(entries))
for _, e := range sorted { for _, e := range entries {
ix.byKey[e.StreetKey] = append(ix.byKey[e.StreetKey], e) ix.byKey[e.StreetKey] = append(ix.byKey[e.StreetKey], e)
if e.CityKey != "" {
if _, ok := ix.cities[e.CityKey]; !ok {
ix.cities[e.CityKey] = e.City
}
}
if _, ok := seen[e.StreetKey]; !ok { if _, ok := seen[e.StreetKey]; !ok {
seen[e.StreetKey] = struct{}{} seen[e.StreetKey] = struct{}{}
ix.streets = append(ix.streets, streetRef{ ix.streets = append(ix.streets, streetRef{
key: e.StreetKey, key: e.StreetKey,
cityKey: e.CityKey,
ordinal: e.Ordinal, ordinal: e.Ordinal,
nameKey: e.NameKey, nameKey: e.NameKey,
streetType: e.StreetType,
}) })
} }
} }
@@ -123,57 +99,22 @@ func NewIndex(version string, entries []address.Entry, opts Options) *Index {
return ix return ix
} }
// entryLess defines the canonical order of registry rows. Address strings are
// the last meaningful tie-breaker; ID makes the order total even for duplicated
// rows.
func entryLess(a, b address.Entry) bool {
switch {
case a.CityKey != b.CityKey:
return a.CityKey < b.CityKey
case a.StreetKey != b.StreetKey:
return a.StreetKey < b.StreetKey
case a.House != b.House:
return a.House < b.House
case a.HouseFrac != b.HouseFrac:
return a.HouseFrac < b.HouseFrac
case a.HouseLetter != b.HouseLetter:
return a.HouseLetter < b.HouseLetter
case a.Building != b.Building:
return a.Building < b.Building
case a.BuildingFrac != b.BuildingFrac:
return a.BuildingFrac < b.BuildingFrac
case a.BuildingLetter != b.BuildingLetter:
return a.BuildingLetter < b.BuildingLetter
case a.Structure != b.Structure:
return a.Structure < b.Structure
case a.Address != b.Address:
return a.Address < b.Address
default:
return a.ID < b.ID
}
}
// Version returns the data version the index was built from. // Version returns the data version the index was built from.
func (ix *Index) Version() string { return ix.version } func (ix *Index) Version() string { return ix.version }
// Size returns the number of indexed entries. // Size returns the number of indexed entries.
func (ix *Index) Size() int { return ix.size } func (ix *Index) Size() int {
n := 0
// Cities returns the distinct canonical city names present in the index, in for _, v := range ix.byKey {
// deterministic order. A city-less index returns an empty slice. n += len(v)
func (ix *Index) Cities() []string {
out := make([]string, 0, len(ix.cities))
for _, name := range ix.cities {
out = append(out, name)
} }
sort.Strings(out) return n
return out
} }
// Resolve resolves a parsed address. It never returns an address that is not // Resolve resolves a parsed address. It never returns an address that is not
// present in the index. // present in the index.
func (ix *Index) Resolve(p address.ParsedAddress) Result { func (ix *Index) Resolve(p address.ParsedAddress) Result {
res := Result{DataVersion: ix.version, City: p.City} res := Result{DataVersion: ix.version}
entries := ix.byKey[p.StreetKey] entries := ix.byKey[p.StreetKey]
similarity := 1.0 similarity := 1.0
@@ -192,11 +133,6 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
res.Status = address.StatusNotFound res.Status = address.StatusNotFound
return res return res
} }
if res.City == "" {
// A city-less query may still match exactly one city; report it so the
// caller does not have to re-derive the city from the candidates.
res.City = commonCity(entries)
}
if !p.HasHouse { if !p.HasHouse {
// Street is known but the caller did not supply a house: safe but // Street is known but the caller did not supply a house: safe but
@@ -212,7 +148,6 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
case 1: case 1:
res.Status = address.StatusFound res.Status = address.StatusFound
res.Address = matched[0].Address res.Address = matched[0].Address
res.City = matched[0].City
res.District = matched[0].District res.District = matched[0].District
res.Confidence = clamp01(similarity) res.Confidence = clamp01(similarity)
return res return res
@@ -227,14 +162,7 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
} }
res.Status = address.StatusPartial res.Status = address.StatusPartial
res.Confidence = clamp01(similarity * 0.9) res.Confidence = clamp01(similarity * 0.9)
// Prefer houses that share the number/fraction/letter: they are the res.Candidates = candidates(entries, similarity, ix.opts.MaxCandidates)
// plausible corrections (a different corpus or structure), while the
// rest of the street is noise.
related := matchHouseUnitAll(entries, p)
if len(related) == 0 {
related = entries
}
res.Candidates = candidates(related, similarity, ix.opts.MaxCandidates)
return res return res
default: default:
res.Status = address.StatusAmbiguous res.Status = address.StatusAmbiguous
@@ -248,16 +176,12 @@ func (ix *Index) Resolve(p address.ParsedAddress) Result {
// the best street when the match is unambiguous, or the list of competing // the best street when the match is unambiguous, or the list of competing
// street keys when several are within the ambiguity margin. // street keys when several are within the ambiguity margin.
func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry, similarity float64, ambiguousKeys []string) { func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry, similarity float64, ambiguousKeys []string) {
if p.NameKey == "" || utf8.RuneCountInString(p.NameKey) > maxFuzzyNameRunes { if p.NameKey == "" {
return nil, 0, nil return nil, 0, nil
} }
best := 0.0 best := 0.0
bestKey := "" bestKey := ""
for _, s := range ix.streets { for _, s := range ix.streets {
// A query that names a city may only match streets in that city.
if p.CityKey != "" && s.cityKey != p.CityKey {
continue
}
// If the query carries an ordinal, the candidate's ordinal must match: // If the query carries an ordinal, the candidate's ordinal must match:
// a user who says "1-й Северный" does not mean the plain "Северная". // a user who says "1-й Северный" does not mean the plain "Северная".
if !ordinalCompatible(p.Ordinal, s.ordinal) { if !ordinalCompatible(p.Ordinal, s.ordinal) {
@@ -273,9 +197,6 @@ func (ix *Index) fuzzyStreets(p address.ParsedAddress) (entries []address.Entry,
} }
for _, s := range ix.streets { for _, s := range ix.streets {
if p.CityKey != "" && s.cityKey != p.CityKey {
continue
}
if !ordinalCompatible(p.Ordinal, s.ordinal) { if !ordinalCompatible(p.Ordinal, s.ordinal) {
continue continue
} }
@@ -312,10 +233,8 @@ func (ix *Index) representatives(keys []string, score float64) []Candidate {
e := es[0] e := es[0]
out = append(out, Candidate{ out = append(out, Candidate{
Address: e.Address, Address: e.Address,
City: e.City,
District: e.District, District: e.District,
House: e.House, House: e.House,
Structure: e.Structure,
Score: clamp01(score), Score: clamp01(score),
}) })
if len(out) == ix.opts.MaxCandidates { if len(out) == ix.opts.MaxCandidates {
@@ -356,16 +275,13 @@ func sameHouseSeveral(entries []address.Entry, p address.ParsedAddress) bool {
// houseUnitEqual compares the parts of a house unit. Unspecified (empty) parts // houseUnitEqual compares the parts of a house unit. Unspecified (empty) parts
// must also be empty on the entry side, so "60/1" never matches "60/1 к148". // must also be empty on the entry side, so "60/1" never matches "60/1 к148".
// The structure ("стр. 2") is part of the identity: a caller who explicitly
// asked for a different structure must not be handed another building.
func houseUnitEqual(e address.Entry, p address.ParsedAddress) bool { func houseUnitEqual(e address.Entry, p address.ParsedAddress) bool {
return e.House == p.House && return e.House == p.House &&
e.HouseFrac == p.HouseFrac && e.HouseFrac == p.HouseFrac &&
e.HouseLetter == p.HouseLetter && e.HouseLetter == p.HouseLetter &&
e.Building == p.Building && e.Building == p.Building &&
e.BuildingFrac == p.BuildingFrac && e.BuildingFrac == p.BuildingFrac &&
e.BuildingLetter == p.BuildingLetter && e.BuildingLetter == p.BuildingLetter
e.Structure == p.Structure
} }
func candidates(entries []address.Entry, score float64, limit int) []Candidate { func candidates(entries []address.Entry, score float64, limit int) []Candidate {
@@ -376,10 +292,8 @@ func candidates(entries []address.Entry, score float64, limit int) []Candidate {
for _, e := range entries { for _, e := range entries {
out = append(out, Candidate{ out = append(out, Candidate{
Address: e.Address, Address: e.Address,
City: e.City,
District: e.District, District: e.District,
House: e.House, House: e.House,
Structure: e.Structure,
Score: clamp01(score), Score: clamp01(score),
}) })
if len(out) == limit { if len(out) == limit {
@@ -398,22 +312,3 @@ func clamp01(v float64) float64 {
} }
return v return v
} }
// commonCity returns the city shared by all entries, or "" when they span
// cities (or carry no city at all).
func commonCity(entries []address.Entry) string {
key, name := "", ""
for _, e := range entries {
if e.CityKey == "" {
continue
}
if key == "" {
key, name = e.CityKey, e.City
continue
}
if e.CityKey != key {
return ""
}
}
return name
}
+4 -19
View File
@@ -1,9 +1,6 @@
package match package match
import ( import "strings"
"strings"
"unicode/utf8"
)
// levenshtein returns the edit distance between two strings, measured in // levenshtein returns the edit distance between two strings, measured in
// runes. It uses the rolling-row formulation: O(min(n,m)) memory. // runes. It uses the rolling-row formulation: O(min(n,m)) memory.
@@ -105,30 +102,18 @@ func min3(a, b, c int) int {
return m return m
} }
// maxTokenRunes is the longest token the fuzzy matcher will compare. Real
// street-name tokens are far shorter; anything longer is noise or abuse.
const maxTokenRunes = 64
// tokenScore scores two name tokens. A single edit is tolerated for tokens of // tokenScore scores two name tokens. A single edit is tolerated for tokens of
// length >= 4 and two edits for length >= 8; otherwise it falls back to the // length >= 4 and two edits for length >= 8; otherwise it falls back to the
// plain ratio. Short names (<= 3 chars) are intentionally not fuzzy-matched: // plain ratio. Short names (<= 3 chars) are intentionally not fuzzy-matched:
// one edit is too large a fraction of the word to be safe. // one edit is too large a fraction of the word to be safe.
//
// Tokens longer than maxTokenRunes are never similar to a real street name.
// Rejecting them up front keeps a pathological input from turning one request
// into millions of Levenshtein cell updates.
func tokenScore(a, b string) float64 { func tokenScore(a, b string) float64 {
if a == b { if a == b {
return 1 return 1
} }
ra, rb := utf8.RuneCountInString(a), utf8.RuneCountInString(b)
if ra == 0 || rb == 0 || ra > maxTokenRunes || rb > maxTokenRunes {
return 0
}
d := levenshtein(a, b) d := levenshtein(a, b)
minLen := ra minLen := len([]rune(a))
if rb < minLen { if lb := len([]rune(b)); lb < minLen {
minLen = rb minLen = lb
} }
switch { switch {
case d == 1 && minLen >= 4: case d == 1 && minLen >= 4:
+9 -100
View File
@@ -20,25 +20,6 @@ type Rewriter interface {
Rewrite(ctx context.Context, raw string) ([]string, error) Rewrite(ctx context.Context, raw string) ([]string, error)
} }
// Parser is the address parser used by the resolver. It is defined here
// (consumer side) so the resolver can be tested with a stub and so a
// deployment can plug in its own dictionary.
type Parser interface {
Parse(raw string) address.ParsedAddress
ParseInCity(raw, city string) address.ParsedAddress
MatchCity(query string) (address.City, bool)
}
// ResolverOptions tunes optional resolver behaviour.
type ResolverOptions struct {
// Parser defaults to address.DefaultParser().
Parser Parser
// OnRewriteError, when set, observes failures of the optional rewriter.
// The resolver always keeps the deterministic result, but operators still
// need to know that the fallback is broken.
OnRewriteError func(error)
}
// Resolver wraps an Index with an optional Rewriter. // Resolver wraps an Index with an optional Rewriter.
// //
// The deterministic path is always tried first. The rewriter is consulted only // The deterministic path is always tried first. The rewriter is consulted only
@@ -48,28 +29,11 @@ type ResolverOptions struct {
type Resolver struct { type Resolver struct {
index *Index index *Index
rewriter Rewriter rewriter Rewriter
parser Parser
onRewriteError func(error)
} }
// NewResolver builds a resolver with the default parser and no rewriter // NewResolver builds a resolver. rewriter may be nil to disable rewriting.
// diagnostics.
func NewResolver(index *Index, rewriter Rewriter) *Resolver { func NewResolver(index *Index, rewriter Rewriter) *Resolver {
return NewResolverWithOptions(index, rewriter, ResolverOptions{}) return &Resolver{index: index, rewriter: rewriter}
}
// NewResolverWithOptions builds a resolver with explicit options.
func NewResolverWithOptions(index *Index, rewriter Rewriter, opts ResolverOptions) *Resolver {
parser := opts.Parser
if parser == nil {
parser = address.DefaultParser()
}
return &Resolver{
index: index,
rewriter: rewriter,
parser: parser,
onRewriteError: opts.OnRewriteError,
}
} }
// Version returns the current data version. // Version returns the current data version.
@@ -78,35 +42,9 @@ func (r *Resolver) Version() string { return r.index.Version() }
// Ready reports whether the resolver has a usable index. // Ready reports whether the resolver has a usable index.
func (r *Resolver) Ready() bool { return r.index != nil && r.index.Size() > 0 } func (r *Resolver) Ready() bool { return r.index != nil && r.index.Size() > 0 }
// Cities returns the canonical city names present in the index.
func (r *Resolver) Cities() []string {
if r.index == nil {
return nil
}
return r.index.Cities()
}
// MatchCity reports whether query is a city name/alias and returns the
// canonical name. It uses the resolver's parser dictionary, not the index, so
// it also recognizes aliases for cities that have no rows yet.
func (r *Resolver) MatchCity(query string) (string, bool) {
city, ok := r.parser.MatchCity(query)
if !ok {
return "", false
}
return city.Name, true
}
// Resolve parses raw and resolves it, consulting the rewriter when enabled. // Resolve parses raw and resolves it, consulting the rewriter when enabled.
func (r *Resolver) Resolve(ctx context.Context, raw string) Result { func (r *Resolver) Resolve(ctx context.Context, raw string) Result {
return r.ResolveInCity(ctx, raw, "") parsed := address.Parse(raw)
}
// ResolveInCity is Resolve with an explicit city hint. The hint overrides a
// city found in the address text and lets clients pass the city as a separate
// field.
func (r *Resolver) ResolveInCity(ctx context.Context, raw, city string) Result {
parsed := r.parse(raw, city)
res := r.index.Resolve(parsed) res := r.index.Resolve(parsed)
if res.Status == address.StatusFound || r.rewriter == nil { if res.Status == address.StatusFound || r.rewriter == nil {
@@ -117,27 +55,13 @@ func (r *Resolver) ResolveInCity(ctx context.Context, raw, city string) Result {
} }
alternatives, err := r.rewriter.Rewrite(ctx, raw) alternatives, err := r.rewriter.Rewrite(ctx, raw)
if err != nil { if err != nil || len(alternatives) == 0 {
if r.onRewriteError != nil {
r.onRewriteError(err)
}
return res
}
if len(alternatives) == 0 {
return res return res
} }
best := res best := res
for _, alt := range alternatives { for _, alt := range alternatives {
variant := r.parse(alt, city) variant := address.Parse(alt)
// The embedded prompt asks the model to strip administrative words
// (including the city), so a correct variant often has no city. It
// inherits the original's city; only a conflicting explicit city is a
// disqualification. Without this, the rewriter silently contributes
// nothing in multi-city deployments (where default_city is empty).
if variant.CityKey == "" && parsed.CityKey != "" {
variant = r.parser.ParseInCity(alt, parsed.City)
}
// The model may fix the spelling of the street, nothing else. // The model may fix the spelling of the street, nothing else.
if !compatibleVariant(parsed, variant) { if !compatibleVariant(parsed, variant) {
continue continue
@@ -150,13 +74,6 @@ func (r *Resolver) ResolveInCity(ctx context.Context, raw, city string) Result {
return best return best
} }
func (r *Resolver) parse(raw, city string) address.ParsedAddress {
if strings.TrimSpace(city) != "" {
return r.parser.ParseInCity(raw, city)
}
return r.parser.Parse(raw)
}
// compatibleVariant reports whether an LLM rewrite may be considered. A rewrite // compatibleVariant reports whether an LLM rewrite may be considered. A rewrite
// is acceptable only when it: // is acceptable only when it:
// - keeps the exact house unit (number, fraction, letter, corpus, structure // - keeps the exact house unit (number, fraction, letter, corpus, structure
@@ -165,9 +82,6 @@ func (r *Resolver) parse(raw, city string) address.ParsedAddress {
// - has the same words in the street name, each within a small edit distance // - has the same words in the street name, each within a small edit distance
// — so a different street can never be substituted. // — so a different street can never be substituted.
func compatibleVariant(orig, variant address.ParsedAddress) bool { func compatibleVariant(orig, variant address.ParsedAddress) bool {
if orig.CityKey != variant.CityKey {
return false
}
if !sameHouseUnit(orig, variant) { if !sameHouseUnit(orig, variant) {
return false return false
} }
@@ -194,13 +108,9 @@ func nearToken(a, b string) bool {
if a == b { if a == b {
return true return true
} }
ra, rb := len([]rune(a)), len([]rune(b)) minLen := len([]rune(a))
if ra == 0 || rb == 0 || ra > maxTokenRunes || rb > maxTokenRunes { if lb := len([]rune(b)); lb < minLen {
return false minLen = lb
}
minLen := ra
if rb < minLen {
minLen = rb
} }
d := damerau(a, b) d := damerau(a, b)
switch { switch {
@@ -219,8 +129,7 @@ func sameHouseUnit(a, b address.ParsedAddress) bool {
a.HouseLetter == b.HouseLetter && a.HouseLetter == b.HouseLetter &&
a.Building == b.Building && a.Building == b.Building &&
a.BuildingFrac == b.BuildingFrac && a.BuildingFrac == b.BuildingFrac &&
a.BuildingLetter == b.BuildingLetter && a.BuildingLetter == b.BuildingLetter
a.Structure == b.Structure
} }
// better ranks results: a stronger status wins, then higher confidence. // better ranks results: a stronger status wins, then higher confidence.
-20
View File
@@ -42,26 +42,6 @@ func TestCompatibleVariantProtectsApartment(t *testing.T) {
} }
} }
// TestCompatibleVariantProtectsCityAndStructure makes sure a rewrite cannot
// move the address to another city or another building.
func TestCompatibleVariantProtectsCityAndStructure(t *testing.T) {
parser := address.NewParser(address.NewDictionary([]address.City{
{Name: "Новосибирск", Aliases: []string{"нск"}},
{Name: "Бердск"},
}), "")
orig := parser.Parse("нск, Сухарная улица 95 стр. 1")
if !compatibleVariant(orig, parser.Parse("Новосибирск, Сухраная улица 95 стр. 1")) {
t.Error("same-city spelling fix must be accepted")
}
if compatibleVariant(orig, parser.Parse("Бердск, Сухарная улица 95 стр. 1")) {
t.Error("city change must be rejected")
}
if compatibleVariant(orig, parser.Parse("нск, Сухарная улица 95 стр. 2")) {
t.Error("structure change must be rejected")
}
}
func TestDamerau(t *testing.T) { func TestDamerau(t *testing.T) {
if got := damerau("гоглоя", "гоголя"); got != 1 { if got := damerau("гоглоя", "гоголя"); got != 1 {
t.Errorf("transposition distance = %d, want 1", got) t.Errorf("transposition distance = %d, want 1", got)
+109
View File
@@ -0,0 +1,109 @@
// Package outages contains the deterministic, pure logic for selecting outages
// that belong to a concrete house unit. It does not talk to the network: the
// portal client lives in an adapter and feeds this package.
package outages
import (
"context"
"strconv"
"strings"
"time"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
)
// Outage is a single outage record as returned by the external portal.
type Outage struct {
Type string `json:"type,omitempty"`
System string `json:"system,omitempty"`
Address string `json:"address,omitempty"`
StartDate string `json:"start_date,omitempty"`
EndDate string `json:"end_date,omitempty"`
Time string `json:"time,omitempty"`
Action string `json:"action,omitempty"`
}
// Source is anything that can answer "what outages are there for this
// canonical address". It is implemented by the ArcGIS portal adapter and by
// fakes in tests.
type Source interface {
Query(ctx context.Context, canonical string) ([]Outage, error)
}
// MatchingHouse keeps only the outages that address the same house unit as the
// target. This is the fix for the classic bug where a substring match returned
// a neighbouring house's outage: matching is done on the parsed structure, not
// on the raw text.
func MatchingHouse(list []Outage, target address.ParsedAddress) []Outage {
out := make([]Outage, 0, len(list))
for _, o := range list {
op := address.Parse(o.Address)
if houseUnitEqual(target, op) {
out = append(out, o)
}
}
return out
}
// houseUnitEqual compares two parsed house units. House number, fraction,
// letter, building (corpus) and its fraction/letter must all match. The
// structure (строение) is deliberately ignored: it is frequently omitted by
// users and does not change which house an outage refers to.
func houseUnitEqual(a, b address.ParsedAddress) bool {
return a.House == b.House &&
a.HouseFrac == b.HouseFrac &&
a.HouseLetter == b.HouseLetter &&
a.Building == b.Building &&
a.BuildingFrac == b.BuildingFrac &&
a.BuildingLetter == b.BuildingLetter
}
// Current keeps outages whose [start, end] range covers now. The portal
// already filters by date, but we re-check locally because the portal's notion
// of "today" and ours must agree (Asia/Novosibirsk, UTC+7).
func Current(list []Outage, now time.Time, loc *time.Location) []Outage {
out := make([]Outage, 0, len(list))
for _, o := range list {
start, okStart := parsePortalTime(o.StartDate, loc)
end, okEnd := parsePortalTime(o.EndDate, loc)
if !okStart || !okEnd {
// Undated records are kept: better to report a possibly-current
// outage than to silently drop it.
out = append(out, o)
continue
}
if (now.Equal(start) || now.After(start)) && (now.Equal(end) || now.Before(end)) {
out = append(out, o)
}
}
return out
}
// parsePortalTime accepts the shapes the portal may produce: epoch
// milliseconds, epoch seconds, RFC3339 and a few common date formats. All
// results are interpreted in loc.
func parsePortalTime(s string, loc *time.Location) (time.Time, bool) {
s = strings.TrimSpace(s)
if s == "" {
return time.Time{}, false
}
if n, err := strconv.ParseInt(s, 10, 64); err == nil {
// Heuristic: 13+ digits is milliseconds, otherwise seconds.
if n > 1e12 {
return time.UnixMilli(n).In(loc), true
}
return time.Unix(n, 0).In(loc), true
}
layouts := []string{
time.RFC3339,
"2006-01-02T15:04:05",
"2006-01-02 15:04:05",
"2006-01-02",
}
for _, layout := range layouts {
if t, err := time.ParseInLocation(layout, s, loc); err == nil {
return t, true
}
}
return time.Time{}, false
}
+57
View File
@@ -0,0 +1,57 @@
package outages
import (
"strconv"
"testing"
"time"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
)
func TestMatchingHouse(t *testing.T) {
list := []Outage{
{Address: "Станционная улица 1/1", System: "Теплоснабжение"},
{Address: "Станционная улица 1/2", System: "Теплоснабжение"},
{Address: "Станционная улица 1/1 к1", System: "Теплоснабжение"},
{Address: "Станционная улица 10", System: "Теплоснабжение"},
}
target := address.Parse("2-я Станционная улица 1/1")
got := MatchingHouse(list, target)
if len(got) != 1 {
t.Fatalf("expected exactly 1 matching outage, got %d: %+v", len(got), got)
}
if got[0].Address != "Станционная улица 1/1" {
t.Errorf("matched wrong outage: %q", got[0].Address)
}
}
func TestMatchingHouseIgnoresStructure(t *testing.T) {
list := []Outage{{Address: "Станционная улица 1/1 стр.2"}}
target := address.Parse("станционная 1/1 стр.3")
if got := MatchingHouse(list, target); len(got) != 1 {
t.Fatalf("structure must not affect house matching; got %d", len(got))
}
}
func TestCurrent(t *testing.T) {
loc := time.FixedZone("NSK", 7*3600)
now := time.Date(2026, 10, 9, 12, 0, 0, 0, loc)
within := now.Add(-time.Hour).UnixMilli()
after := now.Add(time.Hour).UnixMilli()
before := now.Add(-48 * time.Hour).UnixMilli()
list := []Outage{
{Address: "A", StartDate: itoa(within), EndDate: itoa(after)}, // current
{Address: "B", StartDate: itoa(before), EndDate: itoa(before)}, // past
{Address: "C", StartDate: "", EndDate: ""}, // undated -> kept
}
got := Current(list, now, loc)
if len(got) != 2 {
t.Fatalf("expected 2 current outages, got %d: %+v", len(got), got)
}
}
func itoa(v int64) string {
return strconv.FormatInt(v, 10)
}
+8 -34
View File
@@ -28,10 +28,6 @@ import (
type Case struct { type Case struct {
Query string Query string
Expect string Expect string
// ExpectCity is the canonical city of the source entry. Empty for a
// city-less index; otherwise a "found" result in another city is a safety
// violation even when the street and house text look identical.
ExpectCity string
Category string Category string
RequireExact bool RequireExact bool
} }
@@ -45,7 +41,6 @@ type CategoryStat struct {
Ambiguous int `json:"ambiguous"` Ambiguous int `json:"ambiguous"`
NotFound int `json:"not_found"` NotFound int `json:"not_found"`
WrongHouse int `json:"wrong_house"` WrongHouse int `json:"wrong_house"`
WrongCity int `json:"wrong_city"`
WrongStreet int `json:"wrong_street"` WrongStreet int `json:"wrong_street"`
Lossy int `json:"lossy"` Lossy int `json:"lossy"`
} }
@@ -77,19 +72,11 @@ func Run(resolver *match.Resolver, cases []Case) Report {
case address.StatusFound: case address.StatusFound:
rp := address.Parse(res.Address) rp := address.Parse(res.Address)
equivalent := rp.StreetKey == exp.StreetKey && sameUnit(rp, exp) equivalent := rp.StreetKey == exp.StreetKey && sameUnit(rp, exp)
sameCity := c.ExpectCity == "" || res.City == c.ExpectCity
switch { switch {
case equivalent && sameCity: case equivalent:
st.Exact++ st.Exact++
rep.Exact++ rep.Exact++
exact = true exact = true
case !sameCity:
// A wrong city is always a safety violation, even when the
// case is allowed to be lossy: the address text can be
// identical across cities.
st.WrongCity++
rep.Wrong++
rep.WrongExamples = appendExample(rep.WrongExamples, c, res.Address)
case !c.RequireExact: case !c.RequireExact:
// Missing information resolved to a different real address: // Missing information resolved to a different real address:
// expected, not a safety violation. // expected, not a safety violation.
@@ -124,17 +111,14 @@ func Run(resolver *match.Resolver, cases []Case) Report {
return rep return rep
} }
// sameUnit compares house units, including the structure: two buildings that // sameUnit compares house units, ignoring the structure component.
// differ only by "стр." are different addresses.
func sameUnit(a, b address.ParsedAddress) bool { func sameUnit(a, b address.ParsedAddress) bool {
return a.CityKey == b.CityKey && return a.House == b.House &&
a.House == b.House &&
a.HouseFrac == b.HouseFrac && a.HouseFrac == b.HouseFrac &&
a.HouseLetter == b.HouseLetter && a.HouseLetter == b.HouseLetter &&
a.Building == b.Building && a.Building == b.Building &&
a.BuildingFrac == b.BuildingFrac && a.BuildingFrac == b.BuildingFrac &&
a.BuildingLetter == b.BuildingLetter && a.BuildingLetter == b.BuildingLetter
a.Structure == b.Structure
} }
func appendExample(examples []string, c Case, got string) []string { func appendExample(examples []string, c Case, got string) []string {
@@ -201,24 +185,16 @@ func perturbations(e address.Entry, rng *rand.Rand) []Case {
var out []Case var out []Case
add := func(query, category string, requireExact bool) { add := func(query, category string, requireExact bool) {
if query != "" && query != base { if query != "" && query != base {
out = append(out, Case{ out = append(out, Case{Query: query, Expect: base, Category: category, RequireExact: requireExact})
Query: query, Expect: base, ExpectCity: e.City,
Category: category, RequireExact: requireExact,
})
} }
} }
// Baseline: the canonical address itself must always resolve exactly. // Baseline: the canonical address itself must always resolve exactly.
out = append(out, Case{ out = append(out, Case{Query: base, Expect: base, Category: "baseline", RequireExact: true})
Query: base, Expect: base, ExpectCity: e.City,
Category: "baseline", RequireExact: true,
})
add(strings.ToLower(base), "case_lower", true) add(strings.ToLower(base), "case_lower", true)
add(strings.ToUpper(base), "case_upper", true) add(strings.ToUpper(base), "case_upper", true)
if e.City != "" { add("г. Новосибирск, "+base, "city_prefix", true)
add("г. "+e.City+", "+base, "city_prefix", true)
}
add(insertDom(base, e.House), "filler_dom", true) add(insertDom(base, e.House), "filler_dom", true)
add(corruptPunctuation(base), "punctuation", true) add(corruptPunctuation(base), "punctuation", true)
add(addYo(base), "yo", true) add(addYo(base), "yo", true)
@@ -236,9 +212,7 @@ func perturbations(e address.Entry, rng *rand.Rand) []Case {
add(strings.Replace(base, "к"+e.Building, "корпус "+e.Building, 1), "corpus_synonym", true) add(strings.Replace(base, "к"+e.Building, "корпус "+e.Building, 1), "corpus_synonym", true)
} }
if hasStructure(base) { if hasStructure(base) {
// Dropping "стр. N" discards information: the base house may or may add(dropStructure(base), "structure_drop", true)
// not exist on its own, so a different real address is acceptable.
add(dropStructure(base), "structure_drop", false)
} }
if q := mutateLongestToken(base, rng); q != "" { if q := mutateLongestToken(base, rng); q != "" {
add(q, "typo", true) add(q, "typo", true)
-37
View File
@@ -4,9 +4,6 @@ import (
"math/rand" "math/rand"
"testing" "testing"
"unicode/utf8" "unicode/utf8"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
) )
// TestMoveTypeFront guards against the slice-aliasing regression that made the // TestMoveTypeFront guards against the slice-aliasing regression that made the
@@ -50,37 +47,3 @@ func TestMutateOnlyNameTokens(t *testing.T) {
} }
} }
} }
// TestRunDetectsWrongCity guards the multi-city safety net: canonical address
// text is identical across cities, so a found result must be checked against
// the expected city, not only against street/house text.
func TestRunDetectsWrongCity(t *testing.T) {
parser := address.NewParser(address.NewDictionary([]address.City{{Name: "Бердск"}}), "")
p := parser.ParseInCity("Ленина улица 5", "Бердск")
entries := []address.Entry{{
ID: "b1", Address: "Ленина улица 5",
City: p.City, CityKey: p.CityKey,
Ordinal: p.Ordinal, NameKey: p.NameKey, StreetKey: p.StreetKey, House: p.House,
}}
resolver := match.NewResolverWithOptions(
match.NewIndex("test", entries, match.DefaultOptions()),
nil,
match.ResolverOptions{Parser: parser},
)
cases := []Case{
{Query: "Ленина 5", Expect: "Ленина улица 5", ExpectCity: "Новосибирск", Category: "wrong_city", RequireExact: true},
{Query: "Ленина 5", Expect: "Ленина улица 5", ExpectCity: "Бердск", Category: "right_city", RequireExact: true},
}
rep := Run(resolver, cases)
if got := rep.Categories["wrong_city"]; got.WrongCity != 1 || got.Exact != 0 {
t.Fatalf("wrong_city = %+v, want WrongCity=1 Exact=0", got)
}
if got := rep.Categories["right_city"]; got.Exact != 1 || got.WrongCity != 0 {
t.Fatalf("right_city = %+v, want Exact=1 WrongCity=0", got)
}
if rep.Wrong != 1 {
t.Fatalf("report Wrong = %d, want 1", rep.Wrong)
}
}
+1 -1
View File
@@ -29,7 +29,7 @@ func TestRealRegistryParseRate(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("read registry: %v", err) t.Fatalf("read registry: %v", err)
} }
entries, stats := Build(sources, BuildOptions{}) entries, stats := Build(sources)
rate := 0.0 rate := 0.0
if stats.Total > 0 { if stats.Total > 0 {
+13 -69
View File
@@ -20,9 +20,6 @@ type Source struct {
ID string ID string
Address string Address string
District string District string
// City is optional: when the dump is already split per city it may be
// omitted and supplied as the default city to Build.
City string
} }
// Stats summarises a build and, crucially, surfaces rows the parser could not // Stats summarises a build and, crucially, surfaces rows the parser could not
@@ -34,24 +31,13 @@ type Stats struct {
Unparsed int `json:"unparsed"` Unparsed int `json:"unparsed"`
Duplicates int `json:"duplicates"` Duplicates int `json:"duplicates"`
DistinctStreets int `json:"distinct_streets"` DistinctStreets int `json:"distinct_streets"`
Cities map[string]int `json:"cities,omitempty"`
UnparsedSample []string `json:"unparsed_sample,omitempty"` UnparsedSample []string `json:"unparsed_sample,omitempty"`
} }
// BuildOptions controls how source rows become canonical entries.
type BuildOptions struct {
// Parser defaults to address.DefaultParser().
Parser *address.Parser
// DefaultCity is used for rows that neither carry a city field nor name a
// city in the address. Leave it empty for multi-city dumps where every row
// is expected to identify its city.
DefaultCity string
}
// ReadBulkNDJSON reads an Elasticsearch Bulk NDJSON dump: pairs of an action // ReadBulkNDJSON reads an Elasticsearch Bulk NDJSON dump: pairs of an action
// line ({"index":{"_id":...}}) followed by a source line // line ({"index":{"_id":...}}) followed by a source line
// ({"address":...,"district_name":...,"city":...}). Bare source lines (no // ({"address":...,"district_name":...}). Bare source lines (no action line)
// action line) are also accepted, which makes plain JSONL usable too. // are also accepted, which makes plain JSONL usable too.
func ReadBulkNDJSON(r io.Reader) ([]Source, error) { func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
sc := bufio.NewScanner(r) sc := bufio.NewScanner(r)
// 1 MiB per line is far beyond any realistic address row. // 1 MiB per line is far beyond any realistic address row.
@@ -89,7 +75,6 @@ func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
if d, ok := obj["district_name"]; ok { if d, ok := obj["district_name"]; ok {
_ = json.Unmarshal(d, &src.District) _ = json.Unmarshal(d, &src.District)
} }
src.City = firstString(obj, "city", "city_name")
sources = append(sources, src) sources = append(sources, src)
pendingID = "" pendingID = ""
} }
@@ -100,40 +85,17 @@ func ReadBulkNDJSON(r io.Reader) ([]Source, error) {
return sources, nil return sources, nil
} }
// firstString returns the first non-empty string field among names.
func firstString(obj map[string]json.RawMessage, names ...string) string {
for _, name := range names {
raw, ok := obj[name]
if !ok {
continue
}
var s string
if err := json.Unmarshal(raw, &s); err == nil && strings.TrimSpace(s) != "" {
return s
}
}
return ""
}
// Build parses every source row into a canonical Entry and reports statistics. // Build parses every source row into a canonical Entry and reports statistics.
// Rows without a house number are still indexed for street lookups but counted // Rows without a house number are still indexed for street lookups but counted
// as unparsed, since a registry row without a house is suspicious. // as unparsed, since a registry row without a house is suspicious.
func Build(sources []Source, opts BuildOptions) ([]address.Entry, Stats) { func Build(sources []Source) ([]address.Entry, Stats) {
parser := opts.Parser
if parser == nil {
parser = address.NewParser(address.DefaultDictionary(), opts.DefaultCity)
}
entries := make([]address.Entry, 0, len(sources)) entries := make([]address.Entry, 0, len(sources))
streets := make(map[string]struct{}) streets := make(map[string]struct{})
seen := make(map[string]struct{}) seen := make(map[string]struct{})
stats := Stats{Total: len(sources), Cities: make(map[string]int)} stats := Stats{Total: len(sources)}
for _, src := range sources { for _, src := range sources {
// Precedence: an explicit source field wins over a city found in the p := address.Parse(src.Address)
// address text; the parser's default city is the last resort. Passing
// an empty hint is what lets the parser apply that precedence.
cityHint := strings.TrimSpace(src.City)
p := parser.ParseInCity(src.Address, cityHint)
if p.NameKey == "" { if p.NameKey == "" {
stats.Unparsed++ stats.Unparsed++
appendSample(&stats, src.Address) appendSample(&stats, src.Address)
@@ -148,11 +110,9 @@ func Build(sources []Source, opts BuildOptions) ([]address.Entry, Stats) {
} }
entry := address.Entry{ entry := address.Entry{
ID: namespacedID(src.ID, p.CityKey), ID: src.ID,
Address: src.Address, Address: src.Address,
District: src.District, District: src.District,
City: p.City,
CityKey: p.CityKey,
Ordinal: p.Ordinal, Ordinal: p.Ordinal,
NameKey: p.NameKey, NameKey: p.NameKey,
NameDisplay: p.NameDisplay, NameDisplay: p.NameDisplay,
@@ -168,8 +128,9 @@ func Build(sources []Source, opts BuildOptions) ([]address.Entry, Stats) {
} }
// The source registry contains rows that differ only by "стр." // The source registry contains rows that differ only by "стр."
// (structure). They are distinct buildings, so structure is part of // (structure). Since structure does not affect which house an outage
// the de-duplication key and an exact structure query can be honoured. // belongs to, drop such duplicates rather than surfacing them as
// spurious ambiguity at query time.
if key := dedupKey(entry); key != "" { if key := dedupKey(entry); key != "" {
if _, ok := seen[key]; ok { if _, ok := seen[key]; ok {
stats.Duplicates++ stats.Duplicates++
@@ -180,41 +141,24 @@ func Build(sources []Source, opts BuildOptions) ([]address.Entry, Stats) {
entries = append(entries, entry) entries = append(entries, entry)
streets[p.StreetKey] = struct{}{} streets[p.StreetKey] = struct{}{}
stats.Cities[cityLabel(p.City)]++
} }
stats.DistinctStreets = len(streets) stats.DistinctStreets = len(streets)
sort.Strings(stats.UnparsedSample) sort.Strings(stats.UnparsedSample)
return entries, stats return entries, stats
} }
// namespacedID keeps source ids unique when several city registries are merged // dedupKey identifies the same house unit within the same street, ignoring the
// into one database. Source systems often number houses independently. // structure component.
func namespacedID(id, cityKey string) string {
if id == "" || cityKey == "" {
return id
}
return cityKey + "/" + id
}
// dedupKey identifies the same house unit within the same city and street,
// including its structure and building components.
func dedupKey(e address.Entry) string { func dedupKey(e address.Entry) string {
if e.House == "" { if e.House == "" {
return "" return ""
} }
return strings.Join([]string{ return strings.Join([]string{
e.CityKey, e.StreetKey, e.House, e.HouseFrac, e.HouseLetter, e.StreetKey, e.House, e.HouseFrac, e.HouseLetter,
e.Building, e.BuildingFrac, e.BuildingLetter, e.Structure, e.Building, e.BuildingFrac, e.BuildingLetter,
}, "|") }, "|")
} }
func cityLabel(city string) string {
if city == "" {
return "(none)"
}
return city
}
func appendSample(s *Stats, address string) { func appendSample(s *Stats, address string) {
const maxSamples = 20 const maxSamples = 20
if len(s.UnparsedSample) < maxSamples { if len(s.UnparsedSample) < maxSamples {
+3 -73
View File
@@ -3,13 +3,11 @@ package seed
import ( import (
"strings" "strings"
"testing" "testing"
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
) )
func TestReadBulkNDJSON(t *testing.T) { func TestReadBulkNDJSON(t *testing.T) {
input := `{"index": {"_index": "address_registry", "_id": "a1"}} input := `{"index": {"_index": "address_registry", "_id": "a1"}}
{"address": "Станционная улица 1/1", "district_name": "Ленинский", "city": "Новосибирск"} {"address": "Станционная улица 1/1", "district_name": "Ленинский"}
{"index": {"_id": "a2"}} {"index": {"_id": "a2"}}
{"address": "Искитимская улица 5", "district_name": "Кировский"} {"address": "Искитимская улица 5", "district_name": "Кировский"}
@@ -22,7 +20,7 @@ func TestReadBulkNDJSON(t *testing.T) {
if len(src) != 3 { if len(src) != 3 {
t.Fatalf("got %d sources, want 3", len(src)) t.Fatalf("got %d sources, want 3", len(src))
} }
if src[0].ID != "a1" || src[0].District != "Ленинский" || src[0].City != "Новосибирск" { if src[0].ID != "a1" || src[0].District != "Ленинский" {
t.Errorf("first source = %+v", src[0]) t.Errorf("first source = %+v", src[0])
} }
if src[2].Address != "Третья улица 7" { if src[2].Address != "Третья улица 7" {
@@ -36,7 +34,7 @@ func TestBuild(t *testing.T) {
{ID: "2", Address: "Искитимская улица 5"}, {ID: "2", Address: "Искитимская улица 5"},
{ID: "3", Address: "Станционная улица"}, // no house -> unparsed {ID: "3", Address: "Станционная улица"}, // no house -> unparsed
} }
entries, stats := Build(sources, BuildOptions{}) entries, stats := Build(sources)
if stats.Total != 3 { if stats.Total != 3 {
t.Errorf("Total = %d, want 3", stats.Total) t.Errorf("Total = %d, want 3", stats.Total)
@@ -55,71 +53,3 @@ func TestBuild(t *testing.T) {
t.Errorf("DistinctStreets = %d, want 2", stats.DistinctStreets) t.Errorf("DistinctStreets = %d, want 2", stats.DistinctStreets)
} }
} }
func TestBuildKeepsCitiesApart(t *testing.T) {
sources := []Source{
{ID: "1", Address: "Ленина улица 5", City: "Новосибирск"},
{ID: "1", Address: "Ленина улица 5", City: "Бердск"},
{ID: "2", Address: "Ленина улица 7", City: "Новосибирск"},
}
entries, stats := Build(sources, BuildOptions{})
if len(entries) != 3 {
t.Fatalf("entries = %d, want 3: %+v", len(entries), entries)
}
if stats.DistinctStreets != 2 {
t.Errorf("DistinctStreets = %d, want 2 (same street in two cities)", stats.DistinctStreets)
}
if entries[0].StreetKey == entries[1].StreetKey {
t.Errorf("streets from different cities share a key: %q", entries[0].StreetKey)
}
if entries[0].ID == entries[1].ID {
t.Errorf("source ids collided across cities: %q", entries[0].ID)
}
if stats.Cities["Новосибирск"] != 2 || stats.Cities["Бердск"] != 1 {
t.Errorf("city stats = %v", stats.Cities)
}
}
func TestBuildAppliesDefaultCity(t *testing.T) {
sources := []Source{{ID: "1", Address: "Станционная улица 1/1"}}
entries, _ := Build(sources, BuildOptions{DefaultCity: "Новосибирск"})
if len(entries) != 1 {
t.Fatalf("entries = %d", len(entries))
}
if entries[0].City != "Новосибирск" || entries[0].CityKey != "новосибирск" {
t.Errorf("city = %q / %q", entries[0].City, entries[0].CityKey)
}
}
func TestBuildKeepsStructuresDistinct(t *testing.T) {
sources := []Source{
{ID: "1", Address: "Станционная улица 1"},
{ID: "2", Address: "Станционная улица 1 стр. 2"},
}
entries, stats := Build(sources, BuildOptions{})
if len(entries) != 2 {
t.Fatalf("entries = %d, want 2 (structure is part of identity)", len(entries))
}
if stats.Duplicates != 0 {
t.Errorf("Duplicates = %d, want 0", stats.Duplicates)
}
if entries[0].Structure == entries[1].Structure {
t.Errorf("structures collapsed: %+v", entries)
}
}
func TestBuildParsesCityFromAddress(t *testing.T) {
parser := address.NewParser(address.NewDictionary([]address.City{
{Name: "Новосибирск", Aliases: []string{"нск", "новосибирская область"}},
}), "")
sources := []Source{{ID: "1", Address: "нск, Станционная улица 1/1"}}
entries, _ := Build(sources, BuildOptions{Parser: parser})
if len(entries) != 1 {
t.Fatalf("entries = %d", len(entries))
}
if entries[0].City != "Новосибирск" || entries[0].NameKey != "станционная" {
t.Errorf("entry = %+v", entries[0])
}
}
-115
View File
@@ -1,115 +0,0 @@
// Package addrkit is the public library facade of the address normalizer.
//
// The service (cmd/addrkitd) and the CLI use the same domain packages through
// this API. Another project in a different repository can import this package
// directly instead of calling the HTTP service; nothing here does I/O, opens a
// database or depends on the service configuration.
//
// Example:
//
// parser := addrkit.NewParser(
// addrkit.NewDictionary([]addrkit.City{
// {Name: "Новосибирск", Aliases: []string{"нск"}},
// }),
// "",
// )
// parsed := parser.Parse("нск, Ленина 5")
//
// index := addrkit.NewIndex("2026-10-01", entries, addrkit.DefaultOptions())
// resolver := addrkit.NewResolver(index, nil)
// result := resolver.Resolve(context.Background(), "нск, Ленина 5")
package addrkit
import (
"git.totmin.ru/en2zmax/addrkit/internal/domain/address"
"git.totmin.ru/en2zmax/addrkit/internal/domain/match"
)
// Core value types.
type (
// City is one entry of the configurable city dictionary.
City = address.City
// ParsedAddress is the structured form of a free-form address.
ParsedAddress = address.ParsedAddress
// Entry is one canonical registry row.
Entry = address.Entry
// Status is the machine-readable outcome of a resolution.
Status = address.Status
)
// Resolution statuses.
const (
StatusFound = address.StatusFound
StatusPartial = address.StatusPartial
StatusAmbiguous = address.StatusAmbiguous
StatusNotFound = address.StatusNotFound
)
// Dictionary is the immutable vocabulary used by the parser.
type Dictionary = address.Dictionary
// NewDictionary builds a dictionary with the built-in vocabulary and cities.
func NewDictionary(cities []City) *Dictionary {
return address.NewDictionary(cities)
}
// Parser turns free-form address strings into structured addresses.
type Parser = address.Parser
// NewParser builds a parser. defaultCity may be empty.
func NewParser(dict *Dictionary, defaultCity string) *Parser {
return address.NewParser(dict, defaultCity)
}
// Parse uses the shared city-less parser.
func Parse(raw string) ParsedAddress {
return address.Parse(raw)
}
// Matching and resolution.
type (
// Options tunes the index.
Options = match.Options
// Candidate is one plausible match.
Candidate = match.Candidate
// Result is the outcome of a resolution.
Result = match.Result
// Rewriter is the optional query rewriter (for example an LLM).
Rewriter = match.Rewriter
// Index is an immutable in-memory registry index.
Index = match.Index
// Resolver resolves parsed addresses against an index.
Resolver = match.Resolver
)
// ResolverOptions tunes optional resolver behaviour.
type ResolverOptions struct {
// Parser defaults to a city-less parser.
Parser *Parser
// OnRewriteError observes failures of the optional rewriter. The resolver
// always keeps the deterministic result.
OnRewriteError func(error)
}
// DefaultOptions returns the conservative matcher defaults.
func DefaultOptions() Options {
return match.DefaultOptions()
}
// NewIndex builds an index from canonical entries.
func NewIndex(version string, entries []Entry, opts Options) *Index {
return match.NewIndex(version, entries, opts)
}
// NewResolver builds a resolver with the default parser.
func NewResolver(index *Index, rewriter Rewriter) *Resolver {
return match.NewResolver(index, rewriter)
}
// NewResolverWithOptions builds a resolver with explicit options.
func NewResolverWithOptions(index *Index, rewriter Rewriter, opts ResolverOptions) *Resolver {
return match.NewResolverWithOptions(index, rewriter, match.ResolverOptions{
Parser: opts.Parser,
OnRewriteError: opts.OnRewriteError,
})
}
-46
View File
@@ -1,46 +0,0 @@
package addrkit_test
import (
"context"
"testing"
"git.totmin.ru/en2zmax/addrkit/pkg/addrkit"
)
// TestPublicAPI exercises the facade exactly the way another repository would:
// parse a city-aware query and resolve it against an in-memory index.
func TestPublicAPI(t *testing.T) {
dict := addrkit.NewDictionary([]addrkit.City{
{Name: "Новосибирск", Aliases: []string{"нск"}},
{Name: "Бердск"},
})
parser := addrkit.NewParser(dict, "")
entries := make([]addrkit.Entry, 0, 2)
for _, city := range []string{"Новосибирск", "Бердск"} {
p := parser.ParseInCity("Ленина улица 5", city)
entries = append(entries, addrkit.Entry{
ID: city, Address: "Ленина улица 5",
City: p.City, CityKey: p.CityKey,
Ordinal: p.Ordinal, NameKey: p.NameKey, NameDisplay: p.NameDisplay,
StreetType: p.StreetType, StreetKey: p.StreetKey,
House: p.House,
})
}
resolver := addrkit.NewResolverWithOptions(
addrkit.NewIndex("2026-10-01", entries, addrkit.DefaultOptions()),
nil,
addrkit.ResolverOptions{Parser: parser},
)
res := resolver.ResolveInCity(context.Background(), "нск, Ленина 5", "")
if res.Status != addrkit.StatusFound || res.City != "Новосибирск" {
t.Fatalf("result = %+v", res)
}
ambiguous := resolver.Resolve(context.Background(), "Ленина 5")
if ambiguous.Status != addrkit.StatusAmbiguous {
t.Fatalf("cityless result = %+v", ambiguous)
}
}
+31 -53
View File
@@ -1,12 +1,10 @@
openapi: 3.1.0 openapi: 3.1.0
info: info:
title: addrkit title: addrkit
version: "2.0.0" version: "1.0.0"
description: > description: >
Детерминированный многогородской сервис нормализации и поиска адресов. Детерминированный сервис нормализации и поиска адресов. Ответ всегда
Ответ всегда берётся из реестра; LLM не участвует в пути корректности. берётся из реестра; LLM не участвует в пути корректности.
Сервис не знает ни про отключения, ни про другие события: это отдельная
задача, которая может использовать addrkit как нормализатор.
servers: servers:
- url: http://localhost:6171 - url: http://localhost:6171
paths: paths:
@@ -23,15 +21,7 @@ paths:
properties: properties:
address: address:
type: string type: string
maxLength: 512 example: "2-я Станционная улица 1/1 стр."
example: "нск, 2-я Станционная улица 1/1 стр."
city:
type: string
maxLength: 128
description: >
Необязательная подсказка. Переопределяет город, найденный
в тексте адреса.
example: "Новосибирск"
responses: responses:
"200": "200":
description: Результат сопоставления description: Результат сопоставления
@@ -41,54 +31,26 @@ paths:
$ref: "#/components/schemas/ResolveResult" $ref: "#/components/schemas/ResolveResult"
"400": "400":
description: Некорректный запрос description: Некорректный запрос
/v1/city: /v1/outages:
post: post:
summary: Нормализовать только название города summary: Отключения по адресу (после строгого сопоставления дома)
description: >
Отдельный сценарий для проектов, которым нужны названия городов, а не
улицы. Возвращает канонический город по имени или алиасу.
requestBody: requestBody:
required: true required: true
content: content:
application/json: application/json:
schema: schema:
type: object type: object
required: [query] required: [address]
properties: properties:
query: address:
type: string type: string
maxLength: 128
example: "нск"
responses: responses:
"200": "200":
description: Результат description: Канонический адрес и отключения
content: content:
application/json: application/json:
schema: schema:
type: object $ref: "#/components/schemas/OutagesResult"
required: [status]
properties:
status:
type: string
enum: [found, not_found]
city: { type: string }
"400":
description: Некорректный запрос
/v1/cities:
get:
summary: Канонические города, присутствующие в индексе
responses:
"200":
description: Список городов
content:
application/json:
schema:
type: object
required: [cities]
properties:
cities:
type: array
items: { type: string }
/v1/healthz: /v1/healthz:
get: get:
summary: Живость summary: Живость
@@ -116,16 +78,13 @@ components:
schemas: schemas:
Status: Status:
type: string type: string
enum: [found, partial, ambiguous, not_found] enum: [found, partial, ambiguous, not_found, portal_timeout, infra_error]
Candidate: Candidate:
type: object type: object
required: [address, score]
properties: properties:
address: { type: string } address: { type: string }
city: { type: string }
district: { type: string } district: { type: string }
house: { type: string } house: { type: string }
structure: { type: string }
score: { type: number, format: double } score: { type: number, format: double }
ResolveResult: ResolveResult:
type: object type: object
@@ -133,10 +92,29 @@ components:
properties: properties:
status: { $ref: "#/components/schemas/Status" } status: { $ref: "#/components/schemas/Status" }
address: { type: string } address: { type: string }
city: { type: string }
district: { type: string } district: { type: string }
confidence: { type: number, format: double } confidence: { type: number, format: double }
candidates: candidates:
type: array type: array
items: { $ref: "#/components/schemas/Candidate" } items: { $ref: "#/components/schemas/Candidate" }
data_version: { type: string } data_version: { type: string }
Outage:
type: object
properties:
type: { type: string }
system: { type: string }
address: { type: string }
start_date: { type: string }
end_date: { type: string }
action: { type: string }
OutagesResult:
type: object
properties:
status: { $ref: "#/components/schemas/Status" }
address: { type: string }
district: { type: string }
outages:
type: array
items: { $ref: "#/components/schemas/Outage" }
as_of: { type: string }
data_version: { type: string }
-1075
View File
File diff suppressed because it is too large Load Diff