config: split configs per subsystem, extends: inheritance, and a directory config loader #169
No reviewers
Labels
No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
art/ismap!169
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "config-locality"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Config locality: split configs per subsystem, add
extends:and a directory config loaderEditing Sweden's roads used to mean paging through a ~2100-line
conf/semap.yaml. It now meansopening
conf/se/07-roads.yaml, a small file — and increasingly, not opening a file at all butrunning a
yqone-liner.The layout
Reading a config path — a file or a directory of files merged in lexical order — is one function,
so a directory works in either position:
importer import conf/seandinclude: [conf/shared/contracts/]are the same mechanism.mergeConfigmergesinclude:(alatent bug), so a directory config keeps
include: [conf/common.yaml].extends:inheritance (importer/shared.go): a country layer that matches a shared contractinherits the contract's scaffold —
attr_names,attrs,generalization_strategy,stop_z,maxz— for the fields it leaves unset.inheritSharedScaffoldruns beforeapplyCountryinjects the country column, and
wireSharedLayersstill enforces agreement after, with theexact comparison against the contract's country-injected list (an earlier revision had
loosened this; with inheritance ordered correctly the strict check passes for all five configs,
so it is back). Genuinely per-country settings — roads'
merge_lines/merge_from_zoom, ferries'min_length_pixels— stay country-declared.Making the layout survive the tool that edits it
SQL blocks are literal (
query: |), never folded (query: >). This is what makesyqusablehere at all. A folded scalar is re-wrapped at 80 columns whenever anything writes the file, and one
yq -ionconf/se/07-roads.yamlmoved the text after a--comment onto its own line:183 blocks converted, verified three ways: every resolved config is identical outside the SQL
strings; all 155 SQL strings that changed differ only in whitespace and no change lands inside a
quoted literal (checked with a scanner that tracks quote state and skips
--comments); and all66 queries across the five configs
PREPAREagainst the live database, 0 failed. A test rejects afolded block so one cannot come back.
A file the loader would not read is an error, not a skip.
loadConfigDirfiltered on.yamlwhile the single-file loader and the CLI both accept
.yml, so renamingconf/se/07-roads.yamlto
.ymlremoved Sweden's entire roads layer andconfig-checkreportedOK — 14 layers.Lexical file order is the DAG's build order, so which files are read is the map:
.ymlis read,and anything else in a config directory fails loudly.
base.yamlis split the rest of the way. It was the largest file inconf/se— 635 lines, ofwhich
staging:alone was 419 — so "editing roads is a small file" was true while "adding astaging table" was still a 635-line read.
conf/shared/contracts.yaml(1082 lines, 17 contracts)becomes one file per contract, on the same argument: the roads contract is a thing you edit.
yqidioms are documented indocs/importer-cli.md, with the tool requirement in AGENTS.md:mikefarah/yq v4, not the python
yqthat wrapsjq— that one round-trips YAML through JSONand deletes every comment (65 of 138 lines in
conf/se/07-roads.yaml), and the per-layerreasoning next to the query it justifies is the review surface. Every documented command was run
before it was written down.
Safety
(through
loadConfig+prepareConfig) isreflect.DeepEqualto what the old single-fileloader produced, for all four countries, with the only difference being the
extends:fielditself. Re-verified after the
>→|conversion, the base.yaml split and the contracts split.NN-filenames encode, and losing a file was otherwise silent.
make build,make test,config-checkandcheck-queriespass for all five configs.make acceptance: Iceland 38/38, Sweden 44/44, Finland 49/49, Poland 44/45 — the one failure(
R120: every Poland source part becomes a classified footprint,context deadline exceededatits
timeout: 10) reproduces identically on unmodifiedmainand is not from this branch.docs/country-requirements.md: R17 described theconf/<cc>map.yamlnaming convention thischange replaces, and R29's check was
grep -n 'minz:' conf/<cc>map.yaml— a path that nolonger exists, so the check passed without looking at anything. Both fixed; R29 now greps
conf/recursively for a real key.Also
docs/superpowers/is removed — plan artifacts from a planning tool, not project documentation.Merges cleanly with #168; the two were built and tested merged together.
Four fixes to the config split, all of them about the same thing: making the layout hold up under the tool that is supposed to edit it. **A file the loader would not read is now an error, not a skip.** loadConfigDir filtered on `.yaml` while the single-file loader and the CLI both accept `.yml`, so renaming conf/se/07-roads.yaml to .yml removed Sweden's entire roads layer and config-check reported `OK — 14 layers`. Lexical file order is the DAG's build order, so which files are read *is* the map: `.yml` is read, and anything else in a config directory (a `.orig`, a stray note, a subdirectory) fails loudly. **SQL blocks are literal (`query: |`), not folded (`query: >`).** This is what makes `yq` usable here at all. A folded scalar is re-wrapped at 80 columns whenever anything writes the file, and one `yq -i` on conf/se/07-roads.yaml moved the text after a `--` comment onto its own line: WHEN 'överfart och underfart' THEN NULL -- both; drawn at normal level (docs/geo-primer.md) <- no longer a comment; now SQL 183 blocks converted. Verified equivalent three ways: every resolved config is byte-identical outside the SQL strings; all 155 SQL strings that changed differ only in whitespace, and no change lands inside a quoted literal (checked with a scanner that tracks quote state and skips `--` comments); and all 66 queries across the five configs PREPARE against the live database, 0 failed. A test now rejects a folded block so one cannot come back. **base.yaml is split the rest of the way.** It was the largest file in conf/se — 635 lines, of which `staging:` alone was 419 — so "editing Sweden's roads is a small file" was true while "adding a staging table" was still a 635-line read. It is now base.yaml (destination, prefix, country, include), downloads.yaml, staging.yaml and acceptance.yaml, the files the design named. conf/shared/contracts.yaml (1082 lines, 17 contracts) becomes one file per contract, which is the same argument: the roads contract is a thing you edit. Reading a config path — file or directory — is one function now, so `include:` takes a directory too and conf/common.yaml pulls the whole contract set in with one entry. **The attr_names check is strict again.** The split had loosened it to compare both sides with `country` removed; with the scaffold inherited before applyCountry runs, the original exact comparison against the contract's country-injected list passes for all five configs, so it is back and withoutCountry is gone. Also: the layer set and order per country is now asserted (it is what the NN- file names encode, and losing a file is silent otherwise), R17 describes the directory layout it is supposed to be enforcing, R29's check greps `conf/` recursively for a real key instead of a `conf/<cc>map.yaml` path that no longer exists — a check that cannot fail is worse than no check — and the docs point at the file a contract actually lives in.