tile tool: R121 decoded-tile check, standardized benchmarks, size validation (#139) #142
No reviewers
Labels
No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
art/ismap!142
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-139-tile-tool"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
#139: the tile tool — R121 decoded-tile check + standardized benchmarks + size validation
Closes #139
What this is
A read-only Go tool on the importer binary that serves tiles through the same
zxy_<layer>functions mapsrv calls (no HTTP) and adds the three mechanisms theissue scoped in from #127. It is the
importer tilessubcommand (the #129 CLI):importer tiles --check / --bench / --size <config>, with make targetstile-check/tile-bench/tile-size. The branch was rebased onto main after#129 (importer subcommands) landed mid-work, which is why the interface is a
subcommand rather than flat flags.
tiles --check— R121's decoded-tile half: per layer, serve one tile at thecoarsest and one at the finest zoom it draws, decode the MVT in Go, and assert
the tag keys equal the config contract's
attr_namesminusmvt_exclude(plus the machinery columns: the geometry, and
country). A leak(
part_keyoncover,rankonplaces) or a dropped attribute fails therun.
tiles --bench— the standardized benchmarks: the four capitals atz5/z8/z12/z14 plus a z5 whole-country view of each country (the z5 Stockholm
probe included), fetched as 9×9 grids through
zxy_<source>functions, JSONoutput with per-tile byte sizes and latencies.
tiles --size— per-layer/per-zoom heaviest-tile report, the before/afterinstrument #127 will measure against.
Make targets:
make tile-check,make tile-bench(savesbench/results/<date>_<commit>_tiles.json),make tile-size. Docs indocs/importer-cli.md("The tile tool"); the R121 and R83 requirement entries indocs/country-requirements.mdnow cite it.Design decision: why the importer, not mapsrv
The R121 check asserts against the layer contracts —
attr_names,mvt_exclude, theshared_layers:contracts from conf/common.yaml, and thecountrycolumnapplyCountryinjects. The importer is the only component thatparses and validates those (mapsrv's config knows source names, not layers), and
it already has the read-only-against-the-DB shape (
-acceptance-only) and thepool machinery. mapsrv cannot import the importer's structs (both are
package main), so a mapsrv-side tool would have had to re-implement configresolution — the exact duplication the tool is supposed to remove. The decoder
follows
scripts/mvt-decode.py(the project's reference walker) and thehand-rolled protobuf pattern mapsrv's
glyph_check_test.goestablished.Verification
R121 — all five configs pass (130/130 checks)
make tile-check(ismap, semap, plmap, fimap, world): every layer's decoded keysmatch its contract exactly. The two classic leaks are confirmed absent —
coverserves[cover_class, country](nopart_key),placesserves[name, importance, country](norank); every shared layer carriescountry.No layer had a missing function or a key mismatch on the live DB.
Sample (conf/ismap.yaml, 34/34 PASS):
The full 130-row table is one
make tile-checkaway; the only per-configdifference is that ismap.yaml adds its five non-shared layers (contours, peaks,
buildings, airports, airport_points) on top of the twelve shared contracts.
Benchmarks — live DB, JSON saved
bench/results/2026-08-16_683e950_tiles.json(this branch's HEAD is main's683e950; the numbers describe the live DB, as the issue says they should). 20combos × 81 tiles per source function.
zxy_mapwall time 11 s for the wholerun; the z5 whole-country grids are dominated by the ~2 MB Stockholm tiles
(p50 0 ms — most of the 81 tiles are empty ocean — p95 ~350–400 ms).
Heaviest single tile in the run:
zxy_mapz5 (17,8) = 2 385 623 bytes(852 ms).
Size validation — the offenders #127 will work on
make tile-size(full run, 48 s): 17 layers × every zoom they draw, top-5heaviest tiles per layer/zoom. The heaviest tile per layer:
coverz5 is the one to watch: it is 27× the next heaviest tile. The z5Stockholm 8 MB acceptance test still passes (zxy_map(5,17,9) = 1 841 744 bytes,
994 ms) — the historical runaway was fixed, and the tile tool is now what would
have caught it.
Honest limits
per zoom are measured); a tile that clips the middle of one giant polygon
with no feature centroid or bbox corner in it could be missed. The corner tile
of that polygon carries the same bytes and is reported, so the top-5 stays
approximately right — documented in the code and docs.
-tile-bench's 9×9 grids fetch a lot of empty-ocean tiles at z5 by design(that is what a real client sees panning); the per-tile JSON lets a graph
filter them.
coverage (coverage is R119's acceptance tests' job).
Files
importer/tiles.go— decoder, check, bench, size report (+importer/testdata/ferries_z5_17_9.hex, a real 3129-byte ST_AsMVT tile as the golden decode fixture)importer/tiles_test.go— decoder tests (hand-built + real tile), expected-keys logic, tile math; no DB neededimporter/main.go— the-tile-check/-tile-bench/-tile-sizeflagsMakefile—tile-check,tile-bench,tile-sizedocs/importer-cli.md— "The tile tool" section;docs/country-requirements.md— R121/R83 now cite the toolbench/results/2026-08-16_683e950_tiles.json— the committed benchmark run68c6d464a3to5bb050d946# #139: the tile tool — R121 decoded-tile check + standardized benchmarks + size validationto tile tool: R121 decoded-tile check, standardized benchmarks, size validation (#139)Review — the tile tool (#139)
Good tool, and the design decision is right: the R121 check asserts against the layer
contracts, and the importer is the only component that resolves them. Reviewed by
reading the diff and by using it —
tiles --checkandtiles --sizeboth ran againstthe live DB for every config while working #127 on top of this branch.
Fixed on the #127 branch (they were in files that branch already touches)
The docs and the code name flags the #129 rewrite deleted.
docs/importer-cli.mddocuments
-tile-size-top,-tile-size-zoomsand-tile-bench-funcs;tiles.go'sdoc comments say "implements -tile-check / -tile-bench / -tile-size"; and
parseTileSizeZoomsprints-tile-size-zooms %qin two error strings. Copying thedocumented command gives "flag provided but not defined".
The probe assigns features by bbox north-west corner, not centroid.
probeTileandcandidateTilesboth useST_XMin(geom)/ST_YMax(geom), while both code commentsand the doc say "the tile holding the most feature centroids". The corner is a fine
heuristic for "which tile has data" — the documentation is what's wrong.
tiles --checkcould not see a wrong tile. It compares key sets and nothing else.ST_AsMVT'sextentargument is separate fromST_AsMVTGeom's and both default to4096, so a tile carrying 1024-unit coordinates while declaring 4096 decodes perfectly,
has exactly the right keys, passes — and draws at quarter scale in its top-left corner.
That is not hypothetical: it is what #127's first version did, and this check is what
should have caught it before a browser did.
--checknow also asserts that the declaredextent equals
mvtExtent, and that no feature arrives without a geometry (tags withnothing to draw). Both are FAILs; both were silent before.
Worth your attention, not fixed
--sizehas no layer filter.--size-zoomsexists but there is no--size-layer,so measuring one layer's optimization means running all twelve, each doing a full-table
pass. That is the #127 workflow this tool exists for; I ended up writing raw SQL against
zxy_map_coverinstead of using it.candidateTileshas no LIMIT whileprobeTilehas 200 000.probeTilesamples;candidateTilesscans the whole table, for every layer, at every zoom in the window.make tile-sizeonly runs ismap so it is never felt, buttiles --size conf/semap.yamlwalks Sweden's 5.4M-row z13 cover partition.
A failed fetch is recorded as a 0-byte tile.
sizeOneZoomappendsmeasured{cx, cy, 0, 0}on error and the top-N sort then discards it silently. Ameasurement tool should not turn an error into a plausible-looking zero — at minimum it
should be counted and reported the way
--check's MISSING is.The committed bench JSON is 475 904 B / 26 654 lines. The other records in
bench/results/are 1–2 kB, andmake tile-benchadds one of these per run. The 3 240per-tile records are 97% of the file; the same report without the per-tile arrays is
14.5 kB. Either gitignore
*_tiles.json, drop theSetIndent, or commit the combosummaries and keep the per-tile detail local.
parseIntStrictis not strict.fmt.Sscanf(s, "%d", &n)stops at the firstnon-digit and reports success, so
--size-zooms 5xsilently means5, and--size-zooms 9-3xis rejected for the wrong reason.checkTileAtZoomre-implementstileTarget.tableAtZoominline — the same six linesof prefix derivation, ten lines below the method that already does it.
A
static: truelayer would be silently skipped.tileTargetsonly skipsl.Shared != nil; for a static layertableName(maxz)has no_<z>suffix, so theprefix derivation builds a table name that does not exist and the layer reports EMPTY
("no rows in … to probe") rather than being checked or deliberately skipped. Latent —
no config declares one today — but EMPTY is not fatal, so it would never be noticed.
Smaller things.
decodeMVTTilenever returns a non-nil error, and three call sitescheck it.
probeTilecomputesn := math.Pow(2, z)and passes it toSprintfas anargument the format string never references.
benchComboscalls Sweden's whole-countryprobe
stockholm-whole-country(62.4/15.6, the middle of the country) alongsideiceland-,poland-andfinland-whole-country— and that label is in the committedJSON.