art/ismap
1
0
Fork 0
Code Issues 56 Pull requests 1 Projects Releases Packages Wiki Activity Actions

#108: Finland peaks — DEM-derived spot heights from MML dem25m #205

Open
art wants to merge 2 commits from issue-108-fi-peaks into main
pull from: issue-108-fi-peaks
merge into: art:main
Conversation 0 Commits 2 Files changed 14 +1492 -44
art commented 2026-08-21 07:50:27 +00:00
Owner
Copy link

Closes #108

Finland: peaks — DEM-derived spot heights from MML dem25m

The source decision

DEM-derived spot heights from MML's dem25m national elevation model. The issue
asked for a source decision before a layer; the decision is forced by the contract
and the data:

  • MTK korkeuspiste (52210) is empty in the 2025 delivery (measured on disk: 0
    rows in MTK-muut), so the surveyed spot-height class is unavailable.
  • Named fells from paikannimi cannot fill the peaks contract. Two independent
    dead ends: the register has no elevation column (verified in the staging
    schema — R120 requires a valid, non-NULL elevation, and the style renders the
    "NNN m" label from it), and the shared contract has no name column by design
    (conf/shared/contracts/19-peaks.yaml), so the register's names could not be
    carried anyway; wireSharedLayers enforces the contract's attribute list exactly.
  • The DEM path satisfies everything the contract asks — same layer shape as
    Iceland (haedarpunktar) and Sweden (hojdpunkt): geom + elevation, no name,
    elevation-thinned at coarse zoom — and it is national data. This is the issue's
    own recommended default ("peaks = local maxima above a threshold").

So Finland's peaks are the local maxima of MML's 25 m elevation model (the
dem25m COG delivery on Paituli: 84 tiles, 802 MB, EPSG:2393, 16-bit values in
decimetres). The named-fell register remains what places already uses for names;
it is not a peaks source.

The mechanism: importer/dem.go

The importer reads data files natively — no ogr2ogr, no gdal service — and had no
raster reader. dem.go is its first: it parses the tiled 16-bit LZW COGs
(TIFF IFD + geokeys + a faithful TIFF-LZW "early change" decoder, which Go's
stdlib compress/lzw does not implement), decodes the grid, and finds local maxima.

Seam handling is the part that had to be right. Consecutive tiles overlap by
exactly one cell (80 025 m extent on an 80 000 m grid), so a naive per-tile scan
would both miss peaks near seams and invent false ones where a higher neighbour
sits just across the seam. The reader decodes each tile with a radius-cell halo
from its eight neighbours, and a peak is emitted only for cells in the tile's own
extent (the first 3200 cells in each axis) — every cell belongs to exactly one
tile, so no peak is duplicated or lost. The naive per-tile prototype over-counted
by ~24%; the halo version is the honest 26 296.

Parameters (measured over the whole country): radius: 20 cells (500 m
de-clustering — one summit marker per ridge, not one per 25 m cell) and
min_elevation: 200 m (the "above the lowland" floor — below it, Finland's flat
lake plateau produces drumlins, not peaks). Result: 26 296 peaks ≥ 200 m at z14,
7 991 ≥ 300 m at z13, 885 ≥ 500 m at z11 — the same order as Sweden's 33.8k
surveyed spot heights. The coarse z11 floor (≥500 m) matches Iceland's and
Sweden's (R31's cadence).

Config

  • conf/fi/downloads.yaml — 84 dem25m tile entries (one URL per path, the
    mechanism's shape; the comment records the R5 size).
  • conf/fi/staging.yaml — fi_dem_peaks staging: dem_peaks: {radius: 20, min_elevation: 200} on a glob, source_srid: 2393, srs: EPSG:3857.
  • conf/fi/16-peaks.yaml — the layer: extends: peaks, reads fi_dem_peaks,
    zoom filters 300/500, and three acceptance tests (count pin + no NULLs, Halti's
    summit cell ≥ 1300 m, the R31 z11 floor).
  • importer/fetch.go — neededFiles now glob-matches load globs against
    downloads, so a scoped --layer peaks run fetches the 84 tiles.
  • Shared contract comment updated to record that Finland landed as a DEM partition
    and why name stays out for every country.
  • Docs: docs/country-finland.md (catalogue line, source table, known-gaps item),
    docs/architecture.md (dem.go in the source map), docs/workstream-a.md.

Verification against the live database

  • make build, make test — green (plus a new TestLZWDecodeTiff round-trip and
    TestReadDEMPeaks against a synthetic COG; TestNeededFilesGlobMatchesDownloads
    covers the fetch change).

  • importer config-check conf/fi — OK (16 layers, 19 shared contracts, 57 tests).

  • Import: importer import conf/fi/ --layer peaks — 84 tiles fetched through
    the importer's own download path (fetchmeta sidecars written, re-run is a no-op),
    staging built in ~104 s, layer swapped live. Per-zoom row counts:

    zoom fi peaks elevation range NULLs
    z11 885 500–1310 0
    z12 7 991 300–1310 0
    z13 7 991 300–1310 0
    z14 26 296 200–1310 0

  • Partitions at every zoom (map_peaks_11_fi … map_peaks_14_fi), all with gist
    indexes, all SRID 3857 (matching is/se).

  • Decoded tile over Halti (14/9160/3757):
    scripts/mvt-decode.py → layer peaks, 1 feature, elevation=1310, country=fi.
    Halti's summit cell reads 1310 m (the 25 m DEM's value for the summit cell; the
    official height is 1324 m).

  • Real-browser render over CDP (scripts/cdp-map-check.mjs): Halti shows 1
    peak at 1310 m; Saana (Kilpisjärvi) shows 1 peak at 1020 m; Helsinki shows 0
    peaks at z11 — the negative control (the 200 m floor correctly leaves the flat
    south marker-free) and the positive controls (the fells present) both hold.

  • Acceptance tally (per-layer + shared-contract runs, pasted from the run):

    layer peaks (fi) / R120: every Finland DEM peak made it through, with a valid elevation — PASS (peaks=26296, source_peaks=26296, null_elevation=0)
layer peaks (fi) / R120: the highest Finland peak is Halti's summit cell — PASS (max_elevation=1310)
layer peaks (fi) / R31: the z11 floor holds only the ≥500 m summits — PASS (peaks=885, below_500=0)
shared layer peaks / R119: no peaks partition is empty at any zoom z11–z14 — PASS (partitions=12)
shared layer peaks / R120: every country's peaks carry a valid elevation — PASS (countries=3)
config / R105: every Finland staging table is nonzero after load — PASS (tables=18)

    make acceptance runs green except a pre-existing Poland buildings test that
    times out at 10 s against the 5.6 M-row table even on a clean run — unrelated to
    this change.

Notes / caveats

  • The peaks are algorithmic, not surveyed. DEM local maxima above a floor are
    a different population from a cartographer's spot-height survey, which is why the
    contract keeps the door open to a later named-fell layer (a separate shape, not a
    fake partition — the R13 sibling-contract idiom). Nothing here prevents that;
    the register is still the places source.
  • Halti reads 1310 m, not 1324 m: that is the 25 m DEM's own summit-cell value
    (a 25 m cell averages the summit), and the layer's job is to carry the DEM's
    reading, not correct it.
  • The DEM reader holds all 84 tiles in memory (~840 MB peak) for the one-shot
    staging build — the halo scheme needs every tile's header up front. Fine on this
    machine; noted here rather than silently streamed.
  • The Poland buildings acceptance timeout noted above is pre-existing and unrelated.
Closes #108 --- # Finland: peaks — DEM-derived spot heights from MML dem25m ## The source decision **DEM-derived spot heights from MML's `dem25m` national elevation model.** The issue asked for a source decision before a layer; the decision is forced by the contract and the data: - **MTK `korkeuspiste` (52210) is empty** in the 2025 delivery (measured on disk: 0 rows in `MTK-muut`), so the surveyed spot-height class is unavailable. - **Named fells from `paikannimi` cannot fill the peaks contract.** Two independent dead ends: the register has **no elevation column** (verified in the staging schema — R120 requires a valid, non-NULL elevation, and the style renders the "NNN m" label from it), and the shared contract **has no name column by design** (`conf/shared/contracts/19-peaks.yaml`), so the register's names could not be carried anyway; `wireSharedLayers` enforces the contract's attribute list exactly. - **The DEM path satisfies everything the contract asks** — same layer shape as Iceland (`haedarpunktar`) and Sweden (`hojdpunkt`): geom + elevation, no name, elevation-thinned at coarse zoom — and it is national data. This is the issue's own recommended default ("peaks = local maxima above a threshold"). So Finland's peaks are the **local maxima of MML's 25 m elevation model** (the `dem25m` COG delivery on Paituli: 84 tiles, 802 MB, EPSG:2393, 16-bit values in decimetres). The named-fell register remains what `places` already uses for names; it is not a peaks source. ## The mechanism: `importer/dem.go` The importer reads data files natively — no ogr2ogr, no gdal service — and had no raster reader. `dem.go` is its first: it parses the tiled 16-bit LZW COGs (TIFF IFD + geokeys + a faithful TIFF-LZW "early change" decoder, which Go's stdlib `compress/lzw` does not implement), decodes the grid, and finds local maxima. **Seam handling** is the part that had to be right. Consecutive tiles overlap by exactly one cell (80 025 m extent on an 80 000 m grid), so a naive per-tile scan would both miss peaks near seams and invent false ones where a higher neighbour sits just across the seam. The reader decodes each tile with a `radius`-cell halo from its eight neighbours, and a peak is emitted only for cells in the tile's own extent (the first 3200 cells in each axis) — every cell belongs to exactly one tile, so no peak is duplicated or lost. The naive per-tile prototype over-counted by ~24%; the halo version is the honest 26 296. **Parameters** (measured over the whole country): `radius: 20` cells (500 m de-clustering — one summit marker per ridge, not one per 25 m cell) and `min_elevation: 200` m (the "above the lowland" floor — below it, Finland's flat lake plateau produces drumlins, not peaks). Result: 26 296 peaks ≥ 200 m at z14, 7 991 ≥ 300 m at z13, 885 ≥ 500 m at z11 — the same order as Sweden's 33.8k surveyed spot heights. The coarse z11 floor (≥500 m) matches Iceland's and Sweden's (R31's cadence). ## Config - `conf/fi/downloads.yaml` — 84 dem25m tile entries (one URL per path, the mechanism's shape; the comment records the R5 size). - `conf/fi/staging.yaml` — `fi_dem_peaks` staging: `dem_peaks: {radius: 20, min_elevation: 200}` on a glob, `source_srid: 2393`, `srs: EPSG:3857`. - `conf/fi/16-peaks.yaml` — the layer: `extends: peaks`, reads `fi_dem_peaks`, zoom filters 300/500, and three acceptance tests (count pin + no NULLs, Halti's summit cell ≥ 1300 m, the R31 z11 floor). - `importer/fetch.go` — `neededFiles` now glob-matches load globs against downloads, so a scoped `--layer peaks` run fetches the 84 tiles. - Shared contract comment updated to record that Finland landed as a DEM partition and why `name` stays out for every country. - Docs: `docs/country-finland.md` (catalogue line, source table, known-gaps item), `docs/architecture.md` (dem.go in the source map), `docs/workstream-a.md`. ## Verification against the live database - `make build`, `make test` — green (plus a new `TestLZWDecodeTiff` round-trip and `TestReadDEMPeaks` against a synthetic COG; `TestNeededFilesGlobMatchesDownloads` covers the fetch change). - `importer config-check conf/fi` — OK (16 layers, 19 shared contracts, 57 tests). - **Import**: `importer import conf/fi/ --layer peaks` — 84 tiles fetched through the importer's own download path (fetchmeta sidecars written, re-run is a no-op), staging built in ~104 s, layer swapped live. Per-zoom row counts: | zoom | fi peaks | elevation range | NULLs | |---|---|---|---| | z11 | 885 | 500–1310 | 0 | | z12 | 7 991 | 300–1310 | 0 | | z13 | 7 991 | 300–1310 | 0 | | z14 | 26 296 | 200–1310 | 0 | - **Partitions** at every zoom (map_peaks_11_fi … map_peaks_14_fi), all with gist indexes, all SRID 3857 (matching is/se). - **Decoded tile** over Halti (14/9160/3757): `scripts/mvt-decode.py` → layer `peaks`, 1 feature, `elevation=1310`, `country=fi`. Halti's summit cell reads 1310 m (the 25 m DEM's value for the summit cell; the official height is 1324 m). - **Real-browser render** over CDP (`scripts/cdp-map-check.mjs`): Halti shows 1 peak at 1310 m; Saana (Kilpisjärvi) shows 1 peak at 1020 m; Helsinki shows **0** peaks at z11 — the negative control (the 200 m floor correctly leaves the flat south marker-free) and the positive controls (the fells present) both hold. - **Acceptance tally** (per-layer + shared-contract runs, pasted from the run): ``` layer peaks (fi) / R120: every Finland DEM peak made it through, with a valid elevation — PASS (peaks=26296, source_peaks=26296, null_elevation=0) layer peaks (fi) / R120: the highest Finland peak is Halti's summit cell — PASS (max_elevation=1310) layer peaks (fi) / R31: the z11 floor holds only the ≥500 m summits — PASS (peaks=885, below_500=0) shared layer peaks / R119: no peaks partition is empty at any zoom z11–z14 — PASS (partitions=12) shared layer peaks / R120: every country's peaks carry a valid elevation — PASS (countries=3) config / R105: every Finland staging table is nonzero after load — PASS (tables=18) ``` `make acceptance` runs green except a pre-existing Poland buildings test that times out at 10 s against the 5.6 M-row table even on a clean run — unrelated to this change. ## Notes / caveats - **The peaks are algorithmic, not surveyed.** DEM local maxima above a floor are a different population from a cartographer's spot-height survey, which is why the contract keeps the door open to a later named-fell layer (a separate shape, not a fake partition — the R13 sibling-contract idiom). Nothing here prevents that; the register is still the `places` source. - **Halti reads 1310 m**, not 1324 m: that is the 25 m DEM's own summit-cell value (a 25 m cell averages the summit), and the layer's job is to carry the DEM's reading, not correct it. - **The DEM reader holds all 84 tiles in memory** (~840 MB peak) for the one-shot staging build — the halo scheme needs every tile's header up front. Fine on this machine; noted here rather than silently streamed. - The Poland buildings acceptance timeout noted above is pre-existing and unrelated.
The source decision the issue asked for, resolved: MTK's surveyed spot-height
class (korkeuspiste, 52210) is empty in the 2025 delivery, and the named-fell
alternative from the paikannimi register cannot fill the peaks contract — the
register carries no elevation (R120) and the contract has no name column.
So Finland's peaks are the local maxima of MML's 25 m national elevation model
(dem25m on Paituli, 84 COG tiles, 802 MB, EPSG:2393, decimetre values), the
analogue of Iceland's haedarpunktar and Sweden's hojdpunkt.

New importer capability: importer/dem.go reads tiled 16-bit LZW COGs natively
(no ogr2ogr/gdal service) via a new staging `dem_peaks:` load kind. Tiles
overlap by exactly one cell, so each tile is decoded with a radius halo from
its eight neighbours and a peak is emitted only for cells in the tile's own
extent — every cell belongs to exactly one tile, no seam duplication or loss
(verified by a two-tile seam test and by the halo count: 26 296 peaks vs a
per-tile prototype's over-counted 34.5k).

Parameters measured over the whole country: radius 20 cells (500 m) + a 200 m
floor give 26 296 peaks >= 200 m at z14, 7 991 >= 300 m at z13, 885 >= 500 m
at z11 — the same order as Sweden's 33.8k surveyed spot heights; the z11 >=500 m
coarse floor matches Iceland and Sweden (R31). Halti's summit cell reads 1310 m.

Also: neededFiles now glob-matches load globs against downloads so a scoped
`--layer peaks` run fetches the 84 tiles; the fi staging-list acceptance test
gains fi_dem_peaks; configdir_test layer list gains peaks. Docs updated
(country-finland.md, workstream-a.md, architecture.md) and the shared peaks
contract's comment records that the name column stays out for all four
countries.

Verified against the live database: per-zoom counts and NULL checks above, a
decoded MVT over Halti (elevation=1310, country=fi), a real-browser render
showing Halti (1310 m) and Saana (1020 m) present and Helsinki flat-south empty,
and the acceptance tally:
  peaks (fi) R120 count pin PASS (peaks=26296, source_peaks=26296, null=0)
  peaks (fi) R120 Halti PASS (max_elevation=1310)
  peaks (fi) R31 z11 floor PASS (peaks=885, below_500=0)
  shared peaks R119 no empty partition PASS (partitions=12)
  shared peaks R120 valid elevation PASS (countries=3)
  config R105 all fi staging nonzero PASS (tables=18)
Three defects, all found by chasing one report from the map: a 930 m marker at
69.0128/20.9257 standing on ground whose contours reach 725 m.

**The DEM reader mis-decoded every partial edge block.** decodeBlock's comment
asserted that "the LZW stream only carries the real cells" of an edge block, and
it read rows with that narrow stride. TIFF 6.0 requires a tiled image to pad its
edge tiles out to the full tile size, and MML's writer does — checked against the
delivery itself, where the last-column blocks of 31M_DEM2.TIF are 3.49x the
compressed size of the interior ones, not the 0.25x a 129-cell stream would be.
The raster is 3201 cells on a 512 block, so the last column holds 129 real cells
and every row after the first was skewed by 383, filling the grid with real
elevations lifted from progressively further up the raster. That is why the bad
peaks carried plausible heights on the wrong hills — the 930 m terrain is real
and sits 2-4 km away. All five badly-wrong peaks in Kasivarsi were in one file,
all in internal-tile column 6, the partial one. decodeBlock now decodes tw*th and
indexes with a stride of tw, and picks the narrow stride only if the stream
genuinely cannot support the wide one, erroring rather than panicking if neither
fits.

**The seam halo was written upside down.** The padded grid is defined by
gy = gy0 - py, so py is gy0 minus the neighbour cell's global gy; the code
measured from the bottom instead, py = gy - (gy0-ph+1), which is (ph-1) - py,
an exact vertical mirror. The northern neighbour's strip landed on the southern
border. It never corrupted an elevation (the centre-write guard held) but it
made the local-max test consult the wrong rows near every seam, which also means
the "the naive prototype over-counted by ~24%, the halo version is the honest
26 296" claim rested on a halo that was in the wrong place.

**The peaks were not clipped to Finland.** dem25m is 84 square tiles on the KKJ
grid and a square laid over a border does not stop at it, so 532 of 26 296 peaks
(2%) sat in Norway, Sweden or Russia while carrying country = 'fi'. Since country
is the partition key that is Finland's partition claiming another country's
terrain, not a cosmetic error. The layer now joins fi_valtakunta, the same
national outline the country_mask layer is built from.

Tests: the fixture could not express the bug — buildTestCOG writes ONE block
covering the whole raster, so no edge block exists and the stride is never
exercised. Added buildTiledTestCOG (explicit tw/th, edge blocks padded as TIFF
requires) and TestDecodeRegionHandlesPaddedEdgeBlocks over an 11x7 raster with
4x4 blocks, which fails on the old stride and passes on the new one.

The layer's own count pin now compares against the extraction MINUS the clip:
comparing to the raw staging count would fail for exactly the reason the clip
exists. It counts the staging peaks inside the outline and requires the partition
to match, so nothing inside Finland can be silently lost.

Verified on the live DB after a full re-import. z14 25 556 (26 058 extracted,
502 dropped as foreign), z11 772, Halti still 1310, 0 peaks outside the country
mask. Against the contours in Kasivarsi, peaks >= 400 m: mean excess over the
local contour maximum 0 m (was +1), worst +40 m (was +405), none more than 100 m
too high (was 5), and none without contours nearby (was 28). The reported marker
at 69.0128/20.9257 is gone; the nearest peak is now 880 m, 1.8 km away.
acceptance conf/fi 57/57 under -acceptance all; make test green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2QYwuohkM9vWJs6uzsK58
This pull request has changes conflicting with the target branch.
  • docs/country-finland.md
  • importer/configdir_test.go
  • importer/fetch_test.go
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin issue-108-fi-peaks:issue-108-fi-peaks
git switch issue-108-fi-peaks

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff issue-108-fi-peaks
git switch issue-108-fi-peaks
git rebase main
git switch main
git merge --ff-only issue-108-fi-peaks
git switch issue-108-fi-peaks
git rebase main
git switch main
git merge --no-ff issue-108-fi-peaks
git switch main
git merge --squash issue-108-fi-peaks
git switch main
git merge --ff-only issue-108-fi-peaks
git switch main
git merge issue-108-fi-peaks
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
issue is stuck: missing decision, credential, or data (work-issue blocked path)

  
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap!205
No description provided.