art/ismap
1
0
Fork 0
Code Issues 56 Pull requests 1 Projects Releases Packages Wiki Activity Actions

#107: Finland: import MTK-korkeussuhteet — contours (34 GB) #204

Merged
art-bot merged 3 commits from issue-107-fi-contours into main 2026-08-21 08:55:14 +00:00
Conversation 0 Commits 3 Files changed 9 +490 -34
art commented 2026-08-21 05:51:03 +00:00
Owner
Copy link

Finland's contours from MML MTK-korkeussuhteet (issue #107)

Closes #107.

Finland's partition of the shared map_contours_<z> tables (the contours contract,
conf/shared/contracts/18-contours.yaml) from the elevation theme of MML's
Maastotietokanta, MTK-korkeussuhteet 2025-04-03 — the 34.45 GB download the issue
called for ("heavy is the point"). Two commits:

  1. The import (conf/fi/downloads.yaml, staging, 16-contours.yaml, and the importer
    change the download forced).
  2. Fixes found by running it (the index join correction, the deep-tier R120, the
    se–fi border agreement test, docs).

What landed

  • Source (R5): MTK-korkeussuhteet_25-04-03.gpkg, 34 445 299 712 bytes, recorded on
    first fetch. Two staged tables: fi_korkeuskayra (kohdeluokka 52100, 6 865 782 3D
    contour lines, whole-metre height in korkeusarvo millimetres — max 1 327 500 =
    Halti's 1327.5 m) and fi_korkeuslabel (korkeuskayrankorkeusarvo 52191, 212 057
    label points).
  • The download resume feature (importer/fetch.go + fetch_test.go). Paituli resets
    the connection mid-stream on a multi-GB file, so a plain io.Copy restarted the 34 GB
    from zero. download now resumes a dropped transfer from where it stopped
    (Range: bytes=<size>-), bounded by maxDownloadAttempts, keeping the temp file
    between attempts and truncating if a server ignores Range. Covered by
    TestDownloadResumesOnReset (two-half body, connection reset after the first half,
    byte-identical resumed body, no temp file left behind). The 34 GB fetch and the z14
    read rode this path.
  • contours layer. interval_class = 1 is the labelled index set — the lines a
    korkeuskayrankorkeusarvo label references via korkeuskayraviittaus (MML's own
    terrain-selected choice, not a cadence). Zoom ladder (2.5 m cadence, ~23% denser than
    Sweden's 5 m): z14 every line (6 865 782), z13 the labelled set (145 323), z12 the
    100 m index (23 123), z11/z10 a 200 m index floor (7 234) — Finland's genuinely-
    country coarse cadence (500 m, Sweden/Iceland's floor, would be 1 166 lines = only the
    Halti fells).

The join bug the acceptance run caught

The first import joined the label table on its own mtk_id, which matched nothing —
every line came out interval_class = 2, the lower-zoom partitions were empty, and
R119/R120 failed against the live DB. The label's korkeuskayraviittaus (a text
reference) is the pointer to the line's mtk_id; joined correctly it resolves 145 323
of 145 710 distinct references (387 point at lines this product does not carry). The
z14 table was rebuilt from staging with the corrected join and the cascade re-ran —
the second commit documents all of it. The previous worker's z14 load was therefore
redone (the layer's own re-import; it was wrong, not reusable).

The border test (the issue's acceptance criterion)

The shared contours contract gains the se–fi agreement check, mirroring the roads
contract's R126 shape (requires: both partitions, Tornio–Haparanda, skip-not-fail on
a partial database). Agreement is defined on the cadence the two products share:
Sweden draws every 5 m, Finland every 2.5 m, so the check restricts both sides'
distinct 5 m-multiple elevations within 15 km of the border point to the range both
cover and requires the sets identical — each product's outermost contour is excluded as
a boundary effect (Sweden's sea-level 0 and highest 30; Finland's 2 and 27). Measured:
both sides draw exactly {5,10,15,20,25}, stable across radii 5–25 km. It passes on
every config's run (is/se/pl/fi/gr/world), skipping only until both se and fi contours
exist.

Verification against the live database

fi rows per zoom: z14 = 6 865 782, z13 = 145 323, z12 = 23 123, z11 = 7 234,
z10 = 7 234 (z11/z10 both the 200 m index floor; z13 = the resolving labelled set).
Zero NULL elevation and zero NULL/<1 interval_class in z14. R119: all 15
partitions (3 countries × z10–z14) non-empty. Decoded tiles: z14 (Tornio, 26 fine 2.5 m
rungs), z13 (labelled index lines, interval_class=1, elevations 5/10), z10 (northern
fells, 71 lines all interval_class=1, all elevation 200) — each carries elevation +
interval_class + country.

make acceptance-all, per config:

conf/is:     42 test(s): 42 passed, 0 failed, 0 skipped, 0 not run
conf/se:     55 test(s): 55 passed, 0 failed, 0 skipped, 0 not run
conf/pl:     52 test(s): 51 passed, 1 failed  ← the pre-existing Poland buildings
              R120 10 s timeout (sum(ST_NumGeometries) over ~9M footprints), present
              on main and untouched by this change (documented in #101's commit)
conf/fi:     57 test(s): 57 passed, 0 failed, 0 skipped, 0 not run
conf/gr:     60 test(s): 60 passed, 0 failed, 0 skipped, 0 not run
conf/world:  34 test(s): 34 passed, 0 failed, 0 skipped, 0 not run

The fi contours tally lines:

layer contours (fi) / R120: the Finland contours' index rung is exactly the labelled
    set — PASS (11.7s, index_lines=145323, source_index=145323)   [deep tier, timeout 100]
layer contours (fi) / R31: the z11 floor holds only 200 m index lines — PASS
    (lines=7234, not_200=0)
shared layer contours / R119 — PASS (partitions=15)
shared layer contours / R120 — PASS (countries=3, offenders=none)
shared layer contours / the se and fi contours agree at Tornio–Haparanda — PASS
    (fi_5m=5, se_5m=5, fi_range=5..25, se_range=0..30)

The fi R120 mapping pin lives in the deep tier (timeout: 100) for the same reason
Sweden's does: both counts seq-scan 6.87M rows and a 10 s cap flakes (14 s cold).

Notes

  • Docs: country-finland.md marks MTK-korkeussuhteet imported (the catalogue line
    and the R13 contours bullet, including the border-match test); country-requirements.md
    updates the layer inventory and the R13 resolution (only #108 peaks remains for fi).
  • The staged z14 rebuild re-read the 34 GB gpkg (the staging reload is not skipped by a
    single-layer import) — the fetch itself was not re-downloaded.
  • make test passes (configdir_test.go gained contours in the conf/fi list).
Finland's `contours` from MML MTK-korkeussuhteet (issue #107) Closes #107. --- Finland's partition of the shared `map_contours_<z>` tables (the `contours` contract, conf/shared/contracts/18-contours.yaml) from the elevation theme of MML's Maastotietokanta, `MTK-korkeussuhteet` 2025-04-03 — the 34.45 GB download the issue called for ("heavy is the point"). Two commits: 1. **The import** (conf/fi/downloads.yaml, staging, 16-contours.yaml, and the importer change the download forced). 2. **Fixes found by running it** (the index join correction, the deep-tier R120, the se–fi border agreement test, docs). ## What landed - **Source (R5):** `MTK-korkeussuhteet_25-04-03.gpkg`, 34 445 299 712 bytes, recorded on first fetch. Two staged tables: `fi_korkeuskayra` (kohdeluokka 52100, 6 865 782 3D contour lines, whole-metre height in `korkeusarvo` *millimetres* — max 1 327 500 = Halti's 1327.5 m) and `fi_korkeuslabel` (`korkeuskayrankorkeusarvo` 52191, 212 057 label points). - **The `download` resume feature (importer/fetch.go + fetch_test.go).** Paituli resets the connection mid-stream on a multi-GB file, so a plain `io.Copy` restarted the 34 GB from zero. `download` now resumes a dropped transfer from where it stopped (`Range: bytes=<size>-`), bounded by `maxDownloadAttempts`, keeping the temp file between attempts and truncating if a server ignores Range. Covered by `TestDownloadResumesOnReset` (two-half body, connection reset after the first half, byte-identical resumed body, no temp file left behind). The 34 GB fetch and the z14 read rode this path. - **contours layer.** `interval_class = 1` is the labelled index set — the lines a `korkeuskayrankorkeusarvo` label references via `korkeuskayraviittaus` (MML's own terrain-selected choice, not a cadence). Zoom ladder (2.5 m cadence, ~23% denser than Sweden's 5 m): z14 every line (6 865 782), z13 the labelled set (145 323), z12 the 100 m index (23 123), z11/z10 a 200 m index floor (7 234) — Finland's genuinely- country coarse cadence (500 m, Sweden/Iceland's floor, would be 1 166 lines = only the Halti fells). ## The join bug the acceptance run caught The first import joined the label table on its own `mtk_id`, which matched nothing — every line came out `interval_class = 2`, the lower-zoom partitions were empty, and R119/R120 failed against the live DB. The label's `korkeuskayraviittaus` (a text reference) is the pointer to the line's `mtk_id`; joined correctly it resolves 145 323 of 145 710 distinct references (387 point at lines this product does not carry). The z14 table was rebuilt from staging with the corrected join and the cascade re-ran — the second commit documents all of it. **The previous worker's z14 load was therefore redone** (the layer's own re-import; it was wrong, not reusable). ## The border test (the issue's acceptance criterion) The shared contours contract gains the se–fi agreement check, mirroring the roads contract's R126 shape (`requires:` both partitions, Tornio–Haparanda, skip-not-fail on a partial database). Agreement is defined on the cadence the two products share: Sweden draws every 5 m, Finland every 2.5 m, so the check restricts both sides' distinct 5 m-multiple elevations within 15 km of the border point to the range both cover and requires the sets identical — each product's outermost contour is excluded as a boundary effect (Sweden's sea-level 0 and highest 30; Finland's 2 and 27). Measured: **both sides draw exactly {5,10,15,20,25}**, stable across radii 5–25 km. It passes on every config's run (is/se/pl/fi/gr/world), skipping only until both se and fi contours exist. ## Verification against the live database fi rows per zoom: **z14 = 6 865 782, z13 = 145 323, z12 = 23 123, z11 = 7 234, z10 = 7 234** (z11/z10 both the 200 m index floor; z13 = the resolving labelled set). Zero NULL elevation and zero NULL/`<1` interval_class in z14. R119: all 15 partitions (3 countries × z10–z14) non-empty. Decoded tiles: z14 (Tornio, 26 fine 2.5 m rungs), z13 (labelled index lines, interval_class=1, elevations 5/10), z10 (northern fells, 71 lines all interval_class=1, all elevation 200) — each carries elevation + interval_class + country. `make acceptance-all`, per config: ``` conf/is: 42 test(s): 42 passed, 0 failed, 0 skipped, 0 not run conf/se: 55 test(s): 55 passed, 0 failed, 0 skipped, 0 not run conf/pl: 52 test(s): 51 passed, 1 failed ← the pre-existing Poland buildings R120 10 s timeout (sum(ST_NumGeometries) over ~9M footprints), present on main and untouched by this change (documented in #101's commit) conf/fi: 57 test(s): 57 passed, 0 failed, 0 skipped, 0 not run conf/gr: 60 test(s): 60 passed, 0 failed, 0 skipped, 0 not run conf/world: 34 test(s): 34 passed, 0 failed, 0 skipped, 0 not run ``` The fi contours tally lines: ``` layer contours (fi) / R120: the Finland contours' index rung is exactly the labelled set — PASS (11.7s, index_lines=145323, source_index=145323) [deep tier, timeout 100] layer contours (fi) / R31: the z11 floor holds only 200 m index lines — PASS (lines=7234, not_200=0) shared layer contours / R119 — PASS (partitions=15) shared layer contours / R120 — PASS (countries=3, offenders=none) shared layer contours / the se and fi contours agree at Tornio–Haparanda — PASS (fi_5m=5, se_5m=5, fi_range=5..25, se_range=0..30) ``` The fi R120 mapping pin lives in the deep tier (`timeout: 100`) for the same reason Sweden's does: both counts seq-scan 6.87M rows and a 10 s cap flakes (14 s cold). ## Notes - **Docs:** country-finland.md marks MTK-korkeussuhteet imported (the catalogue line and the R13 contours bullet, including the border-match test); country-requirements.md updates the layer inventory and the R13 resolution (only #108 peaks remains for fi). - The staged z14 rebuild re-read the 34 GB gpkg (the staging reload is not skipped by a single-layer import) — the fetch itself was not re-downloaded. - `make test` passes (`configdir_test.go` gained `contours` in the conf/fi list).
Finland's `contours` partition of the shared map_contours_<z> tables (the
`contours` contract in conf/shared/contracts/18-contours.yaml, issue #151) lands
alongside Iceland's and Sweden's. Source: MML Maastotietokanta
MTK-korkeussuhteet 2025-04-03, the elevation theme.

Download and staging (R5): the 34.45 GB GeoPackage (34 445 299 712 bytes,
measured at first fetch) from Paituli over plain HTTPS like every MTK theme. Two
staged feature layers — `korkeuskayra` (kohdeluokka 52100, 6 865 782 3D contour
lines, the height in `korkeusarvo` MEDIUMINT *millimetres*: max 1 327 500 =
Halti's 1327.5 m) and `korkeuskayrankorkeusarvo` (52191, 212 057 label points
referencing their line's mtk_id). Both streamed in batches; EPSG:3067 → 3857.

The 34 GB download forced a real importer change: Paituli resets the connection
mid-stream, so `download` in importer/fetch.go now resumes a dropped transfer
from where it stopped (`Range: bytes=<size>-`) instead of restarting from zero,
bounded by maxDownloadAttempts, with the temp file kept between attempts and
truncated if a server ignores Range (returns 200). Covered by
TestDownloadResumesOnReset, which serves a body in two halves, drops the
connection after the first, and asserts the resumed body is byte-identical and
no temp file is left behind. The z14 load and the fetch itself rode this path.

contours: the contract's interval_class = 1 index rung is exactly the set of
lines a korkeuskayrankorkeusarvo label references (145 710 distinct mtk_ids —
MML's own terrain-selected choice of which contours carry a label and a bolder
stroke; they are not a clean cadence, 20 372 sit off a 10 m multiple); the
remaining 6.7M 2.5 m lines are rung 2, mirroring how Sweden maps stodkurva.
MTK's base interval is 2.5 m (denser than Sweden's 5 m), and Finland's relief is
low — most of the country below 200 m — so the coarse floor cannot be Sweden's
and Iceland's 500 m (1 166 lines, i.e. only the Halti fells). The cascade:
z14 every 2.5 m line (6.87M), z13 the labelled index set (145 710), z12 the
100 m index (elevation % 100 = 0), z11/z10 a 200 m index floor (Finland's
genuinely-country cadence, the deliberate-thinning case R31 exists for).

Acceptance: two fi-specific tests on the layer (the index-rung == labelled-set
mapping pin, and the z11 200 m floor) plus the shared contract's R119/R120 now
run over three countries. configdir_test.go gains contours in the conf/fi list.

The Sweden–Finland contour agreement check at Tornio–Haparanda (the issue's
stated first test of two national products agreeing) is the next commit once the
cascade has run and the se data is in the same DB.
Two corrections found by running the layer and its acceptance tests against the
live database, plus the issue's named acceptance test, plus the docs that record
the R13 status.

THE JOIN WAS WRONG. The first import joined the label table on its own `mtk_id`,
which matched nothing — every line came out interval_class = 2, the cascade's
z13/z12/z11/z10 partitions were empty, and R119/R120 failed. The label's
`korkeuskayraviittaus` (a text reference) is the pointer to the contour line's
`mtk_id`; joined that way it resolves 145 323 of the 145 710 distinct references
(the other 387 point at lines this product does not carry). The z14 table was
rebuilt from staging with the corrected join (verified: interval_class = 1 count
== resolving-reference count == 145 323), and the cascade re-ran.

R120 moved to the deep tier. Its two counts are seq scans over 6.87M rows, so the
10 s cap flakes (14 s cold); `timeout: 100` is the same call Sweden's contours
mapping pin makes for the same reason (conf/se/16-contours.yaml). A CTE runs the
staging-side join once instead of twice.

THE BORDER TEST (the issue's acceptance criterion): the shared contours contract
gains the se–fi agreement check, mirroring the roads contract's R126 shape —
`requires:` both partitions, the Tornio–Haparanda point, skip-not-fail on a
partial database. Agreement is defined on the cadence the two products share:
Sweden draws every 5 m, Finland every 2.5 m, so the check restricts both sides'
distinct 5 m-multiple elevations to the range both cover and requires the sets
identical — each product's outermost contour is excluded as a boundary effect
(Sweden's sea-level 0 and highest 30, Finland's 2 and 27). Measured: both sides
draw exactly {5,10,15,20,25}, stable across radii 5–25 km.

Verified against the live database: fi rows per zoom z14=6 865 782, z13=145 323,
z12=23 123, z11=z10=7 234; zero NULL elevation / interval_class; R119 15/15
partitions non-empty; decoded z14 (fine 2.5 m rungs), z13 (labelled index lines,
interval_class=1) and z10 (200 m floor, all elevation 200) tiles carry
elevation + interval_class + country. `make acceptance-all` on conf/fi: 57/57
passed, 0 skipped/not-run.

Docs: country-finland.md marks MTK-korkeussuhteet imported and updates the R13
contours line (both sides of the border match is the contract test);
country-requirements.md updates the layer inventory and the R13 resolution to
record that Finland's contours landed.
The contours themselves check out against the live DB: 6 865 782 lines, elevation
0–1327 with no NULL or non-positive interval_class, contours around Halti reaching
1327 m (Halti is 1324), and only 41 z10 lines outside the country mask — ordinary
border spill. Three fixes on top, one of them a live bug in a shared code path.

**The zip fetch leaked its temp file.** The resume loop replaced download()'s
blanket `defer os.Remove(tmpName)` with removals on the error paths only. The
rename branch is fine — the temp IS the result — but the zip branch consumes the
temp and never removed it, so every successful .zip fetch left a `.fetch-*.tmp`
behind. Nothing showed on disk yet because this PR's own 34 GB fetch is a .gpkg
and took the rename path; Poland's 380 county zips are ~25 GB of leak the first
time anyone runs `make fetch-bdot10k` on this code. Proven with a new test
(TestDownloadZipLeavesNoTempFile) that failed before the fix.

**The 416 branch claimed a verification that did not exist.** Its comment said
"caller verifies" and the caller did not: a 416 returned complete=true and the
temp file went straight to unzip/rename. A 416 is equally the signature of a temp
left over from a LARGER earlier version of the same URL, which would have
installed a truncated file. Now it HEADs for the real length, accepts only on an
exact match, and otherwise truncates and restarts.

**The border test compared set sizes, not sets.** The issue's acceptance criterion
— the se and fi contours agree at Tornio–Haparanda — used count(DISTINCT elevation)
per side, so {5,10,15,20,30} would have passed against {5,10,15,20,25}: same size,
different terrain, exactly the disagreement the check exists to catch. Now an array
equality. The two reported columns are array_to_string rather than the arrays
themselves, because the runner prints a pgx array as a struct dump
(`&{[{5 2} {10 2}...]}`) and an unreadable failure diagnostic is no diagnostic —
the R145 lesson. Still passes, and now says so legibly: fi_5m=5,10,15,20,25
se_5m=5,10,15,20,25.

Not changed, checked and left alone: `korkeusarvo / 1000` truncates rather than
rounds, so a 2.5 m contour reads 0.5 m low. That is 43.4% of all lines — but only
12 of the 145 323 index lines, and the style labels only index lines
(interval_class = 1), so the visible error is 12 labels nationally. Rounding would
also change which lines the `elevation % 100` rungs select, i.e. a 34 GB re-import
to verify a 12-label improvement.

Verified: make test green including the new fetch test; acceptance conf/fi 57/57
under -acceptance all; acceptance conf/se 53/53 in the default tier.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2QYwuohkM9vWJs6uzsK58
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
issue is stuck: missing decision, credential, or data (work-issue blocked path)

  
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap!204
No description provided.