Closes #14 #116
No reviewers
Labels
No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
art/ismap!116
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-14-sankmark-cover"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #14
#14: Sweden's
sankmarkwetland overlay is carved into cover — inland_marsh and peat_bog land at every zoom, coverage invariant intactThis is piece 1 of #83: Sweden gains the CORINE wetland classes it had no way to express.
The data was already on disk in the Geotorget order (
mark_sverige.gpkg,sankmarklayer)— no new download, no new credential. The source: 1 447 844 polygons, "Sankmark, fast"
(objekttypnr 2651, ×1 308 224) and "Sankmark, våt" (2652, ×139 620), 55 767 km² together,
0 invalid, all in the same EPSG:3006 file as
mark.The design — why the fold is a carve, and how it is done
sankmarkis an overlay, not part of themarkcoverage: a mire sits inside forest oropen land. Inserting it as rows would stack it on its host and break the partition property
ST_CoverageSimplifydepends on — R35's exact prohibition. The fold therefore happens inthree staging steps, all in
conf/semap.yaml:se_sankmark_t10— the layer streamed into PostGIS exactly likese_mark_t10(same gpkg, EPSG:3006 → 3857).
se_wetland_t10— the overlay turned into a proper non-overlapping coverage:per-100-km-grid-cell
ST_UnaryUnionof each class,våtwinning where fast and våtoverlap, each cell's union clipped to its cell square and dumped. 1 386 538 pieces.
se_cover_t10— the carved national coverage: everymarkpolygon minus theunion of the wetland pieces inside it (index-joined LATERAL), plus the wetland pieces
re-emitted as
peat_bog/inland_marshrows. The cover layer now reads this insteadof
se_mark_t10; the class CASE and its WHERE IN list moved here so they stayadjacent (R43).
Two measured properties make the design decisions:
9 738-polygon county sample, ~20% of polygons involved, but the total overlap area is
< 0.2% of the class: digitising slivers. Raw emission would put those overlaps into the
coverage; the union nodes them away. (Nationwide after the union: 307 remaining
self-overlaps, all < 1e-6 m² — floating-point noise.)
ST_Collectover all 1.31M fast polygonsexceeds Postgres' 1 GB array limit (SQLSTATE 54000, hit live). The per-cell union keeps
every collect at sample scale (~14.5k polygons max per cell) and the cascade partitions
by the same 100 km grid anyway. The 0.9% of polygons that cross a cell line are joined
to every cell their bbox touches and clipped to each cell square, so a crossing mire is
tiled exactly: no area dropped, none duplicated.
The review (#116) then forced the hardening that makes the carve a provably valid
coverage — ST_CoverageInvalidEdges on partition 23105 goes from 1 415 invalid-edge rows
(295 061 m) on the first submission to 0 (the pre-carve baseline was 3): the wetland
pieces are snap-rounded to a 1 mm grid after the coverage-clean (exact shared edges, and
no degenerate inputs for GEOS's prepared-intersects), the host fragments are clipped with
ST_ClipByBox2D (the box-clipper, robust where the full overlay threw "side location
conflict" on grazing boundaries), the carve's piece union is ST_UnaryUnion of an
ST_Collect (one noding pass — the ST_Union aggregate's pairwise ClassicUnionStrategy was
the other GEOS hang), every source host is ST_MakeValid-ed once in a materialized CTE
(its checkHolesInShell validity check is quadratic only in the fragment phase, which is
avoided — the 22 invalid hosts' repairs are cheap), and the final per-cell
ST_CoverageClean removes the sub-millimetre difference residue (78 109 slivers < 0.001 m²
→ 456) and fills the gaps their epsilon-drop leaves. The source repair is bound once and
used for both the predicate and the subtraction (R35's stacking prohibition).
The result satisfies R35's checks: 0 invalid rows in both staging outputs, and the
overlay's area is subtracted from its host — area conservation is exact (546 078 km²
of
mark= 490 311 km² carved base + 55 767 km² wetland, to the km² in EPSG:3006).ST_CoverageSimplifyruns cleanly over a full 100 km partition of the combined coverage(20 558 rows, 0 invalid).
Class mapping
peat_bog(412)inland_wetlandwetlandinland_marsh(411)inland_wetlandwetlandBoth values already existed in the shared
landcoverenum and its rollups (the full CLCnomenclature is kept), so the z12/z9 rollup chain needed no change — Sweden's z3 set gains
wetlandfor free, and the written exemption forseinconf/common.yaml(issue #83's"no wetland until that lands") is now history.
Numbers (measured on this run — 2026-08-13, rebuilt after the power outage killed the first attempt at
se_cover_t10)se_mark_t10load (rebuilt by the DAG)se_sankmark_t10loadse_wetland_t10(unions + clip + dump)se_cover_t10(the nationwide carve)The full
-layer coverimport (staging rebuilds + z13 + the whole generalization cascade)took 3 h 29 m (11:26 → 14:55) at
-jobs 4 -batch-rows 10000as the Makefileprescribes — flagged rather than hidden: it is the multi-hour per-layer reimport the note
must record. The z11 step dominates (2 h): it is the first absorption pass over the carved
coverage's ~1.1 M polygons, and absorption is the cascade's expensive operator. All timings
are the stages' own log timestamps.
Verification against the live DB
se_cover_t10, rebuilt): 5 383 862 rows, 0 per-row invalid, 0 NULLs,15 classes (
peat_bog1 252 246,inland_marsh134 172), 456 rows < 0.001 m² (from78 109 pre-review), ST_CoverageInvalidEdges = 0 on the sampled partitions (23105,
22104 — the pre-carve baseline was 3), part_key/centroid mismatches 0 (from 45 714).
map_cover_13_se(live): 15 distinct classes, 5 383 862 rows, 0 invalid(full-scan
ST_IsValid), 0 NULLs — the counts match the staging exactly.peat_bog1 252 246 +inland_marsh134 172) → z10–12inland_wetlandrollup → z3–9wetland;wetlandpresent at z3 (10 774 rows). The z3 set is {agricultural, nature, water, wetland} —
artificial(19 rows) absorbs at the coarse floor, the documented absorption reasonin
conf/common.yaml.(z11) → 242 923 (z10) = 242 923 (z9) → 173 392 (z5) → 109 008 (z4) → 48 853 (z3).
The z13→z9 retention is 83%; the z4→z3 −55% step is the coarse floor absorbing
Sweden's 1.39 M finely-diced mire pieces (mean ~0.04 km² real; only 103 exceed
100 km², while Finland's 100× coarser CORINE wetland retains more with identical
parameters) — measured and filed as issue #117 (R40/R125's check).
the swap (the cascade ran clean to completion).
(the reviewer's own; the pre-carve baseline was 3) — the carve is a valid polygonal
coverage, R35's check met.
mark'sincluded classes, with the excluded "Ej karterat" 2648 tiles outside the count.
/tiles/map/9/288/131showswetland(11 of 184 features, allcountry=se);/tiles/map/13/4632/2016showspeat_bogandinland_marshamong58 features — via
zxy_map_cover, the same path mapsrv serves.and asked the page:
queryRenderedFeatureson thecover_wetlandlayer returnsinland_wetlandat z10 andpeat_bogat z13.ST_CoverageSimplifyran the whole cascade with no errors and 0 invalid rows.What is left out (deliberately)
emitted as-is; the gap pre-exists in
markand emitting the mire there is truer thandropping it.
178 m × ~13 m source sliver (objekttypnr 2640) where GEOS gives inconsistent
relate/intersection answers; it is a source artifact absorbed within a zoom or two and
invisible after z13.
water_bodiesstill readsse_mark_t10raw (its water polygons are not a coverage) andis untouched;
se_coastlikewise.sankmarkforinland_marsh/peat_bog#14Review
The carve-not-insert shape is the right design, the PR body's numbers check out against the live DB exactly (15 classes at z13,
peat_bog1 252 357,inland_marsh134 181), and area is conserved through the cascade: totalsecover is 2 564 244 → 2 564 243 → 2 564 066 → 2 563 315 → 2 563 314 km² for z13→z9, 0.036% lost over five zooms.But R35's check is not met as claimed, and that is the one finding I would hold the PR on.
1. The carve does not produce a valid polygonal coverage —
conf/semap.yaml:284The PR verifies "0 invalid rows" with per-row
ST_IsValid. That is not coverage validity.ST_CoverageInvalidEdgesover partition23105:Each side is clean on its own. All of the invalidity is at the seam between a carved host and the mire re-emitted into its hole, and the control on the last row shows the carve introduces it.
The cause is in the query: the host is differenced against
ST_Union(w.geom)— a re-noded merge of the intersecting pieces — while those same pieces are emitted individually and un-unioned. The hole ring the difference leaves and the boundaries of the individual pieces therefore do not match vertex-for-vertex. The residualST_Overlapsarea is 0.000 m² (sub-micron vertex mismatch, not real overlap), which is why per-row validity and the area arithmetic both look clean.Practical damage today is small — at z12 the resulting overlaps in the same partition total 0.0044 m² — but
ST_CoverageSimplify's stated precondition is a valid coverage, and its behaviour on an invalid one is undefined. The comments atconf/semap.yaml:467and:539("the cascade machinery sees one valid coverage at maxz") assert the thing that is not true.ST_CoverageCleanis available here (PostGIS 3.6.2) and is the direct fix; snap-rounding both sides from one noding operation (ST_Node/ST_ReducePrecisionover the combined overlay) is the other.2. Predicate and subtraction run on different geometries —
conf/semap.yaml:308The LATERAL selects overlay pieces with
ST_Intersects(w.geom, m.geom)against the raw host, but the subtraction isST_Difference(ST_MakeValid(m.geom), …).se_mark_t10holds 22 invalid geometries of 3 495 437 (measured; all self-intersections — one at1934964.96 8974036.90, which threwTopologyException: side location conflictwhen I ran an unrelated intersection over this table during review). For those hosts the predicate can miss a piece that does intersect the repaired geometry: the piece is then not subtracted, yet is still emitted as its own row — the overlay stacked on its host, which is precisely what R35 forbids. BindST_MakeValid(m.geom)once and use it in both places.3. The carve manufactures ~78k slivers the source never contained —
conf/semap.yaml:284se_cover_t10se_wetland_t10Sub-millimetre difference residue, invisible at every zoom, and it feeds the 2-hour z11 absorption pass the PR identifies as the cost driver. An area epsilon on the difference pieces (or snap-rounding before the difference, per finding 1) removes them.
4. The "~1% falls in the source's own coverage gaps" measurement is refuted by the delivered data —
conf/semap.yaml:278,docs/country-sweden.mdSince
cover = mark − (wetland ∩ mark) + wetland, the area of wetland lying outside the coverage is exactlyΣcover − Σmark_included. Measured: 2 564 244 139 003 − 2 564 244 116 410 = 22 593 m², against the ~2 900 km² that 1% of the 292 184 km² wetland would be. The sentence appears in both the config comment and the country doc, and it is the stated justification for a deliberate design decision ("emitted as-is … truer than dropping it"), so it is worth correcting rather than dropping.5.
part_keyfor carved base pieces names a cell they aren't in —conf/semap.yaml:304Base rows take
part_keyfromST_Centroid(m.geom)— the parent mark polygon — while the inserted row is aST_Dumppiece of the carved remainder.The wetland rows are right because they are clipped to cell squares.
part_keyis theST_CoverageSimplifywindow (importer/generalize.go:186) and the absorption neighbour lookup is restricted to the same partition (importer/generalize.go:971,AND n.part_key = c.part_key), so a mis-partitioned piece is simplified with none of its true neighbours present, and when it falls under the area threshold its neighbour search finds nothing — it then survives only if larger than one pixel², otherwise it is dropped, which is the hole the design exists to prevent. Derivingpart_keyfrom the dumped piece in both branches costs nothing.6. Two config comments carry numbers the database contradicts
conf/semap.yaml:469— "~4.8M polygons (3.5M carved base + ~1.3M wetland)". Actual: 5 619 196 = 4 232 658 + 1 386 538. The carve fragments 3.5M source polygons into 4.23M pieces; the comment reports the pre-carve figure. It is the sizing rationale for the-jobs/-batch-rowsbudget, so it is 17% low for anyone tuning from it.docs/country-sweden.md:87— "15 CORINE classes from 17 usableobjekttypnrcodes, plus thesankmarkwetland overlay". The 17markcodes still yield 13; the other two come fromsankmark's 2651/2652, so it is 15 from 19.7. Stale non-conformance row —
docs/country-requirements.md:1290Gap row 10 still reads "Sweden 13 distinct classes … Sweden has no wetland class at all (#14) … neither Sweden nor Poland produces
wetland". All three clauses are now false. Rows 5, 6 and 12 of that table were annotated in place when their issues landed ("Fixed in #81:", "Resolved 2026-08-13:", "fixed in #45"), so the convention says row 10 gets the same treatment — otherwise the next reader of R22 against Sweden finds the requirements doc contradictingconf/common.yaml:308.8. Smaller things
conf/semap.yaml:257).fast_only = F \ Vruns over unions that extend well outside the cell, and only line 265 clips tosq. Since(F\V) ∩ S = (F∩S) \ (V∩S), clipping each class union to the square first is an exact identity over a fraction of the vertices — a straight saving on the 12-minute step, which reruns on every-layer coverimport.conf/semap.yaml:541). The old layer query wasSELECT (ST_Dump(ST_MakeValid(geom))).geom …; the new one is a rawSELECT geom, cover_class, part_keyinto a layer declaringgeom: polygon. Today every row is POLYGON (verified, all 5 619 196), but both upstream dumps can in principle emit linework from aGEOMETRYCOLLECTION, and the wetland branch applies noST_MakeValidat all.ST_CollectionExtract(…, 3)re-establishes the guard.FULL OUTER JOIN … USING (cx, cy)(conf/semap.yaml:262) would remove six hand-writtenCOALESCE(f.cx, v.cx)calls, one of which omitted would silently build a NULL envelope and drop a cell.part_keyformula is now written out four times —conf/semap.yaml:304,:313,conf/fimap.yaml:409— for the value the whole cascade partitions on, with nothing to catch a miss if the cell size or encoding changes.9. Two things above the line of this PR
naturegains 139 341 km² over the same step — a non-dominant class reclassified into the dominant one. R125 is a MUST whose check is exactly "flag any non-dominant class whose share drops by an outsized fraction between adjacent zooms". The PR explains it convincingly as the coarse floor doing its job; the point is that the explanation should be filed and recorded where the next reader of R40/R125 will find it, as Iceland's artificial/agricultural collapses are inconf/common.yaml.docs/workstream-a.md:245nominates this SQL as "the pattern to copy" for Poland's #106. That means re-implementing per-cell union + class priority + clip + carve + class CASE +part_keyby hand inplmap.yaml, including finding 1, which would then need fixing in N places. The importer already owns coverage semantics (generalization_strategy: cover,partition_by,absorb_boundary); an overlay carve belongs there as a declared staging kind — base table, overlay table, class map, priority — rather than as copy-pasted SQL per country.Checked and clean
sea(2631): 8 685 polygons touched, 244 m² total.db: se_mark_t10is thedatabases:source key, not a target table (importer/staging.go:41) — no clobbering ofse_mark_t10; the table name comes fromname:.importer/staging.go:71,:819), which is what makes the LATERAL carve tractable at all.se_cover_t10: 100% POLYGON across all 5 619 196 rows.Edited after running
ST_CoverageInvalidEdges: the first version of this comment said the change satisfied R35, on the strength of the PR's per-rowST_IsValidevidence. Finding 1 corrects that.Reply to the #116 review — all findings addressed, reworked carve verified
Both holding findings and the quality findings are fixed by a reworked
se_wetland_t10/se_cover_t10pair, verified on the live DB (staging built and z13 imported; the cascade is finishing as this is posted).1. Coverage validity (holding) — fixed: 1 415 → 0 invalid-edge rows on partition 23105
The carve now passes
ST_CoverageInvalidEdges— 0 invalid-edge rows on the reviewer's own partition 23105 (and 22104), vs the pre-carve baseline of 3. The changes, each measured on the live DB:ST_CoverageClean(per class). Exact shared edges; the union's residual 307 sub-micron self-overlaps and the degenerate sub-millimetre pieces are gone — the latter being exactly what GEOS 3.14's prepared-intersects path hangs on (measured at 99.9% CPU,PreparedPolygonIntersectsin the stack).markhost is clipped to its 100 km cells first (single-cell hosts pass through; crossing hosts useST_ClipByBox2D— the box-clipper, robust where the full-overlayST_Intersectionthrew "side location conflict" on boundaries grazing the shared line). Every operation below is cell-sized: the per-host lateral of the first submission hung three different ways (the prepared-intersects above;ST_Union'sClassicUnionStrategypairwise noding — replaced byST_UnaryUnion(ST_Collect(…)), one noding pass, after the raw MultiPolygon threw TopologyException on cell 14,83; and agistgettuplewalk for giant host bboxes — gone with cell-sized bboxes).ST_MakeValid-ed once in a materialized CTE (finding 2's binding, and the fix for the 23rd invalid host a first screen missed —ST_IsValidis nondeterministic on borderline geometries, so the repair is applied to all hosts, which a full-table screen showed is cheap).ST_CoverageClean(gapmaximumwidth 1.0) removes the sub-millimetre difference residue (finding 3: 78 109 rows < 0.001 m² → 456) and fills the gaps their epsilon-drop leaves; because pieces never cross cell lines, each clean window is a closed coverage — which is why the result is 0, not ~3.2. Predicate/subtraction on different geometries (holding) — fixed
ST_MakeValid(m.geom)is bound once (materializedmvCTE) and used for both theST_Intersectspredicate and the subtraction — a piece that intersects the repaired host cannot be missed.3. Manufactured slivers — fixed
78 109 rows < 0.001 m² → 456 in the rebuilt staging (0.008% of the old count); the epsilon-drop plus the per-cell clean also shortens the z11 absorption pass.
4. "~1% gap" claim — corrected
Measured
Σcover − Σmark_included= 23 629 m² (0.008% of the wetland area). Corrected inconf/semap.yamlanddocs/country-sweden.md.5. part_key from the parent — fixed
Both branches derive
part_keyfrom the dumped piece's centroid: 45 714 mismatches → 0.6. Stale comment numbers — corrected
"~4.8M polygons" → 5 383 862 = 4 232 658 carved + 1 386 538 wetland... (actual: 3 997 444 carved + 1 386 418 wetland in the rebuilt output — the cell clipping fragments the base differently). "15 from 17" → 15 from 19, in
docs/country-sweden.md.7. Stale non-conformance row — annotated
docs/country-requirements.mdrow 10 marked "Fixed 2026-08-13 in #116"; the z4→z3 collapse filed as #117 with row 10b.8. Smaller items
Per-cell difference after clipping (
(F∩S)\(V∩S)),FULL OUTER JOIN … USING,ST_CollectionExtract(…, 3)guard — done. The four-waypart_keyformula duplication: acknowledged, needs a shared SQL function; tracked in #118.9. Above the line — filed
z4→z3 wetland collapse: #117 with measurements, recorded in
conf/common.yaml. Overlay carve as an importer staging kind for #106: #118;docs/workstream-a.mdpoints at it.One infrastructure finding from the rework
LLVM's JIT compilation of these giant geometry queries hung at 100% CPU for 15+ minutes with zero progress (the same query at
jit=offruns ~1 100 rows/s). PostGIS functions are C — JIT buys nothing.ALTER DATABASE maps SET jit = offis inscripts/setup-db.shand applied to the live DB.The PR body (note) carries the updated design and measured numbers; the zoom-level verification (15 classes at z13, wetland ladder, partitions, decoded tiles, CDP render) is being re-run over the finishing cascade and will be in the final note.