art/ismap
1
0
Fork 0
Code Issues 56 Pull requests 1 Projects Releases Packages Wiki Activity Actions

generalization: the parallel-merge corridor is flat-capped — issue #170 #196

Merged
art merged 1 commit from issue-170-parallel-merge-endcap into main 2026-08-20 17:50:04 +00:00
Conversation 0 Commits 1 Files changed 5 +157 -1
art-bot commented 2026-08-20 17:19:55 +00:00
Collaborator
Copy link

Closes #170

What was wrong

parallel_merge_pixels drops a chain when 95% of its length lies inside a
better-ranked chain's corridor. The corridor was built with ST_Buffer's
default round end caps, so it bulged a full radius past the better-ranked
chain's endpoints — and the coverage test then read "same line at this zoom"
for two situations that are nothing alike:

  • a chain running beside b — the second carriageway the stage exists to
    collapse, and
  • a chain carrying on from b's end, which is the rest of the road.

Any chain shorter than about 2 d that touched a better-ranked one was swallowed
whole. d is one screen pixel, so it doubles every zoom down — 611 m at z8,
1 223 m at z7, 2 446 m at z6 — and each level derives from the one above, so the
loss compounds. Cities are worst because that is where merge_lines cuts a road
into short chains: name, surface, and above all level, which changes at
every interchange bridge.

That is the whole of #170: the E4 in loose pieces at z7 and shattered at z6
(Sundsvall), E4 and E12 the same near Umeå, "all the orange roads chopped up"
around Örebro.

The fix

One argument — ST_Buffer(b.geom, d, 'endcap=flat') in mergeLinesSQL
(importer/generalize.go). The corridor now ends where b ends, so lateral
containment still counts and continuation does not. The ST_DWithin prefilter
is unchanged: a flat corridor is a subset of the round one it bounded.

Shared mechanism, so it applies to every layer that sets parallel_merge_pixels
— roads and railways in all countries, transit in se/pl/fi.

Numbers

Measured on Sweden's z7 roads pass, replaying the real cascade (the replay
reproduces the live tables exactly — z6 E4 came out at 474 features / 2 401 km,
byte-for-byte the live map_roads_6_se):

round caps (before) flat caps (after)
chains deleted at z7 1 718 / 1 641 km 142 / 173 km
of that, with no surviving line within 200 m 1 509 km (92%) 120 km
duplicate motorway carriageway removed at z11 1 244 km 1 167 km

Trunk-and-above network, live, z10 → z6:

before after
se 16 107 → 11 435 km (71%) 16 466 → 14 536 km (88%)
pl 9 936 → 7 078 km (71%) 10 153 → 9 013 km (89%)
fi 19 673 → 17 311 km (88%) 19 882 → 19 169 km (96%)
is 10 456 → 10 054 km (96%) 10 461 → 10 088 km (96%)

Railways gained too, less dramatically (z11 → z6, km): fi 12 778 → 13 819,
pl 31 509 → 32 128, se 21 565 → 21 591. Transit is unchanged to within a
kilometre — its corridors are 0.25–1 px on networks a few hundred km long.

The acceptance test

R147 on the roads contract (conf/shared/contracts/08-roads.yaml), new
requirement in docs/country-requirements.md §11:

at z6, ≥ 98% of each country's z10 trunk-and-above network still has a
trunk-and-above line within half a pixel of it

Half a pixel is the z6 simplification tolerance, so the allowance is
displacement and nothing else. Coverage rather than length retention, because a
collapsed dual carriageway should halve the length and a length test cannot
tell that from a hole. z10 against z6 rather than adjacent levels, because the
defect was invisible one step at a time — z7 against z6 scored 99.1% while the
map was in pieces.

It fails on the old data and passes on the new:

before: FAIL  per_country=fi=98.1%, gr=99.8%, is=99.9%, pl=95.0%, se=92.5%
after:  PASS  per_country=fi=100.0%, gr=99.8%, is=99.9%, pl=100.0%, se=99.9%

(That is not a reconstruction: the fix was stashed, the binary rebuilt, Sweden's
roads regeneralized, the test re-run, and then the fix restored and the data
rebuilt — which is also how the before/after renders below were taken.)

Verification

  • make build, make test — clean. New unit test
    TestParallelCorridorIsFlatCapped locks the end-cap style into the generated
    SQL.

  • Regeneralized: roads z5–13 for is/se/pl/fi, railways z5–11 and transit z9–13
    for se/pl/fi. ~3 min per country for roads, seconds for the rest.

  • Per-zoom sanity, Sweden roads: all ten z5–z14 partitions populated, 0 NULL
    road_class, 0 empty or invalid geometries. Row counts z5–z14: 8 977 / 9 453
    / 16 229 / 16 992 / 41 286 / 45 662 / 94 250 / 628 008 / 2 191 611 /
    3 397 786.

  • Decoded tiles (scripts/mvt-decode.py): z7 70/35 → roads 295 features; z6
    35/17 → roads 587.

  • R101 pixel diff, real Chrome over CDP at the three reported locations,
    before and after, map area only:

    changed road ink gained lost
    Sundsvall z7 0.97% 776 px 252 px
    Umeå z7.78 0.39% 603 px 156 px
    Örebro z6.73 0.61% 2 149 px 93 px

    The gained pixels trace whole corridors — E4 Stockholm–Uppsala–Gävle, E18
    Örebro–Västerås, E20 — filling exactly the gaps that were reported. The lost
    pixels are scattered clusters of ≤ 8 px: label and shield placement shifting,
    and antialiasing. No stretch of road went missing.

Acceptance tallies (fast bucket unless noted):

── conf/is ──   39 test(s): 38 passed, 1 failed, 0 skipped, 0 not run
── conf/se ──   46 test(s): 45 passed, 1 failed, 0 skipped, 0 not run   (--acceptance all)
── conf/pl ──   47 test(s): 44 passed, 2 failed, 0 skipped, 1 not run
── conf/fi ──   50 test(s): 49 passed, 1 failed, 0 skipped, 0 not run
── conf/gr ──   29 test(s): 28 passed, 1 failed, 0 skipped, 0 not run
── conf/world ──29 test(s): 27 passed, 1 failed, 0 skipped, 1 not run

The two failures, both pre-existing and neither mine

  • R25 (level), every config. Greece's roads partition is all-NULL because
    the TEN-T delivery has no grade-separation column. The written exemption for
    that is on gr-roads-rail-waterways, which has not merged; the database has
    Greece's roads from that branch while main's contract does not know about
    them. It fails identically on main with this branch's changes reverted.
  • R120 (Poland buildings), conf/pl. context deadline exceeded, reproducible
    on a re-run in isolation. The buildings layer is untouched here — nothing in
    this change reads or writes it.

Left undone, deliberately

  • Greece's roads and railways still carry the old cascade (its R147 score is
    99.8%, so it passes on the strength of a sparse network, not on the strength
    of the code that built it). conf/gr/04-roads.yaml and 05-railways.yaml
    live on the unmerged gr-roads-rail-waterways branch, so there is no config on
    main to regeneralize from. Whoever lands that branch should run
    importer/importer import --layer roads --regeneralize 5-13 conf/gr and the
    railways equivalent.
  • R95's margin narrowed and this is the real cost of the fix: Sweden's z5
    roads table goes from 1 043 features to 8 977, average feature length 9.1 km →
    1.6 km against R95's 1 km floor, and the z5 Stockholm tile from 335 kB to
    381 kB against a 512 kB budget. Those are the short chains that were being
    deleted — they are the road, and they are contiguous now — but they are
    fragments because level splits a chain at every interchange. Dropping level
    from the merge key at coarse zoom halves the z6 chain count (9 853 → 4 679).
    That is issue #167, which until now had an argument but no number; it now
    has one, and it is worth doing before R95's floor is actually reached.
Closes #170 --- ## What was wrong `parallel_merge_pixels` drops a chain when 95% of its length lies inside a better-ranked chain's corridor. The corridor was built with `ST_Buffer`'s default **round end caps**, so it bulged a full radius past the better-ranked chain's *endpoints* — and the coverage test then read "same line at this zoom" for two situations that are nothing alike: * a chain running **beside** b — the second carriageway the stage exists to collapse, and * a chain **carrying on from** b's end, which is the rest of the road. Any chain shorter than about 2 d that touched a better-ranked one was swallowed whole. `d` is one screen pixel, so it doubles every zoom down — 611 m at z8, 1 223 m at z7, 2 446 m at z6 — and each level derives from the one above, so the loss compounds. Cities are worst because that is where `merge_lines` cuts a road into short chains: `name`, `surface`, and above all `level`, which changes at every interchange bridge. That is the whole of #170: the E4 in loose pieces at z7 and shattered at z6 (Sundsvall), E4 and E12 the same near Umeå, "all the orange roads chopped up" around Örebro. ## The fix One argument — `ST_Buffer(b.geom, d, 'endcap=flat')` in `mergeLinesSQL` (`importer/generalize.go`). The corridor now ends where `b` ends, so lateral containment still counts and continuation does not. The `ST_DWithin` prefilter is unchanged: a flat corridor is a subset of the round one it bounded. Shared mechanism, so it applies to every layer that sets `parallel_merge_pixels` — roads and railways in all countries, transit in se/pl/fi. ## Numbers Measured on Sweden's z7 roads pass, replaying the real cascade (the replay reproduces the live tables exactly — z6 E4 came out at 474 features / 2 401 km, byte-for-byte the live `map_roads_6_se`): | | round caps (before) | flat caps (after) | |---|---|---| | chains deleted at z7 | 1 718 / 1 641 km | 142 / 173 km | | of that, with **no** surviving line within 200 m | 1 509 km (92%) | 120 km | | duplicate motorway carriageway removed at z11 | 1 244 km | 1 167 km | Trunk-and-above network, live, z10 → z6: | | before | after | |---|---|---| | se | 16 107 → 11 435 km (71%) | 16 466 → 14 536 km (88%) | | pl | 9 936 → 7 078 km (71%) | 10 153 → 9 013 km (89%) | | fi | 19 673 → 17 311 km (88%) | 19 882 → 19 169 km (96%) | | is | 10 456 → 10 054 km (96%) | 10 461 → 10 088 km (96%) | Railways gained too, less dramatically (z11 → z6, km): fi 12 778 → 13 819, pl 31 509 → 32 128, se 21 565 → 21 591. Transit is unchanged to within a kilometre — its corridors are 0.25–1 px on networks a few hundred km long. ## The acceptance test **R147** on the `roads` contract (`conf/shared/contracts/08-roads.yaml`), new requirement in `docs/country-requirements.md` §11: > at z6, ≥ 98% of each country's z10 trunk-and-above network still has a > trunk-and-above line within half a pixel of it Half a pixel is the z6 simplification tolerance, so the allowance is displacement and nothing else. Coverage rather than length retention, because a collapsed dual carriageway *should* halve the length and a length test cannot tell that from a hole. z10 against z6 rather than adjacent levels, because the defect was invisible one step at a time — z7 against z6 scored 99.1% while the map was in pieces. It fails on the old data and passes on the new: ``` before: FAIL per_country=fi=98.1%, gr=99.8%, is=99.9%, pl=95.0%, se=92.5% after: PASS per_country=fi=100.0%, gr=99.8%, is=99.9%, pl=100.0%, se=99.9% ``` (That is not a reconstruction: the fix was stashed, the binary rebuilt, Sweden's roads regeneralized, the test re-run, and then the fix restored and the data rebuilt — which is also how the before/after renders below were taken.) ## Verification * `make build`, `make test` — clean. New unit test `TestParallelCorridorIsFlatCapped` locks the end-cap style into the generated SQL. * Regeneralized: roads z5–13 for is/se/pl/fi, railways z5–11 and transit z9–13 for se/pl/fi. ~3 min per country for roads, seconds for the rest. * Per-zoom sanity, Sweden roads: all ten z5–z14 partitions populated, 0 NULL `road_class`, 0 empty or invalid geometries. Row counts z5–z14: 8 977 / 9 453 / 16 229 / 16 992 / 41 286 / 45 662 / 94 250 / 628 008 / 2 191 611 / 3 397 786. * Decoded tiles (`scripts/mvt-decode.py`): z7 70/35 → roads 295 features; z6 35/17 → roads 587. * **R101 pixel diff**, real Chrome over CDP at the three reported locations, before and after, map area only: | | changed | road ink gained | lost | |---|---|---|---| | Sundsvall z7 | 0.97% | 776 px | 252 px | | Umeå z7.78 | 0.39% | 603 px | 156 px | | Örebro z6.73 | 0.61% | 2 149 px | 93 px | The gained pixels trace whole corridors — E4 Stockholm–Uppsala–Gävle, E18 Örebro–Västerås, E20 — filling exactly the gaps that were reported. The lost pixels are scattered clusters of ≤ 8 px: label and shield placement shifting, and antialiasing. No stretch of road went missing. Acceptance tallies (fast bucket unless noted): ``` ── conf/is ── 39 test(s): 38 passed, 1 failed, 0 skipped, 0 not run ── conf/se ── 46 test(s): 45 passed, 1 failed, 0 skipped, 0 not run (--acceptance all) ── conf/pl ── 47 test(s): 44 passed, 2 failed, 0 skipped, 1 not run ── conf/fi ── 50 test(s): 49 passed, 1 failed, 0 skipped, 0 not run ── conf/gr ── 29 test(s): 28 passed, 1 failed, 0 skipped, 0 not run ── conf/world ──29 test(s): 27 passed, 1 failed, 0 skipped, 1 not run ``` ## The two failures, both pre-existing and neither mine * **R25 (`level`), every config.** Greece's roads partition is all-NULL because the TEN-T delivery has no grade-separation column. The written exemption for that is on `gr-roads-rail-waterways`, which has not merged; the database has Greece's roads from that branch while `main`'s contract does not know about them. It fails identically on `main` with this branch's changes reverted. * **R120 (Poland buildings), `conf/pl`.** `context deadline exceeded`, reproducible on a re-run in isolation. The buildings layer is untouched here — nothing in this change reads or writes it. ## Left undone, deliberately * **Greece's roads and railways still carry the old cascade** (its R147 score is 99.8%, so it passes on the strength of a sparse network, not on the strength of the code that built it). `conf/gr/04-roads.yaml` and `05-railways.yaml` live on the unmerged `gr-roads-rail-waterways` branch, so there is no config on `main` to regeneralize from. Whoever lands that branch should run `importer/importer import --layer roads --regeneralize 5-13 conf/gr` and the railways equivalent. * **R95's margin narrowed** and this is the real cost of the fix: Sweden's z5 roads table goes from 1 043 features to 8 977, average feature length 9.1 km → 1.6 km against R95's 1 km floor, and the z5 Stockholm tile from 335 kB to 381 kB against a 512 kB budget. Those are the short chains that were being deleted — they are the road, and they are contiguous now — but they are fragments because `level` splits a chain at every interchange. Dropping `level` from the merge key at coarse zoom halves the z6 chain count (9 853 → 4 679). That is **issue #167**, which until now had an argument but no number; it now has one, and it is worth doing before R95's floor is actually reached.
A road that carries on from another road is not the same line as the road it
carries on from. The parallel-merge stage could not tell the difference, and
deleted the continuation along with the duplicate.

The corridor a chain is tested against was built with ST_Buffer's default round
end caps, so it bulged a full radius past the better-ranked chain's endpoints.
The 95%-coverage test then read "these are the same line at this zoom" for two
situations that are nothing alike: a chain running beside b — the second
carriageway the stage exists to collapse — and a chain continuing on from b's
end, which is the rest of the road. Every chain shorter than about 2 d that
touched a better-ranked one was swallowed whole. d is one screen pixel and
doubles every zoom down (611 m at z8, 1 223 m at z7, 2 446 m at z6), so the
damage compounded through the cascade, and worst in cities, which is exactly
where merge_lines cuts a road into short chains: `name`, `surface` and above all
`level`, a bridge at every interchange.

Measured on Sweden's z7 roads before the fix: the rule deleted 1 718 chains /
1 641 km, of which 1 509 km — 92% — had no surviving line within 200 m. Not
duplicates, holes. Sweden's trunk-and-above network fell from 16 107 km at z10
to 11 435 km at z6; Poland lost the same 29%, Finland 12%. On the map that was
the E4 in loose pieces at z7 and shattered at z6, which is what #170 reported
from three places.

ST_Buffer(b.geom, d, 'endcap=flat') is the fix. The corridor now ends where b
ends, lateral containment still counts, and the same z7 pass deletes 142 chains
/ 173 km. The work the stage is for is untouched: at z11 it still removes
1 167 km of the 1 244 km of duplicate motorway carriageway.

Verified: roads regeneralized z5–13 for is/se/pl/fi, railways z5–11 and transit
z9–13 for se/pl/fi (Greece's roads and railways live on an unmerged branch and
still need the same pass). Sweden's z6 trunk-and-above network is 14 536 km of
16 466 km at z10, 88% against 71% before. Before/after renders of all three
reported locations pixel-diff to road ink *gained* along whole corridors — 2 149
px at Örebro, 776 at Sundsvall, 603 at Umeå — against 93/252/156 px lost in
scattered clusters of ≤ 8 px, which are label shifts and antialiasing (R101).

The invariant is now a test rather than a memory. R147 on the roads contract
asserts that ≥ 98% of each country's z10 trunk-and-above network still has a
trunk-and-above line within half a pixel of it at z6 — half a pixel being the
z6 simplification tolerance, so the allowance is displacement and nothing else.
It fails on the old data with se=92.5%, pl=95.0% and passes on the new with
se=99.9%, pl=100.0%, fi=100.0%, is=99.9%. It is a cascade check that happens to
read roads: any future corridor rule that orphans network fails it.

The cost is chains. Sweden's z5 roads table goes from 1 043 features to 8 977
and its z5 Stockholm tile from 335 kB to 381 kB, against a 512 kB budget, and
R95's margin narrows from 9.1 km average feature length to 1.6 km (floor 1 km).
Most of that is `level` splitting a road at every interchange; dropping it from
the merge key at coarse zoom would halve the z6 chain count (9 853 → 4 679),
which is issue #167 and now has a number attached to it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UGAZTzU6FGJSQKj437rr3L
art merged commit c9f8e48444 into main 2026-08-20 17:50:04 +00:00
art deleted branch issue-170-parallel-merge-endcap 2026-08-20 17:50:04 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
issue is stuck: missing decision, credential, or data (work-issue blocked path)

  
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap!196
No description provided.