art/ismap
1
0
Fork 0
Code Issues 56 Pull requests 1 Projects Releases Packages Wiki Activity Actions

world: carve Natural Earth with Natural Earth, not with our own country masks (#184) #185

Merged
art merged 3 commits from issue-184-world-carve into main 2026-08-20 09:53:27 +00:00
Conversation 0 Commits 3 Files changed 12 +368 -35
art-bot commented 2026-08-20 08:40:02 +00:00
Collaborator
Copy link

Fixes #184. Reported against the Greece pilot (#176) as grey wedges in the Gulf of Corinth,
then again — after the first version of this PR — as big chunks of Norway drawn as water
along its borders with Sweden and Finland. Both are the same root cause, and both are fixed
here. Follow-up: #186.

The bug, in both directions

background_land was Natural Earth land minus the union of every map_country_mask_0
partition: a precise national geometry subtracted from a coarse global one. Measured
before this change:

country NE land left inside it foreign NE land it carved away
se 10 821 km² 1 579 km²
gr 6 429 km² 2 089 km²
fi 6 169 km² 3 987 km²
pl 3 011 km² 6 992 km²
is 0 km² 0 km²

Iceland scores zero on both only because its mask is a 25 km buffered convex hull that
swallows Natural Earth's Iceland whole — not available to a country with land neighbours.

No mask geometry fixes this, and #176 proved it: extending Greece's mask with the WFD
coastal water bodies took the Gulf of Corinth window from 2 429 km² to 40, and the map still
showed a 1 km line straight through it, because coastal water reaches ~2 km and Natural
Earth's error in an open gulf is tens of kilometres.

The fix, part one — carve Natural Earth with Natural Earth

ne_10m_admin_0_countries: same scale as the land layer, same source geometry, one polygon
per country with an ISO code. Subtracting that is exact, because it is subtracting a shape
from itself. Which countries get carved is still driven by map_country_mask_0 — the hole
tracks the data we have, countries go in and out one at a time — but only the country code
is read, never the geometry.

The fix, part two — 1:10m, because the residual scales with generalization

The carve is exact, but Natural Earth's idea of where a border runs is not the national
source's, and the strip between them gets carved with nothing to draw it. That is the Norway
bug. It is not fixable by better national geometry — the error is in the other operand — but
it shrinks with Natural Earth's scale:

country carved land nothing fills, 1:110m at 1:10m
se 11 556 km² (6 074 at a land border) 374
fi 7 938 km² (4 124 at a border) 736
gr 20 527 km² (1 475 at a border; the rest is coast, where sea is right) 2 284
pl 3 011 km² (2 860 at a border) 1 930

~14 500 km² of land border drawn as water → ~3 300, about a 140 m strip, sub-pixel at every
zoom this layer draws. Tile budgets still pass.

Measured and rejected

A protection band along foreign borders — carve Natural Earth's country but stop short
near a neighbour — trades the two errors roughly 1:1 at every width and both scales. At 1:10m
with a 2 km band, Poland's unfilled land drops 1 930 → 626 km² while 3 034 km² of Natural
Earth land comes back inside Poland. Not in the config; the numbers are in the commit
message so nobody rediscovers it.

Verification

Rendered over CDP in real time (R127), the views from both bug reports: the Gulf of Corinth is
water with no wedge and no line; the Athens coast at z10 has no fringe; the Sweden/Norway
border meets Sweden's cover cleanly with Norway's fjords properly resolved; Poland's eastern
border is clean. make acceptance green apart from the pre-existing Poland buildings R120
timeout (#166, fails identically on main). Go build/vet/test green.

Four tests, because the checks this replaces could not fail

The old carve check — "no background land overlaps the mask" — is tautologically true:
background_land is Natural Earth minus the mask. It passed for five countries while the
map was visibly wrong.

  • R144 — every imported country matches exactly one admin-0 polygon (Natural Earth writes
    -99 for some disputed states; an unmatched country is silently never carved), and no
    background land is left inside an imported country.
  • R146 — the test R145 could not be. R145 asks whether a country keeps the land Natural
    Earth says     is its own, so it agreed with Natural Earth's opinion of the border and was
    blind to the Norway bug. R146 asks how much Natural Earth land we carved that the country's
    own boundary disclaims. That is the one place the national mask geometry is still worth
    reading — as a check, not an input.
  • R145 — rewritten as an area identity after the honest per-country sweep turned out to
    cost minutes against 1:10m geometry: background + carve must add back up to Natural Earth
    land. Live: 145 515 244 + 1 326 411 = 146 841 655 km², exact to the km², 0.4 s. It still
    catches what it was written for — an experimental 1 m grid size on ST_Difference cost
    New Zealand 3 813 km², Papua New Guinea 198 and Japan 91, nowhere near any carve,
    because a grid size snaps the inputs too. Caught, measured, reverted.

Also in here

The shapefile loader could not read Natural Earth's admin-0 file at all: its .dbf is
NUL-padded rather than space-padded — 15 917 cells — and Postgres rejects a NUL inside
text outright (SQLSTATE 22021), so the whole COPY died on a file that is not malformed.
sanitizeDBFText strips the padding and replaces invalid UTF-8, with a unit test.

Follow-ups

  • #186 — the residual is a missing maritime boundary. Coasts and land borders ask the same
    question ("Natural Earth says land, our country says nothing") with opposite right answers,
    and only a land+territorial-waters polygon can separate them. Includes the analysis of
    inverting the background (base = land, draw Natural Earth water), which fixes borders
    structurally but moves the same ambiguity to the coast.
  • Poland's 1 930 km² is the outlier by 3× and its land borders render clean, so it is
    probably concentrated somewhere specific and may be Poland's own mask rather than Natural
    Earth. Localisation query in #186.
  • #176 rebases onto this and drops its coastal-water mask work, which this makes unnecessary.
Fixes #184. Reported against the Greece pilot (#176) as grey wedges in the Gulf of Corinth, then again — after the first version of this PR — as **big chunks of Norway drawn as water** along its borders with Sweden and Finland. Both are the same root cause, and both are fixed here. Follow-up: #186. ## The bug, in both directions `background_land` was Natural Earth land minus the union of every `map_country_mask_0` partition: a **precise national geometry subtracted from a coarse global one**. Measured before this change: | country | NE land left **inside** it | foreign NE land it **carved away** | |---|---|---| | `se` | **10 821 km²** | 1 579 km² | | `gr` | **6 429 km²** | 2 089 km² | | `fi` | **6 169 km²** | 3 987 km² | | `pl` | **3 011 km²** | 6 992 km² | | `is` | 0 km² | 0 km² | Iceland scores zero on both **only because its mask is a 25 km buffered convex hull** that swallows Natural Earth's Iceland whole — not available to a country with land neighbours. **No mask geometry fixes this**, and #176 proved it: extending Greece's mask with the WFD coastal water bodies took the Gulf of Corinth window from 2 429 km² to 40, and the map still showed a 1 km line straight through it, because coastal water reaches ~2 km and Natural Earth's error in an open gulf is tens of kilometres. ## The fix, part one — carve Natural Earth with Natural Earth `ne_10m_admin_0_countries`: same scale as the land layer, same source geometry, one polygon per country with an ISO code. Subtracting that is exact, because it is subtracting a shape from itself. Which countries get carved is **still** driven by `map_country_mask_0` — the hole tracks the data we have, countries go in and out one at a time — but only the *country code* is read, never the geometry. ## The fix, part two — 1:10m, because the residual scales with generalization The carve is exact, but Natural Earth's idea of where a **border** runs is not the national source's, and the strip between them gets carved with nothing to draw it. That is the Norway bug. It is not fixable by better national geometry — the error is in the other operand — but it shrinks with Natural Earth's scale: | country | carved land nothing fills, 1:110m | at 1:10m | |---|---|---| | `se` | 11 556 km² (6 074 at a land border) | **374** | | `fi` | 7 938 km² (4 124 at a border) | **736** | | `gr` | 20 527 km² (1 475 at a border; the rest is coast, where sea is right) | **2 284** | | `pl` | 3 011 km² (2 860 at a border) | **1 930** | ~14 500 km² of land border drawn as water → ~3 300, about a 140 m strip, sub-pixel at every zoom this layer draws. Tile budgets still pass. ## Measured and rejected A **protection band** along foreign borders — carve Natural Earth's country but stop short near a neighbour — trades the two errors roughly 1:1 at every width and both scales. At 1:10m with a 2 km band, Poland's unfilled land drops 1 930 → 626 km² while **3 034 km² of Natural Earth land comes back inside Poland**. Not in the config; the numbers are in the commit message so nobody rediscovers it. ## Verification Rendered over CDP in real time (R127), the views from both bug reports: the Gulf of Corinth is water with no wedge and no line; the Athens coast at z10 has no fringe; the Sweden/Norway border meets Sweden's cover cleanly with Norway's fjords properly resolved; Poland's eastern border is clean. `make acceptance` green apart from the pre-existing Poland buildings R120 timeout (#166, fails identically on `main`). Go build/vet/test green. ## Four tests, because the checks this replaces could not fail The old carve check — "no background land overlaps the mask" — is **tautologically true**: `background_land` *is* Natural Earth minus the mask. It passed for five countries while the map was visibly wrong. - **R144** — every imported country matches exactly one admin-0 polygon (Natural Earth writes `-99` for some disputed states; an unmatched country is silently never carved), and no background land is left inside an imported country. - **R146** — the test R145 could not be. R145 asks whether a country keeps the land *Natural Earth says* is its own, so it **agreed with Natural Earth's opinion of the border** and was blind to the Norway bug. R146 asks how much Natural Earth land we carved that the country's **own** boundary disclaims. That is the one place the national mask geometry is still worth reading — as a check, not an input. - **R145** — rewritten as an area identity after the honest per-country sweep turned out to cost minutes against 1:10m geometry: background + carve must add back up to Natural Earth land. Live: 145 515 244 + 1 326 411 = 146 841 655 km², exact to the km², 0.4 s. It still catches what it was written for — an experimental 1 m grid size on `ST_Difference` cost **New Zealand 3 813 km²**, Papua New Guinea 198 and Japan 91, nowhere near any carve, because a grid size snaps the inputs too. Caught, measured, reverted. ## Also in here The shapefile loader could not read Natural Earth's admin-0 file at all: its `.dbf` is **NUL-padded** rather than space-padded — 15 917 cells — and Postgres rejects a NUL inside text outright (SQLSTATE 22021), so the whole COPY died on a file that is not malformed. `sanitizeDBFText` strips the padding and replaces invalid UTF-8, with a unit test. ## Follow-ups - **#186** — the residual is a missing maritime boundary. Coasts and land borders ask the same question ("Natural Earth says land, our country says nothing") with opposite right answers, and only a land+territorial-waters polygon can separate them. Includes the analysis of inverting the background (base = land, draw Natural Earth water), which fixes borders structurally but moves the same ambiguity to the coast. - **Poland's 1 930 km²** is the outlier by 3× and its land borders render clean, so it is probably concentrated somewhere specific and may be Poland's own mask rather than Natural Earth. Localisation query in #186. - #176 rebases onto this and drops its coastal-water mask work, which this makes unnecessary.
background_land was Natural Earth land minus the union of every country mask.
That subtracts a precise national geometry from a coarse global one, and the two
disagree along every kilometre of coast and border, so it fails in both
directions at once — for every country, since the layer was unified. Measured on
the live database:

  country   NE land left INSIDE it     foreign NE land it carved AWAY
  se                 10 821 km²                        1 579 km²
  gr                  6 429 km²                        2 089 km²
  fi                  6 169 km²                        3 987 km²
  pl                  3 011 km²                        6 992 km²
  is                        0 km²                            0 km²

The left column is what you see: grey wedges filling the Gulf of Corinth, blobs
strewn round Crete, a ragged fringe along every coast at high zoom. It was
reported against the Greece pilot, but Greece is not special — Sweden is worse
and nobody had looked. The right column is the same error with the sign flipped,
and nobody looks at that at all: Poland removes 6 992 km² of its neighbours'
background, because a 1:50 000 border crosses Natural Earth's 1:110 m version of
the same border constantly. Iceland scores zero on both only because its mask is
a 25 km buffered convex hull that swallows Natural Earth's Iceland whole, which
is not a trick available to a country with land neighbours.

No mask geometry fixes this, and the Greece pilot proved it: extending the mask
with the WFD coastal water bodies took the Gulf of Corinth window from 2 429 km²
to 40, and the map still looked wrong, because 40 km² along 40 km of gulf is a
1 km line straight through it. Natural Earth's error in the open gulfs is tens of
kilometres; coastal water reaches about two. The country whose data we have is
not the shape Natural Earth thinks it is, and better national geometry does not
converge because the error is in the other operand.

So the carve is ne_110m_admin_0_countries — same scale as the land layer, cut
from the same source geometry, one polygon per country with an ISO code.
Subtracting that is exact, because it is subtracting a shape from itself. Which
countries get carved is still driven by map_country_mask_0, so the hole tracks
the data we have and countries can be added or removed one at a time, but only
the country code is read from it and never the geometry. After: 0.0 km² inside
every imported country (Iceland 4.8 km² of hairline sliver, under its own cover
where nothing draws it) and 0 km² of foreign land carved, across all 170
countries we do not import. Rendered and checked over CDP: the Gulf of Corinth is
water, and the Athens coastline has no fringe.

Three tests, because the check this replaces could not fail. "No background land
overlaps the mask" is tautologically true — background_land IS Natural Earth
minus the mask — so it passed for every country throughout. R144 asserts every
imported country matched an admin-0 polygon (Natural Earth writes -99 for some
disputed states, and an unmatched country is silently never carved) and that no
background land is left inside one. R145 asserts no country we have not imported
loses its land.

R145 earned its keep immediately. Cleaning up the last hairline slivers with a
1 m grid size on ST_Difference cleared them — Iceland to zero, every country
under 0.4 km² — and silently cost New Zealand 3 813 km², Papua New Guinea 198 and
Japan 91, because a grid size snaps the inputs too. Caught, measured, reverted;
the slivers stay.

The shapefile loader could not read ne_110m_admin_0_countries at all: its .dbf is
NUL-padded rather than space-padded, 15 917 cells of it, and Postgres rejects a
NUL inside text outright, so the COPY died on a file that is not malformed.
sanitizeDBFText strips the padding and replaces invalid UTF-8 rather than failing
the load.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8HxnPejASTBhLaSxJKqBA
Carving Natural Earth with Natural Earth is exact, but Natural Earth's idea of
where a border runs is not the national source's idea, and the strip between them
gets carved with nothing to draw it. It renders as sea. At 1:110m that was
14 500 km² of land border drawn as water — most visibly along Norway's borders
with Sweden and Finland, where Natural Earth's line sits about 4 km inside
Sweden's own. Reported from the map, which is the only place it showed:

  country   carved land nothing fills     of which at a land border
  se                    11 556 km²                     6 074 km²
  fi                     7 938 km²                     4 124 km²
  pl                     3 011 km²                     2 860 km²
  gr                    20 527 km²                     1 475 km²

Greece's total is mostly coast, where sea is the right answer. The other three are
neighbours' land drawn as water.

The error is Natural Earth disagreeing with a national source, so it scales with
the generalization. At 1:10m the same measurement is 3 300 km² — se 227, fi 627,
gr 515, pl 1 930 — roughly a 140 m strip, sub-pixel at every zoom this layer
draws. Rendered and checked: the Sweden/Norway border meets Sweden's cover
cleanly, and so does Poland's eastern border. Tile budgets still pass, so the
finer geometry costs nothing that matters at maxz 10.

A protection band along foreign borders was tried and measured and does not work:
carve Natural Earth's country but stop short near a neighbour, and at 1:10m with a
2 km band Poland's unfilled land drops 1 930 -> 626 km² while 3 034 km² of
Natural Earth land comes back inside Poland. Roughly a 1:1 trade at every width
tried, and worse at 1:110m. The band is not in the config; the measurement is in
the commit message so nobody has to rediscover it.

R146 is the test R145 could not be. R145 asks whether a country keeps the land
Natural Earth says is its own, so it agreed with Natural Earth's opinion of the
border — which is the thing that was wrong — and passed throughout. R146 asks the
other question: how much Natural Earth land did we carve that the country's OWN
boundary disclaims. That is the one place the national mask geometry is still
worth reading, and it reads it as a check rather than an input.

R145 itself is rewritten as an area identity. The honest per-country sweep costs
minutes against 1:10m geometry, where the land layer is eleven polygons of several
million vertices; background_land is Natural Earth land minus the carve, so their
areas must add back up to it. Live: 145 515 244 + 1 326 411 = 146 841 655 km²,
exact to the km², in 0.4 s. It still catches what it was written for — the 1 m
grid size that cost New Zealand 3 813 km² is a 4 100 km² imbalance against a
1 000 km² budget.

Poland's 1 930 km² is the remaining outlier by a factor of three and is more
likely its mask than Natural Earth; it gets its own issue rather than a wider
budget here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8HxnPejASTBhLaSxJKqBA
The R146 test landed in conf/world.yaml naming a requirement that did not exist.
It is the one failure mode R144 and R145 cannot see: R145 asks whether a country
keeps the land Natural Earth says is its own, so it agrees with Natural Earth
about where the border runs, which is exactly the thing that is wrong. That is how
14 500 km² of land border came to be drawn as water at 1:110m without a single
test going red — it was found by looking at the map.

The rule also records what the residual is and is not. It is irreducible without a
maritime boundary, because a coast and a land border produce the identical query
result — "Natural Earth says land, our country says nothing" — with opposite
correct answers (#186). And the real fix for one specific border is to import the
country on the other side of it, which makes both sides carve their own Natural
Earth polygon and fills the strip with real cover.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q8HxnPejASTBhLaSxJKqBA
art merged commit d60a13d7b5 into main 2026-08-20 09:53:27 +00:00
art deleted branch issue-184-world-carve 2026-08-20 09:53:27 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
issue is stuck: missing decision, credential, or data (work-issue blocked path)

  
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap!185
No description provided.