background_land carves a precise national mask out of coarse Natural Earth — wrong in both directions, every country #184

New issue

Closed

opened 2026-08-20 08:37:50 +00:00 by art-bot · 0 comments

art-bot commented

2026-08-20 08:37:50 +00:00

Collaborator

background_land is Natural Earth land minus the union of every map_country_mask_0
partition. That subtracts a precise national geometry from a coarse global one, and the
two disagree along every kilometre of coast and border. It fails in both directions at once,
for every country, and has done since the layer was unified.

Measured on the live database:

country	Natural Earth land left inside it	foreign Natural Earth land it carved away
`se`	10 821 km²	1 579 km²
`gr`	6 429 km²	2 089 km²
`fi`	6 169 km²	3 987 km²
`pl`	3 011 km²	6 992 km²
`is`	0 km²	0 km²

The left column is what you see on the map. In Greece it is grey wedges filling the Gulf
of Corinth, blobs strewn around Crete, and a ragged fringe along every coastline at high
zoom. It was reported against the Greece pilot (#176) but Greece is not special — Sweden is
worse, and nobody had looked.

The right column is the same error with the sign flipped, and nobody looks at it at all:
bites taken out of neighbours we have not imported, because a boundary drawn at 1:50 000
crosses Natural Earth's 1:110 m version of the same border constantly. Poland removes
6 992 km² of its neighbours' background.

Iceland scores zero on both only because its mask is a 25 km buffered convex hull that
swallows Natural Earth's Iceland whole. That is not a trick available to a country with land
neighbours: a hull on Greece would carve Natural Earth out from under Albania, North
Macedonia, Bulgaria and Anatolia.

Why no mask geometry can fix it

The Greece pilot tried. The mask was extended with the WFD coastal and transitional water
bodies so it would reach out over the gulfs — 132 041 → 167 843 km², measured, and the Gulf
of Corinth window dropped from 2 429 km² of background land to 40 km². The map still looked
wrong, because 40 km² spread along 40 km of gulf is a 1 km-wide line straight through it,
and because Natural Earth's error in the open gulfs is tens of kilometres while WFD coastal
water reaches about two.

The country whose data we have is simply not the shape Natural Earth thinks it is.
Subtracting one from the other leaves the difference — always, in both directions, at every
zoom. No amount of better national geometry converges, because the error is in the other
operand.

The fix: carve Natural Earth with Natural Earth

Natural Earth also publishes ne_110m_admin_0_countries — the same 1:110 m scale as
ne_110m_land, cut from the same source geometry, one polygon per country with an ISO code.
Subtracting those is exact, because it is subtracting a shape from itself.

Which countries get carved stays driven by map_country_mask_0, so the hole still tracks the
data we actually have and countries can still be added or removed one at a time — but only
the country code is read from it, never the geometry.

Measured after the change: 0.0 km² inside every imported country (Iceland 4.8 km² of
hairline sliver, sitting under its own cover where nothing draws it), and 0 km² of foreign
land carved, checked across all 170 countries we have not imported.

What our own data then does is draw on top, so the coastline you see is the country's own
land cover rather than Natural Earth's. Where Natural Earth claimed land our cover says is
sea, the base map's water shows through — which is right.

Three tests, because the old check could not fail

The carve was "verified" by checking that no background land overlaps the mask. That is
tautologically true — background_land is Natural Earth minus the mask — so it passed
throughout, for every country, while the map was visibly wrong.

R144 — every imported country matches exactly one Natural Earth admin-0 polygon.
Natural Earth writes -99 for a few disputed states; without this a new country is
silently never carved.
R107 — no background land left inside an imported country (the visible bug).
R145 — no country we have not imported loses its Natural Earth land (the invisible
half).

R145 immediately earned its keep: an attempt to clean up the last hairline slivers with a 1 m
grid size on ST_Difference cleared them, and silently cost New Zealand 3 813 km², Papua New
Guinea 198 and Japan 91 — because a grid size snaps the inputs too. Caught, measured,
reverted.

Also fixed on the way

The shapefile loader could not read ne_110m_admin_0_countries at all: its .dbf is
NUL-padded rather than space-padded (15 917 cells), and Postgres rejects a NUL inside text
outright, so the whole COPY died on a file that is not malformed. sanitizeDBFText strips
the padding and replaces invalid UTF-8 instead of failing the load.

`background_land` is Natural Earth land minus the union of every `map_country_mask_0` partition. That subtracts a **precise national geometry from a coarse global one**, and the two disagree along every kilometre of coast and border. It fails in both directions at once, for every country, and has done since the layer was unified. Measured on the live database: | country | Natural Earth land left **inside** it | foreign Natural Earth land it **carved away** | |---|---|---| | `se` | **10 821 km²** | 1 579 km² | | `gr` | **6 429 km²** | 2 089 km² | | `fi` | **6 169 km²** | 3 987 km² | | `pl` | **3 011 km²** | 6 992 km² | | `is` | 0 km² | 0 km² | **The left column is what you see on the map.** In Greece it is grey wedges filling the Gulf of Corinth, blobs strewn around Crete, and a ragged fringe along every coastline at high zoom. It was reported against the Greece pilot (#176) but Greece is not special — Sweden is worse, and nobody had looked. **The right column is the same error with the sign flipped**, and nobody looks at it at all: bites taken out of neighbours we have not imported, because a boundary drawn at 1:50 000 crosses Natural Earth's 1:110 m version of the same border constantly. Poland removes 6 992 km² of its neighbours' background. Iceland scores zero on both only because its mask is a **25 km buffered convex hull** that swallows Natural Earth's Iceland whole. That is not a trick available to a country with land neighbours: a hull on Greece would carve Natural Earth out from under Albania, North Macedonia, Bulgaria and Anatolia. ## Why no mask geometry can fix it The Greece pilot tried. The mask was extended with the WFD coastal and transitional water bodies so it would reach out over the gulfs — 132 041 → 167 843 km², measured, and the Gulf of Corinth window dropped from 2 429 km² of background land to 40 km². **The map still looked wrong**, because 40 km² spread along 40 km of gulf is a 1 km-wide line straight through it, and because Natural Earth's error in the open gulfs is tens of kilometres while WFD coastal water reaches about two. The country whose data we have is simply not the shape Natural Earth thinks it is. Subtracting one from the other leaves the difference — always, in both directions, at every zoom. No amount of better national geometry converges, because the error is in the *other* operand. ## The fix: carve Natural Earth with Natural Earth Natural Earth also publishes `ne_110m_admin_0_countries` — the same 1:110 m scale as `ne_110m_land`, cut from the same source geometry, one polygon per country with an ISO code. Subtracting *those* is exact, because it is subtracting a shape from itself. Which countries get carved stays driven by `map_country_mask_0`, so the hole still tracks the data we actually have and countries can still be added or removed one at a time — but only the **country code** is read from it, never the geometry. Measured after the change: **0.0 km² inside every imported country** (Iceland 4.8 km² of hairline sliver, sitting under its own cover where nothing draws it), and **0 km² of foreign land carved**, checked across all 170 countries we have not imported. What our own data then does is draw on top, so the coastline you see is the country's own land cover rather than Natural Earth's. Where Natural Earth claimed land our cover says is sea, the base map's water shows through — which is right. ## Three tests, because the old check could not fail The carve was "verified" by checking that no background land overlaps the mask. That is **tautologically true** — `background_land` is Natural Earth *minus* the mask — so it passed throughout, for every country, while the map was visibly wrong. - **R144** — every imported country matches exactly one Natural Earth admin-0 polygon. Natural Earth writes `-99` for a few disputed states; without this a new country is silently never carved. - **R107** — no background land left inside an imported country (the visible bug). - **R145** — no country we have *not* imported loses its Natural Earth land (the invisible half). R145 immediately earned its keep: an attempt to clean up the last hairline slivers with a 1 m grid size on `ST_Difference` cleared them, and silently cost New Zealand 3 813 km², Papua New Guinea 198 and Japan 91 — because a grid size snaps the *inputs* too. Caught, measured, reverted. ## Also fixed on the way The shapefile loader could not read `ne_110m_admin_0_countries` at all: its `.dbf` is NUL-padded rather than space-padded (15 917 cells), and Postgres rejects a NUL inside text outright, so the whole COPY died on a file that is not malformed. `sanitizeDBFText` strips the padding and replaces invalid UTF-8 instead of failing the load.

art-bot self-assigned this

2026-08-20 08:37:50 +00:00

art-bot referenced this issue from a commit

2026-08-20 08:39:32 +00:00

world: carve Natural Earth with Natural Earth, not with our own country masks (#184)

art-bot referenced this issue from a pull request that will close it,

2026-08-20 08:40:02 +00:00

world: carve Natural Earth with Natural Earth, not with our own country masks (#184) #185

art-bot referenced this issue from a commit

2026-08-20 08:43:17 +00:00

config: Greece's mask goes back to the land outline — the carve was never its job

art-bot referenced this issue