config: Greece pilot — Natural Earth carve (issue #172) #176
No reviewers
Labels
No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
art/ismap!176
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-172-greece-pilot"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #172 — Greece pilot (the Natural Earth carve is done and verified; the CORINE cover is written but BLOCKED on the CLMS token — see the note). This PR does not auto-close #172; the cover half remains open.
#172: Greece pilot — Natural Earth carve + CORINE landcover
Proves the "add a country" mechanism with the smallest footprint. The mask (the
carve) is done and verified. The cover is written but BLOCKED on the CLMS
token — see the blocked item below.
What changed
New
conf/gr/per R17 (country: gr, R16), served through the sharedmapsource (Greece needs no tile source of its own):
base.yaml— destination,table_prefix: gr,country: gr,include:conf/common.yaml.downloads.yaml— two declared downloads (R10): the ΥΠΕΝ mask WFS (plainanonymous GET, works today) and the CORINE entry (declared as the R10 route but
blocked — see below).
staging.yaml—gr_perifereies(GML national outline, EPSG:4326, reprojectedto 3857; coordinates already in lon/lat order, no axis swap) and
gr_clc(CORINE, blocked).
01-country_mask.yaml— Greece's partition of the sharedmap_country_mask_0,built from the ΥΠΕΝ
perifereies(13 Regions) dissolved to the nationaloutline, with its R119 acceptance test.
02-cover.yaml— the identity CLC→enum.landcovermapping (theconf/fi/03-cover.yaml pattern, R116),
part_keyNULL (Iceland-class size,simplifies in one window), the shared speckle-floor
zoom_filters, and the R34gapless cross-check + R120 no-NULL tests (both
requires:-gated onmap_cover_13_gr, so they skip until cover is importable).acceptance.yaml— the config-level R105 staging-nonzero test (gated ongr_perifereiestoday;gr_clcjoins when its download unblocks).Makefile:
import-gr(imports conf/gr, chainsimport-worldper R107),fetch-gr(fetches the mask WFS; CORINE deliberately excluded — token), and
conf/graddedto
ACCEPTANCE_CONFIGS.Verification (against the live DB)
importer config-check conf/gr— OK (2 layers, 28 acceptance tests, 0 sources).map_country_mask_0_grhas 2 041 disjoint outline pieces, area132 029 km² (Greece's territory incl. territorial sea — official ~132k).
TopologyException(regions share borders whoseedges don't match), fixed by
ST_MakeValidbeforeST_Union, withST_CollectionExtract(..., 3)to drop the repair's stray linework (Iceland'sR139 pattern). Result is
ST_IsValid.import-worldapplied the carve (R107): 0 km² ofbackground_landnowoverlaps the Greek mask; a mainland point (22°E 38°N) is in the mask and no
longer in the background; the z5 tile 17/14 decodes to a single
background_landfeature (the hole).
make testboth pass.Acceptance tallies (
make acceptance; read-only, no fetch/import):The one
conf/plfailure is pre-existing and unrelated to Greece: Poland's heavybuildingsR120 scan (map_buildings_14_pl) blew its 10 s bucket (context deadline exceeded) — a timing flake on millions of rows, not touched by the carve.Blocked: the CORINE cover download
CORINE 2018 for Greece requires a CLMS account + Bearer token to download —
the old EIONET direct-zip URLs are dead (verified: 404), and the current official
route is the CLMS download API
(
land.copernicus.eu/api/@datarequest_post, DatasetID0407d497d3c44bcd93ce8fd5bf78596a, NUTS"GR", OutputFormat GDB/GPKG) or the~4 GB full-Europe GeoPackage — both need the token. The importer's
downloads:mechanism (fetch.go) is a plain unauthenticated GET, so it cannot attach the token.
I did not fake a download. The config declares the CORINE entry as the R10
route with a thorough comment documenting exactly what credential/prompt is
needed; the
gr_clcstaging and the02-cover.yamllayer are written (with thestandard CLC
code_18/geomschema assumed, flagged as unverifiable untilthe file lands). Until
data/gr/clc_gr.gpkgis placed, the cover layer'sacceptance tests skip (correctly gated) and
import-grfails at the cover layer.To unblock: obtain a CLMS token, fetch the Greece CLC 2018 GeoPackage to
data/gr/clc_gr.gpkg, verify the layer name + code column against the deliveredfile, then re-import
coverand run the R34/R120 tests.Not done / note
docs/country-gr.mdpilot section (deliverable 4, R129–R136): that documentlives only on the unmerged #171 branch (
issue-171-greece-catalogue, PR #174).Creating it here would conflict with #171. The pilot's per-layer reasoning is
written next to each query in
conf/gr/instead; the doc section should landwhen #171 merges.
import-grchainsimport-world(R107) and is the declared full import; themask layer was verified via
--layer country_mask+import-world(a per-layerreimport is the expected work; no full rebuild or wide
-regeneralize).world_maskunion in conf/world.yaml dissolves them downstream exactly as forthe other countries.
It looks like quite a bit of the NE land is peeking through around the coasts. We either need to use a maritime border or buffer the coast like we do in Iceland (be careful not to buffer the land border though).
Review pass — pushed one commit with three fixes; the pilot itself verified clean.
Verified against the live database first, since the claims in the body are what a
reviewer would otherwise take on trust:
importer acceptance conf/gris 28/28 green,R34 gapless at 0.75% (cover 131 042 km² vs territory 132 029 km²), R120 zero NULLs,
and z13 carries 42 distinct classes including the Mediterranean ones (
olive_grove,sclerophyllous,rice_fields,salt_marsh).Fixed:
comment did not. Measured figures now name the table they came from.
132 041 km² against Greece's 131 957 km² of land —
perifereiesare land administrativeregions. A mask genuinely covering Greek territorial waters would be several times this,
and R34's land-area cross-check would not sit at 0.75%.
umask 002on the fetch recipe. The GML this target wrote landed 0600 in theshared
/stuff/ismap-datatree, so it is unreadable from the other account and are-import fails with a permission denied that reads like a fetch bug — the trap AGENTS.md
documents. The CLMS provider's own output landed 0664 correctly; only the curl path was
wrong. (The existing file still needs
chmodby its owner; onlyartcan do that.)The one thing I did not change: this branch's
downloads:entry for CORINE declareshttps://land.copernicus.eu/api/@datarequest_postas a plain URL that cannot be fetched.#177 removes it entirely, so fixing it here would only conflict with that rewrite — but it
does mean this branch is not independently mergeable as a fetch path. Worth merging the two
together.
docs/country-gr.md's matching claims (territorial sea,NUTS "GR") are corrected in #183.You were right, and my earlier review was wrong in a way worth naming: I measured the mask
at 132 041 km² against Greece's 131 957 km² of land, saw it contradicted the comment claiming
territorial-sea coverage, and "corrected the comment". The measurement was the bug.
What was wrong
perifereiesis an administrative division of the land. It stops at the shoreline, sothe carve stopped there too, and Natural Earth at 1:110 m does not resolve gulfs, straits or
channels as water. Measured on the live database before the fix:
Nothing caught it, because the obvious carve check is tautologically true:
background_landis Natural Earth minus the union of every mask, so "no background landoverlaps the mask" passes for any mask at all — including an empty one. The
add-countryskill was telling the next country to verify the carve exactly that way; #178 now replaces it.
The fix
R108's first branch, not Iceland's. The same ΥΠΕΝ WFD service the water layers come from also
publishes the coastal water bodies (
gr_cwb_50k_xy_greece, 250 bodies, 38 448 km²) andtransitional water bodies (
gr_twb_50k_greece, 47, 1 128 km²) — named, tagged by riverbasin district, EPSG:2100, same GML shape and axis order as
perifereies. The mask unionsland plus both.
Greece must not use Iceland's buffered hull. Four land borders and a shared Aegean: a hull
would carve Natural Earth land out from under Albania, North Macedonia, Bulgaria and Anatolia
and leave holes with nothing drawn in them. The WFD bodies are national by construction and
stop at the maritime boundary on their own. Measured over-carve onto foreign coasts at Natural
Earth's resolution: 12 km² on the Turkish mainland opposite the Dodecanese, 2 km² on
the Albanian coast, 0 elsewhere — against 14 755 km² correctly carved.
Verified, after a real fetch and a real import
importer acceptance conf/gris 30/30, andimporter tiles conf/gr --checkpasses.Two tests so it stays fixed
against a 1.2 floor; a land-only mask scores 1.00. 0.4 s.
bound. 0.03 s.
Deliberately area comparisons rather than the honest geometric one. "No background land
intersects any coastal water body" needs
ST_MakeValidover 250 multipolygons againstcontinent-sized background polygons, times out at 100 s, and fails on exactly the same
regression these two catch in under half a second.
R34 had to move with the mask
It compared land cover against the mask as its second imported product. That was right while
the mask was the land; against a maritime mask it reads 21.9% off. It now references
gr_perifereies— which is Iceland's shape exactly, since Iceland's R34 referencesmorkforthe same reason: its own mask is a buffered hull. Back to 0.75%. No
ST_MakeValidon thatside —
perifereiesis invalid only in the shared-border way, which a union trips over but anarea sum does not; same 132 029 km² either way, 0.32 s against 7.1 s, which is the difference
between fitting the 10 s bucket and being the next #166.
R105 gains
gr_cwb/gr_twb: an empty one of them degrades the mask straight back to theland-only outline with no other symptom at import time.
The rule this is an instance of is in #183 — R108 gains the missing case and a Check: line
that says the tautological check is not it, and a new R143 says a coverage check references
the land, not the mask. #177 is rebased on this.
63d367e444tofa74bf0a2eThe coastal-water mask did not fix it, and you were right to push back. I have reverted it and
fixed the real cause in #185 (issue #184); this PR is rebased onto that.
What the screenshots actually showed
The khaki wedges are not slivers — they are Natural Earth's 1:110 m land cutting straight
across the gulfs, and the Athens fringe is the same mismatch at hairline width. My acceptance
test measured area, and area cannot see either: 40 km² spread along 40 km of gulf is a
1 km-wide line straight through it, which is exactly what you saw. The test passed while the
map was wrong.
It was never Greece's mask
background_landwas Natural Earth land minus the union of every country mask — a precisenational geometry subtracted from a coarse global one. Measured on the live database:
segrfiplisSweden is worse than Greece. Greece is just where someone looked. And the right-hand column is
the same error with the sign flipped, which nobody had looked at at all — Poland is removing
6 992 km² of its neighbours' background.
Iceland scores zero on both only because its mask is a 25 km buffered convex hull that
swallows Natural Earth's Iceland whole. Your instinct that a hull would break the land borders
was right: on Greece it would carve Natural Earth out from under Albania, North Macedonia,
Bulgaria and Anatolia.
Third party with correct land and maritime borders — you already have it
Natural Earth publishes
ne_110m_admin_0_countries: the same 1:110 m scale as the land layer,cut from the same source geometry, one polygon per country with an ISO code. Carving with
that is exact, because it is subtracting a shape from itself. And it is your other idea too —
which countries get carved is still read from
map_country_mask_0, so countries go in and outone at a time, but only the country code is read, never the geometry.
After: 0.0 km² inside every imported country, 0 km² of foreign land carved across all
170 we do not import. Greece goes from 6 429 km² to 0.03. Rendered over CDP: the Gulf of
Corinth is water, no wedge and no line, and the Athens coast at z10 has no fringe.
What this PR keeps
The mask is
perifereiesdissolved again — 2 041 pieces, 132 041 km², the land. The coastaland transitional water bodies, their two fetches, their staging tables and the Greece-specific
carve tests are gone; the generic ones live on
conf/world.yamlwhere the deciding geometryis. The file keeps the story, because "make the mask maritime" is the obvious wrong answer and
the next country will think of it too.
The Peloponnese window now measures 2.28 km² of background land with the land-only mask,
against 3 159 before — which is the proof the carve no longer depends on this geometry at all.
Still here from the earlier review: the 2 041-piece correction,
umask 002on the fetch, andR34 referencing
gr_perifereiesrather than the mask (R143).