art/ismap
1
0
Fork 0
Code Issues 56 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Greece: peaks — EU-DEM-derived spot heights into the shared peaks layer #206

New issue
Open
opened 2026-08-21 08:01:14 +00:00 by art · 4 comments
art commented 2026-08-21 08:01:14 +00:00
Owner
Copy link

Greece: peaks — EU-DEM-derived spot heights into the shared peaks layer

Scope: Greece's peaks partition of the shared map_peaks_<z> tables, closing the last terrain gap where Iceland has data and no one else. (Siblings: Sweden #101, Finland #108, and the Greece contours issue.)

The source: EU-DEM v1.1 (Copernicus, 25 m — the European Commission's pan-European DEM), fetched for Greece through the existing clms: provider (the same OAuth flow that already fetched the CORINE cover layer). The research gate (done):

  • Greece publishes no national vector spot-height or contour product. The official mapping agency ΓΥΣ holds the national 1:50k DEM/contours but restricted, for-sale. Every contour/elevation layer on data.gov.gr is a municipal WMS raster (Naxos, Syros, Chania, Corfu, Heraklion, Kalamata, Lesvos, +10 more municipalities, varying interval/vintage, no downloads) — not a national dataset (R6 checked against the data.gov.gr catalogue, 2026-08-21).
  • The only official downloadable elevation for Greece is EU-DEM v1.1, the Copernicus pan-European 25 m DEM — the same European Commission family as CORINE. It is not a global DEM (SRTM/ASTER/GLO-30 are rejected under the #84 line); it is the official EU product, cut per-NUTS like the CORINE crop already used.
  • The machinery exists: importer/dem.go (issue #108) is a native Go GeoTIFF COG reader that extracts local-maximum spot heights (radius de-clustering + elevation floor) — the exact shape the shared peaks contract wants (geom + elevation, no name). The importer's first raster stage is reusable as-is; EU-DEM tiles are the same tiled/LZW COG shape family, though the CRS and elevation units must be verified against dem.go's assumptions (Finland: KKJ/EPSG:2393, decimetres — EU-DEM is likely EPSG:3035/3857 and whole metres; if the format differs enough, extend dem.go to a shared generic tile reader rather than duplicating it).

Plan:

  1. Add the EU-DEM Greece crop to conf/gr/downloads.yaml as a clms: entry (dataset_id/download_id for EU-DEM v1.1, NUTS EL — the worker must verify the CLMS API's EU-DEM identifiers and per-NUTS download shape; EU-DEM is distributed per 1°×1° sheet on the portal, so the NUTS-crop pattern may not apply and a per-sheet glob may be needed — verify against the CLMS API docs and the existing clms.go flow before committing).
  2. Reuse dem.go's local-maxima stage for the peaks partition — parameters (radius/floor) tuned for Greek relief (Olympus 2 917 m, Pindus, Taygetos, Crete's Lefka Ori — the floor/thinning cadence should keep the high summits at coarse zoom like Iceland/Sweden/Finland).
  3. Land as conf/gr/NN-peaks.yaml (extends: peaks), with acceptance tests: the shared R119/R120 plus Greece-specific claims (the summit ladder — Olympus is Greece's highest and must appear at coarse zoom; a NULL-elevation check).

Verify against the live database: per-zoom counts, NULLs, partitions at every zoom, a decoded tile, make acceptance. Cite R-numbers from docs/country-requirements.md. Update docs/country-gr.md's layer-by-layer verdict table with the new row.

Greece: peaks — EU-DEM-derived spot heights into the shared peaks layer Scope: Greece's `peaks` partition of the shared `map_peaks_<z>` tables, closing the last terrain gap where Iceland has data and no one else. (Siblings: Sweden #101, Finland #108, and the Greece contours issue.) The source: **EU-DEM v1.1** (Copernicus, 25 m — the European Commission's pan-European DEM), fetched for Greece through the existing `clms:` provider (the same OAuth flow that already fetched the CORINE `cover` layer). The research gate (done): - Greece publishes **no national vector spot-height or contour product**. The official mapping agency ΓΥΣ holds the national 1:50k DEM/contours but restricted, for-sale. Every contour/elevation layer on data.gov.gr is a municipal WMS raster (Naxos, Syros, Chania, Corfu, Heraklion, Kalamata, Lesvos, +10 more municipalities, varying interval/vintage, no downloads) — not a national dataset (R6 checked against the data.gov.gr catalogue, 2026-08-21). - The only official *downloadable* elevation for Greece is EU-DEM v1.1, the Copernicus pan-European 25 m DEM — the same European Commission family as CORINE. It is not a global DEM (SRTM/ASTER/GLO-30 are rejected under the #84 line); it is the official EU product, cut per-NUTS like the CORINE crop already used. - The machinery exists: `importer/dem.go` (issue #108) is a native Go GeoTIFF COG reader that extracts local-maximum spot heights (radius de-clustering + elevation floor) — the exact shape the shared `peaks` contract wants (geom + elevation, no name). The importer's first raster stage is reusable as-is; EU-DEM tiles are the same tiled/LZW COG shape family, though the CRS and elevation units must be verified against dem.go's assumptions (Finland: KKJ/EPSG:2393, decimetres — EU-DEM is likely EPSG:3035/3857 and whole metres; if the format differs enough, extend dem.go to a shared generic tile reader rather than duplicating it). Plan: 1. Add the EU-DEM Greece crop to `conf/gr/downloads.yaml` as a `clms:` entry (dataset_id/download_id for EU-DEM v1.1, NUTS `EL` — the worker must verify the CLMS API's EU-DEM identifiers and per-NUTS download shape; EU-DEM is distributed per 1°×1° sheet on the portal, so the NUTS-crop pattern may not apply and a per-sheet glob may be needed — verify against the CLMS API docs and the existing clms.go flow before committing). 2. Reuse `dem.go`'s local-maxima stage for the `peaks` partition — parameters (radius/floor) tuned for Greek relief (Olympus 2 917 m, Pindus, Taygetos, Crete's Lefka Ori — the floor/thinning cadence should keep the high summits at coarse zoom like Iceland/Sweden/Finland). 3. Land as `conf/gr/NN-peaks.yaml` (`extends: peaks`), with acceptance tests: the shared R119/R120 plus Greece-specific claims (the summit ladder — Olympus is Greece's highest and must appear at coarse zoom; a NULL-elevation check). Verify against the live database: per-zoom counts, NULLs, partitions at every zoom, a decoded tile, `make acceptance`. Cite R-numbers from docs/country-requirements.md. Update `docs/country-gr.md`'s layer-by-layer verdict table with the new row.
art commented 2026-08-21 08:20:50 +00:00
Author
Owner
Copy link

Blocked at the source — EU-DEM v1.1 is retired from every official channel (verified 2026-08-21). No code written.

The issue's premise — "EU-DEM v1.1 fetched for Greece through the existing clms: provider (the same OAuth flow as CORINE)" — does not hold, and it was never a CLMS-API dataset:

  1. CLMS website has retired EU-DEM. The portal lists it under "Products that are no longer disseminated on the CLMS website": "EU-DEM is not maintained anymore and is no longer available on our website." The old product page path is 404.
  2. CLMS API has no EU-DEM dataset. @search?portal_type=DataSet was exhausted (all 241 datasets, 3 batches) plus SearchableText queries (eudem, EU-DEM, digital elevation, elevation model): no EU-DEM/COP-DEM dataset exists, so there is no dataset_id/download_id for a clms: entry and the nuts: crop cannot apply. EU-DEM predates the CLMS 2.0 download API.
  3. EEA datahub marks EU-DEM v1.1 Superseded with no download links; its distribution record points to COP-DEM on the Data Space.
  4. Historical file hosts are dead: published-files.eea.europa.eu and elevationeu.eea.europa.eu are NXDOMAIN; the EEA EU-DEM WCS is 404.
  5. No archive carries the DEM. The Wayback Machine has only the color-shaded preview derivatives (2014); the actual eudem_dem_5deg_*.tif elevation tiles were never archived (CDX verified). A non-official archive would fail R2/R10 anyway.

Recommendation — this needs a source decision, not an implementation:

  • The official successor both CLMS and EEA recommend is COP-DEM (GLO-30/GLO-90) on the Copernicus Data Space Ecosystem. It is a global DEM, which the issue's own research gate rejected ("SRTM/ASTER/GLO-30 are rejected under the #84 line"); whether Copernicus's own GLO-30 is an acceptable successor is the maintainer's call.
  • It needs a new provider: CDSE serves it via S3/STAC with its own OAuth2, not the CLMS clms: flow — a new importer/ fetcher plus a downloads.yaml entry for the ~4–6 GLO-30 tiles over Greece, not a per-NUTS crop.
  • dem.go is not generalized in this run — there is no real tile to verify against, and the repo convention is verify-against-live-data. #207's worker will need: cell size 25 and the decimetre /10 rounding and the 3200-cell (80 km) tile-grid halo/ownership offsets are hardcoded in dem.go; COP-DEM is 30 m, whole metres, EPSG:4326/3857, so each becomes a demPeaksConfig field.

No commits to push, no PR. Handing the source decision back.

**Blocked at the source — EU-DEM v1.1 is retired from every official channel (verified 2026-08-21). No code written.** The issue's premise — "EU-DEM v1.1 fetched for Greece through the existing `clms:` provider (the same OAuth flow as CORINE)" — does not hold, and it was never a CLMS-API dataset: 1. **CLMS website has retired EU-DEM.** The portal lists it under "Products that are no longer disseminated on the CLMS website": *"EU-DEM is not maintained anymore and is no longer available on our website."* The old product page path is 404. 2. **CLMS API has no EU-DEM dataset.** `@search?portal_type=DataSet` was exhausted (all 241 datasets, 3 batches) plus `SearchableText` queries (`eudem`, `EU-DEM`, `digital elevation`, `elevation model`): no EU-DEM/COP-DEM dataset exists, so there is no `dataset_id`/`download_id` for a `clms:` entry and the `nuts:` crop cannot apply. EU-DEM predates the CLMS 2.0 download API. 3. **EEA datahub marks EU-DEM v1.1 Superseded** with no download links; its distribution record points to COP-DEM on the Data Space. 4. **Historical file hosts are dead**: `published-files.eea.europa.eu` and `elevationeu.eea.europa.eu` are NXDOMAIN; the EEA EU-DEM WCS is 404. 5. **No archive carries the DEM.** The Wayback Machine has only the color-shaded preview derivatives (2014); the actual `eudem_dem_5deg_*.tif` elevation tiles were never archived (CDX verified). A non-official archive would fail R2/R10 anyway. **Recommendation — this needs a source decision, not an implementation:** - The official successor both CLMS and EEA recommend is **COP-DEM (GLO-30/GLO-90)** on the Copernicus Data Space Ecosystem. It is a *global* DEM, which the issue's own research gate rejected ("SRTM/ASTER/GLO-30 are rejected under the #84 line"); whether Copernicus's own GLO-30 is an acceptable successor is the maintainer's call. - It needs a **new provider**: CDSE serves it via S3/STAC with its own OAuth2, not the CLMS `clms:` flow — a new `importer/` fetcher plus a `downloads.yaml` entry for the ~4–6 GLO-30 tiles over Greece, not a per-NUTS crop. - **dem.go is not generalized in this run** — there is no real tile to verify against, and the repo convention is verify-against-live-data. #207's worker will need: cell size 25 and the decimetre `/10` rounding and the 3200-cell (80 km) tile-grid halo/ownership offsets are hardcoded in `dem.go`; COP-DEM is 30 m, whole metres, EPSG:4326/3857, so each becomes a `demPeaksConfig` field. No commits to push, no PR. Handing the source decision back.
art commented 2026-08-21 09:10:00 +00:00
Author
Owner
Copy link

Maintainer decision (2026-08-21): approve COP-DEM GLO-30 (the official EU successor to the retired EU-DEM) as Greece's terrain source, fetched through a new CDSE S3/STAC OAuth provider. Re-interprets the #84 line: EU-official (Copernicus), not global-vendor. dem.go generalization to 30 m / whole-metre / EPSG:4326 is in scope. Unblocked.

Maintainer decision (2026-08-21): approve COP-DEM GLO-30 (the official EU successor to the retired EU-DEM) as Greece's terrain source, fetched through a new CDSE S3/STAC OAuth provider. Re-interprets the #84 line: EU-official (Copernicus), not global-vendor. dem.go generalization to 30 m / whole-metre / EPSG:4326 is in scope. Unblocked.
art commented 2026-08-21 10:19:54 +00:00
Author
Owner
Copy link

Blocked at the credential — all the mechanism landed on issue-206-gr-peaks-cop-dem. No PR (there is nothing to import until the credential exists), but the branch is pushed and reviewable.

What is done and verified

  • The CDSE provider (importer/cdse.go): mints a Bearer token at the CDSE realm endpoint (client-credentials or password grant) and signs each S3 object GET with AWS SigV4. Registered in the fetch/list plumbing, the include merge and the neededFiles glob matching. importer list conf/gr cdse shows all 54 orders.
  • importer/dem.go generalized (not copied): COP-DEM's shape (Float32, Deflate+predictor-3, whole metres, 1/3600° EPSG:4326, 3600-cell grid) is dem_peaks: config; the dem25m shape stays the defaults. Verified on the real dem25m tiles: the generalized reader reproduces the documented FI extraction exactly — 26 296 peaks, 885 ≥ 500 m.
  • Greece config: 54 cdse: orders (one per 1° GLO-30 tile the land outline intersects), the gr_dem_peaks staging entry, and conf/gr/10-peaks.yaml (extends: peaks, z13 ≥ 800 / z11 ≥ 1500, three acceptance tests that skip until the partition exists). Docs updated (docs/country-gr.md verdict + catalogue + a Peaks section, importer-cli.md, secrets.example.yaml).
  • Verification: make build, make test, config-check for all five configs, make acceptance → 63 test(s): 59 passed, 0 failed, 3 skipped, 1 not run (the 3 skips are the new GR peaks tests).

Why it is blocked (and precisely what is needed)

CDSE's S3 endpoint is not anonymous (403 without a token, verified live 2026-08-21); the token endpoint rejects empty credentials; the STAC catalogue does not list COP-DEM; and there is no cdse block in conf/secrets.yaml (or anywhere on this machine). To unblock:

cdse:
  cdse_client_id: "..."      # a machine/M2M client (preferred — no 2FA)
  cdse_client_secret: "..."
  # OR cdse_username: "..." / cdse_password: "..."   (EU Login account, 2FA off)
  # OR cdse_s3_access_key: "..." / cdse_s3_secret_key: "..."   (eodata-s3keysmanager)

Then make fetch-gr and importer import conf/gr --layer peaks run the whole pipeline. The one thing the first live fetch must confirm is the SigV4 signing form (token-as-S3-credential vs the S3 keys); the unit tests lock the mechanics.

Two findings worth knowing

  1. The "4–6 GLO-30 tiles" estimate was wrong. A GLO-30 tile is 1°×1°; Greece's land outline intersects 54 tiles, ~1.04 GB (measured against gr_perifereies; the identical product's public AWS copy lists 1036 MB for exactly these 1° cells). The config declares all 54.
  2. The concurrent #108/#205 import died mid-run leaving an orphaned backend holding AccessExclusiveLocks on map_peaks_14 (48+ min); I terminated that dead session so acceptance could run. No leftover locks.
**Blocked at the credential — all the mechanism landed on `issue-206-gr-peaks-cop-dem`.** No PR (there is nothing to import until the credential exists), but the branch is pushed and reviewable. ## What is done and verified - **The CDSE provider** (`importer/cdse.go`): mints a Bearer token at the CDSE realm endpoint (client-credentials or password grant) and signs each S3 object GET with AWS SigV4. Registered in the fetch/list plumbing, the include merge and the `neededFiles` glob matching. `importer list conf/gr cdse` shows all 54 orders. - **`importer/dem.go` generalized** (not copied): COP-DEM's shape (Float32, Deflate+predictor-3, whole metres, 1/3600° EPSG:4326, 3600-cell grid) is `dem_peaks:` config; the dem25m shape stays the defaults. **Verified on the real dem25m tiles: the generalized reader reproduces the documented FI extraction exactly — 26 296 peaks, 885 ≥ 500 m.** - **Greece config**: 54 `cdse:` orders (one per 1° GLO-30 tile the land outline intersects), the `gr_dem_peaks` staging entry, and `conf/gr/10-peaks.yaml` (`extends: peaks`, z13 ≥ 800 / z11 ≥ 1500, three acceptance tests that skip until the partition exists). Docs updated (`docs/country-gr.md` verdict + catalogue + a Peaks section, `importer-cli.md`, `secrets.example.yaml`). - Verification: `make build`, `make test`, config-check for all five configs, `make acceptance` → **63 test(s): 59 passed, 0 failed, 3 skipped, 1 not run** (the 3 skips are the new GR peaks tests). ## Why it is blocked (and precisely what is needed) CDSE's S3 endpoint is **not anonymous** (403 without a token, verified live 2026-08-21); the token endpoint rejects empty credentials; the STAC catalogue does not list COP-DEM; and there is no `cdse` block in `conf/secrets.yaml` (or anywhere on this machine). To unblock: ``` cdse: cdse_client_id: "..." # a machine/M2M client (preferred — no 2FA) cdse_client_secret: "..." # OR cdse_username: "..." / cdse_password: "..." (EU Login account, 2FA off) # OR cdse_s3_access_key: "..." / cdse_s3_secret_key: "..." (eodata-s3keysmanager) ``` Then `make fetch-gr` and `importer import conf/gr --layer peaks` run the whole pipeline. The one thing the first live fetch must confirm is the SigV4 signing form (token-as-S3-credential vs the S3 keys); the unit tests lock the mechanics. ## Two findings worth knowing 1. **The "4–6 GLO-30 tiles" estimate was wrong.** A GLO-30 tile is 1°×1°; Greece's land outline intersects **54 tiles, ~1.04 GB** (measured against `gr_perifereies`; the identical product's public AWS copy lists 1036 MB for exactly these 1° cells). The config declares all 54. 2. The concurrent #108/#205 import died mid-run leaving an orphaned backend holding AccessExclusiveLocks on `map_peaks_14` (48+ min); I terminated that dead session so acceptance could run. No leftover locks.
art commented 2026-08-21 17:48:29 +00:00
Author
Owner
Copy link

Maintainer pause (2026-08-21): implementation paused while the data license of COP-DEM/GLO-30 and the contour/peak generation approach are reviewed. The mechanism branch (issue-206-gr-peaks-cop-dem) stays pushed and unmerged; the only missing piece is the cdse: credential block in conf/secrets.yaml. Not a rejection — a hold for review. #207 waits on this decision.

Maintainer pause (2026-08-21): implementation paused while the data license of COP-DEM/GLO-30 and the contour/peak generation approach are reviewed. The mechanism branch (issue-206-gr-peaks-cop-dem) stays pushed and unmerged; the only missing piece is the cdse: credential block in conf/secrets.yaml. Not a rejection — a hold for review. #207 waits on this decision.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-108-fi-peaks
issue-206-gr-peaks-cop-dem
issue-107-fi-contours
issue-101-se-contours-peaks
issue-152-shared-peaks
issue-151-shared-contours
issue-192-gr-water-bodies
issue-191-gr-transit
issue-193-gr-ferries
issue-102-se-buildings
issue-105-pl-buildings
issue-109-fi-buildings
airports-runways-only
issue-106-pl-airports
issue-110-fi-airports
issue-103-se-airports
issue-45-country-borders
issue-70-strahler-pass-through
issue-79-pl-fi-place-rank
issue-78-is-places-footprint
issue-80-place-casing
issue-82-surface-null
night/issue-47-water-coastline
No results found.
Labels
Clear labels   
blocked
issue is stuck: missing decision, credential, or data (work-issue blocked path)

  
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap#206
No description provided.