#177 - #172 (add-on): the CLMS provider — unblock Greece's cover (R142) - art/ismap

art commented

2026-08-20 00:24:43 +00:00

Owner

Follow-on to the #172 pilot PR (#176): the pilot's one blocked item was the CORINE cover
download, because the old EIONET static URLs are dead and the only official route is the
CLMS download API, which needs a Bearer token the importer's plain-GET downloads:
mechanism cannot supply. This PR lands the token-minting provider that unblocks it — and
then actually fetched and imported the data.

What changed

importer/clms.go — a new provider fetcher, the project's first OAuth2 JWT-bearer
client-credentials flow (R142). Reads the CLMS service key from a clms: block in
conf/secrets.yaml, signs an RS256 JWT grant (stdlib crypto — no new dependency),
exchanges it at @@oauth2-token for a Bearer token, POSTs the Greece crop to
@datarequest_post, polls @datarequest_status_get to completion, and downloads the
.gpkg. Wired into fetch/list, appConfig, include.go and the import fetch phase
alongside geotorget/lastkajen/trafiklab. The token is re-minted on demand rather than
held for the life of the fetch.
importer/clms_test.go — seven tests against local httptest servers: the JWT-bearer
grant (iss/sub/aud claims, grant type, expires_in), missing-credential and bad-key
rejection, the Queued → In_progress → Finished_ok poll transition, terminal failure, and
token re-minting.
conf/gr/downloads.yaml — CORINE moves from a blocked plain downloads: entry to a
clms: block, with the full register/configure instructions.
conf/gr/base.yaml — declares secrets_file: conf/secrets.yaml.
conf/gr/acceptance.yaml — R105 now covers gr_clc as well as gr_perifereies.
conf/gr/staging.yaml, conf/gr/02-cover.yaml — the real layer name and column casing
from the delivered file.
Makefile — fetch-gr runs importer fetch conf/gr clms after the boundary fetch, and
that fetch runs under umask 002 for the shared data tree.
conf/secrets.example.yaml — the clms: block, and the EU Login + CLMS token
registration (the private key is shown once and is unrecoverable).
docs/country-requirements.md — new R142: token-minting providers are importer
fetchers, never a key in config or a URL; CLMS is the reference.
docs/importer-cli.md — the clms provider on both the fetch and list lines.

Verification

The live download ran, and the cover is imported. (An earlier revision of this
description said it had not — that was written before the credential was in place and was
never updated. Corrected here.)

data/gr/clc_gr.gpkg — 169 MB, fetched 2026-08-20T00:49:30Z from
copernicus-fme.eea.europa.eu through the provider, not by hand.
gr_clc staging: 49 056 rows, 0 NULL code_18.
map_cover_*_gr built at all 11 zooms; z13 holds 49 844 rows, 0 NULL cover_class,
42 distinct classes — including the Mediterranean ones Finland never sees
(olive_grove, sclerophyllous, vineyard, rice_fields, salt_marsh).
importer acceptance conf/gr — 28/28 pass, including R34 gapless at 0.75%
(cover 131 042 km² vs territory 132 029 km²), R120 no-NULL, and R105 now reporting
tables=2 where it reported tables=1.
go build ./..., go vet ./..., go test ./... green; the seven CLMS tests all pass.

Field fixes found by running it

NUTS vs ISO. The crop key is the NUTS0 code EL. GR is accepted by the API and
then fails the task with "no data for the selected Area Of Interest".
Transient statuses. A poll that defaults to error fails on Queued, which a fresh
task returns before In_progress. Only terminal statuses error now.
Column casing. The GeoPackage ships Code_18; the staging loader lowercases it, so
the query reads code_18 unquoted. Layer name is U2018_CLC2018_V2020_20u1, not the
assumed clc2018_gr.

Review fixes in the last commit

R105 widened to gr_clc (#181); the access token re-mints instead of being held for the
whole fetch (a raised poll_timeout_seconds would have 401-ed mid-poll); dead clmsCreds
struct removed; hand-rolled urlEncode replaced with url.Values.Encode;
docs/importer-cli.md's list line corrected.

Left undone

docs/country-gr.md's CORINE caveat still described this as a blocker — that is fixed in
#183, which also corrects the catalogue's NUTS "GR" and its roads/rail verdicts.

Rebased onto issue-172-greece-pilot after that branch took its own review fixes.

Follow-on to the #172 pilot PR (#176): the pilot's one blocked item was the CORINE cover download, because the old EIONET static URLs are dead and the only official route is the CLMS download API, which needs a Bearer token the importer's plain-GET `downloads:` mechanism cannot supply. This PR lands the token-minting provider that unblocks it — and then actually fetched and imported the data. ## What changed - **`importer/clms.go`** — a new provider fetcher, the project's first OAuth2 JWT-bearer client-credentials flow (**R142**). Reads the CLMS *service key* from a `clms:` block in `conf/secrets.yaml`, signs an RS256 JWT grant (stdlib crypto — no new dependency), exchanges it at `@@oauth2-token` for a Bearer token, POSTs the Greece crop to `@datarequest_post`, polls `@datarequest_status_get` to completion, and downloads the `.gpkg`. Wired into `fetch`/`list`, `appConfig`, `include.go` and the import fetch phase alongside geotorget/lastkajen/trafiklab. The token is **re-minted on demand** rather than held for the life of the fetch. - **`importer/clms_test.go`** — seven tests against local `httptest` servers: the JWT-bearer grant (iss/sub/aud claims, grant type, `expires_in`), missing-credential and bad-key rejection, the `Queued → In_progress → Finished_ok` poll transition, terminal failure, and token re-minting. - **`conf/gr/downloads.yaml`** — CORINE moves from a blocked plain `downloads:` entry to a `clms:` block, with the full register/configure instructions. - **`conf/gr/base.yaml`** — declares `secrets_file: conf/secrets.yaml`. - **`conf/gr/acceptance.yaml`** — R105 now covers `gr_clc` as well as `gr_perifereies`. - **`conf/gr/staging.yaml`, `conf/gr/02-cover.yaml`** — the real layer name and column casing from the delivered file. - **Makefile** — `fetch-gr` runs `importer fetch conf/gr clms` after the boundary fetch, and that fetch runs under `umask 002` for the shared data tree. - **`conf/secrets.example.yaml`** — the `clms:` block, and the EU Login + CLMS token registration (the private key is shown once and is unrecoverable). - **`docs/country-requirements.md`** — new **R142**: token-minting providers are importer fetchers, never a key in config or a URL; CLMS is the reference. - **`docs/importer-cli.md`** — the `clms` provider on both the `fetch` and `list` lines. ## Verification **The live download ran, and the cover is imported.** (An earlier revision of this description said it had not — that was written before the credential was in place and was never updated. Corrected here.) - `data/gr/clc_gr.gpkg` — 169 MB, fetched `2026-08-20T00:49:30Z` from `copernicus-fme.eea.europa.eu` through the provider, not by hand. - `gr_clc` staging: **49 056 rows, 0 NULL `code_18`**. - `map_cover_*_gr` built at all 11 zooms; z13 holds **49 844 rows, 0 NULL `cover_class`, 42 distinct classes** — including the Mediterranean ones Finland never sees (`olive_grove`, `sclerophyllous`, `vineyard`, `rice_fields`, `salt_marsh`). - `importer acceptance conf/gr` — **28/28 pass**, including R34 gapless at **0.75%** (cover 131 042 km² vs territory 132 029 km²), R120 no-NULL, and R105 now reporting `tables=2` where it reported `tables=1`. - `go build ./...`, `go vet ./...`, `go test ./...` green; the seven CLMS tests all pass. ## Field fixes found by running it - **NUTS vs ISO.** The crop key is the NUTS0 code **`EL`**. `GR` is accepted by the API and then fails the task with "no data for the selected Area Of Interest". - **Transient statuses.** A poll that defaults to error fails on `Queued`, which a fresh task returns before `In_progress`. Only terminal statuses error now. - **Column casing.** The GeoPackage ships `Code_18`; the staging loader lowercases it, so the query reads `code_18` unquoted. Layer name is `U2018_CLC2018_V2020_20u1`, not the assumed `clc2018_gr`. ## Review fixes in the last commit R105 widened to `gr_clc` (#181); the access token re-mints instead of being held for the whole fetch (a raised `poll_timeout_seconds` would have 401-ed mid-poll); dead `clmsCreds` struct removed; hand-rolled `urlEncode` replaced with `url.Values.Encode`; `docs/importer-cli.md`'s `list` line corrected. ## Left undone `docs/country-gr.md`'s CORINE caveat still described this as a blocker — that is fixed in **#183**, which also corrects the catalogue's `NUTS "GR"` and its roads/rail verdicts. Rebased onto `issue-172-greece-pilot` after that branch took its own review fixes.

art added 1 commit

2026-08-20 00:24:43 +00:00

importer: CLMS provider — token-minting download for CORINE land cover (R142, #172 ) 980ff9fec6

Greece's cover was the pilot's one blocked item: the old EIONET static CORINE URLs
are dead and the only official route is the CLMS download API, which needs a Bearer
token the plain-GET `downloads:` mechanism cannot supply. This lands the provider
that unblocks it, and states the general rule it is an instance of.

importer/clms.go is the project's first OAuth2 JWT-bearer client-credentials fetch
(R142): the CLMS *service key* (an RSA PKCS#1 private key plus client_id/user_id/
token_uri, issued once by the portal) lives in a `clms:` block in conf/secrets.yaml;
the provider signs an RS256 JWT grant with it (stdlib crypto — no new dependency),
exchanges it at @@oauth2-token for a short-lived Bearer token, POSTs the Greece
crop to @datarequest_post (DatasetID 0407d497…, NUTS GR, GPKG), polls
@datarequest_status_get to completion, and downloads the .gpkg. Only the service key
ever sits in secrets; the Bearer token lives only in memory. Wired into the import
fetch phase and the `fetch` subcommand (`clms`) alongside geotorget/lastkajen/
trafiklab, via appConfig + include.go.

conf/gr: the CORINE entry moves from a blocked `downloads:` to a `clms:` block;
base.yaml declares secrets_file; the Makefile fetch-gr target runs `importer fetch
conf/gr clms` after the boundary curl. secrets.example.yaml documents the
registration and the block. docs/importer-cli.md lists the clms provider.

R142 in docs/country-requirements.md captures the rule: a token-minting source is an
importer provider, never a key in config or a URL; Lantmäteriet, Trafikverket,
Trafiklab and now Copernicus each get one. clms_test.go verifies the JWT-bearer
grant against a local httptest token endpoint plus credential/key rejection.

Verification: go build/vet/test green; make build green; importer config-check
conf/gr OK (2 layers, 17 shared contracts, 28 acceptance tests); `importer fetch
conf/gr clms` reaches the credential check and fails at the right place because the
real service key is not yet in secrets. The live download is not run — it needs the
real CLMS service key in conf/secrets.yaml; `make fetch-gr` runs it once that is in
place. The delivered .gpkg's internal layer/code column (clc2018_gr, code_18) is
still the assumed standard CLC 2018 schema, to verify on first successful download.

art referenced this pull request

2026-08-20 00:24:51 +00:00

Greece pilot: Natural Earth carve + CORINE landcover (prove the add-a-country mechanism) #172

art force-pushed issue-172-clms-fetcher from 980ff9fec6 to 0fa53f7ed0

2026-08-20 00:25:24 +00:00

Compare

art force-pushed issue-172-clms-fetcher from 0fa53f7ed0 to c8cce66978

2026-08-20 00:29:12 +00:00

Compare

art added 1 commit

2026-08-20 00:55:19 +00:00

importer: CLMS field fixes — NUTS code, transient-status poll, cover schema (R142, #172 ) d47319ed4f

The first live run of the CLMS provider surfaced three things the API docs
did not make obvious:

1. The country crop must use the NUTS code, not the ISO code: Greece is "EL"
   (NUTS0), not "GR". "GR" is accepted by @datarequest_post but the task is
   then Rejected with "no data for the selected Area Of Interest" — the
   rejected task 18859030043's Message, surfaced via the new `list clms`
   debug path.

2. The poll treated every status except Finished_ok/In_progress as an error,
   but a fresh task returns "Queued" before "In_progress". waitForDownloadURL
   now keeps polling on any non-terminal status and only fails on
   Finished_fail/Cancelled/Error/Failed/Rejected (isCLMSFailed). The bug
   stopped make fetch-gr at the very first poll. Covered by
   TestCLMSPollQueuedThenReady, which reproduces the exact Queued→Finished_ok
   field failure.

3. The delivered GeoPackage's schema differs from the assumed standard CLC
   2018 one: the layer is U2018_CLC2018_V2020_20u1 (not clc2018_gr) and the
   code column is Code_18 (which the staging loader lowercases to code_18).
   staging.yaml and the cover query now name the real ones, verified against
   the delivered file and the live staging table.

Also: `list clms` gains a debug path — it lists recent download tasks (and,
via CLMS_TASK_ID, a single task's full JSON) so a rejected or failed download
can be diagnosed without re-downloading; the fetch subcommand's provider list
and docs/importer-cli.md pick up the clms provider for `list`.

Verified against the live DB (the Greece pilot, issue #172): the cover import
lands 49 844 polygons at z13 (131 044 km²), and the R34 cross-check passes —
cover land 131 042 km² vs the mask's national territory 132 029 km², 0.75%
inside the 1% budget. R22 (z3 classes are the level-1 rollup tier, Greece now
one of 5 countries), R119 (no empty cover partition), and R120 (no NULL
cover_class) all pass. After import-world the background carve is real: 0 km²
of background land overlaps the Greece mask at z10 (992 km² at z3 is Natural
Earth 110m coarse-zoom sliver). The CLMS download (162 MB GeoPackage) lands
at data/gr/clc_gr.gpkg.