art/ismap
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Workstream A: catalogue what the downloads contain vs what we import (#16) #51

Merged
art merged 2 commits from night/issue-16-download-catalogue into main 2026-08-05 00:18:14 +00:00
Conversation 2 Commits 2 Files changed 3 +272 -1
art commented 2026-08-04 09:58:33 +00:00
Owner
Copy link

Closes #16

Issue #16 — Workstream A: what our downloads contain vs what we import

Branch: night/issue-16-download-catalogue · Docs: docs/workstream-a.md
(new), plus an index row in AGENTS.md. Docs-only change; no code, no DB.

What this is

The catalogue the issue asked for: per country, available → imported → worth
importing, with every number measured from data/ on disk (scripts/gpkg.py over
all GeoPackages, the file tree, the fetch markers) and matched against what
conf/*.yaml actually reads. Nothing was re-downloaded and nothing was invented —
row counts are as-measured, and the config comments are the authority for what each
layer reads. Finland's catalogue already lived in docs/country-finland.md; this
closes the loop for Iceland, Sweden and Poland and points back at Finland's.

Headline "worth importing" recommendations per country

  • Iceland — örnefni place names. IS 50V carries no names on any feature
    (water, peaks, waterfalls, springs are all NULL), and is_50v_ornefni is the
    register that fixes all of it at once. This is the dataset issue #29 is gated on.
    Correction found: docs/country-iceland.md says "fetched but not imported",
    but the file is not in this data/ tree — make fetch-names has to run before
    any örnefni work. The fetch target and URL are documented in the Makefile/README.
  • Sweden — the Topo 10 text theme is Sweden's örnefni: it is where Swedish
    water names live (semap.yaml says so verbatim), and water_bodies names are all
    NULL today. Then hojd (contours + spot heights) and byggnadsverk (buildings)
    for Iceland-parity terrain. Cheap wins already on disk: hydropunkt springs, and
    the unread SverigepaketTP columns — speed limit, width, ADT, road authority
    (9 of 121 columns are read; the rest is a config edit away). sankmark
    (peat/marsh) is worth it but needs the nationwide ST_Difference design work.
  • Poland — BDOT10k BU* buildings (would give Poland a buildings layer and a
    stations partition — the only country without one) and KU* land-use
    complexes     (airports/ports split, same difference-design cost as sankmark).
    SKJZ.POLOZENIE is the standing bridge/tunnel item, already on disk. All 16
    downloaded classes are staged and read — nothing is wasted, unlike Sweden's dead
    europavagar/rikslansvagar/funkvagklass fetches.
  • Finland — already catalogued: MTK-rakennus (buildings), MTK-korkeussuhteet
    (contours), DR_PYSAKKI (station importance 3).

Notes for the reviewer

  • The catalogue corrects two numbers in country-finland.md: MTK-muut holds 83
    tables measured (doc says 84) and Digiroad holds 33 (confirmed). I left the
    doc alone — a one-line correction could be folded into a later edit.
  • data/se_osm/stockholm_rail.osm is an unused Overpass OSM export from before Topo
    10 was approved. Nothing references it and OSM is out of scope by project rule; it
    should be deleted from disk (flagged in the doc, not deleted by me — data/ is
    gitignored and outside the commit).
  • The one web verification done: the LMÍ geonetwork record confirms örnefni is a
    three-layer (points/lines/polygons) dataset at 1:50 000. The atlas.lmi.is catalog
    page itself 403s to bots, so the fetchable set is documented from the repo's own
    README/Makefile rather than a live listing.
  • Left undone (deliberately, per the issue being an exploration job): no fetch
    targets were added, no layers were imported, and örnefni/the text theme were not
    fetched (re-downloading is out of scope for a catalogue pass). The natural next
    issues: fetch+import örnefni (#29), fetch the Topo 10 text/hojd/byggnadsverk
    themes, drop the three dead Lastkajen fetches, and a BDOT10k BU* evaluation.
Closes #16 --- # Issue #16 — Workstream A: what our downloads contain vs what we import **Branch:** `night/issue-16-download-catalogue` · **Docs:** `docs/workstream-a.md` (new), plus an index row in AGENTS.md. Docs-only change; no code, no DB. ## What this is The catalogue the issue asked for: per country, **available → imported → worth importing**, with every number measured from `data/` on disk (scripts/gpkg.py over all GeoPackages, the file tree, the fetch markers) and matched against what `conf/*.yaml` actually reads. Nothing was re-downloaded and nothing was invented — row counts are as-measured, and the config comments are the authority for what each layer reads. Finland's catalogue already lived in `docs/country-finland.md`; this closes the loop for Iceland, Sweden and Poland and points back at Finland's. ## Headline "worth importing" recommendations per country - **Iceland — örnefni place names.** IS 50V carries **no names on any feature** (water, peaks, waterfalls, springs are all NULL), and `is_50v_ornefni` is the register that fixes all of it at once. This is the dataset issue #29 is gated on. **Correction found:** `docs/country-iceland.md` says "fetched but not imported", but the file is **not in this data/ tree** — `make fetch-names` has to run before any örnefni work. The fetch target and URL are documented in the Makefile/README. - **Sweden — the Topo 10 `text` theme** is Sweden's örnefni: it is where Swedish water names live (semap.yaml says so verbatim), and `water_bodies` names are all NULL today. Then `hojd` (contours + spot heights) and `byggnadsverk` (buildings) for Iceland-parity terrain. Cheap wins already on disk: `hydropunkt` springs, and the unread `SverigepaketTP` columns — speed limit, width, ADT, road authority (9 of 121 columns are read; the rest is a config edit away). `sankmark` (peat/marsh) is worth it but needs the nationwide `ST_Difference` design work. - **Poland — BDOT10k `BU*` buildings** (would give Poland a `buildings` layer and a `stations` partition — the only country without one) and **`KU*` land-use complexes** (airports/ports split, same difference-design cost as sankmark). `SKJZ.POLOZENIE` is the standing bridge/tunnel item, already on disk. All 16 downloaded classes are staged and read — nothing is wasted, unlike Sweden's dead `europavagar`/`rikslansvagar`/`funkvagklass` fetches. - **Finland —** already catalogued: `MTK-rakennus` (buildings), `MTK-korkeussuhteet` (contours), `DR_PYSAKKI` (station importance 3). ## Notes for the reviewer - The catalogue corrects two numbers in country-finland.md: `MTK-muut` holds **83** tables measured (doc says 84) and Digiroad holds **33** (confirmed). I left the doc alone — a one-line correction could be folded into a later edit. - `data/se_osm/stockholm_rail.osm` is an unused Overpass OSM export from before Topo 10 was approved. Nothing references it and OSM is out of scope by project rule; it should be deleted from disk (flagged in the doc, not deleted by me — data/ is gitignored and outside the commit). - The one web verification done: the LMÍ geonetwork record confirms örnefni is a three-layer (points/lines/polygons) dataset at 1:50 000. The atlas.lmi.is catalog page itself 403s to bots, so the fetchable set is documented from the repo's own README/Makefile rather than a live listing. - Left undone (deliberately, per the issue being an exploration job): no fetch targets were added, no layers were imported, and örnefni/the text theme were not fetched (re-downloading is out of scope for a catalogue pass). The natural next issues: fetch+import örnefni (#29), fetch the Topo 10 `text`/`hojd`/`byggnadsverk` themes, drop the three dead Lastkajen fetches, and a BDOT10k `BU*` evaluation.
The deliverable is docs/workstream-a.md, a per-country available -> imported ->
worth-importing table produced by actually enumerating data/ on disk (gpkg.py over
every GeoPackage, the file tree) and matching it against what conf/*.yaml reads.
No re-downloads, no invented numbers. Finland's catalogue already existed in
docs/country-finland.md; this closes the loop for Iceland, Sweden and Poland.

Headline findings:

- Iceland: every IS 50V table is accounted for. The ornefni place-name dataset is
  the single highest-value item in the whole catalogue: IS 50V carries no names on
  any feature, and ornefni fixes unnamed peaks/waterfalls/springs and all of
  Iceland's unlabelled water at once (it unblocks #29). Correction to
  country-iceland.md: the file is NOT on disk in this data/ tree - "fetched but
  not imported" is stale; make fetch-names must run first. mork_sveitarf_flakar
  (69 municipal polygons) is the only candidate for an admin-boundary layer any
  country has.
- Sweden: 9 of SverigepaketTP's 121 columns are read; speed limit, width, ADT and
  road authority sit unread on disk (config-only change to use them). The Topo 10
  `text` theme is Sweden's ornefni - it is where Swedish water names live (all
  NULL today). hojd (contours/peaks) and byggnadsverk (buildings) would give
  Sweden Iceland-parity terrain. europavagar/rikslansvagar/funkvagklass are
  fetched but no layer reads them - candidates to drop from lastkajen.
- Poland: all 16 downloaded BDOT10k classes are staged and read - nothing wasted.
  The gap is what we do not fetch: BU* (buildings, would give Poland buildings and
  a stations partition - the only country without one) and KU* land-use complexes
  (airports/ports split). SKJZ.POLOZENIE is the standing bridge/tunnel item.
- Finland: doc already carries the catalogue; this pass measured MTK-muut at 83
  tables (doc says 84) and confirmed Digiroad at 33.

Also removed from the data inventory: se_osm/stockholm_rail.osm is an unused OSM
Overpass export from before Topo 10 was approved - OSM is out of scope by project
rule and it should be deleted.

Co-Authored-By: Hermes Agent <noreply@nousresearch.com>
art-bot commented 2026-08-04 18:38:05 +00:00
Collaborator
Copy link

Review — #51 (Workstream A download catalogue)

Good deliverable and the right shape for the issue — the verdict column is what turns
this from an inventory into a work list, and the per-row one-liners stay on the right side
of the project's "reasoning lives next to the query" convention. Docs-only, nothing to
build.

The claimed correction was not applied

docs/workstream-a.md line 13:

The two facts this pass corrected there: MTK-muut holds 83 tables on disk (the doc
says 84) and Digiroad KokoSuomi holds 33 (the doc's count, confirmed).

"corrected there" reads as docs/country-finland.md having been updated, but the diff only
touches AGENTS.md and docs/workstream-a.md. docs/country-finland.md:59 still says:

| `MTK-muut` | 5.4 GB | 84 layers — the catch-all theme | ...

Either fix that line or reword to make clear the correction is recorded here and not there.

Stale against #53

Two places will be wrong the moment #53 merges:

  • The Iceland table lists mannvirki_flakar | 110 | yes — places; #53 renames that layer
    to settlements.
  • Work-list item 1, "Iceland — örnefni place names… Needs make fetch-names first; the
    file is not on disk. Unblocks issue #29" — #53 does exactly this and closes #29.

Not a merge conflict, just a factual staleness; whichever lands second needs the touch-up.

Nit

Line 18 pins "on disk" to /home/art/src/ismap/data/. Absolute and user-specific — data/
relative to the repo root is what the rest of the docs use.

Otherwise

The Sweden section is the valuable part: the three dead fetches
(europavagar / rikslansvagar / funkvagklass, 2.6M rows nothing reads) and the
121-column SverigepaketTP inventory are both actionable today, and identifying the Topo 10
text theme as "Sweden's örnefni" is the cross-cutting finding the issue was asking for.
The Poland "none of the 16 is wasted" conclusion is a useful contrast to have measured
rather than assumed.

## Review — #51 (Workstream A download catalogue) Good deliverable and the right shape for the issue — the `verdict` column is what turns this from an inventory into a work list, and the per-row one-liners stay on the right side of the project's "reasoning lives next to the query" convention. Docs-only, nothing to build. ### The claimed correction was not applied `docs/workstream-a.md` line 13: > The two facts this pass corrected there: `MTK-muut` holds **83** tables on disk (the doc > says 84) and Digiroad `KokoSuomi` holds **33** (the doc's count, confirmed). "corrected there" reads as `docs/country-finland.md` having been updated, but the diff only touches `AGENTS.md` and `docs/workstream-a.md`. `docs/country-finland.md:59` still says: ``` | `MTK-muut` | 5.4 GB | 84 layers — the catch-all theme | ... ``` Either fix that line or reword to make clear the correction is recorded here and not there. ### Stale against #53 Two places will be wrong the moment #53 merges: - The Iceland table lists `mannvirki_flakar | 110 | yes — places`; #53 renames that layer to `settlements`. - Work-list item 1, "Iceland — örnefni place names… Needs `make fetch-names` first; the file is not on disk. Unblocks issue #29" — #53 does exactly this and closes #29. Not a merge conflict, just a factual staleness; whichever lands second needs the touch-up. ### Nit Line 18 pins "on disk" to `/home/art/src/ismap/data/`. Absolute and user-specific — `data/` relative to the repo root is what the rest of the docs use. ### Otherwise The Sweden section is the valuable part: the three dead fetches (`europavagar` / `rikslansvagar` / `funkvagklass`, 2.6M rows nothing reads) and the 121-column `SverigepaketTP` inventory are both actionable today, and identifying the Topo 10 `text` theme as "Sweden's örnefni" is the cross-cutting finding the issue was asking for. The Poland "none of the 16 is wasted" conclusion is a useful contrast to have measured rather than assumed.
art force-pushed night/issue-16-download-catalogue from 8e3e0fbeb5 to a104f61786 2026-08-04 23:39:29 +00:00 Compare
art commented 2026-08-04 23:39:42 +00:00
Author
Owner
Copy link

Review addressed in commit a104f61 (branch rebased onto current main; linear history — the force-with-lease was the pre-rebase SHA being replaced).

  1. MTK-muut correction applied where it belongs: docs/country-finland.md:59 now says "83 layers", verified directly against disk (data/fi/fi_mtk_muut.gpkg has 83 non-metadata tables), so workstream-a.md's "corrected there" claim is now true rather than aspirational.
  2. De-staled against #53, using the branch itself as the source of truth: the Iceland table row now reads settlements (mannvirki_flakar's name on that branch — settlement extents + population, distinct from the shared places name layer), and both örnefni passages — the Iceland-section paragraph and work-list item 1 — now record the post-#53 reality: the register is fetched (2026-08-04) and its 44 506 points feed the shared places layer (issue #29 closed); what stays open is the register's lines/polygons (label-anchor machinery) and the spatial join to name peaks/waterfalls/springs on their own features.
  3. Nit: "On disk" now says data/ relative to the repo root — no absolute /home/art path.

Verified by grep across docs/: no "84 layers" for MTK-muut remains, no /home/art/src/ismap/data path remains, and no "not on disk" örnefni claim remains (the only "not on disk" left is the Poland BDOT10k heading, which is a different, legitimate use).

Review addressed in commit a104f61 (branch rebased onto current main; linear history — the force-with-lease was the pre-rebase SHA being replaced). 1. MTK-muut correction applied where it belongs: docs/country-finland.md:59 now says "83 layers", verified directly against disk (data/fi/fi_mtk_muut.gpkg has 83 non-metadata tables), so workstream-a.md's "corrected there" claim is now true rather than aspirational. 2. De-staled against #53, using the branch itself as the source of truth: the Iceland table row now reads `settlements` (mannvirki_flakar's name on that branch — settlement extents + population, distinct from the shared `places` name layer), and both örnefni passages — the Iceland-section paragraph and work-list item 1 — now record the post-#53 reality: the register is fetched (2026-08-04) and its 44 506 points feed the shared `places` layer (issue #29 closed); what stays open is the register's lines/polygons (label-anchor machinery) and the spatial join to name peaks/waterfalls/springs on their own features. 3. Nit: "On disk" now says `data/` relative to the repo root — no absolute /home/art path. Verified by grep across docs/: no "84 layers" for MTK-muut remains, no /home/art/src/ismap/data path remains, and no "not on disk" örnefni claim remains (the only "not on disk" left is the Poland BDOT10k heading, which is a different, legitimate use).
art merged commit a104f61786 into main 2026-08-05 00:18:14 +00:00
art deleted branch night/issue-16-download-catalogue 2026-08-05 00:18:14 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
claude-wip
night-shift

  
map-report
map data feedback (mapsrv)

  
map:is
map data feedback (mapsrv)

  
map:se
map data feedback (mapsrv)

  
needs-review
night-shift

No labels
claude-wip
map-report
map:is
map:se
needs-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
art/ismap!51
No description provided.