Map experiments
  • Go 90.6%
  • HTML 3.5%
  • Makefile 3.1%
  • Python 1.1%
  • JavaScript 0.9%
  • Other 0.8%
Find a file
art-bot (Claude) da9e7b9409 docs: catalogue Norway, Denmark, Czechia and Slovakia — the go/no-go gate for four countries
The add-country gate (R5/R6/R2) for the four countries in issues #210-#213. Nothing is
imported; these are catalogue-and-gate documents in the docs/country-gr.md shape, with the
sources verified live rather than taken from a research report.

What the four verdicts are:

- **Czechia** is a GO on every shared layer from a single source. ZABAGED polohopis covers
  roads (with class *and* number), rail, stations, the Prague metro, trams in seven cities,
  34 river ferries, airports, water, 35 224 springs, 386 waterfalls, 3.9 M building
  polygons and settlements carrying name, population and settlement type in one layer. It
  is anonymous WFS plus a bulk GeoPackage, and its licence is a machine-readable
  per-dataset CC BY 4.0 assertion, which is exactly what Greece's YPEN records lack. No
  new importer provider needed.
- **Norway** is a GO on every layer with no credential anywhere. Kartverket's N50 was
  verified by actually ordering and unpacking two municipalities, so the feature-type names
  in the doc are from delivered GML rather than a product spec: Bygning is gml:Surface
  (polygons), Stasjon carries navn, and ElvMidtlinje/InnsjoMidtlinje give the throughflow
  centrelines a Strahler network needs. Geonorge's order API is Geotorget's shape without
  the OAuth - a POST with no account returned a ReadyForDownload file - so Norway needs one
  new provider (#219) and no secrets. NVDB API v3 is decommissioned; v4 is pinned.
- **Denmark** is a GO split across two access routes, and the split is the trap: the
  anonymous INSPIRE route keeps its attributes in every theme *except* transport, where
  roadlink is 2 506 759 features with no class, no name and no ref. Classes, road numbers,
  station names and building footprints are behind a token (#220). Denmark is a
  building_points country - its 5 994 941 buildings are MultiPoint. Transit is unresolved
  rather than absent.
- **Slovakia** is a GO whose problem is delivery, not data. Only two ZBGIS themes are WFS;
  everything else is behind an ArcGIS MapServer whose /query endpoint answered 503 on every
  attempt. That is recorded as a re-test (#226), not a verdict - the host resolves and its
  metadata returns 200, so this is load-shedding, not an unreachable source. The pilot is
  unblocked anyway: the mask WFS works, and SAZP serves CORINE 2018 with a code_18 column
  over an anonymous CC BY 4.0 WFS, so Slovakia needs neither the clms: provider nor a
  credential.

Three absences are deliberately left as UNRESOLVED rather than written down (Denmark's
water_points and ferries, Slovakia's ferries): R6 says an absence is a claim about the
national dataset, and the provider class lists those claims would rest on have not been
read. An absent verdict is never tested by anything, which is why it gets the higher bar.

R7 is green for all four, checked against the embedded Go font's cmap: Latin Extended-A is
128/128, so Czech e-caron/r-caron/u-ring, Slovak l-caron/l-acute/r-acute and North Sami
d-stroke/eng/t-stroke all render. Recorded in R7's example; the invariant test is #214.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzoRW8cpUdAiQyKcXp1rjL
2026-10-09 06:58:28 +00:00
.bob/context review loop: retire the work-issue skills; fit the operating texts to ismap 2026-10-08 12:13:24 +00:00
.claude docs: add-country becomes a design document, docs/adding-a-country.md 2026-10-09 06:15:39 +00:00
bench #144: commit the first summary-form tile benchmark record 2026-08-16 22:46:12 +00:00
conf importer: enforce every acceptance test's timeout server-side, not just in the client 2026-08-21 18:29:23 +00:00
docs docs: catalogue Norway, Denmark, Czechia and Slovakia — the go/no-go gate for four countries 2026-10-09 06:58:28 +00:00
html Merge main into #152, and retire the is tile source now that it is empty 2026-08-20 21:43:29 +00:00
importer importer: add validation-query, the ad-hoc half of the acceptance bound 2026-08-21 18:32:16 +00:00
mapsrv glyphs: Greek-range invariant test keeps R7 green for onboarding Greece 2026-08-19 23:45:56 +00:00
scripts glyphs: Greek-range invariant test keeps R7 green for onboarding Greece 2026-08-19 23:45:56 +00:00
stylegen stylegen: strict parsing, a table that is a table, and invariants instead of a frozen copy 2026-08-19 22:20:09 +00:00
.gitignore docs: add-country becomes a design document, docs/adding-a-country.md 2026-10-09 06:15:39 +00:00
AGENTS.md docs: catalogue Norway, Denmark, Czechia and Slovakia — the go/no-go gate for four countries 2026-10-09 06:58:28 +00:00
CLAUDE.md docs: rename CLAUDE.md to AGENTS.md; document the linear-history and free-form workflow 2026-08-04 09:03:17 +00:00
Makefile conf/gr: tests that can fail, and mappings that fail closed 2026-08-20 13:11:24 +00:00
opencode.jsonc review loop: retire the work-issue skills; fit the operating texts to ismap 2026-10-08 12:13:24 +00:00
README.md docs: add-country becomes a design document, docs/adding-a-country.md 2026-10-09 06:15:39 +00:00

ismap

How far can we build a map of European countries from official public data, without relying on OpenStreetMap?

That is the question this project exists to answer. It is a PostGIS-based importer and tile server for official national map data, currently covering Iceland (LMÍ), Sweden (Lantmäteriet + Trafikverket), Poland (GUGiK) and Finland (MML + Väylävirasto + SYKE). All four are first-class — none is a test case for the others — and the list is expected to grow, though there is no ambition to cover all of Europe.

Sweden and Finland share a land border, which is how the central claim — that a road class means the same thing in every country — stopped being an argument and became a test: at Tornio–Haparanda the two national road networks meet with a 0 m gap and the same class on both sides. See docs/country-finland.md.

Every layer comes from a national mapping agency or an equivalent official source, plus Natural Earth for the global background. When data is missing or awkward the answer is to find the official source that has it, not to reach for OSM or an OSM derivative.

The project name ismap is obsolete. It stood for Iceland map, which stopped being what this is. A rename is wanted; no replacement has been chosen yet, so read the repository name as a historical accident rather than as a description. It also survives as the default ismap_ table prefix, which shared layers replace with map_. (conf/is and the is map key are not obsolete — there is is just Iceland's country code, same as se and pl. The zxy_is tile-source function is gone as of issue #152: zxy_map serves every country now.)

The end product is a PostgreSQL/PostGIS database populated with correctly imported and generalized map data, served as Mapbox Vector Tiles (MVT). The database schema is the deliverable; the server is a development convenience for looking at the result.

What each country currently imports reflects where importer development went rather than a considered scope — Poland in particular is thin. Feature parity is the goal, as far as each country's data allows. The three maps used to be three parallel stacks sharing an importer; converging them on one MVT source, one style and one set of shared table definitions is done, and docs/unification.md is the record of it.

Architecture

national map data  ──►  importer  ──►  PostGIS (maps db)  ──►  mapsrv  ──►  browser
   (GeoPackage,                            the deliverable      :8080
    shapefile, GML,
    GTFS)

Two Go binaries and a database:

  • importer — the focus of development. Reads the layer config, fetches source data, creates the schema, imports, generalizes across zoom levels, and installs the stored MVT functions.
  • mapsrv — one process serving the whole map from a single origin: vector tiles from the stored functions, MapLibre glyphs and sprites generated on the fly, the static map pages, and the feedback API that files map-data reports as Forgejo issues. It speaks HTTP/1.1 and HTTP/2; a browser only takes the latter over TLS, so set tls_cert/tls_key in conf/mapsrv.yaml if you want it (it is worth less than it sounds — see docs/performance.md).
  • PostGIS — shared map_<layer>_<z> tables partitioned by country, one partition per country per zoom level, holding generalized geometry. Land cover, water, roads, rail and the world background all live there. A handful of Iceland-only layers still use the older per-country <prefix>_<layer>_<z> naming.

Prerequisites

  • Go (1.25+) — the only build dependency; both binaries are pure Go, no cgo
  • PostgreSQL with PostGIS — see below
  • make, curl, unzip
  • Disk: source data is ~900 MB for Iceland, ~25 GB for Sweden, ~11 GB for Poland (from a ~15 GB download) and ~29 GB for Finland, plus the database itself — Poland's land cover alone is ~11 GB imported

First setup

1. PostgreSQL + PostGIS

On Debian/Ubuntu the distribution packages are enough, and the server starts on install:

sudo apt install postgresql postgis

If your distribution has no usable PostGIS package, run one in Docker instead — this is the only thing Docker is needed for, and it is not part of the everyday development path:

docker run -d --name ismap-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD=postgres -v ismap-pgdata:/var/lib/postgresql/data \
  postgis/postgis:18-3.6

2. Tune it for the import

The import is a bulk workload and the stock configuration is sized for something else. In particular the partitioned generalization cascade (Sweden and Poland land cover) will fail on the default 8 MB temp_buffers with "no empty local buffer available".

sudo cp conf/postgresql-tuning.conf /etc/postgresql/18/main/conf.d/
sudo systemctl restart postgresql

The values assume ~30 GB RAM and 8 cores; see the comments in the file to scale them. Iceland alone imports fine without this, just more slowly.

Sweden's and Poland's land cover also need the importer told how much memory it may use: the cascade's peak is roughly -jobs × -batch-rows polygons held at once, and the defaults (8 × 50000) are too generous for them on a machine this size. Import those two with -jobs 4 -batch-rows 10000 (make import-se / make import-pl do this for you). If a cascade step dies rather than errors — the importer says conn is dead — that is a Postgres backend being OOM-killed, and lowering -batch-rows further is the fix.

3. Create the role, database and extension

make setup-db

This is the one step needing postgres superuser rights, so it shells out to sudo -u postgres psql. It creates the maps role, the maps database it owns, and the PostGIS and SFCGAL extensions inside that database — putting PostGIS in the postgres maintenance database instead is the easy mistake here. It is idempotent, so re-running it is safe and cheap. Against the Docker fallback:

SUPER_PSQL="docker exec -i ismap-postgres psql -U postgres" ./scripts/setup-db.sh

4. Import and serve

make import      # fetches any missing source data, then imports Iceland (~10 min)
                 # plus the shared world background, which takes about a second
make start       # mapsrv in the background on :8080
make smoke-test  # verify a tile comes back
make acceptance  # run the data's own acceptance tests against the imported database

Then open http://localhost:8080. Iceland is the quickest country to bring up and needs no credentials, which is why it is the one in the first-run instructions; make import-se, make import-pl and make import-fi are the others (see below for what they need — Finland needs no credentials either, just patience and disk).

Everyday use

make build            # build both binaries
make test             # run the importer and mapsrv unit tests
make acceptance       # run the acceptance tests against the live database (fast buckets)
make acceptance-all   # ... including the 100s and 1000s ones (the full review pass)
make serve            # mapsrv in the foreground (Ctrl-C to stop)
make start/stop/restart   # mapsrv in the background (run/mapsrv.pid, run/mapsrv.log)
make psql             # psql shell on the maps database
make import           # (re)import Iceland
make import-se        # (re)import Sweden
make import-pl        # (re)import Poland — needs make fetch-bdot10k first, ~15 GB
make import-fi        # (re)import Finland — no credentials, first run pulls ~29 GB
make import-world     # (re)build the shared, country-independent layers (~1 s)
make reset-db         # drop every imported table and tile function
make fresh-start      # reset-db, import Iceland + Sweden + world, restart, smoke-test
make graph            # refresh the graphify code index (optional, needs graphify)

make test covers the Go code; make acceptance covers the data. The acceptance tests are declared in conf/*.yaml next to the layers they check — "Sweden's five biggest cities top the place ladder", "no cover partition is empty at any zoom", "both sides of the Tornio–Haparanda bridge carry the same trunk E4" — and the fast ones also run automatically after every layer import. Anything you would otherwise verify by hand in psql belongs there; docs/importer-cli.md has the contract.

Each country import chains import-world itself, so you rarely need to run it directly. It exists because the world background is Natural Earth land minus the union of every country's outline, which changes whenever a country is imported — and the importer's dependency graph is per config, so it can't see that across configs. Rebuilding the world needs no credentials and no country: given an empty set of country outlines it just builds the whole world with no holes.

To iterate on a single layer without re-running the whole cascade, call the importer directly — it resolves the layer's dependencies from the config's import DAG:

importer/importer import --layer waterways conf/is

To iterate on generalization — min_area_pixels, min_width_pixels, tolerances, rollups, zoom filters — add --regeneralize <lo>-<hi> to rebuild only that inclusive zoom window, generalizing down from whatever is already live above it. It skips fetching, staging and the maxz import, none of which a generalization parameter can change:

importer/importer import --layer cover --regeneralize 5-11 -jobs 6 -batch-rows 10000 conf/se
importer/importer import --layer cover --regeneralize 5 conf/se   # just the bottom step

The difference is large for a big country: Sweden's cover is most of an hour before the cascade even starts, while its coarsest step alone re-runs in under a minute.

The maps

One page, html/index.html, serves all of them, selected by ?map=<key>:

Key Map Config Sources
is Iceland conf/is LMÍ IS 50V + land cover (open download)
se Sweden conf/se Lantmäteriet Topo 10/50, Trafikverket NVDB/NJDB, Trafiklab GTFS
pl Poland conf/pl GUGiK BDOT10k + PRG

Every map also draws the layers that have been unified across countries from a shared map source, declared by conf/world.yaml: the low-zoom world background, land cover, and water (lakes and rivers, watercourses, and waterfalls/rapids/springs). All but the background are built by each country's own config into its own partition of one table. Poland is already served entirely from there and has no source of its own.

Iceland needs no credentials. Sweden requires API credentials for Lantmäteriet Geotorget, Trafikverket Lastkajen and Trafiklab, and the map feedback API needs a Forgejo bot token; all of them live in conf/secrets.yaml, which is gitignored. See conf/secrets.example.yaml for the shape. Poland needs no credentials but does need the make fetch-bdot10k bulk download (~15 GB pulled, ~11 GB kept).

Data sources

None of the data files are committed. The importer fetches what a given run needs (see downloads: in each config), re-downloading only when the server's Content-Length or Last-Modified differs from the last fetch, which is recorded in a .fetchmeta sidecar next to each file. The make fetch-* targets below predate that and remain for fetching individual Iceland datasets by hand.

make fetch-data       # all Iceland base datasets (~200 MB, excludes elevation)
make fetch-elevation  # elevation contours (~350 MB, separate due to size)
make fetch-bdot10k    # Poland land cover + water + roads/rail, 380 per-county downloads
                      # (~15 GB pulled; the county zips are deleted after extraction, so
                      #  widening BDOT10K_CLASSES needs the whole set pulled again — the
                      #  marker file is named after the class set to force exactly that)

Iceland datasets

Only Iceland's sources are tabulated here, because they are individually fetchable by hand from open URLs. Sweden's and Poland's are declared in conf/se and conf/pl (downloads:, plus the geotorget:/lastkajen:/trafiklab: provider blocks) and documented in docs/country-sweden.md and docs/country-poland.md.

Makefile target File Source Last updated Description
fetch-lcv is_lcv_lmi_epsg_4326.gpkg LMÍ / natt.is 2023-12-30 Land cover polygons, CORINE/CLC classification
fetch-roads is_50v_samgongur_epsg_4326.gpkg LMÍ / natt.is 2025-07-18 Roads and transportation network
fetch-water is_50v_vatnafar_epsg_4326.gpkg LMÍ / natt.is 2025-02-15 Rivers, lakes, water bodies (~120 MB)
fetch-coastline is_50v_strandlina_epsg_4326.gpkg LMÍ / natt.is 2024-05-15 Coastline
fetch-admin is_50v_mork_epsg_4326.gpkg LMÍ / natt.is 2025-02-27 Municipal and administrative boundaries
fetch-buildings is_50v_mannvirki_epsg_4326.gpkg LMÍ / natt.is 2024-05-19 Buildings and structures
fetch-names is_50v_ornefni_epsg_4326.gpkg LMÍ / natt.is 2026-04-22 Place names (örnefni)
fetch-surface is_50v_yfirbord_epsg_4326.gpkg LMÍ / natt.is 2024-01-12 Land surface classes per ÍST120
fetch-elevation is_50v_haedargogn_epsg_4326.gpkg LMÍ / natt.is 2024-01-02 Elevation contours and spot heights (~350 MB)
fetch-clc clc.csv EEA EIONET — CORINE Land Cover code table

All LMÍ datasets are downloaded from https://atlas.lmi.is/heikir/downloadData/. LMÍ merged into Náttúrufræðistofnun (natt.is) in July 2024; the atlas.lmi.is URLs still work. All data is CC BY 4.0.

Database

host:     127.0.0.1:5432
database: maps
user:     maps
password: maps

Connection string: postgresql://maps:maps@127.0.0.1/maps

These are development credentials for a local database and are checked in deliberately. Do not expose this port beyond localhost.

Developer notes

New to maps and GIS? Start with docs/geo-primer.md — "So you were sucked into a map project", a one-sitting introduction to projections, tiles, zoom levels, generalization, MVT and the ways real-world map data is messy, annotated with where each idea shows up in this codebase. It assumes you can code and knows nothing about maps.

CLAUDE.md is the short orientation and the index to everything else: it says what the project is, the conventions to work by, and which docs/ file holds which mechanism (the import DAG, the generalization strategies, the schema, the shared-layer machinery, the importer's flags, the server, the style).

The rest is under docs/:

  • architecture.md — the data flow end to end: the import dependency DAG, the per-layer pipeline and atomic swap, staging tables, and a map of every source file.
  • schema.md — tables and columns, the shared country-partitioned layer machinery, MVT zoom-range behaviour, derived columns.
  • generalization.md — every cascade strategy and the config keys that tune it.
  • importer-cli.md — running an import: -layer, -regeneralize, the memory budget, postgres tuning, the fetch policy.
  • mapsrv.md — the server: HTTP/2, caching and the data-version ETag, generated glyphs and sprites, the map-report → Forgejo API.
  • style.md — html/style.json and the invariants that break invisibly.
  • .bob/context/ — the operating texts of the bob-agent review loop that works the backlog: the implementer, the two reviewers and the orchestrator.
  • adding-a-country.md — the repeatable shape for adding a country, from the go/no-go catalogue to a verified import.
  • naming.md — why the project is called ismap and what is obsolete about it, plus the table and function naming conventions.
  • country-iceland.md, country-sweden.md, country-poland.md, country-finland.md, country-gr.md — each country's sources, providers, credentials and class mappings. Read the relevant one before touching a country's layers. Finland's is the fullest: it also carries the complete available→imported→worth-importing catalogue for its providers, and the Sweden–Finland border test.
  • unification.md — how the three per-country stacks became one set of shared partitioned tables, one MVT source and one style, and the design decisions behind it. Worth reading before starting work on a shared layer.
  • performance.md — what tiles actually cost and why, with the measurements behind the generalization settings.

The work backlog is Forgejo issues on art/ismap, not a TODO list in a file.