Map experiments
  • Go 86%
  • HTML 6.1%
  • Makefile 3.6%
  • Python 2.1%
  • Shell 1.3%
  • Other 0.9%
Find a file
2026-08-06 18:59:17 +00:00
.claude opencode: project config, /work_issue commands, skill cleanup 2026-08-06 13:28:10 +00:00
.opencode/commands opencode: project config, /work_issue commands, skill cleanup 2026-08-06 13:28:10 +00:00
bench Add UNION ALL MVT function variant for benchmarking 2026-06-04 18:08:49 +02:00
conf #64: answer the #66 review — rail DBSCAN + existence filter, metro dedupe, badge size, stale text 2026-08-06 18:59:17 +00:00
docs #64: answer the #66 review — rail DBSCAN + existence filter, metro dedupe, badge size, stale text 2026-08-06 18:59:17 +00:00
html #64: transit design — unique_icon stations, network-specific badge sprites, docs 2026-08-05 12:18:57 +00:00
importer #64 per-country transit stops: Kraków/Łódź tram feeds, Digiroad tram stops, Warsaw metro finding 2026-08-06 18:59:17 +00:00
mapsrv #64: answer the #66 review — rail DBSCAN + existence filter, metro dedupe, badge size, stale text 2026-08-06 18:59:17 +00:00
scripts Kill the night-shift; issue work now runs on tea and two repo skills 2026-08-04 22:47:21 +00:00
.gitignore graphify: ignore the generated index, and a make target to refresh it 2026-08-04 07:43:31 +00:00
AGENTS.md opencode: project config, /work_issue commands, skill cleanup 2026-08-06 13:28:10 +00:00
CLAUDE.md docs: rename CLAUDE.md to AGENTS.md; document the linear-history and free-form workflow 2026-08-04 09:03:17 +00:00
Makefile #64: drop the OIKM_P file-existence check from fetch-bdot10k — the PL.PZGiK.330 prefix is per-voivodeship (only mazowieckie), so the guard matched 42 of 380 counties and re-downloaded the rest on every run. The marker bump to pt-sw-sk-oikm already encodes the class set and is idempotent by itself; the review's point 9 recommended exactly this. 2026-08-06 18:59:17 +00:00
opencode.jsonc opencode: project config, /work_issue commands, skill cleanup 2026-08-06 13:28:10 +00:00
README.md Kill the night-shift; issue work now runs on tea and two repo skills 2026-08-04 22:47:21 +00:00

ismap

How far can we build a map of European countries from official public data, without relying on OpenStreetMap?

That is the question this project exists to answer. It is a PostGIS-based importer and tile server for official national map data, currently covering Iceland (LMÍ), Sweden (Lantmäteriet + Trafikverket), Poland (GUGiK) and Finland (MML + Väylävirasto + SYKE). All four are first-class — none is a test case for the others — and the list is expected to grow, though there is no ambition to cover all of Europe.

Sweden and Finland share a land border, which is how the central claim — that a road class means the same thing in every country — stopped being an argument and became a test: at TornioHaparanda the two national road networks meet with a 0 m gap and the same class on both sides. See docs/country-finland.md.

Every layer comes from a national mapping agency or an equivalent official source, plus Natural Earth for the global background. When data is missing or awkward the answer is to find the official source that has it, not to reach for OSM or an OSM derivative.

The project name ismap is obsolete. It stood for Iceland map, which stopped being what this is. A rename is wanted; no replacement has been chosen yet, so read the repository name as a historical accident rather than as a description. It also survives as the default ismap_ table prefix, which shared layers replace with map_. (conf/ismap.yaml, the is map key and zxy_is are not obsolete — there is is just Iceland's country code, same as se and pl.)

The end product is a PostgreSQL/PostGIS database populated with correctly imported and generalized map data, served as Mapbox Vector Tiles (MVT). The database schema is the deliverable; the server is a development convenience for looking at the result.

What each country currently imports reflects where importer development went rather than a considered scope — Poland in particular is thin. Feature parity is the goal, as far as each country's data allows. The three maps used to be three parallel stacks sharing an importer; converging them on one MVT source, one style and one set of shared table definitions is done, and docs/unification.md is the record of it.

Architecture

national map data  ──►  importer  ──►  PostGIS (maps db)  ──►  mapsrv  ──►  browser
   (GeoPackage,                            the deliverable      :8080
    shapefile, GML,
    GTFS)

Two Go binaries and a database:

  • importer — the focus of development. Reads the layer config, fetches source data, creates the schema, imports, generalizes across zoom levels, and installs the stored MVT functions.
  • mapsrv — one process serving the whole map from a single origin: vector tiles from the stored functions, MapLibre glyphs and sprites generated on the fly, the static map pages, and the feedback API that files map-data reports as Forgejo issues. It speaks HTTP/1.1 and HTTP/2; a browser only takes the latter over TLS, so set tls_cert/tls_key in conf/mapsrv.yaml if you want it (it is worth less than it sounds — see docs/performance.md).
  • PostGIS — shared map_<layer>_<z> tables partitioned by country, one partition per country per zoom level, holding generalized geometry. Land cover, water, roads, rail and the world background all live there. A handful of Iceland-only layers still use the older per-country <prefix>_<layer>_<z> naming.

Prerequisites

  • Go (1.25+) — the only build dependency; both binaries are pure Go, no cgo
  • PostgreSQL with PostGIS — see below
  • make, curl, unzip
  • Disk: source data is ~900 MB for Iceland, ~25 GB for Sweden, ~11 GB for Poland (from a ~15 GB download) and ~29 GB for Finland, plus the database itself — Poland's land cover alone is ~11 GB imported

First setup

1. PostgreSQL + PostGIS

On Debian/Ubuntu the distribution packages are enough, and the server starts on install:

sudo apt install postgresql postgis

If your distribution has no usable PostGIS package, run one in Docker instead — this is the only thing Docker is needed for, and it is not part of the everyday development path:

docker run -d --name ismap-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD=postgres -v ismap-pgdata:/var/lib/postgresql/data \
  postgis/postgis:18-3.6

2. Tune it for the import

The import is a bulk workload and the stock configuration is sized for something else. In particular the partitioned generalization cascade (Sweden and Poland land cover) will fail on the default 8 MB temp_buffers with "no empty local buffer available".

sudo cp conf/postgresql-tuning.conf /etc/postgresql/18/main/conf.d/
sudo systemctl restart postgresql

The values assume ~30 GB RAM and 8 cores; see the comments in the file to scale them. Iceland alone imports fine without this, just more slowly.

Sweden's and Poland's land cover also need the importer told how much memory it may use: the cascade's peak is roughly -jobs × -batch-rows polygons held at once, and the defaults (8 × 50000) are too generous for them on a machine this size. Import those two with -jobs 4 -batch-rows 10000 (make import-se / make import-pl do this for you). If a cascade step dies rather than errors — the importer says conn is dead — that is a Postgres backend being OOM-killed, and lowering -batch-rows further is the fix.

3. Create the role, database and extension

make setup-db

This is the one step needing postgres superuser rights, so it shells out to sudo -u postgres psql. It creates the maps role, the maps database it owns, and the PostGIS extension inside that database — putting PostGIS in the postgres maintenance database instead is the easy mistake here. It is idempotent, so re-running it is safe and cheap. Against the Docker fallback:

SUPER_PSQL="docker exec -i ismap-postgres psql -U postgres" ./scripts/setup-db.sh

4. Import and serve

make import      # fetches any missing source data, then imports Iceland (~10 min)
                 # plus the shared world background, which takes about a second
make start       # mapsrv in the background on :8080
make smoke-test  # verify a tile comes back

Then open http://localhost:8080. Iceland is the quickest country to bring up and needs no credentials, which is why it is the one in the first-run instructions; make import-se, make import-pl and make import-fi are the others (see below for what they need — Finland needs no credentials either, just patience and disk).

Everyday use

make build            # build both binaries
make test             # run the mapsrv tests
make serve            # mapsrv in the foreground (Ctrl-C to stop)
make start/stop/restart   # mapsrv in the background (run/mapsrv.pid, run/mapsrv.log)
make psql             # psql shell on the maps database
make import           # (re)import Iceland
make import-se        # (re)import Sweden
make import-pl        # (re)import Poland — needs make fetch-bdot10k first, ~15 GB
make import-fi        # (re)import Finland — no credentials, first run pulls ~29 GB
make import-world     # (re)build the shared, country-independent layers (~1 s)
make reset-db         # drop every imported table and tile function
make fresh-start      # reset-db, import Iceland + Sweden + world, restart, smoke-test
make graph            # refresh the graphify code index (optional, needs graphify)

Each country import chains import-world itself, so you rarely need to run it directly. It exists because the world background is Natural Earth land minus the union of every country's outline, which changes whenever a country is imported — and the importer's dependency graph is per config, so it can't see that across configs. Rebuilding the world needs no credentials and no country: given an empty set of country outlines it just builds the whole world with no holes.

To iterate on a single layer without re-running the whole cascade, call the importer directly — it resolves the layer's dependencies from the config's import DAG:

importer/importer -layer waterways conf/ismap.yaml

To iterate on generalizationmin_area_pixels, min_width_pixels, tolerances, rollups, zoom filters — add -regeneralize <lo>-<hi> to rebuild only that inclusive zoom window, generalizing down from whatever is already live above it. It skips fetching, staging and the maxz import, none of which a generalization parameter can change:

importer/importer -layer cover -regeneralize 5-11 -jobs 6 -batch-rows 10000 conf/semap.yaml
importer/importer -layer cover -regeneralize 5 conf/semap.yaml   # just the bottom step

The difference is large for a big country: Sweden's cover is most of an hour before the cascade even starts, while its coarsest step alone re-runs in under a minute.

The maps

One page, html/index.html, serves all of them, selected by ?map=<key>:

Key Map Config Sources
is Iceland conf/ismap.yaml LMÍ IS 50V + land cover (open download)
se Sweden conf/semap.yaml Lantmäteriet Topo 10/50, Trafikverket NVDB/NJDB, Trafiklab GTFS
pl Poland conf/plmap.yaml GUGiK BDOT10k + PRG

Every map also draws the layers that have been unified across countries from a shared map source, declared by conf/world.yaml: the low-zoom world background, land cover, and water (lakes and rivers, watercourses, and waterfalls/rapids/springs). All but the background are built by each country's own config into its own partition of one table. Poland is already served entirely from there and has no source of its own.

Iceland needs no credentials. Sweden requires API credentials for Lantmäteriet Geotorget, Trafikverket Lastkajen and Trafiklab, and the map feedback API needs a Forgejo bot token; all of them live in conf/secrets.yaml, which is gitignored. See conf/secrets.example.yaml for the shape. Poland needs no credentials but does need the make fetch-bdot10k bulk download (~15 GB pulled, ~11 GB kept).

Data sources

None of the data files are committed. The importer fetches what a given run needs (see downloads: in each config), re-downloading only when the server's Content-Length or Last-Modified differs from the last fetch, which is recorded in a .fetchmeta sidecar next to each file. The make fetch-* targets below predate that and remain for fetching individual Iceland datasets by hand.

make fetch-data       # all Iceland base datasets (~200 MB, excludes elevation)
make fetch-elevation  # elevation contours (~350 MB, separate due to size)
make fetch-bdot10k    # Poland land cover + water + roads/rail, 380 per-county downloads
                      # (~15 GB pulled; the county zips are deleted after extraction, so
                      #  widening BDOT10K_CLASSES needs the whole set pulled again — the
                      #  marker file is named after the class set to force exactly that)

Iceland datasets

Only Iceland's sources are tabulated here, because they are individually fetchable by hand from open URLs. Sweden's and Poland's are declared in conf/semap.yaml and conf/plmap.yaml (downloads:, plus the geotorget:/lastkajen:/trafiklab: provider blocks) and documented in docs/country-sweden.md and docs/country-poland.md.

Makefile target File Source Last updated Description
fetch-lcv is_lcv_lmi_epsg_4326.gpkg LMÍ / natt.is 2023-12-30 Land cover polygons, CORINE/CLC classification
fetch-roads is_50v_samgongur_epsg_4326.gpkg LMÍ / natt.is 2025-07-18 Roads and transportation network
fetch-water is_50v_vatnafar_epsg_4326.gpkg LMÍ / natt.is 2025-02-15 Rivers, lakes, water bodies (~120 MB)
fetch-coastline is_50v_strandlina_epsg_4326.gpkg LMÍ / natt.is 2024-05-15 Coastline
fetch-admin is_50v_mork_epsg_4326.gpkg LMÍ / natt.is 2025-02-27 Municipal and administrative boundaries
fetch-buildings is_50v_mannvirki_epsg_4326.gpkg LMÍ / natt.is 2024-05-19 Buildings and structures
fetch-names is_50v_ornefni_epsg_4326.gpkg LMÍ / natt.is 2026-04-22 Place names (örnefni)
fetch-surface is_50v_yfirbord_epsg_4326.gpkg LMÍ / natt.is 2024-01-12 Land surface classes per ÍST120
fetch-elevation is_50v_haedargogn_epsg_4326.gpkg LMÍ / natt.is 2024-01-02 Elevation contours and spot heights (~350 MB)
fetch-clc clc.csv EEA EIONET CORINE Land Cover code table

All LMÍ datasets are downloaded from https://atlas.lmi.is/heikir/downloadData/. LMÍ merged into Náttúrufræðistofnun (natt.is) in July 2024; the atlas.lmi.is URLs still work. All data is CC BY 4.0.

Database

host:     127.0.0.1:5432
database: maps
user:     maps
password: maps

Connection string: postgresql://maps:maps@127.0.0.1/maps

These are development credentials for a local database and are checked in deliberately. Do not expose this port beyond localhost.

Developer notes

New to maps and GIS? Start with docs/geo-primer.md — "So you were sucked into a map project", a one-sitting introduction to projections, tiles, zoom levels, generalization, MVT and the ways real-world map data is messy, annotated with where each idea shows up in this codebase. It assumes you can code and knows nothing about maps.

CLAUDE.md is the short orientation and the index to everything else: it says what the project is, the conventions to work by, and which docs/ file holds which mechanism (the import DAG, the generalization strategies, the schema, the shared-layer machinery, the importer's flags, the server, the style).

The rest is under docs/:

  • architecture.md — the data flow end to end: the import dependency DAG, the per-layer pipeline and atomic swap, staging tables, and a map of every source file.
  • schema.md — tables and columns, the shared country-partitioned layer machinery, MVT zoom-range behaviour, derived columns.
  • generalization.md — every cascade strategy and the config keys that tune it.
  • importer-cli.md — running an import: -layer, -regeneralize, the memory budget, postgres tuning, the fetch policy.
  • mapsrv.md — the server: HTTP/2, caching and the data-version ETag, generated glyphs and sprites, the map-report → Forgejo API.
  • style.mdhtml/style.json and the invariants that break invisibly.
  • .claude/skills/work-issue (one issue end to end: branch, push, PR) and work-issues (triage the backlog and dispatch one subagent per issue), both driving the tea CLI.
  • naming.md — why the project is called ismap and what is obsolete about it, plus the table and function naming conventions.
  • country-iceland.md, country-sweden.md, country-poland.md, country-finland.md — each country's sources, providers, credentials and class mappings. Read the relevant one before touching a country's layers. Finland's is the fullest: it also carries the complete available→imported→worth-importing catalogue for its providers, and the SwedenFinland border test.
  • unification.md — how the three per-country stacks became one set of shared partitioned tables, one MVT source and one style, and the design decisions behind it. Worth reading before starting work on a shared layer.
  • performance.md — what tiles actually cost and why, with the measurements behind the generalization settings.

The work backlog is Forgejo issues on art/ismap, not a TODO list in a file.