art/ismap

Fork 0

cache headers and data versioning #4

New issue

Closed

opened 2026-08-01 09:10:47 +00:00 by art · 2 comments

art commented

2026-08-01 09:10:47 +00:00

Owner

mapsrv needs cache headers/etags. I'm guessing the fonts, icons and style and such should be pretty trivial, but for the actual tiles we need to figure out some data versioning scheme.

art-bot referenced this issue

2026-08-01 19:15:32 +00:00

CLAUDE.md is too long #7

art-bot referenced this issue from a commit

2026-08-01 20:14:53 +00:00

docs: compact CLAUDE.md, split knowledge into docs/, move TODOs to Forgejo

art-bot was assigned by art

2026-08-01 20:24:12 +00:00

art-bot added the

claude-wip

label

2026-08-01 23:38:46 +00:00

art-bot commented

2026-08-01 23:53:57 +00:00

Collaborator

night-shift — worked this issue unattended.

Issue #4 — cache headers and data versioning

Branch: night/issue-4-cache-headers (commit 85ba0d4, not pushed, not merged)
State: done

What changed

New mapsrv/cache.go plus small edits to tiles.go, assets.go, main.go,
sprite.go and compress.go. One policy, applied per handler kind:


tiles, glyphs, sprites	`public, max-age=60, stale-while-revalidate=604800` + weak ETag
`html/` (index, style.json)	`no-cache`, revalidated on `http.FileServer`'s `Last-Modified`
`/api/*`	`no-store`

Fonts/sprites (the "trivial" half): the ETag is a hash of the bytes. They are a
pure function of the mapsrv binary, so content is its own version and a rebuild that
changes one sprite invalidates exactly that sprite. The glyph cache now stores an
asset (content type + body + etag) instead of a bare []byte, so the hash is
computed once per range rather than per request.

Tiles — the data versioning scheme. The tile ETag is a fingerprint of the
database, not of the tile, so a 304 is answered from a cached string with no
ST_AsMVT call and no postgres round trip at all.

The importer already produces the signal for free: every layer goes live by an atomic
swap that DROPs the old table and RENAMEs a freshly built one into its place
(layer.swapSQL / swapSQLShared), so a live table's OID changes on every import,
regeneralize or partition swap and at no other moment. dataVersionSQL md5s those
OIDs together with the source text of the zxy_* functions (which are
CREATE OR REPLACEd, so they keep their OID while changing what they return).
%_new tables are excluded so the version does not flap for the hours an import
spends building them. mapsrv re-reads it at most every 30 s, by whichever request
finds it stale; everyone else keeps serving the version they have rather than
queueing behind a catalog read.

Why the catalog and not a metadata table the importer writes: it works on a
database imported before this code existed (including the current one — no reimport
is pending for this change), it needs no importer change at all, and there is no
second mechanism to keep in sync. Two accepted costs, both documented in CLAUDE.md:
it is one version for the whole database rather than per source (a tile mixes every
layer of its source anyway, and an import touches many layers), and it moves when a
staging table is rebuilt, which changes no served byte — one extra conditional GET
per tile, after an import that was going to invalidate them regardless.

Why a validator and a 60 s window rather than a long max-age: none of these URLs
carries a version. /tiles/map/11/1096/553 means "the current tile", and the only
honest way to keep it current without versioning the URL is to let the client ask.
max-age=60 exists to keep a single panning session off the network, not to cache
across imports.

Two smaller things fell out:

Cache headers are set on the success path only (cacheable() after the query
returns), so a 503 from a missing zxy_* function is never stored under the tile's
URL.
Vary: Accept-Encoding is now set unconditionally by the gzip middleware, not
only when it compresses. It describes what the server looked at, and now that these
responses are cacheable a shared cache must not hand a gzipped tile to a client that
never asked for one. (Pre-existing bug, harmless until now.)

Documented in CLAUDE.md ("Caching, and the data version", under Running the server,
plus the two new files in the key-files list) and measured in docs/performance.md.

Verification

make build, cd mapsrv && go test ./..., go vet ./... — all clean.
New mapsrv/cache_test.go: weak/strong/list/* If-None-Match comparison; a tile
304 served by a server with a nil pool (if the handler reached the database the
test would panic — that is the assertion that a 304 costs no query); data-version
TTL expiry; sprite revalidation and that the two scales do not share an ETag;
static no-cache + Last-Modified → 304; /api/* no-store on both success and
error; Vary present with and without Accept-Encoding.
Ran a second mapsrv on :8099 against the live database (the user's :8080 server
was left running its old binary and untouched). Every route checked with curl:
tile 200 → ETag: W/"36959ccc…" + policy; same ETag back → 304, empty body;
stale ETag → 200. Sprites, glyphs, style.json and /api/issues all as designed.
The fingerprint mapsrv logs at startup is byte-identical to the same query run in
psql. A rolled-back BEGIN; CREATE TABLE …; ROLLBACK probe confirmed the
fingerprint moves when a public table appears and does not move for a _new
table, and that the database was unchanged afterwards. Independently, the tables
with the highest OIDs in the live database are exactly map_roads_*_pl, the ones
the last commit on main swapped — which is the mechanism working.
Headless Chrome over CDP (real time, no virtual-time budget), /?map=se loaded
twice: 66 style layers, 7315 rendered features, no console errors either time, and
sprites came from the browser cache on the second load. (Tile and glyph requests
are issued from MapLibre's worker, so they do not appear in the page session's
Network events — those were verified with curl instead.)
No import was run; none is needed for this change.

Left undone / uncertain

Not measured against a real import. The mechanism is verified by the rollback
probe and by the OID evidence from the last commit's swap, but nobody has watched
the version actually move across a live make import-*. Worth a glance at the
next import: the startup log line mapsrv: data version <hash> should differ
before and after, and make restart is needed for that log line (the value itself
refreshes on its own within 30 s).
One version for the whole database is a deliberate simplification, not an
oversight. If per-source granularity is ever wanted, the honest way is for the
importer to record a version per layer and for mapsrv to learn which layers each
source has — which today it cannot, since conf/mapsrv.yaml lists sources only.
max-age=60 is a judgement call with no data behind it. It is the knob to turn
if this ever goes behind a CDN, and it needs no ETag change.
mapsrv/sprite.go has a pre-existing gofmt complaint (comment alignment in the
shields table) that predates this branch; left alone to keep the diff honest.

**night-shift** — worked this issue unattended. # Issue #4 — cache headers and data versioning **Branch:** `night/issue-4-cache-headers` (commit `85ba0d4`, not pushed, not merged) **State:** done ## What changed New `mapsrv/cache.go` plus small edits to `tiles.go`, `assets.go`, `main.go`, `sprite.go` and `compress.go`. One policy, applied per handler kind: | | | |---|---| | tiles, glyphs, sprites | `public, max-age=60, stale-while-revalidate=604800` + weak ETag | | `html/` (index, style.json) | `no-cache`, revalidated on `http.FileServer`'s `Last-Modified` | | `/api/*` | `no-store` | **Fonts/sprites** (the "trivial" half): the ETag is a hash of the bytes. They are a pure function of the mapsrv binary, so content is its own version and a rebuild that changes one sprite invalidates exactly that sprite. The glyph cache now stores an `asset` (content type + body + etag) instead of a bare `[]byte`, so the hash is computed once per range rather than per request. **Tiles — the data versioning scheme.** The tile ETag is a fingerprint of the *database*, not of the tile, so a 304 is answered from a cached string with no `ST_AsMVT` call and no postgres round trip at all. The importer already produces the signal for free: every layer goes live by an atomic swap that DROPs the old table and RENAMEs a freshly built one into its place (`layer.swapSQL` / `swapSQLShared`), so a live table's OID changes on every import, regeneralize or partition swap and at no other moment. `dataVersionSQL` md5s those OIDs together with the *source text* of the `zxy_*` functions (which are `CREATE OR REPLACE`d, so they keep their OID while changing what they return). `%_new` tables are excluded so the version does not flap for the hours an import spends building them. mapsrv re-reads it at most every 30 s, by whichever request finds it stale; everyone else keeps serving the version they have rather than queueing behind a catalog read. **Why the catalog and not a metadata table the importer writes:** it works on a database imported before this code existed (including the current one — no reimport is pending for this change), it needs no importer change at all, and there is no second mechanism to keep in sync. Two accepted costs, both documented in CLAUDE.md: it is one version for the whole database rather than per source (a tile mixes every layer of its source anyway, and an import touches many layers), and it moves when a *staging* table is rebuilt, which changes no served byte — one extra conditional GET per tile, after an import that was going to invalidate them regardless. **Why a validator and a 60 s window rather than a long max-age:** none of these URLs carries a version. `/tiles/map/11/1096/553` means "the current tile", and the only honest way to keep it current without versioning the URL is to let the client ask. `max-age=60` exists to keep a single panning session off the network, not to cache across imports. Two smaller things fell out: * Cache headers are set on the **success path only** (`cacheable()` after the query returns), so a 503 from a missing `zxy_*` function is never stored under the tile's URL. * `Vary: Accept-Encoding` is now set **unconditionally** by the gzip middleware, not only when it compresses. It describes what the server looked at, and now that these responses are cacheable a shared cache must not hand a gzipped tile to a client that never asked for one. (Pre-existing bug, harmless until now.) Documented in CLAUDE.md ("Caching, and the data version", under Running the server, plus the two new files in the key-files list) and measured in `docs/performance.md`. ## Verification * `make build`, `cd mapsrv && go test ./...`, `go vet ./...` — all clean. * New `mapsrv/cache_test.go`: weak/strong/list/`*` If-None-Match comparison; a tile 304 served by a server with a **nil** pool (if the handler reached the database the test would panic — that is the assertion that a 304 costs no query); data-version TTL expiry; sprite revalidation and that the two scales do not share an ETag; static `no-cache` + `Last-Modified` → 304; `/api/*` `no-store` on both success and error; `Vary` present with and without `Accept-Encoding`. * Ran a second mapsrv on `:8099` against the live database (the user's `:8080` server was left running its old binary and untouched). Every route checked with curl: tile 200 → `ETag: W/"36959ccc…"` + policy; same ETag back → `304`, empty body; stale ETag → 200. Sprites, glyphs, style.json and `/api/issues` all as designed. * The fingerprint mapsrv logs at startup is byte-identical to the same query run in psql. A **rolled-back** `BEGIN; CREATE TABLE …; ROLLBACK` probe confirmed the fingerprint moves when a public table appears and does *not* move for a `_new` table, and that the database was unchanged afterwards. Independently, the tables with the highest OIDs in the live database are exactly `map_roads_*_pl`, the ones the last commit on main swapped — which is the mechanism working. * Headless Chrome over CDP (real time, no virtual-time budget), `/?map=se` loaded twice: 66 style layers, 7315 rendered features, no console errors either time, and sprites came from the browser cache on the second load. (Tile and glyph requests are issued from MapLibre's worker, so they do not appear in the page session's Network events — those were verified with curl instead.) * No import was run; none is needed for this change. ## Left undone / uncertain * **Not measured against a real import.** The mechanism is verified by the rollback probe and by the OID evidence from the last commit's swap, but nobody has watched the version actually move across a live `make import-*`. Worth a glance at the next import: the startup log line `mapsrv: data version <hash>` should differ before and after, and `make restart` is needed for that log line (the value itself refreshes on its own within 30 s). * **One version for the whole database** is a deliberate simplification, not an oversight. If per-source granularity is ever wanted, the honest way is for the importer to record a version per layer and for mapsrv to learn which layers each source has — which today it cannot, since `conf/mapsrv.yaml` lists sources only. * **`max-age=60` is a judgement call** with no data behind it. It is the knob to turn if this ever goes behind a CDN, and it needs no ETag change. * `mapsrv/sprite.go` has a pre-existing `gofmt` complaint (comment alignment in the `shields` table) that predates this branch; left alone to keep the diff honest.

art-bot added

needs-review

and removed

claude-wip

labels

2026-08-01 23:53:57 +00:00

art-bot referenced this issue from a pull request that will close it,

2026-08-02 05:30:07 +00:00

mapsrv: cache headers, ETags, and a data version for tiles #32

art-bot commented

2026-08-02 05:30:07 +00:00

Collaborator

night-shift — worked this issue unattended.

Result is in #32 — #32

Branch night/issue-4-cache-headers; the review note is the pull request description.

**night-shift** — worked this issue unattended. Result is in #32 — https://git.blahonga.org/art/ismap/pulls/32 Branch `night/issue-4-cache-headers`; the review note is the pull request description.

art closed this issue

2026-08-02 06:54:44 +00:00