Catch a layer query reading a column the source fetch does not request

art-bot commented

2026-08-11 08:10:48 +00:00

Collaborator

A layer query can read a column the source fetch does not request, and nothing catches it until someone imports on a clean machine. This shipped once already, in #79.

What happened

#79 gave Poland's places a built-up footprint, anchored with:

WHERE p.rodzajobiektu IN ('miasto','wieś')

The PRNG download 1 160 lines earlier asked the WFS for:

PROPERTYNAME=NAZWAGLOWNA,SKALA_WYSWIETLANIA,msGeometry

The server honours that list strictly, so pl_prng_m1 stages with three columns and the layer query dies:

ERROR: column p.rodzajobiektu does not exist

The column history explains how it got past review:

1d9cbf4 (#29) originally fetched NAZWAGLOWNA,SKALA_WYSWIETLANIA,RODZAJOBIEKTU,STATUSNAZWY,msGeometry.
f27750e (#29) narrowed it to three columns — correct at the time, both dropped columns were unused.
#79 reintroduced a dependency on RODZAJOBIEKTU without restoring it to PROPERTYNAME.

So the query was written against a database whose staging table predated the narrowing. It was verified honestly and the numbers were right; the config simply could not reproduce them. Fixed in 5514a61 by putting the column back in PROPERTYNAME.

Note this is not the stale-file problem — fetch.go records the URL in the .fetchmeta sidecar and re-downloads when it changes, and staging tables are rebuilt from the file on every run. Both of those work. The gap is that nothing connects what a query reads to what the fetch provides.

Why it is worth a guard

The failure is invisible to review: the broken predicate and the unchanged URL are a thousand lines apart in the same file, and every local signal says fine. It only surfaces on a machine that fetches clean — which, for a project where the expensive sources are fetched once and kept, is rarely the machine doing the work. The same shape applies to any narrowing of a source's column set, not just WFS PROPERTYNAME.

Options, cheapest first

PREPARE every query in a check mode. An -check-queries flag that runs PREPARE on each staging and layer query against the live DB and discards it. Postgres resolves column references without executing, so this is near-free and catches unknown columns in every layer at once. Limitation: it validates against whatever is staged now, so it would not have caught this on the author's machine — but it catches it in CI or on any clean box, and it catches typos everywhere else.
Static PROPERTYNAME cross-check. For a download whose URL carries PROPERTYNAME=, parse the list and check it against the <table>.<column> references in queries whose staging: includes the table loaded from that path. Narrow, but it is exactly this bug class and needs no database at all.
Column-set assertion on staging. Let a staging entry declare the columns it must produce, and fail the build if the loaded file lacks one. More typing per source, but it turns a silent absence into a named error at the point of loading.

(1) plus (2) seems like the useful pair — (1) for breadth, (2) because it is the only one that would have failed on the author's box.

Immediate mitigation, no code: when narrowing a source's column list, grep the config for the removed column names before committing.

A layer query can read a column the source fetch does not request, and nothing catches it until someone imports on a clean machine. This shipped once already, in #79. ## What happened `#79` gave Poland's `places` a built-up footprint, anchored with: ```sql WHERE p.rodzajobiektu IN ('miasto','wieś') ``` The PRNG download 1 160 lines earlier asked the WFS for: ``` PROPERTYNAME=NAZWAGLOWNA,SKALA_WYSWIETLANIA,msGeometry ``` The server honours that list strictly, so `pl_prng_m1` stages with three columns and the layer query dies: ``` ERROR: column p.rodzajobiektu does not exist ``` The column history explains how it got past review: * `1d9cbf4` (#29) originally fetched `NAZWAGLOWNA,SKALA_WYSWIETLANIA,RODZAJOBIEKTU,STATUSNAZWY,msGeometry`. * `f27750e` (#29) narrowed it to three columns — correct at the time, both dropped columns were unused. * `#79` reintroduced a dependency on `RODZAJOBIEKTU` without restoring it to `PROPERTYNAME`. So the query was written against a database whose staging table predated the narrowing. It was verified honestly and the numbers were right; the config simply could not reproduce them. Fixed in `5514a61` by putting the column back in `PROPERTYNAME`. Note this is *not* the stale-file problem — `fetch.go` records the URL in the `.fetchmeta` sidecar and re-downloads when it changes, and staging tables are rebuilt from the file on every run. Both of those work. The gap is that nothing connects **what a query reads** to **what the fetch provides**. ## Why it is worth a guard The failure is invisible to review: the broken predicate and the unchanged URL are a thousand lines apart in the same file, and every local signal says fine. It only surfaces on a machine that fetches clean — which, for a project where the expensive sources are fetched once and kept, is rarely the machine doing the work. The same shape applies to any narrowing of a source's column set, not just WFS `PROPERTYNAME`. ## Options, cheapest first 1. **`PREPARE` every query in a check mode.** An `-check-queries` flag that runs `PREPARE` on each staging and layer query against the live DB and discards it. Postgres resolves column references without executing, so this is near-free and catches unknown columns in every layer at once. Limitation: it validates against whatever is staged *now*, so it would not have caught this on the author's machine — but it catches it in CI or on any clean box, and it catches typos everywhere else. 2. **Static `PROPERTYNAME` cross-check.** For a download whose URL carries `PROPERTYNAME=`, parse the list and check it against the `<table>.<column>` references in queries whose `staging:` includes the table loaded from that path. Narrow, but it is exactly this bug class and needs no database at all. 3. **Column-set assertion on staging.** Let a staging entry declare the columns it must produce, and fail the build if the loaded file lacks one. More typing per source, but it turns a silent absence into a named error at the point of loading. (1) plus (2) seems like the useful pair — (1) for breadth, (2) because it is the only one that would have failed on the author's box. Immediate mitigation, no code: when narrowing a source's column list, grep the config for the removed column names before committing.