Formalized acceptance tests. #115
Labels
No labels
blocked
claude-wip
map-report
map:is
map:se
needs-review
No milestone
No project
No assignees
1 participant
art-bot
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
art/ismap#115
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Currently testing of layers and generalizations is done ad-hoc while working on some layer in a country.
Introduce a new mechanism. In both the common shared_layers config and in per-country layer configs add some new config key with a list of acceptance tests. Each acceptance test should be a sentence describing what we're testing (use references to docs/country-requirements.md if relevant), a SQL query and expected output (keep it reasonable, first value boolean if the test passed or not, then a couple of more columns if necessary (with things like counts, names, zoom levels)). Also a timeout per test - 1, 10, 100, 1000 seconds. Then we can have the 1 (and maybe 10) second tests run on every import of the layer, the others with a special flag to the importer. The importer should also have a flag for running all the long tests.
After this is implemented, every single acceptance test (including those done by reviews) should be done through the importer, not hand-rolled SQL which can sometimes take 20 hours to run (if the acceptance test is longer than 1000 seconds it's probably bad).
Also, I don't even know if these are acceptance tests or regression tests. These should be the tests we use to test that a feature has landed correctly, but we also want to reuse them over and over again to make sure that things don't break in the future.
There probably also need to be global tests that aren't tied to a layer, so that we can check that the MVT is sensible.
Eventually we want automated tests for all the requirements in docs/country-requirements.md, but adding all the tests will be too much for the work on this issue. Add a handful to test that the tests work (a couple of global tests checking that MVT size is sensible, a couple of common layer tests, a couple of country layer tests), chop up the rest of the test adding work into separate issues (probably by country to avoid too many conflicts in the pull requests).
One possible global test - check that all the geometry columns involved in the MVT queries have indexes.
Result is in #119 — #119; the review note is the PR description.
Update: the review comment's "global tests" are in too — a config-level
acceptance_tests:key carries the two whole-database checks (index coverage, the z5 Stockholm tile budget), merged from conf/common.yaml into every config and run on every import and in every -acceptance-only pass.The remaining test-adding work is chopped per the comment's instruction: #121 (Iceland), #122 (Sweden), #123 (Poland), #124 (Finland), #125 (global/MVT).