Improvements of ST_AsMVT

Improve buffer and pixel_width calculation for ST_AsMVT

Bug fix tile_to_bbox(): use pixel_width instead of extent. (#333)
Bugfix st as mvt (#334)

Bugfix and improvements of v5.3

Bugfix

Fix wrong variable name (#328) - thanks @lazaa32

Improvements

1. Include all name tags in slice_language_tags (#315) - thanks @zstadler
2. Implement osmfr poly bbox support (#326) - thanks @nyurik
3. remove slash from name (#330)

Generate tiles by PGquery, layer-stats tool, generate bbox

Main features

1. A script to generate vector tiles from PostGIS' ST_AsMVT using tilelive-pgquery (thanks @nyurik) - Thanks @lazaa32 for implement #317 and @smellman, @zstadler and @nyurik for the original script
2. Remove download-osm make-dc, create a simple text file with bbox value instead (#297) - thanks @nyurik
3. New layer-stats tool to explore data (#293) - Thanks @nyurik for implement based on OpenMapTiles / qa

$ docker-compose run --rm openmaptiles-tools layer-stats frequency openmaptiles.yaml aeroway class
======= Analyzing value occurrence in layer 'aeroway' for field [class] =======
class        z10    z11    z12    z13    z14
---------  -----  -----  -----  -----  -----
aerodrome    151    151    154    154    154
apron         56     56    126    224    308
runway       470    470    515    525    528
taxiway     1075   1075   1156   1219   1245
heliport                     1      2      2
helipad                            11    162

Improvements

• Improve delete_empty_keys() performance, add ps to tools (#295) - thanks @nyurik
• Remove the double set of the header Access-Control-Allow-Origin. (#296) - thanks @francois2metz
• preserve mapping order (#302) - thanks @remi-dupre
• Use print_err in places that mix errors and output on stdout. (#307) - thanks @lseelenbinder
• layers without attributes in yaml mapping (#319) - thanks @frodrigo
• Stop parallel import-sql on error (#322) - thanks @frodrigo
• Use pg_roles view instead of pg_authid table (#323) - thanks @baipi

Fixes

• mbtiles-tools fix broken --show-json param (#304) - thanks @nyurik)
• Fixed render scheme in generate-vectortiles (#305) - thanks @kukulich

Clean ups

• #294 (thanks @nyurik), #299 (thanks @erictheise), #301 (thanks @mattcen)

Bugfix release: download-osm imposm config file

• Ensure trailing '/' at the end of replication_url in imposm config file #286 (#287) -- @frodrigo

Fix download-osm bug with some data src not having timestamps

download-osm make-dc for paris area calls osmconvert --out-statistics, which does not produce timestamp max, possibly due to the missing timestamps in the data. As the result, make-dc would crash. Fixing with #292 (@nyurik)

Fix bug with download-osm make-dc not working

Make sure download-osm make-dc ... runs. (#291) - @nyurik

Examine raw MVT tiles, perf, better .mbtiles support

Examine MVT tile content

debug-mvt and mbtiles-tools can now decode MVT tile content, either as a summary table or in an extended view, with optional language names lists.
The tile can be retrieved from Postgres (using layer definition yaml file), from a url/file/STDIN, or from an .mbtiles file (#273, #274) - @nyurik

$ debug-mvt dump "https://tiles.maps.elastic.co/data/v3/6/17/23.pbf" --summary
Uncompressed tile from https://tiles.maps.elastic.co/data/v3/6/17/23.pbf size=68,765 bytes, gzipped=39,095 bytes, 8 layers
Layer                  Extent    Ver    Features                   GeoType    GeoSize  AVG GeoSize    Fields (percentage only if not all features have it)                                        name:* fields
-------------------  --------  -----  ----------  ------------------------  ---------  -------------  ------------------------------------------------------------------------------------------  ---------------
water                    4096      2          29                   POLYGON     11,427  394.0          class, intermittent
waterway                 4096      2           6                LINESTRING        470  78.3           class
landuse                  4096      2          69                   POLYGON      1,287  18.7           class
park                     4096      2          90  POLYGON(51%), POINT(49%)      3,498  38.9           class, name, name_en, name_de, name_int, rank(49%)                                          21 languages
boundary                 4096      2         149                LINESTRING        924  6.2            admin_level, disputed, maritime
transportation           4096      2         745                LINESTRING      8,656  11.6           class
transportation_name      4096      2         182                LINESTRING      4,136  22.7           name_en, name_de, class, ref(99%), ref_length(99%), network(99%), name(55%), name_int(55%)  1 languages
place                    4096      2          24                     POINT         72  3.0            name, name_en, name_de, name_int, class, rank, capital(8%)                                  60 languages

• mbtiles-tools can now show .mbtiles tile content and metadata:

$ mbtiles-tools tile tiles.mbtiles 0/0/0 [--show-names]
  <output similar to the above>

$ mbtiles-tools meta-all tiles.mbtiles --show-ranges
  attribution <a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
  bounds      -180.0,-85.0511,180.0,85.0511
  center      0.0,0.0,4
  description A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
  filesize    192,512
  format      pbf
  id          openmaptiles
  maxzoom     7
  minzoom     0
  mtime       1590629964868 (2020-05-28T01:39:24.868000)
  name        OpenMapTiles
  pixel_scale 256
  version     3.11.0
  json        (The value is a valid JSON, use --show-json for raw dump)
  
  layer                  minZ    maxZ  fields                                                                                                                             description
  -------------------  ------  ------  ---------------------------------------------------------------------------------------------------------------------------------  -----------------------------------------
  water                     0      14  class, intermittent, brunnel                                                                                                       Water polygons representing oceans and l…
  waterway                  0      14  name, name_en, name_de, name_int, class, brunnel, intermittent, name:* (am,ar,az,be,bg,br,bs…)                                     OpenStreetMap [waterways](https://wiki.o…
  landcover                 0      14  class, subclass                                                                                                                    Landcover is used to describe the physic…
  landuse                   0      14  class                                                                                                                              Landuse is used to describe use of land …
  mountain_peak             0      14  osm_id, name, name_en, name_de, name_int, class, ele, ele_ft, rank, name:* (am,ar,az,be,bg,br,bs…)                                 [Natural peaks](http://wiki.openstreetma…
  park                      0      14  class, name, name_en, name_de, name_int, rank, name:* (am,ar,az,be,bg,br,bs…)                                                      The park layer contains parks from OpenS…
  boundary                  0      14  admin_level, disputed, disputed_name, claimed_by, maritime                                                                         Contains administrative boundaries as li…
  aeroway                   0      14  ref, class                                                                                                                         Aeroway polygons based of OpenStreetMap …
  transportation            0      14  class, subclass, oneway, ramp, brunnel, service, layer, level, indoor, bicycle, foot, horse, mtb_scale, surface                    **transportation** contains roads, railw…
  building                  0      14  osm_id, render_height, render_min_height, colour, hide_3d                                                                          All [OSM Buildings](http://wiki.openstre…
  water_name                0      14  osm_id, name, name_en, name_de, name_int, class, intermittent, name:* (am,ar,az,be,bg,br,bs…)                                      Lake center lines for labelling lake bod…
  transportation_name       0      14  name, name_en, name_de, name_int, ref, ref_length, network, class, subclass, layer, level, indoor, name:* (am,ar,az,be,bg,br,bs…)  This is the layer for labelling the high…
  place                     0      14  osm_id, name, name_en, name_de, name_int, class, rank, capital, iso_a2, name:* (am,ar,az,be,bg,br,bs…)                             The place layer consists out of [countri…
  housenumber               0      14  housenumber                                                                                                                        Everything in OpenStreetMap which contai…
  poi                       0      14  osm_id, name, name_en, name_de, name_int, class, subclass, agg_stop, layer, level, indoor, rank, name:* (am,ar,az,be,bg,br,bs…)    [Points of interests](http://wiki.openst…
  aerodrome_label           0      14  osm_id, name, name_en, name_de, name_int, class, iata, icao, ele, ele_ft, name:* (am,ar,az,be,bg,br,bs…)                           [Aerodrome labels](http://wiki.openstree…
  
    Zoom    Tile count  Found tile ranges
  ------  ------------  -------------------
       0             1  0,0 x 0,0
       1             1  1,1 x 1,1
       2             1  2,2 x 2,2
       3             1  4,5 x 4,5
       4             1  8,10 x 8,10
       5             1  16,20 x 16,20
       6             1  33,40 x 33,40
       7             1  66,81 x 66,81

BREAKING

• generate-metadata script has been removed. Use mbtiles-tools instead (#278) - @nyurik

mbtiles-tools

• New --show-ranges param shows mbtiles tile statistics (#279) - @nyurik
• Comprehensive testing for mbtiles-tools utility (#277) - @nyurik

postgis Docker image

• Cleanup postgis docker, bump osml10n ver (#276) - @nyurik
  • Upgrade osml10n PG extension to the faster v2.5.9 (significant performance improvements merged upstream by @nyurik)
  • parametrize base Postgis image version
  • misc cleanup

Misc

• The latest Imposm 0.10.0 is now packaged as a separate set of Docker images, tagged with -imposm10 suffix (#281) - @nyurik
• Add protoc to go-builder in tools (#272) - @nyurik
• Add less utility to the tools image (#275) - @nyurik
• As part of #273, several minor Python dependencies were bumped to the latest versions.
• download-osm --make-dc now adds openmaptiles-tools env vars in the generated docker-compose file (#283) - @nyurik
• Remove obsolete consts.py -- uses values from layer definition instead #284 - @nyurik
• nano was added to the tools container #285 - @nyurik

Bug fixes

• Unable to delete some metadata values using mbtiles-tools meta-set (#271) - @nyurik
• missing EXPIRETILES_DIR env value (#280) - @nyurik

v5.1 back-ported undeclared EXPIRETILES_DIR bugfix

Bugfix release: #282

v5.0 back-ported undeclared EXPIRETILES_DIR fix & ver

Bugfix release: #282 and incorrect version number was updated to 5.0.1

Postgis 3, smarter download-osm, performance improvements

Summary

• Migrate to Postgis 3.0.1 and osml10n v2.5.8, add GZIP extension
• Improved download-osm tool
• Docs and a script to quickly set up Google Cloud testing environment
• A new profiling tool to evaluate PG function performance profile-pg-func
• SQL performance improvements
• Removed unused Natural Earth tables

download-osm Improvements

Many download-osm tool improvements (#255) - @nyurik

• Added --output <file> param to specify the location and the name of the file to save. Also allows --force to override existing file.
• allow area and URL downloading without explicitly specifying the source:

download-osm toronto      # will search Geofabrik, BBBike, and osm.fr
download-osm https://...  # will download from URL

• Auto-detection logic:

  • if starts with http:// or https:// -- try it as a URL
  • look through Geofabrik catalog, matching the last portion of the name (e.g. michigan would match us/michigan) -- this is the same logic as before
  • look through BBBike catalog (string-insensitive comparison, must be a full match)
  • check if osm.fr gives a 200 rather than 404
  • fail otherwise

• added cacheable BBBike catalog (scraped from a single HTML page), allowing non-exact searches (download-osm bbbike toronto instead of Toronto) and catalog listing (download-osm list bbbike)

• better handling of mirrors that have just the latest planet file

• added a few mirrors

• CHANGE: --make-dc will now set OSM_AREA_NAME to the full ID of the area, e.g. for Geofabrik it will be europe/monaco instead of monaco

New profile-pg-func benchmarking utility

Add profile-pg-func to evaluate PG function speed (#263) - @nyurik

profile-pg-func is a micro-benchmarking tool that runs all given functions thousands of times, in several runs, discarding slowest and fastest results. Use it to analyze relative performance between function implementations.

profile-pg-func [--file <file>]... <func>... [--raw]
                [--calls <calls>] [--runs <runs>] [--trim <trim>]
                [--verbose] [--pghost=<host>] [--pgport=<port>] [--dbname=<db>]
                [--user=<user>] [--password=<password>]

<func>        A SQL function call that should be tested, e.g. 'random()'.
              The func code can use call_idx column -- it will be set to  1..{calls}
              If --raw is used, the <func> must be a complete query.
              These magical values will be substituted in <func> before running:
              * {calls} will be replaced with --calls value
              * {run_idx}  will be replaced with the current RUN number (1..{runs})
              * {random_geopoint} will be replaced with a function call to generate
                a random geo point.

Remove unused Natural Earth tables

• Shrink the size of the import-data and postgis-preloaded images by removing all tables that OMT does not use. (#262) - @nyurik
  • ne_10m_admin_0_boundary_lines_disputed_areas
  • ne_10m_rivers_europe
  • ne_10m_rivers_north_america
  • ne_10m_roads
  • ne_10m_roads_north_america
  • ne_10m_urban_areas
  • ne_50m_admin_0_breakaway_disputed_areas_scale_rank
  • ne_50m_admin_1_states_provinces
• remove unneeded clean-natural-earth.sh from the resulting docker images

Other Improvements

• Documentation and a script to quickly set up a testing env in Google Cloud (#267) - @nyurik
• Upgrade to the official Postgis docker image (PG 9.6 + Postgis 3.0) (#268) - @nyurik
• Cleaner generate-tiles (Mapnik) output if FILTER_MAPNIK_OUTPUT is set (#256) - @zstadler
• Upgrade to osml10n v2.5.8 (#269) - @nyurik
• Create directory for state file, if needed (#249) - @zstadler
• Improve sql tests for import-borders (#258) - @nyurik
• Significant SQL performance improvements of CleanNumeric (#264) and LineLabel (#265) - @nyurik
• Migrate relevant functions from the postgis-vt-util lib into tools (#247) - @nyurik

Bug Fixes

• Download state file from geofabrik whether exact name match or not (#251) - @frodrigo