Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: option to translate labels for static maps #999

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

feat: option to translate labels for static maps #999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
57931c8
feat: option to translate labels for static maps
Caerbannog Sep 28, 2023
5b24379
chore: fix linting
Caerbannog Oct 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README_light.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ docker build -t tileserver-gl-light .
[Download from OpenMapTiles.com](https://openmaptiles.com/downloads/planet/) or [create](https://github.com/openmaptiles/openmaptiles) your vector tile, and run following in directory contains your *.mbtiles.

```
docker run --rm -it -v $(pwd):/data -p 8000:80 tileserver-gl-light
docker run --rm -it -v $(pwd):/data -p 8080:8080 tileserver-gl-light
```

## Documentation
You can read full documentation of this project at https://tileserver.readthedocs.io/.
You can read full documentation of this project at https://tileserver.readthedocs.io/.
8 changes: 8 additions & 0 deletions docs/config.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Example:
"serveAllStyles": false,
"serveStaticMaps": true,
"allowRemoteMarkerIcons": true,
"languages": ["en", "fr", "it"],
"tileMargin": 0
},
"styles": {
Expand Down Expand Up @@ -150,6 +151,13 @@ Allows the rendering of marker icons fetched via http(s) hyperlinks.
For security reasons only allow this if you can control the origins from where the markers are fetched!
Default is to disallow fetching of icons from remote sources.

``languages``
--------------

Allows translating labels when rendering static maps. This is a list of allowed languages.
Note that your vector tile source needs to contain the translated labels (e.g. ``name:en``, ``name:fr``...).
Not used by default.

``styles``
==========

Expand Down
13 changes: 7 additions & 6 deletions docs/endpoints.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ WMTS Capabilities

Static images
=============
* Several endpoints:
* There are three endpoints depending on how you want to define the map location:

* ``/styles/{id}/static/{lon},{lat},{zoom}[@{bearing}[,{pitch}]]/{width}x{height}[@2x].{format}`` (center-based)
* ``/styles/{id}/static/{minx},{miny},{maxx},{maxy}/{width}x{height}[@2x].{format}`` (area-based)
* ``/styles/{id}/static/auto/{width}x{height}[@2x].{format}`` (autofit path -- see below)
* Center and zoom level: ``/styles/{id}/static/{lon},{lat},{zoom}[@{bearing}[,{pitch}]]/{width}x{height}[@2x].{format}``
* Bounds, e.g. latitude/longitude of corners: ``/styles/{id}/static/{minx},{miny},{maxx},{maxy}/{width}x{height}[@2x].{format}``
* Autofit to a path: ``/styles/{id}/static/auto/{width}x{height}[@2x].{format}``

* All the static image endpoints additionally support following query parameters:
* All these endpoints accept these query parameters:

* ``path`` - ``((fill|stroke|width)\:[^\|]+\|)*((enc:.+)|((-?\d+\.?\d*,-?\d+\.?\d*\|)+(-?\d+\.?\d*,-?\d+\.?\d*)))``

Expand All @@ -50,7 +50,7 @@ Static images

* e.g. ``path=stroke:yellow|width:2|fill:green|5.9,45.8|5.9,47.8|10.5,47.8|10.5,45.8|5.9,45.8`` or ``path=stroke:blue|width:1|fill:yellow|enc:_p~iF~ps|U_ulLnnqC_mqNvxq`@``

* can be provided multiple times
* can be provided multiple times

* ``latlng`` - indicates coordinates are in ``lat,lng`` order rather than the usual ``lng,lat``
* ``fill`` - color to use as the fill (e.g. ``red``, ``rgba(255,255,255,0.5)``, ``#0000ff``)
Expand Down Expand Up @@ -84,6 +84,7 @@ Static images
* value of ``0.1`` means "add 10% size to each side to make sure the area of interest is nicely visible"

* ``maxzoom`` - Maximum zoom level (only for auto endpoint where zoom level is calculated and not provided)
* ``language`` - Language code to translate all labels from the selected style. If no language is set or if the language is not found in ``options.languages``, no translation happens.

* You can also use (experimental) ``/styles/{id}/static/raw/...`` endpoints with raw spherical mercator coordinates (EPSG:3857) instead of WGS84.

Expand Down
3 changes: 3 additions & 0 deletions public/templates/index.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@
{{#if serving_rendered}}
| <a href="{{public_url}}styles/{{@key}}/wmts.xml{{&../key_query}}">WMTS</a>
{{/if}}
{{#if serving_rendered}}
| <a href="{{public_url}}styles/{{@key}}/static/{{static_center}}/[email protected]{{&../key_query}}">Static</a>
{{/if}}
{{#if xyz_link}}
| <a href="#" onclick="return toggle_xyz('xyz_style_{{@key}}');">XYZ</a>
<input id="xyz_style_{{@key}}" type="text" value="{{&xyz_link}}" style="display:none;" />
Expand Down
85 changes: 73 additions & 12 deletions src/serve_rendered.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import polyline from '@mapbox/polyline';
import proj4 from 'proj4';
import request from 'request';
import { getFontsPbf, getTileUrls, fixTileJSONCenter } from './utils.js';
import translateLayers from './translate_layers.js';

const FLOAT_PATTERN = '[+-]?(?:\\d+|\\d+.?\\d+)';
const PATH_PATTERN =
Expand Down Expand Up @@ -202,6 +203,19 @@ const extractPathsFromQuery = (query, transformer) => {
return paths;
};

/**
* Parses language provided via query.
* @param {object} query Request query parameters.
* @param {object} options Configuration options.
*/
const extractLanguageFromQuery = (query, options) => {
const languages = options.languages || [];
if ('language' in query && languages.includes(query.language)) {
return query.language;
}
return null;
};

/**
* Parses marker options provided via query and sets corresponding attributes
* on marker object.
Expand Down Expand Up @@ -330,7 +344,7 @@ const precisePx = (ll, zoom) => {
};

/**
* Draws a marker in cavans context.
* Draws a marker in canvas context.
* @param {object} ctx Canvas context object.
* @param {object} marker Marker object parsed by extractMarkersFromQuery.
* @param {number} z Map zoom level.
Expand Down Expand Up @@ -646,6 +660,7 @@ export const serve_rendered = {
next,
opt_overlay,
opt_mode = 'tile',
language = null,
) => {
if (
Math.abs(lon) > 180 ||
Expand Down Expand Up @@ -677,7 +692,7 @@ export const serve_rendered = {
if (opt_mode === 'tile' && tileMargin === 0) {
pool = item.map.renderers[scale];
} else {
pool = item.map.renderers_static[scale];
pool = item.map.renderers_static[`${scale}${language}`];
}
pool.acquire((err, renderer) => {
const mlglZ = Math.max(0, z - 1);
Expand Down Expand Up @@ -712,6 +727,7 @@ export const serve_rendered = {

// Fix semi-transparent outlines on raw, premultiplied input
// https://github.com/maptiler/tileserver-gl/issues/350#issuecomment-477857040
// FIXME: unnecessay now? https://github.com/lovell/sharp/issues/1599#issuecomment-837004081
for (let i = 0; i < data.length; i += 4) {
const alpha = data[i + 3];
const norm = alpha / 255;
Expand Down Expand Up @@ -901,6 +917,7 @@ export const serve_rendered = {
}

const paths = extractPathsFromQuery(req.query, transformer);
const language = extractLanguageFromQuery(req.query, options);
const markers = extractMarkersFromQuery(
req.query,
options,
Expand Down Expand Up @@ -935,6 +952,7 @@ export const serve_rendered = {
next,
overlay,
'static',
language,
);
} catch (e) {
next(e);
Expand Down Expand Up @@ -983,6 +1001,7 @@ export const serve_rendered = {
const pitch = 0;

const paths = extractPathsFromQuery(req.query, transformer);
const language = extractLanguageFromQuery(req.query, options);
const markers = extractMarkersFromQuery(
req.query,
options,
Expand Down Expand Up @@ -1016,6 +1035,7 @@ export const serve_rendered = {
next,
overlay,
'static',
language,
);
} catch (e) {
next(e);
Expand Down Expand Up @@ -1077,6 +1097,7 @@ export const serve_rendered = {
: item.dataProjWGStoInternalWGS;

const paths = extractPathsFromQuery(req.query, transformer);
const language = extractLanguageFromQuery(req.query, options);
const markers = extractMarkersFromQuery(
req.query,
options,
Expand Down Expand Up @@ -1150,6 +1171,7 @@ export const serve_rendered = {
next,
overlay,
'static',
language,
);
} catch (e) {
next(e);
Expand Down Expand Up @@ -1184,14 +1206,16 @@ export const serve_rendered = {
};

let styleJSON;
const createPool = (ratio, mode, min, max) => {
const createRenderer = (ratio, createCallback) => {
const createPool = (scale, mode, min, max, language) => {
const createRenderer = (createCallback) => {
const renderer = new mlgl.Map({
mode: mode,
ratio: ratio,
ratio: scale,
request: (req, callback) => {
const protocol = req.url.split(':')[0];
// console.log('Handling request:', req);
if (options.verbose) {
console.log('[VERBOSE] Handling request:', req);
}
if (protocol === 'sprites') {
const dir = options.paths[protocol];
const file = unescape(req.url).substring(protocol.length + 3);
Expand Down Expand Up @@ -1282,7 +1306,9 @@ export const serve_rendered = {
const extension = path.extname(parts.pathname).toLowerCase();
const format = extensionToFormat[extension] || '';
if (err || res.statusCode < 200 || res.statusCode >= 300) {
// console.log('HTTP error', err || res.statusCode);
if (options.verbose) {
console.log('HTTP error', err || res.statusCode);
}
createEmptyResponse(format, '', callback);
return;
}
Expand All @@ -1305,13 +1331,37 @@ export const serve_rendered = {
}
},
});
renderer.load(styleJSON);

if (options.verbose) {
console.log('[VERBOSE] createRenderer', scale, mode, language);
}

let rendererStyle = styleJSON;
if (language) {
const layers = [...styleJSON.layers];
rendererStyle = { ...rendererStyle, layers };

const translator = {
getLayoutProperty: (id, key) =>
layers.find((layer) => layer.id === id).layout[key],
setLayoutProperty: (id, key, value) => {
const i = layers.findIndex((layer) => layer.id === id);
const layer = { ...layers[i] };
layers[i] = layer;
layer.layout = { ...layer.layout, [key]: value };
},
translateLayers,
};
translator.translateLayers(layers, language);
}

renderer.load(rendererStyle);
createCallback(null, renderer);
};
return new advancedPool.Pool({
min: min,
max: max,
create: createRenderer.bind(null, ratio),
create: createRenderer.bind(null),
destroy: (renderer) => {
renderer.release();
},
Expand Down Expand Up @@ -1471,13 +1521,24 @@ export const serve_rendered = {
const j = Math.min(maxPoolSizes.length - 1, s - 1);
const minPoolSize = minPoolSizes[i];
const maxPoolSize = Math.max(minPoolSize, maxPoolSizes[j]);
map.renderers[s] = createPool(s, 'tile', minPoolSize, maxPoolSize);
map.renderers_static[s] = createPool(
const languages = new Set([null, ...(options.languages || [])]);

map.renderers[s] = createPool(
s,
'static',
'tile',
minPoolSize,
maxPoolSize,
null,
);
for (const language of languages) {
map.renderers_static[`${s}${language}`] = createPool(
s,
'static',
language ? 1 : minPoolSize,
language ? 1 : maxPoolSize,
language,
);
}
Comment on lines +1533 to +1541
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why each language needs its own rendering pool. shouldn't the generic static renderer be able to handle any style you sent to it with renderer.load(styleJSON); .

If you wanted more pools due to supporting more languages, you could always still set the maxPoolSizes option.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd caution again adding to many more pools than the user specifies. I think I already pushed that a bit doubling the pool size for tile/static mode,

Sometimes that maxPoolSizes is important for lower power systems, which need to limit the number of cores that can be used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, pool handling was the main point where I had doubts. I suspected the lower power constraint, but needed confirmation and details, so thanks.
I haven't tried calling renderer.load() before each render. That's what you are suggesting, right? I was afraid of the performance penalty. That's the only reason we have separate pools for scale ratios and for 'tile' vs 'static', right? Instead of one big pool with dynamic configuration before each render.
If I remember correctly, renderer.load() causes requests of some assets.
What should I investigate first in your opinion?

Copy link
Collaborator

@acalcutt acalcutt Oct 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The separate pools for static vs tile mode have two do with two issues.

when used as a tile server, maplibre-native tile mode is important to prevent label clipping. when in tile mode, maplibre considers information beyond the edges of the tile, which helps with labels that may go beyond the edge from the next tile (note that the source has to support this bit of extra data, like plantiler and openmaptiles tools do , but tilemaker does not)
maplibre/maplibre-native#284

However we have found that recent changes in maplibre break static images when in tile mode. the reason is that a change has been made when it was still mapbox-native that limited tile mode to only return a single tile. since a static image can span multiple tiles, the resulting image ends up with missing data. to work around this I made static images render in maplibre 'static' mode, which can span multiple tiles. (but does not deal with label clipping in any way)
#608
acalcutt#5

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't doubt we need to have different static and tile modes.
But why can't we have a single pool that constructs a new mlgl.Map for each render? Is that really slow?
I see now that the situation is different for the style (.load() method) than it is for the mode and the scale (which have to be set in the Map constructor). But if it's slow to build new mlgl.Map dynamically, is calling .load() slow too?

Copy link
Collaborator

@acalcutt acalcutt Oct 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure on the performance question about mlgl.Map each render. I do know from basic testing on a raspberry pi that lowering the pool value helped a lot. tileserver-gl defaults to a pool value of 8, but that little pi ran better with a pool of 4, because without that when someone when to a page and had to render a bunch of images, the 4 core cpu was pegged beyond 100% and just decreased the speed of the rendering for all images. at least set to 4 the pi could handle it, and it actually resulted in it rendering faster. I always assumed the pooling was meant as a limiter.

As for callling load, that is what i does now so it seems fine. i guess the question would be does putting the translate down in front of it really slow it down much?

The pooling may also have to do with the way maplibre-native in very linier in the way it operates. I know I had done some basic testing at https://github.com/acalcutt/maplibre-node-test/ and just getting it to do multiple images in a row was a pain. I was working with @tdcosta100 just trying to understand how it worked.

Copy link
Contributor

@tdcosta100 tdcosta100 Oct 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pool is needed because you cannot render multiple images on the same map, that is, each map can render ony one image at a time. And you need different map instances for each resolution, because it's something you configure when instantiating the map and cannot change that later. That said, there are some strategies for rendering different styles (or the same style with different language). Creating lots of instances (using several pools) can lead to consuming lots of memory. One solution is loading the style before the rendering pass, like @acalcutt suggested. This will lead to cache to be cleared everytime, so you will may have performance issues. But I think it's better than creating lots of map instances, unless you specify minPoolSize as 0, so you will create map instances only if you really use them.

}
});

Expand Down
5 changes: 5 additions & 0 deletions src/server.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ function start(opts) {
}

const options = config.options || {};
options.verbose = opts.verbose;
const paths = options.paths || {};
options.paths = paths;
paths.root = path.resolve(
Expand Down Expand Up @@ -441,6 +442,10 @@ function start(opts) {
style.thumbnail = `${center[2]}/${Math.floor(
centerPx[0] / 256,
)}/${Math.floor(centerPx[1] / 256)}.png`;

style.static_center = `${center[0].toFixed(5)},${center[1].toFixed(
5,
)},${center[2]}`;
}

style.xyz_link = getTileUrls(
Expand Down
Loading
Loading