Indic language maps [name only]

[naveenpf]

Need indic langauge maps only for the names.
http://mlm.jochentopf.com/

No of translations would improve if we can showcase indic language maps.

[planemad]

@naveenpf thanks for bringing this up. Any immediate application of this you are seeing?

If we migrate the style from raster to GL, this should be possible. Need to experiment. Reference: https://www.mapbox.com/mapbox-gl-js/example/language-switch/

[naveenpf]

@planemad We had offline discussions, what project we have to take up after state highways in Kerala. Few are interested in taking up translation of the map. It would be good to showcase ml map in osm.in, so that we will know the progress and will help to get more volunteers to contribute.My guess is this will help get more volunteers to contribute in all indic languages.

http://wiki.openstreetmap.org/wiki/Kerala

[planemad]

@naveenpf Amazing! Following progress on the OSM wiki. cc @maning

[planemad]

Tried my hand at a GL Malayalam map

View

• Extracted the data via overpass query name:ml and place and created a Studio tileset using these instructions
• Created a new Studio style with the tileset, replaced the place label layers to use the uploaded data.

Next actions

• Blocked: Migrate default style to GL #15 Migrate the site to GL
• Create a js language switcher to switch to other Indic languages. Reference
• Automate syncing the Mapbox tileset with OSM data

cc @naveenpf

[naveenpf]

Thanks a lot @planemad ... excited to see malayalam map :-) ...

[planemad]

@naveenpf Can you suggest two Malayalam fonts to use. One with a variety of different weights like bold/semibold/light and another visually different to use for seas and oceans, like how we have serif typefaces in latin.

[naveenpf]

I think we can use Rachana Bold and AnjaliOldLipi
http://smc.org.in/fonts/

[santhoshtr]

Note that using good fonts is not enough. As we see in the gif above, the complex script rendering is completely broken. The library that renders the text on the maps should have complex script rendering capability.

[planemad]

@naveenpf updated to Rachana.

@santhoshtr yes, complex script rendering is yet to be built in GL maps mapbox/DEPRECATED-mapbox-gl#4 . Any good example you can point out for a test fixture?

[santhoshtr]

Any good example you can point out for a test fixture?

കോഴിക്കോട് (Kozhikkod/Calicut) should render as

[planemad]

Thanks @santhoshtr . From upstream, seems like we might start to see progress on this by mid April.

[manuvarkey]

Complex rendering is fine in mapbox-classic. See this.
[(https://api.mapbox.com/v4/manuvarkey.2267f3e9/page.html?access_token=pk.eyJ1IjoibWFudXZhcmtleSIsImEiOiJjaWdlMTZsd24xZWZ3NnJtNzk2dXhvcjJmIn0.coLgvL8Ct439UzhAmJwI1w#10/11.9836/75.8620)]

@santhoshtr - please see these bug reports for OsmAnd relating to this issue.
[https://github.com/osmandapp/OsmAnd/issues/2016]
[https://github.com/osmandapp/OsmAnd/issues/1161]

[planemad]

@manuvarkey thats right, the older raster tiles uses mapnik rendering which supports complex shaping. Bringing this support to in browser rendering with GL will take some more time.

For now, the best approach might be to allows layer switching between the english tiles and various languages created by individuals. The style+data updates will be the responsibility of the contributor.

[naveenpf]

@planemad what is your suggestion on making this as www.ml.openstreetmap.in with proper India borders ?

[pnorman]

@manuvarkey thats right, the older raster tiles uses mapnik rendering which supports complex shaping. Bringing this support to in browser rendering with GL will take some more time.

I'd suggest avoiding Mapbox GL unless you have internal information that suggests that mapbox/DEPRECATED-mapbox-gl#4 is close to resolution. Glyph placement, unicode and bidi issues often take a long time to solve, and it's important to have indic scripts rendering right for an Indian site.

With vector tiles like mapbox classic it's still possible to have multiple languages in the vtiles and then just the quick vtile->png step needs to be done for each language.

If you want client-side rendering, I'm not sure if Tangram is any better for bidi support.

[planemad]

🚨 Blocked by complex text support in Mapbox GL. Tracking here: mapbox/mapbox-gl-native#7774

[demonshreder]

I know GL migration is a goal but is it possible to configure current raster tiles to show indic names through changes in style script?

[planemad]

@demonshreder absolutely, given that GL might not get Indic script support soon. Maybe something like https://jsfiddle.net/planemad/Lcvp5p7x/ ?

[demonshreder]

This is great. Can this code be copy pasted for anyone to use? Should we be worried about someone using the same API key, if so I can try to replicate this with openlayers.js.

[planemad]

Trying to think through the options we have for making a multilingual map. See also https://wiki.openstreetmap.org/wiki/Map_internationalization

Goal OSM maps with a language selector to change labels to any Indic language

Client side vector rendering

OSM India has community account vector tile hosting sponsored by Mapbox, so thats a big plus. Its also fairly easy to setup our own vector tiles server if needed. This solves the headache of creating the tiles with multilingual data, but rendering options are limited.

• Mapbox GL would have been preferred due to the extensive capability but Indic script support is still lacking Full "complex text" support: indic scripts, ligatures, kerning, etc. mapbox/mapbox-gl-native#7774
  
  
  Renders as Napolam instead of Nepalam in Tamil
• Openlayers GL same issue as Mapbox GL
• tangram. There are a few demos from the past which had done indic rendering but they all seem down and current development status of the project is unknown :
  • https://thejeshgn.com/2016/09/24/kannada-web-maps-using-tangram-by-mapzen/ @thejeshgn
  • https://github.com/tallytalwar/tangrams-indic @tallytalwar @srikanthlogic

Server side raster rendering

Server side tile renderers can be cumbersome to setup and are resource intensive, however we can be assured that all the languages will be rendered correctly with the mapnik renderer

• Mapbox is our current raster tile host on http://openstreetmap.in/ and supports indic rendering. Unfortunately the raster tile service is getting deprecated ⛔
• maptiler. Their raster tile service seems to be using GL based rasterizing which is identical to Mapbox and does not support the indic text correctly.
• Wikimedia tiles: https://jsfiddle.net/planemad/Lcvp5p7x/22/ has map tiles for every language. Downside is that it does not update regularly and we cannot control any of the styling and modify the borders..
• OSM India tile server: Setting our own tile server seems like the only feasible option if we want control over the data updates and any styling ✅ There are a few options for setting this up openmaptiles, switch2osm

Conclusion

It does not seem like there is good solution that will work out of the box for custom styled indic language maps.

Short term, we could use the Wikimedia indic tiles with a custom boundary overlay https://jsfiddle.net/planemad/Lcvp5p7x/23/ as our multilingual map.

Long term, we should look at setting up our own raster tile server to generate tiles for every indic language. These can probably be just transparent label only tiles which can be overlaid onto any vector basemap. This might be a fairly involved lift and might need some dedicated hands to dig into the technical details. Any volunteers?

[tallytalwar]

Thanks for the tag @planemad. tangram (web) is still in active development, tangram-es is more a hobby project from us core developers now (all development is under Linux Foundation).

tangram (webgl) supports all indic fonts, I had extensively tested this with Brett when we were working on it and did a demo for this, if needed I can cook up something using the mapzen stylesheets.

tangram-es supports all indic too with full bidi, complex text - shaping/positioning support, and both tangram web and native renderers will work with the same stylesheet, just in case osm-in plans to have a mobile app! (An example of an app using tangram-es heavily is https://github.com/westnordost/StreetComplete).

Anyways let me know if you have more questions or want any specific help from me.
Good Luck.

Edit: PS. I kinda disagree with the conclusion, should definitely give tangram a try, we spent a lot of time focusing on language agnostic rendering for both native as well as web map renderers (with completely different underlying approaches for both). tangram web ends up using browser's text rendering capabilities and native renderer we implemented our own using hafbuzz, icu, bidi, etc.

[planemad]

Thanks @tallytalwar, if tangram is under active development thats a big 👍 to use it. If you can help with getting the tangram indic demo running again, that would be a great first step since you would be most knowledgable of what needs to be fixed.

After that it should be fairly trivial to add a language picker to allow users to toggle between options.

[tallytalwar]

Great @planemad, thanks for considering.

I will try to spend sometime this week (time is hard to find these days with covid!! :P) and get to this. I am not a web person, so might be pinging the tangram developers if I am stuck with something, but will keep you guys updated.

Also if in the meantime you or anyone wants to play with things on tangram web: https://tangrams.github.io/refill-style/#3.25/20.36/467.42 (https://github.com/tangrams/refill-style).

[planemad]

@tallytalwar thanks for the links, forked refill-style to https://github.com/osm-in/map with some modifications and it seems to work! Did the old indic demos have some other features that was not in this?

Demo: https://osm-in.github.io/indic-map/

Issues

• The nextzen OSM vector tiles that is used has not been updated since 2017. We would need to replace this with a more updated vector tile source. Changing the source will also require significant map style changes.
  • The tiles do not contain all language labels at all zoom levels.
• Borders need to be fixed as per official ones

[tallytalwar]

Great @planemad . Sorry I haven't been able to work on the indic demo rep, but great refill worked great for you.

On nextzen tiles yeah, switching vector tiles would involve a lot of style changes, unless there is an updated service based on https://github.com/tilezen/vector-datasource. Let me ping a few people here and see what's the status on updates.

Do you have any specific question I can relay?

Again, apologies that I havnt been able to get going on the indic demo.

[tallytalwar]

And for a playground for tweaking map style you can try tangram-play (https://github.com/tangrams/tangram-play).

[tallytalwar]

Let me send you info on borders too, some work was done to get borders working for different conflicting areas of the world in collaboration with here for nextzen vector tiles.. digging this now.

[tallytalwar]

Ok you should be able to follow the instructions from here and make appropriate modifications to the style sheet to reflect the map borders.. https://www.openstreetmap.org/user/nvk/diary/390496

This might be non trivial change so let me know if you need more info here.

[tallytalwar]

Also based on the link above for tilezen (source for nextzen vector tiles), its released quarterly, any specific data you are missing, maybe it's not styled?

[planemad]

tangram-play

@tallytalwar Wonderful. Was able to load the style from the url https://raw.githubusercontent.com/osm-in/map/master/refill-style.yaml in https://play.tangram.city/

This should be good to try out the border changes

Also based on the link above for tilezen (source for nextzen vector tiles), its released quarterly

Quarterly is pretty good., https://www.nextzen.org/ seemed to indicated the OSM data was archived from 2017, although i can see changes made after that. There does not seem any specific update frequency mentioned anywhere.

• Tile source: https://github.com/tilezen/vector-datasource
• Tile docs: https://tilezen.readthedocs.io/en/latest/layers/

any specific data you are missing, maybe it's not styled?

In lower zoom levels, the place data seems to be coming from natural earth which has limited indic language name coverage (only Hindi). What would be the best way then to make a fully localised map at a country level?

Theres a similiar issue on higher zooms with the whosonfirst neighbourhood data that lacks the localisation coverage of OSM places. Prominent area names dont have the local names.

[planemad]

Success! Got the borders updated https://osm-in.github.io/indic-map/

[tallytalwar]

@planemad This is all great, glad all this was helpful to you. Really happy to see tangram being used in osm-in :).

What all is needed next other than wof data and natural earth low zoom data?

[planemad]

What all is needed next other than wof data and natural earth low zoom data?

@tallytalwar Thats seems like the only issue with the tiles. It would be good to know the update frequency and the next release.

On a tangram level, how would one go about implementing dynamic label localisation from Wikidata like https://observablehq.com/@planemad/country-names-in-any-language :

• Get all the tile features with wikidata_id in the map view
• Query Wikidata for localised names for these ids
• Add a dynamic layer to the map and label these features with the localised names