App Translations / Internationalisation

[gfwilliams]

This is a replacement for #136 - without the discussion about how to implement it...

Historically, we used the locale module for translations (http://www.espruino.com/Bangle.js+Locale) but this only provided extremely basic translation (yes/no/etc). While more translations could have been added, these would use up memory and would make apps run slowly.

We are now moving towards translating apps automatically in the app loader during the upload process

It works pretty simply:

• In the app.js file, write any text in English and put /*LANG*/ in front of any text string that should be translated
• Upon upload, the app is scanned for these strings, and they are replaced if they are found in the translation file in the lang folder, for instance https://github.com/espruino/BangleApps/blob/master/lang/de_DE.json

Current status:

• It works. 'Translations' is added to the settings in the About page of the App Loader
• About the only app with /*LANG*/ entries in at the moment is Settings - we need more apps with the LANG tag added
• Translations for different languages are very lacking - run bin/language_scan.js to get an idea. we need more translations! Maybe we want to try doing translations automatically so we have some placeholders?
• Translations are not 'smart'. While they are aware of punctuation at the beginning and end they're not aware of capitalisation - perhaps if this worked better we could get away with less translations required.

Any help that can be provided is hugely welcome! This has been requested a lot so I've done what I can, but my language of non-english languages is seriously lacking.

[HilmarSt]

sorry for this newbie question, but how do I "run bin/language_scan.js" on a windows PC?

[gfwilliams]

You'd have to install node.js (https://nodejs.org/en/) but once you do that, I believe you get a shell window you can open. You can go to the directory and run node bin/language_scan.js

[alessandrococco]

Hi @gfwilliams, I have a doubt about the key of a translation.

language_scan.js returns This will remove everything but the phrase is This will remove everything! (as in E.showPrompt(/*LANG*/'This will remove everything!', ...) What should I use as a key? This will remove everything or This will remove everything!? (I think the latter)

[gfwilliams]

I think you need to use This will remove everything since the actual translation uses the keys without any punctuation.

At some point it could probably be improved but right now that's how it works:

https://github.com/espruino/EspruinoAppLoaderCore/blob/c243e6e71f88358de720ad16ba8515b32b8d650f/js/appinfo.js#L50-L56

[alessandrococco]

Hi @gfwilliams, I came across another special case: if a string contains \n it won't be translated.

In setting/settings.js there's the line E.showPrompt(name + /*LANG*/"\nStay Connectable?". The language_scan tool returns nStay Connectable as key. I tried using \nStay Connectable, nStay Connectable and Stay Connectable but the string is not translated. The same happens with Connect device\nto add to\nwhitelist.

1. Should \n at the beginning and end be handled as punctuation? See translateJS: handle \n at beginning/end of the string EspruinoAppLoaderCore#14
2. How should be handle the \n in the middle of a String?

As a workaround for a \n at the beginning/end I can update the source string from /*LANG*/"\nStay Connectable?" to "\n" + /*LANG*/"Stay Connectable?" but it feels dirty :-D

[gfwilliams]

Ahh, thanks - yes, this is due to the slightly hacky lexer that's in EspruinoTools. Should be fixed now.

[gfwilliams]

Actually it's not fixing this bug - just what you mentioned above :)

[copoer]

@gfwilliams This should hopfully solve all your issues:
#1414

[gfwilliams]

Thanks - just commented on that PR. Needs a few tweaks before I can merge but it looks like a great way to get started with translations

[pebl-hank]

You'd have to install node.js (https://nodejs.org/en/) but once you do that, I believe you get a shell window you can open. You can go to the directory and run node bin/language_scan.js

When I do that, I get
`Initialising Notifications
Initialising Status
Initialising Utils
Initialising Config
Initialising Serial
Initialising CodeWriter
Initialising Modules
Initialising Env
Initialising Assembler
Initialising GetGitHub
Initialising CoreModules
Initialising Pretokenise
Initialising SaveOnSend
Initialising SetTime
apps.json has not been generated, running bin/create_apps_json.sh to build it...
child_process.js:830
throw err;
^

<ref *1> Error: spawnSync C:\myfolder\bjs\bin/../bin/create_apps_json.sh UNKNOWN
at Object.spawnSync (internal/child_process.js:1107:20)
at spawnSync (child_process.js:776:24)
at Object.execFileSync (child_process.js:822:15)
at Object. (C:\myfolder\bjs\bin\language_scan.js:134:18)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:12)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
at internal/main/run_main_module.js:17:47 {
errno: -4094,
code: 'UNKNOWN',
syscall: 'spawnSync C:\myfolder\bjs\bin/../bin/create_apps_json.sh',
path: 'C:\myfolder\bjs\bin/../bin/create_apps_json.sh',
spawnargs: [],
error: [Circular *1],
status: null,
signal: null,
output: null,
pid: 0,
stdout: null,
stderr: null
}`

[samuelstroschein]

Would adding github/inlang/inlang to this repository help with the translation effort?

I could open a PR that adds the config file. Nothing else is required. Contributions open a PR.

[tonykakuuu]

I want to provide a Japanese locale but realized that this not yet possible and groundwork regarding fonts needs to be done. I read through the PR #399 and I'm not sure this is even feasible. Are there any prior resources on this or has anything been done to support Japanese/Chinese fonts beyond implementing image based rendering via Gadgetbridge?

[gfwilliams]

@samuelstroschein I'm not quite sure I understand what that repo really adds? It feels like the current solution with bin/language_scan.js is if anything easier as it even sorts the translations by how many apps use them and will even automatically add translations for you.

@tonykakuuu I'm afraid there's been nothing else added for Japanese/Chinese fonts. The way we can only render one 8 bit codepage at a time causes quite a bit problem - I think the best approach is that done for the image based rendering... I just made a tool (bin/language_render.js) that takes the Japanese translations from lang/unicode-based/ja_JA.json, renders them to images and then uses webtools/imageconverter.js to make them into 1bpp images that can then be written into lang/ja_JA.json.

I can't read japanese, but the result looks quite promising:

If you try the development app loader you should be able to try it out

edit: Ideally we'd have a bitmap font so we could render it better to the low-res 1bpp image. This looks promising if someone wanted to have a try at tweaking language_render.js to load it https://unifoundry.com/unifont/

edit 2: I've just done this - it's much better

[samuelstroschein]

@gfwilliams

Inlang is supposed to become the "localization" layer one can use to reduce the effort of localization from source code-related things to receiving a UI to edit translations in a repository.

For bangle specifically:

1. The custom language_scan.js could be removed and maintenance effort reduced. The upcoming inlang CLI will provide machine translations for missing translations along with additional features.

2. Contributors receive a UI over the translations in this repository.

[gfwilliams]

...receiving a UI to edit translations in a repository.

Thanks - I think in this case it's probably not that useful for us. I mean, using the online editor with https://github.com/espruino/BangleApps/edit/master/lang/de_DE.json is hardly a massive hurdle for most people. It doesn't need a fancy UI - at most I guess a few lines of docs on how to add translations are needed.

Our issue isn't that translation is hard for anyone, it's just that very few people contribute to the translations... We just need people to spend time on that, rather than us spending even more time on infrastructure that nobody is interested in using.

@tonykakuuu as above I've changed the font now so I'd be really interested to see how the japanese translations in the development app loader look to you.

[nxdefiant]

My 2 cents, I just want to say honestly why I do not translate much:

• There is nothing worse then a half English half translated app.
• English has in many cases shorter words then my native language which is beneficial on a small display.
• I still have not found good translations for words like "Timer".
• I may not be able to write (good) English, but I think I can read it. I guess this is valid for rest of the Bangle.js audience.
• It is still good to be able to translate stuff if needed.

[tonykakuuu]

@gfwilliams This is actually brilliant, well done. The translations look mostly correct and I'll go through them in more detail once I have an afternoon.

[gfwilliams]

@tonykakuuu that's great news! Thanks for checking it out!

@nxdefiant Thanks - in Germany in particular it feels like there's a super high English literacy level. And as you say, right now most of Bangle.js users will know some English so I guess it's not so important - but that may change in future. It makes me think that going forward it might make sense to find a way to encourage apps to use icons/images instead of text (especially with a more finger-friendly grid layout), and then those would end up being approachable to everyone without translation

[samuelstroschein]

It makes me think that going forward it might make sense to find a way to encourage apps to use icons/images instead of text

What would be the benefit? Avoiding "too long text" that breaks the user interface?

Our issue isn't that translation is hard for anyone, it's just that very few people contribute to the translations... We just need people to spend time on that, rather than us spending even more time on infrastructure that nobody is interested in using.

Agree, getting contributions is the harder part. We are thinking about ways to lower barriers for translation contributions. Not editing files manually is one of them, but it needs more. For example, automatically showing in the README if translations are missing. A click on that README "field" could directly navigate to the missing translation and thereby lower the barrier to contributing.

[tonykakuuu]

In addition to the translation I attempted to create a jp_JP language (locale) using the language_renderer.js. However using the format abday: "Sun,Mon,Tue,Wed,Thu,Fri,Sat", none of the strings would properly show up in the settings menu or any watchface (mostly just breaking the watchface or displaying UNDEFINED). I can't paste the exact string I used here (because it breaks Githubs markup) but I have it on my public repository if anyone wants to test this.

[gfwilliams]

However using the format abday: "Sun,Mon,Tue,Wed,Thu,Fri,Sat", none of the strings would properly show up in the settings menu

Yes, I think you'd expect that... Right now the code for working out the day from that string is basically abday.split(',')[d.getDay()]: https://github.com/espruino/BangleApps/blob/master/apps/locale/locale.html#L201

The issue is when you embed that as an image, even if you're splitting it out into 7 different images separated by commas (but it might actually just end up as one big image) the binary image data in the string may well have a character , which would throw off the split.

You'd need to rewrite the way the locale module works for Japanese in that case.

What you could do that I guess might be more sensible is to create a whole new app for the Japanese locale (without custom.html) that provides a locale module. You could then reference every day of the week separately with /*LANG*/ in front of it, and ensure that there were translations in the current Japanese translations file, and then the app loader would automatically be able to bring in the images as they were needed.