New builder

[simoncozens]

New builder sketch

The current builder does a good job of taking source files and producing a range of outputs suitable for Google Fonts use. However, it statically encodes the process for turning a given source (Glyphs or Designspace) into a set of output files. We can turn on and off some of those outputs, and we can (minimally) customize the process, but we don't have the ability to introduce new "steps" or transformations. And yet we do have a number of font production and post-production processes which can't be currently achieved with the existing builder:

• A different set of outputs for Noto, requiring its own notobuilder.
• Noto also requires adding Latin subsets (and other non-Latin fonts will require adding a subset of Latin Kernel); this is a source-source (UFO-level) transformation.
• Adding COLRv1 paints using paintcompiler.
• Noto's UI fonts require cmap remapping, baseline shifts and transformations.
• Using hb-subset to slim down / repack the font.
• Handwriting and other "mega-fonts" will require variable font subspacing, feature freezing and more.
• Other users have wanted PS hints and other custom operations.

Currently this requires horrible Makefile hacks or ad-hockery. Rod has also expressed a desire to have a single build process that Just Works to build everything we onboard - and I think it can be done...

To see how to do it, we need to take a step back before we step forwards. I started out by thinking about how an end-user would represent, in a config file, the desired set of build operations for a custom production process. Essentially we are representing a tree structure where different sources are transformed, saved, passed to other operations, saved as a different file, and so on. But the easiest way to represent that tree structure is just to list the operations and let the computer work out the most efficient way to do them. For example, let's compile NotoSerifEthiopic.glyphs into three variable font targets: an unhinted font, a slimmed-down VF for Android, and a hinted VF. We could somehow try to represent "compile the VF, save as unhinted; pass one copy to varLib.subset and save as slim-vf; pass another copy to ttfautohint and save as hinted". But it's actually easier to represent that as a set of distinct operations:

recipe:
  fonts/NotoSansEthiopic/unhinted/variable-ttf/NotoSansEthiopic[wdth,wght].ttf:
    - source: NotoSerifEthiopic.glyphs
    - operation: buildVariable
    - operation: buildStat
    - operation: fix
  fonts/NotoSansEthiopic/unhinted/slim-vf/NotoSansEthiopic[wdth,wght].ttf:
    - source: NotoSerifEthiopic.glyphs
    - operation: buildVariable
    - operation: buildStat
    - operation: subspace
    - operation: hbsubset
  fonts/NotoSansEthiopic/hinted/NotoSansEthiopic[wdth,wght].ttf:
    - source: NotoSerifEthiopic.glyphs
    - operation: buildVariable
    - operation: buildStat
    - operation: fix
    - operation: autohint

and let the computer reconstruct the tree itself.

It turns out to be pretty easy to make the most efficient sequence for this: for each target, walk through the list of operations build a graph from a "source file" node to another "source file" or "binary file" node using the operation as the edge, labeling the final target node with the target's filename. So for target 1:

 [NotoSerifEthiopic.glyphs]
        |
        | buildVariable
        v
[ Binary File Node ]
        |
        | buildStat
        v
[ Binary File Node ]
        |
        | fix
        v
[unhinted/variable-ttf/...]

Then process target 2:

        NotoSerifEthiopic.glyphs
                |
                | buildVariable
                v
        [ Binary File Node ]
                |
                | buildStat
                v
        [ Binary File Node ]
               / \
          fix /   \ subspace
             /     \
[unhinted...]   [ Binary File Node ]
                    |
                    | hbsubset
                    |
                [ slim-vf/... ]   

And so on; finally, label the unnamed file nodes with a temporary filename.

This graph structure can be very naturally transformed into a ninja build file, (let's say each "operation" such as buildVariable is defined by a Python module which ensures that the incoming file and outgoing file are compatible and writes the ninja rules for that transformation) and then we're basically done.

In the configuration YAML, as well as operation we could specify additional arguments passed to the Python modules which implement each operation; we could even have an escape-hatch exec step which calls an arbitrary binary (although we would want to provide modules for most operations that people might want to do on a font):

  fonts/NotoSansMath/hinted/NotoSansMath-Regular.ttf:
    - source: NotoSerifEthiopic.glyphs
    - operation: buildTTF
    - operation: exec
      cmd: python3 scripts/add-math-table.py -o $out $in
    - operation: autohint

Stepping forward

If we do that, we have a very flexible builder, but we've gone backwards in the sense that we're now requiring the user to specify all the steps and all the outputs, when the current builder works out what we need in terms of the desired GF fonts artefacts and directory layouts. Ideally we'd like the new builder config to look like the old config in most cases - the user just provides the names of the source files, and we do the rest in the usual GF-y way.

To fix that, let's have the new builder read in the current source file, examine the sources and add its own recipe field to the in-memory data structure. Any additional entries provided by the user in the recipe field are taken as overrides. We could assume a default value in the config of recipe_provider: googlefonts to use the fontbuilder.recipes.googlefonts module to perform this examination/infilling, allowing for other recipe providers (noto, etc.)

Todos

• Write a ufomerge operation to duplicate the functionality of notobuilder's subsets
• Improve the handling of STAT table generation (only do it once with all VFs, not once per VF)
• Write a cmap remapping / glyph swap operation
• Make sure we can do everything that notobuilder.makeuivf currently does
• Test, test, test

[m4rc1e]

I'm loving the operations part. Having individual pieces which are still super reusable is a major win. I've only skimmed but I assume we can use this new builder as a hot fixer by skipping the fontmake part completely?

[simoncozens]

Yes, I think so. The only fiddly bit is that if you want to hotfix a file in-place, you need to say postprocess: fix instead of operation: fix. This is because with

recipe:
  Foo.ttf:
     - source: Foo.ttf
     - operation: fix

the target of the ninja rule will be Foo.ttf which clearly already exists so ninja will do nothing. If you say

recipe:
  Foo.ttf:
     - source: Foo.ttf
     - postprocess: fix

then the target of the ninja rule will be a "stamp file" and it'll do the right thing.

[simoncozens]

Good grief, it finally passed on Windows. OK, um, I think we're done.

[simoncozens]

Something went wrong with that last (JSON) commit.

[simoncozens]

f86fddf was bad, working on it.

[simoncozens]

Recent builder1 changes to port:

• otfautohint
• WOFF after VTT
• fix_hinted_font after VTT
• removeOutlineOverlaps
• reverseOutlineDirection

[simoncozens]

@m4rc1e I believe this is now ready.

[m4rc1e]

@bramstein This PR is a major overhaul of the existing builder. The aim of this PR is to consolidate all of our custom build chains into one, since we have many. In order to do this, Simon has created a more granular approach to building fonts aka recipes + operations. It shouldn't affect any of your font builds since it is backwards compatible. I'd love you to test this PR just to make sure.

@simoncozens feel free to correct what I've written above or explain more.

[m4rc1e]

cc @IvanUkhov

[bramstein]

@m4rc1e Thanks for letting us know. We'll give it a try (currently upgrading to the latest gftools), and then we can test.

[IvanUkhov]

Perhaps one observations, which might or might not be relevant here, is that cleanUp does not seem to cover it all. In particular, /instance_ttf and *.designspace* are left behind, which was not happening before.

[m4rc1e]

I've just installed a fresh venv and I'm getting the following traceback when testing Dosis

  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/marcfoley/Type/tools/Lib/gftools/builder/__init__.py", line 17, in <module>
    from gftools.builder.schema import BASE_SCHEMA
  File "/Users/marcfoley/Type/tools/Lib/gftools/builder/schema.py", line 18, in <module>
    from gftools.packager import CATEGORIES
  File "/Users/marcfoley/Type/tools/Lib/gftools/packager.py", line 43, in <module>
    from gftools.util import google_fonts as fonts
  File "/Users/marcfoley/Type/tools/Lib/gftools/util/google_fonts.py", line 43, in <module>
    from glyphsets.subsets import SUBSETS
ModuleNotFoundError: No module named 'glyphsets.subsets'

Maybe the glyphsets' api has changed during the development of builder2.

[simoncozens]

Something is wrong with your venv. The builder2 branch has this:

gftools/Lib/gftools/util/google_fonts.py

Line 43 in ed030d9

from google.protobuf import text_format

[m4rc1e]

Been testing it on the upstream repos on my system and they seem to work. No doubt we've missed some edge cases but we can improve it this week.

I suggest we merge this PR and the packager and make the first v1.000 release.

[m4rc1e]

We should support instances, #850

[simoncozens]

Ack, oops. I'll fix this today.