README_BuildProcess.md

Espruino Internals: Build Process

Directories and Files

• ChangeLog: What's new
• boards/: Information on boards, used to auto-generate a lot of the code
• doxygen/: Directory for auto-generated code documentation - see doxygen/README.md for more info
• gen/: Auto-Generated Source Files
• libs/: Optional libraries to include in Espruino (Math, Filesystem, Graphics, etc)
• make/: Makefile fragments for each architecture
• misc/: Other useful things
• scripts/: Scripts for generating files in gen, and for analysing code/compilation/etc
• src/: Main source code
• targetlibs/: Libraries for targeted architectures
• targets/: Specific code for targeted architectures
• tests/: JavaScript Testcases
• benchmark/: JavaScript Benchmarks
• dist_*: files to be copied into distribution zip file
• Makefile: the Makefile which controls the build process

Build Commands

Building for an Espruino board is generally done by typing:

BOARD=BOARDNAME make

And for release, by typing:

RELEASE=1 BOARD=BOARDNAME make

Valid board names are in comments at the head of the Makefile..

Release builds have all the assertions removed, reducing flash usage. Normal builds still have the assertions, and debug builds (with DEBUG=1 have more debug info in the assertions, as well as the debug info needed for proper GDB debugging).

The Make system has the following targets:

• clean - Clean the build
• all (or unspecified) - Build, including binaries needed for flashing the firmware
• flash - Generate the firmware files and flash to the board
• serialflash - Generate the firmware files and flash to the board via serial bootloader (used for STM32, especially the Original Espruino and Espruino Pico)

Build Process

• Overall build logic is in the Makefile.
• The Board Definition File in boards/$BOARD.py is parsed by scripts/get_makefile_decls.py to work out extra definitions for the Makefile (chip family, part number, binary name, whether it has a bootloader), which control the build.
• Architecture-specic build logic is in the make directory - this is called based off family in the Board Definition File in boards/.
• The Makefile adds source files to $SOURCES, and 'wrapper' source files to $WRAPPERSOURCES. Wrapper files are files that contain functions that are exposed to JS.
• The script scripts/build_platform_config.py is run which generates gen/platform_config.h from boards/$BOARD.py - this contains information like the amount of RAM, as well as buffer sizes and the amount of variables that will be stored.
• The script scripts/build_pininfo.py creates the pin definitions and puts them in gen/jspininfo.c
• The script scripts/build_jswrapper.py is then run on $WRAPPERSOURCES - it generates gen/jswrapper.c - a hard-coded symbol table of built-in functions - from the Wrapper files.
• Source files are built
• On embedded targets, a linker file gen/linker.ld is auto-generated by scripts/build_linker.py based on the board definition (available flash, ram, bootloader etc). In some targets (especially non-STM32) a pre-made linker file will be used instead.
• Everything is linked using link time optimisation (where possible) - this helps to inline code that wouldn't otherwise have been inlined, and generally makes for a much more efficient binary.
• The elf file is converted with objdump to:
  • bin (for uploading to Espruino board)
  • hex (for uploading to mbed/etc)
  • lst (assembler listing - used for debugging)
• scripts/check_size.sh does a sanity check of the bin file's size against what's described in the board definition, and fails if it won't fit into available flash memory.

Oddities

• malloc and free are not included, because they're not needed and by default they use a large pool of valuable RAM to speed up memory allocation.
• strcpy, memcpy, memset are all reimplemented in jsutils.h - again because of initial issues with high flash memory usage
• Espruino on STM32 uses its own Maths library for double arithmetic, because when building initially the libraries included by GCC dynamically allocated memory - which pulled in malloc. If we could move back to GCC libraries at some point it'd be a huge bonus

Board Definition File

The board definition files are .py files that reside in the boards folder (for example the Pico one). They're used all over the place:

• To work out the binary's name
• To choose which libraries get built in via the Makefile
• To specify the architecture-specific Makefile fragments are used
• To create gen/platform_config.h via scripts/build_playform_config.py
• To generate the linker file gen/linker.ld using scripts/build_linker.py. Mainly STM32-based targets only.
• To create the pin definitions
• To create the HTML page on each individual board (referenced from the top of the Reference) using scripts/build_board_docs.py
• Converted to JSON (along with the pin declarations) and put on espruino.com to help with autocomplete in the Web IDE (and auto-population of dropdowns in the Blockly editor). Converted by scripts/build_board_json.py

To create a custom board based on one of the distributed board file copy e.g. PICO_P1_3.py to PICO_P1_3_CUSTOM.py and change the content of PICO_P1_3_CUSTOM.py as required for the planed project. The new board file name is than PICO_P1_3_CUSTOM. All created boards with the suffix _CUSTOM.py are ignored by git.

These contain:

info

• Board name and link (for the HTML file)
• boardname - override board name visible at runtime in process.env.BOARD, otherwise the filename of py file is used
• Default console device, pins, and baus rate (the device Espruino goes to when USB is unplugged or not built in)
• variables - The number of variables to use (this depends on the amount of RAM available). Less than 1023 vars use 12 bytes per var, more uses 16 bytes. You have to adjust this such that there is spare room for the stack and static variables (on Espruino boards this means leaving around 16kB free, but on smaller boards it can be reduced a lot)
• bootloader - whether the binary image needs compiling with a special USB-VCP bootloader (Espruino boards only)
• binary_name - the name of the binary that'll be produced
• binaries - available binaries - this is used by the Web IDE to allow the user to choose which binary to upload
• build - controls what gets build via the Makefile:
  • optimizeflags - flags like -O3 to give to the compiler
  • libraries - list of libraries to include - these get transformed into USE_LIBNAME defines
  • makefile - list of commands/definitions to execute in the Makefile

info.makefile definitions

This is a partial list of definitions that can be added in a BOARD.py file's info.build.makefile array, eg: 'DEFINES+=-DBLUETOOTH_NAME_PREFIX=\'"Puck.js"\''

• BLUETOOTH_NAME_PREFIX="..." - NRF52 only: Make the Bluetooth LE device's name BLUETOOTH_NAME_PREFIX followed by the last 2 bytes of the MAC address.
• BLUETOOTH_ADVERTISING_INTERVAL=375 - set the default Bluetooth advertising interval (default 375)
• NFC_DEFAULT_URL="http://foo" - If defined, set the advertised NFC URL to the one given, plus ?a=ble_address. Only do it for a fresh boot - not when code has been saved.
• PIN_NAMES_DIRECT=1 - Package skips out some pins (maybe there's D0,D1,D3 but no D2), so the code must search rather than just offsetting based on pin number.
• DUMP_IGNORE_VARIABLES="...\0" - string containing zero-terminated list of global variable names to ignore when dump() is called. Must be explicityly zero-terminated so there are 2 trailing 0s
• FSMC_BITBANG - if using a built-in FSMC Graphics LCD, don't use the hardware but instead do it in software
• FLASH_64BITS_ALIGNMENT=1 - For testing 64 bit flash writes on linux
• JSMODULESOURCES+=libs/js/.../foo.min.js - include the given JS file as a module that can be used via require("foo")
  • JSMODULESOURCES+=ModuleName:libs/js/.../module.min.js - include as a module used with require("ModuleName")
  • JSMODULESOURCES+=_:libs/js/.../boot.min.js - include as a file that's run at boot time
• JSVAR_MALLOC - Allocate space for variables at jsvInit time, rather than statically
• JSVAR_FORCE_16_BYTE - Force 16 byte JsVars (rather than packing bits to get JsVar size down to the minimum possible)
• USE_FONT_6X8=1 - Also include in a 6x8 fixed width bitmap font
• ESPR_DCDC_ENABLE=1 - On NRF52 use the built-in DCDC converter (requires external hardware)
• ESPR_DCDC_HV_ENABLE=1 - On NRF52840 use the built-in high-voltage (REG0) DCDC converter (requires external hardware)
• ESPR_REGOUT0_1_8V=1 - On NRF52830/40 set the REG0 VCC voltage to 1.8v (the default is 3.3v)
• ESPR_VREF_VDDH=1 - when measuring system voltage (eg for E.getAnalogVRef()) use VDDH, not VDD (only useful if DCDC_HV enabled)
• ESPR_LSE_ENABLE - On NRF52 use an external 32kHz Low Speed External crystal on D0/D1
• ESPR_NO_LOADING_SCREEN - Bangle.js, don't show a 'loading' screen when loading a new app
• ESPR_BOOTLOADER_SPIFLASH - Allow bootloader to flash direct from a file in SPI flash storage
• ESPR_BANGLE_UNISTROKE - Build in 'unistroke' touch gesture recognition
• ESPR_UNICODE_SUPPORT - Build with support for Unicode Strings
• SPIFLASH_SLEEP_CMD - Set if SPI flash needs to be explicitly slept and woken up
• SPIFLASH_READ2X - Enable 2x speed reads of external flash (using MOSI+MOSI as inputs)
• ESPR_JSVAR_FLASH_BUFFER_SIZE=32 - The buffer size in bytes we use when executing/iterating over data in external flash memory (default 16). Should be set based on benchmarks.
• ESPR_PBF_FONTS - Enable support for loading and displaying Pebble-style PBF font files with g.setFontPBF
• ESPR_BLUETOOTH_ANCS - Enable Apple ANCS(notification), AMS and CTS support
• ESPR_USE_STEPPER_TIMER - add builtin Stepper class to handle higher speed stepper handling
• USB_CDC - without this on ESP32C3, USB CDC will not be used for outputting the console

There are some specifically that are useful for cutting a few bytes out of the build:

• SAVE_ON_FLASH - Remove some features (like any ES6 support) to target devices with ~128kB Flash (see next list)
• SAVE_ON_FLASH_EXTREME - Pull out as many features as possible to target devices with ~128kB Flash that also want things like Filesystem support
• ESPR_PACKED_SYMPTR - Packs string offsets of builtin symbols into their function pointers in JswSymPtr, saves 2 bytes per symbol
• JSVAR_FORCE_NO_INLINE - Opposite of JSVAR_FORCE_INLINE. Force getter/setter functions not to be inlined. Saves ~2% code size. Ideally just leave it up to the compiler
• JSVAR_FORCE_INLINE - Opposite of JSVAR_FORCE_NO_INLINE. Force getter/setter functions to be inlined. 2% faster but ~10% extra code size.
• NO_VECTOR_FONT=1 - don't compile in the vector font (this is usually only done for SAVE_ON_FLASH)
• NO_DUMP_HARDWARE_INITIALISATION=1 - don't create lines like digitalWrite(D1,1) for dump()/save() to recreate hardware state
• USE_TAB_COMPLETE=0 - Don't include tab completion (default is yes unless SAVE_ON_FLASH is defined)
• USE_DEBUGGER=0 - Don't include the debugger (default is yes unless SAVE_ON_FLASH is defined)
• USE_NETWORK_JS=0 - Don't include JS networking lib used for handling AT commands (default is yes if networking is enabled)
• ESPR_NO_SOFTWARE_SERIAL - don't build in software serial support
• ESPR_NO_SOFTWARE_I2C - don't build in software I2C support
• ESPR_NO_BLUETOOTH_MESSAGES - don't include text versions of Bluetooth error messages (just the error number)
• ESPR_LIMIT_DATE_RANGE - limits the acceptable range for Date years (saves a few hundred bytes)
• ESPR_NO_REGEX_OPTIMISE - strips out some speed optimisations from the regex library

These are set automatically when SAVE_ON_FLASH is set (see jsutils.h)

• SAVE_ON_FLASH_MATH - Replace some Maths functions that use a bunch of Flash memory (sin/atan/atan2/fft) with slower, smaller versions
• ESPR_NO_GET_SET - No Getter/setter functionality
• ESPR_NO_OBJECT_METHODS - No methods in objects like {method() { ... }}
• ESPR_NO_PROPERTY_SHORTHAND - No property shorthand in objects like {a}
• ESPR_NO_LINE_NUMBERS - disable storing and reporting of Line Numbers. Usually these take 1 var per function, but if we're executing a function from flash we can just work it out from the file when needed
• ESPR_NO_LET_SCOPING - don't create scopes for let (treat it like var, which was the 2v13 and earlier behaviour)
• ESPR_NO_PROMISES - Don't include promise-handling functions
• ESPR_NO_PRETOKENISE - Don't include code to pretokenise functions marked with "ram" - code pretokenised in the IDE can still be executed

chip

• part - Chip part number (this is defined in the compiler - eg if the part is STM32F401CDU6, -DSTM32F401CDU6 is put on the GCC command-line)
• family - Chip family - also defined, but used in the Makefile to define what gets built
• package - Used for ST chips when working out Pin Definitions to ensure that the right pins are included
• ram on chip in KB
• flash on chip in KB
• speed of chip in MHz
• Number of usarts
• Number of spis
• Number of i2cs
• Number of adcs
• Number of dacs
• saved_code - how and where to save JS code in the chip's flash memory
  • The address of the start of the code in flash memory. Note that on STM32 chips this is at 0x08000000 even though code executes from 0x00000000 (the two areas are mirrored but 0x00000000 is faster to execute from while 0x08000000 is what's needed for writing to flash)
  • The page_size in bytes of the first flash page containing code (not used now?)
  • The number of flash pages we're using for code (not used now?)
  • flash_available in KB, used as a sanity check by scripts/check_size.sh after the build completes
• place_text_section what address to start the program code at - This is 0 if not specified, but is useful in devices like the Pico where varying page sizes mean that the program code may be located after the saved program code (and/or bootloader)

board

Names of rows of pins - used solely by scripts/build_board_docs.py to generate the nice HTML description.

There can also be a _css element which contains raw Cascading Style Sheets, which contain carefully tweaked positions of the rows of pins defined in board, to align with the background image in boards/img

devices

This is a list of built-in stuff on the board that is made accessible to Espruino. It's:

• Parsed and turned into a series of #defines in gen/platform_config.h by this code.
• Used for the magenta tags shown near pins in the board's HTML file - that warn users that the pin might be used for some other function.

Stuff you can use is LED1-LED8, BTN1-BTN4, USB, LCD (for boards with FSMC LCDs built in), SD (SD card), JTAG (when JTAG pins are defined, but we need to make sure we leave them alone when the board resets). You can also define your own.

Pin Definitions

The pin definitions are created by scripts/build_pininfo.py and are stored in gen/jspininfo.c. The script calls get_pins in boards/$BOARD.py and a structure of the following form is returned:

[
   { "name":"PD12", "sortingname":"D12", "port":"D", "num":"20", "functions":{}, "csv":{} },
   { "name":"PD13", "sortingname":"D13", "port":"D", "num":"23", "functions":{ "SPI1_SCK":0 }, "csv":{} },
   ...
]

get_pins can define the raw array (like boards/MICROBIT.py), or on boards with STM32 chips like the Pico some utility functions auto-generate the code from CSV files that were copied from ST's datasheets.

• name is the pin name - due to random historical reasons (from ST datasheets) it needs prefixing with P
• sortingname is the name, but padded so that when it's sorted everything appears in the right order
• port is the actual port - on ESP8266 this might not be needed and could just default to D
• num is the pin number - this doesn't have to match D - it's what is needed internally to access the hardware. For instance Olimexino has 'logical' pins that actually map all over the place.
• functions is a map of pin functions to their 'alternate functions' (an STM32 chip thing - STM32F4 chips can have different peripherals on each pin, so the alternate function is a number that you shove in that pin's register in order to connect it to that peripheral). The format, for instance I2C1_SDA is important as it's parsed later and is used to build gen/jspininfo.c. The code to parse them is here
  • You can add "NEGATED":0 to a pin's functions to make Espruino invert that pin in software (to JS 1 will output 0v, 0 will output 3.3v)
  • You can add "NO_BLOCKLY":0 to a pin's functions to stop that pin appearing in the Blockly graphical editor (if the pin is 'internal')
  • You can add "3.3":0 to a pin's functions to ensure that in a board's pinout file (eg https://www.espruino.com/Original#pinout) a pin is shown as only being capable of 3.3v maximum IO
• csv isn't needed, but when using data grabbed from csv files from ST's datasheets like this it contains the raw data for debugging)

Wrapper Files

Wrapper files are normal C source files that contain functions that are exposed to JavaScript code in the interpreter. They're prefixed jswrap_.

See libs/README.md for a short tutorial on how to add your wrapper files.

Each function to be exposed has a specially formatted comment above it containing JSON. The format of this JSON is specified in this file

An example of a wrapper file might be:

#include "jswrap_hello.h"  // We need the declaration of the jswrap_hello_world function
#include "jsinteractive.h" // Pull in the jsiConsolePrint function

// Let's define the JavaScript class that will contain our `world()` method. We'll call it `Hello`
/*JSON{
  "type" : "class",
  "class" : "Hello"
}
Some info about this class
*/

// Now, we define the `jswrap_hello_world` to be a `staticmethod` on the `Hello` class
/*JSON{
  "type" : "staticmethod",
  "class" : "Hello",
  "name" : "world",
  "generate" : "jswrap_hello_world"
}
We can write simple markdown-formatted comments in here. These
then get scanned and turned into the Espruino [Reference](http://www.espruino.com/Reference)

 * Bulletted
 * List
*/
void jswrap_hello_world() {
    jsiConsolePrint("Hello World!\r\n");
}

In the build process, scripts/build_jswrapper.py parses all these files, pulls out the headers, and generates gen/jswrapper.c - a hard-coded symbol table that resides on flash memory. It's consulted whenever

scripts/build_jswrapper.py also needs to be aware of what definitions were passed to the compiler - for instance if SAVE_ON_FLASH is defined, several non-vital functions will not be compiled in (so should not appear in the symbol table).

The wrapper files are also parsed by:

• scripts/build_docs.py, which builds the HTML file used for the Espruino Reference.
• scripts/build_tern_json.js, which generates a JSON description of all functions (along with their documentation) which is used for code completion in the Web IDE and to parse all code examples in order to produce the 'Examples' links in the Reference.