Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix race in tinygo test #3934

Merged
merged 2 commits into from
Oct 4, 2023
Merged

Fix race in tinygo test #3934

merged 2 commits into from
Oct 4, 2023

Conversation

aykevl
Copy link
Member

@aykevl aykevl commented Sep 28, 2023

This happens with tinygo test for example when testing multiple packages at the same time. I found this because the compiler crashed in make tinygo-test-fast:

fatal error: concurrent map writes
fatal error: concurrent map read and map write

goroutine 15 [running]:
github.com/tinygo-org/tinygo/builder.Build({0x40002d0be0, 0xa}, {0x0, 0x0}, {0x4000398048, 0x14}, 0x40003b0000)
        /home/ayke/src/tinygo/tinygo/builder/build.go:131 +0x388
main.buildAndRun({0x40002d0be0, 0xa}, 0x40003b0000, {0x8bc178, 0x40003a4090}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...)
        /home/ayke/src/tinygo/tinygo/main.go:856 +0x45c
main.Test({0x40002d0be0, 0xa}, {0x8bc118, 0x40000b3a08}, {0x0?, 0x0?}, 0x40001e6000, {0x0, 0x0})
        /home/ayke/src/tinygo/tinygo/main.go:270 +0x758
main.main.func3()
        /home/ayke/src/tinygo/tinygo/main.go:1718 +0xe4
created by main.main in goroutine 1
        /home/ayke/src/tinygo/tinygo/main.go:1712 +0x34ec

My solution is essentially to copy the map over instead of modifying it directly.

I've also moved the code up a little, because I think that's a more sensible place (out of the way of the whole package compile logic).

The first commit refactors the goenv.Version logic, which I found while fixing this issue and is a conflicting change (hence why it's in the same PR).

@github-actions
Copy link

github-actions bot commented Sep 28, 2023 •
edited
Loading

Size difference with the dev branch:

Binary size difference 
 before   after   diff
  60960   60960      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adt7410/main.go
   9736    9736      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adxl345/main.go
  13268   13268      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pybadge ./examples/amg88xx
   8724    8724      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/main.go
  11612   11612      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/apds9960/proximity/main.go
   9784    9784      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/itsybitsy-m0/main.go
   8168    8168      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/at24cx/main.go
   8344    8344      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bh1750/main.go
   7632    7632      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/blinkm/main.go
  70492   70492      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pinetime     ./examples/bma42x/main.go
  63444   63444      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmi160/main.go
  27836   27836      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp180/main.go
  63544   63544      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp280/main.go
  12352   12352      0   0.00%  tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/bmp388/main.go
   8128    8128      0   0.00%  tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/sram/main.go
  22100   22100      0   0.00%  tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/time/main.go
  69332   69332      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/ds3231/main.go
   4704    4704      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/easystepper/main.go
  24932   24932      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/espconsole/main.go
  25080   25080      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/esphub/main.go
  24932   24932      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/espstation/main.go
  68844   68844      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/flash/console/spi
  64916   64916      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/flash/console/qspi
   7040    7040      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/gc9a01/main.go
  68148   68148      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/gps/i2c/main.go
  68544   68544      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/gps/uart/main.go
   8372    8372      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/hcsr04/main.go
   5612    5612      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hd44780/customchar/main.go
   5656    5656      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hd44780/text/main.go
  10564   10564      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/hd44780i2c/main.go
  14516   14516      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/hts221/main.go
  16940   16940      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hub75/main.go
  10052   10052      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/basic
  10832   10832      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/ili9341/basic
  29052   29052      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/pyportal_boing
  10080   10080      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/scroll
  10908   10908      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/ili9341/scroll
 263564  263564      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/slideshow
  12052   12052      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/lis3dh/main.go
  13912   13912      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/lps22hb/main.go
  26124   26124      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/lsm303agr/main.go
  12520   12520      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/lsm6ds3/main.go
  10996   10996      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mag3110/main.go
  10176   10176      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp23017/main.go
  10604   10604      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp23017-multiple/main.go
   9816    9816      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp3008/main.go
  66920   66920      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp2515/main.go
  22984   22984      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/microbitmatrix/main.go
  22936   22936      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit-v2 ./examples/microbitmatrix/main.go
   8444    8444      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mma8653/main.go
   8356    8356      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mpu6050/main.go
  75196   75196      0   0.00%  tinygo build -size short -o ./build/test.hex -target=p1am-100 ./examples/p1am/main.go
  12148   12148      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pico ./examples/pca9685/main.go
   6080    6080      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/pcd8544/setbuffer/main.go
   5100    5100      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/pcd8544/setpixel/main.go
   2681    2681      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino ./examples/servo
   7944    7944      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pybadge ./examples/shifter/main.go
  56608   56608      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/sht3x/main.go
  56664   56664      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/sht4x/main.go
  56580   56580      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/shtc3/main.go
   6456    6456      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1306/i2c_128x32/main.go
   5960    5960      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1306/spi_128x64/main.go
   5676    5676      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1331/main.go
   6376    6376      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/st7735/main.go
   6232    6232      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/st7789/main.go
  17156   17156      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/thermistor/main.go
  10312   10312      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-bluefruit ./examples/tone
  10096   10096      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/tm1637/main.go
   9404    9404      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/touch/resistive/fourwire/main.go
  12468   12468      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/touch/resistive/pyportal_touchpaint/main.go
  15804   15804      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/vl53l1x/main.go
  13848   13848      0   0.00%  tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/vl6180x/main.go
   6452    6452      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd2in13/main.go
   6004    6004      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd2in13x/main.go
   6260    6260      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd4in2/main.go
 137504  137504      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/wifinina/ntpclient/main.go
 137512  137512      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/wifinina/udpstation/main.go
 137496  137496      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/wifinina/tcpclient/main.go
 137792  137792      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/wifinina/webclient/main.go
   6924    6924      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/ws2812
   5228    5228      0   0.00%  tinygo build -size short -o ./build/test.bin -target=m5stamp-c3          ./examples/ws2812
  61872   61872      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-nrf52840 ./examples/is31fl3731/main.go
   1549    1549      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino   ./examples/ws2812
    880     880      0   0.00%  tinygo build -size short -o ./build/test.hex -target=digispark ./examples/ws2812
  32248   32248      0   0.00%  tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/bme280/main.go
  16612   16612      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/microphone/main.go
  11268   11268      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/buzzer/main.go
  12976   12976      0   0.00%  tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/veml6070/main.go
   6800    6800      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l293x/simple/main.go
   8720    8720      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l293x/speed/main.go
   6764    6764      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l9110x/simple/main.go
   9336    9336      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l9110x/speed/main.go
   7304    7304      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nucleo-f103rb ./examples/shiftregister/main.go
   6932    6932      0   0.00%  tinygo build -size short -o ./build/test.hex -target=hifive1b ./examples/ssd1351/main.go
  13172   13172      0   0.00%  tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/lis2mdl/main.go
   8428    8428      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/max72xx/main.go
  77152   77152      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/dht/main.go
  36608   36608      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/pcf8523/
  71136   71136      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/alarm/
   7400    7400      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/clkout/
  70768   70768      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/time/
  71056   71056      0   0.00%  tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/timer/
  12076   12076      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pico ./examples/qmi8658c/main.go
   8996    8996      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/ina260/main.go
   9280    9280      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nucleo-l432kc ./examples/aht20/main.go
  72288   72288      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m4 ./examples/sdcard/console/
  82580   82580      0   0.00%  tinygo build -size short -o ./build/test.hex -target=wioterminal ./examples/rtl8720dn/webclient/
  71996   71996      0   0.00%  tinygo build -size short -o ./build/test.hex -target=wioterminal ./examples/rtl8720dn/webserver/
  98644   98644      0   0.00%  tinygo build -size short -o ./build/test.hex -target=wioterminal ./examples/rtl8720dn/mqttsub/
  60756   60756      0   0.00%  tinygo build -size short -o ./build/test.hex -target=feather-m4 ./examples/i2csoft/adt7410/
  10152   10152      0   0.00%  tinygo build -size short -o ./build/test.elf -target=wioterminal ./examples/axp192/m5stack-core2-blinky/
   8916    8916      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/xpt2046/main.go
  14564   14564      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nucleo-wl55jc ./examples/sx126x/lora_rxtx/
  26260   26260      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/ssd1289/main.go
  11188   11188      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pico ./examples/irremote/main.go
  11204   11204      0   0.00%  tinygo build -size short -o ./build/test.hex -target=badger2040 ./examples/uc8151/main.go
  10312   10312      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/scd4x/main.go
   8692    8692      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=circuitplay-express ./examples/makeybutton/main.go
   9576    9576      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ds18b20/main.go
  81312   81312      0   0.00%  tinygo build -size short -o ./build/test.hex -target=nucleo-wl55jc ./examples/lora/lorawan/atcmd/
  15732   15732      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/as560x/main.go
   9800    9800      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/mpu6886/main.go
   7840    7840      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ttp229/main.go
  66056   66056      0   0.00%  tinygo build -size short -o ./build/test.hex -target=pico ./examples/ndir/main_ndir.go
  61368   61368      0   0.00%  tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ndir/main_ndir.go
  64616   64616      0   0.00%  tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ndir/main_ndir.go
   9216    9216      0   0.00%  tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/mpu9150/main.go
3826946 3826946      0   0.00%   sum

@deadprogram
Copy link
Member

@aykevl please rebase against latest dev now. Thanks.

@aykevl
all: refactor goenv.Version to add the git sha1 if needed
8e103a9 
Previously all (except one!) usage of goenv.Version manually added the
git sha1 hash, leading to duplicate code. I've moved this to do it all
in one place, to avoid this duplication.
@aykevl
build: avoid sharing GlobalValues between build instances
b544a5b 
This happens with `tinygo test` for example when testing multiple
packages at the same time. I found this because the compiler crashed in
`make tinygo-test-fast`:

    fatal error: concurrent map writes
    fatal error: concurrent map read and map write

    goroutine 15 [running]:
    github.com/tinygo-org/tinygo/builder.Build({0x40002d0be0, 0xa}, {0x0, 0x0}, {0x4000398048, 0x14}, 0x40003b0000)
            /home/ayke/src/tinygo/tinygo/builder/build.go:131 +0x388
    main.buildAndRun({0x40002d0be0, 0xa}, 0x40003b0000, {0x8bc178, 0x40003a4090}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...)
            /home/ayke/src/tinygo/tinygo/main.go:856 +0x45c
    main.Test({0x40002d0be0, 0xa}, {0x8bc118, 0x40000b3a08}, {0x0?, 0x0?}, 0x40001e6000, {0x0, 0x0})
            /home/ayke/src/tinygo/tinygo/main.go:270 +0x758
    main.main.func3()
            /home/ayke/src/tinygo/tinygo/main.go:1718 +0xe4
    created by main.main in goroutine 1
            /home/ayke/src/tinygo/tinygo/main.go:1712 +0x34ec

My solution is essentially to copy the map over instead of modifying it
directly.

I've also moved the code up a little, because I think that's a more
sensible place (out of the way of the whole package compile logic).
@aykevl
Copy link
Member Author

aykevl commented Oct 4, 2023

Rebased!

@deadprogram
Copy link
Member

Thanks @aykevl now merging.

@deadprogram deadprogram merged commit 8cbfbca into dev Oct 4, 2023
22 checks passed
@deadprogram deadprogram deleted the fix-GlobalValues-race branch October 4, 2023 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@aykevl @deadprogram