ESP32S3: Zephyr freezes during OTA Update over UDP and BLE

[epc-ake]

Discussed in #76302

^{Originally posted by epc-ake July 25, 2024}
Has anyone managed to use mcumgr over udp on a esp32s3?

I'm developing on an esp32s3_devkitm and want to enable OTA firmware updates using mcumgr and mcuboot over a UDP interface. To experiment with this, I modified the prj.conf file of the samples/net/wifi example to enable mcuboot and mcumgr. See the attached file for details. prj.conf

After flashing the firmware and connecting to a Wi-Fi network, I can retrieve image information using the go-app and AuTerm. For example, with the go-app:

mcumgr --conntype udp --connstring=[x.x.0.60]:1337 image list
Images:
 image=0 slot=0
    version: 0.0.0
    bootable: true
    flags: active confirmed
    hash: 60e5eb52f59451a3db2ec9e978b13c0c8485577dd6787684e216069341bdf80b
Split status: N/A (0)

However, when I try to upload an image, the firmware becomes unresponsive and freezes:

./mcumgr --conntype udp --connstring=[x.x.0.60]:1337 image upload zephyr.signed.bin
# starts freezing...

Output zephyr:

*** Booting Zephyr OS build 065fa94c79e5 ***
[00:00:00.241,000] <inf> smp_udp: Started (IPv4)
uart:~$ wifi connect -s ***** -p ***** -k 1
Connection requested
Connected
# requesting image info
[00:00:17.353,000] <inf> net_dhcpv4: Received: x.x.0.60
[00:00:24.301,000] <dbg> mcumgr_img_grp: img_mgmt_active_slot: (0) => 0
[00:00:24.302,000] <inf> mcuboot_util: Image index: 0, Swap type: none
[00:00:24.302,000] <dbg> mcumgr_img_grp: img_mgmt_get_next_boot_slot: (0, *) => slot = 0, type = 0
[00:00:24.302,000] <dbg> mcumgr_img_grp: img_mgmt_active_slot: (0) => 0
uart:~$  # doing image update -> freezing...

I ran the debugger in parallel, and when I interrupted GDB (Ctrl+C) after the firmware started freezing, it pointed to _DoubleExceptionVector.

Output GDB:

Info : [esp32s3.cpu0] Target halted, PC=0x403743C0, debug_reason=00000000
[esp32s3.cpu0] Target halted, PC=0x403743C0, debug_reason=00000000
Info : Set GDB target to 'esp32s3.cpu0'
Set GDB target to 'esp32s3.cpu0'
Info : [esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000
[esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000

Program received signal SIGINT, Interrupt.
_DoubleExceptionVector () at /zephyrproject/zephyr-epc/arch/xtensa/core/xtensa_asm2_util.S:525

Uploading over serial works without any issues, so it seems to be specifically related to the UDP interface.

Does anyone have any ideas on how to debug this?

[github-actions]

Hi @epc-ake! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

[epc-ake]

if I set CONFIG_IMG_ERASE_PROGRESSIVELY=y it uploads at least some data before freezing again.

[epc-ake]

It also freezes during BLE update

[epc-ake]

Same issue over serial as well.
However, it does manage to upload part of the firmware.

[epc-ake]

@sylvioalves can you give us an update on this? In our last conversation on discord you mentioned that you've evaluated a fix for this.

[LeoBRIANDSmile]

Hi, the CPU freezes during which step of OTA update (downloading firmware, erasing flash, writing flash, ...) ? Could you debug this. Because I have a similar issue on an homemade OTA update tool on esp32s3 during the flash erase step. CPU raises FATAL EXCEPTION.

[epc-ake]

I didn't work on this, so no progress here unfortunately.
I think writing to the flash causes zephyr to freeze. So it might be a problem with the flash driver.
#77452 mentions a similar/same bug.

[LeoBRIANDSmile]

It seems to be an issue of flash protection

[epc-ake]

It seems to be an issue of flash protection

This seems to be relevant to the esptool.py.
Overall I was able to partially update the image...

[rftafas]

@epc-ake can you try #78121?

[marekmatej]

@epc-ake are you, by any chance, running the second image on the APPCPU/cpu1 ?

Info : [esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000
[esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000

[epc-ake]

@epc-ake can you try #78121?

Thanks for this. I tried it and it seems to be working for at least UDP.
However, I don't have the time right now to test it in depth...
I will come back to it in 1-2 weeks.

[epc-ake]

@epc-ake are you, by any chance, running the second image on the APPCPU/cpu1 ?

Info : [esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000
[esp32s3.cpu1] Target halted, PC=0x40043A40, debug_reason=00000000

Hmm interesting. I didn't notice that.
Currently I'm only using the PROCPU.
Maybe because I interrupted GDB with (Ctrl+C) it halted on cpu1 by accident?