AI Deck: Function Freezes (Camera + HyperFlash + HyperRAM)

[whoenig]

I am trying to achieve something "simple" for the AI deck and run into very strange issues for a week now.

Goal: Collect some frames and store them on-board (first in HyperRAM, once that is full in HyperFlash). Later, download all the frames from flash over wifi to the computer.

Steps: I changed the aideck-gap8-example wifi-img-streamer, to initialize and use HyperRAM and HyperFlash.

Observations:

• Flash and RAM initialize fine. However, the pi_ram_write, pi_flash_erase, orpi_flash_program sometimes just hang indefinitely (while the rest of the chip seems to be operational, i.e., the LED controlled by the hb_task keeps blinking.
• The hanging mostly occurs when trying to write a captured frame to RAM (or flash). This always happens already on the first frame.
• The official test_hyper_ram_flash example works just fine. The logic for initializing/using HyperRAM/Flash is taken from there.
• Re-arranging code changes the place where the hang occurs. The same holds true for deleting or adding dead code. Hanging is deterministic, even across boards (so this indicates that the slightly different binaries cause some sort of issue).
• The issue stays the same with the latest SDK, as well as the latest docker image from bitcraze.

Debugging is like hunting a ghost, since slight code changes change the outcome. This typically indicates i) HW issues, ii) compiler/SDK issues, iii) memory issues. For ii) I tried the latest SDK. For iii) I tried increasing the FreeRTOS stack sizes as well as just relying on pmsis functions. For allocations I use either global static variables or pmsis_l2_malloc allocated memory. I also played with forced alignment (both static and malloc'ed variables) without a noticeable change.

Any ideas on what might cause this or debugging strategies would be appreciated! I tried to come up with a minimum example that shows some sort of deterministic failure, but as it is with these kinds of issues, the minimal examples (like the test_hyper_ram_flash) usually work.

[knmcguire]

Hi Wolfgang!

Sorry for the delay but we have our quartely meeting this week. I'll try to point @evoggy to this as he is back but probably he won't be able to give you an indepth answer yet until friday and next week.

[whoenig]

Thanks for the heads-up!

[evoggy]

Hi!

I haven't tried writing except for when the bootloader writes to flash, but there's a few things that come to mind. Some of it is basic stuff, but it's always good to double check.

1. Have you tried to lower the FC frequency? Are you running it at 250 MHz? If so can you try it at something lower, like 50 MHz?
2. Have you tried writing data without capturing it from the camera? Just using the functions to write some patterns you can read back from that task.
3. When writing to the HyperFlash, where are you writing the data? Is it across page boundries? Is the page erased first?
4. Did you check the linker file to make sure you're in L2 which is mapped where you're trying to write?

Not sure if I understood correctly, but in your application does it ever work calling the write functions (RAM/FLASH) or do they always hang?

[whoenig]

1. I tried removing the frequency and voltage increase in the example without changes. Is the default 50 MHz, or do I need a special call to lower the frequency?
2. I "ported" the official hyperflash_hyperram example to the ai-deck-examples repo and this one works well. In my example, it sometimes, but not always, helps to remove the camera entirely. The main difference I see is the use of FreeRTOS (the official test just uses the pmsis functions). I haven't yet tried to write an example that simply doesn't use any FreeRTOS. Do you think it is likely that FreeRTOS causes issues?
3. I first erase several sectors (starting from sector10, to not mess with the bootloader/binary). Then I write using program. Each time, I write a frame, so this is potentially crossing sector boundaries (but not during the first call, which often fails!). I also played with the alignment, and tried writing data that is multiples of 16 bytes. Which flash sectors are actually occupied by the bootloader and binary?
4. Nope - could you elaborate a bit / do you have an example? I allocate memory using pmsis_l2_malloc. From that memory I write to either flash or RAM. The RAM address I get from pi_ram_alloc, the flash address is just memory that I erased before (sectors 10 and up).

In my application things work just fine if I remove random lines of code, or sometimes rearrange things (for example, it sometimes seems to matter in which order I initialize RAM, Flash, and camera). So I don't think there is a hardware problem per se, it seems to be more that I somehow cause undefined behavior either on the compiler or on the hardware side. By "sometimes" I mean that it seems to depend on what other code is enabled.

Currently, my "debugging" strategy is to use printf (over UART), and changing the code to see if I can find a minimal example that reproduces the issue. I am not sure if there is a better way (simulator, "valgrind" for gap8) or similar?

[evoggy]

1. No, the default should be 50MHz with the correct voltage
2. Not sure how much FreeRTOS is being used so maybe there's bugs lurking here, but I don't know
3. Each sector is 0x40000 bytes large. The bootloader is on the first one, the binary is on the second one and forward and lastly the filesystem is on the next boundary and after (I think). Sounds like 10+ should be ok. If I remember correctly data needs to be aligned on 8 bytes otherwise it will freeze, but I'm not sure if the data needs to be padded to be a multiple of 8 bytes. This could be a potential issue, but should be the same always if you write the raw images.
4. Yes, this sounds fine. I mixed it up with the bootloader where not all the L1/L2 is mapped in. But it's all mapped in for the application.

The way I've been working is with printouts, I'm not sure if debugging works and I haven't tested the simulator.

I recognize the kind of behavior your describing. Early on with the AI-deck I had similar issues. I'm not 100% sure of the root cause, but from what I remember I had this when running from RAM with JTAG.