Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG][ARL] no FW_READY at boot - ROM status 0x50000005, ROM error 0x0 #8418

Closed
kv2019i opened this issue Oct 30, 2023 · 11 comments
Closed

[BUG][ARL] no FW_READY at boot - ROM status 0x50000005, ROM error 0x0 #8418

kv2019i opened this issue Oct 30, 2023 · 11 comments
Assignees
@abonislawski @tmleman
Labels
ARL Applies to Intel Arrow Lake platform Boot Firmware boot or code signing related. bug Something isn't working as expected core count P1 Blocker bugs or important features
Milestone
v2.9

Comments

@kv2019i
Copy link
Collaborator

kv2019i commented Oct 30, 2023 •
edited
Loading

Describe the bug
FW build from latest SOF main does not boot on ARL system.

Last know good commit 98a20ce (tag: releases/mtl/v3.1.1, origin/mtl-003-drop-stable)

No FW_READY message is receive by kernel, FW state is FW_ENTERED, so ROM
has completed.

To Reproduce
Build firmware and boot system.

Reproduction Rate
100%

Expected behavior
FW should boot.

Impact
Showstopper

Environment

  1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
    • Kernel: -
    • SOF: v2.7.1, main, latest mtl-006-drop-stable
  2. Name of the topology file
    • Topology: sof-hda-generic.tplg
  3. Name of the platform(s) on which the bug is observed.
    • Platform: arl

Logs

[ 3863.060288] snd_sof_intel_hda_common:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1000]=0x1010000 successful
[ 3863.061636] snd_sof_intel_hda_common:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1d1c]=0x8151 successful
[ 3863.061957] snd_sof_intel_hda_common:hda_dsp_stream_hw_params: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x40000 successful
[ 3863.061994] snd_sof_intel_hda_common:hda_dsp_stream_hw_params: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x40000 successful
[ 3863.062005] snd_sof_intel_hda_common:hda_dsp_stream_setup_bdl: sof-audio-pci-intel-mtl 0000:80:1f.3: period_bytes:0x0
[ 3863.062007] snd_sof_intel_hda_common:hda_dsp_stream_setup_bdl: sof-audio-pci-intel-mtl 0000:80:1f.3: periods:1
[ 3863.062077] snd_sof_intel_hda_common:hda_dsp_cl_boot_firmware: sof-audio-pci-intel-mtl 0000:80:1f.3: Attempting iteration 0 of Core En/ROM load...
[ 3863.063409] snd_sof_intel_hda_common:mtl_dsp_core_power_up: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x178d04]=0x2000101 successful
[ 3863.063415] snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:80:1f.3: Primary core power up successful
[ 3863.063428] snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x73214]=0x80000000 successful
[ 3863.063444] snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1800]=0x41 successful
[ 3863.063457] snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1140]=0x1 successful
[ 3863.348296] snd_sof_intel_hda_common:hda_cl_copy_fw: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x180000]=0x5 successful
[ 3863.348313] snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x140000 successful
[ 3863.348321] snd_sof_intel_hda_common:hda_dsp_cl_boot_firmware: sof-audio-pci-intel-mtl 0000:80:1f.3: Firmware download successful, booting...
[ 3865.359486] sof-audio-pci-intel-mtl 0000:80:1f.3: ------------[ DSP dump start ]------------
[ 3865.368023] sof-audio-pci-intel-mtl 0000:80:1f.3: Firmware boot failure due to timeout
[ 3865.375997] sof-audio-pci-intel-mtl 0000:80:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 3865.384202] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM status: 0x5, ROM error: 0x0
[ 3865.391742] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[ 3865.401054] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM feature bit enabled
[ 3865.407895] sof-audio-pci-intel-mtl 0000:80:1f.3: ------------[ DSP dump end ]------------
[ 3865.416219] sof-audio-pci-intel-mtl 0000:80:1f.3: error: failed to boot DSP firmware -5
[ 3865.424834] snd_sof_intel_hda_common:mtl_enable_sdw_irq: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x114c]=0x0 successful
[ 3865.426263] snd_sof_intel_hda_common:mtl_dsp_core_power_down: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x178d04]=0x2000000 successful
[ 3865.427448] snd_sof_intel_hda_common:mtl_power_down_dsp: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1000]=0x0 successful
[ 3865.428042] sof-audio-pci-intel-mtl 0000:80:1f.3: error: sof_probe_work failed err: -5
@kv2019i kv2019i added bug Something isn't working as expected P1 Blocker bugs or important features ARL Applies to Intel Arrow Lake platform labels Oct 30, 2023
@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 30, 2023

FYI @arunt1 @mengdonglin @ujfalusi

@lgirdwood
Copy link
Member

Can we extend the boot timeout and check the status, i.e. are we running on WOVCLK instead of HPCLK
adding @abonislawski @mwasko

@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 30, 2023

First bisect results:

  • v2.4 still work
  • v2.5 fails to boot

@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 30, 2023

Closing as invalid, incorrect configuration targetted.

@kv2019i kv2019i closed this as completed Oct 30, 2023
@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 31, 2023

Reopening, the Zephyr soc code has support for dynamic lookup of the core count, so the mtl built FW is expected to boot.

@kv2019i kv2019i reopened this Oct 31, 2023
@mengdonglin mengdonglin added the Boot Firmware boot or code signing related. label Oct 31, 2023
@mengdonglin mengdonglin changed the title [BUG][ARL] no FW_READY at boot [BUG][ARL] no FW_READY at boot - ROM status 0x50000005, ROM error 0x0 Oct 31, 2023
@abonislawski
Copy link
Member

Worth to check WOVCRO commits on 006 zephyr

@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 31, 2023

Base Zephyr smoke test boot up with SOF main, so this is potentially something on the SOF side. Still trying to get logs out or a proper panic dump.

@kv2019i
Copy link
Collaborator Author

kv2019i commented Oct 31, 2023

@abonislawski and @tmleman root-caused, addition of "watchdog2: watchdog@78500" is the thing that started causing this problem in this configuration.

@abonislawski
Copy link
Member

abonislawski commented Oct 31, 2023 •
edited
Loading

Great, so we just need to limit core count in watchdog

EDIT: verified & working with a test patch:
abonislawski/zephyr@8fe13ad
zephyrproject-rtos/zephyr#64762
zephyrproject-rtos/zephyr#64773

@lgirdwood lgirdwood added this to the v2.9 milestone Nov 10, 2023
@lgirdwood
Copy link
Member

Tagged to v2.9 since not needed in v2.8

kv2019i added a commit to kv2019i/sof that referenced this issue Nov 16, 2023
@kv2019i
west.yml: update Zephyr to 063ce9caf54f
ddcedf6 
Update Zephyr to bring in following Zephyr commit:

155f866ecc2c dts: intel_adsp: ace remove dw watchdog

Link: thesofproject#8418
Signed-off-by: Kai Vehmanen <[email protected]>
@kv2019i kv2019i mentioned this issue Nov 16, 2023
Merged
kv2019i added a commit that referenced this issue Nov 17, 2023
@kv2019i
west.yml: update Zephyr to 063ce9caf54f
8cdae13 
Update Zephyr to bring in following Zephyr commit:

155f866ecc2c dts: intel_adsp: ace remove dw watchdog

Link: #8418
Signed-off-by: Kai Vehmanen <[email protected]>
@keqiaozhang
Copy link
Collaborator

This issue can be closed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abonislawski abonislawski

@tmleman tmleman

Labels
ARL Applies to Intel Arrow Lake platform Boot Firmware boot or code signing related. bug Something isn't working as expected core count P1 Blocker bugs or important features
Projects
None yet
Milestone
v2.9
Development

No branches or pull requests
6 participants
@mengdonglin @lgirdwood @keqiaozhang @abonislawski @kv2019i @tmleman