Occasional test timeouts

[XavierAubert]

Occasional test timeouts

In very rare cases (occurrence of about 1 in 10,000 tests), a combination of different factors causes some tests to take longer than the timeout value specified in the regression file.
Among these factors, there are :

• Interrupt generation delay
• Debug request generation delay
• Number of above events
• Code size
• ...

However, to avoid extending simulation times, it was decided not to increase this timeout value and keep it as is.

Type

• False negative
• core-v-verif environment / tool limitation
• Random issue

Steps to Reproduce

Environment setup:

• Operating System : CentOS 7
• Simulator : Siemens Questa Sim-64 vsim 2023.2_1
• Toolchain: corev-openhw-gcc-centos7-20240530
• ISS : imperas_idv/eng.20240530.0
• core-v-verif : https://github.com/openhwgroup/core-v-verif/tree/a3a121f6424ff972214d245f5a4629e522741fe6
• CV32E40P RTL: https://github.com/openhwgroup/cv32e40p/tree/8f24b1d70dac770553bc8edfd3cef320b729f8eb

to reproduce the issue:

git clone -b cv32e40p/dev --single-branch https://github.com/openhwgroup/core-v-verif core-v-verif
cd core-v-verif
git checkout b9283956e4adfb2b5d30b35ad829789a615723a6
git clone -b dev --single-branch https://github.com/openhwgroup/cv32e40p core-v-cores/cv32e40p
cd core-v-cores/cv32e40p
git checkout bdd52534e28afa4aab4d549f26286355f8b075a8
cd ../../cv32e40p/sim/uvmt

make gen_corev-dv test COREV=YES TEST=corev_rand_interrupt_wfi_mem_stress CFG_PLUSARGS=+UVM_TIMEOUT=50000000 CV_CORE=cv32e40p CFG=pulp_fpu_1cyclat TEST_CFG_FILE=floating_pt_instr_en,disable_all_trn_logs SIMULATOR=vsim USE_ISS=yes COV=YES RUN_INDEX=1377880545 GEN_START_INDEX=1377880545 SEED=1377880545

to reproduce a case that solves the issue (removing optional argument), from cv32e40p/sim/uvmt use this make command instead:

make gen_corev-dv test COREV=YES TEST=corev_rand_interrupt_wfi_mem_stress CV_CORE=cv32e40p CFG=pulp_fpu_1cyclat TEST_CFG_FILE=floating_pt_instr_en,disable_all_trn_logs SIMULATOR=vsim USE_ISS=yes COV=YES RUN_INDEX=1377880545 GEN_START_INDEX=1377880545 SEED=1377880545

The last command runs the test until its end without timeout, at around 70ms simulation time.

Log example:

# UVM_INFO @ 16.800 ns : uvmt_cv32e40p_base_test.sv(309) uvm_test_top [BASE TEST] set load_instr_mem
# UVM_INFO @ 114.300 ns : uvmt_cv32e40p_firmware_test.sv(156) uvm_test_top [TEST] Started RUN
# UVM_FATAL @ 50000000.000 ns : uvm_phase.svh(1506) reporter [PH_TIMEOUT] Explicit timeout of 50000000.000 ns hit, indicating a probable testbench issue
# UVM_INFO @ 50000000.000 ns : uvm_report_server.svh(847) reporter [UVM/REPORT/SERVER]
# --- UVM Report Summary ---

Additional context

In the above steps, ISS and coverage are enabled, but the issue can be reprocued without it since it is only pure simulation time issue.
For reference, the following files are attached:

• test.log is the log file given by the command above
• test-ok-without-timout.log is the log file given by the command above with the optional argument CFG_PLUSARGS=+UVM_TIMEOUT=50000000 removed and without coverage to speed-up running time.

[MikeOpenHWGroup]

Hi @XavierAubert, thanks for this issue, your careful analysis and especially for your example (steps to reproduce). I agree with your strategy to not increase this timeout value and keep it as is.

There is a mechanism in CORE-V-VERIF that allows individual uvm_components to extend the simulation timeout. It was noticed early on that this mechanism replaced "occasional unexpected test timeouts" with "occasional unexpected infinite runtime".