Trap Handler is Triggered During Normal Application Execution

[DS-567]

Hi Stephan,

I have encountered a strange issue with a particular C program that sometimes causes the trap handler to be triggered when a particular load word lw instruction is being executed. Sometimes it executes the lw instruction fine and others triggers the trap handler.

So to explain, I have made some modifications to Neorv32 to achieve fault injection into the program counter and when I run this problematic C code that is shown below, by monitoring the program counter execute_engine.pc and the csr.mcause registers on Vivado ILA, I can see the trap handler gets triggered 5 times with the same cause as 0x05 which is load bus access fault. So I assumed it was my fault injection modifications that was causing this so when testing with a completely unmodified Neorv32 core, the same 5 trap handler triggers still occur at the same instruction!

The Neorv32 version used was taken from Github on 23/09/2023 and the instruction that causes the trigger is at address 0x1D0. Do you experience the same problem when running this same C code? It seems like an address stored in a register must be computed incorrectly / getting corrupted, or I think the insertion function in the C code has a bug potentially?

This is the C code in mention. it is an insertion sorting algorithm that sorts the data array:

#include <neorv32.h>
#include <stdio.h>

// array of numbers to be sorted from smallest to largest (small > large)
int data[] = {16629, 28076, 10730, 8262, 8774, 30424, 3409, 6494, 5335, 6866, 5801, 25230, 3236, 1333, 10796, 18856, 29453, 14350, 8906, 18848, 6784, 26927, 24114, 32503, 7219};

/**********************************************************************//**
 * @name Insertion sort function declaration
 **************************************************************************/
// perform the insertion sort
void insertionSort(int arr[], int size) {
  for (int i = 1; i < size; i++) {
    int tmp = arr[i];
    int j = i - 1;

    while (tmp < arr[j] && j >= 0) {
      arr[j + 1] = arr[j];
      --j;
    }
    arr[j + 1] = tmp;
  }
}

int main() {

  // find the array's length
  int size = sizeof(data) / sizeof(data[0]);

  // call insertion sort function
  insertionSort(data, size);

  return 0;
}

Here is the assembly file (note, the issue occurs from the lw instruction at address 0x1D0 and the trap handler is at 0x13C).

main.elf:     file format elf32-littleriscv


Disassembly of section .text:

00000000 <__crt0_entry>:
   0:	30005073          	csrwi	mstatus,0

00000004 <__crt0_cpu_csr_init>:
   4:	30401073          	csrw	mie,zero
   8:	00000097          	auipc	ra,0x0
   c:	13408093          	addi	ra,ra,308 # 13c <__crt0_trap_handler>
  10:	30509073          	csrw	mtvec,ra

00000014 <__crt0_pointer_init>:
  14:	80002117          	auipc	sp,0x80002
  18:	fe810113          	addi	sp,sp,-24 # 80001ffc <__crt0_stack_begin+0x0>
  1c:	80001197          	auipc	gp,0x80001
  20:	84818193          	addi	gp,gp,-1976 # 80000864 <__crt0_stack_begin+0xffffe868>

00000024 <__crt0_reg_file_init>:
  24:	00000213          	li	tp,0
  28:	00000293          	li	t0,0
  2c:	00000313          	li	t1,0
  30:	00000393          	li	t2,0
  34:	00000413          	li	s0,0
  38:	00000493          	li	s1,0
  3c:	00000813          	li	a6,0
  40:	00000893          	li	a7,0
  44:	00000913          	li	s2,0
  48:	00000993          	li	s3,0
  4c:	00000a13          	li	s4,0
  50:	00000a93          	li	s5,0
  54:	00000b13          	li	s6,0
  58:	00000b93          	li	s7,0
  5c:	00000c13          	li	s8,0
  60:	00000c93          	li	s9,0
  64:	00000d13          	li	s10,0
  68:	00000d93          	li	s11,0
  6c:	00000e13          	li	t3,0
  70:	00000e93          	li	t4,0
  74:	00000f13          	li	t5,0
  78:	00000f93          	li	t6,0

0000007c <__crt0_copy_data>:
  7c:	20c00593          	li	a1,524
  80:	80000617          	auipc	a2,0x80000
  84:	f8060613          	addi	a2,a2,-128 # 80000000 <__crt0_stack_begin+0xffffe004>
  88:	80000697          	auipc	a3,0x80000
  8c:	fdc68693          	addi	a3,a3,-36 # 80000064 <__crt0_stack_begin+0xffffe068>
  90:	00c58e63          	beq	a1,a2,ac <__crt0_clear_bss>

00000094 <__crt0_copy_data_loop>:
  94:	00d65c63          	bge	a2,a3,ac <__crt0_clear_bss>
  98:	0005a703          	lw	a4,0(a1)
  9c:	00e62023          	sw	a4,0(a2)
  a0:	00458593          	addi	a1,a1,4
  a4:	00460613          	addi	a2,a2,4
  a8:	fedff06f          	j	94 <__crt0_copy_data_loop>

000000ac <__crt0_clear_bss>:
  ac:	80000717          	auipc	a4,0x80000
  b0:	fb870713          	addi	a4,a4,-72 # 80000064 <__crt0_stack_begin+0xffffe068>
  b4:	80000797          	auipc	a5,0x80000
  b8:	fb078793          	addi	a5,a5,-80 # 80000064 <__crt0_stack_begin+0xffffe068>

000000bc <__crt0_clear_bss_loop>:
  bc:	00f75863          	bge	a4,a5,cc <__crt0_call_constructors>
  c0:	00072023          	sw	zero,0(a4)
  c4:	00470713          	addi	a4,a4,4
  c8:	ff5ff06f          	j	bc <__crt0_clear_bss_loop>

000000cc <__crt0_call_constructors>:
  cc:	00000417          	auipc	s0,0x0
  d0:	14040413          	addi	s0,s0,320 # 20c <__etext>
  d4:	00000497          	auipc	s1,0x0
  d8:	13848493          	addi	s1,s1,312 # 20c <__etext>

000000dc <__crt0_call_constructors_loop>:
  dc:	00945a63          	bge	s0,s1,f0 <__crt0_call_constructors_loop_end>
  e0:	00042083          	lw	ra,0(s0)
  e4:	000080e7          	jalr	ra
  e8:	00440413          	addi	s0,s0,4
  ec:	ff1ff06f          	j	dc <__crt0_call_constructors_loop>

000000f0 <__crt0_call_constructors_loop_end>:
  f0:	00000513          	li	a0,0
  f4:	00000593          	li	a1,0
  f8:	090000ef          	jal	ra,188 <main>

000000fc <__crt0_main_exit>:
  fc:	30401073          	csrw	mie,zero
 100:	34051073          	csrw	mscratch,a0

00000104 <__crt0_call_destructors>:
 104:	00000417          	auipc	s0,0x0
 108:	10840413          	addi	s0,s0,264 # 20c <__etext>
 10c:	00000497          	auipc	s1,0x0
 110:	10048493          	addi	s1,s1,256 # 20c <__etext>

00000114 <__crt0_call_destructors_loop>:
 114:	00945a63          	bge	s0,s1,128 <__crt0_call_destructors_loop_end>
 118:	00042083          	lw	ra,0(s0)
 11c:	000080e7          	jalr	ra
 120:	00440413          	addi	s0,s0,4
 124:	ff1ff06f          	j	114 <__crt0_call_destructors_loop>

00000128 <__crt0_call_destructors_loop_end>:
 128:	00000093          	li	ra,0
 12c:	00008463          	beqz	ra,134 <__crt0_main_aftermath_end>
 130:	000080e7          	jalr	ra

00000134 <__crt0_main_aftermath_end>:
 134:	10500073          	wfi
 138:	ffdff06f          	j	134 <__crt0_main_aftermath_end>

0000013c <__crt0_trap_handler>:
 13c:	ff810113          	addi	sp,sp,-8
 140:	00812023          	sw	s0,0(sp)
 144:	00912223          	sw	s1,4(sp)
 148:	34202473          	csrr	s0,mcause
 14c:	02044663          	bltz	s0,178 <__crt0_trap_handler_end>
 150:	34102473          	csrr	s0,mepc
 154:	00041483          	lh	s1,0(s0)
 158:	0034f493          	andi	s1,s1,3
 15c:	00240413          	addi	s0,s0,2
 160:	34141073          	csrw	mepc,s0
 164:	00300413          	li	s0,3
 168:	00941863          	bne	s0,s1,178 <__crt0_trap_handler_end>
 16c:	34102473          	csrr	s0,mepc
 170:	00240413          	addi	s0,s0,2
 174:	34141073          	csrw	mepc,s0

00000178 <__crt0_trap_handler_end>:
 178:	00012403          	lw	s0,0(sp)
 17c:	00412483          	lw	s1,4(sp)
 180:	00810113          	addi	sp,sp,8
 184:	30200073          	mret

00000188 <main>:

  // find the array's length
  int size = sizeof(data) / sizeof(data[0]);

  // call insertion sort function
  insertionSort(data, size);
 188:	80000537          	lui	a0,0x80000
int main() {
 18c:	ff010113          	addi	sp,sp,-16
  insertionSort(data, size);
 190:	00050513          	mv	a0,a0
 194:	01900593          	li	a1,25
int main() {
 198:	00112623          	sw	ra,12(sp)
  insertionSort(data, size);
 19c:	014000ef          	jal	ra,1b0 <insertionSort>

  return 0;
}
 1a0:	00c12083          	lw	ra,12(sp)
 1a4:	00000513          	li	a0,0
 1a8:	01010113          	addi	sp,sp,16
 1ac:	00008067          	ret

000001b0 <insertionSort>:
  for (int i = 1; i < size; i++) {
 1b0:	00450693          	addi	a3,a0,4 # 80000004 <__crt0_stack_begin+0xffffe008>
 1b4:	00100713          	li	a4,1
    while (tmp < arr[j] && j >= 0) {
 1b8:	fff00313          	li	t1,-1
  for (int i = 1; i < size; i++) {
 1bc:	00b74463          	blt	a4,a1,1c4 <insertionSort+0x14>
}
 1c0:	00008067          	ret
    int tmp = arr[i];
 1c4:	0006a803          	lw	a6,0(a3)
    int j = i - 1;
 1c8:	fff70793          	addi	a5,a4,-1
    while (tmp < arr[j] && j >= 0) {
 1cc:	00068613          	mv	a2,a3
 1d0:	ffc62883          	lw	a7,-4(a2)         TRAP HANDLER SEEMS TO GET TRIGGERED WHEN THIS IS EXECUTED!
 1d4:	01185663          	bge	a6,a7,1e0 <insertionSort+0x30>
 1d8:	02679263          	bne	a5,t1,1fc <insertionSort+0x4c>
 1dc:	fff00793          	li	a5,-1
    arr[j + 1] = tmp;
 1e0:	00178793          	addi	a5,a5,1
 1e4:	00279793          	slli	a5,a5,0x2
 1e8:	00f507b3          	add	a5,a0,a5
 1ec:	0107a023          	sw	a6,0(a5)
  for (int i = 1; i < size; i++) {
 1f0:	00170713          	addi	a4,a4,1
 1f4:	00468693          	addi	a3,a3,4
 1f8:	fc5ff06f          	j	1bc <insertionSort+0xc>
      arr[j + 1] = arr[j];
 1fc:	01162023          	sw	a7,0(a2)
      --j;
 200:	fff78793          	addi	a5,a5,-1
 204:	ffc60613          	addi	a2,a2,-4
 208:	fc9ff06f          	j	1d0 <insertionSort+0x20>

Here is an image showing the first time the lw instruction is executed at 1D0. You can see mcause eventually gets updated with 0x05 and the next program counter value is 0x13C which is the trap handler base address for this application. Also note a few cycles before the trap handler is triggered, it appears to execute the lw instruction fine with no trap handler getting triggered!

[stnolting]

Hey @DS-567!

I just tested your program and I also get several load access exceptions. I think this is just a coding issue. Your insertion function seems to be accessing an element that is out of bounds.

If have changed this line

    while (tmp < arr[j] && j >= 0) {

to

    while (tmp < arr[j] && j > 0) {

(according to https://en.wikipedia.org/wiki/Insertion_sort). By this, j cannot get negative inside the while-loop and the program happily executes without any exceptions - but I did not check the actual processing result.

[DS-567]

Thanks Stephan,

That seems to be the issue and it still sorts correctly 😃.

[stnolting]

Great to hear! :)