gdbstub and gbd segfaults on different instructions in user space emulation

["Libo Zhou" <zhlb29@foxmail.com>]

Hi all,

I am encountering segmentation fault while porting my custom ISA to QEMU. My custom ISA is VERY VERY simple, it only changes the [31:26] opcode field of LW and SW instructions. The link has my very simple implementation: https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg06976.html

Below is the objdump of the main part of my simple ELF. The dots are the automatically generated nop's by the compiler.
00400090 <main>:
  400090:	23bdffe0    addi    r29,r29,-32
  400094:	7fbe001c    sw    r30,28(r29.
	...
  4000a0:	03a0f021    addu    r30,r29,r0
  4000a4:	20020001    li    r2,1    # int a = 1;
  4000a8:	7fc20010    sw    r2,16(r30)
	...
  4000b4:	20020002    li    r2,2    # int b = 2;
  4000b8:	7fc2000c    sw    r2,12(r30)
	...
  4000c4:	5fc30010    lw    r3,16(r30)
  4000c8:	00000000    nop
  4000cc:	5fc2000c    lw    r2,12(r30)
	...
  4000d8:	00621020    add    r2,r3,r2    # int c = a + b;
  4000dc:	7fc20008    sw    r2,8(r30)
	...
  4000e8:	00001021    addu    r2,r0,r0
  4000ec:	03c0e821    addu    r29,r30,r0
  4000f0:	5fbe001c    lw    r30,28(r29)
  4000f4:	23bd0020    addi    r29,r29,32
  4000f8:	03e00008    jr    r31
  4000fc:	00000000    nop
	...

The code below gives me segfault:
$ ./qemu-mipsel -cpu mycpu testprogram

I have tried 2 ways of debugging it.
Firstly, I connected gdb-multiarch to gdbstub, and I single-stepped the instructions in my ELF. Immediately after the LW instruction, the segfault was thrown. I observed the memory location using 'x' command and found that at least my SW instruction was implemented correctly.
Secondly, I used gdb to directly debug QEMU. I set the breakpoint at function in translate.c:decode_opc. Pressing 'c' should have the same effect as single-stepping instruction in gdbstub. However, the segmentation fault wasn't thrown after LW. It was instead thrown after the 'nop' after 'jr r31' in the objdump.

At this point, I am really stuck. I have spent a long time on this, but I just can't figure out what is going wrong here. If anyone can help me out I would really appreciate it.

Cheers,
Libo Zhou