illegal hardware instruction during MIPS-I ELF linux user emulation

illegal hardware instruction during MIPS-I ELF linux user emulation

[Libo Zhou]

Hi all,

I have an binary file generated by a cross compiler. The 'file' command says

$ file test
test: ELF 32-bit LSB LSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, with debug_info, not stripped.

When I executed it with

$./qemu-mipsel test
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
[1] 11088 illegal hardware instruction (core dumped) ./qemu-mipsel test

However, when I use another cross compiler that generates MIPS32 rel2 binaries, it worked fine.

Can anyone tell me what's going wrong?

Cheers,
Libo Zhou

Re:illegal hardware instruction during MIPS-I ELF linux user emulation

[Libo Zhou]

Any help guys? Is there a way to look at the ELF binary code to see if the instructions have invalid encoding scheme? Since I used a custom compiler that claims to have MIPS-I instructions. I doubt it though because 'file' command gives a reasonable output saying it surely is MIPS-I.

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 13:08, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> Any help guys? Is there a way to look at the ELF binary code to see if the instructions have invalid encoding scheme? Since I used a custom compiler that claims to have MIPS-I instructions. I doubt it though because 'file' command gives a reasonable output saying it surely is MIPS-I.

'file' just looks at the header on the file to see what it
claims to be. It doesn't look through the rest of the file
to check what actual instructions the compiler emitted.
You can use 'objdump' if you want to disassemble a file.

I would start by using the QEMU gdbstub to connect a
MIPS-aware gdb. Then when the SIGILL arrives you can see
what instruction the guest program was trying to execute.

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).
 
Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box?


EDIT: Or my compiler isn't implementing MIPS-I correctly because gdb-multiarch's r3k doesn't recognize some hexes?

[-- Attachment #2: Type: text/html, Size: 1166 bytes --]

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Philippe Mathieu-Daudé]

Hi Libo,

On 9/23/19 10:15 AM, Libo Zhou wrote:
> Hi all,
> 
> I have an binary file generated by a cross compiler. The 'file' command says
> 
> $ file test
> test: ELF 32-bit LSB LSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, with debug_info, not stripped.

If you look at the mips_defs[] array in
target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:

$ git grep .insn_flags target/mips/translate_init.inc.c
translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP,
translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP | ASE_MT,
translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP | ASE_DSP_R2,
translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP
| ASE_DSP_R2 | ASE_DSP_R3 |
translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP |
ASE_DSP_R2,

So currently there is no MIPS-I only CPU.
Note that the code got written with MIPS32 in mind, and implementing
MIPS-I requires a considerable amount of change in the codebase.

> When I executed it with
> 
> $./qemu-mipsel test
> qemu: uncaught target signal 4 (Illegal instruction) - core dumped
> [1] 11088 illegal hardware instruction (core dumped) ./qemu-mipsel test

You can try ./qemu-mipsel -d in_asm,int test to display debugging
information, you might see which instruction trapped.

IMO it is likely the RFE (Return from Exception) instruction.

Regards,

Phil.

> However, when I use another cross compiler that generates MIPS32 rel2 binaries, it worked fine.
> 
> Can anyone tell me what's going wrong?
> 
> Cheers,
> Libo Zhou
> 

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> So currently there is no MIPS-I only CPU.
> Note that the code got written with MIPS32 in mind, and implementing
> MIPS-I requires a considerable amount of change in the codebase.

...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
they?

> IMO it is likely the RFE (Return from Exception) instruction.

It seems unlikely that a linux userspace binary would be trying to
execute RFE...

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Philippe Mathieu-Daudé]

On 9/23/19 4:42 PM, Peter Maydell wrote:
> On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> ...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
> they?

Some MIPS-I instructions where removed for MIPS-II (as RFE) and they are
not implemented. Also some CP0 registers are different.

>> IMO it is likely the RFE (Return from Exception) instruction.
> 
> It seems unlikely that a linux userspace binary would be trying to
> execute RFE...

Oh I thought it was system emulation, indeed it can't be RFE.

One GCC release targetting R3000 (Philips PR31700, Toshiba TX39) doesn't
emit NOP for branch's delay slot. I remember QEMU fails to run the
binaries it generates, but I don't remember how it fails.

Libo, can you provide more information about the cross-compiler you use
and the flags you use when calling it please?

Thanks,

Phil.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> If you look at the mips_defs[] array in
> target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:

> $ git grep .insn_flags target/mips/translate_init.inc.c
> translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
> translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
> translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
> translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
> translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
> translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP,
> translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
> translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_DSP_R2,
> translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
> translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
> translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
> translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
> translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP | ASE_DSP_R2 | ASE_DSP_R3 |
> translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
> translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
> translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
> translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
> translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
> translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
> translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
> translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
> translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
> translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
> translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
> translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
> translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP | ASE_DSP_R2,

> So currently there is no MIPS-I only CPU.
> Note that the code got written with MIPS32 in mind, and implementing
> MIPS-I requires a considerable amount of change in the codebase.

Hi Philippe,

I just figured out what the problem was. The custom compiler I used just modified the opcode fields of sw and lw instructions of MIPS, so QEMU didn't recognize them out of the box.
I just added the support in decode_opc function in translate.c, and I also added my own CPU model in translate_init.inc.c. However, the illegal instruction exception is still there.

I am suspecting that the way I added my own CPU model in translate_init.inc.c is wrong. Below is what I added:
...
+{
+    .name = "MyCPU",
+    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
+},
...
I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

Thanks,
Libo Zhou

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Philippe Mathieu-Daudé]

On 9/26/19 4:31 PM, Libo Zhou wrote:
>> If you look at the mips_defs[] array in
>> target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:
> 
>> $ git grep .insn_flags target/mips/translate_init.inc.c
>> translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
>> translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
>> translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
>> translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
>> translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
>> translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP,
>> translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
>> translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_DSP_R2,
>> translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
>> translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
>> translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
>> translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
>> translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP | ASE_DSP_R2 | ASE_DSP_R3 |
>> translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
>> translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
>> translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
>> translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
>> translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
>> translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
>> translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
>> translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
>> translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
>> translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
>> translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
>> translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
>> translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP | ASE_DSP_R2,
> 
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> Hi Philippe,
> 
> I just figured out what the problem was. The custom compiler I used just modified the opcode fields of sw and lw instructions of MIPS, so QEMU didn't recognize them out of the box.
> I just added the support in decode_opc function in translate.c, and I also added my own CPU model in translate_init.inc.c. However, the illegal instruction exception is still there.
> 
> I am suspecting that the way I added my own CPU model in translate_init.inc.c is wrong. Below is what I added:
> ...
> +{
> +    .name = "MyCPU",
> +    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
> +},
> ...
> I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

Something like that might be acceptable for linux-user.
You should at least set CP0_PRid/Config0/Status_mask.

Look at this patch where Aleksandar accepted the R5900 CPU:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=ed4f49ba9bb56

Can you share what is your CPU?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> > +{
> > +    .name = "MyCPU",
> > +    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
> > +},
> > ...
> > I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

> Something like that might be acceptable for linux-user.
> You should at least set CP0_PRid/Config0/Status_mask.

> Look at this patch where Aleksandar accepted the R5900 CPU:
> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=ed4f49ba9bb56

> Can you share what is your CPU?

It's working now. When I put the -cpu flag at the end of the command line, my cpu was not recognized.

Thanks,
Libo

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

Hi Philippe, Peter,

In target/mips/translate_init.inc.c you can actually see that CPU_MIPS32 ultimately contains ISA_MIPS1. It's just no CPU model explicitly uses ISA_MIPS. But I agree that if my assembly has a removed instruction then it's possible to have an invalid machine code in my program.

As for the cross-compiler, I used a tool chain containing a compiler, an assembler, and a linker. I produced the final ELF like this:

$ mipsel-linux-unknown-gcc -g -S test.c -o test.s
$ mipsel-linux-unknown-as -g test.s -o test.o
$ mipsel-linux-unknown-ld test.o -o test

For my qemu, I did:

$ ../configure --enable-debug --target-list=mipsel-linux-user
$ make -j4

Here are the contents of my source and generated assembly:

$ cat test.c
int main(void)
{
    int a = 1;
    int b = 2;
    int c = a + b;
    return 0;
}

$ cat test.s
	.section .mdebug.abi32
	.previous
	.gnu_attribute 4, 3
	.abicalls
	.section	.debug_abbrev,"",@progbits
$Ldebug_abbrev0:
	.section	.debug_info,"",@progbits
$Ldebug_info0:
	.section	.debug_line,"",@progbits
$Ldebug_line0:
	.text
$Ltext0:
	.align	2
	.globl	main
$LFB0 = .
	.file 1 "test.c"
	.loc 1 2 0
	.cfi_startproc
	.set	nomips16
	.ent	main
	.type	main, @function
main:
	.frame	$fp,32,$31		# vars= 16, regs= 1/0, args= 0, gp= 8
	.mask	0x40000000,-4
	.fmask	0x00000000,0
	.set	noreorder
	.set	nomacro
	
	addi	$sp,$sp,-32
	.cfi_def_cfa_offset 32
	sw	$fp,28($sp)
	nop
	nop
	move	$fp,$sp
	.cfi_offset 30, -4
	.cfi_def_cfa_register 30
	.loc 1 3 0
	li	$2,1			# 0x1
	sw	$2,16($fp)
	nop
	nop
	.loc 1 4 0
	li	$2,2			# 0x2
	sw	$2,12($fp)
	nop
	nop
	.loc 1 5 0
	lw	$3,16($fp)
	lw	$2,12($fp)
	nop
	nop
	add	$2,$3,$2
	sw	$2,8($fp)
	nop
	nop
	.loc 1 6 0
	move	$2,$0
	.loc 1 7 0
	move	$sp,$fp
	lw	$fp,28($sp)
	addi	$sp,$sp,32
	j	$31
	nop
	nop
	nop
	nop
	nop

	.set	macro
	

	.end	main
	.cfi_endproc
$LFE0:
	.size	main, .-main
$Letext0:
	.section	.debug_loc,"",@progbits
$Ldebug_loc0:
$LLST0:
	.4byte	$LFB0-$Ltext0
	.4byte	$LFE0-$Ltext0
	.2byte	0x2
	.byte	0x8e
	.sleb128 32
	.4byte	0x0
	.4byte	0x0
	.section	.debug_info
	.4byte	0x6b
	.2byte	0x2
	.4byte	$Ldebug_abbrev0
	.byte	0x4
	.uleb128 0x1
	.4byte	$LASF0
	.byte	0x1
	.4byte	$LASF1
	.4byte	$LASF2
	.4byte	$Ltext0
	.4byte	$Letext0
	.4byte	$Ldebug_line0
	.uleb128 0x2
	.byte	0x1
	.4byte	$LASF3
	.byte	0x1
	.byte	0x1
	.byte	0x1
	.4byte	0x67
	.4byte	$LFB0
	.4byte	$LFE0
	.4byte	$LLST0
	.4byte	0x67
	.uleb128 0x3
	.ascii	"a\000"
	.byte	0x1
	.byte	0x3
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -16
	.uleb128 0x3
	.ascii	"b\000"
	.byte	0x1
	.byte	0x4
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -20
	.uleb128 0x3
	.ascii	"c\000"
	.byte	0x1
	.byte	0x5
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -24
	.byte	0x0
	.uleb128 0x4
	.byte	0x4
	.byte	0x5
	.ascii	"int\000"
	.byte	0x0
	.section	.debug_abbrev
	.uleb128 0x1
	.uleb128 0x11
	.byte	0x1
	.uleb128 0x25
	.uleb128 0xe
	.uleb128 0x13
	.uleb128 0xb
	.uleb128 0x3
	.uleb128 0xe
	.uleb128 0x1b
	.uleb128 0xe
	.uleb128 0x11
	.uleb128 0x1
	.uleb128 0x12
	.uleb128 0x1
	.uleb128 0x10
	.uleb128 0x6
	.byte	0x0
	.byte	0x0
	.uleb128 0x2
	.uleb128 0x2e
	.byte	0x1
	.uleb128 0x3f
	.uleb128 0xc
	.uleb128 0x3
	.uleb128 0xe
	.uleb128 0x3a
	.uleb128 0xb
	.uleb128 0x3b
	.uleb128 0xb
	.uleb128 0x27
	.uleb128 0xc
	.uleb128 0x49
	.uleb128 0x13
	.uleb128 0x11
	.uleb128 0x1
	.uleb128 0x12
	.uleb128 0x1
	.uleb128 0x40
	.uleb128 0x6
	.uleb128 0x1
	.uleb128 0x13
	.byte	0x0
	.byte	0x0
	.uleb128 0x3
	.uleb128 0x34
	.byte	0x0
	.uleb128 0x3
	.uleb128 0x8
	.uleb128 0x3a
	.uleb128 0xb
	.uleb128 0x3b
	.uleb128 0xb
	.uleb128 0x49
	.uleb128 0x13
	.uleb128 0x2
	.uleb128 0xa
	.byte	0x0
	.byte	0x0
	.uleb128 0x4
	.uleb128 0x24
	.byte	0x0
	.uleb128 0xb
	.uleb128 0xb
	.uleb128 0x3e
	.uleb128 0xb
	.uleb128 0x3
	.uleb128 0x8
	.byte	0x0
	.byte	0x0
	.byte	0x0
	.section	.debug_pubnames,"",@progbits
	.4byte	0x17
	.2byte	0x2
	.4byte	$Ldebug_info0
	.4byte	0x6f
	.4byte	0x25
	.ascii	"main\000"
	.4byte	0x0
	.section	.debug_aranges,"",@progbits
	.4byte	0x1c
	.2byte	0x2
	.4byte	$Ldebug_info0
	.byte	0x4
	.byte	0x0
	.2byte	0x0
	.2byte	0x0
	.4byte	$Ltext0
	.4byte	$Letext0-$Ltext0
	.4byte	0x0
	.4byte	0x0
	.section	.debug_str,"MS",@progbits,1
$LASF0:
	.ascii	"GNU C 4.4.0\000"
$LASF1:
	.ascii	"test.c\000"
$LASF2:
	.ascii	"/export/pfs/home/lte_dsp/zhoulibo/test\000"
$LASF3:
	.ascii	"main\000"
	.ident	"GCC: (GNU) 4.4.0"

One point to note, I did all this compilation process on a remote machine and copied all these files to my own PC with QEMU. The path /export/... is a remote path, but I don't think that's a problem though.
Please let me know if you need more information. I appreciate your help.

Thanks,
Libo Zhou

------------------ Original ------------------
From:  "Philippe Mathieu-Daud ";<philmd@redhat.com>;
Send time: Monday, Sep 23, 2019 10:50 PM
To: "Peter Maydell"<peter.maydell@linaro.org>; 
Cc: "Aleksandar Markovic"<aleksandar.m.mail@gmail.com>; "Libo Zhou"<zhlb29@foxmail.com>; "qemu-devel"<qemu-devel@nongnu.org>; 
Subject:  Re: illegal hardware instruction during MIPS-I ELF linux useremulation



On 9/23/19 4:42 PM, Peter Maydell wrote:
> On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daud  <philmd@redhat.com> wrote:
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> ...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
> they?

Some MIPS-I instructions where removed for MIPS-II (as RFE) and they are
not implemented. Also some CP0 registers are different.

>> IMO it is likely the RFE (Return from Exception) instruction.
> 
> It seems unlikely that a linux userspace binary would be trying to
> execute RFE...

Oh I thought it was system emulation, indeed it can't be RFE.

One GCC release targetting R3000 (Philips PR31700, Toshiba TX39) doesn't
emit NOP for branch's delay slot. I remember QEMU fails to run the
binaries it generates, but I don't remember how it fails.

Libo, can you provide more information about the cross-compiler you use
and the flags you use when calling it please?

Thanks,

Phil.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 17:26, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> Hi Philippe, Peter,
>
> In target/mips/translate_init.inc.c you can actually see that CPU_MIPS32 ultimately contains ISA_MIPS1. It's just no CPU model explicitly uses ISA_MIPS. But I agree that if my assembly has a removed instruction then it's possible to have an invalid machine code in my program.
>
> As for the cross-compiler, I used a tool chain containing a compiler, an assembler, and a linker. I produced the final ELF like this:
>
> $ mipsel-linux-unknown-gcc -g -S test.c -o test.s
> $ mipsel-linux-unknown-as -g test.s -o test.o
> $ mipsel-linux-unknown-ld test.o -o test

Can you run QEMU with some debugging options:

qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
-singlestep test

and then put the resulting debug.log somewhere we can get it?
(it'll probably be quite large)

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> Can you run QEMU with some debugging options:

> qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
> -singlestep test

> and then put the resulting debug.log somewhere we can get it?
> (it'll probably be quite large)

The logging only shows this little information. It seems like only -d cpu works for a short while.

----------------
IN: main
0x00400090:  bovc	sp,sp,0x400014

Trace 0: 0x563b750f7100 [00000000/00400090/0xe2] main
pc=0x00400090 HI=0x00000000 LO=0x00000000 ds 00e2 00000000 0
GPR00: r0 00000000 at 00000000 v0 00000000 v1 00000000
GPR04: a0 00000000 a1 00000000 a2 00000000 a3 00000000
GPR08: t0 00000000 t1 00000000 t2 00000000 t3 00000000
GPR12: t4 00000000 t5 00000000 t6 00000000 t7 00000000
GPR16: s0 00000000 s1 00000000 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000000 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00000000 sp 7ffff090 s8 00000000 ra 00000000
CP0 Status  0x24000010 Cause   0x00000000 EPC    0x00000000
    Config0 0x80000482 Config1 0x9e190c8f LLAddr 0x0000000000000000
    Config2 0x80000000 Config3 0x00000000
    Config4 0x00000000 Config5 0x00000000
----------------
IN: main
0x00400094:  dmult.g	zero,sp,s8

Trace 0: 0x563b750f7240 [00000000/00400094/0xe2] main
pc=0x00400094 HI=0x00000000 LO=0x00000000 ds 00e2 00000000 0
GPR00: r0 00000000 at 00000000 v0 00000000 v1 00000000
GPR04: a0 00000000 a1 00000000 a2 00000000 a3 00000000
GPR08: t0 00000000 t1 00000000 t2 00000000 t3 00000000
GPR12: t4 00000000 t5 00000000 t6 00000000 t7 00000000
GPR16: s0 00000000 s1 00000000 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000000 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00000000 sp 7ffff070 s8 00000000 ra 00000000
CP0 Status  0x24000010 Cause   0x00000000 EPC    0x00000000
    Config0 0x80000482 Config1 0x9e190c8f LLAddr 0x0000000000000000
    Config2 0x80000000 Config3 0x00000000
    Config4 0x00000000 Config5 0x00000000

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> > I would start by using the QEMU gdbstub to connect a
> > MIPS-aware gdb. Then when the SIGILL arrives you can see
> > what instruction the guest program was trying to execute.

> Just tried it and found something interesting.
> I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

> When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

> The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
> 'load word' is shown as '0x5f......';
> 'load immediate' is seen as 'addi';
> 'j' as 'jr';

> When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

> Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?

More updates about this. I just disassembled the unrecognized hex by hand, and figured out that the store word and load word opcodes are not the same as specified in translate.c. While the remaining fields of those unrecognized instructions do match with the source and destination registers.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Philippe Mathieu-Daudé]

On 9/24/19 3:31 PM, Libo Zhou wrote:
>>> I would start by using the QEMU gdbstub to connect a
>>> MIPS-aware gdb. Then when the SIGILL arrives you can see
>>> what instruction the guest program was trying to execute.
> 
>> Just tried it and found something interesting.
>> I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).
> 
>> When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.
> 
>> The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
>> 'load word' is shown as '0x5f......';
>> 'load immediate' is seen as 'addi';
>> 'j' as 'jr';
> 
>> When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).
> 
>> Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?
> 
> More updates about this. I just disassembled the unrecognized hex by hand, and figured out that the store word and load word opcodes are not the same as specified in translate.c. While the remaining fields of those unrecognized instructions do match with the source and destination registers.

What is your compiler/assembler versions (on both machines you used)?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Peter Maydell]

On Tue, 24 Sep 2019 at 02:10, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> > Can you run QEMU with some debugging options:
>
> > qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
> > -singlestep test
>
> > and then put the resulting debug.log somewhere we can get it?
> > (it'll probably be quite large)
>
> The logging only shows this little information. It seems like only -d cpu works for a short while.

> ----------------
> IN: main
> 0x00400094:  dmult.g    zero,sp,s8

I think "dmult.g" is an instruction specific to the Loongson CPU; it
is not in MIPS-I or even any of the other standard MIPS instruction
sets, which is why the default CPU for qemu-mipsel doesn't accept it
and is generating the SIGILL.

Something odd is going on here, because this doesn't seem to match
the disassembly of the test binary that you gave previously.

thanks
-- PMM