Re: illegal hardware instruction during MIPS-I ELF linux useremulation

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

Hi Philippe, Peter,

In target/mips/translate_init.inc.c you can actually see that CPU_MIPS32 ultimately contains ISA_MIPS1. It's just no CPU model explicitly uses ISA_MIPS. But I agree that if my assembly has a removed instruction then it's possible to have an invalid machine code in my program.

As for the cross-compiler, I used a tool chain containing a compiler, an assembler, and a linker. I produced the final ELF like this:

$ mipsel-linux-unknown-gcc -g -S test.c -o test.s
$ mipsel-linux-unknown-as -g test.s -o test.o
$ mipsel-linux-unknown-ld test.o -o test

For my qemu, I did:

$ ../configure --enable-debug --target-list=mipsel-linux-user
$ make -j4

Here are the contents of my source and generated assembly:

$ cat test.c
int main(void)
{
    int a = 1;
    int b = 2;
    int c = a + b;
    return 0;
}

$ cat test.s
	.section .mdebug.abi32
	.previous
	.gnu_attribute 4, 3
	.abicalls
	.section	.debug_abbrev,"",@progbits
$Ldebug_abbrev0:
	.section	.debug_info,"",@progbits
$Ldebug_info0:
	.section	.debug_line,"",@progbits
$Ldebug_line0:
	.text
$Ltext0:
	.align	2
	.globl	main
$LFB0 = .
	.file 1 "test.c"
	.loc 1 2 0
	.cfi_startproc
	.set	nomips16
	.ent	main
	.type	main, @function
main:
	.frame	$fp,32,$31		# vars= 16, regs= 1/0, args= 0, gp= 8
	.mask	0x40000000,-4
	.fmask	0x00000000,0
	.set	noreorder
	.set	nomacro
	
	addi	$sp,$sp,-32
	.cfi_def_cfa_offset 32
	sw	$fp,28($sp)
	nop
	nop
	move	$fp,$sp
	.cfi_offset 30, -4
	.cfi_def_cfa_register 30
	.loc 1 3 0
	li	$2,1			# 0x1
	sw	$2,16($fp)
	nop
	nop
	.loc 1 4 0
	li	$2,2			# 0x2
	sw	$2,12($fp)
	nop
	nop
	.loc 1 5 0
	lw	$3,16($fp)
	lw	$2,12($fp)
	nop
	nop
	add	$2,$3,$2
	sw	$2,8($fp)
	nop
	nop
	.loc 1 6 0
	move	$2,$0
	.loc 1 7 0
	move	$sp,$fp
	lw	$fp,28($sp)
	addi	$sp,$sp,32
	j	$31
	nop
	nop
	nop
	nop
	nop

	.set	macro
	

	.end	main
	.cfi_endproc
$LFE0:
	.size	main, .-main
$Letext0:
	.section	.debug_loc,"",@progbits
$Ldebug_loc0:
$LLST0:
	.4byte	$LFB0-$Ltext0
	.4byte	$LFE0-$Ltext0
	.2byte	0x2
	.byte	0x8e
	.sleb128 32
	.4byte	0x0
	.4byte	0x0
	.section	.debug_info
	.4byte	0x6b
	.2byte	0x2
	.4byte	$Ldebug_abbrev0
	.byte	0x4
	.uleb128 0x1
	.4byte	$LASF0
	.byte	0x1
	.4byte	$LASF1
	.4byte	$LASF2
	.4byte	$Ltext0
	.4byte	$Letext0
	.4byte	$Ldebug_line0
	.uleb128 0x2
	.byte	0x1
	.4byte	$LASF3
	.byte	0x1
	.byte	0x1
	.byte	0x1
	.4byte	0x67
	.4byte	$LFB0
	.4byte	$LFE0
	.4byte	$LLST0
	.4byte	0x67
	.uleb128 0x3
	.ascii	"a\000"
	.byte	0x1
	.byte	0x3
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -16
	.uleb128 0x3
	.ascii	"b\000"
	.byte	0x1
	.byte	0x4
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -20
	.uleb128 0x3
	.ascii	"c\000"
	.byte	0x1
	.byte	0x5
	.4byte	0x67
	.byte	0x2
	.byte	0x91
	.sleb128 -24
	.byte	0x0
	.uleb128 0x4
	.byte	0x4
	.byte	0x5
	.ascii	"int\000"
	.byte	0x0
	.section	.debug_abbrev
	.uleb128 0x1
	.uleb128 0x11
	.byte	0x1
	.uleb128 0x25
	.uleb128 0xe
	.uleb128 0x13
	.uleb128 0xb
	.uleb128 0x3
	.uleb128 0xe
	.uleb128 0x1b
	.uleb128 0xe
	.uleb128 0x11
	.uleb128 0x1
	.uleb128 0x12
	.uleb128 0x1
	.uleb128 0x10
	.uleb128 0x6
	.byte	0x0
	.byte	0x0
	.uleb128 0x2
	.uleb128 0x2e
	.byte	0x1
	.uleb128 0x3f
	.uleb128 0xc
	.uleb128 0x3
	.uleb128 0xe
	.uleb128 0x3a
	.uleb128 0xb
	.uleb128 0x3b
	.uleb128 0xb
	.uleb128 0x27
	.uleb128 0xc
	.uleb128 0x49
	.uleb128 0x13
	.uleb128 0x11
	.uleb128 0x1
	.uleb128 0x12
	.uleb128 0x1
	.uleb128 0x40
	.uleb128 0x6
	.uleb128 0x1
	.uleb128 0x13
	.byte	0x0
	.byte	0x0
	.uleb128 0x3
	.uleb128 0x34
	.byte	0x0
	.uleb128 0x3
	.uleb128 0x8
	.uleb128 0x3a
	.uleb128 0xb
	.uleb128 0x3b
	.uleb128 0xb
	.uleb128 0x49
	.uleb128 0x13
	.uleb128 0x2
	.uleb128 0xa
	.byte	0x0
	.byte	0x0
	.uleb128 0x4
	.uleb128 0x24
	.byte	0x0
	.uleb128 0xb
	.uleb128 0xb
	.uleb128 0x3e
	.uleb128 0xb
	.uleb128 0x3
	.uleb128 0x8
	.byte	0x0
	.byte	0x0
	.byte	0x0
	.section	.debug_pubnames,"",@progbits
	.4byte	0x17
	.2byte	0x2
	.4byte	$Ldebug_info0
	.4byte	0x6f
	.4byte	0x25
	.ascii	"main\000"
	.4byte	0x0
	.section	.debug_aranges,"",@progbits
	.4byte	0x1c
	.2byte	0x2
	.4byte	$Ldebug_info0
	.byte	0x4
	.byte	0x0
	.2byte	0x0
	.2byte	0x0
	.4byte	$Ltext0
	.4byte	$Letext0-$Ltext0
	.4byte	0x0
	.4byte	0x0
	.section	.debug_str,"MS",@progbits,1
$LASF0:
	.ascii	"GNU C 4.4.0\000"
$LASF1:
	.ascii	"test.c\000"
$LASF2:
	.ascii	"/export/pfs/home/lte_dsp/zhoulibo/test\000"
$LASF3:
	.ascii	"main\000"
	.ident	"GCC: (GNU) 4.4.0"

One point to note, I did all this compilation process on a remote machine and copied all these files to my own PC with QEMU. The path /export/... is a remote path, but I don't think that's a problem though.
Please let me know if you need more information. I appreciate your help.

Thanks,
Libo Zhou

------------------ Original ------------------
From:  "Philippe Mathieu-Daud ";<philmd@redhat.com>;
Send time: Monday, Sep 23, 2019 10:50 PM
To: "Peter Maydell"<peter.maydell@linaro.org>; 
Cc: "Aleksandar Markovic"<aleksandar.m.mail@gmail.com>; "Libo Zhou"<zhlb29@foxmail.com>; "qemu-devel"<qemu-devel@nongnu.org>; 
Subject:  Re: illegal hardware instruction during MIPS-I ELF linux useremulation



On 9/23/19 4:42 PM, Peter Maydell wrote:
> On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daud  <philmd@redhat.com> wrote:
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> ...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
> they?

Some MIPS-I instructions where removed for MIPS-II (as RFE) and they are
not implemented. Also some CP0 registers are different.

>> IMO it is likely the RFE (Return from Exception) instruction.
> 
> It seems unlikely that a linux userspace binary would be trying to
> execute RFE...

Oh I thought it was system emulation, indeed it can't be RFE.

One GCC release targetting R3000 (Philips PR31700, Toshiba TX39) doesn't
emit NOP for branch's delay slot. I remember QEMU fails to run the
binaries it generates, but I don't remember how it fails.

Libo, can you provide more information about the cross-compiler you use
and the flags you use when calling it please?

Thanks,

Phil.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 17:26, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> Hi Philippe, Peter,
>
> In target/mips/translate_init.inc.c you can actually see that CPU_MIPS32 ultimately contains ISA_MIPS1. It's just no CPU model explicitly uses ISA_MIPS. But I agree that if my assembly has a removed instruction then it's possible to have an invalid machine code in my program.
>
> As for the cross-compiler, I used a tool chain containing a compiler, an assembler, and a linker. I produced the final ELF like this:
>
> $ mipsel-linux-unknown-gcc -g -S test.c -o test.s
> $ mipsel-linux-unknown-as -g test.s -o test.o
> $ mipsel-linux-unknown-ld test.o -o test

Can you run QEMU with some debugging options:

qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
-singlestep test

and then put the resulting debug.log somewhere we can get it?
(it'll probably be quite large)

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> Can you run QEMU with some debugging options:

> qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
> -singlestep test

> and then put the resulting debug.log somewhere we can get it?
> (it'll probably be quite large)

The logging only shows this little information. It seems like only -d cpu works for a short while.

----------------
IN: main
0x00400090:  bovc	sp,sp,0x400014

Trace 0: 0x563b750f7100 [00000000/00400090/0xe2] main
pc=0x00400090 HI=0x00000000 LO=0x00000000 ds 00e2 00000000 0
GPR00: r0 00000000 at 00000000 v0 00000000 v1 00000000
GPR04: a0 00000000 a1 00000000 a2 00000000 a3 00000000
GPR08: t0 00000000 t1 00000000 t2 00000000 t3 00000000
GPR12: t4 00000000 t5 00000000 t6 00000000 t7 00000000
GPR16: s0 00000000 s1 00000000 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000000 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00000000 sp 7ffff090 s8 00000000 ra 00000000
CP0 Status  0x24000010 Cause   0x00000000 EPC    0x00000000
    Config0 0x80000482 Config1 0x9e190c8f LLAddr 0x0000000000000000
    Config2 0x80000000 Config3 0x00000000
    Config4 0x00000000 Config5 0x00000000
----------------
IN: main
0x00400094:  dmult.g	zero,sp,s8

Trace 0: 0x563b750f7240 [00000000/00400094/0xe2] main
pc=0x00400094 HI=0x00000000 LO=0x00000000 ds 00e2 00000000 0
GPR00: r0 00000000 at 00000000 v0 00000000 v1 00000000
GPR04: a0 00000000 a1 00000000 a2 00000000 a3 00000000
GPR08: t0 00000000 t1 00000000 t2 00000000 t3 00000000
GPR12: t4 00000000 t5 00000000 t6 00000000 t7 00000000
GPR16: s0 00000000 s1 00000000 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000000 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00000000 sp 7ffff070 s8 00000000 ra 00000000
CP0 Status  0x24000010 Cause   0x00000000 EPC    0x00000000
    Config0 0x80000482 Config1 0x9e190c8f LLAddr 0x0000000000000000
    Config2 0x80000000 Config3 0x00000000
    Config4 0x00000000 Config5 0x00000000

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> > I would start by using the QEMU gdbstub to connect a
> > MIPS-aware gdb. Then when the SIGILL arrives you can see
> > what instruction the guest program was trying to execute.

> Just tried it and found something interesting.
> I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

> When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

> The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
> 'load word' is shown as '0x5f......';
> 'load immediate' is seen as 'addi';
> 'j' as 'jr';

> When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

> Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?

More updates about this. I just disassembled the unrecognized hex by hand, and figured out that the store word and load word opcodes are not the same as specified in translate.c. While the remaining fields of those unrecognized instructions do match with the source and destination registers.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Philippe Mathieu-Daudé]

On 9/24/19 3:31 PM, Libo Zhou wrote:
>>> I would start by using the QEMU gdbstub to connect a
>>> MIPS-aware gdb. Then when the SIGILL arrives you can see
>>> what instruction the guest program was trying to execute.
> 
>> Just tried it and found something interesting.
>> I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).
> 
>> When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.
> 
>> The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
>> 'load word' is shown as '0x5f......';
>> 'load immediate' is seen as 'addi';
>> 'j' as 'jr';
> 
>> When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).
> 
>> Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box? Or is it possible that my compiler doesn't implement MIPS-I correctly?
> 
> More updates about this. I just disassembled the unrecognized hex by hand, and figured out that the store word and load word opcodes are not the same as specified in translate.c. While the remaining fields of those unrecognized instructions do match with the source and destination registers.

What is your compiler/assembler versions (on both machines you used)?

Re: illegal hardware instruction during MIPS-I ELF linuxuseremulation

[Libo Zhou]

> > More updates about this. I just disassembled the unrecognized hex by hand, and figured out that the store word and load word opcodes are not the same as specified in translate.c. While the remaining fields of those unrecognized instructions do match with the source and destination registers.

> What is your compiler/assembler versions (on both machines you used)?

I don't have access to the machine for now and I may not remember the exact version numbers.

The cross compiler I used is a custom compiler based on gcc 4.4.0 (vaguely remember). It generated MIPS-I code that didn't work on QEMU. Specifically some generated opcodes didn't match those in target/mips/translate.c. However, I just checked Wikipedia's MIPS-I opcode table and I think QEMU implements it correctly. The single and double floating point opcode looked a little off for me though, but I didn't use FP ops in my case.

On my own PC I used mipsel-linux-gnu-gcc version 7.4.0. It just worked fine on QEMU.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Peter Maydell]

On Tue, 24 Sep 2019 at 02:10, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> > Can you run QEMU with some debugging options:
>
> > qemu-mipsel -d in_asm,exec,cpu,unimp,guest_errors,nochain -D debug.log
> > -singlestep test
>
> > and then put the resulting debug.log somewhere we can get it?
> > (it'll probably be quite large)
>
> The logging only shows this little information. It seems like only -d cpu works for a short while.

> ----------------
> IN: main
> 0x00400094:  dmult.g    zero,sp,s8

I think "dmult.g" is an instruction specific to the Loongson CPU; it
is not in MIPS-I or even any of the other standard MIPS instruction
sets, which is why the default CPU for qemu-mipsel doesn't accept it
and is generating the SIGILL.

Something odd is going on here, because this doesn't seem to match
the disassembly of the test binary that you gave previously.

thanks
-- PMM