general protection fault, probably for non-canonical address in pick_next_task_fair()

general protection fault, probably for non-canonical address in pick_next_task_fair()

[Breno Leitao]

I've been running some stress test using stress-ng with a kernel with some
debug options enabled, such as KASAN and friends (See the config below).

I saw it in rc4 and the decode instructions are a bit off (as it is here
also - search for mavabs in dmesg below and you will find something as `(bad)`,
so I though it was a machine issue. But now I see it again, and I am sharing
for awareness.

This is happening in upstream kernel against the following commit
d206a76d7d2726 ("Linux 6.8-rc6")

This is the exercpt that shows before the crash:

	general protection fault, probably for non-canonical address 0xdffffc0000000014: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
	KASAN: null-ptr-deref in range [0x00000000000000a0-0x00000000000000a7]

This is the stack that is getting it

	? __die_body (arch/x86/kernel/dumpstack.c:421)
	? die_addr (arch/x86/kernel/dumpstack.c:460)
	? exc_general_protection (arch/x86/kernel/traps.c:? arch/x86/kernel/traps.c:643)
	? asm_exc_general_protection (arch/x86/include/asm/idtentry.h:564)
	? pick_next_task_fair (kernel/sched/sched.h:1453 kernel/sched/fair.c:8435)
	? pick_next_task_fair (kernel/sched/fair.c:5463 kernel/sched/fair.c:8434)
	? update_rq_clock_task (kernel/sched/core.c:?)
	__schedule (kernel/sched/core.c:6022 kernel/sched/core.c:6545 kernel/sched/core.c:6691)
	schedule (kernel/sched/core.c:6803 kernel/sched/core.c:6817)
	syscall_exit_to_user_mode (kernel/entry/common.c:98 include/linux/entry-common.h:328 kernel/entry/common.c:201 kernel/entry/common.c:212) 
	do_syscall_64 (arch/x86/entry/common.c:102)
	? irqentry_exit_to_user_mode (kernel/entry/common.c:228)
	entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:129)

Full dmesg: https://paste.mozilla.org/RiLnt4QO#
Configs: https://paste.mozilla.org/XJ9wbdRp

Re: general protection fault, probably for non-canonical address in pick_next_task_fair()

[Thomas Gleixner]

On Thu, Feb 29 2024 at 07:55, Breno Leitao wrote:
> I've been running some stress test using stress-ng with a kernel with some
> debug options enabled, such as KASAN and friends (See the config below).
>
> I saw it in rc4 and the decode instructions are a bit off (as it is here
> also - search for mavabs in dmesg below and you will find something as `(bad)`,
> so I though it was a machine issue. But now I see it again, and I am sharing
> for awareness.

The (bad) is after the faulting instruction, but gives an hint:

  2e:	0f 84 67 ff ff ff    	je     0xffffffffffffff9b
  34:	48 89 ef             	mov    %rbp,%rdi
  37:	e8 cf 70 76 00       	call   0x76710b
  3c:	e9                   	.byte 0xe9

That's an invalid opcode, which means that memory is corrupted.

Thanks,

        tglx

Re: general protection fault, probably for non-canonical address in pick_next_task_fair()

[Abel Wu]

Hi Breno, this seems to be a known issue under discussion.

https://lore.kernel.org/lkml/202401301012.2ed95df0-oliver.sang@intel.com/
https://lore.kernel.org/lkml/20240226082349.302363-1-yu.c.chen@intel.com/

On 2/29/24 11:55 PM, Breno Leitao Wrote:
> I've been running some stress test using stress-ng with a kernel with some
> debug options enabled, such as KASAN and friends (See the config below).
> 
> I saw it in rc4 and the decode instructions are a bit off (as it is here
> also - search for mavabs in dmesg below and you will find something as `(bad)`,
> so I though it was a machine issue. But now I see it again, and I am sharing
> for awareness.
> 
> This is happening in upstream kernel against the following commit
> d206a76d7d2726 ("Linux 6.8-rc6")
> 
> This is the exercpt that shows before the crash:
> 
> 	general protection fault, probably for non-canonical address 0xdffffc0000000014: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
> 	KASAN: null-ptr-deref in range [0x00000000000000a0-0x00000000000000a7]
> 
> This is the stack that is getting it
> 
> 	? __die_body (arch/x86/kernel/dumpstack.c:421)
> 	? die_addr (arch/x86/kernel/dumpstack.c:460)
> 	? exc_general_protection (arch/x86/kernel/traps.c:? arch/x86/kernel/traps.c:643)
> 	? asm_exc_general_protection (arch/x86/include/asm/idtentry.h:564)
> 	? pick_next_task_fair (kernel/sched/sched.h:1453 kernel/sched/fair.c:8435)
> 	? pick_next_task_fair (kernel/sched/fair.c:5463 kernel/sched/fair.c:8434)
> 	? update_rq_clock_task (kernel/sched/core.c:?)
> 	__schedule (kernel/sched/core.c:6022 kernel/sched/core.c:6545 kernel/sched/core.c:6691)
> 	schedule (kernel/sched/core.c:6803 kernel/sched/core.c:6817)
> 	syscall_exit_to_user_mode (kernel/entry/common.c:98 include/linux/entry-common.h:328 kernel/entry/common.c:201 kernel/entry/common.c:212)
> 	do_syscall_64 (arch/x86/entry/common.c:102)
> 	? irqentry_exit_to_user_mode (kernel/entry/common.c:228)
> 	entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:129)
> 
> Full dmesg: https://paste.mozilla.org/RiLnt4QO#
> Configs: https://paste.mozilla.org/XJ9wbdRp
> 

Re: general protection fault, probably for non-canonical address in pick_next_task_fair()

[Abel Wu]

(+ Chen Yu, Oliver Sang)

On 2/29/24 11:55 PM, Breno Leitao Wrote:
> I've been running some stress test using stress-ng with a kernel with some
> debug options enabled, such as KASAN and friends (See the config below).
> 
> I saw it in rc4 and the decode instructions are a bit off (as it is here
> also - search for mavabs in dmesg below and you will find something as `(bad)`,
> so I though it was a machine issue. But now I see it again, and I am sharing
> for awareness.
> 
> This is happening in upstream kernel against the following commit
> d206a76d7d2726 ("Linux 6.8-rc6")
> 
> This is the exercpt that shows before the crash:
> 
> 	general protection fault, probably for non-canonical address 0xdffffc0000000014: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
> 	KASAN: null-ptr-deref in range [0x00000000000000a0-0x00000000000000a7]
> 
> This is the stack that is getting it
> 
> 	? __die_body (arch/x86/kernel/dumpstack.c:421)
> 	? die_addr (arch/x86/kernel/dumpstack.c:460)
> 	? exc_general_protection (arch/x86/kernel/traps.c:? arch/x86/kernel/traps.c:643)
> 	? asm_exc_general_protection (arch/x86/include/asm/idtentry.h:564)
> 	? pick_next_task_fair (kernel/sched/sched.h:1453 kernel/sched/fair.c:8435)
> 	? pick_next_task_fair (kernel/sched/fair.c:5463 kernel/sched/fair.c:8434)
> 	? update_rq_clock_task (kernel/sched/core.c:?)
> 	__schedule (kernel/sched/core.c:6022 kernel/sched/core.c:6545 kernel/sched/core.c:6691)
> 	schedule (kernel/sched/core.c:6803 kernel/sched/core.c:6817)
> 	syscall_exit_to_user_mode (kernel/entry/common.c:98 include/linux/entry-common.h:328 kernel/entry/common.c:201 kernel/entry/common.c:212)
> 	do_syscall_64 (arch/x86/entry/common.c:102)
> 	? irqentry_exit_to_user_mode (kernel/entry/common.c:228)
> 	entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:129)
> 
> Full dmesg: https://paste.mozilla.org/RiLnt4QO#
> Configs: https://paste.mozilla.org/XJ9wbdRp
> 

Re: general protection fault, probably for non-canonical address in pick_next_task_fair()

[Chen Yu]

On 2024-03-01 at 11:47:05 +0800, Abel Wu wrote:
> (+ Chen Yu, Oliver Sang)
> 
> On 2/29/24 11:55 PM, Breno Leitao Wrote:
> > I've been running some stress test using stress-ng with a kernel with some
> > debug options enabled, such as KASAN and friends (See the config below).
> > 
> > I saw it in rc4 and the decode instructions are a bit off (as it is here
> > also - search for mavabs in dmesg below and you will find something as `(bad)`,
> > so I though it was a machine issue. But now I see it again, and I am sharing
> > for awareness.
> > 
> > This is happening in upstream kernel against the following commit
> > d206a76d7d2726 ("Linux 6.8-rc6")
> > 
> > This is the exercpt that shows before the crash:
> > 
> > 	general protection fault, probably for non-canonical address 0xdffffc0000000014: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
> > 	KASAN: null-ptr-deref in range [0x00000000000000a0-0x00000000000000a7]
> > 
> > This is the stack that is getting it
> > 
> > 	? __die_body (arch/x86/kernel/dumpstack.c:421)
> > 	? die_addr (arch/x86/kernel/dumpstack.c:460)
> > 	? exc_general_protection (arch/x86/kernel/traps.c:? arch/x86/kernel/traps.c:643)
> > 	? asm_exc_general_protection (arch/x86/include/asm/idtentry.h:564)
> > 	? pick_next_task_fair (kernel/sched/sched.h:1453 kernel/sched/fair.c:8435)

Seems to be the same reason pick_eevdf returns NULL.. it panic here..
cfs_rq = group_cfs_rq(se);

I remember lkp has regular stress-ng test for regression test, but
not detect this yet.

thanks,
Chenyu