mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Guenter Roeck]

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during
boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c
("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch
fixes the problem.

Bisect log is attached.

Guenter

---
# bad: [494a19b0aa3c905b6c5221f49058526896b06a23] Add linux-next specific files for 20170526
# good: [08332893e37af6ae779367e78e444f8f9571511d] Linux 4.12-rc2
git bisect start 'HEAD' 'v4.12-rc2'
# good: [61f91dc3a7c15bdc06e746b0fa30a2b125bd41b6] Merge remote-tracking branch 'drm/drm-next'
git bisect good 61f91dc3a7c15bdc06e746b0fa30a2b125bd41b6
# bad: [854cfdd6fc9298018f6625d5e495405b2687b13c] Merge remote-tracking branch 'tip/auto-latest'
git bisect bad 854cfdd6fc9298018f6625d5e495405b2687b13c
# good: [bb4b4365c7a19d593d190ec6a380a7835b20e3fc] Merge remote-tracking branch 'sound/for-next'
git bisect good bb4b4365c7a19d593d190ec6a380a7835b20e3fc
# good: [830aade0f6121bf1bfa6504aa588b23469694803] Merge remote-tracking branch 'battery/for-next'
git bisect good 830aade0f6121bf1bfa6504aa588b23469694803
# bad: [0f90e8b13c713120ae0d2485533e5acf956f8321] Merge branch 'x86/boot'
git bisect bad 0f90e8b13c713120ae0d2485533e5acf956f8321
# good: [d04e31a23c3c828456cb5613f391ce4ac4e5765f] cpufreq/pasemi: Adjust system_state check
git bisect good d04e31a23c3c828456cb5613f391ce4ac4e5765f
# bad: [a451941cac2792181be56857e70233cbb9957cd2] Merge branch 'perf/urgent'
git bisect bad a451941cac2792181be56857e70233cbb9957cd2
# bad: [2c510368365cd0ce0c48c3f97a7048dc07ccd94a] Merge branch 'locking/urgent'
git bisect bad 2c510368365cd0ce0c48c3f97a7048dc07ccd94a
# good: [b9de5068b2d7c22de1bb89d62a7119a6b8dff88f] x86/ioapic: Remove unused IO_APIC_irq_trigger() function
git bisect good b9de5068b2d7c22de1bb89d62a7119a6b8dff88f
# bad: [79824ccf1691d0b8d4ef83d5d212ba0b31d8081e] Merge branch 'core/urgent'
git bisect bad 79824ccf1691d0b8d4ef83d5d212ba0b31d8081e
# bad: [4d6501dce079c1eb6bf0b1d8f528a5e81770109e] kthread: Fix use-after-free if kthread fork fails
git bisect bad 4d6501dce079c1eb6bf0b1d8f528a5e81770109e
# first bad commit: [4d6501dce079c1eb6bf0b1d8f528a5e81770109e] kthread: Fix use-after-free if kthread fork fails

Re: mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Vegard Nossum]

On 05/27/17 19:56, Guenter Roeck wrote:
> Hi,
> 
> my qemu testis of mips images are failing in -next. Symptom is a hang during
> boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.
> 
> I bisected the problem in next-20170526. It points to commit 4d6501dce079c
> ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch
> fixes the problem.
> 
> Bisect log is attached.

Hi,

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce
it on mips so I will try to debug.

Are you sure it's this commit, though? I checked out linus/master and
I get a boot hang even after reverting it.


Vegard

Re: mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Guenter Roeck]

+cc Linus, as mainline is now affected.

On 05/28/2017 04:45 AM, Vegard Nossum wrote:
> On 05/27/17 19:56, Guenter Roeck wrote:
>> Hi,
>>
>> my qemu testis of mips images are failing in -next. Symptom is a hang during
>> boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.
>>
>> I bisected the problem in next-20170526. It points to commit 4d6501dce079c
>> ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch
>> fixes the problem.
>>
>> Bisect log is attached.
> 
> Hi,
> 
> Thanks for the report and sorry for the breakage :-/
> 
> I can't immediately spot what's going wrong, but I am able to reproduce
> it on mips so I will try to debug.
> 
> Are you sure it's this commit, though? I checked out linus/master and
> I get a boot hang even after reverting it.
> 

Reverting the patch fixed the problem for me in -next. The last mainline build
on my test system was with v4.12-rc2-228-gc86daad, which was fine except for
the known build breakages (frv, hexagon) and the known tracebacks
in crisv32
	WARNING: CPU: 0 PID: 1 at drivers/mtd/mtdcore.c:734 mtd_device_parse_register+0x17a/0x1aa
and ppc
	refcount_t: underflow; use-after-free.
	WARNING: CPU: 0 PID: 1 at lib/refcount.c:184 refcount_sub_and_test+0x84/0xe0

It may well be that something else is broken in mainline. The build of
v4.12-rc2-284-g249f1ef on my builders isn't complete yet. We'll see in
a couple of hours if there is other damage. I'll bisect later today if
necessary.

Guenter

Re: mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Vegard Nossum]

On 05/28/17 13:45, Vegard Nossum wrote:
> On 05/27/17 19:56, Guenter Roeck wrote:
>> Hi,
>>
>> my qemu testis of mips images are failing in -next. Symptom is a hang 
>> during
>> boot; see http://kerneltests.org/builders/qemu-mips-next for some 
>> examples.
>>
>> I bisected the problem in next-20170526. It points to commit 
>> 4d6501dce079c
>> ("kthread: Fix use-after-free if kthread fork fails"). Reverting that 
>> patch
>> fixes the problem.
>>
>> Bisect log is attached.
> 
> Hi,
> 
> Thanks for the report and sorry for the breakage :-/
> 
> I can't immediately spot what's going wrong, but I am able to reproduce
> it on mips so I will try to debug.
> 
> Are you sure it's this commit, though? I checked out linus/master and
> I get a boot hang even after reverting it.

My mistake; I ran into a different bug which made me think it was
hanging when it wasn't.

However, I think I found the problem; does this patch fix it for you too?

diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c
index 918d4c73e951..5351e1f3950d 100644
--- a/arch/mips/kernel/process.c
+++ b/arch/mips/kernel/process.c
@@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, 
unsigned long usp,
         struct thread_info *ti = task_thread_info(p);
         struct pt_regs *childregs, *regs = current_pt_regs();
         unsigned long childksp;
-       p->set_child_tid = p->clear_child_tid = NULL;

         childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;

The problem is that when we moved the p->{set,clear}_child_tid
assignments inside copy_process(), the above assignments would clear
them out. The assignments only exist on mips and openrisc (which would
need the same patch), which explains why I didn't see it in my x86
testing. I think the patch above should be safe given that we're now
always setting these fields in copy_process() at an appropriate moment.

Looks like those assignments came from commit 3c37026d43c47 ("NPTL,
round one."); Ralf?

Oleg?


Vegard

Re: mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Guenter Roeck]

On 05/28/2017 07:49 AM, Vegard Nossum wrote:
> On 05/28/17 13:45, Vegard Nossum wrote:
>> On 05/27/17 19:56, Guenter Roeck wrote:
>>> Hi,
>>>
>>> my qemu testis of mips images are failing in -next. Symptom is a hang during
>>> boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.
>>>
>>> I bisected the problem in next-20170526. It points to commit 4d6501dce079c
>>> ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch
>>> fixes the problem.
>>>
>>> Bisect log is attached.
>>
>> Hi,
>>
>> Thanks for the report and sorry for the breakage :-/
>>
>> I can't immediately spot what's going wrong, but I am able to reproduce
>> it on mips so I will try to debug.
>>
>> Are you sure it's this commit, though? I checked out linus/master and
>> I get a boot hang even after reverting it.
> 
> My mistake; I ran into a different bug which made me think it was
> hanging when it wasn't.
> 
> However, I think I found the problem; does this patch fix it for you too?
> 

I'll give it a try.

I tried my qemu emulation on mainline after reverting your patch; it passed.
What kernel configuration and qemu command line did you use in your test ?

> diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c
> index 918d4c73e951..5351e1f3950d 100644
> --- a/arch/mips/kernel/process.c
> +++ b/arch/mips/kernel/process.c
> @@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp,
>          struct thread_info *ti = task_thread_info(p);
>          struct pt_regs *childregs, *regs = current_pt_regs();
>          unsigned long childksp;
> -       p->set_child_tid = p->clear_child_tid = NULL;
> 
>          childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
> 
> The problem is that when we moved the p->{set,clear}_child_tid
> assignments inside copy_process(), the above assignments would clear
> them out. The assignments only exist on mips and openrisc (which would
> need the same patch), which explains why I didn't see it in my x86

Interestingly, my openrisc test passed. Of course, that is just a boot test,
so it may not hit the problem.

Thanks,
Guenter

> testing. I think the patch above should be safe given that we're now
> always setting these fields in copy_process() at an appropriate moment.
> 
> Looks like those assignments came from commit 3c37026d43c47 ("NPTL,
> round one."); Ralf?
> 
> Oleg?
> 
> 
> Vegard
> 

Re: mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

[Guenter Roeck]

On 05/28/2017 07:49 AM, Vegard Nossum wrote:
> On 05/28/17 13:45, Vegard Nossum wrote:
>> On 05/27/17 19:56, Guenter Roeck wrote:
>>> Hi,
>>>
>>> my qemu testis of mips images are failing in -next. Symptom is a hang during
>>> boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.
>>>
>>> I bisected the problem in next-20170526. It points to commit 4d6501dce079c
>>> ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch
>>> fixes the problem.
>>>
>>> Bisect log is attached.
>>
>> Hi,
>>
>> Thanks for the report and sorry for the breakage :-/
>>
>> I can't immediately spot what's going wrong, but I am able to reproduce
>> it on mips so I will try to debug.
>>
>> Are you sure it's this commit, though? I checked out linus/master and
>> I get a boot hang even after reverting it.
> 
> My mistake; I ran into a different bug which made me think it was
> hanging when it wasn't.
> 
> However, I think I found the problem; does this patch fix it for you too?
> 
> diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c
> index 918d4c73e951..5351e1f3950d 100644
> --- a/arch/mips/kernel/process.c
> +++ b/arch/mips/kernel/process.c
> @@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp,
>          struct thread_info *ti = task_thread_info(p);
>          struct pt_regs *childregs, *regs = current_pt_regs();
>          unsigned long childksp;
> -       p->set_child_tid = p->clear_child_tid = NULL;
> 
>          childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
> 

Confirmed; that patch fixes the problem for me.

Guenter

> The problem is that when we moved the p->{set,clear}_child_tid
> assignments inside copy_process(), the above assignments would clear
> them out. The assignments only exist on mips and openrisc (which would
> need the same patch), which explains why I didn't see it in my x86
> testing. I think the patch above should be safe given that we're now
> always setting these fields in copy_process() at an appropriate moment.
> 
> Looks like those assignments came from commit 3c37026d43c47 ("NPTL,
> round one."); Ralf?
> 
> Oleg?
> 
> 
> Vegard
>