Re: boot time scheduling hile atomic

Re: boot time scheduling hile atomic

[_evil]

Joerg Dorchain wrote:
>>>
>>>i don't know why you're getting this error, but seeing "swapper" to be
>>>involved here: can you compile with CONFIG_SWAP=n and boot this one?
>>
>>I'll do tonight.
> 
> Did so, no effect on this. IMHO "swapper" is a somewhat misguiding name
> for the idle task with pid 0.

yes, i just guessed. sadly, it did not help :-\
and i can't decode the backtrace either.

does the system halt after the message? if not, we a re not in a hurry 
and if you have time you could possbily track it down a bit to 1) the 
kernelversion when it happened first and/or 2) find the .config option, 
that causes this error.

if the box halts after this, and no guru from linuxppc-dev is available, 
maybe you should try and post on linux-kernel too.

just my 2c,
Christian.

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi

Joerg Dorchain wrote:
> On Thu, Mar 03, 2005 at 03:18:32PM +0100, Joerg Dorchain wrote:
> 
>>>>scheduling while atomic: swapper/0x00000002/0
>>>>Call trace:
>>>> [c0007620] dump_stack+0x18/0x28
>>>> [c01de704] schedule+0x678/0x67c
>>>> [c0004500] syscall_exit_work+0x108/0x10c
>>>> [c02a97b4] proc_root_init+0x168/0x174
>>>> [ff847288] 0xff847288
>>>> [c02945e8] start_kernel+0x144/0x170
>>>> [00003a30] 0x3a30
>>
>>I'll do tonight.
> 
> I am more wondering what in proc_root_init or immediately after it
> causes a call to the exit_work syscall.
> 
This is not called by proc_root_init.
It seems be called by timer interrupt
(I confirmed it by viewing trap number in
interrupt context with ICE).

As long as I investigate the problem,
proc_root_init is called twice.
At first time, it's called from
start_kernel in init/main.c.
Second case, it seems to be called by some kind of
init call facility.

proc_root_init is not init_call function, I could not
figure out concretely why it called twice.

I will investigate the problem as long as I can.

I wishes this report to become your help.

Regards,
-- 
Takeharu KATO
Fujitsu Limited
Email:kato.takeharu at jp.fujitsu.com

Re: boot time scheduling hile atomic

[Joerg Dorchain]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Thu, Mar 10, 2005 at 01:51:27PM +0900, Takeharu KATO wrote:
> > I am more wondering what in proc_root_init or immediately after it
> > causes a call to the exit_work syscall.
> > 
> This is not called by proc_root_init.
> It seems be called by timer interrupt
> (I confirmed it by viewing trap number in
> interrupt context with ICE).

I came to the same idea. This explains as well why the i386 party does
not notice the effect, as their first interrupt happen later (different
HZ value)

At this point, I was out of clue, so thank you for your help.

Bye,

Joerg

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Joerg:

> I came to the same idea. This explains as well why the i386 party does
> not notice the effect, as their first interrupt happen later (different
> HZ value)
> 
As far as I know, current HZ value for PPC is 1000
as same as i386 family.

Please see following:

-- include/asm-i386/param.h
   1 #ifndef _ASMi386_PARAM_H
   2 #define _ASMi386_PARAM_H
   3
   4 #ifdef __KERNEL__
   5 # define HZ             1000            /* Internal kernel timer 
frequency */
-- include/asm-i386/param.h

-- include/asm-ppc/param.h

   1 #ifndef _ASM_PPC_PARAM_H
   2 #define _ASM_PPC_PARAM_H
   3
   4 #ifdef __KERNEL__
   5 #define HZ              1000            /* internal timer frequency */
   6 #define USER_HZ         100             /* for user interfaces in 
"ticks" */
-- include/asm-ppc/param.h

FYI, Kernel preemption ought not to be taken place at this point,
because preemption count is a positive number.

According to my observation, the problem occur on second call of
proc_root_init.But I could not who call it.
IIRC,the value of LR(link register) pointed at the start of BSS.

Regards,



-- 
Takeharu KATO

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Joerg:

First of all, I am sorry to making mistake on my stack analysis.
After all, I found what is wrong.

The syscall_exit_work is called(jumped) by fork, strictly speaking,
this is caused by kernel_thead for init process.
At this point, kernel set a need_resched flag in thread_info struct
(flags member).
So, the system executes syscall_exit_work when
the swapper thread exited from syscall.

However, the syscall_exit_work routine does not check
preempt counts when it calls schedule function.

FYI, I post the patch which fix this problem.
I've confirmed the problem has been removed by this patch,
but I can not figure out whether it is truely
correct or not.

Please review this.

Signed-off-by: Takeharu KATO <kato.takeharu@jp.fujitsu.com>
--- linux-2.6.11/arch/ppc/kernel/entry.S        2005-03-09 12:57:32.000000000 +0900
+++ linux-2.6.11-wdt/arch/ppc/kernel/entry.S    2005-03-11 14:39:16.345490784 +0900
@@ -314,10 +314,16 @@ syscall_exit_work:
         SYNC
         MTMSRD(r10)             /* disable interrupts again */
         rlwinm  r12,r1,0,0,18   /* current_thread_info() */
-       lwz     r9,TI_FLAGS(r12)
  5:
+#ifdef CONFIG_PREEMPT
+       lwz     r0,TI_PREEMPT(r12)
+       cmpwi   0,r0,0          /* if non-zero, we can not reschedule here */
+       bne     6f
+#endif
+       lwz     r9,TI_FLAGS(r12)
         andi.   r0,r9,_TIF_NEED_RESCHED
         bne     1f
+6:
         lwz     r5,_MSR(r1)
         andi.   r5,r5,MSR_PR
         beq     syscall_exit_cont

-- 
Takeharu KATO
Fujitsu Limited
Email:kato.takeharu at jp.fujitsu.com

Re: boot time scheduling hile atomic

[Benjamin Herrenschmidt]

On Fri, 2005-03-11 at 15:19 +0900, Takeharu KATO wrote:
> Hi Joerg:
> 
> First of all, I am sorry to making mistake on my stack analysis.
> After all, I found what is wrong.
> 
> The syscall_exit_work is called(jumped) by fork, strictly speaking,
> this is caused by kernel_thead for init process.
> At this point, kernel set a need_resched flag in thread_info struct
> (flags member).
> So, the system executes syscall_exit_work when
> the swapper thread exited from syscall.
> 
> However, the syscall_exit_work routine does not check
> preempt counts when it calls schedule function.
> 
> FYI, I post the patch which fix this problem.
> I've confirmed the problem has been removed by this patch,
> but I can not figure out whether it is truely
> correct or not.

I think the problem is more why are we calling a syscall while the
preempt count is up ?

Ben.

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Benjamin:

Thank you for your response.

> I think the problem is more why are we calling a syscall while the
> preempt count is up ?
> 
Because, start_kernel(line:455) in init/main.c count up this.
-- start_kernel (init/main.c)
451         /*
452          * Disable preemption - early bootup scheduling is extremely
453          * fragile until we cpu_idle() for the first time.
454          */
455         preempt_disable();
456         build_all_zonelists();
-- start_kernel(init/main.c)

Preemption is disabled until it enters the body of idle task from rest_init.
Please see below:
-- rest_init (init/main.c)
379 static void noinline rest_init(void)
380         __releases(kernel_lock)
381 {
382         kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
383         numa_default_policy();
384         unlock_kernel();
385         preempt_enable_no_resched();
386         cpu_idle();
387 }
--
I think that the kernel assumes that init thread(the kernel thread which become to
init process later) is kicked by idle process(cpu_idle).

But, the kernel invoke a system call trap when it execute kernel_thread call
(at rest_init line:382) on PowerPC.
The preemption count has positive value here.
So, PowerPC linux kernel invoke system call when the preempt count is up.

Regards,

-- 
Takeharu KATO
Fujitsu Limited
Email:kato.takeharu at jp.fujitsu.com

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Benjamin:

I forgot writing that I've sent the revised edition of
the patch to the PPC-Dev ML.
Please see the patch as ``boot time scheduling w hile atomic fix''
which I posted in 2005-03-15 from Patch tracking
system for PowerPC Linux as follows.

http://ozlabs.org/ppc32-patches/

It changes the implementation of kernel_thread function
to be performed with function call.

Regards,

Takeharu KATO wrote:

> But, the kernel invoke a system call trap when it execute kernel_thread call
> (at rest_init line:382) on PowerPC.
> The preemption count has positive value here.
> So, PowerPC linux kernel invoke system call when the preempt count is up.
> 

-- 
Takeharu KATO
Fujitsu Limited
Email:kato.takeharu at jp.fujitsu.com

Re: boot time scheduling hile atomic

[Benjamin Herrenschmidt]

On Wed, 2005-03-16 at 14:49 +0900, Takeharu KATO wrote:
> Hi Benjamin:
> 
> I forgot writing that I've sent the revised edition of
> the patch to the PPC-Dev ML.
> Please see the patch as ``boot time scheduling w hile atomic fix''
> which I posted in 2005-03-15 from Patch tracking
> system for PowerPC Linux as follows.
> 
> http://ozlabs.org/ppc32-patches/
> 
> It changes the implementation of kernel_thread function
> to be performed with function call.

I prefer your previous patch. Paul hacked it a bit and will commit his
modified version.

Ben.

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Benjamin:

I can not figure out why my previous patch is more preferable than
later one.
Could you explain the reason of that please.

Regards,

>>
>>http://ozlabs.org/ppc32-patches/
>>
>>It changes the implementation of kernel_thread function
>>to be performed with function call.
> 
> 
> I prefer your previous patch. Paul hacked it a bit and will commit his
> modified version.
> 
> Ben.
> 

-- 
Takeharu KATO

Re: boot time scheduling hile atomic

[Benjamin Herrenschmidt]

On Wed, 2005-03-16 at 23:07 +0900, Takeharu KATO wrote:
> Hi Benjamin:
> 
> I can not figure out why my previous patch is more preferable than
> later one.
> Could you explain the reason of that please.

There may be other kernel bits calling syscall, it sounds saner to
always make sure we don't reschedule when in atomic (though it's dubious
that it's allowed to do syscalls in atomic regions, let's be safe
there).

In fact, we are thinking about consolidating the syscall return path and
the normal irq return path like we did on ppc64.

Ben.

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi Benjamin:

Benjamin Herrenschmidt wrote:

> 
> There may be other kernel bits calling syscall, it sounds saner to
> always make sure we don't reschedule when in atomic (though it's dubious
> that it's allowed to do syscalls in atomic regions, let's be safe
> there).
> 
> In fact, we are thinking about consolidating the syscall return path and
> the normal irq return path like we did on ppc64.
> 
Thkans, I make sense.
Would you send me Paul's patch?

Regards,

-- 
Takeharu KATO
Fujitsu Limited
Email:kato.takeharu at jp.fujitsu.com

RE: boot time scheduling hile atomic

[Rune Torgersen]

Any patch forthcomming soon?=20

> -----Original Message-----
> From: linuxppc-dev-bounces@ozlabs.org=20
> [mailto:linuxppc-dev-bounces@ozlabs.org] On Behalf Of=20
> Benjamin Herrenschmidt
> Sent: Wednesday, March 16, 2005 16:29
> To: Takeharu KATO
> Cc: linuxppc-dev list
> Subject: Re: boot time scheduling hile atomic
>=20
> On Wed, 2005-03-16 at 23:07 +0900, Takeharu KATO wrote:
> > Hi Benjamin:
> >=20
> > I can not figure out why my previous patch is more preferable than
> > later one.
> > Could you explain the reason of that please.
>=20
> There may be other kernel bits calling syscall, it sounds saner to
> always make sure we don't reschedule when in atomic (though=20
> it's dubious
> that it's allowed to do syscalls in atomic regions, let's be safe
> there).
>=20
> In fact, we are thinking about consolidating the syscall=20
> return path and
> the normal irq return path like we did on ppc64.
>=20
> Ben.
>=20
>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>=20
>=20

Re: boot time scheduling hile atomic

[Takeharu KATO]

Hi

IMHO, it is more preferable to change kernel thread creation mechanism adapting to
other architecture(in stead of tweaking exception handling) to solve the issue at least.
Of course, cleaning up the exception codes to adapt to ppc64  is also preferable from
code maintenance view.

Anyway, please try to apply the patch as follow:
http://ozlabs.org/ppc32-patches/patch.pl?id=318

Rune Torgersen wrote:
> Any patch forthcomming soon? 
> 
> 
>>-----Original Message-----
>>From: linuxppc-dev-bounces@ozlabs.org 
>>[mailto:linuxppc-dev-bounces@ozlabs.org] On Behalf Of 
>>Benjamin Herrenschmidt
>>Sent: Wednesday, March 16, 2005 16:29
>>To: Takeharu KATO
>>Cc: linuxppc-dev list
>>Subject: Re: boot time scheduling hile atomic
>>
>>On Wed, 2005-03-16 at 23:07 +0900, Takeharu KATO wrote:
>>
>>>Hi Benjamin:
>>>
>>>I can not figure out why my previous patch is more preferable than
>>>later one.
>>>Could you explain the reason of that please.
>>
>>There may be other kernel bits calling syscall, it sounds saner to
>>always make sure we don't reschedule when in atomic (though 
>>it's dubious
>>that it's allowed to do syscalls in atomic regions, let's be safe
>>there).
>>
>>In fact, we are thinking about consolidating the syscall 
>>return path and
>>the normal irq return path like we did on ppc64.
>>
>>Ben.
>>
>>
>>_______________________________________________
>>Linuxppc-dev mailing list
>>Linuxppc-dev@ozlabs.org
>>https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>
>>
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>