Re: Patch Upstream: cputimer: Cure lock inversion

Re: Patch Upstream: cputimer: Cure lock inversion

[Greg KH]

On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
> commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Mon, 17 Oct 2011 11:50:30 +0200
> Subject: cputimer: Cure lock inversion
> 
> There's a lock inversion between the cputimer->lock and rq->lock;
> notably the two callchains involved are:
> 
>  update_rlimit_cpu()
>    sighand->siglock
>    set_process_cpu_timer()
>      cpu_timer_sample_group()
>        thread_group_cputimer()
>          cputimer->lock
>          thread_group_cputime()
>            task_sched_runtime()
>              ->pi_lock
>              rq->lock
> 
>  scheduler_tick()
>    rq->lock
>    task_tick_fair()
>      update_curr()
>        account_group_exec()
>          cputimer->lock
> 
> Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
> the second one is keeping up-to-date.
> 
> This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
> SMP accounting oddities").

There is no such patch in Linus's tree that I can find.  So, what
problem is this really trying to cure here and what kernel did it show
up in?

confused,

greg k-h

Re: Patch Upstream: cputimer: Cure lock inversion

[Josh Boyer]

On Wed, Oct 19, 2011 at 11:49 AM, Greg KH <greg@kroah.com> wrote:
> On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
>> commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
>> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
>> Date: Mon, 17 Oct 2011 11:50:30 +0200
>> Subject: cputimer: Cure lock inversion
>>
>> There's a lock inversion between the cputimer->lock and rq->lock;
>> notably the two callchains involved are:
>>
>>  update_rlimit_cpu()
>>    sighand->siglock
>>    set_process_cpu_timer()
>>      cpu_timer_sample_group()
>>        thread_group_cputimer()
>>          cputimer->lock
>>          thread_group_cputime()
>>            task_sched_runtime()
>>              ->pi_lock
>>              rq->lock
>>
>>  scheduler_tick()
>>    rq->lock
>>    task_tick_fair()
>>      update_curr()
>>        account_group_exec()
>>          cputimer->lock
>>
>> Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
>> the second one is keeping up-to-date.
>>
>> This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
>> SMP accounting oddities").
>
> There is no such patch in Linus's tree that I can find.  So, what
> problem is this really trying to cure here and what kernel did it show
> up in?

Uh...

bcd5cff7216f9b2de0a148cc355eac199dc6f1cf is the upstream commit (post -rc10).

This thread covers the conversation (it's long):

http://thread.gmane.org/gmane.linux.kernel/1199406/focus=1204676

josh

Re: Patch Upstream: cputimer: Cure lock inversion

[Greg KH]

On Wed, Oct 19, 2011 at 11:55:01AM -0400, Josh Boyer wrote:
> On Wed, Oct 19, 2011 at 11:49 AM, Greg KH <greg@kroah.com> wrote:
> > On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
> >> commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
> >> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> >> Date: Mon, 17 Oct 2011 11:50:30 +0200
> >> Subject: cputimer: Cure lock inversion
> >>
> >> There's a lock inversion between the cputimer->lock and rq->lock;
> >> notably the two callchains involved are:
> >>
> >>  update_rlimit_cpu()
> >>    sighand->siglock
> >>    set_process_cpu_timer()
> >>      cpu_timer_sample_group()
> >>        thread_group_cputimer()
> >>          cputimer->lock
> >>          thread_group_cputime()
> >>            task_sched_runtime()
> >>              ->pi_lock
> >>              rq->lock
> >>
> >>  scheduler_tick()
> >>    rq->lock
> >>    task_tick_fair()
> >>      update_curr()
> >>        account_group_exec()
> >>          cputimer->lock
> >>
> >> Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
> >> the second one is keeping up-to-date.
> >>
> >> This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
> >> SMP accounting oddities").
> >
> > There is no such patch in Linus's tree that I can find.  So, what
> > problem is this really trying to cure here and what kernel did it show
> > up in?
> 
> Uh...
> 
> bcd5cff7216f9b2de0a148cc355eac199dc6f1cf is the upstream commit (post -rc10).

No, I understand that this is the commit I just referenced.

I'm talking about the "This problem was introduced..." line in the
commit.  I want to find out what was the original problem that this
patch is fixing, to determine how far back in the -stable series I need
to backport this to.

The issue is that there is no e8abccb7193 ("posix-cpu-timers: Cure
SMP accounting oddities") commit that I can see in Linus's tree right
now.

> This thread covers the conversation (it's long):
> 
> http://thread.gmane.org/gmane.linux.kernel/1199406/focus=1204676

Ugh, I'll go dig, but help would be appreciated...

greg k-h

Re: Patch Upstream: cputimer: Cure lock inversion

[Josh Boyer]

On Wed, Oct 19, 2011 at 12:00 PM, Greg KH <greg@kroah.com> wrote:
> On Wed, Oct 19, 2011 at 11:55:01AM -0400, Josh Boyer wrote:
>> On Wed, Oct 19, 2011 at 11:49 AM, Greg KH <greg@kroah.com> wrote:
>> > On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
>> >> commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
>> >> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
>> >> Date: Mon, 17 Oct 2011 11:50:30 +0200
>> >> Subject: cputimer: Cure lock inversion
>> >>
>> >> There's a lock inversion between the cputimer->lock and rq->lock;
>> >> notably the two callchains involved are:
>> >>
>> >>  update_rlimit_cpu()
>> >>    sighand->siglock
>> >>    set_process_cpu_timer()
>> >>      cpu_timer_sample_group()
>> >>        thread_group_cputimer()
>> >>          cputimer->lock
>> >>          thread_group_cputime()
>> >>            task_sched_runtime()
>> >>              ->pi_lock
>> >>              rq->lock
>> >>
>> >>  scheduler_tick()
>> >>    rq->lock
>> >>    task_tick_fair()
>> >>      update_curr()
>> >>        account_group_exec()
>> >>          cputimer->lock
>> >>
>> >> Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
>> >> the second one is keeping up-to-date.
>> >>
>> >> This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
>> >> SMP accounting oddities").
>> >
>> > There is no such patch in Linus's tree that I can find.  So, what
>> > problem is this really trying to cure here and what kernel did it show
>> > up in?
>>
>> Uh...
>>
>> bcd5cff7216f9b2de0a148cc355eac199dc6f1cf is the upstream commit (post -rc10).
>
> No, I understand that this is the commit I just referenced.
>
> I'm talking about the "This problem was introduced..." line in the
> commit.  I want to find out what was the original problem that this
> patch is fixing, to determine how far back in the -stable series I need
> to backport this to.

Ah, I misunderstood.

> The issue is that there is no e8abccb7193 ("posix-cpu-timers: Cure
> SMP accounting oddities") commit that I can see in Linus's tree right
> now.
>
>> This thread covers the conversation (it's long):
>>
>> http://thread.gmane.org/gmane.linux.kernel/1199406/focus=1204676
>
> Ugh, I'll go dig, but help would be appreciated...

My guess is they meant d670ec13178d0fd868

josh

Re: Patch Upstream: cputimer: Cure lock inversion

[Peter Zijlstra]

On Wed, 2011-10-19 at 12:03 -0400, Josh Boyer wrote:
> > Ugh, I'll go dig, but help would be appreciated...
> 
> My guess is they meant d670ec13178d0fd868
> 
Your guess is right, no idea how I messed that up, my git tree contains
both commits and they're similar but different version of the same
patch.

:-(

Re: Patch Upstream: cputimer: Cure lock inversion

[Greg KH]

On Wed, Oct 19, 2011 at 06:19:16PM +0200, Peter Zijlstra wrote:
> On Wed, 2011-10-19 at 12:03 -0400, Josh Boyer wrote:
> > > Ugh, I'll go dig, but help would be appreciated...
> > 
> > My guess is they meant d670ec13178d0fd868
> > 
> Your guess is right, no idea how I messed that up, my git tree contains
> both commits and they're similar but different version of the same
> patch.

No problem, just got confused here,

greg k-h

Re: Patch Upstream: cputimer: Cure lock inversion

[Thomas Gleixner]

On Wed, 19 Oct 2011, Greg KH wrote:

> On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
> > commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
> > From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > Date: Mon, 17 Oct 2011 11:50:30 +0200
> > Subject: cputimer: Cure lock inversion
> > 
> > There's a lock inversion between the cputimer->lock and rq->lock;
> > notably the two callchains involved are:
> > 
> >  update_rlimit_cpu()
> >    sighand->siglock
> >    set_process_cpu_timer()
> >      cpu_timer_sample_group()
> >        thread_group_cputimer()
> >          cputimer->lock
> >          thread_group_cputime()
> >            task_sched_runtime()
> >              ->pi_lock
> >              rq->lock
> > 
> >  scheduler_tick()
> >    rq->lock
> >    task_tick_fair()
> >      update_curr()
> >        account_group_exec()
> >          cputimer->lock
> > 
> > Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
> > the second one is keeping up-to-date.
> > 
> > This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
> > SMP accounting oddities").
> 
> There is no such patch in Linus's tree that I can find.  So, what
> problem is this really trying to cure here and what kernel did it show
> up in?

Oops. It's in 3.0.7

commit 249cf808ba1a0d403fe7c476a74b66e2bc0a8e53
Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date:   Thu Sep 1 12:42:04 2011 +0200

    posix-cpu-timers: Cure SMP wobbles
    
    commit d670ec13178d0fd8680e6742a2bc6e04f28f87d8 upstream.

and that patch introduced the above deadlock, which is cured by:

commit bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date:   Mon Oct 17 11:50:30 2011 +0200

    cputimer: Cure lock inversion

Thanks,

	tglx

Re: Patch Upstream: cputimer: Cure lock inversion

[Greg KH]

On Wed, Oct 19, 2011 at 09:41:52PM +0200, Thomas Gleixner wrote:
> On Wed, 19 Oct 2011, Greg KH wrote:
> 
> > On Wed, Oct 19, 2011 at 11:39:14AM -0400, Gregs git-bot wrote:
> > > commit: bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
> > > From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > Date: Mon, 17 Oct 2011 11:50:30 +0200
> > > Subject: cputimer: Cure lock inversion
> > > 
> > > There's a lock inversion between the cputimer->lock and rq->lock;
> > > notably the two callchains involved are:
> > > 
> > >  update_rlimit_cpu()
> > >    sighand->siglock
> > >    set_process_cpu_timer()
> > >      cpu_timer_sample_group()
> > >        thread_group_cputimer()
> > >          cputimer->lock
> > >          thread_group_cputime()
> > >            task_sched_runtime()
> > >              ->pi_lock
> > >              rq->lock
> > > 
> > >  scheduler_tick()
> > >    rq->lock
> > >    task_tick_fair()
> > >      update_curr()
> > >        account_group_exec()
> > >          cputimer->lock
> > > 
> > > Where the first one is enabling a CLOCK_PROCESS_CPUTIME_ID timer, and
> > > the second one is keeping up-to-date.
> > > 
> > > This problem was introduced by e8abccb7193 ("posix-cpu-timers: Cure
> > > SMP accounting oddities").
> > 
> > There is no such patch in Linus's tree that I can find.  So, what
> > problem is this really trying to cure here and what kernel did it show
> > up in?
> 
> Oops. It's in 3.0.7
> 
> commit 249cf808ba1a0d403fe7c476a74b66e2bc0a8e53
> Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date:   Thu Sep 1 12:42:04 2011 +0200
> 
>     posix-cpu-timers: Cure SMP wobbles
>     
>     commit d670ec13178d0fd8680e6742a2bc6e04f28f87d8 upstream.
> 
> and that patch introduced the above deadlock, which is cured by:
> 
> commit bcd5cff7216f9b2de0a148cc355eac199dc6f1cf
> Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date:   Mon Oct 17 11:50:30 2011 +0200
> 
>     cputimer: Cure lock inversion

Thanks for the explanation.  I should have it all straightened out now.

greg k-h