Possible sandybridge livelock issue

Possible sandybridge livelock issue

[James Bottomley]

We've just come off a large round of debugging a kswapd problem over on
linux-mm:

http://marc.info/?t=130392066000001

The upshot was that kswapd wasn't being allowed to sleep (which we're
now fixing).  However, in spite of intensive efforts, the actual hang
was only reproducible on sandybridge laptops.

When the hang occurred, kswapd basically pegged one core in 100% system
time.  This looks like there's something specific to sandybridge that
causes this type of bad interaction.  I was wondering if it could be
something to to with a scheduling problem in turbo mode?  Once kswapd
goes flat out, the core its on will kick into turbo mode, which causes
it to get preferentially scheduled there, leading to the live lock.

The only evidence I have to support this theory is that when I reproduce
the problem with PREEMPT, the core pegs at 100% system time and stays
there even if I turn off the load.  However, if I can execute work that
causes kswapd to be kicked off the core it's running on, it will calm
back down and go to sleep.

James

Possible sandybridge livelock issue

[James Bottomley]

We've just come off a large round of debugging a kswapd problem over on
linux-mm:

http://marc.info/?t=130392066000001

The upshot was that kswapd wasn't being allowed to sleep (which we're
now fixing).  However, in spite of intensive efforts, the actual hang
was only reproducible on sandybridge laptops.

When the hang occurred, kswapd basically pegged one core in 100% system
time.  This looks like there's something specific to sandybridge that
causes this type of bad interaction.  I was wondering if it could be
something to to with a scheduling problem in turbo mode?  Once kswapd
goes flat out, the core its on will kick into turbo mode, which causes
it to get preferentially scheduled there, leading to the live lock.

The only evidence I have to support this theory is that when I reproduce
the problem with PREEMPT, the core pegs at 100% system time and stays
there even if I turn off the load.  However, if I can execute work that
causes kswapd to be kicked off the core it's running on, it will calm
back down and go to sleep.

James


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[Andi Kleen]

James Bottomley <James.Bottomley@HansenPartnership.com> writes:
>
> When the hang occurred, kswapd basically pegged one core in 100% system
> time.  This looks like there's something specific to sandybridge that
> causes this type of bad interaction.  I was wondering if it could be
> something to to with a scheduling problem in turbo mode?  Once kswapd
> goes flat out, the core its on will kick into turbo mode, which causes
> it to get preferentially scheduled there, leading to the live lock.

Sounds unlikely to me.

Turbo mode does not affect the scheduler and the cores are (reasonably) 
independent.


> The only evidence I have to support this theory is that when I reproduce
> the problem with PREEMPT, the core pegs at 100% system time and stays
> there even if I turn off the load.  However, if I can execute work that
> causes kswapd to be kicked off the core it's running on, it will calm
> back down and go to sleep.

Turbo mode just makes the CPU faster, but it should not change 
the scheduler decisions.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only

Re: Possible sandybridge livelock issue

[Andi Kleen]

James Bottomley <James.Bottomley@HansenPartnership.com> writes:
>
> When the hang occurred, kswapd basically pegged one core in 100% system
> time.  This looks like there's something specific to sandybridge that
> causes this type of bad interaction.  I was wondering if it could be
> something to to with a scheduling problem in turbo mode?  Once kswapd
> goes flat out, the core its on will kick into turbo mode, which causes
> it to get preferentially scheduled there, leading to the live lock.

Sounds unlikely to me.

Turbo mode does not affect the scheduler and the cores are (reasonably) 
independent.


> The only evidence I have to support this theory is that when I reproduce
> the problem with PREEMPT, the core pegs at 100% system time and stays
> there even if I turn off the load.  However, if I can execute work that
> causes kswapd to be kicked off the core it's running on, it will calm
> back down and go to sleep.

Turbo mode just makes the CPU faster, but it should not change 
the scheduler decisions.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[Christoph Lameter]

On Fri, 13 May 2011, Andi Kleen wrote:

> Turbo mode just makes the CPU faster, but it should not change
> the scheduler decisions.

I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).

Re: Possible sandybridge livelock issue

[Christoph Lameter]

On Fri, 13 May 2011, Andi Kleen wrote:

> Turbo mode just makes the CPU faster, but it should not change
> the scheduler decisions.

I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[Andi Kleen]

Christoph Lameter <cl@linux.com> writes:

> On Fri, 13 May 2011, Andi Kleen wrote:
>
>> Turbo mode just makes the CPU faster, but it should not change
>> the scheduler decisions.
>
> I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
> 2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).

It still doesn't make a lot of sense to blame the CPU for this.
This is just not the level how CPU problems would likely appear.

Can you figure out better what the kswapd is doing?

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only

Re: Possible sandybridge livelock issue

[Andi Kleen]

Christoph Lameter <cl@linux.com> writes:

> On Fri, 13 May 2011, Andi Kleen wrote:
>
>> Turbo mode just makes the CPU faster, but it should not change
>> the scheduler decisions.
>
> I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
> 2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).

It still doesn't make a lot of sense to blame the CPU for this.
This is just not the level how CPU problems would likely appear.

Can you figure out better what the kswapd is doing?

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[James Bottomley]

On Fri, 2011-05-13 at 11:23 -0700, Andi Kleen wrote:
> Christoph Lameter <cl@linux.com> writes:
> 
> > On Fri, 13 May 2011, Andi Kleen wrote:
> >
> >> Turbo mode just makes the CPU faster, but it should not change
> >> the scheduler decisions.
> >
> > I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
> > 2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).
> 
> It still doesn't make a lot of sense to blame the CPU for this.
> This is just not the level how CPU problems would likely appear.
> 
> Can you figure out better what the kswapd is doing?

We have ... it was the thread in the first email.  We don't need a fix
for the kswapd issue, what we're warning about is a potential
sandybridge problem.

The facts are that only sandybridge systems livelocked in the kswapd
problem ... no other systems could reproduce it, although they did see
heavy CPU time accumulate to kswapd.  And this is with a gang of mm
people trying to reproduce the problem on non-sandybridge systems.

On the sandybridge systems that livelocked, it was sometimes possible to
release the lock by pushing kswapd off the cpu it was hogging.

If you think the theory about why this happend to be wrong, fine ...
come up with another one.  The facts are as above and only sandybridge
systems seem to be affected.

James

Re: Possible sandybridge livelock issue

[James Bottomley]

On Fri, 2011-05-13 at 11:23 -0700, Andi Kleen wrote:
> Christoph Lameter <cl@linux.com> writes:
> 
> > On Fri, 13 May 2011, Andi Kleen wrote:
> >
> >> Turbo mode just makes the CPU faster, but it should not change
> >> the scheduler decisions.
> >
> > I also have similar issues with Sandybridge on Ubuntu 11.04 and kernels
> > 2.6.38 as well as 2.6.39 (standard ubuntu kernel configs).
> 
> It still doesn't make a lot of sense to blame the CPU for this.
> This is just not the level how CPU problems would likely appear.
> 
> Can you figure out better what the kswapd is doing?

We have ... it was the thread in the first email.  We don't need a fix
for the kswapd issue, what we're warning about is a potential
sandybridge problem.

The facts are that only sandybridge systems livelocked in the kswapd
problem ... no other systems could reproduce it, although they did see
heavy CPU time accumulate to kswapd.  And this is with a gang of mm
people trying to reproduce the problem on non-sandybridge systems.

On the sandybridge systems that livelocked, it was sometimes possible to
release the lock by pushing kswapd off the cpu it was hogging.

If you think the theory about why this happend to be wrong, fine ...
come up with another one.  The facts are as above and only sandybridge
systems seem to be affected.

James


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[Ingo Molnar]

* James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > Can you figure out better what the kswapd is doing?
> 
> We have ... it was the thread in the first email.  We don't need a fix for 
> the kswapd issue, what we're warning about is a potential sandybridge 
> problem.
> 
> The facts are that only sandybridge systems livelocked in the kswapd problem 
> ... no other systems could reproduce it, although they did see heavy CPU time 
> accumulate to kswapd.  And this is with a gang of mm people trying to 
> reproduce the problem on non-sandybridge systems.
> 
> On the sandybridge systems that livelocked, it was sometimes possible to 
> release the lock by pushing kswapd off the cpu it was hogging.

It's not uncommon at all to see certain races (or even livelocks) only with the 
latest and greatest CPUs.

I have a first-gen CPU system that when i got it a couple of years ago 
triggered like a dozen Linux kernel races and bugs possible theoretically on 
all other CPUs but not reported on any other Linux system up to that point, 
*ever* - and some of those bugs were many years old.

> If you think the theory about why this happend to be wrong, fine ... come up 
> with another one.  The facts are as above and only sandybridge systems seem 
> to be affected.

I can see at least four other plausible hypotheses, all matching the facts as 
you laid them out:

 - i could be a bug/race in the kswapd code.

 - it could be that the race window needs a certain level of instruction 
   parallelism - which occurs with a higher likelyhood on Sandybridge.

 - it could be that Sandybridge CPUs keep dirty cachelines owned a bit longer 
   than other CPUs, making an existing livelock bug in the kernel code easier 
   to trigger.

 - a hardware bug: if cacheline ownership is not arbitrated between 
   nodes/cpus/cores fairly (enough) and a specific CPU can monopolize a 
   cacheline for a very long time if only it keeps modifying it in an 
   aggressive enough kswapd loop.

Note, since each of these hypotheses has a specific non-zero chance of being 
the objective truth, your hypothesis might in the end turn out to be the right 
one and might turn into a proven scientific theory: CPU and scheduler bugs do 
happen after all.

The other hypotheses i outlined have non-zero chances as well: kswapd bugs do 
happen as well and various CPU timing differences do tend to occur as well.

But above you seem to be confused about how supporting facts and hypotheses 
relate to each other: you seemed to imply that because your facts support your 
hypothesis the ball is somehow on the other side. As things stand now we 
clearly need more facts, to exclude more of the many possibilities.

So i wanted to clear up these basics of science first, before any of us wastes 
too much time on writing mails and such. Oh ... never mind ;-)

Thanks,

	Ingo

Re: Possible sandybridge livelock issue

[Ingo Molnar]

* James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > Can you figure out better what the kswapd is doing?
> 
> We have ... it was the thread in the first email.  We don't need a fix for 
> the kswapd issue, what we're warning about is a potential sandybridge 
> problem.
> 
> The facts are that only sandybridge systems livelocked in the kswapd problem 
> ... no other systems could reproduce it, although they did see heavy CPU time 
> accumulate to kswapd.  And this is with a gang of mm people trying to 
> reproduce the problem on non-sandybridge systems.
> 
> On the sandybridge systems that livelocked, it was sometimes possible to 
> release the lock by pushing kswapd off the cpu it was hogging.

It's not uncommon at all to see certain races (or even livelocks) only with the 
latest and greatest CPUs.

I have a first-gen CPU system that when i got it a couple of years ago 
triggered like a dozen Linux kernel races and bugs possible theoretically on 
all other CPUs but not reported on any other Linux system up to that point, 
*ever* - and some of those bugs were many years old.

> If you think the theory about why this happend to be wrong, fine ... come up 
> with another one.  The facts are as above and only sandybridge systems seem 
> to be affected.

I can see at least four other plausible hypotheses, all matching the facts as 
you laid them out:

 - i could be a bug/race in the kswapd code.

 - it could be that the race window needs a certain level of instruction 
   parallelism - which occurs with a higher likelyhood on Sandybridge.

 - it could be that Sandybridge CPUs keep dirty cachelines owned a bit longer 
   than other CPUs, making an existing livelock bug in the kernel code easier 
   to trigger.

 - a hardware bug: if cacheline ownership is not arbitrated between 
   nodes/cpus/cores fairly (enough) and a specific CPU can monopolize a 
   cacheline for a very long time if only it keeps modifying it in an 
   aggressive enough kswapd loop.

Note, since each of these hypotheses has a specific non-zero chance of being 
the objective truth, your hypothesis might in the end turn out to be the right 
one and might turn into a proven scientific theory: CPU and scheduler bugs do 
happen after all.

The other hypotheses i outlined have non-zero chances as well: kswapd bugs do 
happen as well and various CPU timing differences do tend to occur as well.

But above you seem to be confused about how supporting facts and hypotheses 
relate to each other: you seemed to imply that because your facts support your 
hypothesis the ball is somehow on the other side. As things stand now we 
clearly need more facts, to exclude more of the many possibilities.

So i wanted to clear up these basics of science first, before any of us wastes 
too much time on writing mails and such. Oh ... never mind ;-)

Thanks,

	Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Possible sandybridge livelock issue

[Ingo Molnar]

* James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> We've just come off a large round of debugging a kswapd problem over on
> linux-mm:
> 
> http://marc.info/?t=130392066000001
> 
> The upshot was that kswapd wasn't being allowed to sleep (which we're
> now fixing).  However, in spite of intensive efforts, the actual hang
> was only reproducible on sandybridge laptops.
> 
> When the hang occurred, kswapd basically pegged one core in 100% system
> time.  This looks like there's something specific to sandybridge that
> causes this type of bad interaction.  I was wondering if it could be
> something to to with a scheduling problem in turbo mode?  Once kswapd
> goes flat out, the core its on will kick into turbo mode, which causes
> it to get preferentially scheduled there, leading to the live lock.

There's no explicit 'schedule Sandybridge differently' logic in the scheduler.

Thus turbo mode can only affect scheduling by executing code faster. Executing 
faster *does* mean more scheduling on that CPU: it's faster to do work so it's 
faster back to idle again.

I.e. i can see Sandybridge being special only due to timing and performance 
differences.

> The only evidence I have to support this theory is that when I reproduce the 
> problem with PREEMPT, the core pegs at 100% system time and stays there even 
> if I turn off the load.  However, if I can execute work that causes kswapd to 
> be kicked off the core it's running on, it will calm back down and go to 
> sleep.

At first sight this looks like some sort of kswapd problem: if you put kswapd 
into TASK_*INTERRUPTIBLE and schedule() it then the scheduler won't keep it 
running, on Sandybridge or elsewhere. The scheduler can't magically make kswapd 
runnable unless there's some big bug in it. So you first need to examine why 
kswapd never schedules to idle.

Thanks,

	Ingo

Re: Possible sandybridge livelock issue

[Ingo Molnar]

* James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> We've just come off a large round of debugging a kswapd problem over on
> linux-mm:
> 
> http://marc.info/?t=130392066000001
> 
> The upshot was that kswapd wasn't being allowed to sleep (which we're
> now fixing).  However, in spite of intensive efforts, the actual hang
> was only reproducible on sandybridge laptops.
> 
> When the hang occurred, kswapd basically pegged one core in 100% system
> time.  This looks like there's something specific to sandybridge that
> causes this type of bad interaction.  I was wondering if it could be
> something to to with a scheduling problem in turbo mode?  Once kswapd
> goes flat out, the core its on will kick into turbo mode, which causes
> it to get preferentially scheduled there, leading to the live lock.

There's no explicit 'schedule Sandybridge differently' logic in the scheduler.

Thus turbo mode can only affect scheduling by executing code faster. Executing 
faster *does* mean more scheduling on that CPU: it's faster to do work so it's 
faster back to idle again.

I.e. i can see Sandybridge being special only due to timing and performance 
differences.

> The only evidence I have to support this theory is that when I reproduce the 
> problem with PREEMPT, the core pegs at 100% system time and stays there even 
> if I turn off the load.  However, if I can execute work that causes kswapd to 
> be kicked off the core it's running on, it will calm back down and go to 
> sleep.

At first sight this looks like some sort of kswapd problem: if you put kswapd 
into TASK_*INTERRUPTIBLE and schedule() it then the scheduler won't keep it 
running, on Sandybridge or elsewhere. The scheduler can't magically make kswapd 
runnable unless there's some big bug in it. So you first need to examine why 
kswapd never schedules to idle.

Thanks,

	Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>