[BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Thomas Lange]

Commit 305e683 introduced a wrap bug that causes task scheduling to fail
after sched_clock() wrap. On a 1000 HZ system with 32bit jiffies, this
occurs after 49.7 days.

Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.

Symptoms include one task getting all available cpu time while others get
_none_. Setting niceness seems to make things even worse. Running this code
in a new process after wrap completely lock up user space, thus triggering a
watchdog reboot:
{ nice(1); while(1); }

To reproduce bug in reasonable time, one can up HZ. With 16000 HZ, bug occurs
after 3.1 days.
Modifying sched_clock() to wrap when jiffies does triggers bug after 5 mins.

The basic problem seems to be that rq->clock_task get stuck forever with a
really high value when rq->clock starts over from 0.

This fix solves that problem:

diff --git a/kernel/sched.c b/kernel/sched.c
index d40d662..883448f 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -657,6 +657,8 @@ inline void update_rq_clock(struct rq *rq)
        if (!rq->skip_clock_update)
                rq->clock = sched_clock_cpu(cpu_of(rq));
        irq_time = irq_time_cpu(cpu);
+       if (rq->clock < rq->clock_task)
+               rq->clock_task = 0;
        if (rq->clock - irq_time > rq->clock_task)
                rq->clock_task = rq->clock - irq_time;

I can create a proper patch if the above is acceptable.

A more appropriate solution would perhaps be to pull some additional sched
commits into stable branch, like fe44d62 and friends. I don't know enough
about scheduler internals to tell.

All tests were performed on mips32 systems, but all systems with 32bit
jiffies should be affected.

/Thomas

Re: [BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Greg KH]

On Sun, Jun 24, 2012 at 07:32:11PM +0200, Thomas Lange wrote:
> Commit 305e683 introduced a wrap bug that causes task scheduling to fail
> after sched_clock() wrap. On a 1000 HZ system with 32bit jiffies, this
> occurs after 49.7 days.
> 
> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.

Is this an issue in 3.5-rc3?

2.6.35-stable really isn't maintained anymore, I would suggest upgrading
to a more modern kernel version.

thanks,

greg k-h

Re: [BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Peter Zijlstra]

On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.

Ok, so nobody cares about that.. what does something recent like 3.5-rc4
do?

If that's fixed, find the patch that fixes it. If not, we'll have a
look.

Re: [BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Peter Zijlstra]

On Mon, 2012-06-25 at 11:45 +0200, Peter Zijlstra wrote:
> On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
> > Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
> 
> Ok, so nobody cares about that.. what does something recent like 3.5-rc4
> do?
> 
> If that's fixed, find the patch that fixes it. If not, we'll have a
> look.


If anything, I think something like the below ought to cure things.

---
 kernel/sched/clock.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index c685e31..a1a128a 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -74,7 +74,7 @@
  */
 unsigned long long __attribute__((weak)) sched_clock(void)
 {
-	return (unsigned long long)(jiffies - INITIAL_JIFFIES)
+	return (unsigned long long)(get_jiffies_64() - INITIAL_JIFFIES)
 					* (NSEC_PER_SEC / HZ);
 }
 EXPORT_SYMBOL_GPL(sched_clock);

Re: [BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Thomas Lange]

On 2012-06-25 12:38, Peter Zijlstra wrote:
> On Mon, 2012-06-25 at 11:45 +0200, Peter Zijlstra wrote:
>> On Sun, 2012-06-24 at 19:32 +0200, Thomas Lange wrote:
>>> Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
>>
>> Ok, so nobody cares about that.. what does something recent like 3.5-rc4
>> do?

I haven't tested yet. I will try to find the time to do that.

>> If that's fixed, find the patch that fixes it. If not, we'll have a
>> look.

Commit fe44d62 removed the if-clause that caused the problem. Most likely, this
solved the problem.

> 
> 
> If anything, I think something like the below ought to cure things.

I agree. Either we make sure that sched_clock() never wraps or if it wraps,
that it wraps shortly after boot to catch problems early.

> 
> ---
>  kernel/sched/clock.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
> index c685e31..a1a128a 100644
> --- a/kernel/sched/clock.c
> +++ b/kernel/sched/clock.c
> @@ -74,7 +74,7 @@
>   */
>  unsigned long long __attribute__((weak)) sched_clock(void)
>  {
> -	return (unsigned long long)(jiffies - INITIAL_JIFFIES)
> +	return (unsigned long long)(get_jiffies_64() - INITIAL_JIFFIES)
>  					* (NSEC_PER_SEC / HZ);
>  }
>  EXPORT_SYMBOL_GPL(sched_clock);

/Thomas

Re: [BUG] sched: clock wrap bug in 2.6.35-stable kills scheduling

[Peter Zijlstra]

On Mon, 2012-06-25 at 20:49 +0200, Thomas Lange wrote:
> Commit fe44d62 removed the if-clause that caused the problem. Most likely, this
> solved the problem.

Ah, probably. IIRC we also fixed the short wrap of those arm clocks
since.

> > If anything, I think something like the below ought to cure things.
> 
> I agree. Either we make sure that sched_clock() never wraps or if it wraps,
> that it wraps shortly after boot to catch problems early.

Right, sched_clock() should only ever wrap on the full u64.