Re: [PATCH 1/1] sched: prevent divide by zero error in cpu_avg_load_per_task

Ingo Molnar
2008-11-27T09:30:46Z

* Steven Rostedt <rostedt@goodmis.org> wrote:

> From: Steven Rostedt <rostedt@goodmis.org>
>
> Impact: fix to divide by zero
>
> While testing the branch profiler, I hit this crash:
>
> divide error: 0000 [#1] PREEMPT SMP
> [...]
> Call Trace:
> <IRQ> <0> [<ffffffff8024fd43>] find_busiest_group+0x3e5/0xcaa
> [<ffffffff8025da75>] rebalance_domains+0x2da/0xa21

> The code for cpu_avg_load_per_task has:
>
> if (rq->nr_running)
> rq->avg_load_per_task = rq->load.weight / rq->nr_running;
>
> The runqueue lock is not held here, and there is nothing that
> prevents the rq->nr_running from going to zero after it passes the
> if condition.
>
> The branch profiler simply made the race window bigger.
>
> This patch saves off the rq->nr_running to a local variable and uses
> that for both the condition and the division.

good catch! Applied to tip/sched/urgent, thanks Steve!

the rebalancer scans remote runqueues without holding the runqueue
lock for performance reasons, so nr_running indeed has to be loaded
into a local variable here.

I think it could hit anywhere upstream as well, even without the
branch tracer: depends on the register pressure and GCC's choices for
reloading that register. If say FRAME_POINTER is enabled and we are
running with some more agressive optimization or just an older/suckier
GCC that does a spurious reload, then this might happen too.

it's not a classic race, it's an SMP bug that will either trigger or
not trigger at all, depending on compiler behavior.

Ingo