Re: [PATCH] Fix WARN_ON / WARN_ON_ONCE regression

Tim Chen
2006-10-04T21:32:52Z

On Wed, 2006-10-04 at 10:34 -0700, Andrew Morton wrote:

>
> I wonder if we just got unlucky and that particular benchmark with that
> particular kernel build just happens to reach the cache system's
> associativity threshold, and this one extra cacheline took it over the
> edge. Or something.
>

I took a look at the "L2 cache lines evicted" data from emon. It
increases by 17 million per cpu core during the 20 second tbench run
with the WARN_ON_ONCE patch, along with 11 million increase in "L2 cache
miss". So perhaps the __warn_once variable got evicted. No theory at
least point on how it got evicted.

Also did an experiment to locate the callsite causing cache miss with
the original WARN_ON_ONCE patch. I turned off WARN_ON_ONCE one by one in
kernel/softirq.c where almost all call for WARN_ON_ONCE originated. I
found that with each turnoff of WARN_ON_ONCE, the L2 cache miss go down
by 1 to 2 million, L2 cache line eviction goes down by 2 to 3 million.

One particular WARN_ON_ONCE in local_bh_enable contributed the most: 6
million L2 cache miss and 10 million L2 cache line evictions. That
particular WARN_ON_ONCE also has the most effect on tbench throughput
(accounting for 70% of the tbench throughput reduction).

-- Code snippet --
void local_bh_enable(void)
{
#ifdef CONFIG_TRACE_IRQFLAGS
unsigned long flags;

WARN_ON_ONCE(in_irq());
#endif
WARN_ON_ONCE(irqs_disabled()); <------- most degradation here.
CONFIG_TRACE_IRQFLAGS off

#ifdef CONFIG_TRACE_IRQFLAGS
local_irq_save(flags);
#endif
/*

Thanks.

Tim