Re: upstream regression (IO-APIC?)

Alok Kataria
2008-11-03T17:49:44Z

On Sun, 2008-11-02 at 12:24 -0800, Bartlomiej Zolnierkiewicz wrote:
> On Sunday 02 November 2008, Bartlomiej Zolnierkiewicz wrote:
> > On Thursday 30 October 2008, Robert Hancock wrote:
> > > Bartlomiej Zolnierkiewicz wrote:
> > > > The current Linus tree as of commit e946217e4fdaa67681bbabfa8e6b18641921f750
> > > > is broken for me. I get either the following panic (see log from qemu below)
> > > > or lost IRQs on ATA init... Is this a known issue?
> > > >
> > > > PS The tree that I used before and was supposedly good (sorry, I'm too tired
> > > > to verify it now) had commit 57f8f7b60db6f1ed2c6918ab9230c4623a9dbe37 at head.
> >
> > Unfortunately 57f8f7b60db6f1ed2c6918ab9230c4623a9dbe37 (v2.6.28-rc1)
> > is also bad. Bisecting it further was a real pain (i.e. I hit broken
> > build with x86 irqbalance changes, broken build with netfilter nat
> > changes and jbd journal problem). In the end it turned out that 2.6.27
> > is bad too! However with 2.6.27 the panic occurs only once per several
> > attempts and if there is no panic kernel boots normally (no lost IRQs).
> >
> > [...]
> >
> > I finally managed to narrow it down to change making x86 use tsc_khz
> > for loops_per_jiffy -- commit 3da757daf86e498872855f0b5e101f763ba79499
> > ("x86: use cpu_khz for loops_per_jiffy calculation"). This approach
> > seems too simplistic (as I see now Arjan & Pavel expressed concerns
> > about it back when the patch was posted initially [1][2]). Also it
> > would probably be preferred to re-use existing preset_lpj variable
> > (just like KVM does it for similar purpose [3]) instead of adding a
> > lpj_tsc one and increasing complexity.
>
> It turned out that I can boot a kernel with different config with
> HZ == 250 just fine and switching to HZ == 1000 makes it fail.
>
>
> Looking into it some more:
>
> HZ == 250 kernel (good):
>
> Calibrating delay loop (skipped), value calculated using timer frequency.. 2986.79 BogoMIPS (lpj=5973580)
>
> HZ == 1000 kernel (bad):
>
> Calibrating delay loop (skipped), using tsc calculated value.. 2990.35 BogoMIPS (lpj=1495176)
>
> HZ == 1000 kernel with hackyfix (good):
>
> Calibrating delay using timer specific routine.. 3016.68 BogoMIPS (lpj=6033376)
>
>
> Argggh... lpj is used for udelay() & friends so this bug is quite
> dangerous (since udelay() & friends are used for hardware delays)...
>
> [ The commit works for HZ == 250 because it does tsc_khz * 1000 / HZ,
> tsc_khz * 4 => lpj assumption holds true and there is no frequency
> scaling at boot. ]
>
> The quick fix would be to replace 1000 / HZ by the magic number "4"

That's not right, the magic number 4 thing would not be correct.
On one of my systems for eg, i get this in dmesg

Detected 2010.400 MHz processor.
...
Calibrating delay using timer specific routine.. 4022.47 BogoMIPS
(lpj=2011235)

This is with an earlier kernel, the HZ value is 1000. And the lpj value
that we get from the calculation of (tsc_khz * 1000)/HZ is correct in
this case. And on all the systems that i have checked this assumption
holds true.

One of the things that i suspect is that you are not using delay_tsc in
this case, i.e. tsc is not used for delay which is causing that panic

can you please try the patch below on your system ?

[test-patch]

Index: linux-2.6/arch/x86/kernel/tsc.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/tsc.c 2008-10-15 10:51:14.000000000 -0700
+++ linux-2.6/arch/x86/kernel/tsc.c 2008-11-03 09:43:01.000000000 -0800
@@ -847,10 +847,6 @@
cpu_khz = calibrate_cpu();
#endif

- lpj = ((u64)tsc_khz * 1000);
- do_div(lpj, HZ);
- lpj_fine = lpj;
-
printk("Detected %lu.%03lu MHz processor.\n",
(unsigned long)cpu_khz / 1000,
(unsigned long)cpu_khz % 1000);
@@ -871,6 +867,10 @@
tsc_disabled = 0;

use_tsc_delay();
+ lpj = ((u64)tsc_khz * 1000);
+ do_div(lpj, HZ);
+ lpj_fine = lpj;
+
/* Check and install the TSC clocksource */
dmi_check_system(bad_tsc_dmi_table);
check_system_tsc_reliable();

> but the major question is whether can we reliably depend on the tsc_khz
> for lpj?

If the patch above doesn't help, I think the answer to your question is
- not on some particular hardware, but we would know.
Btw, what h/w are you running this on ?

Thanks,
Alok
>
> Thanks,
> Bart