EHCI + APIC errors = no usb goodness

EHCI + APIC errors = no usb goodness

[Greg KH]

I've had a laptop here that has had some "issues" in the past (time
running double speed, XFree doesn't work, etc.)

Now I'm down to the last problem, USB doesn't work, which is a bit of a
pain for me :)

Anyway, below is the kernel log from 2.6.16-rc1-mm2 (contains the latest
acpi tree, which I thought might help out.)  This log is when I modprobe
ehci-hcd.  The interesting thing is the APIC error, and then the fact
that the interrupt screams off into oblivion.  I can provide early
kernel boot info too, if you think that might help out.

Any thoughts?

I _think_ a real old 2.6 kernel used to work on this box, so I'll do a
bit of searching to see if I can find one and then try bisecting from
there...

Andi, I know you have been doing some apic rework for x86-64 (which this
processor is, but I'm still running it in 32bit mode.)  Do you happen to
have any patch that I could test out?

thanks,

greg k-h


[   87.324320] ehci_hcd: block sizes: qh 128 qtd 96 itd 192 sitd 96
[   87.324356] ACPI: PCI Interrupt 0000:00:13.2[A] -> GSI 19 (level, low) -> IRQ 16
[   87.324371] ehci_hcd 0000:00:13.2: EHCI Host Controller
[   87.324379] ehci_hcd 0000:00:13.2: reset hcs_params 0x2408 dbg=0 cc=2 pcc=4 ordered !ppc ports=8
[   87.324385] ehci_hcd 0000:00:13.2: reset hcc_params a012 thresh 1 uframes 256/512/1024
[   87.324426] ehci_hcd 0000:00:13.2: capability 0001 at a0
[   87.324436] ehci_hcd 0000:00:13.2: MWI active
[   87.324589] drivers/usb/core/inode.c: creating file 'devices'
[   87.324594] drivers/usb/core/inode.c: creating file '001'
[   87.324598] ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 1
[   87.324613] ehci_hcd 0000:00:13.2: irq 16, io mem 0xfbdff000
[   87.324623] ehci_hcd 0000:00:13.2: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
[   87.324633] ehci_hcd 0000:00:13.2: init command 010009 (park)=0 ithresh=1 period=256 RUN
[   87.324643] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[   87.324682] usb usb1: default language 0x0409
[   87.324695] usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
[   87.324700] usb usb1: Product: EHCI Host Controller
[   87.324704] usb usb1: Manufacturer: Linux 2.6.16-rc1-mm2 ehci_hcd
[   87.324709] usb usb1: SerialNumber: 0000:00:13.2
[   87.324727] usb usb1: uevent
[   87.324764] usb usb1: device is self-powered
[   87.324769] usb usb1: configuration #1 chosen from 1 choice
[   87.324781] usb usb1: adding 1-0:1.0 (config #1, interface 0)
[   87.324793] usb 1-0:1.0: uevent
[   87.324818] hub 1-0:1.0: usb_probe_interface
[   87.324822] hub 1-0:1.0: usb_probe_interface - got id
[   87.324827] hub 1-0:1.0: USB hub found
[   87.324841] hub 1-0:1.0: 8 ports detected
[   87.324845] hub 1-0:1.0: standalone hub
[   87.324849] hub 1-0:1.0: no power switching (usb 1.0)
[   87.324854] hub 1-0:1.0: individual port over-current protection
[   87.324858] hub 1-0:1.0: Single TT
[   87.324863] hub 1-0:1.0: TT requires at most 8 FS bit times (666 ns)
[   87.324867] hub 1-0:1.0: power on to power good time: 20ms
[   87.324876] hub 1-0:1.0: local power source is good
[   87.406180] APIC error on CPU0: 00(40)
[   87.426282] drivers/usb/core/inode.c: creating file '001'
[   87.426333] hub 1-0:1.0: state 5 ports 8 chg 0000 evt 0000
[   87.712002] APIC error on CPU0: 40(40)
[   87.774743] irq 16: nobody cared (try booting with the "irqpoll" option)
[   87.774748]  <c013e14a> __report_bad_irq+0x2a/0x90   <c013dae9> handle_IRQ_event+0x39/0x70
[   87.774759]  <c013e270> note_interrupt+0xa0/0x100   <c013dbd4> __do_IRQ+0xb4/0xc0
[   87.774767]  <c0105709> do_IRQ+0x19/0x30   <c0103a46> common_interrupt+0x1a/0x20
[   87.774775] handlers:
[   87.774777] [<dc83bc40>] (usb_hcd_irq+0x0/0x70 [usbcore])
[   87.774796] Disabling IRQ #16

Re: EHCI + APIC errors = no usb goodness

[Pete Zaitcev]

On Mon, 23 Jan 2006 13:04:43 -0800, Greg KH <gregkh@suse.de> wrote:

> Now I'm down to the last problem, USB doesn't work, which is a bit of a
> pain for me :)

> [   87.406180] APIC error on CPU0: 00(40)
> [   87.426282] drivers/usb/core/inode.c: creating file '001'
> [   87.426333] hub 1-0:1.0: state 5 ports 8 chg 0000 evt 0000
> [   87.712002] APIC error on CPU0: 40(40)
> [   87.774743] irq 16: nobody cared (try booting with the "irqpoll" option)

Why do you even enable APIC on an old laptop? We tried it a few times,
it's just not possible. I'd say, about one in ten to one in five of 2002
vintage laptops will not even boot, let alone work when APIC is enabled.
Some of them are well known, like Dell Lattitude 610. I expect this to
change with dual-core laptops, but for now the rule is: UP kernel,
No APIC, for distro kernels at least.

-- Pete

Re: EHCI + APIC errors = no usb goodness

[Greg KH]

On Mon, Jan 23, 2006 at 01:25:54PM -0800, Pete Zaitcev wrote:
> On Mon, 23 Jan 2006 13:04:43 -0800, Greg KH <gregkh@suse.de> wrote:
> 
> > Now I'm down to the last problem, USB doesn't work, which is a bit of a
> > pain for me :)
> 
> > [   87.406180] APIC error on CPU0: 00(40)
> > [   87.426282] drivers/usb/core/inode.c: creating file '001'
> > [   87.426333] hub 1-0:1.0: state 5 ports 8 chg 0000 evt 0000
> > [   87.712002] APIC error on CPU0: 40(40)
> > [   87.774743] irq 16: nobody cared (try booting with the "irqpoll" option)
> 
> Why do you even enable APIC on an old laptop? We tried it a few times,
> it's just not possible. I'd say, about one in ten to one in five of 2002
> vintage laptops will not even boot, let alone work when APIC is enabled.
> Some of them are well known, like Dell Lattitude 610. I expect this to
> change with dual-core laptops, but for now the rule is: UP kernel,
> No APIC, for distro kernels at least.

Hm, it's a brand-new laptop, and all of my other boxes work with ioapic
on UP kernels.  I didn't think to turn it off, I'll go rebuild and see
if that helps.

oh, and 2.6.13 seems to work just fine, with ioapic enabled...

thanks,

greg k-h

Re: EHCI + APIC errors = no usb goodness

[Pete Zaitcev]

On Mon, 23 Jan 2006 13:41:15 -0800, Greg KH <gregkh@suse.de> wrote:

> Hm, it's a brand-new laptop [...]
> oh, and 2.6.13 seems to work just fine, with ioapic enabled...

Sorry, I didn't catch that. You wrote "a real old 2.6 kernel worked" and
I thought you hark back to 2.6.5 or something. Never mind then.

Oh and by the way, the traceback you posted was 32-bit.

-- Pete

Re: EHCI + APIC errors = no usb goodness

[Greg KH]

On Mon, Jan 23, 2006 at 01:50:18PM -0800, Pete Zaitcev wrote:
> On Mon, 23 Jan 2006 13:41:15 -0800, Greg KH <gregkh@suse.de> wrote:
> 
> > Hm, it's a brand-new laptop [...]
> > oh, and 2.6.13 seems to work just fine, with ioapic enabled...
> 
> Sorry, I didn't catch that. You wrote "a real old 2.6 kernel worked" and
> I thought you hark back to 2.6.5 or something. Never mind then.

Sorry, I think 2.6.12 is "real old" sometimes, and that is what I was
thinking of :)

> Oh and by the way, the traceback you posted was 32-bit.

Yeah, I'm running in 32bit mode, haven't gotten around to trying 64bit
mode out on this box yet...

thanks,

greg k-h

Re: EHCI + APIC errors = no usb goodness

[Greg KH]

On Mon, Jan 23, 2006 at 01:41:15PM -0800, Greg KH wrote:
> On Mon, Jan 23, 2006 at 01:25:54PM -0800, Pete Zaitcev wrote:
> > On Mon, 23 Jan 2006 13:04:43 -0800, Greg KH <gregkh@suse.de> wrote:
> > 
> > > Now I'm down to the last problem, USB doesn't work, which is a bit of a
> > > pain for me :)
> > 
> > > [   87.406180] APIC error on CPU0: 00(40)
> > > [   87.426282] drivers/usb/core/inode.c: creating file '001'
> > > [   87.426333] hub 1-0:1.0: state 5 ports 8 chg 0000 evt 0000
> > > [   87.712002] APIC error on CPU0: 40(40)
> > > [   87.774743] irq 16: nobody cared (try booting with the "irqpoll" option)
> > 
> > Why do you even enable APIC on an old laptop? We tried it a few times,
> > it's just not possible. I'd say, about one in ten to one in five of 2002
> > vintage laptops will not even boot, let alone work when APIC is enabled.
> > Some of them are well known, like Dell Lattitude 610. I expect this to
> > change with dual-core laptops, but for now the rule is: UP kernel,
> > No APIC, for distro kernels at least.
> 
> Hm, it's a brand-new laptop, and all of my other boxes work with ioapic
> on UP kernels.  I didn't think to turn it off, I'll go rebuild and see
> if that helps.

Nope, didn't help, no more APIC errors though, but the usb interrupt is
still disabled.

I'll go have fun with 'git bisect' and see what happens...

thanks,

greg k-h

Re: EHCI + APIC errors = no usb goodness

[David Brownell]

On Monday 23 January 2006 1:04 pm, Greg KH wrote:

> Any thoughts?

Not particularly; clearly the EHCI driver enabled IRQs, since that's
not done until after the "USB 2.0 started, EHCI 1.00, driver 10 Dec 2004"
message prints.  And it shouldn't be an SMI issue, as might be improved
by that patch I sent this AM.

According to arch/i386/kernel/apic.c that "0x40" APIC error bit means
it got an illegal vector ... sounds to me like IRQ setup issues, since
USB code doesn't know about such things (they're not even exposed from
the arch irq handling code).

Try sticking a message where ehci_irq() returns IRQ_NONE and see what
IRQ status is being reported to EHCI.

- Dave

Re: EHCI + APIC errors = no usb goodness

[Andi Kleen]

On Monday 23 January 2006 22:04, Greg KH wrote:
> I've had a laptop here that has had some "issues" in the past (time
> running double speed, XFree doesn't work, etc.)
> 
> Now I'm down to the last problem, USB doesn't work, which is a bit of a
> pain for me :)
> 
> Anyway, below is the kernel log from 2.6.16-rc1-mm2 (contains the latest
> acpi tree, which I thought might help out.)  This log is when I modprobe
> ehci-hcd.  The interesting thing is the APIC error, 

That was the laptop with ATI chipset right? Most of them have routing
troubles with the timer interrupt. I finally gave up trying to fix
them and just switched over to using the APIC timer which is run
by the CPU and not dependent on chipsets. Use the latest
patch from my x86-64 queue ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/

This should both eliminate the APIC errors and likely any timing problems
for 64bit kernels.

> Andi, I know you have been doing some apic rework for x86-64 (which this
> processor is, but I'm still running it in 32bit mode.)  Do you happen to
> have any patch that I could test out?

Ah, 32bit? For that just run without APIC.

-Andi

Re: EHCI + APIC errors = no usb goodness

[Kurt Wall]

On Tue, Jan 24, 2006 at 07:22:20AM +0100, Andi Kleen took 33 lines to write:
> 
> That was the laptop with ATI chipset right? Most of them have routing
> troubles with the timer interrupt. I finally gave up trying to fix
> them and just switched over to using the APIC timer which is run
> by the CPU and not dependent on chipsets. Use the latest
> patch from my x86-64 queue ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/
> 
> This should both eliminate the APIC errors and likely any timing problems
> for 64bit kernels.

I can confirm these two paragraphs. Andi's patche(s) worked for me on
2.6.16-rc1.

> Ah, 32bit? For that just run without APIC.

Kurt
-- 
A billion here, a couple of billion there -- first thing you know it
adds up to be real money.
		-- Senator Everett McKinley Dirksen

Re: EHCI + APIC errors = no usb goodness

[Greg KH]

On Tue, Jan 24, 2006 at 07:22:20AM +0100, Andi Kleen wrote:
> On Monday 23 January 2006 22:04, Greg KH wrote:
> > I've had a laptop here that has had some "issues" in the past (time
> > running double speed, XFree doesn't work, etc.)
> > 
> > Now I'm down to the last problem, USB doesn't work, which is a bit of a
> > pain for me :)
> > 
> > Anyway, below is the kernel log from 2.6.16-rc1-mm2 (contains the latest
> > acpi tree, which I thought might help out.)  This log is when I modprobe
> > ehci-hcd.  The interesting thing is the APIC error, 
> 
> That was the laptop with ATI chipset right? Most of them have routing
> troubles with the timer interrupt. I finally gave up trying to fix
> them and just switched over to using the APIC timer which is run
> by the CPU and not dependent on chipsets. Use the latest
> patch from my x86-64 queue ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/

Yes, this is the same laptop.

Turns out this was a bug in the USB EHCI bios handoff logic, a newer
patch from David fixed it.  The APIC stuff was just a false alarm, and
for now, I've just turned it off, thanks to Pete reminding me about it.

When I get this laptop converted to 64bit, I'll try out your patches.

thanks,

greg k-h

RE: EHCI + APIC errors = no usb goodness

[Brown, Len]

 
>Anyway, below is the kernel log from 2.6.16-rc1-mm2 (contains 
>the latest acpi tree, which I thought might help out.)

I don't recall any boot-time ACPI interrupt configuration
changes recently, so I wouldn't be surprised if you get
the same behaviour back several kernel versions.

if it comes up in IOAPIC mode when "acpi=off", then
that would be a clue -- but it seems that recent
systems don't include MPS, and thus when ACPI is off,
so it IOAPIC.

That said, it should also boot with ACPI in "noapic" mode,
and if it doesn't, then that could indicate an ACPI-specific issue
that may carry over into ACPI+ioapic mode.

-Len