* Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter
[not found] ` <20081112.041143.11487260.davem@davemloft.net>
@ 2008-11-12 21:31 ` Jesper Dangaard Brouer
2008-11-12 23:10 ` Matheos Worku
[not found] ` <1226572171.6834.87.camel@localhost.localdomain>
1 sibling, 1 reply; 11+ messages in thread
From: Jesper Dangaard Brouer @ 2008-11-12 21:31 UTC (permalink / raw)
To: netdev, linux-kernel
Hi Google,
On Wed, 12 Nov 2008, David Miller wrote:
> Guess what that does? The packet counters live in the upper
> 32-bits and the MARK bits live in the lower 32-bits of the
> TX_CS register.
>
> So it first reads the packet counters, and as a side effect that
> clears the MARK bits in the TX_CS register. So when we read the lower
> 32-bits the MARK bits are always seen as zero.
For the thorough reader, the TX_CS Transmit Control and Status register is
described in table 26-15 page 761-762 in the PDF document titled:
"UltraSPARC T2 supplement to UltraSPARC architecture 2007", downloadable
from:
http://opensparc-t2.sunsource.net/specs/UST2-UASuppl-current-draft-HP-EXT.pdf
Cheers,
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter
2008-11-12 21:31 ` NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter Jesper Dangaard Brouer
@ 2008-11-12 23:10 ` Matheos Worku
0 siblings, 0 replies; 11+ messages in thread
From: Matheos Worku @ 2008-11-12 23:10 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: netdev, linux-kernel
Jesper Dangaard Brouer wrote:
>
> Hi Google,
>
> On Wed, 12 Nov 2008, David Miller wrote:
>
>> Guess what that does? The packet counters live in the upper
>> 32-bits and the MARK bits live in the lower 32-bits of the
>> TX_CS register.
>>
>> So it first reads the packet counters, and as a side effect that
>> clears the MARK bits in the TX_CS register. So when we read the lower
>> 32-bits the MARK bits are always seen as zero.
>
>
> For the thorough reader, the TX_CS Transmit Control and Status
> register is described in table 26-15 page 761-762 in the PDF document
> titled: "UltraSPARC T2 supplement to UltraSPARC architecture 2007",
> downloadable from:
> http://opensparc-t2.sunsource.net/specs/UST2-UASuppl-current-draft-HP-EXT.pdf
>
>
> Cheers,
> Jesper Brouer
>
> --
> -------------------------------------------------------------------
> MSc. Master of Computer Science
> Dept. of Computer Science, University of Copenhagen
> Author of http://www.adsl-optimizer.dk
> -------------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
The niu/neptune HW puts some requirement on 32 bit reads of 64 bit
registers. You need to read the lower 32 bits first and then the upper
32 bits. The same ordering applies to writes as well.
On some 64 bit platforms, the 64 bit reads are split into two 32 bit
reads as well, regardless of the OS.
Regards
Matheos
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf + regression IRQs)
[not found] ` <20081113.141513.116537651.davem@davemloft.net>
@ 2008-11-19 22:58 ` Jesper Dangaard Brouer
2008-11-19 23:11 ` David Miller
0 siblings, 1 reply; 11+ messages in thread
From: Jesper Dangaard Brouer @ 2008-11-19 22:58 UTC (permalink / raw)
To: David Miller; +Cc: Jesper Dangaard Brouer, netdev, linux-kernel
On Thu, 13 Nov 2008, David Miller wrote:
> From: Jesper Dangaard Brouer <jdb@comx.dk>
> Date: Thu, 13 Nov 2008 11:29:31 +0100
>
>> Although I'm not happy about the new perf numbers, as I now on a SMP
>> system only can route approx 290 kpps, remember I could route 319 kpps
>> using a single CPU nosmp kernel.
>
> That unfortunately (can be) the cost of SMP :-/
[Regression]
Well that was not the real cause of the performance loss. Because on
kernel 2.6.27 I get really good performance (900-1200kpps) compared to
2.6.28 (git net-2.6).
The cause of this problem (tracked down together with Robert Olsson) is
that on 2.6.28 I have a lot less IRQs available. It seems max 34 IRQs.
Due the reduced number of IRQs the NIU driver cannot get enough IRQs to
the interfaces, and starts to use "IO-APIC" based IRQs.
On kernel 2.6.28:
My eth2 is using 10 IRQs all "PCI-MSI-edge".
BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
with the usb driver...
Think thats must be my performance problem on 2.6.28.
> With multi-flow tests, Robert Olsson is getting 4.2 mpps rates with
> NIU and pktgen. That's what this card is designed for, good
> multi-flow workload performance, rather than striving for maximum
> single-flow performance.
[Packet performance]
Yes, I know, I do use pktgen and multi-flows (rand dest IP+port).
For the two drivers NIU and Suns NXGE, my packet per sec performance is
now, on 2.6.27 (with backported NIU fixes).
With NIU driver I can route 900 kpps.
With NXGE driver (and enqueue=NULL hack) I can route 1200 kpps.
Actually I think I can go higher, because I'm limited by my packet rate
generator. I use pktgen (with rand dst IP+port) and can only generate 1200
kpps.
(I have actually ordered some new hardware, so I can get a faster pktgen
machine and perhaps test it as a router too. Also ordered the hardware
because I want to test PCI-express v.2.0. I have a prototype 12-port
gigabit NIC (from hotlava systems) that support PCIe v.2.0 and has 6x
82575 chips (4RX/4TX queues))
Hilsen
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf + regression IRQs)
2008-11-19 22:58 ` NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf + regression IRQs) Jesper Dangaard Brouer
@ 2008-11-19 23:11 ` David Miller
2008-11-20 19:48 ` Regression: Bisected, IRQ and MSI allocations screwed without sparse irq Jesper Dangaard Brouer
0 siblings, 1 reply; 11+ messages in thread
From: David Miller @ 2008-11-19 23:11 UTC (permalink / raw)
To: hawk; +Cc: jdb, netdev, linux-kernel
From: Jesper Dangaard Brouer <hawk@diku.dk>
Date: Wed, 19 Nov 2008 23:58:12 +0100 (CET)
> Well that was not the real cause of the performance loss. Because
> on kernel 2.6.27 I get really good performance (900-1200kpps)
> compared to 2.6.28 (git net-2.6).
>
> The cause of this problem (tracked down together with Robert Olsson)
> is that on 2.6.28 I have a lot less IRQs available. It seems max 34
> IRQs.
>
> Due the reduced number of IRQs the NIU driver cannot get enough IRQs
> to the interfaces, and starts to use "IO-APIC" based IRQs.
This is almost certainly related to the driver unload bug.
I know you ran into unbuildable/unbootable kernels during a bisect,
but you really need to track down this regression.
There were a lot of IRQ changes, especially on x86. The sequence is
something like:
1) dyn irqs
2) APIC/IO_APIC handling integration
3) by-hand REVERT of dyn irqs, it was done by hand in order to not
lose the #2 changes
4) interrupt remapping support
^ permalink raw reply [flat|nested] 11+ messages in thread
* Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-19 23:11 ` David Miller
@ 2008-11-20 19:48 ` Jesper Dangaard Brouer
2008-11-21 0:34 ` Thomas Gleixner
0 siblings, 1 reply; 11+ messages in thread
From: Jesper Dangaard Brouer @ 2008-11-20 19:48 UTC (permalink / raw)
To: Thomas Gleixner
Cc: David Miller, Jesper Dangaard Brouer, netdev, linux-kernel,
Robert Olsson
[-- Attachment #1: Type: TEXT/PLAIN, Size: 4263 bytes --]
Hi Thomas Gleixner,
I have bisected a regression to your commit
3235e936c0cc3589309280b6f59e5096779adae3,
"x86: remove sparse irq from Kconfig".
Its actually not necessary your fault, as your commit simply removes
the config option HAVE_SPARSE_IRQ. This revels the bug / regression
I'm exposted to.
Guess I should bisect again to find the exact faulty commit, but I'm
rather sick of bisecting at the moment, and though you might have a
better idea whats going wrong. I would rather spend my time
performance tuning the multiqueue routing code...
[The regression]:
During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
get really good performance (900-1200kpps) compared to 2.6.28 (davem
git net-2.6).
The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs. Due the reduced number of IRQs the NIU driver cannot get
enough IRQs to the interfaces, and starts to use "IO-APIC" based
IRQs.
On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
the usb driver. That my performance problem on 2.6.28.
[Other related bugs]:
Is that unloading the "niu" driver will give a kernel BUG during
deallocation og MSI interrupts. (See dmesg output below if interested)
(I have attached full bisect history)
Cheers,
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------
On Wed, 19 Nov 2008, David Miller wrote:
> From: Jesper Dangaard Brouer <hawk@diku.dk>
> Date: Wed, 19 Nov 2008 23:58:12 +0100 (CET)
>
>> Well that was not the real cause of the performance loss. Because
>> on kernel 2.6.27 I get really good performance (900-1200kpps)
>> compared to 2.6.28 (git net-2.6).
>>
>> The cause of this problem (tracked down together with Robert Olsson)
>> is that on 2.6.28 I have a lot less IRQs available. It seems max 34
>> IRQs.
>>
>> Due the reduced number of IRQs the NIU driver cannot get enough IRQs
>> to the interfaces, and starts to use "IO-APIC" based IRQs.
>
> This is almost certainly related to the driver unload bug.
>
> I know you ran into unbuildable/unbootable kernels during a bisect,
> but you really need to track down this regression.
------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo
niu(-)
Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0
EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
[<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
[<c021ef41>] ? pci_disable_msix+0x31/0x40
[<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
[<c0219c29>] ? pci_device_remove+0x19/0x40
[<c026b0d4>] ? __device_release_driver+0x54/0x80
[<c026b197>] ? driver_detach+0x97/0xa0
[<c026a475>] ? bus_remove_driver+0x75/0xa0
[<c026b609>] ? driver_unregister+0x39/0x40
[<c0219e51>] ? pci_unregister_driver+0x21/0x80
[<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
[<c0145d74>] ? sys_delete_module+0x114/0x1d0
[<c016810a>] ? remove_vma+0x3a/0x50
[<c0168c29>] ? do_munmap+0x189/0x1e0
[<c0103229>] ? sysenter_do_call+0x12/0x21
[<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b
14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe
55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]---
[-- Attachment #2: Type: TEXT/plain, Size: 32509 bytes --]
~~ -*-text-*-
-------------------------------------------------------
Bisecting IRQ change:
What change reduced the IRQs
-------------------------------------------------------
Jesper Dangaard Brouer (jdb@comx.dk)
-------------------------------------------------------
$LastChangedRevision: 786 $
$Date: 2008-11-20 20:44:51 +0100 (Thu, 20 Nov 2008) $
-------------------------------------------------------
git clone
~~~~~~~~~
+---------
cd /var/kernels/git/davem
git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect-irqs
+---------
Description / Reason to find
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
During my testing of the Sun Neptune based NICs.
On kernel 2.6.27 I get really good performance (900-1200kpps)
compared to 2.6.28 (git net-2.6).
The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs.
Due the reduced number of IRQs the NIU driver cannot get enough IRQs
to the interfaces, and starts to use "IO-APIC" based IRQs.
On kernel 2.6.28:
My eth2 is using 10 IRQs all "PCI-MSI-edge".
BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
with the usb driver...
Think thats must be my performance problem on 2.6.28.
Known: Good and bad
~~~~~~~~~~~~~~~~~~~
GOOD:
git bisect good v2.6.27
BAD:
git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
[92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch
'tracing-v28-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
HiSTORY:
~~~~~~~~
+--------
cd /var/kernels/git/davem/net-2.6-bisect-irqs/
git bisect start
git bisect good v2.6.27
+--------
+--------------
git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
Bisecting: 3220 revisions left to test after this
[af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
+--------------
CONFIG_LOCALVERSION="-bisect"
+-------------
cp ../net-2.6-bisect/.config .
script make_oldconfig_01
make oldconfig
exit
#Script done, file is make_oldconfig_01
+-------------
+----------------
time make -j6 bzImage modules
#
#real 9m22.739s
#user 16m56.776s
#sys 1m4.672s
+----------------
Booted kernel: GOOD: irqs and (niu rmmod good)
+----------------
git bisect good
Bisecting: 1614 revisions left to test after this
[36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
+----------------
Compiling:
+----------------
time make -j6 bzImage modules
+----------------
Booted kernel: GOOD: irqs and (niu rmmod good)
+-----------
git bisect good
Bisecting: 807 revisions left to test after this
[1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
+-----------
Compiling...
+----------------
time make -j6 bzImage modules
#real 10m1.561s
#user 17m23.293s
#sys 1m5.744s
+----------------
Installing...
Booted kernel:
+----
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #3 SMP PREEMPT Thu Nov 20 12:33:02 CET 2008 i686 GNU/Linux
+----
Results: GOOD: irqs and (niu rmmod good)
+------
git bisect good
Bisecting: 403 revisions left to test after this
[1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
+------
Compiling...
+----------------
time make -j6 bzImage modules
#real 10m9.371s
#user 17m21.781s
#sys 1m6.052s
+----------------
Installing...
Booting ...
+-------
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #4 SMP PREEMPT Thu Nov 20 12:50:39 CET 2008 i686 GNU/Linux
+-------
Results: GOOD: irqs and (niu rmmod good)
+-------
git-bisect good
Bisecting: 223 revisions left to test after this
[dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
+-------
+----------------
time make -j6 bzImage modules
+----------------
Booting ...
+--------
Linux dcu-router-ng 2.6.27-bisect #5 SMP PREEMPT Thu Nov 20 13:58:34 CET 2008 i686 GNU/Linux
+--------
Results: BAD: irqs and (niu rmmod also BAD)
+-------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 125 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 1 2 2 IO-APIC-edge serial
8: 0 2 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 103 108 108 112 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6, eth0
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, eth3
20: 0 0 0 0 PCI-MSI-edge eth2
21: 0 0 0 0 PCI-MSI-edge eth2
22: 24 23 23 23 IO-APIC-fasteoi uhci_hcd:usb5, eth2
23: 0 0 0 0 PCI-MSI-edge eth2
24: 0 0 0 0 PCI-MSI-edge eth2
25: 0 0 0 0 PCI-MSI-edge eth2
26: 0 0 0 0 PCI-MSI-edge eth2
27: 0 0 0 0 PCI-MSI-edge eth2
28: 0 0 0 0 PCI-MSI-edge eth2
29: 0 0 0 0 PCI-MSI-edge eth2
30: 0 0 0 0 PCI-MSI-edge eth2
31: 0 0 0 0 PCI-MSI-edge eth2
32: 0 0 0 0 PCI-MSI-edge eth2
34: 271 268 268 264 PCI-MSI-edge cciss0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3301 2970 2594 2389 Local timer interrupts
RES: 28 560 6 13 Rescheduling interrupts
CAL: 50 104 99 62 Function call interrupts
TLB: 241 224 287 279 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+-------
OUTPUT "rmmod niu" (gives segfault) and "dmesg"
+-------
------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-)
Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0
EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
[<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
[<c021ef41>] ? pci_disable_msix+0x31/0x40
[<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
[<c0219c29>] ? pci_device_remove+0x19/0x40
[<c026b0d4>] ? __device_release_driver+0x54/0x80
[<c026b197>] ? driver_detach+0x97/0xa0
[<c026a475>] ? bus_remove_driver+0x75/0xa0
[<c026b609>] ? driver_unregister+0x39/0x40
[<c0219e51>] ? pci_unregister_driver+0x21/0x80
[<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
[<c0145d74>] ? sys_delete_module+0x114/0x1d0
[<c016810a>] ? remove_vma+0x3a/0x50
[<c0168c29>] ? do_munmap+0x189/0x1e0
[<c0103229>] ? sysenter_do_call+0x12/0x21
[<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]---
+-------
+------
git-bisect bad
Bisecting: 89 revisions left to test after this
[db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
+------
Related config change? (make oldconfig)
+------
script make_oldconfig_02
make oldconfig
Script done, file is make_oldconfig_02
+------
+------
Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) ? ?Y
This enables support for sparse irq, esp for msi/msi-x. the irq
number will be bus/dev/fn + 12bit. You may need if you have lots of
cards supports msi-x installed.
If you don't know what to do here, say Y.
+------
Compiling...
+----------------
time make -j6 bzImage modules
#
#real 9m29.556s
#user 17m10.396s
#sys 1m5.056s
+----------------
Booting ...
+-------
Linux dcu-router-ng 2.6.27-bisect #6 SMP PREEMPT Thu Nov 20 14:25:40 CET 2008 i686 GNU/Linux
+-------
The output from /proc/interrupts changed, very weird!
BUT eth3 does use a "PCI-MSI-edge" interrupt.
Guess this is a GOOD state even though it looks weird.
Unloading NIU driver also GOOD.
+---------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0x0: 124 1 0 0 IO-APIC-edge timer
0x1: 1 0 0 1 IO-APIC-edge i8042
0x3: 2 1 2 2 IO-APIC-edge serial
0x8: 1 0 0 1 IO-APIC-edge rtc
0x9: 0 0 0 0 IO-APIC-fasteoi acpi
0xc: 0 1 2 1 IO-APIC-edge i8042
0x10: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
0x11: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
0x12: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
0x6000fe: 288 289 290 290 PCI-MSI-edge cciss0
0x16: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
0xb00100: 0 0 0 0 PCI-MSI-edge eth2
0xb000ff: 0 0 0 0 PCI-MSI-edge eth2
0xb000fe: 0 0 0 0 PCI-MSI-edge eth2
0xb000fd: 0 0 0 0 PCI-MSI-edge eth2
0xb000fc: 0 0 0 0 PCI-MSI-edge eth2
0xb000fb: 0 0 0 0 PCI-MSI-edge eth2
0xb000fa: 0 0 0 0 PCI-MSI-edge eth2
0xb000f9: 0 0 0 0 PCI-MSI-edge eth2
0xb000f8: 0 0 0 0 PCI-MSI-edge eth2
0xb000f7: 0 0 0 0 PCI-MSI-edge eth2
0xb000f6: 0 0 0 0 PCI-MSI-edge eth2
0xb000f5: 0 0 0 0 PCI-MSI-edge eth2
0xb000f4: 0 0 0 0 PCI-MSI-edge eth2
0xb01100: 0 0 0 0 PCI-MSI-edge eth3
0xb010ff: 0 0 0 0 PCI-MSI-edge eth3
0xb010fe: 0 0 0 0 PCI-MSI-edge eth3
0xb010fd: 0 0 0 0 PCI-MSI-edge eth3
0xb010fc: 0 0 0 0 PCI-MSI-edge eth3
0xb010fb: 0 0 0 0 PCI-MSI-edge eth3
0xb010fa: 0 0 0 0 PCI-MSI-edge eth3
0xb010f9: 0 0 0 0 PCI-MSI-edge eth3
0xb010f8: 0 0 0 0 PCI-MSI-edge eth3
0xb010f7: 0 0 0 0 PCI-MSI-edge eth3
0xb010f6: 0 0 0 0 PCI-MSI-edge eth3
0x13: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
0x300100: 210 210 210 208 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3630 3265 3103 2711 Local timer interrupts
RES: 34 226 12 417 Rescheduling interrupts
CAL: 89 55 90 78 Function call interrupts
TLB: 253 205 311 267 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+---------
Guess it a GOOD situation...
+------
git-bisect good
Bisecting: 44 revisions left to test after this
[ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
+------
Compiling ...
+--------
time make -j6 bzImage modules
#real 9m28.062s
#user 17m7.492s
#sys 1m4.248s
+--------
Installing ...
Booting ...
+------
Linux dcu-router-ng 2.6.27-bisect #7 SMP PREEMPT Thu Nov 20 14:52:45 CET 2008 i686 GNU/Linux
+------
Still looks GOOD (/proc/interrupts still looks weird).
And rmmod NIU driver GOOD.
+------
git-bisect good
Bisecting: 22 revisions left to test after this
[922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
+------
Compiling ...
+--------
time make -j6 bzImage modules
#real 0m34.622s
#user 0m41.139s
#sys 0m5.812s
+--------
Install ...
Booting ...
+-----
Linux dcu-router-ng 2.6.27-bisect #8 SMP PREEMPT Thu Nov 20 15:04:11 CET 2008 i686 GNU/Linux
+-----
Looks GOOD, and /proc/interrupts changed again! Now the interrupts
are not i HEX anymore, but in decimal, but still strange/large
numbers for MSI.
Unloading NIU driver GOOD.
+------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 2 1 2 IO-APIC-edge serial
8: 0 0 1 1 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 828 821 828 823 PCI-MSI-edge cciss0
22: 23 24 24 22 IO-APIC-fasteoi uhci_hcd:usb5
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
3145984: 9993 9994 9987 9993 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 10075 11448 8001 8787 Local timer interrupts
RES: 297 17 349 26 Rescheduling interrupts
CAL: 173 189 95 173 Function call interrupts
TLB: 299 259 330 345 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+------
+-------
git-bisect good
Bisecting: 11 revisions left to test after this
[a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
+-------
Compiling
+--------
time make -j6 bzImage modules
#real 9m30.767s
#user 17m10.808s
#sys 1m6.388s
+--------
Installing ...
Booting ...
+-----
Linux dcu-router-ng 2.6.27-bisect #9 SMP PREEMPT Thu Nov 20 15:28:17 CET 2008 i686 GNU/Linux
+-----
BAD kernel version, max IRQ is 34. And eth3 got assigned a
IO-APIC-fasteoi shared with uhci_hcd:usb2.
Also BAD unloading of NIU driver.
BUG is some where in between:
git log 922402f15a85f7a064926eb1db68cc52bc4d4a91..a1aca5de08a0cb840a90fb3f729a5940f8d21185 | grep ^commit | wc -l
11 commits
+-------
git-bisect bad
Bisecting: 5 revisions left to test after this
[3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
+-------
Compiling...
+--------
time make -j6 bzImage modules
+--------
Install ...
Booting
+------
Linux dcu-router-ng 2.6.27-bisect #10 SMP PREEMPT Thu Nov 20 15:56:10 CET 2008 i686 GNU/Linux
+------
BAD kernel.
BAD rmmod NIU driver.
+---------
git bisect bad
Bisecting: 2 revisions left to test after this
[4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
+---------
Compiling...
+------
time make -j6 bzImage modules
#real 7m23.814s
#user 12m15.718s
#sys 0m42.183s
+------
Config change prompting:
+-----
Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) Y
+-----
Installing ...
Booting ...
+-------
Linux dcu-router-ng 2.6.27-bisect #11 SMP PREEMPT Thu Nov 20 16:19:29 CET 2008 i686 GNU/Linux
+-------
GOOD!!!
+--------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 2 2 2 IO-APIC-edge serial
8: 1 0 0 1 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 285 288 293 287 PCI-MSI-edge cciss0
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
22: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
3145984: 244 242 238 241 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3715 3104 2853 2542 Local timer interrupts
RES: 88 52 280 258 Rescheduling interrupts
CAL: 76 75 93 59 Function call interrupts
TLB: 245 241 312 283 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+---------
+---------
git-bisect good
Bisecting: 1 revisions left to test after this
[7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
+----------
Compiling ...
+------
time make -j6 bzImage modules
+------
Install...
Boot ...
+-------
Linux dcu-router-ng 2.6.27-bisect #12 SMP PREEMPT Thu Nov 20 16:33:11 CET 2008 i686 GNU/Linux
+-------
+--------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 1 2 2 2 IO-APIC-edge serial
8: 2 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 268 269 268 267 PCI-MSI-edge cciss0
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
22: 25 23 24 25 IO-APIC-fasteoi uhci_hcd:usb5
3145984: 175 174 176 178 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3508 2902 2765 2489 Local timer interrupts
RES: 238 35 461 6 Rescheduling interrupts
CAL: 61 90 59 81 Function call interrupts
TLB: 257 220 299 300 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+--------
GOOD.
+----------
git-bisect good
3235e936c0cc3589309280b6f59e5096779adae3 is first bad commit
commit 3235e936c0cc3589309280b6f59e5096779adae3
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Oct 15 13:16:00 2008 +0200
x86: remove sparse irq from Kconfig
This code is not ready yet.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
:040000 040000 6043e32465556e828de0fbb6aa497b277239af01 2dd75ba207990d83a3a4c7b7b16abccfe2d5e10d M arch
+--------
Found bad commit:
3235e936c0cc3589309280b6f59e5096779adae3
Git bisect LOG
~~~~~~~~~~~~~~
+-------
git-bisect log
git-bisect start
# good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
# bad: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
# good: [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
git-bisect good af5c2bd16ac2e5688c3bf46ea1f95112d696d294
# good: [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git-bisect good 36ac1d2f323f8bf8bc10c25b88f617657720e241
# good: [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
git-bisect good 1aece34833721d64eb33fc15cd923c727296d3d3
# good: [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
git-bisect good 1d9a8a47d659f053abeca9ece45651b4d94780c8
# bad: [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
git-bisect bad dd3a1db900f2a215a7d7dd71b836e149a6cf5fed
# good: [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
git-bisect good db4b5525caafd846ec20f95afbc6403c792e22cf
# good: [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
git-bisect good ba374c9baef910fbc5373541d98c50f15e82c3f8
# good: [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
git-bisect good 922402f15a85f7a064926eb1db68cc52bc4d4a91
# bad: [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
git-bisect bad a1aca5de08a0cb840a90fb3f729a5940f8d21185
# bad: [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
git-bisect bad 3235e936c0cc3589309280b6f59e5096779adae3
# good: [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
git-bisect good 4c66a73f0796dacc2ff0d4af75794ec843ceb3d1
# good: [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
git-bisect good 7ef0c30dbf96a8d9a234e90c248eb19df3c031be
+-------
Email
~~~~~
To:
Thomas Gleixner <tglx@linutronix.de>
David Miller <davem@davemloft.net>,
Jesper Dangaard Brouer <jdb@comx.dk>,
netdev <netdev@vger.kernel.org>,
linux-kernel@vger.kernel.org,
Robert Olsson <Robert.Olsson@data.slu.se>
Subj.:
Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
Hi Thomas Gleixner,
I have bisected a regression to your commit
3235e936c0cc3589309280b6f59e5096779adae3, "x86: remove sparse irq
from Kconfig".
Its actually not necessary your fault, as your commit simply removes
the config option HAVE_SPARSE_IRQ. This revels the bug / regression
I'm exposted to.
Guess I should bisect again to find the exact faulty commit, but I'm
rather sick of bisecting at the moment, and though you might have a
better idea whats going wrong. I would rather spend my time
performance tuning the multiqueue routing code...
[The regression]:
During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
get really good performance (900-1200kpps) compared to 2.6.28 (davem
git net-2.6).
The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs. Due the reduced number of IRQs the NIU driver cannot get
enough IRQs to the interfaces, and starts to use "IO-APIC" based
IRQs.
On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
the usb driver. That my performance problem on 2.6.28.
[Other related bugs]:
Is that unloading the "niu" driver will give a kernel BUG.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-20 19:48 ` Regression: Bisected, IRQ and MSI allocations screwed without sparse irq Jesper Dangaard Brouer
@ 2008-11-21 0:34 ` Thomas Gleixner
2008-11-21 10:33 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 11+ messages in thread
From: Thomas Gleixner @ 2008-11-21 0:34 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: David Miller, Jesper Dangaard Brouer, netdev, LKML, Robert Olsson
Jesper,
On Thu, 20 Nov 2008, Jesper Dangaard Brouer wrote:
> I have bisected a regression to your commit
> 3235e936c0cc3589309280b6f59e5096779adae3,
> "x86: remove sparse irq from Kconfig".
>
> Its actually not necessary your fault, as your commit simply removes
> the config option HAVE_SPARSE_IRQ. This revels the bug / regression
> I'm exposted to.
Yup, the bisect result is pretty useless.
> The cause of this problem (tracked down together with Robert Olsson)
> is that on 2.6.28 I have a lot less IRQs available. It seems max 34
> IRQs. Due the reduced number of IRQs the NIU driver cannot get
> enough IRQs to the interfaces, and starts to use "IO-APIC" based
> IRQs.
Can you please try the attached patch ?
Thanks,
tglx
-----
arch/x86/kernel/io_apic.c | 22 +---------------------
1 file changed, 1 insertion(+), 21 deletions(-)
Index: linux-2.6/arch/x86/kernel/io_apic.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/io_apic.c
+++ linux-2.6/arch/x86/kernel/io_apic.c
@@ -3594,27 +3594,7 @@ int __init io_apic_get_redir_entries (in
int __init probe_nr_irqs(void)
{
- int idx;
- int nr = 0;
-#ifndef CONFIG_XEN
- int nr_min = 32;
-#else
- int nr_min = NR_IRQS;
-#endif
-
- for (idx = 0; idx < nr_ioapics; idx++)
- nr += io_apic_get_redir_entries(idx) + 1;
-
- /* double it for hotplug and msi and nmi */
- nr <<= 1;
-
- /* something wrong ? */
- if (nr < nr_min)
- nr = nr_min;
- if (WARN_ON(nr > NR_IRQS))
- nr = NR_IRQS;
-
- return nr;
+ return NR_IRQS;
}
/* --------------------------------------------------------------------------
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-21 0:34 ` Thomas Gleixner
@ 2008-11-21 10:33 ` Jesper Dangaard Brouer
2008-11-21 16:40 ` Thomas Gleixner
0 siblings, 1 reply; 11+ messages in thread
From: Jesper Dangaard Brouer @ 2008-11-21 10:33 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Jesper Dangaard Brouer, David Miller, netdev, LKML, Robert Olsson
On Thu, 2008-11-20 at 16:34 -0800, Thomas Gleixner wrote:
> On Thu, 20 Nov 2008, Jesper Dangaard Brouer wrote:
> > I have bisected a regression to your commit
> > 3235e936c0cc3589309280b6f59e5096779adae3,
> > "x86: remove sparse irq from Kconfig".
> >
> > Its actually not necessary your fault, as your commit simply removes
> > the config option HAVE_SPARSE_IRQ. This revels the bug / regression
> > I'm exposted to.
>
> Yup, the bisect result is pretty useless.
>
> > The cause of this problem (tracked down together with Robert Olsson)
> > is that on 2.6.28 I have a lot less IRQs available. It seems max 34
> > IRQs. Due the reduced number of IRQs the NIU driver cannot get
> > enough IRQs to the interfaces, and starts to use "IO-APIC" based
> > IRQs.
>
> Can you please try the attached patch ?
I have tried the patch and it solved the problem! :-)
I'll gladly test other patches from your. Guess this patch needs to be
brushed up before a mainline patch is ready.
My hardware is a HP ProLiant DL380-G5.
> -----
> arch/x86/kernel/io_apic.c | 22 +---------------------
> 1 file changed, 1 insertion(+), 21 deletions(-)
>
> Index: linux-2.6/arch/x86/kernel/io_apic.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/io_apic.c
> +++ linux-2.6/arch/x86/kernel/io_apic.c
> @@ -3594,27 +3594,7 @@ int __init io_apic_get_redir_entries (in
>
> int __init probe_nr_irqs(void)
> {
> - int idx;
> - int nr = 0;
> -#ifndef CONFIG_XEN
> - int nr_min = 32;
> -#else
> - int nr_min = NR_IRQS;
> -#endif
> -
> - for (idx = 0; idx < nr_ioapics; idx++)
> - nr += io_apic_get_redir_entries(idx) + 1;
> -
> - /* double it for hotplug and msi and nmi */
> - nr <<= 1;
> -
> - /* something wrong ? */
> - if (nr < nr_min)
> - nr = nr_min;
> - if (WARN_ON(nr > NR_IRQS))
> - nr = NR_IRQS;
> -
> - return nr;
> + return NR_IRQS;
> }
>
--
Med venlig hilsen / Best regards
Jesper Brouer
ComX Networks A/S
Linux Network developer
Cand. Scient Datalog / MSc.
Author of http://adsl-optimizer.dk
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-21 10:33 ` Jesper Dangaard Brouer
@ 2008-11-21 16:40 ` Thomas Gleixner
2008-11-21 19:35 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 11+ messages in thread
From: Thomas Gleixner @ 2008-11-21 16:40 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Jesper Dangaard Brouer, David Miller, netdev, LKML, Robert Olsson
On Fri, 21 Nov 2008, Jesper Dangaard Brouer wrote:
> > Can you please try the attached patch ?
>
> I have tried the patch and it solved the problem! :-)
>
> I'll gladly test other patches from your. Guess this patch needs to be
> brushed up before a mainline patch is ready.
Ok, I queue it for mainline. This solves just the number of irqs
limitation, the rmmod problem still persists, right ?
Thanks,
tglx
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-21 16:40 ` Thomas Gleixner
@ 2008-11-21 19:35 ` Jesper Dangaard Brouer
2008-11-21 21:11 ` Thomas Gleixner
2008-11-21 23:06 ` David Miller
0 siblings, 2 replies; 11+ messages in thread
From: Jesper Dangaard Brouer @ 2008-11-21 19:35 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Jesper Dangaard Brouer, David Miller, netdev, LKML, Robert Olsson,
Ingo Molnar
On Fri, 21 Nov 2008, Thomas Gleixner wrote:
> On Fri, 21 Nov 2008, Jesper Dangaard Brouer wrote:
>>> Can you please try the attached patch ?
>>
>> I have tried the patch and it solved the problem! :-)
>>
>> I'll gladly test other patches from your. Guess this patch needs to be
>> brushed up before a mainline patch is ready.
>
> Ok, I queue it for mainline. This solves just the number of irqs
> limitation, the rmmod problem still persists, right ?
It solves both the irq limit and the NIU driver unload bug.
We should give it a good description.
I have cooked up a patch with a description below, will you accept that?
Who's tree do you want it to go upsteam via?
(You are listed as one of the X86 maintainers, but Ingo's tree seems more
up-to-date. My patch below is agains DaveM's tree)
Cheers,
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------
Fixing irq limit and NIU driver unload bug.
Removing the config option HAVE_SPARSE_IRQ (commit
3235e936c0cc3589309280b6f59e5096779adae3) revealed a regression that
limited the number of irqs on the system.
Besides limiting the number of IRQ, this also caused unloading of the
NIU driver to fail during msi_free_irqs(). The reduced number of IRQs
caused the NIU driver to use "IO-APIC" based IRQs instead of
"PCI-MSI-edge".
This patch changes probe_nr_irqs() to return NR_IRQS, which is
basically the same as the NOT CONFIG_X86_IO_APIC case. Thus being
fairly safe.
Thus, solving both the irq limit and the NIU driver unload bug.
Tested-by: Jesper Dangaard Brouer <hawk@comx.dk>
Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/kernel/io_apic.c | 22 +---------------------
1 files changed, 1 insertions(+), 21 deletions(-)
diff --git a/arch/x86/kernel/io_apic.c b/arch/x86/kernel/io_apic.c
index c9513e1..1fec0f9 100644
--- a/arch/x86/kernel/io_apic.c
+++ b/arch/x86/kernel/io_apic.c
@@ -3608,27 +3608,7 @@ int __init io_apic_get_redir_entries (int ioapic)
int __init probe_nr_irqs(void)
{
- int idx;
- int nr = 0;
-#ifndef CONFIG_XEN
- int nr_min = 32;
-#else
- int nr_min = NR_IRQS;
-#endif
-
- for (idx = 0; idx < nr_ioapics; idx++)
- nr += io_apic_get_redir_entries(idx) + 1;
-
- /* double it for hotplug and msi and nmi */
- nr <<= 1;
-
- /* something wrong ? */
- if (nr < nr_min)
- nr = nr_min;
- if (WARN_ON(nr > NR_IRQS))
- nr = NR_IRQS;
-
- return nr;
+ return NR_IRQS;
}
/* --------------------------------------------------------------------------
--
1.5.4.2
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-21 19:35 ` Jesper Dangaard Brouer
@ 2008-11-21 21:11 ` Thomas Gleixner
2008-11-21 23:06 ` David Miller
1 sibling, 0 replies; 11+ messages in thread
From: Thomas Gleixner @ 2008-11-21 21:11 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Jesper Dangaard Brouer, David Miller, netdev, LKML, Robert Olsson,
Ingo Molnar
On Fri, 21 Nov 2008, Jesper Dangaard Brouer wrote:
> > Ok, I queue it for mainline. This solves just the number of irqs
> > limitation, the rmmod problem still persists, right ?
>
> It solves both the irq limit and the NIU driver unload bug.
I don't believe that it solves it. It hides it at the best.
> We should give it a good description.
> I have cooked up a patch with a description below, will you accept that?
>
> Who's tree do you want it to go upsteam via?
> (You are listed as one of the X86 maintainers, but Ingo's tree seems more
> up-to-date. My patch below is agains DaveM's tree)
I queued it already with a description of the irq nr. problem.
The rmmod problem is something different and should be investigated
thoroughly instead of declaring it solved by magic.
Thanks,
tglx
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq
2008-11-21 19:35 ` Jesper Dangaard Brouer
2008-11-21 21:11 ` Thomas Gleixner
@ 2008-11-21 23:06 ` David Miller
1 sibling, 0 replies; 11+ messages in thread
From: David Miller @ 2008-11-21 23:06 UTC (permalink / raw)
To: hawk; +Cc: tglx, jdb, netdev, linux-kernel, Robert.Olsson, mingo
From: Jesper Dangaard Brouer <hawk@diku.dk>
Date: Fri, 21 Nov 2008 20:35:32 +0100 (CET)
> On Fri, 21 Nov 2008, Thomas Gleixner wrote:
>
> > On Fri, 21 Nov 2008, Jesper Dangaard Brouer wrote:
> >>> Can you please try the attached patch ?
> >>
> >> I have tried the patch and it solved the problem! :-)
> >>
> >> I'll gladly test other patches from your. Guess this patch needs to be
> >> brushed up before a mainline patch is ready.
> >
> > Ok, I queue it for mainline. This solves just the number of irqs
> > limitation, the rmmod problem still persists, right ?
>
> It solves both the irq limit and the NIU driver unload bug.
I think it "solves" the unload BUG because the driver never has
to fallback to IO_APIC irqs and abort trying to use MSI-X
any longer.
Only the IRQ limit bug is fixed by Thomas's patch.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-11-21 23:07 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20081112.014923.68124784.davem@davemloft.net>
[not found] ` <1226487710.6834.53.camel@localhost.localdomain>
[not found] ` <20081112.035240.226243372.davem@davemloft.net>
[not found] ` <20081112.041143.11487260.davem@davemloft.net>
2008-11-12 21:31 ` NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter Jesper Dangaard Brouer
2008-11-12 23:10 ` Matheos Worku
[not found] ` <1226572171.6834.87.camel@localhost.localdomain>
[not found] ` <20081113.141513.116537651.davem@davemloft.net>
2008-11-19 22:58 ` NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf + regression IRQs) Jesper Dangaard Brouer
2008-11-19 23:11 ` David Miller
2008-11-20 19:48 ` Regression: Bisected, IRQ and MSI allocations screwed without sparse irq Jesper Dangaard Brouer
2008-11-21 0:34 ` Thomas Gleixner
2008-11-21 10:33 ` Jesper Dangaard Brouer
2008-11-21 16:40 ` Thomas Gleixner
2008-11-21 19:35 ` Jesper Dangaard Brouer
2008-11-21 21:11 ` Thomas Gleixner
2008-11-21 23:06 ` David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox