* Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 21:06 UTC (permalink / raw)
To: Linux Network Developers; +Cc: brice, gallatin
I've run into a major receive side performance issue with multi-10-GigE
on a NUMA system. The system is using a SuperMicro X8DAH+-F motherboard
with 2 3.2 GHz quad-core Intel Xeon 5580 processors and 12 GB of
1333 MHz DDR3 memory. It is a Fedora 10 system but using the latest
2.6.29.6 kernel from Fedora 11 (originally tried the 2.6.27.29 kernel
from Fedora 10).
The test setup is:
i7test1----(6)----xeontest1----(6)----i7test2
10-GigE 10-GigE
So xeontest1 has 6 dual-port Myricom 10-GigE NICs for a total
of 12 10-GigE interfaces. eth2 through eth7 (which are on the
second Intel 5520 I/O Hub) are connected to i7test1 while
eth8 through eth13 (which are on the first Intel 5520 I/O Hub)
are connected to i7test2.
Previous direct testing between i7test1 and i7test2 (which use an
Asus P6T6 WS Revolution motherboard) demonstrated that they could
achieve ~70 Gbps performance for either transmit or receive using
8 10-GigE interfaces.
The transmit side performance of xeontest1 is fantastic:
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -xc6/3 -p5012 192.168.12.11 &
n12: 9648.0522 MB / 10.00 sec = 8091.4066 Mbps 49 %TX 26 %RX 0 retrans 0.18 msRTT
n9: 11130.5320 MB / 10.01 sec = 9328.3224 Mbps 47 %TX 37 %RX 0 retrans 0.19 msRTT
n11: 9418.1250 MB / 10.00 sec = 7897.5848 Mbps 50 %TX 30 %RX 0 retrans 0.18 msRTT
n10: 9279.4758 MB / 10.01 sec = 7778.7146 Mbps 49 %TX 28 %RX 0 retrans 0.12 msRTT
n8: 11142.6574 MB / 10.01 sec = 9340.3789 Mbps 47 %TX 35 %RX 0 retrans 0.18 msRTT
n13: 9422.1492 MB / 10.01 sec = 7897.4115 Mbps 49 %TX 25 %RX 0 retrans 0.17 msRTT
n3: 11471.2500 MB / 10.01 sec = 9613.9477 Mbps 49 %TX 32 %RX 0 retrans 0.15 msRTT
n6: 9339.6354 MB / 10.01 sec = 7828.5345 Mbps 50 %TX 25 %RX 0 retrans 0.19 msRTT
n4: 9093.2500 MB / 10.01 sec = 7624.1589 Mbps 49 %TX 28 %RX 0 retrans 0.15 msRTT
n5: 9121.8367 MB / 10.01 sec = 7646.8646 Mbps 50 %TX 29 %RX 0 retrans 0.17 msRTT
n7: 9292.2500 MB / 10.01 sec = 7789.1574 Mbps 49 %TX 26 %RX 0 retrans 0.17 msRTT
n2: 11487.1150 MB / 10.01 sec = 9627.2690 Mbps 49 %TX 46 %RX 0 retrans 0.19 msRTT
Aggregate performance: 100.4637 Gbps
The problem is with the receive side performance.
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n11: 6983.6359 MB / 10.09 sec = 5803.2293 Mbps 13 %TX 26 %RX 0 retrans 0.11 msRTT
n10: 7000.1557 MB / 10.11 sec = 5807.5978 Mbps 13 %TX 26 %RX 0 retrans 0.12 msRTT
n9: 2451.7206 MB / 10.21 sec = 2014.8397 Mbps 4 %TX 13 %RX 0 retrans 0.11 msRTT
n13: 2453.0887 MB / 10.20 sec = 2016.8751 Mbps 3 %TX 11 %RX 0 retrans 0.10 msRTT
n12: 2446.5303 MB / 10.24 sec = 2004.4638 Mbps 4 %TX 11 %RX 0 retrans 0.10 msRTT
n8: 2462.5890 MB / 10.26 sec = 2014.0272 Mbps 3 %TX 11 %RX 0 retrans 0.12 msRTT
n4: 2763.5091 MB / 10.26 sec = 2258.4871 Mbps 4 %TX 14 %RX 0 retrans 0.10 msRTT
n5: 2770.0887 MB / 10.28 sec = 2261.2562 Mbps 4 %TX 15 %RX 0 retrans 0.10 msRTT
n2: 1777.7277 MB / 10.32 sec = 1444.9054 Mbps 2 %TX 11 %RX 0 retrans 0.11 msRTT
n6: 1772.7962 MB / 10.31 sec = 1442.0346 Mbps 3 %TX 10 %RX 0 retrans 0.11 msRTT
n3: 1779.4535 MB / 10.32 sec = 1446.0090 Mbps 2 %TX 11 %RX 0 retrans 0.15 msRTT
n7: 1770.8359 MB / 10.35 sec = 1435.4757 Mbps 2 %TX 11 %RX 0 retrans 0.12 msRTT
Aggregate performance: 29.9492 Gbps
I suspected that this was because the memory being allocated by the
myri10ge driver was not being allocated on the optimum NUMA node.
BTW the NUMA nodes on the system are 0 and 2 instead of 0 and 1 which
is what I would have expected, but this is my first experience with
a NUMA system.
Based upon a patch by Peter Zijlstra that I discovered through Google
searching, I tried patching the myri10ge driver to change its memory
allocation of memory pages from alloc_pages() to alloc_pages_node()
and specifying the NUMA node of the parent device of the Myricom 10-GigE
device, which IIUC should be the PCIe switch. This didn't help.
This could be because I discovered that if I did:
find /sys -name numa_node -exec grep . {} /dev/null \;
that the numa_node associated with all the PCI devices was always 0,
and if IIUC then I believe some of the PCI devices should have been
associated with NUMA node 2. Perhaps this is what is causing all
the memory pages allocated by the myri10ge driver to be on NUMA
node 0, and thus causing the major performance issue.
To kludge around this, I made a different patch to the myri10ge driver.
This time I hardcoded the NUMA node in the call to alloc_pages_node()
to 2 for devices with an IRQ between 113 and 118 (eth2 through eth7)
and to 0 for devices with an IRQ between 119 and 124 (eth8 through eth13).
This is of course very specific to our specific system (NUMA node ids
and Myricom 10-GigE device IRQs), and is not something that would be
generically applicable. But it was useful as a test, and it did
improve the receive side performance substantially!
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n5: 8221.2911 MB / 10.09 sec = 6836.0343 Mbps 17 %TX 31 %RX 0 retrans 0.12 msRTT
n4: 8237.9524 MB / 10.10 sec = 6840.2379 Mbps 16 %TX 31 %RX 0 retrans 0.11 msRTT
n11: 7935.3750 MB / 10.11 sec = 6586.2476 Mbps 15 %TX 29 %RX 0 retrans 0.16 msRTT
n2: 4543.1621 MB / 10.13 sec = 3763.0669 Mbps 9 %TX 21 %RX 0 retrans 0.12 msRTT
n10: 7916.3925 MB / 10.13 sec = 6555.5210 Mbps 15 %TX 28 %RX 0 retrans 0.13 msRTT
n7: 4558.4817 MB / 10.14 sec = 3771.6557 Mbps 7 %TX 22 %RX 0 retrans 0.10 msRTT
n13: 4390.1875 MB / 10.14 sec = 3633.6421 Mbps 6 %TX 21 %RX 0 retrans 0.12 msRTT
n3: 4572.6478 MB / 10.15 sec = 3778.2596 Mbps 9 %TX 21 %RX 0 retrans 0.14 msRTT
n6: 4564.4776 MB / 10.14 sec = 3774.4373 Mbps 9 %TX 21 %RX 0 retrans 0.11 msRTT
n8: 4409.8551 MB / 10.16 sec = 3642.1920 Mbps 8 %TX 19 %RX 0 retrans 0.12 msRTT
n9: 4412.7836 MB / 10.16 sec = 3643.7788 Mbps 8 %TX 20 %RX 0 retrans 0.14 msRTT
n12: 4413.4061 MB / 10.16 sec = 3645.2544 Mbps 8 %TX 21 %RX 0 retrans 0.11 msRTT
Aggregate performance: 56.4703 Gbps
This was basically double the previous receive side performance
without the patch.
I don't know if this is fundamentally a myri10ge driver issue or
some underlying Linux kernel issue, so it's not clear to me what
a proper fix would be.
Finally, while definitely a major improvement, I think it should be
possible to do even better, since we achieved 70 Gbps in the i7 to i7
tests, and probably could have done 80 Gbps except for an Asus
motherboard restriction with the interconnect between the Intel X58
and Nvidia NF200 chips. It's definitely a big step in the right
direction though if this issue can be resolved.
Any help greatly appreicated in advance.
-Thanks
-Bill
^ permalink raw reply
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Paul Congdon (UC Davis) @ 2009-08-07 21:06 UTC (permalink / raw)
To: shemminger, anna.fischer
Cc: arnd, davem, netdev, bridge, adobriyan, virtualization
In-Reply-To: <D2E118A1E7268F449C469EA2F301D92902BC39B2@exil.voltaire.com>
[-- Attachment #1.1: Type: text/plain, Size: 1122 bytes --]
Yaron,
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
Agreed that the hardware solution is preferred so the macvlan implementation doesn’t really matter. If we are talking SR-IOV, then it is direct mapped, regardless of whether there is a VEB or VEPA in the hardware below, so you are bypassing the bridge software code also.
I disagree that adding the multicast handling is simple – while not conceptually hard, it will basically require you to put an address table into the macvlan implementation – if you have that, then why not have just used the one already in the bridge code. If you hook a VEPA up to a non-hairpin mode external bridge, you get the macvlan capability as well.
It also seems to me like the special macvlan interfaces for KVM don’t apply to XEN or a non-virtualized environment? Or more has to be written to make that work? If it is in the bridge code, you get all of this re-use.
[-- Attachment #1.2: Type: text/html, Size: 6831 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge
^ permalink raw reply
* [PATCH net-next] myri10ge: improve parity error detection and recovery
From: Brice Goglin @ 2009-08-07 20:44 UTC (permalink / raw)
To: David S. Miller; +Cc: Linux Network Development list
Improve myri10ge parity error detection and recovery:
1) Don't restore PCI config space to a rebooted NIC until AFTER the
host is quiescent.
2) Let myri10ge_close() know the NIC is dead, so it won't waste time
waiting for a dead nic to respond to MXGEFW_CMD_ETHERNET_DOWN
3) When the NIC is quiet (link down, or otherwise idle link) use
a pci config space read to detect a rebooted NIC. Otherwise
we might never notice that a NIC rebooted
Signed-off-by: Andrew Gallatin <gallatin@myri.com>
Signed-off-by: Brice Goglin <brice@myri.com>
diff -ur /home/bgoglin/src/git/net-next-2.6/drivers/net/myri10ge/myri10ge.c linux-tmp/drivers/net/myri10ge/myri10ge.c
--- net-next-2.6/drivers/net/myri10ge/myri10ge.c 2009-07-22 14:39:45.000000000 +0200
+++ linux-tmp/drivers/net/myri10ge/myri10ge.c 2009-08-07 22:30:05.000000000 +0200
@@ -75,7 +75,7 @@
#include "myri10ge_mcp.h"
#include "myri10ge_mcp_gen_header.h"
-#define MYRI10GE_VERSION_STR "1.5.0-1.418"
+#define MYRI10GE_VERSION_STR "1.5.0-1.432"
MODULE_DESCRIPTION("Myricom 10G driver (10GbE)");
MODULE_AUTHOR("Maintainer: help@myri.com");
@@ -188,6 +188,7 @@
dma_addr_t fw_stats_bus;
int watchdog_tx_done;
int watchdog_tx_req;
+ int watchdog_rx_done;
#ifdef CONFIG_MYRI10GE_DCA
int cached_dca_tag;
int cpu;
@@ -256,6 +257,7 @@
u32 link_changes;
u32 msg_enable;
unsigned int board_number;
+ int rebooted;
};
static char *myri10ge_fw_unaligned = "myri10ge_ethp_z8e.dat";
@@ -2552,17 +2554,22 @@
netif_carrier_off(dev);
netif_tx_stop_all_queues(dev);
- old_down_cnt = mgp->down_cnt;
- mb();
- status = myri10ge_send_cmd(mgp, MXGEFW_CMD_ETHERNET_DOWN, &cmd, 0);
- if (status)
- printk(KERN_ERR "myri10ge: %s: Couldn't bring down link\n",
- dev->name);
-
- wait_event_timeout(mgp->down_wq, old_down_cnt != mgp->down_cnt, HZ);
- if (old_down_cnt == mgp->down_cnt)
- printk(KERN_ERR "myri10ge: %s never got down irq\n", dev->name);
+ if (mgp->rebooted == 0) {
+ old_down_cnt = mgp->down_cnt;
+ mb();
+ status =
+ myri10ge_send_cmd(mgp, MXGEFW_CMD_ETHERNET_DOWN, &cmd, 0);
+ if (status)
+ printk(KERN_ERR
+ "myri10ge: %s: Couldn't bring down link\n",
+ dev->name);
+ wait_event_timeout(mgp->down_wq, old_down_cnt != mgp->down_cnt,
+ HZ);
+ if (old_down_cnt == mgp->down_cnt)
+ printk(KERN_ERR "myri10ge: %s never got down irq\n",
+ dev->name);
+ }
netif_tx_disable(dev);
myri10ge_free_irq(mgp);
for (i = 0; i < mgp->num_slices; i++)
@@ -3427,12 +3434,13 @@
container_of(work, struct myri10ge_priv, watchdog_work);
struct myri10ge_tx_buf *tx;
u32 reboot;
- int status;
+ int status, rebooted;
int i;
u16 cmd, vendor;
mgp->watchdog_resets++;
pci_read_config_word(mgp->pdev, PCI_COMMAND, &cmd);
+ rebooted = 0;
if ((cmd & PCI_COMMAND_MASTER) == 0) {
/* Bus master DMA disabled? Check to see
* if the card rebooted due to a parity error
@@ -3444,9 +3452,12 @@
myri10ge_reset_recover ? " " : " not");
if (myri10ge_reset_recover == 0)
return;
-
+ rtnl_lock();
+ mgp->rebooted = 1;
+ rebooted = 1;
+ myri10ge_close(mgp->dev);
myri10ge_reset_recover--;
-
+ mgp->rebooted = 0;
/*
* A rebooted nic will come back with config space as
* it was after power was applied to PCIe bus.
@@ -3494,8 +3505,10 @@
}
}
- rtnl_lock();
- myri10ge_close(mgp->dev);
+ if (!rebooted) {
+ rtnl_lock();
+ myri10ge_close(mgp->dev);
+ }
status = myri10ge_load_firmware(mgp, 1);
if (status != 0)
printk(KERN_ERR "myri10ge: %s: failed to load firmware\n",
@@ -3516,12 +3529,14 @@
{
struct myri10ge_priv *mgp;
struct myri10ge_slice_state *ss;
- int i, reset_needed;
+ int i, reset_needed, busy_slice_cnt;
u32 rx_pause_cnt;
+ u16 cmd;
mgp = (struct myri10ge_priv *)arg;
rx_pause_cnt = ntohl(mgp->ss[0].fw_stats->dropped_pause);
+ busy_slice_cnt = 0;
for (i = 0, reset_needed = 0;
i < mgp->num_slices && reset_needed == 0; ++i) {
@@ -3559,8 +3574,22 @@
reset_needed = 1;
}
}
+ if (ss->watchdog_tx_done != ss->tx.done ||
+ ss->watchdog_rx_done != ss->rx_done.cnt) {
+ busy_slice_cnt++;
+ }
ss->watchdog_tx_done = ss->tx.done;
ss->watchdog_tx_req = ss->tx.req;
+ ss->watchdog_rx_done = ss->rx_done.cnt;
+ }
+ /* if we've sent or received no traffic, poll the NIC to
+ * ensure it is still there. Otherwise, we risk not noticing
+ * an error in a timely fashion */
+ if (busy_slice_cnt == 0) {
+ pci_read_config_word(mgp->pdev, PCI_COMMAND, &cmd);
+ if ((cmd & PCI_COMMAND_MASTER) == 0) {
+ reset_needed = 1;
+ }
}
mgp->watchdog_pause = rx_pause_cnt;
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 21:18 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807170600.9a2eff2e.billfink@mindspring.com>
Bill Fink wrote:
> This could be because I discovered that if I did:
>
> find /sys -name numa_node -exec grep . {} /dev/null \;
>
> that the numa_node associated with all the PCI devices was always 0,
> and if IIUC then I believe some of the PCI devices should have been
> associated with NUMA node 2. Perhaps this is what is causing all
> the memory pages allocated by the myri10ge driver to be on NUMA
> node 0, and thus causing the major performance issue.
>
I've seen some cases in the past where numa_node was always 0 on
quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
later kernels thanks to patches from Yinghai Lu (CC'ed).
Is the corresponding local_cpus sysfs file wrong as well ?
Maybe your kernel doesn't properly handle the NUMA location of PCI
devices on Nehalem machines yet?
Brice
^ permalink raw reply
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Stephen Hemminger @ 2009-08-07 21:36 UTC (permalink / raw)
To: Paul Congdon (UC Davis)
Cc: anna.fischer, bridge, netdev, virtualization, davem, kaber,
adobriyan, arnd
In-Reply-To: <000001ca17a3$03f841f0$0be8c5d0$@edu>
On Fri, 7 Aug 2009 14:06:58 -0700
"Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
> Yaron,
>
>
> The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
>
> Agreed that the hardware solution is preferred so the macvlan implementation doesn’t really matter. If we are talking SR-IOV, then it is direct mapped, regardless of whether there is a VEB or VEPA in the hardware below, so you are bypassing the bridge software code also.
>
> I disagree that adding the multicast handling is simple – while not conceptually hard, it will basically require you to put an address table into the macvlan implementation – if you have that, then why not have just used the one already in the bridge code. If you hook a VEPA up to a non-hairpin mode external bridge, you get the macvlan capability as well.
I have a patch that forwards all multicast packets, and another that does
proper forwarding. It should have worked that way in original macvlan, the
current behavior is really a bug.
--
^ permalink raw reply
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-08-07 21:38 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Paul Congdon (UC Davis), drobbins, 'Fischer, Anna',
herbert, mst, netdev, bridge, linux-kernel, ogerlitz, evb, davem
In-Reply-To: <20090807123554.7c2bc27c@nehalam>
On Friday 07 August 2009, Stephen Hemminger wrote:
> On Fri, 7 Aug 2009 12:10:07 -0700
> "Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
>
> > Responding to Daniel's questions...
> >
> > > I have some general questions about the intended use and benefits of
> > > VEPA, from an IT perspective:
> > >
> > > In which virtual machine setups and technologies do you forsee this
> > > interface being used?
> >
> > The benefit of VEPA is the coordination and unification with the
> > external network switch. So, in environments where you are
> > needing/wanting your feature rich, wire speed, external network
> > device (firewall/switch/IPS/content-filter) to provide consistent
> > policy enforcement, and you want your VMs traffic to be subject to
> > that enforcement, you will want their traffic directed externally.
> > Perhaps you have some VMs that are on a DMZ or clustering an
> > application or implementing a multi-tier application where you
> > would normally place a firewall in-between the tiers.
>
> I do have to raise the point that Linux is perfectly capable of keeping up without
> the need of an external switch. Whether you want policy external or internal is
> a architecture decision that should not be driven by mis-information about performance.
In general, I agree that Linux on a decent virtual machine host will be
able to handle forwarding of network data fast enough, often faster than
the external connectivity allows if it needs to transmit every frame twice.
However, there is a tradeoff between CPU cycles and I/O bandwidth. If your
application needs lots of CPU but you have spare capacity on the PCI bus, the
network wire and the external switch, VEPA can also be a win on the performance
side. As always, performance depends on the application, even if it's not the
main driving factor here.
Arnd <><
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 21:51 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7C9A14.7070600@inria.fr>
On Fri, 07 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> > This could be because I discovered that if I did:
> >
> > find /sys -name numa_node -exec grep . {} /dev/null \;
> >
> > that the numa_node associated with all the PCI devices was always 0,
> > and if IIUC then I believe some of the PCI devices should have been
> > associated with NUMA node 2. Perhaps this is what is causing all
> > the memory pages allocated by the myri10ge driver to be on NUMA
> > node 0, and thus causing the major performance issue.
> >
>
> I've seen some cases in the past where numa_node was always 0 on
> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
> later kernels thanks to patches from Yinghai Lu (CC'ed).
By later kernels do you mean 2.6.30 or 2.6.31?
> Is the corresponding local_cpus sysfs file wrong as well ?
All sysfs local_cpus values are the same (00000000,000000ff),
so yes they are also wrong.
> Maybe your kernel doesn't properly handle the NUMA location of PCI
> devices on Nehalem machines yet?
I assume so, unless there's some secret NUMA system setting that
I'm unaware of that would affect this and needs changing for my
setup.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 21:53 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807175112.a1f57407.billfink@mindspring.com>
Bill Fink wrote:
>> I've seen some cases in the past where numa_node was always 0 on
>> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
>> later kernels thanks to patches from Yinghai Lu (CC'ed).
>>
>
> By later kernels do you mean 2.6.30 or 2.6.31?
>
No, I meant "later than when the problem occured". I was using 2.6.22 at
this point and the problem was fixed somewhere around 2.6.25.
>> Is the corresponding local_cpus sysfs file wrong as well ?
>>
>
> All sysfs local_cpus values are the same (00000000,000000ff),
> so yes they are also wrong.
>
And hyperthreading is enabled, right?
Brice
^ permalink raw reply
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-08-07 22:05 UTC (permalink / raw)
To: Paul Congdon (UC Davis)
Cc: drobbins, 'Fischer, Anna', herbert, mst, netdev, bridge,
linux-kernel, ogerlitz, evb, davem
In-Reply-To: <004f01ca1792$b24a7a90$16df6fb0$@edu>
On Friday 07 August 2009, Paul Congdon (UC Davis) wrote:
> Responding to Daniel's questions...
Thanks for the detailed responses. I'll add some more about the
specifics of the macvlan implementation that differs from the
bridge based VEPA implementation.
> > Is this new interface to be used within a virtual machine or
> > container, on the master node, or both?
>
> It is really an interface to a new type of virtual switch. When
> you create virtual network, I would imagine it being a new mode
> of operation (bridge, NAT, VEPA, etc).
I think the question was whether the patch needs to applied in the
host or the guest. Both the implementation that you and Anna did
and the one that I posted only apply to the *host* (master node),
the virtual machine does not need to know about it.
> > What interface(s) would need to be configured for a single virtual
> > machine to use VEPA to access the network?
>
> It would be the same as if that machine were configure to use a
> bridge to access the network, but the bridge mode would be different.
Right, with the bridge based VEPA, you would set up a kvm guest
or a container with the regular tools, then use the sysfs interface
to put the bridge device into VEPA mode.
With the macvlan based mode, you use 'ip link' to add a new tap
device to an external network interface and not use a bridge at
all. Then you configure KVM to use that tap device instead of the
regular bridge/tap setup.
> > What are the current flexibility, security or performance limitations
> > of tun/tap and bridge that make this new interface necessary or
> > beneficial?
>
> If you have VMs that will be communicating with one another on
> the same physical machine, and you want their traffic to be
> exposed to an in-line network device such as a application
> firewall/IPS/content-filter (without this feature) you will have
> to have this device co-located within the same physical server.
> This will use up CPU cycles that you presumable purchased to run
> applications, it will require a lot of consistent configuration
> on all physical machines, it could invoke potentially a lot of
> software licensing, additional cost, etc.. Everything would
> need to be replicated on each physical machine. With the VEPA
> capability, you can leverage all this functionality in an
> external network device and have it managed and configured in
> one place. The external implementation is likely a higher
> performance, silicon based implementation. It should make it
> easier to migrate machines from one physical server to another
> and maintain the same network policy enforcement.
It's worth noting that depending on your network connectivity,
performance is likely to go down significantly with VEPA over the
existing bridge/tap setup, because all frames have to be sent
twice through an external wire that has a limited capacity, so you may
lose inter-guest bandwidth and get more latency in many cases, while
you free up CPU cycles. With the bridge based VEPA, you might not
even gain many cycles because much of the overhead is still there.
On the cost side, external switches can also get quite expensive
compared to x86 servers.
IMHO the real win of VEPA is on the management side, where you can
use a single set of tool for managing the network, rather than
having your network admins deal with both the external switches
and the setup of linux netfilter rules etc.
The macvlan based VEPA has the same features as the bridge based
VEPA, but much simpler code, which allows a number of shortcuts
to save CPU cycles.
> The isolation in the outbound direction is created by the way frames
> are forwarded. They are simply dropped on the wire, so no VMs can
> talk directly to one another without their traffic first going
> external. In the inbound direction, the isolation is created using
> the forwarding table.
Right. Note that in the macvlan case, the filtering on inbound data is an inherent
part of the macvlan setup, it does use the dynamic forwarding table of the
bridge driver.
> > Are there any associated user-space tools required for configuring a
> > VEPA?
> >
>
> The standard brctl utility has been augmented to enable/disable the capability.
That is for the bridge based VEPA, while my patch uses the 'ip link'
command that ships with most distros. It does not need any modifications
right now, but might need them if we add other features like support for
multiple MAC addresses in a single guest.
Arnd <><
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 22:08 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7CA24E.4080503@inria.fr>
On Fri, 07 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> >> I've seen some cases in the past where numa_node was always 0 on
> >> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
> >> later kernels thanks to patches from Yinghai Lu (CC'ed).
> >
> > By later kernels do you mean 2.6.30 or 2.6.31?
>
> No, I meant "later than when the problem occured". I was using 2.6.22 at
> this point and the problem was fixed somewhere around 2.6.25.
OK. The tests were run on a 2.6.29.6 kernel so presumably should
have included the fix you mentioned.
> >> Is the corresponding local_cpus sysfs file wrong as well ?
> >
> > All sysfs local_cpus values are the same (00000000,000000ff),
> > so yes they are also wrong.
>
> And hyperthreading is enabled, right?
No, hyperthreading is disabled. It's a dual quad-core system so there
are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-07 22:12 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090807170600.9a2eff2e.billfink@mindspring.com>
On Fri, Aug 07, 2009 at 05:06:00PM -0400, Bill Fink wrote:
> I've run into a major receive side performance issue with multi-10-GigE
> on a NUMA system. The system is using a SuperMicro X8DAH+-F motherboard
> with 2 3.2 GHz quad-core Intel Xeon 5580 processors and 12 GB of
> 1333 MHz DDR3 memory. It is a Fedora 10 system but using the latest
> 2.6.29.6 kernel from Fedora 11 (originally tried the 2.6.27.29 kernel
> from Fedora 10).
>
> The test setup is:
>
> i7test1----(6)----xeontest1----(6)----i7test2
> 10-GigE 10-GigE
>
> So xeontest1 has 6 dual-port Myricom 10-GigE NICs for a total
> of 12 10-GigE interfaces. eth2 through eth7 (which are on the
> second Intel 5520 I/O Hub) are connected to i7test1 while
> eth8 through eth13 (which are on the first Intel 5520 I/O Hub)
> are connected to i7test2.
>
> Previous direct testing between i7test1 and i7test2 (which use an
> Asus P6T6 WS Revolution motherboard) demonstrated that they could
> achieve ~70 Gbps performance for either transmit or receive using
> 8 10-GigE interfaces.
>
> The transmit side performance of xeontest1 is fantastic:
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -xc6/3 -p5012 192.168.12.11 &
> n12: 9648.0522 MB / 10.00 sec = 8091.4066 Mbps 49 %TX 26 %RX 0 retrans 0.18 msRTT
> n9: 11130.5320 MB / 10.01 sec = 9328.3224 Mbps 47 %TX 37 %RX 0 retrans 0.19 msRTT
> n11: 9418.1250 MB / 10.00 sec = 7897.5848 Mbps 50 %TX 30 %RX 0 retrans 0.18 msRTT
> n10: 9279.4758 MB / 10.01 sec = 7778.7146 Mbps 49 %TX 28 %RX 0 retrans 0.12 msRTT
> n8: 11142.6574 MB / 10.01 sec = 9340.3789 Mbps 47 %TX 35 %RX 0 retrans 0.18 msRTT
> n13: 9422.1492 MB / 10.01 sec = 7897.4115 Mbps 49 %TX 25 %RX 0 retrans 0.17 msRTT
> n3: 11471.2500 MB / 10.01 sec = 9613.9477 Mbps 49 %TX 32 %RX 0 retrans 0.15 msRTT
> n6: 9339.6354 MB / 10.01 sec = 7828.5345 Mbps 50 %TX 25 %RX 0 retrans 0.19 msRTT
> n4: 9093.2500 MB / 10.01 sec = 7624.1589 Mbps 49 %TX 28 %RX 0 retrans 0.15 msRTT
> n5: 9121.8367 MB / 10.01 sec = 7646.8646 Mbps 50 %TX 29 %RX 0 retrans 0.17 msRTT
> n7: 9292.2500 MB / 10.01 sec = 7789.1574 Mbps 49 %TX 26 %RX 0 retrans 0.17 msRTT
> n2: 11487.1150 MB / 10.01 sec = 9627.2690 Mbps 49 %TX 46 %RX 0 retrans 0.19 msRTT
>
> Aggregate performance: 100.4637 Gbps
>
> The problem is with the receive side performance.
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
> n11: 6983.6359 MB / 10.09 sec = 5803.2293 Mbps 13 %TX 26 %RX 0 retrans 0.11 msRTT
> n10: 7000.1557 MB / 10.11 sec = 5807.5978 Mbps 13 %TX 26 %RX 0 retrans 0.12 msRTT
> n9: 2451.7206 MB / 10.21 sec = 2014.8397 Mbps 4 %TX 13 %RX 0 retrans 0.11 msRTT
> n13: 2453.0887 MB / 10.20 sec = 2016.8751 Mbps 3 %TX 11 %RX 0 retrans 0.10 msRTT
> n12: 2446.5303 MB / 10.24 sec = 2004.4638 Mbps 4 %TX 11 %RX 0 retrans 0.10 msRTT
> n8: 2462.5890 MB / 10.26 sec = 2014.0272 Mbps 3 %TX 11 %RX 0 retrans 0.12 msRTT
> n4: 2763.5091 MB / 10.26 sec = 2258.4871 Mbps 4 %TX 14 %RX 0 retrans 0.10 msRTT
> n5: 2770.0887 MB / 10.28 sec = 2261.2562 Mbps 4 %TX 15 %RX 0 retrans 0.10 msRTT
> n2: 1777.7277 MB / 10.32 sec = 1444.9054 Mbps 2 %TX 11 %RX 0 retrans 0.11 msRTT
> n6: 1772.7962 MB / 10.31 sec = 1442.0346 Mbps 3 %TX 10 %RX 0 retrans 0.11 msRTT
> n3: 1779.4535 MB / 10.32 sec = 1446.0090 Mbps 2 %TX 11 %RX 0 retrans 0.15 msRTT
> n7: 1770.8359 MB / 10.35 sec = 1435.4757 Mbps 2 %TX 11 %RX 0 retrans 0.12 msRTT
>
> Aggregate performance: 29.9492 Gbps
>
> I suspected that this was because the memory being allocated by the
> myri10ge driver was not being allocated on the optimum NUMA node.
> BTW the NUMA nodes on the system are 0 and 2 instead of 0 and 1 which
> is what I would have expected, but this is my first experience with
> a NUMA system.
>
> Based upon a patch by Peter Zijlstra that I discovered through Google
> searching, I tried patching the myri10ge driver to change its memory
> allocation of memory pages from alloc_pages() to alloc_pages_node()
> and specifying the NUMA node of the parent device of the Myricom 10-GigE
> device, which IIUC should be the PCIe switch. This didn't help.
>
> This could be because I discovered that if I did:
>
> find /sys -name numa_node -exec grep . {} /dev/null \;
>
> that the numa_node associated with all the PCI devices was always 0,
> and if IIUC then I believe some of the PCI devices should have been
> associated with NUMA node 2. Perhaps this is what is causing all
> the memory pages allocated by the myri10ge driver to be on NUMA
> node 0, and thus causing the major performance issue.
>
> To kludge around this, I made a different patch to the myri10ge driver.
> This time I hardcoded the NUMA node in the call to alloc_pages_node()
> to 2 for devices with an IRQ between 113 and 118 (eth2 through eth7)
> and to 0 for devices with an IRQ between 119 and 124 (eth8 through eth13).
> This is of course very specific to our specific system (NUMA node ids
> and Myricom 10-GigE device IRQs), and is not something that would be
> generically applicable. But it was useful as a test, and it did
> improve the receive side performance substantially!
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
> n5: 8221.2911 MB / 10.09 sec = 6836.0343 Mbps 17 %TX 31 %RX 0 retrans 0.12 msRTT
> n4: 8237.9524 MB / 10.10 sec = 6840.2379 Mbps 16 %TX 31 %RX 0 retrans 0.11 msRTT
> n11: 7935.3750 MB / 10.11 sec = 6586.2476 Mbps 15 %TX 29 %RX 0 retrans 0.16 msRTT
> n2: 4543.1621 MB / 10.13 sec = 3763.0669 Mbps 9 %TX 21 %RX 0 retrans 0.12 msRTT
> n10: 7916.3925 MB / 10.13 sec = 6555.5210 Mbps 15 %TX 28 %RX 0 retrans 0.13 msRTT
> n7: 4558.4817 MB / 10.14 sec = 3771.6557 Mbps 7 %TX 22 %RX 0 retrans 0.10 msRTT
> n13: 4390.1875 MB / 10.14 sec = 3633.6421 Mbps 6 %TX 21 %RX 0 retrans 0.12 msRTT
> n3: 4572.6478 MB / 10.15 sec = 3778.2596 Mbps 9 %TX 21 %RX 0 retrans 0.14 msRTT
> n6: 4564.4776 MB / 10.14 sec = 3774.4373 Mbps 9 %TX 21 %RX 0 retrans 0.11 msRTT
> n8: 4409.8551 MB / 10.16 sec = 3642.1920 Mbps 8 %TX 19 %RX 0 retrans 0.12 msRTT
> n9: 4412.7836 MB / 10.16 sec = 3643.7788 Mbps 8 %TX 20 %RX 0 retrans 0.14 msRTT
> n12: 4413.4061 MB / 10.16 sec = 3645.2544 Mbps 8 %TX 21 %RX 0 retrans 0.11 msRTT
>
> Aggregate performance: 56.4703 Gbps
>
> This was basically double the previous receive side performance
> without the patch.
>
> I don't know if this is fundamentally a myri10ge driver issue or
> some underlying Linux kernel issue, so it's not clear to me what
> a proper fix would be.
>
> Finally, while definitely a major improvement, I think it should be
> possible to do even better, since we achieved 70 Gbps in the i7 to i7
> tests, and probably could have done 80 Gbps except for an Asus
> motherboard restriction with the interconnect between the Intel X58
> and Nvidia NF200 chips. It's definitely a big step in the right
> direction though if this issue can be resolved.
>
> Any help greatly appreicated in advance.
>
> -Thanks
>
> -Bill
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
You're timing is impeccable! I just posted a patch for an ftrace module to help
detect just these kind of conditions:
http://marc.info/?l=linux-netdev&m=124967650218846&w=2
Hope that helps you out
Neil
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 22:17 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807180840.b27ce794.billfink@mindspring.com>
Bill Fink wrote:
> OK. The tests were run on a 2.6.29.6 kernel so presumably should
> have included the fix you mentioned.
>
Yes, but I wanted to emphasize that new platforms sometime need some new
code to handle this kind of things. Some Nehalem-specific changes might
be needed now.
>>>> Is the corresponding local_cpus sysfs file wrong as well ?
>>>>
>>> All sysfs local_cpus values are the same (00000000,000000ff),
>>> so yes they are also wrong.
>>>
>> And hyperthreading is enabled, right?
>>
>
> No, hyperthreading is disabled. It's a dual quad-core system so there
> are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
>
So numa_node says that the device is close to node 0 while local_cpus
says that it's close to all 8 cores ie close to both node0 and node2
(which may well be wrong as well).
Brice
^ permalink raw reply
* 2009 Prize Award!!
From: kpairote @ 2009-08-07 21:50 UTC (permalink / raw)
2009 Prize Award!!
Winning/Confirm!!
This email is to notify you that you have won an Award Sum of
€500,000.00 (Five Hundred Thousand Euros)in an E-mail program held in
Den Haag, The Netherlands.
Please contact the claim officer with your winning info. REF
NUM.(NL80246) BATCH NUM.(EU-0375908NL) TICKET NUM.(460205)Name;Mr.Paul
Hanson,TEL.+31-643608961,E-mail:spbg09@aol.nl.
Yours Sincerely,
Mrs.Sandra Sleeswijk,
Public Relation Officer.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
-- Computer Center PSU PATTANI --
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 22:55 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7CA7F0.60704@inria.fr>
On Sat, 08 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> > OK. The tests were run on a 2.6.29.6 kernel so presumably should
> > have included the fix you mentioned.
>
> Yes, but I wanted to emphasize that new platforms sometime need some new
> code to handle this kind of things. Some Nehalem-specific changes might
> be needed now.
Thanks for the clarification.
> >>>> Is the corresponding local_cpus sysfs file wrong as well ?
> >>>>
> >>> All sysfs local_cpus values are the same (00000000,000000ff),
> >>> so yes they are also wrong.
> >>>
> >> And hyperthreading is enabled, right?
> >>
> >
> > No, hyperthreading is disabled. It's a dual quad-core system so there
> > are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
>
> So numa_node says that the device is close to node 0 while local_cpus
> says that it's close to all 8 cores ie close to both node0 and node2
> (which may well be wrong as well).
I believe it is wrong. The basic system arcitecture is:
Memory----CPU1----QPI----CPU2----Memory
| |
| |
QPI QPI
| |
| |
5520----QPI----5520
|||| ||||
|||| ||||
|||| ||||
PCIe PCIe
There are 2 x8, 1 x16, and 1 x4 PCIe 2.0 interfaces on each of the
Intel 5520 I/O Hubs. The Myricom dual-port 10-GigE NICs are in the
six x8 or better slots. eth2 through eth7 are on the second
Intel 5520 I/O Hub, so they should presumably show up on NUMA node 2,
and have local CPUs 1, 3, 5, and 7. eth8 through eth13 are on the
first Intel 5520 I/O Hub, and thus should be on NUMA node 0 with
local CPUs 0, 2, 4, and 6 (CPU info derived from /proc/cpinfo).
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-08 0:54 UTC (permalink / raw)
To: Neil Horman; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090807221211.GA16874@localhost.localdomain>
On Fri, 7 Aug 2009, Neil Horman wrote:
> You're timing is impeccable! I just posted a patch for an ftrace module to help
> detect just these kind of conditions:
> http://marc.info/?l=linux-netdev&m=124967650218846&w=2
>
> Hope that helps you out
> Neil
Thanks! It could be helpful. Do you have a pointer to documentation
on how to use it? And does it require the latest GIT kernel or could
it possibly be used with a 2.6.29.6 kernel?
-Bill
^ permalink raw reply
* Re: [PATCH] korina: Read buffer overflow
From: Phil Sutter @ 2009-08-08 0:48 UTC (permalink / raw)
To: Roel Kluin; +Cc: netdev, Andrew Morton, David S. Miller, florian
In-Reply-To: <4A7C494B.2060204@gmail.com>
Hi,
On Fri, Aug 07, 2009 at 05:33:31PM +0200, Roel Kluin wrote:
> If the loop breaks with an i of 0, then we read lp->rd_ring[-1].
Indeed.
> Signed-off-by: Roel Kluin <roel.kluin@gmail.com>
> ---
> Should we clean up like this? please review
>
> diff --git a/drivers/net/korina.c b/drivers/net/korina.c
> index b4cf602..b965b2b 100644
> --- a/drivers/net/korina.c
> +++ b/drivers/net/korina.c
> @@ -754,7 +754,7 @@ static void korina_alloc_ring(struct net_device *dev)
> {
> struct korina_private *lp = netdev_priv(dev);
> struct sk_buff *skb;
> - int i;
> + int i, j;
>
> /* Initialize the transmit descriptors */
> for (i = 0; i < KORINA_NUM_TDS; i++) {
> @@ -771,7 +771,7 @@ static void korina_alloc_ring(struct net_device *dev)
> for (i = 0; i < KORINA_NUM_RDS; i++) {
> skb = dev_alloc_skb(KORINA_RBSIZE + 2);
> if (!skb)
> - break;
> + goto err_free;
This implies that all KORINA_NUM_TDS receive descriptors need to be
available for the driver to work.
> skb_reserve(skb, 2);
> lp->rx_skb[i] = skb;
> lp->rd_ring[i].control = DMA_DESC_IOD |
> @@ -790,6 +790,12 @@ static void korina_alloc_ring(struct net_device *dev)
> lp->rx_chain_head = 0;
> lp->rx_chain_tail = 0;
> lp->rx_chain_status = desc_empty;
> +err_free:
> + for (j = 0; j < i; j++) {
> + lp->rd_ring[j].control = 0;
> + dev_kfree_skb_any(lp->rx_skb[j]);
> + lp->rx_skb[j] = NULL;
> + }
> }
Also I guess there should be some error handling, as the driver probably
wont be useful without a single receive descriptor. What do you think
about the following:
| diff --git a/drivers/net/korina.c b/drivers/net/korina.c
| index a2701f5..d1c5276 100644
| --- a/drivers/net/korina.c
| +++ b/drivers/net/korina.c
| @@ -741,7 +741,7 @@ static struct ethtool_ops netdev_ethtool_ops = {
| .get_link = netdev_get_link,
| };
|
| -static void korina_alloc_ring(struct net_device *dev)
| +static int korina_alloc_ring(struct net_device *dev)
| {
| struct korina_private *lp = netdev_priv(dev);
| struct sk_buff *skb;
| @@ -772,6 +772,13 @@ static void korina_alloc_ring(struct net_device *dev)
| lp->rd_ring[i].ca = CPHYSADDR(skb->data);
| lp->rd_ring[i].link = CPHYSADDR(&lp->rd_ring[i+1]);
| }
| + if (!i) {
| + printk(KERN_ERR DRV_NAME "%s: could not allocate a single"
| + " receive descriptor\n", dev->name)
| + return 1;
| + }
| + printk(KERN_DEBUG DRV_NAME "%s: allocated %d receive descriptors\n",
| + dev->name, i);
|
| /* loop back receive descriptors, so the last
| * descriptor points to the first one */
| @@ -782,6 +789,8 @@ static void korina_alloc_ring(struct net_device *dev)
| lp->rx_chain_head = 0;
| lp->rx_chain_tail = 0;
| lp->rx_chain_status = desc_empty;
| +
| + return 0;
| }
|
| static void korina_free_ring(struct net_device *dev)
| @@ -824,7 +833,8 @@ static int korina_init(struct net_device *dev)
| writel(ETH_INT_FC_EN, &lp->eth_regs->ethintfc);
|
| /* Allocate rings */
| - korina_alloc_ring(dev);
| + if (korina_alloc_ring(dev))
| + return 1;
|
| writel(0, &lp->rx_dma_regs->dmas);
| /* Start Rx DMA */
Greetings, Phil
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-08 1:35 UTC (permalink / raw)
To: Andrew Gallatin; +Cc: Brice Goglin, Linux Network Developers, Yinghai Lu
In-Reply-To: <4A7CCEFC.7020308@myri.com>
On Fri, 07 Aug 2009, Andrew Gallatin wrote:
> Bill Fink wrote:
>
> > All sysfs local_cpus values are the same (00000000,000000ff),
> > so yes they are also wrong.
>
> How were you handling IRQ binding? If local_cpus is wrong,
> the irqbalance will not be able to make good decisions about
> where to bind the NICs' IRQs. Did you try manually binding
> each NICs's interrupt to a separate CPU on the correct node?
Yes, all the NIC IRQs were bound to a CPU on the local NUMA node,
and the nuttcp application had its CPU affinity set to the same
CPU with its memory affinity bound to the same local NUMA node.
And the irqbalance daemon wasn't running.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Andrew Gallatin @ 2009-08-08 1:03 UTC (permalink / raw)
To: Bill Fink; +Cc: Brice Goglin, Linux Network Developers, Yinghai Lu
In-Reply-To: <20090807175112.a1f57407.billfink@mindspring.com>
Bill Fink wrote:
> All sysfs local_cpus values are the same (00000000,000000ff),
> so yes they are also wrong.
How were you handling IRQ binding? If local_cpus is wrong,
the irqbalance will not be able to make good decisions about
where to bind the NICs' IRQs. Did you try manually binding
each NICs's interrupt to a separate CPU on the correct node?
Regards,
Drew
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-08 1:56 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090807205442.32918186.billfink@mindspring.com>
On Fri, Aug 07, 2009 at 08:54:42PM -0400, Bill Fink wrote:
> On Fri, 7 Aug 2009, Neil Horman wrote:
>
> > You're timing is impeccable! I just posted a patch for an ftrace module to help
> > detect just these kind of conditions:
> > http://marc.info/?l=linux-netdev&m=124967650218846&w=2
> >
> > Hope that helps you out
> > Neil
>
> Thanks! It could be helpful. Do you have a pointer to documentation
> on how to use it? And does it require the latest GIT kernel or could
> it possibly be used with a 2.6.29.6 kernel?
>
> -Bill
>
It should apply to 2.6.29.6 no problem (might take a little massaging, but not
much).
No docs I'm afraid (sorry, I'm horrible about that)
Using it is easy though:
1) Patch, build and boot the kernel (make sure to have
CONFIG_SKB_SOURCES_TRACER, along with the other FTRACE requisite options)
2) mount -t debugfs nodev /sys/kernel/debug
3) cd /sys/kernel/debug/tracing
4) echo skb_sources > ./current_tracer
5) echo 1 > trace
6) cat ./trace
Step 5 clears the trace buffer. Step 6 provides you a list list this
PID ANID CNID RXQ CCPU LEN
Where:
PID - The process receiving an skb
ANID - The node which the skb being received was allocated on
CNID - The node which the process is running when it read this skb
RQQ - The NIC receive queue that received this skb
CCPU - The cpu the process was running on when it read the skb in question
LEN - The length of the skb being received
Each entry in the list denotes a unique skb (obviously), and with a clever awk
script you can identify which nodes each process in your system is receiving
frames from, so that you can use numactl or taskset to bias that process to run
on the same nodes cpus.
Note that step (6) wil show a larger list each time you cat that file (as trace
records aren't removed during a read. Step 5 is what actually clears the trace
buffer and resets the list length to zero.
Hope that helps. Please feel free to email me if you have any questions.
Regards
Neil
^ permalink raw reply
* [RESEND PATCH] gianfar: keep vlan related state when restart
From: Yong Zhang @ 2009-08-08 2:36 UTC (permalink / raw)
To: davem, afleming, dai.haruki; +Cc: netdev, linux-kernel
If vlan has been enabled. ifdown followed by ifup will lost hardware
related state.
Also remove duplicated operation in gfar_vlan_rx_register().
Signed-off-by: Yong Zhang <yong.zhang0@gmail.com>
---
drivers/net/gianfar.c | 12 ++++++++++--
1 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
index f8ffcbf..e212f2c 100644
--- a/drivers/net/gianfar.c
+++ b/drivers/net/gianfar.c
@@ -936,6 +936,7 @@ int startup_gfar(struct net_device *dev)
struct gfar __iomem *regs = priv->regs;
int err = 0;
u32 rctrl = 0;
+ u32 tctrl = 0;
u32 attrs = 0;
gfar_write(®s->imask, IMASK_INIT_CLEAR);
@@ -1111,11 +1112,19 @@ int startup_gfar(struct net_device *dev)
rctrl |= RCTRL_PADDING(priv->padding);
}
+ /* keep vlan related bits if it's enabled */
+ if (priv->vlgrp) {
+ rctrl |= RCTRL_VLEX | RCTRL_PRSDEP_INIT;
+ tctrl |= TCTRL_VLINS;
+ }
+
/* Init rctrl based on our settings */
gfar_write(&priv->regs->rctrl, rctrl);
if (dev->features & NETIF_F_IP_CSUM)
- gfar_write(&priv->regs->tctrl, TCTRL_INIT_CSUM);
+ tctrl |= TCTRL_INIT_CSUM;
+
+ gfar_write(&priv->regs->tctrl, tctrl);
/* Set the extraction length and index */
attrs = ATTRELI_EL(priv->rx_stash_size) |
@@ -1450,7 +1459,6 @@ static void gfar_vlan_rx_register(struct net_device *dev,
/* Enable VLAN tag extraction */
tempval = gfar_read(&priv->regs->rctrl);
- tempval |= RCTRL_VLEX;
tempval |= (RCTRL_VLEX | RCTRL_PRSDEP_INIT);
gfar_write(&priv->regs->rctrl, tempval);
} else {
--
1.6.0.4
^ permalink raw reply related
* [PATCH] drivers/net/tokenring: Use status field rather than state field
From: Julia Lawall @ 2009-08-08 6:47 UTC (permalink / raw)
To: netdev, linux-kernel, kernel-janitors
From: Julia Lawall <julia@diku.dk>
In general in this file, it is the status field, not the state field, that
contains values like OPEN and CLOSED. Indeed, in the first error case, it
is the field status that is initialized. I have thus assumed that all of
the error handling code should be the same, and moved it to the end of the
function to emphasize its commonality.
Signed-off-by: Julia Lawall <julia@diku.dk>
---
drivers/net/tokenring/smctr.c | 36 +++++--------------
1 files changed, 10 insertions(+), 26 deletions(-)
diff --git a/drivers/net/tokenring/smctr.c b/drivers/net/tokenring/smctr.c
index 6515894..f789f51 100644
--- a/drivers/net/tokenring/smctr.c
+++ b/drivers/net/tokenring/smctr.c
@@ -3091,11 +3091,7 @@ static int smctr_lobe_media_test(struct net_device *dev)
/* Setup the lobe media test. */
smctr_lobe_media_test_cmd(dev);
if(smctr_wait_cmd(dev))
- {
- smctr_reset_adapter(dev);
- tp->status = CLOSED;
- return (LOBE_MEDIA_TEST_FAILED);
- }
+ goto err;
/* Tx lobe media test frames. */
for(i = 0; i < 1500; ++i)
@@ -3103,20 +3099,12 @@ static int smctr_lobe_media_test(struct net_device *dev)
if(smctr_send_lobe_media_test(dev))
{
if(perror)
- {
- smctr_reset_adapter(dev);
- tp->state = CLOSED;
- return (LOBE_MEDIA_TEST_FAILED);
- }
+ goto err;
else
{
perror = 1;
if(smctr_lobe_media_test_cmd(dev))
- {
- smctr_reset_adapter(dev);
- tp->state = CLOSED;
- return (LOBE_MEDIA_TEST_FAILED);
- }
+ goto err;
}
}
}
@@ -3124,28 +3112,24 @@ static int smctr_lobe_media_test(struct net_device *dev)
if(smctr_send_dat(dev))
{
if(smctr_send_dat(dev))
- {
- smctr_reset_adapter(dev);
- tp->state = CLOSED;
- return (LOBE_MEDIA_TEST_FAILED);
- }
+ goto err;
}
/* Check if any frames received during test. */
if((tp->rx_fcb_curr[MAC_QUEUE]->frame_status)
|| (tp->rx_fcb_curr[NON_MAC_QUEUE]->frame_status))
- {
- smctr_reset_adapter(dev);
- tp->state = CLOSED;
- return (LOBE_MEDIA_TEST_FAILED);
- }
+ goto err;
/* Set receive mask to "Promisc" mode. */
tp->receive_mask = saved_rcv_mask;
smctr_chg_rx_mask(dev);
- return (0);
+ return 0;
+err:
+ smctr_reset_adapter(dev);
+ tp->status = CLOSED;
+ return LOBE_MEDIA_TEST_FAILED;
}
static int smctr_lobe_media_test_cmd(struct net_device *dev)
^ permalink raw reply related
* [PATCH] drivres/net: Change constant name
From: Julia Lawall @ 2009-08-08 7:53 UTC (permalink / raw)
To: netdev, linux-kernel, kernel-janitors
From: Julia Lawall <julia@diku.dk>
In this series of tests the constants have the form XRXMAC_STATUS, except
in this one case. The values of XRXMAC_STAT_MSK_RXOCTET_CNT_EXP and
XRXMAC_STATUS_RXOCTET_CNT_EXP are furthermore the same.
Signed-off-by: Julia Lawall <julia@diku.dk>
---
drivers/net/niu.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/net/niu.c b/drivers/net/niu.c
index d2146d4..3ada7ea 100644
--- a/drivers/net/niu.c
+++ b/drivers/net/niu.c
@@ -4015,7 +4015,7 @@ static void niu_xmac_interrupt(struct niu *np)
mp->rx_hist_cnt6 += RXMAC_HIST_CNT6_COUNT;
if (val & XRXMAC_STATUS_RXHIST7_CNT_EXP)
mp->rx_hist_cnt7 += RXMAC_HIST_CNT7_COUNT;
- if (val & XRXMAC_STAT_MSK_RXOCTET_CNT_EXP)
+ if (val & XRXMAC_STATUS_RXOCTET_CNT_EXP)
mp->rx_octets += RXMAC_BT_CNT_COUNT;
if (val & XRXMAC_STATUS_CVIOLERR_CNT_EXP)
mp->rx_code_violations += RXMAC_CD_VIO_CNT_COUNT;
^ permalink raw reply related
* 6lowpan: IPv6 encapsulation question
From: Dmitry Eremin-Solenikov @ 2009-08-08 8:23 UTC (permalink / raw)
To: netdev; +Cc: pekkas
Hi,
We are going to work on the implementation of 6lowpan: the encapsulation
for IPv6 packets for IEEE 802.15.4 networks. RFC 4944 and several
following drafts from 6lowpan WG define several types of IPv6/UDP
headers compression, special fragmentation, ND, etc.
The question is: what is the most clean way to implement 6lowpan and to
hook it into the Linux kernel.
We've identified the following possible ways:
* Implement it just as a special net_device, accepting only IPv6 trafic,
and splitting/merging it on the top of our IEEE 802.15.4 net_devices
* Implement transmission/fragmentation by hooking into neighbouring output
routines
* Implement transmission/fragmentation by hooking into dst->output
routines
* Just hook into IPv6 stack by checking for ARPHRD_IEEE802154 devices
* ???
--
With best wishes
Dmitry
^ permalink raw reply
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Benny Amorsen @ 2009-08-08 8:50 UTC (permalink / raw)
To: Fischer, Anna
Cc: arnd@arndb.de, Yaron Haviv, bridge@lists.linux-foundation.org,
virtualization@lists.linux-foundation.org, adobriyan@gmail.com,
Paul Congdon (UC Davis), netdev@vger.kernel.org,
evb@yahoogroups.com, davem@davemloft.net
In-Reply-To: <0199E0D51A61344794750DC57738F58E6D6A6CD803__29862.6656564467$1249679159$gmane$org@GVW1118EXC.americas.hpqcorp.net>
"Fischer, Anna" <anna.fischer@hp.com> writes:
> If you do have a SRIOV NIC that supports VEPA, then I would think that
> you do not have QEMU or macvtap in the setup any more though. Simply
> because in that case the VM can directly access the VF on the physical
> device. That would be ideal.
I'm just trying to understand how this all works, so I'm probably asking
a stupid question:
Would a SRIOV NIC with VEPA support show up as multiple devices? I.e.
would I get e.g. eth0-eth7 for a NIC with support for 8 virtual
interfaces? Would they have different MAC addresses?
/Benny
^ permalink raw reply
* RE: [RESEND PATCH] gianfar: keep vlan related state when restart
From: Haruki Dai-R35557 @ 2009-08-08 9:07 UTC (permalink / raw)
To: Yong Zhang, davem, Fleming Andy-AFLEMING; +Cc: netdev, linux-kernel
In-Reply-To: <1249699012-26379-1-git-send-email-yong.zhang0@gmail.com>
> -----Original Message-----
> From: Yong Zhang [mailto:yong.zhang0@gmail.com]
> Sent: Friday, August 07, 2009 9:37 PM
> To: davem@davemloft.net; Fleming Andy-AFLEMING; Haruki Dai-R35557
> Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: [RESEND PATCH] gianfar: keep vlan related state when restart
>
> If vlan has been enabled. ifdown followed by ifup will lost hardware
> related state.
>
> Also remove duplicated operation in gfar_vlan_rx_register().
>
> Signed-off-by: Yong Zhang <yong.zhang0@gmail.com>
Thanks for the catch. This patch looks fine to me.
Acked-by: Dai Haruki <dai.haruki@freescale.com>
> ---
> drivers/net/gianfar.c | 12 ++++++++++--
> 1 files changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
> index f8ffcbf..e212f2c 100644
> --- a/drivers/net/gianfar.c
> +++ b/drivers/net/gianfar.c
> @@ -936,6 +936,7 @@ int startup_gfar(struct net_device *dev)
> struct gfar __iomem *regs = priv->regs;
> int err = 0;
> u32 rctrl = 0;
> + u32 tctrl = 0;
> u32 attrs = 0;
>
> gfar_write(®s->imask, IMASK_INIT_CLEAR);
> @@ -1111,11 +1112,19 @@ int startup_gfar(struct net_device *dev)
> rctrl |= RCTRL_PADDING(priv->padding);
> }
>
> + /* keep vlan related bits if it's enabled */
> + if (priv->vlgrp) {
> + rctrl |= RCTRL_VLEX | RCTRL_PRSDEP_INIT;
> + tctrl |= TCTRL_VLINS;
> + }
> +
> /* Init rctrl based on our settings */
> gfar_write(&priv->regs->rctrl, rctrl);
>
> if (dev->features & NETIF_F_IP_CSUM)
> - gfar_write(&priv->regs->tctrl, TCTRL_INIT_CSUM);
> + tctrl |= TCTRL_INIT_CSUM;
> +
> + gfar_write(&priv->regs->tctrl, tctrl);
>
> /* Set the extraction length and index */
> attrs = ATTRELI_EL(priv->rx_stash_size) |
> @@ -1450,7 +1459,6 @@ static void gfar_vlan_rx_register(struct
net_device *dev,
>
> /* Enable VLAN tag extraction */
> tempval = gfar_read(&priv->regs->rctrl);
> - tempval |= RCTRL_VLEX;
> tempval |= (RCTRL_VLEX | RCTRL_PRSDEP_INIT);
> gfar_write(&priv->regs->rctrl, tempval);
> } else {
> --
> 1.6.0.4
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox