* Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 21:06 UTC (permalink / raw)
To: Linux Network Developers; +Cc: brice, gallatin
I've run into a major receive side performance issue with multi-10-GigE
on a NUMA system. The system is using a SuperMicro X8DAH+-F motherboard
with 2 3.2 GHz quad-core Intel Xeon 5580 processors and 12 GB of
1333 MHz DDR3 memory. It is a Fedora 10 system but using the latest
2.6.29.6 kernel from Fedora 11 (originally tried the 2.6.27.29 kernel
from Fedora 10).
The test setup is:
i7test1----(6)----xeontest1----(6)----i7test2
10-GigE 10-GigE
So xeontest1 has 6 dual-port Myricom 10-GigE NICs for a total
of 12 10-GigE interfaces. eth2 through eth7 (which are on the
second Intel 5520 I/O Hub) are connected to i7test1 while
eth8 through eth13 (which are on the first Intel 5520 I/O Hub)
are connected to i7test2.
Previous direct testing between i7test1 and i7test2 (which use an
Asus P6T6 WS Revolution motherboard) demonstrated that they could
achieve ~70 Gbps performance for either transmit or receive using
8 10-GigE interfaces.
The transmit side performance of xeontest1 is fantastic:
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -xc6/3 -p5012 192.168.12.11 &
n12: 9648.0522 MB / 10.00 sec = 8091.4066 Mbps 49 %TX 26 %RX 0 retrans 0.18 msRTT
n9: 11130.5320 MB / 10.01 sec = 9328.3224 Mbps 47 %TX 37 %RX 0 retrans 0.19 msRTT
n11: 9418.1250 MB / 10.00 sec = 7897.5848 Mbps 50 %TX 30 %RX 0 retrans 0.18 msRTT
n10: 9279.4758 MB / 10.01 sec = 7778.7146 Mbps 49 %TX 28 %RX 0 retrans 0.12 msRTT
n8: 11142.6574 MB / 10.01 sec = 9340.3789 Mbps 47 %TX 35 %RX 0 retrans 0.18 msRTT
n13: 9422.1492 MB / 10.01 sec = 7897.4115 Mbps 49 %TX 25 %RX 0 retrans 0.17 msRTT
n3: 11471.2500 MB / 10.01 sec = 9613.9477 Mbps 49 %TX 32 %RX 0 retrans 0.15 msRTT
n6: 9339.6354 MB / 10.01 sec = 7828.5345 Mbps 50 %TX 25 %RX 0 retrans 0.19 msRTT
n4: 9093.2500 MB / 10.01 sec = 7624.1589 Mbps 49 %TX 28 %RX 0 retrans 0.15 msRTT
n5: 9121.8367 MB / 10.01 sec = 7646.8646 Mbps 50 %TX 29 %RX 0 retrans 0.17 msRTT
n7: 9292.2500 MB / 10.01 sec = 7789.1574 Mbps 49 %TX 26 %RX 0 retrans 0.17 msRTT
n2: 11487.1150 MB / 10.01 sec = 9627.2690 Mbps 49 %TX 46 %RX 0 retrans 0.19 msRTT
Aggregate performance: 100.4637 Gbps
The problem is with the receive side performance.
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n11: 6983.6359 MB / 10.09 sec = 5803.2293 Mbps 13 %TX 26 %RX 0 retrans 0.11 msRTT
n10: 7000.1557 MB / 10.11 sec = 5807.5978 Mbps 13 %TX 26 %RX 0 retrans 0.12 msRTT
n9: 2451.7206 MB / 10.21 sec = 2014.8397 Mbps 4 %TX 13 %RX 0 retrans 0.11 msRTT
n13: 2453.0887 MB / 10.20 sec = 2016.8751 Mbps 3 %TX 11 %RX 0 retrans 0.10 msRTT
n12: 2446.5303 MB / 10.24 sec = 2004.4638 Mbps 4 %TX 11 %RX 0 retrans 0.10 msRTT
n8: 2462.5890 MB / 10.26 sec = 2014.0272 Mbps 3 %TX 11 %RX 0 retrans 0.12 msRTT
n4: 2763.5091 MB / 10.26 sec = 2258.4871 Mbps 4 %TX 14 %RX 0 retrans 0.10 msRTT
n5: 2770.0887 MB / 10.28 sec = 2261.2562 Mbps 4 %TX 15 %RX 0 retrans 0.10 msRTT
n2: 1777.7277 MB / 10.32 sec = 1444.9054 Mbps 2 %TX 11 %RX 0 retrans 0.11 msRTT
n6: 1772.7962 MB / 10.31 sec = 1442.0346 Mbps 3 %TX 10 %RX 0 retrans 0.11 msRTT
n3: 1779.4535 MB / 10.32 sec = 1446.0090 Mbps 2 %TX 11 %RX 0 retrans 0.15 msRTT
n7: 1770.8359 MB / 10.35 sec = 1435.4757 Mbps 2 %TX 11 %RX 0 retrans 0.12 msRTT
Aggregate performance: 29.9492 Gbps
I suspected that this was because the memory being allocated by the
myri10ge driver was not being allocated on the optimum NUMA node.
BTW the NUMA nodes on the system are 0 and 2 instead of 0 and 1 which
is what I would have expected, but this is my first experience with
a NUMA system.
Based upon a patch by Peter Zijlstra that I discovered through Google
searching, I tried patching the myri10ge driver to change its memory
allocation of memory pages from alloc_pages() to alloc_pages_node()
and specifying the NUMA node of the parent device of the Myricom 10-GigE
device, which IIUC should be the PCIe switch. This didn't help.
This could be because I discovered that if I did:
find /sys -name numa_node -exec grep . {} /dev/null \;
that the numa_node associated with all the PCI devices was always 0,
and if IIUC then I believe some of the PCI devices should have been
associated with NUMA node 2. Perhaps this is what is causing all
the memory pages allocated by the myri10ge driver to be on NUMA
node 0, and thus causing the major performance issue.
To kludge around this, I made a different patch to the myri10ge driver.
This time I hardcoded the NUMA node in the call to alloc_pages_node()
to 2 for devices with an IRQ between 113 and 118 (eth2 through eth7)
and to 0 for devices with an IRQ between 119 and 124 (eth8 through eth13).
This is of course very specific to our specific system (NUMA node ids
and Myricom 10-GigE device IRQs), and is not something that would be
generically applicable. But it was useful as a test, and it did
improve the receive side performance substantially!
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n5: 8221.2911 MB / 10.09 sec = 6836.0343 Mbps 17 %TX 31 %RX 0 retrans 0.12 msRTT
n4: 8237.9524 MB / 10.10 sec = 6840.2379 Mbps 16 %TX 31 %RX 0 retrans 0.11 msRTT
n11: 7935.3750 MB / 10.11 sec = 6586.2476 Mbps 15 %TX 29 %RX 0 retrans 0.16 msRTT
n2: 4543.1621 MB / 10.13 sec = 3763.0669 Mbps 9 %TX 21 %RX 0 retrans 0.12 msRTT
n10: 7916.3925 MB / 10.13 sec = 6555.5210 Mbps 15 %TX 28 %RX 0 retrans 0.13 msRTT
n7: 4558.4817 MB / 10.14 sec = 3771.6557 Mbps 7 %TX 22 %RX 0 retrans 0.10 msRTT
n13: 4390.1875 MB / 10.14 sec = 3633.6421 Mbps 6 %TX 21 %RX 0 retrans 0.12 msRTT
n3: 4572.6478 MB / 10.15 sec = 3778.2596 Mbps 9 %TX 21 %RX 0 retrans 0.14 msRTT
n6: 4564.4776 MB / 10.14 sec = 3774.4373 Mbps 9 %TX 21 %RX 0 retrans 0.11 msRTT
n8: 4409.8551 MB / 10.16 sec = 3642.1920 Mbps 8 %TX 19 %RX 0 retrans 0.12 msRTT
n9: 4412.7836 MB / 10.16 sec = 3643.7788 Mbps 8 %TX 20 %RX 0 retrans 0.14 msRTT
n12: 4413.4061 MB / 10.16 sec = 3645.2544 Mbps 8 %TX 21 %RX 0 retrans 0.11 msRTT
Aggregate performance: 56.4703 Gbps
This was basically double the previous receive side performance
without the patch.
I don't know if this is fundamentally a myri10ge driver issue or
some underlying Linux kernel issue, so it's not clear to me what
a proper fix would be.
Finally, while definitely a major improvement, I think it should be
possible to do even better, since we achieved 70 Gbps in the i7 to i7
tests, and probably could have done 80 Gbps except for an Asus
motherboard restriction with the interconnect between the Intel X58
and Nvidia NF200 chips. It's definitely a big step in the right
direction though if this issue can be resolved.
Any help greatly appreicated in advance.
-Thanks
-Bill
^ permalink raw reply
* RE: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Fischer, Anna @ 2009-08-07 21:00 UTC (permalink / raw)
To: Yaron Haviv, evb@yahoogroups.com, shemminger@linux-foundation.org
Cc: bridge@lists.linux-foundation.org, netdev@vger.kernel.org,
virtualization@lists.linux-foundation.org, davem@davemloft.net,
kaber@trash.net, adobriyan@gmail.com, arnd@arndb.de,
Paul Congdon (UC Davis)
In-Reply-To: <D2E118A1E7268F449C469EA2F301D92902BC39B2@exil.voltaire.com>
Hi Yaron,
Yes, I also believe that VEPA + SRIOV can potentially, in some deployments, achieve better performance than a bridge/tap configuration, especially when you run multiple VMs and if you want to enable more sophisticated network processing in the data path.
If you do have a SRIOV NIC that supports VEPA, then I would think that you do not have QEMU or macvtap in the setup any more though. Simply because in that case the VM can directly access the VF on the physical device. That would be ideal.
I do think that the macvtap driver is a good addition as a simple and fast virtual network I/O interface, in case you do not need full bridge functionality. It does seem to assume though that the virtualization software uses QEMU/tap interfaces. How would this work with a Xen para-virtualized network interface? I guess there would need to be yet another driver?
Anna
--
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: 07 August 2009 21:36
To: evb@yahoogroups.com; shemminger@linux-foundation.org; Fischer, Anna
Cc: bridge@lists.linux-foundation.org; netdev@vger.kernel.org; virtualization@lists.linux-foundation.org; davem@davemloft.net; kaber@trash.net; adobriyan@gmail.com; arnd@arndb.de
Subject: Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
Paul,
I also think that bridge may not be the right place for VEPA, but rather a simpler sw/hw mux
Although the VEPA support may reside in multiple places (I.e. also in the bridge)
As Arnd pointed out Or already added an extension to qemu that allow direct guest virtual NIC mapping to an interface device (vs using tap), this was done specifically to address VEPA, and result in much faster performance and lower cpu overhead (Or and some others are planning additional meaningful performance optimizations)
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
It may be counter intuitive for some, but we expect the (completed) qemu VEPA mode + SR-IOV + certain switches with hairpin (vepa) mode to perform faster than using bridge+tap even for connecting 2 VMs on the same host
Yaron
Sent from BlackBerry
________________________________________
From: evb@yahoogroups.com
To: 'Stephen Hemminger' ; 'Fischer, Anna'
Cc: bridge@lists.linux-foundation.org ; linux-kernel@vger.kernel.org ; netdev@vger.kernel.org ; virtualization@lists.linux-foundation.org ; evb@yahoogroups.com ; davem@davemloft.net ; kaber@trash.net ; adobriyan@gmail.com ; 'Arnd Bergmann'
Sent: Fri Aug 07 21:58:00 2009
Subject: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
>
> After reading more about this, I am not convinced this should be part
> of the bridge code. The bridge code really consists of two parts:
> forwarding table and optional spanning tree. Well the VEPA code short
> circuits both of these; it can't imagine it working with STP turned
> on. The only part of bridge code that really gets used by this are the
> receive packet hooks and the crufty old API.
>
> So instead of adding more stuff to existing bridge code, why not have
> a new driver for just VEPA. You could do it with a simple version of
> macvlan type driver.
Stephen,
Thanks for your comments and questions. We do believe the bridge code is
the right place for this, so I'd like to embellish on that a bit more to
help persuade you. Sorry for the long winded response, but here are some
thoughts:
- First and foremost, VEPA is going to be a standard addition to the IEEE
802.1Q specification. The working group agreed at the last meeting to
pursue a project to augment the bridge standard with hairpin mode (aka
reflective relay) and a remote filtering service (VEPA). See for details:
http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709
-v01.pdf
- The VEPA functionality was really a pretty small change to the code with
low risk and wouldn't seem to warrant an entire new driver or module.
- There are good use cases where VMs will want to have some of their
interfaces attached to bridges and others to bridges operating in VEPA mode.
In other words, we see simultaneous operation of the bridge code and VEPA
occurring, so having as much of the underlying code as common as possible
would seem to be beneficial.
- By augmenting the bridge code with VEPA there is a great amount of re-use
achieved. It works wherever the bridge code works and doesn't need anything
special to support KVM, XEN, and all the hooks, etc...
- The hardware vendors building SR-IOV NICs with embedded switches will be
adding VEPA mode, so by keeping the bridge module in sync would be
consistent with this trend and direction. It will be possible to extend the
hardware implementations by cascading a software bridge and/or VEPA, so
being in sync with the architecture would make this more consistent.
- The forwarding table is still needed and used on inbound traffic to
deliver frames to the correct virtual interfaces and to filter any reflected
frames. A new driver would have to basically implement an equivalent
forwarding table anyway. As I understand the current macvlan type driver,
it wouldn't filter multicast frames properly without such a table.
- It seems the hairpin mode would be needed in the bridge module whether
VEPA was added to the bridge module or a new driver. Having the associated
changes together in the same code could aid in understanding and deployment.
As I understand the macvlan code, it currently doesn't allow two VMs on the
same machine to communicate with one another. I could imagine a hairpin
mode on the adjacent bridge making this possible, but the macvlan code would
need to be updated to filter reflected frames so a source did not receive
his own packet. I could imagine this being done as well, but to also
support selective multicast usage, something similar to the bridge
forwarding table would be needed. I think putting VEPA into a new driver
would cause you to implement many things the bridge code already supports.
Given that we expect the bridge standard to ultimately include VEPA, and the
new functions are basic forwarding operations, it seems to make most sense
to keep this consistent with the bridge module.
Paul
^ permalink raw reply
* Re: [PATCH 3/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:44 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / consumption correlator
Add ftracer module to kernel to print out a list that correlates a process id,
an skb it read, and the numa nodes on wich the process was running when it was
read along with the numa node the skbuff was allocated on.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
kernel/trace/Makefile | 1
kernel/trace/trace.h | 19 ++++
kernel/trace/trace_skb_sources.c | 154 +++++++++++++++++++++++++++++++++++++++
net/core/datagram.c | 3
4 files changed, 177 insertions(+)
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index 844164d..ee5e5b1 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o
ifeq ($(CONFIG_BLOCK),y)
obj-$(CONFIG_EVENT_TRACING) += blktrace.o
endif
+obj-$(CONFIG_SKB_SOURCES_TRACER) += trace_skb_sources.o
obj-$(CONFIG_EVENT_TRACING) += trace_events.o
obj-$(CONFIG_EVENT_TRACING) += trace_export.o
obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 3548ae5..8c1d458 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -11,6 +11,7 @@
#include <trace/boot.h>
#include <linux/kmemtrace.h>
#include <trace/power.h>
+#include <trace/events/skb.h>
#include <linux/trace_seq.h>
#include <linux/ftrace_event.h>
@@ -40,6 +41,7 @@ enum trace_type {
TRACE_KMEM_FREE,
TRACE_POWER,
TRACE_BLK,
+ TRACE_SKB_SOURCE,
__TRACE_LAST_TYPE,
};
@@ -171,6 +173,21 @@ struct trace_power {
struct power_trace state_data;
};
+struct skb_record {
+ pid_t pid; /* pid of the copying process */
+ int anid; /* node where skb was allocated */
+ int cnid; /* node to which skb was copied in userspace */
+ char ifname[IFNAMSIZ]; /* Name of the receiving interface */
+ int rx_queue; /* The rx queue the skb was received on */
+ int ccpu; /* Cpu the application got this frame from */
+ int len; /* length of the data copied */
+};
+
+struct trace_skb_event {
+ struct trace_entry ent;
+ struct skb_record event_data;
+};
+
enum kmemtrace_type_id {
KMEMTRACE_TYPE_KMALLOC = 0, /* kmalloc() or kfree(). */
KMEMTRACE_TYPE_CACHE, /* kmem_cache_*(). */
@@ -323,6 +340,8 @@ extern void __ftrace_bad_type(void);
TRACE_SYSCALL_ENTER); \
IF_ASSIGN(var, ent, struct syscall_trace_exit, \
TRACE_SYSCALL_EXIT); \
+ IF_ASSIGN(var, ent, struct trace_skb_event, \
+ TRACE_SKB_SOURCE); \
__ftrace_bad_type(); \
} while (0)
diff --git a/kernel/trace/trace_skb_sources.c b/kernel/trace/trace_skb_sources.c
new file mode 100644
index 0000000..4ba3671
--- /dev/null
+++ b/kernel/trace/trace_skb_sources.c
@@ -0,0 +1,154 @@
+/*
+ * ring buffer based tracer for analyzing per-socket skb sources
+ *
+ * Neil Horman <nhorman@tuxdriver.com>
+ * Copyright (C) 2009
+ *
+ *
+ */
+
+#include <linux/init.h>
+#include <linux/debugfs.h>
+#include <trace/events/skb.h>
+#include <linux/kallsyms.h>
+#include <linux/module.h>
+#include <linux/hardirq.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+
+#include "trace.h"
+#include "trace_output.h"
+
+EXPORT_TRACEPOINT_SYMBOL_GPL(skb_copy_datagram_iovec);
+
+static struct trace_array *skb_trace;
+static int __read_mostly trace_skb_source_enabled;
+
+static void probe_skb_dequeue(const struct sk_buff *skb, int len)
+{
+ struct ring_buffer_event *event;
+ struct trace_skb_event *entry;
+ struct trace_array *tr = skb_trace;
+ struct net_device *dev;
+
+ if (!trace_skb_source_enabled)
+ return;
+
+ if (in_interrupt())
+ return;
+
+ event = trace_buffer_lock_reserve(tr, TRACE_SKB_SOURCE,
+ sizeof(*entry), 0, 0);
+ if (!event)
+ return;
+ entry = ring_buffer_event_data(event);
+
+ entry->event_data.pid = current->pid;
+ entry->event_data.anid = page_to_nid(virt_to_page(skb->data));
+ entry->event_data.cnid = cpu_to_node(smp_processor_id());
+ entry->event_data.len = len;
+ entry->event_data.rx_queue = skb->queue_mapping;
+ entry->event_data.ccpu = smp_processor_id();
+
+ dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
+ if (dev) {
+ memcpy(entry->event_data.ifname, dev->name, IFNAMSIZ);
+ dev_put(dev);
+ } else {
+ strcpy(entry->event_data.ifname, "Unknown");
+ }
+
+ trace_buffer_unlock_commit(tr, event, 0, 0);
+}
+
+static int tracing_skb_source_register(void)
+{
+ int ret;
+
+ ret = register_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
+ if (ret)
+ pr_info("skb source trace: Couldn't activate dequeue tracepoint");
+
+ return ret;
+}
+
+static void start_skb_source_trace(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 1;
+}
+
+static void stop_skb_source_trace(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 0;
+}
+
+static void skb_source_trace_reset(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 0;
+ unregister_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
+}
+
+
+static int skb_source_trace_init(struct trace_array *tr)
+{
+ int cpu;
+ skb_trace = tr;
+
+ trace_skb_source_enabled = 1;
+ tracing_skb_source_register();
+
+ for_each_cpu(cpu, cpu_possible_mask)
+ tracing_reset(tr, cpu);
+ return 0;
+}
+
+static enum print_line_t skb_source_print_line(struct trace_iterator *iter)
+{
+ int ret = 0;
+ struct trace_entry *entry = iter->ent;
+ struct trace_skb_event *event;
+ struct skb_record *record;
+ struct trace_seq *s = &iter->seq;
+
+ trace_assign_type(event, entry);
+ record = &event->event_data;
+ if (entry->type != TRACE_SKB_SOURCE)
+ return TRACE_TYPE_UNHANDLED;
+
+ ret = trace_seq_printf(s, " %d %d %d %s %d %d %d\n",
+ record->pid,
+ record->anid,
+ record->cnid,
+ record->ifname,
+ record->rx_queue,
+ record->ccpu,
+ record->len);
+
+ if (!ret)
+ return TRACE_TYPE_PARTIAL_LINE;
+
+ return TRACE_TYPE_HANDLED;
+}
+
+static void skb_source_print_header(struct seq_file *s)
+{
+ seq_puts(s, "# PID ANID CNID IFC RXQ CCPU LEN\n");
+ seq_puts(s, "# | | | | | | |\n");
+}
+
+static struct tracer skb_source_tracer __read_mostly =
+{
+ .name = "skb_sources",
+ .init = skb_source_trace_init,
+ .start = start_skb_source_trace,
+ .stop = stop_skb_source_trace,
+ .reset = skb_source_trace_reset,
+ .print_line = skb_source_print_line,
+ .print_header = skb_source_print_header,
+};
+
+static int init_skb_source_trace(void)
+{
+ return register_tracer(&skb_source_tracer);
+}
+device_initcall(init_skb_source_trace);
diff --git a/net/core/datagram.c b/net/core/datagram.c
index b0fe692..1c6cf3a 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -55,6 +55,7 @@
#include <net/checksum.h>
#include <net/sock.h>
#include <net/tcp_states.h>
+#include <trace/events/skb.h>
/*
* Is a socket 'connection oriented' ?
@@ -284,6 +285,8 @@ int skb_copy_datagram_iovec(const struct sk_buff *skb, int offset,
int i, copy = start - offset;
struct sk_buff *frag_iter;
+ trace_skb_copy_datagram_iovec(skb, len);
+
/* Copy header. */
if (copy > 0) {
if (copy > len)
^ permalink raw reply related
* [PATCH 5/6] [NET] ixp2000/enp2611: don't set non-existent member get_stats
From: Uwe Kleine-König @ 2009-08-07 20:42 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, rt-users, Russell King, Dmitry Baryshkov, Lennert Buytenhek,
netdev
In-Reply-To: <20090807203939.GA19374@pengutronix.de>
This fixes a build failure for 2.6.31-rc5 (ARCH=arm, ixp2000_defconfig):
CC drivers/net/ixp2000/enp2611.o
drivers/net/ixp2000/enp2611.c: In function 'enp2611_init_module':
drivers/net/ixp2000/enp2611.c:213: error: 'struct net_device' has no member named 'get_stats'
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Dmitry Baryshkov <dbaryshkov@gmail.com>
Cc: Lennert Buytenhek <kernel@wantstofly.org>
Cc: netdev@vger.kernel.org
---
Hello,
obviously this has the downside that the stats won't work, I let the fix
for someone else :-)
Best regards
Uwe
---
drivers/net/ixp2000/enp2611.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/net/ixp2000/enp2611.c b/drivers/net/ixp2000/enp2611.c
index b02a981..d6ef176 100644
--- a/drivers/net/ixp2000/enp2611.c
+++ b/drivers/net/ixp2000/enp2611.c
@@ -210,7 +210,7 @@ static int __init enp2611_init_module(void)
return -ENOMEM;
}
- nds[i]->get_stats = enp2611_get_stats;
+ /* nds[i]->get_stats = enp2611_get_stats; */
pm3386_init_port(i);
pm3386_get_mac(i, nds[i]->dev_addr);
}
--
1.6.3.3
^ permalink raw reply related
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Yaron Haviv @ 2009-08-07 20:35 UTC (permalink / raw)
To: evb, shemminger, anna.fischer
Cc: arnd, davem, netdev, bridge, adobriyan, virtualization
[-- Attachment #1.1: Type: text/plain, Size: 9517 bytes --]
Paul,
I also think that bridge may not be the right place for VEPA, but rather a simpler sw/hw mux
Although the VEPA support may reside in multiple places (I.e. also in the bridge)
As Arnd pointed out Or already added an extension to qemu that allow direct guest virtual NIC mapping to an interface device (vs using tap), this was done specifically to address VEPA, and result in much faster performance and lower cpu overhead (Or and some others are planning additional meaningful performance optimizations)
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
It may be counter intuitive for some, but we expect the (completed) qemu VEPA mode + SR-IOV + certain switches with hairpin (vepa) mode to perform faster than using bridge+tap even for connecting 2 VMs on the same host
Yaron
Sent from BlackBerry
________________________________
From: evb@yahoogroups.com
To: 'Stephen Hemminger' ; 'Fischer, Anna'
Cc: bridge@lists.linux-foundation.org ; linux-kernel@vger.kernel.org ; netdev@vger.kernel.org ; virtualization@lists.linux-foundation.org ; evb@yahoogroups.com ; davem@davemloft.net ; kaber@trash.net ; adobriyan@gmail.com ; 'Arnd Bergmann'
Sent: Fri Aug 07 21:58:00 2009
Subject: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
>
> After reading more about this, I am not convinced this should be part
> of the bridge code. The bridge code really consists of two parts:
> forwarding table and optional spanning tree. Well the VEPA code short
> circuits both of these; it can't imagine it working with STP turned
> on. The only part of bridge code that really gets used by this are the
> receive packet hooks and the crufty old API.
>
> So instead of adding more stuff to existing bridge code, why not have
> a new driver for just VEPA. You could do it with a simple version of
> macvlan type driver.
Stephen,
Thanks for your comments and questions. We do believe the bridge code is
the right place for this, so I'd like to embellish on that a bit more to
help persuade you. Sorry for the long winded response, but here are some
thoughts:
- First and foremost, VEPA is going to be a standard addition to the IEEE
802.1Q specification. The working group agreed at the last meeting to
pursue a project to augment the bridge standard with hairpin mode (aka
reflective relay) and a remote filtering service (VEPA). See for details:
http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709 <http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709>
-v01.pdf
- The VEPA functionality was really a pretty small change to the code with
low risk and wouldn't seem to warrant an entire new driver or module.
- There are good use cases where VMs will want to have some of their
interfaces attached to bridges and others to bridges operating in VEPA mode.
In other words, we see simultaneous operation of the bridge code and VEPA
occurring, so having as much of the underlying code as common as possible
would seem to be beneficial.
- By augmenting the bridge code with VEPA there is a great amount of re-use
achieved. It works wherever the bridge code works and doesn't need anything
special to support KVM, XEN, and all the hooks, etc...
- The hardware vendors building SR-IOV NICs with embedded switches will be
adding VEPA mode, so by keeping the bridge module in sync would be
consistent with this trend and direction. It will be possible to extend the
hardware implementations by cascading a software bridge and/or VEPA, so
being in sync with the architecture would make this more consistent.
- The forwarding table is still needed and used on inbound traffic to
deliver frames to the correct virtual interfaces and to filter any reflected
frames. A new driver would have to basically implement an equivalent
forwarding table anyway. As I understand the current macvlan type driver,
it wouldn't filter multicast frames properly without such a table.
- It seems the hairpin mode would be needed in the bridge module whether
VEPA was added to the bridge module or a new driver. Having the associated
changes together in the same code could aid in understanding and deployment.
As I understand the macvlan code, it currently doesn't allow two VMs on the
same machine to communicate with one another. I could imagine a hairpin
mode on the adjacent bridge making this possible, but the macvlan code would
need to be updated to filter reflected frames so a source did not receive
his own packet. I could imagine this being done as well, but to also
support selective multicast usage, something similar to the bridge
forwarding table would be needed. I think putting VEPA into a new driver
would cause you to implement many things the bridge code already supports.
Given that we expect the bridge standard to ultimately include VEPA, and the
new functions are basic forwarding operations, it seems to make most sense
to keep this consistent with the bridge module.
Paul
__._,_.___
Messages in this topic <http://groups.yahoo.com/group/evb/message/167;_ylc=X3oDMTMzb3FibzIzBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARtc2dJZAMyMTIEc2VjA2Z0cgRzbGsDdnRwYwRzdGltZQMxMjQ5NjcxNTEwBHRwY0lkAzE2Nw--> (9) Reply (via web post) <http://groups.yahoo.com/group/evb/post;_ylc=X3oDMTJwcDZzNTZqBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARtc2dJZAMyMTIEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxMjQ5NjcxNTEw?act=reply&messageNum=212> | Start a new topic <http://groups.yahoo.com/group/evb/post;_ylc=X3oDMTJmZW52ZmhiBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNudHBjBHN0aW1lAzEyNDk2NzE1MTA->
Messages <http://groups.yahoo.com/group/evb/messages;_ylc=X3oDMTJmODJ0MmU2BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNtc2dzBHN0aW1lAzEyNDk2NzE1MTA-> | Files <http://groups.yahoo.com/group/evb/files;_ylc=X3oDMTJnaGdsYXI4BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNmaWxlcwRzdGltZQMxMjQ5NjcxNTEw> | Photos <http://groups.yahoo.com/group/evb/photos;_ylc=X3oDMTJmYm90MmpqBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNwaG90BHN0aW1lAzEyNDk2NzE1MTA-> | Links <http://groups.yahoo.com/group/evb/links;_ylc=X3oDMTJnbWdyaTZnBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNsaW5rcwRzdGltZQMxMjQ5NjcxNTEw> | Database <http://groups.yahoo.com/group/evb/database;_ylc=X3oDMTJkZW1ka3FhBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNkYgRzdGltZQMxMjQ5NjcxNTEw> | Polls <http://groups.yahoo.com/group/evb/polls;_ylc=X3oDMTJnMG9lZTJuBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNwb2xscwRzdGltZQMxMjQ5NjcxNTEw> | Members <http://groups.yahoo.com/group/evb/members;_ylc=X3oDMTJmMWdwYXViBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNtYnJzBHN0aW1lAzEyNDk2NzE1MTA-> | Calendar <http://groups.yahoo.com/group/evb/calendar;_ylc=X3oDMTJlZnQ1N25iBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNjYWwEc3RpbWUDMTI0OTY3MTUxMA-->
Yahoo! Groups <http://groups.yahoo.com/;_ylc=X3oDMTJlNDhoZDY1BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNnZnAEc3RpbWUDMTI0OTY3MTUxMA-->
Change settings via the Web <http://groups.yahoo.com/group/evb/join;_ylc=X3oDMTJna2g4aW9zBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNzdG5ncwRzdGltZQMxMjQ5NjcxNTEw> (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest <mailto:evb-digest@yahoogroups.com?subject=Email Delivery: Digest> | Switch format to Traditional <mailto:evb-traditional@yahoogroups.com?subject=Change Delivery Format: Traditional>
Visit Your Group <http://groups.yahoo.com/group/evb;_ylc=X3oDMTJlN2ZwMTRxBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNocGYEc3RpbWUDMTI0OTY3MTUxMA--> | Yahoo! Groups Terms of Use <http://docs.yahoo.com/info/terms/> | Unsubscribe <mailto:evb-unsubscribe@yahoogroups.com?subject=>
Recent Activity
Visit Your Group <http://groups.yahoo.com/group/evb;_ylc=X3oDMTJmYW91dGs2BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDdnRsBHNsawN2Z2hwBHN0aW1lAzEyNDk2NzE1MTA->
Give Back
Yahoo! for Good <http://us.lrd.yahoo.com/_ylc=X3oDMTJuam45aG04BF9TAzk3MzU5NzE0BF9wAzEEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDYnJhbmQEc3RpbWUDMTI0OTY3MTUxMA--;_ylg=1/SIG=11314uv3k/**http%3A//brand.yahoo.com/forgood>
Get inspired
by a good cause.
Y! Toolbar
Get it Free! <http://us.lrd.yahoo.com/_ylc=X3oDMTJwbGY0NzUzBF9TAzk3MzU5NzE0BF9wAzIEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDdG9vbGJhcgRzdGltZQMxMjQ5NjcxNTEw;_ylg=1/SIG=11c6dvmk9/**http%3A//toolbar.yahoo.com/%3F.cpdl=ygrps>
easy 1-click access
to your groups.
Yahoo! Groups
Start a group <http://groups.yahoo.com/start;_ylc=X3oDMTJwdjNqdTNiBF9TAzk3MzU5NzE0BF9wAzMEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDZ3JvdXBzMgRzdGltZQMxMjQ5NjcxNTEw>
in 3 easy steps.
Connect with others.
.
<http://geo.yahoo.com/serv?s=97359714/grpId=23896447/grpspId=1705004750/msgId=212/stime=1249671510/nc1=1/nc2=2/nc3=3>
__,_._,___
[-- Attachment #1.2: Type: text/html, Size: 18843 bytes --]
[-- Attachment #2: Type: text/plain, Size: 184 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH 2/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:30 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / consumption corelator - Add config option
This patch adds a Kconfig option to enable the addtition of the skb source
tracer.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Kconfig | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 1551f47..1aeec05 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -234,6 +234,16 @@ config BOOT_TRACER
You must pass in ftrace=initcall to the kernel command line
to enable this on bootup.
+config SKB_SOURCES_TRACER
+ bool "Trace skb source information
+ select GENERIC_TRACER
+ help
+ This tracer helps developers/sysadmins correlate skb allocation and
+ consumption. The idea being that some processes will primarily consume data
+ that was allocated on certain numa nodes. By being able to visualize which
+ nodes the data was allocated on, a sysadmin or developer can optimize the
+ scheduling of those processes to cut back on cross node chatter.
+
config TRACE_BRANCH_PROFILING
bool
select GENERIC_TRACER
^ permalink raw reply related
* Re: [PATCH 1/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:28 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / cosumption tracer - Add consumption tracepoint
This patch adds a tracepoint to skb_copy_datagram_iovec, which is called each
time a userspace process copies a frame from a socket receive queue to a user
space buffer. It allows us to hook in and examine each sk_buff that the system
receives on a per-socket bases, and can be use to compile a list of which skb's
were received by which processes.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
include/trace/events/skb.h | 20 ++++++++++++++++++++
net/core/datagram.c | 3 +++
2 files changed, 23 insertions(+)
diff --git a/include/trace/events/skb.h b/include/trace/events/skb.h
index 1e8fabb..bdc5c1c 100644
--- a/include/trace/events/skb.h
+++ b/include/trace/events/skb.h
@@ -2,6 +2,7 @@
#define _TRACE_SKB_H
#include <linux/skbuff.h>
+#include <linux/netdevice.h>
#include <linux/tracepoint.h>
#undef TRACE_SYSTEM
@@ -34,6 +35,25 @@ TRACE_EVENT(kfree_skb,
__entry->skbaddr, __entry->protocol, __entry->location)
);
+TRACE_EVENT(skb_copy_datagram_iovec,
+
+ TP_PROTO(const struct sk_buff *skb, int len),
+
+ TP_ARGS(skb, len),
+
+ TP_STRUCT__entry(
+ __field( const void *, skbaddr )
+ __field( int, len )
+ ),
+
+ TP_fast_assign(
+ __entry->skbaddr = skb;
+ __entry->len = len;
+ ),
+
+ TP_printk("skbaddr=%p len=%d", __entry->skbaddr, __entry->len)
+);
+
#endif /* _TRACE_SKB_H */
/* This part must be outside protection */
diff --git a/net/core/datagram.c b/net/core/datagram.c
index b0fe692..1c6cf3a 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -55,6 +55,7 @@
#include <net/checksum.h>
#include <net/sock.h>
#include <net/tcp_states.h>
+#include <trace/events/skb.h>
/*
* Is a socket 'connection oriented' ?
@@ -284,6 +285,8 @@ int skb_copy_datagram_iovec(const struct sk_buff *skb, int offset,
int i, copy = start - offset;
struct sk_buff *frag_iter;
+ trace_skb_copy_datagram_iovec(skb, len);
+
/* Copy header. */
if (copy > 0) {
if (copy > len)
^ permalink raw reply related
* [PATCH 0/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:21 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt, nhorman
Hey all-
I put out an RFC about this awhile ago and didn't get any loud screams,
so I've gone ahead and implemented it
Currently, our network infrastructure allows net device drivers to
allocate skbs based on the the numa node the device itself is local to. This of
course cuts down on cross numa chatter when the device is DMA-ing network
traffic to the driver. Unfortuantely no such corresponding infrastrucuture
exists at the process level. The scheduler has no insight into the numa
locality of incomming data packets for a given process (and arguably it
shouldn't), and so there is every chance that a process will run on a different
numa node than the packets that its receiving lives on, creating cross numa node
traffic.
This patch aims to provide userspace with the opportunity to optimize
that scheduling. It consists of a tracepoint and an ftrace module which exports
a history of the packets each process receives, along with the numa node each
packet was received on, as well as the numa node the process was running on when
it copied the buffer to user space. With this information, exported via the
ftrace infrastructure to user space, a sysadim can identify high prirority
processes, and optimize their scheduling so that they are more likely to run on
the same node that they are primarily receiving data on, thereby cutting down
cross numa node traffic.
Tested by me, working well, applies against the head of the net-next tree
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
^ permalink raw reply
* RE: [Bridge] [PATCH] macvlan: add tap device backend
From: david @ 2009-08-07 20:17 UTC (permalink / raw)
To: Fischer, Anna
Cc: Stephen Hemminger, Paul Congdon (UC Davis), drobbins@funtoo.org,
'Arnd Bergmann', herbert@gondor.apana.org.au,
mst@redhat.com, netdev@vger.kernel.org,
bridge@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
ogerlitz@voltaire.com, evb@yahoogroups.com, davem@davemloft.net
In-Reply-To: <0199E0D51A61344794750DC57738F58E6D6A6CD7FE@GVW1118EXC.americas.hpqcorp.net>
On Fri, 7 Aug 2009, Fischer, Anna wrote:
> Subject: RE: [Bridge] [PATCH] macvlan: add tap device backend
>
>> Subject: Re: [Bridge] [PATCH] macvlan: add tap device backend
>>
>> On Fri, 7 Aug 2009 12:10:07 -0700
>> "Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
>>
>>> Responding to Daniel's questions...
>>>
>>>> I have some general questions about the intended use and benefits
>> of
>>>> VEPA, from an IT perspective:
>>>>
>>>> In which virtual machine setups and technologies do you forsee this
>>>> interface being used?
>>>
>>> The benefit of VEPA is the coordination and unification with the
>> external network switch. So, in environments where you are
>> needing/wanting your feature rich, wire speed, external network device
>> (firewall/switch/IPS/content-filter) to provide consistent policy
>> enforcement, and you want your VMs traffic to be subject to that
>> enforcement, you will want their traffic directed externally. Perhaps
>> you have some VMs that are on a DMZ or clustering an application or
>> implementing a multi-tier application where you would normally place a
>> firewall in-between the tiers.
>>
>> I do have to raise the point that Linux is perfectly capable of keeping
>> up without
>> the need of an external switch. Whether you want policy external or
>> internal is
>> a architecture decision that should not be driven by mis-information
>> about performance.
>
> VEPA is not only about enabling faster packet processing (like firewall/switch/IPS/content-filter etc) by doing this on the external switch.
>
> Due to rather low performance of software-based I/O virtualization approaches a lot of effort has recently been going into hardware-based implementations of virtual network interfaces like SRIOV NICs provide. Without VEPA, such a NIC would have to implement sophisticated virtual switching capabilities. VEPA however is very simple and therefore perfectly suited for a hardware-based implementation. So in the future, it will give you direct I/O like performance and all the capabilities your adjacent switch provides.
>
the performance overhead isn't from switching the packets, it's from
running the firewall/IDS/etc software on the same system.
with VEPA the communications from one VM to another VM running on the same
host will be forced to go out the interface to the datacenter switching
fabric. The overall performance of the network link will be slightly
slower, but it allows for other devices to be inserted into the path.
this is something that I would want available if I were to start using VMs
for things. I don't want to have to duplicate my IDS/firewalling functions
within each host system as well as having them as part of the switching
fabric.
David Lang
^ permalink raw reply
* Re: [PATCH 2/7 v3] initdev:kernel:USB init device discovery notification
From: Alan Jenkins @ 2009-08-07 20:09 UTC (permalink / raw)
To: David VomLehn
Cc: Alan Stern, linux-kernel, akpm, linux-usb, greg, linux-scsi,
netdev, arjan
In-Reply-To: <20090807174209.GB30450@cuplxvomd02.corp.sa.net>
David VomLehn wrote:
> On Fri, Aug 07, 2009 at 06:17:51PM +0100, Alan Jenkins wrote:
>
>> On 8/7/09, Alan Stern <stern@rowland.harvard.edu> wrote:
>>
>>> On Fri, 7 Aug 2009, Alan Jenkins wrote:
>>>
>>>
>>>>> +/* The mask of possible USB boot devices depends on what drivers and
>>>>> + * options have been configured into the kernel. There are too many
>>>>> + * USB network config options to list here, so just assume it is always
>>>>> + * possible to have a USB network device.
>>>>> + */
>>>>>
>>>> That could be fixed though. Make the "USB Network Adaptors" menu into
>>>> an option.
> I remember thinking about this and seem to recall that Alan Stern and I
> tossed this idea around, but didn't act on it. Such a thing holds potential
> for avoiding an unnecessary delay during boot. Whether it actually makes
> things faster in a given configuration will depend on that configuration's
> critical path, but I would welcome such a patch on the basis that it's
> unlikely to hurt and could improve boot times in some circumstances.
>
Bother, I see why this doesn't work now. There are all the USB
*wireless* network drivers outside the menu.
Thanks anyway
Alan
^ permalink raw reply
* RE: [Bridge] [PATCH] macvlan: add tap device backend
From: Paul Congdon (UC Davis) @ 2009-08-07 19:47 UTC (permalink / raw)
To: 'Stephen Hemminger'
Cc: drobbins, 'Fischer, Anna', 'Arnd Bergmann',
herbert, mst, netdev, bridge, linux-kernel, ogerlitz, evb, davem
In-Reply-To: <20090807123554.7c2bc27c@nehalam>
>
> I do have to raise the point that Linux is perfectly capable of keeping
> up without
> the need of an external switch. Whether you want policy external or
> internal is
> a architecture decision that should not be driven by mis-information
> about performance.
No argument here. I agree that you can do a lot in Linux. It is, as you
say, an architecture decision, that can be enabled with this addition mode
of operation. Without the mode of forcing things external, however, you
would always need to put this function internal or play games with VLANs
overlapping to get traffic to forward the way you want it.
Paul
^ permalink raw reply
* Re: [evb] Re: [PATCH][RFC] net/bridge: add basic VEPA support
From: Paul Congdon (UC Davis) @ 2009-08-07 19:44 UTC (permalink / raw)
To: evb, 'Stephen Hemminger'
Cc: 'Fischer, Anna', netdev, bridge, linux-kernel,
virtualization, 'Or Gerlitz', adobriyan, davem
In-Reply-To: <200908071329.44437.arnd@arndb.de>
[-- Attachment #1.1: Type: text/plain, Size: 862 bytes --]
Arnd,
I don't think your scheme works too well because broadcast packet coming
from other interfaces on br0 would get replicated and sent across the wire
to ethB multiple times.
Paul
That way you should be able to do something
like:
Host A Host B
/- nalvcam0 -\ /- macvlan0 - 192.168.1.1
br0 -| |- ethA === ethB -|
\- nalvcam1 -/ \- macvlan1 - 192.168.1.2
Now assuming that macvlan0 and macvlan1 are in different
network namespaces or belong to different KVM guests, these
guests would be able to communicate with each other through
the bridge on host A, which can set the policy (using ebtables)
for this communication and get interface statistics on its
nalvcam interfaces. Also, instead of having the br0, Host A could
assign an IP addresses to the two nalvcam interfaces that host
B has, and use IP forwarding between the guests of host B.
[-- Attachment #1.2: Type: text/html, Size: 9293 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge
^ permalink raw reply
* RE: [Bridge] [PATCH] macvlan: add tap device backend
From: Fischer, Anna @ 2009-08-07 19:44 UTC (permalink / raw)
To: Stephen Hemminger, Paul Congdon (UC Davis)
Cc: drobbins@funtoo.org, 'Paul Congdon (UC Davis)',
'Arnd Bergmann', herbert@gondor.apana.org.au,
mst@redhat.com, netdev@vger.kernel.org,
bridge@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
ogerlitz@voltaire.com, evb@yahoogroups.com, davem@davemloft.net
In-Reply-To: <20090807123554.7c2bc27c@nehalam>
> Subject: Re: [Bridge] [PATCH] macvlan: add tap device backend
>
> On Fri, 7 Aug 2009 12:10:07 -0700
> "Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
>
> > Responding to Daniel's questions...
> >
> > > I have some general questions about the intended use and benefits
> of
> > > VEPA, from an IT perspective:
> > >
> > > In which virtual machine setups and technologies do you forsee this
> > > interface being used?
> >
> > The benefit of VEPA is the coordination and unification with the
> external network switch. So, in environments where you are
> needing/wanting your feature rich, wire speed, external network device
> (firewall/switch/IPS/content-filter) to provide consistent policy
> enforcement, and you want your VMs traffic to be subject to that
> enforcement, you will want their traffic directed externally. Perhaps
> you have some VMs that are on a DMZ or clustering an application or
> implementing a multi-tier application where you would normally place a
> firewall in-between the tiers.
>
> I do have to raise the point that Linux is perfectly capable of keeping
> up without
> the need of an external switch. Whether you want policy external or
> internal is
> a architecture decision that should not be driven by mis-information
> about performance.
VEPA is not only about enabling faster packet processing (like firewall/switch/IPS/content-filter etc) by doing this on the external switch.
Due to rather low performance of software-based I/O virtualization approaches a lot of effort has recently been going into hardware-based implementations of virtual network interfaces like SRIOV NICs provide. Without VEPA, such a NIC would have to implement sophisticated virtual switching capabilities. VEPA however is very simple and therefore perfectly suited for a hardware-based implementation. So in the future, it will give you direct I/O like performance and all the capabilities your adjacent switch provides.
Anna
^ permalink raw reply
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Stephen Hemminger @ 2009-08-07 19:35 UTC (permalink / raw)
To: Paul Congdon (UC Davis)
Cc: drobbins, 'Paul Congdon (UC Davis)',
'Fischer, Anna', 'Arnd Bergmann', herbert, mst,
netdev, bridge, linux-kernel, ogerlitz, evb, davem
In-Reply-To: <004f01ca1792$b24a7a90$16df6fb0$@edu>
On Fri, 7 Aug 2009 12:10:07 -0700
"Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
> Responding to Daniel's questions...
>
> > I have some general questions about the intended use and benefits of
> > VEPA, from an IT perspective:
> >
> > In which virtual machine setups and technologies do you forsee this
> > interface being used?
>
> The benefit of VEPA is the coordination and unification with the external network switch. So, in environments where you are needing/wanting your feature rich, wire speed, external network device (firewall/switch/IPS/content-filter) to provide consistent policy enforcement, and you want your VMs traffic to be subject to that enforcement, you will want their traffic directed externally. Perhaps you have some VMs that are on a DMZ or clustering an application or implementing a multi-tier application where you would normally place a firewall in-between the tiers.
I do have to raise the point that Linux is perfectly capable of keeping up without
the need of an external switch. Whether you want policy external or internal is
a architecture decision that should not be driven by mis-information about performance.
^ permalink raw reply
* RE: [Bridge] [PATCH] macvlan: add tap device backend
From: Paul Congdon (UC Davis) @ 2009-08-07 19:10 UTC (permalink / raw)
To: drobbins
Cc: 'Paul Congdon (UC Davis)', 'Fischer, Anna',
'Arnd Bergmann', herbert, mst, netdev, bridge,
linux-kernel, ogerlitz, evb, davem
In-Reply-To: <0199E0D51A61344794750DC57738F58E6D6A6CD7F6@GVW1118EXC.americas.hpqcorp.net>
Responding to Daniel's questions...
> I have some general questions about the intended use and benefits of
> VEPA, from an IT perspective:
>
> In which virtual machine setups and technologies do you forsee this
> interface being used?
The benefit of VEPA is the coordination and unification with the external network switch. So, in environments where you are needing/wanting your feature rich, wire speed, external network device (firewall/switch/IPS/content-filter) to provide consistent policy enforcement, and you want your VMs traffic to be subject to that enforcement, you will want their traffic directed externally. Perhaps you have some VMs that are on a DMZ or clustering an application or implementing a multi-tier application where you would normally place a firewall in-between the tiers.
> Is this new interface to be used within a virtual machine or
> container, on the master node, or both?
It is really an interface to a new type of virtual switch. When you create virtual network, I would imagine it being a new mode of operation (bridge, NAT, VEPA, etc).
> What interface(s) would need to be configured for a single virtual
> machine to use VEPA to access the network?
It would be the same as if that machine were configure to use a bridge to access the network, but the bridge mode would be different.
> What are the current flexibility, security or performance limitations
> of tun/tap and bridge that make this new interface necessary or
> beneficial?
If you have VMs that will be communicating with one another on the same physical machine, and you want their traffic to be exposed to an in-line network device such as a application firewall/IPS/content-filter (without this feature) you will have to have this device co-located within the same physical server. This will use up CPU cycles that you presumable purchased to run applications, it will require a lot of consistent configuration on all physical machines, it could invoke potentially a lot of software licensing, additional cost, etc.. Everything would need to be replicated on each physical machine. With the VEPA capability, you can leverage all this functionality in an external network device and have it managed and configured in one place. The external implementation is likely a
higher performance, silicon based implementation. It should make it easier to migrate machines from one physical server to another and maintain the same network policy enforcement.
> Is this new interface useful at all for VPN solutions or is it
> *specifically* targeted for connecting virtual machines to the
> network?
I'm not sure I see the benefit for VPN solutions, but I'd have to understand the deployment scenario better. Certainly this is targeting connecting VMs to the adjacent physical LAN.
> Is this essentially a bridge with layer-2 isolation for the virtual
> machine interfaces built-in? If isolation is provided, what mechanism
> is used to accomplish this, and how secure is it?
That might be an over simplification, but you can achieve layer-2 isolation if you connect to a standard external switch. If that switch has 'hairpin' forwarding, then the VMs can talk at L2, but their traffic is forced through the bridge. If that bridge is a security device (e.g. firewall), then their traffic is exposed to that.
The isolation in the outbound direction is created by the way frames are forwarded. They are simply dropped on the wire, so no VMs can talk directly to one another without their traffic first going external. In the inbound direction, the isolation is created using the forwarding table.
> Does VEPA look like a regular ethernet interface (eth0) on the virtual
> machine side?
Yes
> Are there any associated user-space tools required for configuring a
> VEPA?
>
The standard brctl utility has been augmented to enable/disable the capability.
> Do you have any HOWTO-style documentation that would demonstrate how
> this interface would be used in production? Or a FAQ?
>
None yet.
> This seems like a very interesting effort but I don't quite have a
> good grasp of VEPA's benefits and limitations -- I imagine that others
> are in the same boat too.
>
There are some seminar slides available on the IEEE 802.1 web-site and elsewhere. The patch had a reference to a seminar, but here is another one you might find helpful:
http://www.internet2.edu/presentations/jt2009jul/20090719-congdon.pdf
I'm happy to try to explain further...
Paul
^ permalink raw reply
* RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Paul Congdon (UC Davis) @ 2009-08-07 18:58 UTC (permalink / raw)
To: 'Stephen Hemminger', 'Fischer, Anna'
Cc: bridge, linux-kernel, netdev, virtualization, evb, davem, kaber,
adobriyan, 'Arnd Bergmann'
In-Reply-To: <20090806210002.75beb7c6@nehalam>
>
> After reading more about this, I am not convinced this should be part
> of the bridge code. The bridge code really consists of two parts:
> forwarding table and optional spanning tree. Well the VEPA code short
> circuits both of these; it can't imagine it working with STP turned
> on. The only part of bridge code that really gets used by this are the
> receive packet hooks and the crufty old API.
>
> So instead of adding more stuff to existing bridge code, why not have
> a new driver for just VEPA. You could do it with a simple version of
> macvlan type driver.
Stephen,
Thanks for your comments and questions. We do believe the bridge code is
the right place for this, so I'd like to embellish on that a bit more to
help persuade you. Sorry for the long winded response, but here are some
thoughts:
- First and foremost, VEPA is going to be a standard addition to the IEEE
802.1Q specification. The working group agreed at the last meeting to
pursue a project to augment the bridge standard with hairpin mode (aka
reflective relay) and a remote filtering service (VEPA). See for details:
http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709
-v01.pdf
- The VEPA functionality was really a pretty small change to the code with
low risk and wouldn't seem to warrant an entire new driver or module.
- There are good use cases where VMs will want to have some of their
interfaces attached to bridges and others to bridges operating in VEPA mode.
In other words, we see simultaneous operation of the bridge code and VEPA
occurring, so having as much of the underlying code as common as possible
would seem to be beneficial.
- By augmenting the bridge code with VEPA there is a great amount of re-use
achieved. It works wherever the bridge code works and doesn't need anything
special to support KVM, XEN, and all the hooks, etc...
- The hardware vendors building SR-IOV NICs with embedded switches will be
adding VEPA mode, so by keeping the bridge module in sync would be
consistent with this trend and direction. It will be possible to extend the
hardware implementations by cascading a software bridge and/or VEPA, so
being in sync with the architecture would make this more consistent.
- The forwarding table is still needed and used on inbound traffic to
deliver frames to the correct virtual interfaces and to filter any reflected
frames. A new driver would have to basically implement an equivalent
forwarding table anyway. As I understand the current macvlan type driver,
it wouldn't filter multicast frames properly without such a table.
- It seems the hairpin mode would be needed in the bridge module whether
VEPA was added to the bridge module or a new driver. Having the associated
changes together in the same code could aid in understanding and deployment.
As I understand the macvlan code, it currently doesn't allow two VMs on the
same machine to communicate with one another. I could imagine a hairpin
mode on the adjacent bridge making this possible, but the macvlan code would
need to be updated to filter reflected frames so a source did not receive
his own packet. I could imagine this being done as well, but to also
support selective multicast usage, something similar to the bridge
forwarding table would be needed. I think putting VEPA into a new driver
would cause you to implement many things the bridge code already supports.
Given that we expect the bridge standard to ultimately include VEPA, and the
new functions are basic forwarding operations, it seems to make most sense
to keep this consistent with the bridge module.
Paul
^ permalink raw reply
* pull request: wireless-2.6 2009-08-07
From: John W. Linville @ 2009-08-07 18:39 UTC (permalink / raw)
To: davem; +Cc: linux-wireless, netdev, linux-kernel
Dave,
Here are a few more late-comers intended for 2.6.31. They are basically
simple one-liners (or might-as-well-be-one-liners), most fixing potential
security or crash issues or avoiding potentially obscure bugs.
Please let me know if there are problems!
Thanks,
John
---
Individual patches are available here:
http://www.kernel.org/pub/linux/kernel/people/linville/wireless-2.6/
---
The following changes since commit 0bf52b981770cbf006323bab5177f2858a196766:
Ingo Molnar (1):
net: Fix spinlock use in alloc_netdev_mq()
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git master
Bob Dunlop (1):
libertas: correct packing of rxpd structure
Lennert Buytenhek (5):
mwl8k: fix NULL pointer dereference on receive out-of-memory
mwl8k: add various missing GET_HW_SPEC endian conversions
mwl8k: call pci_unmap_single() before accessing command structure again
mwl8k: prevent crash in ->configure_filter() if no interface was added
mwl8k: prevent module unload hang
Zhu Yi (1):
ipw2x00: Write outside array bounds
drivers/net/wireless/ipw2x00/ipw2200.c | 2 +-
drivers/net/wireless/libertas/hostcmd.h | 4 ++--
drivers/net/wireless/mwl8k.c | 31 +++++++++++++++++++------------
3 files changed, 22 insertions(+), 15 deletions(-)
diff --git a/drivers/net/wireless/ipw2x00/ipw2200.c b/drivers/net/wireless/ipw2x00/ipw2200.c
index 44c29b3..6dcac73 100644
--- a/drivers/net/wireless/ipw2x00/ipw2200.c
+++ b/drivers/net/wireless/ipw2x00/ipw2200.c
@@ -6226,7 +6226,7 @@ static void ipw_add_scan_channels(struct ipw_priv *priv,
};
u8 channel;
- while (channel_index < IPW_SCAN_CHANNELS) {
+ while (channel_index < IPW_SCAN_CHANNELS - 1) {
channel =
priv->speed_scan[priv->speed_scan_pos];
if (channel == 0) {
diff --git a/drivers/net/wireless/libertas/hostcmd.h b/drivers/net/wireless/libertas/hostcmd.h
index 0a2e291..c8a1998 100644
--- a/drivers/net/wireless/libertas/hostcmd.h
+++ b/drivers/net/wireless/libertas/hostcmd.h
@@ -56,8 +56,8 @@ struct rxpd {
u8 bss_type;
/* BSS number */
u8 bss_num;
- } bss;
- } u;
+ } __attribute__ ((packed)) bss;
+ } __attribute__ ((packed)) u;
/* SNR */
u8 snr;
diff --git a/drivers/net/wireless/mwl8k.c b/drivers/net/wireless/mwl8k.c
index a263d5c..83967af 100644
--- a/drivers/net/wireless/mwl8k.c
+++ b/drivers/net/wireless/mwl8k.c
@@ -261,7 +261,7 @@ struct mwl8k_vif {
*/
};
-#define MWL8K_VIF(_vif) (struct mwl8k_vif *)(&((_vif)->drv_priv))
+#define MWL8K_VIF(_vif) ((struct mwl8k_vif *)&((_vif)->drv_priv))
static const struct ieee80211_channel mwl8k_channels[] = {
{ .center_freq = 2412, .hw_value = 1, },
@@ -1012,6 +1012,8 @@ static int rxq_process(struct ieee80211_hw *hw, int index, int limit)
rmb();
skb = rxq->rx_skb[rxq->rx_head];
+ if (skb == NULL)
+ break;
rxq->rx_skb[rxq->rx_head] = NULL;
rxq->rx_head = (rxq->rx_head + 1) % MWL8K_RX_DESCS;
@@ -1591,6 +1593,9 @@ static int mwl8k_post_cmd(struct ieee80211_hw *hw, struct mwl8k_cmd_pkt *cmd)
timeout = wait_for_completion_timeout(&cmd_wait,
msecs_to_jiffies(MWL8K_CMD_TIMEOUT_MS));
+ pci_unmap_single(priv->pdev, dma_addr, dma_size,
+ PCI_DMA_BIDIRECTIONAL);
+
result = &cmd->result;
if (!timeout) {
spin_lock_irq(&priv->fw_lock);
@@ -1610,8 +1615,6 @@ static int mwl8k_post_cmd(struct ieee80211_hw *hw, struct mwl8k_cmd_pkt *cmd)
*result);
}
- pci_unmap_single(priv->pdev, dma_addr, dma_size,
- PCI_DMA_BIDIRECTIONAL);
return rc;
}
@@ -1654,18 +1657,18 @@ static int mwl8k_cmd_get_hw_spec(struct ieee80211_hw *hw)
memset(cmd->perm_addr, 0xff, sizeof(cmd->perm_addr));
cmd->ps_cookie = cpu_to_le32(priv->cookie_dma);
cmd->rx_queue_ptr = cpu_to_le32(priv->rxq[0].rx_desc_dma);
- cmd->num_tx_queues = MWL8K_TX_QUEUES;
+ cmd->num_tx_queues = cpu_to_le32(MWL8K_TX_QUEUES);
for (i = 0; i < MWL8K_TX_QUEUES; i++)
cmd->tx_queue_ptrs[i] = cpu_to_le32(priv->txq[i].tx_desc_dma);
- cmd->num_tx_desc_per_queue = MWL8K_TX_DESCS;
- cmd->total_rx_desc = MWL8K_RX_DESCS;
+ cmd->num_tx_desc_per_queue = cpu_to_le32(MWL8K_TX_DESCS);
+ cmd->total_rx_desc = cpu_to_le32(MWL8K_RX_DESCS);
rc = mwl8k_post_cmd(hw, &cmd->header);
if (!rc) {
SET_IEEE80211_PERM_ADDR(hw, cmd->perm_addr);
priv->num_mcaddrs = le16_to_cpu(cmd->num_mcaddrs);
- priv->fw_rev = cmd->fw_rev;
+ priv->fw_rev = le32_to_cpu(cmd->fw_rev);
priv->hw_rev = cmd->hw_rev;
priv->region_code = le16_to_cpu(cmd->region_code);
}
@@ -3216,15 +3219,19 @@ static int mwl8k_configure_filter_wt(struct work_struct *wt)
struct dev_addr_list *mclist = worker->mclist;
struct mwl8k_priv *priv = hw->priv;
- struct mwl8k_vif *mv_vif;
int rc = 0;
if (changed_flags & FIF_BCN_PRBRESP_PROMISC) {
if (*total_flags & FIF_BCN_PRBRESP_PROMISC)
rc = mwl8k_cmd_set_pre_scan(hw);
else {
- mv_vif = MWL8K_VIF(priv->vif);
- rc = mwl8k_cmd_set_post_scan(hw, mv_vif->bssid);
+ u8 *bssid;
+
+ bssid = "\x00\x00\x00\x00\x00\x00";
+ if (priv->vif != NULL)
+ bssid = MWL8K_VIF(priv->vif)->bssid;
+
+ rc = mwl8k_cmd_set_post_scan(hw, bssid);
}
}
@@ -3726,6 +3733,8 @@ static void __devexit mwl8k_remove(struct pci_dev *pdev)
ieee80211_stop_queues(hw);
+ ieee80211_unregister_hw(hw);
+
/* Remove tx reclaim tasklet */
tasklet_kill(&priv->tx_reclaim_task);
@@ -3739,8 +3748,6 @@ static void __devexit mwl8k_remove(struct pci_dev *pdev)
for (i = 0; i < MWL8K_TX_QUEUES; i++)
mwl8k_txq_reclaim(hw, i, 1);
- ieee80211_unregister_hw(hw);
-
for (i = 0; i < MWL8K_TX_QUEUES; i++)
mwl8k_txq_deinit(hw, i);
--
John W. Linville Someday the world will need a hero, and you
linville@tuxdriver.com might be all we have. Be ready.
^ permalink raw reply related
* Re: [PATCH 4/7] vbus-proxy: add a pci-to-vbus bridge
From: Gregory Haskins @ 2009-08-07 18:25 UTC (permalink / raw)
To: Ira W. Snyder; +Cc: Arnd Bergmann, alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090807155500.GB12561@ovro.caltech.edu>
[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]
Ira W. Snyder wrote:
<big snip>
> With my powerpc hardware, it should be easily possible to have at least
> 6 devices, each with two virtqueues, one for tx and one for rx. (The
> limit is caused by the amount of distinct kick() events I can generate.)
> This could allow many combinations of devices, such as:
>
> * 1 virtio-net, 1 virtio-console
> * 3 virtio-net, 2 virtio-console
> * 6 virtio-net
> * etc.
Note that the vbus "connector" design abstract details such as kick
events, so you are not limited by the number of physical interrupts of
your hardware, per se.
You can actually have an arbitrary width namespace for your virtqueues
running over a single interrupt, if you were so inclined. (Note you can
also have a 1:1 with interrupts if you like, too)
If you look at the design of the vbus-pcibridge connector, it actually
aggregates the entire namespace into between 1 and 8 interrupts
(depending on availability, and separated by priority level). If
multiple vectors are not available, the entire protocol can fallback to
run over a single vector when necessary.
Kind Regards,
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* Re: [PATCH 0/7] AlacrityVM guest drivers
From: Gregory Haskins @ 2009-08-07 18:04 UTC (permalink / raw)
To: Anthony Liguori
Cc: Michael S. Tsirkin, Gregory Haskins, linux-kernel,
alacrityvm-devel, netdev, kvm
In-Reply-To: <4A7C4C50.5040503@codemonkey.ws>
[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]
Anthony Liguori wrote:
> Gregory Haskins wrote:
>> That said, note that the graphs were from earlier kernel runs (2.6.28,
>> 29-rc8). The most recent data I can find that I published is for
>> 2.6.29, announced with the vbus-v3 release back in April:
>>
>> http://lkml.org/lkml/2009/4/21/408
>>
>> In it, the virtio-net throughput numbers are substantially higher and
>> possibly more in line with your expectations (4.5gb/s) (though notably
>> still lagging venet, which weighed in at 5.6gb/s).
>>
>
> Okay, that makes more sense. Would be nice to update the graphs as they
> make virtio look really, really bad :-)
Yeah, they are certainly ripe for an update. (Note that I was
unilaterally stale on venet numbers, too) ;)
>
>> Generally, I find that the virtio-net exhibits non-deterministic results
>> from release to release. I suspect (as we have discussed) the
>> tx-mitigation scheme. Some releases buffer the daylights out of the
>> stream, and virtio gets close(r) throughput (e.g. 4.5g vs 5.8g, but
>> absolutely terrible latency (4000us vs 65us). Other releases it seems
>> to operate with more of a compromise (1.3gb/s vs 3.8gb/s, but 350us vs
>> 85us).
>>
>
> Are you using kvm modules or a new kernel?
I just build the entire kernel from git.
> There was some timer
> infrastructure changes around 28/29 and it's possible that the system
> your on is now detecting an hpet which will result in a better time
> source. That could have an affect on mitigation.
Yeah, I suspect you are right. I always kept the .config and machine
constant, but I *do* bounce around kernel versions so perhaps I got
hosed by a make-oldconfig cycle somewhere along the way.
>
>> If there is another patch-series/tree I should be using for comparison,
>> please point me at it.
>>
>
> No, I think it's fair to look at upstream Linux. Looking at the latest
> bits would be nice though because there are some virtio friendly changes
> recently like MSI-x and GRO.
Yeah, I will definitely include kvm.git in addition to whatever is
current from Linus. I already have adopted using the latest
qemu-kvm.git into my workflow.
Regards,
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* [PATCH] yellowfin: Fix buffer underrun after dev_alloc_skb() failure
From: Roel Kluin @ 2009-08-07 17:57 UTC (permalink / raw)
To: netdev, Andrew Morton, David S. Miller
When dev_alloc_skb fails in the first iteration, a buffer underrun occurs.
Signed-off-by: Roel Kluin <roel.kluin@gmail.com>
---
Is this the best way to fix this? Untested, please review.
diff --git a/drivers/net/yellowfin.c b/drivers/net/yellowfin.c
index a075801..a82ac32 100644
--- a/drivers/net/yellowfin.c
+++ b/drivers/net/yellowfin.c
@@ -728,7 +728,7 @@ static void yellowfin_tx_timeout(struct net_device *dev)
static void yellowfin_init_ring(struct net_device *dev)
{
struct yellowfin_private *yp = netdev_priv(dev);
- int i;
+ int i, j;
yp->tx_full = 0;
yp->cur_rx = yp->cur_tx = 0;
@@ -753,6 +753,11 @@ static void yellowfin_init_ring(struct net_device *dev)
yp->rx_ring[i].addr = cpu_to_le32(pci_map_single(yp->pci_dev,
skb->data, yp->rx_buf_sz, PCI_DMA_FROMDEVICE));
}
+ if (i != RX_RING_SIZE) {
+ for (j = 0; j < i; j++)
+ dev_kfree_skb(yp->rx_skbuff[j]);
+ return;
+ }
yp->rx_ring[i-1].dbdma_cmd = cpu_to_le32(CMD_STOP);
yp->dirty_rx = (unsigned int)(i - RX_RING_SIZE);
^ permalink raw reply related
* [net-2.6 PATCH 2/2] e1000e: fix potential NVM corruption on ICH9 with 8K bank size
From: Jeff Kirsher @ 2009-08-07 17:41 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Bruce Allan, Jeff Kirsher
In-Reply-To: <20090807174136.31114.96240.stgit@localhost.localdomain>
From: Bruce Allan <bruce.w.allan@intel.com>
The bank offset was being incorrectly calculated on ICH9 parts with a bank
size of 8K (instead of the more common 4K bank) which would cause any NVM
writes to be done on the wrong address after switching from bank 1 to bank
0. Additionally, assume we are meant to use bank 0 if a valid bank is not
detected, and remove the unnecessary acquisition of the SW/FW/HW semaphore
when writing to the shadow ram version of the NVM image.
Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/e1000e/ich8lan.c | 50 ++++++++++--------------------------------
1 files changed, 12 insertions(+), 38 deletions(-)
diff --git a/drivers/net/e1000e/ich8lan.c b/drivers/net/e1000e/ich8lan.c
index dd61e7e..99df2ab 100644
--- a/drivers/net/e1000e/ich8lan.c
+++ b/drivers/net/e1000e/ich8lan.c
@@ -338,10 +338,7 @@ static s32 e1000_init_nvm_params_ich8lan(struct e1000_hw *hw)
{
struct e1000_nvm_info *nvm = &hw->nvm;
struct e1000_dev_spec_ich8lan *dev_spec = &hw->dev_spec.ich8lan;
- union ich8_hws_flash_status hsfsts;
- u32 gfpreg;
- u32 sector_base_addr;
- u32 sector_end_addr;
+ u32 gfpreg, sector_base_addr, sector_end_addr;
u16 i;
/* Can't read flash registers if the register set isn't mapped. */
@@ -375,20 +372,6 @@ static s32 e1000_init_nvm_params_ich8lan(struct e1000_hw *hw)
/* Adjust to word count */
nvm->flash_bank_size /= sizeof(u16);
- /*
- * Make sure the flash bank size does not overwrite the 4k
- * sector ranges. We may have 64k allotted to us but we only care
- * about the first 2 4k sectors. Therefore, if we have anything less
- * than 64k set in the HSFSTS register, we will reduce the bank size
- * down to 4k and let the rest remain unused. If berasesz == 3, then
- * we are working in 64k mode. Otherwise we are not.
- */
- if (nvm->flash_bank_size > E1000_ICH8_SHADOW_RAM_WORDS) {
- hsfsts.regval = er16flash(ICH_FLASH_HSFSTS);
- if (hsfsts.hsf_status.berasesz != 3)
- nvm->flash_bank_size = E1000_ICH8_SHADOW_RAM_WORDS;
- }
-
nvm->word_size = E1000_ICH8_SHADOW_RAM_WORDS;
/* Clear shadow ram */
@@ -1324,7 +1307,7 @@ static s32 e1000_read_nvm_ich8lan(struct e1000_hw *hw, u16 offset, u16 words,
struct e1000_nvm_info *nvm = &hw->nvm;
struct e1000_dev_spec_ich8lan *dev_spec = &hw->dev_spec.ich8lan;
u32 act_offset;
- s32 ret_val;
+ s32 ret_val = 0;
u32 bank = 0;
u16 i, word;
@@ -1339,12 +1322,15 @@ static s32 e1000_read_nvm_ich8lan(struct e1000_hw *hw, u16 offset, u16 words,
goto out;
ret_val = e1000_valid_nvm_bank_detect_ich8lan(hw, &bank);
- if (ret_val)
- goto release;
+ if (ret_val) {
+ hw_dbg(hw, "Could not detect valid bank, assuming bank 0\n");
+ bank = 0;
+ }
act_offset = (bank) ? nvm->flash_bank_size : 0;
act_offset += offset;
+ ret_val = 0;
for (i = 0; i < words; i++) {
if ((dev_spec->shadow_ram) &&
(dev_spec->shadow_ram[offset+i].modified)) {
@@ -1359,7 +1345,6 @@ static s32 e1000_read_nvm_ich8lan(struct e1000_hw *hw, u16 offset, u16 words,
}
}
-release:
e1000_release_swflag_ich8lan(hw);
out:
@@ -1610,7 +1595,6 @@ static s32 e1000_write_nvm_ich8lan(struct e1000_hw *hw, u16 offset, u16 words,
{
struct e1000_nvm_info *nvm = &hw->nvm;
struct e1000_dev_spec_ich8lan *dev_spec = &hw->dev_spec.ich8lan;
- s32 ret_val;
u16 i;
if ((offset >= nvm->word_size) || (words > nvm->word_size - offset) ||
@@ -1619,17 +1603,11 @@ static s32 e1000_write_nvm_ich8lan(struct e1000_hw *hw, u16 offset, u16 words,
return -E1000_ERR_NVM;
}
- ret_val = e1000_acquire_swflag_ich8lan(hw);
- if (ret_val)
- return ret_val;
-
for (i = 0; i < words; i++) {
dev_spec->shadow_ram[offset+i].modified = 1;
dev_spec->shadow_ram[offset+i].value = data[i];
}
- e1000_release_swflag_ich8lan(hw);
-
return 0;
}
@@ -1670,8 +1648,8 @@ static s32 e1000_update_nvm_checksum_ich8lan(struct e1000_hw *hw)
*/
ret_val = e1000_valid_nvm_bank_detect_ich8lan(hw, &bank);
if (ret_val) {
- e1000_release_swflag_ich8lan(hw);
- goto out;
+ hw_dbg(hw, "Could not detect valid bank, assuming bank 0\n");
+ bank = 0;
}
if (bank == 0) {
@@ -2057,12 +2035,8 @@ static s32 e1000_erase_flash_bank_ich8lan(struct e1000_hw *hw, u32 bank)
iteration = 1;
break;
case 2:
- if (hw->mac.type == e1000_ich9lan) {
- sector_size = ICH_FLASH_SEG_SIZE_8K;
- iteration = flash_bank_size / ICH_FLASH_SEG_SIZE_8K;
- } else {
- return -E1000_ERR_NVM;
- }
+ sector_size = ICH_FLASH_SEG_SIZE_8K;
+ iteration = 1;
break;
case 3:
sector_size = ICH_FLASH_SEG_SIZE_64K;
@@ -2074,7 +2048,7 @@ static s32 e1000_erase_flash_bank_ich8lan(struct e1000_hw *hw, u32 bank)
/* Start with the base address, then add the sector offset. */
flash_linear_addr = hw->nvm.flash_base_addr;
- flash_linear_addr += (bank) ? (sector_size * iteration) : 0;
+ flash_linear_addr += (bank) ? flash_bank_size : 0;
for (j = 0; j < iteration ; j++) {
do {
^ permalink raw reply related
* Re: [PATCH 2/7 v3] initdev:kernel:USB init device discovery notification
From: David VomLehn @ 2009-08-07 17:42 UTC (permalink / raw)
To: Alan Jenkins
Cc: Alan Stern, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
linux-usb-u79uwXL29TY76Z2rM5mHXA, greg-U8xfFu+wG4EAvxtiuMwx3w,
linux-scsi-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA,
arjan-wEGCiKHe2LqWVfeAwA7xHQ
In-Reply-To: <9b2b86520908071017h606e595fr4488d6ff61cb337c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Fri, Aug 07, 2009 at 06:17:51PM +0100, Alan Jenkins wrote:
> On 8/7/09, Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org> wrote:
> > On Fri, 7 Aug 2009, Alan Jenkins wrote:
> >
> >> > +/* The mask of possible USB boot devices depends on what drivers and
> >> > + * options have been configured into the kernel. There are too many
> >> > + * USB network config options to list here, so just assume it is always
> >> > + * possible to have a USB network device.
> >> > + */
> >>
> >> That could be fixed though. Make the "USB Network Adaptors" menu into
> >> an option. Then you just have e.g. "CONFIG_NETDEV_USB", plus the two
> >> class devices outside the menu (CONFIG_USB_ACM and CONFIG_USB_WDM).
>
> Oops, scratch WDM. It doesn't actually provide a network device
> (otherwise it would depend on NET).
>
> > That could be done. It might even meet with the approval of the
> > networking maintainer, independent of this patch series.
> >
> > But what if somebody adds yet another networking class driver outside
> > the menu?
>
> Hmm.
>
> Ok, how about a single (hidden) USB_HAS_NETDEV. NETDEV_USB and
> USB_ACM select USB_HAS_NETDEV.
>
> Textual juxtaposition or simple copy+paste inside class/Kconfig should
> help it get picked up. And I'd add a comment pointing the bewildered
> to the definition of usb_initdev_mask.
>
> I'll write a patch for your inspection this evening.
I remember thinking about this and seem to recall that Alan Stern and I
tossed this idea around, but didn't act on it. Such a thing holds potential
for avoiding an unnecessary delay during boot. Whether it actually makes
things faster in a given configuration will depend on that configuration's
critical path, but I would welcome such a patch on the basis that it's
unlikely to hurt and could improve boot times in some circumstances.
> Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [net-2.6 PATCH 1/2] e1000e: fix acquisition of SW/FW/HW semaphore for ICHx parts
From: Jeff Kirsher @ 2009-08-07 17:41 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Bruce Allan, Jeff Kirsher
From: Bruce Allan <bruce.w.allan@intel.com>
For ICHx parts, write the EXTCNF_CTRL.SWFLAG bit once when trying to
acquire the SW/FW/HW semaphore instead of multiple times to prevent the
hardware from having problems (especially for systems with manageability
enabled), and extend the timeout for the hardware to set the SWFLAG bit.
Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/e1000e/ich8lan.c | 44 ++++++++++++++++++++++++++++++------------
1 files changed, 31 insertions(+), 13 deletions(-)
diff --git a/drivers/net/e1000e/ich8lan.c b/drivers/net/e1000e/ich8lan.c
index d56c747..dd61e7e 100644
--- a/drivers/net/e1000e/ich8lan.c
+++ b/drivers/net/e1000e/ich8lan.c
@@ -594,8 +594,8 @@ static DEFINE_MUTEX(nvm_mutex);
**/
static s32 e1000_acquire_swflag_ich8lan(struct e1000_hw *hw)
{
- u32 extcnf_ctrl;
- u32 timeout = PHY_CFG_TIMEOUT;
+ u32 extcnf_ctrl, timeout = PHY_CFG_TIMEOUT;
+ s32 ret_val = 0;
might_sleep();
@@ -603,28 +603,46 @@ static s32 e1000_acquire_swflag_ich8lan(struct e1000_hw *hw)
while (timeout) {
extcnf_ctrl = er32(EXTCNF_CTRL);
+ if (!(extcnf_ctrl & E1000_EXTCNF_CTRL_SWFLAG))
+ break;
- if (!(extcnf_ctrl & E1000_EXTCNF_CTRL_SWFLAG)) {
- extcnf_ctrl |= E1000_EXTCNF_CTRL_SWFLAG;
- ew32(EXTCNF_CTRL, extcnf_ctrl);
+ mdelay(1);
+ timeout--;
+ }
+
+ if (!timeout) {
+ hw_dbg(hw, "SW/FW/HW has locked the resource for too long.\n");
+ ret_val = -E1000_ERR_CONFIG;
+ goto out;
+ }
+
+ timeout = PHY_CFG_TIMEOUT * 2;
+
+ extcnf_ctrl |= E1000_EXTCNF_CTRL_SWFLAG;
+ ew32(EXTCNF_CTRL, extcnf_ctrl);
+
+ while (timeout) {
+ extcnf_ctrl = er32(EXTCNF_CTRL);
+ if (extcnf_ctrl & E1000_EXTCNF_CTRL_SWFLAG)
+ break;
- extcnf_ctrl = er32(EXTCNF_CTRL);
- if (extcnf_ctrl & E1000_EXTCNF_CTRL_SWFLAG)
- break;
- }
mdelay(1);
timeout--;
}
if (!timeout) {
- hw_dbg(hw, "FW or HW has locked the resource for too long.\n");
+ hw_dbg(hw, "Failed to acquire the semaphore.\n");
extcnf_ctrl &= ~E1000_EXTCNF_CTRL_SWFLAG;
ew32(EXTCNF_CTRL, extcnf_ctrl);
- mutex_unlock(&nvm_mutex);
- return -E1000_ERR_CONFIG;
+ ret_val = -E1000_ERR_CONFIG;
+ goto out;
}
- return 0;
+out:
+ if (ret_val)
+ mutex_unlock(&nvm_mutex);
+
+ return ret_val;
}
/**
^ permalink raw reply related
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Daniel Robbins @ 2009-08-07 17:35 UTC (permalink / raw)
To: Arnd Bergmann
Cc: netdev, Herbert Xu, Michael S. Tsirkin, Fischer, Anna, bridge,
linux-kernel, David S. Miller", Or Gerlitz,
Edge Virtual Bridging
In-Reply-To: <1249595428-21594-1-git-send-email-arnd@arndb.de>
On Thu, Aug 6, 2009 at 3:50 PM, Arnd Bergmann<arnd@arndb.de> wrote:
> This is a first prototype of a new interface into the network
> stack, to eventually replace tun/tap and the bridge driver
> in certain virtual machine setups.
I have some general questions about the intended use and benefits of
VEPA, from an IT perspective:
In which virtual machine setups and technologies do you forsee this
interface being used?
Is this new interface to be used within a virtual machine or
container, on the master node, or both?
What interface(s) would need to be configured for a single virtual
machine to use VEPA to access the network?
What are the current flexibility, security or performance limitations
of tun/tap and bridge that make this new interface necessary or
beneficial?
Is this new interface useful at all for VPN solutions or is it
*specifically* targeted for connecting virtual machines to the
network?
Is this essentially a bridge with layer-2 isolation for the virtual
machine interfaces built-in? If isolation is provided, what mechanism
is used to accomplish this, and how secure is it?
Does VEPA look like a regular ethernet interface (eth0) on the virtual
machine side?
Are there any associated user-space tools required for configuring a VEPA?
Do you have any HOWTO-style documentation that would demonstrate how
this interface would be used in production? Or a FAQ?
This seems like a very interesting effort but I don't quite have a
good grasp of VEPA's benefits and limitations -- I imagine that others
are in the same boat too.
Best Regards,
Daniel
^ permalink raw reply
* Re: [PATCH 2/7 v3] initdev:kernel:USB init device discovery notification
From: Alan Jenkins @ 2009-08-07 17:17 UTC (permalink / raw)
To: Alan Stern
Cc: David VomLehn, linux-kernel, akpm, linux-usb, greg, linux-scsi,
netdev, arjan
In-Reply-To: <Pine.LNX.4.44L0.0908071142260.3063-100000@iolanthe.rowland.org>
On 8/7/09, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Fri, 7 Aug 2009, Alan Jenkins wrote:
>
>> > +/* The mask of possible USB boot devices depends on what drivers and
>> > + * options have been configured into the kernel. There are too many
>> > + * USB network config options to list here, so just assume it is always
>> > + * possible to have a USB network device.
>> > + */
>>
>> That could be fixed though. Make the "USB Network Adaptors" menu into
>> an option. Then you just have e.g. "CONFIG_NETDEV_USB", plus the two
>> class devices outside the menu (CONFIG_USB_ACM and CONFIG_USB_WDM).
Oops, scratch WDM. It doesn't actually provide a network device
(otherwise it would depend on NET).
> That could be done. It might even meet with the approval of the
> networking maintainer, independent of this patch series.
>
> But what if somebody adds yet another networking class driver outside
> the menu?
Hmm.
Ok, how about a single (hidden) USB_HAS_NETDEV. NETDEV_USB and
USB_ACM select USB_HAS_NETDEV.
Textual juxtaposition or simple copy+paste inside class/Kconfig should
help it get picked up. And I'd add a comment pointing the bewildered
to the definition of usb_initdev_mask.
I'll write a patch for your inspection this evening.
Alan
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox