* [PATCH 0/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:21 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt, nhorman
Hey all-
I put out an RFC about this awhile ago and didn't get any loud screams,
so I've gone ahead and implemented it
Currently, our network infrastructure allows net device drivers to
allocate skbs based on the the numa node the device itself is local to. This of
course cuts down on cross numa chatter when the device is DMA-ing network
traffic to the driver. Unfortuantely no such corresponding infrastrucuture
exists at the process level. The scheduler has no insight into the numa
locality of incomming data packets for a given process (and arguably it
shouldn't), and so there is every chance that a process will run on a different
numa node than the packets that its receiving lives on, creating cross numa node
traffic.
This patch aims to provide userspace with the opportunity to optimize
that scheduling. It consists of a tracepoint and an ftrace module which exports
a history of the packets each process receives, along with the numa node each
packet was received on, as well as the numa node the process was running on when
it copied the buffer to user space. With this information, exported via the
ftrace infrastructure to user space, a sysadim can identify high prirority
processes, and optimize their scheduling so that they are more likely to run on
the same node that they are primarily receiving data on, thereby cutting down
cross numa node traffic.
Tested by me, working well, applies against the head of the net-next tree
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
^ permalink raw reply
* Re: [PATCH 1/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:28 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / cosumption tracer - Add consumption tracepoint
This patch adds a tracepoint to skb_copy_datagram_iovec, which is called each
time a userspace process copies a frame from a socket receive queue to a user
space buffer. It allows us to hook in and examine each sk_buff that the system
receives on a per-socket bases, and can be use to compile a list of which skb's
were received by which processes.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
include/trace/events/skb.h | 20 ++++++++++++++++++++
net/core/datagram.c | 3 +++
2 files changed, 23 insertions(+)
diff --git a/include/trace/events/skb.h b/include/trace/events/skb.h
index 1e8fabb..bdc5c1c 100644
--- a/include/trace/events/skb.h
+++ b/include/trace/events/skb.h
@@ -2,6 +2,7 @@
#define _TRACE_SKB_H
#include <linux/skbuff.h>
+#include <linux/netdevice.h>
#include <linux/tracepoint.h>
#undef TRACE_SYSTEM
@@ -34,6 +35,25 @@ TRACE_EVENT(kfree_skb,
__entry->skbaddr, __entry->protocol, __entry->location)
);
+TRACE_EVENT(skb_copy_datagram_iovec,
+
+ TP_PROTO(const struct sk_buff *skb, int len),
+
+ TP_ARGS(skb, len),
+
+ TP_STRUCT__entry(
+ __field( const void *, skbaddr )
+ __field( int, len )
+ ),
+
+ TP_fast_assign(
+ __entry->skbaddr = skb;
+ __entry->len = len;
+ ),
+
+ TP_printk("skbaddr=%p len=%d", __entry->skbaddr, __entry->len)
+);
+
#endif /* _TRACE_SKB_H */
/* This part must be outside protection */
diff --git a/net/core/datagram.c b/net/core/datagram.c
index b0fe692..1c6cf3a 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -55,6 +55,7 @@
#include <net/checksum.h>
#include <net/sock.h>
#include <net/tcp_states.h>
+#include <trace/events/skb.h>
/*
* Is a socket 'connection oriented' ?
@@ -284,6 +285,8 @@ int skb_copy_datagram_iovec(const struct sk_buff *skb, int offset,
int i, copy = start - offset;
struct sk_buff *frag_iter;
+ trace_skb_copy_datagram_iovec(skb, len);
+
/* Copy header. */
if (copy > 0) {
if (copy > len)
^ permalink raw reply related
* Re: [PATCH 2/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:30 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / consumption corelator - Add config option
This patch adds a Kconfig option to enable the addtition of the skb source
tracer.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Kconfig | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 1551f47..1aeec05 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -234,6 +234,16 @@ config BOOT_TRACER
You must pass in ftrace=initcall to the kernel command line
to enable this on bootup.
+config SKB_SOURCES_TRACER
+ bool "Trace skb source information
+ select GENERIC_TRACER
+ help
+ This tracer helps developers/sysadmins correlate skb allocation and
+ consumption. The idea being that some processes will primarily consume data
+ that was allocated on certain numa nodes. By being able to visualize which
+ nodes the data was allocated on, a sysadmin or developer can optimize the
+ scheduling of those processes to cut back on cross node chatter.
+
config TRACE_BRANCH_PROFILING
bool
select GENERIC_TRACER
^ permalink raw reply related
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Yaron Haviv @ 2009-08-07 20:35 UTC (permalink / raw)
To: evb, shemminger, anna.fischer
Cc: arnd, davem, netdev, bridge, adobriyan, virtualization
[-- Attachment #1.1: Type: text/plain, Size: 9517 bytes --]
Paul,
I also think that bridge may not be the right place for VEPA, but rather a simpler sw/hw mux
Although the VEPA support may reside in multiple places (I.e. also in the bridge)
As Arnd pointed out Or already added an extension to qemu that allow direct guest virtual NIC mapping to an interface device (vs using tap), this was done specifically to address VEPA, and result in much faster performance and lower cpu overhead (Or and some others are planning additional meaningful performance optimizations)
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
It may be counter intuitive for some, but we expect the (completed) qemu VEPA mode + SR-IOV + certain switches with hairpin (vepa) mode to perform faster than using bridge+tap even for connecting 2 VMs on the same host
Yaron
Sent from BlackBerry
________________________________
From: evb@yahoogroups.com
To: 'Stephen Hemminger' ; 'Fischer, Anna'
Cc: bridge@lists.linux-foundation.org ; linux-kernel@vger.kernel.org ; netdev@vger.kernel.org ; virtualization@lists.linux-foundation.org ; evb@yahoogroups.com ; davem@davemloft.net ; kaber@trash.net ; adobriyan@gmail.com ; 'Arnd Bergmann'
Sent: Fri Aug 07 21:58:00 2009
Subject: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
>
> After reading more about this, I am not convinced this should be part
> of the bridge code. The bridge code really consists of two parts:
> forwarding table and optional spanning tree. Well the VEPA code short
> circuits both of these; it can't imagine it working with STP turned
> on. The only part of bridge code that really gets used by this are the
> receive packet hooks and the crufty old API.
>
> So instead of adding more stuff to existing bridge code, why not have
> a new driver for just VEPA. You could do it with a simple version of
> macvlan type driver.
Stephen,
Thanks for your comments and questions. We do believe the bridge code is
the right place for this, so I'd like to embellish on that a bit more to
help persuade you. Sorry for the long winded response, but here are some
thoughts:
- First and foremost, VEPA is going to be a standard addition to the IEEE
802.1Q specification. The working group agreed at the last meeting to
pursue a project to augment the bridge standard with hairpin mode (aka
reflective relay) and a remote filtering service (VEPA). See for details:
http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709 <http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709>
-v01.pdf
- The VEPA functionality was really a pretty small change to the code with
low risk and wouldn't seem to warrant an entire new driver or module.
- There are good use cases where VMs will want to have some of their
interfaces attached to bridges and others to bridges operating in VEPA mode.
In other words, we see simultaneous operation of the bridge code and VEPA
occurring, so having as much of the underlying code as common as possible
would seem to be beneficial.
- By augmenting the bridge code with VEPA there is a great amount of re-use
achieved. It works wherever the bridge code works and doesn't need anything
special to support KVM, XEN, and all the hooks, etc...
- The hardware vendors building SR-IOV NICs with embedded switches will be
adding VEPA mode, so by keeping the bridge module in sync would be
consistent with this trend and direction. It will be possible to extend the
hardware implementations by cascading a software bridge and/or VEPA, so
being in sync with the architecture would make this more consistent.
- The forwarding table is still needed and used on inbound traffic to
deliver frames to the correct virtual interfaces and to filter any reflected
frames. A new driver would have to basically implement an equivalent
forwarding table anyway. As I understand the current macvlan type driver,
it wouldn't filter multicast frames properly without such a table.
- It seems the hairpin mode would be needed in the bridge module whether
VEPA was added to the bridge module or a new driver. Having the associated
changes together in the same code could aid in understanding and deployment.
As I understand the macvlan code, it currently doesn't allow two VMs on the
same machine to communicate with one another. I could imagine a hairpin
mode on the adjacent bridge making this possible, but the macvlan code would
need to be updated to filter reflected frames so a source did not receive
his own packet. I could imagine this being done as well, but to also
support selective multicast usage, something similar to the bridge
forwarding table would be needed. I think putting VEPA into a new driver
would cause you to implement many things the bridge code already supports.
Given that we expect the bridge standard to ultimately include VEPA, and the
new functions are basic forwarding operations, it seems to make most sense
to keep this consistent with the bridge module.
Paul
__._,_.___
Messages in this topic <http://groups.yahoo.com/group/evb/message/167;_ylc=X3oDMTMzb3FibzIzBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARtc2dJZAMyMTIEc2VjA2Z0cgRzbGsDdnRwYwRzdGltZQMxMjQ5NjcxNTEwBHRwY0lkAzE2Nw--> (9) Reply (via web post) <http://groups.yahoo.com/group/evb/post;_ylc=X3oDMTJwcDZzNTZqBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARtc2dJZAMyMTIEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxMjQ5NjcxNTEw?act=reply&messageNum=212> | Start a new topic <http://groups.yahoo.com/group/evb/post;_ylc=X3oDMTJmZW52ZmhiBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNudHBjBHN0aW1lAzEyNDk2NzE1MTA->
Messages <http://groups.yahoo.com/group/evb/messages;_ylc=X3oDMTJmODJ0MmU2BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNtc2dzBHN0aW1lAzEyNDk2NzE1MTA-> | Files <http://groups.yahoo.com/group/evb/files;_ylc=X3oDMTJnaGdsYXI4BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNmaWxlcwRzdGltZQMxMjQ5NjcxNTEw> | Photos <http://groups.yahoo.com/group/evb/photos;_ylc=X3oDMTJmYm90MmpqBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNwaG90BHN0aW1lAzEyNDk2NzE1MTA-> | Links <http://groups.yahoo.com/group/evb/links;_ylc=X3oDMTJnbWdyaTZnBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNsaW5rcwRzdGltZQMxMjQ5NjcxNTEw> | Database <http://groups.yahoo.com/group/evb/database;_ylc=X3oDMTJkZW1ka3FhBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNkYgRzdGltZQMxMjQ5NjcxNTEw> | Polls <http://groups.yahoo.com/group/evb/polls;_ylc=X3oDMTJnMG9lZTJuBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNwb2xscwRzdGltZQMxMjQ5NjcxNTEw> | Members <http://groups.yahoo.com/group/evb/members;_ylc=X3oDMTJmMWdwYXViBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNtYnJzBHN0aW1lAzEyNDk2NzE1MTA-> | Calendar <http://groups.yahoo.com/group/evb/calendar;_ylc=X3oDMTJlZnQ1N25iBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNjYWwEc3RpbWUDMTI0OTY3MTUxMA-->
Yahoo! Groups <http://groups.yahoo.com/;_ylc=X3oDMTJlNDhoZDY1BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNnZnAEc3RpbWUDMTI0OTY3MTUxMA-->
Change settings via the Web <http://groups.yahoo.com/group/evb/join;_ylc=X3oDMTJna2g4aW9zBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNzdG5ncwRzdGltZQMxMjQ5NjcxNTEw> (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest <mailto:evb-digest@yahoogroups.com?subject=Email Delivery: Digest> | Switch format to Traditional <mailto:evb-traditional@yahoogroups.com?subject=Change Delivery Format: Traditional>
Visit Your Group <http://groups.yahoo.com/group/evb;_ylc=X3oDMTJlN2ZwMTRxBF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDZnRyBHNsawNocGYEc3RpbWUDMTI0OTY3MTUxMA--> | Yahoo! Groups Terms of Use <http://docs.yahoo.com/info/terms/> | Unsubscribe <mailto:evb-unsubscribe@yahoogroups.com?subject=>
Recent Activity
Visit Your Group <http://groups.yahoo.com/group/evb;_ylc=X3oDMTJmYW91dGs2BF9TAzk3MzU5NzE0BGdycElkAzIzODk2NDQ3BGdycHNwSWQDMTcwNTAwNDc1MARzZWMDdnRsBHNsawN2Z2hwBHN0aW1lAzEyNDk2NzE1MTA->
Give Back
Yahoo! for Good <http://us.lrd.yahoo.com/_ylc=X3oDMTJuam45aG04BF9TAzk3MzU5NzE0BF9wAzEEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDYnJhbmQEc3RpbWUDMTI0OTY3MTUxMA--;_ylg=1/SIG=11314uv3k/**http%3A//brand.yahoo.com/forgood>
Get inspired
by a good cause.
Y! Toolbar
Get it Free! <http://us.lrd.yahoo.com/_ylc=X3oDMTJwbGY0NzUzBF9TAzk3MzU5NzE0BF9wAzIEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDdG9vbGJhcgRzdGltZQMxMjQ5NjcxNTEw;_ylg=1/SIG=11c6dvmk9/**http%3A//toolbar.yahoo.com/%3F.cpdl=ygrps>
easy 1-click access
to your groups.
Yahoo! Groups
Start a group <http://groups.yahoo.com/start;_ylc=X3oDMTJwdjNqdTNiBF9TAzk3MzU5NzE0BF9wAzMEZ3JwSWQDMjM4OTY0NDcEZ3Jwc3BJZAMxNzA1MDA0NzUwBHNlYwNuY21vZARzbGsDZ3JvdXBzMgRzdGltZQMxMjQ5NjcxNTEw>
in 3 easy steps.
Connect with others.
.
<http://geo.yahoo.com/serv?s=97359714/grpId=23896447/grpspId=1705004750/msgId=212/stime=1249671510/nc1=1/nc2=2/nc3=3>
__,_._,___
[-- Attachment #1.2: Type: text/html, Size: 18843 bytes --]
[-- Attachment #2: Type: text/plain, Size: 184 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* [PATCH 5/6] [NET] ixp2000/enp2611: don't set non-existent member get_stats
From: Uwe Kleine-König @ 2009-08-07 20:42 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, rt-users, Russell King, Dmitry Baryshkov, Lennert Buytenhek,
netdev
In-Reply-To: <20090807203939.GA19374@pengutronix.de>
This fixes a build failure for 2.6.31-rc5 (ARCH=arm, ixp2000_defconfig):
CC drivers/net/ixp2000/enp2611.o
drivers/net/ixp2000/enp2611.c: In function 'enp2611_init_module':
drivers/net/ixp2000/enp2611.c:213: error: 'struct net_device' has no member named 'get_stats'
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Dmitry Baryshkov <dbaryshkov@gmail.com>
Cc: Lennert Buytenhek <kernel@wantstofly.org>
Cc: netdev@vger.kernel.org
---
Hello,
obviously this has the downside that the stats won't work, I let the fix
for someone else :-)
Best regards
Uwe
---
drivers/net/ixp2000/enp2611.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/net/ixp2000/enp2611.c b/drivers/net/ixp2000/enp2611.c
index b02a981..d6ef176 100644
--- a/drivers/net/ixp2000/enp2611.c
+++ b/drivers/net/ixp2000/enp2611.c
@@ -210,7 +210,7 @@ static int __init enp2611_init_module(void)
return -ENOMEM;
}
- nds[i]->get_stats = enp2611_get_stats;
+ /* nds[i]->get_stats = enp2611_get_stats; */
pm3386_init_port(i);
pm3386_get_mac(i, nds[i]->dev_addr);
}
--
1.6.3.3
^ permalink raw reply related
* Re: [PATCH 3/3] net: Add ftracer to help optimize process scheduling based on incomming frame allocations
From: Neil Horman @ 2009-08-07 20:44 UTC (permalink / raw)
To: netdev; +Cc: davem, rostedt
In-Reply-To: <20090807202130.GA26677@hmsreliant.think-freely.org>
skb allocation / consumption correlator
Add ftracer module to kernel to print out a list that correlates a process id,
an skb it read, and the numa nodes on wich the process was running when it was
read along with the numa node the skbuff was allocated on.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
kernel/trace/Makefile | 1
kernel/trace/trace.h | 19 ++++
kernel/trace/trace_skb_sources.c | 154 +++++++++++++++++++++++++++++++++++++++
net/core/datagram.c | 3
4 files changed, 177 insertions(+)
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index 844164d..ee5e5b1 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o
ifeq ($(CONFIG_BLOCK),y)
obj-$(CONFIG_EVENT_TRACING) += blktrace.o
endif
+obj-$(CONFIG_SKB_SOURCES_TRACER) += trace_skb_sources.o
obj-$(CONFIG_EVENT_TRACING) += trace_events.o
obj-$(CONFIG_EVENT_TRACING) += trace_export.o
obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 3548ae5..8c1d458 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -11,6 +11,7 @@
#include <trace/boot.h>
#include <linux/kmemtrace.h>
#include <trace/power.h>
+#include <trace/events/skb.h>
#include <linux/trace_seq.h>
#include <linux/ftrace_event.h>
@@ -40,6 +41,7 @@ enum trace_type {
TRACE_KMEM_FREE,
TRACE_POWER,
TRACE_BLK,
+ TRACE_SKB_SOURCE,
__TRACE_LAST_TYPE,
};
@@ -171,6 +173,21 @@ struct trace_power {
struct power_trace state_data;
};
+struct skb_record {
+ pid_t pid; /* pid of the copying process */
+ int anid; /* node where skb was allocated */
+ int cnid; /* node to which skb was copied in userspace */
+ char ifname[IFNAMSIZ]; /* Name of the receiving interface */
+ int rx_queue; /* The rx queue the skb was received on */
+ int ccpu; /* Cpu the application got this frame from */
+ int len; /* length of the data copied */
+};
+
+struct trace_skb_event {
+ struct trace_entry ent;
+ struct skb_record event_data;
+};
+
enum kmemtrace_type_id {
KMEMTRACE_TYPE_KMALLOC = 0, /* kmalloc() or kfree(). */
KMEMTRACE_TYPE_CACHE, /* kmem_cache_*(). */
@@ -323,6 +340,8 @@ extern void __ftrace_bad_type(void);
TRACE_SYSCALL_ENTER); \
IF_ASSIGN(var, ent, struct syscall_trace_exit, \
TRACE_SYSCALL_EXIT); \
+ IF_ASSIGN(var, ent, struct trace_skb_event, \
+ TRACE_SKB_SOURCE); \
__ftrace_bad_type(); \
} while (0)
diff --git a/kernel/trace/trace_skb_sources.c b/kernel/trace/trace_skb_sources.c
new file mode 100644
index 0000000..4ba3671
--- /dev/null
+++ b/kernel/trace/trace_skb_sources.c
@@ -0,0 +1,154 @@
+/*
+ * ring buffer based tracer for analyzing per-socket skb sources
+ *
+ * Neil Horman <nhorman@tuxdriver.com>
+ * Copyright (C) 2009
+ *
+ *
+ */
+
+#include <linux/init.h>
+#include <linux/debugfs.h>
+#include <trace/events/skb.h>
+#include <linux/kallsyms.h>
+#include <linux/module.h>
+#include <linux/hardirq.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+
+#include "trace.h"
+#include "trace_output.h"
+
+EXPORT_TRACEPOINT_SYMBOL_GPL(skb_copy_datagram_iovec);
+
+static struct trace_array *skb_trace;
+static int __read_mostly trace_skb_source_enabled;
+
+static void probe_skb_dequeue(const struct sk_buff *skb, int len)
+{
+ struct ring_buffer_event *event;
+ struct trace_skb_event *entry;
+ struct trace_array *tr = skb_trace;
+ struct net_device *dev;
+
+ if (!trace_skb_source_enabled)
+ return;
+
+ if (in_interrupt())
+ return;
+
+ event = trace_buffer_lock_reserve(tr, TRACE_SKB_SOURCE,
+ sizeof(*entry), 0, 0);
+ if (!event)
+ return;
+ entry = ring_buffer_event_data(event);
+
+ entry->event_data.pid = current->pid;
+ entry->event_data.anid = page_to_nid(virt_to_page(skb->data));
+ entry->event_data.cnid = cpu_to_node(smp_processor_id());
+ entry->event_data.len = len;
+ entry->event_data.rx_queue = skb->queue_mapping;
+ entry->event_data.ccpu = smp_processor_id();
+
+ dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
+ if (dev) {
+ memcpy(entry->event_data.ifname, dev->name, IFNAMSIZ);
+ dev_put(dev);
+ } else {
+ strcpy(entry->event_data.ifname, "Unknown");
+ }
+
+ trace_buffer_unlock_commit(tr, event, 0, 0);
+}
+
+static int tracing_skb_source_register(void)
+{
+ int ret;
+
+ ret = register_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
+ if (ret)
+ pr_info("skb source trace: Couldn't activate dequeue tracepoint");
+
+ return ret;
+}
+
+static void start_skb_source_trace(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 1;
+}
+
+static void stop_skb_source_trace(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 0;
+}
+
+static void skb_source_trace_reset(struct trace_array *tr)
+{
+ trace_skb_source_enabled = 0;
+ unregister_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
+}
+
+
+static int skb_source_trace_init(struct trace_array *tr)
+{
+ int cpu;
+ skb_trace = tr;
+
+ trace_skb_source_enabled = 1;
+ tracing_skb_source_register();
+
+ for_each_cpu(cpu, cpu_possible_mask)
+ tracing_reset(tr, cpu);
+ return 0;
+}
+
+static enum print_line_t skb_source_print_line(struct trace_iterator *iter)
+{
+ int ret = 0;
+ struct trace_entry *entry = iter->ent;
+ struct trace_skb_event *event;
+ struct skb_record *record;
+ struct trace_seq *s = &iter->seq;
+
+ trace_assign_type(event, entry);
+ record = &event->event_data;
+ if (entry->type != TRACE_SKB_SOURCE)
+ return TRACE_TYPE_UNHANDLED;
+
+ ret = trace_seq_printf(s, " %d %d %d %s %d %d %d\n",
+ record->pid,
+ record->anid,
+ record->cnid,
+ record->ifname,
+ record->rx_queue,
+ record->ccpu,
+ record->len);
+
+ if (!ret)
+ return TRACE_TYPE_PARTIAL_LINE;
+
+ return TRACE_TYPE_HANDLED;
+}
+
+static void skb_source_print_header(struct seq_file *s)
+{
+ seq_puts(s, "# PID ANID CNID IFC RXQ CCPU LEN\n");
+ seq_puts(s, "# | | | | | | |\n");
+}
+
+static struct tracer skb_source_tracer __read_mostly =
+{
+ .name = "skb_sources",
+ .init = skb_source_trace_init,
+ .start = start_skb_source_trace,
+ .stop = stop_skb_source_trace,
+ .reset = skb_source_trace_reset,
+ .print_line = skb_source_print_line,
+ .print_header = skb_source_print_header,
+};
+
+static int init_skb_source_trace(void)
+{
+ return register_tracer(&skb_source_tracer);
+}
+device_initcall(init_skb_source_trace);
diff --git a/net/core/datagram.c b/net/core/datagram.c
index b0fe692..1c6cf3a 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -55,6 +55,7 @@
#include <net/checksum.h>
#include <net/sock.h>
#include <net/tcp_states.h>
+#include <trace/events/skb.h>
/*
* Is a socket 'connection oriented' ?
@@ -284,6 +285,8 @@ int skb_copy_datagram_iovec(const struct sk_buff *skb, int offset,
int i, copy = start - offset;
struct sk_buff *frag_iter;
+ trace_skb_copy_datagram_iovec(skb, len);
+
/* Copy header. */
if (copy > 0) {
if (copy > len)
^ permalink raw reply related
* RE: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Fischer, Anna @ 2009-08-07 21:00 UTC (permalink / raw)
To: Yaron Haviv, evb@yahoogroups.com, shemminger@linux-foundation.org
Cc: bridge@lists.linux-foundation.org, netdev@vger.kernel.org,
virtualization@lists.linux-foundation.org, davem@davemloft.net,
kaber@trash.net, adobriyan@gmail.com, arnd@arndb.de,
Paul Congdon (UC Davis)
In-Reply-To: <D2E118A1E7268F449C469EA2F301D92902BC39B2@exil.voltaire.com>
Hi Yaron,
Yes, I also believe that VEPA + SRIOV can potentially, in some deployments, achieve better performance than a bridge/tap configuration, especially when you run multiple VMs and if you want to enable more sophisticated network processing in the data path.
If you do have a SRIOV NIC that supports VEPA, then I would think that you do not have QEMU or macvtap in the setup any more though. Simply because in that case the VM can directly access the VF on the physical device. That would be ideal.
I do think that the macvtap driver is a good addition as a simple and fast virtual network I/O interface, in case you do not need full bridge functionality. It does seem to assume though that the virtualization software uses QEMU/tap interfaces. How would this work with a Xen para-virtualized network interface? I guess there would need to be yet another driver?
Anna
--
From: Yaron Haviv [mailto:yaronh@voltaire.com]
Sent: 07 August 2009 21:36
To: evb@yahoogroups.com; shemminger@linux-foundation.org; Fischer, Anna
Cc: bridge@lists.linux-foundation.org; netdev@vger.kernel.org; virtualization@lists.linux-foundation.org; davem@davemloft.net; kaber@trash.net; adobriyan@gmail.com; arnd@arndb.de
Subject: Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
Paul,
I also think that bridge may not be the right place for VEPA, but rather a simpler sw/hw mux
Although the VEPA support may reside in multiple places (I.e. also in the bridge)
As Arnd pointed out Or already added an extension to qemu that allow direct guest virtual NIC mapping to an interface device (vs using tap), this was done specifically to address VEPA, and result in much faster performance and lower cpu overhead (Or and some others are planning additional meaningful performance optimizations)
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
It may be counter intuitive for some, but we expect the (completed) qemu VEPA mode + SR-IOV + certain switches with hairpin (vepa) mode to perform faster than using bridge+tap even for connecting 2 VMs on the same host
Yaron
Sent from BlackBerry
________________________________________
From: evb@yahoogroups.com
To: 'Stephen Hemminger' ; 'Fischer, Anna'
Cc: bridge@lists.linux-foundation.org ; linux-kernel@vger.kernel.org ; netdev@vger.kernel.org ; virtualization@lists.linux-foundation.org ; evb@yahoogroups.com ; davem@davemloft.net ; kaber@trash.net ; adobriyan@gmail.com ; 'Arnd Bergmann'
Sent: Fri Aug 07 21:58:00 2009
Subject: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
>
> After reading more about this, I am not convinced this should be part
> of the bridge code. The bridge code really consists of two parts:
> forwarding table and optional spanning tree. Well the VEPA code short
> circuits both of these; it can't imagine it working with STP turned
> on. The only part of bridge code that really gets used by this are the
> receive packet hooks and the crufty old API.
>
> So instead of adding more stuff to existing bridge code, why not have
> a new driver for just VEPA. You could do it with a simple version of
> macvlan type driver.
Stephen,
Thanks for your comments and questions. We do believe the bridge code is
the right place for this, so I'd like to embellish on that a bit more to
help persuade you. Sorry for the long winded response, but here are some
thoughts:
- First and foremost, VEPA is going to be a standard addition to the IEEE
802.1Q specification. The working group agreed at the last meeting to
pursue a project to augment the bridge standard with hairpin mode (aka
reflective relay) and a remote filtering service (VEPA). See for details:
http://www.ieee802.org/1/files/public/docs2009/new-evb-congdon-evbPar5C-0709
-v01.pdf
- The VEPA functionality was really a pretty small change to the code with
low risk and wouldn't seem to warrant an entire new driver or module.
- There are good use cases where VMs will want to have some of their
interfaces attached to bridges and others to bridges operating in VEPA mode.
In other words, we see simultaneous operation of the bridge code and VEPA
occurring, so having as much of the underlying code as common as possible
would seem to be beneficial.
- By augmenting the bridge code with VEPA there is a great amount of re-use
achieved. It works wherever the bridge code works and doesn't need anything
special to support KVM, XEN, and all the hooks, etc...
- The hardware vendors building SR-IOV NICs with embedded switches will be
adding VEPA mode, so by keeping the bridge module in sync would be
consistent with this trend and direction. It will be possible to extend the
hardware implementations by cascading a software bridge and/or VEPA, so
being in sync with the architecture would make this more consistent.
- The forwarding table is still needed and used on inbound traffic to
deliver frames to the correct virtual interfaces and to filter any reflected
frames. A new driver would have to basically implement an equivalent
forwarding table anyway. As I understand the current macvlan type driver,
it wouldn't filter multicast frames properly without such a table.
- It seems the hairpin mode would be needed in the bridge module whether
VEPA was added to the bridge module or a new driver. Having the associated
changes together in the same code could aid in understanding and deployment.
As I understand the macvlan code, it currently doesn't allow two VMs on the
same machine to communicate with one another. I could imagine a hairpin
mode on the adjacent bridge making this possible, but the macvlan code would
need to be updated to filter reflected frames so a source did not receive
his own packet. I could imagine this being done as well, but to also
support selective multicast usage, something similar to the bridge
forwarding table would be needed. I think putting VEPA into a new driver
would cause you to implement many things the bridge code already supports.
Given that we expect the bridge standard to ultimately include VEPA, and the
new functions are basic forwarding operations, it seems to make most sense
to keep this consistent with the bridge module.
Paul
^ permalink raw reply
* Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 21:06 UTC (permalink / raw)
To: Linux Network Developers; +Cc: brice, gallatin
I've run into a major receive side performance issue with multi-10-GigE
on a NUMA system. The system is using a SuperMicro X8DAH+-F motherboard
with 2 3.2 GHz quad-core Intel Xeon 5580 processors and 12 GB of
1333 MHz DDR3 memory. It is a Fedora 10 system but using the latest
2.6.29.6 kernel from Fedora 11 (originally tried the 2.6.27.29 kernel
from Fedora 10).
The test setup is:
i7test1----(6)----xeontest1----(6)----i7test2
10-GigE 10-GigE
So xeontest1 has 6 dual-port Myricom 10-GigE NICs for a total
of 12 10-GigE interfaces. eth2 through eth7 (which are on the
second Intel 5520 I/O Hub) are connected to i7test1 while
eth8 through eth13 (which are on the first Intel 5520 I/O Hub)
are connected to i7test2.
Previous direct testing between i7test1 and i7test2 (which use an
Asus P6T6 WS Revolution motherboard) demonstrated that they could
achieve ~70 Gbps performance for either transmit or receive using
8 10-GigE interfaces.
The transmit side performance of xeontest1 is fantastic:
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -xc6/3 -p5012 192.168.12.11 &
n12: 9648.0522 MB / 10.00 sec = 8091.4066 Mbps 49 %TX 26 %RX 0 retrans 0.18 msRTT
n9: 11130.5320 MB / 10.01 sec = 9328.3224 Mbps 47 %TX 37 %RX 0 retrans 0.19 msRTT
n11: 9418.1250 MB / 10.00 sec = 7897.5848 Mbps 50 %TX 30 %RX 0 retrans 0.18 msRTT
n10: 9279.4758 MB / 10.01 sec = 7778.7146 Mbps 49 %TX 28 %RX 0 retrans 0.12 msRTT
n8: 11142.6574 MB / 10.01 sec = 9340.3789 Mbps 47 %TX 35 %RX 0 retrans 0.18 msRTT
n13: 9422.1492 MB / 10.01 sec = 7897.4115 Mbps 49 %TX 25 %RX 0 retrans 0.17 msRTT
n3: 11471.2500 MB / 10.01 sec = 9613.9477 Mbps 49 %TX 32 %RX 0 retrans 0.15 msRTT
n6: 9339.6354 MB / 10.01 sec = 7828.5345 Mbps 50 %TX 25 %RX 0 retrans 0.19 msRTT
n4: 9093.2500 MB / 10.01 sec = 7624.1589 Mbps 49 %TX 28 %RX 0 retrans 0.15 msRTT
n5: 9121.8367 MB / 10.01 sec = 7646.8646 Mbps 50 %TX 29 %RX 0 retrans 0.17 msRTT
n7: 9292.2500 MB / 10.01 sec = 7789.1574 Mbps 49 %TX 26 %RX 0 retrans 0.17 msRTT
n2: 11487.1150 MB / 10.01 sec = 9627.2690 Mbps 49 %TX 46 %RX 0 retrans 0.19 msRTT
Aggregate performance: 100.4637 Gbps
The problem is with the receive side performance.
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n11: 6983.6359 MB / 10.09 sec = 5803.2293 Mbps 13 %TX 26 %RX 0 retrans 0.11 msRTT
n10: 7000.1557 MB / 10.11 sec = 5807.5978 Mbps 13 %TX 26 %RX 0 retrans 0.12 msRTT
n9: 2451.7206 MB / 10.21 sec = 2014.8397 Mbps 4 %TX 13 %RX 0 retrans 0.11 msRTT
n13: 2453.0887 MB / 10.20 sec = 2016.8751 Mbps 3 %TX 11 %RX 0 retrans 0.10 msRTT
n12: 2446.5303 MB / 10.24 sec = 2004.4638 Mbps 4 %TX 11 %RX 0 retrans 0.10 msRTT
n8: 2462.5890 MB / 10.26 sec = 2014.0272 Mbps 3 %TX 11 %RX 0 retrans 0.12 msRTT
n4: 2763.5091 MB / 10.26 sec = 2258.4871 Mbps 4 %TX 14 %RX 0 retrans 0.10 msRTT
n5: 2770.0887 MB / 10.28 sec = 2261.2562 Mbps 4 %TX 15 %RX 0 retrans 0.10 msRTT
n2: 1777.7277 MB / 10.32 sec = 1444.9054 Mbps 2 %TX 11 %RX 0 retrans 0.11 msRTT
n6: 1772.7962 MB / 10.31 sec = 1442.0346 Mbps 3 %TX 10 %RX 0 retrans 0.11 msRTT
n3: 1779.4535 MB / 10.32 sec = 1446.0090 Mbps 2 %TX 11 %RX 0 retrans 0.15 msRTT
n7: 1770.8359 MB / 10.35 sec = 1435.4757 Mbps 2 %TX 11 %RX 0 retrans 0.12 msRTT
Aggregate performance: 29.9492 Gbps
I suspected that this was because the memory being allocated by the
myri10ge driver was not being allocated on the optimum NUMA node.
BTW the NUMA nodes on the system are 0 and 2 instead of 0 and 1 which
is what I would have expected, but this is my first experience with
a NUMA system.
Based upon a patch by Peter Zijlstra that I discovered through Google
searching, I tried patching the myri10ge driver to change its memory
allocation of memory pages from alloc_pages() to alloc_pages_node()
and specifying the NUMA node of the parent device of the Myricom 10-GigE
device, which IIUC should be the PCIe switch. This didn't help.
This could be because I discovered that if I did:
find /sys -name numa_node -exec grep . {} /dev/null \;
that the numa_node associated with all the PCI devices was always 0,
and if IIUC then I believe some of the PCI devices should have been
associated with NUMA node 2. Perhaps this is what is causing all
the memory pages allocated by the myri10ge driver to be on NUMA
node 0, and thus causing the major performance issue.
To kludge around this, I made a different patch to the myri10ge driver.
This time I hardcoded the NUMA node in the call to alloc_pages_node()
to 2 for devices with an IRQ between 113 and 118 (eth2 through eth7)
and to 0 for devices with an IRQ between 119 and 124 (eth8 through eth13).
This is of course very specific to our specific system (NUMA node ids
and Myricom 10-GigE device IRQs), and is not something that would be
generically applicable. But it was useful as a test, and it did
improve the receive side performance substantially!
[root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
n5: 8221.2911 MB / 10.09 sec = 6836.0343 Mbps 17 %TX 31 %RX 0 retrans 0.12 msRTT
n4: 8237.9524 MB / 10.10 sec = 6840.2379 Mbps 16 %TX 31 %RX 0 retrans 0.11 msRTT
n11: 7935.3750 MB / 10.11 sec = 6586.2476 Mbps 15 %TX 29 %RX 0 retrans 0.16 msRTT
n2: 4543.1621 MB / 10.13 sec = 3763.0669 Mbps 9 %TX 21 %RX 0 retrans 0.12 msRTT
n10: 7916.3925 MB / 10.13 sec = 6555.5210 Mbps 15 %TX 28 %RX 0 retrans 0.13 msRTT
n7: 4558.4817 MB / 10.14 sec = 3771.6557 Mbps 7 %TX 22 %RX 0 retrans 0.10 msRTT
n13: 4390.1875 MB / 10.14 sec = 3633.6421 Mbps 6 %TX 21 %RX 0 retrans 0.12 msRTT
n3: 4572.6478 MB / 10.15 sec = 3778.2596 Mbps 9 %TX 21 %RX 0 retrans 0.14 msRTT
n6: 4564.4776 MB / 10.14 sec = 3774.4373 Mbps 9 %TX 21 %RX 0 retrans 0.11 msRTT
n8: 4409.8551 MB / 10.16 sec = 3642.1920 Mbps 8 %TX 19 %RX 0 retrans 0.12 msRTT
n9: 4412.7836 MB / 10.16 sec = 3643.7788 Mbps 8 %TX 20 %RX 0 retrans 0.14 msRTT
n12: 4413.4061 MB / 10.16 sec = 3645.2544 Mbps 8 %TX 21 %RX 0 retrans 0.11 msRTT
Aggregate performance: 56.4703 Gbps
This was basically double the previous receive side performance
without the patch.
I don't know if this is fundamentally a myri10ge driver issue or
some underlying Linux kernel issue, so it's not clear to me what
a proper fix would be.
Finally, while definitely a major improvement, I think it should be
possible to do even better, since we achieved 70 Gbps in the i7 to i7
tests, and probably could have done 80 Gbps except for an Asus
motherboard restriction with the interconnect between the Intel X58
and Nvidia NF200 chips. It's definitely a big step in the right
direction though if this issue can be resolved.
Any help greatly appreicated in advance.
-Thanks
-Bill
^ permalink raw reply
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Paul Congdon (UC Davis) @ 2009-08-07 21:06 UTC (permalink / raw)
To: shemminger, anna.fischer
Cc: arnd, davem, netdev, bridge, adobriyan, virtualization
In-Reply-To: <D2E118A1E7268F449C469EA2F301D92902BC39B2@exil.voltaire.com>
[-- Attachment #1.1: Type: text/plain, Size: 1122 bytes --]
Yaron,
The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
Agreed that the hardware solution is preferred so the macvlan implementation doesn’t really matter. If we are talking SR-IOV, then it is direct mapped, regardless of whether there is a VEB or VEPA in the hardware below, so you are bypassing the bridge software code also.
I disagree that adding the multicast handling is simple – while not conceptually hard, it will basically require you to put an address table into the macvlan implementation – if you have that, then why not have just used the one already in the bridge code. If you hook a VEPA up to a non-hairpin mode external bridge, you get the macvlan capability as well.
It also seems to me like the special macvlan interfaces for KVM don’t apply to XEN or a non-virtualized environment? Or more has to be written to make that work? If it is in the bridge code, you get all of this re-use.
[-- Attachment #1.2: Type: text/html, Size: 6831 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge
^ permalink raw reply
* [PATCH net-next] myri10ge: improve parity error detection and recovery
From: Brice Goglin @ 2009-08-07 20:44 UTC (permalink / raw)
To: David S. Miller; +Cc: Linux Network Development list
Improve myri10ge parity error detection and recovery:
1) Don't restore PCI config space to a rebooted NIC until AFTER the
host is quiescent.
2) Let myri10ge_close() know the NIC is dead, so it won't waste time
waiting for a dead nic to respond to MXGEFW_CMD_ETHERNET_DOWN
3) When the NIC is quiet (link down, or otherwise idle link) use
a pci config space read to detect a rebooted NIC. Otherwise
we might never notice that a NIC rebooted
Signed-off-by: Andrew Gallatin <gallatin@myri.com>
Signed-off-by: Brice Goglin <brice@myri.com>
diff -ur /home/bgoglin/src/git/net-next-2.6/drivers/net/myri10ge/myri10ge.c linux-tmp/drivers/net/myri10ge/myri10ge.c
--- net-next-2.6/drivers/net/myri10ge/myri10ge.c 2009-07-22 14:39:45.000000000 +0200
+++ linux-tmp/drivers/net/myri10ge/myri10ge.c 2009-08-07 22:30:05.000000000 +0200
@@ -75,7 +75,7 @@
#include "myri10ge_mcp.h"
#include "myri10ge_mcp_gen_header.h"
-#define MYRI10GE_VERSION_STR "1.5.0-1.418"
+#define MYRI10GE_VERSION_STR "1.5.0-1.432"
MODULE_DESCRIPTION("Myricom 10G driver (10GbE)");
MODULE_AUTHOR("Maintainer: help@myri.com");
@@ -188,6 +188,7 @@
dma_addr_t fw_stats_bus;
int watchdog_tx_done;
int watchdog_tx_req;
+ int watchdog_rx_done;
#ifdef CONFIG_MYRI10GE_DCA
int cached_dca_tag;
int cpu;
@@ -256,6 +257,7 @@
u32 link_changes;
u32 msg_enable;
unsigned int board_number;
+ int rebooted;
};
static char *myri10ge_fw_unaligned = "myri10ge_ethp_z8e.dat";
@@ -2552,17 +2554,22 @@
netif_carrier_off(dev);
netif_tx_stop_all_queues(dev);
- old_down_cnt = mgp->down_cnt;
- mb();
- status = myri10ge_send_cmd(mgp, MXGEFW_CMD_ETHERNET_DOWN, &cmd, 0);
- if (status)
- printk(KERN_ERR "myri10ge: %s: Couldn't bring down link\n",
- dev->name);
-
- wait_event_timeout(mgp->down_wq, old_down_cnt != mgp->down_cnt, HZ);
- if (old_down_cnt == mgp->down_cnt)
- printk(KERN_ERR "myri10ge: %s never got down irq\n", dev->name);
+ if (mgp->rebooted == 0) {
+ old_down_cnt = mgp->down_cnt;
+ mb();
+ status =
+ myri10ge_send_cmd(mgp, MXGEFW_CMD_ETHERNET_DOWN, &cmd, 0);
+ if (status)
+ printk(KERN_ERR
+ "myri10ge: %s: Couldn't bring down link\n",
+ dev->name);
+ wait_event_timeout(mgp->down_wq, old_down_cnt != mgp->down_cnt,
+ HZ);
+ if (old_down_cnt == mgp->down_cnt)
+ printk(KERN_ERR "myri10ge: %s never got down irq\n",
+ dev->name);
+ }
netif_tx_disable(dev);
myri10ge_free_irq(mgp);
for (i = 0; i < mgp->num_slices; i++)
@@ -3427,12 +3434,13 @@
container_of(work, struct myri10ge_priv, watchdog_work);
struct myri10ge_tx_buf *tx;
u32 reboot;
- int status;
+ int status, rebooted;
int i;
u16 cmd, vendor;
mgp->watchdog_resets++;
pci_read_config_word(mgp->pdev, PCI_COMMAND, &cmd);
+ rebooted = 0;
if ((cmd & PCI_COMMAND_MASTER) == 0) {
/* Bus master DMA disabled? Check to see
* if the card rebooted due to a parity error
@@ -3444,9 +3452,12 @@
myri10ge_reset_recover ? " " : " not");
if (myri10ge_reset_recover == 0)
return;
-
+ rtnl_lock();
+ mgp->rebooted = 1;
+ rebooted = 1;
+ myri10ge_close(mgp->dev);
myri10ge_reset_recover--;
-
+ mgp->rebooted = 0;
/*
* A rebooted nic will come back with config space as
* it was after power was applied to PCIe bus.
@@ -3494,8 +3505,10 @@
}
}
- rtnl_lock();
- myri10ge_close(mgp->dev);
+ if (!rebooted) {
+ rtnl_lock();
+ myri10ge_close(mgp->dev);
+ }
status = myri10ge_load_firmware(mgp, 1);
if (status != 0)
printk(KERN_ERR "myri10ge: %s: failed to load firmware\n",
@@ -3516,12 +3529,14 @@
{
struct myri10ge_priv *mgp;
struct myri10ge_slice_state *ss;
- int i, reset_needed;
+ int i, reset_needed, busy_slice_cnt;
u32 rx_pause_cnt;
+ u16 cmd;
mgp = (struct myri10ge_priv *)arg;
rx_pause_cnt = ntohl(mgp->ss[0].fw_stats->dropped_pause);
+ busy_slice_cnt = 0;
for (i = 0, reset_needed = 0;
i < mgp->num_slices && reset_needed == 0; ++i) {
@@ -3559,8 +3574,22 @@
reset_needed = 1;
}
}
+ if (ss->watchdog_tx_done != ss->tx.done ||
+ ss->watchdog_rx_done != ss->rx_done.cnt) {
+ busy_slice_cnt++;
+ }
ss->watchdog_tx_done = ss->tx.done;
ss->watchdog_tx_req = ss->tx.req;
+ ss->watchdog_rx_done = ss->rx_done.cnt;
+ }
+ /* if we've sent or received no traffic, poll the NIC to
+ * ensure it is still there. Otherwise, we risk not noticing
+ * an error in a timely fashion */
+ if (busy_slice_cnt == 0) {
+ pci_read_config_word(mgp->pdev, PCI_COMMAND, &cmd);
+ if ((cmd & PCI_COMMAND_MASTER) == 0) {
+ reset_needed = 1;
+ }
}
mgp->watchdog_pause = rx_pause_cnt;
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 21:18 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807170600.9a2eff2e.billfink@mindspring.com>
Bill Fink wrote:
> This could be because I discovered that if I did:
>
> find /sys -name numa_node -exec grep . {} /dev/null \;
>
> that the numa_node associated with all the PCI devices was always 0,
> and if IIUC then I believe some of the PCI devices should have been
> associated with NUMA node 2. Perhaps this is what is causing all
> the memory pages allocated by the myri10ge driver to be on NUMA
> node 0, and thus causing the major performance issue.
>
I've seen some cases in the past where numa_node was always 0 on
quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
later kernels thanks to patches from Yinghai Lu (CC'ed).
Is the corresponding local_cpus sysfs file wrong as well ?
Maybe your kernel doesn't properly handle the NUMA location of PCI
devices on Nehalem machines yet?
Brice
^ permalink raw reply
* Re: [evb] RE: [PATCH][RFC] net/bridge: add basic VEPA support
From: Stephen Hemminger @ 2009-08-07 21:36 UTC (permalink / raw)
To: Paul Congdon (UC Davis)
Cc: anna.fischer, bridge, netdev, virtualization, davem, kaber,
adobriyan, arnd
In-Reply-To: <000001ca17a3$03f841f0$0be8c5d0$@edu>
On Fri, 7 Aug 2009 14:06:58 -0700
"Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
> Yaron,
>
>
> The interface multiplexing can be achieved using macvlan driver or using an SR-IOV capable NIC (the preferred option), macvlan may need to be extended to support VEPA multicast handling, this looks like a rather simple task
>
> Agreed that the hardware solution is preferred so the macvlan implementation doesn’t really matter. If we are talking SR-IOV, then it is direct mapped, regardless of whether there is a VEB or VEPA in the hardware below, so you are bypassing the bridge software code also.
>
> I disagree that adding the multicast handling is simple – while not conceptually hard, it will basically require you to put an address table into the macvlan implementation – if you have that, then why not have just used the one already in the bridge code. If you hook a VEPA up to a non-hairpin mode external bridge, you get the macvlan capability as well.
I have a patch that forwards all multicast packets, and another that does
proper forwarding. It should have worked that way in original macvlan, the
current behavior is really a bug.
--
^ permalink raw reply
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-08-07 21:38 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Paul Congdon (UC Davis), drobbins, 'Fischer, Anna',
herbert, mst, netdev, bridge, linux-kernel, ogerlitz, evb, davem
In-Reply-To: <20090807123554.7c2bc27c@nehalam>
On Friday 07 August 2009, Stephen Hemminger wrote:
> On Fri, 7 Aug 2009 12:10:07 -0700
> "Paul Congdon \(UC Davis\)" <ptcongdon@ucdavis.edu> wrote:
>
> > Responding to Daniel's questions...
> >
> > > I have some general questions about the intended use and benefits of
> > > VEPA, from an IT perspective:
> > >
> > > In which virtual machine setups and technologies do you forsee this
> > > interface being used?
> >
> > The benefit of VEPA is the coordination and unification with the
> > external network switch. So, in environments where you are
> > needing/wanting your feature rich, wire speed, external network
> > device (firewall/switch/IPS/content-filter) to provide consistent
> > policy enforcement, and you want your VMs traffic to be subject to
> > that enforcement, you will want their traffic directed externally.
> > Perhaps you have some VMs that are on a DMZ or clustering an
> > application or implementing a multi-tier application where you
> > would normally place a firewall in-between the tiers.
>
> I do have to raise the point that Linux is perfectly capable of keeping up without
> the need of an external switch. Whether you want policy external or internal is
> a architecture decision that should not be driven by mis-information about performance.
In general, I agree that Linux on a decent virtual machine host will be
able to handle forwarding of network data fast enough, often faster than
the external connectivity allows if it needs to transmit every frame twice.
However, there is a tradeoff between CPU cycles and I/O bandwidth. If your
application needs lots of CPU but you have spare capacity on the PCI bus, the
network wire and the external switch, VEPA can also be a win on the performance
side. As always, performance depends on the application, even if it's not the
main driving factor here.
Arnd <><
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 21:51 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7C9A14.7070600@inria.fr>
On Fri, 07 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> > This could be because I discovered that if I did:
> >
> > find /sys -name numa_node -exec grep . {} /dev/null \;
> >
> > that the numa_node associated with all the PCI devices was always 0,
> > and if IIUC then I believe some of the PCI devices should have been
> > associated with NUMA node 2. Perhaps this is what is causing all
> > the memory pages allocated by the myri10ge driver to be on NUMA
> > node 0, and thus causing the major performance issue.
> >
>
> I've seen some cases in the past where numa_node was always 0 on
> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
> later kernels thanks to patches from Yinghai Lu (CC'ed).
By later kernels do you mean 2.6.30 or 2.6.31?
> Is the corresponding local_cpus sysfs file wrong as well ?
All sysfs local_cpus values are the same (00000000,000000ff),
so yes they are also wrong.
> Maybe your kernel doesn't properly handle the NUMA location of PCI
> devices on Nehalem machines yet?
I assume so, unless there's some secret NUMA system setting that
I'm unaware of that would affect this and needs changing for my
setup.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 21:53 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807175112.a1f57407.billfink@mindspring.com>
Bill Fink wrote:
>> I've seen some cases in the past where numa_node was always 0 on
>> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
>> later kernels thanks to patches from Yinghai Lu (CC'ed).
>>
>
> By later kernels do you mean 2.6.30 or 2.6.31?
>
No, I meant "later than when the problem occured". I was using 2.6.22 at
this point and the problem was fixed somewhere around 2.6.25.
>> Is the corresponding local_cpus sysfs file wrong as well ?
>>
>
> All sysfs local_cpus values are the same (00000000,000000ff),
> so yes they are also wrong.
>
And hyperthreading is enabled, right?
Brice
^ permalink raw reply
* Re: [Bridge] [PATCH] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-08-07 22:05 UTC (permalink / raw)
To: Paul Congdon (UC Davis)
Cc: drobbins, 'Fischer, Anna', herbert, mst, netdev, bridge,
linux-kernel, ogerlitz, evb, davem
In-Reply-To: <004f01ca1792$b24a7a90$16df6fb0$@edu>
On Friday 07 August 2009, Paul Congdon (UC Davis) wrote:
> Responding to Daniel's questions...
Thanks for the detailed responses. I'll add some more about the
specifics of the macvlan implementation that differs from the
bridge based VEPA implementation.
> > Is this new interface to be used within a virtual machine or
> > container, on the master node, or both?
>
> It is really an interface to a new type of virtual switch. When
> you create virtual network, I would imagine it being a new mode
> of operation (bridge, NAT, VEPA, etc).
I think the question was whether the patch needs to applied in the
host or the guest. Both the implementation that you and Anna did
and the one that I posted only apply to the *host* (master node),
the virtual machine does not need to know about it.
> > What interface(s) would need to be configured for a single virtual
> > machine to use VEPA to access the network?
>
> It would be the same as if that machine were configure to use a
> bridge to access the network, but the bridge mode would be different.
Right, with the bridge based VEPA, you would set up a kvm guest
or a container with the regular tools, then use the sysfs interface
to put the bridge device into VEPA mode.
With the macvlan based mode, you use 'ip link' to add a new tap
device to an external network interface and not use a bridge at
all. Then you configure KVM to use that tap device instead of the
regular bridge/tap setup.
> > What are the current flexibility, security or performance limitations
> > of tun/tap and bridge that make this new interface necessary or
> > beneficial?
>
> If you have VMs that will be communicating with one another on
> the same physical machine, and you want their traffic to be
> exposed to an in-line network device such as a application
> firewall/IPS/content-filter (without this feature) you will have
> to have this device co-located within the same physical server.
> This will use up CPU cycles that you presumable purchased to run
> applications, it will require a lot of consistent configuration
> on all physical machines, it could invoke potentially a lot of
> software licensing, additional cost, etc.. Everything would
> need to be replicated on each physical machine. With the VEPA
> capability, you can leverage all this functionality in an
> external network device and have it managed and configured in
> one place. The external implementation is likely a higher
> performance, silicon based implementation. It should make it
> easier to migrate machines from one physical server to another
> and maintain the same network policy enforcement.
It's worth noting that depending on your network connectivity,
performance is likely to go down significantly with VEPA over the
existing bridge/tap setup, because all frames have to be sent
twice through an external wire that has a limited capacity, so you may
lose inter-guest bandwidth and get more latency in many cases, while
you free up CPU cycles. With the bridge based VEPA, you might not
even gain many cycles because much of the overhead is still there.
On the cost side, external switches can also get quite expensive
compared to x86 servers.
IMHO the real win of VEPA is on the management side, where you can
use a single set of tool for managing the network, rather than
having your network admins deal with both the external switches
and the setup of linux netfilter rules etc.
The macvlan based VEPA has the same features as the bridge based
VEPA, but much simpler code, which allows a number of shortcuts
to save CPU cycles.
> The isolation in the outbound direction is created by the way frames
> are forwarded. They are simply dropped on the wire, so no VMs can
> talk directly to one another without their traffic first going
> external. In the inbound direction, the isolation is created using
> the forwarding table.
Right. Note that in the macvlan case, the filtering on inbound data is an inherent
part of the macvlan setup, it does use the dynamic forwarding table of the
bridge driver.
> > Are there any associated user-space tools required for configuring a
> > VEPA?
> >
>
> The standard brctl utility has been augmented to enable/disable the capability.
That is for the bridge based VEPA, while my patch uses the 'ip link'
command that ships with most distros. It does not need any modifications
right now, but might need them if we add other features like support for
multiple MAC addresses in a single guest.
Arnd <><
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 22:08 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7CA24E.4080503@inria.fr>
On Fri, 07 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> >> I've seen some cases in the past where numa_node was always 0 on
> >> quad-Opteron machines with a PCI bus on node 1. IIRC it got fixed in
> >> later kernels thanks to patches from Yinghai Lu (CC'ed).
> >
> > By later kernels do you mean 2.6.30 or 2.6.31?
>
> No, I meant "later than when the problem occured". I was using 2.6.22 at
> this point and the problem was fixed somewhere around 2.6.25.
OK. The tests were run on a 2.6.29.6 kernel so presumably should
have included the fix you mentioned.
> >> Is the corresponding local_cpus sysfs file wrong as well ?
> >
> > All sysfs local_cpus values are the same (00000000,000000ff),
> > so yes they are also wrong.
>
> And hyperthreading is enabled, right?
No, hyperthreading is disabled. It's a dual quad-core system so there
are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-07 22:12 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090807170600.9a2eff2e.billfink@mindspring.com>
On Fri, Aug 07, 2009 at 05:06:00PM -0400, Bill Fink wrote:
> I've run into a major receive side performance issue with multi-10-GigE
> on a NUMA system. The system is using a SuperMicro X8DAH+-F motherboard
> with 2 3.2 GHz quad-core Intel Xeon 5580 processors and 12 GB of
> 1333 MHz DDR3 memory. It is a Fedora 10 system but using the latest
> 2.6.29.6 kernel from Fedora 11 (originally tried the 2.6.27.29 kernel
> from Fedora 10).
>
> The test setup is:
>
> i7test1----(6)----xeontest1----(6)----i7test2
> 10-GigE 10-GigE
>
> So xeontest1 has 6 dual-port Myricom 10-GigE NICs for a total
> of 12 10-GigE interfaces. eth2 through eth7 (which are on the
> second Intel 5520 I/O Hub) are connected to i7test1 while
> eth8 through eth13 (which are on the first Intel 5520 I/O Hub)
> are connected to i7test2.
>
> Previous direct testing between i7test1 and i7test2 (which use an
> Asus P6T6 WS Revolution motherboard) demonstrated that they could
> achieve ~70 Gbps performance for either transmit or receive using
> 8 10-GigE interfaces.
>
> The transmit side performance of xeontest1 is fantastic:
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -xc6/3 -p5012 192.168.12.11 &
> n12: 9648.0522 MB / 10.00 sec = 8091.4066 Mbps 49 %TX 26 %RX 0 retrans 0.18 msRTT
> n9: 11130.5320 MB / 10.01 sec = 9328.3224 Mbps 47 %TX 37 %RX 0 retrans 0.19 msRTT
> n11: 9418.1250 MB / 10.00 sec = 7897.5848 Mbps 50 %TX 30 %RX 0 retrans 0.18 msRTT
> n10: 9279.4758 MB / 10.01 sec = 7778.7146 Mbps 49 %TX 28 %RX 0 retrans 0.12 msRTT
> n8: 11142.6574 MB / 10.01 sec = 9340.3789 Mbps 47 %TX 35 %RX 0 retrans 0.18 msRTT
> n13: 9422.1492 MB / 10.01 sec = 7897.4115 Mbps 49 %TX 25 %RX 0 retrans 0.17 msRTT
> n3: 11471.2500 MB / 10.01 sec = 9613.9477 Mbps 49 %TX 32 %RX 0 retrans 0.15 msRTT
> n6: 9339.6354 MB / 10.01 sec = 7828.5345 Mbps 50 %TX 25 %RX 0 retrans 0.19 msRTT
> n4: 9093.2500 MB / 10.01 sec = 7624.1589 Mbps 49 %TX 28 %RX 0 retrans 0.15 msRTT
> n5: 9121.8367 MB / 10.01 sec = 7646.8646 Mbps 50 %TX 29 %RX 0 retrans 0.17 msRTT
> n7: 9292.2500 MB / 10.01 sec = 7789.1574 Mbps 49 %TX 26 %RX 0 retrans 0.17 msRTT
> n2: 11487.1150 MB / 10.01 sec = 9627.2690 Mbps 49 %TX 46 %RX 0 retrans 0.19 msRTT
>
> Aggregate performance: 100.4637 Gbps
>
> The problem is with the receive side performance.
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
> n11: 6983.6359 MB / 10.09 sec = 5803.2293 Mbps 13 %TX 26 %RX 0 retrans 0.11 msRTT
> n10: 7000.1557 MB / 10.11 sec = 5807.5978 Mbps 13 %TX 26 %RX 0 retrans 0.12 msRTT
> n9: 2451.7206 MB / 10.21 sec = 2014.8397 Mbps 4 %TX 13 %RX 0 retrans 0.11 msRTT
> n13: 2453.0887 MB / 10.20 sec = 2016.8751 Mbps 3 %TX 11 %RX 0 retrans 0.10 msRTT
> n12: 2446.5303 MB / 10.24 sec = 2004.4638 Mbps 4 %TX 11 %RX 0 retrans 0.10 msRTT
> n8: 2462.5890 MB / 10.26 sec = 2014.0272 Mbps 3 %TX 11 %RX 0 retrans 0.12 msRTT
> n4: 2763.5091 MB / 10.26 sec = 2258.4871 Mbps 4 %TX 14 %RX 0 retrans 0.10 msRTT
> n5: 2770.0887 MB / 10.28 sec = 2261.2562 Mbps 4 %TX 15 %RX 0 retrans 0.10 msRTT
> n2: 1777.7277 MB / 10.32 sec = 1444.9054 Mbps 2 %TX 11 %RX 0 retrans 0.11 msRTT
> n6: 1772.7962 MB / 10.31 sec = 1442.0346 Mbps 3 %TX 10 %RX 0 retrans 0.11 msRTT
> n3: 1779.4535 MB / 10.32 sec = 1446.0090 Mbps 2 %TX 11 %RX 0 retrans 0.15 msRTT
> n7: 1770.8359 MB / 10.35 sec = 1435.4757 Mbps 2 %TX 11 %RX 0 retrans 0.12 msRTT
>
> Aggregate performance: 29.9492 Gbps
>
> I suspected that this was because the memory being allocated by the
> myri10ge driver was not being allocated on the optimum NUMA node.
> BTW the NUMA nodes on the system are 0 and 2 instead of 0 and 1 which
> is what I would have expected, but this is my first experience with
> a NUMA system.
>
> Based upon a patch by Peter Zijlstra that I discovered through Google
> searching, I tried patching the myri10ge driver to change its memory
> allocation of memory pages from alloc_pages() to alloc_pages_node()
> and specifying the NUMA node of the parent device of the Myricom 10-GigE
> device, which IIUC should be the PCIe switch. This didn't help.
>
> This could be because I discovered that if I did:
>
> find /sys -name numa_node -exec grep . {} /dev/null \;
>
> that the numa_node associated with all the PCI devices was always 0,
> and if IIUC then I believe some of the PCI devices should have been
> associated with NUMA node 2. Perhaps this is what is causing all
> the memory pages allocated by the myri10ge driver to be on NUMA
> node 0, and thus causing the major performance issue.
>
> To kludge around this, I made a different patch to the myri10ge driver.
> This time I hardcoded the NUMA node in the call to alloc_pages_node()
> to 2 for devices with an IRQ between 113 and 118 (eth2 through eth7)
> and to 0 for devices with an IRQ between 119 and 124 (eth8 through eth13).
> This is of course very specific to our specific system (NUMA node ids
> and Myricom 10-GigE device IRQs), and is not something that would be
> generically applicable. But it was useful as a test, and it did
> improve the receive side performance substantially!
>
> [root@xeontest1 ~]# numactl --membind=2 nuttcp -In2 -r -xc1/0 -p5001 192.168.1.10 & numactl --membind=2 nuttcp -In3 -r -xc3/0 -p5002 192.168.2.10 & numactl --membind=2 nuttcp -In4 -r -xc5/1 -p5003 192.168.3.10 & numactl --membind=2 nuttcp -In5 -r -xc7/1 -p5004 192.168.4.10 & nuttcp -In8 -r -xc0/0 -p5007 192.168.7.11 & nuttcp -In9 -r -xc2/0 -p5008 192.168.8.11 & nuttcp -In10 -r -xc4/1 -p5009 192.168.9.11 & nuttcp -In11 -r -xc6/1 -p5010 192.168.10.11 & numactl --membind=2 nuttcp -In6 -r -xc5/2 -p5005 192.168.5.10 & numactl --membind=2 nuttcp -In7 -r -xc7/3 -p5006 192.168.6.10 & nuttcp -In12 -r -xc4/2 -p5011 192.168.11.11 & nuttcp -In13 -r -xc6/3 -p5012 192.168.12.11 &
> n5: 8221.2911 MB / 10.09 sec = 6836.0343 Mbps 17 %TX 31 %RX 0 retrans 0.12 msRTT
> n4: 8237.9524 MB / 10.10 sec = 6840.2379 Mbps 16 %TX 31 %RX 0 retrans 0.11 msRTT
> n11: 7935.3750 MB / 10.11 sec = 6586.2476 Mbps 15 %TX 29 %RX 0 retrans 0.16 msRTT
> n2: 4543.1621 MB / 10.13 sec = 3763.0669 Mbps 9 %TX 21 %RX 0 retrans 0.12 msRTT
> n10: 7916.3925 MB / 10.13 sec = 6555.5210 Mbps 15 %TX 28 %RX 0 retrans 0.13 msRTT
> n7: 4558.4817 MB / 10.14 sec = 3771.6557 Mbps 7 %TX 22 %RX 0 retrans 0.10 msRTT
> n13: 4390.1875 MB / 10.14 sec = 3633.6421 Mbps 6 %TX 21 %RX 0 retrans 0.12 msRTT
> n3: 4572.6478 MB / 10.15 sec = 3778.2596 Mbps 9 %TX 21 %RX 0 retrans 0.14 msRTT
> n6: 4564.4776 MB / 10.14 sec = 3774.4373 Mbps 9 %TX 21 %RX 0 retrans 0.11 msRTT
> n8: 4409.8551 MB / 10.16 sec = 3642.1920 Mbps 8 %TX 19 %RX 0 retrans 0.12 msRTT
> n9: 4412.7836 MB / 10.16 sec = 3643.7788 Mbps 8 %TX 20 %RX 0 retrans 0.14 msRTT
> n12: 4413.4061 MB / 10.16 sec = 3645.2544 Mbps 8 %TX 21 %RX 0 retrans 0.11 msRTT
>
> Aggregate performance: 56.4703 Gbps
>
> This was basically double the previous receive side performance
> without the patch.
>
> I don't know if this is fundamentally a myri10ge driver issue or
> some underlying Linux kernel issue, so it's not clear to me what
> a proper fix would be.
>
> Finally, while definitely a major improvement, I think it should be
> possible to do even better, since we achieved 70 Gbps in the i7 to i7
> tests, and probably could have done 80 Gbps except for an Asus
> motherboard restriction with the interconnect between the Intel X58
> and Nvidia NF200 chips. It's definitely a big step in the right
> direction though if this issue can be resolved.
>
> Any help greatly appreicated in advance.
>
> -Thanks
>
> -Bill
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
You're timing is impeccable! I just posted a patch for an ftrace module to help
detect just these kind of conditions:
http://marc.info/?l=linux-netdev&m=124967650218846&w=2
Hope that helps you out
Neil
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Brice Goglin @ 2009-08-07 22:17 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <20090807180840.b27ce794.billfink@mindspring.com>
Bill Fink wrote:
> OK. The tests were run on a 2.6.29.6 kernel so presumably should
> have included the fix you mentioned.
>
Yes, but I wanted to emphasize that new platforms sometime need some new
code to handle this kind of things. Some Nehalem-specific changes might
be needed now.
>>>> Is the corresponding local_cpus sysfs file wrong as well ?
>>>>
>>> All sysfs local_cpus values are the same (00000000,000000ff),
>>> so yes they are also wrong.
>>>
>> And hyperthreading is enabled, right?
>>
>
> No, hyperthreading is disabled. It's a dual quad-core system so there
> are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
>
So numa_node says that the device is close to node 0 while local_cpus
says that it's close to all 8 cores ie close to both node0 and node2
(which may well be wrong as well).
Brice
^ permalink raw reply
* 2009 Prize Award!!
From: kpairote @ 2009-08-07 21:50 UTC (permalink / raw)
2009 Prize Award!!
Winning/Confirm!!
This email is to notify you that you have won an Award Sum of
€500,000.00 (Five Hundred Thousand Euros)in an E-mail program held in
Den Haag, The Netherlands.
Please contact the claim officer with your winning info. REF
NUM.(NL80246) BATCH NUM.(EU-0375908NL) TICKET NUM.(460205)Name;Mr.Paul
Hanson,TEL.+31-643608961,E-mail:spbg09@aol.nl.
Yours Sincerely,
Mrs.Sandra Sleeswijk,
Public Relation Officer.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
-- Computer Center PSU PATTANI --
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-07 22:55 UTC (permalink / raw)
To: Brice Goglin; +Cc: Linux Network Developers, Yinghai Lu, gallatin
In-Reply-To: <4A7CA7F0.60704@inria.fr>
On Sat, 08 Aug 2009, Brice Goglin wrote:
> Bill Fink wrote:
> > OK. The tests were run on a 2.6.29.6 kernel so presumably should
> > have included the fix you mentioned.
>
> Yes, but I wanted to emphasize that new platforms sometime need some new
> code to handle this kind of things. Some Nehalem-specific changes might
> be needed now.
Thanks for the clarification.
> >>>> Is the corresponding local_cpus sysfs file wrong as well ?
> >>>>
> >>> All sysfs local_cpus values are the same (00000000,000000ff),
> >>> so yes they are also wrong.
> >>>
> >> And hyperthreading is enabled, right?
> >>
> >
> > No, hyperthreading is disabled. It's a dual quad-core system so there
> > are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
>
> So numa_node says that the device is close to node 0 while local_cpus
> says that it's close to all 8 cores ie close to both node0 and node2
> (which may well be wrong as well).
I believe it is wrong. The basic system arcitecture is:
Memory----CPU1----QPI----CPU2----Memory
| |
| |
QPI QPI
| |
| |
5520----QPI----5520
|||| ||||
|||| ||||
|||| ||||
PCIe PCIe
There are 2 x8, 1 x16, and 1 x4 PCIe 2.0 interfaces on each of the
Intel 5520 I/O Hubs. The Myricom dual-port 10-GigE NICs are in the
six x8 or better slots. eth2 through eth7 are on the second
Intel 5520 I/O Hub, so they should presumably show up on NUMA node 2,
and have local CPUs 1, 3, 5, and 7. eth8 through eth13 are on the
first Intel 5520 I/O Hub, and thus should be on NUMA node 0 with
local CPUs 0, 2, 4, and 6 (CPU info derived from /proc/cpinfo).
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-08 0:54 UTC (permalink / raw)
To: Neil Horman; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090807221211.GA16874@localhost.localdomain>
On Fri, 7 Aug 2009, Neil Horman wrote:
> You're timing is impeccable! I just posted a patch for an ftrace module to help
> detect just these kind of conditions:
> http://marc.info/?l=linux-netdev&m=124967650218846&w=2
>
> Hope that helps you out
> Neil
Thanks! It could be helpful. Do you have a pointer to documentation
on how to use it? And does it require the latest GIT kernel or could
it possibly be used with a 2.6.29.6 kernel?
-Bill
^ permalink raw reply
* Re: [PATCH] korina: Read buffer overflow
From: Phil Sutter @ 2009-08-08 0:48 UTC (permalink / raw)
To: Roel Kluin; +Cc: netdev, Andrew Morton, David S. Miller, florian
In-Reply-To: <4A7C494B.2060204@gmail.com>
Hi,
On Fri, Aug 07, 2009 at 05:33:31PM +0200, Roel Kluin wrote:
> If the loop breaks with an i of 0, then we read lp->rd_ring[-1].
Indeed.
> Signed-off-by: Roel Kluin <roel.kluin@gmail.com>
> ---
> Should we clean up like this? please review
>
> diff --git a/drivers/net/korina.c b/drivers/net/korina.c
> index b4cf602..b965b2b 100644
> --- a/drivers/net/korina.c
> +++ b/drivers/net/korina.c
> @@ -754,7 +754,7 @@ static void korina_alloc_ring(struct net_device *dev)
> {
> struct korina_private *lp = netdev_priv(dev);
> struct sk_buff *skb;
> - int i;
> + int i, j;
>
> /* Initialize the transmit descriptors */
> for (i = 0; i < KORINA_NUM_TDS; i++) {
> @@ -771,7 +771,7 @@ static void korina_alloc_ring(struct net_device *dev)
> for (i = 0; i < KORINA_NUM_RDS; i++) {
> skb = dev_alloc_skb(KORINA_RBSIZE + 2);
> if (!skb)
> - break;
> + goto err_free;
This implies that all KORINA_NUM_TDS receive descriptors need to be
available for the driver to work.
> skb_reserve(skb, 2);
> lp->rx_skb[i] = skb;
> lp->rd_ring[i].control = DMA_DESC_IOD |
> @@ -790,6 +790,12 @@ static void korina_alloc_ring(struct net_device *dev)
> lp->rx_chain_head = 0;
> lp->rx_chain_tail = 0;
> lp->rx_chain_status = desc_empty;
> +err_free:
> + for (j = 0; j < i; j++) {
> + lp->rd_ring[j].control = 0;
> + dev_kfree_skb_any(lp->rx_skb[j]);
> + lp->rx_skb[j] = NULL;
> + }
> }
Also I guess there should be some error handling, as the driver probably
wont be useful without a single receive descriptor. What do you think
about the following:
| diff --git a/drivers/net/korina.c b/drivers/net/korina.c
| index a2701f5..d1c5276 100644
| --- a/drivers/net/korina.c
| +++ b/drivers/net/korina.c
| @@ -741,7 +741,7 @@ static struct ethtool_ops netdev_ethtool_ops = {
| .get_link = netdev_get_link,
| };
|
| -static void korina_alloc_ring(struct net_device *dev)
| +static int korina_alloc_ring(struct net_device *dev)
| {
| struct korina_private *lp = netdev_priv(dev);
| struct sk_buff *skb;
| @@ -772,6 +772,13 @@ static void korina_alloc_ring(struct net_device *dev)
| lp->rd_ring[i].ca = CPHYSADDR(skb->data);
| lp->rd_ring[i].link = CPHYSADDR(&lp->rd_ring[i+1]);
| }
| + if (!i) {
| + printk(KERN_ERR DRV_NAME "%s: could not allocate a single"
| + " receive descriptor\n", dev->name)
| + return 1;
| + }
| + printk(KERN_DEBUG DRV_NAME "%s: allocated %d receive descriptors\n",
| + dev->name, i);
|
| /* loop back receive descriptors, so the last
| * descriptor points to the first one */
| @@ -782,6 +789,8 @@ static void korina_alloc_ring(struct net_device *dev)
| lp->rx_chain_head = 0;
| lp->rx_chain_tail = 0;
| lp->rx_chain_status = desc_empty;
| +
| + return 0;
| }
|
| static void korina_free_ring(struct net_device *dev)
| @@ -824,7 +833,8 @@ static int korina_init(struct net_device *dev)
| writel(ETH_INT_FC_EN, &lp->eth_regs->ethintfc);
|
| /* Allocate rings */
| - korina_alloc_ring(dev);
| + if (korina_alloc_ring(dev))
| + return 1;
|
| writel(0, &lp->rx_dma_regs->dmas);
| /* Start Rx DMA */
Greetings, Phil
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Bill Fink @ 2009-08-08 1:35 UTC (permalink / raw)
To: Andrew Gallatin; +Cc: Brice Goglin, Linux Network Developers, Yinghai Lu
In-Reply-To: <4A7CCEFC.7020308@myri.com>
On Fri, 07 Aug 2009, Andrew Gallatin wrote:
> Bill Fink wrote:
>
> > All sysfs local_cpus values are the same (00000000,000000ff),
> > so yes they are also wrong.
>
> How were you handling IRQ binding? If local_cpus is wrong,
> the irqbalance will not be able to make good decisions about
> where to bind the NICs' IRQs. Did you try manually binding
> each NICs's interrupt to a separate CPU on the correct node?
Yes, all the NIC IRQs were bound to a CPU on the local NUMA node,
and the nuttcp application had its CPU affinity set to the same
CPU with its memory affinity bound to the same local NUMA node.
And the irqbalance daemon wasn't running.
-Thanks
-Bill
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Andrew Gallatin @ 2009-08-08 1:03 UTC (permalink / raw)
To: Bill Fink; +Cc: Brice Goglin, Linux Network Developers, Yinghai Lu
In-Reply-To: <20090807175112.a1f57407.billfink@mindspring.com>
Bill Fink wrote:
> All sysfs local_cpus values are the same (00000000,000000ff),
> so yes they are also wrong.
How were you handling IRQ binding? If local_cpus is wrong,
the irqbalance will not be able to make good decisions about
where to bind the NICs' IRQs. Did you try manually binding
each NICs's interrupt to a separate CPU on the correct node?
Regards,
Drew
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox