* Re: [PATCH NEXT 2/4] netxen: implement generic pcie semaphore functions
From: David Miller @ 2009-08-26 22:37 UTC (permalink / raw)
To: dhananjay; +Cc: davem, netdev
In-Reply-To: <1251177808-16450-2-git-send-email-dhananjay@netxen.com>
From: Dhananjay Phadke <dhananjay@netxen.com>
Date: Mon, 24 Aug 2009 22:23:26 -0700
> Implement common function for locking/unlocking 8 hardware
> semaphores used for serializing access to shared resouces
> on a NIC board by different PCI functions.
>
> As by definition, callers of these semaphore API can be
> put to sleep till the semaphore is locked.
>
> Signed-off-by: Dhananjay Phadke <dhananjay@netxen.com>
Applied.
^ permalink raw reply
* Re: [PATCH NEXT 1/4] netxen: remove unused code
From: David Miller @ 2009-08-26 22:37 UTC (permalink / raw)
To: dhananjay; +Cc: davem, netdev
In-Reply-To: <1251177808-16450-1-git-send-email-dhananjay@netxen.com>
From: Dhananjay Phadke <dhananjay@netxen.com>
Date: Mon, 24 Aug 2009 22:23:25 -0700
> Remove code for phy access on unreleased NX2031 based quad-gig board.
>
> NX3031 based production quad-gig boards do not require direct phy
> access by driver.
>
> Signed-off-by: Dhananjay Phadke <dhananjay@netxen.com>
Applied.
^ permalink raw reply
* Re: [PATCH 0/9] drivers/net/s2io.c: Cleanups
From: David Miller @ 2009-08-26 22:33 UTC (permalink / raw)
To: joe
Cc: netdev, anil.murthy, sreenivasa.honnur, sivakumar.subramani,
santosh.rastapur, ram.vepa, akpm, linux-next
In-Reply-To: <cover.1251170438.git.joe@perches.com>
From: Joe Perches <joe@perches.com>
Date: Mon, 24 Aug 2009 20:29:39 -0700
> Just a few cleanups, compiled, untested. No hardware.
>
> Joe Perches (9):
> s2io.c: Use const for strings
> s2io.c: Shorten code line length by using intermediate pointers
> s2io.c: Use calculated size in kmallocs
> s2io.c: use kzalloc
> s2io.c: Make more conforming to normal kernel style
> s2io.c: convert printks to pr_<level>
> s2io.c: fix spelling explaination
> s2io.c: Standardize statistics accessors
> s2io.c: Convert skipped nic->config.tx_cfg[i]. to tx_cfg->
Since this is a pretty serious set of cleanups, I'd like to
let the S2IO driver folks have some time to look at this and
at least have a chance to ACK/NACK them.
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Sridhar Samudrala @ 2009-08-26 22:11 UTC (permalink / raw)
To: Christoph Lameter
Cc: David Stevens, David S. Miller, Eric Dumazet, netdev, niv
In-Reply-To: <alpine.DEB.1.10.0908261506590.9933@gentwo.org>
On Wed, 2009-08-26 at 15:09 -0400, Christoph Lameter wrote:
> On Wed, 26 Aug 2009, Sridhar Samudrala wrote:
>
> > > They are reported for IP and UDP.
> > Not clear what you meant by this.
>
> The SNMP and UDP statistics show the loss. qdisc level does not show the
> loss.
> > > root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show
> > > qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> > > 1 1 1 1
> > > Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0)
> > > rate 0bit 0pps backlog 0b 0p requeues 0
> >
> > Even the Sent count seems to be too low. Are you looking at the right
> > device?
>
> I would think that tc displays all queues? It says eth0 and eth0 is the
> device that we sent the data out on.
>
> > So based on the current analysis, the packets are getting dropped after
> > the call to ip_local_out() in ip_push_pending_frames(). ip_local_out()
> > is failing with NET_XMIT_DROP. But we are not sure where they are
> > getting dropped. Is that right?
>
> ip_local_out is returning ENOBUFS. Something at the qdisc layer is
> dropping the packet and not incrementing counters.
Is the ENOBUFS return with your/Eric's patch? I thought you were
were seeing NET_XMIT_DROP without any patches.
>
> > I think we need to figure out where they are getting dropped and then
> > decide on the appropriate counter to be incremented.
>
> Right. Where in the qdisc layer do drops occur?
The normal path where the packets are dropped when the tx qlen is exceeded is
pfifo_fast_enqueue() -> qdisc_drop()
In this path, drops are counted.
The other place is in dev_queue_xmit(), but you are not hitting that case too.
So it looks like there is another place where they are getting dropped.
Thanks
Sridhar
^ permalink raw reply
* [PATCH] Re: iproute2 / tbf with large burst seems broken again
From: Jarek Poplawski @ 2009-08-26 21:59 UTC (permalink / raw)
To: Denys Fedoryschenko; +Cc: netdev
In-Reply-To: <200908251518.00762.denys@visp.net.lb>
Denys Fedoryschenko wrote, On 08/25/2009 02:18 PM:
...
> Just maybe good idea in next kernels to document this or handle overflow, that
> it will give warning in kernel. Otherwise user will have non-functional qdisc
> that will not pass traffic.
OK, if this patch works for you send your Tested-by, please.
Thanks,
Jarek P.
---------------->
pkt_sched: Warn on tokens overflow in tbf_dequeue
After recent change of scheduling resolution some "extreme" configs
can cause token overflows with stalls. This patch warns when tokens
were negative before accounting for a packet length.
Reported-by: Denys Fedoryschenko <denys@visp.net.lb>
Signed-off-by: Jarek Poplawski <jarkao2@gmail.com>
---
net/sched/sch_tbf.c | 6 ++++++
1 files changed, 6 insertions(+), 0 deletions(-)
diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c
index e22dfe8..ef191c4 100644
--- a/net/sched/sch_tbf.c
+++ b/net/sched/sch_tbf.c
@@ -191,6 +191,12 @@ static struct sk_buff *tbf_dequeue(struct Qdisc* sch)
return skb;
}
+ if (unlikely((long)(toks + L2T(q, len)) < 0 ||
+ (long)(ptoks + L2T_P(q, len) < 0)))
+ if (net_ratelimit())
+ pr_warning("tbf: tokens overflow;"
+ " fix limits.\n");
+
qdisc_watchdog_schedule(&q->watchdog,
now + max_t(long, -toks, -ptoks));
^ permalink raw reply related
* [PATCH] scripts/get_maintainer.pl: Add maintainers in order listed in matched section
From: Joe Perches @ 2009-08-26 21:48 UTC (permalink / raw)
To: LKML
Cc: Andrew Morton, Greg KH, Ben Hutchings, David Miller, netdev,
linux-net-drivers
In-Reply-To: <cover.1249525834.git.joe@perches.com>
This patch is on top of patches that are already queued in mm.
Previous behavior was "bottom-up" in each section from the pattern "F:"
entry that matched. Now information is entered into the various lists
in the "as entered" order for each matched section.
This also allows the F: entry to be put anywhere in a section, not just
as the last entries in the section.
And a couple of improvements:
Don't alphabetically sort before outputting the matched scm, status,
subsystem and web sections.
Ignore content after a single email address so these entries are acceptable
M: name <address> whatever other comment
And a fix:
Make an M: entry without a name again use the name from an immediately
preceding P: line if it exists.
Signed-off-by: Joe Perches <joe@perches.com>
---
scripts/get_maintainer.pl | 70 ++++++++++++++++++++++++++++++++++----------
1 files changed, 54 insertions(+), 16 deletions(-)
diff --git a/scripts/get_maintainer.pl b/scripts/get_maintainer.pl
index b8d1223..b54db05 100755
--- a/scripts/get_maintainer.pl
+++ b/scripts/get_maintainer.pl
@@ -313,22 +313,22 @@ if ($email || $email_list) {
}
if ($scm) {
- @scm = sort_and_uniq(@scm);
+ @scm = uniq(@scm);
output(@scm);
}
if ($status) {
- @status = sort_and_uniq(@status);
+ @status = uniq(@status);
output(@status);
}
if ($subsystem) {
- @subsystem = sort_and_uniq(@subsystem);
+ @subsystem = uniq(@subsystem);
output(@subsystem);
}
if ($web) {
- @web = sort_and_uniq(@web);
+ @web = uniq(@web);
output(@web);
}
@@ -442,7 +442,7 @@ sub parse_email {
$address = $2;
} elsif ($formatted_email =~ /^\s*<(.+\@\S*)>.*$/) {
$address = $1;
- } elsif ($formatted_email =~ /^(.+\@\S*)$/) {
+ } elsif ($formatted_email =~ /^(.+\@\S*).*$/) {
$address = $1;
}
@@ -485,12 +485,46 @@ sub format_email {
return $formatted_email;
}
-sub add_categories {
+sub find_starting_index {
+
+ my ($index) = @_;
+
+ while ($index > 0) {
+ my $tv = $typevalue[$index];
+ if (!($tv =~ m/^(\C):\s*(.*)/)) {
+ last;
+ }
+ $index--;
+ }
+
+ return $index;
+}
+
+sub find_ending_index {
my ($index) = @_;
- $index = $index - 1;
- while ($index >= 0) {
+ while ($index < @typevalue) {
my $tv = $typevalue[$index];
+ if (!($tv =~ m/^(\C):\s*(.*)/)) {
+ last;
+ }
+ $index++;
+ }
+
+ return $index;
+}
+
+sub add_categories {
+ my ($index) = @_;
+
+ my $i;
+ my $start = find_starting_index($index);
+ my $end = find_ending_index($index);
+
+ push(@subsystem, $typevalue[$start]);
+
+ for ($i = $start + 1; $i < $end; $i++) {
+ my $tv = $typevalue[$i];
if ($tv =~ m/^(\C):\s*(.*)/) {
my $ptype = $1;
my $pvalue = $2;
@@ -513,11 +547,12 @@ sub add_categories {
} elsif ($ptype eq "M") {
my ($name, $address) = parse_email($pvalue);
if ($name eq "") {
- if ($index >= 0) {
- my $tv = $typevalue[$index - 1];
+ if ($i > 0) {
+ my $tv = $typevalue[$i - 1];
if ($tv =~ m/^(\C):\s*(.*)/) {
if ($1 eq "P") {
$name = $2;
+ $pvalue = format_email($name, $address);
}
}
}
@@ -532,11 +567,6 @@ sub add_categories {
} elsif ($ptype eq "S") {
push(@status, $pvalue);
}
-
- $index--;
- } else {
- push(@subsystem,$tv);
- $index = -1;
}
}
}
@@ -559,6 +589,10 @@ sub push_email_address {
my ($name, $address) = parse_email($line);
+ if ($address eq "") {
+ return 0;
+ }
+
if (!$email_remove_duplicates) {
push(@email_to, format_email($name, $address));
} elsif (!email_inuse($name, $address)) {
@@ -566,6 +600,8 @@ sub push_email_address {
$email_hash_name{$name}++;
$email_hash_address{$address}++;
}
+
+ return 1;
}
sub push_email_addresses {
@@ -581,7 +617,9 @@ sub push_email_addresses {
push_email_address($entry);
}
} else {
- warn("Invalid MAINTAINERS address: '" . $address . "'\n");
+ if (!push_email_address($address)) {
+ warn("Invalid MAINTAINERS address: '" . $address . "'\n");
+ }
}
}
--
1.6.3.1.10.g659a0.dirty
^ permalink raw reply related
* Re: [PATCH] r8169: Reduce looping in the interrupt handler.
From: Eric W. Biederman @ 2009-08-26 21:40 UTC (permalink / raw)
To: Francois Romieu
Cc: David Dillow, Michael Riepe, Michael Buesch, Rui Santos,
Michael B??ker, linux-kernel, netdev
In-Reply-To: <20090826213024.GA20428@electric-eye.fr.zoreil.com>
Francois Romieu <romieu@fr.zoreil.com> writes:
> Eric W. Biederman <ebiederm@xmission.com> :
> [...]
>> It is a bit weird but it also means we aren't playing silly games
>> with status inside the loop. So if we go through the loop we ack
>> everything in status.
>
> I fear we have some longstanding problem anyway :
>
> 1. quiescent state
> 2. packets are received
> 3. rtl8169_interrupt schedules napi, clears IntrStatus and exits
> 4. packets are received and some non-napi event happens
> 5. rtl8169_interrupt wakes up, reads IntrStatus and goes on...
> 6. rtl8169_poll wakes up, processes Rx and Tx napi events and goes on...
> 7. tp->intr_mask still equals ~tp->napi_event : rtl8169_interrupt
> handler does not even try to schedule napi.
> 8. more packets are received
> 9. rtl8169_interrupt clears IntrStatus
> a. rtl8169_poll reenables napi scheduling, updates IntrMask and exits
> b. rtl8169_interrupt reads a perfectly clean IntrStatus and exits
That would not surprise me.
Right now I really don't have much more test bandwidth. So I tried
for something simple that would address my problem without
fundamentally changing the already tested logic. I am not seeing any
of the weird corner cases where we get confused. The changes to fix
that problem is totally killing my ability to use the NIC, because it
loops way to much.
Perhaps we should unconditionally ack everything after changing the
interrupt mask? If that would prevent races it sounds like a simple fix.
Eric
^ permalink raw reply
* Re: [PATCH] r8169: Reduce looping in the interrupt handler.
From: Francois Romieu @ 2009-08-26 21:30 UTC (permalink / raw)
To: Eric W. Biederman
Cc: David Dillow, Michael Riepe, Michael Buesch, Rui Santos,
Michael B??ker, linux-kernel, netdev
In-Reply-To: <m11vmy737x.fsf@fess.ebiederm.org>
Eric W. Biederman <ebiederm@xmission.com> :
[...]
> It is a bit weird but it also means we aren't playing silly games
> with status inside the loop. So if we go through the loop we ack
> everything in status.
I fear we have some longstanding problem anyway :
1. quiescent state
2. packets are received
3. rtl8169_interrupt schedules napi, clears IntrStatus and exits
4. packets are received and some non-napi event happens
5. rtl8169_interrupt wakes up, reads IntrStatus and goes on...
6. rtl8169_poll wakes up, processes Rx and Tx napi events and goes on...
7. tp->intr_mask still equals ~tp->napi_event : rtl8169_interrupt
handler does not even try to schedule napi.
8. more packets are received
9. rtl8169_interrupt clears IntrStatus
a. rtl8169_poll reenables napi scheduling, updates IntrMask and exits
b. rtl8169_interrupt reads a perfectly clean IntrStatus and exits
--
Ueimor
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 20:40 UTC (permalink / raw)
To: Neil Horman
Cc: David Miller, rostedt, fweisbec, billfink, netdev, brice,
gallatin
In-Reply-To: <20090826202344.GE10816@hmsreliant.think-freely.org>
* Neil Horman <nhorman@tuxdriver.com> wrote:
> On Wed, Aug 26, 2009 at 09:48:35PM +0200, Ingo Molnar wrote:
> >
> > * David Miller <davem@davemloft.net> wrote:
> >
> > > From: Ingo Molnar <mingo@elte.hu>
> > > Date: Wed, 26 Aug 2009 21:08:30 +0200
> > >
> > > > Sigh, no. Please re-read the past discussions about this.
> > > > trace_skb_sources.c is a hack and should be converted to generic
> > > > tracepoints. Is there anything in it that cannot be expressed in
> > > > terms of TRACE_EVENT()?
> > >
> > > Neil explained why he needed to implement it this way in his reply
> > > to Steven Rostedt. I attach it here for your convenience.
> >
> > thanks. The argument is invalid:
>
> Just because you assert that doesn't make it so, Ingo.
I stand by that statement, the argument is invalid, for the many
reasons i outlined in my previous mails. (you'd have gotten those
same arguments had you submitted that patch to the folks who
maintain kernel/trace/)
> > > > BTW, why not just do this as events? Or was this just a easy way
> > > > to communicate with the user space tools?
> > >
> > > Thats exactly why I did it. the idea is for me to now write a
> > > user space tool that lets me analyze the events and ajust process
> > > scheduling to optimize the rx path. Neil
> >
> > All tooling (in fact _more_ tooling) can be done based on generic,
> > TRACE_EVENT() based tracepoints. Generic tracepoints are far more
> > available, have a generalized format with format parsers and user
> > tooling implemented, etc. etc.
>
> Then why allow for ftrace modules at all? [...]
We routinely reject trivial plugins like yours and ask people to use
the proper mechanism: TRACE_EVENT().
We are also converting non-trivial plugins to generic tracepoints. A
recent example are the system call tracepoints, but we also
converted blktrace and kmemtrace to generic tracepoints.
But trace_skb_sources.c got committed to the networking tree,
without review and acks from the tracing folks. Now you are
unwilling to fix it and that's not very constructive.
> [...] I grant that the skb ftracer is a bit trivial at the moment
> for an ftrace module, but I really prefer to leave it is so that I
> can expand it with additional tracepoints. And looking at them,
> anything you've said above applies to any of the currently
> implemented ftrace modules. If you're so adamant that we should
> just do everything with TRACE_EVENT log messages, then lets get
> rid of the ftrace infrastructure all together. Until we do that,
> however, I like my skb tracer just as it is.
You dont seem to be aware of the breath of features and capabilities
that TRACE_EVENT() based tooling allows us to do. Please see my
previous mail about an (incomplete) list.
( One item i forgot to mention there: using them you can for example
trace full workloads, such as a kernel build - without other
workloads mixed into that trace. Etc. etc. - the list goes on. )
Ingo
^ permalink raw reply
* [PATCH] bonding: bond_check_dev_link() needs to check for netif_running()
From: Petri Gynther @ 2009-08-26 20:35 UTC (permalink / raw)
To: Stephen Hemminger, David S. Miller; +Cc: netdev
bond_check_dev_link() needs to check for netif_running() before checking
the link status of a slave. This needs to be done because some interfaces
keep reporting netif_carrier_ok() == TRUE even after "ifconfig down".
Signed-off-by: Petri Gynther <pgynther@google.com>
---
drivers/net/bonding/bond_main.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index aa1be1f..0835cbe 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -695,6 +695,9 @@ static int bond_check_dev_link(struct bonding *bond,
struct ifreq ifr;
struct mii_ioctl_data *mii;
+ if (!netif_running(slave_dev))
+ return 0;
+
if (bond->params.use_carrier)
return netif_carrier_ok(slave_dev) ? BMSR_LSTATUS : 0;
--
1.5.4.3
^ permalink raw reply related
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 20:28 UTC (permalink / raw)
To: David Miller
Cc: nhorman, rostedt, fweisbec, billfink, netdev, brice, gallatin
In-Reply-To: <20090826194835.GA16508@elte.hu>
* Ingo Molnar <mingo@elte.hu> wrote:
> * David Miller <davem@davemloft.net> wrote:
>
> > From: Ingo Molnar <mingo@elte.hu>
> > Date: Wed, 26 Aug 2009 21:08:30 +0200
> >
> > > Sigh, no. Please re-read the past discussions about this.
> > > trace_skb_sources.c is a hack and should be converted to generic
> > > tracepoints. Is there anything in it that cannot be expressed in
> > > terms of TRACE_EVENT()?
> >
> > Neil explained why he needed to implement it this way in his reply
> > to Steven Rostedt. I attach it here for your convenience.
>
> thanks. The argument is invalid:
>
> > > BTW, why not just do this as events? Or was this just a easy way
> > > to communicate with the user space tools?
> >
> > Thats exactly why I did it. the idea is for me to now write a
> > user space tool that lets me analyze the events and ajust process
> > scheduling to optimize the rx path. Neil
>
> All tooling (in fact _more_ tooling) can be done based on generic,
> TRACE_EVENT() based tracepoints. Generic tracepoints are far more
> available, have a generalized format with format parsers and user
> tooling implemented, etc. etc.
To expand on the 'etc. etc.'.
Right now we already have once TRACE_EVENT() based generic
tracepoint for skbs - the skb_free one in
include/trace/events/skb.h.
Here's a list of examples of what that single generic tracepoint
allows us to do, which Neil's kernel/trace/trace_skb_sources.c code
cannot do:
- structured format/field description:
aldebaran:~> cat /debug/tracing/events/skb/kfree_skb/format
name: kfree_skb
ID: 603
format:
field:unsigned short common_type; offset:0; size:2;
field:unsigned char common_flags; offset:2; size:1;
field:unsigned char common_preempt_count; offset:3; size:1;
field:int common_pid; offset:4; size:4;
field:int common_tgid; offset:8; size:4;
field:void * skbaddr; offset:16; size:8;
field:unsigned short protocol; offset:24; size:2;
field:void * location; offset:32; size:8;
print fmt: "skbaddr=%p protocol=%u location=%p", REC->skbaddr, REC->protocol, REC->location
The advantages of that are numerous: we have a user-space parser
for that, so new tracepoints or changes to tracepoints can be
propagated across the tooling automatically. (see below examples
about how this works in practice)
- perfcounters integration:
- it's enumerated and visible in the list of tracepoints:
aldebaran:~> perf list 2>&1 | grep skb
skb:kfree_skb [Tracepoint event]
- the tracepoint can be used for statistics (perf stat):
aldebaran:~> perf stat -e skb:kfree_skb -a sleep 1
Performance counter stats for 'sleep 1':
- noise analysis:
aldebaran:~> perf stat --repeat 10 -e skb:kfree_skb -a sleep 1
Performance counter stats for 'sleep 1' (10 runs):
25 skb:kfree_skb ( +- 7.692% )
- the tracepoint can be used for profiling:
aldebaran:~> perf top -e skb:kfree_skb -c 1
------------------------------------------------------------------------------
PerfTop: 334 irqs/sec kernel: 0.3% [1 skb:kfree_skb], (all, 16 CPUs)
------------------------------------------------------------------------------
samples pcnt RIP kernel function
______ _______ _____ ________________ _______________
23.00 - 100.0% - ffffffff81266828 : store_bind
- can be used to do call-graph profiling that captures kernel
and user-space call-graphs as well:
aldebaran:~> perf record --call-graph -e skb:kfree_skb -c 1 -f -a sleep 1
[ perf record: Captured and wrote 0.035 MB perf.data (~1547 samples) ]
aldebaran:~> perf report
...
# Samples: 4102
#
# Overhead Command Shared Object Symbol
# ........ ............... ........................................................................................................ ......
#
88.44% distccd 3641efb1d0 [.] 0x00003641efb1d0
3.07% Xorg 3641ed6590 [.] 0x00003641ed6590
2.51% at-spi-registry 3642a0db50 [.] 0x00003642a0db50
2.24% sshd /lib64/libc-2.8.so [.] __libc_read
0.73% sshd 7f71d4e69590 [.] 0x007f71d4e69590
0.63% init [kernel] [k] store_bind
0.56% sshd /lib64/libc-2.8.so [.] __recvmsg
0.49% gnome-settings- 3642a0db8b [.] 0x00003642a0db8b
0.39% sshd /lib64/libc-2.8.so [.] __GI___libc_connect
0.39% sshd /lib64/libc-2.8.so [.] __sendto_nocancel
0.15% id /lib64/libc-2.8.so [.] __GI___libc_connect
|
|--50.00%-- get_mapping
| __nscd_get_map_ref
|
--50.00%-- __nscd_open_socket
0.10% metacity 3641ed6590 [.] 0x00003641ed6590
0.07% gdm-simple-gree 3642a0db8b [.] 0x00003642a0db8b
|
|--66.67%-- 0x3641ed65cb
|
--33.33%-- 0x3642a0db8b
0.05% bash /lib64/libc-2.8.so [.] __GI___libc_connect
|
|--50.00%-- get_mapping
| __nscd_get_map_ref
|
--50.00%-- __nscd_open_socket
0.05% :3129 /lib64/libc-2.8.so [.] __GI___libc_connect
|
|--50.00%-- get_mapping
| __nscd_get_map_ref
|
--50.00%-- __nscd_open_socket
0.05% :3098 /lib64/libc-2.8.so [.] __GI___libc_connect
|
|--50.00%-- get_mapping
| __nscd_get_map_ref
|
--50.00%-- __nscd_open_socket
0.02% init [kernel] [k] bind_con_driver
0.02% gnome-power-man 3642a0db50 [.] 0x00003642a0db50
0.02% cc1 /opt/crosstool/gcc-4.2.2-glibc-2.3.6/i686-unknown-linux-gnu/libexec/gcc/i686-unknown-linux-gnu/4.2.2/cc1 [.] num_positive
- can be used to capture traces to user-space and analyze them
there:
aldebaran:/home/mingo> perf record -e skb:kfree_skb:r -c 1 -R -f -a sleep 10
[ perf record: Captured and wrote 4.426 MB perf.data (~193365 samples) ]
aldebaran:/home/mingo> perf trace
version = 0.5B6
init-0 [000] 0.000000: kfree_skb: skbaddr=0xffff8801bcc15300 protocol=2048 location=0xffffffff81461c94
Xorg-4411 [000] 0.000000: kfree_skb: skbaddr=0xffff8801bb955a00 protocol=0 location=0xffffffff814e8aff
at-spi-registry-4948 [000] 0.000000: kfree_skb: skbaddr=0xffff8801bb955a00 protocol=0 location=0xffffffff814e8aff
...
- generic tracepoints can be available with lots of other
tracepoints at once - while the skb_sources plugin is exclusive.
(no other plugin can be active at the same time) Generic
tracepoints have separate toggles - any sub-set of tracepoints
can be active at any time.
- per tracepoint filter expressions support, such as:
aldebaran:/debug/tracing/events/skb/kfree_skb> echo 'protocol == 0 && common_pid == 123' > filter
aldebaran:/debug/tracing/events/skb/kfree_skb> cat filter protocol == 0 && common_pid == 123
protocol == 0 && common_pid == 123
When this filter is modified, the kernel creates a (safe) list of
(atomically evaluatable) predicaments from the expression and the
data is filtered before it's traced.
The filter engine works in process, softirq, IRQ, NMI and any
other context and is very fast as well. (no parsing overhead in
the fastpath - we pre-parse the expression and break it down.)
In other words, generic tracepoints are _vastly_ superior to the
skb_sources plugin, and this fact is obvious to all tracing
developers, that's why every tracing developer who commented on this
thread asked (in a rather befuddled way) "why not TRACE_EVENT()?".
And note that the above examples were based on a _single_ existing
generic tracepoint of very limited utility - and still it already
allowed a lot of interesting data to be captured. If we had a more
comprehensive set of skb tracepoints, a whole lot of interesting
possibilities would open up ...
All in one, we dont do new ftrace plugins that can be done via
generic tracepoints - we only limit ftrace plugins to vastly
different things like the function tracer or the latency tracer.
That's why we have things like a tracing tree and a review process,
to address such issues before patches get committed.
David, please sort this out before sending any bits in this area to
Linus, Neil's response is basically "i want it this way" which is
not really acceptable - the maintainers of kernel/trace/* dont want
it this way, for very good technical reasons.
The skb_sources hack should be converted to a proper
TRACE_EVENT(skb_dequeue) tracepoint. Also, as we offered it on the
onset, we'd be glad to help out with the conversion. I can do a
patch if nobody volunteers.
Plus we'd like to encourage more TRACE_EVENT() networking
tracepoints like the existing skb_free. They are a great tool.
Ingo
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-26 20:23 UTC (permalink / raw)
To: Ingo Molnar
Cc: David Miller, rostedt, fweisbec, billfink, netdev, brice,
gallatin
In-Reply-To: <20090826194835.GA16508@elte.hu>
On Wed, Aug 26, 2009 at 09:48:35PM +0200, Ingo Molnar wrote:
>
> * David Miller <davem@davemloft.net> wrote:
>
> > From: Ingo Molnar <mingo@elte.hu>
> > Date: Wed, 26 Aug 2009 21:08:30 +0200
> >
> > > Sigh, no. Please re-read the past discussions about this.
> > > trace_skb_sources.c is a hack and should be converted to generic
> > > tracepoints. Is there anything in it that cannot be expressed in
> > > terms of TRACE_EVENT()?
> >
> > Neil explained why he needed to implement it this way in his reply
> > to Steven Rostedt. I attach it here for your convenience.
>
> thanks. The argument is invalid:
>
Just because you assert that doesn't make it so, Ingo.
> > > BTW, why not just do this as events? Or was this just a easy way
> > > to communicate with the user space tools?
> >
> > Thats exactly why I did it. the idea is for me to now write a
> > user space tool that lets me analyze the events and ajust process
> > scheduling to optimize the rx path. Neil
>
> All tooling (in fact _more_ tooling) can be done based on generic,
> TRACE_EVENT() based tracepoints. Generic tracepoints are far more
> available, have a generalized format with format parsers and user
> tooling implemented, etc. etc.
>
Then why allow for ftrace modules at all? I grant that the skb ftracer is a bit
trivial at the moment for an ftrace module, but I really prefer to leave it is
so that I can expand it with additional tracepoints. And looking at them,
anything you've said above applies to any of the currently implemented ftrace
modules. If you're so adamant that we should just do everything with
TRACE_EVENT log messages, then lets get rid of the ftrace infrastructure all
together. Until we do that, however, I like my skb tracer just as it is.
Neil
> Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply
* Re: 2.6.31-rc7-git2: Reported regressions 2.6.29 -> 2.6.30
From: Rafael J. Wysocki @ 2009-08-26 20:11 UTC (permalink / raw)
To: Rafał Miłecki
Cc: Linux Kernel Mailing List, DRI, Linux SCSI List,
Network Development, Linux Wireless List, Natalie Protasevich,
Linux ACPI, Andrew Morton, Kernel Testers List, Linus Torvalds,
Linux PM List, Len Brown
In-Reply-To: <b170af450908260147o60427997ne6b9872198893c41@mail.gmail.com>
On Wednesday 26 August 2009, Rafał Miłecki wrote:
> 2009/8/25 Rafael J. Wysocki <rjw@sisk.pl>:
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=13514
> > Subject : acer_wmi causes stack corruption
> > Submitter : Rus <harbour@sfinx.od.ua>
> > Date : 2009-06-12 08:13 (75 days old)
>
> It has patch, just Len doesn't seem to... don't know, read the topic?
> http://patchwork.kernel.org/patch/29082/
>
> Can we ping Len somehow to push this patch directly to Linus's tree?
I have updated the bug entry, hopefully Len will notice it now.
Thanks,
Rafael
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: AlacrityVM benchmark numbers updated
From: Gregory Haskins @ 2009-08-26 20:05 UTC (permalink / raw)
To: Avi Kivity
Cc: alacrityvm-devel, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, Michael S. Tsirkin, netdev
In-Reply-To: <4A958BA0.2000801@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 688 bytes --]
Avi Kivity wrote:
> On 08/26/2009 09:42 PM, Gregory Haskins wrote:
>> Actually, I have already look at this and it does indeed seem better to
>> use switch_mm+gupf() over gup() by quite a large margin. You could then
>> couple that with your DMA-engine idea to potentially gain even more
>> benefits (though probably not for networking since most NICs have their
>> own DMA engine anyway).
>>
>>
>
> For tx, we'll just go copyless once we plumb the destructors properly.
> But for rx on a shared interface it is impossible to avoid the copy.
> You can only choose if you want it done by the cpu or a local dma engine.
>
>
Yep, agree on both counts.
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* Re: [PATCH] r8169: Reduce looping in the interrupt handler.
From: Eric W. Biederman @ 2009-08-26 20:02 UTC (permalink / raw)
To: David Dillow
Cc: Francois Romieu, Michael Riepe, Michael Buesch, Rui Santos,
Michael B??ker, linux-kernel, netdev
In-Reply-To: <1251295175.14241.11.camel@obelisk.thedillows.org>
David Dillow <dave@thedillows.org> writes:
> On Wed, 2009-08-26 at 09:56 -0400, David Dillow wrote:
>> On Wed, 2009-08-26 at 00:58 -0700, Eric W. Biederman wrote:
>> > diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
>> > index 3b19e0c..2214945 100644
>> > --- a/drivers/net/r8169.c
>> > +++ b/drivers/net/r8169.c
>>
>> > +
>> > + /* Ignore the parts of status that reflect more than
>> > + * the enabled interrupts.
>> > + */
>> > + smp_rmb();
>> > + if (!(status & tp->intr_mask & tp->intr_event))
>> > + break;
>> > }
>>
>> This looks like an odd construct, since we're just about to go back the
>> while condition up top -- why not just mask it here and let the loop
>> handle it naturally?
>
> Never mind, I see what you are doing -- avoiding a false loop if we get
> status == 0xffff. I still don't like the aesthetics of it, but it makes
> sense, and I'll blame it on the card. :)
>
> I should really get some caffeine before posting...
It is a bit weird but it also means we aren't playing silly games
with status inside the loop. So if we go through the loop we ack
everything in status.
Eric
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-26 20:01 UTC (permalink / raw)
To: Ingo Molnar
Cc: David S. Miller, Steven Rostedt, Frédéric Weisbecker,
Bill Fink, Linux Network Developers, brice, gallatin
In-Reply-To: <20090826190830.GF13632@elte.hu>
On Wed, Aug 26, 2009 at 09:08:30PM +0200, Ingo Molnar wrote:
>
> * Neil Horman <nhorman@tuxdriver.com> wrote:
>
> > On Wed, Aug 26, 2009 at 08:15:02PM +0200, Ingo Molnar wrote:
> > >
> > > * Neil Horman <nhorman@tuxdriver.com> wrote:
> > >
> > > > On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> > > > > On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > > > > > On Fri, 21 Aug 2009, Neil Horman wrote:
> > > > > >
> > > > > > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > > > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > > > > > >
> > > > > > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > > > > > >
> > > > > > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > > > > > and never produced a crashdump. I tried this several times and
> > > > > > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > > > > > worked just fine.
> > > > > > > > >
> > > > > > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > > > > > >
> > > > > > > > There was a backtrace on the screen but I didn't have a chance to
> > > > > > > > record it. BTW did anyone ever think to print the backtrace in
> > > > > > > > reverse (first to some reserved memory and then output to the display)
> > > > > > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > > > > > the screen?
> > > > > > > >
> > > > > > > The real solution is to use a console to which the output doesn't scroll off the
> > > > > > > screen. Normally people use a serial console they can log, or a RAC card that
> > > > > > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > > > > > vt iirc to allow for scrolling.
> > > > > >
> > > > > > None of our Asus P6T6 systems have serial consoles. I don't know of
> > > > > > any RAC cards for them either, nor are there spare PCI slots available
> > > > > > in many cases. I wouldn't think the Shift-PageUp trick would work
> > > > > > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > > > > > out netconsole yet either, but I'm not sure it would help either in a
> > > > > > case like this that was a network related kernel crash.
> > > > > >
> > > > > Any USB ports that you can attach a serial dongle to? That would work as well,
> > > > > or, as previously mentioned, netconsole also does the trick.
> > > > >
> > > > > > In any case, a simple kernel command line that would provide a reversed
> > > > > > backtrace would be a simple thing to facilitate Linux users providing
> > > > > > useful info to Linux kernel developers in helping to debug kernel
> > > > > > problems. The most useful info would still be on the screen, so it
> > > > > > could be transcribed or a photo image of the screen could be taken.
> > > > > >
> > > > > I understand what your saying, I'm just saying there are currently several
> > > > > options for you that have already solved this problem in differnt ways.
> > > > >
> > > > > > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > > > > > does have a serial console, and after a fair amount of effort I was
> > > > > > able to get it to work as desired, and was able to finally capture
> > > > > > a backtrace of the kernel oops. BTW I believe the reason the
> > > > > > kexec/kdump didn't work was probably because it couldn't find
> > > > > > a /proc/vmcore file, although I don't know why that would be,
> > > > > > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > > > > > up normally if it fails to find the /proc/vmcore file (or it's
> > > > > > zero size).
> > > > > >
> > > > > I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> > > > > happy to look into it further.
> > > > >
> > > > > > The following shows a simple ping test usage of the skb_sources
> > > > > > tracing feature:
> > > > > >
> > > > > > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > > > > > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > > > > > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > > > > > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > > > > > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > > > > > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > > > > > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> > > > > >
> > > > > > --- 192.168.1.10 ping statistics ---
> > > > > > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > > > > > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> > > > > >
> > > > > > [root@xeontest1 tracing]# cat trace
> > > > > > # tracer: skb_sources
> > > > > > #
> > > > > > # PID ANID CNID IFC RXQ CCPU LEN
> > > > > > # | | | | | | |
> > > > > > 4217 1 1 eth2 0 4 1500
> > > > > > 4217 1 1 eth2 0 4 1500
> > > > > > 4217 1 1 eth2 0 4 1500
> > > > > > 4217 1 1 eth2 0 4 1500
> > > > > > 4217 1 1 eth2 0 4 1500
> > > > > >
> > > > > > All is as was expected.
> > > > > >
> > > > > > But if I try an actual nuttcp performance test (even rate limited
> > > > > > to 1 Mbps), I get the following kernel oops:
> > > > > >
> > > > > thank you, I think I see the problem, I'll have a patch for you in just a bit
> > > > >
> > > > > Thanks
> > > > > Neil
> > > > >
> > > > > > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > > > > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > > > > > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > > PGD 337d12067 PUD 337d11067 PMD 0
> > > > > > Oops: 0000 [#1] SMP
> > > > > > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > > > > > CPU 4
> > > > > > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > > > > > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > > > > > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > > > > > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > > > > > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > > > > > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > > > > > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > > > > > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > > > > > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > > > > > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > > > > > Stack:
> > > > > > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > > > > > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > > > > > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > > > > > Call Trace:
> > > > > > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > > > > > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > > > > > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > > > > > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > > > > > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > > > > > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > > > > > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > > > > > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > > > > > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > > > > > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > > > > > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > > > > > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > > > > > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > > > > > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > > > > > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > > > > > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > > > > > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > > > > > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > > > > > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > > RSP <ffff8801a5811a88>
> > > > > > CR2: 0000000000000038
> > > > > >
> > > > > > -Thanks
> > > > > >
> > > > > > -Bill
> > > > > >
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > > > the body of a message to majordomo@vger.kernel.org
> > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > > >
> > > >
> > > >
> > > > Here you go, I think this will fix your oops.
> > > >
> > > >
> > > > Fix NULL pointer deref in skb sources ftracer
> > > >
> > > > Its possible that skb->sk will be null in this path, so we shouldn't just assume
> > > > we can pass it to sock_net
> > > >
> > > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > > >
> > > > trace_skb_sources.c | 6 ++++--
> > > > 1 file changed, 4 insertions(+), 2 deletions(-)
> > >
> > > ok if this is just a temporary fix until TRACE_EVENT() is done, but
> > > we'll get rid of this and do TRACE_EVENT() before net-next-2.6 it's
> > > pushed to .32, right?
> >
> > Not sure that the two are related. I think you meant to send this
> > to the other thread, didnt you?
>
> Sigh, no. Please re-read the past discussions about this.
> trace_skb_sources.c is a hack and should be converted to generic
> tracepoints. Is there anything in it that cannot be expressed in
> terms of TRACE_EVENT()?
>
As David noted in my previous posting, no, I don't intend to change this. It
would certainly be possible to express this in terms of just a TRACE_EVENT, but
it would much more complex and messy for any user space tool to do so, IMHO. SO
I'd like to leave it as it is. To say its a hack as it is would really be to
say any of the current ftrace modules are a hack, as all of them could just as
easily be expressed as a series of trace events which were later parsed by a
user space tool.
I thought you're comments were related to the conversion of the napi_poll
tracepoint to a TRACE_EVENT structure, Which is current in progress.
Best
Neil
> Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 19:48 UTC (permalink / raw)
To: David Miller
Cc: nhorman, rostedt, fweisbec, billfink, netdev, brice, gallatin
In-Reply-To: <20090826.123631.79533250.davem@davemloft.net>
* David Miller <davem@davemloft.net> wrote:
> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 26 Aug 2009 21:08:30 +0200
>
> > Sigh, no. Please re-read the past discussions about this.
> > trace_skb_sources.c is a hack and should be converted to generic
> > tracepoints. Is there anything in it that cannot be expressed in
> > terms of TRACE_EVENT()?
>
> Neil explained why he needed to implement it this way in his reply
> to Steven Rostedt. I attach it here for your convenience.
thanks. The argument is invalid:
> > BTW, why not just do this as events? Or was this just a easy way
> > to communicate with the user space tools?
>
> Thats exactly why I did it. the idea is for me to now write a
> user space tool that lets me analyze the events and ajust process
> scheduling to optimize the rx path. Neil
All tooling (in fact _more_ tooling) can be done based on generic,
TRACE_EVENT() based tracepoints. Generic tracepoints are far more
available, have a generalized format with format parsers and user
tooling implemented, etc. etc.
Ingo
^ permalink raw reply
* [PATCH net-next 2/2] cnic: Put uio init in separate function.
From: Michael Chan @ 2009-08-26 19:49 UTC (permalink / raw)
To: davem; +Cc: netdev, Michael Chan
In-Reply-To: <1251316163-9359-1-git-send-email-mchan@broadcom.com>
This will allow the 10G iSCSI code to reuse the function.
Signed-off-by: Michael Chan <mchan@broadcom.com>
---
drivers/net/cnic.c | 68 +++++++++++++++++++++++++++++++--------------------
1 files changed, 41 insertions(+), 27 deletions(-)
diff --git a/drivers/net/cnic.c b/drivers/net/cnic.c
index 039900c..f8a0923 100644
--- a/drivers/net/cnic.c
+++ b/drivers/net/cnic.c
@@ -762,33 +762,14 @@ static int cnic_alloc_l2_rings(struct cnic_dev *dev, int pages)
return 0;
}
-static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
-{
+static int cnic_alloc_uio(struct cnic_dev *dev) {
struct cnic_local *cp = dev->cnic_priv;
struct uio_info *uinfo;
int ret;
- ret = cnic_alloc_dma(dev, &cp->kwq_info, KWQ_PAGE_CNT, 1);
- if (ret)
- goto error;
- cp->kwq = (struct kwqe **) cp->kwq_info.pg_arr;
-
- ret = cnic_alloc_dma(dev, &cp->kcq_info, KCQ_PAGE_CNT, 1);
- if (ret)
- goto error;
- cp->kcq = (struct kcqe **) cp->kcq_info.pg_arr;
-
- ret = cnic_alloc_context(dev);
- if (ret)
- goto error;
-
- ret = cnic_alloc_l2_rings(dev, 2);
- if (ret)
- goto error;
-
uinfo = kzalloc(sizeof(*uinfo), GFP_ATOMIC);
if (!uinfo)
- goto error;
+ return -ENOMEM;
uinfo->mem[0].addr = dev->netdev->base_addr;
uinfo->mem[0].internal_addr = dev->regview;
@@ -796,10 +777,15 @@ static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
uinfo->mem[0].memtype = UIO_MEM_PHYS;
uinfo->mem[1].addr = (unsigned long) cp->status_blk & PAGE_MASK;
- if (cp->ethdev->drv_state & CNIC_DRV_STATE_USING_MSIX)
- uinfo->mem[1].size = BNX2_SBLK_MSIX_ALIGN_SIZE * 9;
- else
- uinfo->mem[1].size = BNX2_SBLK_MSIX_ALIGN_SIZE;
+ if (test_bit(CNIC_F_BNX2_CLASS, &dev->flags)) {
+ if (cp->ethdev->drv_state & CNIC_DRV_STATE_USING_MSIX)
+ uinfo->mem[1].size = BNX2_SBLK_MSIX_ALIGN_SIZE * 9;
+ else
+ uinfo->mem[1].size = BNX2_SBLK_MSIX_ALIGN_SIZE;
+
+ uinfo->name = "bnx2_cnic";
+ }
+
uinfo->mem[1].memtype = UIO_MEM_LOGICAL;
uinfo->mem[2].addr = (unsigned long) cp->l2_ring;
@@ -810,7 +796,6 @@ static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
uinfo->mem[3].size = cp->l2_buf_size;
uinfo->mem[3].memtype = UIO_MEM_LOGICAL;
- uinfo->name = "bnx2_cnic";
uinfo->version = CNIC_MODULE_VERSION;
uinfo->irq = UIO_IRQ_CUSTOM;
@@ -822,10 +807,39 @@ static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
ret = uio_register_device(&dev->pcidev->dev, uinfo);
if (ret) {
kfree(uinfo);
- goto error;
+ return ret;
}
cp->cnic_uinfo = uinfo;
+ return 0;
+}
+
+static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
+{
+ struct cnic_local *cp = dev->cnic_priv;
+ int ret;
+
+ ret = cnic_alloc_dma(dev, &cp->kwq_info, KWQ_PAGE_CNT, 1);
+ if (ret)
+ goto error;
+ cp->kwq = (struct kwqe **) cp->kwq_info.pg_arr;
+
+ ret = cnic_alloc_dma(dev, &cp->kcq_info, KCQ_PAGE_CNT, 1);
+ if (ret)
+ goto error;
+ cp->kcq = (struct kcqe **) cp->kcq_info.pg_arr;
+
+ ret = cnic_alloc_context(dev);
+ if (ret)
+ goto error;
+
+ ret = cnic_alloc_l2_rings(dev, 2);
+ if (ret)
+ goto error;
+
+ ret = cnic_alloc_uio(dev);
+ if (ret)
+ goto error;
return 0;
--
1.6.4.GIT
^ permalink raw reply related
* [PATCH net-next 1/2] cnic: Put rx/tx ring allocation in separate function.
From: Michael Chan @ 2009-08-26 19:49 UTC (permalink / raw)
To: davem; +Cc: netdev, Michael Chan
This will allow the 10G iSCSI code to reuse the function.
Signed-off-by: Michael Chan <mchan@broadcom.com>
---
drivers/net/cnic.c | 33 ++++++++++++++++++++++-----------
1 files changed, 22 insertions(+), 11 deletions(-)
diff --git a/drivers/net/cnic.c b/drivers/net/cnic.c
index 4869d77..039900c 100644
--- a/drivers/net/cnic.c
+++ b/drivers/net/cnic.c
@@ -742,6 +742,26 @@ static int cnic_alloc_context(struct cnic_dev *dev)
return 0;
}
+static int cnic_alloc_l2_rings(struct cnic_dev *dev, int pages)
+{
+ struct cnic_local *cp = dev->cnic_priv;
+
+ cp->l2_ring_size = pages * BCM_PAGE_SIZE;
+ cp->l2_ring = pci_alloc_consistent(dev->pcidev, cp->l2_ring_size,
+ &cp->l2_ring_map);
+ if (!cp->l2_ring)
+ return -ENOMEM;
+
+ cp->l2_buf_size = (cp->l2_rx_ring_size + 1) * cp->l2_single_buf_size;
+ cp->l2_buf_size = PAGE_ALIGN(cp->l2_buf_size);
+ cp->l2_buf = pci_alloc_consistent(dev->pcidev, cp->l2_buf_size,
+ &cp->l2_buf_map);
+ if (!cp->l2_buf)
+ return -ENOMEM;
+
+ return 0;
+}
+
static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
{
struct cnic_local *cp = dev->cnic_priv;
@@ -762,17 +782,8 @@ static int cnic_alloc_bnx2_resc(struct cnic_dev *dev)
if (ret)
goto error;
- cp->l2_ring_size = 2 * BCM_PAGE_SIZE;
- cp->l2_ring = pci_alloc_consistent(dev->pcidev, cp->l2_ring_size,
- &cp->l2_ring_map);
- if (!cp->l2_ring)
- goto error;
-
- cp->l2_buf_size = (cp->l2_rx_ring_size + 1) * cp->l2_single_buf_size;
- cp->l2_buf_size = PAGE_ALIGN(cp->l2_buf_size);
- cp->l2_buf = pci_alloc_consistent(dev->pcidev, cp->l2_buf_size,
- &cp->l2_buf_map);
- if (!cp->l2_buf)
+ ret = cnic_alloc_l2_rings(dev, 2);
+ if (ret)
goto error;
uinfo = kzalloc(sizeof(*uinfo), GFP_ATOMIC);
--
1.6.4.GIT
^ permalink raw reply related
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: David Miller @ 2009-08-26 19:36 UTC (permalink / raw)
To: mingo; +Cc: nhorman, rostedt, fweisbec, billfink, netdev, brice, gallatin
In-Reply-To: <20090826190830.GF13632@elte.hu>
From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 26 Aug 2009 21:08:30 +0200
> Sigh, no. Please re-read the past discussions about this.
> trace_skb_sources.c is a hack and should be converted to generic
> tracepoints. Is there anything in it that cannot be expressed in
> terms of TRACE_EVENT()?
Neil explained why he needed to implement it this way in his
reply to Steven Rostedt. I attach it here for your
convenience.
Subject: Re: [PATCH 3/3] net: skb ftracer - Add actual ftrace code to kernel (v3)
From: Neil Horman <nhorman@tuxdriver.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: netdev@vger.kernel.org, davem@davemloft.net
Date: Tue, 18 Aug 2009 12:39:58 -0400
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Mew: tab/spc characters on Subject: are simplified.
On Mon, Aug 17, 2009 at 04:55:38PM -0400, Steven Rostedt wrote:
>
> Hi Neil!
>
> Sorry for the late reply, I've been on vacation for the last week.
>
> On Thu, 13 Aug 2009, Neil Horman wrote:
>
> > skb allocation / consumption correlator
> >
> > Add ftracer module to kernel to print out a list that correlates a process id,
> > an skb it read, and the numa nodes on wich the process was running when it was
> > read along with the numa node the skbuff was allocated on.
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> >
> >
> > Makefile | 1
> > trace.h | 19 ++++++
> > trace_skb_sources.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 3 files changed, 174 insertions(+)
> >
> > diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> > index 844164d..ee5e5b1 100644
> > --- a/kernel/trace/Makefile
> > +++ b/kernel/trace/Makefile
> > @@ -49,6 +49,7 @@ obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o
> > ifeq ($(CONFIG_BLOCK),y)
> > obj-$(CONFIG_EVENT_TRACING) += blktrace.o
> > endif
> > +obj-$(CONFIG_SKB_SOURCES_TRACER) += trace_skb_sources.o
> > obj-$(CONFIG_EVENT_TRACING) += trace_events.o
> > obj-$(CONFIG_EVENT_TRACING) += trace_export.o
> > obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
> > diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
> > index 8b9f4f6..8a6281b 100644
> > --- a/kernel/trace/trace.h
> > +++ b/kernel/trace/trace.h
> > @@ -11,6 +11,7 @@
> > #include <trace/boot.h>
> > #include <linux/kmemtrace.h>
> > #include <trace/power.h>
> > +#include <trace/events/skb.h>
> >
> > #include <linux/trace_seq.h>
> > #include <linux/ftrace_event.h>
> > @@ -40,6 +41,7 @@ enum trace_type {
> > TRACE_KMEM_FREE,
> > TRACE_POWER,
> > TRACE_BLK,
> > + TRACE_SKB_SOURCE,
> >
> > __TRACE_LAST_TYPE,
> > };
> > @@ -171,6 +173,21 @@ struct trace_power {
> > struct power_trace state_data;
> > };
> >
> > +struct skb_record {
> > + pid_t pid; /* pid of the copying process */
> > + int anid; /* node where skb was allocated */
> > + int cnid; /* node to which skb was copied in userspace */
> > + char ifname[IFNAMSIZ]; /* Name of the receiving interface */
> > + int rx_queue; /* The rx queue the skb was received on */
> > + int ccpu; /* Cpu the application got this frame from */
> > + int len; /* length of the data copied */
> > +};
> > +
> > +struct trace_skb_event {
> > + struct trace_entry ent;
> > + struct skb_record event_data;
> > +};
> > +
> > enum kmemtrace_type_id {
> > KMEMTRACE_TYPE_KMALLOC = 0, /* kmalloc() or kfree(). */
> > KMEMTRACE_TYPE_CACHE, /* kmem_cache_*(). */
> > @@ -323,6 +340,8 @@ extern void __ftrace_bad_type(void);
> > TRACE_SYSCALL_ENTER); \
> > IF_ASSIGN(var, ent, struct syscall_trace_exit, \
> > TRACE_SYSCALL_EXIT); \
> > + IF_ASSIGN(var, ent, struct trace_skb_event, \
> > + TRACE_SKB_SOURCE); \
> > __ftrace_bad_type(); \
> > } while (0)
> >
> > diff --git a/kernel/trace/trace_skb_sources.c b/kernel/trace/trace_skb_sources.c
> > new file mode 100644
> > index 0000000..4ba3671
> > --- /dev/null
> > +++ b/kernel/trace/trace_skb_sources.c
> > @@ -0,0 +1,154 @@
> > +/*
> > + * ring buffer based tracer for analyzing per-socket skb sources
> > + *
> > + * Neil Horman <nhorman@tuxdriver.com>
> > + * Copyright (C) 2009
> > + *
> > + *
> > + */
> > +
> > +#include <linux/init.h>
> > +#include <linux/debugfs.h>
> > +#include <trace/events/skb.h>
> > +#include <linux/kallsyms.h>
> > +#include <linux/module.h>
> > +#include <linux/hardirq.h>
> > +#include <linux/netdevice.h>
> > +#include <net/sock.h>
> > +
> > +#include "trace.h"
> > +#include "trace_output.h"
> > +
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(skb_copy_datagram_iovec);
> > +
> > +static struct trace_array *skb_trace;
> > +static int __read_mostly trace_skb_source_enabled;
> > +
> > +static void probe_skb_dequeue(const struct sk_buff *skb, int len)
> > +{
> > + struct ring_buffer_event *event;
> > + struct trace_skb_event *entry;
> > + struct trace_array *tr = skb_trace;
> > + struct net_device *dev;
> > +
> > + if (!trace_skb_source_enabled)
> > + return;
> > +
> > + if (in_interrupt())
> > + return;
>
> Is there a reason for not doing this in an interrupt?
>
Because the idea is to correlate skb consumption to a process. If we get in
this tracepoint in an interrupt, it doesn't make sense to record.
> > +
> > + event = trace_buffer_lock_reserve(tr, TRACE_SKB_SOURCE,
> > + sizeof(*entry), 0, 0);
> > + if (!event)
> > + return;
> > + entry = ring_buffer_event_data(event);
> > +
> > + entry->event_data.pid = current->pid;
>
> Note, the trace_buffer_lock_reserve will record the current pid, thus you
> do not need to record it here.
>
> > + entry->event_data.anid = page_to_nid(virt_to_page(skb->data));
> > + entry->event_data.cnid = cpu_to_node(smp_processor_id());
> > + entry->event_data.len = len;
> > + entry->event_data.rx_queue = skb->queue_mapping;
> > + entry->event_data.ccpu = smp_processor_id();
>
> Also, the cpu is recorded in the ring buffer. They are per cpu ring
> buffers and that determines the cpu it was recorded on.
>
> > +
> > + dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
> > + if (dev) {
> > + memcpy(entry->event_data.ifname, dev->name, IFNAMSIZ);
> > + dev_put(dev);
> > + } else {
> > + strcpy(entry->event_data.ifname, "Unknown");
> > + }
> > +
> > + trace_buffer_unlock_commit(tr, event, 0, 0);
> > +}
> > +
> > +static int tracing_skb_source_register(void)
> > +{
> > + int ret;
> > +
> > + ret = register_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
> > + if (ret)
> > + pr_info("skb source trace: Couldn't activate dequeue tracepoint");
> > +
> > + return ret;
> > +}
> > +
> > +static void start_skb_source_trace(struct trace_array *tr)
> > +{
> > + trace_skb_source_enabled = 1;
> > +}
> > +
> > +static void stop_skb_source_trace(struct trace_array *tr)
> > +{
> > + trace_skb_source_enabled = 0;
> > +}
> > +
> > +static void skb_source_trace_reset(struct trace_array *tr)
> > +{
> > + trace_skb_source_enabled = 0;
> > + unregister_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
> > +}
> > +
> > +
> > +static int skb_source_trace_init(struct trace_array *tr)
> > +{
> > + int cpu;
> > + skb_trace = tr;
> > +
> > + trace_skb_source_enabled = 1;
> > + tracing_skb_source_register();
> > +
> > + for_each_cpu(cpu, cpu_possible_mask)
> > + tracing_reset(tr, cpu);
> > + return 0;
> > +}
> > +
> > +static enum print_line_t skb_source_print_line(struct trace_iterator *iter)
> > +{
> > + int ret = 0;
> > + struct trace_entry *entry = iter->ent;
>
> iter->cpu has the cpu that trace was recorded on.
> entry->pid has the pid of the process that did the recording.
>
ok, I'll clean this up in a subsequent patch, since davem has already rolled
them in.
> > + struct trace_skb_event *event;
> > + struct skb_record *record;
> > + struct trace_seq *s = &iter->seq;
> > +
> > + trace_assign_type(event, entry);
> > + record = &event->event_data;
> > + if (entry->type != TRACE_SKB_SOURCE)
> > + return TRACE_TYPE_UNHANDLED;
> > +
> > + ret = trace_seq_printf(s, " %d %d %d %s %d %d %d\n",
> > + record->pid,
> > + record->anid,
> > + record->cnid,
> > + record->ifname,
> > + record->rx_queue,
> > + record->ccpu,
> > + record->len);
> > +
> > + if (!ret)
> > + return TRACE_TYPE_PARTIAL_LINE;
> > +
> > + return TRACE_TYPE_HANDLED;
> > +}
> > +
> > +static void skb_source_print_header(struct seq_file *s)
> > +{
> > + seq_puts(s, "# PID ANID CNID IFC RXQ CCPU LEN\n");
> > + seq_puts(s, "# | | | | | | |\n");
> > +}
> > +
> > +static struct tracer skb_source_tracer __read_mostly =
> > +{
> > + .name = "skb_sources",
> > + .init = skb_source_trace_init,
> > + .start = start_skb_source_trace,
> > + .stop = stop_skb_source_trace,
> > + .reset = skb_source_trace_reset,
> > + .print_line = skb_source_print_line,
> > + .print_header = skb_source_print_header,
> > +};
> > +
> > +static int init_skb_source_trace(void)
> > +{
> > + return register_tracer(&skb_source_tracer);
> > +}
> > +device_initcall(init_skb_source_trace);
> >
>
> BTW, why not just do this as events? Or was this just a easy way to
> communicate with the user space tools?
>
Thats exactly why I did it. the idea is for me to now write a user space tool
that lets me analyze the events and ajust process scheduling to optimize the rx
path.
Neil
> -- Steve
>
>
^ permalink raw reply
* Re: AlacrityVM benchmark numbers updated
From: Avi Kivity @ 2009-08-26 19:23 UTC (permalink / raw)
To: Gregory Haskins
Cc: alacrityvm-devel, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, Michael S. Tsirkin, netdev
In-Reply-To: <4A95822D.9060207@gmail.com>
On 08/26/2009 09:42 PM, Gregory Haskins wrote:
> Actually, I have already look at this and it does indeed seem better to
> use switch_mm+gupf() over gup() by quite a large margin. You could then
> couple that with your DMA-engine idea to potentially gain even more
> benefits (though probably not for networking since most NICs have their
> own DMA engine anyway).
>
>
For tx, we'll just go copyless once we plumb the destructors properly.
But for rx on a shared interface it is impossible to avoid the copy.
You can only choose if you want it done by the cpu or a local dma engine.
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply
* [PATCH] [resend] Remove useless condition: max_bonds is of type int and cannot be greater than INT_MAX.
From: Nicolas de Pesloüan @ 2009-08-26 19:22 UTC (permalink / raw)
To: David Miller, Jay Vosburgh; +Cc: bonding-devel, netdev
Remove useless condition: max_bonds is of type int and cannot be greater than
INT_MAX.
Signed-off-by: Nicolas de Pesloüan <nicolas.2p.debian@free.fr>
---
drivers/net/bonding/bond_main.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 4798d30..edd71a1 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -4755,7 +4755,7 @@ static int bond_check_params(struct bond_params *params)
params->ad_select = BOND_AD_STABLE;
}
- if (max_bonds < 0 || max_bonds > INT_MAX) {
+ if (max_bonds < 0) {
pr_warning(DRV_NAME
": Warning: max_bonds (%d) not in range %d-%d, so it "
"was reset to BOND_DEFAULT_MAX_BONDS (%d)\n",
^ permalink raw reply related
* Re: Page allocation failures in guest
From: David Miller @ 2009-08-26 19:22 UTC (permalink / raw)
To: rusty
Cc: drzeus-list, avi, minchan.kim, kvm, linux-kernel, linux-mm,
fengguang.wu, kosaki.motohiro, riel, netdev
In-Reply-To: <200908262148.59664.rusty@rustcorp.com.au>
From: Rusty Russell <rusty@rustcorp.com.au>
Date: Wed, 26 Aug 2009 21:48:58 +0930
> Dave, can you push this to Linus ASAP?
Ok.
> Subject: virtio: net refill on out-of-memory
>
> If we run out of memory, use keventd to fill the buffer. There's a
> report of this happening: "Page allocation failures in guest",
> Message-ID: <20090713115158.0a4892b0@mjolnir.ossman.eu>
>
> Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Applied, thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-26 19:09 UTC (permalink / raw)
To: Sridhar Samudrala
Cc: David Stevens, David S. Miller, Eric Dumazet, netdev, niv
In-Reply-To: <1251309040.10599.34.camel@w-sridhar.beaverton.ibm.com>
On Wed, 26 Aug 2009, Sridhar Samudrala wrote:
> > They are reported for IP and UDP.
> Not clear what you meant by this.
The SNMP and UDP statistics show the loss. qdisc level does not show the
loss.
> > root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show
> > qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> > 1 1 1 1
> > Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0)
> > rate 0bit 0pps backlog 0b 0p requeues 0
>
> Even the Sent count seems to be too low. Are you looking at the right
> device?
I would think that tc displays all queues? It says eth0 and eth0 is the
device that we sent the data out on.
> So based on the current analysis, the packets are getting dropped after
> the call to ip_local_out() in ip_push_pending_frames(). ip_local_out()
> is failing with NET_XMIT_DROP. But we are not sure where they are
> getting dropped. Is that right?
ip_local_out is returning ENOBUFS. Something at the qdisc layer is
dropping the packet and not incrementing counters.
> I think we need to figure out where they are getting dropped and then
> decide on the appropriate counter to be incremented.
Right. Where in the qdisc layer do drops occur?
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 19:08 UTC (permalink / raw)
To: Neil Horman, David S. Miller, Steven Rostedt,
=?unknown-8bit?B?RnLDqWTDqXJpYw==?= Weisbecker
Cc: Bill Fink, Linux Network Developers, brice, gallatin
In-Reply-To: <20090826190435.GC10816@hmsreliant.think-freely.org>
* Neil Horman <nhorman@tuxdriver.com> wrote:
> On Wed, Aug 26, 2009 at 08:15:02PM +0200, Ingo Molnar wrote:
> >
> > * Neil Horman <nhorman@tuxdriver.com> wrote:
> >
> > > On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> > > > On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > > > > On Fri, 21 Aug 2009, Neil Horman wrote:
> > > > >
> > > > > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > > > > >
> > > > > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > > > > >
> > > > > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > > > > and never produced a crashdump. I tried this several times and
> > > > > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > > > > worked just fine.
> > > > > > > >
> > > > > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > > > > >
> > > > > > > There was a backtrace on the screen but I didn't have a chance to
> > > > > > > record it. BTW did anyone ever think to print the backtrace in
> > > > > > > reverse (first to some reserved memory and then output to the display)
> > > > > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > > > > the screen?
> > > > > > >
> > > > > > The real solution is to use a console to which the output doesn't scroll off the
> > > > > > screen. Normally people use a serial console they can log, or a RAC card that
> > > > > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > > > > vt iirc to allow for scrolling.
> > > > >
> > > > > None of our Asus P6T6 systems have serial consoles. I don't know of
> > > > > any RAC cards for them either, nor are there spare PCI slots available
> > > > > in many cases. I wouldn't think the Shift-PageUp trick would work
> > > > > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > > > > out netconsole yet either, but I'm not sure it would help either in a
> > > > > case like this that was a network related kernel crash.
> > > > >
> > > > Any USB ports that you can attach a serial dongle to? That would work as well,
> > > > or, as previously mentioned, netconsole also does the trick.
> > > >
> > > > > In any case, a simple kernel command line that would provide a reversed
> > > > > backtrace would be a simple thing to facilitate Linux users providing
> > > > > useful info to Linux kernel developers in helping to debug kernel
> > > > > problems. The most useful info would still be on the screen, so it
> > > > > could be transcribed or a photo image of the screen could be taken.
> > > > >
> > > > I understand what your saying, I'm just saying there are currently several
> > > > options for you that have already solved this problem in differnt ways.
> > > >
> > > > > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > > > > does have a serial console, and after a fair amount of effort I was
> > > > > able to get it to work as desired, and was able to finally capture
> > > > > a backtrace of the kernel oops. BTW I believe the reason the
> > > > > kexec/kdump didn't work was probably because it couldn't find
> > > > > a /proc/vmcore file, although I don't know why that would be,
> > > > > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > > > > up normally if it fails to find the /proc/vmcore file (or it's
> > > > > zero size).
> > > > >
> > > > I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> > > > happy to look into it further.
> > > >
> > > > > The following shows a simple ping test usage of the skb_sources
> > > > > tracing feature:
> > > > >
> > > > > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > > > > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> > > > >
> > > > > --- 192.168.1.10 ping statistics ---
> > > > > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > > > > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> > > > >
> > > > > [root@xeontest1 tracing]# cat trace
> > > > > # tracer: skb_sources
> > > > > #
> > > > > # PID ANID CNID IFC RXQ CCPU LEN
> > > > > # | | | | | | |
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > >
> > > > > All is as was expected.
> > > > >
> > > > > But if I try an actual nuttcp performance test (even rate limited
> > > > > to 1 Mbps), I get the following kernel oops:
> > > > >
> > > > thank you, I think I see the problem, I'll have a patch for you in just a bit
> > > >
> > > > Thanks
> > > > Neil
> > > >
> > > > > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > > > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > > > > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > PGD 337d12067 PUD 337d11067 PMD 0
> > > > > Oops: 0000 [#1] SMP
> > > > > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > > > > CPU 4
> > > > > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > > > > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > > > > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > > > > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > > > > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > > > > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > > > > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > > > > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > > > > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > > > > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > > > > Stack:
> > > > > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > > > > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > > > > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > > > > Call Trace:
> > > > > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > > > > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > > > > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > > > > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > > > > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > > > > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > > > > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > > > > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > > > > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > > > > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > > > > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > > > > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > > > > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > > > > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > > > > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > > > > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > > > > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > > > > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > > > > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > RSP <ffff8801a5811a88>
> > > > > CR2: 0000000000000038
> > > > >
> > > > > -Thanks
> > > > >
> > > > > -Bill
> > > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > >
> > > Here you go, I think this will fix your oops.
> > >
> > >
> > > Fix NULL pointer deref in skb sources ftracer
> > >
> > > Its possible that skb->sk will be null in this path, so we shouldn't just assume
> > > we can pass it to sock_net
> > >
> > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > >
> > > trace_skb_sources.c | 6 ++++--
> > > 1 file changed, 4 insertions(+), 2 deletions(-)
> >
> > ok if this is just a temporary fix until TRACE_EVENT() is done, but
> > we'll get rid of this and do TRACE_EVENT() before net-next-2.6 it's
> > pushed to .32, right?
>
> Not sure that the two are related. I think you meant to send this
> to the other thread, didnt you?
Sigh, no. Please re-read the past discussions about this.
trace_skb_sources.c is a hack and should be converted to generic
tracepoints. Is there anything in it that cannot be expressed in
terms of TRACE_EVENT()?
Ingo
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox