* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Gregory Haskins @ 2009-08-17 19:35 UTC (permalink / raw)
To: Ingo Molnar
Cc: Avi Kivity, Gregory Haskins, kvm, alacrityvm-devel, linux-kernel,
netdev
In-Reply-To: <20090817151449.GA19384@elte.hu>
[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]
Ingo Molnar wrote:
> * Gregory Haskins <gregory.haskins@gmail.com> wrote:
>
>> Avi Kivity wrote:
>>> On 08/17/2009 05:16 PM, Gregory Haskins wrote:
>>>>> My opinion is that this is a duplication of effort and we'd be better
>>>>> off if everyone contributed to enhancing virtio, which already has
>>>>> widely deployed guest drivers and non-Linux guest support.
>>>>>
>>>>> It may have merit if it is proven that it is technically superior to
>>>>> virtio (and I don't mean some benchmark in some point in time; I mean
>>>>> design wise). So far I haven't seen any indications that it is.
>>>>>
>>>>>
>>>> The design is very different, so hopefully I can start to convince you
>>>> why it might be interesting.
>>>>
>>> We've been through this before I believe. If you can point out
>>> specific differences that make venet outperform virtio-net I'll
>>> be glad to hear (and steal) them though.
>> You sure know how to convince someone to collaborate with you, eh?
>>
>> Unforunately, i've answered that question numerous times, but it
>> apparently falls on deaf ears.
>
> I'm trying to find the relevant discussion. The link you gave in the
> previous mail:
>
> http://lkml.org/lkml/2009/4/21/408
>
> does not offer any design analysis of vbus versus virtio, and why
> the only fix to virtio is vbus. It offers a comparison and a blanket
> statement that vbus is superior but no arguments.
>
> (If you've already explained in a past thread then please give me an
> URL to that reply if possible, or forward me that prior reply.
> Thanks!)
Sorry, it was a series of long threads from quite a while back. I will
see if I can find some references, but it might be easier to just start
fresh (see the last reply I sent).
Kind Regards,
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-17 20:01 UTC (permalink / raw)
To: netdev; +Cc: Eric Dumazet
If I use a large send queue
(echo 2000000 >/proc/sys/net/core/wmem_default)
then lots of packet loss results if I try to send more packets than
the bandwidth of the wire (300 bytes the maximum rate is 341k pps). With a
size of 2M for the output buffer the TX ring is overrun.
But this loss is nowhere seen in any counter increments:
Receiver: Listening to control channel 239.0.192.1
Receiver: Subscribing to 1 MC addresses 239.0.192-254.2-254 offset 0
origin 10.2.36.111
TotalMsg Lost SeqErr Msg/Sec Min/us Avg/us Max/us StdDev Kbytes Idle Samples
3118667 532979 25341 308774 1734.96 1784.72 1904.04 44.11 0.0 0 10
3415674 585146 27642 341569 1788.90 1858.40 1905.43 34.32 102470.4 0 10
3449840 591012 27844 341568 1941.12 1987.45 2040.16 32.73 102470.4 0 9
3449819 591093 27993 341567 2024.34 2036.00 2044.24 6.48 102470.0 0 5
3415693 585268 27633 341568 2010.57 2017.84 2025.10 7.27 102470.4 0 2
ifconfig
eth0 Link encap:Ethernet HWaddr 00:21:9b:8f:a1:40
inet addr:10.2.36.110 Bcast:10.2.36.255 Mask:255.255.255.0
inet6 addr: fe80::221:9bff:fe8f:a140/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:174716487 errors:0 dropped:0 overruns:0 frame:0
TX packets:1379 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:60451720777 (56.3 GiB) TX bytes:225841 (220.5 KiB)
Interrupt:36 Memory:d6000000-d6012800
If I then reduce the queue size to 20k
(echo 20000 >/proc/sys/net/core/wmem_default)
the loss does either not occur or I see corresponding loss in the switch
stats:
clameter@rd-strategy3:~$ bin/mcast
Receiver: Listening to control channel 239.0.192.1
Receiver: Subscribing to 1 MC addresses 239.0.192-254.2-254 offset 0
origin 10.2.36.111
TotalMsg Lost SeqErr Msg/Sec Min/us Avg/us Max/us StdDev Kbytes
Idle Samples
3126997 0 0 309598 109.57 191.69 404.73 79.55 0.0 0 10
3449790 5 5 341566 238.60 324.98 407.74 57.31 102469.6 0 9
3449843 94 94 341569 412.31 412.32 412.33 0.01 102470.4 0 2
3449859 76 76 341569 379.78 399.74 419.65 16.28 102470.6 0 3
3415644 92 92 341569 408.63 414.63 417.93 4.25 102470.5 0 3
3413885 3 3 341386 93.53 145.58 207.28 36.85 102415.8 0 10
3449832 0 0 341568 207.06 282.44 348.13 45.80 102470.4 0 10
The TX ring is not overrun in that case since the queue size reduces the
maximum objects in flight to below 255 (TX ring size of broadcom). So the
application is throttled.
It seems that Linux does not report the UDP packet loss above that is due
to overrunning the TX ring? Why is that?
^ permalink raw reply
* Re: [PATCH v3 3/6] vbus: add a "vbus-proxy" bus model for vbus_driver objects
From: Gregory Haskins @ 2009-08-17 20:17 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Anthony Liguori, Ingo Molnar, Gregory Haskins, kvm, Avi Kivity,
alacrityvm-devel, linux-kernel, netdev
In-Reply-To: <20090817174142.GA11140@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3195 bytes --]
Michael S. Tsirkin wrote:
> On Mon, Aug 17, 2009 at 10:14:56AM -0400, Gregory Haskins wrote:
>> Case in point: Take an upstream kernel and you can modprobe the
>> vbus-pcibridge in and virtio devices will work over that transport
>> unmodified.
>>
>> See http://lkml.org/lkml/2009/8/6/244 for details.
>
> The modprobe you are talking about would need
> to be done in guest kernel, correct?
Yes, and your point is? "unmodified" (pardon the psuedo pun) modifies
"virtio", not "guest". It means you can take an off-the-shelf kernel
with off-the-shelf virtio (ala distro-kernel) and modprobe
vbus-pcibridge and get alacrityvm acceleration. It is not a design goal
of mine to forbid the loading of a new driver, so I am ok with that
requirement.
>
>> OTOH, Michael's patch is purely targeted at improving virtio-net on kvm,
>> and its likewise constrained by various limitations of that decision
>> (such as its reliance of the PCI model, and the kvm memory scheme).
>
> vhost is actually not related to PCI in any way. It simply leaves all
> setup for userspace to do. And the memory scheme was intentionally
> separated from kvm so that it can easily support e.g. lguest.
>
I think you have missed my point. I mean that vhost requires a separate
bus-model (ala qemu-pci). And no, your memory scheme is not separated,
at least, not very well. It still assumes memory-regions and
copy_to_user(), which is very kvm-esque. Vbus has people using things
like userspace containers (no regions), and physical hardware (dma
controllers, so no regions or copy_to_user) so your scheme quickly falls
apart once you get away from KVM.
Don't get me wrong: That design may have its place. Perhaps you only
care about fixing KVM, which is a perfectly acceptable strategy. Its
just not a strategy that I think is the best approach. Essentially you
are promoting the proliferation of competing backends, and I am trying
to unify them (which is ironic that this thread started with concerns I
was fragmenting things ;).
The bottom line is, you have a simpler solution that is more finely
targeted at KVM and virtio-networking. It fixes probably a lot of
problems with the existing implementation, but it still has limitations.
OTOH, what I am promoting is more complex, but more flexible. That is
the tradeoff. You can't have both ;) So do not for one second think
that what you implemented is equivalent, because they are not.
In fact, I believe I warned you about this potential problem when you
decided to implement your own version. I think I said something to the
effect of "you will either have a subset of functionality, or you will
ultimately reinvent what I did". Right now you are in the subset phase.
Perhaps someday you will be in the complete-reinvent phase. Why you
wanted to go that route when I had already worked though the issues is
something perhaps only you will ever know, but I'm sure you had your
reasons. But do note you could have saved yourself grief by reusing my
already implemented and tested variant, as I politely offered to work
with you on making it meet your needs.
Kind Regards
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* [PATCH -next] phonet: fix build when PROC_FS is disabled
From: Randy Dunlap @ 2009-08-17 20:35 UTC (permalink / raw)
To: Linux Kernel Mailing List, linux-next, Netdev; +Cc: David Miller, Andrew Morton
From: Randy Dunlap <randy.dunlap@oracle.com>
Fix phonet build when PROC_FS is not enabled:
net/built-in.o: In function `pn_sock_open':
socket.c:(.text+0x23c649): undefined reference to `seq_open_net'
net/built-in.o:(.rodata+0x21018): undefined reference to `seq_release_net'
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
---
net/phonet/socket.c | 2 ++
1 file changed, 2 insertions(+)
--- linux-next-20090817.orig/net/phonet/socket.c
+++ linux-next-20090817/net/phonet/socket.c
@@ -413,6 +413,7 @@ found:
}
EXPORT_SYMBOL(pn_sock_get_port);
+#ifdef CONFIG_PROC_FS
static struct sock *pn_sock_get_idx(struct seq_file *seq, loff_t pos)
{
struct net *net = seq_file_net(seq);
@@ -509,3 +510,4 @@ const struct file_operations pn_sock_seq
.llseek = seq_lseek,
.release = seq_release_net,
};
+#endif
--
~Randy
LPC 2009, Sept. 23-25, Portland, Oregon
http://linuxplumbersconf.org/2009/
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Nivedita Singhvi @ 2009-08-17 20:40 UTC (permalink / raw)
To: Christoph Lameter; +Cc: netdev, Eric Dumazet
In-Reply-To: <alpine.DEB.1.10.0908171548040.20399@gentwo.org>
Christoph Lameter wrote:
> If I use a large send queue
>
> (echo 2000000 >/proc/sys/net/core/wmem_default)
>
> then lots of packet loss results if I try to send more packets than
> the bandwidth of the wire (300 bytes the maximum rate is 341k pps). With a
> size of 2M for the output buffer the TX ring is overrun.
>
> But this loss is nowhere seen in any counter increments:
Is there any chance this is getting dropped at qdisc?
Can you check with tc?
thanks,
Nivedita
^ permalink raw reply
* Re: [PATCH -next] phonet: fix build when PROC_FS is disabled
From: Rémi Denis-Courmont @ 2009-08-17 20:43 UTC (permalink / raw)
To: Randy Dunlap
Cc: Linux Kernel Mailing List, linux-next, Netdev, David Miller,
Andrew Morton
In-Reply-To: <4A89BF25.9010404@oracle.com>
Le lundi 17 août 2009 23:35:49 Randy Dunlap, vous avez écrit :
> From: Randy Dunlap <randy.dunlap@oracle.com>
>
> Fix phonet build when PROC_FS is not enabled:
>
> net/built-in.o: In function `pn_sock_open':
> socket.c:(.text+0x23c649): undefined reference to `seq_open_net'
> net/built-in.o:(.rodata+0x21018): undefined reference to `seq_release_net'
>
> Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
Acked-by: Rémi Denis-Courmont <remi.denis-courmont@nokia.com>
--
Rémi Denis-Courmont
http://www.remlab.net/
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-17 20:46 UTC (permalink / raw)
To: Nivedita Singhvi; +Cc: netdev, Eric Dumazet
In-Reply-To: <4A89C026.4030402@us.ibm.com>
On Mon, 17 Aug 2009, Nivedita Singhvi wrote:
> Is there any chance this is getting dropped at qdisc?
> Can you check with tc?
I checked and the counter there is zero.
:/home/clameter# tc -s qdisc show
qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
1 1 1 1 1
Sent 3042712708 bytes 8896862 pkt (dropped 0, overlimits 0 requeues 56303)
rate 0bit 0pps backlog 0b 0p requeues 56303
^ permalink raw reply
* Re: [PATCH 3/3] net: skb ftracer - Add actual ftrace code to kernel (v3)
From: Steven Rostedt @ 2009-08-17 20:55 UTC (permalink / raw)
To: Neil Horman; +Cc: netdev, davem
In-Reply-To: <20090813152356.GC16682@hmsreliant.think-freely.org>
Hi Neil!
Sorry for the late reply, I've been on vacation for the last week.
On Thu, 13 Aug 2009, Neil Horman wrote:
> skb allocation / consumption correlator
>
> Add ftracer module to kernel to print out a list that correlates a process id,
> an skb it read, and the numa nodes on wich the process was running when it was
> read along with the numa node the skbuff was allocated on.
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>
>
> Makefile | 1
> trace.h | 19 ++++++
> trace_skb_sources.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 174 insertions(+)
>
> diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> index 844164d..ee5e5b1 100644
> --- a/kernel/trace/Makefile
> +++ b/kernel/trace/Makefile
> @@ -49,6 +49,7 @@ obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o
> ifeq ($(CONFIG_BLOCK),y)
> obj-$(CONFIG_EVENT_TRACING) += blktrace.o
> endif
> +obj-$(CONFIG_SKB_SOURCES_TRACER) += trace_skb_sources.o
> obj-$(CONFIG_EVENT_TRACING) += trace_events.o
> obj-$(CONFIG_EVENT_TRACING) += trace_export.o
> obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
> diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
> index 8b9f4f6..8a6281b 100644
> --- a/kernel/trace/trace.h
> +++ b/kernel/trace/trace.h
> @@ -11,6 +11,7 @@
> #include <trace/boot.h>
> #include <linux/kmemtrace.h>
> #include <trace/power.h>
> +#include <trace/events/skb.h>
>
> #include <linux/trace_seq.h>
> #include <linux/ftrace_event.h>
> @@ -40,6 +41,7 @@ enum trace_type {
> TRACE_KMEM_FREE,
> TRACE_POWER,
> TRACE_BLK,
> + TRACE_SKB_SOURCE,
>
> __TRACE_LAST_TYPE,
> };
> @@ -171,6 +173,21 @@ struct trace_power {
> struct power_trace state_data;
> };
>
> +struct skb_record {
> + pid_t pid; /* pid of the copying process */
> + int anid; /* node where skb was allocated */
> + int cnid; /* node to which skb was copied in userspace */
> + char ifname[IFNAMSIZ]; /* Name of the receiving interface */
> + int rx_queue; /* The rx queue the skb was received on */
> + int ccpu; /* Cpu the application got this frame from */
> + int len; /* length of the data copied */
> +};
> +
> +struct trace_skb_event {
> + struct trace_entry ent;
> + struct skb_record event_data;
> +};
> +
> enum kmemtrace_type_id {
> KMEMTRACE_TYPE_KMALLOC = 0, /* kmalloc() or kfree(). */
> KMEMTRACE_TYPE_CACHE, /* kmem_cache_*(). */
> @@ -323,6 +340,8 @@ extern void __ftrace_bad_type(void);
> TRACE_SYSCALL_ENTER); \
> IF_ASSIGN(var, ent, struct syscall_trace_exit, \
> TRACE_SYSCALL_EXIT); \
> + IF_ASSIGN(var, ent, struct trace_skb_event, \
> + TRACE_SKB_SOURCE); \
> __ftrace_bad_type(); \
> } while (0)
>
> diff --git a/kernel/trace/trace_skb_sources.c b/kernel/trace/trace_skb_sources.c
> new file mode 100644
> index 0000000..4ba3671
> --- /dev/null
> +++ b/kernel/trace/trace_skb_sources.c
> @@ -0,0 +1,154 @@
> +/*
> + * ring buffer based tracer for analyzing per-socket skb sources
> + *
> + * Neil Horman <nhorman@tuxdriver.com>
> + * Copyright (C) 2009
> + *
> + *
> + */
> +
> +#include <linux/init.h>
> +#include <linux/debugfs.h>
> +#include <trace/events/skb.h>
> +#include <linux/kallsyms.h>
> +#include <linux/module.h>
> +#include <linux/hardirq.h>
> +#include <linux/netdevice.h>
> +#include <net/sock.h>
> +
> +#include "trace.h"
> +#include "trace_output.h"
> +
> +EXPORT_TRACEPOINT_SYMBOL_GPL(skb_copy_datagram_iovec);
> +
> +static struct trace_array *skb_trace;
> +static int __read_mostly trace_skb_source_enabled;
> +
> +static void probe_skb_dequeue(const struct sk_buff *skb, int len)
> +{
> + struct ring_buffer_event *event;
> + struct trace_skb_event *entry;
> + struct trace_array *tr = skb_trace;
> + struct net_device *dev;
> +
> + if (!trace_skb_source_enabled)
> + return;
> +
> + if (in_interrupt())
> + return;
Is there a reason for not doing this in an interrupt?
> +
> + event = trace_buffer_lock_reserve(tr, TRACE_SKB_SOURCE,
> + sizeof(*entry), 0, 0);
> + if (!event)
> + return;
> + entry = ring_buffer_event_data(event);
> +
> + entry->event_data.pid = current->pid;
Note, the trace_buffer_lock_reserve will record the current pid, thus you
do not need to record it here.
> + entry->event_data.anid = page_to_nid(virt_to_page(skb->data));
> + entry->event_data.cnid = cpu_to_node(smp_processor_id());
> + entry->event_data.len = len;
> + entry->event_data.rx_queue = skb->queue_mapping;
> + entry->event_data.ccpu = smp_processor_id();
Also, the cpu is recorded in the ring buffer. They are per cpu ring
buffers and that determines the cpu it was recorded on.
> +
> + dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
> + if (dev) {
> + memcpy(entry->event_data.ifname, dev->name, IFNAMSIZ);
> + dev_put(dev);
> + } else {
> + strcpy(entry->event_data.ifname, "Unknown");
> + }
> +
> + trace_buffer_unlock_commit(tr, event, 0, 0);
> +}
> +
> +static int tracing_skb_source_register(void)
> +{
> + int ret;
> +
> + ret = register_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
> + if (ret)
> + pr_info("skb source trace: Couldn't activate dequeue tracepoint");
> +
> + return ret;
> +}
> +
> +static void start_skb_source_trace(struct trace_array *tr)
> +{
> + trace_skb_source_enabled = 1;
> +}
> +
> +static void stop_skb_source_trace(struct trace_array *tr)
> +{
> + trace_skb_source_enabled = 0;
> +}
> +
> +static void skb_source_trace_reset(struct trace_array *tr)
> +{
> + trace_skb_source_enabled = 0;
> + unregister_trace_skb_copy_datagram_iovec(probe_skb_dequeue);
> +}
> +
> +
> +static int skb_source_trace_init(struct trace_array *tr)
> +{
> + int cpu;
> + skb_trace = tr;
> +
> + trace_skb_source_enabled = 1;
> + tracing_skb_source_register();
> +
> + for_each_cpu(cpu, cpu_possible_mask)
> + tracing_reset(tr, cpu);
> + return 0;
> +}
> +
> +static enum print_line_t skb_source_print_line(struct trace_iterator *iter)
> +{
> + int ret = 0;
> + struct trace_entry *entry = iter->ent;
iter->cpu has the cpu that trace was recorded on.
entry->pid has the pid of the process that did the recording.
> + struct trace_skb_event *event;
> + struct skb_record *record;
> + struct trace_seq *s = &iter->seq;
> +
> + trace_assign_type(event, entry);
> + record = &event->event_data;
> + if (entry->type != TRACE_SKB_SOURCE)
> + return TRACE_TYPE_UNHANDLED;
> +
> + ret = trace_seq_printf(s, " %d %d %d %s %d %d %d\n",
> + record->pid,
> + record->anid,
> + record->cnid,
> + record->ifname,
> + record->rx_queue,
> + record->ccpu,
> + record->len);
> +
> + if (!ret)
> + return TRACE_TYPE_PARTIAL_LINE;
> +
> + return TRACE_TYPE_HANDLED;
> +}
> +
> +static void skb_source_print_header(struct seq_file *s)
> +{
> + seq_puts(s, "# PID ANID CNID IFC RXQ CCPU LEN\n");
> + seq_puts(s, "# | | | | | | |\n");
> +}
> +
> +static struct tracer skb_source_tracer __read_mostly =
> +{
> + .name = "skb_sources",
> + .init = skb_source_trace_init,
> + .start = start_skb_source_trace,
> + .stop = stop_skb_source_trace,
> + .reset = skb_source_trace_reset,
> + .print_line = skb_source_print_line,
> + .print_header = skb_source_print_header,
> +};
> +
> +static int init_skb_source_trace(void)
> +{
> + return register_tracer(&skb_source_tracer);
> +}
> +device_initcall(init_skb_source_trace);
>
BTW, why not just do this as events? Or was this just a easy way to
communicate with the user space tools?
-- Steve
^ permalink raw reply
* Re: [Bonding-devel] [PATCH net-next-2.6] bonding: introduce primary_lazy option
From: Nicolas de Pesloüan @ 2009-08-17 20:55 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, fubar, bonding-devel
In-Reply-To: <20090817114938.GA3416@psychotron.englab.brq.redhat.com>
Jiri Pirko a écrit :
> Fri, Aug 14, 2009 at 06:27:03PM CEST, nicolas.2p.debian@free.fr wrote:
>> Jiri Pirko wrote:
>>> Thu, Aug 13, 2009 at 09:41:02PM CEST, nicolas.2p.debian@free.fr wrote:
>>>> Jiri Pirko wrote:
>>>>> In some cases there is not desirable to switch back to primary interface when
>>>>> it's link recovers and rather stay wiith currently active one. We need to avoid
>>>>> packetloss as much as we can in some cases. This is solved by introducing
>>>>> primary_lazy option. Note that enslaved primary slave is set as current
>>>>> active no matter what.
>>>> May I suggest that instead of creating a new option to better define how
>>>> the "primary" option is expected to behave for active-backup mode, we
>>>> try the "weight" slave option I proposed in the thread "alternative
>>>> to primary" earlier this year ?
>>>>
>>>> http://sourceforge.net/mailarchive/forum.php?thread_name=49D5357E.4020201%40free.fr&forum_name=bonding-devel
>>> This link does not work for me :(
>> Nor for me... Sourceforge apparently decided to drop the bonding-devel
>> list archive just now. 'hope the list archive will be back soon.
>>
>> Originally, the proposed "weight" option for slaves was designed just to
>> provide a way to better define which slave should become active when the
>> active one just went down. As you know, the current "primary" option
>> does not allow for a predictable selection of the new active slave when
>> the primary loose connectivity. The new active slave is chosen "at
>> random" between the remaining slaves.
>>
>> After a short thread, involving Jay Vosburg and Andy Gospodarek, we end
>> up with a general configuration interface, that provide a way to tune
>> many things in slave management :
>>
>> - Active slave selection in active/backup mode, even in the presence of
>> more than two slaves.
>> - Active aggregator selection in 802.3ad mode.
>> - Load balancing tuning for most load balancing modes.
>>
>> The sysfs interface would be /sys/class/net/eth0/bonding/weight. Writing
>> a number there would give a "user supplied weight" to a slave. The speed
>> and link state of the slave would give a "natural weight" for the slave.
>> And the "effective weight" would be computed every time one of user
>> supplied or natural weight change (upon speed or link state changes) and
>> would be used everywhere we need a slave weight.
>>
>> I suggest that :
>> - slave's natural weight = speed of the slave if link UP, else 0.
>> - slave's effective weight = slave's natural weight * slave's user
>> supplied weight.
>> - aggregator's effective weight = sum of the effective weights of the
>> slaves inside the aggregator.
>>
>> For the active/backup mode, the exact behavior would be :
>>
>> - When the active slave disappear, the new active slave is the one whose
>> effective weight is the highest.
>> - When a slave comes back, it only becomes active if its effective
>> weight is strictly higher than the one of the current active slave.
>> (This stop the flip-flop risk you stated).
>> - To keep the old "primary" option, we simply give a very high user
>> supplied weight to the primary slave. Jay suggested :
>> #define BOND_PRIMARY_PRIO 0x80000000
>> user_supplied_weight &= BOND_PRIMARY_PRIO /* to set the primary */
>> user_supplied_weight &= ~BOND_PRIMAY_PRIO /* to clear the primary */
>>
>> The same apply to aggregator : Every time a slave enter (link UP) or
>> leave (link DOWN) an aggregator, the aggregator effective weight is
>> recomputed. Then, if an aggregator exist with an strictly higher
>> effective weight than the current active one, the new best aggregator
>> becomes active.
>>
>> For others modes, the weight might be used later to tune the load
>> balancing logic in some way.
>>
>> A default value of 1 for slave weight would cause slave speed to be used
>> alone, hence the "natural weight".
>>
>
> I read your text and also the original list thread and I must say I see no
> solution in this "weight" parameter for this issue. Because it's desired for one
> link to stay active even if second come up, these 2 must have the same weight.
> But imagine 3 links of the same weight. In that case you cannot insure that the
> "primary one" will be chosen as active (see my picture in the reply to Jay's
> post). Correct me if I'm wrong but for that what I want to fix by primary_lazy
> option, your proposed weight option has no effect.
>
> Therefor I still think the primary_lazy is the only solution now.
>
> Jirka
Hi Jirka,
From your previous posts (first one and reply to Jay), I understand
that your want to achieve the following behavior :
eth0 is primary and active.
eth1 is allowed to be active is eth0 is down.
Also, eth1 should stay active, even if eth0 comes back up.
Switch active to eth0 if eth1 eventually fall down.
Switch active to eth2 only if both eth0 and eth1 are down.
eth0 eth1 eth2
UP(curr) UP UP
DOWN UP(curr) UP
UP UP(curr) UP
UP(curr) DOWN UP
DOWN DOWN UP(curr)
Using weight, the following setup should give this result :
echo 1000 > /sys/class/net/eth0/bonding/weight
echo 1000 > /sys/class/net/eth1/bonding/weight
echo 1 > /sys/class/net/eth2/bonding/weight
echo eth0 > /sys/class/net/bond0/bonding/active_slave
I hope this is clear now.
Nicolas.
>
>>>> Giving the same "weight" to two different slaves means "chose at random
>>>> on startup and keep the active one until it fails". And if the "at
>>>> random" behavior is not appropriate, one can force the active slave
>>>> using what Jay suggested (/sys/class/net/bond0/bonding/active).
>>>>
>>>> The proposed "weight" slave's option is able to prevent the slaves from
>>>> flip-flopping, by stating the fact that two slaves share the same
>>>> "primary" level, and may provide several other enhancements as
>>>> described in the thread.
>>>>
>>> Although I cannot reach the thread, this looks interesting. But I'm not sure it
>>> has real benefits over primary_lazy option (and it doesn't solve initial curr
>>> active slave setup)
>> You are right, it doesn't solve the initial active slave selection. But
>> why would it be so important to properly select the initial active
>> slave, if you feel comfortable with staying with a new active slave,
>> after a failure and return of the original active slave ? This kind of
>> failures may last for only a few seconds (just unplugging and plugging
>> back the wire), and you configuration may then stay with the new active
>> slave "forever". If "forever" is acceptable, may be "at startup" is
>> acceptable too. :-)
>>
>> From my point of view (and Andy Gospodarek apparently agreed), the real
>> benefits of the weight slave option is that is it more generic and allow
>> for later usage in other modes, that we don't anticipate for now.
>>
>> Quoted from a mail from Andy Gospodarek in the original thread :
>>
>> "I really have no objection to that. Adding this as a base part of
>> bonding for a few modes with known features would be a nice start.
>> I'm sure others will be kind enough to send suggestions or patches for
>> ways this could benefit other modes."
>>
>> Nicolas.
>
^ permalink raw reply
* [RFC net-next PATCH 0/4] qlge: Performance changes for qlge.
From: Ron Mercer @ 2009-08-17 21:08 UTC (permalink / raw)
To: davem; +Cc: netdev, ron.mercer
All,
Looking for feedback on changes for qlge. The goal is to:
1) Do TX completions in send path (with cleaner timer).
2) Change RSS queue count to match MSIx vector count instead
of CPU count. Some platforms didn't offer enough vectors
for our previous approach.
3) Change large RX buffer logic to use either multiple pages
or chunks of pages based on MTU and system page size.
Examples:
64k Pages with 1500 MTU. The RX buffers size would be
2048 bytes and there would be 32 per page.
4k pages with 9000 MTU. The RX buffer size would be 16k,
or 4 pages per buffer.
Regards,
Ron Mercer
^ permalink raw reply
* [RFC net-next PATCH 1/4] qlge: Remove workqueue usage from data path.
From: Ron Mercer @ 2009-08-17 21:08 UTC (permalink / raw)
To: davem; +Cc: netdev, ron.mercer
In-Reply-To: <1250543329-15123-1-git-send-email-ron.mercer@qlogic.com>
Driver was using workqueues for TX completions and for default
(broadcast/multicast) RX completions. This change moves default
RX to NAPI context and TX to interrupt context.
Signed-off-by: Ron Mercer <ron.mercer@qlogic.com>
---
drivers/net/qlge/qlge.h | 2 -
drivers/net/qlge/qlge_main.c | 115 +++++++++---------------------------------
2 files changed, 24 insertions(+), 93 deletions(-)
diff --git a/drivers/net/qlge/qlge.h b/drivers/net/qlge/qlge.h
index 6ed5317..e0b9330 100644
--- a/drivers/net/qlge/qlge.h
+++ b/drivers/net/qlge/qlge.h
@@ -1292,7 +1292,6 @@ struct rx_ring {
u32 cpu; /* Which CPU this should run on. */
char name[IFNAMSIZ + 5];
struct napi_struct napi;
- struct delayed_work rx_work;
u8 reserved;
struct ql_adapter *qdev;
};
@@ -1519,7 +1518,6 @@ struct ql_adapter {
union flash_params flash;
struct net_device_stats stats;
- struct workqueue_struct *q_workqueue;
struct workqueue_struct *workqueue;
struct delayed_work asic_reset_work;
struct delayed_work mpi_reset_work;
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index 3a271af..c551ac3 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -1682,7 +1682,7 @@ static void ql_process_mac_tx_intr(struct ql_adapter *qdev,
ql_unmap_send(qdev, tx_ring_desc, tx_ring_desc->map_cnt);
qdev->stats.tx_bytes += (tx_ring_desc->skb)->len;
qdev->stats.tx_packets++;
- dev_kfree_skb(tx_ring_desc->skb);
+ dev_kfree_skb_any(tx_ring_desc->skb);
tx_ring_desc->skb = NULL;
if (unlikely(mac_rsp->flags1 & (OB_MAC_IOCB_RSP_E |
@@ -1928,35 +1928,11 @@ static void ql_vlan_rx_kill_vid(struct net_device *ndev, u16 vid)
}
-/* Worker thread to process a given rx_ring that is dedicated
- * to outbound completions.
- */
-static void ql_tx_clean(struct work_struct *work)
-{
- struct rx_ring *rx_ring =
- container_of(work, struct rx_ring, rx_work.work);
- ql_clean_outbound_rx_ring(rx_ring);
- ql_enable_completion_interrupt(rx_ring->qdev, rx_ring->irq);
-
-}
-
-/* Worker thread to process a given rx_ring that is dedicated
- * to inbound completions.
- */
-static void ql_rx_clean(struct work_struct *work)
-{
- struct rx_ring *rx_ring =
- container_of(work, struct rx_ring, rx_work.work);
- ql_clean_inbound_rx_ring(rx_ring, 64);
- ql_enable_completion_interrupt(rx_ring->qdev, rx_ring->irq);
-}
-
/* MSI-X Multiple Vector Interrupt Handler for outbound completions. */
static irqreturn_t qlge_msix_tx_isr(int irq, void *dev_id)
{
struct rx_ring *rx_ring = dev_id;
- queue_delayed_work_on(rx_ring->cpu, rx_ring->qdev->q_workqueue,
- &rx_ring->rx_work, 0);
+ ql_clean_outbound_rx_ring(rx_ring);
return IRQ_HANDLED;
}
@@ -1978,7 +1954,7 @@ static irqreturn_t qlge_isr(int irq, void *dev_id)
struct rx_ring *rx_ring = dev_id;
struct ql_adapter *qdev = rx_ring->qdev;
struct intr_context *intr_context = &qdev->intr_context[0];
- u32 var;
+ u32 var, mask;
int i;
int work_done = 0;
@@ -2020,41 +1996,26 @@ static irqreturn_t qlge_isr(int irq, void *dev_id)
}
/*
- * Check the default queue and wake handler if active.
+ * Start NAPI for rx or handler for TX for each active queue.
*/
- rx_ring = &qdev->rx_ring[0];
- if (ql_read_sh_reg(rx_ring->prod_idx_sh_reg) != rx_ring->cnsmr_idx) {
- QPRINTK(qdev, INTR, INFO, "Waking handler for rx_ring[0].\n");
- ql_disable_completion_interrupt(qdev, intr_context->intr);
- queue_delayed_work_on(smp_processor_id(), qdev->q_workqueue,
- &rx_ring->rx_work, 0);
- work_done++;
- }
-
- if (!test_bit(QL_MSIX_ENABLED, &qdev->flags)) {
- /*
- * Start the DPC for each active queue.
- */
- for (i = 1; i < qdev->rx_ring_count; i++) {
+ mask = ql_read32(qdev, ISR1);
+ for (i = 0; i < qdev->rx_ring_count; i++) {
+ if (mask & (1 << i)) {
rx_ring = &qdev->rx_ring[i];
- if (ql_read_sh_reg(rx_ring->prod_idx_sh_reg) !=
- rx_ring->cnsmr_idx) {
- QPRINTK(qdev, INTR, INFO,
- "Waking handler for rx_ring[%d].\n", i);
+ QPRINTK(qdev, INTR, INFO,
+ "Waking handler for rx_ring[%d].\n", i);
+ if (rx_ring->type == TX_Q)
+ ql_clean_outbound_rx_ring(rx_ring);
+ else {
ql_disable_completion_interrupt(qdev,
- intr_context->
- intr);
- if (i < qdev->rss_ring_first_cq_id)
- queue_delayed_work_on(rx_ring->cpu,
- qdev->q_workqueue,
- &rx_ring->rx_work,
- 0);
- else
- napi_schedule(&rx_ring->napi);
- work_done++;
+ intr_context->
+ intr);
+ napi_schedule(&rx_ring->napi);
}
+ work_done++;
}
}
+
ql_enable_completion_interrupt(qdev, intr_context->intr);
return work_done ? IRQ_HANDLED : IRQ_NONE;
}
@@ -2706,32 +2667,15 @@ static int ql_start_rx_ring(struct ql_adapter *qdev, struct rx_ring *rx_ring)
}
switch (rx_ring->type) {
case TX_Q:
- /* If there's only one interrupt, then we use
- * worker threads to process the outbound
- * completion handling rx_rings. We do this so
- * they can be run on multiple CPUs. There is
- * room to play with this more where we would only
- * run in a worker if there are more than x number
- * of outbound completions on the queue and more
- * than one queue active. Some threshold that
- * would indicate a benefit in spite of the cost
- * of a context switch.
- * If there's more than one interrupt, then the
- * outbound completions are processed in the ISR.
- */
- if (!test_bit(QL_MSIX_ENABLED, &qdev->flags))
- INIT_DELAYED_WORK(&rx_ring->rx_work, ql_tx_clean);
- else {
- /* With all debug warnings on we see a WARN_ON message
- * when we free the skb in the interrupt context.
- */
- INIT_DELAYED_WORK(&rx_ring->rx_work, ql_tx_clean);
- }
cqicb->irq_delay = cpu_to_le16(qdev->tx_coalesce_usecs);
cqicb->pkt_delay = cpu_to_le16(qdev->tx_max_coalesced_frames);
break;
case DEFAULT_Q:
- INIT_DELAYED_WORK(&rx_ring->rx_work, ql_rx_clean);
+ /* Inbound completion handling rx_rings run in
+ * separate NAPI contexts.
+ */
+ netif_napi_add(qdev->ndev, &rx_ring->napi, ql_napi_poll_msix,
+ 64);
cqicb->irq_delay = 0;
cqicb->pkt_delay = 0;
break;
@@ -3340,16 +3284,11 @@ static int ql_adapter_down(struct ql_adapter *qdev)
cancel_delayed_work_sync(&qdev->mpi_idc_work);
cancel_delayed_work_sync(&qdev->mpi_port_cfg_work);
- /* The default queue at index 0 is always processed in
- * a workqueue.
- */
- cancel_delayed_work_sync(&qdev->rx_ring[0].rx_work);
-
/* The rest of the rx_rings are processed in
* a workqueue only if it's a single interrupt
* environment (MSI/Legacy).
*/
- for (i = 1; i < qdev->rx_ring_count; i++) {
+ for (i = 0; i < qdev->rx_ring_count; i++) {
rx_ring = &qdev->rx_ring[i];
/* Only the RSS rings use NAPI on multi irq
* environment. Outbound completion processing
@@ -3357,8 +3296,6 @@ static int ql_adapter_down(struct ql_adapter *qdev)
*/
if (i >= qdev->rss_ring_first_cq_id) {
napi_disable(&rx_ring->napi);
- } else {
- cancel_delayed_work_sync(&rx_ring->rx_work);
}
}
@@ -3845,10 +3782,7 @@ static void ql_release_all(struct pci_dev *pdev)
destroy_workqueue(qdev->workqueue);
qdev->workqueue = NULL;
}
- if (qdev->q_workqueue) {
- destroy_workqueue(qdev->q_workqueue);
- qdev->q_workqueue = NULL;
- }
+
if (qdev->reg_base)
iounmap(qdev->reg_base);
if (qdev->doorbell_area)
@@ -3962,7 +3896,6 @@ static int __devinit ql_init_device(struct pci_dev *pdev,
*/
qdev->rx_csum = 1;
- qdev->q_workqueue = create_workqueue(ndev->name);
qdev->workqueue = create_singlethread_workqueue(ndev->name);
INIT_DELAYED_WORK(&qdev->asic_reset_work, ql_asic_reset_work);
INIT_DELAYED_WORK(&qdev->mpi_reset_work, ql_mpi_reset_work);
--
1.6.0.2
^ permalink raw reply related
* [RFC net-next PATCH 2/4] qlge: Move TX completion processing to send path.
From: Ron Mercer @ 2009-08-17 21:08 UTC (permalink / raw)
To: davem; +Cc: netdev, ron.mercer
In-Reply-To: <1250543329-15123-1-git-send-email-ron.mercer@qlogic.com>
Signed-off-by: Ron Mercer <ron.mercer@qlogic.com>
---
drivers/net/qlge/qlge.h | 5 ++-
drivers/net/qlge/qlge_main.c | 59 +++++++++++++++++++++++++++++++++++++-----
2 files changed, 55 insertions(+), 9 deletions(-)
diff --git a/drivers/net/qlge/qlge.h b/drivers/net/qlge/qlge.h
index e0b9330..975590c 100644
--- a/drivers/net/qlge/qlge.h
+++ b/drivers/net/qlge/qlge.h
@@ -1205,6 +1205,7 @@ struct bq_desc {
};
#define QL_TXQ_IDX(qdev, skb) (smp_processor_id()%(qdev->tx_ring_count))
+#define TXQ_CLEAN_TIME (HZ/4)
struct tx_ring {
/*
@@ -1224,11 +1225,11 @@ struct tx_ring {
u8 wq_id; /* queue id for this entry */
u8 reserved1[3];
struct tx_ring_desc *q; /* descriptor list for the queue */
- spinlock_t lock;
atomic_t tx_count; /* counts down for every outstanding IO */
atomic_t queue_stopped; /* Turns queue off when full. */
- struct delayed_work tx_work;
+ struct netdev_queue *txq;
struct ql_adapter *qdev;
+ struct timer_list txq_clean_timer;
};
/*
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index c551ac3..d9b22da 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -1682,7 +1682,7 @@ static void ql_process_mac_tx_intr(struct ql_adapter *qdev,
ql_unmap_send(qdev, tx_ring_desc, tx_ring_desc->map_cnt);
qdev->stats.tx_bytes += (tx_ring_desc->skb)->len;
qdev->stats.tx_packets++;
- dev_kfree_skb_any(tx_ring_desc->skb);
+ dev_kfree_skb(tx_ring_desc->skb);
tx_ring_desc->skb = NULL;
if (unlikely(mac_rsp->flags1 & (OB_MAC_IOCB_RSP_E |
@@ -1797,22 +1797,40 @@ static int ql_clean_outbound_rx_ring(struct rx_ring *rx_ring)
ql_update_cq(rx_ring);
prod = ql_read_sh_reg(rx_ring->prod_idx_sh_reg);
}
+ if (!count)
+ return count;
ql_write_cq_idx(rx_ring);
tx_ring = &qdev->tx_ring[net_rsp->txq_idx];
- if (__netif_subqueue_stopped(qdev->ndev, tx_ring->wq_id) &&
- net_rsp != NULL) {
+ if (netif_tx_queue_stopped(tx_ring->txq)) {
if (atomic_read(&tx_ring->queue_stopped) &&
(atomic_read(&tx_ring->tx_count) > (tx_ring->wq_len / 4)))
/*
* The queue got stopped because the tx_ring was full.
* Wake it up, because it's now at least 25% empty.
*/
- netif_wake_subqueue(qdev->ndev, tx_ring->wq_id);
+ if (netif_running(qdev->ndev)) {
+ netif_tx_wake_queue(tx_ring->txq);
+ atomic_dec(&tx_ring->queue_stopped);
+ }
}
return count;
}
+static void ql_txq_clean_timer(unsigned long data)
+{
+ struct tx_ring *tx_ring = (struct tx_ring *)data;
+ struct ql_adapter *qdev = tx_ring->qdev;
+ struct rx_ring *rx_ring = &qdev->rx_ring[tx_ring->cq_id];
+
+ if (__netif_tx_trylock(tx_ring->txq)) {
+ ql_clean_outbound_rx_ring(rx_ring);
+ __netif_tx_unlock(tx_ring->txq);
+ }
+ mod_timer(&tx_ring->txq_clean_timer, jiffies + TXQ_CLEAN_TIME);
+
+}
+
static int ql_clean_inbound_rx_ring(struct rx_ring *rx_ring, int budget)
{
struct ql_adapter *qdev = rx_ring->qdev;
@@ -2005,7 +2023,7 @@ static irqreturn_t qlge_isr(int irq, void *dev_id)
QPRINTK(qdev, INTR, INFO,
"Waking handler for rx_ring[%d].\n", i);
if (rx_ring->type == TX_Q)
- ql_clean_outbound_rx_ring(rx_ring);
+ continue;
else {
ql_disable_completion_interrupt(qdev,
intr_context->
@@ -2107,11 +2125,17 @@ static int qlge_send(struct sk_buff *skb, struct net_device *ndev)
if (skb_padto(skb, ETH_ZLEN))
return NETDEV_TX_OK;
+ /* If there is at least 16 entries to clean then
+ * go do it.
+ */
+ if (tx_ring->wq_len - atomic_read(&tx_ring->tx_count) > 16)
+ ql_clean_outbound_rx_ring(&qdev->rx_ring[tx_ring->cq_id]);
+
if (unlikely(atomic_read(&tx_ring->tx_count) < 2)) {
QPRINTK(qdev, TX_QUEUED, INFO,
"%s: shutting down tx queue %d du to lack of resources.\n",
__func__, tx_ring_idx);
- netif_stop_subqueue(ndev, tx_ring->wq_id);
+ netif_tx_stop_queue(tx_ring->txq);
atomic_inc(&tx_ring->queue_stopped);
return NETDEV_TX_BUSY;
}
@@ -2128,6 +2152,8 @@ static int qlge_send(struct sk_buff *skb, struct net_device *ndev)
tx_ring_desc->skb = skb;
mac_iocb_ptr->frame_len = cpu_to_le16((u16) skb->len);
+ /* Disable completion interrupt for this packet. */
+ mac_iocb_ptr->flags1 |= OB_MAC_IOCB_REQ_I;
if (qdev->vlgrp && vlan_tx_tag_present(skb)) {
QPRINTK(qdev, TX_QUEUED, DEBUG, "Adding a vlan tag %d.\n",
@@ -2153,13 +2179,20 @@ static int qlge_send(struct sk_buff *skb, struct net_device *ndev)
tx_ring->prod_idx++;
if (tx_ring->prod_idx == tx_ring->wq_len)
tx_ring->prod_idx = 0;
+ atomic_dec(&tx_ring->tx_count);
wmb();
+ /* Run the destructor before telling the DMA engine about
+ * the packet to make sure it doesn't complete and get
+ * freed prematurely.
+ */
+ if (likely(!skb_shared(skb)))
+ skb_orphan(skb);
+
ql_write_db_reg(tx_ring->prod_idx, tx_ring->prod_idx_db_reg);
QPRINTK(qdev, TX_QUEUED, DEBUG, "tx queued, slot %d, len %d\n",
tx_ring->prod_idx, skb->len);
- atomic_dec(&tx_ring->tx_count);
return NETDEV_TX_OK;
}
@@ -2727,6 +2760,8 @@ static int ql_start_tx_ring(struct ql_adapter *qdev, struct tx_ring *tx_ring)
*/
tx_ring->cnsmr_idx_sh_reg = shadow_reg;
tx_ring->cnsmr_idx_sh_reg_dma = shadow_reg_dma;
+ *tx_ring->cnsmr_idx_sh_reg = 0;
+ tx_ring->txq = netdev_get_tx_queue(qdev->ndev, tx_ring->wq_id);
wqicb->len = cpu_to_le16(tx_ring->wq_len | Q_LEN_V | Q_LEN_CPP_CONT);
wqicb->flags = cpu_to_le16(Q_FLAGS_LC |
@@ -2746,6 +2781,7 @@ static int ql_start_tx_ring(struct ql_adapter *qdev, struct tx_ring *tx_ring)
return err;
}
QPRINTK(qdev, IFUP, DEBUG, "Successfully loaded WQICB.\n");
+ mod_timer(&tx_ring->txq_clean_timer, jiffies + TXQ_CLEAN_TIME);
return err;
}
@@ -3299,6 +3335,12 @@ static int ql_adapter_down(struct ql_adapter *qdev)
}
}
+ /* Delete the timers used for cleaning up
+ * TX completions.
+ */
+ for (i = 0; i < qdev->tx_ring_count; i++)
+ del_timer_sync(&qdev->tx_ring[i].txq_clean_timer);
+
clear_bit(QL_ADAPTER_UP, &qdev->flags);
ql_disable_interrupts(qdev);
@@ -3438,6 +3480,9 @@ static int ql_configure_rings(struct ql_adapter *qdev)
* immediately after the default Q ID, which is zero.
*/
tx_ring->cq_id = i + 1;
+ init_timer(&tx_ring->txq_clean_timer);
+ tx_ring->txq_clean_timer.data = (unsigned long)tx_ring;
+ tx_ring->txq_clean_timer.function = ql_txq_clean_timer;
}
for (i = 0; i < qdev->rx_ring_count; i++) {
--
1.6.0.2
^ permalink raw reply related
* [RFC net-next PATCH 3/4] qlge: Change rx_ring to MSIX vector mapping.
From: Ron Mercer @ 2009-08-17 21:08 UTC (permalink / raw)
To: davem; +Cc: netdev, ron.mercer
In-Reply-To: <1250543329-15123-1-git-send-email-ron.mercer@qlogic.com>
Currently there is one default ring, one outbound rx ring for each
CPU, and one RSS ring for each CPU.
This patch allocates one RSS ring for each MSIx vector,
and adds the default ring's functionality to the first
RSS ring.
Using up an MSIX vector on sideband work is expensive on platforms
that have limited resources.
Signed-off-by: Ron Mercer <ron.mercer@qlogic.com>
---
drivers/net/qlge/qlge.h | 10 +-
drivers/net/qlge/qlge_dbg.c | 8 +-
drivers/net/qlge/qlge_ethtool.c | 40 +++---
drivers/net/qlge/qlge_main.c | 266 +++++++++++++++++----------------------
4 files changed, 141 insertions(+), 183 deletions(-)
diff --git a/drivers/net/qlge/qlge.h b/drivers/net/qlge/qlge.h
index 975590c..5559c80 100644
--- a/drivers/net/qlge/qlge.h
+++ b/drivers/net/qlge/qlge.h
@@ -34,7 +34,7 @@
#define QLGE_DEVICE_ID_8000 0x8000
#define MAX_CPUS 8
#define MAX_TX_RINGS MAX_CPUS
-#define MAX_RX_RINGS ((MAX_CPUS * 2) + 1)
+#define MAX_RX_RINGS (MAX_CPUS * 2)
#define NUM_TX_RING_ENTRIES 256
#define NUM_RX_RING_ENTRIES 256
@@ -1236,7 +1236,6 @@ struct tx_ring {
* Type of inbound queue.
*/
enum {
- DEFAULT_Q = 2, /* Handles slow queue and chip/MPI events. */
TX_Q = 3, /* Handles outbound completions. */
RX_Q = 4, /* Handles inbound completions. */
};
@@ -1288,7 +1287,7 @@ struct rx_ring {
u32 sbq_free_cnt; /* free buffer desc cnt */
/* Misc. handler elements. */
- u32 type; /* Type of queue, tx, rx, or default. */
+ u32 type; /* Type of queue, tx, rx. */
u32 irq; /* Which vector this ring is assigned. */
u32 cpu; /* Which CPU this should run on. */
char name[IFNAMSIZ + 5];
@@ -1486,11 +1485,9 @@ struct ql_adapter {
struct intr_context intr_context[MAX_RX_RINGS];
int tx_ring_count; /* One per online CPU. */
- u32 rss_ring_first_cq_id;/* index of first inbound (rss) rx_ring */
- u32 rss_ring_count; /* One per online CPU. */
+ u32 rss_ring_count; /* One per irq vector. */
/*
* rx_ring_count =
- * one default queue +
* (CPU count * outbound completion rx_ring) +
* (CPU count * inbound (RSS) completion rx_ring)
*/
@@ -1502,7 +1499,6 @@ struct ql_adapter {
struct tx_ring tx_ring[MAX_TX_RINGS];
int rx_csum;
- u32 default_rx_queue;
u16 rx_coalesce_usecs; /* cqicb->int_delay */
u16 rx_max_coalesced_frames; /* cqicb->pkt_int_delay */
diff --git a/drivers/net/qlge/qlge_dbg.c b/drivers/net/qlge/qlge_dbg.c
index 40a70c3..c6d4db5 100644
--- a/drivers/net/qlge/qlge_dbg.c
+++ b/drivers/net/qlge/qlge_dbg.c
@@ -418,13 +418,9 @@ void ql_dump_qdev(struct ql_adapter *qdev)
printk(KERN_ERR PFX "qdev->intr_count = %d.\n", qdev->intr_count);
printk(KERN_ERR PFX "qdev->tx_ring = %p.\n",
qdev->tx_ring);
- printk(KERN_ERR PFX "qdev->rss_ring_first_cq_id = %d.\n",
- qdev->rss_ring_first_cq_id);
printk(KERN_ERR PFX "qdev->rss_ring_count = %d.\n",
qdev->rss_ring_count);
printk(KERN_ERR PFX "qdev->rx_ring = %p.\n", qdev->rx_ring);
- printk(KERN_ERR PFX "qdev->default_rx_queue = %d.\n",
- qdev->default_rx_queue);
printk(KERN_ERR PFX "qdev->xg_sem_mask = 0x%08x.\n",
qdev->xg_sem_mask);
printk(KERN_ERR PFX "qdev->port_link_up = 0x%08x.\n",
@@ -545,8 +541,8 @@ void ql_dump_rx_ring(struct rx_ring *rx_ring)
printk(KERN_ERR PFX
"===================== Dumping rx_ring %d ===============.\n",
rx_ring->cq_id);
- printk(KERN_ERR PFX "Dumping rx_ring %d, type = %s%s%s.\n",
- rx_ring->cq_id, rx_ring->type == DEFAULT_Q ? "DEFAULT" : "",
+ printk(KERN_ERR PFX "Dumping rx_ring %d, type = %s%s.\n",
+ rx_ring->cq_id,
rx_ring->type == TX_Q ? "OUTBOUND COMPLETIONS" : "",
rx_ring->type == RX_Q ? "INBOUND_COMPLETIONS" : "");
printk(KERN_ERR PFX "rx_ring->cqicb = %p.\n", &rx_ring->cqicb);
diff --git a/drivers/net/qlge/qlge_ethtool.c b/drivers/net/qlge/qlge_ethtool.c
index eb6a9ee..57faa54 100644
--- a/drivers/net/qlge/qlge_ethtool.c
+++ b/drivers/net/qlge/qlge_ethtool.c
@@ -46,20 +46,21 @@ static int ql_update_ring_coalescing(struct ql_adapter *qdev)
return status;
spin_lock(&qdev->hw_lock);
- /* Skip the default queue, and update the outbound handler
+ /* Update the inbound (RSS) handler
* queues if they changed.
*/
- cqicb = (struct cqicb *)&qdev->rx_ring[1];
- if (le16_to_cpu(cqicb->irq_delay) != qdev->tx_coalesce_usecs ||
- le16_to_cpu(cqicb->pkt_delay) != qdev->tx_max_coalesced_frames) {
- for (i = 1; i < qdev->rss_ring_first_cq_id; i++, rx_ring++) {
+ cqicb = (struct cqicb *)&qdev->rx_ring[0];
+ if (le16_to_cpu(cqicb->irq_delay) != qdev->rx_coalesce_usecs ||
+ le16_to_cpu(cqicb->pkt_delay) !=
+ qdev->rx_max_coalesced_frames) {
+ for (i = 0; i < qdev->rss_ring_count; i++, rx_ring++) {
rx_ring = &qdev->rx_ring[i];
- cqicb = (struct cqicb *)rx_ring;
- cqicb->irq_delay = cpu_to_le16(qdev->tx_coalesce_usecs);
+ cqicb = &rx_ring->cqicb;
+ cqicb->irq_delay = cpu_to_le16(qdev->rx_coalesce_usecs);
cqicb->pkt_delay =
- cpu_to_le16(qdev->tx_max_coalesced_frames);
+ cpu_to_le16(qdev->rx_max_coalesced_frames);
cqicb->flags = FLAGS_LI;
- status = ql_write_cfg(qdev, cqicb, sizeof(*cqicb),
+ status = ql_write_cfg(qdev, cqicb, sizeof(cqicb),
CFG_LCQ, rx_ring->cq_id);
if (status) {
QPRINTK(qdev, IFUP, ERR,
@@ -69,20 +70,21 @@ static int ql_update_ring_coalescing(struct ql_adapter *qdev)
}
}
- /* Update the inbound (RSS) handler queues if they changed. */
- cqicb = (struct cqicb *)&qdev->rx_ring[qdev->rss_ring_first_cq_id];
- if (le16_to_cpu(cqicb->irq_delay) != qdev->rx_coalesce_usecs ||
- le16_to_cpu(cqicb->pkt_delay) != qdev->rx_max_coalesced_frames) {
- for (i = qdev->rss_ring_first_cq_id;
- i <= qdev->rss_ring_first_cq_id + qdev->rss_ring_count;
+ /* Update the outbound handler queues if they changed. */
+ cqicb = (struct cqicb *)&qdev->rx_ring[qdev->rss_ring_count];
+ if (le16_to_cpu(cqicb->irq_delay) != qdev->tx_coalesce_usecs ||
+ le16_to_cpu(cqicb->pkt_delay) !=
+ qdev->tx_max_coalesced_frames) {
+ for (i = qdev->rss_ring_count;
+ i < qdev->rx_ring_count;
i++) {
rx_ring = &qdev->rx_ring[i];
- cqicb = (struct cqicb *)rx_ring;
- cqicb->irq_delay = cpu_to_le16(qdev->rx_coalesce_usecs);
+ cqicb = &rx_ring->cqicb;
+ cqicb->irq_delay = cpu_to_le16(qdev->tx_coalesce_usecs);
cqicb->pkt_delay =
- cpu_to_le16(qdev->rx_max_coalesced_frames);
+ cpu_to_le16(qdev->tx_max_coalesced_frames);
cqicb->flags = FLAGS_LI;
- status = ql_write_cfg(qdev, cqicb, sizeof(*cqicb),
+ status = ql_write_cfg(qdev, cqicb, sizeof(cqicb),
CFG_LCQ, rx_ring->cq_id);
if (status) {
QPRINTK(qdev, IFUP, ERR,
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index d9b22da..dcac9f2 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -370,9 +370,7 @@ static int ql_set_mac_addr_reg(struct ql_adapter *qdev, u8 *addr, u32 type,
cam_output = (CAM_OUT_ROUTE_NIC |
(qdev->
func << CAM_OUT_FUNC_SHIFT) |
- (qdev->
- rss_ring_first_cq_id <<
- CAM_OUT_CQ_ID_SHIFT));
+ (0 << CAM_OUT_CQ_ID_SHIFT));
if (qdev->vlgrp)
cam_output |= CAM_OUT_RV;
/* route to NIC core */
@@ -616,9 +614,8 @@ u32 ql_enable_completion_interrupt(struct ql_adapter *qdev, u32 intr)
unsigned long hw_flags = 0;
struct intr_context *ctx = qdev->intr_context + intr;
- if (likely(test_bit(QL_MSIX_ENABLED, &qdev->flags) && intr)) {
- /* Always enable if we're MSIX multi interrupts and
- * it's not the default (zeroeth) interrupt.
+ if (likely(test_bit(QL_MSIX_ENABLED, &qdev->flags))) {
+ /* Always enable if we're MSIX multi interrupts.
*/
ql_write32(qdev, INTR_EN,
ctx->intr_en_mask);
@@ -644,7 +641,7 @@ static u32 ql_disable_completion_interrupt(struct ql_adapter *qdev, u32 intr)
/* HW disables for us if we're MSIX multi interrupts and
* it's not the default (zeroeth) interrupt.
*/
- if (likely(test_bit(QL_MSIX_ENABLED, &qdev->flags) && intr))
+ if (likely(test_bit(QL_MSIX_ENABLED, &qdev->flags)))
return 0;
ctx = qdev->intr_context + intr;
@@ -1649,8 +1646,7 @@ static void ql_process_mac_rx_intr(struct ql_adapter *qdev,
qdev->stats.rx_packets++;
qdev->stats.rx_bytes += skb->len;
- skb_record_rx_queue(skb,
- rx_ring->cq_id - qdev->rss_ring_first_cq_id);
+ skb_record_rx_queue(skb, rx_ring->cq_id);
if (skb->ip_summed == CHECKSUM_UNNECESSARY) {
if (qdev->vlgrp &&
(ib_mac_rsp->flags2 & IB_MAC_IOCB_RSP_V) &&
@@ -1946,19 +1942,55 @@ static void ql_vlan_rx_kill_vid(struct net_device *ndev, u16 vid)
}
-/* MSI-X Multiple Vector Interrupt Handler for outbound completions. */
-static irqreturn_t qlge_msix_tx_isr(int irq, void *dev_id)
+/* MSI-X Multiple Vector Interrupt Handler for inbound completions. */
+static irqreturn_t qlge_msix_rx_isr(int irq, void *dev_id)
{
struct rx_ring *rx_ring = dev_id;
- ql_clean_outbound_rx_ring(rx_ring);
+ napi_schedule(&rx_ring->napi);
return IRQ_HANDLED;
}
/* MSI-X Multiple Vector Interrupt Handler for inbound completions. */
-static irqreturn_t qlge_msix_rx_isr(int irq, void *dev_id)
+static irqreturn_t qlge_msix_dflt_rx_isr(int irq, void *dev_id)
{
struct rx_ring *rx_ring = dev_id;
- napi_schedule(&rx_ring->napi);
+ struct ql_adapter *qdev = rx_ring->qdev;
+ u32 var;
+
+ var = ql_read32(qdev, STS);
+ /*
+ * Check for fatal error.
+ */
+ if (var & STS_FE) {
+ ql_queue_asic_error(qdev);
+ QPRINTK(qdev, INTR, ERR, "Got fatal error, STS = %x.\n", var);
+ var = ql_read32(qdev, ERR_STS);
+ QPRINTK(qdev, INTR, ERR,
+ "Resetting chip. Error Status Register = 0x%x\n", var);
+ return IRQ_HANDLED;
+ }
+
+ /*
+ * Check MPI processor activity.
+ */
+ if ((var & STS_PI) &&
+ (ql_read32(qdev, INTR_MASK) & INTR_MASK_PI)) {
+ /*
+ * We've got an async event or mailbox completion.
+ * Handle it and clear the source of the interrupt.
+ */
+ QPRINTK(qdev, INTR, ERR, "Got MPI processor interrupt.\n");
+ queue_delayed_work(qdev->workqueue,
+ &qdev->mpi_work, 0);
+ return IRQ_HANDLED;
+ }
+
+ /*
+ * Start NAPI if there's work to do.
+ */
+ if (ql_read_sh_reg(rx_ring->prod_idx_sh_reg) !=
+ rx_ring->cnsmr_idx)
+ napi_schedule(&rx_ring->napi);
return IRQ_HANDLED;
}
@@ -1972,8 +2004,7 @@ static irqreturn_t qlge_isr(int irq, void *dev_id)
struct rx_ring *rx_ring = dev_id;
struct ql_adapter *qdev = rx_ring->qdev;
struct intr_context *intr_context = &qdev->intr_context[0];
- u32 var, mask;
- int i;
+ u32 var;
int work_done = 0;
spin_lock(&qdev->hw_lock);
@@ -2016,22 +2047,14 @@ static irqreturn_t qlge_isr(int irq, void *dev_id)
/*
* Start NAPI for rx or handler for TX for each active queue.
*/
- mask = ql_read32(qdev, ISR1);
- for (i = 0; i < qdev->rx_ring_count; i++) {
- if (mask & (1 << i)) {
- rx_ring = &qdev->rx_ring[i];
- QPRINTK(qdev, INTR, INFO,
- "Waking handler for rx_ring[%d].\n", i);
- if (rx_ring->type == TX_Q)
- continue;
- else {
- ql_disable_completion_interrupt(qdev,
- intr_context->
- intr);
- napi_schedule(&rx_ring->napi);
- }
- work_done++;
- }
+ var = ql_read32(qdev, ISR1);
+ if (var) {
+ QPRINTK(qdev, INTR, INFO,
+ "Waking handler for rx_ring[0].\n");
+ ql_disable_completion_interrupt(qdev,
+ intr_context->intr);
+ napi_schedule(&rx_ring->napi);
+ work_done++;
}
ql_enable_completion_interrupt(qdev, intr_context->intr);
@@ -2612,6 +2635,7 @@ static int ql_start_rx_ring(struct ql_adapter *qdev, struct rx_ring *rx_ring)
/* Set up the shadow registers for this ring. */
rx_ring->prod_idx_sh_reg = shadow_reg;
rx_ring->prod_idx_sh_reg_dma = shadow_reg_dma;
+ *rx_ring->prod_idx_sh_reg = 0;
shadow_reg += sizeof(u64);
shadow_reg_dma += sizeof(u64);
rx_ring->lbq_base_indirect = shadow_reg;
@@ -2702,15 +2726,7 @@ static int ql_start_rx_ring(struct ql_adapter *qdev, struct rx_ring *rx_ring)
case TX_Q:
cqicb->irq_delay = cpu_to_le16(qdev->tx_coalesce_usecs);
cqicb->pkt_delay = cpu_to_le16(qdev->tx_max_coalesced_frames);
- break;
- case DEFAULT_Q:
- /* Inbound completion handling rx_rings run in
- * separate NAPI contexts.
- */
- netif_napi_add(qdev->ndev, &rx_ring->napi, ql_napi_poll_msix,
- 64);
- cqicb->irq_delay = 0;
- cqicb->pkt_delay = 0;
+ cqicb->msix_vect = 0;
break;
case RX_Q:
/* Inbound completion handling rx_rings run in
@@ -2800,15 +2816,14 @@ static void ql_disable_msix(struct ql_adapter *qdev)
static void ql_enable_msix(struct ql_adapter *qdev)
{
- int i;
+ int i, err;
- qdev->intr_count = 1;
/* Get the MSIX vectors. */
if (irq_type == MSIX_IRQ) {
/* Try to alloc space for the msix struct,
* if it fails then go to MSI/legacy.
*/
- qdev->msi_x_entry = kcalloc(qdev->rx_ring_count,
+ qdev->msi_x_entry = kcalloc(qdev->intr_count,
sizeof(struct msix_entry),
GFP_KERNEL);
if (!qdev->msi_x_entry) {
@@ -2816,26 +2831,39 @@ static void ql_enable_msix(struct ql_adapter *qdev)
goto msi;
}
- for (i = 0; i < qdev->rx_ring_count; i++)
+ for (i = 0; i < qdev->intr_count; i++) {
qdev->msi_x_entry[i].entry = i;
+ qdev->msi_x_entry[i].vector = 0;
+ }
- if (!pci_enable_msix
- (qdev->pdev, qdev->msi_x_entry, qdev->rx_ring_count)) {
- set_bit(QL_MSIX_ENABLED, &qdev->flags);
- qdev->intr_count = qdev->rx_ring_count;
- QPRINTK(qdev, IFUP, DEBUG,
- "MSI-X Enabled, got %d vectors.\n",
- qdev->intr_count);
- return;
- } else {
+ /* Loop to get our vectors. We start with
+ * what we want and settle for what we get.
+ */
+ do {
+ err = pci_enable_msix(qdev->pdev,
+ qdev->msi_x_entry, qdev->intr_count);
+ if (err > 0)
+ qdev->intr_count = err;
+ } while (err > 0);
+
+ if (err < 0) {
+ pci_disable_msix(qdev->pdev);
kfree(qdev->msi_x_entry);
qdev->msi_x_entry = NULL;
QPRINTK(qdev, IFUP, WARNING,
"MSI-X Enable failed, trying MSI.\n");
+ qdev->intr_count = 1;
irq_type = MSI_IRQ;
+ } else if (err == 0) {
+ set_bit(QL_MSIX_ENABLED, &qdev->flags);
+ QPRINTK(qdev, IFUP, INFO,
+ "MSI-X Enabled, got %d vectors.\n",
+ qdev->intr_count);
+ return;
}
}
msi:
+ qdev->intr_count = 1;
if (irq_type == MSI_IRQ) {
if (!pci_enable_msi(qdev->pdev)) {
set_bit(QL_MSI_ENABLED, &qdev->flags);
@@ -2845,7 +2873,7 @@ msi:
}
}
irq_type = LEG_IRQ;
- QPRINTK(qdev, IFUP, DEBUG, "Running with legacy interrupts.\n");
+ QPRINTK(qdev, IFUP, INFO, "Running with legacy interrupts.\n");
}
/*
@@ -2859,13 +2887,10 @@ static void ql_resolve_queues_to_irqs(struct ql_adapter *qdev)
int i = 0;
struct intr_context *intr_context = &qdev->intr_context[0];
- ql_enable_msix(qdev);
-
if (likely(test_bit(QL_MSIX_ENABLED, &qdev->flags))) {
/* Each rx_ring has it's
* own intr_context since we have separate
* vectors for each queue.
- * This only true when MSI-X is enabled.
*/
for (i = 0; i < qdev->intr_count; i++, intr_context++) {
qdev->rx_ring[i].irq = i;
@@ -2887,25 +2912,18 @@ static void ql_resolve_queues_to_irqs(struct ql_adapter *qdev)
INTR_EN_TYPE_MASK | INTR_EN_INTR_MASK |
INTR_EN_TYPE_READ | INTR_EN_IHD_MASK | INTR_EN_IHD |
i;
-
if (i == 0) {
- /*
- * Default queue handles bcast/mcast plus
- * async events. Needs buffers.
+ /* The first vector/queue handles
+ * broadcast/multicast, fatal errors,
+ * and firmware events. This in addition
+ * to normal inbound NAPI processing.
*/
- intr_context->handler = qlge_isr;
- sprintf(intr_context->name, "%s-default-queue",
- qdev->ndev->name);
- } else if (i < qdev->rss_ring_first_cq_id) {
- /*
- * Outbound queue is for outbound completions only.
- */
- intr_context->handler = qlge_msix_tx_isr;
- sprintf(intr_context->name, "%s-tx-%d",
+ intr_context->handler = qlge_msix_dflt_rx_isr;
+ sprintf(intr_context->name, "%s-rx-%d",
qdev->ndev->name, i);
} else {
/*
- * Inbound queues handle unicast frames only.
+ * Inbound queues handle unicast frames.
*/
intr_context->handler = qlge_msix_rx_isr;
sprintf(intr_context->name, "%s-rx-%d",
@@ -2935,7 +2953,7 @@ static void ql_resolve_queues_to_irqs(struct ql_adapter *qdev)
*/
intr_context->handler = qlge_isr;
sprintf(intr_context->name, "%s-single_irq", qdev->ndev->name);
- for (i = 0; i < qdev->rx_ring_count; i++)
+ for (i = 0; i < qdev->rss_ring_count; i++)
qdev->rx_ring[i].irq = 0;
}
}
@@ -2986,11 +3004,10 @@ static int ql_request_irq(struct ql_adapter *qdev)
goto err_irq;
} else {
QPRINTK(qdev, IFUP, DEBUG,
- "Hooked intr %d, queue type %s%s%s, with name %s.\n",
+ "Hooked intr %d, queue type %s%s, "
+ "with name %s.\n",
i,
qdev->rx_ring[i].type ==
- DEFAULT_Q ? "DEFAULT_Q" : "",
- qdev->rx_ring[i].type ==
TX_Q ? "TX_Q" : "",
qdev->rx_ring[i].type ==
RX_Q ? "RX_Q" : "", intr_context->name);
@@ -3016,10 +3033,8 @@ static int ql_request_irq(struct ql_adapter *qdev)
goto err_irq;
QPRINTK(qdev, IFUP, ERR,
- "Hooked intr %d, queue type %s%s%s, with name %s.\n",
+ "Hooked intr %d, queue type %s%s, with name %s.\n",
i,
- qdev->rx_ring[0].type ==
- DEFAULT_Q ? "DEFAULT_Q" : "",
qdev->rx_ring[0].type == TX_Q ? "TX_Q" : "",
qdev->rx_ring[0].type == RX_Q ? "RX_Q" : "",
intr_context->name);
@@ -3042,7 +3057,7 @@ static int ql_start_rss(struct ql_adapter *qdev)
memset((void *)ricb, 0, sizeof(*ricb));
- ricb->base_cq = qdev->rss_ring_first_cq_id | RSS_L4K;
+ ricb->base_cq = RSS_L4K;
ricb->flags =
(RSS_L6K | RSS_LI | RSS_LB | RSS_LM | RSS_RI4 | RSS_RI6 | RSS_RT4 |
RSS_RT6);
@@ -3244,7 +3259,7 @@ static int ql_adapter_initialize(struct ql_adapter *qdev)
}
/* Start NAPI for the RSS queues. */
- for (i = qdev->rss_ring_first_cq_id; i < qdev->rx_ring_count; i++) {
+ for (i = 0; i < qdev->rss_ring_count; i++) {
QPRINTK(qdev, IFUP, DEBUG, "Enabling NAPI for rx_ring[%d].\n",
i);
napi_enable(&qdev->rx_ring[i].napi);
@@ -3306,7 +3321,6 @@ static void ql_display_dev_info(struct net_device *ndev)
static int ql_adapter_down(struct ql_adapter *qdev)
{
int i, status = 0;
- struct rx_ring *rx_ring;
ql_link_off(qdev);
@@ -3320,20 +3334,12 @@ static int ql_adapter_down(struct ql_adapter *qdev)
cancel_delayed_work_sync(&qdev->mpi_idc_work);
cancel_delayed_work_sync(&qdev->mpi_port_cfg_work);
- /* The rest of the rx_rings are processed in
- * a workqueue only if it's a single interrupt
- * environment (MSI/Legacy).
- */
- for (i = 0; i < qdev->rx_ring_count; i++) {
- rx_ring = &qdev->rx_ring[i];
/* Only the RSS rings use NAPI on multi irq
* environment. Outbound completion processing
* is done in interrupt context.
*/
- if (i >= qdev->rss_ring_first_cq_id) {
- napi_disable(&rx_ring->napi);
- }
- }
+ for (i = 0; i < qdev->rss_ring_count; i++)
+ napi_disable(&qdev->rx_ring[i].napi);
/* Delete the timers used for cleaning up
* TX completions.
@@ -3349,7 +3355,7 @@ static int ql_adapter_down(struct ql_adapter *qdev)
/* Call netif_napi_del() from common point.
*/
- for (i = qdev->rss_ring_first_cq_id; i < qdev->rx_ring_count; i++)
+ for (i = 0; i < qdev->rss_ring_count; i++)
netif_napi_del(&qdev->rx_ring[i].napi);
ql_free_rx_buffers(qdev);
@@ -3428,43 +3434,20 @@ static int ql_configure_rings(struct ql_adapter *qdev)
int i;
struct rx_ring *rx_ring;
struct tx_ring *tx_ring;
- int cpu_cnt = num_online_cpus();
-
- /*
- * For each processor present we allocate one
- * rx_ring for outbound completions, and one
- * rx_ring for inbound completions. Plus there is
- * always the one default queue. For the CPU
- * counts we end up with the following rx_rings:
- * rx_ring count =
- * one default queue +
- * (CPU count * outbound completion rx_ring) +
- * (CPU count * inbound (RSS) completion rx_ring)
- * To keep it simple we limit the total number of
- * queues to < 32, so we truncate CPU to 8.
- * This limitation can be removed when requested.
- */
-
- if (cpu_cnt > MAX_CPUS)
- cpu_cnt = MAX_CPUS;
+ int cpu_cnt = min(MAX_CPUS, (int)num_online_cpus());
- /*
- * rx_ring[0] is always the default queue.
- */
/* Allocate outbound completion ring for each CPU. */
qdev->tx_ring_count = cpu_cnt;
- /* Allocate inbound completion (RSS) ring for each CPU. */
qdev->rss_ring_count = cpu_cnt;
- /* cq_id for the first inbound ring handler. */
- qdev->rss_ring_first_cq_id = cpu_cnt + 1;
+ qdev->intr_count = qdev->rss_ring_count;
+ /* Try to get a vector for each RSS queue (per CPU). */
+ ql_enable_msix(qdev);
/*
- * qdev->rx_ring_count:
- * Total number of rx_rings. This includes the one
- * default queue, a number of outbound completion
- * handler rx_rings, and the number of inbound
- * completion handler rx_rings.
+ * We might not have gotten what we wanted.
+ * Downshift to what we were actually given
*/
- qdev->rx_ring_count = qdev->tx_ring_count + qdev->rss_ring_count + 1;
+ qdev->rss_ring_count = qdev->intr_count;
+ qdev->rx_ring_count = qdev->tx_ring_count + qdev->rss_ring_count;
for (i = 0; i < qdev->tx_ring_count; i++) {
tx_ring = &qdev->tx_ring[i];
@@ -3477,9 +3460,9 @@ static int ql_configure_rings(struct ql_adapter *qdev)
/*
* The completion queue ID for the tx rings start
- * immediately after the default Q ID, which is zero.
+ * immediately after the rss rings.
*/
- tx_ring->cq_id = i + 1;
+ tx_ring->cq_id = qdev->rss_ring_count + i;
init_timer(&tx_ring->txq_clean_timer);
tx_ring->txq_clean_timer.data = (unsigned long)tx_ring;
tx_ring->txq_clean_timer.function = ql_txq_clean_timer;
@@ -3487,14 +3470,12 @@ static int ql_configure_rings(struct ql_adapter *qdev)
for (i = 0; i < qdev->rx_ring_count; i++) {
rx_ring = &qdev->rx_ring[i];
- memset((void *)rx_ring, 0, sizeof(*rx_ring));
+ memset(rx_ring, 0, sizeof(*rx_ring));
rx_ring->qdev = qdev;
rx_ring->cq_id = i;
- rx_ring->cpu = i % cpu_cnt; /* CPU to run handler on. */
- if (i == 0) { /* Default queue at index 0. */
+ if (i < qdev->rss_ring_count) {
/*
- * Default queue handles bcast/mcast plus
- * async events. Needs buffers.
+ * Inbound (RSS) queues.
*/
rx_ring->cq_len = qdev->rx_ring_size;
rx_ring->cq_size =
@@ -3507,12 +3488,11 @@ static int ql_configure_rings(struct ql_adapter *qdev)
rx_ring->sbq_size =
rx_ring->sbq_len * sizeof(__le64);
rx_ring->sbq_buf_size = SMALL_BUFFER_SIZE * 2;
- rx_ring->type = DEFAULT_Q;
- } else if (i < qdev->rss_ring_first_cq_id) {
+ rx_ring->type = RX_Q;
+ } else {
/*
- * Outbound queue handles outbound completions only.
+ * Outbound queues handling outbound completions only.
*/
- /* outbound cq is same size as tx_ring it services. */
rx_ring->cq_len = qdev->tx_ring_size;
rx_ring->cq_size =
rx_ring->cq_len * sizeof(struct ql_net_rsp_iocb);
@@ -3523,22 +3503,6 @@ static int ql_configure_rings(struct ql_adapter *qdev)
rx_ring->sbq_size = 0;
rx_ring->sbq_buf_size = 0;
rx_ring->type = TX_Q;
- } else { /* Inbound completions (RSS) queues */
- /*
- * Inbound queues handle unicast frames only.
- */
- rx_ring->cq_len = qdev->rx_ring_size;
- rx_ring->cq_size =
- rx_ring->cq_len * sizeof(struct ql_net_rsp_iocb);
- rx_ring->lbq_len = NUM_LARGE_BUFFERS;
- rx_ring->lbq_size =
- rx_ring->lbq_len * sizeof(__le64);
- rx_ring->lbq_buf_size = LARGE_BUFFER_SIZE;
- rx_ring->sbq_len = NUM_SMALL_BUFFERS;
- rx_ring->sbq_size =
- rx_ring->sbq_len * sizeof(__le64);
- rx_ring->sbq_buf_size = SMALL_BUFFER_SIZE * 2;
- rx_ring->type = RX_Q;
}
}
return 0;
--
1.6.0.2
^ permalink raw reply related
* [RFC net-next PATCH 4/4] qlge: Change large rx buffer logic.
From: Ron Mercer @ 2009-08-17 21:08 UTC (permalink / raw)
To: davem; +Cc: netdev, ron.mercer
In-Reply-To: <1250543329-15123-1-git-send-email-ron.mercer@qlogic.com>
Currently we use a single page per large rx buffer. This patch changes
this to use either chunks of pages for small MTU or multiple pages for
jumbo MTU depending on the page size.
Examples:
64k pages at 1500 MTU:
large buffers will be 2048 bytes in length and there will be 32
per page.
4k pages at 9000 MTU:
large buffer will use 4 pages as one large buffer.
Signed-off-by: Ron Mercer <ron.mercer@qlogic.com>
---
drivers/net/qlge/qlge.h | 16 ++-
drivers/net/qlge/qlge_main.c | 302 +++++++++++++++++++++++++++++-------------
2 files changed, 224 insertions(+), 94 deletions(-)
diff --git a/drivers/net/qlge/qlge.h b/drivers/net/qlge/qlge.h
index 5559c80..c731a05 100644
--- a/drivers/net/qlge/qlge.h
+++ b/drivers/net/qlge/qlge.h
@@ -53,8 +53,8 @@
#define RX_RING_SHADOW_SPACE (sizeof(u64) + \
MAX_DB_PAGES_PER_BQ(NUM_SMALL_BUFFERS) * sizeof(u64) + \
MAX_DB_PAGES_PER_BQ(NUM_LARGE_BUFFERS) * sizeof(u64))
-#define SMALL_BUFFER_SIZE 256
-#define LARGE_BUFFER_SIZE PAGE_SIZE
+#define SMALL_BUFFER_SIZE 512
+#define SMALL_BUF_MAP_SIZE (SMALL_BUFFER_SIZE / 2)
#define MAX_SPLIT_SIZE 1023
#define QLGE_SB_PAD 32
@@ -1193,9 +1193,17 @@ struct tx_ring_desc {
struct tx_ring_desc *next;
};
+struct page_chunk {
+ struct page *page; /* master page */
+ char *va; /* virt addr for this chunk */
+ u64 map; /* mapping for master */
+ unsigned int offset; /* offset for this chunk */
+ unsigned int last_flag; /* flag set for last chunk in page */
+};
+
struct bq_desc {
union {
- struct page *lbq_page;
+ struct page_chunk pg_chunk;
struct sk_buff *skb;
} p;
__le64 *addr;
@@ -1264,6 +1272,7 @@ struct rx_ring {
dma_addr_t lbq_base_dma;
void *lbq_base_indirect;
dma_addr_t lbq_base_indirect_dma;
+ struct page_chunk pg_chunk; /* current page for chunks */
struct bq_desc *lbq; /* array of control blocks */
void __iomem *lbq_prod_idx_db_reg; /* PCI doorbell mem area + 0x18 */
u32 lbq_prod_idx; /* current sw prod idx */
@@ -1497,6 +1506,7 @@ struct ql_adapter {
struct rx_ring rx_ring[MAX_RX_RINGS];
struct tx_ring tx_ring[MAX_TX_RINGS];
+ unsigned int lbq_buf_order;
int rx_csum;
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index dcac9f2..a057ae1 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -996,6 +996,11 @@ end:
return status;
}
+static inline unsigned int ql_lbq_block_size(struct ql_adapter *qdev)
+{
+ return PAGE_SIZE << qdev->lbq_buf_order;
+}
+
/* Get the next large buffer. */
static struct bq_desc *ql_get_curr_lbuf(struct rx_ring *rx_ring)
{
@@ -1007,6 +1012,28 @@ static struct bq_desc *ql_get_curr_lbuf(struct rx_ring *rx_ring)
return lbq_desc;
}
+static struct bq_desc *ql_get_curr_lchunk(struct ql_adapter *qdev,
+ struct rx_ring *rx_ring)
+{
+ struct bq_desc *lbq_desc = ql_get_curr_lbuf(rx_ring);
+
+ pci_dma_sync_single_for_cpu(qdev->pdev,
+ pci_unmap_addr(lbq_desc, mapaddr),
+ rx_ring->lbq_buf_size,
+ PCI_DMA_FROMDEVICE);
+
+ /* If it's the last chunk of our master page then
+ * we unmap it.
+ */
+ if ((lbq_desc->p.pg_chunk.offset + rx_ring->lbq_buf_size)
+ == ql_lbq_block_size(qdev))
+ pci_unmap_page(qdev->pdev,
+ lbq_desc->p.pg_chunk.map,
+ ql_lbq_block_size(qdev),
+ PCI_DMA_FROMDEVICE);
+ return lbq_desc;
+}
+
/* Get the next small buffer. */
static struct bq_desc *ql_get_curr_sbuf(struct rx_ring *rx_ring)
{
@@ -1034,6 +1061,53 @@ static void ql_write_cq_idx(struct rx_ring *rx_ring)
ql_write_db_reg(rx_ring->cnsmr_idx, rx_ring->cnsmr_idx_db_reg);
}
+static int ql_get_next_chunk(struct ql_adapter *qdev, struct rx_ring *rx_ring,
+ struct bq_desc *lbq_desc)
+{
+ if (!rx_ring->pg_chunk.page) {
+ u64 map;
+ rx_ring->pg_chunk.page = alloc_pages(__GFP_COLD | __GFP_COMP |
+ GFP_ATOMIC,
+ qdev->lbq_buf_order);
+ if (unlikely(!rx_ring->pg_chunk.page)) {
+ QPRINTK(qdev, DRV, ERR,
+ "page allocation failed.\n");
+ return -ENOMEM;
+ }
+ rx_ring->pg_chunk.offset = 0;
+ map = pci_map_page(qdev->pdev, rx_ring->pg_chunk.page,
+ 0, ql_lbq_block_size(qdev),
+ PCI_DMA_FROMDEVICE);
+ if (pci_dma_mapping_error(qdev->pdev, map)) {
+ __free_pages(rx_ring->pg_chunk.page,
+ qdev->lbq_buf_order);
+ QPRINTK(qdev, DRV, ERR,
+ "PCI mapping failed.\n");
+ return -ENOMEM;
+ }
+ rx_ring->pg_chunk.map = map;
+ rx_ring->pg_chunk.va = page_address(rx_ring->pg_chunk.page);
+ }
+
+ /* Copy the current master pg_chunk info
+ * to the current descriptor.
+ */
+ lbq_desc->p.pg_chunk = rx_ring->pg_chunk;
+
+ /* Adjust the master page chunk for next
+ * buffer get.
+ */
+ rx_ring->pg_chunk.offset += rx_ring->lbq_buf_size;
+ if (rx_ring->pg_chunk.offset == ql_lbq_block_size(qdev)) {
+ rx_ring->pg_chunk.page = NULL;
+ lbq_desc->p.pg_chunk.last_flag = 1;
+ } else {
+ rx_ring->pg_chunk.va += rx_ring->lbq_buf_size;
+ get_page(rx_ring->pg_chunk.page);
+ lbq_desc->p.pg_chunk.last_flag = 0;
+ }
+ return 0;
+}
/* Process (refill) a large buffer queue. */
static void ql_update_lbq(struct ql_adapter *qdev, struct rx_ring *rx_ring)
{
@@ -1043,39 +1117,30 @@ static void ql_update_lbq(struct ql_adapter *qdev, struct rx_ring *rx_ring)
u64 map;
int i;
- while (rx_ring->lbq_free_cnt > 16) {
+ while (rx_ring->lbq_free_cnt > 32) {
for (i = 0; i < 16; i++) {
QPRINTK(qdev, RX_STATUS, DEBUG,
"lbq: try cleaning clean_idx = %d.\n",
clean_idx);
lbq_desc = &rx_ring->lbq[clean_idx];
- if (lbq_desc->p.lbq_page == NULL) {
- QPRINTK(qdev, RX_STATUS, DEBUG,
- "lbq: getting new page for index %d.\n",
- lbq_desc->index);
- lbq_desc->p.lbq_page = alloc_page(GFP_ATOMIC);
- if (lbq_desc->p.lbq_page == NULL) {
- rx_ring->lbq_clean_idx = clean_idx;
- QPRINTK(qdev, RX_STATUS, ERR,
- "Couldn't get a page.\n");
- return;
- }
- map = pci_map_page(qdev->pdev,
- lbq_desc->p.lbq_page,
- 0, PAGE_SIZE,
- PCI_DMA_FROMDEVICE);
- if (pci_dma_mapping_error(qdev->pdev, map)) {
- rx_ring->lbq_clean_idx = clean_idx;
- put_page(lbq_desc->p.lbq_page);
- lbq_desc->p.lbq_page = NULL;
- QPRINTK(qdev, RX_STATUS, ERR,
- "PCI mapping failed.\n");
+ if (ql_get_next_chunk(qdev, rx_ring, lbq_desc)) {
+ QPRINTK(qdev, IFUP, ERR,
+ "Could not get a page chunk.\n");
return;
}
- pci_unmap_addr_set(lbq_desc, mapaddr, map);
- pci_unmap_len_set(lbq_desc, maplen, PAGE_SIZE);
- *lbq_desc->addr = cpu_to_le64(map);
- }
+
+ map = lbq_desc->p.pg_chunk.map +
+ lbq_desc->p.pg_chunk.offset;
+ pci_unmap_addr_set(lbq_desc, mapaddr, map);
+ pci_unmap_len_set(lbq_desc,
+ maplen,
+ rx_ring->lbq_buf_size);
+
+ *lbq_desc->addr = cpu_to_le64(map);
+
+ pci_dma_sync_single_for_device(qdev->pdev, map,
+ rx_ring->lbq_buf_size,
+ PCI_DMA_FROMDEVICE);
clean_idx++;
if (clean_idx == rx_ring->lbq_len)
clean_idx = 0;
@@ -1118,7 +1183,7 @@ static void ql_update_sbq(struct ql_adapter *qdev, struct rx_ring *rx_ring)
sbq_desc->index);
sbq_desc->p.skb =
netdev_alloc_skb(qdev->ndev,
- rx_ring->sbq_buf_size);
+ SMALL_BUFFER_SIZE);
if (sbq_desc->p.skb == NULL) {
QPRINTK(qdev, PROBE, ERR,
"Couldn't get an skb.\n");
@@ -1128,8 +1193,8 @@ static void ql_update_sbq(struct ql_adapter *qdev, struct rx_ring *rx_ring)
skb_reserve(sbq_desc->p.skb, QLGE_SB_PAD);
map = pci_map_single(qdev->pdev,
sbq_desc->p.skb->data,
- rx_ring->sbq_buf_size /
- 2, PCI_DMA_FROMDEVICE);
+ rx_ring->sbq_buf_size,
+ PCI_DMA_FROMDEVICE);
if (pci_dma_mapping_error(qdev->pdev, map)) {
QPRINTK(qdev, IFUP, ERR, "PCI mapping failed.\n");
rx_ring->sbq_clean_idx = clean_idx;
@@ -1139,7 +1204,7 @@ static void ql_update_sbq(struct ql_adapter *qdev, struct rx_ring *rx_ring)
}
pci_unmap_addr_set(sbq_desc, mapaddr, map);
pci_unmap_len_set(sbq_desc, maplen,
- rx_ring->sbq_buf_size / 2);
+ rx_ring->sbq_buf_size);
*sbq_desc->addr = cpu_to_le64(map);
}
@@ -1451,48 +1516,41 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
* chain it to the header buffer's skb and let
* it rip.
*/
- lbq_desc = ql_get_curr_lbuf(rx_ring);
- pci_unmap_page(qdev->pdev,
- pci_unmap_addr(lbq_desc,
- mapaddr),
- pci_unmap_len(lbq_desc, maplen),
- PCI_DMA_FROMDEVICE);
+ lbq_desc = ql_get_curr_lchunk(qdev, rx_ring);
QPRINTK(qdev, RX_STATUS, DEBUG,
- "Chaining page to skb.\n");
- skb_fill_page_desc(skb, 0, lbq_desc->p.lbq_page,
- 0, length);
+ "Chaining page at offset = %d,"
+ "for %d bytes to skb.\n",
+ lbq_desc->p.pg_chunk.offset, length);
+ skb_fill_page_desc(skb, 0, lbq_desc->p.pg_chunk.page,
+ lbq_desc->p.pg_chunk.offset,
+ length);
skb->len += length;
skb->data_len += length;
skb->truesize += length;
- lbq_desc->p.lbq_page = NULL;
} else {
/*
* The headers and data are in a single large buffer. We
* copy it to a new skb and let it go. This can happen with
* jumbo mtu on a non-TCP/UDP frame.
*/
- lbq_desc = ql_get_curr_lbuf(rx_ring);
+ lbq_desc = ql_get_curr_lchunk(qdev, rx_ring);
skb = netdev_alloc_skb(qdev->ndev, length);
if (skb == NULL) {
- QPRINTK(qdev, PROBE, DEBUG,
+ QPRINTK(qdev, PROBE, ERR,
"No skb available, drop the packet.\n");
return NULL;
}
- pci_unmap_page(qdev->pdev,
- pci_unmap_addr(lbq_desc,
- mapaddr),
- pci_unmap_len(lbq_desc, maplen),
- PCI_DMA_FROMDEVICE);
skb_reserve(skb, NET_IP_ALIGN);
QPRINTK(qdev, RX_STATUS, DEBUG,
"%d bytes of headers and data in large. Chain page to new skb and pull tail.\n", length);
- skb_fill_page_desc(skb, 0, lbq_desc->p.lbq_page,
- 0, length);
+ skb_fill_page_desc(skb, 0,
+ lbq_desc->p.pg_chunk.page,
+ lbq_desc->p.pg_chunk.offset,
+ length);
skb->len += length;
skb->data_len += length;
skb->truesize += length;
length -= length;
- lbq_desc->p.lbq_page = NULL;
__pskb_pull_tail(skb,
(ib_mac_rsp->flags2 & IB_MAC_IOCB_RSP_V) ?
VLAN_ETH_HLEN : ETH_HLEN);
@@ -1509,8 +1567,7 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
* frames. If the MTU goes up we could
* eventually be in trouble.
*/
- int size, offset, i = 0;
- __le64 *bq, bq_array[8];
+ int size, i = 0;
sbq_desc = ql_get_curr_sbuf(rx_ring);
pci_unmap_single(qdev->pdev,
pci_unmap_addr(sbq_desc, mapaddr),
@@ -1529,37 +1586,25 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
QPRINTK(qdev, RX_STATUS, DEBUG,
"%d bytes of headers & data in chain of large.\n", length);
skb = sbq_desc->p.skb;
- bq = &bq_array[0];
- memcpy(bq, skb->data, sizeof(bq_array));
sbq_desc->p.skb = NULL;
skb_reserve(skb, NET_IP_ALIGN);
- } else {
- QPRINTK(qdev, RX_STATUS, DEBUG,
- "Headers in small, %d bytes of data in chain of large.\n", length);
- bq = (__le64 *)sbq_desc->p.skb->data;
}
while (length > 0) {
- lbq_desc = ql_get_curr_lbuf(rx_ring);
- pci_unmap_page(qdev->pdev,
- pci_unmap_addr(lbq_desc,
- mapaddr),
- pci_unmap_len(lbq_desc,
- maplen),
- PCI_DMA_FROMDEVICE);
- size = (length < PAGE_SIZE) ? length : PAGE_SIZE;
- offset = 0;
+ lbq_desc = ql_get_curr_lchunk(qdev, rx_ring);
+ size = (length < rx_ring->lbq_buf_size) ? length :
+ rx_ring->lbq_buf_size;
QPRINTK(qdev, RX_STATUS, DEBUG,
"Adding page %d to skb for %d bytes.\n",
i, size);
- skb_fill_page_desc(skb, i, lbq_desc->p.lbq_page,
- offset, size);
+ skb_fill_page_desc(skb, i,
+ lbq_desc->p.pg_chunk.page,
+ lbq_desc->p.pg_chunk.offset,
+ size);
skb->len += size;
skb->data_len += size;
skb->truesize += size;
length -= size;
- lbq_desc->p.lbq_page = NULL;
- bq++;
i++;
}
__pskb_pull_tail(skb, (ib_mac_rsp->flags2 & IB_MAC_IOCB_RSP_V) ?
@@ -1576,7 +1621,7 @@ static void ql_process_mac_rx_intr(struct ql_adapter *qdev,
struct net_device *ndev = qdev->ndev;
struct sk_buff *skb = NULL;
u16 vlan_id = (le16_to_cpu(ib_mac_rsp->vlan_id) &
- IB_MAC_IOCB_RSP_VLAN_MASK)
+ IB_MAC_IOCB_RSP_VLAN_MASK);
QL_DUMP_IB_MAC_RSP(ib_mac_rsp);
@@ -2322,25 +2367,34 @@ err:
return -ENOMEM;
}
-static void ql_free_lbq_buffers(struct ql_adapter *qdev, struct rx_ring *rx_ring)
+static void ql_free_lbq_buffers(struct ql_adapter *qdev,
+ struct rx_ring *rx_ring)
{
- int i;
struct bq_desc *lbq_desc;
- for (i = 0; i < rx_ring->lbq_len; i++) {
- lbq_desc = &rx_ring->lbq[i];
- if (lbq_desc->p.lbq_page) {
- pci_unmap_page(qdev->pdev,
- pci_unmap_addr(lbq_desc, mapaddr),
- pci_unmap_len(lbq_desc, maplen),
- PCI_DMA_FROMDEVICE);
+ uint32_t curr_idx, clean_idx;
+
+ curr_idx = rx_ring->lbq_curr_idx;
+ clean_idx = rx_ring->lbq_clean_idx;
+ while (curr_idx != clean_idx) {
+ lbq_desc = &rx_ring->lbq[curr_idx];
- put_page(lbq_desc->p.lbq_page);
- lbq_desc->p.lbq_page = NULL;
+ if (lbq_desc->p.pg_chunk.last_flag) {
+ pci_unmap_page(qdev->pdev,
+ lbq_desc->p.pg_chunk.map,
+ ql_lbq_block_size(qdev),
+ PCI_DMA_FROMDEVICE);
+ lbq_desc->p.pg_chunk.last_flag = 0;
}
+
+ put_page(lbq_desc->p.pg_chunk.page);
+ lbq_desc->p.pg_chunk.page = NULL;
+
+ if (++curr_idx == rx_ring->lbq_len)
+ curr_idx = 0;
+
}
}
-
static void ql_free_sbq_buffers(struct ql_adapter *qdev, struct rx_ring *rx_ring)
{
int i;
@@ -2713,7 +2767,7 @@ static int ql_start_rx_ring(struct ql_adapter *qdev, struct rx_ring *rx_ring)
cqicb->sbq_addr =
cpu_to_le64(rx_ring->sbq_base_indirect_dma);
cqicb->sbq_buf_size =
- cpu_to_le16((u16)(rx_ring->sbq_buf_size/2));
+ cpu_to_le16((u16)(rx_ring->sbq_buf_size));
bq_len = (rx_ring->sbq_len == 65536) ? 0 :
(u16) rx_ring->sbq_len;
cqicb->sbq_len = cpu_to_le16(bq_len);
@@ -2732,6 +2786,7 @@ static int ql_start_rx_ring(struct ql_adapter *qdev, struct rx_ring *rx_ring)
/* Inbound completion handling rx_rings run in
* separate NAPI contexts.
*/
+ rx_ring->napi.dev = qdev->ndev;
netif_napi_add(qdev->ndev, &rx_ring->napi, ql_napi_poll_msix,
64);
cqicb->irq_delay = cpu_to_le16(qdev->rx_coalesce_usecs);
@@ -3210,7 +3265,7 @@ static int ql_adapter_initialize(struct ql_adapter *qdev)
ql_write32(qdev, FSC, mask | value);
ql_write32(qdev, SPLT_HDR, SPLT_HDR_EP |
- min(SMALL_BUFFER_SIZE, MAX_SPLIT_SIZE));
+ min(SMALL_BUF_MAP_SIZE, MAX_SPLIT_SIZE));
/* Start up the rx queues. */
for (i = 0; i < qdev->rx_ring_count; i++) {
@@ -3435,6 +3490,10 @@ static int ql_configure_rings(struct ql_adapter *qdev)
struct rx_ring *rx_ring;
struct tx_ring *tx_ring;
int cpu_cnt = min(MAX_CPUS, (int)num_online_cpus());
+ unsigned int lbq_buf_len =
+ (qdev->ndev->mtu > 1500) ? 16384 : 2048;
+
+ qdev->lbq_buf_order = get_order(lbq_buf_len);
/* Allocate outbound completion ring for each CPU. */
qdev->tx_ring_count = cpu_cnt;
@@ -3483,11 +3542,11 @@ static int ql_configure_rings(struct ql_adapter *qdev)
rx_ring->lbq_len = NUM_LARGE_BUFFERS;
rx_ring->lbq_size =
rx_ring->lbq_len * sizeof(__le64);
- rx_ring->lbq_buf_size = LARGE_BUFFER_SIZE;
+ rx_ring->lbq_buf_size = (u16)lbq_buf_len;
rx_ring->sbq_len = NUM_SMALL_BUFFERS;
rx_ring->sbq_size =
rx_ring->sbq_len * sizeof(__le64);
- rx_ring->sbq_buf_size = SMALL_BUFFER_SIZE * 2;
+ rx_ring->sbq_buf_size = SMALL_BUF_MAP_SIZE;
rx_ring->type = RX_Q;
} else {
/*
@@ -3532,14 +3591,64 @@ error_up:
return err;
}
+static int ql_change_rx_buffers(struct ql_adapter *qdev)
+{
+ struct rx_ring *rx_ring;
+ int i, status;
+ u32 lbq_buf_len;
+
+ /* Wait for an oustanding reset to complete. */
+ if (!test_bit(QL_ADAPTER_UP, &qdev->flags)) {
+ int i = 3;
+ while (i-- && !test_bit(QL_ADAPTER_UP, &qdev->flags)) {
+ QPRINTK(qdev, IFUP, ERR,
+ "Waiting for adapter UP...\n");
+ ssleep(1);
+ }
+
+ if (!i) {
+ QPRINTK(qdev, IFUP, ERR,
+ "Timed out waiting for adapter UP\n");
+ return -ETIMEDOUT;
+ }
+ }
+
+ status = ql_adapter_down(qdev);
+ if (status)
+ goto error;
+
+ /* Get the new rx buffer size. */
+ lbq_buf_len = (qdev->ndev->mtu > 1500) ? 16384 : 2048;
+ qdev->lbq_buf_order = get_order(lbq_buf_len);
+
+ for (i = 0; i < qdev->rss_ring_count; i++) {
+ rx_ring = &qdev->rx_ring[i];
+ /* Set the new size. */
+ rx_ring->lbq_buf_size = lbq_buf_len;
+ }
+
+ status = ql_adapter_up(qdev);
+ if (status)
+ goto error;
+
+ return status;
+error:
+ QPRINTK(qdev, IFUP, ALERT,
+ "Driver up/down cycle failed, closing device.\n");
+ rtnl_lock();
+ set_bit(QL_ADAPTER_UP, &qdev->flags);
+ dev_close(qdev->ndev);
+ rtnl_unlock();
+ return status;
+}
+
static int qlge_change_mtu(struct net_device *ndev, int new_mtu)
{
struct ql_adapter *qdev = netdev_priv(ndev);
+ int status = 0;
if (ndev->mtu == 1500 && new_mtu == 9000) {
QPRINTK(qdev, IFUP, ERR, "Changing to jumbo MTU.\n");
- queue_delayed_work(qdev->workqueue,
- &qdev->mpi_port_cfg_work, 0);
} else if (ndev->mtu == 9000 && new_mtu == 1500) {
QPRINTK(qdev, IFUP, ERR, "Changing to normal MTU.\n");
} else if ((ndev->mtu == 1500 && new_mtu == 1500) ||
@@ -3547,8 +3656,19 @@ static int qlge_change_mtu(struct net_device *ndev, int new_mtu)
return 0;
} else
return -EINVAL;
+
+ if (!netif_running(qdev->ndev)) {
+ ndev->mtu = new_mtu;
+ return 0;
+ }
+
ndev->mtu = new_mtu;
- return 0;
+ status = ql_change_rx_buffers(qdev);
+ if (status)
+ QPRINTK(qdev, IFUP, ERR,
+ "Changing MTU failed.\n");
+
+ return status;
}
static struct net_device_stats *qlge_get_stats(struct net_device
--
1.6.0.2
^ permalink raw reply related
* RE: [RFC] bridge: prevent hairpin and STP problems?
From: Fischer, Anna @ 2009-08-17 21:16 UTC (permalink / raw)
To: Stephen Hemminger, David Miller
Cc: netdev@vger.kernel.org, ptcongdon@ucdavis.edu,
evb@yahoogroups.com, bridge@lists.linux-foundation.org,
kaber@trash.net, arnd@arndb.de, Dickson, Mike (ISS Software),
adobriyan@gmail.com, bridge@osdl.org, Arnd Bergmann
In-Reply-To: <20090814144133.34ac9d94@nehalam>
> Subject: [RFC] bridge: prevent hairpin and STP problems?
>
> Do we need to add this to block Spanning Tree from being enabled
> with hairpin mode? I am not sure what the exact usage of hairpin
> mode and if it is possible to create loops and get STP confusion.
>
> For comment only, do not apply as is.
Your patch disables STP on the whole bridge if one or more ports
are set to hairpin mode.
However, I don't really see that this is necessary.
A hairpin mode port should not reflect BPDUs, because otherwise the
connected port would think it has detected a loop. The hairpin mode
port should still be able to generate BPDUs though, and in any case
the bridge should still be able to run STP.
The hairpin patch we submitted reflects packets on the forwarding /
data path whereas BPDUs are processed with a separate hook, so we
should not be reflecting BPDUs back out of a hairpin mode port.
Anna
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Sridhar Samudrala @ 2009-08-17 21:50 UTC (permalink / raw)
To: Christoph Lameter; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <alpine.DEB.1.10.0908171642500.20399@gentwo.org>
On Mon, 2009-08-17 at 16:46 -0400, Christoph Lameter wrote:
> On Mon, 17 Aug 2009, Nivedita Singhvi wrote:
>
> > Is there any chance this is getting dropped at qdisc?
> > Can you check with tc?
>
> I checked and the counter there is zero.
>
> :/home/clameter# tc -s qdisc show
>
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 3042712708 bytes 8896862 pkt (dropped 0, overlimits 0 requeues 56303)
> rate 0bit 0pps backlog 0b 0p requeues 56303
What about ethtool -S ? Does it report any errors?
Thanks
Sridhar
^ permalink raw reply
* Re: mlx4 2.6.31-rc5: SW2HW_EQ failed.
From: Roland Dreier @ 2009-08-17 22:04 UTC (permalink / raw)
To: Christoph Lameter; +Cc: netdev, Yevgeny Petrilin
In-Reply-To: <alpine.DEB.1.10.0908171523330.20399@gentwo.org>
> mlx4 fails to initialize here:
>
>
> [ 9.973940] mlx4_core 0000:04:00.0: irq 93 for MSI/MSI-X
> [ 9.983108] sr 1:0:0:0: Attached scsi CD-ROM sr0
> [ 9.988209] ses 0:0:32:0: Attached scsi generic sg0 type 13
> [ 9.999376] sd 0:2:0:0: Attached scsi generic sg1 type 0
> [ 10.010024] sr 1:0:0:0: Attached scsi generic sg2 type 5
> [ 10.256371] mlx4_core 0000:04:00.0: SW2HW_EQ failed (-5)
> [ 10.270103] mlx4_core 0000:04:00.0: Failed to initialize event queue
> table, aborting.
> [ 10.288768] mlx4_core 0000:04:00.0: PCI INT A disabled
> [ 10.299057] mlx4_core: probe of 0000:04:00.0 failed with error -5
Thanks for the report... could you try loading mlx4_core with
debug_level=1 to see if anything interesting comes out? The kernel log
here indicates that the device FW is giving us "internal error" when we
try to initialize event queues.
Also what kernel is this with? Anything unusual about the system (arch
!= x86, lots of CPUs or RAM, etc)?
One stab in the dark would be to try a423b8a0 ("mlx4_core: Allocate and
map sufficient ICM memory for EQ context") from the for-next branch of
my infiniband.git kernel.org tree. I would only think that matters if
you have 32 or more CPUs, but maybe you do...
- R.
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-17 22:13 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <1250545839.25939.21.camel@w-sridhar.beaverton.ibm.com>
On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
> What about ethtool -S ? Does it report any errors?
Neither. This is is a broadcom bnx2 NIC.
^ permalink raw reply
* Re: mlx4 2.6.31-rc5: SW2HW_EQ failed.
From: Christoph Lameter @ 2009-08-17 22:17 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, Yevgeny Petrilin
In-Reply-To: <ada7hx22j3g.fsf@cisco.com>
On Mon, 17 Aug 2009, Roland Dreier wrote:
>
> > mlx4 fails to initialize here:
> >
> >
> > [ 9.973940] mlx4_core 0000:04:00.0: irq 93 for MSI/MSI-X
> > [ 9.983108] sr 1:0:0:0: Attached scsi CD-ROM sr0
> > [ 9.988209] ses 0:0:32:0: Attached scsi generic sg0 type 13
> > [ 9.999376] sd 0:2:0:0: Attached scsi generic sg1 type 0
> > [ 10.010024] sr 1:0:0:0: Attached scsi generic sg2 type 5
> > [ 10.256371] mlx4_core 0000:04:00.0: SW2HW_EQ failed (-5)
> > [ 10.270103] mlx4_core 0000:04:00.0: Failed to initialize event queue
> > table, aborting.
> > [ 10.288768] mlx4_core 0000:04:00.0: PCI INT A disabled
> > [ 10.299057] mlx4_core: probe of 0000:04:00.0 failed with error -5
>
> Thanks for the report... could you try loading mlx4_core with
> debug_level=1 to see if anything interesting comes out? The kernel log
> here indicates that the device FW is giving us "internal error" when we
> try to initialize event queues.
Device FW??? The log you wanted follows at the end of this message.
> Also what kernel is this with? Anything unusual about the system (arch
> != x86, lots of CPUs or RAM, etc)?
Dell R620 two quad nehalems. Build with standard debian kernel config.
> One stab in the dark would be to try a423b8a0 ("mlx4_core: Allocate and
> map sufficient ICM memory for EQ context") from the for-next branch of
> my infiniband.git kernel.org tree. I would only think that matters if
> you have 32 or more CPUs, but maybe you do...
We have 16 processors.
[ 7423.298136] mlx4_core: Mellanox ConnectX core driver v0.01 (May 1,
2007)
[ 7423.298137] mlx4_core: Initializing 0000:04:00.0
[ 7423.298147] mlx4_core 0000:04:00.0: PCI INT A -> GSI 38 (level, low) ->
IRQ 38
[ 7423.298165] mlx4_core 0000:04:00.0: setting latency timer to 64
[ 7424.298240] mlx4_core 0000:04:00.0: FW version 2.6.000 (cmd intf rev
3), max commands 16
[ 7424.298242] mlx4_core 0000:04:00.0: Catastrophic error buffer at
0x1f020, size 0x10, BAR 0
[ 7424.298243] mlx4_core 0000:04:00.0: FW size 385 KB
[ 7424.298245] mlx4_core 0000:04:00.0: Clear int @ f0058, BAR 0
[ 7424.299848] mlx4_core 0000:04:00.0: Mapped 26 chunks/6168 KB for FW.
[ 7424.921833] mlx4_core 0000:04:00.0: BlueFlame available (reg size 512,
regs/page 256)
[ 7424.921952] mlx4_core 0000:04:00.0: Base MM extensions: flags 00000cc0,
rsvd L_Key 00000500
[ 7424.921954] mlx4_core 0000:04:00.0: Max ICM size 4294967296 MB
[ 7424.921955] mlx4_core 0000:04:00.0: Max QPs: 16777216, reserved QPs:
64, entry size: 256
[ 7424.921957] mlx4_core 0000:04:00.0: Max SRQs: 16777216, reserved SRQs:
64, entry size: 128
[ 7424.921959] mlx4_core 0000:04:00.0: Max CQs: 16777216, reserved CQs:
128, entry size: 128
[ 7424.921960] mlx4_core 0000:04:00.0: Max EQs: 512, reserved EQs: 4,
entry size: 128
[ 7424.921961] mlx4_core 0000:04:00.0: reserved MPTs: 16, reserved MTTs:
16
[ 7424.921963] mlx4_core 0000:04:00.0: Max PDs: 8388608, reserved PDs: 4,
reserved UARs: 1
[ 7424.921964] mlx4_core 0000:04:00.0: Max QP/MCG: 8388608, reserved MGMs:
0
[ 7424.921966] mlx4_core 0000:04:00.0: Max CQEs: 4194304, max WQEs: 16384,
max SRQ WQEs: 16384
[ 7424.921967] mlx4_core 0000:04:00.0: Local CA ACK delay: 15, max MTU:
4096, port width cap: 3
[ 7424.921969] mlx4_core 0000:04:00.0: Max SQ desc size: 1008, max SQ S/G:
62
[ 7424.921970] mlx4_core 0000:04:00.0: Max RQ desc size: 512, max RQ S/G:
32
[ 7424.921971] mlx4_core 0000:04:00.0: Max GSO size: 131072
[ 7424.921972] mlx4_core 0000:04:00.0: DEV_CAP flags:
[ 7424.921974] mlx4_core 0000:04:00.0: RC transport
[ 7424.921975] mlx4_core 0000:04:00.0: UC transport
[ 7424.921976] mlx4_core 0000:04:00.0: UD transport
[ 7424.921977] mlx4_core 0000:04:00.0: XRC transport
[ 7424.921978] mlx4_core 0000:04:00.0: FCoIB support
[ 7424.921979] mlx4_core 0000:04:00.0: SRQ support
[ 7424.921980] mlx4_core 0000:04:00.0: IPoIB checksum offload
[ 7424.921981] mlx4_core 0000:04:00.0: P_Key violation counter
[ 7424.921982] mlx4_core 0000:04:00.0: Q_Key violation counter
[ 7424.921983] mlx4_core 0000:04:00.0: DPDP
[ 7424.921984] mlx4_core 0000:04:00.0: APM support
[ 7424.921985] mlx4_core 0000:04:00.0: Atomic ops support
[ 7424.921986] mlx4_core 0000:04:00.0: Address vector port checking
support
[ 7424.921988] mlx4_core 0000:04:00.0: UD multicast support
[ 7424.921989] mlx4_core 0000:04:00.0: Router support
[ 7424.921993] mlx4_core 0000:04:00.0: profile[ 0] ( CMPT): 2^26
entries @ 0x 0, size 0x 100000000
[ 7424.921995] mlx4_core 0000:04:00.0: profile[ 1] (RDMARC): 2^21
entries @ 0x 100000000, size 0x 4000000
[ 7424.921997] mlx4_core 0000:04:00.0: profile[ 2] ( MTT): 2^20
entries @ 0x 104000000, size 0x 4000000
[ 7424.921999] mlx4_core 0000:04:00.0: profile[ 3] ( QP): 2^17
entries @ 0x 108000000, size 0x 2000000
[ 7424.922001] mlx4_core 0000:04:00.0: profile[ 4] ( ALTC): 2^17
entries @ 0x 10a000000, size 0x 800000
[ 7424.922003] mlx4_core 0000:04:00.0: profile[ 5] ( SRQ): 2^16
entries @ 0x 10a800000, size 0x 800000
[ 7424.922005] mlx4_core 0000:04:00.0: profile[ 6] ( CQ): 2^16
entries @ 0x 10b000000, size 0x 800000
[ 7424.922007] mlx4_core 0000:04:00.0: profile[ 7] ( DMPT): 2^17
entries @ 0x 10b800000, size 0x 800000
[ 7424.922009] mlx4_core 0000:04:00.0: profile[ 8] ( MCG): 2^13
entries @ 0x 10c000000, size 0x 200000
[ 7424.922011] mlx4_core 0000:04:00.0: profile[ 9] ( AUXC): 2^17
entries @ 0x 10c200000, size 0x 20000
[ 7424.922013] mlx4_core 0000:04:00.0: profile[10] ( EQ): 2^06
entries @ 0x 10c220000, size 0x 2000
[ 7424.922014] mlx4_core 0000:04:00.0: HCA context memory: reserving
4393096 KB
[ 7424.922034] mlx4_core 0000:04:00.0: 4393096 KB of HCA context requires
8620 KB aux memory.
[ 7424.942888] mlx4_core 0000:04:00.0: Mapped 37 chunks/8620 KB for ICM
aux.
[ 7424.943998] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 0 for
ICM.
[ 7424.945080] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 40000000
for ICM.
[ 7424.946162] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 80000000
for ICM.
[ 7424.946192] mlx4_core 0000:04:00.0: Mapped 1 chunks/4 KB at c0000000
for ICM.
[ 7424.946221] mlx4_core 0000:04:00.0: Mapped page at 1a79c4000 to
10c220000 for ICM.
[ 7424.947283] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 104000000
for ICM.
[ 7424.948380] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10b800000
for ICM.
[ 7424.949441] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 108000000
for ICM.
[ 7424.949976] mlx4_core 0000:04:00.0: Mapped 1 chunks/128 KB at 10c200000
for ICM.
[ 7424.951037] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10a000000
for ICM.
[ 7424.952098] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 100000000
for ICM.
[ 7424.953159] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10b000000
for ICM.
[ 7424.954219] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10a800000
for ICM.
[ 7424.955279] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c000000
for ICM.
[ 7424.956339] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c040000
for ICM.
[ 7424.957399] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c080000
for ICM.
[ 7424.958458] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c0c0000
for ICM.
[ 7424.959519] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c100000
for ICM.
[ 7424.960581] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c140000
for ICM.
[ 7424.961641] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c180000
for ICM.
[ 7424.962702] mlx4_core 0000:04:00.0: Mapped 1 chunks/256 KB at 10c1c0000
for ICM.
[ 7425.199430] mlx4_core 0000:04:00.0: irq 70 for MSI/MSI-X
[ 7425.199432] mlx4_core 0000:04:00.0: irq 71 for MSI/MSI-X
[ 7425.199434] mlx4_core 0000:04:00.0: irq 72 for MSI/MSI-X
[ 7425.199436] mlx4_core 0000:04:00.0: irq 73 for MSI/MSI-X
[ 7425.199437] mlx4_core 0000:04:00.0: irq 74 for MSI/MSI-X
[ 7425.199439] mlx4_core 0000:04:00.0: irq 75 for MSI/MSI-X
[ 7425.199441] mlx4_core 0000:04:00.0: irq 76 for MSI/MSI-X
[ 7425.199443] mlx4_core 0000:04:00.0: irq 77 for MSI/MSI-X
[ 7425.199445] mlx4_core 0000:04:00.0: irq 78 for MSI/MSI-X
[ 7425.199446] mlx4_core 0000:04:00.0: irq 79 for MSI/MSI-X
[ 7425.199448] mlx4_core 0000:04:00.0: irq 80 for MSI/MSI-X
[ 7425.199450] mlx4_core 0000:04:00.0: irq 81 for MSI/MSI-X
[ 7425.199452] mlx4_core 0000:04:00.0: irq 82 for MSI/MSI-X
[ 7425.199454] mlx4_core 0000:04:00.0: irq 83 for MSI/MSI-X
[ 7425.199456] mlx4_core 0000:04:00.0: irq 84 for MSI/MSI-X
[ 7425.199457] mlx4_core 0000:04:00.0: irq 85 for MSI/MSI-X
[ 7425.199459] mlx4_core 0000:04:00.0: irq 86 for MSI/MSI-X
[ 7425.199461] mlx4_core 0000:04:00.0: irq 87 for MSI/MSI-X
[ 7425.199463] mlx4_core 0000:04:00.0: irq 88 for MSI/MSI-X
[ 7425.199464] mlx4_core 0000:04:00.0: irq 89 for MSI/MSI-X
[ 7425.199466] mlx4_core 0000:04:00.0: irq 90 for MSI/MSI-X
[ 7425.199468] mlx4_core 0000:04:00.0: irq 91 for MSI/MSI-X
[ 7425.199470] mlx4_core 0000:04:00.0: irq 92 for MSI/MSI-X
[ 7425.199472] mlx4_core 0000:04:00.0: irq 93 for MSI/MSI-X
[ 7425.199474] mlx4_core 0000:04:00.0: irq 94 for MSI/MSI-X
[ 7425.199475] mlx4_core 0000:04:00.0: irq 95 for MSI/MSI-X
[ 7425.199477] mlx4_core 0000:04:00.0: irq 96 for MSI/MSI-X
[ 7425.199479] mlx4_core 0000:04:00.0: irq 97 for MSI/MSI-X
[ 7425.199481] mlx4_core 0000:04:00.0: irq 98 for MSI/MSI-X
[ 7425.199483] mlx4_core 0000:04:00.0: irq 99 for MSI/MSI-X
[ 7425.199485] mlx4_core 0000:04:00.0: irq 100 for MSI/MSI-X
[ 7425.199487] mlx4_core 0000:04:00.0: irq 101 for MSI/MSI-X
[ 7425.199488] mlx4_core 0000:04:00.0: irq 102 for MSI/MSI-X
[ 7425.472921] mlx4_core 0000:04:00.0: SW2HW_EQ failed (-5)
[ 7425.476030] mlx4_core 0000:04:00.0: Failed to initialize event queue
table, aborting.
[ 7425.494648] mlx4_core 0000:04:00.0: PCI INT A disabled
[ 7425.494660] mlx4_core: probe of 0000:04:00.0 failed with error -5
^ permalink raw reply
* [PATCH V2] lib/vsprintf.c: Add "%pI6c" - print pointer as compressed ipv6 address
From: Joe Perches @ 2009-08-17 22:29 UTC (permalink / raw)
To: Jens Rosenboom; +Cc: David Miller, chuck.lever, brian.haley, netdev
In-Reply-To: <1250522309.16632.45.camel@fnki-nb00130>
On Mon, 2009-08-17 at 17:18 +0200, Jens Rosenboom wrote:
> Two small optimizations:
Thanks. Another is to use the calculated "longest" to
increment i after the :: instead of counting 0's again.
Another possibility might be to make all the ip formatting
in vsprintf #ifdef'd on CONFIG_NET.
> Also I'm wondering whether it makes sense to pull the format code
> checking into all the sub-routines.
It isn't clear to me what you're asking for.
> It might be easier to maintain if it
> is all kept together in pointer().
Can you explain more thoroughly please?
If you mean not passing const char *fmt to the sub-routines,
I think not doing so makes it harder to maintain and extend.
I think it will be useful to extend the 'S' resource_string
capability to use fmt to allow specific bits to be selected
of the resource identifier to be printed.
See the idea in: http://lkml.org/lkml/2009/4/17/105
Here's the modified patch with your suggestions:
cheers,
Signed-off-by: Joe Perches <joe@perches.com>
diff --git a/lib/vsprintf.c b/lib/vsprintf.c
index 756ccaf..cb8a112 100644
--- a/lib/vsprintf.c
+++ b/lib/vsprintf.c
@@ -25,6 +25,7 @@
#include <linux/kallsyms.h>
#include <linux/uaccess.h>
#include <linux/ioport.h>
+#include <net/addrconf.h>
#include <asm/page.h> /* for PAGE_SIZE */
#include <asm/div64.h>
@@ -630,60 +631,156 @@ static char *resource_string(char *buf, char *end, struct resource *res,
}
static char *mac_address_string(char *buf, char *end, u8 *addr,
- struct printf_spec spec)
+ struct printf_spec spec, const char *fmt)
{
- char mac_addr[6 * 3]; /* (6 * 2 hex digits), 5 colons and trailing zero */
+ char mac_addr[sizeof("xx:xx:xx:xx:xx:xx")];
char *p = mac_addr;
int i;
for (i = 0; i < 6; i++) {
p = pack_hex_byte(p, addr[i]);
- if (!(spec.flags & SPECIAL) && i != 5)
+ if (fmt[0] == 'M' && i != 5)
*p++ = ':';
}
*p = '\0';
- spec.flags &= ~SPECIAL;
return string(buf, end, mac_addr, spec);
}
-static char *ip6_addr_string(char *buf, char *end, u8 *addr,
- struct printf_spec spec)
+static char *ip4_string(char *p, const u8 *addr, bool leading_zeros)
+{
+ int i;
+
+ for (i = 0; i < 4; i++) {
+ char temp[3]; /* hold each IP quad in reverse order */
+ int digits = put_dec_trunc(temp, addr[i]) - temp;
+ if (leading_zeros) {
+ if (digits < 3)
+ *p++ = '0';
+ if (digits < 2)
+ *p++ = '0';
+ }
+ /* reverse the digits in the quad */
+ while (digits--)
+ *p++ = temp[digits];
+ if (i < 3)
+ *p++ = '.';
+ }
+
+ *p = '\0';
+ return p;
+}
+
+static char *ip6_compressed_string(char *p, const struct in6_addr *addr)
{
- char ip6_addr[8 * 5]; /* (8 * 4 hex digits), 7 colons and trailing zero */
- char *p = ip6_addr;
int i;
+ int j;
+ int range;
+ unsigned char zerolength[8];
+ int longest = 1;
+ int colonpos = -1;
+ u16 word;
+ u8 hi;
+ u8 lo;
+ bool needcolon = false;
+ bool useIPv4 = ipv6_addr_v4mapped(addr) || ipv6_addr_is_isatap(addr);
+
+ memset(zerolength, 0, sizeof(zerolength));
+
+ if (useIPv4)
+ range = 6;
+ else
+ range = 8;
+
+ /* find position of longest 0 run */
+ for (i = 0; i < range; i++) {
+ for (j = i; j < range; j++) {
+ if (addr->s6_addr16[j] != 0)
+ break;
+ zerolength[i]++;
+ }
+ }
+ for (i = 0; i < range; i++) {
+ if (zerolength[i] > longest) {
+ longest = zerolength[i];
+ colonpos = i;
+ }
+ }
+
+ /* emit address */
+ for (i = 0; i < range; i++) {
+ if (i == colonpos) {
+ if (needcolon || i == 0)
+ *p++ = ':';
+ *p++ = ':';
+ needcolon = false;
+ i += longest - 1;
+ continue;
+ }
+ if (needcolon) {
+ *p++ = ':';
+ needcolon = false;
+ }
+ /* hex u16 without leading 0s */
+ word = ntohs(addr->s6_addr16[i]);
+ hi = word >> 8;
+ lo = word & 0xff;
+ if (hi) {
+ if (hi > 0x0f)
+ p = pack_hex_byte(p, hi);
+ else
+ *p++ = hex_asc_lo(hi);
+ }
+ if (hi || lo > 0x0f)
+ p = pack_hex_byte(p, lo);
+ else
+ *p++ = hex_asc_lo(lo);
+ needcolon = true;
+ }
+
+ if (useIPv4) {
+ if (needcolon)
+ *p++ = ':';
+ p = ip4_string(p, &addr->s6_addr[12], false);
+ }
+ *p = '\0';
+ return p;
+}
+
+static char *ip6_string(char *p, const struct in6_addr *addr, const char *fmt)
+{
+ int i;
for (i = 0; i < 8; i++) {
- p = pack_hex_byte(p, addr[2 * i]);
- p = pack_hex_byte(p, addr[2 * i + 1]);
- if (!(spec.flags & SPECIAL) && i != 7)
+ p = pack_hex_byte(p, addr->s6_addr[2 * i]);
+ p = pack_hex_byte(p, addr->s6_addr[2 * i + 1]);
+ if (fmt[0] == 'I' && i != 7)
*p++ = ':';
}
+
*p = '\0';
- spec.flags &= ~SPECIAL;
+ return p;
+}
+
+static char *ip6_addr_string(char *buf, char *end, const u8 *addr,
+ struct printf_spec spec, const char *fmt)
+{
+ char ip6_addr[sizeof("xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255")];
+
+ if (fmt[0] == 'I' && fmt[2] == 'c')
+ ip6_compressed_string(ip6_addr, (const struct in6_addr *)addr);
+ else
+ ip6_string(ip6_addr, (const struct in6_addr *)addr, fmt);
return string(buf, end, ip6_addr, spec);
}
-static char *ip4_addr_string(char *buf, char *end, u8 *addr,
- struct printf_spec spec)
+static char *ip4_addr_string(char *buf, char *end, const u8 *addr,
+ struct printf_spec spec, const char *fmt)
{
- char ip4_addr[4 * 4]; /* (4 * 3 decimal digits), 3 dots and trailing zero */
- char temp[3]; /* hold each IP quad in reverse order */
- char *p = ip4_addr;
- int i, digits;
+ char ip4_addr[sizeof("255.255.255.255")];
- for (i = 0; i < 4; i++) {
- digits = put_dec_trunc(temp, addr[i]) - temp;
- /* reverse the digits in the quad */
- while (digits--)
- *p++ = temp[digits];
- if (i != 3)
- *p++ = '.';
- }
- *p = '\0';
- spec.flags &= ~SPECIAL;
+ ip4_string(ip4_addr, addr, fmt[0] == 'i');
return string(buf, end, ip4_addr, spec);
}
@@ -702,11 +799,15 @@ static char *ip4_addr_string(char *buf, char *end, u8 *addr,
* addresses (not the name nor the flags)
* - 'M' For a 6-byte MAC address, it prints the address in the
* usual colon-separated hex notation
- * - 'I' [46] for IPv4/IPv6 addresses printed in the usual way (dot-separated
- * decimal for v4 and colon separated network-order 16 bit hex for v6)
- * - 'i' [46] for 'raw' IPv4/IPv6 addresses, IPv6 omits the colons, IPv4 is
- * currently the same
- *
+ * - 'm' For a 6-byte MAC address, it prints the hex address without colons
+ * - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
+ * IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
+ * IPv6 uses colon separated network-order 16 bit hex with leading 0's
+ * - 'i' [46] for 'raw' IPv4/IPv6 addresses
+ * IPv6 omits the colons (01020304...0f)
+ * IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
+ * - 'I6c' for IPv6 addresses printed as specified by
+ * http://www.ietf.org/id/draft-kawamura-ipv6-text-representation-03.txt
* Note: The difference between 'S' and 'F' is that on ia64 and ppc64
* function pointers are really function descriptors, which contain a
* pointer to the real address.
@@ -726,20 +827,24 @@ static char *pointer(const char *fmt, char *buf, char *end, void *ptr,
return symbol_string(buf, end, ptr, spec, *fmt);
case 'R':
return resource_string(buf, end, ptr, spec);
- case 'm':
- spec.flags |= SPECIAL;
- /* Fallthrough */
- case 'M':
- return mac_address_string(buf, end, ptr, spec);
- case 'i':
- spec.flags |= SPECIAL;
- /* Fallthrough */
- case 'I':
- if (fmt[1] == '6')
- return ip6_addr_string(buf, end, ptr, spec);
- if (fmt[1] == '4')
- return ip4_addr_string(buf, end, ptr, spec);
- spec.flags &= ~SPECIAL;
+ case 'M': /* Colon separated: 00:01:02:03:04:05 */
+ case 'm': /* Contiguous: 000102030405 */
+ return mac_address_string(buf, end, ptr, spec, fmt);
+ case 'I': /* Formatted IP supported
+ * 4: 1.2.3.4
+ * 6: 0001:0203:...:0708
+ * 6c: 1::708 or 1::1.2.3.4
+ */
+ case 'i': /* Contiguous:
+ * 4: 001.002.003.004
+ * 6: 000102...0f
+ */
+ switch (fmt[1]) {
+ case '6':
+ return ip6_addr_string(buf, end, ptr, spec, fmt);
+ case '4':
+ return ip4_addr_string(buf, end, ptr, spec, fmt);
+ }
break;
}
spec.flags |= SMALL;
cheers, Joe
^ permalink raw reply related
* Re: [RFC] bridge: prevent hairpin and STP problems?
From: Stephen Hemminger @ 2009-08-17 22:37 UTC (permalink / raw)
To: Fischer, Anna
Cc: David Miller, netdev@vger.kernel.org, ptcongdon@ucdavis.edu,
evb@yahoogroups.com, bridge@lists.linux-foundation.org,
kaber@trash.net, arnd@arndb.de, Dickson, Mike (ISS Software),
adobriyan@gmail.com, bridge@osdl.org
In-Reply-To: <0199E0D51A61344794750DC57738F58E6D6B0C79D7@GVW1118EXC.americas.hpqcorp.net>
On Mon, 17 Aug 2009 21:16:04 +0000
"Fischer, Anna" <anna.fischer@hp.com> wrote:
> > Subject: [RFC] bridge: prevent hairpin and STP problems?
> >
> > Do we need to add this to block Spanning Tree from being enabled
> > with hairpin mode? I am not sure what the exact usage of hairpin
> > mode and if it is possible to create loops and get STP confusion.
> >
> > For comment only, do not apply as is.
>
> Your patch disables STP on the whole bridge if one or more ports
> are set to hairpin mode.
>
> However, I don't really see that this is necessary.
>
> A hairpin mode port should not reflect BPDUs, because otherwise the
> connected port would think it has detected a loop. The hairpin mode
> port should still be able to generate BPDUs though, and in any case
> the bridge should still be able to run STP.
>
> The hairpin patch we submitted reflects packets on the forwarding /
> data path whereas BPDUs are processed with a separate hook, so we
> should not be reflecting BPDUs back out of a hairpin mode port.
So if user is using hairpin properly, the STP would work. In fact
it would be a good thing since it would detect looping configurations.
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Sridhar Samudrala @ 2009-08-17 22:43 UTC (permalink / raw)
To: Christoph Lameter; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <alpine.DEB.1.10.0908171813070.15956@gentwo.org>
On Mon, 2009-08-17 at 18:13 -0400, Christoph Lameter wrote:
> On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
>
> > What about ethtool -S ? Does it report any errors?
>
> Neither. This is is a broadcom bnx2 NIC.
Are you sure the packets are dropped at the sender?
Another place where packet drops/errors are counted is
/proc/net/softnet_stat
It tracks some counters that could result in drops. I thought these are
all receive statistics. But looks like cpu_collision is a tx stat. The
name of the structure is netif_rx_stat and it includes cpu_collison
counter.
Thanks
Sridhar
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-17 22:52 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <1250549034.25939.30.camel@w-sridhar.beaverton.ibm.com>
On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
> On Mon, 2009-08-17 at 18:13 -0400, Christoph Lameter wrote:
> > On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
> >
> > > What about ethtool -S ? Does it report any errors?
> >
> > Neither. This is is a broadcom bnx2 NIC.
>
> Are you sure the packets are dropped at the sender?
Yes I am sending 400k messages from the app and the receiver only gets
341k @300 byte (which is the line rate). There is no way that the 400k get
over the line. Also if I reduce SO_SNDBUF then both receiver and
sender get down to 341k.
I added the output of ethtool -S at the end.
The mcast tool can be had from http://gentwo.org/ll or from my directory
on www.kernel.org.
> Another place where packet drops/errors are counted is
> /proc/net/softnet_stat
> It tracks some counters that could result in drops. I thought these are
> all receive statistics. But looks like cpu_collision is a tx stat. The
> name of the structure is netif_rx_stat and it includes cpu_collison
> counter.
How do I decode that information?
clameter@rd-gateway3:~$ cat /proc/net/softnet_stat
00000a2f 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
0000000c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000018 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000006 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
rd-gateway3:/home/clameter# ethtool -S eth0
NIC statistics:
rx_bytes: 380526
rx_error_bytes: 0
tx_bytes: 6299438984
tx_error_bytes: 0
rx_ucast_packets: 2373
rx_mcast_packets: 168
rx_bcast_packets: 5
tx_ucast_packets: 1972
tx_mcast_packets: 18205522
tx_bcast_packets: 0
tx_mac_errors: 0
tx_carrier_errors: 0
rx_crc_errors: 0
rx_align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
tx_deferred: 0
tx_excess_collisions: 0
tx_late_collisions: 0
tx_total_collisions: 0
rx_fragments: 0
rx_jabbers: 0
rx_undersize_packets: 0
rx_oversize_packets: 0
rx_64_byte_packets: 375
rx_65_to_127_byte_packets: 1857
rx_128_to_255_byte_packets: 168
rx_256_to_511_byte_packets: 39
rx_512_to_1023_byte_packets: 12
rx_1024_to_1522_byte_packets: 95
rx_1523_to_9022_byte_packets: 0
tx_64_byte_packets: 221
tx_65_to_127_byte_packets: 1413
tx_128_to_255_byte_packets: 240
tx_256_to_511_byte_packets: 18205519
tx_512_to_1023_byte_packets: 18
tx_1024_to_1522_byte_packets: 83
tx_1523_to_9022_byte_packets: 0
rx_xon_frames: 0
rx_xoff_frames: 0
tx_xon_frames: 0
tx_xoff_frames: 0
rx_mac_ctrl_frames: 0
rx_filtered_packets: 16002
rx_discards: 0
rx_fw_discards: 0
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-17 22:57 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <alpine.DEB.1.10.0908171848370.15956@gentwo.org>
Argh I guess you want the receiver. The last msg was the sender.
clameter@rd-strategy3:~$ cat /proc/net/softnet_stat
0115d345 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
0000000c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000018 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Sridhar Samudrala @ 2009-08-18 0:12 UTC (permalink / raw)
To: Christoph Lameter; +Cc: Nivedita Singhvi, netdev, Eric Dumazet
In-Reply-To: <alpine.DEB.1.10.0908171848370.15956@gentwo.org>
On Mon, 2009-08-17 at 18:52 -0400, Christoph Lameter wrote:
> On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
>
> > On Mon, 2009-08-17 at 18:13 -0400, Christoph Lameter wrote:
> > > On Mon, 17 Aug 2009, Sridhar Samudrala wrote:
> > >
> > > > What about ethtool -S ? Does it report any errors?
> > >
> > > Neither. This is is a broadcom bnx2 NIC.
> >
> > Are you sure the packets are dropped at the sender?
>
> Yes I am sending 400k messages from the app and the receiver only gets
> 341k @300 byte (which is the line rate). There is no way that the 400k get
> over the line. Also if I reduce SO_SNDBUF then both receiver and
> sender get down to 341k.
>
> I added the output of ethtool -S at the end.
>
> The mcast tool can be had from http://gentwo.org/ll or from my directory
> on www.kernel.org.
>
> > Another place where packet drops/errors are counted is
> > /proc/net/softnet_stat
> > It tracks some counters that could result in drops. I thought these are
> > all receive statistics. But looks like cpu_collision is a tx stat. The
> > name of the structure is netif_rx_stat and it includes cpu_collison
> > counter.
>
> How do I decode that information?
total dropped time_squeeze 0 0 0 0 0 cpu_collision
The first 3 are rx stats and the last one is a tx stat.
Anyway, only the first field(total packets received) seems to be non-zero
in your softnet_stat output on both sender and receiver.
So it is possible that there is some other place in the stack where the packets
are gettting dropped but not counted.
-Sridhar
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox