* Re: Is this skb recycle buffer helpful to improve Linux network stack performance?
From: Eric Dumazet @ 2005-10-08 22:22 UTC (permalink / raw)
To: LeoY; +Cc: netdev, linux-kernel
In-Reply-To: <BAY108-DAV4008FD962D8FEC0497CC193870@phx.gbl>
LeoY a écrit :
> Hi Eric,
>
> Thanks for your reply.
> 1. For the packet size, this idea is targeting some specific network
> card driver and we assume all the packet size will not exceed 2KB.
> 2. Currently only Uni-Processor is considered(Hyper-threading is also
> disabled), I will add spin lock part once it works on the UP.
>
1) Even on Uni-Processor, you still need to protect against IRQS
2) The big cost of kmalloc()/kfree() come from the
local_irq_save(flags)/local_irq_restore(flags)
Once you add them in your code, you will discover you gain nothing compared to
kmalloc()/kfree() that already use a 'ring buffer' : More over, the slab
implementation is designed to have separate 'ring buffer' (one for each cpu),
so it is probably better than a 'central ring buffer'
Eric
^ permalink raw reply
* [RTI #2053] AutoReply: exception
From: RTI support via RT @ 2005-10-09 4:01 UTC (permalink / raw)
To: netdev
In-Reply-To: <200510090402.j99422SP016667@bigbend.rti.com>
Greetings!
This message has been automatically generated in response to the
creation of a trouble ticket regarding:
"exception",
a summary of which appears below.
There is no need to reply to this message right now. Your ticket has been
assigned an ID of [RTI #2053].
Please include the string:
[RTI #2053]
in the subject line of all future correspondence about this issue. To do so,
you may reply to this message.
Thank you,
Real-Time Innovations support team
<support@rti.com>
-------------------------------------------------------------------------
take it easy!
^ permalink raw reply
* friable No delay dispatch! Accept your antibiotics quicker. synthesis
From: ambrose cravens @ 2005-10-10 4:54 UTC (permalink / raw)
To: Dean Puller, owner-linux-xfs, gomez, blair, devfs, craig, leonard,
ogl-sample, netdev
Constraint to acquire meddicaation, acquisite those all here.
Heal your annxiety by taking our effective meddicaation.
Wholly Tracked Packages will arrive to you within 40hrs.
Treat your syndrome at much economical tags.
Free meddicaal details with our accredited doctor.
3 wordings for you! You are surprising. Howard K --NJ.
http://uk.geocities.com/snazzysmashingb/?exfo
"You used to teach girls," she said, "If you could only have taught me, I
could have learnt from you. I am so very miserable, and I like you so much."
bluethner dsavage cmdi sz04 esingers felip
"Is it you, indeed, Professor Wilson? I was afraid that you might get here
before I did. I was detained at a concert, and Bartley telephoned that he
would be late. Thomas will show you your room. Had you rather have your tea
brought to you there, or will you have it down here with me, while we wait
for Bartley?"
Wilson was pleased to find that he had been the cause of her rapid walk,
and with her he was even more vastly pleased than before. He followed her
through the drawing-room into the library, where the wide back windows
looked out upon the garden and the sunset and a fine stretch of
silver-colored river. A harp-shaped elm stood stripped against the
pale-colored evening sky, with ragged last year's birds' nests in its forks,
and through the bare branches the evening star quivered in the misty air.
The long brown room breathed the peace of a rich and amply guarded quiet.
Tea was brought in immediately and placed in front of the wood fire. Mrs.
Alexander sat down in a high-backed chair and began to pour it, while Wilson
sank into a low seat opposite her and took his cup with a great sense of
ease and harmony and comfort.
^ permalink raw reply
* Re: [PATCH] sis900: come alive after temporary memory shortage
From: Daniele Venzano @ 2005-10-10 8:37 UTC (permalink / raw)
To: Vasily Averin; +Cc: Andrew Morton, Jeff Garzik, NetDev, Stanislav Protassov
In-Reply-To: <4347C50A.9010501@sw.ru>
On Sat, Oct 08, 2005 at 05:09:30PM +0400, Vasily Averin wrote:
> Daniele, could you please check our new patch carefully?
This time I tested it as thoroughly as possible. It kept working during
all the time the memory was filling up, until the OOM killer woke up
from his dark pit. After OOM frenzy, the driver was still alive and
kicking.
> Andrew, could you please drop our old patch and replace it by the new one?
Yes, this one is better, thanks.
> Patch solves following problems:
> 1) Forgotten counter incrementation in sis900_rx() in case
> it doesn't get memory for skb, that leads to whole interface failure.
> Problem is accompanied with messages:
> eth0: Memory squeeze,deferring packet.
> eth0: NULL pointer encountered in Rx ring, skipping
> 2) If counter cur_rx overflows and there'll be temporary memory problems
> buffer can't be recreated later, when memory IS available.
> 3) Limit the work in handler to prevent the endless packets processing if
> new packets are generated faster then handled.
>
> Signed-off-by: Konstantin Khorenko <khorenko@sw.ru>
> Signed-off-by: Vasily Averin <vvs@sw.ru>
Signed-off-by: Daniele Venzano <venza@brownhat.org>
--- a/drivers/net/sis900.c 2005-10-08 12:22:53.000000000 +0400
+++ b/drivers/net/sis900.c 2005-10-08 15:12:29.000000000 +0400
@@ -1696,15 +1696,20 @@ static int sis900_rx(struct net_device *
long ioaddr = net_dev->base_addr;
unsigned int entry = sis_priv->cur_rx % NUM_RX_DESC;
u32 rx_status = sis_priv->rx_ring[entry].cmdsts;
+ int rx_work_limit;
if (netif_msg_rx_status(sis_priv))
printk(KERN_DEBUG "sis900_rx, cur_rx:%4.4d, dirty_rx:%4.4d "
"status:0x%8.8x\n",
sis_priv->cur_rx, sis_priv->dirty_rx, rx_status);
+ rx_work_limit = sis_priv->dirty_rx + NUM_RX_DESC - sis_priv->cur_rx;
while (rx_status & OWN) {
unsigned int rx_size;
+ if (--rx_work_limit < 0)
+ break;
+
rx_size = (rx_status & DSIZE) - CRC_SIZE;
if (rx_status & (ABORT|OVERRUN|TOOLONG|RUNT|RXISERR|CRCERR|FAERR)) {
@@ -1732,9 +1737,11 @@ static int sis900_rx(struct net_device *
we are working on NULL sk_buff :-( */
if (sis_priv->rx_skbuff[entry] == NULL) {
if (netif_msg_rx_err(sis_priv))
- printk(KERN_INFO "%s: NULL pointer "
- "encountered in Rx ring, skipping\n",
- net_dev->name);
+ printk(KERN_WARNING "%s: NULL pointer "
+ "encountered in Rx ring\n"
+ "cur_rx:%4.4d, dirty_rx:%4.4d\n",
+ net_dev->name, sis_priv->cur_rx,
+ sis_priv->dirty_rx);
break;
}
@@ -1770,6 +1777,7 @@ static int sis900_rx(struct net_device *
sis_priv->rx_ring[entry].cmdsts = 0;
sis_priv->rx_ring[entry].bufptr = 0;
sis_priv->stats.rx_dropped++;
+ sis_priv->cur_rx++;
break;
}
skb->dev = net_dev;
@@ -1787,7 +1795,7 @@ static int sis900_rx(struct net_device *
/* refill the Rx buffer, what if the rate of refilling is slower
* than consuming ?? */
- for (;sis_priv->cur_rx - sis_priv->dirty_rx > 0; sis_priv->dirty_rx++) {
+ for (; sis_priv->cur_rx != sis_priv->dirty_rx; sis_priv->dirty_rx++) {
struct sk_buff *skb;
entry = sis_priv->dirty_rx % NUM_RX_DESC;
^ permalink raw reply
* Re: 32 bit (socket layer) ioctl emulation for 64 bit kernels- Question regarding...
From: Arnd Bergmann @ 2005-10-10 11:48 UTC (permalink / raw)
To: spereira; +Cc: linux-kernel, netdev
In-Reply-To: <3ad486780510092121h78a522cat11f33581dfc670dc@mail.gmail.com>
On Maandag 10 Oktober 2005 06:21, spereira wrote:
>
> Is there currently an alternative to register_ioctl32_conversion that
> would help achive 32 bit ioctl emulation at the socket layer?
> Any suggestions/advice whould be much appreciated.
The correct solution would be to add the missing functionality to
net/socket.c and move over the implementation of SIOC* from
fs/compat_ioctl.c. Getting the code path right is a little tricky,
but I think a patch to fix this up would be appreciated.
As a start, you could define a compat_sock_ioctl along the
lines of compat_blkdev_ioctl and add your own handlers to the
x25_proto_ops, but IMHO it would makes sense to get rid of stuff
like dev_ifsioc from fs/compat_ioctl.c at the same time by
introducing a new compat_dev_ioctl called from compat_sock_ioctl.
Arnd <><
^ permalink raw reply
* [PATCH] pcnet32 does not use the PROM address on powerpc
From: Olaf Hering @ 2005-10-10 14:23 UTC (permalink / raw)
To: netdev, linuxppc64-dev
In-Reply-To: <20051008120316.GA12122@suse.de>
On Sat, Oct 08, Olaf Hering wrote:
> On Sat, Oct 08, Olaf Hering wrote:
>
> > I have a 44p 270, which gets all 0xFF as MAC address if I power if off
> > and on again. Further reboots do not fix it.
> > But it does get the correct one if I boot into SMS and do a netboot,
> > further reboots will always get the correct MAC address
The CSR contains garbage after a coldboot on RS/6000.
One some systems (like my 44p 270) the MAC address is all FF,
on others (like my B50) it is ff:ff:ff:fd:ff:6b.
It can eventually be fixed by loading pcnet32, set the interface
into the UP state, rmmod pcnet32 and load it again. But this worked
only on the 270.
Only netbooting after a cold start provides the correct MAC address
via prom and CSR. This makes it very unreliable.
I dont know why the MAC is stored in two different places. Remove
the special case for powerpc, which was added in early 2.4 development.
Signed-off-by: Olaf Hering <olh@suse.de>
drivers/net/pcnet32.c | 5 -----
1 files changed, 5 deletions(-)
Index: linux-2.6.14-rc3/drivers/net/pcnet32.c
===================================================================
--- linux-2.6.14-rc3.orig/drivers/net/pcnet32.c
+++ linux-2.6.14-rc3/drivers/net/pcnet32.c
@@ -1172,12 +1172,7 @@ pcnet32_probe1(unsigned long ioaddr, int
if (memcmp(promaddr, dev->dev_addr, 6)
|| !is_valid_ether_addr(dev->dev_addr)) {
-#ifndef __powerpc__
if (is_valid_ether_addr(promaddr)) {
-#else
- if (!is_valid_ether_addr(dev->dev_addr)
- && is_valid_ether_addr(promaddr)) {
-#endif
if (pcnet32_debug & NETIF_MSG_PROBE) {
printk(" warning: CSR address invalid,\n");
printk(KERN_INFO " using instead PROM address of");
--
short story of a lazy sysadmin:
alias appserv=wotan
^ permalink raw reply
* Re: [ANNOUNCE] iproute2 version (050929)
From: Stephen Hemminger @ 2005-10-10 14:40 UTC (permalink / raw)
To: Krzysztof Oledzki; +Cc: netdev, lartc
In-Reply-To: <Pine.LNX.4.62.0510101232310.1343@bizon.gios.gov.pl>
On Mon, 10 Oct 2005 12:34:51 +0200 (CEST)
Krzysztof Oledzki <olel@ans.pl> wrote:
>
>
> On Wed, 5 Oct 2005, Stephen Hemminger wrote:
>
> > On Sat, 1 Oct 2005 23:34:25 +0200 (CEST)
> > Krzysztof Oledzki <olel@ans.pl> wrote:
> >
> >>
> >>
> >> On Fri, 30 Sep 2005, Stephen Hemminger wrote:
> >>
> >>> There is an new minor update to iproute2 utilities available:
> >>> http://developer.osdl.org/dev/iproute2/download/iproute2-050929.tar.gz
> >>
> >>
> >> It hangs on "ip rule flush". Tested on 2.6.13.2.
> >>
> >>
> >> Best regards,
> >>
> >>
> >> Krzysztof Olędzki
> >
> > Fixed by this patch (in next release)
>
> Not sure... Just checked iproute2-ss051007 and it still hangs on "ip rule
> flush" with ~100% cpu load.
>
Are you running as root? If not it will loop forever.
^ permalink raw reply
* [PATCH] Documentation/networking/README.ipw2100
From: Alejandro Bonilla @ 2005-10-10 17:05 UTC (permalink / raw)
To: linux-kernel; +Cc: netdev, jketreno, jgarzik
Hi,
This updates the Documentation so that IPW2100 can be well documented for
2.6.14, the Kconfig points to this file but there is no information about the
firmware.
Please apply.
Signed-off-by: Alejandro Bonilla <abonilla@linuxwireless.org>
diff --git a/Documentation/networking/README.ipw2100
b/Documentation/networking/README.ipw2100
--- a/Documentation/networking/README.ipw2100
+++ b/Documentation/networking/README.ipw2100
@@ -143,7 +143,12 @@ firmware image to load into the wireless
You can obtain these images from <http://ipw2100.sf.net/firmware.php>.
-See INSTALL for instructions on installing the firmware.
+The firmware package should be extracted where your hotplug firmware agent
+is looking:
+
+% grep FIRMWARE /etc/hotplug/firmware.agent
+
+The most common path is /lib/firmware but the firmware.agent will tell.
===========================
^ permalink raw reply
* [PATCH] Documentation/networking/README.ipw2200
From: Alejandro Bonilla @ 2005-10-10 17:18 UTC (permalink / raw)
To: linux-kernel; +Cc: netdev, jketreno, jgarzik
Hi,
This patch is different from the previous one. This is for IPW2200. ;-)
It updates the information/README to match the Kconfig info of the IPW2200.
Please apply to have a better documented 2.6.14
I dunno if this is either handled by Netdev or LKML because is about
Documentation/
Thanks,
.Alejandro
Signed-off-by: Alejandro Bonilla <abonilla@linuxwireless.org>
diff --git a/Documentation/networking/README.ipw2200
b/Documentation/networking/README.ipw2200
--- a/Documentation/networking/README.ipw2200
+++ b/Documentation/networking/README.ipw2200
@@ -1,5 +1,5 @@
-Intel(R) PRO/Wireless 2915ABG Driver for Linux in support of:
+Intel(R) PRO/Wireless 2200BG Driver for Linux in support of:
Intel(R) PRO/Wireless 2200BG Network Connection
Intel(R) PRO/Wireless 2915ABG Network Connection
@@ -27,7 +27,8 @@ Index
1.4. Sysfs Helper Files
2. About the Version Numbers
3. Support
-4. License
+4. Firmware
+5. License
1. Introduction
@@ -272,7 +273,24 @@ For general information and support, go
http://ipw2200.sf.net/
-4. License
+4. Firmware
+-----------------------------------------------
+
+As the firmware is licensed under a restricted use license, it can not be
+included within the kernel sources. To enable the IPW2200 you will need a
+firmware image to load into the wireless NIC's processors.
+
+You can obtain these images from <http://ipw2200.sf.net/firmware.php>.
+
+The firmware package should be extracted where your hotplug firmware agent
+is looking:
+
+% grep FIRMWARE /etc/hotplug/firmware.agent
+
+The most common path is /lib/firmware but the firmware.agent will tell.
+
+
+5. License
-----------------------------------------------
Copyright(c) 2003 - 2005 Intel Corporation. All rights reserved.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-10 18:23 UTC (permalink / raw)
To: Matt Leininger; +Cc: netdev, openib-general
In-Reply-To: <1128738350.13945.369.camel@localhost>
> 2.6.12-rc5 in-kernel 1 405 <<<<<
> 2.6.12-rc4 in-kernel 1 470 <<<<<
I was optimistic when I saw this, because the changeover to git
occurred with 2.6.12-rc2, so I thought I could use git bisect to track
down exactly when the performance regression happened.
However, I haven't been able to get numbers that are stable enough to
track this down. I have two systems, both HP DL145s with dual Opteron
875s and two-port mem-free PCI Express HCAs. I use MSI-X with the
completion interrupt affinity set to CPU 0, and "taskset 2" to run
netserver and netperf on CPU 1.
With default netperf parameters (just "-H otherguy") I get numbers
between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
The numbers are quite consistent between reboots, but if I reboot the
system (even keeping the kernel identical), I see large performance
changes. Presumably something is happening like the cache coloring of
some hot data structures changing semi-randomly depending on the
timing of various initialations.
Matt, how stable are your numbers?
- R.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Michael S. Tsirkin @ 2005-10-10 20:03 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <521x2tgrim.fsf@cisco.com>
Hi Roland,
Quoting r. Roland Dreier <rolandd@cisco.com>:
> However, I haven't been able to get numbers that are stable enough to
> track this down.
Disabling irq balancing sometimes helps me make the numbers more stable.
Hope this helps,
--
MST
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Rick Jones @ 2005-10-10 20:17 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <521x2tgrim.fsf@cisco.com>
Roland Dreier wrote:
> > 2.6.12-rc5 in-kernel 1 405 <<<<<
> > 2.6.12-rc4 in-kernel 1 470 <<<<<
>
> I was optimistic when I saw this, because the changeover to git
> occurred with 2.6.12-rc2, so I thought I could use git bisect to track
> down exactly when the performance regression happened.
>
> However, I haven't been able to get numbers that are stable enough to
> track this down. I have two systems, both HP DL145s with dual Opteron
> 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the
> completion interrupt affinity set to CPU 0, and "taskset 2" to run
> netserver and netperf on CPU 1.
>
> With default netperf parameters (just "-H otherguy") I get numbers
> between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
> The numbers are quite consistent between reboots, but if I reboot the
> system (even keeping the kernel identical), I see large performance
> changes. Presumably something is happening like the cache coloring of
> some hot data structures changing semi-randomly depending on the
> timing of various initialations.
Which rev of netperf are you using, and areyou using the "confidence intervals"
options (-i, -I)? for a long time, the linux-unique behaviour of returning the
overhead bytes for SO_[SND|RCV]BUF and them being 2X what one gives in
setsockopt() gave netperf some trouble - the socket buffer would double in size
each iteration on a confidence interval run. Later netperf versions (late 2.3,
and 2.4.X) have a kludge for this.
Slightly related to that, IIRC, the linux receiver code adjusts the advertised
window as the connection goes along - how far the receive code opens the window
may change from run to run - might that have an effect? If there is a way to
get the linux receiver to simply advertise the full window from the beginning
that might help minimize the number of variables.
Are there large changes in service demand along with the large performance changes?
FWIW, on later netperfs the -T option should allow you to specify the CPU on
which netperf and/or netserver run, although I've had some trouble reliably
detecting the right sched_setaffinity syntax among the releases.
rick jones
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-10 20:58 UTC (permalink / raw)
To: Rick Jones; +Cc: netdev, openib-general
In-Reply-To: <434ACC74.3020404@hp.com>
Rick> Which rev of netperf are you using, and areyou using the
Rick> "confidence intervals" options (-i, -I)? for a long time,
Rick> the linux-unique behaviour of returning the overhead bytes
Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in
Rick> setsockopt() gave netperf some trouble - the socket buffer
Rick> would double in size each iteration on a confidence interval
Rick> run. Later netperf versions (late 2.3, and 2.4.X) have a
Rick> kludge for this.
I believe it's netperf 2.2.
I'm not using any confidence interval stuff. However, the variation
is not between single runs of netperf -- if I do 5 runs of netperf in
a row, I get roughly the same number from each run. For example, I
might see something like
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 3869.82
and then
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 3862.41
for two successive runs. However, if I reboot the system into the
same kernel (ie everything set up exactly the same), the same
invocation of netperf might give
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 4389.20
Rick> Are there large changes in service demand along with the
Rick> large performance changes?
Not sure. How do I have netperf report service demand?
- R.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-10 21:03 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: netdev, openib-general
In-Reply-To: <20051010200321.GC6633@mellanox.co.il>
Michael> Disabling irq balancing sometimes helps me make the
Michael> numbers more stable.
I don't think that's an issue. I'm running on x86_64, which I don't
think has the kernel irq balancer, and I'm not running a userspace IRQ
balancer. I can see all the mthca interrupts going to the CPU I set
through the smp_affinity file.
- R.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Rick Jones @ 2005-10-10 21:22 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <52irw5f5t0.fsf@cisco.com>
Roland Dreier wrote:
> Rick> Which rev of netperf are you using, and areyou using the
> Rick> "confidence intervals" options (-i, -I)? for a long time,
> Rick> the linux-unique behaviour of returning the overhead bytes
> Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in
> Rick> setsockopt() gave netperf some trouble - the socket buffer
> Rick> would double in size each iteration on a confidence interval
> Rick> run. Later netperf versions (late 2.3, and 2.4.X) have a
> Rick> kludge for this.
>
> I believe it's netperf 2.2.
That's rather old. I literally just put 2.4.1 out on ftp.cup.hp.com - probably
better to use that if possible. Not that it will change the variability just
that I like it when people are up-to-date on the versions :) If nothing else,
the 2.4.X version(s) have a much improved (hopefully) manual in doc/
[If you are really maschochistic, the very first release of netperf 4.0.0 source
has happened. I can make no guarantees as to its actually working at the moment
though :) Netperf4 is going to be the stream for the multiple-connection,
multiple system tests rather than the single-connection nature of netperf2]
> I'm not using any confidence interval stuff. However, the variation
> is not between single runs of netperf -- if I do 5 runs of netperf in
> a row, I get roughly the same number from each run. For example, I
> might see something like
>
> TCP STREAM TEST to 192.168.145.2 : histogram
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 3869.82
>
> and then
>
> TCP STREAM TEST to 192.168.145.2 : histogram
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 3862.41
>
> for two successive runs. However, if I reboot the system into the
> same kernel (ie everything set up exactly the same), the same
> invocation of netperf might give
>
> TCP STREAM TEST to 192.168.145.2 : histogram
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 4389.20
>
> Rick> Are there large changes in service demand along with the
> Rick> large performance changes?
>
> Not sure. How do I have netperf report service demand?
Ask for CPU utilization with -c (local) and -C (remote). The /proc/stat stuff
used by Linux does not need calibration (IIRC) so you don't have to worry about
that.
If cache effects are involved, you can make netperf "harder" or "easier" on the
caches by altering the size of the send and/or recv buffer rings. By default
they are one more than the socket buffer size divided by the send size, but you
can make them larger or smaller with the -W option.
These days I use a 128K socket buffer and 32K send for the "canonical" (although
not default :) netperf TCP_STREAM test:
netperf -H remote -c -C -- -s 128K -S 128K -m 32K
In netperf-speak K == 1024, k == 1000, M == 2^20, m == 10^6, G == 2^40, g == 10^9...
rick jones
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Grant Grundler @ 2005-10-10 21:26 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <521x2tgrim.fsf@cisco.com>
On Mon, Oct 10, 2005 at 11:23:45AM -0700, Roland Dreier wrote:
> > 2.6.12-rc5 in-kernel 1 405 <<<<<
> > 2.6.12-rc4 in-kernel 1 470 <<<<<
>
> I was optimistic when I saw this, because the changeover to git
> occurred with 2.6.12-rc2, so I thought I could use git bisect to track
> down exactly when the performance regression happened.
>
> However, I haven't been able to get numbers that are stable enough to
> track this down. I have two systems, both HP DL145s with dual Opteron
> 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the
> completion interrupt affinity set to CPU 0, and "taskset 2" to run
> netserver and netperf on CPU 1.
As you know, opteron boxes are NUMA. I think you want MSI-X interrupt
bound to the same CPU that's connected to the IO. Is CPU 0 closer to IO?
I would bind netperf to CPU0 and netserver to CPU 1 on each box respectively.
Or just try all 4 combinations to see which combinations are CPU bound
vs memory/IO bound.
> With default netperf parameters (just "-H otherguy") I get numbers
> between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
> The numbers are quite consistent between reboots, but if I reboot the
> system (even keeping the kernel identical), I see large performance
> changes.
I gather you meant "tests" in the first phrase? (vs reboot).
> Presumably something is happening like the cache coloring of
> some hot data structures changing semi-randomly depending on the
> timing of various initialations.
My guess is based on the same premise.
The mem-free card will be very sensitive to were it's control data
is allocated. Is either box configured to interleave memory from
both CPUs?
If it's interleaving, every other cacheline will be "local".
Can you disable interleave and try different netperf/server
bindings as suggested above?
hth,
grant
^ permalink raw reply
* Re: [ANNOUNCE] iproute2 version (050929)
From: Stephen Hemminger @ 2005-10-10 23:16 UTC (permalink / raw)
To: hadi; +Cc: netdev, lartc, Krzysztof Oledzki
In-Reply-To: <1128976620.6353.3.camel@localhost.localdomain>
On Mon, 10 Oct 2005 16:37:00 -0400
jamal <hadi@cyberus.ca> wrote:
> On Mon, 2005-10-10 at 22:08 +0200, Krzysztof Oledzki wrote:
> >
>
> > >
> > > Why are you trying to flush those tables?
> > To install a new set of rules.
> >
>
>
> flush should kill everything if i am not mistaken.
> Are you replacing the main table etc as well?
>
> > > Try to downgrade the kernel and see what happens - go as far as 2.6.12
> > But why? It works with older (ss050330) version of iproute2 even on
> > 2.6.13.3.
> >
>
> Ok, sorry i wasnt aware of that. Try the attached patch.
>
> cheers,
> jamal
Look like the -batch code that keeps the RTNL handle open was
breaking this. The patch shows the general idea, but it still has
problems because I think it would still not work if used in
-batch situation.
--
Stephen Hemminger <shemminger@osdl.org>
OSDL http://developer.osdl.org/~shemminger
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Matt Leininger @ 2005-10-10 23:25 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <521x2tgrim.fsf@cisco.com>
On Mon, 2005-10-10 at 11:23 -0700, Roland Dreier wrote:
> > 2.6.12-rc5 in-kernel 1 405 <<<<<
> > 2.6.12-rc4 in-kernel 1 470 <<<<<
>
> I was optimistic when I saw this, because the changeover to git
> occurred with 2.6.12-rc2, so I thought I could use git bisect to track
> down exactly when the performance regression happened.
>
> However, I haven't been able to get numbers that are stable enough to
> track this down. I have two systems, both HP DL145s with dual Opteron
> 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the
> completion interrupt affinity set to CPU 0, and "taskset 2" to run
> netserver and netperf on CPU 1.
>
> With default netperf parameters (just "-H otherguy") I get numbers
> between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
> The numbers are quite consistent between reboots, but if I reboot the
> system (even keeping the kernel identical), I see large performance
> changes. Presumably something is happening like the cache coloring of
> some hot data structures changing semi-randomly depending on the
> timing of various initialations.
>
> Matt, how stable are your numbers?
Pretty consistent. Here are a few runs with 2.6.12-rc5 with reboots
in between each run. I'm using netperf-2.3pl1.
Run 1:
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 410302.39 99.89 92.09 4.869
4.489
Run 2: (after another reboot)
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 409510.33 99.89 91.59 4.879
4.473
Run 3: (after reboot)
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 404354.11 99.89 91.39 4.941
4.520
I see the same variance in netperf results if I don't reboot between
runs.
- Matt
>
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Grant Grundler @ 2005-10-10 23:30 UTC (permalink / raw)
To: Grant Grundler; +Cc: netdev, openib-general
In-Reply-To: <20051010212652.GG9613@esmail.cup.hp.com>
On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote:
...
> If it's interleaving, every other cacheline will be "local".
ISTR AMD64 was page-interleaved but then got confused by documents
describing "128-bit" 2-way interleave. I now realize the 128bit
is refering to interleave between two "banks" of memory behind
each memory controller. ie 2 * 128-bit provides in the 32-byte
cacheline size that most x86 programs expect.
Anyway, I'm hoping that we'll see a consistent result if node interleave
is turned off.
sorry for the confusion,
grant
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-10 23:38 UTC (permalink / raw)
To: Matt Leininger; +Cc: netdev, openib-general
In-Reply-To: <1128986707.13945.424.camel@localhost>
Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5
Matt> with reboots in between each run. I'm using netperf-2.3pl1.
That's interesting. I'm guessing you're using mem-ful HCAs?
Given that your results are more stable than mine, if you're up for
it, you could install git, clone Linus's tree, and then do a git
bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression
to a single commit (if in fact that's possible).
- R.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Matt Leininger @ 2005-10-10 23:44 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev, openib-general
In-Reply-To: <52br1xdjtm.fsf@cisco.com>
On Mon, 2005-10-10 at 16:38 -0700, Roland Dreier wrote:
> Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5
> Matt> with reboots in between each run. I'm using netperf-2.3pl1.
>
> That's interesting. I'm guessing you're using mem-ful HCAs?
Yes, I'm using mem-full HCAs. I could try reflashing the firmware for
memfree if that's of interest.
>
> Given that your results are more stable than mine, if you're up for
> it, you could install git, clone Linus's tree, and then do a git
> bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression
> to a single commit (if in fact that's possible).
I was hoping someone else would do this. :)
I'll start working on it tomorrow if no one else gets to it.
Thanks,
- Matt
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-10 23:53 UTC (permalink / raw)
To: Matt Leininger; +Cc: netdev, openib-general
In-Reply-To: <1128987897.13952.441.camel@localhost>
Matt> Yes, I'm using mem-full HCAs. I could try reflashing the
Matt> firmware for memfree if that's of interest.
No, probably not. If I get a chance I'll do the opposite (flash
mem-free -> mem-full, since my HCAs do have memory) and see if it
makes my results stable.
Matt> I was hoping someone else would do this. :) I'll start
Matt> working on it tomorrow if no one else gets to it.
I might get a chance to do it tonight... I'll post if I do.
- R.
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Andi Kleen @ 2005-10-11 0:51 UTC (permalink / raw)
To: Grant Grundler; +Cc: netdev, openib-general
In-Reply-To: <20051010233054.GA11213@esmail.cup.hp.com>
On Tuesday 11 October 2005 01:30, Grant Grundler wrote:
> On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote:
> ...
>
> > If it's interleaving, every other cacheline will be "local".
>
> ISTR AMD64 was page-interleaved but then got confused by documents
> describing "128-bit" 2-way interleave. I now realize the 128bit
> is refering to interleave between two "banks" of memory behind
> each memory controller. ie 2 * 128-bit provides in the 32-byte
> cacheline size that most x86 programs expect.
The cache line size on K7 and K8 is 64 bytes.
> Anyway, I'm hoping that we'll see a consistent result if node interleave
> is turned off.
Yes usually a good idea.
-Andi
^ permalink raw reply
* IPW Question on menuconfig
From: Alejandro Bonilla Beeche @ 2005-10-11 3:00 UTC (permalink / raw)
To: netdev, linux-kernel; +Cc: James Ketrenos
Hi,
I was just git updating 2.6.14-rc4 and I didn't have the
CONFIG_IEEE80211 module selected, then I went into Device Drivers /
Network devices / Wireless, and the IPW2100 wasn't in the list. I went,
then Modularized the IEEE80211 and then IPW2100 was there. I think is
kind of estrange that IPW2100 wouldn't show just because it has a
dependencie, shouldn't IPW2100 show in the list, and if you select it,
then it would also select CONFIG_IEEE80211?
Just a dumb question cause I dunno how is normally done or the policy to
do this.
.Alejandro
^ permalink raw reply
* Re: Timeline of IPoIB performance
From: Roland Dreier @ 2005-10-11 4:03 UTC (permalink / raw)
To: Matt Leininger; +Cc: netdev, openib-general
In-Reply-To: <527jcldj4n.fsf@cisco.com>
Roland> I might get a chance to do it tonight... I'll post if I do.
I'm giving it a shot but I just can't reproduce this well on my
systems. I do see a pretty big regression between 2.6.12-rc4 and
2.6.14-rc2, but 2.6.12-rc5 looks OK on my systems.
I reflashed to FW 4.7.0 (mem-ful) and built netperf 2.4.1.
With 2.6.12-rc4 I've seen runs as slow as:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.00 553.71 37.46 -1.00 2.642 -1.000
and with 2.6.12-rc5 I've seen runs as fast as:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.00 581.82 39.58 -1.00 2.657 -1.000
so not much difference there. With 2.6.14-rc2, the best of 10 runs was:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.01 497.00 39.71 -1.00 3.121 -1.000
so we've definitely lost something there.
Time to do some more bisecting...
- R.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox