* Re: gigabit trouble
[not found] ` <20040729210401.A32456@electric-eye.fr.zoreil.com>
@ 2004-07-29 22:41 ` Bart Alewijnse
2004-07-30 15:15 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-07-29 22:41 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel
I was going to do an exhaustive test, but because my computer stopped
running completely, I'll reply to the one or two bits I can now.
On Thu, 29 Jul 2004 21:04:01 +0200, Francois Romieu
<romieu@fr.zoreil.com> wrote:
> Bart Alewijnse <scarfboy@gmail.com> :
> [...]
> > I run gentoo on both, which until yesterday was 2.6.7-ck5 (on both),
> > and currently run 2.6.7-mm6 (again, both), as I saw the suggestion
> > somewhere it had better support for the card - something about a new
> > net card inferface that's nicer to interrupts.
>
> NAPI support for r8169 is available in recent -mm kernel and there is
> a small (though noticeable) optimization wrt to interrupt disabling.
Well, I noticed a max of 6500 interrupts/s on eth1 on both computers, with or
without napi - and that 6500 is also the figure of packets per second.
So I'm slightly dubious. (notice 6500 *1500 bytes is about 10MB/s)
> [...]
> > So, question one - how do I see the link speed under linux, and how,
> > if at all, do I control it?
>
> ethtool
Thanks. That wasn't the problem - the line speed's a gbit.
> [...]
> > Disturbingly, in such a linux-to-linux speed test, my new computer
> > froze.As in, in text mode, have the screen freeze and apparently be
> > half written full of nonsense.
>
> These messages would be welcome (pen/paper/serial line/image/log file
> or whatever).
No messages, no oops, no log messages that I noticed.
It was video memory corruption in text mode.
As to the rest, I'll do it when I revive my computer. Right now, I'm
thinking the power supply may be dodgy. I'll see if I can get
a minimum to run off my even older 235Watt. But as a note,
with two cards, two cdroms and a hard drive less, it was still
making the noise. I think it still is now, but since no OS actually
boots completely right now, I can't say for sure.
I'll do a memtest, that makes sense.
--Bart
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-29 22:41 ` gigabit trouble Bart Alewijnse
@ 2004-07-30 15:15 ` Bart Alewijnse
2004-07-30 18:54 ` Francois Romieu
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-07-30 15:15 UTC (permalink / raw)
To: netdev
On Fri, 30 Jul 2004 00:41:03 +0200, Bart Alewijnse <scarfboy@gmail.com> wrote:
> I was going to do an exhaustive test, but because my computer stopped
> running completely, I'll reply to the one or two bits I can now.
...which was unrelated. Although I can't get the noise anymore
now either. I'm not too happy, I like my problems reproducable.
Of course, in the course of tring to fix the nonbootability, I changed
a bunch of things. Hrm.
I'll try getting it back later.
> > > So, question one - how do I see the link speed under linux, and how,
> > > if at all, do I control it?
> >
> > ethtool
> Thanks. That wasn't the problem - the line speed's a gbit.
Definately. I've now seen speeds up to 22MB/s, but only in pure
network benching (netio), and only with udp, although I guess
that makes a some sense.
(Also, for some reason it makes a respectable difference which
computer I run the netio server on. I mean, netio measures
speed both ways on one run, and it's different depending on
where I run it. I guess that suggests io limiting)
So the hanging around ten MB/s was just coincidence - it still
does it in both nfs and samba, but I guess it's io limited
*somewhere*, but trying to figure out why doesn't belong
in this list, I guess. Or maybe it does; I've always wondered
why samba hangs around 3, 4, 5, maybe 6 MB/s on a 100mbit
link - which with netio shows that it can go at 11MB/s, its full
speed - and even nfs rarely tops above eight. On my friend's
setup too, and he *does* have respectable hardware in his
server:)
I assume the 20MB/s top speed on my gbit cards is io limiting,
and possibly the fact that they're 32bit cards in 33mhz slots.
Still, it's far from impressive. Any suggestions (on where to
go) about improving it?
Anyhow, thanks for the help so far.
--Bart Alewijnse
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-30 15:15 ` Bart Alewijnse
@ 2004-07-30 18:54 ` Francois Romieu
2004-07-30 21:03 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Francois Romieu @ 2004-07-30 18:54 UTC (permalink / raw)
To: Bart Alewijnse; +Cc: netdev
Bart Alewijnse <scarfboy@gmail.com> :
[...]
> Definately. I've now seen speeds up to 22MB/s, but only in pure
> network benching (netio), and only with udp, although I guess
> that makes a some sense.
It seems low. On a non-napi setup, I'd expect your celeron box to
stand at least 20~30 kIRQ/s if the hardware is not flawed.
[...]
> I assume the 20MB/s top speed on my gbit cards is io limiting,
> and possibly the fact that they're 32bit cards in 33mhz slots.
> Still, it's far from impressive. Any suggestions (on where to
> go) about improving it?
See Documentation/networking/ip-sysctl.txt and the r/wmem parameters
for a start. The 2.6.8-rc2-mm1 version of the r8169 driver is suggested.
A dumb 'vmstat 1' output during test could give some hint.
Can I assume that the systems are stable if not fast ?
--
Ueimor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-30 18:54 ` Francois Romieu
@ 2004-07-30 21:03 ` Bart Alewijnse
2004-07-30 21:41 ` Francois Romieu
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-07-30 21:03 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
You are aware that this is a celeron at 400 mhz, and not whatever 400
is or possibly isn't in that annoying new naming scheme? (Just
checking...)
Both systems are not overclocked, so are stable in that respect. The
nonbooting thing was a loose ground wire touching my hard drive
jumpers, making it think it was a slave. I'm surprised winxp managed
to do anything at all. I've used both these systems for ages, never
had any real trouble except for the strange pci slot/irq problems in
my new computer, but that was mostly a work-or-not thing - although it
also increases pci latency. I'm still not sure whether it's a card
that's in there, a card that was in there, (I replaced my sound card,
and can suddently get a much lower midi-to-wave playing latency - but
the sound card has been known to click and pop randomly over several
reconfigurations the last few days) or just a screwy motherboard. I
should hope it's not the last, it's a brand board.
Anyhow, on transmit from the celeron box, under extreme benchy
circumstrances, I've seen it around 16Kints/s on transmit and 13k on
receive. But under everyday nfs/samba, 6400 is about the best it does
either way.
The maximum amount of interrupts/s I've seen so far 22302 (including
the 1000 in the kernel timer, 'course). Nothing much else changes,
except the context switches raising from an idle 10~30 to 1000~1500
to, sometimes 30000. Actually, using netio seems to busy the system
completely:
# top -d 0.5 | grep Cpu\(
Cpu(s): 8.8% us, 36.8% sy, 0.0% ni, 1.5% id, 0.0% wa, 13.2% hi, 39.7% si
Cpu(s): 7.5% us, 38.8% sy, 0.0% ni, 0.0% id, 0.0% wa, 14.9% hi, 38.8% si
Cpu(s): 5.7% us, 41.4% sy, 0.0% ni, 0.0% id, 0.0% wa, 14.9% hi, 37.9% si
Cpu(s): 5.0% us, 44.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 14.5% hi, 36.5% si
Cpu(s): 4.4% us, 50.5% sy, 0.0% ni, 1.1% id, 0.0% wa, 12.1% hi, 31.9% si
Cpu(s): 3.8% us, 50.9% sy, 0.0% ni, 0.0% id, 0.0% wa, 17.0% hi, 28.3% si
But a to-winxp smb file transfer, hangin around 7, 8MB/s, doesn't:
Cpu(s): 3.8% us, 50.0% sy, 0.0% ni, 17.9% id, 0.0% wa, 7.5% hi, 20.8% si
Cpu(s): 5.0% us, 43.0% sy, 0.0% ni, 27.0% id, 0.0% wa, 7.0% hi, 18.0% si
Cpu(s): 4.8% us, 40.4% sy, 0.0% ni, 29.8% id, 0.0% wa, 5.8% hi, 19.2% si
Cpu(s): 5.1% us, 43.9% sy, 0.0% ni, 26.5% id, 0.0% wa, 5.1% hi, 19.4% si
Cpu(s): 5.3% us, 42.5% sy, 0.0% ni, 26.5% id, 0.0% wa, 6.2% hi, 19.5% si
Cpu(s): 5.9% us, 43.1% sy, 0.0% ni, 25.5% id, 1.0% wa, 5.9% hi, 18.6% si
Cpu(s): 4.6% us, 41.7% sy, 0.0% ni, 25.9% id, 0.0% wa, 5.6% hi, 22.2% si
Cpu(s): 6.2% us, 44.6% sy, 0.0% ni, 23.2% id, 0.0% wa, 5.4% hi, 20.5% si
Cpu(s): 4.2% us, 34.7% sy, 0.0% ni, 38.9% id, 0.0% wa, 5.3% hi, 16.8% si
It's not too consistent all over either; windows netio transmits
slowly when packet sizes are >1k (the mtu?), as low as 5MB/s, while
the receive speed for the same packet size goes at ~22MB
Ooh, finally, a proper kernel pan --er, bug, it says. On my old
computer this time.
I know, incidentally, that the memory works; I recently did a three-hour memtest
on this computer. Picture here:
http://www.scarfboy.com/other/panic-30jul.gif
This may be due to fiddling with said wmem, etc values, I set some of them
considerably larger. I did get a few percent apparent speed increase,
incidentally, though that may have been wishful thinking.
I guess I should try >= 2.6.8-rc2-mm1 next?
--Bart Alewijnse
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-30 21:03 ` Bart Alewijnse
@ 2004-07-30 21:41 ` Francois Romieu
2004-07-31 19:51 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Francois Romieu @ 2004-07-30 21:41 UTC (permalink / raw)
To: Bart Alewijnse; +Cc: netdev, linux-kernel
Bart Alewijnse <scarfboy@gmail.com> :
> You are aware that this is a celeron at 400 mhz, and not whatever 400
> is or possibly isn't in that annoying new naming scheme? (Just
> checking...)
Yes. It is good enough to handle some network traffic.
[...]
> Anyhow, on transmit from the celeron box, under extreme benchy
> circumstrances, I've seen it around 16Kints/s on transmit and 13k on
> receive. But under everyday nfs/samba, 6400 is about the best it does
> either way.
The figures does not seem bad.
I am curious: which chipset does the motherboard include ?
An 'lspci -vx' sums it quite well.
[...]
> This may be due to fiddling with said wmem, etc values, I set some of them
It should not.
> considerably larger. I did get a few percent apparent speed increase,
> incidentally, though that may have been wishful thinking.
>
> I guess I should try >= 2.6.8-rc2-mm1 next?
Please. Do not compile in preempt/ipv6/smp. SMP is supposed to be safe
but you do not need it and it will not make your r8169 faster if you have
only one CPU. I'd welcome a 'vmstat 1' output as it gives the bi/bo.
--
Ueimor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-30 21:41 ` Francois Romieu
@ 2004-07-31 19:51 ` Bart Alewijnse
2004-07-31 21:18 ` Francois Romieu
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-07-31 19:51 UTC (permalink / raw)
To: netdev, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2854 bytes --]
On Fri, 30 Jul 2004 23:41:20 +0200, Francois Romieu
<romieu@fr.zoreil.com> wrote:
> [...]
> > Anyhow, on transmit from the celeron box, under extreme benchy
> > circumstrances, I've seen it around 16Kints/s on transmit and 13k on
> > receive. But under everyday nfs/samba, 6400 is about the best it does
> > either way.
>
> The figures does not seem bad.
Except they're about a factor 1.2 to 1.7 faster than my 100mbit, in practical
(network filesystem) application. That's very short of spectacular.
> I am curious: which chipset does the motherboard include ?
> An 'lspci -vx' sums it quite well.
Ermm... VIA, I believe it was called Apollo Pro. VT82C693.
But as the rest may matter, the full lspci -vx listing's attached.
> [...]
> > This may be due to fiddling with said wmem, etc values, I set some of them
> It should not.
If my computer did what it should do, would I be here?:)
> > considerably larger. I did get a few percent apparent speed increase,
> > incidentally, though that may have been wishful thinking.
> >
> > I guess I should try >= 2.6.8-rc2-mm1 next?
>
> Please. Do not compile in preempt/ipv6/smp. SMP is supposed to be safe
> but you do not need it and it will not make your r8169 faster if you have
> only one CPU. I'd welcome a 'vmstat 1' output as it gives the bi/bo.
ipv6 was nonsense anyhow (and I think loaded as a module), smp was force
of habit as something kernely ages back wanted it. But is there really no
point to preempting in a server?
Right, both computers now run 2.6.8-rc2-mm1. It's better. Roughly
speaking, the top *benchmark* speeds, rouding slightly up, are:
udp: 33MB/s (using ~26Kints) one way, 12MB/s (9Kints/s) the other,
tcp: 22MB/s (16Kints)one way, 12MB/s (9Kints/s) the other.
(The 9Kints ca be 12 when the packet size is 1K or 2K)
Oddly enough, the slow direction is sending from my new computer
(XP1700, KT333, 1GB ram) to my old (both running linux and the
mentioned kernel). I'm fairly sure this is due to the fact that my
asus motherboard is crap, and I guess I'll have to settle for it,
unless one of you is very good at black magic. Because trust me, the
way hardware and chooses not to work, apparently based on slots and/or
irq's is just *weird*. Bah.
Practical speeds increased far less dramatically. nfs now sustains at
10MB/s, with the occasional 12. Samba increased less, about half a meg
a second, it sustains speeds slightly over 8MB/s, to both linux and
windows.
I guess no network filesystems focuses on speed? I mean, I did always
wonder why nfs and samba on 100mbit transfers averaged around half the
line speed, and that's counting large files (no tcp startup to blame).
I don't suppose any of you know a good windows nfs client.
I guess I'll have to settle for this. I'm just annoyed that the 'giga'
is basically a joke, especially on my setup.
--Bart Alewijnse
[-- Attachment #2: lspci-vx.txt --]
[-- Type: text/plain, Size: 3651 bytes --]
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT82C693A/694x [Apollo PRO133x] (rev 06)
Flags: bus master, medium devsel, latency 0
Memory at dc000000 (32-bit, prefetchable)
Capabilities: [a0] AGP version 1.0
00: 06 11 91 06 06 00 90 a2 06 00 00 06 00 00 00 00
10: 08 00 00 dc 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 a0 00 00 00 00 00 00 00 00 00 00 00
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT82C598/694x [Apollo MVP3/Pro133x AGP] (prog-if 00 [Normal decode])
Flags: bus master, 66Mhz, medium devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
00: 06 11 98 85 07 00 20 22 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 f0 00 00 00
20: f0 ff 00 00 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 00
0000:00:07.0 ISA bridge: VIA Technologies, Inc. VT82C596 ISA [Mobile South] (rev 11)
Subsystem: VIA Technologies, Inc. VT82C596/A/B PCI to ISA Bridge
Flags: bus master, stepping, medium devsel, latency 0
00: 06 11 96 05 87 00 00 02 11 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000:00:07.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) (prog-if 8a [Master SecP PriP])
Flags: bus master, medium devsel, latency 32
I/O ports at e000 [size=16]
00: 06 11 71 05 07 00 80 02 06 8a 01 01 00 20 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 ff 00 00 00
0000:00:07.3 Host bridge: VIA Technologies, Inc. VT82C596 Power Management (rev 20)
Flags: medium devsel
00: 06 11 51 30 00 00 80 02 20 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000:00:09.0 VGA compatible controller: S3 Inc. 86c325 [ViRGE] (rev 06) (prog-if 00 [VGA])
Flags: bus master, medium devsel, latency 32, IRQ 15
Memory at d8000000 (32-bit, non-prefetchable)
00: 33 53 31 56 07 00 00 02 06 00 00 03 00 20 00 00
10: 00 00 00 d8 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 0f 01 04 ff
0000:00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Flags: bus master, medium devsel, latency 32, IRQ 11
I/O ports at e800
Memory at df001000 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
00: ec 10 39 81 07 00 90 02 10 00 00 02 00 20 00 00
10: 01 e8 00 00 00 10 00 df 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ec 10 39 81
30: 00 00 00 00 50 00 00 00 00 00 00 00 0b 01 20 40
0000:00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
Flags: bus master, 66Mhz, medium devsel, latency 32, IRQ 10
I/O ports at ec00 [size=de000000]
Memory at df000000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 00010000 [disabled]
Capabilities: [dc] Power Management version 2
00: ec 10 69 81 07 00 b0 02 10 00 00 02 08 20 00 00
10: 01 ec 00 00 00 00 00 df 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ec 10 69 81
30: 00 00 00 de dc 00 00 00 00 00 00 00 0a 01 20 40
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-31 19:51 ` Bart Alewijnse
@ 2004-07-31 21:18 ` Francois Romieu
2004-08-01 19:03 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Francois Romieu @ 2004-07-31 21:18 UTC (permalink / raw)
To: Bart Alewijnse; +Cc: netdev, linux-kernel
Bart Alewijnse <scarfboy@gmail.com> :
[...]
> of habit as something kernely ages back wanted it. But is there really no
> point to preempting in a server?
I don't know: I have no URL for the figures at hand.
> Right, both computers now run 2.6.8-rc2-mm1. It's better. Roughly
> speaking, the top *benchmark* speeds, rouding slightly up, are:
> udp: 33MB/s (using ~26Kints) one way, 12MB/s (9Kints/s) the other,
> tcp: 22MB/s (16Kints)one way, 12MB/s (9Kints/s) the other.
>
> (The 9Kints ca be 12 when the packet size is 1K or 2K)
Can you give a try to a napi version of the module (it is a compile-time
option) ?
You may want higher {r/w}mem_max as well as renicing ksoftirqd with
a strongly < 0 value on the celeron client.
> Oddly enough, the slow direction is sending from my new computer
> (XP1700, KT333, 1GB ram) to my old (both running linux and the
> mentioned kernel). I'm fairly sure this is due to the fact that my
The r8169 driver has been reported to be faster on Rx than on Tx so your
observation makes sense.
[...]
> I guess I'll have to settle for this. I'm just annoyed that the 'giga'
> is basically a joke, especially on my setup.
It should be possible to make things slightly better for the celeron box
as a client. I'll cook up a patch.
Would you be kind enough to do some ftp transfer test where the celeron
box retrieves data and send the 'vmstat 1' output of the client during
the test ?
--
Ueimor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-07-31 21:18 ` Francois Romieu
@ 2004-08-01 19:03 ` Bart Alewijnse
2004-08-03 2:47 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-08-01 19:03 UTC (permalink / raw)
To: netdev, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 3253 bytes --]
On Sat, 31 Jul 2004 23:18:36 +0200, Francois Romieu
<romieu@fr.zoreil.com> wrote:
> > Right, both computers now run 2.6.8-rc2-mm1. It's better. Roughly
> > speaking, the top *benchmark* speeds, rouding slightly up, are:
> > udp: 33MB/s (using ~26Kints) one way, 12MB/s (9Kints/s) the other,
> > tcp: 22MB/s (16Kints)one way, 12MB/s (9Kints/s) the other.
> >
> > (The 9Kints ca be 12 when the packet size is 1K or 2K)
>
> Can you give a try to a napi version of the module (it is a compile-time
> option) ?
> You may want higher {r/w}mem_max as well as renicing ksoftirqd with
> a strongly < 0 value on the celeron client.
That's with NAPI on both. I notice r/wmem has an effect, but it's not much.
Although on my new computer it seems moot as there's something weird
going on with interrupts with my botherboard.
> > Oddly enough, the slow direction is sending from my new computer
> > (XP1700, KT333, 1GB ram) to my old (both running linux and the
> > mentioned kernel). I'm fairly sure this is due to the fact that my
>
> The r8169 driver has been reported to be faster on Rx than on Tx so your
> observation makes sense.
How does that make sense? When one side receives, the other sends, hrm?
They're two identical cards, it should be entirely symmetrical assuming
equal hardware - not faster on the years older hardware.
> [...]
> > I guess I'll have to settle for this. I'm just annoyed that the 'giga'
> > is basically a joke, especially on my setup.
>
> It should be possible to make things slightly better for the celeron box
> as a client. I'll cook up a patch.
The celeron box is the server, that's the entire point. Anyhow, it's
much faster in transmission than my new computer right now due to said
mobo problem, so I *want* it as the server.
> Would you be kind enough to do some ftp transfer test where the celeron
> box retrieves data and send the 'vmstat 1' output of the client during
> the test ?
Sure, but I can tell you right now with reasonable certaintly that my
new computer
won't top 9000interrupts/sec, i.e. 9000 packets per second, and therefore do
the 12MB/s at most, and probably less; interruptwise, my new computer
is the bottleneck, and I'm guessing UDP is faster because TCP is
limited by the amount of packets, or in the other direction ACKs, it
can send per second.
Transfers to the celeron are a relatively pointless measure because
of my new computer being horribly interrupt limited at the moment.
That may have started when I installed the gbit card, but mostly because it
disturbs the mobo's precious and unguessable hardware balance. I'll try figuring
out if I can solve it, but basically it just involves swapping around
cards until it
works better, so that'ld take a while.
Attached are the vmstats for an ftp and nfs transfer, as measured from
my old computer (the nfs and ftp server). Both were going at a pretty
low speed, sub-9000 packets; this may have to do with drive
fragmentation, my hard drives are rather full at the moment.
Also, the logs on the client (my new computer) while sending data to
my old one. These were made at a different time, and I believe the
transfer rate was better (6MB for ftp, 9MB for nfs) than for the
-to-old- logs, which were worse, and varied more.
--Bart
[-- Attachment #2: ftp-put-to-old-computer__lin-mm-to-lin-mm_larger --]
[-- Type: application/octet-stream, Size: 4268 bytes --]
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 48532 29648 175464 0 0 101 31 273 169 15 2 80 2
0 0 0 48532 29648 175468 0 0 0 0 1007 9 0 1 99 0
1 0 0 45796 29648 178212 0 0 0 0 2873 35 2 33 65 0
1 0 0 39216 29648 184532 0 0 0 0 5131 36 7 76 17 0
1 0 0 32820 29648 191348 0 0 0 0 4938 52 7 77 16 0
0 1 0 28788 29908 195028 0 0 0 4628 3276 41 4 47 26 23
1 1 0 24308 29908 199020 0 0 0 8656 4211 40 8 90 0 2
1 0 0 20596 29908 202724 0 0 0 6500 3964 257 6 94 0 0
1 0 0 13940 29908 209404 0 0 0 0 5994 14 8 92 0 0
1 0 0 7216 29908 216116 0 0 0 8 5996 12 6 94 0 0
1 0 0 2408 29840 221084 0 0 0 8 5931 28 8 92 0 0
1 0 0 2856 29740 221320 0 0 0 1184 5574 51 5 95 0 0
1 1 0 2536 30400 221400 0 0 0 6524 4886 90 6 94 0 0
1 2 0 2856 30276 221580 0 0 0 7780 4279 73 7 93 0 0
1 2 0 2408 30248 222368 0 0 0 6368 4251 73 5 95 0 0
1 2 0 2244 29904 223260 0 0 0 6344 4952 68 6 94 0 0
1 2 0 3116 29636 222960 0 0 0 5244 4163 81 8 92 0 0
1 1 0 2472 29528 223968 0 0 0 5664 4731 101 7 93 0 0
1 0 0 2920 29416 224020 0 0 0 1268 5124 194 6 94 0 0
1 0 0 2984 29132 224628 0 0 0 636 5825 40 8 92 0 0
1 0 0 2280 28944 225796 0 0 0 6668 4807 73 6 94 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 2260 28668 226352 0 0 0 3976 4973 77 7 93 0 0
1 0 0 2408 28420 226664 0 0 0 5320 4514 96 6 94 0 0
1 1 0 2664 28368 226480 0 0 0 19264 3261 44 6 94 0 0
1 1 0 2476 28288 226732 0 0 0 16184 2124 252 7 93 0 0
1 0 0 2920 28056 226968 0 0 0 536 3898 67 7 93 0 0
1 0 0 2804 27696 227700 0 0 0 8 5863 53 6 94 0 0
1 0 0 2984 27504 227928 0 0 0 8 5929 70 7 93 0 0
1 0 0 2920 27008 228648 0 0 0 4 5949 53 9 91 0 0
1 0 0 2408 26840 229468 0 0 0 1436 5543 50 6 94 0 0
2 0 0 2792 26808 228560 0 0 0 35532 3685 46 8 92 0 0
1 1 0 3064 26716 228916 0 0 0 0 2085 217 8 92 0 0
1 0 0 2984 26520 229572 0 0 0 160 3751 69 6 94 0 0
1 0 0 2536 26352 230324 0 0 0 8 5953 50 8 92 0 0
1 0 0 2920 24116 232332 0 0 0 8 5894 53 7 93 0 0
1 0 0 3048 22420 234180 0 0 0 4 5945 48 6 94 0 0
1 0 0 3112 21064 235396 0 0 0 1348 5476 55 7 91 2 0
1 1 0 1944 20572 236508 0 0 0 35652 3628 52 6 94 0 0
0 2 0 2132 20056 237048 0 0 1064 0 1125 150 7 22 0 70
1 1 0 3188 18984 236408 0 0 224 0 1077 108 61 10 0 29
0 0 0 4616 18984 236408 0 0 0 144 1034 34 2 2 83 13
0 0 0 4616 18984 236408 0 0 0 0 1006 10 0 0 100 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 178516 18984 63676 0 0 0 0 1009 12 0 33 67 0
0 0 0 177572 20180 63680 0 0 48 1192 1033 73 1 4 70 25
0 0 0 177572 20180 63680 0 0 0 0 1008 10 0 0 100 0
0 0 0 177892 20216 63680 0 0 0 1124 1224 39 0 3 40 57
0 0 0 177892 20216 63684 0 0 0 0 1067 13 0 1 85 14
0 0 0 177892 20220 63688 0 0 8 0 1011 16 0 0 99 1
[-- Attachment #3: nfs-to-old-computer__lin-mm-to-lin-mm --]
[-- Type: application/octet-stream, Size: 4139 bytes --]
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 178240 20360 63688 0 0 101 33 276 169 15 2 80 2
0 0 0 178240 20400 63696 0 0 8 68 1019 27 0 1 96 3
0 0 0 178240 20400 63696 0 0 0 0 1006 8 0 0 100 0
0 0 0 178240 20400 63696 0 0 0 0 1008 14 0 0 100 0
0 0 0 178240 20400 63696 0 0 0 0 1004 12 0 0 100 0
0 0 0 178240 20400 63696 0 0 0 0 1005 14 0 0 100 0
0 0 0 178240 20412 63696 0 0 0 20 1011 20 0 0 100 0
0 0 0 178176 20460 63696 0 0 4 72 1024 37 0 0 99 1
1 0 0 171012 20468 70512 0 0 0 0 2589 222 0 75 25 0
1 0 0 162244 20476 79280 0 0 0 0 5497 289 0 100 0 0
1 0 0 153732 20484 88048 0 0 0 0 5618 1214 0 100 0 0
1 1 0 145540 21016 95280 0 0 0 5848 4887 528 1 99 0 0
1 1 0 139268 21020 101552 0 0 0 5276 4414 1260 0 100 0 0
1 2 0 132228 21028 108656 0 0 0 2828 4992 5500 0 100 0 0
1 2 0 125380 21036 115408 0 0 0 4320 4885 6024 0 100 0 0
1 2 0 119044 21040 121840 0 0 0 5760 5352 5829 0 100 0 0
1 2 0 113988 21048 126832 0 0 0 6196 4268 4731 1 99 0 0
2 1 0 109764 21052 130888 0 0 0 5224 4011 3766 0 64 0 36
1 0 0 103876 21056 136812 0 0 0 7876 4831 5618 0 100 0 0
1 0 0 96516 21064 143916 0 0 0 0 5529 6671 0 100 0 0
1 0 0 91332 21068 149068 0 0 0 9348 3804 4864 0 100 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 0 82948 21076 157064 0 0 0 6724 6287 7503 0 100 0 0
1 1 0 79364 21356 159848 0 0 0 36220 1967 2719 0 100 0 0
1 1 0 76228 21360 163336 0 0 0 0 2064 3335 0 100 0 0
1 0 0 71620 21364 168224 0 0 0 280 3422 4576 1 99 0 0
1 0 0 67716 21424 172092 0 0 0 10808 3681 3775 0 56 0 44
1 0 0 59396 21432 180284 0 0 0 0 6356 7676 0 100 0 0
1 0 0 51012 21440 188536 0 0 0 0 6379 7739 0 100 0 0
1 0 0 42628 21448 196836 0 0 0 0 6465 7772 0 100 0 0
1 1 0 35460 21532 203256 0 0 0 33336 4836 5986 0 100 0 0
1 1 0 32196 21536 206852 0 0 0 0 2180 3430 0 100 0 0
1 0 0 27908 21540 211252 0 0 0 376 3021 4165 0 100 0 0
1 0 0 19652 21548 219428 0 0 0 4 6154 7661 0 100 0 0
1 0 0 11524 21556 227504 0 0 0 4 6416 7563 0 100 0 0
1 0 0 3128 21564 235772 0 0 0 0 6432 7746 1 99 0 0
3 0 0 2352 17428 240988 0 0 0 3604 5411 5982 0 100 0 0
0 2 0 2816 16080 241064 0 0 0 32860 4643 3364 0 85 0 15
2 1 0 2832 15604 241944 0 0 0 292 2091 2591 0 84 0 16
2 1 0 2600 14836 243504 0 0 0 3640 3358 4167 0 100 0 0
2 0 0 2856 13836 244664 0 0 0 5796 4396 1931 0 100 0 0
0 1 0 2224 13532 245740 0 0 0 4224 2970 2312 0 35 0 65
0 1 0 3060 13404 244828 0 0 0 8028 1660 52 0 5 0 95
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 2736 13380 245708 0 0 0 3148 3310 2317 0 42 0 58
1 0 0 2856 12616 246500 0 0 0 0 6892 5792 0 100 0 0
1 1 0 2732 12396 247544 0 0 0 2624 6287 1131 0 100 0 0
1 1 0 2536 12100 248196 0 0 0 11404 3940 251 0 100 0 0
[-- Attachment #4: clientlog-ftpput --]
[-- Type: application/octet-stream, Size: 3635 bytes --]
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 198312 70336 258240 0 0 299 85 1536 767 8 6 78 8
0 0 0 198228 70336 258308 0 0 68 0 1437 463 3 1 94 2
0 0 0 198228 70336 258308 0 0 0 0 2013 1589 11 6 83 0
0 1 0 196652 70388 259820 0 0 1506 92 2329 540 2 3 87 8
0 0 0 190240 70392 266208 0 0 6414 0 6567 808 0 11 84 5
0 0 0 183604 70400 272864 0 0 6662 0 6618 814 0 10 84 6
0 0 0 179236 70404 277212 0 0 4357 0 5303 1305 6 10 81 3
0 0 0 175024 70408 281424 0 0 4228 0 4545 712 1 7 89 3
0 0 0 167832 70428 288136 0 0 6718 20 7369 1026 8 13 73 6
0 0 0 160492 70432 294796 0 0 6663 0 6648 906 0 9 85 5
0 0 0 153236 70440 301384 0 0 6534 0 6612 905 1 11 83 5
0 0 0 145748 70448 308108 0 0 6791 0 6696 926 1 11 81 7
0 0 0 142612 70448 310964 0 0 2818 0 3477 648 1 5 92 2
0 0 0 142612 70460 310952 0 0 0 20 1112 500 0 0 100 0
0 0 0 135956 70468 316996 0 0 6022 0 6129 856 1 10 84 5
0 0 0 128724 70472 323520 0 0 6535 0 6670 888 1 13 80 6
0 0 0 121364 70480 330176 0 0 6662 0 6606 894 0 11 83 6
0 0 0 114004 70488 336832 0 0 6663 0 6678 925 1 12 81 6
0 0 0 107156 70504 342936 0 0 6150 20 6360 867 1 10 82 6
0 0 0 104340 70508 345516 0 0 2562 0 3200 685 1 5 91 3
0 0 0 101396 70508 348168 0 0 2691 0 3338 645 0 3 95 2
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 94164 70516 354756 0 0 6534 0 6698 904 1 11 82 6
0 0 0 86740 70524 361412 0 0 6663 0 6668 935 4 12 77 6
0 0 0 79380 70540 368060 0 0 6662 20 6651 897 1 11 83 5
0 0 0 72020 70548 374716 0 0 6663 0 6685 903 1 11 82 6
0 0 0 66720 70552 379676 0 0 4997 0 5777 871 2 8 85 4
1 0 0 64148 70556 381984 0 0 2306 0 3309 1039 5 6 87 2
0 0 0 60764 70556 385044 0 0 3075 0 4013 631 1 6 91 2
0 0 0 53404 70576 391688 0 0 6662 20 7302 1390 5 15 74 6
1 0 0 46412 70580 398008 0 0 6278 0 6824 1459 6 15 75 4
0 0 0 46412 70580 398008 0 0 0 0 1191 400 1 1 98 0
0 0 0 46412 70580 398008 0 0 0 0 1091 396 0 1 99 0
0 0 0 46412 70580 398008 0 0 0 0 1113 436 1 0 99 0
0 0 0 46412 70592 397996 0 0 0 20 1127 563 2 0 98 0
0 0 0 46412 70592 397996 0 0 0 0 1171 617 3 1 96 0
0 0 0 46428 70592 397996 0 0 0 0 1300 580 3 1 96 0
0 0 0 46428 70592 397996 0 0 0 0 1097 409 1 0 99 0
0 0 0 46428 70592 397996 0 0 0 0 1203 804 5 1 94 0
0 0 0 46300 70640 398016 0 0 0 112 1236 1273 11 3 86 0
0 0 0 46300 70640 398016 0 0 0 0 1568 1574 13 4 83 0
0 0 0 46620 70640 398016 0 0 0 0 2298 1439 12 6 82 0
0 0 0 46648 70640 398016 0 0 0 0 1408 908 5 2 93 0
[-- Attachment #5: clientlog-nfsput --]
[-- Type: application/octet-stream, Size: 4266 bytes --]
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 16776 292012 166988 0 0 276 87 1521 770 8 6 78 8
0 0 0 16748 292012 166988 0 0 0 0 1476 416 3 1 96 0
0 0 0 16748 292012 166988 0 0 0 0 2640 1564 12 6 82 0
0 0 0 16748 292012 166988 0 0 0 0 1104 426 1 0 99 0
0 0 0 16748 292012 166988 0 0 0 0 1219 435 1 1 98 0
0 1 0 15356 292052 168308 0 0 744 80 1259 488 1 2 93 4
0 1 0 5036 286580 184524 0 0 9476 0 1284 622 3 14 0 83
0 1 0 4588 275636 198868 0 0 9348 0 1301 664 4 17 0 79
0 1 0 4972 269376 206352 0 0 4625 0 1263 564 1 8 0 91
1 1 0 5248 264268 212208 0 0 3725 0 1495 687 5 8 0 87
1 1 0 5292 238764 243764 0 0 19358 16 2129 1886 16 38 0 47
0 1 0 5164 212332 274888 0 0 19613 8 2160 1023 7 36 0 57
0 1 0 5008 199712 289140 0 0 14739 0 3938 921 6 32 0 62
0 1 0 5344 180980 312020 0 0 18578 0 8116 1149 6 44 0 49
0 1 0 4960 173048 322876 0 0 8852 0 7988 1086 4 25 0 71
0 1 0 5360 163640 334188 0 0 8474 0 6578 1012 5 22 0 73
0 2 0 5288 154676 346280 0 0 9730 4 4463 915 4 23 0 72
0 2 0 5348 145140 358332 0 0 7812 12 7056 1168 4 25 0 71
0 1 0 5368 139152 366564 0 0 5065 0 7798 1062 3 18 0 79
0 1 0 5012 133780 372820 0 0 3855 0 7854 995 3 18 0 79
0 1 0 5012 130208 377004 0 0 2571 0 7841 967 3 14 0 83
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 0 5136 128292 379056 0 0 1161 0 7270 2039 12 16 0 72
0 1 0 4760 120656 388392 0 0 5904 0 4059 805 4 15 0 81
0 2 0 5536 111688 397496 0 0 5762 12 4195 899 3 15 0 82
1 1 0 5528 108032 401764 0 0 2721 4 3605 851 3 11 0 86
0 1 0 5276 100952 410204 0 0 5243 0 7641 1045 2 20 0 78
0 1 0 4636 95532 417188 0 0 4289 0 7621 1042 3 15 0 81
0 1 0 5612 90808 421436 0 0 2781 0 7609 1053 2 14 0 84
0 1 0 5276 80724 433696 0 0 7256 4 7748 1093 3 22 0 74
0 0 0 4908 69872 446044 0 0 7533 20 5074 948 4 17 61 17
0 0 0 4908 69872 446044 0 0 0 0 3803 610 0 3 97 0
0 0 0 4908 69872 446044 0 0 0 0 5064 713 1 3 96 0
0 0 0 4972 69872 446044 0 0 0 0 6978 864 0 8 92 0
0 0 0 5164 69872 446044 0 0 0 0 7493 914 0 6 94 0
0 0 0 5164 69884 446032 0 0 0 20 1207 419 1 1 98 0
0 0 0 6380 69884 446032 0 0 0 0 4992 712 0 6 94 0
0 0 0 6508 69884 446032 0 0 0 0 7394 915 0 5 95 0
0 0 0 6572 69884 446032 0 0 0 0 3858 619 1 5 94 0
0 0 0 6636 69884 446100 0 0 12 0 5700 820 1 4 94 1
0 0 0 7600 69928 446056 0 0 0 104 5860 783 1 7 92 0
0 0 0 8116 69928 446056 0 0 0 0 5489 762 3 6 91 0
0 0 0 8440 69928 446056 0 0 0 0 6020 852 1 6 93 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 8568 69928 446056 0 0 0 0 5433 822 1 5 94 0
0 0 0 10488 69944 446040 0 0 7 0 2099 728 4 6 86 4
0 0 0 10480 69956 446096 0 0 0 20 1224 952 4 1 95 0
0 0 0 10480 69956 446096 0 0 0 0 1283 465 2 1 97 0
0 0 0 199252 69956 257600 0 0 0 0 1779 1300 9 7 84 0
0 0 0 199252 69956 257600 0 0 0 0 1445 1205 19 4 77 0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-08-01 19:03 ` Bart Alewijnse
@ 2004-08-03 2:47 ` Bart Alewijnse
2004-08-03 7:48 ` Francois Romieu
0 siblings, 1 reply; 11+ messages in thread
From: Bart Alewijnse @ 2004-08-03 2:47 UTC (permalink / raw)
To: netdev, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]
Okay, after I Did Stuff(tm) to my new computer, it's capable of
generating approximately 50Kints/s, which transates to 66MB/s in a
network benchmark (Actually, netio misreports the speeds, probably
because of an overflow; I was amused)
There's two howevers and one unfortunately, though.
The howevers:
- it's not in the direction I want -- meaning I should update, which I
can live with. But:
- the bench speed going from my old computer wend down from ~33mb to
~20MB/s which I can't make any sense of at all, and the practical
speeds from the server didn't change for the better, possibly for the
worse. I'm not sure, because of...
The unfortunately:
- After a few minutes, my old computer kernel paniced. I assume this
is because it can't handle the incoming traffic, as its own pace of
sending (and therefore probably handling interrupts) is more like
20kints/sec than 50. I couldn't tell anything vmstat-wise as during
the data going in that direction (un udp; tcp hung around 17MB both
ways - which is better thn before, but *still* short of impressive),
the ssh shells just stopped responding, and I guess so did vmstat.
It's probably being battered in hardware interrupts, I saw high
figures. Quite possibly this has to do with renicing ksoftircq
(negatively) as someone suggested. However, I am still of the opinion
that a difference in processor speeds should not crash anything.
The panic looks a lot like the last one; same kernel (napi still
enabled for the 8169). Image attached.
I'ld like to know what's wrong here, so that I can avoid it. I can
deal and live with stupid speeds better than crashing serves. Makes me
grumble less, y'know.
--Bart Alewijnse
[-- Attachment #2: panic2.gif --]
[-- Type: image/gif, Size: 84567 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-08-03 2:47 ` Bart Alewijnse
@ 2004-08-03 7:48 ` Francois Romieu
2004-08-03 12:32 ` Bart Alewijnse
0 siblings, 1 reply; 11+ messages in thread
From: Francois Romieu @ 2004-08-03 7:48 UTC (permalink / raw)
To: Bart Alewijnse; +Cc: netdev, linux-kernel
Bart Alewijnse <scarfboy@gmail.com> :
[...]
> The panic looks a lot like the last one; same kernel (napi still
> enabled for the 8169). Image attached.
The irq rate are strangely high for a napi version of the r8169 driver.
Can you describe your test commands so that I reproduce these here ?
--
Ueimor
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: gigabit trouble
2004-08-03 7:48 ` Francois Romieu
@ 2004-08-03 12:32 ` Bart Alewijnse
0 siblings, 0 replies; 11+ messages in thread
From: Bart Alewijnse @ 2004-08-03 12:32 UTC (permalink / raw)
To: netdev, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]
Quite simply a 'netio -s' on one computer, and a 'netio -u
192.168.1.whichever' on the other. I was monitoring with 'vmstat 1'
for hardware stuff and bmon to see the actual speed and had a ssh
open.
The things I've been fiddling with recently to see if it'd make a difference:
- Renicing of ksoftirqc to anything between -5 and -17, I don't
remember exactly.
- Disabling tcp window scaling. I read somewhere that that may help
throughhput, but it may be a stupid move.
- Some barely thought about (I did read the following:
http://lists.samba.org/archive/samba/2003-December/077198.html)
window memory buffer size changes, with a 'so that that at least won't
be a bottleneck' angle:
--------------------
echo 400000 > /proc/sys/net/core/wmem_default
echo 512000 > /proc/sys/net/core/wmem_max
echo 400000 > /proc/sys/net/core/rmem_default
echo 512000 > /proc/sys/net/core/rmem_max
echo 98304 512000 640000 > /proc/sys/net/ipv4/tcp_mem
echo 98304 512000 640000 > /proc/sys/net/ipv4/tcp_wmem
echo 98304 512000 640000 > /proc/sys/net/ipv4/tcp_rmem
echo 98304 512000 640000 > /proc/sys/net/ipv4/tcp_mem
-----------
It seems that windows (in which I have jumbo frames on) doesn' quite
need as many interrupts, although I'm not sure that was for the same
speed, as I had another kernel panic (attached, this one has a small
visible trace). I was running the windows version of netio as a
server, and doing a client test with udp.
--Bart Alewijnse
[-- Attachment #2: panic3.gif --]
[-- Type: image/gif, Size: 95552 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2004-08-03 12:32 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <b71082d8040729094537e59a11@mail.gmail.com>
[not found] ` <20040729210401.A32456@electric-eye.fr.zoreil.com>
2004-07-29 22:41 ` gigabit trouble Bart Alewijnse
2004-07-30 15:15 ` Bart Alewijnse
2004-07-30 18:54 ` Francois Romieu
2004-07-30 21:03 ` Bart Alewijnse
2004-07-30 21:41 ` Francois Romieu
2004-07-31 19:51 ` Bart Alewijnse
2004-07-31 21:18 ` Francois Romieu
2004-08-01 19:03 ` Bart Alewijnse
2004-08-03 2:47 ` Bart Alewijnse
2004-08-03 7:48 ` Francois Romieu
2004-08-03 12:32 ` Bart Alewijnse
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).