Re: netatalk slow after system upgrade (possibly kernel problem?)

Re: netatalk slow after system upgrade (possibly kernel problem?)

[Andrew Morton]

(cc netdev)

> On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie <michael.monnerie@it-management.at> wrote:
> Dear lists,
> 
> I've been spending a LOT of time trying to find out where's the problem, 
> but can't find it and therefore seek urgent help now. We have the 
> following system:
> 
> Server with VMware server
> -> VM running a webserver and netatalk
> -> 2 other VMs not related
> 
> The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from 
> SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 
> 10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since 
> this upgrade, Apple clients (MacOS X) now do READ very very slowly 
> (about 512KB/s over the gigabit LAN), while writing to the server still 
> is normal (>20MB/s). I've even retried with the newest kernel 
> 2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control 
> (cubic, reno, bic, etc.) and nothing helps. I've then tried to install 
> Samba and found that we have similar problems reading with it from 
> MacOS clients. Now I'm pretty sure it should be something with the 
> linux kernel, but I don't understand what.
> 
> Here are the wireshark dumps in pcap format:
> http://zmi.at/x/atalk-write-fast.pcap
> -> you can see writing to the server (192.168.120.9) is normal and fast
> 
> http://zmi.at/x/atalk-read-slow.pcap
> -> reading is horribly slow. Lots of "unknown", because of netatalk or 
> what?
> 
> http://zmi.at/x/unknown-atalk.pcap
> -> another dump while reading, you see "unknown" reads. I'm not sure if 
> it's just wireshark not understanding the packets or netatalk.
> 
> And trying with samba:
> http://zmi.at/x/smb-read-slow.pcap
> http://zmi.at/x/smb-write-quick.pcap
> you can see that it's also slow.
> 
> Now why did it work with the old 2.6.13 kernel? I still have that old 
> VM, and when I start it, it is always perfectly fast. Only newer 
> versions are slow. Can somebody give me a hint please?

It would be interesting if this could be repeated on bare hardware, so we
can eliminate the possibility that it is some weird interaction with vmware.

Re: netatalk slow after system upgrade (possibly kernel problem?)

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 3792 bytes --]

(I let full quote here for reference, my text is on the bottom)

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.

I tested now on the same machine, outside of VMware: It is blazing fast, 
just as it should be - kernel release 2.6.18.8-0.5-ZMI, where -ZMI 
means I compiled the kernel myself. Still, it runs quick within the VM 
with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more recent 
kernels (I couldn't test every combination of course). 

I tried but cannot compile 2.6.13 within SUSE 10.3, it gives lots of 
errors because libs changed.

It would be very nice to find the source of the problem. I uploaded the 
kernel config options files onto my server for reference, HTH:

VMware with SUSE 10.0 and fast netatalk/samba:
http://zmi.at/x/config2.6.13-15.15.txt

VMware host, outside of VMware, also fast:
http://zmi.at/x/config2.6.18.8-host.txt

VMware with SUSE 10.2 and kernel 2.6.22-9-ZMI, slow:
http://zmi.at/x/config2.6.22.9-slow.txt

VMware with SUSE 10.3 and kernel 2.6.23.13-ZMI, slow:
http://zmi.at/x/config2.6.23.13-slow.txt

The problem seems to be somewhere between the kernel and VMware, but I 
do not know what to do.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 176 bytes --]

_______________________________________________
Netatalk-admins mailing list
Netatalk-admins@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/netatalk-admins

Re: [Netatalk-admins] netatalk slow after system upgrade (possibly kernel problem?)

[Michael Monnerie]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

I received this e-mail, which could be the resolution to this problem. I 
will test it with the customer tomorrow.

Is there a workaround possible from the Linux server, instead of 
modifying all clients? As it runs with kernel 2.6.13, but not with 
2.6.23, maybe some behaviour change via sysctl could help here.

On Mittwoch, 6. Februar 2008 Ralph Böhme wrote:
> Am 05.02.2008 um 16:26 schrieb Michael Monnerie:
> > Still, it runs quick within the VM
> > with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more
> > recent kernels (I couldn't test every combination of course).
>
> Use the sniffer, Luke!
>
> If you find an ongoing sequence of ~5 packets sent by the server,
> ~200ms pause, client ACK, next turn, then you might me experiencing
> this:
>
> http://www.helios.de/support/ti.phtml?110
>
> Proposed solution:
> sysctl -w net.inet.tcp.delayed_ack=2 to be set on the client side.
> Note: the issue should be resolved with OS X 10.5.
>
> It seems as if /some/ server side TCP stacks don't like OS X 10.4
> delayed ACKs implementation.
>
> HTH,
> -Ralph



-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: netatalk slow after system upgrade (possibly kernel problem?)

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 3400 bytes --]

(full quote below, because this message was in January)

I did some new tests, see http://zmi.at/x/wire.delayack2.pcap
Dump from SUSE10.3, Apple client had delayed ACK set to 2 with
sysctl -w net.inet.tcp.delayed_ack=2, and also MacOS10.5 has this 
problem. It was suggested I could have this problem:
http://www.helios.de/support/ti.phtml?110

On the same hardware, I can start the old SUSE10.0 at the same time in 
parallel with the new SUSE10.3, both are virtual machines in VMware, 
and the old machine is always fast and the new one always slow. From 
the dump I would say it's the fault of MacOS - sometimes a very long 
delay before an ACK. But then again, it's fast when copying to the old 
SUSE linux box. Very strange.

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.



-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 176 bytes --]

_______________________________________________
Netatalk-admins mailing list
Netatalk-admins@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/netatalk-admins