slow performance on disk/network i/o full speed after drop_caches

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Dear list,

i hope this is the correct list to write to if it would be nice to give 
me a hint where i can ask.

Kernel: 2.6.38

I'm seeing some strange problems on some of our servers after upgrading 
to 2.6.38.

I'm copying a 1GB file via scp from Machine A to Machine B. When B is 
freshly booted the file transfer is done with about 80 to 85 Mb/s. I can 
repeat that various times to performance degrease.

Then after some days copying is only done with about 900kb/s up to 3Mb/s 
going up and down while transfering the file.

When i then do drop_caches it works again on 80Mb/s.

sync && echo 3 >/proc/sys/vm/drop_caches && sleep 2 && echo 0 
 >/proc/sys/vm/drop_caches

Attached is also an output of meminfo before and after drop_caches.

What's going on here? MemFree is pretty high.

Please CC me i'm not on list.

Greets Stefan

# before drop_caches

# cat /proc/meminfo
MemTotal:        8185544 kB
MemFree:         6670292 kB
Buffers:          105164 kB
Cached:           166672 kB
SwapCached:            0 kB
Active:           728308 kB
Inactive:         567428 kB
Active(anon):     639204 kB
Inactive(anon):   394932 kB
Active(file):      89104 kB
Inactive(file):   172496 kB
Unevictable:        2976 kB
Mlocked:            2992 kB
SwapTotal:       1464316 kB
SwapFree:        1464316 kB
Dirty:                52 kB
Writeback:             0 kB
AnonPages:       1026920 kB
Mapped:            54208 kB
Shmem:              8380 kB
Slab:              80724 kB
SReclaimable:      22844 kB
SUnreclaim:        57880 kB
KernelStack:        2872 kB
PageTables:        35448 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5557088 kB
Committed_AS:    6187972 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      292360 kB
VmallocChunk:   34359425327 kB
HardwareCorrupted:     0 kB
DirectMap4k:        5632 kB
DirectMap2M:     2082816 kB
DirectMap1G:     6291456 kB

# cat /proc/meminfo
MemTotal:        8185544 kB
MemFree:         6888060 kB
Buffers:             372 kB
Cached:            61492 kB
SwapCached:            0 kB
Active:           659156 kB
Inactive:         426664 kB
Active(anon):     638892 kB
Inactive(anon):   395200 kB
Active(file):      20264 kB
Inactive(file):    31464 kB
Unevictable:        2976 kB
Mlocked:            2992 kB
SwapTotal:       1464316 kB
SwapFree:        1464316 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       1026952 kB
Mapped:            54236 kB
Shmem:              8316 kB
Slab:              70616 kB
SReclaimable:      12264 kB
SUnreclaim:        58352 kB
KernelStack:        2864 kB
PageTables:        35448 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5557088 kB
Committed_AS:    6187932 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      292360 kB
VmallocChunk:   34359425327 kB
HardwareCorrupted:     0 kB
DirectMap4k:        5632 kB
DirectMap2M:     2082816 kB
DirectMap1G:     6291456 kB