Re: Network performance - iperf

[Eric Dumazet <eric.dumazet@gmail.com>]

Le lundi 29 mars 2010 à 13:33 +0200, Michal Simek a écrit :
> Hi All,
> 
> I am doing several network benchmarks on Microblaze cpu with MMU.
> I am seeing one issue which is weird and I would like know where the 
> problem is.
> I am using the same hw design and the same Linux kernel. I have done 
> only change in memory size (in DTS).
> 
> 32MB: 18.3Mb/s
> 64MB: 15.2Mb/s
> 128MB: 10.6Mb/s
> 256MB: 3.8Mb/s
> 
> There is huge difference between systems with 32MB and 256MB ram.
> 
> I am running iperf TCP tests with these commands.
> On x86: iperf -c 192.168.0.105 -i 5 -t 50
> On microblaze: iperf -s
> 
> I look at pte misses which are the same on all configurations which 
> means that the number of do_page_fault exceptions is the same on all 
> configurations.
> I added some hooks to low level kernel code to be able to see number of 
> tlb misses. There is big differences between number of misses on system 
> with 256MB and 32MB. I measured two kernel settings. First column is 
> kernel with asm optimized memcpy/memmove function and the second is 
> without optimization. (Kernel with asm optimized lib functions is 30% 
> faster than system without optimization)
> 
> 32MB: 12703	13641
> 64MB: 1021750	655644
> 128MB: 1031644	531879
> 256MB: 1011322	430027
> 
> Most of them are data tlb misses. Microblaze MMU doesn't use any LRU 
> mechanism to find TLB victim that's why we there is naive TLB 
> replacement strategy based on incrementing counter. We using 2 tlbs for 
> kernel itself which are not updated that's why we can use "only" 62 TLBs 
> from 64.
> 

This probably has nothing to do with tcp stack, but trashing tlb on some
pathological cases (you have 62 entries, thats good for working size up
to 248 Kbytes, all included (program stack, program static & dynamic
data), given microblaze 4Kbytes page size.

You could try :

echo "4096 8192 32768" >/proc/sys/net/ipv4/tcp_rmem

to reduce memory footprint of iperf (or use iperf parameters)

Of course, I suppose kernel memory is 32 MB max, if you use only two
tlbs (16 Mbytes each) for kernel...