Peter Zijlstra wrote:
> On Mon, 2008-08-25 at 13:22 +0300, Török Edwin wrote:
>
>   
>> Well, the real program (clamd) that this testprogram tries to simulate 
>> does an mmap for almost every file, and I have lots of small files.
>> 6.5G, 114122 files, average size 57k.
>>
>> I'll run latencytop again, last time it has showed 100ms - 500ms latency 
>>     

Latencytop output attached.
There is 4 - 60 ms latency for mmap/munmap, and the more threads there 
are the total latency gets higher (latencytop says sum was ~480ms).

Running with MaxThreads 4 gets me 300-400% CPU usage, but with 
MaxThreads 8 CPU usage drops to around 120-250%.
Now, maxthreads 4 looks like a good choice from a CPU usage point of 
view, but is actually bad because it means that threads gets stuck in 
iowait, and the CPU won't have anything to do. MaxThreads 8 looked like 
a good alternative to fill the iowait gaps, but we run into the mmap_sem 
issue.
In a real world environment MaxThreads influences how many mails you can 
process in parallel with your MTA, so generally it should be as high as 
possible.

On 2.6.27-rc4:

MaxThreads 4 time, empty database (all cached, almost no I/O):
1m9s

MaxThreads 4 time, after echo 3>/proc/sys/vm/drop_caches:
1m29s

MaxThreads 8 time, empty database (all cached, almost no I/O):
2m16s

MaxThreads 8 time, after echo 3>/proc/sys/vm/drop_caches:
2m15s


Of course running with a full database will give different results, so 
I'll do some timing with that too (will take a little longer though).

>> for clamd, and it was about mmap, I'll provide you with the exact output.
>>     
>
> Right - does it make sense to teach clamav about pread() ?

If it is preferred over mmap, then maybe yes.

Peter Zijlstra wrote:
> OK, I'll poke a little more at is later today to see if I can spot
> something

Thanks!

Best regards,
--Edwin