Re: high system cpu load during intense disk i/o

[Dimitrios Apostolou <jimis@gmx.net>]

Rafał Bilski wrote:
>> Hello and thanks for your reply. 
> Hello again,
>> The cron job that is running every 10 min on my system is mpop (a 
>> fetchmail-like program) and another running every 5 min is mrtg. Both 
>> normally finish within 1-2 seconds.
>> The fact that these simple cron jobs don't finish ever is certainly 
>> because of the high system CPU load. If you see the two_discs_bad.txt 
>> which I attached on my original message, you'll see that *vmlinux*, 
>> and specifically the *scheduler*, take up most time.
>> And the fact that this happens only when running two i/o processes but 
>> when running only one everything is absolutely snappy (not at all 
>> slow, see one_disc.txt), makes me sure that this is a kernel bug. I'd 
>> be happy to help but I need some guidance to pinpoint the problem. 
> In Your oprofile output I find "acpi_pm_read" particulary interesting. 
> Unlike other VIA chipsets, which I know, Your doesn't use VLink to 
> connect northbridge to southbridge. Instead PCI bus connects these two. 
> As You probably know maximal PCI throughtput is 133MiB/s. In theory. In 
> practice probably less.
> ACPI registers are located on southbridge. This probably means that 
> processor needs access to PCI bus in order to read ACPI timer register.
> Now some math. 20GiB disk probably can send data at 20MiB/s rate. 200GiB 
> disk probably about 40MiB/s. So 20+2*40=100MiB/s. I think that this 
> could explain why simple inl() call takes so much time and why Your 
> system isn't very responsive.
>> Thanks, Dimitris
> Let me know if You find my theory amazing or amusing.

Hello Rafal,

I find your theory very nice, but unfortunately I don't think it applies 
here. As you can see from the vmstat outputs the write throughput is 
about 15MB/s for both disks. When reading I get about 30MB/s again from 
both disks. The other disk, the small one, is mostly idle, except for 
writing little bits and bytes now and then. Since the problem occurs 
when writing, 15MB/s is just too little I think for the PCI bus.

However I find it quite possible to have reached the throughput limit 
because of software (driver) problems. I have done various testing 
(mostly "hdparm -tT" with exactly the same PC and disks since about 
kernel 2.6.8 (maybe even earlier). I remember with certainty that read 
throughput the early days was about 50MB/s for each of the big disks, 
and combined with RAID 0 I got ~75MB/s. Those figures have been dropping 
gradually with each new kernel release and the situation today, with 
2.6.22, is that hdparm gives maximum throughput 20MB/s for each disk, 
and for RAID 0 too!

I have been ignoring these performance regressions because of no 
stability problems until now. So could it be that I'm reaching the 
20MB/s driver limit and some requests take too long to be served?


Dimitris