2.6.21.1 - 97% wait time on IDE operations

2.6.21.1 - 97% wait time on IDE operations

[Tommy Vercetti]

Hi folks, 

I was trying to get answer to my question around, but no one knows.
I do have DMA turned on, etc, yet - on extensive harddrive operations wait 
time is 90+% , which means that machine is waiting, rather than doing 
something meanwhile. (I guess). 
Can someone describe to me , in more detail why is that happening, and what 
steps should I consider to avoid it ? I couldn't find any answers that would 
have help me on net.

thanks.
-- 
Vercetti

Re: 2.6.21.1 - 97% wait time on IDE operations

[Paolo Ornati]

On Sat, 26 May 2007 15:05:38 +0200
Tommy Vercetti <vercetti@zlew.org> wrote:

> I was trying to get answer to my question around, but no one knows.
> I do have DMA turned on, etc, yet - on extensive harddrive operations wait 
> time is 90+% , which means that machine is waiting, rather than doing 
> something meanwhile. (I guess). 
> Can someone describe to me , in more detail why is that happening, and what 
> steps should I consider to avoid it ? I couldn't find any answers that would 
> have help me on net.

It means that the disk is slow and the CPU is fast... so while the disk
is busy seeking and reading data the CPU has nothing to do but wait for
it.

Idle == CPU has nothing to do
Waiting == CPU has nothing to do, but it will have as soon as the slow
	disk (or whatever) delivers data


Anyway 97% is quite high... what CPU / Hard Disk do you have?

What kernel version?
I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler)
Filesystem?

And what time of "operations" are you doing?

-- 
	Paolo Ornati
	Linux 2.6.22-rc3-cfs-v14-gf193016a on x86_64

Re: 2.6.21.1 - 97% wait time on IDE operations

[Tommy Vercetti]

On Saturday 26 May 2007 15:52, Paolo Ornati wrote:
> On Sat, 26 May 2007 15:05:38 +0200
> It means that the disk is slow and the CPU is fast... so while the disk
> is busy seeking and reading data the CPU has nothing to do but wait for
> it.
>
> Idle == CPU has nothing to do
> Waiting == CPU has nothing to do, but it will have as soon as the slow
> 	disk (or whatever) delivers data
mhm.

> Anyway 97% is quite high... what CPU / Hard Disk do you have?
Intel(R) Pentium(R) M processor 1400MHz
TOSHIBA MK8025GAS

> What kernel version?
2.6.21.1
> I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler)
gj@puppet:~$ cat /sys/block/hda/queue/scheduler
noop anticipatory deadline [cfq]

> Filesystem?
reiser3
> And what time of "operations" are you doing?
apt-get install, vmware

-- 
Vercetti

Re: 2.6.21.1 - 97% wait time on IDE operations

[Paolo Ornati]

On Sat, 26 May 2007 16:07:49 +0200
Tommy Vercetti <vercetti@zlew.org> wrote:

> > Anyway 97% is quite high... what CPU / Hard Disk do you have?  
> Intel(R) Pentium(R) M processor 1400MHz
> TOSHIBA MK8025GAS
> 
> > What kernel version?  
> 2.6.21.1
> > I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler)  
> gj@puppet:~$ cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
> 
> > Filesystem?  
> reiser3
> > And what time of "operations" are you doing?  
> apt-get install, vmware

It's probably just a slow disk... try hdparm as suggested by Ray and
see if they look sane (PS: just seen the numbers and looks sane to me).

If they are and you need more performance buy a better HD :)

Improvements from kernel side are possible but don't expect too much.


Things to check/try:

1) FS: is it in good shape? Or is it full and fragmented?

2) FS mount options, try adding everything that can reduce accesses
(noatime, nodiratime)

3) try experimental patches such as Adaptive Readahead, it seems to
help some workload (it was there in older -mm, now there's a much
simplier readahead replacement in 2.6.22-rc2-mm1 called "on-demand
readahead", maybe that helps too?)

-- 
	Paolo Ornati
	Linux 2.6.22-rc3-cfs-v14-gf193016a on x86_64

Re: 2.6.21.1 - 97% wait time on IDE operations

[Julio M. Merino Vidal]

On 26/05/2007, at 15:05, Tommy Vercetti wrote:

> Hi folks,
>
> I was trying to get answer to my question around, but no one knows.
> I do have DMA turned on, etc, yet - on extensive harddrive  
> operations wait
> time is 90+% , which means that machine is waiting, rather than doing
> something meanwhile. (I guess).
> Can someone describe to me , in more detail why is that happening,  
> and what
> steps should I consider to avoid it ? I couldn't find any answers  
> that would
> have help me on net.

I see in your other post that you have a laptop disk drive and, based  
on its specs, it is a 4200RPM one.  Laptop drives are incredibly slow  
for random disk access -- I have one and hate it with passion... even  
though it is slightly better (5400RPM)...

You mention that you are doing expensive disk operations.  If these  
involve a lot of disk seeks (such as, for example, a "cvs update" on  
a large source tree), disk performance will be very very poor and any  
other operation you attempt on it will be slow as well.  These kind  
of drives are the major bottleneck of current laptop machines, in my  
opinion.

(I just wanted to make this clear wrt the hardware.  I do not know if  
Linux is also involved or not.)

-- 
Julio M. Merino Vidal <jmerino@ac.upc.edu>

Re: 2.6.21.1 - 97% wait time on IDE operations

[Ray Lee]

On 5/26/07, Tommy Vercetti <vercetti@zlew.org> wrote:
> I was trying to get answer to my question around, but no one knows.
> I do have DMA turned on, etc, yet - on extensive harddrive operations wait

You may have DMA turned on, but it may be ineffectual if you don't
have the right IDE driver handing your disk.

> time is 90+% , which means that machine is waiting, rather than doing
> something meanwhile. (I guess).

What does hdparm -tT /dev/hda say? If the numbers are reasonable
(~400MB/s for cache reads, ~25MB/s for buffered disk reads), then
you're just doing a lot of seeking on the drive, and that takes a long
time. (Drives can seek about 100 times a second. If every seek reads
only 4k of data, that's 400k of data processed per second, or a lot of
time spent with the CPU waiting on I/O.)

Re: 2.6.21.1 - 97% wait time on IDE operations

[Tommy Vercetti]

in terms of driver, what do you mean ?
this is my lspci:
00:00.0 Host bridge: Intel Corporation 82855PM Processor to I/O Controller 
(rev 21)
00:01.0 PCI bridge: Intel Corporation 82855PM Processor to AGP Controller (rev 
21)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) 
USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) 
USB UHCI Controller #2 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI 
Controller (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 83)
00:1f.0 ISA bridge: Intel Corporation 82801DBM (ICH4-M) LPC Interface Bridge 
(rev 03)
00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller (rev 
03)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM 
(ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 03)
00:1f.6 Modem: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 
Modem Controller (rev 03)
01:00.0 VGA compatible controller: nVidia Corporation NV34M [GeForce FX 
Go5200] (rev a1)
02:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 
Controller (PHY/Link)
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (MOB) 
Ethernet Controller (rev 83)
02:0a.0 Network controller: Intel Corporation PRO/Wireless LAN 2100 3B Mini 
PCI Adapter (rev 04)
02:0b.0 CardBus bridge: Toshiba America Info Systems ToPIC100 PCI to Cardbus 
Bridge with ZV Support (rev 33)
02:0d.0 System peripheral: Toshiba America Info Systems SD TypA Controller 
(rev 05)
03:00.0 Ethernet controller: Atheros Communications, Inc. AR5212 802.11abg NIC 
(rev 01)


modules list, and config on request (don't want to pollute devs mailboxes).

/dev/hda:
 Timing cached reads:   732 MB in  2.00 seconds = 365.74 MB/sec
 Timing buffered disk reads:   76 MB in  3.00 seconds =  25.31 MB/sec
root@puppet:~# htparm -i -v /dev/hda
-su: htparm: command not found
root@puppet:~# hdparm -i -v /dev/hda

/dev/hda:
 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 65535/16/63, sectors = 156301488, start = 0

 Model=TOSHIBA MK8025GAS, FwRev=KA023A, SerialNo=95NX6153S
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=48
 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=16
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=156301488
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  sdma0 sdma1 sdma2 mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma3 udma4 *udma5
 AdvancedPM=yes: unknown setting WriteCache=enabled
 Drive conforms to: Unspecified:  ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 
ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6

 * signifies the current active mode




-- 
Vercetti

Re: 2.6.21.1 - 97% wait time on IDE operations

[Ray Lee]

On 5/26/07, Tommy Vercetti <vercetti@zlew.org> wrote:
> in terms of driver, what do you mean ?

The software driver that handles the specific chipset for that drive.
The thing that shows up when you do an lsmod, assuming you have the
driver compiled as a module.

> /dev/hda:
>  Timing cached reads:   732 MB in  2.00 seconds = 365.74 MB/sec
>  Timing buffered disk reads:   76 MB in  3.00 seconds =  25.31 MB/sec

That looks fine. Your filesystem may just be very fragmented. I'm not
familiar with how reiser3 filesystems age, so if you've had the system
installed for quite some time, it may be time to back it up and
reinstall. Dunno.

All of Paolo's suggestions look good, BTW. Sorry I couldn't be more help.

Re: 2.6.21.1 - 97% wait time on IDE operations

[Tommy Vercetti]

On Saturday 26 May 2007 18:07, Ray Lee wrote:
> On 5/26/07, Tommy Vercetti <vercetti@zlew.org> wrote:
> > in terms of driver, what do you mean ?
>
> The software driver that handles the specific chipset for that drive.
> The thing that shows up when you do an lsmod, assuming you have the
> driver compiled as a module.

root@puppet:~# dmesg |grep -i ide
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
    ide0: BM-DMA at 0x1840-0x1847, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x1848-0x184f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
ide1 at 0x170-0x177,0x376 on irq 15

it's compiled in. 
you reckon I should use something else for this platform ? 

> > /dev/hda:
> >  Timing cached reads:   732 MB in  2.00 seconds = 365.74 MB/sec
> >  Timing buffered disk reads:   76 MB in  3.00 seconds =  25.31 MB/sec
>
> That looks fine. Your filesystem may just be very fragmented. I'm not
> familiar with how reiser3 filesystems age, so if you've had the system
> installed for quite some time, it may be time to back it up and
> reinstall. Dunno.
yeah, and they say unix FSes don't need defragmentation.... my bottom...

-- 
Vercetti

Re: 2.6.21.1 - 97% wait time on IDE operations

[Alan Cox]

> > That looks fine. Your filesystem may just be very fragmented. I'm not
> > familiar with how reiser3 filesystems age, so if you've had the system
> > installed for quite some time, it may be time to back it up and
> > reinstall. Dunno.
> yeah, and they say unix FSes don't need defragmentation.... my bottom...

Provided they don't get within about 5-10% of full a lot of the time Unix
file systems generally don't  - especially with preallocators. Reiser3 is
not a traditional unix file system, its very different to almost any
other fs, and it does need defragmenting.

Re: 2.6.21.1 - 97% wait time on IDE operations

[Clemens Koller]

Hi, Alan!

Alan Cox schrieb:
>> yeah, and they say unix FSes don't need defragmentation.... my bottom...
> 
> Provided they don't get within about 5-10% of full a lot of the time Unix
> file systems generally don't  - especially with preallocators. Reiser3 is
> not a traditional unix file system, its very different to almost any
> other fs, and it does need defragmenting.

Just FYI:
http://ck.kolivas.org/apps/defrag/
http://sourceforge.net/projects/davtools/
defragmentation seems to be a minor issue.

After reiser3 (kernel 2.6.18.1 or so) crashed(*) badly with
some data loss on a production server can you recommed
a _safe_ and good performing filesystem for an
compile and test server (lots of small files).

(*) rm ~/Maildir while it was fed with courier-imap and
pressed CTRL-C to interrupt that trashed some of the files in
~/Maildir. It needed a --rebuild-tree.


Best greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm-technology.com
Phone: +49-89-741518-50
Fax: +49-89-741518-19

Re: 2.6.21.1 - 97% wait time on IDE operations

[Benny Amorsen]

>>>>> "AC" == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

AC> Provided they don't get within about 5-10% of full a lot of the
AC> time Unix file systems generally don't

That's a big if right there. For servers it isn't a problem, few
people can get capacity right to withing 10%, so you never let a
server run full. Desktops/laptops on the other hand spend most of
their lives between 80% and 100%.


/Benny

Re: 2.6.21.1 - 97% wait time on IDE operations

[Alan Cox]

O> That's a big if right there. For servers it isn't a problem, few
> people can get capacity right to withing 10%, so you never let a
> server run full. Desktops/laptops on the other hand spend most of
> their lives between 80% and 100%.

Modern desktop patterns are very different to older ones - the disk fill
is almost entirely continuous writes of large files (OGG, MP3, Movies
etc). In addition the default ext3 behaviour reserves the last part of
the disk for root - so usually it doesn't get below 5% free as only root
can steal that space.

Back when you had a 40MB /home the usage tended to be multiple parallel
writers, permanently at 90%+ (with a "please remove unused files" motd).

Alan

Re: 2.6.21.1 - 97% wait time on IDE operations

[Guillaume Chazarain]

Hi,

> wait time is 90+% , which means that machine is waiting, rather than doing
> something meanwhile. (I guess).

Yes, the machine spends its time waiting for the disk to do its things
as it does not have anything else to do. Nothing to worry about. If
you want, launch some CPU bound process at the same time and the wait
time will be replaced by some user+system time but you'll gain
nothing.

Hope this helps.

-- 
Guillaume

Re: 2.6.21.1 - 97% wait time on IDE operations

[Bill Davidsen]

Tommy Vercetti wrote:
> Hi folks, 
> 
> I was trying to get answer to my question around, but no one knows.
> I do have DMA turned on, etc, yet - on extensive harddrive operations wait 
> time is 90+% , which means that machine is waiting, rather than doing 
> something meanwhile. (I guess). 
> Can someone describe to me , in more detail why is that happening, and what 
> steps should I consider to avoid it ? I couldn't find any answers that would 
> have help me on net.
> 
> thanks.

 From later posts I suspect that your disk performance just sucks, but 
do  use one of the monitoring tools and follow the actual disk work, in 
term of seeks and transfer rate.

I've been looking at high iowait while disk and network are (nearly) 
idle, but it sounds as if you just have a bad match of CPU and disk speed.

Can you borrow a USB drive to use for some testing? USB 2 needed to be 
faster than your old 4200 rpm drive.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot