high cpu load with sw raid1

high cpu load with sw raid1

[David Hajek]

Hi,

I have linux box with 70GB SW Raid1. This box runs for half
a year without problems but now I meet the high cpu load 
problems. I suspect that it can be caused by not enough 
free disk space on this md device. I see following:

1 GB free  - load > 5
5 GB free  - load < 1

I have to notice that this box is rather under heavy load
(1 GB cvs tree, nfs homes etc.) My question is whether this 
load can depend on available disk space because I do not
see any suspect processes that can cause such a high load.

Kernel: 2.2.19
Patches: lfs + md + ide
RH6.2 + glibc-2.2.12

Thanks.

--
David Hajek
hajek@idoox.com 
- Heisenberg may have slept here...

Re: high cpu load with sw raid1

[Jakob Østergaard]

On Thu, Sep 20, 2001 at 10:26:16AM +0200, David Hajek wrote:
> Hi,
> 
> I have linux box with 70GB SW Raid1. This box runs for half
> a year without problems but now I meet the high cpu load 
> problems. I suspect that it can be caused by not enough 
> free disk space on this md device. I see following:
> 
> 1 GB free  - load > 5
> 5 GB free  - load < 1

RAID does not know about "free space" (that is a filesystem thing), so that
would be either some strange interaction between the filesystem and lower
layers, or a measurement error - I guess.

High fragmentation could lead to extra filesystem activity, but that's not
really something the RAID can influence.

> 
> I have to notice that this box is rather under heavy load
> (1 GB cvs tree, nfs homes etc.) My question is whether this 
> load can depend on available disk space because I do not
> see any suspect processes that can cause such a high load.
> 
> Kernel: 2.2.19
> Patches: lfs + md + ide
> RH6.2 + glibc-2.2.12

Please check that your disks are using DMA (hdparm /dev/hdX). You should
see something like:

[root@eagle /root]# hdparm /dev/hda

/dev/hda:
 multcount    = 16 (on)
 I/O support  =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)                <=========== INDICATES DMA
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 782/128/63, sectors = 6306048, start = 0

Without DMA you will see high CPU load from accessing the disks, regardless of
free space.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: high cpu load with sw raid1

[David Hajek]

Inlined.

On Thu, Sep 20, 2001, Jakob Řstergaard wrote:
> On Thu, Sep 20, 2001 at 10:26:16AM +0200, David Hajek wrote:
> > Hi,
> > 
> > I have linux box with 70GB SW Raid1. This box runs for half
> > a year without problems but now I meet the high cpu load 
> > problems. I suspect that it can be caused by not enough 
> > free disk space on this md device. I see following:
> > 
> > 1 GB free  - load > 5
> > 5 GB free  - load < 1
> 
> RAID does not know about "free space" (that is a filesystem thing), so that
> would be either some strange interaction between the filesystem and lower
> layers, or a measurement error - I guess.
> 
> High fragmentation could lead to extra filesystem activity, but that's not
> really something the RAID can influence.

Why not? I do not say that it is cause be the RAID. It can be caused
by filesystem. Do you think that fsck helps to defrag filesystem?

> 
> Please check that your disks are using DMA (hdparm /dev/hdX). You should
> see something like:
> 
> [root@eagle /root]# hdparm /dev/hda
> 
> /dev/hda:
>  multcount    = 16 (on)
>  I/O support  =  0 (default 16-bit)
>  unmaskirq    =  0 (off)
>  using_dma    =  1 (on)                <=========== INDICATES DMA
>  keepsettings =  0 (off)
>  nowerr       =  0 (off)
>  readonly     =  0 (off)
>  readahead    =  8 (on)
>  geometry     = 782/128/63, sectors = 6306048, start = 0
> 
> Without DMA you will see high CPU load from accessing the disks, regardless of
> free space.

I use SCSI disks for this raid. Here is the output:

[root@marvin /root]# hdparm /dev/sda

/dev/sda:
 readonly     =  0 (off)
 geometry     = 8924/255/63, sectors = 143374738, start = 0
[root@marvin /root]# hdparm -t /dev/sda

  /dev/sda:
   Timing buffered disk reads:  64 MB in  1.87 seconds = 34.22 MB/sec
[root@marvin /root]#   

I think disk speeds are ok - it works for really fine for half a year.

-- 
David Hajek
hajek@idoox.com                	     GSM: +420 604 352968
- You can't cheat the phone company.

Re: high cpu load with sw raid1

[Jakob Østergaard]

On Thu, Sep 20, 2001 at 11:23:42AM +0200, David Hajek wrote:
> Inlined.
> 
> On Thu, Sep 20, 2001, Jakob Rstergaard wrote:
> > On Thu, Sep 20, 2001 at 10:26:16AM +0200, David Hajek wrote:
> > > Hi,
> > > 
> > > I have linux box with 70GB SW Raid1. This box runs for half
> > > a year without problems but now I meet the high cpu load 
> > > problems. I suspect that it can be caused by not enough 
> > > free disk space on this md device. I see following:
> > > 
> > > 1 GB free  - load > 5
> > > 5 GB free  - load < 1
> > 
> > RAID does not know about "free space" (that is a filesystem thing), so that
> > would be either some strange interaction between the filesystem and lower
> > layers, or a measurement error - I guess.
> > 
> > High fragmentation could lead to extra filesystem activity, but that's not
> > really something the RAID can influence.
> 
> Why not? I do not say that it is cause be the RAID. It can be caused
> by filesystem. Do you think that fsck helps to defrag filesystem?

Ok, I was just trying to rule out the RAID.

I know for a fact that fsck does not defrag the filesystem.  However, if you
do run fsck, it will tell you how much is fragmented.

A few percent shouldn't matter.  But see what the fragmentation is when you
have only one gig available, compared to when you have five.

...
> 
> I use SCSI disks for this raid. Here is the output:
> 
> [root@marvin /root]# hdparm /dev/sda
> 
> /dev/sda:
>  readonly     =  0 (off)
>  geometry     = 8924/255/63, sectors = 143374738, start = 0
> [root@marvin /root]# hdparm -t /dev/sda
> 
>   /dev/sda:
>    Timing buffered disk reads:  64 MB in  1.87 seconds = 34.22 MB/sec
> [root@marvin /root]#   

Oh, silly me.  I'm mixing up mails here...  Sure, with SCSI you'll be fine.

> 
> I think disk speeds are ok - it works for really fine for half a year.

Yep.

If everything is slowed down by disk seeks due to high fragmentation, you
should actually be able to hear your disks seek like mad when the load is high.
Throughput doesn't matter much in that case - have you noticed the sound of
disk-heads (sonic booms, heads catching on fire, etc.)  ?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: high cpu load with sw raid1

[Andreas Dilger]

On Sep 20, 2001  10:26 +0200, David Hajek wrote:
> I have linux box with 70GB SW Raid1. This box runs for half
> a year without problems but now I meet the high cpu load 
> problems. I suspect that it can be caused by not enough 
> free disk space on this md device. I see following:
> 
> 1 GB free  - load > 5
> 5 GB free  - load < 1

What filesystem are you using?  If it is reiserfs, and you have < 10%
of the disk free, it is very unhappy.  A patch to fix this is available.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: high cpu load with sw raid1

[David Hajek]

On Thu, Sep 20, 2001, Andreas Dilger wrote:
> On Sep 20, 2001  10:26 +0200, David Hajek wrote:
> > I have linux box with 70GB SW Raid1. This box runs for half
> > a year without problems but now I meet the high cpu load 
> > problems. I suspect that it can be caused by not enough 
> > free disk space on this md device. I see following:
> > 
> > 1 GB free  - load > 5
> > 5 GB free  - load < 1
> 
> What filesystem are you using?  If it is reiserfs, and you have < 10%
> of the disk free, it is very unhappy.  A patch to fix this is available.
> 

I'm using ext2. I suspect high ext2 fragmentation, because when
there are 'only' 1GB free the disk is _really_ busy. I doubt
that it takes lot of time to find free blocks. 

-- 
David Hajek
hajek@idoox.com                	     GSM: +420 604 352968
- VMS is like a nightmare about RXS-11M.

Re: high cpu load with sw raid1

[Andreas Dilger]

On Sep 21, 2001  09:07 +0200, David Hajek wrote:
> On Thu, Sep 20, 2001, Andreas Dilger wrote:
> > On Sep 20, 2001  10:26 +0200, David Hajek wrote:
> > > I have linux box with 70GB SW Raid1. This box runs for half
> > > a year without problems but now I meet the high cpu load 
> > > problems. I suspect that it can be caused by not enough 
> > > free disk space on this md device. I see following:
> > > 
> > > 1 GB free  - load > 5
> > > 5 GB free  - load < 1
> > 
> > What filesystem are you using?  If it is reiserfs, and you have < 10%
> > of the disk free, it is very unhappy.  A patch to fix this is available.
> 
> I'm using ext2. I suspect high ext2 fragmentation, because when
> there are 'only' 1GB free the disk is _really_ busy. I doubt
> that it takes lot of time to find free blocks. 

OK, I just re-read your initial posting, and see you have a 70GB RAID,
so 1GB free is about 1.4% free, which makes for bad performance no
matter what filesystem you have.  In general, ext2 will have this 1%
free space spread evenly across all of the disk, so while 1GB is still
a lot of space, it is still a nearly full filesystem.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert