Slow reconstruction

Slow reconstruction

[John Rowe]

First, thanks to all the authors of software RAID, which works really
well. 

I have just one problem(!) and as disks get bigger it gets worse:
reconstruction takes a long time. At the moment I've just had a disk
fail on a 5x250 GB RAID5 array and it's going to take eight days to
recover. Is there anything I can do to speed things up?

FWIW, each disk is the master drive on an IDE card, two of the disks are
shared with the RAID1 root partition and one of the 5 devices is a RAID0
stripe. The system isn't doing anything else, it's just used for backup.

Thanks again.

John

Re: Slow reconstruction

[Mike Hardy]

John Rowe wrote:
> First, thanks to all the authors of software RAID, which works really
> well. 
> 
> I have just one problem(!) and as disks get bigger it gets worse:
> reconstruction takes a long time. At the moment I've just had a disk
> fail on a 5x250 GB RAID5 array and it's going to take eight days to
> recover. Is there anything I can do to speed things up?
> 
> FWIW, each disk is the master drive on an IDE card, two of the disks are
> shared with the RAID1 root partition and one of the 5 devices is a RAID0
> stripe. The system isn't doing anything else, it's just used for backup.

That is an astonishingly slow reconstruction time.

Have you looked at hdparm informational output on each of the drives?
dmesg, grepping for the hard drive initialization information?

With speeds like that, I suspect you don't have DMA enabled for some
reason. Maybe someone else has seen something more exactly matching
this, but when speeds are this off, that's what its been for me in the past.

Running hdparm -Tt /dev/hdXX can tell you what the disks can do
individually, I'd start there, and look at the hdparm -i (or -I?) output
to see what they were set at, and start tuning the interface.

Hopefully that turns something up quickly.

-Mike

Re: Slow reconstruction

[John Rowe]

Thanks for all the suggestions. 

/proc/sys/dev/raid/speed_limit_max had the value "10000", changing it to
200000 didn't make a difference. hdparm looks OK to me:

/dev/hdm1:
 Timing buffer-cache reads:   128 MB in  1.19 seconds =107.56 MB/sec
 Timing buffered disk reads:  64 MB in  3.13 seconds = 20.45 MB/sec

/dev/hdo1:
 Timing buffer-cache reads:   128 MB in  1.17 seconds =109.40 MB/sec
 Timing buffered disk reads:  64 MB in  3.85 seconds = 16.62 MB/sec
rsync:/root# hdparm -Tt /dev/hda4

/dev/hda4:
 Timing buffer-cache reads:   128 MB in  1.15 seconds =111.30 MB/sec
 Timing buffered disk reads:  64 MB in  4.31 seconds = 14.85 MB/sec

/dev/hdc4:
 Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
 Timing buffered disk reads:  64 MB in  4.79 seconds = 13.36 MB/sec

/dev/hdi3:
 Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
 Timing buffered disk reads:  64 MB in  2.73 seconds = 23.44 MB/sec

/dev/hdk1:
 Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
 Timing buffered disk reads:  64 MB in  2.72 seconds = 23.53 MB/sec


DMA is enabled for all six disks.

If it helps, here is my raidtab. I wonder a little about the 64K chunk
size on the stripe but a 128K size on the RAID5. Is this good or bad?

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
    device          /dev/hda2
    raid-disk     0
    device          /dev/hdc2
    raid-disk     1


raiddev             /dev/md2
raid-level                  0
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
    device          /dev/hdm1
    raid-disk     0
    device          /dev/hdo1
    raid-disk     1

raiddev             /dev/md3
raid-level                  5
nr-raid-disks               5
chunk-size                  128k
parity-algorithm        left-symmetric
persistent-superblock       1
nr-spare-disks              0
    device          /dev/hda4
    raid-disk     0
    device          /dev/hdc4
    raid-disk     1
    device          /dev/hdi3
    raid-disk     2
    device          /dev/hdk1
    raid-disk     3
    device          /dev/md2
    raid-disk     4


Comments gratefully received.

John

Re: Slow reconstruction

[seth vidal]

On Wed, 2005-06-29 at 18:31 +0100, John Rowe wrote:
> Thanks for all the suggestions. 
> 
> /proc/sys/dev/raid/speed_limit_max had the value "10000", changing it to
> 200000 didn't make a difference. hdparm looks OK to me:
> 

did you raise speed_limit_min??

if you didn't then you're probably being hit by the 'only use idle io'
problem.

-sv

Re: Slow reconstruction

[John Rowe]

> 
> did you raise speed_limit_min??
> 
> if you didn't then you're probably being hit by the 'only use idle io'
> problem.

That's fixed it! It's now going along at around 5.7 MB/s which isn't
huge but is around 11 times what it was. Many thanks.

But I'm a bit surprised this makes a difference because there is no
other real IO going on.

John

Re: Slow reconstruction

[KELEMEN Peter]

* John Rowe (rowe@excc.ex.ac.uk) [20050629 18:31]:

> Timing buffered disk reads:  64 MB in  3.13 seconds = 20.45 MB/sec
> Timing buffered disk reads:  64 MB in  3.85 seconds = 16.62 MB/sec
> Timing buffered disk reads:  64 MB in  4.31 seconds = 14.85 MB/sec
> Timing buffered disk reads:  64 MB in  4.79 seconds = 13.36 MB/sec
> Timing buffered disk reads:  64 MB in  2.73 seconds = 23.44 MB/sec
> Timing buffered disk reads:  64 MB in  2.72 seconds = 23.53 MB/sec

These are very low values.  A modern 250G drive should exceed
50MB/sec.  What kind of IDE controller do you use?

Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \     Peter.Kelemen@cern.ch
.+'         `+...+'         `+...+'         `+...+'         `+...+'
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Slow reconstruction

[John Rowe]

> These are very low values.  A modern 250G drive should exceed
> 50MB/sec.  What kind of IDE controller do you use?

None of it is very modern - it's put together from what we had lying
around :-(

(Don't even start..) I'll see if it's any different when the array is
reconstructed.

John

Re: Slow reconstruction

[KELEMEN Peter]

* John Rowe (rowe@excc.ex.ac.uk) [20050629 18:52]:

> None of it is very modern - it's put together from what we had
> lying around :-(

It is most likely that the controllers (and/or respective drivers)
are the source of your problem.  I'd say the controllers are
UDMA(66) only.

Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \     Peter.Kelemen@cern.ch
.+'         `+...+'         `+...+'         `+...+'         `+...+'
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Slow reconstruction

[Tyler]

For 250GB drives, those Buffered disk reads seem rather slow.. they 
should be in the 35-50 range.  I would check your dma settings with 
hdparm as someone already suggested.  What speed of processor is in the 
server, and what type of IDE cards/controllers?  Maybe also check your 
/proc/interrupts for errors.. maybe you have an IRQ conflict.  Also 
double check your dmesg output.  What kernel are you using?

Tyler.

John Rowe wrote:

>Thanks for all the suggestions. 
>
>/proc/sys/dev/raid/speed_limit_max had the value "10000", changing it to
>200000 didn't make a difference. hdparm looks OK to me:
>
>/dev/hdm1:
> Timing buffer-cache reads:   128 MB in  1.19 seconds =107.56 MB/sec
> Timing buffered disk reads:  64 MB in  3.13 seconds = 20.45 MB/sec
>
>/dev/hdo1:
> Timing buffer-cache reads:   128 MB in  1.17 seconds =109.40 MB/sec
> Timing buffered disk reads:  64 MB in  3.85 seconds = 16.62 MB/sec
>rsync:/root# hdparm -Tt /dev/hda4
>
>/dev/hda4:
> Timing buffer-cache reads:   128 MB in  1.15 seconds =111.30 MB/sec
> Timing buffered disk reads:  64 MB in  4.31 seconds = 14.85 MB/sec
>
>/dev/hdc4:
> Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
> Timing buffered disk reads:  64 MB in  4.79 seconds = 13.36 MB/sec
>
>/dev/hdi3:
> Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
> Timing buffered disk reads:  64 MB in  2.73 seconds = 23.44 MB/sec
>
>/dev/hdk1:
> Timing buffer-cache reads:   128 MB in  1.16 seconds =110.34 MB/sec
> Timing buffered disk reads:  64 MB in  2.72 seconds = 23.53 MB/sec
>
>
>DMA is enabled for all six disks.
>
>If it helps, here is my raidtab. I wonder a little about the 64K chunk
>size on the stripe but a 128K size on the RAID5. Is this good or bad?
>
>raiddev             /dev/md0
>raid-level                  1
>nr-raid-disks               2
>chunk-size                  64k
>persistent-superblock       1
>nr-spare-disks              0
>    device          /dev/hda2
>    raid-disk     0
>    device          /dev/hdc2
>    raid-disk     1
>
>
>raiddev             /dev/md2
>raid-level                  0
>nr-raid-disks               2
>chunk-size                  64k
>persistent-superblock       1
>nr-spare-disks              0
>    device          /dev/hdm1
>    raid-disk     0
>    device          /dev/hdo1
>    raid-disk     1
>
>raiddev             /dev/md3
>raid-level                  5
>nr-raid-disks               5
>chunk-size                  128k
>parity-algorithm        left-symmetric
>persistent-superblock       1
>nr-spare-disks              0
>    device          /dev/hda4
>    raid-disk     0
>    device          /dev/hdc4
>    raid-disk     1
>    device          /dev/hdi3
>    raid-disk     2
>    device          /dev/hdk1
>    raid-disk     3
>    device          /dev/md2
>    raid-disk     4
>
>
>Comments gratefully received.
>
>John
>
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  
>

Re: Slow reconstruction

[seth vidal]

On Wed, 2005-06-29 at 17:10 +0100, John Rowe wrote:
> First, thanks to all the authors of software RAID, which works really
> well. 
> 
> I have just one problem(!) and as disks get bigger it gets worse:
> reconstruction takes a long time. At the moment I've just had a disk
> fail on a 5x250 GB RAID5 array and it's going to take eight days to
> recover. Is there anything I can do to speed things up?
> 
> FWIW, each disk is the master drive on an IDE card, two of the disks are
> shared with the RAID1 root partition and one of the 5 devices is a RAID0
> stripe. The system isn't doing anything else, it's just used for backup.
> 

you know about /proc/sys/dev/raid/speed_limit_max and speed_limit_min,
right?

echo 40000 > /proc/sys/dev/raid/speed_limit_max

see if it goes faster, then.

-sv