Re: Raid10 multi core scaling

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3732 bytes --]

On Mon, 02 Dec 2013 09:19:45 +0100 David Brown <david.brown@hesbynett.no>
wrote:

> On 02/12/13 07:22, NeilBrown wrote:
> > On Tue, 26 Nov 2013 10:58:59 +0000 Pedro Teixeira <finas@aeiou.pt> wrote:
> > 
> >>    I created a Raid10 array with 16 sata 1TB disks and the array
> >> performance
> >> seems to be limited by the md0_raid10 taking 99% of one core and not
> >> scalling to other cores. I tried overclocing the cpu cores and this lead to
> >> a small increase in performance ( but md0_raid10 keeps eating 99% of one
> >> core ).
> > 
> > Are you really talking about general array performance, or just resync
> > performance?
> > 
> > Because md0_raid10 doesn't do much work for normal IO so that should scale to
> > multiple processors.
> > I'm not sure that optimising resync to use more than one processor is really
> > much of a priority - is it?
> > 
> > NeilBrown
> 
> I think that depends a bit on whether we are talking about md raid10, or
> raid1+0.  For md raid10, resyncs and rebuilds are always going to take a
> bit of time - data has to be copied back and forth across the same
> disks, with a lot of head movement (especially for "far" layout).  I
> don't see that it would take much cpu time, however.  And this sort of
> layout is mostly for small arrays - it gives great performance when you
> have two drives, and perhaps up to about 4 drives.  But after that,
> people probably want raid1+0 layouts (i.e., raid0 stripes of raid1 pairs).
> 
> For raid1 + 0, resync time /is/ important - it is one of the reasons
> people pick it, especially the time to resync a single replaced drive.
> But again, I can't see how that would take cpu time.
> 
> I suppose that when doing the initial sync of this 16 disk array, you
> have to read in all data from 8 disks and write it out to the other 8
> disks.  That's a lot of IO passing through the cpus, even if there is no
> calculations going on.  But I thought each raid1 pair already had its
> own thread, so this would be scaled across 8 cores?  Is md raid10
> limited to just one thread?

For recovery the data should not pass through the CPU.  But it still has to
handle thousands of IO requests.
For resync, we read both copies and compare only writing if they differ.
This can be more efficient when the device is a lot slower than the CPU.

md/raid10 is currently limited to just one thread.

NeilBrown

> 
> mvh.,
> 
> David
> 
> 
> > 
> > 
> >>
> >>    I'm using:
> >>     - a phenom X6 at 3600mhz
> >>     - 16 seagate SSHDs ( sata3 7200RPM with 8GB ssd cache )
> >>     - 4x marvell 9230 sata3 controllers ( 4 ports each ) pcie 2.0 2x
> >> lanes.
> >>     - 8GB ram
> >>     - custom 3.12 kernel and mdadm compiled from latest source
> >>
> >>    what I did to test performance was to force a check on the array, and
> >> this
> >> leads to mdadm reporting a speed of about 990000K/sec. The hard disks
> >> report a 54% utilization. ( Overclocking the cpu by 200mhz increases the
> >> resync speed a bit and the hdd's utilizartion to about 58% )
> >>
> >>    If I do the same with a raid5 array instead of raid10, them resync
> >> speed
> >> will be almost double of raid10, the harddisk utilization reported will be
> >> 98-100% and I can see at least two cores being used.
> >>
> >> ________________________________________________________________________________
> >> Mensagem enviada através do email grátis AEIOU
> >> http://www.aeiou.pt
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]