Re: high throughput storage server?

[Roberto Spadim <roberto@spadim.com.br>]

with 2 disks md raid0 i get 400MB/s SAS 10krpm 6gb/s channel
you will need at last 10000/400*2=25*2=50 disks to get a start number
memory/cpu/network speed?
memory must allow more than 10gb/s (ddr3 can do this, i don't know if
enabled ecc will be a problem or not, check with memtest86+)
cpu? hummm i don't know very well how to help here, since it's just
read and write memory/interfaces (network/disks), maybe a 'magic'
number like: 3ghz * 64bits/8bits=24.000 (maybe 24gbits/s) i don't know
how to estimate... but i think you will need a multicore cpu... maybe
one for network one for disks one for mdadm one for nfs and one for
linux, >=5 cores at least with 3ghz 64bits each (maybe starting with
xeon 6cores with hyper thread)
it's just a idea how to estimate, it's not correct/true/real
i think it's better contact ibm/dell/hp/compaq/texas/anyother and talk
about the problem, post results here, this is a nice hardware question
:)
don't tell about software raid, just the hardware to allow this
bandwidth (10gb/s) and share files

2011/3/20 NeilBrown <neilb@suse.de>:
> On Sat, 19 Mar 2011 20:34:26 -0500 Stan Hoeppner <stan@hardwarefreak.com>
> wrote:
>
>> NeilBrown put forth on 3/18/2011 5:01 PM:
>> > On Fri, 18 Mar 2011 10:43:43 -0500 Stan Hoeppner <stan@hardwarefreak.com>
>> > wrote:
>> >
>> >> Christoph Hellwig put forth on 3/18/2011 9:05 AM:
>> >>
>> >> Thanks for the confirmations and explanations.
>> >>
>> >>> The kernel is pretty smart in placement of user and page cache data, but
>> >>> it can't really second guess your intention.  With the numactl tool you
>> >>> can help it doing the proper placement for you workload.  Note that the
>> >>> choice isn't always trivial - a numa system tends to have memory on
>> >>> multiple nodes, so you'll either have to find a good partitioning of
>> >>> your workload or live with off-node references.  I don't think
>> >>> partitioning NFS workloads is trivial, but then again I'm not a
>> >>> networking expert.
>> >>
>> >> Bringing mdraid back into the fold, I'm wondering what kinda of load the
>> >> mdraid threads would place on a system of the caliber needed to push
>> >> 10GB/s NFS.
>> >>
>> >> Neil, I spent quite a bit of time yesterday spec'ing out what I believe
>> >
>> > Addressing me directly in an email that wasn't addressed to me directly seem
>> > a bit ... odd.  Maybe that is just me.
>>
>> I guess that depends on one's perspective.  Is it the content of email
>> To: and Cc: headers that matters, or the substance of the list
>> discussion thread?  You are the lead developer and maintainer of Linux
>> mdraid AFAIK.  Thus I would have assumed that directly addressing a
>> question to you within any given list thread was acceptable, regardless
>> of whose address was where in the email headers.
>
> This assumes that I read every email on this list.  I certainly do read a lot,
> but I tend to tune out of threads that don't seem particularly interesting -
> and configuring hardware is only vaguely interesting to me - and I am sure
> there are people on the list with more experience.
>
> But whatever... there is certainly more chance of me missing something that
> isn't directly addressed to me (such messages get filed differently).
>
>
>>
>> >> How much of each core's cycles will we consume with normal random read
>> >
>> > For RAID10, the md thread plays no part in reads.  Which ever thread
>> > submitted the read submits it all the way down to the relevant member device.
>> > If the read fails the thread will come in to play.
>>
>> So with RIAD10 read scalability is in essence limited to the execution
>> rate of the block device layer code and the interconnect b/w required.
>
> Correct.
>
>>
>> > For writes, the thread is used primarily to make sure the writes are properly
>> > orders w.r.t. bitmap updates.  I could probably remove that requirement if a
>> > bitmap was not in use...
>>
>> How compute intensive is this thread during writes, if at all, at
>> extreme IO bandwidth rates?
>
> Not compute intensive at all - just single threaded.  So it will only
> dispatch a single request at a time.  Whether single threading the writes is
> good or bad is not something that I'm completely clear on.  It seems bad in
> the sense that modern machines have lots of CPUs and we are fore-going any
> possible benefits of parallelism.  However the current VM seems to do all
> (or most) writeout from a single thread per device - the 'bdi' threads.
> So maybe keeping it single threaded in the md level is perfectly natural and
> avoids cache bouncing...
>
>
>>
>> >> operations assuming 10GB/s of continuous aggregate throughput?  Would
>> >> the mdraid threads consume sufficient cycles that when combined with
>> >> network stack processing and interrupt processing, that 16 cores at
>> >> 2.4GHz would be insufficient?  If so, would bumping the two sockets up
>> >> to 24 cores at 2.1GHz be enough for the total workload?  Or, would we
>> >> need to move to a 4 socket system with 32 or 48 cores?
>> >>
>> >> Is this possibly a situation where mdraid just isn't suitable due to the
>> >> CPU, memory, and interconnect bandwidth demands, making hardware RAID
>> >> the only real option?
>> >
>> > I'm sorry, but I don't do resource usage estimates or comparisons with
>> > hardware raid.  I just do software design and coding.
>>
>> I probably worded this question very poorly and have possibly made
>> unfair assumptions about mdraid performance.
>>
>> >>     And if it does requires hardware RAID, would it
>> >> be possible to stick 16 block devices together in a --linear mdraid
>> >> array and maintain the 10GB/s performance?  Or, would the single
>> >> --linear array be processed by a single thread?  If so, would a single
>> >> 2.4GHz core be able to handle an mdraid --leaner thread managing 8
>> >> devices at 10GB/s aggregate?
>> >
>> > There is no thread for linear or RAID0.
>>
>> What kernel code is responsible for the concatenation and striping
>> operations of mdraid linear and RAID0 if not an mdraid thread?
>>
>
> When the VM or filesystem or whatever wants to start an IO request, it calls
> into the md code to find out how big it is allowed to make that request.  The
> md code returns a number which ensures that the request will end up being
> mapped onto just one drive (at least in the majority of cases).
> The VM or filesystem builds up the request (a struct bio) to at most that
> size and hands it to md.  md simply assigns a different target device and
> offset in that device to the request, and hands it over the the target device.
>
> So whatever thread it was that started the request carries it all the way
> down to the device which is a member of the RAID array (for RAID0/linear).
> Typically it then gets placed on a queue, and an interrupt handler takes it
> off the queue and acts upon it.
>
> So - no separate md thread.
>
> RAID1 and RAID10 make only limited use of their thread, doing as much of the
> work as possible in the original calling thread.
> RAID4/5/6 do lots of work in the md thread.  The calling thread just finds a
> place in the stripe cache to attach the request, attaches it, and signals the
> thread.
> (Though reads on a non-degraded array can by-pass the cache and are handled
> much like reads on RAID0).
>
>> > If you want to share load over a number of devices, you would normally use
>> > RAID0.  However if the load had a high thread count and the filesystem
>> > distributed IO evenly across the whole device space, then linear might work
>> > for you.
>>
>> In my scenario I'm thinking I'd want to stay away RAID0 because of the
>> multi-level stripe width issues of double nested RAID (RAID0 over
>> RAID10).  I assumed linear would be the way to go, as my scenario calls
>> for using XFS.  Using 32 allocation groups should evenly spread the
>> load, which is ~50 NFS clients.
>
> You may well be right.
>
>>
>> What I'm trying to figure out is how much CPU time I am going to need for:
>>
>> 1.  Aggregate 10GB/s IO rate
>> 2.  mdraid managing 384 drives
>>     A.  16 mdraid10 arrays of 24 drives each
>>     B.  mdraid linear concatenating the 16 arrays
>
> I very much doubt that CPU is going to be an issue.  Memory bandwidth might -
> but I'm only really guessing here, so it is probably time to stop.
>
>
>>
>> Thanks for your input Neil.
>>
> Pleasure.
>
> NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html