cpu/memory use

cpu/memory use

[Roberto Spadim]

Hi, i wass thinking about cpu/memory use
i have a dual (three four) cpu
could i make linux to only use cpu1 for raid? and others to anythink
else, don´t migrate raid between cpus... and don´t allow others
programs to use raid cpu...
is it possible?
is it dificult?
it´s more linux related feature, not raid related, but could we implement it?
what about memory usage? how many memory software raid use? is it per
device, per raid, does it have a hard limit (offcourse)? could we
calculate it? for example i want raid1 with two disks of 1tb, how many
memory should i buy?

-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cpu/memory use

[NeilBrown]

On Wed, 19 Jan 2011 22:14:45 -0200 Roberto Spadim <roberto@spadim.com.br>
wrote:

> Hi, i wass thinking about cpu/memory use
> i have a dual (three four) cpu
> could i make linux to only use cpu1 for raid? and others to anythink
> else, don´t migrate raid between cpus... and don´t allow others
> programs to use raid cpu...
> is it possible?
> is it dificult?
> it´s more linux related feature, not raid related, but could we implement it?
> what about memory usage? how many memory software raid use? is it per
> device, per raid, does it have a hard limit (offcourse)? could we
> calculate it? for example i want raid1 with two disks of 1tb, how many
> memory should i buy?
> 

For levels other than RAID4/5/6, md/raid does not use any significant amount
of CPU or memory.

For RAID4/5/6, md's use for CPU is single-threads so it will only use a
single CPU - which ever one the scheduler allocates it to from time to time.

The only room for improvement that I can see would be to allow the 'xor'
calculation to be run in parallel on multiple CPUs and that would only help
if the storage devices were nearly as fast as a CPU.  Where we have tried
parallelising xor, it has only made things slower. 


For your particular question about RAID1 - use a RAID1 across two devices
would use less than 100K more than using just one of the devices.
During resync it might use as much as a couple of megabytes of extra memory.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cpu/memory use

[Roberto Spadim]

hummmm nice
i asked it because there´s many questions about
is hardware raid better than software raid?
i think that software is easier to implement and cpu (intel x86) is
easier to upgrade than a arm(fpga or another) hardware raid cpu

but thinking about raid software i never read anything about cpu use/memory use
now i see it´s very small footprint for raid0/raid1, just checksum
make thinks slower raid != 0/1

could we implement a "real" hardware raid?
what´s "real"? a cpu just for raid, a memory just for raid (fixed size)
maybe with linux cpu afinity, and linux max memory use?
ok it´s not a good ide, but for benchmarks it´s the best scenario, and
we will never get a "on high load your software raid can be slower..."
got the problem? it´s not a numeric optimization, it´s just a human
feeling (social? professional?) "optimization"
and we know that we will always have a portion of memory and a portion
of cpu just for raid =)

2011/1/19 NeilBrown <neilb@suse.de>:
> On Wed, 19 Jan 2011 22:14:45 -0200 Roberto Spadim <roberto@spadim.com.br>
> wrote:
>
>> Hi, i wass thinking about cpu/memory use
>> i have a dual (three four) cpu
>> could i make linux to only use cpu1 for raid? and others to anythink
>> else, don´t migrate raid between cpus... and don´t allow others
>> programs to use raid cpu...
>> is it possible?
>> is it dificult?
>> it´s more linux related feature, not raid related, but could we implement it?
>> what about memory usage? how many memory software raid use? is it per
>> device, per raid, does it have a hard limit (offcourse)? could we
>> calculate it? for example i want raid1 with two disks of 1tb, how many
>> memory should i buy?
>>
>
> For levels other than RAID4/5/6, md/raid does not use any significant amount
> of CPU or memory.
>
> For RAID4/5/6, md's use for CPU is single-threads so it will only use a
> single CPU - which ever one the scheduler allocates it to from time to time.
>
> The only room for improvement that I can see would be to allow the 'xor'
> calculation to be run in parallel on multiple CPUs and that would only help
> if the storage devices were nearly as fast as a CPU.  Where we have tried
> parallelising xor, it has only made things slower.
>
>
> For your particular question about RAID1 - use a RAID1 across two devices
> would use less than 100K more than using just one of the devices.
> During resync it might use as much as a couple of megabytes of extra memory.
>
> NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cpu/memory use

[David Brown]

On 20/01/2011 04:33, Roberto Spadim wrote:
> hummmm nice
> i asked it because there´s many questions about
> is hardware raid better than software raid?
> i think that software is easier to implement and cpu (intel x86) is
> easier to upgrade than a arm(fpga or another) hardware raid cpu
>
> but thinking about raid software i never read anything about cpu use/memory use
> now i see it´s very small footprint for raid0/raid1, just checksum
> make thinks slower raid != 0/1
>
> could we implement a "real" hardware raid?

"Hardware raid" means there is dedicated hardware that handles things 
like parity calculations and raid1 duplication - the host cpu only sees 
the un-raided transactions.  Hardware raid controllers will also often 
have battery backup to make sure stripes are consistent in the event of 
failures.

You can't make software raid work as "real hardware raid", because it 
doesn't have these hardware features.

Software raid can easily be as fast, or often significantly faster, than 
hardware raid cards.  The host processor is typically much faster at 
parity calculations than raid card processors.  I haven't done 
benchmarks (and have no hardware raid card for comparison), but I expect 
that with a modern cpu and motherboard, the performance bottleneck would 
be PCIExpress IO if you have a lot of duplication (for example, raid 15).

I don't think cpu core affinity would make a significant difference, but 
it should be easy enough to try it.  And I'm sure md raid already uses 
non-paged kernel memory as necessary (and I believe it is possible to 
tune the number of stripe buffers used for raid5/6).  Linux will always 
use as much free main memory as it can for disk caching, which will be 
far more than you will get on a hardware raid card.


If you set up a Linux box as a NAS with mdadm raid, so that it is just a 
disk server (iSCSI, for example), and have an UPS, then you effectively 
have a hardware raid system.

> what´s "real"? a cpu just for raid, a memory just for raid (fixed size)
> maybe with linux cpu afinity, and linux max memory use?
> ok it´s not a good ide, but for benchmarks it´s the best scenario, and
> we will never get a "on high load your software raid can be slower..."
> got the problem? it´s not a numeric optimization, it´s just a human
> feeling (social? professional?) "optimization"
> and we know that we will always have a portion of memory and a portion
> of cpu just for raid =)
>
> 2011/1/19 NeilBrown<neilb@suse.de>:
>> On Wed, 19 Jan 2011 22:14:45 -0200 Roberto Spadim<roberto@spadim.com.br>
>> wrote:
>>
>>> Hi, i wass thinking about cpu/memory use
>>> i have a dual (three four) cpu
>>> could i make linux to only use cpu1 for raid? and others to anythink
>>> else, don´t migrate raid between cpus... and don´t allow others
>>> programs to use raid cpu...
>>> is it possible?
>>> is it dificult?
>>> it´s more linux related feature, not raid related, but could we implement it?
>>> what about memory usage? how many memory software raid use? is it per
>>> device, per raid, does it have a hard limit (offcourse)? could we
>>> calculate it? for example i want raid1 with two disks of 1tb, how many
>>> memory should i buy?
>>>
>>
>> For levels other than RAID4/5/6, md/raid does not use any significant amount
>> of CPU or memory.
>>
>> For RAID4/5/6, md's use for CPU is single-threads so it will only use a
>> single CPU - which ever one the scheduler allocates it to from time to time.
>>
>> The only room for improvement that I can see would be to allow the 'xor'
>> calculation to be run in parallel on multiple CPUs and that would only help
>> if the storage devices were nearly as fast as a CPU.  Where we have tried
>> parallelising xor, it has only made things slower.
>>
>>
>> For your particular question about RAID1 - use a RAID1 across two devices
>> would use less than 100K more than using just one of the devices.
>> During resync it might use as much as a couple of megabytes of extra memory.
>>
>> NeilBrown
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cpu/memory use

[Roberto Spadim]

yes, it's what i think too
i will try some userspace cpu affinity to be sure that benchmarks are
'real' enoght
another question... when using beowolf, or another "cpu hot plug"
(network cpu), can raid don't use these "external" cpus, just local
ones?

2011/1/20 David Brown <david@westcontrol.com>:
> On 20/01/2011 04:33, Roberto Spadim wrote:
>>
>> hummmm nice
>> i asked it because there´s many questions about
>> is hardware raid better than software raid?
>> i think that software is easier to implement and cpu (intel x86) is
>> easier to upgrade than a arm(fpga or another) hardware raid cpu
>>
>> but thinking about raid software i never read anything about cpu
>> use/memory use
>> now i see it´s very small footprint for raid0/raid1, just checksum
>> make thinks slower raid != 0/1
>>
>> could we implement a "real" hardware raid?
>
> "Hardware raid" means there is dedicated hardware that handles things like
> parity calculations and raid1 duplication - the host cpu only sees the
> un-raided transactions.  Hardware raid controllers will also often have
> battery backup to make sure stripes are consistent in the event of failures.
>
> You can't make software raid work as "real hardware raid", because it
> doesn't have these hardware features.
>
> Software raid can easily be as fast, or often significantly faster, than
> hardware raid cards.  The host processor is typically much faster at parity
> calculations than raid card processors.  I haven't done benchmarks (and have
> no hardware raid card for comparison), but I expect that with a modern cpu
> and motherboard, the performance bottleneck would be PCIExpress IO if you
> have a lot of duplication (for example, raid 15).
>
> I don't think cpu core affinity would make a significant difference, but it
> should be easy enough to try it.  And I'm sure md raid already uses
> non-paged kernel memory as necessary (and I believe it is possible to tune
> the number of stripe buffers used for raid5/6).  Linux will always use as
> much free main memory as it can for disk caching, which will be far more
> than you will get on a hardware raid card.
>
>
> If you set up a Linux box as a NAS with mdadm raid, so that it is just a
> disk server (iSCSI, for example), and have an UPS, then you effectively have
> a hardware raid system.
>
>> what´s "real"? a cpu just for raid, a memory just for raid (fixed size)
>> maybe with linux cpu afinity, and linux max memory use?
>> ok it´s not a good ide, but for benchmarks it´s the best scenario, and
>> we will never get a "on high load your software raid can be slower..."
>> got the problem? it´s not a numeric optimization, it´s just a human
>> feeling (social? professional?) "optimization"
>> and we know that we will always have a portion of memory and a portion
>> of cpu just for raid =)
>>
>> 2011/1/19 NeilBrown<neilb@suse.de>:
>>>
>>> On Wed, 19 Jan 2011 22:14:45 -0200 Roberto Spadim<roberto@spadim.com.br>
>>> wrote:
>>>
>>>> Hi, i wass thinking about cpu/memory use
>>>> i have a dual (three four) cpu
>>>> could i make linux to only use cpu1 for raid? and others to anythink
>>>> else, don´t migrate raid between cpus... and don´t allow others
>>>> programs to use raid cpu...
>>>> is it possible?
>>>> is it dificult?
>>>> it´s more linux related feature, not raid related, but could we
>>>> implement it?
>>>> what about memory usage? how many memory software raid use? is it per
>>>> device, per raid, does it have a hard limit (offcourse)? could we
>>>> calculate it? for example i want raid1 with two disks of 1tb, how many
>>>> memory should i buy?
>>>>
>>>
>>> For levels other than RAID4/5/6, md/raid does not use any significant
>>> amount
>>> of CPU or memory.
>>>
>>> For RAID4/5/6, md's use for CPU is single-threads so it will only use a
>>> single CPU - which ever one the scheduler allocates it to from time to
>>> time.
>>>
>>> The only room for improvement that I can see would be to allow the 'xor'
>>> calculation to be run in parallel on multiple CPUs and that would only
>>> help
>>> if the storage devices were nearly as fast as a CPU.  Where we have tried
>>> parallelising xor, it has only made things slower.
>>>
>>>
>>> For your particular question about RAID1 - use a RAID1 across two devices
>>> would use less than 100K more than using just one of the devices.
>>> During resync it might use as much as a couple of megabytes of extra
>>> memory.
>>>
>>> NeilBrown
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html