[PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Dan Williams]

I am pleased to release this latest spin of the raid acceleration
patches for merge consideration.  This release aims to address all
pending review items including MD bug fixes and async_tx api changes
from Neil, and concerns on channel management from Chris and others.

Data integrity tests using home grown scripts and 'iozone -V' are
passing.  I am open to suggestions for additional testing criteria.  I
have also verified that git bisect is not broken by this set.

The short log below highlights the most recent changes.  The patches
will be sent as a reply to this message, and they are also available via
git:

	git pull git://lost.foo-projects.org/~dwillia2/git/iop md-accel-linus

Additional comments and feedback welcome.

Thanks,
Dan

--
01/16: dmaengine: add base support for the async_tx api
	* convert channel capabilities to a 'cpumask_t like' bitmap
02/16: dmaengine: move channel management to the client
	* this patch is new to this series
03/16: ARM: Add drivers/dma to arch/arm/Kconfig
04/16: dmaengine: add the async_tx api
	* remove the per operation type list, and distribute operation
	  capabilities evenly amongst the available channels
	* simplify async_tx_find_channel to optimize the fast path
05/16: md: add raid5_run_ops and support routines
	* explicitly handle the 2-disk raid5 case (xor becomes memcpy)
	* fix race between async engines and bi_end_io call for reads,
	  Neil Brown
	* remove unnecessary spin_lock from ops_complete_biofill
	* remove test_and_set/test_and_clear BUG_ONs, Neil Brown
	* remove explicit interrupt handling, Neil Brown
06/16: md: use raid5_run_ops for stripe cache operations
07/16: md: move write operations to raid5_run_ops
	* remove test_and_set/test_and_clear BUG_ONs, Neil Brown
08/16: md: move raid5 compute block operations to raid5_run_ops
	* remove the req_compute BUG_ON
09/16: md: move raid5 parity checks to raid5_run_ops
	* remove test_and_set/test_and_clear BUG_ONs, Neil Brown
10/16: md: satisfy raid5 read requests via raid5_run_ops
	* cleanup to_read and to_fill accounting
	* do not fail reads that have reached the cache
11/16: md: use async_tx and raid5_run_ops for raid5 expansion operations
12/16: md: move raid5 io requests to raid5_run_ops
13/16: md: remove raid5 compute_block and compute_parity5
14/16: dmaengine: driver for the iop32x, iop33x, and iop13xx raid engines
	* fix locking bug in iop_adma_alloc_chan_resources, Benjamin
	  Herrenschmidt
	* convert capabilities over to dma_cap_mask_t
15/16: iop13xx: Surface the iop13xx adma units to the iop-adma driver
16/16: iop3xx: Surface the iop3xx DMA and AAU units to the iop-adma driver

(previous release: http://marc.info/?l=linux-raid&m=117463257423193&w=2)

Re: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Nick Piggin]

Dan Williams wrote:
> I am pleased to release this latest spin of the raid acceleration
> patches for merge consideration.  This release aims to address all
> pending review items including MD bug fixes and async_tx api changes
> from Neil, and concerns on channel management from Chris and others.
> 
> Data integrity tests using home grown scripts and 'iozone -V' are
> passing.  I am open to suggestions for additional testing criteria.

Do you have performance numbers?

-- 
SUSE Labs, Novell Inc.

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Williams, Dan J]

> From: Nick Piggin [mailto:nickpiggin@yahoo.com.au]
> > I am pleased to release this latest spin of the raid acceleration
> > patches for merge consideration.  This release aims to address all
> > pending review items including MD bug fixes and async_tx api changes
> > from Neil, and concerns on channel management from Chris and others.
> >
> > Data integrity tests using home grown scripts and 'iozone -V' are
> > passing.  I am open to suggestions for additional testing criteria.
> 
> Do you have performance numbers?
> 
Patch #4 outlines the throughput gains as reported by tiobench.  20-30%
for sequential writes, and 40-55% for degraded reads.

Here are some recent screen captures of a resync operation with and
without offload, this is on an iop13xx platform:

mdadm --create /dev/md0 /dev/sd[abc] missing -n 4 -l 5
mdadm --add /dev/md0 /dev/sdd

---
CONFIG_DMA_ENGINE = n

top - 00:01:39 up 1 min,  1 user,  load average: 0.77, 0.20, 0.06
Tasks:  50 total,   1 running,  49 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3% us, 47.4% sy,  0.0% ni, 49.0% id,  0.0% wa,  0.7% hi,
2.6% si
Mem:   2074836k total,    36276k used,  2038560k free,        0k buffers
Swap:        0k total,        0k used,        0k free,    16560k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1307 root      11  -5     0    0    0 S 48.0  0.0   0:14.14 md0_raid5
 1319 root      15   0  2260 1052  868 R  0.3  0.1   0:00.12 top

iq81340mc:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdd[4] sdc[2] sdb[1]
      468872448 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  0.7% (1104000/156290816)
finish=108.1min speed=23919K/sec

---
CONFIG_DMA_ENGINE = y && CONFIG_INTEL_IOP_ADMA = y

top - 00:06:21 up 6 min,  1 user,  load average: 1.10, 0.68, 0.29
Tasks:  51 total,   1 running,  50 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5% us,  7.5% sy,  0.1% ni, 85.8% id,  1.9% wa,  0.5% hi,
3.5% si
Mem:   2074776k total,    40520k used,  2034256k free,        0k buffers
Swap:        0k total,        0k used,        0k free,    19448k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1354 root      10  -5     0    0    0 S 11.6  0.0   0:29.32 md0_raid5
 1491 root      18   0  2256  964  780 R  1.9  0.0   0:00.03 top
	
iq81340mc:/mnt# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid5 sdd[4] sdc[2] sdb[1] sda[0]
      468872448 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  3.5% (5474916/156290816)
finish=52.2min speed=48061K/sec

Some older iozone data is available here:
https://sourceforge.net/project/showfiles.php?group_id=115074&package_id
=203776

Regards,
Dan

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Justin Piszcz]

On Wed, 2 May 2007, Williams, Dan J wrote:

>> From: Nick Piggin [mailto:nickpiggin@yahoo.com.au]
>>> I am pleased to release this latest spin of the raid acceleration
>>> patches for merge consideration.  This release aims to address all
>>> pending review items including MD bug fixes and async_tx api changes
>>> from Neil, and concerns on channel management from Chris and others.
>>>
>>> Data integrity tests using home grown scripts and 'iozone -V' are
>>> passing.  I am open to suggestions for additional testing criteria.
>>
>> Do you have performance numbers?
>>
> Patch #4 outlines the throughput gains as reported by tiobench.  20-30%
> for sequential writes, and 40-55% for degraded reads.
>
> Here are some recent screen captures of a resync operation with and
> without offload, this is on an iop13xx platform:
>
> mdadm --create /dev/md0 /dev/sd[abc] missing -n 4 -l 5
> mdadm --add /dev/md0 /dev/sdd
>
> ---
> CONFIG_DMA_ENGINE = n
>
> top - 00:01:39 up 1 min,  1 user,  load average: 0.77, 0.20, 0.06
> Tasks:  50 total,   1 running,  49 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.3% us, 47.4% sy,  0.0% ni, 49.0% id,  0.0% wa,  0.7% hi,
> 2.6% si
> Mem:   2074836k total,    36276k used,  2038560k free,        0k buffers
> Swap:        0k total,        0k used,        0k free,    16560k cached
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 1307 root      11  -5     0    0    0 S 48.0  0.0   0:14.14 md0_raid5
> 1319 root      15   0  2260 1052  868 R  0.3  0.1   0:00.12 top
>
> iq81340mc:~# cat /proc/mdstat
> Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> md0 : active raid5 sda[0] sdd[4] sdc[2] sdb[1]
>      468872448 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
>      [>....................]  recovery =  0.7% (1104000/156290816)
> finish=108.1min speed=23919K/sec
>
> ---
> CONFIG_DMA_ENGINE = y && CONFIG_INTEL_IOP_ADMA = y
>
> top - 00:06:21 up 6 min,  1 user,  load average: 1.10, 0.68, 0.29
> Tasks:  51 total,   1 running,  50 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.5% us,  7.5% sy,  0.1% ni, 85.8% id,  1.9% wa,  0.5% hi,
> 3.5% si
> Mem:   2074776k total,    40520k used,  2034256k free,        0k buffers
> Swap:        0k total,        0k used,        0k free,    19448k cached
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 1354 root      10  -5     0    0    0 S 11.6  0.0   0:29.32 md0_raid5
> 1491 root      18   0  2256  964  780 R  1.9  0.0   0:00.03 top
>
> iq81340mc:/mnt# cat /proc/mdstat
> Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> md0 : active raid5 sdd[4] sdc[2] sdb[1] sda[0]
>      468872448 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
>      [>....................]  recovery =  3.5% (5474916/156290816)
> finish=52.2min speed=48061K/sec
>
> Some older iozone data is available here:
> https://sourceforge.net/project/showfiles.php?group_id=115074&package_id
> =203776
>
> Regards,
> Dan
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I have not been following this closely, must you have an 
CONFIG_INTEL_IOP_ADMA piece of hardware and/or chipset to use this 
feature or can regular desktop users take hold of it as well?

Justin.

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Williams, Dan J]

> From: Justin Piszcz [mailto:jpiszcz@lucidpixels.com]
> I have not been following this closely, must you have an
> CONFIG_INTEL_IOP_ADMA piece of hardware and/or chipset to use this
> feature or can regular desktop users take hold of it as well?
> 
Currently this feature is available on the iop series of processors,
e.g. http://www.intel.com/design/iio/iop348.htm

And there is a driver in the works to support the 440spe platform
(http://marc.info/?l=linux-raid&m=117400143317440&w=2)

Desktops currently take advantage of architecture specific SIMD
instructions to accelerate xor calculations.

> Justin.

Dan

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Justin Piszcz]

On Wed, 2 May 2007, Williams, Dan J wrote:

>> From: Justin Piszcz [mailto:jpiszcz@lucidpixels.com]
>> I have not been following this closely, must you have an
>> CONFIG_INTEL_IOP_ADMA piece of hardware and/or chipset to use this
>> feature or can regular desktop users take hold of it as well?
>>
> Currently this feature is available on the iop series of processors,
> e.g. http://www.intel.com/design/iio/iop348.htm
>
> And there is a driver in the works to support the 440spe platform
> (http://marc.info/?l=linux-raid&m=117400143317440&w=2)
>
> Desktops currently take advantage of architecture specific SIMD
> instructions to accelerate xor calculations.
>
>> Justin.
>
> Dan
>

Ahh-- these are the family of chips in the Areca controllers (or close to 
them)-- so md/software raid will utilize the HW functions of the chip? 
If so, that is pretty slick :)

Justin.

Re: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Dagfinn Ilmari Mannsåker]

"Williams, Dan J" <dan.j.williams@intel.com> writes:

>> From: Justin Piszcz [mailto:jpiszcz@lucidpixels.com]
>> I have not been following this closely, must you have an
>> CONFIG_INTEL_IOP_ADMA piece of hardware and/or chipset to use this
>> feature or can regular desktop users take hold of it as well?
>> 
> Currently this feature is available on the iop series of processors,
> e.g. http://www.intel.com/design/iio/iop348.htm
>
> And there is a driver in the works to support the 440spe platform
> (http://marc.info/?l=linux-raid&m=117400143317440&w=2)
>
> Desktops currently take advantage of architecture specific SIMD
> instructions to accelerate xor calculations.

How about hard drive controllers with RAID accelleration features (but
not full hardware RAID capability) such as the Promise SX4
(http://linux-ata.org/faq-sata-raid.html#sx4)?

-- 
ilmari
"A disappointingly low fraction of the human race is,
 at any given time, on fire." - Stig Sandbeck Mathisen

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Williams, Dan J]

> From: Dagfinn Ilmari Mannsåker [mailto:ilmari@ilmari.org]
> How about hard drive controllers with RAID accelleration features (but
> not full hardware RAID capability) such as the Promise SX4
> (http://linux-ata.org/faq-sata-raid.html#sx4)?
> 
See this thread:
http://marc.info/?l=linux-kernel&m=116793705601271&w=2
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Ronen Shitrit]

Hi

The resync numbers you sent, looks very promising :)
Do you have any performance numbers that you can share for these set of
patches, which shows the Rd/Wr IO bandwidth.

Thanks
Ronen Shitrit

RE: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Tomasz Chmielewski]

Ronen Shitrit wrote:

> The resync numbers you sent, looks very promising :)
> Do you have any performance numbers that you can share for these set of
> patches, which shows the Rd/Wr IO bandwidth.

I have some simple tests made with hdparm, with the results I don't 
understand.

We see hdparm results are fine if we access the whole device:

thecus:~# hdparm -Tt /dev/sdd

/dev/sdd:
  Timing cached reads:   392 MB in  2.00 seconds = 195.71 MB/sec
  Timing buffered disk reads:  146 MB in  3.01 seconds =  48.47 MB/sec


But are 10 times worse (Timing buffered disk reads) when we access 
partitions:

thecus:/# hdparm -Tt /dev/sdc1 /dev/sdd1

/dev/sdc1:
  Timing cached reads:   396 MB in  2.01 seconds = 197.18 MB/sec
  Timing buffered disk reads:   16 MB in  3.32 seconds =   4.83 MB/sec

/dev/sdd1:
  Timing cached reads:   394 MB in  2.00 seconds = 196.89 MB/sec
  Timing buffered disk reads:   16 MB in  3.13 seconds =   5.11 MB/sec


Why is it so much worse?


I used 2.6.21-iop1 patches from http://sf.net/projects/xscaleiop; right 
now I use 2.6.17-iop1, for which the results are ~35 MB/s when accessing 
a device (/dev/sdd) or a partition (/dev/sdd1).


In kernel config, I enabled Intel DMA engines.

The device I use is Thecus n4100, it is "Platform: IQ31244 (XScale)", 
and has 600 MHz CPU.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [PATCH 00/16] raid acceleration and asynchronous offload api for 2.6.22

[Tomasz Chmielewski]

Tomasz Chmielewski schrieb:
> Ronen Shitrit wrote:
> 
>> The resync numbers you sent, looks very promising :)
>> Do you have any performance numbers that you can share for these set of
>> patches, which shows the Rd/Wr IO bandwidth.
> 
> I have some simple tests made with hdparm, with the results I don't 
> understand.
> 
> We see hdparm results are fine if we access the whole device:
> 
> thecus:~# hdparm -Tt /dev/sdd
> 
> /dev/sdd:
>  Timing cached reads:   392 MB in  2.00 seconds = 195.71 MB/sec
>  Timing buffered disk reads:  146 MB in  3.01 seconds =  48.47 MB/sec
> 
> 
> But are 10 times worse (Timing buffered disk reads) when we access 
> partitions:

There seems to be another side effect when comparing DMA engine in 
2.6.17-iop1 to 2.6.21-iop1: network performance.


For simple network tests, I use "netperf" tool to measure network 
performance.

With 2.6.17-iop1 and all DMA offloading options enabled (selectable in 
System type ---> IOP3xx Implementation Options  --->), I get nearly 25 
MB/s throughput.

With 2.6.21-iop1 and all DMA offloading optons enabled (moved to Device 
Drivers  ---> DMA Engine support  --->), I get only about 10 MB/s 
throughput.
Additionally, on 2.6.21-iop1, I get lots of "dma_cookie < 0" printed by 
the kernel.


-- 
Tomasz Chmielewski
http://wpkg.org