async_tx and RAID HW xor engine

async_tx and RAID HW xor engine

[Rajasekhar Pulluru]

Hi,

I am trying to understand how are hardware xor engine's utilized by
async_tx module. I don't get the exact link that connects async_tx and
hardware xor engine.
Could someone help me understand it.

Registration of a RAID HW XOR engine happens in
drivers/dma/<soc-xor.c> using dma_async_device_register(). Does this
registration function connect the async_tx api's
to use the HW XOR engine? Or is there anything we need to support to
make async_tx api's to take advantage of the hardware XOR engine?

Thanks & Regards,
Rajasekhar

Re: async_tx and RAID HW xor engine

[Dan Williams]

[ adding Russell and Neil for comment about the proposed change to async_tx ]

On Mon, Apr 30, 2012 at 3:05 AM, Rajasekhar Pulluru
<pullururajasekhar@gmail.com> wrote:
> Hi,
>
> I am trying to understand how are hardware xor engine's utilized by
> async_tx module. I don't get the exact link that connects async_tx and
> hardware xor engine.
> Could someone help me understand it.
>
> Registration of a RAID HW XOR engine happens in
> drivers/dma/<soc-xor.c> using dma_async_device_register(). Does this
> registration function connect the async_tx api's
> to use the HW XOR engine?

Yes, the channel registers its capabilities with dmaengine, and then
async_tx asks dmaengine (via dma_find_channel) for a resource to carry
out the operation(s).

> Or is there anything we need to support to
> make async_tx api's to take advantage of the hardware XOR engine?

Registration with dma engine should be all that is needed, but a word
of caution that the channel switching mechanism has been found to be
incompatible with the dma mapping api.  So, unless your dma channel
supports all the operations needed for raid, do not rely on async_tx's
channel switching capabilities.

An architecture where this is known to be a problem is ARM, but I
wonder if we should take the following global step until this is fixed
properly?

diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index cf9da36..b812b6b 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -280,6 +280,7 @@ config NET_DMA
 config ASYNC_TX_DMA
        bool "Async_tx: Offload support for the async_tx api"
        depends on DMA_ENGINE
+       depends on !ASYNC_TX_ENABLE_CHANNEL_SWITCH
        help
          This allows the async_tx api to take advantage of offload engines for
          memcpy, memset, xor, and raid6 p+q operations.  If your platform has

Re: async_tx and RAID HW xor engine

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2504 bytes --]

On Mon, 30 Apr 2012 13:31:41 -0700 Dan Williams <dan.j.williams@intel.com>
wrote:

> [ adding Russell and Neil for comment about the proposed change to async_tx ]

I've managed to avoid learning much about the internal details of async_tx
and adma but I thought I'd have a quick look.

Doesn't this Kconfig change simply disable ASYNC_TX_DMA on any device that
selects ASNC_TX_ENABLE_CHANNEL_SWITCH.  i.e.
  INTEL_IOP_ADMA
  FSL_DMA
  MX_XOR
  AMCC_PPC440_SPA_ADMA
Is that what you want to do?  Disable all of those?
I guess if they don't work then that is the right thing to do....

NeilBrown




> 
> On Mon, Apr 30, 2012 at 3:05 AM, Rajasekhar Pulluru
> <pullururajasekhar@gmail.com> wrote:
> > Hi,
> >
> > I am trying to understand how are hardware xor engine's utilized by
> > async_tx module. I don't get the exact link that connects async_tx and
> > hardware xor engine.
> > Could someone help me understand it.
> >
> > Registration of a RAID HW XOR engine happens in
> > drivers/dma/<soc-xor.c> using dma_async_device_register(). Does this
> > registration function connect the async_tx api's
> > to use the HW XOR engine?
> 
> Yes, the channel registers its capabilities with dmaengine, and then
> async_tx asks dmaengine (via dma_find_channel) for a resource to carry
> out the operation(s).
> 
> > Or is there anything we need to support to
> > make async_tx api's to take advantage of the hardware XOR engine?
> 
> Registration with dma engine should be all that is needed, but a word
> of caution that the channel switching mechanism has been found to be
> incompatible with the dma mapping api.  So, unless your dma channel
> supports all the operations needed for raid, do not rely on async_tx's
> channel switching capabilities.
> 
> An architecture where this is known to be a problem is ARM, but I
> wonder if we should take the following global step until this is fixed
> properly?
> 
> diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
> index cf9da36..b812b6b 100644
> --- a/drivers/dma/Kconfig
> +++ b/drivers/dma/Kconfig
> @@ -280,6 +280,7 @@ config NET_DMA
>  config ASYNC_TX_DMA
>         bool "Async_tx: Offload support for the async_tx api"
>         depends on DMA_ENGINE
> +       depends on !ASYNC_TX_ENABLE_CHANNEL_SWITCH
>         help
>           This allows the async_tx api to take advantage of offload engines for
>           memcpy, memset, xor, and raid6 p+q operations.  If your platform has


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: async_tx and RAID HW xor engine

[Dan Williams]

On Mon, Apr 30, 2012 at 5:49 PM, NeilBrown <neilb@suse.de> wrote:
> On Mon, 30 Apr 2012 13:31:41 -0700 Dan Williams <dan.j.williams@intel.com>
> wrote:
>
>> [ adding Russell and Neil for comment about the proposed change to async_tx ]
>
> I've managed to avoid learning much about the internal details of async_tx
> and adma but I thought I'd have a quick look.
>
> Doesn't this Kconfig change simply disable ASYNC_TX_DMA on any device that
> selects ASNC_TX_ENABLE_CHANNEL_SWITCH.  i.e.
>  INTEL_IOP_ADMA
>  FSL_DMA
>  MX_XOR
>  AMCC_PPC440_SPA_ADMA
> Is that what you want to do?  Disable all of those?
> I guess if they don't work then that is the right thing to do....

Now that I refresh my memory on the full scope of the problem, this
change would help, but would not be sufficient.  The undefined
behavior that the code currently relies on is overlapping dma-mappings
to the same buffer (happens to work on x86, known to be broken on ARM
v6).  So md/raid456 and async_tx actually need to conscious of up to
three dma mapping cases:

1/ pure software raid
2/ offload to a dma domain that can handle a copy+xor/pq+validate
chain (dma map the stripe once)
    2a/ single domain single channel
    2b/ single domain multiple channels - channel switch but reuse the
initial dma mapping for all operations
3/ offload to collection of dma domains with differing capabilities
(channel switch + remap the stripe every time we cross a domain
boundary)

The biggest difference between 2 and 3 is that the entire chain of
operations can be prepped and submitted to hardware up front in 2, but
needs to wait for the completion of the previous operation before
re-mapping in 3.

An ulterior motive of adding "depends on
!ASYNC_TX_ENABLE_CHANNEL_SWITCH" is to elicit some help with this
rework from other driver maintainers since I'm not even sure 3 is a
configuration the current engines will present.

--
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html