Question on IOAT

Question on IOAT

[Olaf Kirch]

Hi,

I looked into the IOAT code today as I'm trying to find out how to add support
for it to NFS. I ran into this piece of code, which waits for the DMA
operation to complete:

                while (dma_async_memcpy_complete(tp->ucopy.dma_chan,
                                                 tp->ucopy.dma_cookie, &done,
                                                 &used) == DMA_IN_PROGRESS) {
                        /* do partial cleanup of sk_async_wait_queue */
                        while ((skb = skb_peek(&sk->sk_async_wait_queue)) &&
                               (dma_async_is_complete(skb->dma_cookie, done,
                                                      used) == DMA_SUCCESS)) {
                                __skb_dequeue(&sk->sk_async_wait_queue);
                                kfree_skb(skb);
                        }
                }

Nowhere in the dma_async_*complete functions can I see any code
that would sleep if the DMA is not yet complete. Am I missing something,
or are we really busy-waiting on the DMA engine? Wouldn't this kind of
defeat the purpose of freeing up the CPU from the chores of memcpying?

I also checked the code in ioatdma.c - I would have expected there to
be some kind of interrupt handler that kicks the upper layers when a
DMA operation completes. But the interrupt handler seems to be for
error reporting exclusively... 

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir@lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

Re: Question on IOAT

[Chris Leech]

On 2/5/07, Olaf Kirch <olaf.kirch@oracle.com> wrote:

> Nowhere in the dma_async_*complete functions can I see any code
> that would sleep if the DMA is not yet complete. Am I missing something,
> or are we really busy-waiting on the DMA engine? Wouldn't this kind of
> defeat the purpose of freeing up the CPU from the chores of memcpying?

It is busy waiting, but only because the TCP socket use initiates the
DMA copies from the softirq and they have time to complete during the
switch back to application context.  Going back to sleep and creating
more context switching made things worse.  I'm working on seeing if
completion interrupts could be used with a better thought out
implementation, the performance implications aren't fully clear to me
yet.

For other uses, interrupts are probably desired.

> I also checked the code in ioatdma.c - I would have expected there to
> be some kind of interrupt handler that kicks the upper layers when a
> DMA operation completes. But the interrupt handler seems to be for
> error reporting exclusively...

It's just not there now, but it can be added easily, it's one bit in
the descriptor and a register read in the interrupt handler to see
which channel(s) need attention.

- Chris

Re: Question on IOAT

[Olaf Kirch]

On Monday 05 February 2007 19:24, Chris Leech wrote:
> It is busy waiting, but only because the TCP socket use initiates the
> DMA copies from the softirq and they have time to complete during the
> switch back to application context.

But to me it looks as if the code in tcp_recvmsg will also initiate DMA
copies when the user calls recvmsg and there is data queued on the
socket. So in that case, what you end up doing in the same process
context is queue DMA ops -> issue pending -> busy wait. In that case,
sleeping would sound like the right approach, wouldn't it?

> Going back to sleep and creating 
> more context switching made things worse.  I'm working on seeing if
> completion interrupts could be used with a better thought out
> implementation, the performance implications aren't fully clear to me
> yet.
>
> For other uses, interrupts are probably desired.

The NFS client's RPC code hooks into the data_ready callback,
tries to find the task that the reply is going to, and copies the data.
So per your argument above, for the NFS client it should be okay to
busy wait after switching to the task.

This would leave the NFS server, which basically uses some kind
of poll/recvmsg scheme. In essence, it doesn't call recvmsg until
the data has been queued. I guess this kind of code would benefit
from using async IO - then the stack can DMA directly into the
buffer and then wake up the process.

> It's just not there now, but it can be added easily, it's one bit in
> the descriptor and a register read in the interrupt handler to see
> which channel(s) need attention.

Hm, okay.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir@lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax