5.10 breaks rxe

5.10 breaks rxe

[Bob Pearson]

Jason,

Looking at the current tree post 5.10.0-rc1 rxe is pretty broken.

The first problem I have run into is caused by the following recent patch:

   commit f959dcd6ddfd29235030e8026471ac1b022ad2b0
   Author: Thomas Tai <thomas.tai@oracle.com>
   Date:   Thu Sep 17 18:43:03 2020 +0200

       dma-direct: Fix potential NULL pointer dereference
    
       When booting the kernel v5.9-rc4 on a VM, the kernel would panic when
       printing a warning message in swiotlb_map(). The dev->dma_mask must not
       be a NULL pointer when calling the dma mapping layer. A NULL pointer
       check can potentially avoid the panic.
    
       Signed-off-by: Thomas Tai <thomas.tai@oracle.com>
       Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
       Signed-off-by: Christoph Hellwig <hch@lst.de>

which causes a warning if ib_device->dma_device->dma_mask is NULL which it is at the moment for rxe.

But, rxe never does any DMA. It only copies data in software. It does however sometimes do this copy at interrupt level and will fail badly if the users memory is paged out. Rxe just follows the HW rdma model and pins MRs which solves this problem. The original rxe code does packet processing in tasklets which are software interrupts or on the NAPI thread which is also an interrupt.

Rxe calls ib_register_device() with a NULL 3rd argument which is supported. But in this case, ib_register_device just sets dma_device to point at &ib_device->dev which is sort of a dummy device and never sets dma_mask to anything. This now fails because of the above patch.

I can think of three approaches to fixing this problem:

1. Fix ib_register_device to correctly handle a NULL dma_device argument (if anyone besides rxe uses it)
2. Fix rxe to provide a valid dma_device. Currently rxe assumes that ib_device->dev will be it and sets dev.parent to point at the dma_device behind the network device that has rxe installed on it.
3. (Much harder) Strip out all the memory pinning and DMA support and perform the copies in work queues which can sleep. The problem is that rdma code basically assumes that DMA is always there.

This may be easier than I am seeing. Let me know what you suggest.

Thanks,

Bob

Re: 5.10 breaks rxe

[Bob Pearson]

On 10/29/20 4:08 PM, Jason Gunthorpe wrote:
> Parav will fix it, see the note from Christoph about using dma coerce and mask
> 
> Get Outlook for Android<https://aka.ms/ghei36>
> 

I went ahead and submitted a one line patch that fixes it in rxe. Could do same thing in core.

> ________________________________
> From: Bob Pearson <rpearsonhpe@gmail.com>
> Sent: Thursday, October 29, 2020 5:37:10 PM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: linux-rdma@vger.kernel.org <linux-rdma@vger.kernel.org>
> Subject: 5.10 breaks rxe
> 
> Jason,
> 
> Looking at the current tree post 5.10.0-rc1 rxe is pretty broken.
> 
> The first problem I have run into is caused by the following recent patch:
> 
>    commit f959dcd6ddfd29235030e8026471ac1b022ad2b0
>    Author: Thomas Tai <thomas.tai@oracle.com>
>    Date:   Thu Sep 17 18:43:03 2020 +0200
> 
>        dma-direct: Fix potential NULL pointer dereference
> 
>        When booting the kernel v5.9-rc4 on a VM, the kernel would panic when
>        printing a warning message in swiotlb_map(). The dev->dma_mask must not
>        be a NULL pointer when calling the dma mapping layer. A NULL pointer
>        check can potentially avoid the panic.
> 
>        Signed-off-by: Thomas Tai <thomas.tai@oracle.com>
>        Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>        Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> which causes a warning if ib_device->dma_device->dma_mask is NULL which it is at the moment for rxe.
> 
> But, rxe never does any DMA. It only copies data in software. It does however sometimes do this copy at interrupt level and will fail badly if the users memory is paged out. Rxe just follows the HW rdma model and pins MRs which solves this problem. The original rxe code does packet processing in tasklets which are software interrupts or on the NAPI thread which is also an interrupt.
> 
> Rxe calls ib_register_device() with a NULL 3rd argument which is supported. But in this case, ib_register_device just sets dma_device to point at &ib_device->dev which is sort of a dummy device and never sets dma_mask to anything. This now fails because of the above patch.
> 
> I can think of three approaches to fixing this problem:
> 
> 1. Fix ib_register_device to correctly handle a NULL dma_device argument (if anyone besides rxe uses it)
> 2. Fix rxe to provide a valid dma_device. Currently rxe assumes that ib_device->dev will be it and sets dev.parent to point at the dma_device behind the network device that has rxe installed on it.
> 3. (Much harder) Strip out all the memory pinning and DMA support and perform the copies in work queues which can sleep. The problem is that rdma code basically assumes that DMA is always there.
> 
> This may be easier than I am seeing. Let me know what you suggest.
> 
> Thanks,
> 
> Bob
>