Re: [PATCH v2] powerpc/pseries/iommu: switch to Default DMA window during kdump

Re: [PATCH v2] powerpc/pseries/iommu: switch to Default DMA window during kdump

[Ritesh Harjani]

Gaurav Batra <gbatra@linux.ibm.com> writes:

> In PowerPC (pseries) a non-virtualized adapter will have 2 DMA windows -
> 2GB default and a larger Dynamic DMA Window (DDW). DDW is large enough to
> map total RAM to a device.
>
> During normal functioning of OS, since RAM is pre-mapped, 2GB default
> window is not used. The only scenario it might get used is when buffers in
> pmemory are mapped to the device for DMA.
>
> As of today, during kdump, during early device discovery, pci_dma_find()
> finds that the device has 2 DMA windows. It selects to use DDW. This is a
> kdump path and DMA window is needed for IO to the device.
>
> Although commit 09a3c1e46142 ("powerpc/pseries/iommu: IOMMU table is not
> initialized for kdump over SR-IOV") fixed an issue during kdump with SR-IOV
> case, but this also made the kdump prefer DDW over the default DMA window
> when both are present (dedicated adapter case). Since the DDW is fully
> mapped by the previous kernel, iommu_table_clear() can free only
> KDUMP_MIN_TCE_ENTRIES (2048) TCEs for use by kdump kernel.
>
> This is not enough when the dump device is NVMe over Fibre Channel.
> Because nvme-fc driver DMA-maps the cmds and resp IUs of every
> pre-allocated request and each such mapping consumes roughly:
>
>     32 (IO queues, one per cpus = nr_cpus) *
>     64 (queue_depth, blk-mq kdump limit) *
>     2 (cmd+resp) = 4096
>
> This is already double of what we have without counting admin queues and
> lpfc driver's own allocations / mapping requirement. Hence this results
> into iommu_alloc failures like -
>
> lpfc 0153:70:00.0: iommu_alloc failed,
> tbl 0000000034ebcf5e vaddr 00000000d814df0b npages 1
> lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
> lpfc 0153:70:00.0: iommu_alloc failed,
> tbl 0000000034ebcf5e vaddr 000000009779e4d2 npages 1
> lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
>
> iommu_map_phys+0x1c4/0x1f0 (unreliable)
> dma_iommu_map_phys+0x54/0xa0
> dma_map_phys+0x3f8/0x590
> __nvme_fc_init_request+0x110/0x300 [nvme_fc]
> nvme_fc_init_request+0x60/0xb8 [nvme_fc]
> blk_mq_alloc_map_and_rqs+0x388/0x510
> blk_mq_alloc_tag_set+0x2a4/0x5f0
> nvme_alloc_io_tag_set+0xe0/0x1e0 [nvme_core]
> nvme_fc_connect_ctrl_work+0x85c/0xdac [nvme_fc]
> process_one_work+0x1e4/0x5a0
> worker_thread+0x1ec/0x3e0
>
> Increasing the number of free TCE entries in iommu_table_clear() will
> increase the probability of hitting EEH since there could still be some
> active IOs from the previous life of the kernel.
>
> Hence this patch partially reverts the previous fixes commit and
> switches the kdump's default back to 2GB default DMA window instead of
> DDW window. This window will mostly be empty. Or, could be slightly used
> if buffers in pmemory were mapped for IO.

BTW w/o this fix the only TCE window that kdump & kexec uses is DDW
which won't work for 32-bit capable DMA devices correct? So I think this
should also fix the kexec & kdump case where we have a 32-bit DMA device
- which otherwise will fail to function after kexec with the following
error:

"Warning: IOMMU offset too big for device mask"
"mask: 0xffffffff, table offset: 0x800000000000000"

Thoughts?

>
> Fixes: 09a3c1e46142 ("powerpc/pseries/iommu: IOMMU table is not initialized for kdump over SR-IOV")
> Cc: stable@vger.kernel.org
> Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com>
> Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>