Re: [PATCH] powerpc/pseries/iommu: switch to Default DMA window during kdump

[Gaurav Batra <gbatra@linux.ibm.com>]

Hello Ritesh,

Thanks a lot for reviewing this patch. My responses are inline

On 7/30/26 10:09 AM, Ritesh Harjani (IBM) wrote:
> Hi Gaurav,
>
> Thanks for the patch. Few observation, inputs and queries -
>
> Gaurav Batra <gbatra@linux.ibm.com> writes:
>
>> In PowerPC (pseries) a non-virtualized adapter will have 2 DMA windows -
>> 2GB default and a larger Dynamic DMA Window (DDW). DDW is large enough to
>> map total RAM to a device.
>>
>> During normal functioning of OS, since RAM is pre-mapped, 2GB default
>> window is not used. The only scenario it might get used is when buffers in
>> pmemory are mapped to the device for DMA.
>>
>> As of today, during kdump, during early device discovery, pci_dma_find()
>> finds that the device has 2 DMA windows. It selects to use DDW. This is a
>> kdump path and DMA window is needed for IO to the device.
>>
>> Since, in the previous life of the LPAR (before panic), RAM was pre-mapped
>> via DDW, the DDW is completely full. So, in the iommu table initialization
>> code, iommu_table_clear() frees KDUMP_MIN_TCE_ENTRIES (2K) number of
>> TCEs.
>>
>> But, it seems these are not enough for NVMe over Fibre-channel. When kdump
>> is trying to save vmcore on storage device, which is NVMe-FC, the
>> TCE usage is much more than 2K number of entries. The driver is mapping
>> a lot more buffers for DMA. After all the TCEs are consumed, iommu returns
>> iommu_alloc failures and the driver is not able to further map buffers for
>> IO. kdump fails to copy vmcore to NVMe-FC storage device.
> Some context I collected while reviewing this patch -
>
> 1. kdump environment generally prefers configurations so that we could
>     avoid issues like memory allocation failures. I guess, we don't want to
>     be running the system with max configurations - that is also the reason
>     why distros keep nr_cpus to a lower value during kdump case.
>     For e.g. see this [1]:
>          scsi: lpfc: Limit xri count for kdump environment
>
>          scsi-mq operation inherently performs pre-allocation of resources for
>          blk-mq request queues. Even though the kdump environment reduces the
>          configuration to a single CPU, thus 1 hardware queue, which helps
>          significantly, the resources are still rather large due to the per request
>          allocations.
>          <...>
>     [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=31f06d2e73726160645f8d9976a0b3f42e136da7
>
> 2. While looking more into this call stack path I also saw -
>              /*
>              * If a crashdump is active, then we are potentially in a very
>          -	 * memory constrained environment. Limit us to 1 queue and
>          -	 * 64 tags to prevent using too much memory.
>          +	 * memory constrained environment. Limit us to  64 tags to prevent
>          +	 * using too much memory.
>              */
>          -	if (is_kdump_kernel()) {
>          -		set->nr_hw_queues = 1;
>          -		set->nr_maps = 1;
>          +	if (is_kdump_kernel())
>                  set->queue_depth = min(64U, set->queue_depth);
>          -	}
>          +
>          @@ -4515,7 +4513,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
>                                  GFP_KERNEL, set->numa_node);
>                  if (!set->map[i].mq_map)
>                      goto out_free_mq_map;
>          -		set->map[i].nr_queues = is_kdump_kernel() ? 1 : set->nr_hw_queues;
>          +		set->map[i].nr_queues = set->nr_hw_queues;
>              }
>
>              blk_mq_update_queue_map(set);
>
>     So looks like we already reduce blk-mq queue_depth to 64 in
>     blk_mq_alloc_tag_set(), no matter what the queue_depth is passed to
>     us by the driver.
>
> 3. Also with above patch from v6.9 onwards, we made nr_queues as
>     set->nr_hw_queues, whereas earlier it was clamped to 1. Because the
>     code assumes that in kdump kernel we boot with nr_cpus=1..
>          blk-mq: don't change nr_hw_queues and nr_maps for kdump kernel
>
>          For most of ARCHs, 'nr_cpus=1' is passed for kdump kernel, so
>          nr_hw_queues for each mapping is supposed to be 1 already.
>
>          More importantly, this way may cause trouble for driver, because blk-mq and
>          driver see different queue mapping since driver should setup hardware
>          queue setting before calling into allocating blk-mq tagset.
>
>          So not overriding nr_hw_queues and nr_maps for kdump kernel.
>
>     [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ec30b461f3d067bd322a6c3a6c5105746ed9bf14

I agree with your assessment that kdump executes in a very limited 
environment

and drivers make an effort to use less resources. The goal is to dump 
memory and

not performance.

>
> So looking at above, I had few questions -
> 1. In your kdump kernel how many nr_cpus you are booting up with?
>     Recently what I heard rhel/sles might be using nr_cpus=16/32. @Sourabh?
>     So the calculation in nvme-fc driver then becomes:
>       32(nr_cpus) * 64(queue_depth) * 2(cmd+resp) = 4096
>
>    If you are able to reproduce this issue 100% of the time - then can
>    you try kdump with nr_cpus=1 and see whether it fixes your iommu alloc
>    failure?

Initially, when I reproduced this issue, I was not passing any value to

nr_cpus to the kdump kernel. I made changes to /etc/sysconfig/kdump to pass

nr_cpus=1. With this the kdump was successful. I tried with 
nr_cpus=16/32. These

were successful as well. Though, in these cases, I did notice a few 
iommu_alloc failures

(maybe < 10), vmcore was gathered successfully.

I started to see the issue with nr_cpus=64. My LPAR is configured with 
max cpus = 64. So,

earlier, when I was not specifying nr_cpus in the /etc/sysconfig/kdump, 
kdump could be

defaulting to 64 CPUs and hence allocating more resources during kdump.

> 2. Also were there more than 1 controller attached? Which can change the
>    above calculation then.

Only 1 controller. Here is the output of lscpi

ltcd41-lp11:~ # lspci
0153:70:00.0 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series 
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.1 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series 
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.2 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series 
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.3 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series 
32Gb/64Gb Fibre Channel Adapter (rev 10)

>
> Looking at the lpfc and nvme-fc driver - a lot of the calculation are
> based on nr_online_cpus. I somehow think if we clamp that value of
> nr_online_cpus, we should stop seeing these alloc failures.
> Thoughts?
>
> Hopefully, if you can work on some above points further, it will also
> explain why are we seeing this failures only now. Is this something that
> has caused an issue after RHEL/SLES moved to nr_cpus=16/32?

it seems this got exposed because in my test LPAR, nr_cpus=1/16/32 was not

getting passed to the kdump kernel.

> Or was it after this commit from v6.9?
>     ec30b461f3d: ("blk-mq: don't change nr_hw_queues and nr_maps for kdump kernel")
>
>> Here are the driver logs and stack
>>
>> lpfc 0153:70:00.0: iommu_alloc failed,
>> tbl 0000000034ebcf5e vaddr 00000000d814df0b npages 1
>> lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
>> lpfc 0153:70:00.0: iommu_alloc failed,
>> tbl 0000000034ebcf5e vaddr 000000009779e4d2 npages 1
>> lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
>>
>> iommu_map_phys+0x1c4/0x1f0 (unreliable)
>> dma_iommu_map_phys+0x54/0xa0
>> dma_map_phys+0x3f8/0x590
>> __nvme_fc_init_request+0x110/0x300 [nvme_fc]
>> nvme_fc_init_request+0x60/0xb8 [nvme_fc]
>> blk_mq_alloc_map_and_rqs+0x388/0x510
>> blk_mq_alloc_tag_set+0x2a4/0x5f0
>> nvme_alloc_io_tag_set+0xe0/0x1e0 [nvme_core]
>> nvme_fc_connect_ctrl_work+0x85c/0xdac [nvme_fc]
>> process_one_work+0x1e4/0x5a0
>> worker_thread+0x1ec/0x3e0
>>
>> Increasing the number of free TCE entries in iommu_table_clear() will
>> increase the probability of hitting EEH since there could still be some
>> active IOs from the previous life of the kernel.
>>
>> Instead, during kdump, we can switch to default 2GB DMA window. This window
>> will mostly be empty. Or, could be slightly used if buffers in pmemory
>> were mapped for IO.
>>
> This will still remain a problem when we have SR-IOV adapter attached
> correct? Because in that case we only get 1 window, so we anyway can't
> use default window in kdump case. Correct?

you are right. The patch is fixing the dedicated adapter path only by 
switching to

default window for kdump. Before I submitted the patch, I did try SR-IOV 
path as well.

Here, I assigned a virtualized adapter to LPAR and gathered kdump over 
NFS. I checked the

footprint of DMA buffers in this path. They were not much. I think, I 
did sent these details

in my emails (the discussion/advice).

As of now SR-IOV path doesn't seems to be of concern. But, I think, the 
correct overall fix

should be to maintain the DDW state --> if it is pre-mapped DDW, 
transfer this knowledge to kdump.

With this, the DDW will be intact and buffers pre-mapped, as before. 
But, this requires more work

and thorough testing by FVT/ISST. So, I kept this for later. For now, 
switching to default window

seems to me the least invasive fix for this very narrow problem.

Your insight and thoughts?


Thanks a lot

Gaurav

>
> -ritesh
>