Re: [PATCH v11 00/11] iommu/arm-smmu-v3: Adopt the crashed kernel's stream table for kdump

[Shuai Xue <xueshuai@linux.alibaba.com>]

On 10/6/26 3:30 AM, Nicolin Chen wrote:
> When transitioning to a kdump kernel, the primary kernel might have crashed
> while endpoint devices were actively bus-mastering DMA. Currently, the SMMU
> driver aggressively resets the hardware during probe by clearing CR0_SMMUEN
> and setting the Global Bypass Attribute (GBPA) to ABORT.
> 
> In a kdump scenario, this aggressive reset is highly destructive:
> a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal
>     PCIe AER or SErrors that may panic the kdump kernel
> b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass
>     the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.
> 
> To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact
> and avoid modifying STRTAB_BASE. This allows HW to continue translating in-
> flight DMA using the crashed kernel's page tables until the endpoint device
> drivers probe and quiesce their respective hardware.
> 
> However, the ARM SMMUv3 architecture specification states that updating the
> SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.
> 
> This leaves a kdump kernel no choice but to adopt the stream table from the
> crashed kernel.
> 
> In this series:
>   - Introduce an ARM_SMMU_OPT_KDUMP_ADOPT
>   - Skip SMMUEN and STRTAB_BASE resets in arm_smmu_device_reset()
>   - Skip EVENTQ/PRIQ setup including interrupts and their handlers
>   - Memremap the crashed kernel's stream tables into the kdump kernel [*]
>   - Reserve the crashed kernel's in-use ASIDs and VMIDs, preventing any TLB
>     aliasing with the kdump kernel's own domains
>   - Defer any default domain attachment to retain STEs until device drivers
>     explicitly request it.
> Most of the new code is in arm-smmu-v3-kexec.c. A hidden ARM_SMMU_V3_KEXEC
> config symbol (def_bool CRASH_DUMP) controls its build.
> 
> The common helpers parse and walk the crashed kernel's stream/CD tables and
> reserve its in-use IDs. The kdump code is guarded by CONFIG_CRASH_DUMP. The
> helpers can also support the proposed SMMUv3 Live Update work:
> https://lore.kernel.org/all/20260929071950.2710070-1-praan@google.com/
> 
> [*] For verification reasons, this series only fixes coherent SMMUs.
> 
> For non-ARM_SMMU_OPT_KDUMP_ADOPT cases, keep a status quo since the commit
> 3f54c447df34f ("iommu/arm-smmu-v3: Don't disable SMMU in kdump kernel"):
> full reset followed by driver-initiated reattach, potentially rejecting any
> in-flight DMA.
> 
> Note that this series is no longer treated as a bug fix, since it has grown
> fairly big and most of the kdump code now resides in a separate C file. For
> folks interested in back-porting the change: a v6.12+ kernel (since commit
> 85196f54743d ("iommu/arm-smmu-v3: Reorganize struct arm_smmu_strtab_cfg"))
> would be still compatible with this series.
> 
> This is on Github:
> https://github.com/nicolinc/iommufd/commits/smmuv3_kdump-v11
> 
> Changelog
> v11
>   * Rebase on arm/smmu/updates branch
>   * Add Tested-by from Breno and Cristian
>   * Add Reviewed/Acked-by from Jason and Kiryl
>   * Update commit messages and Assisted-by tags
>   * New prep patch: swap EVTQ/GERROR MSI indexes
>   * Skip EVTQ's CFG0 write and vector allocation
>   * Merge arm-smmu-v3-kdump code into arm-smmu-v3-kexec
>   * Fold patches accordingly as static functions require callers
Hi Nicolin,

We tested this series (v11, backported onto our 6.6-based kernel) on two
arm64 server platforms that differ in how PCIe errors are reported: one
with firmware-first reporting (GHES/CPER), the other with OS-first AER
plus DPC. The test triggers a kdump crash (echo c > /proc/sysrq-trigger)
on an otherwise idle machine and compares the capture kernel's dmesg
between the unpatched baseline and the kernel carrying this series.

On the firmware-first platform (10 SMMUv3 instances):

   Unpatched, every SMMU logs:

     arm-smmu-v3.0.auto: SMMU currently enabled! Resetting...

   and the destructive reset immediately aborts in-flight DMA. The capture
   kernel then drowns in a storm of GHES/CPER records together with PCIe
   AER "Poisoned TLP" (Data Link Layer) errors from the devices that were
   bus-mastering at crash time. Devices behind the reset SMMUs fail to
   recover in the capture kernel, and in some runs the capture kernel
   hangs in the error storm instead of saving the vmcore.

   With the series applied, all 10 SMMUs log:

     arm-smmu-v3.0.auto: kdump: retaining SMMUEN for in-flight DMA

   and the capture kernel boots with zero AER/GHES error records. A
   complete vmcore is saved and the machine reboots normally.

On the OS-first AER/DPC platform (8 SMMUv3 instances):

   Unpatched: 8x "SMMU currently enabled! Resetting..." followed within
   seconds by PCIe AER Completion Abort errors reported by the root
   complex, along with DPC containment events and EDR notifications. The
   NICs behind the contained link never recover in the capture kernel.

   With the series applied: 8x "kdump: retaining SMMUEN for in-flight DMA",
   no PCIe error events at all, a complete vmcore and a clean reboot.

The baseline symptom is load-independent: it reproduces on idle machines,
since even an idle server has periodic DMA in flight (NIC rings, NVMe
admin queues, etc.).

Overall, on both platforms this series turns a capture-kernel boot that
used to be buried under fatal PCIe error storms into a quiet one that
reliably produces a usable vmcore.

Please feel free to add:

Tested-by: Shuai Xue <xueshuai@linux.alibaba.com>

Thanks,
Shuai Xue