[PATCH] powerpc/iommu: Do not do platform domain attach atctions after probe

[PATCH] powerpc/iommu: Do not do platform domain attach atctions after probe

[Jason Gunthorpe]

POWER throws a splat at boot, it looks like the DMA ops were probably
changed while a driver was attached. Something is still weird about how
power sequences its bootup. Previously this was hidden since the core
iommu code did nothing during probe, now it calls
spapr_tce_platform_iommu_attach_dev().

Make spapr_tce_platform_iommu_attach_dev() do nothing on the probe time
call like it did before.

  WARNING: CPU: 0 PID: 8 at arch/powerpc/kernel/iommu.c:407 __iommu_free+0x1e4/0x1f0
  Modules linked in: sd_mod t10_pi crc64_rocksoft crc64 sg ibmvfc mlx5_core(+) scsi_transport_fc ibmveth mlxfw psample dm_multipath dm_mirror dm_region_hash dm_log dm_mod fuse
  CPU: 0 PID: 8 Comm: kworker/0:0 Not tainted 6.6.0-rc3-next-20230929-auto #1
  Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1030.30 (NH1030_062) hv:phyp pSeries
  Workqueue: events work_for_cpu_fn
  NIP:  c00000000005f6d4 LR: c00000000005f6d0 CTR: 00000000005ca81c
  REGS: c000000003a27890 TRAP: 0700   Not tainted (6.6.0-rc3-next-20230929-auto)
  MSR:  800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 48000824  XER: 00000008
  CFAR: c00000000020f738 IRQMASK: 0
  GPR00: c00000000005f6d0 c000000003a27b30 c000000001481800 000000000000017
  GPR04: 00000000ffff7fff c000000003a27950 c000000003a27948 0000000000000027
  GPR08: c000000c18c07c10 0000000000000001 0000000000000027 c000000002ac8a08
  GPR12: 0000000000000000 c000000002ff0000 c00000000019cc88 c000000003042300
  GPR16: 0000000000000000 0000000000000000 0000000000000000 c000000003071ab0
  GPR20: c00000000349f80d c000000003215440 c000000003215480 61c8864680b583eb
  GPR24: 0000000000000000 000000007fffffff 0800000020000000 0000000000000010
  GPR28: 0000000000020000 0000800000020000 c00000000c5dc800 c00000000c5dc880
  NIP [c00000000005f6d4] __iommu_free+0x1e4/0x1f0
  LR [c00000000005f6d0] __iommu_free+0x1e0/0x1f0
  Call Trace:
  [c000000003a27b30] [c00000000005f6d0] __iommu_free+0x1e0/0x1f0 (unreliable)
  [c000000003a27bc0] [c00000000005f848] iommu_free+0x28/0x70
  [c000000003a27bf0] [c000000000061518] iommu_free_coherent+0x68/0xa0
  [c000000003a27c20] [c00000000005e8d4] dma_iommu_free_coherent+0x24/0x40
  [c000000003a27c40] [c00000000024698c] dma_free_attrs+0x10c/0x140
  [c000000003a27c90] [c008000000dcb8d4] mlx5_cmd_cleanup+0x5c/0x90 [mlx5_core]
  [c000000003a27cc0] [c008000000dc45a0] mlx5_mdev_uninit+0xc8/0x100 [mlx5_core]
  [c000000003a27d00] [c008000000dc4ac4] probe_one+0x3ec/0x530 [mlx5_core]
  [c000000003a27d90] [c0000000008c5edc] local_pci_probe+0x6c/0x110
  [c000000003a27e10] [c000000000189c98] work_for_cpu_fn+0x38/0x60
  [c000000003a27e40] [c00000000018d1d0] process_scheduled_works+0x230/0x4f0
  [c000000003a27f10] [c00000000018ff14] worker_thread+0x1e4/0x500
  [c000000003a27f90] [c00000000019cdb8] kthread+0x138/0x140
  [c000000003a27fe0] [c00000000000df98] start_kernel_thread+0x14/0x18
  Code: 481b004d 60000000 e89e0028 3c62ffe0 3863dd20 481b0039 60000000 e89e0038 3c62ffe0 3863dd38 481b0025 60000000 <0fe00000> 4bffff20 60000000 3c4c0142
  ---[ end trace 0000000000000000 ]---
  iommu_free: invalid entry
      entry     = 0x8000000203d0
      dma_addr  = 0x8000000203d0000
      Table     = 0xc00000000c5dc800
      bus#      = 0x1
      size      = 0x20000
      startOff  = 0x800000000000
      index     = 0x70200016

Fixes: 2ad56efa80db ("powerpc/iommu: Setup a default domain and remove set_platform_dma_ops")
Reported-by: Tasmiya Nalatwad <tasmiya@linux.vnet.ibm.com>
Link: https://lore.kernel.org/r/d06cee81-c47f-9d62-dfc6-4c77b60058db@linux.vnet.ibm.com
Tested-by: Tasmiya Nalatwad <tasmiya@linux.vnet.ibm.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 arch/powerpc/kernel/iommu.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

Joerg, this would need to go to your tree, thanks

diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index d6ad3fde85a212..bf19932147517b 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -1280,13 +1280,19 @@ struct iommu_table_group_ops spapr_tce_table_group_ops = {
 /*
  * A simple iommu_ops to allow less cruft in generic VFIO code.
  */
-static int spapr_tce_platform_iommu_attach_dev(struct iommu_domain *dom,
-					       struct device *dev)
+static int
+spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain,
+				    struct device *dev)
 {
+	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
 	struct iommu_group *grp = iommu_group_get(dev);
 	struct iommu_table_group *table_group;
 	int ret = -EINVAL;
 
+	/* At first attach the ownership is already set */
+	if (!domain)
+		return 0;
+
 	if (!grp)
 		return -ENODEV;
 

base-commit: 20700abdf2393c8c0bd3a100e3111d0fcd3d5bf3
-- 
2.42.0

Re: [PATCH] powerpc/iommu: Do not do platform domain attach atctions after probe

[Joerg Roedel]

On Thu, Oct 05, 2023 at 10:35:11AM -0300, Jason Gunthorpe wrote:
>  arch/powerpc/kernel/iommu.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)

Applied, thanks.

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman