From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6FBE28C860 for ; Fri, 2 Jan 2026 18:41:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767379278; cv=none; b=EiIJtA3H5UV/6vlf6923NmM0CUn2rR/+krCMaR30NzhuSqAY8K/zKxgNsvCROzsh2Xh/slqDtk6Ty4WIe9tFe+8LlA8xWHtUBSyutjm8a7gU7tPFrn+oezWOcqqeg2yzRz+EcH/stPIn2CbUZc9o/1YrvN31E781MS72YCFRhDs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767379278; c=relaxed/simple; bh=bXmVqZi5/XxGwHoRNC2qo8trHPL/JUWYt6dxPjrbbI4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Lvzi89+aVdg104llDMjsl52RhGGSJ+3f1Y/N1DIKW/zebZxkwuaL2T/xy3xRKqvtyavs6Mz4Ek3Ye3kpAIxB9RXUGo/hNtRoQTbWkXHAp2GuwXB5NwwyFxiwwDcRYRNHAYGjXxswXJqKkmT1TG6iIiWcGeoIfmtJ/+fBBKjeg6Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=XEm80rH/; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="XEm80rH/" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-8b2d7c38352so17697185a.0 for ; Fri, 02 Jan 2026 10:41:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1767379275; x=1767984075; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZbKNq3OzvNgOXwfOYEfmEuaPS05vXN54NlE6TZcUdp4=; b=XEm80rH/f7ajg/ABVoHYzDA8sIgm6xlGcxnhG2f0hdkom1SGb0cE0nkVasqgIe0B0+ BIFw9IbtoYI6qODNUx3EfM0fhXr39MO9/j+p09Jey7uVqee2lZm2Ubc9BsNs6z06BcPf RsKl3Rv9um4zoMQD111N6s71gm+duh4XTb1d/oupq1ClwBFjU+7CE82qHQDIv5nqIWYq OW8SR16+NPp5T0+3JfccAS/MQAiY6jHaqYOK3icGgbwILEjjmfHFNJKxd4NAa6ldsvrK Z7DMTGGohcmCMXp8wbb2PyX9BRqPoDSZMPGgUkTL7Nv7hrJfAd6ddoKbSxe2l9kxLCYR xXyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767379275; x=1767984075; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZbKNq3OzvNgOXwfOYEfmEuaPS05vXN54NlE6TZcUdp4=; b=svMozv/VkEXmN/kqvNunlL+viERJhZBG4CFBKqrO5pTQupgk74mtI5eYy94Gi8jRQY WHZbwZWt8cJEF57HP9f1YlpzZqK8nAQ2B30SY9dhbZ9ESlvO9y713xn/dyBGlGeWd4lf qlz7ZhUPFqPyJdp5EEtYfOeCFdu6HPqb2SSpztv6LUyt5S1o+H8WmjtZuGJ9XO+NIdtr 1w3sFXzj8gQvP2eyq3m7YiaxHMESOAnl0fSLeDAzMlaZc+Z2lA9yI5GrBqns9sYRcJso PgO3vO2BINojRQQje88/mszCd5fsfmQoDvftf4duGnJ0ouonJJCv1Q1lFB0ndy+6ctve XA6A== X-Forwarded-Encrypted: i=1; AJvYcCVo21WBzSng+8cWM2rG/Qql13BYIzHMBQb/VYRQw0ySYHleilclUB8F5Sh8im4IXD3SBBplm7FTmFyAXiE=@vger.kernel.org X-Gm-Message-State: AOJu0YzJDqzbLaS5xkgqYxXF2vzqVWh80cS0MNxsENWnnIXeTSmfQjH1 UNJag9iRG4NQUK1kIw1qIBAQEmkOBBe7wJP3J0HrSKg9Q5oTaHCiqPho2QsCJ3l7CVyJuSMHvhT TuxcC X-Gm-Gg: AY/fxX5sLjsoHkFnWpZy/hydIK1yvwstfUNmPp0YPFL1pUKBbTQM4TjHSHyJrFcaOi6 sp9qugAED3w5SDgyAShaV8ncCp3s8w9k5aAHvAUvxw3NNzcDgiHFN+TaKSj28QgEMp5mDTuTygg 6uAaWThXAEi2IBJmUpJpNORx49Phlsppyta/hZXIx12NXe9jlTyowuT7rQLnapcijXJmPbQAO55 nd3uQ7ghb2kCNojOoGpT+2Jlx8hsAN/Bl0legkVIzmHeOvXLBCc5jOf3rBs2eA/oZq5/NBNHJqF UwwOvlRcyobF3Q78tWhMjzTKxH2oGQUCFjjqz+CgwprHbj0SXJvQzaZ4uC5zU3/6w5XioCIyNPl trbJ3NPb2fzWq5JoUiMapi1qKb7wCvhZs+6B/TTEtYSfP+Ojby/9FPaEnqLf2pO+6QxsZKANhoW fSr8QqQoR+dd2NvXIEiYyAwu+m1uu9BcbKgFyVBoIQY1SAGCmHUrlgOY/a10DKW990XlM= X-Google-Smtp-Source: AGHT+IFf1uKGIwRHqgDuPoJHVm8is9Z+z+ElqpNBRFLqGHyGLWLw+ECPqzKaTxkx0tZNT3guYluMgA== X-Received: by 2002:a05:620a:7111:b0:8a3:90cb:9224 with SMTP id af79cd13be357-8c3568a0131mr102879985a.2.1767379274814; Fri, 02 Jan 2026 10:41:14 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-162-112-119.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.112.119]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8c09678afa6sm3293504685a.2.2026.01.02.10.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jan 2026 10:41:14 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1vbk5J-00000000neJ-2b6O; Fri, 02 Jan 2026 14:41:13 -0400 Date: Fri, 2 Jan 2026 14:41:13 -0400 From: Jason Gunthorpe To: Dawei Li Cc: will@kernel.org, robin.murphy@arm.com, joro@8bytes.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, set_pte_at@outlook.com, stable@vger.kernel.org Subject: Re: [PATCH] iommu/arm-smmu-v3: Maintain valid access attributes for non-coherent SMMU Message-ID: <20260102184113.GA125261@ziepe.ca> References: <20251229002354.162872-1-dawei.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20251229002354.162872-1-dawei.li@linux.dev> On Mon, Dec 29, 2025 at 08:23:54AM +0800, Dawei Li wrote: > According to SMMUv3 architecture specification, IO-coherent access for > SMMU is supported for: > - Translation table walks. > - Fetches of L1STD, STE, L1CD and CD. > - Command queue, Event queue and PRI queue access. > - GERROR, CMD_SYNC, Event queue and PRI queue MSIs, if supported I was recently looking at this too.. IMHO this is not really a clean description of what this patch is doing. I would write this description as: When the SMMU does a DMA for itself it can set various memory access attributes which control how the interconnect should execute the DMA. Linux uses these to differentiate DMA that must snoop the cache and DMA that must bypass it because Linux has allocated non-coherent on the CPU. In Table "13.8 Attributes for SMMU-originated accesses" each of the different types of DMA is categorized and the specific bits controlling the memory attribute for the fetch are identified. Make this consisent globally. If Linux has cache flushed the buffer, or allocated a DMA incoherenet buffer, then it should set the non-caching memory attribute so the DMA matches. This is important for some of the allocations where Linux is currently allocating DMA coherent memory, meaning nothing has made the CPU cache coherent and doing any coherent access to that memory may result in cache inconsistencies. This may solve problems in systems where the SMMU driver thinks the SMMU is non-coherent, but in fact, the SMMU and the interconnect selectively supports coherence and setting the wrong memory attributes will cause non-working cached access. [and then if you have a specific SOC that shows an issue please describe the HW] > +static __always_inline bool smmu_coherent(struct arm_smmu_device *smmu) > +{ > + return !!(smmu->features & ARM_SMMU_FEAT_COHERENCY); > +} > + > /* High-level queue accessors */ > -static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) > +static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent, > + struct arm_smmu_device *smmu) > { > memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); > cmd[0] |= FIELD_PREP(CMDQ_0_OP, ent->opcode); > @@ -358,8 +364,13 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) > } else { > cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); > } > - cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); > - cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); > + if (smmu_coherent(smmu)) { > + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); > + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); > + } else { > + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_OSH); > + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OINC); > + } And then please go through your patch and add comments actually explaining what the DMA is and what memory is being reached by it - since it is not always very clear from the ARM mnemonics For instance, this is: /* DMA for "CMDQ MSI" which targets q->base_dma allocated by arm_smmu_init_one_queue() */ > @@ -1612,11 +1624,18 @@ void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, > (cd_table->cdtab_dma & STRTAB_STE_0_S1CTXPTR_MASK) | > FIELD_PREP(STRTAB_STE_0_S1CDMAX, cd_table->s1cdmax)); > > + if (smmu_coherent(smmu)) { > + val = FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | > + FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | > + FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH); > + } else { > + val = FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_NC) | > + FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_NC) | > + FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_OSH); > + } This one is "CD fetch" allocated by arm_smmu_alloc_cd_ptr() etc And note that the above will need this hunk too: +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c @@ -432,6 +432,14 @@ size_t arm_smmu_get_viommu_size(struct device *dev, !(smmu->features & ARM_SMMU_FEAT_S2FWB)) return 0; + /* + * When running non-coherent we can't suppot S2FWB since it will also + * force a coherent CD fetch, aside from the question of what + * S2FWB/CANWBS even does with non-coherent SMMUs. + */ + if (!smmu_coherent(smmu)) + return 0; > @@ -3746,7 +3765,7 @@ int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, > q->cons_reg = page + cons_off; > q->ent_dwords = dwords; > > - q->q_base = Q_BASE_RWA; > + q->q_base = smmu_coherent(smmu) ? Q_BASE_RWA : 0; CMDQ fetch, though do we even need to manage RWA? Isn't it ignored if IC/OC/SH are set to their non-cachable values? etc.. Jason