Re: [RFC PATCH v1 04/38] tsm: Support DMA Allocation from private memory

[Aneesh Kumar K.V <aneesh.kumar@kernel.org>]

Jason Gunthorpe <jgg@ziepe.ca> writes:

> On Sat, Aug 02, 2025 at 02:14:20PM +0530, Aneesh Kumar K.V wrote:
>> Jason Gunthorpe <jgg@ziepe.ca> writes:
>> 
>> > On Tue, Jul 29, 2025 at 01:53:10PM +0530, Aneesh Kumar K.V wrote:
>> >> Jason Gunthorpe <jgg@ziepe.ca> writes:
>> >> 
>> >> > On Mon, Jul 28, 2025 at 07:21:41PM +0530, Aneesh Kumar K.V (Arm) wrote:
>> >> >> @@ -48,3 +49,12 @@ int set_memory_decrypted(unsigned long addr, int numpages)
>> >> >>  	return crypt_ops->decrypt(addr, numpages);
>> >> >>  }
>> >> >>  EXPORT_SYMBOL_GPL(set_memory_decrypted);
>> >> >> +
>> >> >> +bool force_dma_unencrypted(struct device *dev)
>> >> >> +{
>> >> >> +	if (dev->tdi_enabled)
>> >> >> +		return false;
>> >> >
>> >> > Is this OK? I see code like this:
>> >> >
>> >> > static inline dma_addr_t phys_to_dma_direct(struct device *dev,
>> >> > 		phys_addr_t phys)
>> >> > {
>> >> > 	if (force_dma_unencrypted(dev))
>> >> > 		return phys_to_dma_unencrypted(dev, phys);
>> >> > 	return phys_to_dma(dev, phys);
>> >> >
>> >> > What are the ARM rules for generating dma addreses?
>> >> >
>> >> > 1) Device is T=0, memory is unencrypted, call dma_addr_unencrypted()
>> >> >    and do "top bit IBA set"
>> >> >
>> >> > 2) Device is T=1, memory is encrypted, use the phys_to_dma() normally
>> >> >
>> >> > 3) Device it T=1, memory is uncrypted, use the phys_to_dma()
>> >> >    normally??? Seems odd, I would have guessed the DMA address sould
>> >> >    be the same as case #1?
>> >> >
>> >> > Can you document this in a comment?
>> >> >
>> >> 
>> >> If a device is operating in secure mode (T=1), it is currently assumed
>> >> that only access to private (encrypted) memory is supported.
>> >
>> > No, this is no how the PCI specs were written as far as I
>> > understand. The XT bit thing is supposed to add more fine grained
>> > device side control over what memory the DMA can target. T alone does
>> > not do that.
>> >
>> >> It is unclear whether devices would need to perform DMA to shared
>> >> (unencrypted) memory while operating in this mode, as TLPs with T=1
>> >> are generally expected to target private memory.
>> >
>> > PCI SIG supports it, kernel should support it.
>> >
>> 
>> Would we also need a separate DMA allocation API for allocating
>> addresses intended to be shared with the non-secure hypervisor?
>> 
>> Are there any existing drivers in the kernel that already require such
>> shared allocations, which I could use as a reference?
>
> The most likely case in the near term is PCI P2P to shared MMIO.
>
> I don't know any way to allocate shared memory in a driver??
>
> At the bare minimum this patch should be documenting the correct
> architecture and outlining any gaps in the current implementation.
>
> I also don't really understand what the above code is even
> doing.. Isn't the design on ARM that the IPA always encodes the
> shared/private in the top bit?
>
> How do we get a shared page that does not already have a phys_addr_t
> in the shared IPA? Shouldn't the kernel have switched to the shared
> IPA alias when it returned the swiotlb buffer? eg why do we need to do:
>
> #define dma_addr_unencrypted(x)		((x) | PROT_NS_SHARED)
>

swiotlb virt addr is updated in the direct map page table such that we
have the correct attribute set. For ex: swiotlb_update_mem_attributes
uses set_memory_decrypted() to mark the memory as shared.

	set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT);

However, when mapping swiotlb regions to obtain a `dma_addr_t`, we still
need to explicitly convert the physical address:

swiotlb_map()
	swiotlb_addr = swiotlb_tbl_map_single(dev, paddr, size, 0, dir, attrs);
        ...

	/* Ensure that the address returned is DMA'ble */
	dma_addr = phys_to_dma_unencrypted(dev, swiotlb_addr);

Note that we don’t update the phys_addr_t to set the top
bit. For reference:

	tlb_addr = slot_addr(pool->start, index) + offset;

-aneesh