From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D16C934750D for ; Wed, 25 Mar 2026 19:23:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774466639; cv=none; b=MQQeCBwMCFQbQe4MdsJIQkXluecSBhCzMROr0jIAdx5/XmorpmEM91jqdNBORRhiashQ38PDgCFnq8XYtwW32l/KR4vtxFTP/sgXX7yPXi9pwMs2w+nQE3X7ZZ0nny6IP0SjIQh0y2Ef8X5b37sE9xjM63OXXYUPD4nIFMM5IAE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774466639; c=relaxed/simple; bh=frY0xcNYgk1ejLroxKj/T9fAx5Z9H2r/8bvmlT1Hw3Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Apj1guFtmE1NV6Aa5vZ4L+yUUSbtBJYUfsZZFq2QKeSYYL61Vkn8RdziBi4vR+cbv6hokSET8DnPlBQ2S/c2vlAtMBnoQ8011emwXD0wi85Cw0Lr2G5QHujCluZJpSLSq0aXv0Tt36F5oxaN/iFkEyIxHwkr353rB6XkViVGGic= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=resnulli.us; spf=none smtp.mailfrom=resnulli.us; dkim=pass (2048-bit key) header.d=resnulli-us.20230601.gappssmtp.com header.i=@resnulli-us.20230601.gappssmtp.com header.b=mAndURXp; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=resnulli.us Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=resnulli.us Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=resnulli-us.20230601.gappssmtp.com header.i=@resnulli-us.20230601.gappssmtp.com header.b="mAndURXp" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-43b45bb7548so103482f8f.1 for ; Wed, 25 Mar 2026 12:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20230601.gappssmtp.com; s=20230601; t=1774466636; x=1775071436; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=brIT3WgYpimrqZgaDvvmaI9KpVmIklFHTAJQwyM6HwI=; b=mAndURXpFknpo8jHzSrSnPCIjAr/qoZxHz0ZmN3DJcf5/IqL+Yh22Eh8bRw9mYDk7a rBhwybrDIAZ+rEYfW5U+Co6V3bcutQlruylADLLhBiLdmmxLpAZ4fzUzg7U50ZIACGo7 dKExZsp0ujK6xXLGNUARFgllEJoTADz55PVnmZlBHZ3HO9CuU5YGt0uruYL+NAjLgWtS EAR4y7fS0xj8R1TVmnPEtThpUgiZ5oGG2KfvOzaV5NX7ehhQkaovOWpmu/A+ADmAHKnR 9I12xxKhSzKGUh43DGdyJMrDzAvBLlPnWNtDtFAnLLfRRqc/WT3m66hPHe7vw2TW5nEP h1uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774466636; x=1775071436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=brIT3WgYpimrqZgaDvvmaI9KpVmIklFHTAJQwyM6HwI=; b=ZYUpV8ZN+aQbIsM7tPoKJexBB8Yf9fimAr5nJS+QNGW59m1yvAtsNJcZYWwHUVMV+p HxtPrh0tHG8GUjy4yHuG0rbOfp/HgNPTA92kGl+xskkgaXWIS3lGOdxgjUWjYdHFg9bA oWDhQRpHwTpt/GiTNv52yv+PIz6aDdAy8ypZBUdLU7dAeh4GFZtN8IWwzRB6YD/Pcf2e yVuGr17+ru7kFEcqCjaSg54xtPmIy8VjllygiSxVo0VE4y83Iy8iEAyounewbQwlIsyU EkBsgYEcPPjINGXRwngVYzhHjdwyEZNslyukIlC6pubnsFJwQCuPtGx00JUIDioQsFfV htbg== X-Forwarded-Encrypted: i=1; AJvYcCVSfn8I0Gn+iXIC+L39q98NFGHXCERMQY/Sgn/XPw/iqF9iQtMwkSaibs3wFndUOAclsZPRlhMFs1/3@lists.linux.dev X-Gm-Message-State: AOJu0Yz9bcahBOHX6DsK8WgVWgjh0Tcxg6/0/zhPwNTB2wArE1SMXUsc T5EzjO4bAHDar6tXEIzILsGNyhUzBp+s0mCiiWSBzlOcJQo62ifrPyvNZ4iAfkFjHms= X-Gm-Gg: ATEYQzx9NvO9XtFt/DU+mDjjmxudQEMiwzQsx7S/SYq/6LFIYsA/jw4Wg1s+zO1Ve73 hY0NPkT/IvtLtrtQ2oBXyAsUhqCZCML8cAMLjibBbBEaEkBessC2nF2VovJHXTNbqOKY/hL5GIg J8oJX8SRM1ITblied+ImiK8p1yPuXj6UBriuwhSoC+wjrSDG9IXVBITJw7OjBj1n2z2Eo1l0UJa x0iAy4agp5OJXPWKwAwJ8lnbq2kkTTWnrJ24zq+ZgRVw7ynXF+5mLJ46DZZ/pWZmV04VDWWBirq Y5jELbZyHN97UudGY/J8nqFrJxiH6i2Cbyla7s2rPGw7aOvEzFHc9moai5S79svz0297iss0cY3 BBQw0szsxHCCBnUBmzl0U3b2jp8yhuBUwrtFjmLofCKU2xfCueA4ExTUSb3KAvcfIidyxdM8bca ynwRS5QlpSGMKeGBMtEA== X-Received: by 2002:a05:6000:18a8:b0:439:b440:b8a2 with SMTP id ffacd0b85a97d-43b88a0d156mr6816531f8f.28.1774466636179; Wed, 25 Mar 2026 12:23:56 -0700 (PDT) Received: from localhost ([140.209.217.211]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43b919df7dcsm2215681f8f.27.2026.03.25.12.23.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Mar 2026 12:23:55 -0700 (PDT) From: Jiri Pirko To: dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, iommu@lists.linux.dev, linux-media@vger.kernel.org Cc: sumit.semwal@linaro.org, benjamin.gaignard@collabora.com, Brian.Starkey@arm.com, jstultz@google.com, tjmercier@google.com, christian.koenig@amd.com, m.szyprowski@samsung.com, robin.murphy@arm.com, jgg@ziepe.ca, leon@kernel.org, sean.anderson@linux.dev, ptesarik@suse.com, catalin.marinas@arm.com, aneesh.kumar@kernel.org, suzuki.poulose@arm.com, steven.price@arm.com, thomas.lendacky@amd.com, john.allen@amd.com, ashish.kalra@amd.com, suravee.suthikulpanit@amd.com, linux-coco@lists.linux.dev Subject: [PATCH v5 2/2] dma-buf: heaps: system: add system_cc_shared heap for explicitly shared memory Date: Wed, 25 Mar 2026 20:23:52 +0100 Message-ID: <20260325192352.437608-3-jiri@resnulli.us> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20260325192352.437608-1-jiri@resnulli.us> References: <20260325192352.437608-1-jiri@resnulli.us> Precedence: bulk X-Mailing-List: linux-coco@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Jiri Pirko Add a new "system_cc_shared" dma-buf heap to allow userspace to allocate shared (decrypted) memory for confidential computing (CoCo) VMs. On CoCo VMs, guest memory is private by default. The hardware uses an encryption bit in page table entries (C-bit on AMD SEV, "shared" bit on Intel TDX) to control whether a given memory access is private or shared. The kernel's direct map is set up as private, so pages returned by alloc_pages() are private in the direct map by default. To make this memory usable for devices that do not support DMA to private memory (no TDISP support), it has to be explicitly shared. A couple of things are needed to properly handle shared memory for the dma-buf use case: - set_memory_decrypted() on the direct map after allocation: Besides clearing the encryption bit in the direct map PTEs, this also notifies the hypervisor about the page state change. On free, the inverse set_memory_encrypted() must be called before returning pages to the allocator. If re-encryption fails, pages are intentionally leaked to prevent shared memory from being reused as private. - pgprot_decrypted() for userspace and kernel virtual mappings: Any new mapping of the shared pages, be it to userspace via mmap or to kernel vmalloc space via vmap, creates PTEs independent of the direct map. These must also have the encryption bit cleared, otherwise accesses through them would see encrypted (garbage) data. - DMA_ATTR_CC_SHARED for DMA mapping: Since the pages are already shared, the DMA API needs to be informed via DMA_ATTR_CC_SHARED so it can map them correctly as unencrypted for device access. On non-CoCo VMs, the system_cc_shared heap is not registered to prevent misuse by userspace that does not understand the security implications of explicitly shared memory. Signed-off-by: Jiri Pirko --- v4->v5: - bools renamed: s/decrypted/cc_decrypted/ - other renames: s/decrypted/decrypted/ - this included name of the heap v2->v3: - removed couple of leftovers from headers v1->v2: - fixed build errors on s390 by including mem_encrypt.h - converted system heap flag implementation to a separate heap --- drivers/dma-buf/heaps/system_heap.c | 103 ++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 5 deletions(-) diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index b3650d8fd651..03c2b87cb111 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -10,17 +10,25 @@ * Andrew F. Davis */ +#include #include #include #include #include #include +#include #include +#include #include +#include #include #include #include +struct system_heap_priv { + bool cc_shared; +}; + struct system_heap_buffer { struct dma_heap *heap; struct list_head attachments; @@ -29,6 +37,7 @@ struct system_heap_buffer { struct sg_table sg_table; int vmap_cnt; void *vaddr; + bool cc_shared; }; struct dma_heap_attachment { @@ -36,6 +45,7 @@ struct dma_heap_attachment { struct sg_table table; struct list_head list; bool mapped; + bool cc_shared; }; #define LOW_ORDER_GFP (GFP_HIGHUSER | __GFP_ZERO) @@ -52,6 +62,34 @@ static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP}; static const unsigned int orders[] = {8, 4, 0}; #define NUM_ORDERS ARRAY_SIZE(orders) +static int system_heap_set_page_decrypted(struct page *page) +{ + unsigned long addr = (unsigned long)page_address(page); + unsigned int nr_pages = 1 << compound_order(page); + int ret; + + ret = set_memory_decrypted(addr, nr_pages); + if (ret) + pr_warn_ratelimited("dma-buf system heap: failed to decrypt page at %p\n", + page_address(page)); + + return ret; +} + +static int system_heap_set_page_encrypted(struct page *page) +{ + unsigned long addr = (unsigned long)page_address(page); + unsigned int nr_pages = 1 << compound_order(page); + int ret; + + ret = set_memory_encrypted(addr, nr_pages); + if (ret) + pr_warn_ratelimited("dma-buf system heap: failed to re-encrypt page at %p, leaking memory\n", + page_address(page)); + + return ret; +} + static int dup_sg_table(struct sg_table *from, struct sg_table *to) { struct scatterlist *sg, *new_sg; @@ -90,6 +128,7 @@ static int system_heap_attach(struct dma_buf *dmabuf, a->dev = attachment->dev; INIT_LIST_HEAD(&a->list); a->mapped = false; + a->cc_shared = buffer->cc_shared; attachment->priv = a; @@ -119,9 +158,11 @@ static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attac { struct dma_heap_attachment *a = attachment->priv; struct sg_table *table = &a->table; + unsigned long attrs; int ret; - ret = dma_map_sgtable(attachment->dev, table, direction, 0); + attrs = a->cc_shared ? DMA_ATTR_CC_SHARED : 0; + ret = dma_map_sgtable(attachment->dev, table, direction, attrs); if (ret) return ERR_PTR(ret); @@ -188,8 +229,13 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) unsigned long addr = vma->vm_start; unsigned long pgoff = vma->vm_pgoff; struct scatterlist *sg; + pgprot_t prot; int i, ret; + prot = vma->vm_page_prot; + if (buffer->cc_shared) + prot = pgprot_decrypted(prot); + for_each_sgtable_sg(table, sg, i) { unsigned long n = sg->length >> PAGE_SHIFT; @@ -206,8 +252,7 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) if (addr + size > vma->vm_end) size = vma->vm_end - addr; - ret = remap_pfn_range(vma, addr, page_to_pfn(page), - size, vma->vm_page_prot); + ret = remap_pfn_range(vma, addr, page_to_pfn(page), size, prot); if (ret) return ret; @@ -225,6 +270,7 @@ static void *system_heap_do_vmap(struct system_heap_buffer *buffer) struct page **pages = vmalloc(sizeof(struct page *) * npages); struct page **tmp = pages; struct sg_page_iter piter; + pgprot_t prot; void *vaddr; if (!pages) @@ -235,7 +281,10 @@ static void *system_heap_do_vmap(struct system_heap_buffer *buffer) *tmp++ = sg_page_iter_page(&piter); } - vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL); + prot = PAGE_KERNEL; + if (buffer->cc_shared) + prot = pgprot_decrypted(prot); + vaddr = vmap(pages, npages, VM_MAP, prot); vfree(pages); if (!vaddr) @@ -296,6 +345,14 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf) for_each_sgtable_sg(table, sg, i) { struct page *page = sg_page(sg); + /* + * Intentionally leak pages that cannot be re-encrypted + * to prevent shared memory from being reused. + */ + if (buffer->cc_shared && + system_heap_set_page_encrypted(page)) + continue; + __free_pages(page, compound_order(page)); } sg_free_table(table); @@ -347,6 +404,8 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, DEFINE_DMA_BUF_EXPORT_INFO(exp_info); unsigned long size_remaining = len; unsigned int max_order = orders[0]; + struct system_heap_priv *priv = dma_heap_get_drvdata(heap); + bool cc_shared = priv->cc_shared; struct dma_buf *dmabuf; struct sg_table *table; struct scatterlist *sg; @@ -362,6 +421,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, mutex_init(&buffer->lock); buffer->heap = heap; buffer->len = len; + buffer->cc_shared = cc_shared; INIT_LIST_HEAD(&pages); i = 0; @@ -396,6 +456,14 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, list_del(&page->lru); } + if (cc_shared) { + for_each_sgtable_sg(table, sg, i) { + ret = system_heap_set_page_decrypted(sg_page(sg)); + if (ret) + goto free_pages; + } + } + /* create the dmabuf */ exp_info.exp_name = dma_heap_get_name(heap); exp_info.ops = &system_heap_buf_ops; @@ -413,6 +481,13 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, for_each_sgtable_sg(table, sg, i) { struct page *p = sg_page(sg); + /* + * Intentionally leak pages that cannot be re-encrypted + * to prevent shared memory from being reused. + */ + if (buffer->cc_shared && + system_heap_set_page_encrypted(p)) + continue; __free_pages(p, compound_order(p)); } sg_free_table(table); @@ -428,6 +503,14 @@ static const struct dma_heap_ops system_heap_ops = { .allocate = system_heap_allocate, }; +static struct system_heap_priv system_heap_priv = { + .cc_shared = false, +}; + +static struct system_heap_priv system_heap_cc_shared_priv = { + .cc_shared = true, +}; + static int __init system_heap_create(void) { struct dma_heap_export_info exp_info; @@ -435,8 +518,18 @@ static int __init system_heap_create(void) exp_info.name = "system"; exp_info.ops = &system_heap_ops; - exp_info.priv = NULL; + exp_info.priv = &system_heap_priv; + + sys_heap = dma_heap_add(&exp_info); + if (IS_ERR(sys_heap)) + return PTR_ERR(sys_heap); + + if (IS_ENABLED(CONFIG_HIGHMEM) || + !cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + return 0; + exp_info.name = "system_cc_shared"; + exp_info.priv = &system_heap_cc_shared_priv; sys_heap = dma_heap_add(&exp_info); if (IS_ERR(sys_heap)) return PTR_ERR(sys_heap); -- 2.51.1