From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63A41606A320 for ; Wed, 18 Nov 2026 15:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=mUCgV5Uo Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 19E2510E153; Wed, 18 Nov 2026 15:02:09 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9438510E153 for ; Wed, 18 Nov 2026 15:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1795014123; x=1826550123; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=kpgKsN1Wy7ZsN5WYBqODY8ojepjK480nsr7fnI3mMTo=; b=mUCgV5UoL5xnaBIiOE3tRlHulo7zVbqMIzOvG8DUs71rX+F169p5tg3t 0xvE7dqFKWvSjnKAwDQrSYnkGpjQceAPsWAQPHLHSqAm2rkQfe3l5E85C iNXgzYmhuAHJRBpDTKX4I1g/k5B/UbD1tiTYxxG6sZMaDTf8K3/zmgM69 UwxppplC/awq8KG6+IE9m/l+KQqdmb/JMErOwg1/KZxwoqoNAi6wyIdjm sF3cQaK+aJxiQZiucxCTgcAnYWdtwQf9MDxiXDpCZjhnJmCZvZeXnv5Cu +DiNNgnzmvlnB91p0uSvPsuWPiVLW1QyeZ2o2JNUhr/YZcZVkASINP3T4 A==; X-CSE-ConnectionGUID: IFpAH1PMTHmphPRHq2vV+Q== X-CSE-MsgGUID: TGxquoSYSeOCFnK6IDnmGw== X-IronPort-AV: E=McAfee;i="6800,10657,11970"; a="3844073" X-IronPort-AV: E=Sophos;i="6.27,219,1787036400"; d="scan'208";a="3844073" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Nov 2026 07:01:28 -0800 X-CSE-ConnectionGUID: tK6nWilUSHqaSrgtoVlcpg== X-CSE-MsgGUID: 63tTN+zPT7eFfSijHc6ODQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.27,219,1787036400"; d="scan'208";a="10499815" Received: from abityuts-desk1.ger.corp.intel.com (HELO [10.245.244.222]) ([10.245.244.222]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Nov 2026 07:01:25 -0800 Message-ID: Subject: Re: [PATCH v3] drm/xe: Add a per-client debugfs file reporting bo NUMA node distribution From: Thomas =?ISO-8859-1?Q?Hellstr=F6m?= To: intel-xe@lists.freedesktop.org Cc: Himal Prasad Ghimiray , Rodrigo Vivi , Matthew Brost , Matthew Auld , Andi Shyti Date: Wed, 18 Nov 2026 16:01:22 +0100 In-Reply-To: <20261118134203.646500-1-thomas.hellstrom@linux.intel.com> References: <20261118134203.646500-1-thomas.hellstrom@linux.intel.com> Organization: Intel Sweden AB, Registration Number: 556189-6027 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.58.3 (3.58.3-2.fc43) X-Archived-By: MIME-Version: 1.0 X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Intel-xe" X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 Subject above should ofc be v4. /Thomas On Wed, 2026-11-18 at 14:42 +0100, Thomas Hellstr=C3=B6m wrote: > When debugging NUMA placement of bo system memory, there is currently > no way to tell on which nodes the pages backing a bo reside. The > /proc//numa_maps interface doesn't help since bo CPU mappings > are > VM_PFNMAP, and clients may have thousands of bos, so a dump of all > bos > is undesirable. >=20 > Add a "bo_numa" file to the drm core per-client debugfs directory, > client-/. The file implements a request-response protocol per > open > file: Writing a GEM handle valid for the client takes a snapshot, > under > the bo's dma_resv lock, of the bo size, placement, and the number of > pages on each NUMA node. A bo whose pages are swapped out, backed up > or purged is reported as swapped/purged, and one whose pages are not > populated as not populated. Subsequent reads return that snapshot. > Once the > snapshot has been completely read it is discarded, and a new handle > may be written. Until then writes fail with -EBUSY, so that a pending > snapshot can't be replaced before it is consumed. This allows > querying > many bos using a single open file. >=20 > The write-request / read-response protocol per open file follows the > simple_transaction helpers in fs/libfs.c, used by e.g. nfsd and > selinuxfs, and amdgpu already populates the per-client debugfs > directory with its vm_pagetable_info file. >=20 > For sg bos, typically imported dma-bufs, the backing pages are owned > by the exporter and can't be inspected, so only size and placement > are > reported. >=20 > Example: > =C2=A0 exec 3<>/sys/kernel/debug/dri/client-/bo_numa > =C2=A0 echo >&3 > =C2=A0 cat <&3 > =C2=A0 echo >&3 > =C2=A0 cat <&3 > =C2=A0 exec 3>&- >=20 > The client id can be obtained from the drm-client-id fdinfo key. >=20 > v2: > - Use drm_coredump_printer_is_full() (Andi Shyti) > - Drop the Assisted-by tag and add a Signed-off-by (Andi Shyti) >=20 > v3: > - Report swapped or purged bos as swapped/purged (Sashiko) >=20 > v4: > - Also report bos purged via xe_ttm_bo_purge() as swapped/purged > (Sashiko) >=20 > Cc: Himal Prasad Ghimiray > Cc: Rodrigo Vivi > Cc: Matthew Brost > Cc: Matthew Auld > Cc: Andi Shyti > Signed-off-by: Thomas Hellstr=C3=B6m > Reviewed-by: Himal Prasad Ghimiray > # v3 > --- > =C2=A0drivers/gpu/drm/xe/xe_bo.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 113 ++++= +++++++++++++++++++ > =C2=A0drivers/gpu/drm/xe/xe_bo.h=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2= =A0 6 ++ > =C2=A0drivers/gpu/drm/xe/xe_debugfs.c | 157 > ++++++++++++++++++++++++++++++++ > =C2=A0drivers/gpu/drm/xe/xe_debugfs.h |=C2=A0=C2=A0 3 + > =C2=A0drivers/gpu/drm/xe/xe_device.c=C2=A0 |=C2=A0=C2=A0 2 + > =C2=A05 files changed, 281 insertions(+) >=20 > diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c > index 42a6aa1a2e4a..353ac1e9a9d6 100644 > --- a/drivers/gpu/drm/xe/xe_bo.c > +++ b/drivers/gpu/drm/xe/xe_bo.c > @@ -6,12 +6,14 @@ > =C2=A0#include "xe_bo.h" > =C2=A0 > =C2=A0#include > +#include > =C2=A0#include > =C2=A0 > =C2=A0#include > =C2=A0#include > =C2=A0#include > =C2=A0#include > +#include > =C2=A0#include > =C2=A0#include > =C2=A0#include > @@ -4248,6 +4250,117 @@ void > xe_bo_runtime_pm_release_mmap_offset(struct xe_bo *bo) > =C2=A0 list_del_init(&bo->vram_userfault_link); > =C2=A0} > =C2=A0 > +#if IS_ENABLED(CONFIG_DEBUG_FS) > +static void xe_bo_count_numa_pages(struct ttm_tt *tt, unsigned long > *nodes) > +{ > + pgoff_t i; > + > + for (i =3D 0; i < tt->num_pages; ++i) > + nodes[page_to_nid(tt->pages[i])]++; > +} > + > +static void xe_bo_print_numa_info_locked(struct xe_bo *bo, unsigned > long *nodes, > + struct drm_printer *p) > +{ > + struct ttm_resource *res =3D bo->ttm.resource; > + struct ttm_tt *tt =3D bo->ttm.ttm; > + unsigned long total =3D 0; > + const char *name; > + int nid; > + > + xe_bo_assert_held(bo); > + > + name =3D res ? xe_mem_type_to_name[res->mem_type] : NULL; > + drm_printf(p, "size: %zu\n", xe_bo_size(bo)); > + if (name) > + drm_printf(p, "placement: %s\n", name); > + else if (res) > + drm_printf(p, "placement: mem_type %u\n", res- > >mem_type); > + else > + drm_puts(p, "placement: none\n"); > + > + /* The pages of sg bos are owned by the exporter and not > visible. */ > + if (bo->ttm.type =3D=3D ttm_bo_type_sg) { > + drm_puts(p, "pages: unknown (sg bo)\n"); > + return; > + } > + > + /* > + * Purging via xe_ttm_bo_purge() guts the bo without setting > any > + * swapped or backed-up flag on the tt. > + */ > + if (xe_bo_is_purged(bo) || (tt && ttm_tt_is_swapped(tt))) { > + drm_puts(p, "pages: swapped/purged\n"); > + return; > + } > + > + if (tt && !ttm_tt_is_populated(tt)) { > + drm_puts(p, "pages: not populated\n"); > + return; > + } > + > + if (tt) > + xe_bo_count_numa_pages(tt, nodes); > + > + for_each_node(nid) { > + if (!nodes[nid] && !node_online(nid)) > + continue; > + drm_printf(p, "node%d: %lu\n", nid, nodes[nid]); > + total +=3D nodes[nid]; > + } > + > + drm_printf(p, "total: %lu\n", total); > +} > + > +/** > + * xe_bo_print_numa_info() - Print the NUMA node distribution of a > bo > + * @file: The drm_file of the client owning @handle. > + * @handle: The GEM handle of the bo. > + * @p: The drm_printer to print to. > + * > + * Print the size and placement of the bo and, for bos with > + * driver-allocated system pages, the number of pages residing on > each > + * NUMA node. If the bo's pages are currently swapped out, backed up > or > + * purged by the shrinker, the bo is reported as swapped/purged, and > if > + * they are not populated, as not populated. For sg bos, typically > + * imported dma-bufs, the backing pages are owned by the exporter > and are > + * reported as unknown. The information is gathered under the bo's > + * dma_resv lock and thus represents a consistent snapshot. > + * > + * Return: 0 on success, -ENOENT if @handle doesn't refer to a bo, > + * negative error code on other failures. > + */ > +int xe_bo_print_numa_info(struct drm_file *file, u32 handle, > + =C2=A0 struct drm_printer *p) > +{ > + struct drm_gem_object *obj; > + unsigned long *nodes; > + int ret; > + > + nodes =3D kcalloc(nr_node_ids, sizeof(*nodes), > GFP_KERNEL_ACCOUNT); > + if (!nodes) > + return -ENOMEM; > + > + obj =3D drm_gem_object_lookup(file, handle); > + if (!obj) { > + ret =3D -ENOENT; > + goto out_free; > + } > + > + ret =3D dma_resv_lock_interruptible(obj->resv, NULL); > + if (!ret) { > + xe_bo_print_numa_info_locked(gem_to_xe_bo(obj), > nodes, p); > + dma_resv_unlock(obj->resv); > + } > + > + drm_gem_object_put(obj); > +out_free: > + kfree(nodes); > + > + return ret; > +} > +#endif > + > =C2=A0#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST) > =C2=A0#include "tests/xe_bo.c" > =C2=A0#endif > diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h > index 77b3a02e9b8e..13197e4e0358 100644 > --- a/drivers/gpu/drm/xe/xe_bo.h > +++ b/drivers/gpu/drm/xe/xe_bo.h > @@ -112,6 +112,7 @@ enum xe_madv_purgeable_state { > =C2=A0 XE_MADV_PURGEABLE_PURGED, > =C2=A0}; > =C2=A0 > +struct drm_printer; > =C2=A0struct sg_table; > =C2=A0 > =C2=A0struct xe_bo *xe_bo_alloc(void); > @@ -458,6 +459,11 @@ int xe_bo_dumb_create(struct drm_file > *file_priv, > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct drm_device *dev, > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct drm_mode_create_dumb *args)= ; > =C2=A0 > +#if IS_ENABLED(CONFIG_DEBUG_FS) > +int xe_bo_print_numa_info(struct drm_file *file, u32 handle, > + =C2=A0 struct drm_printer *p); > +#endif > + > =C2=A0bool xe_bo_needs_ccs_pages(struct xe_bo *bo); > =C2=A0 > =C2=A0int xe_bo_decompress(struct xe_bo *bo); > diff --git a/drivers/gpu/drm/xe/xe_debugfs.c > b/drivers/gpu/drm/xe/xe_debugfs.c > index e59d4964d811..1a74a7106462 100644 > --- a/drivers/gpu/drm/xe/xe_debugfs.c > +++ b/drivers/gpu/drm/xe/xe_debugfs.c > @@ -11,6 +11,8 @@ > =C2=A0#include > =C2=A0 > =C2=A0#include > +#include > +#include > =C2=A0 > =C2=A0#include "regs/xe_pmt.h" > =C2=A0#include "xe_bo.h" > @@ -785,3 +787,158 @@ void xe_debugfs_register(struct xe_device *xe) > =C2=A0 else if (IS_SRIOV_VF(xe)) > =C2=A0 xe_sriov_vf_debugfs_register(xe, root); > =C2=A0} > + > +/* > + * Per-client "bo_numa" file: Report the NUMA node distribution of > the > + * system pages backing a single bo. > + * > + * The file implements a request-response protocol: write() takes a > GEM > + * handle, valid for the client, and takes a snapshot of the bo's > page > + * distribution under the bo's dma_resv lock. Subsequent read()s > return > + * that snapshot. Once the snapshot has been completely read, it is > + * discarded and a new handle may be written. Until then, writes > return > + * -EBUSY, so that a pending snapshot is never replaced before it is > + * consumed. > + * > + * See xe_bo_print_numa_info() for the snapshot contents. > + */ > +struct bo_numa_snapshot { > + /* Serializes reads and writes on the open file. */ > + struct mutex lock; > + /* The pending formatted snapshot, or NULL if none. */ > + char *buf; > + /* Length of the formatted snapshot. */ > + size_t len; > +}; > + > +static int bo_numa_open(struct inode *inode, struct file *f) > +{ > + struct bo_numa_snapshot *snap; > + > + snap =3D kzalloc_obj(*snap, GFP_KERNEL_ACCOUNT); > + if (!snap) > + return -ENOMEM; > + > + mutex_init(&snap->lock); > + f->private_data =3D snap; > + > + return nonseekable_open(inode, f); > +} > + > +static int bo_numa_release(struct inode *inode, struct file *f) > +{ > + struct bo_numa_snapshot *snap =3D f->private_data; > + > + kvfree(snap->buf); > + mutex_destroy(&snap->lock); > + kfree(snap); > + > + return 0; > +} > + > +static ssize_t bo_numa_read(struct file *f, char __user *ubuf, > size_t size, > + =C2=A0=C2=A0=C2=A0 loff_t *pos) > +{ > + struct bo_numa_snapshot *snap =3D f->private_data; > + ssize_t ret =3D 0; > + > + mutex_lock(&snap->lock); > + if (snap->buf) { > + ret =3D simple_read_from_buffer(ubuf, size, pos, snap- > >buf, > + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snap->len); > + if (*pos >=3D snap->len) { > + kvfree(snap->buf); > + snap->buf =3D NULL; > + } > + } > + mutex_unlock(&snap->lock); > + > + return ret; > +} > + > +static ssize_t bo_numa_write(struct file *f, const char __user > *ubuf, > + =C2=A0=C2=A0=C2=A0=C2=A0 size_t size, loff_t *pos) > +{ > + struct drm_file *file =3D file_inode(f)->i_private; > + struct bo_numa_snapshot *snap =3D f->private_data; > + struct drm_print_iterator iter; > + struct drm_printer p; > + size_t bufsize =3D PAGE_SIZE / 2; > + char *buf =3D NULL; > + u32 handle; > + int ret; > + > + ret =3D kstrtou32_from_user(ubuf, size, 0, &handle); > + if (ret) > + return ret; > + > + ret =3D mutex_lock_interruptible(&snap->lock); > + if (ret) > + return ret; > + > + if (snap->buf) { > + ret =3D -EBUSY; > + goto out_unlock; > + } > + > + /* > + * Retry with a larger buffer until the snapshot fits. The > coredump > + * printer is used as a generic bounded buffer printer, and > reports > + * when its buffer is full. > + */ > + do { > + kvfree(buf); > + bufsize *=3D 2; > + buf =3D kvzalloc(bufsize, GFP_KERNEL_ACCOUNT); > + if (!buf) { > + ret =3D -ENOMEM; > + goto out_unlock; > + } > + > + iter =3D (struct drm_print_iterator) { > + .data =3D buf, > + .remain =3D bufsize, > + }; > + p =3D drm_coredump_printer(&iter); > + ret =3D xe_bo_print_numa_info(file, handle, &p); > + if (ret) > + goto out_free; > + } while (drm_coredump_printer_is_full(&p)); > + > + snap->buf =3D buf; > + snap->len =3D bufsize - iter.remain; > + *pos =3D 0; > + buf =3D NULL; > + > +out_free: > + kvfree(buf); > +out_unlock: > + mutex_unlock(&snap->lock); > + > + return ret ?: size; > +} > + > +static const struct file_operations bo_numa_fops =3D { > + .owner =3D THIS_MODULE, > + .open =3D bo_numa_open, > + .release =3D bo_numa_release, > + .read =3D bo_numa_read, > + .write =3D bo_numa_write, > +}; > + > +/** > + * xe_debugfs_client_add() - Add xe-specific per-client debugfs > files > + * @file: The drm_file of the client. > + * > + * Populate the drm core per-client debugfs directory with xe- > specific > + * files. The files are removed by the drm core together with the > + * directory when @file is released. > + */ > +void xe_debugfs_client_add(struct drm_file *file) > +{ > + if (IS_ERR_OR_NULL(file->debugfs_client)) > + return; > + > + debugfs_create_file("bo_numa", 0600, file->debugfs_client, > file, > + =C2=A0=C2=A0=C2=A0 &bo_numa_fops); > +} > diff --git a/drivers/gpu/drm/xe/xe_debugfs.h > b/drivers/gpu/drm/xe/xe_debugfs.h > index 88d91c78036b..90effba271d9 100644 > --- a/drivers/gpu/drm/xe/xe_debugfs.h > +++ b/drivers/gpu/drm/xe/xe_debugfs.h > @@ -8,6 +8,7 @@ > =C2=A0 > =C2=A0#include > =C2=A0 > +struct drm_file; > =C2=A0struct xe_device; > =C2=A0 > =C2=A0#ifdef CONFIG_DEBUG_FS > @@ -16,12 +17,14 @@ bool xe_fault_csc_hw_error(void); > =C2=A0bool xe_fault_wedge_cold_reset(void); > =C2=A0bool xe_fault_mempage_offline(void); > =C2=A0void xe_debugfs_register(struct xe_device *xe); > +void xe_debugfs_client_add(struct drm_file *file); > =C2=A0#else > =C2=A0static inline bool xe_fault_gt_reset(void) { return false; } > =C2=A0static inline bool xe_fault_csc_hw_error(void) { return false; } > =C2=A0static inline bool xe_fault_wedge_cold_reset(void) { return false; = } > =C2=A0static inline bool xe_fault_mempage_offline(void) { return false; } > =C2=A0static inline void xe_debugfs_register(struct xe_device *xe) { } > +static inline void xe_debugfs_client_add(struct drm_file *file) { } > =C2=A0#endif > =C2=A0 > =C2=A0#endif > diff --git a/drivers/gpu/drm/xe/xe_device.c > b/drivers/gpu/drm/xe/xe_device.c > index 616e02066ae6..f5a581d39c96 100644 > --- a/drivers/gpu/drm/xe/xe_device.c > +++ b/drivers/gpu/drm/xe/xe_device.c > @@ -127,6 +127,8 @@ static int xe_file_open(struct drm_device *dev, > struct drm_file *file) > =C2=A0 put_task_struct(task); > =C2=A0 } > =C2=A0 > + xe_debugfs_client_add(file); > + > =C2=A0 return 0; > =C2=A0} > =C2=A0