[PATCH v16 0/7] TTM shrinker helpers and xe buffer object shrinker

[PATCH v16 0/7] TTM shrinker helpers and xe buffer object shrinker

[Thomas Hellström]

This series implements TTM shrinker / eviction helpers and an xe bo
shrinker. It builds on a previous series, *and obsoletes that one*.

https://lore.kernel.org/linux-mm/b7491378-defd-4f1c-31e2-29e4c77e2d67@amd.com/T/

Where the comment about layering
https://lore.kernel.org/linux-mm/b7491378-defd-4f1c-31e2-29e4c77e2d67@amd.com/T/#ma918844aa8a6efe8768fdcda0c6590d5c93850c9

now addressed, and this version also implements shmem objects for backup
rather than direct swap-cache insertions, which was used in the previuos
series. It turns out that with per-page backup / shrinking, shmem objects
appears to work just as well as direct swap-cache insertions with the
added benefit that was introduced in the previous TTM shrinker series to
avoid running out of swap entries isn't really needed.

The series earlier consisted of a LRU traversal part and the current part.
The LRU traversal part is merged, but is still mentioned in the history
below.

Patch 1 introduces a backup implemententaion.

Patch 2 introduces functionality in the ttm_pool code for page-by-page shrinking
and recovery. It avoids having to temporarily allocate a huge amount of
memory to be able to shrink a buffer object. It also introduces the
possibility to immediately write-back pages if needed.

Patch 3 Adds a simple error injection to the above code to help increase
test coverage.

Patch 4 Implements a macro for LRU iteration.

Patch 5 Introduces driver-facing helpers for shrinking.

Patch 6 Implements the xe bo shrinker.

Patch 7 Increases (removes) the XE_PL_TT watermark.

v2:
- Squash obsolete revision history in the patch commit messages.
- Fix a couple of review comments by Christian
- Don't store the mem_type in the TTM managers but in the
  resource cursor.
- Rename introduced TTM *back_up* function names to *backup*
- Add ttm pool recovery fault injection.
- Shrinker xe kunit test
- Various bugfixes

v3:
- Address some review comments from Matthew Brost and Christian König.
- Use the restartable LRU walk for TTM swapping and eviction.
- Provide a POC drm_exec locking implementation for exhaustive
  eviction. (Christian König).

v4:
- Remove the RFC exhaustive eviction part. While the path to exhaustive
  eviction is pretty clear and demonstrated in v3, there is still some
  drm_exec work that needs to be agreed and implemented.
- Add shrinker power management. On some hw we need to wake when shrinking.
- Fix the lru walker helper for -EALREADY errors.
- Add drm/xe: Increase the XE_PL_TT watermark.

v5:
- Update also TTM kunit tests
- Handle ghost- and zombie objects in the shrinker.
- A couple of compile- and UAF fixes reported by Kernel Build Robot and
  Dan Carpenter.

v6:
- Address review comments from Matthew Brost on the
  restartable LRU traversal path.

v7:
- Split out TTM restartable LRU traversal path and merge that.
- Adapt the review comments on that series.

v8:
- Address review comments from Matthew Brost as detailed in the
  respective patches.

v9:
- Rebase and fix compilation errors

v10:
- Use a LRU iteration macro rather than a function with a callback.
- Rebasing and cleanups
- Address some additional review comments from Matt Brost.
- Drop the shrinker selftest. It was already merged as a swapout
  self-test.

v11:
- Move more core interaction to additional TTM helpers.
- Don't back up without __GFP_FS, and don't start writeback without __GFP_IO.
- Rebase.

v12:
- Fix an indentation flaw.
- Rebase

v13:
- Remove the backup base-class, and use direct calls for ttm_backup
  (Christian König).
- Rebase on the ttm_backup changes.
- Move shrunken bos from the LRU list to the unevictable list.
- Provide an accessor function with sanity checks to set the
- ttm_tt::backup field.
- Update documentation.

v14:
- Update documentation of ttm_backup_bytes_avail().
- Work around converting between struct file * and struct ttm-backup *.
- Don't set up backup for imported buffers.

v15:
- ttm_backup return value change. (Christian König)
- ttm_pool restore/alloc implementation rework (Christian König)
- Documentation update.
- Remove cond_resched (Christian König)

v16:
- Two patches merged. Rebase.

Cc: Christian König <christian.koenig@amd.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Paulo Zanoni <paulo.r.zanoni@intel.com>
Cc: <dri-devel@lists.freedesktop.org>

Thomas Hellström (7):
  drm/ttm: Provide a shmem backup implementation
  drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages
  drm/ttm: Use fault-injection to test error paths
  drm/ttm: Add a macro to perform LRU iteration
  drm/ttm: Add helpers for shrinking
  drm/xe: Add a shrinker for xe bos
  drm/xe: Increase the XE_PL_TT watermark

 drivers/gpu/drm/ttm/Makefile         |   2 +-
 drivers/gpu/drm/ttm/ttm_backup.c     | 207 ++++++++++
 drivers/gpu/drm/ttm/ttm_bo_util.c    | 247 +++++++++++-
 drivers/gpu/drm/ttm/ttm_pool.c       | 577 ++++++++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_tt.c         |  83 ++++
 drivers/gpu/drm/xe/Makefile          |   1 +
 drivers/gpu/drm/xe/tests/xe_bo.c     |   6 +-
 drivers/gpu/drm/xe/xe_bo.c           | 202 +++++++++-
 drivers/gpu/drm/xe/xe_bo.h           |  36 ++
 drivers/gpu/drm/xe/xe_device.c       |   8 +
 drivers/gpu/drm/xe/xe_device_types.h |   2 +
 drivers/gpu/drm/xe/xe_shrinker.c     | 258 ++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h     |  18 +
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c  |   3 +-
 include/drm/ttm/ttm_backup.h         |  74 ++++
 include/drm/ttm/ttm_bo.h             |  93 +++++
 include/drm/ttm/ttm_pool.h           |   8 +
 include/drm/ttm/ttm_tt.h             |  69 +++-
 18 files changed, 1814 insertions(+), 80 deletions(-)
 create mode 100644 drivers/gpu/drm/ttm/ttm_backup.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
 create mode 100644 include/drm/ttm/ttm_backup.h

-- 
2.48.1

[PATCH v16 1/7] drm/ttm: Provide a shmem backup implementation

[Thomas Hellström]

Provide a standalone shmem backup implementation.
Given the ttm_backup interface, this could
later on be extended to providing other backup
implementation than shmem, with one use-case being
GPU swapout to a user-provided fd.

v5:
- Fix a UAF. (kernel test robot, Dan Carptenter)
v6:
- Rename ttm_backup_shmem_copy_page() function argument
  (Matthew Brost)
- Add some missing documentation
v8:
- Use folio_file_page to get to the page we want to writeback
  instead of using the first page of the folio.
v13:
- Remove the base class abstraction (Christian König)
- Include ttm_backup_bytes_avail().
v14:
- Fix kerneldoc for ttm_backup_bytes_avail() (0-day)
- Work around casting of __randomize_layout struct pointer (0-day)
v15:
- Return negative error code from ttm_backup_backup_page()
  (Christian König)
- Doc fixes. (Christian König).

Cc: Christian König <christian.koenig@amd.com>
Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <dri-devel@lists.freedesktop.org>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v13
---
 drivers/gpu/drm/ttm/Makefile     |   2 +-
 drivers/gpu/drm/ttm/ttm_backup.c | 207 +++++++++++++++++++++++++++++++
 include/drm/ttm/ttm_backup.h     |  74 +++++++++++
 3 files changed, 282 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/ttm/ttm_backup.c
 create mode 100644 include/drm/ttm/ttm_backup.h

diff --git a/drivers/gpu/drm/ttm/Makefile b/drivers/gpu/drm/ttm/Makefile
index dad298127226..40d07a35293a 100644
--- a/drivers/gpu/drm/ttm/Makefile
+++ b/drivers/gpu/drm/ttm/Makefile
@@ -4,7 +4,7 @@
 
 ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \
 	ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \
-	ttm_device.o ttm_sys_manager.o
+	ttm_device.o ttm_sys_manager.o ttm_backup.o
 ttm-$(CONFIG_AGP) += ttm_agp_backend.o
 
 obj-$(CONFIG_DRM_TTM) += ttm.o
diff --git a/drivers/gpu/drm/ttm/ttm_backup.c b/drivers/gpu/drm/ttm/ttm_backup.c
new file mode 100644
index 000000000000..93c007f18855
--- /dev/null
+++ b/drivers/gpu/drm/ttm/ttm_backup.c
@@ -0,0 +1,207 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_backup.h>
+#include <linux/page-flags.h>
+#include <linux/swap.h>
+
+/*
+ * Casting from randomized struct file * to struct ttm_backup * is fine since
+ * struct ttm_backup is never defined nor dereferenced.
+ */
+static struct file *ttm_backup_to_file(struct ttm_backup *backup)
+{
+	return (void *)backup;
+}
+
+static struct ttm_backup *ttm_file_to_backup(struct file *file)
+{
+	return (void *)file;
+}
+
+/*
+ * Need to map shmem indices to handle since a handle value
+ * of 0 means error, following the swp_entry_t convention.
+ */
+static unsigned long ttm_backup_shmem_idx_to_handle(pgoff_t idx)
+{
+	return (unsigned long)idx + 1;
+}
+
+static pgoff_t ttm_backup_handle_to_shmem_idx(pgoff_t handle)
+{
+	return handle - 1;
+}
+
+/**
+ * ttm_backup_drop() - release memory associated with a handle
+ * @backup: The struct backup pointer used to obtain the handle
+ * @handle: The handle obtained from the @backup_page function.
+ */
+void ttm_backup_drop(struct ttm_backup *backup, pgoff_t handle)
+{
+	loff_t start = ttm_backup_handle_to_shmem_idx(handle);
+
+	start <<= PAGE_SHIFT;
+	shmem_truncate_range(file_inode(ttm_backup_to_file(backup)), start,
+			     start + PAGE_SIZE - 1);
+}
+
+/**
+ * ttm_backup_copy_page() - Copy the contents of a previously backed
+ * up page
+ * @backup: The struct backup pointer used to back up the page.
+ * @dst: The struct page to copy into.
+ * @handle: The handle returned when the page was backed up.
+ * @intr: Try to perform waits interruptable or at least killable.
+ *
+ * Return: 0 on success, Negative error code on failure, notably
+ * -EINTR if @intr was set to true and a signal is pending.
+ */
+int ttm_backup_copy_page(struct ttm_backup *backup, struct page *dst,
+			 pgoff_t handle, bool intr)
+{
+	struct file *filp = ttm_backup_to_file(backup);
+	struct address_space *mapping = filp->f_mapping;
+	struct folio *from_folio;
+	pgoff_t idx = ttm_backup_handle_to_shmem_idx(handle);
+
+	from_folio = shmem_read_folio(mapping, idx);
+	if (IS_ERR(from_folio))
+		return PTR_ERR(from_folio);
+
+	copy_highpage(dst, folio_file_page(from_folio, idx));
+	folio_put(from_folio);
+
+	return 0;
+}
+
+/**
+ * ttm_backup_backup_page() - Backup a page
+ * @backup: The struct backup pointer to use.
+ * @page: The page to back up.
+ * @writeback: Whether to perform immediate writeback of the page.
+ * This may have performance implications.
+ * @idx: A unique integer for each page and each struct backup.
+ * This allows the backup implementation to avoid managing
+ * its address space separately.
+ * @page_gfp: The gfp value used when the page was allocated.
+ * This is used for accounting purposes.
+ * @alloc_gfp: The gfp to be used when allocating memory.
+ *
+ * Context: If called from reclaim context, the caller needs to
+ * assert that the shrinker gfp has __GFP_FS set, to avoid
+ * deadlocking on lock_page(). If @writeback is set to true and
+ * called from reclaim context, the caller also needs to assert
+ * that the shrinker gfp has __GFP_IO set, since without it,
+ * we're not allowed to start backup IO.
+ *
+ * Return: A handle on success. Negative error code on failure.
+ *
+ * Note: This function could be extended to back up a folio and
+ * implementations would then split the folio internally if needed.
+ * Drawback is that the caller would then have to keep track of
+ * the folio size- and usage.
+ */
+s64
+ttm_backup_backup_page(struct ttm_backup *backup, struct page *page,
+		       bool writeback, pgoff_t idx, gfp_t page_gfp,
+		       gfp_t alloc_gfp)
+{
+	struct file *filp = ttm_backup_to_file(backup);
+	struct address_space *mapping = filp->f_mapping;
+	unsigned long handle = 0;
+	struct folio *to_folio;
+	int ret;
+
+	to_folio = shmem_read_folio_gfp(mapping, idx, alloc_gfp);
+	if (IS_ERR(to_folio))
+		return PTR_ERR(to_folio);
+
+	folio_mark_accessed(to_folio);
+	folio_lock(to_folio);
+	folio_mark_dirty(to_folio);
+	copy_highpage(folio_file_page(to_folio, idx), page);
+	handle = ttm_backup_shmem_idx_to_handle(idx);
+
+	if (writeback && !folio_mapped(to_folio) &&
+	    folio_clear_dirty_for_io(to_folio)) {
+		struct writeback_control wbc = {
+			.sync_mode = WB_SYNC_NONE,
+			.nr_to_write = SWAP_CLUSTER_MAX,
+			.range_start = 0,
+			.range_end = LLONG_MAX,
+			.for_reclaim = 1,
+		};
+		folio_set_reclaim(to_folio);
+		ret = mapping->a_ops->writepage(folio_file_page(to_folio, idx), &wbc);
+		if (!folio_test_writeback(to_folio))
+			folio_clear_reclaim(to_folio);
+		/*
+		 * If writepage succeeds, it unlocks the folio.
+		 * writepage() errors are otherwise dropped, since writepage()
+		 * is only best effort here.
+		 */
+		if (ret)
+			folio_unlock(to_folio);
+	} else {
+		folio_unlock(to_folio);
+	}
+
+	folio_put(to_folio);
+
+	return handle;
+}
+
+/**
+ * ttm_backup_fini() - Free the struct backup resources after last use.
+ * @backup: Pointer to the struct backup whose resources to free.
+ *
+ * After a call to this function, it's illegal to use the @backup pointer.
+ */
+void ttm_backup_fini(struct ttm_backup *backup)
+{
+	fput(ttm_backup_to_file(backup));
+}
+
+/**
+ * ttm_backup_bytes_avail() - Report the approximate number of bytes of backup space
+ * left for backup.
+ *
+ * This function is intended also for driver use to indicate whether a
+ * backup attempt is meaningful.
+ *
+ * Return: An approximate size of backup space available.
+ */
+u64 ttm_backup_bytes_avail(void)
+{
+	/*
+	 * The idea behind backing up to shmem is that shmem objects may
+	 * eventually be swapped out. So no point swapping out if there
+	 * is no or low swap-space available. But the accuracy of this
+	 * number also depends on shmem actually swapping out backed-up
+	 * shmem objects without too much buffering.
+	 */
+	return (u64)get_nr_swap_pages() << PAGE_SHIFT;
+}
+EXPORT_SYMBOL_GPL(ttm_backup_bytes_avail);
+
+/**
+ * ttm_backup_shmem_create() - Create a shmem-based struct backup.
+ * @size: The maximum size (in bytes) to back up.
+ *
+ * Create a backup utilizing shmem objects.
+ *
+ * Return: A pointer to a struct ttm_backup on success,
+ * an error pointer on error.
+ */
+struct ttm_backup *ttm_backup_shmem_create(loff_t size)
+{
+	struct file *filp;
+
+	filp = shmem_file_setup("ttm shmem backup", size, 0);
+
+	return ttm_file_to_backup(filp);
+}
diff --git a/include/drm/ttm/ttm_backup.h b/include/drm/ttm/ttm_backup.h
new file mode 100644
index 000000000000..24ad120b8827
--- /dev/null
+++ b/include/drm/ttm/ttm_backup.h
@@ -0,0 +1,74 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _TTM_BACKUP_H_
+#define _TTM_BACKUP_H_
+
+#include <linux/mm_types.h>
+#include <linux/shmem_fs.h>
+
+struct ttm_backup;
+
+/**
+ * ttm_backup_handle_to_page_ptr() - Convert handle to struct page pointer
+ * @handle: The handle to convert.
+ *
+ * Converts an opaque handle received from the
+ * struct ttm_backoup_ops::backup_page() function to an (invalid)
+ * struct page pointer suitable for a struct page array.
+ *
+ * Return: An (invalid) struct page pointer.
+ */
+static inline struct page *
+ttm_backup_handle_to_page_ptr(unsigned long handle)
+{
+	return (struct page *)(handle << 1 | 1);
+}
+
+/**
+ * ttm_backup_page_ptr_is_handle() - Whether a struct page pointer is a handle
+ * @page: The struct page pointer to check.
+ *
+ * Return: true if the struct page pointer is a handld returned from
+ * ttm_backup_handle_to_page_ptr(). False otherwise.
+ */
+static inline bool ttm_backup_page_ptr_is_handle(const struct page *page)
+{
+	return (unsigned long)page & 1;
+}
+
+/**
+ * ttm_backup_page_ptr_to_handle() - Convert a struct page pointer to a handle
+ * @page: The struct page pointer to convert
+ *
+ * Return: The handle that was previously used in
+ * ttm_backup_handle_to_page_ptr() to obtain a struct page pointer, suitable
+ * for use as argument in the struct ttm_backup_ops drop() or
+ * copy_backed_up_page() functions.
+ */
+static inline unsigned long
+ttm_backup_page_ptr_to_handle(const struct page *page)
+{
+	WARN_ON(!ttm_backup_page_ptr_is_handle(page));
+	return (unsigned long)page >> 1;
+}
+
+void ttm_backup_drop(struct ttm_backup *backup, pgoff_t handle);
+
+int ttm_backup_copy_page(struct ttm_backup *backup, struct page *dst,
+			 pgoff_t handle, bool intr);
+
+s64
+ttm_backup_backup_page(struct ttm_backup *backup, struct page *page,
+		       bool writeback, pgoff_t idx, gfp_t page_gfp,
+		       gfp_t alloc_gfp);
+
+void ttm_backup_fini(struct ttm_backup *backup);
+
+u64 ttm_backup_bytes_avail(void);
+
+struct ttm_backup *ttm_backup_shmem_create(loff_t size);
+
+#endif
-- 
2.48.1

Re: [PATCH v16 1/7] drm/ttm: Provide a shmem backup implementation

[Christian König]

Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> Provide a standalone shmem backup implementation.
> Given the ttm_backup interface, this could
> later on be extended to providing other backup
> implementation than shmem, with one use-case being
> GPU swapout to a user-provided fd.
>
> v5:
> - Fix a UAF. (kernel test robot, Dan Carptenter)
> v6:
> - Rename ttm_backup_shmem_copy_page() function argument
>    (Matthew Brost)
> - Add some missing documentation
> v8:
> - Use folio_file_page to get to the page we want to writeback
>    instead of using the first page of the folio.
> v13:
> - Remove the base class abstraction (Christian König)
> - Include ttm_backup_bytes_avail().
> v14:
> - Fix kerneldoc for ttm_backup_bytes_avail() (0-day)
> - Work around casting of __randomize_layout struct pointer (0-day)
> v15:
> - Return negative error code from ttm_backup_backup_page()
>    (Christian König)
> - Doc fixes. (Christian König).
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: <dri-devel@lists.freedesktop.org>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v13

Reviewed-by: Christian König <christian.koenig@amd.com>

> ---
>   drivers/gpu/drm/ttm/Makefile     |   2 +-
>   drivers/gpu/drm/ttm/ttm_backup.c | 207 +++++++++++++++++++++++++++++++
>   include/drm/ttm/ttm_backup.h     |  74 +++++++++++
>   3 files changed, 282 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/gpu/drm/ttm/ttm_backup.c
>   create mode 100644 include/drm/ttm/ttm_backup.h
>
> diff --git a/drivers/gpu/drm/ttm/Makefile b/drivers/gpu/drm/ttm/Makefile
> index dad298127226..40d07a35293a 100644
> --- a/drivers/gpu/drm/ttm/Makefile
> +++ b/drivers/gpu/drm/ttm/Makefile
> @@ -4,7 +4,7 @@
>   
>   ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \
>   	ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \
> -	ttm_device.o ttm_sys_manager.o
> +	ttm_device.o ttm_sys_manager.o ttm_backup.o
>   ttm-$(CONFIG_AGP) += ttm_agp_backend.o
>   
>   obj-$(CONFIG_DRM_TTM) += ttm.o
> diff --git a/drivers/gpu/drm/ttm/ttm_backup.c b/drivers/gpu/drm/ttm/ttm_backup.c
> new file mode 100644
> index 000000000000..93c007f18855
> --- /dev/null
> +++ b/drivers/gpu/drm/ttm/ttm_backup.c
> @@ -0,0 +1,207 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright © 2024 Intel Corporation
> + */
> +
> +#include <drm/ttm/ttm_backup.h>
> +#include <linux/page-flags.h>
> +#include <linux/swap.h>
> +
> +/*
> + * Casting from randomized struct file * to struct ttm_backup * is fine since
> + * struct ttm_backup is never defined nor dereferenced.
> + */
> +static struct file *ttm_backup_to_file(struct ttm_backup *backup)
> +{
> +	return (void *)backup;
> +}
> +
> +static struct ttm_backup *ttm_file_to_backup(struct file *file)
> +{
> +	return (void *)file;
> +}
> +
> +/*
> + * Need to map shmem indices to handle since a handle value
> + * of 0 means error, following the swp_entry_t convention.
> + */
> +static unsigned long ttm_backup_shmem_idx_to_handle(pgoff_t idx)
> +{
> +	return (unsigned long)idx + 1;
> +}
> +
> +static pgoff_t ttm_backup_handle_to_shmem_idx(pgoff_t handle)
> +{
> +	return handle - 1;
> +}
> +
> +/**
> + * ttm_backup_drop() - release memory associated with a handle
> + * @backup: The struct backup pointer used to obtain the handle
> + * @handle: The handle obtained from the @backup_page function.
> + */
> +void ttm_backup_drop(struct ttm_backup *backup, pgoff_t handle)
> +{
> +	loff_t start = ttm_backup_handle_to_shmem_idx(handle);
> +
> +	start <<= PAGE_SHIFT;
> +	shmem_truncate_range(file_inode(ttm_backup_to_file(backup)), start,
> +			     start + PAGE_SIZE - 1);
> +}
> +
> +/**
> + * ttm_backup_copy_page() - Copy the contents of a previously backed
> + * up page
> + * @backup: The struct backup pointer used to back up the page.
> + * @dst: The struct page to copy into.
> + * @handle: The handle returned when the page was backed up.
> + * @intr: Try to perform waits interruptable or at least killable.
> + *
> + * Return: 0 on success, Negative error code on failure, notably
> + * -EINTR if @intr was set to true and a signal is pending.
> + */
> +int ttm_backup_copy_page(struct ttm_backup *backup, struct page *dst,
> +			 pgoff_t handle, bool intr)
> +{
> +	struct file *filp = ttm_backup_to_file(backup);
> +	struct address_space *mapping = filp->f_mapping;
> +	struct folio *from_folio;
> +	pgoff_t idx = ttm_backup_handle_to_shmem_idx(handle);
> +
> +	from_folio = shmem_read_folio(mapping, idx);
> +	if (IS_ERR(from_folio))
> +		return PTR_ERR(from_folio);
> +
> +	copy_highpage(dst, folio_file_page(from_folio, idx));
> +	folio_put(from_folio);
> +
> +	return 0;
> +}
> +
> +/**
> + * ttm_backup_backup_page() - Backup a page
> + * @backup: The struct backup pointer to use.
> + * @page: The page to back up.
> + * @writeback: Whether to perform immediate writeback of the page.
> + * This may have performance implications.
> + * @idx: A unique integer for each page and each struct backup.
> + * This allows the backup implementation to avoid managing
> + * its address space separately.
> + * @page_gfp: The gfp value used when the page was allocated.
> + * This is used for accounting purposes.
> + * @alloc_gfp: The gfp to be used when allocating memory.
> + *
> + * Context: If called from reclaim context, the caller needs to
> + * assert that the shrinker gfp has __GFP_FS set, to avoid
> + * deadlocking on lock_page(). If @writeback is set to true and
> + * called from reclaim context, the caller also needs to assert
> + * that the shrinker gfp has __GFP_IO set, since without it,
> + * we're not allowed to start backup IO.
> + *
> + * Return: A handle on success. Negative error code on failure.
> + *
> + * Note: This function could be extended to back up a folio and
> + * implementations would then split the folio internally if needed.
> + * Drawback is that the caller would then have to keep track of
> + * the folio size- and usage.
> + */
> +s64
> +ttm_backup_backup_page(struct ttm_backup *backup, struct page *page,
> +		       bool writeback, pgoff_t idx, gfp_t page_gfp,
> +		       gfp_t alloc_gfp)
> +{
> +	struct file *filp = ttm_backup_to_file(backup);
> +	struct address_space *mapping = filp->f_mapping;
> +	unsigned long handle = 0;
> +	struct folio *to_folio;
> +	int ret;
> +
> +	to_folio = shmem_read_folio_gfp(mapping, idx, alloc_gfp);
> +	if (IS_ERR(to_folio))
> +		return PTR_ERR(to_folio);
> +
> +	folio_mark_accessed(to_folio);
> +	folio_lock(to_folio);
> +	folio_mark_dirty(to_folio);
> +	copy_highpage(folio_file_page(to_folio, idx), page);
> +	handle = ttm_backup_shmem_idx_to_handle(idx);
> +
> +	if (writeback && !folio_mapped(to_folio) &&
> +	    folio_clear_dirty_for_io(to_folio)) {
> +		struct writeback_control wbc = {
> +			.sync_mode = WB_SYNC_NONE,
> +			.nr_to_write = SWAP_CLUSTER_MAX,
> +			.range_start = 0,
> +			.range_end = LLONG_MAX,
> +			.for_reclaim = 1,
> +		};
> +		folio_set_reclaim(to_folio);
> +		ret = mapping->a_ops->writepage(folio_file_page(to_folio, idx), &wbc);
> +		if (!folio_test_writeback(to_folio))
> +			folio_clear_reclaim(to_folio);
> +		/*
> +		 * If writepage succeeds, it unlocks the folio.
> +		 * writepage() errors are otherwise dropped, since writepage()
> +		 * is only best effort here.
> +		 */
> +		if (ret)
> +			folio_unlock(to_folio);
> +	} else {
> +		folio_unlock(to_folio);
> +	}
> +
> +	folio_put(to_folio);
> +
> +	return handle;
> +}
> +
> +/**
> + * ttm_backup_fini() - Free the struct backup resources after last use.
> + * @backup: Pointer to the struct backup whose resources to free.
> + *
> + * After a call to this function, it's illegal to use the @backup pointer.
> + */
> +void ttm_backup_fini(struct ttm_backup *backup)
> +{
> +	fput(ttm_backup_to_file(backup));
> +}
> +
> +/**
> + * ttm_backup_bytes_avail() - Report the approximate number of bytes of backup space
> + * left for backup.
> + *
> + * This function is intended also for driver use to indicate whether a
> + * backup attempt is meaningful.
> + *
> + * Return: An approximate size of backup space available.
> + */
> +u64 ttm_backup_bytes_avail(void)
> +{
> +	/*
> +	 * The idea behind backing up to shmem is that shmem objects may
> +	 * eventually be swapped out. So no point swapping out if there
> +	 * is no or low swap-space available. But the accuracy of this
> +	 * number also depends on shmem actually swapping out backed-up
> +	 * shmem objects without too much buffering.
> +	 */
> +	return (u64)get_nr_swap_pages() << PAGE_SHIFT;
> +}
> +EXPORT_SYMBOL_GPL(ttm_backup_bytes_avail);
> +
> +/**
> + * ttm_backup_shmem_create() - Create a shmem-based struct backup.
> + * @size: The maximum size (in bytes) to back up.
> + *
> + * Create a backup utilizing shmem objects.
> + *
> + * Return: A pointer to a struct ttm_backup on success,
> + * an error pointer on error.
> + */
> +struct ttm_backup *ttm_backup_shmem_create(loff_t size)
> +{
> +	struct file *filp;
> +
> +	filp = shmem_file_setup("ttm shmem backup", size, 0);
> +
> +	return ttm_file_to_backup(filp);
> +}
> diff --git a/include/drm/ttm/ttm_backup.h b/include/drm/ttm/ttm_backup.h
> new file mode 100644
> index 000000000000..24ad120b8827
> --- /dev/null
> +++ b/include/drm/ttm/ttm_backup.h
> @@ -0,0 +1,74 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2024 Intel Corporation
> + */
> +
> +#ifndef _TTM_BACKUP_H_
> +#define _TTM_BACKUP_H_
> +
> +#include <linux/mm_types.h>
> +#include <linux/shmem_fs.h>
> +
> +struct ttm_backup;
> +
> +/**
> + * ttm_backup_handle_to_page_ptr() - Convert handle to struct page pointer
> + * @handle: The handle to convert.
> + *
> + * Converts an opaque handle received from the
> + * struct ttm_backoup_ops::backup_page() function to an (invalid)
> + * struct page pointer suitable for a struct page array.
> + *
> + * Return: An (invalid) struct page pointer.
> + */
> +static inline struct page *
> +ttm_backup_handle_to_page_ptr(unsigned long handle)
> +{
> +	return (struct page *)(handle << 1 | 1);
> +}
> +
> +/**
> + * ttm_backup_page_ptr_is_handle() - Whether a struct page pointer is a handle
> + * @page: The struct page pointer to check.
> + *
> + * Return: true if the struct page pointer is a handld returned from
> + * ttm_backup_handle_to_page_ptr(). False otherwise.
> + */
> +static inline bool ttm_backup_page_ptr_is_handle(const struct page *page)
> +{
> +	return (unsigned long)page & 1;
> +}
> +
> +/**
> + * ttm_backup_page_ptr_to_handle() - Convert a struct page pointer to a handle
> + * @page: The struct page pointer to convert
> + *
> + * Return: The handle that was previously used in
> + * ttm_backup_handle_to_page_ptr() to obtain a struct page pointer, suitable
> + * for use as argument in the struct ttm_backup_ops drop() or
> + * copy_backed_up_page() functions.
> + */
> +static inline unsigned long
> +ttm_backup_page_ptr_to_handle(const struct page *page)
> +{
> +	WARN_ON(!ttm_backup_page_ptr_is_handle(page));
> +	return (unsigned long)page >> 1;
> +}
> +
> +void ttm_backup_drop(struct ttm_backup *backup, pgoff_t handle);
> +
> +int ttm_backup_copy_page(struct ttm_backup *backup, struct page *dst,
> +			 pgoff_t handle, bool intr);
> +
> +s64
> +ttm_backup_backup_page(struct ttm_backup *backup, struct page *page,
> +		       bool writeback, pgoff_t idx, gfp_t page_gfp,
> +		       gfp_t alloc_gfp);
> +
> +void ttm_backup_fini(struct ttm_backup *backup);
> +
> +u64 ttm_backup_bytes_avail(void);
> +
> +struct ttm_backup *ttm_backup_shmem_create(loff_t size);
> +
> +#endif

[PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Thomas Hellström]

Provide a helper to shrink ttm_tt page-vectors on a per-page
basis. A ttm_backup backend could then in theory get away with
allocating a single temporary page for each struct ttm_tt.

This is accomplished by splitting larger pages before trying to
back them up.

In the future we could allow ttm_backup to handle backing up
large pages as well, but currently there's no benefit in
doing that, since the shmem backup backend would have to
split those anyway to avoid allocating too much temporary
memory, and if the backend instead inserts pages into the
swap-cache, those are split on reclaim by the core.

Due to potential backup- and recover errors, allow partially swapped
out struct ttm_tt's, although mark them as swapped out stopping them
from being swapped out a second time. More details in the ttm_pool.c
DOC section.

v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface.
v8:
- Use a struct for flags for readability (Matt Brost)
- Address misc other review comments (Matt Brost)
v9:
- Update the kerneldoc for the ttm_tt::backup field.
v10:
- Rebase.
v13:
- Rebase on ttm_backup interface change. Update kerneldoc.
- Rebase and adjust ttm_tt_is_swapped().
v15:
- Rebase on ttm_backup return value change.
- Rebase on previous restructuring of ttm_pool_alloc()
- Rework the ttm_pool backup interface (Christian König)
- Remove cond_resched() (Christian König)
- Get rid of the need to allocate an intermediate page array
  when restoring a multi-order page (Christian König)
- Update documentation.

Cc: Christian König <christian.koenig@amd.com>
Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <dri-devel@lists.freedesktop.org>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 554 +++++++++++++++++++++++++++++----
 drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
 include/drm/ttm/ttm_pool.h     |   8 +
 include/drm/ttm/ttm_tt.h       |  67 +++-
 4 files changed, 629 insertions(+), 54 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index c9eba76d5143..ffb7abf52bab 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -41,6 +41,7 @@
 #include <asm/set_memory.h>
 #endif
 
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <drm/ttm/ttm_bo.h>
@@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
 	enum ttm_caching tt_caching;
 };
 
+/**
+ * struct ttm_pool_tt_restore - State representing restore from backup
+ * @pool: The pool used for page allocation while restoring.
+ * @snapshot_alloc: A snapshot of the most recent struct ttm_pool_alloc_state.
+ * @alloced_page: Pointer to the page most recently allocated from a pool or system.
+ * @first_dma: The dma address corresponding to @alloced_page if dma_mapping
+ * is requested.
+ * @alloced_pages: The number of allocated pages present in the struct ttm_tt
+ * page vector from this restore session.
+ * @restored_pages: The number of 4K pages restored for @alloced_page (which
+ * is typically a multi-order page).
+ * @page_caching: The struct ttm_tt requested caching
+ * @order: The order of @alloced_page.
+ *
+ * Recovery from backup might fail when we've recovered less than the
+ * full ttm_tt. In order not to loose any data (yet), keep information
+ * around that allows us to restart a failed ttm backup recovery.
+ */
+struct ttm_pool_tt_restore {
+	struct ttm_pool *pool;
+	struct ttm_pool_alloc_state snapshot_alloc;
+	struct page *alloced_page;
+	dma_addr_t first_dma;
+	pgoff_t alloced_pages;
+	pgoff_t restored_pages;
+	enum ttm_caching page_caching;
+	unsigned int order;
+};
+
 static unsigned long page_pool_size;
 
 MODULE_PARM_DESC(page_pool_size, "Number of pages in the WC/UC/DMA pool");
@@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct ttm_pool_alloc_state *alloc)
 	return 0;
 }
 
-/* Map pages of 1 << order size and fill the DMA address array  */
+/* DMA Map pages of 1 << order size and return the resulting dma_address. */
 static int ttm_pool_map(struct ttm_pool *pool, unsigned int order,
-			struct page *p, dma_addr_t **dma_addr)
+			struct page *p, dma_addr_t *dma_addr)
 {
 	dma_addr_t addr;
-	unsigned int i;
 
 	if (pool->use_dma_alloc) {
 		struct ttm_pool_dma *dma = (void *)p->private;
@@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool *pool, unsigned int order,
 			return -EFAULT;
 	}
 
-	for (i = 1 << order; i ; --i) {
-		*(*dma_addr)++ = addr;
-		addr += PAGE_SIZE;
-	}
+	*dma_addr = addr;
 
 	return 0;
 }
@@ -371,6 +397,190 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 	return p->private;
 }
 
+/*
+ * Split larger pages so that we can free each PAGE_SIZE page as soon
+ * as it has been backed up, in order to avoid memory pressure during
+ * reclaim.
+ */
+static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct page *p)
+{
+	unsigned int order = ttm_pool_page_order(pool, p);
+	pgoff_t nr;
+
+	if (!order)
+		return;
+
+	split_page(p, order);
+	nr = 1UL << order;
+	while (nr--)
+		(p++)->private = 0;
+}
+
+/**
+ * DOC: Partial backup and restoration of a struct ttm_tt.
+ *
+ * Swapout using ttm_backup_backup_page() and swapin using
+ * ttm_backup_copy_page() may fail.
+ * The former most likely due to lack of swap-space or memory, the latter due
+ * to lack of memory or because of signal interruption during waits.
+ *
+ * Backup failure is easily handled by using a ttm_tt pages vector that holds
+ * both backup handles and page pointers. This has to be taken into account when
+ * restoring such a ttm_tt from backup, and when freeing it while backed up.
+ * When restoring, for simplicity, new pages are actually allocated from the
+ * pool and the contents of any old pages are copied in and then the old pages
+ * are released.
+ *
+ * For restoration failures, the struct ttm_pool_tt_restore holds sufficient state
+ * to be able to resume an interrupted restore, and that structure is freed once
+ * the restoration is complete. If the struct ttm_tt is destroyed while there
+ * is a valid struct ttm_pool_tt_restore attached, that is also properly taken
+ * care of.
+ */
+
+/* Is restore ongoing for the currently allocated page? */
+static bool ttm_pool_restore_valid(const struct ttm_pool_tt_restore *restore)
+{
+	return restore && restore->restored_pages < (1 << restore->order);
+}
+
+/* DMA unmap and free a multi-order page, either to the relevant pool or to system. */
+static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool, struct page *page,
+				       const dma_addr_t *dma_addr, enum ttm_caching caching)
+{
+	struct ttm_pool_type *pt = NULL;
+	unsigned int order;
+	pgoff_t nr;
+
+	if (pool) {
+		order = ttm_pool_page_order(pool, page);
+		nr = (1UL << order);
+		if (dma_addr)
+			ttm_pool_unmap(pool, *dma_addr, nr);
+
+		pt = ttm_pool_select_type(pool, caching, order);
+	} else {
+		order = page->private;
+		nr = (1UL << order);
+	}
+
+	if (pt)
+		ttm_pool_type_give(pt, page);
+	else
+		ttm_pool_free_page(pool, caching, order, page);
+
+	return nr;
+}
+
+/* Populate the page-array using the most recent allocated multi-order page. */
+static void ttm_pool_allocated_page_commit(struct page *allocated,
+					   dma_addr_t first_dma,
+					   struct ttm_pool_alloc_state *alloc,
+					   pgoff_t nr)
+{
+	pgoff_t i;
+
+	for (i = 0; i < nr; ++i)
+		*alloc->pages++ = allocated++;
+
+	alloc->remaining_pages -= nr;
+
+	if (!alloc->dma_addr)
+		return;
+
+	for (i = 0; i < nr; ++i) {
+		*alloc->dma_addr++ = first_dma;
+		first_dma += PAGE_SIZE;
+	}
+}
+
+/*
+ * When restoring, restore backed-up content to the newly allocated page and
+ * if successful, populate the page-table and dma-address arrays.
+ */
+static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,
+				   struct ttm_backup *backup,
+				   const struct ttm_operation_ctx *ctx,
+				   struct ttm_pool_alloc_state *alloc)
+
+{
+	pgoff_t i, nr = 1UL << restore->order;
+	struct page **first_page = alloc->pages;
+	struct page *p;
+	int ret = 0;
+
+	for (i = restore->restored_pages; i < nr; ++i) {
+		p = first_page[i];
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
+
+			if (handle == 0) {
+				restore->restored_pages++;
+				continue;
+			}
+
+			ret = ttm_backup_copy_page(backup, restore->alloced_page + i,
+						   handle, ctx->interruptible);
+			if (ret)
+				break;
+
+			ttm_backup_drop(backup, handle);
+		} else if (p) {
+			/*
+			 * We could probably avoid splitting the old page
+			 * using clever logic, but ATM we don't care, as
+			 * we prioritize releasing memory ASAP. Note that
+			 * here, the old retained page is always write-back
+			 * cached.
+			 */
+			ttm_pool_split_for_swap(restore->pool, p);
+			copy_highpage(restore->alloced_page + i, p);
+			__free_pages(p, 0);
+		}
+
+		restore->restored_pages++;
+		first_page[i] = ttm_backup_handle_to_page_ptr(0);
+	}
+
+	if (ret) {
+		if (!restore->restored_pages) {
+			dma_addr_t *dma_addr = alloc->dma_addr ? &restore->first_dma : NULL;
+
+			ttm_pool_unmap_and_free(restore->pool, restore->alloced_page,
+						dma_addr, restore->page_caching);
+			restore->restored_pages = nr;
+		}
+		return ret;
+	}
+
+	ttm_pool_allocated_page_commit(restore->alloced_page, restore->first_dma,
+				       alloc, nr);
+	if (restore->page_caching == alloc->tt_caching || PageHighMem(restore->alloced_page))
+		alloc->caching_divide = alloc->pages;
+	restore->snapshot_alloc = *alloc;
+	restore->alloced_pages += nr;
+
+	return 0;
+}
+
+/* If restoring, save information needed for ttm_pool_restore_commit(). */
+static void
+ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned int order,
+				struct page *p,
+				enum ttm_caching page_caching,
+				dma_addr_t first_dma,
+				struct ttm_pool_tt_restore *restore,
+				const struct ttm_pool_alloc_state *alloc)
+{
+	restore->pool = pool;
+	restore->order = order;
+	restore->restored_pages = 0;
+	restore->page_caching = page_caching;
+	restore->first_dma = first_dma;
+	restore->alloced_page = p;
+	restore->snapshot_alloc = *alloc;
+}
+
 /*
  * Called when we got a page, either from a pool or newly allocated.
  * if needed, dma map the page and populate the dma address array.
@@ -380,10 +590,11 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
  */
 static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 				   struct page *p, enum ttm_caching page_caching,
-				   struct ttm_pool_alloc_state *alloc)
+				   struct ttm_pool_alloc_state *alloc,
+				   struct ttm_pool_tt_restore *restore)
 {
-	pgoff_t i, nr = 1UL << order;
 	bool caching_consistent;
+	dma_addr_t first_dma;
 	int r = 0;
 
 	caching_consistent = (page_caching == alloc->tt_caching) || PageHighMem(p);
@@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 	}
 
 	if (alloc->dma_addr) {
-		r = ttm_pool_map(pool, order, p, &alloc->dma_addr);
+		r = ttm_pool_map(pool, order, p, &first_dma);
 		if (r)
 			return r;
 	}
 
-	alloc->remaining_pages -= nr;
-	for (i = 0; i < nr; ++i)
-		*alloc->pages++ = p++;
+	if (restore) {
+		ttm_pool_page_allocated_restore(pool, order, p, page_caching,
+						first_dma, restore, alloc);
+	} else {
+		ttm_pool_allocated_page_commit(p, first_dma, alloc, 1UL << order);
 
-	if (caching_consistent)
-		alloc->caching_divide = alloc->pages;
+		if (caching_consistent)
+			alloc->caching_divide = alloc->pages;
+	}
 
 	return 0;
 }
@@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt,
 				pgoff_t start_page, pgoff_t end_page)
 {
 	struct page **pages = &tt->pages[start_page];
-	unsigned int order;
+	struct ttm_backup *backup = tt->backup;
 	pgoff_t i, nr;
 
 	for (i = start_page; i < end_page; i += nr, pages += nr) {
-		struct ttm_pool_type *pt = NULL;
+		struct page *p = *pages;
 
-		order = ttm_pool_page_order(pool, *pages);
-		nr = (1UL << order);
-		if (tt->dma_address)
-			ttm_pool_unmap(pool, tt->dma_address[i], nr);
+		nr = 1;
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
 
-		pt = ttm_pool_select_type(pool, caching, order);
-		if (pt)
-			ttm_pool_type_give(pt, *pages);
-		else
-			ttm_pool_free_page(pool, caching, order, *pages);
+			if (handle != 0)
+				ttm_backup_drop(backup, handle);
+		} else if (p) {
+			dma_addr_t *dma_addr = tt->dma_address ?
+				tt->dma_address + i : NULL;
+
+			nr = ttm_pool_unmap_and_free(pool, p, dma_addr, caching);
+		}
 	}
 }
 
@@ -467,22 +683,11 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
-/**
- * ttm_pool_alloc - Fill a ttm_tt object
- *
- * @pool: ttm_pool to use
- * @tt: ttm_tt object to fill
- * @ctx: operation context
- *
- * Fill the ttm_tt object with pages and also make sure to DMA map them when
- * necessary.
- *
- * Returns: 0 on successe, negative error code otherwise.
- */
-int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
-		   struct ttm_operation_ctx *ctx)
+static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
+			    const struct ttm_operation_ctx *ctx,
+			    struct ttm_pool_alloc_state *alloc,
+			    struct ttm_pool_tt_restore *restore)
 {
-	struct ttm_pool_alloc_state alloc;
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	struct page *p;
 	int r;
 
-	ttm_pool_alloc_state_init(tt, &alloc);
-
-	WARN_ON(!alloc.remaining_pages || ttm_tt_is_populated(tt));
-	WARN_ON(alloc.dma_addr && !pool->dev);
+	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
+	WARN_ON(alloc->dma_addr && !pool->dev);
 
 	if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
 		gfp_flags |= __GFP_ZERO;
@@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 	page_caching = tt->caching;
 	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, &alloc);
-	     alloc.remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, &alloc)) {
+	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	     alloc->remaining_pages;
+	     order = ttm_pool_alloc_find_order(order, alloc)) {
 		struct ttm_pool_type *pt;
 
 		/* First, try to allocate a page from a pool if one exists. */
@@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			r = -ENOMEM;
 			goto error_free_all;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, &alloc);
+		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
+					    restore);
 		if (r)
 			goto error_free_page;
+
+		if (ttm_pool_restore_valid(restore)) {
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			if (r)
+				goto error_free_all;
+		}
 	}
 
-	r = ttm_pool_apply_caching(&alloc);
+	r = ttm_pool_apply_caching(alloc);
 	if (r)
 		goto error_free_all;
 
+	kfree(tt->restore);
+	tt->restore = NULL;
+
 	return 0;
 
 error_free_page:
 	ttm_pool_free_page(pool, page_caching, order, p);
 
 error_free_all:
-	caching_divide = alloc.caching_divide - tt->pages;
+	if (tt->restore)
+		return r;
+
+	caching_divide = alloc->caching_divide - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
-			    tt->num_pages - alloc.remaining_pages);
+			    tt->num_pages - alloc->remaining_pages);
 
 	return r;
 }
+
+/**
+ * ttm_pool_alloc - Fill a ttm_tt object
+ *
+ * @pool: ttm_pool to use
+ * @tt: ttm_tt object to fill
+ * @ctx: operation context
+ *
+ * Fill the ttm_tt object with pages and also make sure to DMA map them when
+ * necessary.
+ *
+ * Returns: 0 on successe, negative error code otherwise.
+ */
+int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
+		   struct ttm_operation_ctx *ctx)
+{
+	struct ttm_pool_alloc_state alloc;
+
+	if (WARN_ON(ttm_tt_is_backed_up(tt)))
+		return -EINVAL;
+
+	ttm_pool_alloc_state_init(tt, &alloc);
+
+	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+}
 EXPORT_SYMBOL(ttm_pool_alloc);
 
+/**
+ * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring previously backed-up
+ * content.
+ *
+ * @pool: ttm_pool to use
+ * @tt: ttm_tt object to fill
+ * @ctx: operation context
+ *
+ * Fill the ttm_tt object with pages and also make sure to DMA map them when
+ * necessary. Read in backed-up content.
+ *
+ * Returns: 0 on successe, negative error code otherwise.
+ */
+int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
+			       const struct ttm_operation_ctx *ctx)
+{
+	struct ttm_pool_alloc_state alloc;
+
+	if (WARN_ON(!ttm_tt_is_backed_up(tt)))
+		return -EINVAL;
+
+	if (!tt->restore) {
+		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
+
+		ttm_pool_alloc_state_init(tt, &alloc);
+		if (ctx->gfp_retry_mayfail)
+			gfp |= __GFP_RETRY_MAYFAIL;
+
+		tt->restore = kzalloc(sizeof(*tt->restore), gfp);
+		if (!tt->restore)
+			return -ENOMEM;
+
+		tt->restore->snapshot_alloc = alloc;
+		tt->restore->pool = pool;
+		tt->restore->restored_pages = 1;
+	} else {
+		struct ttm_pool_tt_restore *restore = tt->restore;
+		int ret;
+
+		alloc = restore->snapshot_alloc;
+		if (ttm_pool_restore_valid(tt->restore)) {
+			ret = ttm_pool_restore_commit(restore, tt->backup, ctx, &alloc);
+			if (ret)
+				return ret;
+		}
+		if (!alloc.remaining_pages)
+			return 0;
+	}
+
+	return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt->restore);
+}
+
 /**
  * ttm_pool_free - Free the backing pages from a ttm_tt object
  *
@@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)
 }
 EXPORT_SYMBOL(ttm_pool_free);
 
+/**
+ * ttm_pool_drop_backed_up() - Release content of a swapped-out struct ttm_tt
+ * @tt: The struct ttm_tt.
+ *
+ * Release handles with associated content or any remaining pages of
+ * a backed-up struct ttm_tt.
+ */
+void ttm_pool_drop_backed_up(struct ttm_tt *tt)
+{
+	struct ttm_pool_tt_restore *restore;
+	pgoff_t start_page = 0;
+
+	WARN_ON(!ttm_tt_is_backed_up(tt));
+
+	restore = tt->restore;
+
+	/*
+	 * Unmap and free any uncommitted restore page.
+	 * any tt page-array backup entries already read back has
+	 * been cleared already
+	 */
+	if (ttm_pool_restore_valid(restore)) {
+		dma_addr_t *dma_addr = tt->dma_address ? &restore->first_dma : NULL;
+
+		ttm_pool_unmap_and_free(restore->pool, restore->alloced_page,
+					dma_addr, restore->page_caching);
+		restore->restored_pages = 1UL << restore->order;
+	}
+
+	/*
+	 * If a restore is ongoing, part of the tt pages may have a
+	 * caching different than writeback.
+	 */
+	if (restore) {
+		pgoff_t mid = restore->snapshot_alloc.caching_divide - tt->pages;
+
+		start_page = restore->alloced_pages;
+		WARN_ON(mid > start_page);
+		/* Pages that might be dma-mapped and non-cached */
+		ttm_pool_free_range(restore->pool, tt, tt->caching,
+				    0, mid);
+		/* Pages that might be dma-mapped but cached */
+		ttm_pool_free_range(restore->pool, tt, ttm_cached,
+				    mid, restore->alloced_pages);
+		kfree(restore);
+		tt->restore = NULL;
+	}
+
+	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt->num_pages);
+}
+
+/**
+ * ttm_pool_backup() - Back up or purge a struct ttm_tt
+ * @pool: The pool used when allocating the struct ttm_tt.
+ * @tt: The struct ttm_tt.
+ * @flags: Flags to govern the backup behaviour.
+ *
+ * Back up or purge a struct ttm_tt. If @purge is true, then
+ * all pages will be freed directly to the system rather than to the pool
+ * they were allocated from, making the function behave similarly to
+ * ttm_pool_free(). If @purge is false the pages will be backed up instead,
+ * exchanged for handles.
+ * A subsequent call to ttm_pool_restore_and_alloc() will then read back the content and
+ * a subsequent call to ttm_pool_drop_backed_up() will drop it.
+ * If backup of a page fails for whatever reason, @ttm will still be
+ * partially backed up, retaining those pages for which backup fails.
+ * In that case, this function can be retried, possibly after freeing up
+ * memory resources.
+ *
+ * Return: Number of pages actually backed up or freed, or negative
+ * error code on error.
+ */
+long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
+		     const struct ttm_backup_flags *flags)
+{
+	struct ttm_backup *backup = tt->backup;
+	struct page *page;
+	unsigned long handle;
+	gfp_t alloc_gfp;
+	gfp_t gfp;
+	int ret = 0;
+	pgoff_t shrunken = 0;
+	pgoff_t i, num_pages;
+
+	if (WARN_ON(ttm_tt_is_backed_up(tt)))
+		return -EINVAL;
+
+	if ((!ttm_backup_bytes_avail() && !flags->purge) ||
+	    pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
+		return -EBUSY;
+
+#ifdef CONFIG_X86
+	/* Anything returned to the system needs to be cached. */
+	if (tt->caching != ttm_cached)
+		set_pages_array_wb(tt->pages, tt->num_pages);
+#endif
+
+	if (tt->dma_address || flags->purge) {
+		for (i = 0; i < tt->num_pages; i += num_pages) {
+			unsigned int order;
+
+			page = tt->pages[i];
+			if (unlikely(!page)) {
+				num_pages = 1;
+				continue;
+			}
+
+			order = ttm_pool_page_order(pool, page);
+			num_pages = 1UL << order;
+			if (tt->dma_address)
+				ttm_pool_unmap(pool, tt->dma_address[i],
+					       num_pages);
+			if (flags->purge) {
+				shrunken += num_pages;
+				page->private = 0;
+				__free_pages(page, order);
+				memset(tt->pages + i, 0,
+				       num_pages * sizeof(*tt->pages));
+			}
+		}
+	}
+
+	if (flags->purge)
+		return shrunken;
+
+	if (pool->use_dma32)
+		gfp = GFP_DMA32;
+	else
+		gfp = GFP_HIGHUSER;
+
+	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
+
+	for (i = 0; i < tt->num_pages; ++i) {
+		s64 shandle;
+
+		page = tt->pages[i];
+		if (unlikely(!page))
+			continue;
+
+		ttm_pool_split_for_swap(pool, page);
+
+		shandle = ttm_backup_backup_page(backup, page, flags->writeback, i,
+						 gfp, alloc_gfp);
+		if (shandle < 0) {
+			/* We allow partially shrunken tts */
+			ret = shandle;
+			break;
+		}
+		handle = shandle;
+		tt->pages[i] = ttm_backup_handle_to_page_ptr(handle);
+		put_page(page);
+		shrunken++;
+	}
+
+	return shrunken ? shrunken : ret;
+}
+
 /**
  * ttm_pool_init - Initialize a pool
  *
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 3baf215eca23..00b7c28f2329 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -40,6 +40,7 @@
 #include <drm/drm_cache.h>
 #include <drm/drm_device.h>
 #include <drm/drm_util.h>
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_bo.h>
 #include <drm/ttm/ttm_tt.h>
 
@@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm,
 	ttm->swap_storage = NULL;
 	ttm->sg = bo->sg;
 	ttm->caching = caching;
+	ttm->restore = NULL;
+	ttm->backup = NULL;
 }
 
 int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
@@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
 		fput(ttm->swap_storage);
 	ttm->swap_storage = NULL;
 
+	if (ttm_tt_is_backed_up(ttm))
+		ttm_pool_drop_backed_up(ttm);
+	if (ttm->backup) {
+		ttm_backup_fini(ttm->backup);
+		ttm->backup = NULL;
+	}
+
 	if (ttm->pages)
 		kvfree(ttm->pages);
 	else
@@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
 
+/**
+ * ttm_tt_backup() - Helper to back up a struct ttm_tt.
+ * @bdev: The TTM device.
+ * @tt: The struct ttm_tt.
+ * @flags: Flags that govern the backup behaviour.
+ *
+ * Update the page accounting and call ttm_pool_shrink_tt to free pages
+ * or back them up.
+ *
+ * Return: Number of pages freed or swapped out, or negative error code on
+ * error.
+ */
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
+		   const struct ttm_backup_flags flags)
+{
+	long ret;
+
+	if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
+		return 0;
+
+	ret = ttm_pool_backup(&bdev->pool, tt, &flags);
+	if (ret > 0) {
+		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
+		tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
+	}
+
+	return ret;
+}
+
+int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
+		   const struct ttm_operation_ctx *ctx)
+{
+	int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt, ctx);
+
+	if (ret)
+		return ret;
+
+	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_tt_restore);
+
 /**
  * ttm_tt_swapout - swap out tt object
  *
@@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
 		goto error;
 
 	ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
+	ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 	if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
 		ret = ttm_tt_swapin(ttm);
 		if (unlikely(ret != 0)) {
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 160d954a261e..54cd34a6e4c0 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -33,6 +33,7 @@
 
 struct device;
 struct seq_file;
+struct ttm_backup_flags;
 struct ttm_operation_ctx;
 struct ttm_pool;
 struct ttm_tt;
@@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
 
 int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);
 
+void ttm_pool_drop_backed_up(struct ttm_tt *tt);
+
+long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
+		     const struct ttm_backup_flags *flags);
+int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
+			       const struct ttm_operation_ctx *ctx);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 991edafdb2dd..c736c01ac2ca 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -32,11 +32,13 @@
 #include <drm/ttm/ttm_caching.h>
 #include <drm/ttm/ttm_kmap_iter.h>
 
+struct ttm_backup;
 struct ttm_device;
 struct ttm_tt;
 struct ttm_resource;
 struct ttm_buffer_object;
 struct ttm_operation_ctx;
+struct ttm_pool_tt_restore;
 
 /**
  * struct ttm_tt - This is a structure holding the pages, caching- and aperture
@@ -85,17 +87,22 @@ struct ttm_tt {
 	 * fault handling abuses the DMA api a bit and dma_map_attrs can't be
 	 * used to assure pgprot always matches.
 	 *
+	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
+	 * struct ttm_tt has been (possibly partially) backed up.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
+	 *
 	 */
 #define TTM_TT_FLAG_SWAPPED		BIT(0)
 #define TTM_TT_FLAG_ZERO_ALLOC		BIT(1)
 #define TTM_TT_FLAG_EXTERNAL		BIT(2)
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
+#define TTM_TT_FLAG_BACKED_UP	        BIT(5)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -105,11 +112,20 @@ struct ttm_tt {
 	dma_addr_t *dma_address;
 	/** @swap_storage: Pointer to shmem struct file for swap storage. */
 	struct file *swap_storage;
+	/**
+	 * @backup: Pointer to backup struct for backed up tts.
+	 * Could be unified with @swap_storage. Meanwhile, the driver's
+	 * ttm_tt_create() callback is responsible for assigning
+	 * this field.
+	 */
+	struct ttm_backup *backup;
 	/**
 	 * @caching: The current caching state of the pages, see enum
 	 * ttm_caching.
 	 */
 	enum ttm_caching caching;
+	/** @restore: Partial restoration from backup state. TTM private */
+	struct ttm_pool_tt_restore *restore;
 };
 
 /**
@@ -129,9 +145,38 @@ static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
 }
 
+/**
+ * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or backed up
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if swapped or backed up, false otherwise.
+ */
 static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
 {
-	return tt->page_flags & TTM_TT_FLAG_SWAPPED;
+	return tt->page_flags & (TTM_TT_FLAG_SWAPPED | TTM_TT_FLAG_BACKED_UP);
+}
+
+/**
+ * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if swapped or backed up, false otherwise.
+ */
+static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
+}
+
+/**
+ * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
+ * @tt: The struct ttm_tt.
+ *
+ * Drivers can use this functionto clear the backed-up status,
+ * for example before destroying or re-validating a purged tt.
+ */
+static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
+{
+	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
 /**
@@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 					    struct ttm_tt *tt);
 unsigned long ttm_tt_pages_limit(void);
+
+/**
+ * struct ttm_backup_flags - Flags to govern backup behaviour.
+ * @purge: Free pages without backing up. Bypass pools.
+ * @writeback: Attempt to copy contents directly to swap space, even
+ * if that means blocking on writes to external memory.
+ */
+struct ttm_backup_flags {
+	u32 purge : 1;
+	u32 writeback : 1;
+};
+
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
+		   const struct ttm_backup_flags flags);
+
+int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
+		   const struct ttm_operation_ctx *ctx);
+
 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>
 
-- 
2.48.1

Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Christian König]

Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> Provide a helper to shrink ttm_tt page-vectors on a per-page
> basis. A ttm_backup backend could then in theory get away with
> allocating a single temporary page for each struct ttm_tt.
>
> This is accomplished by splitting larger pages before trying to
> back them up.
>
> In the future we could allow ttm_backup to handle backing up
> large pages as well, but currently there's no benefit in
> doing that, since the shmem backup backend would have to
> split those anyway to avoid allocating too much temporary
> memory, and if the backend instead inserts pages into the
> swap-cache, those are split on reclaim by the core.
>
> Due to potential backup- and recover errors, allow partially swapped
> out struct ttm_tt's, although mark them as swapped out stopping them
> from being swapped out a second time. More details in the ttm_pool.c
> DOC section.
>
> v2:
> - A couple of cleanups and error fixes in ttm_pool_back_up_tt.
> - s/back_up/backup/
> - Add a writeback parameter to the exported interface.
> v8:
> - Use a struct for flags for readability (Matt Brost)
> - Address misc other review comments (Matt Brost)
> v9:
> - Update the kerneldoc for the ttm_tt::backup field.
> v10:
> - Rebase.
> v13:
> - Rebase on ttm_backup interface change. Update kerneldoc.
> - Rebase and adjust ttm_tt_is_swapped().
> v15:
> - Rebase on ttm_backup return value change.
> - Rebase on previous restructuring of ttm_pool_alloc()
> - Rework the ttm_pool backup interface (Christian König)
> - Remove cond_resched() (Christian König)
> - Get rid of the need to allocate an intermediate page array
>    when restoring a multi-order page (Christian König)
> - Update documentation.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: <dri-devel@lists.freedesktop.org>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Brost <matthew.brost@intel.com>

I've tried to wrap my head around all of this like twenty times in the 
last three month, but was always interrupted at some point.

Feel free to add Acked-by: Christian Koenig <christian.koenig@amd.com>.

Sorry,
Christian.

> ---
>   drivers/gpu/drm/ttm/ttm_pool.c | 554 +++++++++++++++++++++++++++++----
>   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
>   include/drm/ttm/ttm_pool.h     |   8 +
>   include/drm/ttm/ttm_tt.h       |  67 +++-
>   4 files changed, 629 insertions(+), 54 deletions(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index c9eba76d5143..ffb7abf52bab 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> @@ -41,6 +41,7 @@
>   #include <asm/set_memory.h>
>   #endif
>   
> +#include <drm/ttm/ttm_backup.h>
>   #include <drm/ttm/ttm_pool.h>
>   #include <drm/ttm/ttm_tt.h>
>   #include <drm/ttm/ttm_bo.h>
> @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
>   	enum ttm_caching tt_caching;
>   };
>   
> +/**
> + * struct ttm_pool_tt_restore - State representing restore from backup
> + * @pool: The pool used for page allocation while restoring.
> + * @snapshot_alloc: A snapshot of the most recent struct ttm_pool_alloc_state.
> + * @alloced_page: Pointer to the page most recently allocated from a pool or system.
> + * @first_dma: The dma address corresponding to @alloced_page if dma_mapping
> + * is requested.
> + * @alloced_pages: The number of allocated pages present in the struct ttm_tt
> + * page vector from this restore session.
> + * @restored_pages: The number of 4K pages restored for @alloced_page (which
> + * is typically a multi-order page).
> + * @page_caching: The struct ttm_tt requested caching
> + * @order: The order of @alloced_page.
> + *
> + * Recovery from backup might fail when we've recovered less than the
> + * full ttm_tt. In order not to loose any data (yet), keep information
> + * around that allows us to restart a failed ttm backup recovery.
> + */
> +struct ttm_pool_tt_restore {
> +	struct ttm_pool *pool;
> +	struct ttm_pool_alloc_state snapshot_alloc;
> +	struct page *alloced_page;
> +	dma_addr_t first_dma;
> +	pgoff_t alloced_pages;
> +	pgoff_t restored_pages;
> +	enum ttm_caching page_caching;
> +	unsigned int order;
> +};
> +
>   static unsigned long page_pool_size;
>   
>   MODULE_PARM_DESC(page_pool_size, "Number of pages in the WC/UC/DMA pool");
> @@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct ttm_pool_alloc_state *alloc)
>   	return 0;
>   }
>   
> -/* Map pages of 1 << order size and fill the DMA address array  */
> +/* DMA Map pages of 1 << order size and return the resulting dma_address. */
>   static int ttm_pool_map(struct ttm_pool *pool, unsigned int order,
> -			struct page *p, dma_addr_t **dma_addr)
> +			struct page *p, dma_addr_t *dma_addr)
>   {
>   	dma_addr_t addr;
> -	unsigned int i;
>   
>   	if (pool->use_dma_alloc) {
>   		struct ttm_pool_dma *dma = (void *)p->private;
> @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool *pool, unsigned int order,
>   			return -EFAULT;
>   	}
>   
> -	for (i = 1 << order; i ; --i) {
> -		*(*dma_addr)++ = addr;
> -		addr += PAGE_SIZE;
> -	}
> +	*dma_addr = addr;
>   
>   	return 0;
>   }
> @@ -371,6 +397,190 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>   	return p->private;
>   }
>   
> +/*
> + * Split larger pages so that we can free each PAGE_SIZE page as soon
> + * as it has been backed up, in order to avoid memory pressure during
> + * reclaim.
> + */
> +static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct page *p)
> +{
> +	unsigned int order = ttm_pool_page_order(pool, p);
> +	pgoff_t nr;
> +
> +	if (!order)
> +		return;
> +
> +	split_page(p, order);
> +	nr = 1UL << order;
> +	while (nr--)
> +		(p++)->private = 0;
> +}
> +
> +/**
> + * DOC: Partial backup and restoration of a struct ttm_tt.
> + *
> + * Swapout using ttm_backup_backup_page() and swapin using
> + * ttm_backup_copy_page() may fail.
> + * The former most likely due to lack of swap-space or memory, the latter due
> + * to lack of memory or because of signal interruption during waits.
> + *
> + * Backup failure is easily handled by using a ttm_tt pages vector that holds
> + * both backup handles and page pointers. This has to be taken into account when
> + * restoring such a ttm_tt from backup, and when freeing it while backed up.
> + * When restoring, for simplicity, new pages are actually allocated from the
> + * pool and the contents of any old pages are copied in and then the old pages
> + * are released.
> + *
> + * For restoration failures, the struct ttm_pool_tt_restore holds sufficient state
> + * to be able to resume an interrupted restore, and that structure is freed once
> + * the restoration is complete. If the struct ttm_tt is destroyed while there
> + * is a valid struct ttm_pool_tt_restore attached, that is also properly taken
> + * care of.
> + */
> +
> +/* Is restore ongoing for the currently allocated page? */
> +static bool ttm_pool_restore_valid(const struct ttm_pool_tt_restore *restore)
> +{
> +	return restore && restore->restored_pages < (1 << restore->order);
> +}
> +
> +/* DMA unmap and free a multi-order page, either to the relevant pool or to system. */
> +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool, struct page *page,
> +				       const dma_addr_t *dma_addr, enum ttm_caching caching)
> +{
> +	struct ttm_pool_type *pt = NULL;
> +	unsigned int order;
> +	pgoff_t nr;
> +
> +	if (pool) {
> +		order = ttm_pool_page_order(pool, page);
> +		nr = (1UL << order);
> +		if (dma_addr)
> +			ttm_pool_unmap(pool, *dma_addr, nr);
> +
> +		pt = ttm_pool_select_type(pool, caching, order);
> +	} else {
> +		order = page->private;
> +		nr = (1UL << order);
> +	}
> +
> +	if (pt)
> +		ttm_pool_type_give(pt, page);
> +	else
> +		ttm_pool_free_page(pool, caching, order, page);
> +
> +	return nr;
> +}
> +
> +/* Populate the page-array using the most recent allocated multi-order page. */
> +static void ttm_pool_allocated_page_commit(struct page *allocated,
> +					   dma_addr_t first_dma,
> +					   struct ttm_pool_alloc_state *alloc,
> +					   pgoff_t nr)
> +{
> +	pgoff_t i;
> +
> +	for (i = 0; i < nr; ++i)
> +		*alloc->pages++ = allocated++;
> +
> +	alloc->remaining_pages -= nr;
> +
> +	if (!alloc->dma_addr)
> +		return;
> +
> +	for (i = 0; i < nr; ++i) {
> +		*alloc->dma_addr++ = first_dma;
> +		first_dma += PAGE_SIZE;
> +	}
> +}
> +
> +/*
> + * When restoring, restore backed-up content to the newly allocated page and
> + * if successful, populate the page-table and dma-address arrays.
> + */
> +static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,
> +				   struct ttm_backup *backup,
> +				   const struct ttm_operation_ctx *ctx,
> +				   struct ttm_pool_alloc_state *alloc)
> +
> +{
> +	pgoff_t i, nr = 1UL << restore->order;
> +	struct page **first_page = alloc->pages;
> +	struct page *p;
> +	int ret = 0;
> +
> +	for (i = restore->restored_pages; i < nr; ++i) {
> +		p = first_page[i];
> +		if (ttm_backup_page_ptr_is_handle(p)) {
> +			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
> +
> +			if (handle == 0) {
> +				restore->restored_pages++;
> +				continue;
> +			}
> +
> +			ret = ttm_backup_copy_page(backup, restore->alloced_page + i,
> +						   handle, ctx->interruptible);
> +			if (ret)
> +				break;
> +
> +			ttm_backup_drop(backup, handle);
> +		} else if (p) {
> +			/*
> +			 * We could probably avoid splitting the old page
> +			 * using clever logic, but ATM we don't care, as
> +			 * we prioritize releasing memory ASAP. Note that
> +			 * here, the old retained page is always write-back
> +			 * cached.
> +			 */
> +			ttm_pool_split_for_swap(restore->pool, p);
> +			copy_highpage(restore->alloced_page + i, p);
> +			__free_pages(p, 0);
> +		}
> +
> +		restore->restored_pages++;
> +		first_page[i] = ttm_backup_handle_to_page_ptr(0);
> +	}
> +
> +	if (ret) {
> +		if (!restore->restored_pages) {
> +			dma_addr_t *dma_addr = alloc->dma_addr ? &restore->first_dma : NULL;
> +
> +			ttm_pool_unmap_and_free(restore->pool, restore->alloced_page,
> +						dma_addr, restore->page_caching);
> +			restore->restored_pages = nr;
> +		}
> +		return ret;
> +	}
> +
> +	ttm_pool_allocated_page_commit(restore->alloced_page, restore->first_dma,
> +				       alloc, nr);
> +	if (restore->page_caching == alloc->tt_caching || PageHighMem(restore->alloced_page))
> +		alloc->caching_divide = alloc->pages;
> +	restore->snapshot_alloc = *alloc;
> +	restore->alloced_pages += nr;
> +
> +	return 0;
> +}
> +
> +/* If restoring, save information needed for ttm_pool_restore_commit(). */
> +static void
> +ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned int order,
> +				struct page *p,
> +				enum ttm_caching page_caching,
> +				dma_addr_t first_dma,
> +				struct ttm_pool_tt_restore *restore,
> +				const struct ttm_pool_alloc_state *alloc)
> +{
> +	restore->pool = pool;
> +	restore->order = order;
> +	restore->restored_pages = 0;
> +	restore->page_caching = page_caching;
> +	restore->first_dma = first_dma;
> +	restore->alloced_page = p;
> +	restore->snapshot_alloc = *alloc;
> +}
> +
>   /*
>    * Called when we got a page, either from a pool or newly allocated.
>    * if needed, dma map the page and populate the dma address array.
> @@ -380,10 +590,11 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>    */
>   static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
>   				   struct page *p, enum ttm_caching page_caching,
> -				   struct ttm_pool_alloc_state *alloc)
> +				   struct ttm_pool_alloc_state *alloc,
> +				   struct ttm_pool_tt_restore *restore)
>   {
> -	pgoff_t i, nr = 1UL << order;
>   	bool caching_consistent;
> +	dma_addr_t first_dma;
>   	int r = 0;
>   
>   	caching_consistent = (page_caching == alloc->tt_caching) || PageHighMem(p);
> @@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
>   	}
>   
>   	if (alloc->dma_addr) {
> -		r = ttm_pool_map(pool, order, p, &alloc->dma_addr);
> +		r = ttm_pool_map(pool, order, p, &first_dma);
>   		if (r)
>   			return r;
>   	}
>   
> -	alloc->remaining_pages -= nr;
> -	for (i = 0; i < nr; ++i)
> -		*alloc->pages++ = p++;
> +	if (restore) {
> +		ttm_pool_page_allocated_restore(pool, order, p, page_caching,
> +						first_dma, restore, alloc);
> +	} else {
> +		ttm_pool_allocated_page_commit(p, first_dma, alloc, 1UL << order);
>   
> -	if (caching_consistent)
> -		alloc->caching_divide = alloc->pages;
> +		if (caching_consistent)
> +			alloc->caching_divide = alloc->pages;
> +	}
>   
>   	return 0;
>   }
> @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt,
>   				pgoff_t start_page, pgoff_t end_page)
>   {
>   	struct page **pages = &tt->pages[start_page];
> -	unsigned int order;
> +	struct ttm_backup *backup = tt->backup;
>   	pgoff_t i, nr;
>   
>   	for (i = start_page; i < end_page; i += nr, pages += nr) {
> -		struct ttm_pool_type *pt = NULL;
> +		struct page *p = *pages;
>   
> -		order = ttm_pool_page_order(pool, *pages);
> -		nr = (1UL << order);
> -		if (tt->dma_address)
> -			ttm_pool_unmap(pool, tt->dma_address[i], nr);
> +		nr = 1;
> +		if (ttm_backup_page_ptr_is_handle(p)) {
> +			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
>   
> -		pt = ttm_pool_select_type(pool, caching, order);
> -		if (pt)
> -			ttm_pool_type_give(pt, *pages);
> -		else
> -			ttm_pool_free_page(pool, caching, order, *pages);
> +			if (handle != 0)
> +				ttm_backup_drop(backup, handle);
> +		} else if (p) {
> +			dma_addr_t *dma_addr = tt->dma_address ?
> +				tt->dma_address + i : NULL;
> +
> +			nr = ttm_pool_unmap_and_free(pool, p, dma_addr, caching);
> +		}
>   	}
>   }
>   
> @@ -467,22 +683,11 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
>   	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
>   }
>   
> -/**
> - * ttm_pool_alloc - Fill a ttm_tt object
> - *
> - * @pool: ttm_pool to use
> - * @tt: ttm_tt object to fill
> - * @ctx: operation context
> - *
> - * Fill the ttm_tt object with pages and also make sure to DMA map them when
> - * necessary.
> - *
> - * Returns: 0 on successe, negative error code otherwise.
> - */
> -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> -		   struct ttm_operation_ctx *ctx)
> +static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> +			    const struct ttm_operation_ctx *ctx,
> +			    struct ttm_pool_alloc_state *alloc,
> +			    struct ttm_pool_tt_restore *restore)
>   {
> -	struct ttm_pool_alloc_state alloc;
>   	enum ttm_caching page_caching;
>   	gfp_t gfp_flags = GFP_USER;
>   	pgoff_t caching_divide;
> @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>   	struct page *p;
>   	int r;
>   
> -	ttm_pool_alloc_state_init(tt, &alloc);
> -
> -	WARN_ON(!alloc.remaining_pages || ttm_tt_is_populated(tt));
> -	WARN_ON(alloc.dma_addr && !pool->dev);
> +	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
> +	WARN_ON(alloc->dma_addr && !pool->dev);
>   
>   	if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
>   		gfp_flags |= __GFP_ZERO;
> @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>   
>   	page_caching = tt->caching;
>   	allow_pools = true;
> -	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, &alloc);
> -	     alloc.remaining_pages;
> -	     order = ttm_pool_alloc_find_order(order, &alloc)) {
> +	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
> +	     alloc->remaining_pages;
> +	     order = ttm_pool_alloc_find_order(order, alloc)) {
>   		struct ttm_pool_type *pt;
>   
>   		/* First, try to allocate a page from a pool if one exists. */
> @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>   			r = -ENOMEM;
>   			goto error_free_all;
>   		}
> -		r = ttm_pool_page_allocated(pool, order, p, page_caching, &alloc);
> +		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
> +					    restore);
>   		if (r)
>   			goto error_free_page;
> +
> +		if (ttm_pool_restore_valid(restore)) {
> +			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
> +			if (r)
> +				goto error_free_all;
> +		}
>   	}
>   
> -	r = ttm_pool_apply_caching(&alloc);
> +	r = ttm_pool_apply_caching(alloc);
>   	if (r)
>   		goto error_free_all;
>   
> +	kfree(tt->restore);
> +	tt->restore = NULL;
> +
>   	return 0;
>   
>   error_free_page:
>   	ttm_pool_free_page(pool, page_caching, order, p);
>   
>   error_free_all:
> -	caching_divide = alloc.caching_divide - tt->pages;
> +	if (tt->restore)
> +		return r;
> +
> +	caching_divide = alloc->caching_divide - tt->pages;
>   	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
>   	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
> -			    tt->num_pages - alloc.remaining_pages);
> +			    tt->num_pages - alloc->remaining_pages);
>   
>   	return r;
>   }
> +
> +/**
> + * ttm_pool_alloc - Fill a ttm_tt object
> + *
> + * @pool: ttm_pool to use
> + * @tt: ttm_tt object to fill
> + * @ctx: operation context
> + *
> + * Fill the ttm_tt object with pages and also make sure to DMA map them when
> + * necessary.
> + *
> + * Returns: 0 on successe, negative error code otherwise.
> + */
> +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> +		   struct ttm_operation_ctx *ctx)
> +{
> +	struct ttm_pool_alloc_state alloc;
> +
> +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> +		return -EINVAL;
> +
> +	ttm_pool_alloc_state_init(tt, &alloc);
> +
> +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> +}
>   EXPORT_SYMBOL(ttm_pool_alloc);
>   
> +/**
> + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring previously backed-up
> + * content.
> + *
> + * @pool: ttm_pool to use
> + * @tt: ttm_tt object to fill
> + * @ctx: operation context
> + *
> + * Fill the ttm_tt object with pages and also make sure to DMA map them when
> + * necessary. Read in backed-up content.
> + *
> + * Returns: 0 on successe, negative error code otherwise.
> + */
> +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> +			       const struct ttm_operation_ctx *ctx)
> +{
> +	struct ttm_pool_alloc_state alloc;
> +
> +	if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> +		return -EINVAL;
> +
> +	if (!tt->restore) {
> +		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> +
> +		ttm_pool_alloc_state_init(tt, &alloc);
> +		if (ctx->gfp_retry_mayfail)
> +			gfp |= __GFP_RETRY_MAYFAIL;
> +
> +		tt->restore = kzalloc(sizeof(*tt->restore), gfp);
> +		if (!tt->restore)
> +			return -ENOMEM;
> +
> +		tt->restore->snapshot_alloc = alloc;
> +		tt->restore->pool = pool;
> +		tt->restore->restored_pages = 1;
> +	} else {
> +		struct ttm_pool_tt_restore *restore = tt->restore;
> +		int ret;
> +
> +		alloc = restore->snapshot_alloc;
> +		if (ttm_pool_restore_valid(tt->restore)) {
> +			ret = ttm_pool_restore_commit(restore, tt->backup, ctx, &alloc);
> +			if (ret)
> +				return ret;
> +		}
> +		if (!alloc.remaining_pages)
> +			return 0;
> +	}
> +
> +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt->restore);
> +}
> +
>   /**
>    * ttm_pool_free - Free the backing pages from a ttm_tt object
>    *
> @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)
>   }
>   EXPORT_SYMBOL(ttm_pool_free);
>   
> +/**
> + * ttm_pool_drop_backed_up() - Release content of a swapped-out struct ttm_tt
> + * @tt: The struct ttm_tt.
> + *
> + * Release handles with associated content or any remaining pages of
> + * a backed-up struct ttm_tt.
> + */
> +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> +{
> +	struct ttm_pool_tt_restore *restore;
> +	pgoff_t start_page = 0;
> +
> +	WARN_ON(!ttm_tt_is_backed_up(tt));
> +
> +	restore = tt->restore;
> +
> +	/*
> +	 * Unmap and free any uncommitted restore page.
> +	 * any tt page-array backup entries already read back has
> +	 * been cleared already
> +	 */
> +	if (ttm_pool_restore_valid(restore)) {
> +		dma_addr_t *dma_addr = tt->dma_address ? &restore->first_dma : NULL;
> +
> +		ttm_pool_unmap_and_free(restore->pool, restore->alloced_page,
> +					dma_addr, restore->page_caching);
> +		restore->restored_pages = 1UL << restore->order;
> +	}
> +
> +	/*
> +	 * If a restore is ongoing, part of the tt pages may have a
> +	 * caching different than writeback.
> +	 */
> +	if (restore) {
> +		pgoff_t mid = restore->snapshot_alloc.caching_divide - tt->pages;
> +
> +		start_page = restore->alloced_pages;
> +		WARN_ON(mid > start_page);
> +		/* Pages that might be dma-mapped and non-cached */
> +		ttm_pool_free_range(restore->pool, tt, tt->caching,
> +				    0, mid);
> +		/* Pages that might be dma-mapped but cached */
> +		ttm_pool_free_range(restore->pool, tt, ttm_cached,
> +				    mid, restore->alloced_pages);
> +		kfree(restore);
> +		tt->restore = NULL;
> +	}
> +
> +	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt->num_pages);
> +}
> +
> +/**
> + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> + * @pool: The pool used when allocating the struct ttm_tt.
> + * @tt: The struct ttm_tt.
> + * @flags: Flags to govern the backup behaviour.
> + *
> + * Back up or purge a struct ttm_tt. If @purge is true, then
> + * all pages will be freed directly to the system rather than to the pool
> + * they were allocated from, making the function behave similarly to
> + * ttm_pool_free(). If @purge is false the pages will be backed up instead,
> + * exchanged for handles.
> + * A subsequent call to ttm_pool_restore_and_alloc() will then read back the content and
> + * a subsequent call to ttm_pool_drop_backed_up() will drop it.
> + * If backup of a page fails for whatever reason, @ttm will still be
> + * partially backed up, retaining those pages for which backup fails.
> + * In that case, this function can be retried, possibly after freeing up
> + * memory resources.
> + *
> + * Return: Number of pages actually backed up or freed, or negative
> + * error code on error.
> + */
> +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
> +		     const struct ttm_backup_flags *flags)
> +{
> +	struct ttm_backup *backup = tt->backup;
> +	struct page *page;
> +	unsigned long handle;
> +	gfp_t alloc_gfp;
> +	gfp_t gfp;
> +	int ret = 0;
> +	pgoff_t shrunken = 0;
> +	pgoff_t i, num_pages;
> +
> +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> +		return -EINVAL;
> +
> +	if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> +	    pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> +		return -EBUSY;
> +
> +#ifdef CONFIG_X86
> +	/* Anything returned to the system needs to be cached. */
> +	if (tt->caching != ttm_cached)
> +		set_pages_array_wb(tt->pages, tt->num_pages);
> +#endif
> +
> +	if (tt->dma_address || flags->purge) {
> +		for (i = 0; i < tt->num_pages; i += num_pages) {
> +			unsigned int order;
> +
> +			page = tt->pages[i];
> +			if (unlikely(!page)) {
> +				num_pages = 1;
> +				continue;
> +			}
> +
> +			order = ttm_pool_page_order(pool, page);
> +			num_pages = 1UL << order;
> +			if (tt->dma_address)
> +				ttm_pool_unmap(pool, tt->dma_address[i],
> +					       num_pages);
> +			if (flags->purge) {
> +				shrunken += num_pages;
> +				page->private = 0;
> +				__free_pages(page, order);
> +				memset(tt->pages + i, 0,
> +				       num_pages * sizeof(*tt->pages));
> +			}
> +		}
> +	}
> +
> +	if (flags->purge)
> +		return shrunken;
> +
> +	if (pool->use_dma32)
> +		gfp = GFP_DMA32;
> +	else
> +		gfp = GFP_HIGHUSER;
> +
> +	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
> +
> +	for (i = 0; i < tt->num_pages; ++i) {
> +		s64 shandle;
> +
> +		page = tt->pages[i];
> +		if (unlikely(!page))
> +			continue;
> +
> +		ttm_pool_split_for_swap(pool, page);
> +
> +		shandle = ttm_backup_backup_page(backup, page, flags->writeback, i,
> +						 gfp, alloc_gfp);
> +		if (shandle < 0) {
> +			/* We allow partially shrunken tts */
> +			ret = shandle;
> +			break;
> +		}
> +		handle = shandle;
> +		tt->pages[i] = ttm_backup_handle_to_page_ptr(handle);
> +		put_page(page);
> +		shrunken++;
> +	}
> +
> +	return shrunken ? shrunken : ret;
> +}
> +
>   /**
>    * ttm_pool_init - Initialize a pool
>    *
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 3baf215eca23..00b7c28f2329 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -40,6 +40,7 @@
>   #include <drm/drm_cache.h>
>   #include <drm/drm_device.h>
>   #include <drm/drm_util.h>
> +#include <drm/ttm/ttm_backup.h>
>   #include <drm/ttm/ttm_bo.h>
>   #include <drm/ttm/ttm_tt.h>
>   
> @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm,
>   	ttm->swap_storage = NULL;
>   	ttm->sg = bo->sg;
>   	ttm->caching = caching;
> +	ttm->restore = NULL;
> +	ttm->backup = NULL;
>   }
>   
>   int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
> @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
>   		fput(ttm->swap_storage);
>   	ttm->swap_storage = NULL;
>   
> +	if (ttm_tt_is_backed_up(ttm))
> +		ttm_pool_drop_backed_up(ttm);
> +	if (ttm->backup) {
> +		ttm_backup_fini(ttm->backup);
> +		ttm->backup = NULL;
> +	}
> +
>   	if (ttm->pages)
>   		kvfree(ttm->pages);
>   	else
> @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
>   }
>   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
>   
> +/**
> + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> + * @bdev: The TTM device.
> + * @tt: The struct ttm_tt.
> + * @flags: Flags that govern the backup behaviour.
> + *
> + * Update the page accounting and call ttm_pool_shrink_tt to free pages
> + * or back them up.
> + *
> + * Return: Number of pages freed or swapped out, or negative error code on
> + * error.
> + */
> +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> +		   const struct ttm_backup_flags flags)
> +{
> +	long ret;
> +
> +	if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> +		return 0;
> +
> +	ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> +	if (ret > 0) {
> +		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> +		tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> +	}
> +
> +	return ret;
> +}
> +
> +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> +		   const struct ttm_operation_ctx *ctx)
> +{
> +	int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt, ctx);
> +
> +	if (ret)
> +		return ret;
> +
> +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL(ttm_tt_restore);
> +
>   /**
>    * ttm_tt_swapout - swap out tt object
>    *
> @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
>   		goto error;
>   
>   	ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> +	ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
>   	if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
>   		ret = ttm_tt_swapin(ttm);
>   		if (unlikely(ret != 0)) {
> diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
> index 160d954a261e..54cd34a6e4c0 100644
> --- a/include/drm/ttm/ttm_pool.h
> +++ b/include/drm/ttm/ttm_pool.h
> @@ -33,6 +33,7 @@
>   
>   struct device;
>   struct seq_file;
> +struct ttm_backup_flags;
>   struct ttm_operation_ctx;
>   struct ttm_pool;
>   struct ttm_tt;
> @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
>   
>   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);
>   
> +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> +
> +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
> +		     const struct ttm_backup_flags *flags);
> +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> +			       const struct ttm_operation_ctx *ctx);
> +
>   int ttm_pool_mgr_init(unsigned long num_pages);
>   void ttm_pool_mgr_fini(void);
>   
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index 991edafdb2dd..c736c01ac2ca 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -32,11 +32,13 @@
>   #include <drm/ttm/ttm_caching.h>
>   #include <drm/ttm/ttm_kmap_iter.h>
>   
> +struct ttm_backup;
>   struct ttm_device;
>   struct ttm_tt;
>   struct ttm_resource;
>   struct ttm_buffer_object;
>   struct ttm_operation_ctx;
> +struct ttm_pool_tt_restore;
>   
>   /**
>    * struct ttm_tt - This is a structure holding the pages, caching- and aperture
> @@ -85,17 +87,22 @@ struct ttm_tt {
>   	 * fault handling abuses the DMA api a bit and dma_map_attrs can't be
>   	 * used to assure pgprot always matches.
>   	 *
> +	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
> +	 * struct ttm_tt has been (possibly partially) backed up.
> +	 *
>   	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
>   	 * set by TTM after ttm_tt_populate() has successfully returned, and is
>   	 * then unset when TTM calls ttm_tt_unpopulate().
> +	 *
>   	 */
>   #define TTM_TT_FLAG_SWAPPED		BIT(0)
>   #define TTM_TT_FLAG_ZERO_ALLOC		BIT(1)
>   #define TTM_TT_FLAG_EXTERNAL		BIT(2)
>   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
>   #define TTM_TT_FLAG_DECRYPTED		BIT(4)
> +#define TTM_TT_FLAG_BACKED_UP	        BIT(5)
>   
> -#define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
> +#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
>   	uint32_t page_flags;
>   	/** @num_pages: Number of pages in the page array. */
>   	uint32_t num_pages;
> @@ -105,11 +112,20 @@ struct ttm_tt {
>   	dma_addr_t *dma_address;
>   	/** @swap_storage: Pointer to shmem struct file for swap storage. */
>   	struct file *swap_storage;
> +	/**
> +	 * @backup: Pointer to backup struct for backed up tts.
> +	 * Could be unified with @swap_storage. Meanwhile, the driver's
> +	 * ttm_tt_create() callback is responsible for assigning
> +	 * this field.
> +	 */
> +	struct ttm_backup *backup;
>   	/**
>   	 * @caching: The current caching state of the pages, see enum
>   	 * ttm_caching.
>   	 */
>   	enum ttm_caching caching;
> +	/** @restore: Partial restoration from backup state. TTM private */
> +	struct ttm_pool_tt_restore *restore;
>   };
>   
>   /**
> @@ -129,9 +145,38 @@ static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
>   	return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
>   }
>   
> +/**
> + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or backed up
> + * @tt: The struct ttm_tt.
> + *
> + * Return: true if swapped or backed up, false otherwise.
> + */
>   static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
>   {
> -	return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> +	return tt->page_flags & (TTM_TT_FLAG_SWAPPED | TTM_TT_FLAG_BACKED_UP);
> +}
> +
> +/**
> + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> + * @tt: The struct ttm_tt.
> + *
> + * Return: true if swapped or backed up, false otherwise.
> + */
> +static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
> +{
> +	return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> +}
> +
> +/**
> + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
> + * @tt: The struct ttm_tt.
> + *
> + * Drivers can use this functionto clear the backed-up status,
> + * for example before destroying or re-validating a purged tt.
> + */
> +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
> +{
> +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
>   }
>   
>   /**
> @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
>   					    struct ttm_tt *tt);
>   unsigned long ttm_tt_pages_limit(void);
> +
> +/**
> + * struct ttm_backup_flags - Flags to govern backup behaviour.
> + * @purge: Free pages without backing up. Bypass pools.
> + * @writeback: Attempt to copy contents directly to swap space, even
> + * if that means blocking on writes to external memory.
> + */
> +struct ttm_backup_flags {
> +	u32 purge : 1;
> +	u32 writeback : 1;
> +};
> +
> +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> +		   const struct ttm_backup_flags flags);
> +
> +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> +		   const struct ttm_operation_ctx *ctx);
> +
>   #if IS_ENABLED(CONFIG_AGP)
>   #include <linux/agp_backend.h>
>   

Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Thomas Hellström]

Hi, Christian,

On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
> Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> > Provide a helper to shrink ttm_tt page-vectors on a per-page
> > basis. A ttm_backup backend could then in theory get away with
> > allocating a single temporary page for each struct ttm_tt.
> > 
> > This is accomplished by splitting larger pages before trying to
> > back them up.
> > 
> > In the future we could allow ttm_backup to handle backing up
> > large pages as well, but currently there's no benefit in
> > doing that, since the shmem backup backend would have to
> > split those anyway to avoid allocating too much temporary
> > memory, and if the backend instead inserts pages into the
> > swap-cache, those are split on reclaim by the core.
> > 
> > Due to potential backup- and recover errors, allow partially
> > swapped
> > out struct ttm_tt's, although mark them as swapped out stopping
> > them
> > from being swapped out a second time. More details in the
> > ttm_pool.c
> > DOC section.
> > 
> > v2:
> > - A couple of cleanups and error fixes in ttm_pool_back_up_tt.
> > - s/back_up/backup/
> > - Add a writeback parameter to the exported interface.
> > v8:
> > - Use a struct for flags for readability (Matt Brost)
> > - Address misc other review comments (Matt Brost)
> > v9:
> > - Update the kerneldoc for the ttm_tt::backup field.
> > v10:
> > - Rebase.
> > v13:
> > - Rebase on ttm_backup interface change. Update kerneldoc.
> > - Rebase and adjust ttm_tt_is_swapped().
> > v15:
> > - Rebase on ttm_backup return value change.
> > - Rebase on previous restructuring of ttm_pool_alloc()
> > - Rework the ttm_pool backup interface (Christian König)
> > - Remove cond_resched() (Christian König)
> > - Get rid of the need to allocate an intermediate page array
> >    when restoring a multi-order page (Christian König)
> > - Update documentation.
> > 
> > Cc: Christian König <christian.koenig@amd.com>
> > Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> > Cc: Matthew Brost <matthew.brost@intel.com>
> > Cc: <dri-devel@lists.freedesktop.org>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Reviewed-by: Matthew Brost <matthew.brost@intel.com>
> 
> I've tried to wrap my head around all of this like twenty times in
> the 
> last three month, but was always interrupted at some point.
> 
> Feel free to add Acked-by: Christian Koenig
> <christian.koenig@amd.com>.
> 
> Sorry,
> Christian.

Thanks a lot for all reviewing and comments so far. There are two TTM
patches left in the series that don't have an ack by you:

https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
and

https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17

None of them particularly big considering the amount of doc text.

It'd be great if those could have an ack as well so we could finally
merge this series.

Thanks,
Thomas






> 
> > ---
> >   drivers/gpu/drm/ttm/ttm_pool.c | 554
> > +++++++++++++++++++++++++++++----
> >   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
> >   include/drm/ttm/ttm_pool.h     |   8 +
> >   include/drm/ttm/ttm_tt.h       |  67 +++-
> >   4 files changed, 629 insertions(+), 54 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
> > b/drivers/gpu/drm/ttm/ttm_pool.c
> > index c9eba76d5143..ffb7abf52bab 100644
> > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > @@ -41,6 +41,7 @@
> >   #include <asm/set_memory.h>
> >   #endif
> >   
> > +#include <drm/ttm/ttm_backup.h>
> >   #include <drm/ttm/ttm_pool.h>
> >   #include <drm/ttm/ttm_tt.h>
> >   #include <drm/ttm/ttm_bo.h>
> > @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
> >   	enum ttm_caching tt_caching;
> >   };
> >   
> > +/**
> > + * struct ttm_pool_tt_restore - State representing restore from
> > backup
> > + * @pool: The pool used for page allocation while restoring.
> > + * @snapshot_alloc: A snapshot of the most recent struct
> > ttm_pool_alloc_state.
> > + * @alloced_page: Pointer to the page most recently allocated from
> > a pool or system.
> > + * @first_dma: The dma address corresponding to @alloced_page if
> > dma_mapping
> > + * is requested.
> > + * @alloced_pages: The number of allocated pages present in the
> > struct ttm_tt
> > + * page vector from this restore session.
> > + * @restored_pages: The number of 4K pages restored for
> > @alloced_page (which
> > + * is typically a multi-order page).
> > + * @page_caching: The struct ttm_tt requested caching
> > + * @order: The order of @alloced_page.
> > + *
> > + * Recovery from backup might fail when we've recovered less than
> > the
> > + * full ttm_tt. In order not to loose any data (yet), keep
> > information
> > + * around that allows us to restart a failed ttm backup recovery.
> > + */
> > +struct ttm_pool_tt_restore {
> > +	struct ttm_pool *pool;
> > +	struct ttm_pool_alloc_state snapshot_alloc;
> > +	struct page *alloced_page;
> > +	dma_addr_t first_dma;
> > +	pgoff_t alloced_pages;
> > +	pgoff_t restored_pages;
> > +	enum ttm_caching page_caching;
> > +	unsigned int order;
> > +};
> > +
> >   static unsigned long page_pool_size;
> >   
> >   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
> > WC/UC/DMA pool");
> > @@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct
> > ttm_pool_alloc_state *alloc)
> >   	return 0;
> >   }
> >   
> > -/* Map pages of 1 << order size and fill the DMA address array  */
> > +/* DMA Map pages of 1 << order size and return the resulting
> > dma_address. */
> >   static int ttm_pool_map(struct ttm_pool *pool, unsigned int
> > order,
> > -			struct page *p, dma_addr_t **dma_addr)
> > +			struct page *p, dma_addr_t *dma_addr)
> >   {
> >   	dma_addr_t addr;
> > -	unsigned int i;
> >   
> >   	if (pool->use_dma_alloc) {
> >   		struct ttm_pool_dma *dma = (void *)p->private;
> > @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool *pool,
> > unsigned int order,
> >   			return -EFAULT;
> >   	}
> >   
> > -	for (i = 1 << order; i ; --i) {
> > -		*(*dma_addr)++ = addr;
> > -		addr += PAGE_SIZE;
> > -	}
> > +	*dma_addr = addr;
> >   
> >   	return 0;
> >   }
> > @@ -371,6 +397,190 @@ static unsigned int
> > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> >   	return p->private;
> >   }
> >   
> > +/*
> > + * Split larger pages so that we can free each PAGE_SIZE page as
> > soon
> > + * as it has been backed up, in order to avoid memory pressure
> > during
> > + * reclaim.
> > + */
> > +static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct
> > page *p)
> > +{
> > +	unsigned int order = ttm_pool_page_order(pool, p);
> > +	pgoff_t nr;
> > +
> > +	if (!order)
> > +		return;
> > +
> > +	split_page(p, order);
> > +	nr = 1UL << order;
> > +	while (nr--)
> > +		(p++)->private = 0;
> > +}
> > +
> > +/**
> > + * DOC: Partial backup and restoration of a struct ttm_tt.
> > + *
> > + * Swapout using ttm_backup_backup_page() and swapin using
> > + * ttm_backup_copy_page() may fail.
> > + * The former most likely due to lack of swap-space or memory, the
> > latter due
> > + * to lack of memory or because of signal interruption during
> > waits.
> > + *
> > + * Backup failure is easily handled by using a ttm_tt pages vector
> > that holds
> > + * both backup handles and page pointers. This has to be taken
> > into account when
> > + * restoring such a ttm_tt from backup, and when freeing it while
> > backed up.
> > + * When restoring, for simplicity, new pages are actually
> > allocated from the
> > + * pool and the contents of any old pages are copied in and then
> > the old pages
> > + * are released.
> > + *
> > + * For restoration failures, the struct ttm_pool_tt_restore holds
> > sufficient state
> > + * to be able to resume an interrupted restore, and that structure
> > is freed once
> > + * the restoration is complete. If the struct ttm_tt is destroyed
> > while there
> > + * is a valid struct ttm_pool_tt_restore attached, that is also
> > properly taken
> > + * care of.
> > + */
> > +
> > +/* Is restore ongoing for the currently allocated page? */
> > +static bool ttm_pool_restore_valid(const struct
> > ttm_pool_tt_restore *restore)
> > +{
> > +	return restore && restore->restored_pages < (1 << restore-
> > >order);
> > +}
> > +
> > +/* DMA unmap and free a multi-order page, either to the relevant
> > pool or to system. */
> > +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool,
> > struct page *page,
> > +				       const dma_addr_t *dma_addr,
> > enum ttm_caching caching)
> > +{
> > +	struct ttm_pool_type *pt = NULL;
> > +	unsigned int order;
> > +	pgoff_t nr;
> > +
> > +	if (pool) {
> > +		order = ttm_pool_page_order(pool, page);
> > +		nr = (1UL << order);
> > +		if (dma_addr)
> > +			ttm_pool_unmap(pool, *dma_addr, nr);
> > +
> > +		pt = ttm_pool_select_type(pool, caching, order);
> > +	} else {
> > +		order = page->private;
> > +		nr = (1UL << order);
> > +	}
> > +
> > +	if (pt)
> > +		ttm_pool_type_give(pt, page);
> > +	else
> > +		ttm_pool_free_page(pool, caching, order, page);
> > +
> > +	return nr;
> > +}
> > +
> > +/* Populate the page-array using the most recent allocated multi-
> > order page. */
> > +static void ttm_pool_allocated_page_commit(struct page *allocated,
> > +					   dma_addr_t first_dma,
> > +					   struct
> > ttm_pool_alloc_state *alloc,
> > +					   pgoff_t nr)
> > +{
> > +	pgoff_t i;
> > +
> > +	for (i = 0; i < nr; ++i)
> > +		*alloc->pages++ = allocated++;
> > +
> > +	alloc->remaining_pages -= nr;
> > +
> > +	if (!alloc->dma_addr)
> > +		return;
> > +
> > +	for (i = 0; i < nr; ++i) {
> > +		*alloc->dma_addr++ = first_dma;
> > +		first_dma += PAGE_SIZE;
> > +	}
> > +}
> > +
> > +/*
> > + * When restoring, restore backed-up content to the newly
> > allocated page and
> > + * if successful, populate the page-table and dma-address arrays.
> > + */
> > +static int ttm_pool_restore_commit(struct ttm_pool_tt_restore
> > *restore,
> > +				   struct ttm_backup *backup,
> > +				   const struct ttm_operation_ctx
> > *ctx,
> > +				   struct ttm_pool_alloc_state
> > *alloc)
> > +
> > +{
> > +	pgoff_t i, nr = 1UL << restore->order;
> > +	struct page **first_page = alloc->pages;
> > +	struct page *p;
> > +	int ret = 0;
> > +
> > +	for (i = restore->restored_pages; i < nr; ++i) {
> > +		p = first_page[i];
> > +		if (ttm_backup_page_ptr_is_handle(p)) {
> > +			unsigned long handle =
> > ttm_backup_page_ptr_to_handle(p);
> > +
> > +			if (handle == 0) {
> > +				restore->restored_pages++;
> > +				continue;
> > +			}
> > +
> > +			ret = ttm_backup_copy_page(backup,
> > restore->alloced_page + i,
> > +						   handle, ctx-
> > >interruptible);
> > +			if (ret)
> > +				break;
> > +
> > +			ttm_backup_drop(backup, handle);
> > +		} else if (p) {
> > +			/*
> > +			 * We could probably avoid splitting the
> > old page
> > +			 * using clever logic, but ATM we don't
> > care, as
> > +			 * we prioritize releasing memory ASAP.
> > Note that
> > +			 * here, the old retained page is always
> > write-back
> > +			 * cached.
> > +			 */
> > +			ttm_pool_split_for_swap(restore->pool, p);
> > +			copy_highpage(restore->alloced_page + i,
> > p);
> > +			__free_pages(p, 0);
> > +		}
> > +
> > +		restore->restored_pages++;
> > +		first_page[i] = ttm_backup_handle_to_page_ptr(0);
> > +	}
> > +
> > +	if (ret) {
> > +		if (!restore->restored_pages) {
> > +			dma_addr_t *dma_addr = alloc->dma_addr ?
> > &restore->first_dma : NULL;
> > +
> > +			ttm_pool_unmap_and_free(restore->pool,
> > restore->alloced_page,
> > +						dma_addr, restore-
> > >page_caching);
> > +			restore->restored_pages = nr;
> > +		}
> > +		return ret;
> > +	}
> > +
> > +	ttm_pool_allocated_page_commit(restore->alloced_page,
> > restore->first_dma,
> > +				       alloc, nr);
> > +	if (restore->page_caching == alloc->tt_caching ||
> > PageHighMem(restore->alloced_page))
> > +		alloc->caching_divide = alloc->pages;
> > +	restore->snapshot_alloc = *alloc;
> > +	restore->alloced_pages += nr;
> > +
> > +	return 0;
> > +}
> > +
> > +/* If restoring, save information needed for
> > ttm_pool_restore_commit(). */
> > +static void
> > +ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned
> > int order,
> > +				struct page *p,
> > +				enum ttm_caching page_caching,
> > +				dma_addr_t first_dma,
> > +				struct ttm_pool_tt_restore
> > *restore,
> > +				const struct ttm_pool_alloc_state
> > *alloc)
> > +{
> > +	restore->pool = pool;
> > +	restore->order = order;
> > +	restore->restored_pages = 0;
> > +	restore->page_caching = page_caching;
> > +	restore->first_dma = first_dma;
> > +	restore->alloced_page = p;
> > +	restore->snapshot_alloc = *alloc;
> > +}
> > +
> >   /*
> >    * Called when we got a page, either from a pool or newly
> > allocated.
> >    * if needed, dma map the page and populate the dma address
> > array.
> > @@ -380,10 +590,11 @@ static unsigned int
> > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> >    */
> >   static int ttm_pool_page_allocated(struct ttm_pool *pool,
> > unsigned int order,
> >   				   struct page *p, enum
> > ttm_caching page_caching,
> > -				   struct ttm_pool_alloc_state
> > *alloc)
> > +				   struct ttm_pool_alloc_state
> > *alloc,
> > +				   struct ttm_pool_tt_restore
> > *restore)
> >   {
> > -	pgoff_t i, nr = 1UL << order;
> >   	bool caching_consistent;
> > +	dma_addr_t first_dma;
> >   	int r = 0;
> >   
> >   	caching_consistent = (page_caching == alloc->tt_caching)
> > || PageHighMem(p);
> > @@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct
> > ttm_pool *pool, unsigned int order,
> >   	}
> >   
> >   	if (alloc->dma_addr) {
> > -		r = ttm_pool_map(pool, order, p, &alloc-
> > >dma_addr);
> > +		r = ttm_pool_map(pool, order, p, &first_dma);
> >   		if (r)
> >   			return r;
> >   	}
> >   
> > -	alloc->remaining_pages -= nr;
> > -	for (i = 0; i < nr; ++i)
> > -		*alloc->pages++ = p++;
> > +	if (restore) {
> > +		ttm_pool_page_allocated_restore(pool, order, p,
> > page_caching,
> > +						first_dma,
> > restore, alloc);
> > +	} else {
> > +		ttm_pool_allocated_page_commit(p, first_dma,
> > alloc, 1UL << order);
> >   
> > -	if (caching_consistent)
> > -		alloc->caching_divide = alloc->pages;
> > +		if (caching_consistent)
> > +			alloc->caching_divide = alloc->pages;
> > +	}
> >   
> >   	return 0;
> >   }
> > @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct
> > ttm_pool *pool, struct ttm_tt *tt,
> >   				pgoff_t start_page, pgoff_t
> > end_page)
> >   {
> >   	struct page **pages = &tt->pages[start_page];
> > -	unsigned int order;
> > +	struct ttm_backup *backup = tt->backup;
> >   	pgoff_t i, nr;
> >   
> >   	for (i = start_page; i < end_page; i += nr, pages += nr) {
> > -		struct ttm_pool_type *pt = NULL;
> > +		struct page *p = *pages;
> >   
> > -		order = ttm_pool_page_order(pool, *pages);
> > -		nr = (1UL << order);
> > -		if (tt->dma_address)
> > -			ttm_pool_unmap(pool, tt->dma_address[i],
> > nr);
> > +		nr = 1;
> > +		if (ttm_backup_page_ptr_is_handle(p)) {
> > +			unsigned long handle =
> > ttm_backup_page_ptr_to_handle(p);
> >   
> > -		pt = ttm_pool_select_type(pool, caching, order);
> > -		if (pt)
> > -			ttm_pool_type_give(pt, *pages);
> > -		else
> > -			ttm_pool_free_page(pool, caching, order,
> > *pages);
> > +			if (handle != 0)
> > +				ttm_backup_drop(backup, handle);
> > +		} else if (p) {
> > +			dma_addr_t *dma_addr = tt->dma_address ?
> > +				tt->dma_address + i : NULL;
> > +
> > +			nr = ttm_pool_unmap_and_free(pool, p,
> > dma_addr, caching);
> > +		}
> >   	}
> >   }
> >   
> > @@ -467,22 +683,11 @@ static unsigned int
> > ttm_pool_alloc_find_order(unsigned int highest,
> >   	return min_t(unsigned int, highest, __fls(alloc-
> > >remaining_pages));
> >   }
> >   
> > -/**
> > - * ttm_pool_alloc - Fill a ttm_tt object
> > - *
> > - * @pool: ttm_pool to use
> > - * @tt: ttm_tt object to fill
> > - * @ctx: operation context
> > - *
> > - * Fill the ttm_tt object with pages and also make sure to DMA map
> > them when
> > - * necessary.
> > - *
> > - * Returns: 0 on successe, negative error code otherwise.
> > - */
> > -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > -		   struct ttm_operation_ctx *ctx)
> > +static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
> > *tt,
> > +			    const struct ttm_operation_ctx *ctx,
> > +			    struct ttm_pool_alloc_state *alloc,
> > +			    struct ttm_pool_tt_restore *restore)
> >   {
> > -	struct ttm_pool_alloc_state alloc;
> >   	enum ttm_caching page_caching;
> >   	gfp_t gfp_flags = GFP_USER;
> >   	pgoff_t caching_divide;
> > @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > struct ttm_tt *tt,
> >   	struct page *p;
> >   	int r;
> >   
> > -	ttm_pool_alloc_state_init(tt, &alloc);
> > -
> > -	WARN_ON(!alloc.remaining_pages ||
> > ttm_tt_is_populated(tt));
> > -	WARN_ON(alloc.dma_addr && !pool->dev);
> > +	WARN_ON(!alloc->remaining_pages ||
> > ttm_tt_is_populated(tt));
> > +	WARN_ON(alloc->dma_addr && !pool->dev);
> >   
> >   	if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
> >   		gfp_flags |= __GFP_ZERO;
> > @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > struct ttm_tt *tt,
> >   
> >   	page_caching = tt->caching;
> >   	allow_pools = true;
> > -	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > &alloc);
> > -	     alloc.remaining_pages;
> > -	     order = ttm_pool_alloc_find_order(order, &alloc)) {
> > +	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > alloc);
> > +	     alloc->remaining_pages;
> > +	     order = ttm_pool_alloc_find_order(order, alloc)) {
> >   		struct ttm_pool_type *pt;
> >   
> >   		/* First, try to allocate a page from a pool if
> > one exists. */
> > @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > struct ttm_tt *tt,
> >   			r = -ENOMEM;
> >   			goto error_free_all;
> >   		}
> > -		r = ttm_pool_page_allocated(pool, order, p,
> > page_caching, &alloc);
> > +		r = ttm_pool_page_allocated(pool, order, p,
> > page_caching, alloc,
> > +					    restore);
> >   		if (r)
> >   			goto error_free_page;
> > +
> > +		if (ttm_pool_restore_valid(restore)) {
> > +			r = ttm_pool_restore_commit(restore, tt-
> > >backup, ctx, alloc);
> > +			if (r)
> > +				goto error_free_all;
> > +		}
> >   	}
> >   
> > -	r = ttm_pool_apply_caching(&alloc);
> > +	r = ttm_pool_apply_caching(alloc);
> >   	if (r)
> >   		goto error_free_all;
> >   
> > +	kfree(tt->restore);
> > +	tt->restore = NULL;
> > +
> >   	return 0;
> >   
> >   error_free_page:
> >   	ttm_pool_free_page(pool, page_caching, order, p);
> >   
> >   error_free_all:
> > -	caching_divide = alloc.caching_divide - tt->pages;
> > +	if (tt->restore)
> > +		return r;
> > +
> > +	caching_divide = alloc->caching_divide - tt->pages;
> >   	ttm_pool_free_range(pool, tt, tt->caching, 0,
> > caching_divide);
> >   	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
> > -			    tt->num_pages -
> > alloc.remaining_pages);
> > +			    tt->num_pages - alloc-
> > >remaining_pages);
> >   
> >   	return r;
> >   }
> > +
> > +/**
> > + * ttm_pool_alloc - Fill a ttm_tt object
> > + *
> > + * @pool: ttm_pool to use
> > + * @tt: ttm_tt object to fill
> > + * @ctx: operation context
> > + *
> > + * Fill the ttm_tt object with pages and also make sure to DMA map
> > them when
> > + * necessary.
> > + *
> > + * Returns: 0 on successe, negative error code otherwise.
> > + */
> > +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > +		   struct ttm_operation_ctx *ctx)
> > +{
> > +	struct ttm_pool_alloc_state alloc;
> > +
> > +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > +		return -EINVAL;
> > +
> > +	ttm_pool_alloc_state_init(tt, &alloc);
> > +
> > +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> > +}
> >   EXPORT_SYMBOL(ttm_pool_alloc);
> >   
> > +/**
> > + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
> > previously backed-up
> > + * content.
> > + *
> > + * @pool: ttm_pool to use
> > + * @tt: ttm_tt object to fill
> > + * @ctx: operation context
> > + *
> > + * Fill the ttm_tt object with pages and also make sure to DMA map
> > them when
> > + * necessary. Read in backed-up content.
> > + *
> > + * Returns: 0 on successe, negative error code otherwise.
> > + */
> > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > ttm_tt *tt,
> > +			       const struct ttm_operation_ctx
> > *ctx)
> > +{
> > +	struct ttm_pool_alloc_state alloc;
> > +
> > +	if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> > +		return -EINVAL;
> > +
> > +	if (!tt->restore) {
> > +		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> > +
> > +		ttm_pool_alloc_state_init(tt, &alloc);
> > +		if (ctx->gfp_retry_mayfail)
> > +			gfp |= __GFP_RETRY_MAYFAIL;
> > +
> > +		tt->restore = kzalloc(sizeof(*tt->restore), gfp);
> > +		if (!tt->restore)
> > +			return -ENOMEM;
> > +
> > +		tt->restore->snapshot_alloc = alloc;
> > +		tt->restore->pool = pool;
> > +		tt->restore->restored_pages = 1;
> > +	} else {
> > +		struct ttm_pool_tt_restore *restore = tt->restore;
> > +		int ret;
> > +
> > +		alloc = restore->snapshot_alloc;
> > +		if (ttm_pool_restore_valid(tt->restore)) {
> > +			ret = ttm_pool_restore_commit(restore, tt-
> > >backup, ctx, &alloc);
> > +			if (ret)
> > +				return ret;
> > +		}
> > +		if (!alloc.remaining_pages)
> > +			return 0;
> > +	}
> > +
> > +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
> > >restore);
> > +}
> > +
> >   /**
> >    * ttm_pool_free - Free the backing pages from a ttm_tt object
> >    *
> > @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool,
> > struct ttm_tt *tt)
> >   }
> >   EXPORT_SYMBOL(ttm_pool_free);
> >   
> > +/**
> > + * ttm_pool_drop_backed_up() - Release content of a swapped-out
> > struct ttm_tt
> > + * @tt: The struct ttm_tt.
> > + *
> > + * Release handles with associated content or any remaining pages
> > of
> > + * a backed-up struct ttm_tt.
> > + */
> > +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> > +{
> > +	struct ttm_pool_tt_restore *restore;
> > +	pgoff_t start_page = 0;
> > +
> > +	WARN_ON(!ttm_tt_is_backed_up(tt));
> > +
> > +	restore = tt->restore;
> > +
> > +	/*
> > +	 * Unmap and free any uncommitted restore page.
> > +	 * any tt page-array backup entries already read back has
> > +	 * been cleared already
> > +	 */
> > +	if (ttm_pool_restore_valid(restore)) {
> > +		dma_addr_t *dma_addr = tt->dma_address ? &restore-
> > >first_dma : NULL;
> > +
> > +		ttm_pool_unmap_and_free(restore->pool, restore-
> > >alloced_page,
> > +					dma_addr, restore-
> > >page_caching);
> > +		restore->restored_pages = 1UL << restore->order;
> > +	}
> > +
> > +	/*
> > +	 * If a restore is ongoing, part of the tt pages may have
> > a
> > +	 * caching different than writeback.
> > +	 */
> > +	if (restore) {
> > +		pgoff_t mid = restore-
> > >snapshot_alloc.caching_divide - tt->pages;
> > +
> > +		start_page = restore->alloced_pages;
> > +		WARN_ON(mid > start_page);
> > +		/* Pages that might be dma-mapped and non-cached
> > */
> > +		ttm_pool_free_range(restore->pool, tt, tt-
> > >caching,
> > +				    0, mid);
> > +		/* Pages that might be dma-mapped but cached */
> > +		ttm_pool_free_range(restore->pool, tt, ttm_cached,
> > +				    mid, restore->alloced_pages);
> > +		kfree(restore);
> > +		tt->restore = NULL;
> > +	}
> > +
> > +	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt-
> > >num_pages);
> > +}
> > +
> > +/**
> > + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> > + * @pool: The pool used when allocating the struct ttm_tt.
> > + * @tt: The struct ttm_tt.
> > + * @flags: Flags to govern the backup behaviour.
> > + *
> > + * Back up or purge a struct ttm_tt. If @purge is true, then
> > + * all pages will be freed directly to the system rather than to
> > the pool
> > + * they were allocated from, making the function behave similarly
> > to
> > + * ttm_pool_free(). If @purge is false the pages will be backed up
> > instead,
> > + * exchanged for handles.
> > + * A subsequent call to ttm_pool_restore_and_alloc() will then
> > read back the content and
> > + * a subsequent call to ttm_pool_drop_backed_up() will drop it.
> > + * If backup of a page fails for whatever reason, @ttm will still
> > be
> > + * partially backed up, retaining those pages for which backup
> > fails.
> > + * In that case, this function can be retried, possibly after
> > freeing up
> > + * memory resources.
> > + *
> > + * Return: Number of pages actually backed up or freed, or
> > negative
> > + * error code on error.
> > + */
> > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
> > +		     const struct ttm_backup_flags *flags)
> > +{
> > +	struct ttm_backup *backup = tt->backup;
> > +	struct page *page;
> > +	unsigned long handle;
> > +	gfp_t alloc_gfp;
> > +	gfp_t gfp;
> > +	int ret = 0;
> > +	pgoff_t shrunken = 0;
> > +	pgoff_t i, num_pages;
> > +
> > +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > +		return -EINVAL;
> > +
> > +	if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> > +	    pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> > +		return -EBUSY;
> > +
> > +#ifdef CONFIG_X86
> > +	/* Anything returned to the system needs to be cached. */
> > +	if (tt->caching != ttm_cached)
> > +		set_pages_array_wb(tt->pages, tt->num_pages);
> > +#endif
> > +
> > +	if (tt->dma_address || flags->purge) {
> > +		for (i = 0; i < tt->num_pages; i += num_pages) {
> > +			unsigned int order;
> > +
> > +			page = tt->pages[i];
> > +			if (unlikely(!page)) {
> > +				num_pages = 1;
> > +				continue;
> > +			}
> > +
> > +			order = ttm_pool_page_order(pool, page);
> > +			num_pages = 1UL << order;
> > +			if (tt->dma_address)
> > +				ttm_pool_unmap(pool, tt-
> > >dma_address[i],
> > +					       num_pages);
> > +			if (flags->purge) {
> > +				shrunken += num_pages;
> > +				page->private = 0;
> > +				__free_pages(page, order);
> > +				memset(tt->pages + i, 0,
> > +				       num_pages * sizeof(*tt-
> > >pages));
> > +			}
> > +		}
> > +	}
> > +
> > +	if (flags->purge)
> > +		return shrunken;
> > +
> > +	if (pool->use_dma32)
> > +		gfp = GFP_DMA32;
> > +	else
> > +		gfp = GFP_HIGHUSER;
> > +
> > +	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
> > __GFP_RETRY_MAYFAIL;
> > +
> > +	for (i = 0; i < tt->num_pages; ++i) {
> > +		s64 shandle;
> > +
> > +		page = tt->pages[i];
> > +		if (unlikely(!page))
> > +			continue;
> > +
> > +		ttm_pool_split_for_swap(pool, page);
> > +
> > +		shandle = ttm_backup_backup_page(backup, page,
> > flags->writeback, i,
> > +						 gfp, alloc_gfp);
> > +		if (shandle < 0) {
> > +			/* We allow partially shrunken tts */
> > +			ret = shandle;
> > +			break;
> > +		}
> > +		handle = shandle;
> > +		tt->pages[i] =
> > ttm_backup_handle_to_page_ptr(handle);
> > +		put_page(page);
> > +		shrunken++;
> > +	}
> > +
> > +	return shrunken ? shrunken : ret;
> > +}
> > +
> >   /**
> >    * ttm_pool_init - Initialize a pool
> >    *
> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > b/drivers/gpu/drm/ttm/ttm_tt.c
> > index 3baf215eca23..00b7c28f2329 100644
> > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > @@ -40,6 +40,7 @@
> >   #include <drm/drm_cache.h>
> >   #include <drm/drm_device.h>
> >   #include <drm/drm_util.h>
> > +#include <drm/ttm/ttm_backup.h>
> >   #include <drm/ttm/ttm_bo.h>
> >   #include <drm/ttm/ttm_tt.h>
> >   
> > @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt
> > *ttm,
> >   	ttm->swap_storage = NULL;
> >   	ttm->sg = bo->sg;
> >   	ttm->caching = caching;
> > +	ttm->restore = NULL;
> > +	ttm->backup = NULL;
> >   }
> >   
> >   int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
> > @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
> >   		fput(ttm->swap_storage);
> >   	ttm->swap_storage = NULL;
> >   
> > +	if (ttm_tt_is_backed_up(ttm))
> > +		ttm_pool_drop_backed_up(ttm);
> > +	if (ttm->backup) {
> > +		ttm_backup_fini(ttm->backup);
> > +		ttm->backup = NULL;
> > +	}
> > +
> >   	if (ttm->pages)
> >   		kvfree(ttm->pages);
> >   	else
> > @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
> >   }
> >   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
> >   
> > +/**
> > + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> > + * @bdev: The TTM device.
> > + * @tt: The struct ttm_tt.
> > + * @flags: Flags that govern the backup behaviour.
> > + *
> > + * Update the page accounting and call ttm_pool_shrink_tt to free
> > pages
> > + * or back them up.
> > + *
> > + * Return: Number of pages freed or swapped out, or negative error
> > code on
> > + * error.
> > + */
> > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > +		   const struct ttm_backup_flags flags)
> > +{
> > +	long ret;
> > +
> > +	if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> > +		return 0;
> > +
> > +	ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> > +	if (ret > 0) {
> > +		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> > +		tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > +		   const struct ttm_operation_ctx *ctx)
> > +{
> > +	int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
> > ctx);
> > +
> > +	if (ret)
> > +		return ret;
> > +
> > +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > +
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL(ttm_tt_restore);
> > +
> >   /**
> >    * ttm_tt_swapout - swap out tt object
> >    *
> > @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
> >   		goto error;
> >   
> >   	ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> > +	ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> >   	if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
> >   		ret = ttm_tt_swapin(ttm);
> >   		if (unlikely(ret != 0)) {
> > diff --git a/include/drm/ttm/ttm_pool.h
> > b/include/drm/ttm/ttm_pool.h
> > index 160d954a261e..54cd34a6e4c0 100644
> > --- a/include/drm/ttm/ttm_pool.h
> > +++ b/include/drm/ttm/ttm_pool.h
> > @@ -33,6 +33,7 @@
> >   
> >   struct device;
> >   struct seq_file;
> > +struct ttm_backup_flags;
> >   struct ttm_operation_ctx;
> >   struct ttm_pool;
> >   struct ttm_tt;
> > @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
> >   
> >   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);
> >   
> > +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> > +
> > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
> > +		     const struct ttm_backup_flags *flags);
> > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > ttm_tt *tt,
> > +			       const struct ttm_operation_ctx
> > *ctx);
> > +
> >   int ttm_pool_mgr_init(unsigned long num_pages);
> >   void ttm_pool_mgr_fini(void);
> >   
> > diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> > index 991edafdb2dd..c736c01ac2ca 100644
> > --- a/include/drm/ttm/ttm_tt.h
> > +++ b/include/drm/ttm/ttm_tt.h
> > @@ -32,11 +32,13 @@
> >   #include <drm/ttm/ttm_caching.h>
> >   #include <drm/ttm/ttm_kmap_iter.h>
> >   
> > +struct ttm_backup;
> >   struct ttm_device;
> >   struct ttm_tt;
> >   struct ttm_resource;
> >   struct ttm_buffer_object;
> >   struct ttm_operation_ctx;
> > +struct ttm_pool_tt_restore;
> >   
> >   /**
> >    * struct ttm_tt - This is a structure holding the pages,
> > caching- and aperture
> > @@ -85,17 +87,22 @@ struct ttm_tt {
> >   	 * fault handling abuses the DMA api a bit and
> > dma_map_attrs can't be
> >   	 * used to assure pgprot always matches.
> >   	 *
> > +	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
> > if the
> > +	 * struct ttm_tt has been (possibly partially) backed up.
> > +	 *
> >   	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT
> > USE. This is
> >   	 * set by TTM after ttm_tt_populate() has successfully
> > returned, and is
> >   	 * then unset when TTM calls ttm_tt_unpopulate().
> > +	 *
> >   	 */
> >   #define TTM_TT_FLAG_SWAPPED		BIT(0)
> >   #define TTM_TT_FLAG_ZERO_ALLOC		BIT(1)
> >   #define TTM_TT_FLAG_EXTERNAL		BIT(2)
> >   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
> >   #define TTM_TT_FLAG_DECRYPTED		BIT(4)
> > +#define TTM_TT_FLAG_BACKED_UP	        BIT(5)
> >   
> > -#define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
> > +#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
> >   	uint32_t page_flags;
> >   	/** @num_pages: Number of pages in the page array. */
> >   	uint32_t num_pages;
> > @@ -105,11 +112,20 @@ struct ttm_tt {
> >   	dma_addr_t *dma_address;
> >   	/** @swap_storage: Pointer to shmem struct file for swap
> > storage. */
> >   	struct file *swap_storage;
> > +	/**
> > +	 * @backup: Pointer to backup struct for backed up tts.
> > +	 * Could be unified with @swap_storage. Meanwhile, the
> > driver's
> > +	 * ttm_tt_create() callback is responsible for assigning
> > +	 * this field.
> > +	 */
> > +	struct ttm_backup *backup;
> >   	/**
> >   	 * @caching: The current caching state of the pages, see
> > enum
> >   	 * ttm_caching.
> >   	 */
> >   	enum ttm_caching caching;
> > +	/** @restore: Partial restoration from backup state. TTM
> > private */
> > +	struct ttm_pool_tt_restore *restore;
> >   };
> >   
> >   /**
> > @@ -129,9 +145,38 @@ static inline bool ttm_tt_is_populated(struct
> > ttm_tt *tt)
> >   	return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
> >   }
> >   
> > +/**
> > + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or
> > backed up
> > + * @tt: The struct ttm_tt.
> > + *
> > + * Return: true if swapped or backed up, false otherwise.
> > + */
> >   static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
> >   {
> > -	return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> > +	return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
> > TTM_TT_FLAG_BACKED_UP);
> > +}
> > +
> > +/**
> > + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> > + * @tt: The struct ttm_tt.
> > + *
> > + * Return: true if swapped or backed up, false otherwise.
> > + */
> > +static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
> > +{
> > +	return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> > +}
> > +
> > +/**
> > + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
> > + * @tt: The struct ttm_tt.
> > + *
> > + * Drivers can use this functionto clear the backed-up status,
> > + * for example before destroying or re-validating a purged tt.
> > + */
> > +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
> > +{
> > +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> >   }
> >   
> >   /**
> > @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long num_pages,
> > unsigned long num_dma32_pages);
> >   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
> > ttm_kmap_iter_tt *iter_tt,
> >   					    struct ttm_tt *tt);
> >   unsigned long ttm_tt_pages_limit(void);
> > +
> > +/**
> > + * struct ttm_backup_flags - Flags to govern backup behaviour.
> > + * @purge: Free pages without backing up. Bypass pools.
> > + * @writeback: Attempt to copy contents directly to swap space,
> > even
> > + * if that means blocking on writes to external memory.
> > + */
> > +struct ttm_backup_flags {
> > +	u32 purge : 1;
> > +	u32 writeback : 1;
> > +};
> > +
> > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > +		   const struct ttm_backup_flags flags);
> > +
> > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > +		   const struct ttm_operation_ctx *ctx);
> > +
> >   #if IS_ENABLED(CONFIG_AGP)
> >   #include <linux/agp_backend.h>
> >   
> 

RESEND Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Thomas Hellström]

Hi, Christian,

Ping? I'd really want to get this in before -rc6

Thanks,
Thomas



On Tue, 2025-02-18 at 16:40 +0100, Thomas Hellström wrote:
> Hi, Christian,
> 
> On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
> > Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> > > Provide a helper to shrink ttm_tt page-vectors on a per-page
> > > basis. A ttm_backup backend could then in theory get away with
> > > allocating a single temporary page for each struct ttm_tt.
> > > 
> > > This is accomplished by splitting larger pages before trying to
> > > back them up.
> > > 
> > > In the future we could allow ttm_backup to handle backing up
> > > large pages as well, but currently there's no benefit in
> > > doing that, since the shmem backup backend would have to
> > > split those anyway to avoid allocating too much temporary
> > > memory, and if the backend instead inserts pages into the
> > > swap-cache, those are split on reclaim by the core.
> > > 
> > > Due to potential backup- and recover errors, allow partially
> > > swapped
> > > out struct ttm_tt's, although mark them as swapped out stopping
> > > them
> > > from being swapped out a second time. More details in the
> > > ttm_pool.c
> > > DOC section.
> > > 
> > > v2:
> > > - A couple of cleanups and error fixes in ttm_pool_back_up_tt.
> > > - s/back_up/backup/
> > > - Add a writeback parameter to the exported interface.
> > > v8:
> > > - Use a struct for flags for readability (Matt Brost)
> > > - Address misc other review comments (Matt Brost)
> > > v9:
> > > - Update the kerneldoc for the ttm_tt::backup field.
> > > v10:
> > > - Rebase.
> > > v13:
> > > - Rebase on ttm_backup interface change. Update kerneldoc.
> > > - Rebase and adjust ttm_tt_is_swapped().
> > > v15:
> > > - Rebase on ttm_backup return value change.
> > > - Rebase on previous restructuring of ttm_pool_alloc()
> > > - Rework the ttm_pool backup interface (Christian König)
> > > - Remove cond_resched() (Christian König)
> > > - Get rid of the need to allocate an intermediate page array
> > >    when restoring a multi-order page (Christian König)
> > > - Update documentation.
> > > 
> > > Cc: Christian König <christian.koenig@amd.com>
> > > Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > Cc: <dri-devel@lists.freedesktop.org>
> > > Signed-off-by: Thomas Hellström
> > > <thomas.hellstrom@linux.intel.com>
> > > Reviewed-by: Matthew Brost <matthew.brost@intel.com>
> > 
> > I've tried to wrap my head around all of this like twenty times in
> > the 
> > last three month, but was always interrupted at some point.
> > 
> > Feel free to add Acked-by: Christian Koenig
> > <christian.koenig@amd.com>.
> > 
> > Sorry,
> > Christian.
> 
> Thanks a lot for all reviewing and comments so far. There are two TTM
> patches left in the series that don't have an ack by you:
> 
> https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
> and
> 
> https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17
> 
> None of them particularly big considering the amount of doc text.
> 
> It'd be great if those could have an ack as well so we could finally
> merge this series.
> 
> Thanks,
> Thomas
> 
> 
> 
> 
> 
> 
> > 
> > > ---
> > >   drivers/gpu/drm/ttm/ttm_pool.c | 554
> > > +++++++++++++++++++++++++++++----
> > >   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
> > >   include/drm/ttm/ttm_pool.h     |   8 +
> > >   include/drm/ttm/ttm_tt.h       |  67 +++-
> > >   4 files changed, 629 insertions(+), 54 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
> > > b/drivers/gpu/drm/ttm/ttm_pool.c
> > > index c9eba76d5143..ffb7abf52bab 100644
> > > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > > @@ -41,6 +41,7 @@
> > >   #include <asm/set_memory.h>
> > >   #endif
> > >   
> > > +#include <drm/ttm/ttm_backup.h>
> > >   #include <drm/ttm/ttm_pool.h>
> > >   #include <drm/ttm/ttm_tt.h>
> > >   #include <drm/ttm/ttm_bo.h>
> > > @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
> > >   	enum ttm_caching tt_caching;
> > >   };
> > >   
> > > +/**
> > > + * struct ttm_pool_tt_restore - State representing restore from
> > > backup
> > > + * @pool: The pool used for page allocation while restoring.
> > > + * @snapshot_alloc: A snapshot of the most recent struct
> > > ttm_pool_alloc_state.
> > > + * @alloced_page: Pointer to the page most recently allocated
> > > from
> > > a pool or system.
> > > + * @first_dma: The dma address corresponding to @alloced_page if
> > > dma_mapping
> > > + * is requested.
> > > + * @alloced_pages: The number of allocated pages present in the
> > > struct ttm_tt
> > > + * page vector from this restore session.
> > > + * @restored_pages: The number of 4K pages restored for
> > > @alloced_page (which
> > > + * is typically a multi-order page).
> > > + * @page_caching: The struct ttm_tt requested caching
> > > + * @order: The order of @alloced_page.
> > > + *
> > > + * Recovery from backup might fail when we've recovered less
> > > than
> > > the
> > > + * full ttm_tt. In order not to loose any data (yet), keep
> > > information
> > > + * around that allows us to restart a failed ttm backup
> > > recovery.
> > > + */
> > > +struct ttm_pool_tt_restore {
> > > +	struct ttm_pool *pool;
> > > +	struct ttm_pool_alloc_state snapshot_alloc;
> > > +	struct page *alloced_page;
> > > +	dma_addr_t first_dma;
> > > +	pgoff_t alloced_pages;
> > > +	pgoff_t restored_pages;
> > > +	enum ttm_caching page_caching;
> > > +	unsigned int order;
> > > +};
> > > +
> > >   static unsigned long page_pool_size;
> > >   
> > >   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
> > > WC/UC/DMA pool");
> > > @@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct
> > > ttm_pool_alloc_state *alloc)
> > >   	return 0;
> > >   }
> > >   
> > > -/* Map pages of 1 << order size and fill the DMA address array 
> > > */
> > > +/* DMA Map pages of 1 << order size and return the resulting
> > > dma_address. */
> > >   static int ttm_pool_map(struct ttm_pool *pool, unsigned int
> > > order,
> > > -			struct page *p, dma_addr_t **dma_addr)
> > > +			struct page *p, dma_addr_t *dma_addr)
> > >   {
> > >   	dma_addr_t addr;
> > > -	unsigned int i;
> > >   
> > >   	if (pool->use_dma_alloc) {
> > >   		struct ttm_pool_dma *dma = (void *)p->private;
> > > @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool
> > > *pool,
> > > unsigned int order,
> > >   			return -EFAULT;
> > >   	}
> > >   
> > > -	for (i = 1 << order; i ; --i) {
> > > -		*(*dma_addr)++ = addr;
> > > -		addr += PAGE_SIZE;
> > > -	}
> > > +	*dma_addr = addr;
> > >   
> > >   	return 0;
> > >   }
> > > @@ -371,6 +397,190 @@ static unsigned int
> > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > >   	return p->private;
> > >   }
> > >   
> > > +/*
> > > + * Split larger pages so that we can free each PAGE_SIZE page as
> > > soon
> > > + * as it has been backed up, in order to avoid memory pressure
> > > during
> > > + * reclaim.
> > > + */
> > > +static void ttm_pool_split_for_swap(struct ttm_pool *pool,
> > > struct
> > > page *p)
> > > +{
> > > +	unsigned int order = ttm_pool_page_order(pool, p);
> > > +	pgoff_t nr;
> > > +
> > > +	if (!order)
> > > +		return;
> > > +
> > > +	split_page(p, order);
> > > +	nr = 1UL << order;
> > > +	while (nr--)
> > > +		(p++)->private = 0;
> > > +}
> > > +
> > > +/**
> > > + * DOC: Partial backup and restoration of a struct ttm_tt.
> > > + *
> > > + * Swapout using ttm_backup_backup_page() and swapin using
> > > + * ttm_backup_copy_page() may fail.
> > > + * The former most likely due to lack of swap-space or memory,
> > > the
> > > latter due
> > > + * to lack of memory or because of signal interruption during
> > > waits.
> > > + *
> > > + * Backup failure is easily handled by using a ttm_tt pages
> > > vector
> > > that holds
> > > + * both backup handles and page pointers. This has to be taken
> > > into account when
> > > + * restoring such a ttm_tt from backup, and when freeing it
> > > while
> > > backed up.
> > > + * When restoring, for simplicity, new pages are actually
> > > allocated from the
> > > + * pool and the contents of any old pages are copied in and then
> > > the old pages
> > > + * are released.
> > > + *
> > > + * For restoration failures, the struct ttm_pool_tt_restore
> > > holds
> > > sufficient state
> > > + * to be able to resume an interrupted restore, and that
> > > structure
> > > is freed once
> > > + * the restoration is complete. If the struct ttm_tt is
> > > destroyed
> > > while there
> > > + * is a valid struct ttm_pool_tt_restore attached, that is also
> > > properly taken
> > > + * care of.
> > > + */
> > > +
> > > +/* Is restore ongoing for the currently allocated page? */
> > > +static bool ttm_pool_restore_valid(const struct
> > > ttm_pool_tt_restore *restore)
> > > +{
> > > +	return restore && restore->restored_pages < (1 <<
> > > restore-
> > > > order);
> > > +}
> > > +
> > > +/* DMA unmap and free a multi-order page, either to the relevant
> > > pool or to system. */
> > > +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool,
> > > struct page *page,
> > > +				       const dma_addr_t
> > > *dma_addr,
> > > enum ttm_caching caching)
> > > +{
> > > +	struct ttm_pool_type *pt = NULL;
> > > +	unsigned int order;
> > > +	pgoff_t nr;
> > > +
> > > +	if (pool) {
> > > +		order = ttm_pool_page_order(pool, page);
> > > +		nr = (1UL << order);
> > > +		if (dma_addr)
> > > +			ttm_pool_unmap(pool, *dma_addr, nr);
> > > +
> > > +		pt = ttm_pool_select_type(pool, caching, order);
> > > +	} else {
> > > +		order = page->private;
> > > +		nr = (1UL << order);
> > > +	}
> > > +
> > > +	if (pt)
> > > +		ttm_pool_type_give(pt, page);
> > > +	else
> > > +		ttm_pool_free_page(pool, caching, order, page);
> > > +
> > > +	return nr;
> > > +}
> > > +
> > > +/* Populate the page-array using the most recent allocated
> > > multi-
> > > order page. */
> > > +static void ttm_pool_allocated_page_commit(struct page
> > > *allocated,
> > > +					   dma_addr_t first_dma,
> > > +					   struct
> > > ttm_pool_alloc_state *alloc,
> > > +					   pgoff_t nr)
> > > +{
> > > +	pgoff_t i;
> > > +
> > > +	for (i = 0; i < nr; ++i)
> > > +		*alloc->pages++ = allocated++;
> > > +
> > > +	alloc->remaining_pages -= nr;
> > > +
> > > +	if (!alloc->dma_addr)
> > > +		return;
> > > +
> > > +	for (i = 0; i < nr; ++i) {
> > > +		*alloc->dma_addr++ = first_dma;
> > > +		first_dma += PAGE_SIZE;
> > > +	}
> > > +}
> > > +
> > > +/*
> > > + * When restoring, restore backed-up content to the newly
> > > allocated page and
> > > + * if successful, populate the page-table and dma-address
> > > arrays.
> > > + */
> > > +static int ttm_pool_restore_commit(struct ttm_pool_tt_restore
> > > *restore,
> > > +				   struct ttm_backup *backup,
> > > +				   const struct
> > > ttm_operation_ctx
> > > *ctx,
> > > +				   struct ttm_pool_alloc_state
> > > *alloc)
> > > +
> > > +{
> > > +	pgoff_t i, nr = 1UL << restore->order;
> > > +	struct page **first_page = alloc->pages;
> > > +	struct page *p;
> > > +	int ret = 0;
> > > +
> > > +	for (i = restore->restored_pages; i < nr; ++i) {
> > > +		p = first_page[i];
> > > +		if (ttm_backup_page_ptr_is_handle(p)) {
> > > +			unsigned long handle =
> > > ttm_backup_page_ptr_to_handle(p);
> > > +
> > > +			if (handle == 0) {
> > > +				restore->restored_pages++;
> > > +				continue;
> > > +			}
> > > +
> > > +			ret = ttm_backup_copy_page(backup,
> > > restore->alloced_page + i,
> > > +						   handle, ctx-
> > > > interruptible);
> > > +			if (ret)
> > > +				break;
> > > +
> > > +			ttm_backup_drop(backup, handle);
> > > +		} else if (p) {
> > > +			/*
> > > +			 * We could probably avoid splitting the
> > > old page
> > > +			 * using clever logic, but ATM we don't
> > > care, as
> > > +			 * we prioritize releasing memory ASAP.
> > > Note that
> > > +			 * here, the old retained page is always
> > > write-back
> > > +			 * cached.
> > > +			 */
> > > +			ttm_pool_split_for_swap(restore->pool,
> > > p);
> > > +			copy_highpage(restore->alloced_page + i,
> > > p);
> > > +			__free_pages(p, 0);
> > > +		}
> > > +
> > > +		restore->restored_pages++;
> > > +		first_page[i] =
> > > ttm_backup_handle_to_page_ptr(0);
> > > +	}
> > > +
> > > +	if (ret) {
> > > +		if (!restore->restored_pages) {
> > > +			dma_addr_t *dma_addr = alloc->dma_addr ?
> > > &restore->first_dma : NULL;
> > > +
> > > +			ttm_pool_unmap_and_free(restore->pool,
> > > restore->alloced_page,
> > > +						dma_addr,
> > > restore-
> > > > page_caching);
> > > +			restore->restored_pages = nr;
> > > +		}
> > > +		return ret;
> > > +	}
> > > +
> > > +	ttm_pool_allocated_page_commit(restore->alloced_page,
> > > restore->first_dma,
> > > +				       alloc, nr);
> > > +	if (restore->page_caching == alloc->tt_caching ||
> > > PageHighMem(restore->alloced_page))
> > > +		alloc->caching_divide = alloc->pages;
> > > +	restore->snapshot_alloc = *alloc;
> > > +	restore->alloced_pages += nr;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/* If restoring, save information needed for
> > > ttm_pool_restore_commit(). */
> > > +static void
> > > +ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned
> > > int order,
> > > +				struct page *p,
> > > +				enum ttm_caching page_caching,
> > > +				dma_addr_t first_dma,
> > > +				struct ttm_pool_tt_restore
> > > *restore,
> > > +				const struct
> > > ttm_pool_alloc_state
> > > *alloc)
> > > +{
> > > +	restore->pool = pool;
> > > +	restore->order = order;
> > > +	restore->restored_pages = 0;
> > > +	restore->page_caching = page_caching;
> > > +	restore->first_dma = first_dma;
> > > +	restore->alloced_page = p;
> > > +	restore->snapshot_alloc = *alloc;
> > > +}
> > > +
> > >   /*
> > >    * Called when we got a page, either from a pool or newly
> > > allocated.
> > >    * if needed, dma map the page and populate the dma address
> > > array.
> > > @@ -380,10 +590,11 @@ static unsigned int
> > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > >    */
> > >   static int ttm_pool_page_allocated(struct ttm_pool *pool,
> > > unsigned int order,
> > >   				   struct page *p, enum
> > > ttm_caching page_caching,
> > > -				   struct ttm_pool_alloc_state
> > > *alloc)
> > > +				   struct ttm_pool_alloc_state
> > > *alloc,
> > > +				   struct ttm_pool_tt_restore
> > > *restore)
> > >   {
> > > -	pgoff_t i, nr = 1UL << order;
> > >   	bool caching_consistent;
> > > +	dma_addr_t first_dma;
> > >   	int r = 0;
> > >   
> > >   	caching_consistent = (page_caching == alloc->tt_caching)
> > > > > PageHighMem(p);
> > > @@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct
> > > ttm_pool *pool, unsigned int order,
> > >   	}
> > >   
> > >   	if (alloc->dma_addr) {
> > > -		r = ttm_pool_map(pool, order, p, &alloc-
> > > > dma_addr);
> > > +		r = ttm_pool_map(pool, order, p, &first_dma);
> > >   		if (r)
> > >   			return r;
> > >   	}
> > >   
> > > -	alloc->remaining_pages -= nr;
> > > -	for (i = 0; i < nr; ++i)
> > > -		*alloc->pages++ = p++;
> > > +	if (restore) {
> > > +		ttm_pool_page_allocated_restore(pool, order, p,
> > > page_caching,
> > > +						first_dma,
> > > restore, alloc);
> > > +	} else {
> > > +		ttm_pool_allocated_page_commit(p, first_dma,
> > > alloc, 1UL << order);
> > >   
> > > -	if (caching_consistent)
> > > -		alloc->caching_divide = alloc->pages;
> > > +		if (caching_consistent)
> > > +			alloc->caching_divide = alloc->pages;
> > > +	}
> > >   
> > >   	return 0;
> > >   }
> > > @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct
> > > ttm_pool *pool, struct ttm_tt *tt,
> > >   				pgoff_t start_page, pgoff_t
> > > end_page)
> > >   {
> > >   	struct page **pages = &tt->pages[start_page];
> > > -	unsigned int order;
> > > +	struct ttm_backup *backup = tt->backup;
> > >   	pgoff_t i, nr;
> > >   
> > >   	for (i = start_page; i < end_page; i += nr, pages += nr)
> > > {
> > > -		struct ttm_pool_type *pt = NULL;
> > > +		struct page *p = *pages;
> > >   
> > > -		order = ttm_pool_page_order(pool, *pages);
> > > -		nr = (1UL << order);
> > > -		if (tt->dma_address)
> > > -			ttm_pool_unmap(pool, tt->dma_address[i],
> > > nr);
> > > +		nr = 1;
> > > +		if (ttm_backup_page_ptr_is_handle(p)) {
> > > +			unsigned long handle =
> > > ttm_backup_page_ptr_to_handle(p);
> > >   
> > > -		pt = ttm_pool_select_type(pool, caching, order);
> > > -		if (pt)
> > > -			ttm_pool_type_give(pt, *pages);
> > > -		else
> > > -			ttm_pool_free_page(pool, caching, order,
> > > *pages);
> > > +			if (handle != 0)
> > > +				ttm_backup_drop(backup, handle);
> > > +		} else if (p) {
> > > +			dma_addr_t *dma_addr = tt->dma_address ?
> > > +				tt->dma_address + i : NULL;
> > > +
> > > +			nr = ttm_pool_unmap_and_free(pool, p,
> > > dma_addr, caching);
> > > +		}
> > >   	}
> > >   }
> > >   
> > > @@ -467,22 +683,11 @@ static unsigned int
> > > ttm_pool_alloc_find_order(unsigned int highest,
> > >   	return min_t(unsigned int, highest, __fls(alloc-
> > > > remaining_pages));
> > >   }
> > >   
> > > -/**
> > > - * ttm_pool_alloc - Fill a ttm_tt object
> > > - *
> > > - * @pool: ttm_pool to use
> > > - * @tt: ttm_tt object to fill
> > > - * @ctx: operation context
> > > - *
> > > - * Fill the ttm_tt object with pages and also make sure to DMA
> > > map
> > > them when
> > > - * necessary.
> > > - *
> > > - * Returns: 0 on successe, negative error code otherwise.
> > > - */
> > > -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > -		   struct ttm_operation_ctx *ctx)
> > > +static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
> > > *tt,
> > > +			    const struct ttm_operation_ctx *ctx,
> > > +			    struct ttm_pool_alloc_state *alloc,
> > > +			    struct ttm_pool_tt_restore *restore)
> > >   {
> > > -	struct ttm_pool_alloc_state alloc;
> > >   	enum ttm_caching page_caching;
> > >   	gfp_t gfp_flags = GFP_USER;
> > >   	pgoff_t caching_divide;
> > > @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > struct ttm_tt *tt,
> > >   	struct page *p;
> > >   	int r;
> > >   
> > > -	ttm_pool_alloc_state_init(tt, &alloc);
> > > -
> > > -	WARN_ON(!alloc.remaining_pages ||
> > > ttm_tt_is_populated(tt));
> > > -	WARN_ON(alloc.dma_addr && !pool->dev);
> > > +	WARN_ON(!alloc->remaining_pages ||
> > > ttm_tt_is_populated(tt));
> > > +	WARN_ON(alloc->dma_addr && !pool->dev);
> > >   
> > >   	if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
> > >   		gfp_flags |= __GFP_ZERO;
> > > @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > struct ttm_tt *tt,
> > >   
> > >   	page_caching = tt->caching;
> > >   	allow_pools = true;
> > > -	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > &alloc);
> > > -	     alloc.remaining_pages;
> > > -	     order = ttm_pool_alloc_find_order(order, &alloc)) {
> > > +	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > alloc);
> > > +	     alloc->remaining_pages;
> > > +	     order = ttm_pool_alloc_find_order(order, alloc)) {
> > >   		struct ttm_pool_type *pt;
> > >   
> > >   		/* First, try to allocate a page from a pool if
> > > one exists. */
> > > @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > struct ttm_tt *tt,
> > >   			r = -ENOMEM;
> > >   			goto error_free_all;
> > >   		}
> > > -		r = ttm_pool_page_allocated(pool, order, p,
> > > page_caching, &alloc);
> > > +		r = ttm_pool_page_allocated(pool, order, p,
> > > page_caching, alloc,
> > > +					    restore);
> > >   		if (r)
> > >   			goto error_free_page;
> > > +
> > > +		if (ttm_pool_restore_valid(restore)) {
> > > +			r = ttm_pool_restore_commit(restore, tt-
> > > > backup, ctx, alloc);
> > > +			if (r)
> > > +				goto error_free_all;
> > > +		}
> > >   	}
> > >   
> > > -	r = ttm_pool_apply_caching(&alloc);
> > > +	r = ttm_pool_apply_caching(alloc);
> > >   	if (r)
> > >   		goto error_free_all;
> > >   
> > > +	kfree(tt->restore);
> > > +	tt->restore = NULL;
> > > +
> > >   	return 0;
> > >   
> > >   error_free_page:
> > >   	ttm_pool_free_page(pool, page_caching, order, p);
> > >   
> > >   error_free_all:
> > > -	caching_divide = alloc.caching_divide - tt->pages;
> > > +	if (tt->restore)
> > > +		return r;
> > > +
> > > +	caching_divide = alloc->caching_divide - tt->pages;
> > >   	ttm_pool_free_range(pool, tt, tt->caching, 0,
> > > caching_divide);
> > >   	ttm_pool_free_range(pool, tt, ttm_cached,
> > > caching_divide,
> > > -			    tt->num_pages -
> > > alloc.remaining_pages);
> > > +			    tt->num_pages - alloc-
> > > > remaining_pages);
> > >   
> > >   	return r;
> > >   }
> > > +
> > > +/**
> > > + * ttm_pool_alloc - Fill a ttm_tt object
> > > + *
> > > + * @pool: ttm_pool to use
> > > + * @tt: ttm_tt object to fill
> > > + * @ctx: operation context
> > > + *
> > > + * Fill the ttm_tt object with pages and also make sure to DMA
> > > map
> > > them when
> > > + * necessary.
> > > + *
> > > + * Returns: 0 on successe, negative error code otherwise.
> > > + */
> > > +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > +		   struct ttm_operation_ctx *ctx)
> > > +{
> > > +	struct ttm_pool_alloc_state alloc;
> > > +
> > > +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > +		return -EINVAL;
> > > +
> > > +	ttm_pool_alloc_state_init(tt, &alloc);
> > > +
> > > +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> > > +}
> > >   EXPORT_SYMBOL(ttm_pool_alloc);
> > >   
> > > +/**
> > > + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
> > > previously backed-up
> > > + * content.
> > > + *
> > > + * @pool: ttm_pool to use
> > > + * @tt: ttm_tt object to fill
> > > + * @ctx: operation context
> > > + *
> > > + * Fill the ttm_tt object with pages and also make sure to DMA
> > > map
> > > them when
> > > + * necessary. Read in backed-up content.
> > > + *
> > > + * Returns: 0 on successe, negative error code otherwise.
> > > + */
> > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > ttm_tt *tt,
> > > +			       const struct ttm_operation_ctx
> > > *ctx)
> > > +{
> > > +	struct ttm_pool_alloc_state alloc;
> > > +
> > > +	if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> > > +		return -EINVAL;
> > > +
> > > +	if (!tt->restore) {
> > > +		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> > > +
> > > +		ttm_pool_alloc_state_init(tt, &alloc);
> > > +		if (ctx->gfp_retry_mayfail)
> > > +			gfp |= __GFP_RETRY_MAYFAIL;
> > > +
> > > +		tt->restore = kzalloc(sizeof(*tt->restore),
> > > gfp);
> > > +		if (!tt->restore)
> > > +			return -ENOMEM;
> > > +
> > > +		tt->restore->snapshot_alloc = alloc;
> > > +		tt->restore->pool = pool;
> > > +		tt->restore->restored_pages = 1;
> > > +	} else {
> > > +		struct ttm_pool_tt_restore *restore = tt-
> > > >restore;
> > > +		int ret;
> > > +
> > > +		alloc = restore->snapshot_alloc;
> > > +		if (ttm_pool_restore_valid(tt->restore)) {
> > > +			ret = ttm_pool_restore_commit(restore,
> > > tt-
> > > > backup, ctx, &alloc);
> > > +			if (ret)
> > > +				return ret;
> > > +		}
> > > +		if (!alloc.remaining_pages)
> > > +			return 0;
> > > +	}
> > > +
> > > +	return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
> > > > restore);
> > > +}
> > > +
> > >   /**
> > >    * ttm_pool_free - Free the backing pages from a ttm_tt object
> > >    *
> > > @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool,
> > > struct ttm_tt *tt)
> > >   }
> > >   EXPORT_SYMBOL(ttm_pool_free);
> > >   
> > > +/**
> > > + * ttm_pool_drop_backed_up() - Release content of a swapped-out
> > > struct ttm_tt
> > > + * @tt: The struct ttm_tt.
> > > + *
> > > + * Release handles with associated content or any remaining
> > > pages
> > > of
> > > + * a backed-up struct ttm_tt.
> > > + */
> > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> > > +{
> > > +	struct ttm_pool_tt_restore *restore;
> > > +	pgoff_t start_page = 0;
> > > +
> > > +	WARN_ON(!ttm_tt_is_backed_up(tt));
> > > +
> > > +	restore = tt->restore;
> > > +
> > > +	/*
> > > +	 * Unmap and free any uncommitted restore page.
> > > +	 * any tt page-array backup entries already read back
> > > has
> > > +	 * been cleared already
> > > +	 */
> > > +	if (ttm_pool_restore_valid(restore)) {
> > > +		dma_addr_t *dma_addr = tt->dma_address ?
> > > &restore-
> > > > first_dma : NULL;
> > > +
> > > +		ttm_pool_unmap_and_free(restore->pool, restore-
> > > > alloced_page,
> > > +					dma_addr, restore-
> > > > page_caching);
> > > +		restore->restored_pages = 1UL << restore->order;
> > > +	}
> > > +
> > > +	/*
> > > +	 * If a restore is ongoing, part of the tt pages may
> > > have
> > > a
> > > +	 * caching different than writeback.
> > > +	 */
> > > +	if (restore) {
> > > +		pgoff_t mid = restore-
> > > > snapshot_alloc.caching_divide - tt->pages;
> > > +
> > > +		start_page = restore->alloced_pages;
> > > +		WARN_ON(mid > start_page);
> > > +		/* Pages that might be dma-mapped and non-cached
> > > */
> > > +		ttm_pool_free_range(restore->pool, tt, tt-
> > > > caching,
> > > +				    0, mid);
> > > +		/* Pages that might be dma-mapped but cached */
> > > +		ttm_pool_free_range(restore->pool, tt,
> > > ttm_cached,
> > > +				    mid, restore-
> > > >alloced_pages);
> > > +		kfree(restore);
> > > +		tt->restore = NULL;
> > > +	}
> > > +
> > > +	ttm_pool_free_range(NULL, tt, ttm_cached, start_page,
> > > tt-
> > > > num_pages);
> > > +}
> > > +
> > > +/**
> > > + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> > > + * @pool: The pool used when allocating the struct ttm_tt.
> > > + * @tt: The struct ttm_tt.
> > > + * @flags: Flags to govern the backup behaviour.
> > > + *
> > > + * Back up or purge a struct ttm_tt. If @purge is true, then
> > > + * all pages will be freed directly to the system rather than to
> > > the pool
> > > + * they were allocated from, making the function behave
> > > similarly
> > > to
> > > + * ttm_pool_free(). If @purge is false the pages will be backed
> > > up
> > > instead,
> > > + * exchanged for handles.
> > > + * A subsequent call to ttm_pool_restore_and_alloc() will then
> > > read back the content and
> > > + * a subsequent call to ttm_pool_drop_backed_up() will drop it.
> > > + * If backup of a page fails for whatever reason, @ttm will
> > > still
> > > be
> > > + * partially backed up, retaining those pages for which backup
> > > fails.
> > > + * In that case, this function can be retried, possibly after
> > > freeing up
> > > + * memory resources.
> > > + *
> > > + * Return: Number of pages actually backed up or freed, or
> > > negative
> > > + * error code on error.
> > > + */
> > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
> > > +		     const struct ttm_backup_flags *flags)
> > > +{
> > > +	struct ttm_backup *backup = tt->backup;
> > > +	struct page *page;
> > > +	unsigned long handle;
> > > +	gfp_t alloc_gfp;
> > > +	gfp_t gfp;
> > > +	int ret = 0;
> > > +	pgoff_t shrunken = 0;
> > > +	pgoff_t i, num_pages;
> > > +
> > > +	if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > +		return -EINVAL;
> > > +
> > > +	if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> > > +	    pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> > > +		return -EBUSY;
> > > +
> > > +#ifdef CONFIG_X86
> > > +	/* Anything returned to the system needs to be cached.
> > > */
> > > +	if (tt->caching != ttm_cached)
> > > +		set_pages_array_wb(tt->pages, tt->num_pages);
> > > +#endif
> > > +
> > > +	if (tt->dma_address || flags->purge) {
> > > +		for (i = 0; i < tt->num_pages; i += num_pages) {
> > > +			unsigned int order;
> > > +
> > > +			page = tt->pages[i];
> > > +			if (unlikely(!page)) {
> > > +				num_pages = 1;
> > > +				continue;
> > > +			}
> > > +
> > > +			order = ttm_pool_page_order(pool, page);
> > > +			num_pages = 1UL << order;
> > > +			if (tt->dma_address)
> > > +				ttm_pool_unmap(pool, tt-
> > > > dma_address[i],
> > > +					       num_pages);
> > > +			if (flags->purge) {
> > > +				shrunken += num_pages;
> > > +				page->private = 0;
> > > +				__free_pages(page, order);
> > > +				memset(tt->pages + i, 0,
> > > +				       num_pages * sizeof(*tt-
> > > > pages));
> > > +			}
> > > +		}
> > > +	}
> > > +
> > > +	if (flags->purge)
> > > +		return shrunken;
> > > +
> > > +	if (pool->use_dma32)
> > > +		gfp = GFP_DMA32;
> > > +	else
> > > +		gfp = GFP_HIGHUSER;
> > > +
> > > +	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
> > > __GFP_RETRY_MAYFAIL;
> > > +
> > > +	for (i = 0; i < tt->num_pages; ++i) {
> > > +		s64 shandle;
> > > +
> > > +		page = tt->pages[i];
> > > +		if (unlikely(!page))
> > > +			continue;
> > > +
> > > +		ttm_pool_split_for_swap(pool, page);
> > > +
> > > +		shandle = ttm_backup_backup_page(backup, page,
> > > flags->writeback, i,
> > > +						 gfp,
> > > alloc_gfp);
> > > +		if (shandle < 0) {
> > > +			/* We allow partially shrunken tts */
> > > +			ret = shandle;
> > > +			break;
> > > +		}
> > > +		handle = shandle;
> > > +		tt->pages[i] =
> > > ttm_backup_handle_to_page_ptr(handle);
> > > +		put_page(page);
> > > +		shrunken++;
> > > +	}
> > > +
> > > +	return shrunken ? shrunken : ret;
> > > +}
> > > +
> > >   /**
> > >    * ttm_pool_init - Initialize a pool
> > >    *
> > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > index 3baf215eca23..00b7c28f2329 100644
> > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > @@ -40,6 +40,7 @@
> > >   #include <drm/drm_cache.h>
> > >   #include <drm/drm_device.h>
> > >   #include <drm/drm_util.h>
> > > +#include <drm/ttm/ttm_backup.h>
> > >   #include <drm/ttm/ttm_bo.h>
> > >   #include <drm/ttm/ttm_tt.h>
> > >   
> > > @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt
> > > *ttm,
> > >   	ttm->swap_storage = NULL;
> > >   	ttm->sg = bo->sg;
> > >   	ttm->caching = caching;
> > > +	ttm->restore = NULL;
> > > +	ttm->backup = NULL;
> > >   }
> > >   
> > >   int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object
> > > *bo,
> > > @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
> > >   		fput(ttm->swap_storage);
> > >   	ttm->swap_storage = NULL;
> > >   
> > > +	if (ttm_tt_is_backed_up(ttm))
> > > +		ttm_pool_drop_backed_up(ttm);
> > > +	if (ttm->backup) {
> > > +		ttm_backup_fini(ttm->backup);
> > > +		ttm->backup = NULL;
> > > +	}
> > > +
> > >   	if (ttm->pages)
> > >   		kvfree(ttm->pages);
> > >   	else
> > > @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
> > >   }
> > >   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
> > >   
> > > +/**
> > > + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> > > + * @bdev: The TTM device.
> > > + * @tt: The struct ttm_tt.
> > > + * @flags: Flags that govern the backup behaviour.
> > > + *
> > > + * Update the page accounting and call ttm_pool_shrink_tt to
> > > free
> > > pages
> > > + * or back them up.
> > > + *
> > > + * Return: Number of pages freed or swapped out, or negative
> > > error
> > > code on
> > > + * error.
> > > + */
> > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > > +		   const struct ttm_backup_flags flags)
> > > +{
> > > +	long ret;
> > > +
> > > +	if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> > > +		return 0;
> > > +
> > > +	ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> > > +	if (ret > 0) {
> > > +		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> > > +		tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> > > +	}
> > > +
> > > +	return ret;
> > > +}
> > > +
> > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > > +		   const struct ttm_operation_ctx *ctx)
> > > +{
> > > +	int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
> > > ctx);
> > > +
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL(ttm_tt_restore);
> > > +
> > >   /**
> > >    * ttm_tt_swapout - swap out tt object
> > >    *
> > > @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
> > >   		goto error;
> > >   
> > >   	ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> > > +	ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > >   	if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
> > >   		ret = ttm_tt_swapin(ttm);
> > >   		if (unlikely(ret != 0)) {
> > > diff --git a/include/drm/ttm/ttm_pool.h
> > > b/include/drm/ttm/ttm_pool.h
> > > index 160d954a261e..54cd34a6e4c0 100644
> > > --- a/include/drm/ttm/ttm_pool.h
> > > +++ b/include/drm/ttm/ttm_pool.h
> > > @@ -33,6 +33,7 @@
> > >   
> > >   struct device;
> > >   struct seq_file;
> > > +struct ttm_backup_flags;
> > >   struct ttm_operation_ctx;
> > >   struct ttm_pool;
> > >   struct ttm_tt;
> > > @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
> > >   
> > >   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file
> > > *m);
> > >   
> > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> > > +
> > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
> > > +		     const struct ttm_backup_flags *flags);
> > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > ttm_tt *tt,
> > > +			       const struct ttm_operation_ctx
> > > *ctx);
> > > +
> > >   int ttm_pool_mgr_init(unsigned long num_pages);
> > >   void ttm_pool_mgr_fini(void);
> > >   
> > > diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> > > index 991edafdb2dd..c736c01ac2ca 100644
> > > --- a/include/drm/ttm/ttm_tt.h
> > > +++ b/include/drm/ttm/ttm_tt.h
> > > @@ -32,11 +32,13 @@
> > >   #include <drm/ttm/ttm_caching.h>
> > >   #include <drm/ttm/ttm_kmap_iter.h>
> > >   
> > > +struct ttm_backup;
> > >   struct ttm_device;
> > >   struct ttm_tt;
> > >   struct ttm_resource;
> > >   struct ttm_buffer_object;
> > >   struct ttm_operation_ctx;
> > > +struct ttm_pool_tt_restore;
> > >   
> > >   /**
> > >    * struct ttm_tt - This is a structure holding the pages,
> > > caching- and aperture
> > > @@ -85,17 +87,22 @@ struct ttm_tt {
> > >   	 * fault handling abuses the DMA api a bit and
> > > dma_map_attrs can't be
> > >   	 * used to assure pgprot always matches.
> > >   	 *
> > > +	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
> > > if the
> > > +	 * struct ttm_tt has been (possibly partially) backed
> > > up.
> > > +	 *
> > >   	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT
> > > USE. This is
> > >   	 * set by TTM after ttm_tt_populate() has successfully
> > > returned, and is
> > >   	 * then unset when TTM calls ttm_tt_unpopulate().
> > > +	 *
> > >   	 */
> > >   #define TTM_TT_FLAG_SWAPPED		BIT(0)
> > >   #define TTM_TT_FLAG_ZERO_ALLOC		BIT(1)
> > >   #define TTM_TT_FLAG_EXTERNAL		BIT(2)
> > >   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
> > >   #define TTM_TT_FLAG_DECRYPTED		BIT(4)
> > > +#define TTM_TT_FLAG_BACKED_UP	        BIT(5)
> > >   
> > > -#define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
> > > +#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
> > >   	uint32_t page_flags;
> > >   	/** @num_pages: Number of pages in the page array. */
> > >   	uint32_t num_pages;
> > > @@ -105,11 +112,20 @@ struct ttm_tt {
> > >   	dma_addr_t *dma_address;
> > >   	/** @swap_storage: Pointer to shmem struct file for swap
> > > storage. */
> > >   	struct file *swap_storage;
> > > +	/**
> > > +	 * @backup: Pointer to backup struct for backed up tts.
> > > +	 * Could be unified with @swap_storage. Meanwhile, the
> > > driver's
> > > +	 * ttm_tt_create() callback is responsible for assigning
> > > +	 * this field.
> > > +	 */
> > > +	struct ttm_backup *backup;
> > >   	/**
> > >   	 * @caching: The current caching state of the pages, see
> > > enum
> > >   	 * ttm_caching.
> > >   	 */
> > >   	enum ttm_caching caching;
> > > +	/** @restore: Partial restoration from backup state. TTM
> > > private */
> > > +	struct ttm_pool_tt_restore *restore;
> > >   };
> > >   
> > >   /**
> > > @@ -129,9 +145,38 @@ static inline bool
> > > ttm_tt_is_populated(struct
> > > ttm_tt *tt)
> > >   	return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
> > >   }
> > >   
> > > +/**
> > > + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or
> > > backed up
> > > + * @tt: The struct ttm_tt.
> > > + *
> > > + * Return: true if swapped or backed up, false otherwise.
> > > + */
> > >   static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
> > >   {
> > > -	return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> > > +	return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
> > > TTM_TT_FLAG_BACKED_UP);
> > > +}
> > > +
> > > +/**
> > > + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> > > + * @tt: The struct ttm_tt.
> > > + *
> > > + * Return: true if swapped or backed up, false otherwise.
> > > + */
> > > +static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
> > > +{
> > > +	return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> > > +}
> > > +
> > > +/**
> > > + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
> > > + * @tt: The struct ttm_tt.
> > > + *
> > > + * Drivers can use this functionto clear the backed-up status,
> > > + * for example before destroying or re-validating a purged tt.
> > > + */
> > > +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
> > > +{
> > > +	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > >   }
> > >   
> > >   /**
> > > @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long
> > > num_pages,
> > > unsigned long num_dma32_pages);
> > >   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
> > > ttm_kmap_iter_tt *iter_tt,
> > >   					    struct ttm_tt *tt);
> > >   unsigned long ttm_tt_pages_limit(void);
> > > +
> > > +/**
> > > + * struct ttm_backup_flags - Flags to govern backup behaviour.
> > > + * @purge: Free pages without backing up. Bypass pools.
> > > + * @writeback: Attempt to copy contents directly to swap space,
> > > even
> > > + * if that means blocking on writes to external memory.
> > > + */
> > > +struct ttm_backup_flags {
> > > +	u32 purge : 1;
> > > +	u32 writeback : 1;
> > > +};
> > > +
> > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > > +		   const struct ttm_backup_flags flags);
> > > +
> > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > > +		   const struct ttm_operation_ctx *ctx);
> > > +
> > >   #if IS_ENABLED(CONFIG_AGP)
> > >   #include <linux/agp_backend.h>
> > >   
> > 
> 

Re: RESEND Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Dave Airlie]

I've looked over the two patches mentioned here, I think they have
seen enough time and we need to unblock,

Please add and merge them:
Acked-by: Dave Airlie <airlied@redhat.com>

Dave.

On Tue, 25 Feb 2025 at 18:44, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> Hi, Christian,
>
> Ping? I'd really want to get this in before -rc6
>
> Thanks,
> Thomas
>
>
>
> On Tue, 2025-02-18 at 16:40 +0100, Thomas Hellström wrote:
> > Hi, Christian,
> >
> > On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
> > > Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> > > > Provide a helper to shrink ttm_tt page-vectors on a per-page
> > > > basis. A ttm_backup backend could then in theory get away with
> > > > allocating a single temporary page for each struct ttm_tt.
> > > >
> > > > This is accomplished by splitting larger pages before trying to
> > > > back them up.
> > > >
> > > > In the future we could allow ttm_backup to handle backing up
> > > > large pages as well, but currently there's no benefit in
> > > > doing that, since the shmem backup backend would have to
> > > > split those anyway to avoid allocating too much temporary
> > > > memory, and if the backend instead inserts pages into the
> > > > swap-cache, those are split on reclaim by the core.
> > > >
> > > > Due to potential backup- and recover errors, allow partially
> > > > swapped
> > > > out struct ttm_tt's, although mark them as swapped out stopping
> > > > them
> > > > from being swapped out a second time. More details in the
> > > > ttm_pool.c
> > > > DOC section.
> > > >
> > > > v2:
> > > > - A couple of cleanups and error fixes in ttm_pool_back_up_tt.
> > > > - s/back_up/backup/
> > > > - Add a writeback parameter to the exported interface.
> > > > v8:
> > > > - Use a struct for flags for readability (Matt Brost)
> > > > - Address misc other review comments (Matt Brost)
> > > > v9:
> > > > - Update the kerneldoc for the ttm_tt::backup field.
> > > > v10:
> > > > - Rebase.
> > > > v13:
> > > > - Rebase on ttm_backup interface change. Update kerneldoc.
> > > > - Rebase and adjust ttm_tt_is_swapped().
> > > > v15:
> > > > - Rebase on ttm_backup return value change.
> > > > - Rebase on previous restructuring of ttm_pool_alloc()
> > > > - Rework the ttm_pool backup interface (Christian König)
> > > > - Remove cond_resched() (Christian König)
> > > > - Get rid of the need to allocate an intermediate page array
> > > >    when restoring a multi-order page (Christian König)
> > > > - Update documentation.
> > > >
> > > > Cc: Christian König <christian.koenig@amd.com>
> > > > Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> > > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > > Cc: <dri-devel@lists.freedesktop.org>
> > > > Signed-off-by: Thomas Hellström
> > > > <thomas.hellstrom@linux.intel.com>
> > > > Reviewed-by: Matthew Brost <matthew.brost@intel.com>
> > >
> > > I've tried to wrap my head around all of this like twenty times in
> > > the
> > > last three month, but was always interrupted at some point.
> > >
> > > Feel free to add Acked-by: Christian Koenig
> > > <christian.koenig@amd.com>.
> > >
> > > Sorry,
> > > Christian.
> >
> > Thanks a lot for all reviewing and comments so far. There are two TTM
> > patches left in the series that don't have an ack by you:
> >
> > https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
> > and
> >
> > https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17
> >
> > None of them particularly big considering the amount of doc text.
> >
> > It'd be great if those could have an ack as well so we could finally
> > merge this series.
> >
> > Thanks,
> > Thomas
> >
> >
> >
> >
> >
> >
> > >
> > > > ---
> > > >   drivers/gpu/drm/ttm/ttm_pool.c | 554
> > > > +++++++++++++++++++++++++++++----
> > > >   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
> > > >   include/drm/ttm/ttm_pool.h     |   8 +
> > > >   include/drm/ttm/ttm_tt.h       |  67 +++-
> > > >   4 files changed, 629 insertions(+), 54 deletions(-)
> > > >
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > index c9eba76d5143..ffb7abf52bab 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > @@ -41,6 +41,7 @@
> > > >   #include <asm/set_memory.h>
> > > >   #endif
> > > >
> > > > +#include <drm/ttm/ttm_backup.h>
> > > >   #include <drm/ttm/ttm_pool.h>
> > > >   #include <drm/ttm/ttm_tt.h>
> > > >   #include <drm/ttm/ttm_bo.h>
> > > > @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
> > > >           enum ttm_caching tt_caching;
> > > >   };
> > > >
> > > > +/**
> > > > + * struct ttm_pool_tt_restore - State representing restore from
> > > > backup
> > > > + * @pool: The pool used for page allocation while restoring.
> > > > + * @snapshot_alloc: A snapshot of the most recent struct
> > > > ttm_pool_alloc_state.
> > > > + * @alloced_page: Pointer to the page most recently allocated
> > > > from
> > > > a pool or system.
> > > > + * @first_dma: The dma address corresponding to @alloced_page if
> > > > dma_mapping
> > > > + * is requested.
> > > > + * @alloced_pages: The number of allocated pages present in the
> > > > struct ttm_tt
> > > > + * page vector from this restore session.
> > > > + * @restored_pages: The number of 4K pages restored for
> > > > @alloced_page (which
> > > > + * is typically a multi-order page).
> > > > + * @page_caching: The struct ttm_tt requested caching
> > > > + * @order: The order of @alloced_page.
> > > > + *
> > > > + * Recovery from backup might fail when we've recovered less
> > > > than
> > > > the
> > > > + * full ttm_tt. In order not to loose any data (yet), keep
> > > > information
> > > > + * around that allows us to restart a failed ttm backup
> > > > recovery.
> > > > + */
> > > > +struct ttm_pool_tt_restore {
> > > > + struct ttm_pool *pool;
> > > > + struct ttm_pool_alloc_state snapshot_alloc;
> > > > + struct page *alloced_page;
> > > > + dma_addr_t first_dma;
> > > > + pgoff_t alloced_pages;
> > > > + pgoff_t restored_pages;
> > > > + enum ttm_caching page_caching;
> > > > + unsigned int order;
> > > > +};
> > > > +
> > > >   static unsigned long page_pool_size;
> > > >
> > > >   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
> > > > WC/UC/DMA pool");
> > > > @@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct
> > > > ttm_pool_alloc_state *alloc)
> > > >           return 0;
> > > >   }
> > > >
> > > > -/* Map pages of 1 << order size and fill the DMA address array
> > > > */
> > > > +/* DMA Map pages of 1 << order size and return the resulting
> > > > dma_address. */
> > > >   static int ttm_pool_map(struct ttm_pool *pool, unsigned int
> > > > order,
> > > > -                 struct page *p, dma_addr_t **dma_addr)
> > > > +                 struct page *p, dma_addr_t *dma_addr)
> > > >   {
> > > >           dma_addr_t addr;
> > > > - unsigned int i;
> > > >
> > > >           if (pool->use_dma_alloc) {
> > > >                   struct ttm_pool_dma *dma = (void *)p->private;
> > > > @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool
> > > > *pool,
> > > > unsigned int order,
> > > >                           return -EFAULT;
> > > >           }
> > > >
> > > > - for (i = 1 << order; i ; --i) {
> > > > -         *(*dma_addr)++ = addr;
> > > > -         addr += PAGE_SIZE;
> > > > - }
> > > > + *dma_addr = addr;
> > > >
> > > >           return 0;
> > > >   }
> > > > @@ -371,6 +397,190 @@ static unsigned int
> > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > >           return p->private;
> > > >   }
> > > >
> > > > +/*
> > > > + * Split larger pages so that we can free each PAGE_SIZE page as
> > > > soon
> > > > + * as it has been backed up, in order to avoid memory pressure
> > > > during
> > > > + * reclaim.
> > > > + */
> > > > +static void ttm_pool_split_for_swap(struct ttm_pool *pool,
> > > > struct
> > > > page *p)
> > > > +{
> > > > + unsigned int order = ttm_pool_page_order(pool, p);
> > > > + pgoff_t nr;
> > > > +
> > > > + if (!order)
> > > > +         return;
> > > > +
> > > > + split_page(p, order);
> > > > + nr = 1UL << order;
> > > > + while (nr--)
> > > > +         (p++)->private = 0;
> > > > +}
> > > > +
> > > > +/**
> > > > + * DOC: Partial backup and restoration of a struct ttm_tt.
> > > > + *
> > > > + * Swapout using ttm_backup_backup_page() and swapin using
> > > > + * ttm_backup_copy_page() may fail.
> > > > + * The former most likely due to lack of swap-space or memory,
> > > > the
> > > > latter due
> > > > + * to lack of memory or because of signal interruption during
> > > > waits.
> > > > + *
> > > > + * Backup failure is easily handled by using a ttm_tt pages
> > > > vector
> > > > that holds
> > > > + * both backup handles and page pointers. This has to be taken
> > > > into account when
> > > > + * restoring such a ttm_tt from backup, and when freeing it
> > > > while
> > > > backed up.
> > > > + * When restoring, for simplicity, new pages are actually
> > > > allocated from the
> > > > + * pool and the contents of any old pages are copied in and then
> > > > the old pages
> > > > + * are released.
> > > > + *
> > > > + * For restoration failures, the struct ttm_pool_tt_restore
> > > > holds
> > > > sufficient state
> > > > + * to be able to resume an interrupted restore, and that
> > > > structure
> > > > is freed once
> > > > + * the restoration is complete. If the struct ttm_tt is
> > > > destroyed
> > > > while there
> > > > + * is a valid struct ttm_pool_tt_restore attached, that is also
> > > > properly taken
> > > > + * care of.
> > > > + */
> > > > +
> > > > +/* Is restore ongoing for the currently allocated page? */
> > > > +static bool ttm_pool_restore_valid(const struct
> > > > ttm_pool_tt_restore *restore)
> > > > +{
> > > > + return restore && restore->restored_pages < (1 <<
> > > > restore-
> > > > > order);
> > > > +}
> > > > +
> > > > +/* DMA unmap and free a multi-order page, either to the relevant
> > > > pool or to system. */
> > > > +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool,
> > > > struct page *page,
> > > > +                                const dma_addr_t
> > > > *dma_addr,
> > > > enum ttm_caching caching)
> > > > +{
> > > > + struct ttm_pool_type *pt = NULL;
> > > > + unsigned int order;
> > > > + pgoff_t nr;
> > > > +
> > > > + if (pool) {
> > > > +         order = ttm_pool_page_order(pool, page);
> > > > +         nr = (1UL << order);
> > > > +         if (dma_addr)
> > > > +                 ttm_pool_unmap(pool, *dma_addr, nr);
> > > > +
> > > > +         pt = ttm_pool_select_type(pool, caching, order);
> > > > + } else {
> > > > +         order = page->private;
> > > > +         nr = (1UL << order);
> > > > + }
> > > > +
> > > > + if (pt)
> > > > +         ttm_pool_type_give(pt, page);
> > > > + else
> > > > +         ttm_pool_free_page(pool, caching, order, page);
> > > > +
> > > > + return nr;
> > > > +}
> > > > +
> > > > +/* Populate the page-array using the most recent allocated
> > > > multi-
> > > > order page. */
> > > > +static void ttm_pool_allocated_page_commit(struct page
> > > > *allocated,
> > > > +                                    dma_addr_t first_dma,
> > > > +                                    struct
> > > > ttm_pool_alloc_state *alloc,
> > > > +                                    pgoff_t nr)
> > > > +{
> > > > + pgoff_t i;
> > > > +
> > > > + for (i = 0; i < nr; ++i)
> > > > +         *alloc->pages++ = allocated++;
> > > > +
> > > > + alloc->remaining_pages -= nr;
> > > > +
> > > > + if (!alloc->dma_addr)
> > > > +         return;
> > > > +
> > > > + for (i = 0; i < nr; ++i) {
> > > > +         *alloc->dma_addr++ = first_dma;
> > > > +         first_dma += PAGE_SIZE;
> > > > + }
> > > > +}
> > > > +
> > > > +/*
> > > > + * When restoring, restore backed-up content to the newly
> > > > allocated page and
> > > > + * if successful, populate the page-table and dma-address
> > > > arrays.
> > > > + */
> > > > +static int ttm_pool_restore_commit(struct ttm_pool_tt_restore
> > > > *restore,
> > > > +                            struct ttm_backup *backup,
> > > > +                            const struct
> > > > ttm_operation_ctx
> > > > *ctx,
> > > > +                            struct ttm_pool_alloc_state
> > > > *alloc)
> > > > +
> > > > +{
> > > > + pgoff_t i, nr = 1UL << restore->order;
> > > > + struct page **first_page = alloc->pages;
> > > > + struct page *p;
> > > > + int ret = 0;
> > > > +
> > > > + for (i = restore->restored_pages; i < nr; ++i) {
> > > > +         p = first_page[i];
> > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > +                 unsigned long handle =
> > > > ttm_backup_page_ptr_to_handle(p);
> > > > +
> > > > +                 if (handle == 0) {
> > > > +                         restore->restored_pages++;
> > > > +                         continue;
> > > > +                 }
> > > > +
> > > > +                 ret = ttm_backup_copy_page(backup,
> > > > restore->alloced_page + i,
> > > > +                                            handle, ctx-
> > > > > interruptible);
> > > > +                 if (ret)
> > > > +                         break;
> > > > +
> > > > +                 ttm_backup_drop(backup, handle);
> > > > +         } else if (p) {
> > > > +                 /*
> > > > +                  * We could probably avoid splitting the
> > > > old page
> > > > +                  * using clever logic, but ATM we don't
> > > > care, as
> > > > +                  * we prioritize releasing memory ASAP.
> > > > Note that
> > > > +                  * here, the old retained page is always
> > > > write-back
> > > > +                  * cached.
> > > > +                  */
> > > > +                 ttm_pool_split_for_swap(restore->pool,
> > > > p);
> > > > +                 copy_highpage(restore->alloced_page + i,
> > > > p);
> > > > +                 __free_pages(p, 0);
> > > > +         }
> > > > +
> > > > +         restore->restored_pages++;
> > > > +         first_page[i] =
> > > > ttm_backup_handle_to_page_ptr(0);
> > > > + }
> > > > +
> > > > + if (ret) {
> > > > +         if (!restore->restored_pages) {
> > > > +                 dma_addr_t *dma_addr = alloc->dma_addr ?
> > > > &restore->first_dma : NULL;
> > > > +
> > > > +                 ttm_pool_unmap_and_free(restore->pool,
> > > > restore->alloced_page,
> > > > +                                         dma_addr,
> > > > restore-
> > > > > page_caching);
> > > > +                 restore->restored_pages = nr;
> > > > +         }
> > > > +         return ret;
> > > > + }
> > > > +
> > > > + ttm_pool_allocated_page_commit(restore->alloced_page,
> > > > restore->first_dma,
> > > > +                                alloc, nr);
> > > > + if (restore->page_caching == alloc->tt_caching ||
> > > > PageHighMem(restore->alloced_page))
> > > > +         alloc->caching_divide = alloc->pages;
> > > > + restore->snapshot_alloc = *alloc;
> > > > + restore->alloced_pages += nr;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +/* If restoring, save information needed for
> > > > ttm_pool_restore_commit(). */
> > > > +static void
> > > > +ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned
> > > > int order,
> > > > +                         struct page *p,
> > > > +                         enum ttm_caching page_caching,
> > > > +                         dma_addr_t first_dma,
> > > > +                         struct ttm_pool_tt_restore
> > > > *restore,
> > > > +                         const struct
> > > > ttm_pool_alloc_state
> > > > *alloc)
> > > > +{
> > > > + restore->pool = pool;
> > > > + restore->order = order;
> > > > + restore->restored_pages = 0;
> > > > + restore->page_caching = page_caching;
> > > > + restore->first_dma = first_dma;
> > > > + restore->alloced_page = p;
> > > > + restore->snapshot_alloc = *alloc;
> > > > +}
> > > > +
> > > >   /*
> > > >    * Called when we got a page, either from a pool or newly
> > > > allocated.
> > > >    * if needed, dma map the page and populate the dma address
> > > > array.
> > > > @@ -380,10 +590,11 @@ static unsigned int
> > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > >    */
> > > >   static int ttm_pool_page_allocated(struct ttm_pool *pool,
> > > > unsigned int order,
> > > >                                      struct page *p, enum
> > > > ttm_caching page_caching,
> > > > -                            struct ttm_pool_alloc_state
> > > > *alloc)
> > > > +                            struct ttm_pool_alloc_state
> > > > *alloc,
> > > > +                            struct ttm_pool_tt_restore
> > > > *restore)
> > > >   {
> > > > - pgoff_t i, nr = 1UL << order;
> > > >           bool caching_consistent;
> > > > + dma_addr_t first_dma;
> > > >           int r = 0;
> > > >
> > > >           caching_consistent = (page_caching == alloc->tt_caching)
> > > > > > PageHighMem(p);
> > > > @@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct
> > > > ttm_pool *pool, unsigned int order,
> > > >           }
> > > >
> > > >           if (alloc->dma_addr) {
> > > > -         r = ttm_pool_map(pool, order, p, &alloc-
> > > > > dma_addr);
> > > > +         r = ttm_pool_map(pool, order, p, &first_dma);
> > > >                   if (r)
> > > >                           return r;
> > > >           }
> > > >
> > > > - alloc->remaining_pages -= nr;
> > > > - for (i = 0; i < nr; ++i)
> > > > -         *alloc->pages++ = p++;
> > > > + if (restore) {
> > > > +         ttm_pool_page_allocated_restore(pool, order, p,
> > > > page_caching,
> > > > +                                         first_dma,
> > > > restore, alloc);
> > > > + } else {
> > > > +         ttm_pool_allocated_page_commit(p, first_dma,
> > > > alloc, 1UL << order);
> > > >
> > > > - if (caching_consistent)
> > > > -         alloc->caching_divide = alloc->pages;
> > > > +         if (caching_consistent)
> > > > +                 alloc->caching_divide = alloc->pages;
> > > > + }
> > > >
> > > >           return 0;
> > > >   }
> > > > @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct
> > > > ttm_pool *pool, struct ttm_tt *tt,
> > > >                                   pgoff_t start_page, pgoff_t
> > > > end_page)
> > > >   {
> > > >           struct page **pages = &tt->pages[start_page];
> > > > - unsigned int order;
> > > > + struct ttm_backup *backup = tt->backup;
> > > >           pgoff_t i, nr;
> > > >
> > > >           for (i = start_page; i < end_page; i += nr, pages += nr)
> > > > {
> > > > -         struct ttm_pool_type *pt = NULL;
> > > > +         struct page *p = *pages;
> > > >
> > > > -         order = ttm_pool_page_order(pool, *pages);
> > > > -         nr = (1UL << order);
> > > > -         if (tt->dma_address)
> > > > -                 ttm_pool_unmap(pool, tt->dma_address[i],
> > > > nr);
> > > > +         nr = 1;
> > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > +                 unsigned long handle =
> > > > ttm_backup_page_ptr_to_handle(p);
> > > >
> > > > -         pt = ttm_pool_select_type(pool, caching, order);
> > > > -         if (pt)
> > > > -                 ttm_pool_type_give(pt, *pages);
> > > > -         else
> > > > -                 ttm_pool_free_page(pool, caching, order,
> > > > *pages);
> > > > +                 if (handle != 0)
> > > > +                         ttm_backup_drop(backup, handle);
> > > > +         } else if (p) {
> > > > +                 dma_addr_t *dma_addr = tt->dma_address ?
> > > > +                         tt->dma_address + i : NULL;
> > > > +
> > > > +                 nr = ttm_pool_unmap_and_free(pool, p,
> > > > dma_addr, caching);
> > > > +         }
> > > >           }
> > > >   }
> > > >
> > > > @@ -467,22 +683,11 @@ static unsigned int
> > > > ttm_pool_alloc_find_order(unsigned int highest,
> > > >           return min_t(unsigned int, highest, __fls(alloc-
> > > > > remaining_pages));
> > > >   }
> > > >
> > > > -/**
> > > > - * ttm_pool_alloc - Fill a ttm_tt object
> > > > - *
> > > > - * @pool: ttm_pool to use
> > > > - * @tt: ttm_tt object to fill
> > > > - * @ctx: operation context
> > > > - *
> > > > - * Fill the ttm_tt object with pages and also make sure to DMA
> > > > map
> > > > them when
> > > > - * necessary.
> > > > - *
> > > > - * Returns: 0 on successe, negative error code otherwise.
> > > > - */
> > > > -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > > -            struct ttm_operation_ctx *ctx)
> > > > +static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
> > > > *tt,
> > > > +                     const struct ttm_operation_ctx *ctx,
> > > > +                     struct ttm_pool_alloc_state *alloc,
> > > > +                     struct ttm_pool_tt_restore *restore)
> > > >   {
> > > > - struct ttm_pool_alloc_state alloc;
> > > >           enum ttm_caching page_caching;
> > > >           gfp_t gfp_flags = GFP_USER;
> > > >           pgoff_t caching_divide;
> > > > @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > > struct ttm_tt *tt,
> > > >           struct page *p;
> > > >           int r;
> > > >
> > > > - ttm_pool_alloc_state_init(tt, &alloc);
> > > > -
> > > > - WARN_ON(!alloc.remaining_pages ||
> > > > ttm_tt_is_populated(tt));
> > > > - WARN_ON(alloc.dma_addr && !pool->dev);
> > > > + WARN_ON(!alloc->remaining_pages ||
> > > > ttm_tt_is_populated(tt));
> > > > + WARN_ON(alloc->dma_addr && !pool->dev);
> > > >
> > > >           if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
> > > >                   gfp_flags |= __GFP_ZERO;
> > > > @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > > struct ttm_tt *tt,
> > > >
> > > >           page_caching = tt->caching;
> > > >           allow_pools = true;
> > > > - for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > &alloc);
> > > > -      alloc.remaining_pages;
> > > > -      order = ttm_pool_alloc_find_order(order, &alloc)) {
> > > > + for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > alloc);
> > > > +      alloc->remaining_pages;
> > > > +      order = ttm_pool_alloc_find_order(order, alloc)) {
> > > >                   struct ttm_pool_type *pt;
> > > >
> > > >                   /* First, try to allocate a page from a pool if
> > > > one exists. */
> > > > @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > > struct ttm_tt *tt,
> > > >                           r = -ENOMEM;
> > > >                           goto error_free_all;
> > > >                   }
> > > > -         r = ttm_pool_page_allocated(pool, order, p,
> > > > page_caching, &alloc);
> > > > +         r = ttm_pool_page_allocated(pool, order, p,
> > > > page_caching, alloc,
> > > > +                                     restore);
> > > >                   if (r)
> > > >                           goto error_free_page;
> > > > +
> > > > +         if (ttm_pool_restore_valid(restore)) {
> > > > +                 r = ttm_pool_restore_commit(restore, tt-
> > > > > backup, ctx, alloc);
> > > > +                 if (r)
> > > > +                         goto error_free_all;
> > > > +         }
> > > >           }
> > > >
> > > > - r = ttm_pool_apply_caching(&alloc);
> > > > + r = ttm_pool_apply_caching(alloc);
> > > >           if (r)
> > > >                   goto error_free_all;
> > > >
> > > > + kfree(tt->restore);
> > > > + tt->restore = NULL;
> > > > +
> > > >           return 0;
> > > >
> > > >   error_free_page:
> > > >           ttm_pool_free_page(pool, page_caching, order, p);
> > > >
> > > >   error_free_all:
> > > > - caching_divide = alloc.caching_divide - tt->pages;
> > > > + if (tt->restore)
> > > > +         return r;
> > > > +
> > > > + caching_divide = alloc->caching_divide - tt->pages;
> > > >           ttm_pool_free_range(pool, tt, tt->caching, 0,
> > > > caching_divide);
> > > >           ttm_pool_free_range(pool, tt, ttm_cached,
> > > > caching_divide,
> > > > -                     tt->num_pages -
> > > > alloc.remaining_pages);
> > > > +                     tt->num_pages - alloc-
> > > > > remaining_pages);
> > > >
> > > >           return r;
> > > >   }
> > > > +
> > > > +/**
> > > > + * ttm_pool_alloc - Fill a ttm_tt object
> > > > + *
> > > > + * @pool: ttm_pool to use
> > > > + * @tt: ttm_tt object to fill
> > > > + * @ctx: operation context
> > > > + *
> > > > + * Fill the ttm_tt object with pages and also make sure to DMA
> > > > map
> > > > them when
> > > > + * necessary.
> > > > + *
> > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > + */
> > > > +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > > +            struct ttm_operation_ctx *ctx)
> > > > +{
> > > > + struct ttm_pool_alloc_state alloc;
> > > > +
> > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > +         return -EINVAL;
> > > > +
> > > > + ttm_pool_alloc_state_init(tt, &alloc);
> > > > +
> > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> > > > +}
> > > >   EXPORT_SYMBOL(ttm_pool_alloc);
> > > >
> > > > +/**
> > > > + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
> > > > previously backed-up
> > > > + * content.
> > > > + *
> > > > + * @pool: ttm_pool to use
> > > > + * @tt: ttm_tt object to fill
> > > > + * @ctx: operation context
> > > > + *
> > > > + * Fill the ttm_tt object with pages and also make sure to DMA
> > > > map
> > > > them when
> > > > + * necessary. Read in backed-up content.
> > > > + *
> > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > + */
> > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > > ttm_tt *tt,
> > > > +                        const struct ttm_operation_ctx
> > > > *ctx)
> > > > +{
> > > > + struct ttm_pool_alloc_state alloc;
> > > > +
> > > > + if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> > > > +         return -EINVAL;
> > > > +
> > > > + if (!tt->restore) {
> > > > +         gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> > > > +
> > > > +         ttm_pool_alloc_state_init(tt, &alloc);
> > > > +         if (ctx->gfp_retry_mayfail)
> > > > +                 gfp |= __GFP_RETRY_MAYFAIL;
> > > > +
> > > > +         tt->restore = kzalloc(sizeof(*tt->restore),
> > > > gfp);
> > > > +         if (!tt->restore)
> > > > +                 return -ENOMEM;
> > > > +
> > > > +         tt->restore->snapshot_alloc = alloc;
> > > > +         tt->restore->pool = pool;
> > > > +         tt->restore->restored_pages = 1;
> > > > + } else {
> > > > +         struct ttm_pool_tt_restore *restore = tt-
> > > > >restore;
> > > > +         int ret;
> > > > +
> > > > +         alloc = restore->snapshot_alloc;
> > > > +         if (ttm_pool_restore_valid(tt->restore)) {
> > > > +                 ret = ttm_pool_restore_commit(restore,
> > > > tt-
> > > > > backup, ctx, &alloc);
> > > > +                 if (ret)
> > > > +                         return ret;
> > > > +         }
> > > > +         if (!alloc.remaining_pages)
> > > > +                 return 0;
> > > > + }
> > > > +
> > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
> > > > > restore);
> > > > +}
> > > > +
> > > >   /**
> > > >    * ttm_pool_free - Free the backing pages from a ttm_tt object
> > > >    *
> > > > @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool,
> > > > struct ttm_tt *tt)
> > > >   }
> > > >   EXPORT_SYMBOL(ttm_pool_free);
> > > >
> > > > +/**
> > > > + * ttm_pool_drop_backed_up() - Release content of a swapped-out
> > > > struct ttm_tt
> > > > + * @tt: The struct ttm_tt.
> > > > + *
> > > > + * Release handles with associated content or any remaining
> > > > pages
> > > > of
> > > > + * a backed-up struct ttm_tt.
> > > > + */
> > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> > > > +{
> > > > + struct ttm_pool_tt_restore *restore;
> > > > + pgoff_t start_page = 0;
> > > > +
> > > > + WARN_ON(!ttm_tt_is_backed_up(tt));
> > > > +
> > > > + restore = tt->restore;
> > > > +
> > > > + /*
> > > > +  * Unmap and free any uncommitted restore page.
> > > > +  * any tt page-array backup entries already read back
> > > > has
> > > > +  * been cleared already
> > > > +  */
> > > > + if (ttm_pool_restore_valid(restore)) {
> > > > +         dma_addr_t *dma_addr = tt->dma_address ?
> > > > &restore-
> > > > > first_dma : NULL;
> > > > +
> > > > +         ttm_pool_unmap_and_free(restore->pool, restore-
> > > > > alloced_page,
> > > > +                                 dma_addr, restore-
> > > > > page_caching);
> > > > +         restore->restored_pages = 1UL << restore->order;
> > > > + }
> > > > +
> > > > + /*
> > > > +  * If a restore is ongoing, part of the tt pages may
> > > > have
> > > > a
> > > > +  * caching different than writeback.
> > > > +  */
> > > > + if (restore) {
> > > > +         pgoff_t mid = restore-
> > > > > snapshot_alloc.caching_divide - tt->pages;
> > > > +
> > > > +         start_page = restore->alloced_pages;
> > > > +         WARN_ON(mid > start_page);
> > > > +         /* Pages that might be dma-mapped and non-cached
> > > > */
> > > > +         ttm_pool_free_range(restore->pool, tt, tt-
> > > > > caching,
> > > > +                             0, mid);
> > > > +         /* Pages that might be dma-mapped but cached */
> > > > +         ttm_pool_free_range(restore->pool, tt,
> > > > ttm_cached,
> > > > +                             mid, restore-
> > > > >alloced_pages);
> > > > +         kfree(restore);
> > > > +         tt->restore = NULL;
> > > > + }
> > > > +
> > > > + ttm_pool_free_range(NULL, tt, ttm_cached, start_page,
> > > > tt-
> > > > > num_pages);
> > > > +}
> > > > +
> > > > +/**
> > > > + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> > > > + * @pool: The pool used when allocating the struct ttm_tt.
> > > > + * @tt: The struct ttm_tt.
> > > > + * @flags: Flags to govern the backup behaviour.
> > > > + *
> > > > + * Back up or purge a struct ttm_tt. If @purge is true, then
> > > > + * all pages will be freed directly to the system rather than to
> > > > the pool
> > > > + * they were allocated from, making the function behave
> > > > similarly
> > > > to
> > > > + * ttm_pool_free(). If @purge is false the pages will be backed
> > > > up
> > > > instead,
> > > > + * exchanged for handles.
> > > > + * A subsequent call to ttm_pool_restore_and_alloc() will then
> > > > read back the content and
> > > > + * a subsequent call to ttm_pool_drop_backed_up() will drop it.
> > > > + * If backup of a page fails for whatever reason, @ttm will
> > > > still
> > > > be
> > > > + * partially backed up, retaining those pages for which backup
> > > > fails.
> > > > + * In that case, this function can be retried, possibly after
> > > > freeing up
> > > > + * memory resources.
> > > > + *
> > > > + * Return: Number of pages actually backed up or freed, or
> > > > negative
> > > > + * error code on error.
> > > > + */
> > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
> > > > +              const struct ttm_backup_flags *flags)
> > > > +{
> > > > + struct ttm_backup *backup = tt->backup;
> > > > + struct page *page;
> > > > + unsigned long handle;
> > > > + gfp_t alloc_gfp;
> > > > + gfp_t gfp;
> > > > + int ret = 0;
> > > > + pgoff_t shrunken = 0;
> > > > + pgoff_t i, num_pages;
> > > > +
> > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > +         return -EINVAL;
> > > > +
> > > > + if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> > > > +     pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> > > > +         return -EBUSY;
> > > > +
> > > > +#ifdef CONFIG_X86
> > > > + /* Anything returned to the system needs to be cached.
> > > > */
> > > > + if (tt->caching != ttm_cached)
> > > > +         set_pages_array_wb(tt->pages, tt->num_pages);
> > > > +#endif
> > > > +
> > > > + if (tt->dma_address || flags->purge) {
> > > > +         for (i = 0; i < tt->num_pages; i += num_pages) {
> > > > +                 unsigned int order;
> > > > +
> > > > +                 page = tt->pages[i];
> > > > +                 if (unlikely(!page)) {
> > > > +                         num_pages = 1;
> > > > +                         continue;
> > > > +                 }
> > > > +
> > > > +                 order = ttm_pool_page_order(pool, page);
> > > > +                 num_pages = 1UL << order;
> > > > +                 if (tt->dma_address)
> > > > +                         ttm_pool_unmap(pool, tt-
> > > > > dma_address[i],
> > > > +                                        num_pages);
> > > > +                 if (flags->purge) {
> > > > +                         shrunken += num_pages;
> > > > +                         page->private = 0;
> > > > +                         __free_pages(page, order);
> > > > +                         memset(tt->pages + i, 0,
> > > > +                                num_pages * sizeof(*tt-
> > > > > pages));
> > > > +                 }
> > > > +         }
> > > > + }
> > > > +
> > > > + if (flags->purge)
> > > > +         return shrunken;
> > > > +
> > > > + if (pool->use_dma32)
> > > > +         gfp = GFP_DMA32;
> > > > + else
> > > > +         gfp = GFP_HIGHUSER;
> > > > +
> > > > + alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
> > > > __GFP_RETRY_MAYFAIL;
> > > > +
> > > > + for (i = 0; i < tt->num_pages; ++i) {
> > > > +         s64 shandle;
> > > > +
> > > > +         page = tt->pages[i];
> > > > +         if (unlikely(!page))
> > > > +                 continue;
> > > > +
> > > > +         ttm_pool_split_for_swap(pool, page);
> > > > +
> > > > +         shandle = ttm_backup_backup_page(backup, page,
> > > > flags->writeback, i,
> > > > +                                          gfp,
> > > > alloc_gfp);
> > > > +         if (shandle < 0) {
> > > > +                 /* We allow partially shrunken tts */
> > > > +                 ret = shandle;
> > > > +                 break;
> > > > +         }
> > > > +         handle = shandle;
> > > > +         tt->pages[i] =
> > > > ttm_backup_handle_to_page_ptr(handle);
> > > > +         put_page(page);
> > > > +         shrunken++;
> > > > + }
> > > > +
> > > > + return shrunken ? shrunken : ret;
> > > > +}
> > > > +
> > > >   /**
> > > >    * ttm_pool_init - Initialize a pool
> > > >    *
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > index 3baf215eca23..00b7c28f2329 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > @@ -40,6 +40,7 @@
> > > >   #include <drm/drm_cache.h>
> > > >   #include <drm/drm_device.h>
> > > >   #include <drm/drm_util.h>
> > > > +#include <drm/ttm/ttm_backup.h>
> > > >   #include <drm/ttm/ttm_bo.h>
> > > >   #include <drm/ttm/ttm_tt.h>
> > > >
> > > > @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt
> > > > *ttm,
> > > >           ttm->swap_storage = NULL;
> > > >           ttm->sg = bo->sg;
> > > >           ttm->caching = caching;
> > > > + ttm->restore = NULL;
> > > > + ttm->backup = NULL;
> > > >   }
> > > >
> > > >   int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object
> > > > *bo,
> > > > @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
> > > >                   fput(ttm->swap_storage);
> > > >           ttm->swap_storage = NULL;
> > > >
> > > > + if (ttm_tt_is_backed_up(ttm))
> > > > +         ttm_pool_drop_backed_up(ttm);
> > > > + if (ttm->backup) {
> > > > +         ttm_backup_fini(ttm->backup);
> > > > +         ttm->backup = NULL;
> > > > + }
> > > > +
> > > >           if (ttm->pages)
> > > >                   kvfree(ttm->pages);
> > > >           else
> > > > @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
> > > >   }
> > > >   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
> > > >
> > > > +/**
> > > > + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> > > > + * @bdev: The TTM device.
> > > > + * @tt: The struct ttm_tt.
> > > > + * @flags: Flags that govern the backup behaviour.
> > > > + *
> > > > + * Update the page accounting and call ttm_pool_shrink_tt to
> > > > free
> > > > pages
> > > > + * or back them up.
> > > > + *
> > > > + * Return: Number of pages freed or swapped out, or negative
> > > > error
> > > > code on
> > > > + * error.
> > > > + */
> > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > > > +            const struct ttm_backup_flags flags)
> > > > +{
> > > > + long ret;
> > > > +
> > > > + if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> > > > +         return 0;
> > > > +
> > > > + ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> > > > + if (ret > 0) {
> > > > +         tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> > > > +         tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> > > > + }
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > > > +            const struct ttm_operation_ctx *ctx)
> > > > +{
> > > > + int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
> > > > ctx);
> > > > +
> > > > + if (ret)
> > > > +         return ret;
> > > > +
> > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +EXPORT_SYMBOL(ttm_tt_restore);
> > > > +
> > > >   /**
> > > >    * ttm_tt_swapout - swap out tt object
> > > >    *
> > > > @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
> > > >                   goto error;
> > > >
> > > >           ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> > > > + ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > >           if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
> > > >                   ret = ttm_tt_swapin(ttm);
> > > >                   if (unlikely(ret != 0)) {
> > > > diff --git a/include/drm/ttm/ttm_pool.h
> > > > b/include/drm/ttm/ttm_pool.h
> > > > index 160d954a261e..54cd34a6e4c0 100644
> > > > --- a/include/drm/ttm/ttm_pool.h
> > > > +++ b/include/drm/ttm/ttm_pool.h
> > > > @@ -33,6 +33,7 @@
> > > >
> > > >   struct device;
> > > >   struct seq_file;
> > > > +struct ttm_backup_flags;
> > > >   struct ttm_operation_ctx;
> > > >   struct ttm_pool;
> > > >   struct ttm_tt;
> > > > @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
> > > >
> > > >   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file
> > > > *m);
> > > >
> > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> > > > +
> > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
> > > > +              const struct ttm_backup_flags *flags);
> > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > > ttm_tt *tt,
> > > > +                        const struct ttm_operation_ctx
> > > > *ctx);
> > > > +
> > > >   int ttm_pool_mgr_init(unsigned long num_pages);
> > > >   void ttm_pool_mgr_fini(void);
> > > >
> > > > diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> > > > index 991edafdb2dd..c736c01ac2ca 100644
> > > > --- a/include/drm/ttm/ttm_tt.h
> > > > +++ b/include/drm/ttm/ttm_tt.h
> > > > @@ -32,11 +32,13 @@
> > > >   #include <drm/ttm/ttm_caching.h>
> > > >   #include <drm/ttm/ttm_kmap_iter.h>
> > > >
> > > > +struct ttm_backup;
> > > >   struct ttm_device;
> > > >   struct ttm_tt;
> > > >   struct ttm_resource;
> > > >   struct ttm_buffer_object;
> > > >   struct ttm_operation_ctx;
> > > > +struct ttm_pool_tt_restore;
> > > >
> > > >   /**
> > > >    * struct ttm_tt - This is a structure holding the pages,
> > > > caching- and aperture
> > > > @@ -85,17 +87,22 @@ struct ttm_tt {
> > > >            * fault handling abuses the DMA api a bit and
> > > > dma_map_attrs can't be
> > > >            * used to assure pgprot always matches.
> > > >            *
> > > > +  * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
> > > > if the
> > > > +  * struct ttm_tt has been (possibly partially) backed
> > > > up.
> > > > +  *
> > > >            * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT
> > > > USE. This is
> > > >            * set by TTM after ttm_tt_populate() has successfully
> > > > returned, and is
> > > >            * then unset when TTM calls ttm_tt_unpopulate().
> > > > +  *
> > > >            */
> > > >   #define TTM_TT_FLAG_SWAPPED             BIT(0)
> > > >   #define TTM_TT_FLAG_ZERO_ALLOC          BIT(1)
> > > >   #define TTM_TT_FLAG_EXTERNAL            BIT(2)
> > > >   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE   BIT(3)
> > > >   #define TTM_TT_FLAG_DECRYPTED           BIT(4)
> > > > +#define TTM_TT_FLAG_BACKED_UP            BIT(5)
> > > >
> > > > -#define TTM_TT_FLAG_PRIV_POPULATED       BIT(5)
> > > > +#define TTM_TT_FLAG_PRIV_POPULATED       BIT(6)
> > > >           uint32_t page_flags;
> > > >           /** @num_pages: Number of pages in the page array. */
> > > >           uint32_t num_pages;
> > > > @@ -105,11 +112,20 @@ struct ttm_tt {
> > > >           dma_addr_t *dma_address;
> > > >           /** @swap_storage: Pointer to shmem struct file for swap
> > > > storage. */
> > > >           struct file *swap_storage;
> > > > + /**
> > > > +  * @backup: Pointer to backup struct for backed up tts.
> > > > +  * Could be unified with @swap_storage. Meanwhile, the
> > > > driver's
> > > > +  * ttm_tt_create() callback is responsible for assigning
> > > > +  * this field.
> > > > +  */
> > > > + struct ttm_backup *backup;
> > > >           /**
> > > >            * @caching: The current caching state of the pages, see
> > > > enum
> > > >            * ttm_caching.
> > > >            */
> > > >           enum ttm_caching caching;
> > > > + /** @restore: Partial restoration from backup state. TTM
> > > > private */
> > > > + struct ttm_pool_tt_restore *restore;
> > > >   };
> > > >
> > > >   /**
> > > > @@ -129,9 +145,38 @@ static inline bool
> > > > ttm_tt_is_populated(struct
> > > > ttm_tt *tt)
> > > >           return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
> > > >   }
> > > >
> > > > +/**
> > > > + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or
> > > > backed up
> > > > + * @tt: The struct ttm_tt.
> > > > + *
> > > > + * Return: true if swapped or backed up, false otherwise.
> > > > + */
> > > >   static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
> > > >   {
> > > > - return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> > > > + return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
> > > > TTM_TT_FLAG_BACKED_UP);
> > > > +}
> > > > +
> > > > +/**
> > > > + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> > > > + * @tt: The struct ttm_tt.
> > > > + *
> > > > + * Return: true if swapped or backed up, false otherwise.
> > > > + */
> > > > +static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
> > > > +{
> > > > + return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> > > > +}
> > > > +
> > > > +/**
> > > > + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
> > > > + * @tt: The struct ttm_tt.
> > > > + *
> > > > + * Drivers can use this functionto clear the backed-up status,
> > > > + * for example before destroying or re-validating a purged tt.
> > > > + */
> > > > +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
> > > > +{
> > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > >   }
> > > >
> > > >   /**
> > > > @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long
> > > > num_pages,
> > > > unsigned long num_dma32_pages);
> > > >   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
> > > > ttm_kmap_iter_tt *iter_tt,
> > > >                                               struct ttm_tt *tt);
> > > >   unsigned long ttm_tt_pages_limit(void);
> > > > +
> > > > +/**
> > > > + * struct ttm_backup_flags - Flags to govern backup behaviour.
> > > > + * @purge: Free pages without backing up. Bypass pools.
> > > > + * @writeback: Attempt to copy contents directly to swap space,
> > > > even
> > > > + * if that means blocking on writes to external memory.
> > > > + */
> > > > +struct ttm_backup_flags {
> > > > + u32 purge : 1;
> > > > + u32 writeback : 1;
> > > > +};
> > > > +
> > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
> > > > +            const struct ttm_backup_flags flags);
> > > > +
> > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
> > > > +            const struct ttm_operation_ctx *ctx);
> > > > +
> > > >   #if IS_ENABLED(CONFIG_AGP)
> > > >   #include <linux/agp_backend.h>
> > > >
> > >
> >
>

Re: RESEND Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Christian König]

Yeah, it's sad but I probably won't find time for a deeper review anytime soon.

So feel free to add my Acked-by as well.

Christian.

Am 05.03.25 um 04:01 schrieb Dave Airlie:
> I've looked over the two patches mentioned here, I think they have
> seen enough time and we need to unblock,
>
> Please add and merge them:
> Acked-by: Dave Airlie <airlied@redhat.com>
>
> Dave.
>
> On Tue, 25 Feb 2025 at 18:44, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
>> Hi, Christian,
>>
>> Ping? I'd really want to get this in before -rc6
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Tue, 2025-02-18 at 16:40 +0100, Thomas Hellström wrote:
>>> Hi, Christian,
>>>
>>> On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
>>>> Am 30.01.25 um 11:13 schrieb Thomas Hellström:
>>>>> Provide a helper to shrink ttm_tt page-vectors on a per-page
>>>>> basis. A ttm_backup backend could then in theory get away with
>>>>> allocating a single temporary page for each struct ttm_tt.
>>>>>
>>>>> This is accomplished by splitting larger pages before trying to
>>>>> back them up.
>>>>>
>>>>> In the future we could allow ttm_backup to handle backing up
>>>>> large pages as well, but currently there's no benefit in
>>>>> doing that, since the shmem backup backend would have to
>>>>> split those anyway to avoid allocating too much temporary
>>>>> memory, and if the backend instead inserts pages into the
>>>>> swap-cache, those are split on reclaim by the core.
>>>>>
>>>>> Due to potential backup- and recover errors, allow partially
>>>>> swapped
>>>>> out struct ttm_tt's, although mark them as swapped out stopping
>>>>> them
>>>>> from being swapped out a second time. More details in the
>>>>> ttm_pool.c
>>>>> DOC section.
>>>>>
>>>>> v2:
>>>>> - A couple of cleanups and error fixes in ttm_pool_back_up_tt.
>>>>> - s/back_up/backup/
>>>>> - Add a writeback parameter to the exported interface.
>>>>> v8:
>>>>> - Use a struct for flags for readability (Matt Brost)
>>>>> - Address misc other review comments (Matt Brost)
>>>>> v9:
>>>>> - Update the kerneldoc for the ttm_tt::backup field.
>>>>> v10:
>>>>> - Rebase.
>>>>> v13:
>>>>> - Rebase on ttm_backup interface change. Update kerneldoc.
>>>>> - Rebase and adjust ttm_tt_is_swapped().
>>>>> v15:
>>>>> - Rebase on ttm_backup return value change.
>>>>> - Rebase on previous restructuring of ttm_pool_alloc()
>>>>> - Rework the ttm_pool backup interface (Christian König)
>>>>> - Remove cond_resched() (Christian König)
>>>>> - Get rid of the need to allocate an intermediate page array
>>>>>    when restoring a multi-order page (Christian König)
>>>>> - Update documentation.
>>>>>
>>>>> Cc: Christian König <christian.koenig@amd.com>
>>>>> Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
>>>>> Cc: Matthew Brost <matthew.brost@intel.com>
>>>>> Cc: <dri-devel@lists.freedesktop.org>
>>>>> Signed-off-by: Thomas Hellström
>>>>> <thomas.hellstrom@linux.intel.com>
>>>>> Reviewed-by: Matthew Brost <matthew.brost@intel.com>
>>>> I've tried to wrap my head around all of this like twenty times in
>>>> the
>>>> last three month, but was always interrupted at some point.
>>>>
>>>> Feel free to add Acked-by: Christian Koenig
>>>> <christian.koenig@amd.com>.
>>>>
>>>> Sorry,
>>>> Christian.
>>> Thanks a lot for all reviewing and comments so far. There are two TTM
>>> patches left in the series that don't have an ack by you:
>>>
>>> https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
>>> and
>>>
>>> https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17
>>>
>>> None of them particularly big considering the amount of doc text.
>>>
>>> It'd be great if those could have an ack as well so we could finally
>>> merge this series.
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>>
>>>>> ---
>>>>>   drivers/gpu/drm/ttm/ttm_pool.c | 554
>>>>> +++++++++++++++++++++++++++++----
>>>>>   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
>>>>>   include/drm/ttm/ttm_pool.h     |   8 +
>>>>>   include/drm/ttm/ttm_tt.h       |  67 +++-
>>>>>   4 files changed, 629 insertions(+), 54 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
>>>>> b/drivers/gpu/drm/ttm/ttm_pool.c
>>>>> index c9eba76d5143..ffb7abf52bab 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_pool.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
>>>>> @@ -41,6 +41,7 @@
>>>>>   #include <asm/set_memory.h>
>>>>>   #endif
>>>>>
>>>>> +#include <drm/ttm/ttm_backup.h>
>>>>>   #include <drm/ttm/ttm_pool.h>
>>>>>   #include <drm/ttm/ttm_tt.h>
>>>>>   #include <drm/ttm/ttm_bo.h>
>>>>> @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
>>>>>           enum ttm_caching tt_caching;
>>>>>   };
>>>>>
>>>>> +/**
>>>>> + * struct ttm_pool_tt_restore - State representing restore from
>>>>> backup
>>>>> + * @pool: The pool used for page allocation while restoring.
>>>>> + * @snapshot_alloc: A snapshot of the most recent struct
>>>>> ttm_pool_alloc_state.
>>>>> + * @alloced_page: Pointer to the page most recently allocated
>>>>> from
>>>>> a pool or system.
>>>>> + * @first_dma: The dma address corresponding to @alloced_page if
>>>>> dma_mapping
>>>>> + * is requested.
>>>>> + * @alloced_pages: The number of allocated pages present in the
>>>>> struct ttm_tt
>>>>> + * page vector from this restore session.
>>>>> + * @restored_pages: The number of 4K pages restored for
>>>>> @alloced_page (which
>>>>> + * is typically a multi-order page).
>>>>> + * @page_caching: The struct ttm_tt requested caching
>>>>> + * @order: The order of @alloced_page.
>>>>> + *
>>>>> + * Recovery from backup might fail when we've recovered less
>>>>> than
>>>>> the
>>>>> + * full ttm_tt. In order not to loose any data (yet), keep
>>>>> information
>>>>> + * around that allows us to restart a failed ttm backup
>>>>> recovery.
>>>>> + */
>>>>> +struct ttm_pool_tt_restore {
>>>>> + struct ttm_pool *pool;
>>>>> + struct ttm_pool_alloc_state snapshot_alloc;
>>>>> + struct page *alloced_page;
>>>>> + dma_addr_t first_dma;
>>>>> + pgoff_t alloced_pages;
>>>>> + pgoff_t restored_pages;
>>>>> + enum ttm_caching page_caching;
>>>>> + unsigned int order;
>>>>> +};
>>>>> +
>>>>>   static unsigned long page_pool_size;
>>>>>
>>>>>   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
>>>>> WC/UC/DMA pool");
>>>>> @@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct
>>>>> ttm_pool_alloc_state *alloc)
>>>>>           return 0;
>>>>>   }
>>>>>
>>>>> -/* Map pages of 1 << order size and fill the DMA address array
>>>>> */
>>>>> +/* DMA Map pages of 1 << order size and return the resulting
>>>>> dma_address. */
>>>>>   static int ttm_pool_map(struct ttm_pool *pool, unsigned int
>>>>> order,
>>>>> -                 struct page *p, dma_addr_t **dma_addr)
>>>>> +                 struct page *p, dma_addr_t *dma_addr)
>>>>>   {
>>>>>           dma_addr_t addr;
>>>>> - unsigned int i;
>>>>>
>>>>>           if (pool->use_dma_alloc) {
>>>>>                   struct ttm_pool_dma *dma = (void *)p->private;
>>>>> @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool
>>>>> *pool,
>>>>> unsigned int order,
>>>>>                           return -EFAULT;
>>>>>           }
>>>>>
>>>>> - for (i = 1 << order; i ; --i) {
>>>>> -         *(*dma_addr)++ = addr;
>>>>> -         addr += PAGE_SIZE;
>>>>> - }
>>>>> + *dma_addr = addr;
>>>>>
>>>>>           return 0;
>>>>>   }
>>>>> @@ -371,6 +397,190 @@ static unsigned int
>>>>> ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>>>>>           return p->private;
>>>>>   }
>>>>>
>>>>> +/*
>>>>> + * Split larger pages so that we can free each PAGE_SIZE page as
>>>>> soon
>>>>> + * as it has been backed up, in order to avoid memory pressure
>>>>> during
>>>>> + * reclaim.
>>>>> + */
>>>>> +static void ttm_pool_split_for_swap(struct ttm_pool *pool,
>>>>> struct
>>>>> page *p)
>>>>> +{
>>>>> + unsigned int order = ttm_pool_page_order(pool, p);
>>>>> + pgoff_t nr;
>>>>> +
>>>>> + if (!order)
>>>>> +         return;
>>>>> +
>>>>> + split_page(p, order);
>>>>> + nr = 1UL << order;
>>>>> + while (nr--)
>>>>> +         (p++)->private = 0;
>>>>> +}
>>>>> +
>>>>> +/**
>>>>> + * DOC: Partial backup and restoration of a struct ttm_tt.
>>>>> + *
>>>>> + * Swapout using ttm_backup_backup_page() and swapin using
>>>>> + * ttm_backup_copy_page() may fail.
>>>>> + * The former most likely due to lack of swap-space or memory,
>>>>> the
>>>>> latter due
>>>>> + * to lack of memory or because of signal interruption during
>>>>> waits.
>>>>> + *
>>>>> + * Backup failure is easily handled by using a ttm_tt pages
>>>>> vector
>>>>> that holds
>>>>> + * both backup handles and page pointers. This has to be taken
>>>>> into account when
>>>>> + * restoring such a ttm_tt from backup, and when freeing it
>>>>> while
>>>>> backed up.
>>>>> + * When restoring, for simplicity, new pages are actually
>>>>> allocated from the
>>>>> + * pool and the contents of any old pages are copied in and then
>>>>> the old pages
>>>>> + * are released.
>>>>> + *
>>>>> + * For restoration failures, the struct ttm_pool_tt_restore
>>>>> holds
>>>>> sufficient state
>>>>> + * to be able to resume an interrupted restore, and that
>>>>> structure
>>>>> is freed once
>>>>> + * the restoration is complete. If the struct ttm_tt is
>>>>> destroyed
>>>>> while there
>>>>> + * is a valid struct ttm_pool_tt_restore attached, that is also
>>>>> properly taken
>>>>> + * care of.
>>>>> + */
>>>>> +
>>>>> +/* Is restore ongoing for the currently allocated page? */
>>>>> +static bool ttm_pool_restore_valid(const struct
>>>>> ttm_pool_tt_restore *restore)
>>>>> +{
>>>>> + return restore && restore->restored_pages < (1 <<
>>>>> restore-
>>>>>> order);
>>>>> +}
>>>>> +
>>>>> +/* DMA unmap and free a multi-order page, either to the relevant
>>>>> pool or to system. */
>>>>> +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool *pool,
>>>>> struct page *page,
>>>>> +                                const dma_addr_t
>>>>> *dma_addr,
>>>>> enum ttm_caching caching)
>>>>> +{
>>>>> + struct ttm_pool_type *pt = NULL;
>>>>> + unsigned int order;
>>>>> + pgoff_t nr;
>>>>> +
>>>>> + if (pool) {
>>>>> +         order = ttm_pool_page_order(pool, page);
>>>>> +         nr = (1UL << order);
>>>>> +         if (dma_addr)
>>>>> +                 ttm_pool_unmap(pool, *dma_addr, nr);
>>>>> +
>>>>> +         pt = ttm_pool_select_type(pool, caching, order);
>>>>> + } else {
>>>>> +         order = page->private;
>>>>> +         nr = (1UL << order);
>>>>> + }
>>>>> +
>>>>> + if (pt)
>>>>> +         ttm_pool_type_give(pt, page);
>>>>> + else
>>>>> +         ttm_pool_free_page(pool, caching, order, page);
>>>>> +
>>>>> + return nr;
>>>>> +}
>>>>> +
>>>>> +/* Populate the page-array using the most recent allocated
>>>>> multi-
>>>>> order page. */
>>>>> +static void ttm_pool_allocated_page_commit(struct page
>>>>> *allocated,
>>>>> +                                    dma_addr_t first_dma,
>>>>> +                                    struct
>>>>> ttm_pool_alloc_state *alloc,
>>>>> +                                    pgoff_t nr)
>>>>> +{
>>>>> + pgoff_t i;
>>>>> +
>>>>> + for (i = 0; i < nr; ++i)
>>>>> +         *alloc->pages++ = allocated++;
>>>>> +
>>>>> + alloc->remaining_pages -= nr;
>>>>> +
>>>>> + if (!alloc->dma_addr)
>>>>> +         return;
>>>>> +
>>>>> + for (i = 0; i < nr; ++i) {
>>>>> +         *alloc->dma_addr++ = first_dma;
>>>>> +         first_dma += PAGE_SIZE;
>>>>> + }
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * When restoring, restore backed-up content to the newly
>>>>> allocated page and
>>>>> + * if successful, populate the page-table and dma-address
>>>>> arrays.
>>>>> + */
>>>>> +static int ttm_pool_restore_commit(struct ttm_pool_tt_restore
>>>>> *restore,
>>>>> +                            struct ttm_backup *backup,
>>>>> +                            const struct
>>>>> ttm_operation_ctx
>>>>> *ctx,
>>>>> +                            struct ttm_pool_alloc_state
>>>>> *alloc)
>>>>> +
>>>>> +{
>>>>> + pgoff_t i, nr = 1UL << restore->order;
>>>>> + struct page **first_page = alloc->pages;
>>>>> + struct page *p;
>>>>> + int ret = 0;
>>>>> +
>>>>> + for (i = restore->restored_pages; i < nr; ++i) {
>>>>> +         p = first_page[i];
>>>>> +         if (ttm_backup_page_ptr_is_handle(p)) {
>>>>> +                 unsigned long handle =
>>>>> ttm_backup_page_ptr_to_handle(p);
>>>>> +
>>>>> +                 if (handle == 0) {
>>>>> +                         restore->restored_pages++;
>>>>> +                         continue;
>>>>> +                 }
>>>>> +
>>>>> +                 ret = ttm_backup_copy_page(backup,
>>>>> restore->alloced_page + i,
>>>>> +                                            handle, ctx-
>>>>>> interruptible);
>>>>> +                 if (ret)
>>>>> +                         break;
>>>>> +
>>>>> +                 ttm_backup_drop(backup, handle);
>>>>> +         } else if (p) {
>>>>> +                 /*
>>>>> +                  * We could probably avoid splitting the
>>>>> old page
>>>>> +                  * using clever logic, but ATM we don't
>>>>> care, as
>>>>> +                  * we prioritize releasing memory ASAP.
>>>>> Note that
>>>>> +                  * here, the old retained page is always
>>>>> write-back
>>>>> +                  * cached.
>>>>> +                  */
>>>>> +                 ttm_pool_split_for_swap(restore->pool,
>>>>> p);
>>>>> +                 copy_highpage(restore->alloced_page + i,
>>>>> p);
>>>>> +                 __free_pages(p, 0);
>>>>> +         }
>>>>> +
>>>>> +         restore->restored_pages++;
>>>>> +         first_page[i] =
>>>>> ttm_backup_handle_to_page_ptr(0);
>>>>> + }
>>>>> +
>>>>> + if (ret) {
>>>>> +         if (!restore->restored_pages) {
>>>>> +                 dma_addr_t *dma_addr = alloc->dma_addr ?
>>>>> &restore->first_dma : NULL;
>>>>> +
>>>>> +                 ttm_pool_unmap_and_free(restore->pool,
>>>>> restore->alloced_page,
>>>>> +                                         dma_addr,
>>>>> restore-
>>>>>> page_caching);
>>>>> +                 restore->restored_pages = nr;
>>>>> +         }
>>>>> +         return ret;
>>>>> + }
>>>>> +
>>>>> + ttm_pool_allocated_page_commit(restore->alloced_page,
>>>>> restore->first_dma,
>>>>> +                                alloc, nr);
>>>>> + if (restore->page_caching == alloc->tt_caching ||
>>>>> PageHighMem(restore->alloced_page))
>>>>> +         alloc->caching_divide = alloc->pages;
>>>>> + restore->snapshot_alloc = *alloc;
>>>>> + restore->alloced_pages += nr;
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> +/* If restoring, save information needed for
>>>>> ttm_pool_restore_commit(). */
>>>>> +static void
>>>>> +ttm_pool_page_allocated_restore(struct ttm_pool *pool, unsigned
>>>>> int order,
>>>>> +                         struct page *p,
>>>>> +                         enum ttm_caching page_caching,
>>>>> +                         dma_addr_t first_dma,
>>>>> +                         struct ttm_pool_tt_restore
>>>>> *restore,
>>>>> +                         const struct
>>>>> ttm_pool_alloc_state
>>>>> *alloc)
>>>>> +{
>>>>> + restore->pool = pool;
>>>>> + restore->order = order;
>>>>> + restore->restored_pages = 0;
>>>>> + restore->page_caching = page_caching;
>>>>> + restore->first_dma = first_dma;
>>>>> + restore->alloced_page = p;
>>>>> + restore->snapshot_alloc = *alloc;
>>>>> +}
>>>>> +
>>>>>   /*
>>>>>    * Called when we got a page, either from a pool or newly
>>>>> allocated.
>>>>>    * if needed, dma map the page and populate the dma address
>>>>> array.
>>>>> @@ -380,10 +590,11 @@ static unsigned int
>>>>> ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>>>>>    */
>>>>>   static int ttm_pool_page_allocated(struct ttm_pool *pool,
>>>>> unsigned int order,
>>>>>                                      struct page *p, enum
>>>>> ttm_caching page_caching,
>>>>> -                            struct ttm_pool_alloc_state
>>>>> *alloc)
>>>>> +                            struct ttm_pool_alloc_state
>>>>> *alloc,
>>>>> +                            struct ttm_pool_tt_restore
>>>>> *restore)
>>>>>   {
>>>>> - pgoff_t i, nr = 1UL << order;
>>>>>           bool caching_consistent;
>>>>> + dma_addr_t first_dma;
>>>>>           int r = 0;
>>>>>
>>>>>           caching_consistent = (page_caching == alloc->tt_caching)
>>>>>>> PageHighMem(p);
>>>>> @@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct
>>>>> ttm_pool *pool, unsigned int order,
>>>>>           }
>>>>>
>>>>>           if (alloc->dma_addr) {
>>>>> -         r = ttm_pool_map(pool, order, p, &alloc-
>>>>>> dma_addr);
>>>>> +         r = ttm_pool_map(pool, order, p, &first_dma);
>>>>>                   if (r)
>>>>>                           return r;
>>>>>           }
>>>>>
>>>>> - alloc->remaining_pages -= nr;
>>>>> - for (i = 0; i < nr; ++i)
>>>>> -         *alloc->pages++ = p++;
>>>>> + if (restore) {
>>>>> +         ttm_pool_page_allocated_restore(pool, order, p,
>>>>> page_caching,
>>>>> +                                         first_dma,
>>>>> restore, alloc);
>>>>> + } else {
>>>>> +         ttm_pool_allocated_page_commit(p, first_dma,
>>>>> alloc, 1UL << order);
>>>>>
>>>>> - if (caching_consistent)
>>>>> -         alloc->caching_divide = alloc->pages;
>>>>> +         if (caching_consistent)
>>>>> +                 alloc->caching_divide = alloc->pages;
>>>>> + }
>>>>>
>>>>>           return 0;
>>>>>   }
>>>>> @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct
>>>>> ttm_pool *pool, struct ttm_tt *tt,
>>>>>                                   pgoff_t start_page, pgoff_t
>>>>> end_page)
>>>>>   {
>>>>>           struct page **pages = &tt->pages[start_page];
>>>>> - unsigned int order;
>>>>> + struct ttm_backup *backup = tt->backup;
>>>>>           pgoff_t i, nr;
>>>>>
>>>>>           for (i = start_page; i < end_page; i += nr, pages += nr)
>>>>> {
>>>>> -         struct ttm_pool_type *pt = NULL;
>>>>> +         struct page *p = *pages;
>>>>>
>>>>> -         order = ttm_pool_page_order(pool, *pages);
>>>>> -         nr = (1UL << order);
>>>>> -         if (tt->dma_address)
>>>>> -                 ttm_pool_unmap(pool, tt->dma_address[i],
>>>>> nr);
>>>>> +         nr = 1;
>>>>> +         if (ttm_backup_page_ptr_is_handle(p)) {
>>>>> +                 unsigned long handle =
>>>>> ttm_backup_page_ptr_to_handle(p);
>>>>>
>>>>> -         pt = ttm_pool_select_type(pool, caching, order);
>>>>> -         if (pt)
>>>>> -                 ttm_pool_type_give(pt, *pages);
>>>>> -         else
>>>>> -                 ttm_pool_free_page(pool, caching, order,
>>>>> *pages);
>>>>> +                 if (handle != 0)
>>>>> +                         ttm_backup_drop(backup, handle);
>>>>> +         } else if (p) {
>>>>> +                 dma_addr_t *dma_addr = tt->dma_address ?
>>>>> +                         tt->dma_address + i : NULL;
>>>>> +
>>>>> +                 nr = ttm_pool_unmap_and_free(pool, p,
>>>>> dma_addr, caching);
>>>>> +         }
>>>>>           }
>>>>>   }
>>>>>
>>>>> @@ -467,22 +683,11 @@ static unsigned int
>>>>> ttm_pool_alloc_find_order(unsigned int highest,
>>>>>           return min_t(unsigned int, highest, __fls(alloc-
>>>>>> remaining_pages));
>>>>>   }
>>>>>
>>>>> -/**
>>>>> - * ttm_pool_alloc - Fill a ttm_tt object
>>>>> - *
>>>>> - * @pool: ttm_pool to use
>>>>> - * @tt: ttm_tt object to fill
>>>>> - * @ctx: operation context
>>>>> - *
>>>>> - * Fill the ttm_tt object with pages and also make sure to DMA
>>>>> map
>>>>> them when
>>>>> - * necessary.
>>>>> - *
>>>>> - * Returns: 0 on successe, negative error code otherwise.
>>>>> - */
>>>>> -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>>>>> -            struct ttm_operation_ctx *ctx)
>>>>> +static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
>>>>> *tt,
>>>>> +                     const struct ttm_operation_ctx *ctx,
>>>>> +                     struct ttm_pool_alloc_state *alloc,
>>>>> +                     struct ttm_pool_tt_restore *restore)
>>>>>   {
>>>>> - struct ttm_pool_alloc_state alloc;
>>>>>           enum ttm_caching page_caching;
>>>>>           gfp_t gfp_flags = GFP_USER;
>>>>>           pgoff_t caching_divide;
>>>>> @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool,
>>>>> struct ttm_tt *tt,
>>>>>           struct page *p;
>>>>>           int r;
>>>>>
>>>>> - ttm_pool_alloc_state_init(tt, &alloc);
>>>>> -
>>>>> - WARN_ON(!alloc.remaining_pages ||
>>>>> ttm_tt_is_populated(tt));
>>>>> - WARN_ON(alloc.dma_addr && !pool->dev);
>>>>> + WARN_ON(!alloc->remaining_pages ||
>>>>> ttm_tt_is_populated(tt));
>>>>> + WARN_ON(alloc->dma_addr && !pool->dev);
>>>>>
>>>>>           if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
>>>>>                   gfp_flags |= __GFP_ZERO;
>>>>> @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool,
>>>>> struct ttm_tt *tt,
>>>>>
>>>>>           page_caching = tt->caching;
>>>>>           allow_pools = true;
>>>>> - for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
>>>>> &alloc);
>>>>> -      alloc.remaining_pages;
>>>>> -      order = ttm_pool_alloc_find_order(order, &alloc)) {
>>>>> + for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
>>>>> alloc);
>>>>> +      alloc->remaining_pages;
>>>>> +      order = ttm_pool_alloc_find_order(order, alloc)) {
>>>>>                   struct ttm_pool_type *pt;
>>>>>
>>>>>                   /* First, try to allocate a page from a pool if
>>>>> one exists. */
>>>>> @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool,
>>>>> struct ttm_tt *tt,
>>>>>                           r = -ENOMEM;
>>>>>                           goto error_free_all;
>>>>>                   }
>>>>> -         r = ttm_pool_page_allocated(pool, order, p,
>>>>> page_caching, &alloc);
>>>>> +         r = ttm_pool_page_allocated(pool, order, p,
>>>>> page_caching, alloc,
>>>>> +                                     restore);
>>>>>                   if (r)
>>>>>                           goto error_free_page;
>>>>> +
>>>>> +         if (ttm_pool_restore_valid(restore)) {
>>>>> +                 r = ttm_pool_restore_commit(restore, tt-
>>>>>> backup, ctx, alloc);
>>>>> +                 if (r)
>>>>> +                         goto error_free_all;
>>>>> +         }
>>>>>           }
>>>>>
>>>>> - r = ttm_pool_apply_caching(&alloc);
>>>>> + r = ttm_pool_apply_caching(alloc);
>>>>>           if (r)
>>>>>                   goto error_free_all;
>>>>>
>>>>> + kfree(tt->restore);
>>>>> + tt->restore = NULL;
>>>>> +
>>>>>           return 0;
>>>>>
>>>>>   error_free_page:
>>>>>           ttm_pool_free_page(pool, page_caching, order, p);
>>>>>
>>>>>   error_free_all:
>>>>> - caching_divide = alloc.caching_divide - tt->pages;
>>>>> + if (tt->restore)
>>>>> +         return r;
>>>>> +
>>>>> + caching_divide = alloc->caching_divide - tt->pages;
>>>>>           ttm_pool_free_range(pool, tt, tt->caching, 0,
>>>>> caching_divide);
>>>>>           ttm_pool_free_range(pool, tt, ttm_cached,
>>>>> caching_divide,
>>>>> -                     tt->num_pages -
>>>>> alloc.remaining_pages);
>>>>> +                     tt->num_pages - alloc-
>>>>>> remaining_pages);
>>>>>           return r;
>>>>>   }
>>>>> +
>>>>> +/**
>>>>> + * ttm_pool_alloc - Fill a ttm_tt object
>>>>> + *
>>>>> + * @pool: ttm_pool to use
>>>>> + * @tt: ttm_tt object to fill
>>>>> + * @ctx: operation context
>>>>> + *
>>>>> + * Fill the ttm_tt object with pages and also make sure to DMA
>>>>> map
>>>>> them when
>>>>> + * necessary.
>>>>> + *
>>>>> + * Returns: 0 on successe, negative error code otherwise.
>>>>> + */
>>>>> +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>>>>> +            struct ttm_operation_ctx *ctx)
>>>>> +{
>>>>> + struct ttm_pool_alloc_state alloc;
>>>>> +
>>>>> + if (WARN_ON(ttm_tt_is_backed_up(tt)))
>>>>> +         return -EINVAL;
>>>>> +
>>>>> + ttm_pool_alloc_state_init(tt, &alloc);
>>>>> +
>>>>> + return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
>>>>> +}
>>>>>   EXPORT_SYMBOL(ttm_pool_alloc);
>>>>>
>>>>> +/**
>>>>> + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
>>>>> previously backed-up
>>>>> + * content.
>>>>> + *
>>>>> + * @pool: ttm_pool to use
>>>>> + * @tt: ttm_tt object to fill
>>>>> + * @ctx: operation context
>>>>> + *
>>>>> + * Fill the ttm_tt object with pages and also make sure to DMA
>>>>> map
>>>>> them when
>>>>> + * necessary. Read in backed-up content.
>>>>> + *
>>>>> + * Returns: 0 on successe, negative error code otherwise.
>>>>> + */
>>>>> +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
>>>>> ttm_tt *tt,
>>>>> +                        const struct ttm_operation_ctx
>>>>> *ctx)
>>>>> +{
>>>>> + struct ttm_pool_alloc_state alloc;
>>>>> +
>>>>> + if (WARN_ON(!ttm_tt_is_backed_up(tt)))
>>>>> +         return -EINVAL;
>>>>> +
>>>>> + if (!tt->restore) {
>>>>> +         gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
>>>>> +
>>>>> +         ttm_pool_alloc_state_init(tt, &alloc);
>>>>> +         if (ctx->gfp_retry_mayfail)
>>>>> +                 gfp |= __GFP_RETRY_MAYFAIL;
>>>>> +
>>>>> +         tt->restore = kzalloc(sizeof(*tt->restore),
>>>>> gfp);
>>>>> +         if (!tt->restore)
>>>>> +                 return -ENOMEM;
>>>>> +
>>>>> +         tt->restore->snapshot_alloc = alloc;
>>>>> +         tt->restore->pool = pool;
>>>>> +         tt->restore->restored_pages = 1;
>>>>> + } else {
>>>>> +         struct ttm_pool_tt_restore *restore = tt-
>>>>>> restore;
>>>>> +         int ret;
>>>>> +
>>>>> +         alloc = restore->snapshot_alloc;
>>>>> +         if (ttm_pool_restore_valid(tt->restore)) {
>>>>> +                 ret = ttm_pool_restore_commit(restore,
>>>>> tt-
>>>>>> backup, ctx, &alloc);
>>>>> +                 if (ret)
>>>>> +                         return ret;
>>>>> +         }
>>>>> +         if (!alloc.remaining_pages)
>>>>> +                 return 0;
>>>>> + }
>>>>> +
>>>>> + return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
>>>>>> restore);
>>>>> +}
>>>>> +
>>>>>   /**
>>>>>    * ttm_pool_free - Free the backing pages from a ttm_tt object
>>>>>    *
>>>>> @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool,
>>>>> struct ttm_tt *tt)
>>>>>   }
>>>>>   EXPORT_SYMBOL(ttm_pool_free);
>>>>>
>>>>> +/**
>>>>> + * ttm_pool_drop_backed_up() - Release content of a swapped-out
>>>>> struct ttm_tt
>>>>> + * @tt: The struct ttm_tt.
>>>>> + *
>>>>> + * Release handles with associated content or any remaining
>>>>> pages
>>>>> of
>>>>> + * a backed-up struct ttm_tt.
>>>>> + */
>>>>> +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
>>>>> +{
>>>>> + struct ttm_pool_tt_restore *restore;
>>>>> + pgoff_t start_page = 0;
>>>>> +
>>>>> + WARN_ON(!ttm_tt_is_backed_up(tt));
>>>>> +
>>>>> + restore = tt->restore;
>>>>> +
>>>>> + /*
>>>>> +  * Unmap and free any uncommitted restore page.
>>>>> +  * any tt page-array backup entries already read back
>>>>> has
>>>>> +  * been cleared already
>>>>> +  */
>>>>> + if (ttm_pool_restore_valid(restore)) {
>>>>> +         dma_addr_t *dma_addr = tt->dma_address ?
>>>>> &restore-
>>>>>> first_dma : NULL;
>>>>> +
>>>>> +         ttm_pool_unmap_and_free(restore->pool, restore-
>>>>>> alloced_page,
>>>>> +                                 dma_addr, restore-
>>>>>> page_caching);
>>>>> +         restore->restored_pages = 1UL << restore->order;
>>>>> + }
>>>>> +
>>>>> + /*
>>>>> +  * If a restore is ongoing, part of the tt pages may
>>>>> have
>>>>> a
>>>>> +  * caching different than writeback.
>>>>> +  */
>>>>> + if (restore) {
>>>>> +         pgoff_t mid = restore-
>>>>>> snapshot_alloc.caching_divide - tt->pages;
>>>>> +
>>>>> +         start_page = restore->alloced_pages;
>>>>> +         WARN_ON(mid > start_page);
>>>>> +         /* Pages that might be dma-mapped and non-cached
>>>>> */
>>>>> +         ttm_pool_free_range(restore->pool, tt, tt-
>>>>>> caching,
>>>>> +                             0, mid);
>>>>> +         /* Pages that might be dma-mapped but cached */
>>>>> +         ttm_pool_free_range(restore->pool, tt,
>>>>> ttm_cached,
>>>>> +                             mid, restore-
>>>>>> alloced_pages);
>>>>> +         kfree(restore);
>>>>> +         tt->restore = NULL;
>>>>> + }
>>>>> +
>>>>> + ttm_pool_free_range(NULL, tt, ttm_cached, start_page,
>>>>> tt-
>>>>>> num_pages);
>>>>> +}
>>>>> +
>>>>> +/**
>>>>> + * ttm_pool_backup() - Back up or purge a struct ttm_tt
>>>>> + * @pool: The pool used when allocating the struct ttm_tt.
>>>>> + * @tt: The struct ttm_tt.
>>>>> + * @flags: Flags to govern the backup behaviour.
>>>>> + *
>>>>> + * Back up or purge a struct ttm_tt. If @purge is true, then
>>>>> + * all pages will be freed directly to the system rather than to
>>>>> the pool
>>>>> + * they were allocated from, making the function behave
>>>>> similarly
>>>>> to
>>>>> + * ttm_pool_free(). If @purge is false the pages will be backed
>>>>> up
>>>>> instead,
>>>>> + * exchanged for handles.
>>>>> + * A subsequent call to ttm_pool_restore_and_alloc() will then
>>>>> read back the content and
>>>>> + * a subsequent call to ttm_pool_drop_backed_up() will drop it.
>>>>> + * If backup of a page fails for whatever reason, @ttm will
>>>>> still
>>>>> be
>>>>> + * partially backed up, retaining those pages for which backup
>>>>> fails.
>>>>> + * In that case, this function can be retried, possibly after
>>>>> freeing up
>>>>> + * memory resources.
>>>>> + *
>>>>> + * Return: Number of pages actually backed up or freed, or
>>>>> negative
>>>>> + * error code on error.
>>>>> + */
>>>>> +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
>>>>> +              const struct ttm_backup_flags *flags)
>>>>> +{
>>>>> + struct ttm_backup *backup = tt->backup;
>>>>> + struct page *page;
>>>>> + unsigned long handle;
>>>>> + gfp_t alloc_gfp;
>>>>> + gfp_t gfp;
>>>>> + int ret = 0;
>>>>> + pgoff_t shrunken = 0;
>>>>> + pgoff_t i, num_pages;
>>>>> +
>>>>> + if (WARN_ON(ttm_tt_is_backed_up(tt)))
>>>>> +         return -EINVAL;
>>>>> +
>>>>> + if ((!ttm_backup_bytes_avail() && !flags->purge) ||
>>>>> +     pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
>>>>> +         return -EBUSY;
>>>>> +
>>>>> +#ifdef CONFIG_X86
>>>>> + /* Anything returned to the system needs to be cached.
>>>>> */
>>>>> + if (tt->caching != ttm_cached)
>>>>> +         set_pages_array_wb(tt->pages, tt->num_pages);
>>>>> +#endif
>>>>> +
>>>>> + if (tt->dma_address || flags->purge) {
>>>>> +         for (i = 0; i < tt->num_pages; i += num_pages) {
>>>>> +                 unsigned int order;
>>>>> +
>>>>> +                 page = tt->pages[i];
>>>>> +                 if (unlikely(!page)) {
>>>>> +                         num_pages = 1;
>>>>> +                         continue;
>>>>> +                 }
>>>>> +
>>>>> +                 order = ttm_pool_page_order(pool, page);
>>>>> +                 num_pages = 1UL << order;
>>>>> +                 if (tt->dma_address)
>>>>> +                         ttm_pool_unmap(pool, tt-
>>>>>> dma_address[i],
>>>>> +                                        num_pages);
>>>>> +                 if (flags->purge) {
>>>>> +                         shrunken += num_pages;
>>>>> +                         page->private = 0;
>>>>> +                         __free_pages(page, order);
>>>>> +                         memset(tt->pages + i, 0,
>>>>> +                                num_pages * sizeof(*tt-
>>>>>> pages));
>>>>> +                 }
>>>>> +         }
>>>>> + }
>>>>> +
>>>>> + if (flags->purge)
>>>>> +         return shrunken;
>>>>> +
>>>>> + if (pool->use_dma32)
>>>>> +         gfp = GFP_DMA32;
>>>>> + else
>>>>> +         gfp = GFP_HIGHUSER;
>>>>> +
>>>>> + alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
>>>>> __GFP_RETRY_MAYFAIL;
>>>>> +
>>>>> + for (i = 0; i < tt->num_pages; ++i) {
>>>>> +         s64 shandle;
>>>>> +
>>>>> +         page = tt->pages[i];
>>>>> +         if (unlikely(!page))
>>>>> +                 continue;
>>>>> +
>>>>> +         ttm_pool_split_for_swap(pool, page);
>>>>> +
>>>>> +         shandle = ttm_backup_backup_page(backup, page,
>>>>> flags->writeback, i,
>>>>> +                                          gfp,
>>>>> alloc_gfp);
>>>>> +         if (shandle < 0) {
>>>>> +                 /* We allow partially shrunken tts */
>>>>> +                 ret = shandle;
>>>>> +                 break;
>>>>> +         }
>>>>> +         handle = shandle;
>>>>> +         tt->pages[i] =
>>>>> ttm_backup_handle_to_page_ptr(handle);
>>>>> +         put_page(page);
>>>>> +         shrunken++;
>>>>> + }
>>>>> +
>>>>> + return shrunken ? shrunken : ret;
>>>>> +}
>>>>> +
>>>>>   /**
>>>>>    * ttm_pool_init - Initialize a pool
>>>>>    *
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
>>>>> b/drivers/gpu/drm/ttm/ttm_tt.c
>>>>> index 3baf215eca23..00b7c28f2329 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_tt.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>>>>> @@ -40,6 +40,7 @@
>>>>>   #include <drm/drm_cache.h>
>>>>>   #include <drm/drm_device.h>
>>>>>   #include <drm/drm_util.h>
>>>>> +#include <drm/ttm/ttm_backup.h>
>>>>>   #include <drm/ttm/ttm_bo.h>
>>>>>   #include <drm/ttm/ttm_tt.h>
>>>>>
>>>>> @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct ttm_tt
>>>>> *ttm,
>>>>>           ttm->swap_storage = NULL;
>>>>>           ttm->sg = bo->sg;
>>>>>           ttm->caching = caching;
>>>>> + ttm->restore = NULL;
>>>>> + ttm->backup = NULL;
>>>>>   }
>>>>>
>>>>>   int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object
>>>>> *bo,
>>>>> @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
>>>>>                   fput(ttm->swap_storage);
>>>>>           ttm->swap_storage = NULL;
>>>>>
>>>>> + if (ttm_tt_is_backed_up(ttm))
>>>>> +         ttm_pool_drop_backed_up(ttm);
>>>>> + if (ttm->backup) {
>>>>> +         ttm_backup_fini(ttm->backup);
>>>>> +         ttm->backup = NULL;
>>>>> + }
>>>>> +
>>>>>           if (ttm->pages)
>>>>>                   kvfree(ttm->pages);
>>>>>           else
>>>>> @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
>>>>>   }
>>>>>   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
>>>>>
>>>>> +/**
>>>>> + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
>>>>> + * @bdev: The TTM device.
>>>>> + * @tt: The struct ttm_tt.
>>>>> + * @flags: Flags that govern the backup behaviour.
>>>>> + *
>>>>> + * Update the page accounting and call ttm_pool_shrink_tt to
>>>>> free
>>>>> pages
>>>>> + * or back them up.
>>>>> + *
>>>>> + * Return: Number of pages freed or swapped out, or negative
>>>>> error
>>>>> code on
>>>>> + * error.
>>>>> + */
>>>>> +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
>>>>> +            const struct ttm_backup_flags flags)
>>>>> +{
>>>>> + long ret;
>>>>> +
>>>>> + if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
>>>>> +         return 0;
>>>>> +
>>>>> + ret = ttm_pool_backup(&bdev->pool, tt, &flags);
>>>>> + if (ret > 0) {
>>>>> +         tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
>>>>> +         tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
>>>>> + }
>>>>> +
>>>>> + return ret;
>>>>> +}
>>>>> +
>>>>> +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
>>>>> +            const struct ttm_operation_ctx *ctx)
>>>>> +{
>>>>> + int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
>>>>> ctx);
>>>>> +
>>>>> + if (ret)
>>>>> +         return ret;
>>>>> +
>>>>> + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +EXPORT_SYMBOL(ttm_tt_restore);
>>>>> +
>>>>>   /**
>>>>>    * ttm_tt_swapout - swap out tt object
>>>>>    *
>>>>> @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device *bdev,
>>>>>                   goto error;
>>>>>
>>>>>           ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
>>>>> + ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
>>>>>           if (unlikely(ttm->page_flags & TTM_TT_FLAG_SWAPPED)) {
>>>>>                   ret = ttm_tt_swapin(ttm);
>>>>>                   if (unlikely(ret != 0)) {
>>>>> diff --git a/include/drm/ttm/ttm_pool.h
>>>>> b/include/drm/ttm/ttm_pool.h
>>>>> index 160d954a261e..54cd34a6e4c0 100644
>>>>> --- a/include/drm/ttm/ttm_pool.h
>>>>> +++ b/include/drm/ttm/ttm_pool.h
>>>>> @@ -33,6 +33,7 @@
>>>>>
>>>>>   struct device;
>>>>>   struct seq_file;
>>>>> +struct ttm_backup_flags;
>>>>>   struct ttm_operation_ctx;
>>>>>   struct ttm_pool;
>>>>>   struct ttm_tt;
>>>>> @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
>>>>>
>>>>>   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file
>>>>> *m);
>>>>>
>>>>> +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
>>>>> +
>>>>> +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
>>>>> +              const struct ttm_backup_flags *flags);
>>>>> +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
>>>>> ttm_tt *tt,
>>>>> +                        const struct ttm_operation_ctx
>>>>> *ctx);
>>>>> +
>>>>>   int ttm_pool_mgr_init(unsigned long num_pages);
>>>>>   void ttm_pool_mgr_fini(void);
>>>>>
>>>>> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
>>>>> index 991edafdb2dd..c736c01ac2ca 100644
>>>>> --- a/include/drm/ttm/ttm_tt.h
>>>>> +++ b/include/drm/ttm/ttm_tt.h
>>>>> @@ -32,11 +32,13 @@
>>>>>   #include <drm/ttm/ttm_caching.h>
>>>>>   #include <drm/ttm/ttm_kmap_iter.h>
>>>>>
>>>>> +struct ttm_backup;
>>>>>   struct ttm_device;
>>>>>   struct ttm_tt;
>>>>>   struct ttm_resource;
>>>>>   struct ttm_buffer_object;
>>>>>   struct ttm_operation_ctx;
>>>>> +struct ttm_pool_tt_restore;
>>>>>
>>>>>   /**
>>>>>    * struct ttm_tt - This is a structure holding the pages,
>>>>> caching- and aperture
>>>>> @@ -85,17 +87,22 @@ struct ttm_tt {
>>>>>            * fault handling abuses the DMA api a bit and
>>>>> dma_map_attrs can't be
>>>>>            * used to assure pgprot always matches.
>>>>>            *
>>>>> +  * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
>>>>> if the
>>>>> +  * struct ttm_tt has been (possibly partially) backed
>>>>> up.
>>>>> +  *
>>>>>            * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT
>>>>> USE. This is
>>>>>            * set by TTM after ttm_tt_populate() has successfully
>>>>> returned, and is
>>>>>            * then unset when TTM calls ttm_tt_unpopulate().
>>>>> +  *
>>>>>            */
>>>>>   #define TTM_TT_FLAG_SWAPPED             BIT(0)
>>>>>   #define TTM_TT_FLAG_ZERO_ALLOC          BIT(1)
>>>>>   #define TTM_TT_FLAG_EXTERNAL            BIT(2)
>>>>>   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE   BIT(3)
>>>>>   #define TTM_TT_FLAG_DECRYPTED           BIT(4)
>>>>> +#define TTM_TT_FLAG_BACKED_UP            BIT(5)
>>>>>
>>>>> -#define TTM_TT_FLAG_PRIV_POPULATED       BIT(5)
>>>>> +#define TTM_TT_FLAG_PRIV_POPULATED       BIT(6)
>>>>>           uint32_t page_flags;
>>>>>           /** @num_pages: Number of pages in the page array. */
>>>>>           uint32_t num_pages;
>>>>> @@ -105,11 +112,20 @@ struct ttm_tt {
>>>>>           dma_addr_t *dma_address;
>>>>>           /** @swap_storage: Pointer to shmem struct file for swap
>>>>> storage. */
>>>>>           struct file *swap_storage;
>>>>> + /**
>>>>> +  * @backup: Pointer to backup struct for backed up tts.
>>>>> +  * Could be unified with @swap_storage. Meanwhile, the
>>>>> driver's
>>>>> +  * ttm_tt_create() callback is responsible for assigning
>>>>> +  * this field.
>>>>> +  */
>>>>> + struct ttm_backup *backup;
>>>>>           /**
>>>>>            * @caching: The current caching state of the pages, see
>>>>> enum
>>>>>            * ttm_caching.
>>>>>            */
>>>>>           enum ttm_caching caching;
>>>>> + /** @restore: Partial restoration from backup state. TTM
>>>>> private */
>>>>> + struct ttm_pool_tt_restore *restore;
>>>>>   };
>>>>>
>>>>>   /**
>>>>> @@ -129,9 +145,38 @@ static inline bool
>>>>> ttm_tt_is_populated(struct
>>>>> ttm_tt *tt)
>>>>>           return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
>>>>>   }
>>>>>
>>>>> +/**
>>>>> + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out or
>>>>> backed up
>>>>> + * @tt: The struct ttm_tt.
>>>>> + *
>>>>> + * Return: true if swapped or backed up, false otherwise.
>>>>> + */
>>>>>   static inline bool ttm_tt_is_swapped(const struct ttm_tt *tt)
>>>>>   {
>>>>> - return tt->page_flags & TTM_TT_FLAG_SWAPPED;
>>>>> + return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
>>>>> TTM_TT_FLAG_BACKED_UP);
>>>>> +}
>>>>> +
>>>>> +/**
>>>>> + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
>>>>> + * @tt: The struct ttm_tt.
>>>>> + *
>>>>> + * Return: true if swapped or backed up, false otherwise.
>>>>> + */
>>>>> +static inline bool ttm_tt_is_backed_up(const struct ttm_tt *tt)
>>>>> +{
>>>>> + return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
>>>>> +}
>>>>> +
>>>>> +/**
>>>>> + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up status
>>>>> + * @tt: The struct ttm_tt.
>>>>> + *
>>>>> + * Drivers can use this functionto clear the backed-up status,
>>>>> + * for example before destroying or re-validating a purged tt.
>>>>> + */
>>>>> +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
>>>>> +{
>>>>> + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
>>>>>   }
>>>>>
>>>>>   /**
>>>>> @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long
>>>>> num_pages,
>>>>> unsigned long num_dma32_pages);
>>>>>   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
>>>>> ttm_kmap_iter_tt *iter_tt,
>>>>>                                               struct ttm_tt *tt);
>>>>>   unsigned long ttm_tt_pages_limit(void);
>>>>> +
>>>>> +/**
>>>>> + * struct ttm_backup_flags - Flags to govern backup behaviour.
>>>>> + * @purge: Free pages without backing up. Bypass pools.
>>>>> + * @writeback: Attempt to copy contents directly to swap space,
>>>>> even
>>>>> + * if that means blocking on writes to external memory.
>>>>> + */
>>>>> +struct ttm_backup_flags {
>>>>> + u32 purge : 1;
>>>>> + u32 writeback : 1;
>>>>> +};
>>>>> +
>>>>> +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,
>>>>> +            const struct ttm_backup_flags flags);
>>>>> +
>>>>> +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt *tt,
>>>>> +            const struct ttm_operation_ctx *ctx);
>>>>> +
>>>>>   #if IS_ENABLED(CONFIG_AGP)
>>>>>   #include <linux/agp_backend.h>
>>>>>

Re: RESEND Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Thomas Hellström]

On Wed, 2025-03-05 at 10:10 +0100, Christian König wrote:
> Yeah, it's sad but I probably won't find time for a deeper review
> anytime soon.
> 
> So feel free to add my Acked-by as well.
> 
> Christian.

Thanks, Christian, I added those acks and pushed the series.

I discussed a bit with Dave and Sima on how to avoid similar stalls in
the future, and the suggestion was to drop the maintainer ack
requirement for TTM, and instead use the "seek rough consensus, don't
push anything that would cause immediate protests." policy of core DRM,
still running major designs with you.

Would that work for you?

Thanks,
Thomas




> 
> Am 05.03.25 um 04:01 schrieb Dave Airlie:
> > I've looked over the two patches mentioned here, I think they have
> > seen enough time and we need to unblock,
> > 
> > Please add and merge them:
> > Acked-by: Dave Airlie <airlied@redhat.com>
> > 
> > Dave.
> > 
> > On Tue, 25 Feb 2025 at 18:44, Thomas Hellström
> > <thomas.hellstrom@linux.intel.com> wrote:
> > > Hi, Christian,
> > > 
> > > Ping? I'd really want to get this in before -rc6
> > > 
> > > Thanks,
> > > Thomas
> > > 
> > > 
> > > 
> > > On Tue, 2025-02-18 at 16:40 +0100, Thomas Hellström wrote:
> > > > Hi, Christian,
> > > > 
> > > > On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
> > > > > Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> > > > > > Provide a helper to shrink ttm_tt page-vectors on a per-
> > > > > > page
> > > > > > basis. A ttm_backup backend could then in theory get away
> > > > > > with
> > > > > > allocating a single temporary page for each struct ttm_tt.
> > > > > > 
> > > > > > This is accomplished by splitting larger pages before
> > > > > > trying to
> > > > > > back them up.
> > > > > > 
> > > > > > In the future we could allow ttm_backup to handle backing
> > > > > > up
> > > > > > large pages as well, but currently there's no benefit in
> > > > > > doing that, since the shmem backup backend would have to
> > > > > > split those anyway to avoid allocating too much temporary
> > > > > > memory, and if the backend instead inserts pages into the
> > > > > > swap-cache, those are split on reclaim by the core.
> > > > > > 
> > > > > > Due to potential backup- and recover errors, allow
> > > > > > partially
> > > > > > swapped
> > > > > > out struct ttm_tt's, although mark them as swapped out
> > > > > > stopping
> > > > > > them
> > > > > > from being swapped out a second time. More details in the
> > > > > > ttm_pool.c
> > > > > > DOC section.
> > > > > > 
> > > > > > v2:
> > > > > > - A couple of cleanups and error fixes in
> > > > > > ttm_pool_back_up_tt.
> > > > > > - s/back_up/backup/
> > > > > > - Add a writeback parameter to the exported interface.
> > > > > > v8:
> > > > > > - Use a struct for flags for readability (Matt Brost)
> > > > > > - Address misc other review comments (Matt Brost)
> > > > > > v9:
> > > > > > - Update the kerneldoc for the ttm_tt::backup field.
> > > > > > v10:
> > > > > > - Rebase.
> > > > > > v13:
> > > > > > - Rebase on ttm_backup interface change. Update kerneldoc.
> > > > > > - Rebase and adjust ttm_tt_is_swapped().
> > > > > > v15:
> > > > > > - Rebase on ttm_backup return value change.
> > > > > > - Rebase on previous restructuring of ttm_pool_alloc()
> > > > > > - Rework the ttm_pool backup interface (Christian König)
> > > > > > - Remove cond_resched() (Christian König)
> > > > > > - Get rid of the need to allocate an intermediate page
> > > > > > array
> > > > > >    when restoring a multi-order page (Christian König)
> > > > > > - Update documentation.
> > > > > > 
> > > > > > Cc: Christian König <christian.koenig@amd.com>
> > > > > > Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> > > > > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > > > > Cc: <dri-devel@lists.freedesktop.org>
> > > > > > Signed-off-by: Thomas Hellström
> > > > > > <thomas.hellstrom@linux.intel.com>
> > > > > > Reviewed-by: Matthew Brost <matthew.brost@intel.com>
> > > > > I've tried to wrap my head around all of this like twenty
> > > > > times in
> > > > > the
> > > > > last three month, but was always interrupted at some point.
> > > > > 
> > > > > Feel free to add Acked-by: Christian Koenig
> > > > > <christian.koenig@amd.com>.
> > > > > 
> > > > > Sorry,
> > > > > Christian.
> > > > Thanks a lot for all reviewing and comments so far. There are
> > > > two TTM
> > > > patches left in the series that don't have an ack by you:
> > > > 
> > > > https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
> > > > and
> > > > 
> > > > https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17
> > > > 
> > > > None of them particularly big considering the amount of doc
> > > > text.
> > > > 
> > > > It'd be great if those could have an ack as well so we could
> > > > finally
> > > > merge this series.
> > > > 
> > > > Thanks,
> > > > Thomas
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > > > ---
> > > > > >   drivers/gpu/drm/ttm/ttm_pool.c | 554
> > > > > > +++++++++++++++++++++++++++++----
> > > > > >   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
> > > > > >   include/drm/ttm/ttm_pool.h     |   8 +
> > > > > >   include/drm/ttm/ttm_tt.h       |  67 +++-
> > > > > >   4 files changed, 629 insertions(+), 54 deletions(-)
> > > > > > 
> > > > > > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > > b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > > index c9eba76d5143..ffb7abf52bab 100644
> > > > > > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > > @@ -41,6 +41,7 @@
> > > > > >   #include <asm/set_memory.h>
> > > > > >   #endif
> > > > > > 
> > > > > > +#include <drm/ttm/ttm_backup.h>
> > > > > >   #include <drm/ttm/ttm_pool.h>
> > > > > >   #include <drm/ttm/ttm_tt.h>
> > > > > >   #include <drm/ttm/ttm_bo.h>
> > > > > > @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
> > > > > >           enum ttm_caching tt_caching;
> > > > > >   };
> > > > > > 
> > > > > > +/**
> > > > > > + * struct ttm_pool_tt_restore - State representing restore
> > > > > > from
> > > > > > backup
> > > > > > + * @pool: The pool used for page allocation while
> > > > > > restoring.
> > > > > > + * @snapshot_alloc: A snapshot of the most recent struct
> > > > > > ttm_pool_alloc_state.
> > > > > > + * @alloced_page: Pointer to the page most recently
> > > > > > allocated
> > > > > > from
> > > > > > a pool or system.
> > > > > > + * @first_dma: The dma address corresponding to
> > > > > > @alloced_page if
> > > > > > dma_mapping
> > > > > > + * is requested.
> > > > > > + * @alloced_pages: The number of allocated pages present
> > > > > > in the
> > > > > > struct ttm_tt
> > > > > > + * page vector from this restore session.
> > > > > > + * @restored_pages: The number of 4K pages restored for
> > > > > > @alloced_page (which
> > > > > > + * is typically a multi-order page).
> > > > > > + * @page_caching: The struct ttm_tt requested caching
> > > > > > + * @order: The order of @alloced_page.
> > > > > > + *
> > > > > > + * Recovery from backup might fail when we've recovered
> > > > > > less
> > > > > > than
> > > > > > the
> > > > > > + * full ttm_tt. In order not to loose any data (yet), keep
> > > > > > information
> > > > > > + * around that allows us to restart a failed ttm backup
> > > > > > recovery.
> > > > > > + */
> > > > > > +struct ttm_pool_tt_restore {
> > > > > > + struct ttm_pool *pool;
> > > > > > + struct ttm_pool_alloc_state snapshot_alloc;
> > > > > > + struct page *alloced_page;
> > > > > > + dma_addr_t first_dma;
> > > > > > + pgoff_t alloced_pages;
> > > > > > + pgoff_t restored_pages;
> > > > > > + enum ttm_caching page_caching;
> > > > > > + unsigned int order;
> > > > > > +};
> > > > > > +
> > > > > >   static unsigned long page_pool_size;
> > > > > > 
> > > > > >   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
> > > > > > WC/UC/DMA pool");
> > > > > > @@ -199,12 +229,11 @@ static int
> > > > > > ttm_pool_apply_caching(struct
> > > > > > ttm_pool_alloc_state *alloc)
> > > > > >           return 0;
> > > > > >   }
> > > > > > 
> > > > > > -/* Map pages of 1 << order size and fill the DMA address
> > > > > > array
> > > > > > */
> > > > > > +/* DMA Map pages of 1 << order size and return the
> > > > > > resulting
> > > > > > dma_address. */
> > > > > >   static int ttm_pool_map(struct ttm_pool *pool, unsigned
> > > > > > int
> > > > > > order,
> > > > > > -                 struct page *p, dma_addr_t **dma_addr)
> > > > > > +                 struct page *p, dma_addr_t *dma_addr)
> > > > > >   {
> > > > > >           dma_addr_t addr;
> > > > > > - unsigned int i;
> > > > > > 
> > > > > >           if (pool->use_dma_alloc) {
> > > > > >                   struct ttm_pool_dma *dma = (void *)p-
> > > > > > >private;
> > > > > > @@ -218,10 +247,7 @@ static int ttm_pool_map(struct
> > > > > > ttm_pool
> > > > > > *pool,
> > > > > > unsigned int order,
> > > > > >                           return -EFAULT;
> > > > > >           }
> > > > > > 
> > > > > > - for (i = 1 << order; i ; --i) {
> > > > > > -         *(*dma_addr)++ = addr;
> > > > > > -         addr += PAGE_SIZE;
> > > > > > - }
> > > > > > + *dma_addr = addr;
> > > > > > 
> > > > > >           return 0;
> > > > > >   }
> > > > > > @@ -371,6 +397,190 @@ static unsigned int
> > > > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > > > >           return p->private;
> > > > > >   }
> > > > > > 
> > > > > > +/*
> > > > > > + * Split larger pages so that we can free each PAGE_SIZE
> > > > > > page as
> > > > > > soon
> > > > > > + * as it has been backed up, in order to avoid memory
> > > > > > pressure
> > > > > > during
> > > > > > + * reclaim.
> > > > > > + */
> > > > > > +static void ttm_pool_split_for_swap(struct ttm_pool *pool,
> > > > > > struct
> > > > > > page *p)
> > > > > > +{
> > > > > > + unsigned int order = ttm_pool_page_order(pool, p);
> > > > > > + pgoff_t nr;
> > > > > > +
> > > > > > + if (!order)
> > > > > > +         return;
> > > > > > +
> > > > > > + split_page(p, order);
> > > > > > + nr = 1UL << order;
> > > > > > + while (nr--)
> > > > > > +         (p++)->private = 0;
> > > > > > +}
> > > > > > +
> > > > > > +/**
> > > > > > + * DOC: Partial backup and restoration of a struct ttm_tt.
> > > > > > + *
> > > > > > + * Swapout using ttm_backup_backup_page() and swapin using
> > > > > > + * ttm_backup_copy_page() may fail.
> > > > > > + * The former most likely due to lack of swap-space or
> > > > > > memory,
> > > > > > the
> > > > > > latter due
> > > > > > + * to lack of memory or because of signal interruption
> > > > > > during
> > > > > > waits.
> > > > > > + *
> > > > > > + * Backup failure is easily handled by using a ttm_tt
> > > > > > pages
> > > > > > vector
> > > > > > that holds
> > > > > > + * both backup handles and page pointers. This has to be
> > > > > > taken
> > > > > > into account when
> > > > > > + * restoring such a ttm_tt from backup, and when freeing
> > > > > > it
> > > > > > while
> > > > > > backed up.
> > > > > > + * When restoring, for simplicity, new pages are actually
> > > > > > allocated from the
> > > > > > + * pool and the contents of any old pages are copied in
> > > > > > and then
> > > > > > the old pages
> > > > > > + * are released.
> > > > > > + *
> > > > > > + * For restoration failures, the struct
> > > > > > ttm_pool_tt_restore
> > > > > > holds
> > > > > > sufficient state
> > > > > > + * to be able to resume an interrupted restore, and that
> > > > > > structure
> > > > > > is freed once
> > > > > > + * the restoration is complete. If the struct ttm_tt is
> > > > > > destroyed
> > > > > > while there
> > > > > > + * is a valid struct ttm_pool_tt_restore attached, that is
> > > > > > also
> > > > > > properly taken
> > > > > > + * care of.
> > > > > > + */
> > > > > > +
> > > > > > +/* Is restore ongoing for the currently allocated page? */
> > > > > > +static bool ttm_pool_restore_valid(const struct
> > > > > > ttm_pool_tt_restore *restore)
> > > > > > +{
> > > > > > + return restore && restore->restored_pages < (1 <<
> > > > > > restore-
> > > > > > > order);
> > > > > > +}
> > > > > > +
> > > > > > +/* DMA unmap and free a multi-order page, either to the
> > > > > > relevant
> > > > > > pool or to system. */
> > > > > > +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool
> > > > > > *pool,
> > > > > > struct page *page,
> > > > > > +                                const dma_addr_t
> > > > > > *dma_addr,
> > > > > > enum ttm_caching caching)
> > > > > > +{
> > > > > > + struct ttm_pool_type *pt = NULL;
> > > > > > + unsigned int order;
> > > > > > + pgoff_t nr;
> > > > > > +
> > > > > > + if (pool) {
> > > > > > +         order = ttm_pool_page_order(pool, page);
> > > > > > +         nr = (1UL << order);
> > > > > > +         if (dma_addr)
> > > > > > +                 ttm_pool_unmap(pool, *dma_addr, nr);
> > > > > > +
> > > > > > +         pt = ttm_pool_select_type(pool, caching, order);
> > > > > > + } else {
> > > > > > +         order = page->private;
> > > > > > +         nr = (1UL << order);
> > > > > > + }
> > > > > > +
> > > > > > + if (pt)
> > > > > > +         ttm_pool_type_give(pt, page);
> > > > > > + else
> > > > > > +         ttm_pool_free_page(pool, caching, order, page);
> > > > > > +
> > > > > > + return nr;
> > > > > > +}
> > > > > > +
> > > > > > +/* Populate the page-array using the most recent allocated
> > > > > > multi-
> > > > > > order page. */
> > > > > > +static void ttm_pool_allocated_page_commit(struct page
> > > > > > *allocated,
> > > > > > +                                    dma_addr_t first_dma,
> > > > > > +                                    struct
> > > > > > ttm_pool_alloc_state *alloc,
> > > > > > +                                    pgoff_t nr)
> > > > > > +{
> > > > > > + pgoff_t i;
> > > > > > +
> > > > > > + for (i = 0; i < nr; ++i)
> > > > > > +         *alloc->pages++ = allocated++;
> > > > > > +
> > > > > > + alloc->remaining_pages -= nr;
> > > > > > +
> > > > > > + if (!alloc->dma_addr)
> > > > > > +         return;
> > > > > > +
> > > > > > + for (i = 0; i < nr; ++i) {
> > > > > > +         *alloc->dma_addr++ = first_dma;
> > > > > > +         first_dma += PAGE_SIZE;
> > > > > > + }
> > > > > > +}
> > > > > > +
> > > > > > +/*
> > > > > > + * When restoring, restore backed-up content to the newly
> > > > > > allocated page and
> > > > > > + * if successful, populate the page-table and dma-address
> > > > > > arrays.
> > > > > > + */
> > > > > > +static int ttm_pool_restore_commit(struct
> > > > > > ttm_pool_tt_restore
> > > > > > *restore,
> > > > > > +                            struct ttm_backup *backup,
> > > > > > +                            const struct
> > > > > > ttm_operation_ctx
> > > > > > *ctx,
> > > > > > +                            struct ttm_pool_alloc_state
> > > > > > *alloc)
> > > > > > +
> > > > > > +{
> > > > > > + pgoff_t i, nr = 1UL << restore->order;
> > > > > > + struct page **first_page = alloc->pages;
> > > > > > + struct page *p;
> > > > > > + int ret = 0;
> > > > > > +
> > > > > > + for (i = restore->restored_pages; i < nr; ++i) {
> > > > > > +         p = first_page[i];
> > > > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > > > +                 unsigned long handle =
> > > > > > ttm_backup_page_ptr_to_handle(p);
> > > > > > +
> > > > > > +                 if (handle == 0) {
> > > > > > +                         restore->restored_pages++;
> > > > > > +                         continue;
> > > > > > +                 }
> > > > > > +
> > > > > > +                 ret = ttm_backup_copy_page(backup,
> > > > > > restore->alloced_page + i,
> > > > > > +                                            handle, ctx-
> > > > > > > interruptible);
> > > > > > +                 if (ret)
> > > > > > +                         break;
> > > > > > +
> > > > > > +                 ttm_backup_drop(backup, handle);
> > > > > > +         } else if (p) {
> > > > > > +                 /*
> > > > > > +                  * We could probably avoid splitting the
> > > > > > old page
> > > > > > +                  * using clever logic, but ATM we don't
> > > > > > care, as
> > > > > > +                  * we prioritize releasing memory ASAP.
> > > > > > Note that
> > > > > > +                  * here, the old retained page is always
> > > > > > write-back
> > > > > > +                  * cached.
> > > > > > +                  */
> > > > > > +                 ttm_pool_split_for_swap(restore->pool,
> > > > > > p);
> > > > > > +                 copy_highpage(restore->alloced_page + i,
> > > > > > p);
> > > > > > +                 __free_pages(p, 0);
> > > > > > +         }
> > > > > > +
> > > > > > +         restore->restored_pages++;
> > > > > > +         first_page[i] =
> > > > > > ttm_backup_handle_to_page_ptr(0);
> > > > > > + }
> > > > > > +
> > > > > > + if (ret) {
> > > > > > +         if (!restore->restored_pages) {
> > > > > > +                 dma_addr_t *dma_addr = alloc->dma_addr ?
> > > > > > &restore->first_dma : NULL;
> > > > > > +
> > > > > > +                 ttm_pool_unmap_and_free(restore->pool,
> > > > > > restore->alloced_page,
> > > > > > +                                         dma_addr,
> > > > > > restore-
> > > > > > > page_caching);
> > > > > > +                 restore->restored_pages = nr;
> > > > > > +         }
> > > > > > +         return ret;
> > > > > > + }
> > > > > > +
> > > > > > + ttm_pool_allocated_page_commit(restore->alloced_page,
> > > > > > restore->first_dma,
> > > > > > +                                alloc, nr);
> > > > > > + if (restore->page_caching == alloc->tt_caching ||
> > > > > > PageHighMem(restore->alloced_page))
> > > > > > +         alloc->caching_divide = alloc->pages;
> > > > > > + restore->snapshot_alloc = *alloc;
> > > > > > + restore->alloced_pages += nr;
> > > > > > +
> > > > > > + return 0;
> > > > > > +}
> > > > > > +
> > > > > > +/* If restoring, save information needed for
> > > > > > ttm_pool_restore_commit(). */
> > > > > > +static void
> > > > > > +ttm_pool_page_allocated_restore(struct ttm_pool *pool,
> > > > > > unsigned
> > > > > > int order,
> > > > > > +                         struct page *p,
> > > > > > +                         enum ttm_caching page_caching,
> > > > > > +                         dma_addr_t first_dma,
> > > > > > +                         struct ttm_pool_tt_restore
> > > > > > *restore,
> > > > > > +                         const struct
> > > > > > ttm_pool_alloc_state
> > > > > > *alloc)
> > > > > > +{
> > > > > > + restore->pool = pool;
> > > > > > + restore->order = order;
> > > > > > + restore->restored_pages = 0;
> > > > > > + restore->page_caching = page_caching;
> > > > > > + restore->first_dma = first_dma;
> > > > > > + restore->alloced_page = p;
> > > > > > + restore->snapshot_alloc = *alloc;
> > > > > > +}
> > > > > > +
> > > > > >   /*
> > > > > >    * Called when we got a page, either from a pool or newly
> > > > > > allocated.
> > > > > >    * if needed, dma map the page and populate the dma
> > > > > > address
> > > > > > array.
> > > > > > @@ -380,10 +590,11 @@ static unsigned int
> > > > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > > > >    */
> > > > > >   static int ttm_pool_page_allocated(struct ttm_pool *pool,
> > > > > > unsigned int order,
> > > > > >                                      struct page *p, enum
> > > > > > ttm_caching page_caching,
> > > > > > -                            struct ttm_pool_alloc_state
> > > > > > *alloc)
> > > > > > +                            struct ttm_pool_alloc_state
> > > > > > *alloc,
> > > > > > +                            struct ttm_pool_tt_restore
> > > > > > *restore)
> > > > > >   {
> > > > > > - pgoff_t i, nr = 1UL << order;
> > > > > >           bool caching_consistent;
> > > > > > + dma_addr_t first_dma;
> > > > > >           int r = 0;
> > > > > > 
> > > > > >           caching_consistent = (page_caching == alloc-
> > > > > > >tt_caching)
> > > > > > > > PageHighMem(p);
> > > > > > @@ -395,17 +606,20 @@ static int
> > > > > > ttm_pool_page_allocated(struct
> > > > > > ttm_pool *pool, unsigned int order,
> > > > > >           }
> > > > > > 
> > > > > >           if (alloc->dma_addr) {
> > > > > > -         r = ttm_pool_map(pool, order, p, &alloc-
> > > > > > > dma_addr);
> > > > > > +         r = ttm_pool_map(pool, order, p, &first_dma);
> > > > > >                   if (r)
> > > > > >                           return r;
> > > > > >           }
> > > > > > 
> > > > > > - alloc->remaining_pages -= nr;
> > > > > > - for (i = 0; i < nr; ++i)
> > > > > > -         *alloc->pages++ = p++;
> > > > > > + if (restore) {
> > > > > > +         ttm_pool_page_allocated_restore(pool, order, p,
> > > > > > page_caching,
> > > > > > +                                         first_dma,
> > > > > > restore, alloc);
> > > > > > + } else {
> > > > > > +         ttm_pool_allocated_page_commit(p, first_dma,
> > > > > > alloc, 1UL << order);
> > > > > > 
> > > > > > - if (caching_consistent)
> > > > > > -         alloc->caching_divide = alloc->pages;
> > > > > > +         if (caching_consistent)
> > > > > > +                 alloc->caching_divide = alloc->pages;
> > > > > > + }
> > > > > > 
> > > > > >           return 0;
> > > > > >   }
> > > > > > @@ -428,22 +642,24 @@ static void
> > > > > > ttm_pool_free_range(struct
> > > > > > ttm_pool *pool, struct ttm_tt *tt,
> > > > > >                                   pgoff_t start_page,
> > > > > > pgoff_t
> > > > > > end_page)
> > > > > >   {
> > > > > >           struct page **pages = &tt->pages[start_page];
> > > > > > - unsigned int order;
> > > > > > + struct ttm_backup *backup = tt->backup;
> > > > > >           pgoff_t i, nr;
> > > > > > 
> > > > > >           for (i = start_page; i < end_page; i += nr, pages
> > > > > > += nr)
> > > > > > {
> > > > > > -         struct ttm_pool_type *pt = NULL;
> > > > > > +         struct page *p = *pages;
> > > > > > 
> > > > > > -         order = ttm_pool_page_order(pool, *pages);
> > > > > > -         nr = (1UL << order);
> > > > > > -         if (tt->dma_address)
> > > > > > -                 ttm_pool_unmap(pool, tt->dma_address[i],
> > > > > > nr);
> > > > > > +         nr = 1;
> > > > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > > > +                 unsigned long handle =
> > > > > > ttm_backup_page_ptr_to_handle(p);
> > > > > > 
> > > > > > -         pt = ttm_pool_select_type(pool, caching, order);
> > > > > > -         if (pt)
> > > > > > -                 ttm_pool_type_give(pt, *pages);
> > > > > > -         else
> > > > > > -                 ttm_pool_free_page(pool, caching, order,
> > > > > > *pages);
> > > > > > +                 if (handle != 0)
> > > > > > +                         ttm_backup_drop(backup, handle);
> > > > > > +         } else if (p) {
> > > > > > +                 dma_addr_t *dma_addr = tt->dma_address ?
> > > > > > +                         tt->dma_address + i : NULL;
> > > > > > +
> > > > > > +                 nr = ttm_pool_unmap_and_free(pool, p,
> > > > > > dma_addr, caching);
> > > > > > +         }
> > > > > >           }
> > > > > >   }
> > > > > > 
> > > > > > @@ -467,22 +683,11 @@ static unsigned int
> > > > > > ttm_pool_alloc_find_order(unsigned int highest,
> > > > > >           return min_t(unsigned int, highest, __fls(alloc-
> > > > > > > remaining_pages));
> > > > > >   }
> > > > > > 
> > > > > > -/**
> > > > > > - * ttm_pool_alloc - Fill a ttm_tt object
> > > > > > - *
> > > > > > - * @pool: ttm_pool to use
> > > > > > - * @tt: ttm_tt object to fill
> > > > > > - * @ctx: operation context
> > > > > > - *
> > > > > > - * Fill the ttm_tt object with pages and also make sure to
> > > > > > DMA
> > > > > > map
> > > > > > them when
> > > > > > - * necessary.
> > > > > > - *
> > > > > > - * Returns: 0 on successe, negative error code otherwise.
> > > > > > - */
> > > > > > -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
> > > > > > *tt,
> > > > > > -            struct ttm_operation_ctx *ctx)
> > > > > > +static int __ttm_pool_alloc(struct ttm_pool *pool, struct
> > > > > > ttm_tt
> > > > > > *tt,
> > > > > > +                     const struct ttm_operation_ctx *ctx,
> > > > > > +                     struct ttm_pool_alloc_state *alloc,
> > > > > > +                     struct ttm_pool_tt_restore *restore)
> > > > > >   {
> > > > > > - struct ttm_pool_alloc_state alloc;
> > > > > >           enum ttm_caching page_caching;
> > > > > >           gfp_t gfp_flags = GFP_USER;
> > > > > >           pgoff_t caching_divide;
> > > > > > @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool
> > > > > > *pool,
> > > > > > struct ttm_tt *tt,
> > > > > >           struct page *p;
> > > > > >           int r;
> > > > > > 
> > > > > > - ttm_pool_alloc_state_init(tt, &alloc);
> > > > > > -
> > > > > > - WARN_ON(!alloc.remaining_pages ||
> > > > > > ttm_tt_is_populated(tt));
> > > > > > - WARN_ON(alloc.dma_addr && !pool->dev);
> > > > > > + WARN_ON(!alloc->remaining_pages ||
> > > > > > ttm_tt_is_populated(tt));
> > > > > > + WARN_ON(alloc->dma_addr && !pool->dev);
> > > > > > 
> > > > > >           if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
> > > > > >                   gfp_flags |= __GFP_ZERO;
> > > > > > @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool
> > > > > > *pool,
> > > > > > struct ttm_tt *tt,
> > > > > > 
> > > > > >           page_caching = tt->caching;
> > > > > >           allow_pools = true;
> > > > > > - for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > > > &alloc);
> > > > > > -      alloc.remaining_pages;
> > > > > > -      order = ttm_pool_alloc_find_order(order, &alloc)) {
> > > > > > + for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > > > alloc);
> > > > > > +      alloc->remaining_pages;
> > > > > > +      order = ttm_pool_alloc_find_order(order, alloc)) {
> > > > > >                   struct ttm_pool_type *pt;
> > > > > > 
> > > > > >                   /* First, try to allocate a page from a
> > > > > > pool if
> > > > > > one exists. */
> > > > > > @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool
> > > > > > *pool,
> > > > > > struct ttm_tt *tt,
> > > > > >                           r = -ENOMEM;
> > > > > >                           goto error_free_all;
> > > > > >                   }
> > > > > > -         r = ttm_pool_page_allocated(pool, order, p,
> > > > > > page_caching, &alloc);
> > > > > > +         r = ttm_pool_page_allocated(pool, order, p,
> > > > > > page_caching, alloc,
> > > > > > +                                     restore);
> > > > > >                   if (r)
> > > > > >                           goto error_free_page;
> > > > > > +
> > > > > > +         if (ttm_pool_restore_valid(restore)) {
> > > > > > +                 r = ttm_pool_restore_commit(restore, tt-
> > > > > > > backup, ctx, alloc);
> > > > > > +                 if (r)
> > > > > > +                         goto error_free_all;
> > > > > > +         }
> > > > > >           }
> > > > > > 
> > > > > > - r = ttm_pool_apply_caching(&alloc);
> > > > > > + r = ttm_pool_apply_caching(alloc);
> > > > > >           if (r)
> > > > > >                   goto error_free_all;
> > > > > > 
> > > > > > + kfree(tt->restore);
> > > > > > + tt->restore = NULL;
> > > > > > +
> > > > > >           return 0;
> > > > > > 
> > > > > >   error_free_page:
> > > > > >           ttm_pool_free_page(pool, page_caching, order, p);
> > > > > > 
> > > > > >   error_free_all:
> > > > > > - caching_divide = alloc.caching_divide - tt->pages;
> > > > > > + if (tt->restore)
> > > > > > +         return r;
> > > > > > +
> > > > > > + caching_divide = alloc->caching_divide - tt->pages;
> > > > > >           ttm_pool_free_range(pool, tt, tt->caching, 0,
> > > > > > caching_divide);
> > > > > >           ttm_pool_free_range(pool, tt, ttm_cached,
> > > > > > caching_divide,
> > > > > > -                     tt->num_pages -
> > > > > > alloc.remaining_pages);
> > > > > > +                     tt->num_pages - alloc-
> > > > > > > remaining_pages);
> > > > > >           return r;
> > > > > >   }
> > > > > > +
> > > > > > +/**
> > > > > > + * ttm_pool_alloc - Fill a ttm_tt object
> > > > > > + *
> > > > > > + * @pool: ttm_pool to use
> > > > > > + * @tt: ttm_tt object to fill
> > > > > > + * @ctx: operation context
> > > > > > + *
> > > > > > + * Fill the ttm_tt object with pages and also make sure to
> > > > > > DMA
> > > > > > map
> > > > > > them when
> > > > > > + * necessary.
> > > > > > + *
> > > > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > > > + */
> > > > > > +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt
> > > > > > *tt,
> > > > > > +            struct ttm_operation_ctx *ctx)
> > > > > > +{
> > > > > > + struct ttm_pool_alloc_state alloc;
> > > > > > +
> > > > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > > > +         return -EINVAL;
> > > > > > +
> > > > > > + ttm_pool_alloc_state_init(tt, &alloc);
> > > > > > +
> > > > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> > > > > > +}
> > > > > >   EXPORT_SYMBOL(ttm_pool_alloc);
> > > > > > 
> > > > > > +/**
> > > > > > + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
> > > > > > previously backed-up
> > > > > > + * content.
> > > > > > + *
> > > > > > + * @pool: ttm_pool to use
> > > > > > + * @tt: ttm_tt object to fill
> > > > > > + * @ctx: operation context
> > > > > > + *
> > > > > > + * Fill the ttm_tt object with pages and also make sure to
> > > > > > DMA
> > > > > > map
> > > > > > them when
> > > > > > + * necessary. Read in backed-up content.
> > > > > > + *
> > > > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > > > + */
> > > > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool,
> > > > > > struct
> > > > > > ttm_tt *tt,
> > > > > > +                        const struct ttm_operation_ctx
> > > > > > *ctx)
> > > > > > +{
> > > > > > + struct ttm_pool_alloc_state alloc;
> > > > > > +
> > > > > > + if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> > > > > > +         return -EINVAL;
> > > > > > +
> > > > > > + if (!tt->restore) {
> > > > > > +         gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> > > > > > +
> > > > > > +         ttm_pool_alloc_state_init(tt, &alloc);
> > > > > > +         if (ctx->gfp_retry_mayfail)
> > > > > > +                 gfp |= __GFP_RETRY_MAYFAIL;
> > > > > > +
> > > > > > +         tt->restore = kzalloc(sizeof(*tt->restore),
> > > > > > gfp);
> > > > > > +         if (!tt->restore)
> > > > > > +                 return -ENOMEM;
> > > > > > +
> > > > > > +         tt->restore->snapshot_alloc = alloc;
> > > > > > +         tt->restore->pool = pool;
> > > > > > +         tt->restore->restored_pages = 1;
> > > > > > + } else {
> > > > > > +         struct ttm_pool_tt_restore *restore = tt-
> > > > > > > restore;
> > > > > > +         int ret;
> > > > > > +
> > > > > > +         alloc = restore->snapshot_alloc;
> > > > > > +         if (ttm_pool_restore_valid(tt->restore)) {
> > > > > > +                 ret = ttm_pool_restore_commit(restore,
> > > > > > tt-
> > > > > > > backup, ctx, &alloc);
> > > > > > +                 if (ret)
> > > > > > +                         return ret;
> > > > > > +         }
> > > > > > +         if (!alloc.remaining_pages)
> > > > > > +                 return 0;
> > > > > > + }
> > > > > > +
> > > > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
> > > > > > > restore);
> > > > > > +}
> > > > > > +
> > > > > >   /**
> > > > > >    * ttm_pool_free - Free the backing pages from a ttm_tt
> > > > > > object
> > > > > >    *
> > > > > > @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool
> > > > > > *pool,
> > > > > > struct ttm_tt *tt)
> > > > > >   }
> > > > > >   EXPORT_SYMBOL(ttm_pool_free);
> > > > > > 
> > > > > > +/**
> > > > > > + * ttm_pool_drop_backed_up() - Release content of a
> > > > > > swapped-out
> > > > > > struct ttm_tt
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + *
> > > > > > + * Release handles with associated content or any
> > > > > > remaining
> > > > > > pages
> > > > > > of
> > > > > > + * a backed-up struct ttm_tt.
> > > > > > + */
> > > > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> > > > > > +{
> > > > > > + struct ttm_pool_tt_restore *restore;
> > > > > > + pgoff_t start_page = 0;
> > > > > > +
> > > > > > + WARN_ON(!ttm_tt_is_backed_up(tt));
> > > > > > +
> > > > > > + restore = tt->restore;
> > > > > > +
> > > > > > + /*
> > > > > > +  * Unmap and free any uncommitted restore page.
> > > > > > +  * any tt page-array backup entries already read back
> > > > > > has
> > > > > > +  * been cleared already
> > > > > > +  */
> > > > > > + if (ttm_pool_restore_valid(restore)) {
> > > > > > +         dma_addr_t *dma_addr = tt->dma_address ?
> > > > > > &restore-
> > > > > > > first_dma : NULL;
> > > > > > +
> > > > > > +         ttm_pool_unmap_and_free(restore->pool, restore-
> > > > > > > alloced_page,
> > > > > > +                                 dma_addr, restore-
> > > > > > > page_caching);
> > > > > > +         restore->restored_pages = 1UL << restore->order;
> > > > > > + }
> > > > > > +
> > > > > > + /*
> > > > > > +  * If a restore is ongoing, part of the tt pages may
> > > > > > have
> > > > > > a
> > > > > > +  * caching different than writeback.
> > > > > > +  */
> > > > > > + if (restore) {
> > > > > > +         pgoff_t mid = restore-
> > > > > > > snapshot_alloc.caching_divide - tt->pages;
> > > > > > +
> > > > > > +         start_page = restore->alloced_pages;
> > > > > > +         WARN_ON(mid > start_page);
> > > > > > +         /* Pages that might be dma-mapped and non-cached
> > > > > > */
> > > > > > +         ttm_pool_free_range(restore->pool, tt, tt-
> > > > > > > caching,
> > > > > > +                             0, mid);
> > > > > > +         /* Pages that might be dma-mapped but cached */
> > > > > > +         ttm_pool_free_range(restore->pool, tt,
> > > > > > ttm_cached,
> > > > > > +                             mid, restore-
> > > > > > > alloced_pages);
> > > > > > +         kfree(restore);
> > > > > > +         tt->restore = NULL;
> > > > > > + }
> > > > > > +
> > > > > > + ttm_pool_free_range(NULL, tt, ttm_cached, start_page,
> > > > > > tt-
> > > > > > > num_pages);
> > > > > > +}
> > > > > > +
> > > > > > +/**
> > > > > > + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> > > > > > + * @pool: The pool used when allocating the struct ttm_tt.
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + * @flags: Flags to govern the backup behaviour.
> > > > > > + *
> > > > > > + * Back up or purge a struct ttm_tt. If @purge is true,
> > > > > > then
> > > > > > + * all pages will be freed directly to the system rather
> > > > > > than to
> > > > > > the pool
> > > > > > + * they were allocated from, making the function behave
> > > > > > similarly
> > > > > > to
> > > > > > + * ttm_pool_free(). If @purge is false the pages will be
> > > > > > backed
> > > > > > up
> > > > > > instead,
> > > > > > + * exchanged for handles.
> > > > > > + * A subsequent call to ttm_pool_restore_and_alloc() will
> > > > > > then
> > > > > > read back the content and
> > > > > > + * a subsequent call to ttm_pool_drop_backed_up() will
> > > > > > drop it.
> > > > > > + * If backup of a page fails for whatever reason, @ttm
> > > > > > will
> > > > > > still
> > > > > > be
> > > > > > + * partially backed up, retaining those pages for which
> > > > > > backup
> > > > > > fails.
> > > > > > + * In that case, this function can be retried, possibly
> > > > > > after
> > > > > > freeing up
> > > > > > + * memory resources.
> > > > > > + *
> > > > > > + * Return: Number of pages actually backed up or freed, or
> > > > > > negative
> > > > > > + * error code on error.
> > > > > > + */
> > > > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt
> > > > > > *tt,
> > > > > > +              const struct ttm_backup_flags *flags)
> > > > > > +{
> > > > > > + struct ttm_backup *backup = tt->backup;
> > > > > > + struct page *page;
> > > > > > + unsigned long handle;
> > > > > > + gfp_t alloc_gfp;
> > > > > > + gfp_t gfp;
> > > > > > + int ret = 0;
> > > > > > + pgoff_t shrunken = 0;
> > > > > > + pgoff_t i, num_pages;
> > > > > > +
> > > > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > > > +         return -EINVAL;
> > > > > > +
> > > > > > + if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> > > > > > +     pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> > > > > > +         return -EBUSY;
> > > > > > +
> > > > > > +#ifdef CONFIG_X86
> > > > > > + /* Anything returned to the system needs to be cached.
> > > > > > */
> > > > > > + if (tt->caching != ttm_cached)
> > > > > > +         set_pages_array_wb(tt->pages, tt->num_pages);
> > > > > > +#endif
> > > > > > +
> > > > > > + if (tt->dma_address || flags->purge) {
> > > > > > +         for (i = 0; i < tt->num_pages; i += num_pages) {
> > > > > > +                 unsigned int order;
> > > > > > +
> > > > > > +                 page = tt->pages[i];
> > > > > > +                 if (unlikely(!page)) {
> > > > > > +                         num_pages = 1;
> > > > > > +                         continue;
> > > > > > +                 }
> > > > > > +
> > > > > > +                 order = ttm_pool_page_order(pool, page);
> > > > > > +                 num_pages = 1UL << order;
> > > > > > +                 if (tt->dma_address)
> > > > > > +                         ttm_pool_unmap(pool, tt-
> > > > > > > dma_address[i],
> > > > > > +                                        num_pages);
> > > > > > +                 if (flags->purge) {
> > > > > > +                         shrunken += num_pages;
> > > > > > +                         page->private = 0;
> > > > > > +                         __free_pages(page, order);
> > > > > > +                         memset(tt->pages + i, 0,
> > > > > > +                                num_pages * sizeof(*tt-
> > > > > > > pages));
> > > > > > +                 }
> > > > > > +         }
> > > > > > + }
> > > > > > +
> > > > > > + if (flags->purge)
> > > > > > +         return shrunken;
> > > > > > +
> > > > > > + if (pool->use_dma32)
> > > > > > +         gfp = GFP_DMA32;
> > > > > > + else
> > > > > > +         gfp = GFP_HIGHUSER;
> > > > > > +
> > > > > > + alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
> > > > > > __GFP_RETRY_MAYFAIL;
> > > > > > +
> > > > > > + for (i = 0; i < tt->num_pages; ++i) {
> > > > > > +         s64 shandle;
> > > > > > +
> > > > > > +         page = tt->pages[i];
> > > > > > +         if (unlikely(!page))
> > > > > > +                 continue;
> > > > > > +
> > > > > > +         ttm_pool_split_for_swap(pool, page);
> > > > > > +
> > > > > > +         shandle = ttm_backup_backup_page(backup, page,
> > > > > > flags->writeback, i,
> > > > > > +                                          gfp,
> > > > > > alloc_gfp);
> > > > > > +         if (shandle < 0) {
> > > > > > +                 /* We allow partially shrunken tts */
> > > > > > +                 ret = shandle;
> > > > > > +                 break;
> > > > > > +         }
> > > > > > +         handle = shandle;
> > > > > > +         tt->pages[i] =
> > > > > > ttm_backup_handle_to_page_ptr(handle);
> > > > > > +         put_page(page);
> > > > > > +         shrunken++;
> > > > > > + }
> > > > > > +
> > > > > > + return shrunken ? shrunken : ret;
> > > > > > +}
> > > > > > +
> > > > > >   /**
> > > > > >    * ttm_pool_init - Initialize a pool
> > > > > >    *
> > > > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > > index 3baf215eca23..00b7c28f2329 100644
> > > > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > > @@ -40,6 +40,7 @@
> > > > > >   #include <drm/drm_cache.h>
> > > > > >   #include <drm/drm_device.h>
> > > > > >   #include <drm/drm_util.h>
> > > > > > +#include <drm/ttm/ttm_backup.h>
> > > > > >   #include <drm/ttm/ttm_bo.h>
> > > > > >   #include <drm/ttm/ttm_tt.h>
> > > > > > 
> > > > > > @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct
> > > > > > ttm_tt
> > > > > > *ttm,
> > > > > >           ttm->swap_storage = NULL;
> > > > > >           ttm->sg = bo->sg;
> > > > > >           ttm->caching = caching;
> > > > > > + ttm->restore = NULL;
> > > > > > + ttm->backup = NULL;
> > > > > >   }
> > > > > > 
> > > > > >   int ttm_tt_init(struct ttm_tt *ttm, struct
> > > > > > ttm_buffer_object
> > > > > > *bo,
> > > > > > @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
> > > > > >                   fput(ttm->swap_storage);
> > > > > >           ttm->swap_storage = NULL;
> > > > > > 
> > > > > > + if (ttm_tt_is_backed_up(ttm))
> > > > > > +         ttm_pool_drop_backed_up(ttm);
> > > > > > + if (ttm->backup) {
> > > > > > +         ttm_backup_fini(ttm->backup);
> > > > > > +         ttm->backup = NULL;
> > > > > > + }
> > > > > > +
> > > > > >           if (ttm->pages)
> > > > > >                   kvfree(ttm->pages);
> > > > > >           else
> > > > > > @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
> > > > > >   }
> > > > > >   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
> > > > > > 
> > > > > > +/**
> > > > > > + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> > > > > > + * @bdev: The TTM device.
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + * @flags: Flags that govern the backup behaviour.
> > > > > > + *
> > > > > > + * Update the page accounting and call ttm_pool_shrink_tt
> > > > > > to
> > > > > > free
> > > > > > pages
> > > > > > + * or back them up.
> > > > > > + *
> > > > > > + * Return: Number of pages freed or swapped out, or
> > > > > > negative
> > > > > > error
> > > > > > code on
> > > > > > + * error.
> > > > > > + */
> > > > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt
> > > > > > *tt,
> > > > > > +            const struct ttm_backup_flags flags)
> > > > > > +{
> > > > > > + long ret;
> > > > > > +
> > > > > > + if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> > > > > > +         return 0;
> > > > > > +
> > > > > > + ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> > > > > > + if (ret > 0) {
> > > > > > +         tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> > > > > > +         tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> > > > > > + }
> > > > > > +
> > > > > > + return ret;
> > > > > > +}
> > > > > > +
> > > > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt
> > > > > > *tt,
> > > > > > +            const struct ttm_operation_ctx *ctx)
> > > > > > +{
> > > > > > + int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
> > > > > > ctx);
> > > > > > +
> > > > > > + if (ret)
> > > > > > +         return ret;
> > > > > > +
> > > > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > > > +
> > > > > > + return 0;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL(ttm_tt_restore);
> > > > > > +
> > > > > >   /**
> > > > > >    * ttm_tt_swapout - swap out tt object
> > > > > >    *
> > > > > > @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device
> > > > > > *bdev,
> > > > > >                   goto error;
> > > > > > 
> > > > > >           ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> > > > > > + ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > > >           if (unlikely(ttm->page_flags &
> > > > > > TTM_TT_FLAG_SWAPPED)) {
> > > > > >                   ret = ttm_tt_swapin(ttm);
> > > > > >                   if (unlikely(ret != 0)) {
> > > > > > diff --git a/include/drm/ttm/ttm_pool.h
> > > > > > b/include/drm/ttm/ttm_pool.h
> > > > > > index 160d954a261e..54cd34a6e4c0 100644
> > > > > > --- a/include/drm/ttm/ttm_pool.h
> > > > > > +++ b/include/drm/ttm/ttm_pool.h
> > > > > > @@ -33,6 +33,7 @@
> > > > > > 
> > > > > >   struct device;
> > > > > >   struct seq_file;
> > > > > > +struct ttm_backup_flags;
> > > > > >   struct ttm_operation_ctx;
> > > > > >   struct ttm_pool;
> > > > > >   struct ttm_tt;
> > > > > > @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool
> > > > > > *pool);
> > > > > > 
> > > > > >   int ttm_pool_debugfs(struct ttm_pool *pool, struct
> > > > > > seq_file
> > > > > > *m);
> > > > > > 
> > > > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> > > > > > +
> > > > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt
> > > > > > *ttm,
> > > > > > +              const struct ttm_backup_flags *flags);
> > > > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool,
> > > > > > struct
> > > > > > ttm_tt *tt,
> > > > > > +                        const struct ttm_operation_ctx
> > > > > > *ctx);
> > > > > > +
> > > > > >   int ttm_pool_mgr_init(unsigned long num_pages);
> > > > > >   void ttm_pool_mgr_fini(void);
> > > > > > 
> > > > > > diff --git a/include/drm/ttm/ttm_tt.h
> > > > > > b/include/drm/ttm/ttm_tt.h
> > > > > > index 991edafdb2dd..c736c01ac2ca 100644
> > > > > > --- a/include/drm/ttm/ttm_tt.h
> > > > > > +++ b/include/drm/ttm/ttm_tt.h
> > > > > > @@ -32,11 +32,13 @@
> > > > > >   #include <drm/ttm/ttm_caching.h>
> > > > > >   #include <drm/ttm/ttm_kmap_iter.h>
> > > > > > 
> > > > > > +struct ttm_backup;
> > > > > >   struct ttm_device;
> > > > > >   struct ttm_tt;
> > > > > >   struct ttm_resource;
> > > > > >   struct ttm_buffer_object;
> > > > > >   struct ttm_operation_ctx;
> > > > > > +struct ttm_pool_tt_restore;
> > > > > > 
> > > > > >   /**
> > > > > >    * struct ttm_tt - This is a structure holding the pages,
> > > > > > caching- and aperture
> > > > > > @@ -85,17 +87,22 @@ struct ttm_tt {
> > > > > >            * fault handling abuses the DMA api a bit and
> > > > > > dma_map_attrs can't be
> > > > > >            * used to assure pgprot always matches.
> > > > > >            *
> > > > > > +  * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
> > > > > > if the
> > > > > > +  * struct ttm_tt has been (possibly partially) backed
> > > > > > up.
> > > > > > +  *
> > > > > >            * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only.
> > > > > > DO NOT
> > > > > > USE. This is
> > > > > >            * set by TTM after ttm_tt_populate() has
> > > > > > successfully
> > > > > > returned, and is
> > > > > >            * then unset when TTM calls ttm_tt_unpopulate().
> > > > > > +  *
> > > > > >            */
> > > > > >   #define TTM_TT_FLAG_SWAPPED             BIT(0)
> > > > > >   #define TTM_TT_FLAG_ZERO_ALLOC          BIT(1)
> > > > > >   #define TTM_TT_FLAG_EXTERNAL            BIT(2)
> > > > > >   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE   BIT(3)
> > > > > >   #define TTM_TT_FLAG_DECRYPTED           BIT(4)
> > > > > > +#define TTM_TT_FLAG_BACKED_UP            BIT(5)
> > > > > > 
> > > > > > -#define TTM_TT_FLAG_PRIV_POPULATED       BIT(5)
> > > > > > +#define TTM_TT_FLAG_PRIV_POPULATED       BIT(6)
> > > > > >           uint32_t page_flags;
> > > > > >           /** @num_pages: Number of pages in the page
> > > > > > array. */
> > > > > >           uint32_t num_pages;
> > > > > > @@ -105,11 +112,20 @@ struct ttm_tt {
> > > > > >           dma_addr_t *dma_address;
> > > > > >           /** @swap_storage: Pointer to shmem struct file
> > > > > > for swap
> > > > > > storage. */
> > > > > >           struct file *swap_storage;
> > > > > > + /**
> > > > > > +  * @backup: Pointer to backup struct for backed up tts.
> > > > > > +  * Could be unified with @swap_storage. Meanwhile, the
> > > > > > driver's
> > > > > > +  * ttm_tt_create() callback is responsible for assigning
> > > > > > +  * this field.
> > > > > > +  */
> > > > > > + struct ttm_backup *backup;
> > > > > >           /**
> > > > > >            * @caching: The current caching state of the
> > > > > > pages, see
> > > > > > enum
> > > > > >            * ttm_caching.
> > > > > >            */
> > > > > >           enum ttm_caching caching;
> > > > > > + /** @restore: Partial restoration from backup state. TTM
> > > > > > private */
> > > > > > + struct ttm_pool_tt_restore *restore;
> > > > > >   };
> > > > > > 
> > > > > >   /**
> > > > > > @@ -129,9 +145,38 @@ static inline bool
> > > > > > ttm_tt_is_populated(struct
> > > > > > ttm_tt *tt)
> > > > > >           return tt->page_flags &
> > > > > > TTM_TT_FLAG_PRIV_POPULATED;
> > > > > >   }
> > > > > > 
> > > > > > +/**
> > > > > > + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out
> > > > > > or
> > > > > > backed up
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + *
> > > > > > + * Return: true if swapped or backed up, false otherwise.
> > > > > > + */
> > > > > >   static inline bool ttm_tt_is_swapped(const struct ttm_tt
> > > > > > *tt)
> > > > > >   {
> > > > > > - return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> > > > > > + return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
> > > > > > TTM_TT_FLAG_BACKED_UP);
> > > > > > +}
> > > > > > +
> > > > > > +/**
> > > > > > + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + *
> > > > > > + * Return: true if swapped or backed up, false otherwise.
> > > > > > + */
> > > > > > +static inline bool ttm_tt_is_backed_up(const struct ttm_tt
> > > > > > *tt)
> > > > > > +{
> > > > > > + return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> > > > > > +}
> > > > > > +
> > > > > > +/**
> > > > > > + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up
> > > > > > status
> > > > > > + * @tt: The struct ttm_tt.
> > > > > > + *
> > > > > > + * Drivers can use this functionto clear the backed-up
> > > > > > status,
> > > > > > + * for example before destroying or re-validating a purged
> > > > > > tt.
> > > > > > + */
> > > > > > +static inline void ttm_tt_clear_backed_up(struct ttm_tt
> > > > > > *tt)
> > > > > > +{
> > > > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > > >   }
> > > > > > 
> > > > > >   /**
> > > > > > @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long
> > > > > > num_pages,
> > > > > > unsigned long num_dma32_pages);
> > > > > >   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
> > > > > > ttm_kmap_iter_tt *iter_tt,
> > > > > >                                               struct ttm_tt
> > > > > > *tt);
> > > > > >   unsigned long ttm_tt_pages_limit(void);
> > > > > > +
> > > > > > +/**
> > > > > > + * struct ttm_backup_flags - Flags to govern backup
> > > > > > behaviour.
> > > > > > + * @purge: Free pages without backing up. Bypass pools.
> > > > > > + * @writeback: Attempt to copy contents directly to swap
> > > > > > space,
> > > > > > even
> > > > > > + * if that means blocking on writes to external memory.
> > > > > > + */
> > > > > > +struct ttm_backup_flags {
> > > > > > + u32 purge : 1;
> > > > > > + u32 writeback : 1;
> > > > > > +};
> > > > > > +
> > > > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt
> > > > > > *tt,
> > > > > > +            const struct ttm_backup_flags flags);
> > > > > > +
> > > > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt
> > > > > > *tt,
> > > > > > +            const struct ttm_operation_ctx *ctx);
> > > > > > +
> > > > > >   #if IS_ENABLED(CONFIG_AGP)
> > > > > >   #include <linux/agp_backend.h>
> > > > > > 
> 

Re: RESEND Re: [PATCH v16 2/7] drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages

[Thomas Hellström]

On Wed, 2025-03-05 at 13:01 +1000, Dave Airlie wrote:
> I've looked over the two patches mentioned here, I think they have
> seen enough time and we need to unblock,
> 
> Please add and merge them:
> Acked-by: Dave Airlie <airlied@redhat.com>
> 
> Dave.

Thanks for unblocking, Dave. Pushed to drm-misc-next yesterday.

/Thomas




> 
> On Tue, 25 Feb 2025 at 18:44, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> > 
> > Hi, Christian,
> > 
> > Ping? I'd really want to get this in before -rc6
> > 
> > Thanks,
> > Thomas
> > 
> > 
> > 
> > On Tue, 2025-02-18 at 16:40 +0100, Thomas Hellström wrote:
> > > Hi, Christian,
> > > 
> > > On Wed, 2025-02-05 at 15:02 +0100, Christian König wrote:
> > > > Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> > > > > Provide a helper to shrink ttm_tt page-vectors on a per-page
> > > > > basis. A ttm_backup backend could then in theory get away
> > > > > with
> > > > > allocating a single temporary page for each struct ttm_tt.
> > > > > 
> > > > > This is accomplished by splitting larger pages before trying
> > > > > to
> > > > > back them up.
> > > > > 
> > > > > In the future we could allow ttm_backup to handle backing up
> > > > > large pages as well, but currently there's no benefit in
> > > > > doing that, since the shmem backup backend would have to
> > > > > split those anyway to avoid allocating too much temporary
> > > > > memory, and if the backend instead inserts pages into the
> > > > > swap-cache, those are split on reclaim by the core.
> > > > > 
> > > > > Due to potential backup- and recover errors, allow partially
> > > > > swapped
> > > > > out struct ttm_tt's, although mark them as swapped out
> > > > > stopping
> > > > > them
> > > > > from being swapped out a second time. More details in the
> > > > > ttm_pool.c
> > > > > DOC section.
> > > > > 
> > > > > v2:
> > > > > - A couple of cleanups and error fixes in
> > > > > ttm_pool_back_up_tt.
> > > > > - s/back_up/backup/
> > > > > - Add a writeback parameter to the exported interface.
> > > > > v8:
> > > > > - Use a struct for flags for readability (Matt Brost)
> > > > > - Address misc other review comments (Matt Brost)
> > > > > v9:
> > > > > - Update the kerneldoc for the ttm_tt::backup field.
> > > > > v10:
> > > > > - Rebase.
> > > > > v13:
> > > > > - Rebase on ttm_backup interface change. Update kerneldoc.
> > > > > - Rebase and adjust ttm_tt_is_swapped().
> > > > > v15:
> > > > > - Rebase on ttm_backup return value change.
> > > > > - Rebase on previous restructuring of ttm_pool_alloc()
> > > > > - Rework the ttm_pool backup interface (Christian König)
> > > > > - Remove cond_resched() (Christian König)
> > > > > - Get rid of the need to allocate an intermediate page array
> > > > >    when restoring a multi-order page (Christian König)
> > > > > - Update documentation.
> > > > > 
> > > > > Cc: Christian König <christian.koenig@amd.com>
> > > > > Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> > > > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > > > Cc: <dri-devel@lists.freedesktop.org>
> > > > > Signed-off-by: Thomas Hellström
> > > > > <thomas.hellstrom@linux.intel.com>
> > > > > Reviewed-by: Matthew Brost <matthew.brost@intel.com>
> > > > 
> > > > I've tried to wrap my head around all of this like twenty times
> > > > in
> > > > the
> > > > last three month, but was always interrupted at some point.
> > > > 
> > > > Feel free to add Acked-by: Christian Koenig
> > > > <christian.koenig@amd.com>.
> > > > 
> > > > Sorry,
> > > > Christian.
> > > 
> > > Thanks a lot for all reviewing and comments so far. There are two
> > > TTM
> > > patches left in the series that don't have an ack by you:
> > > 
> > > https://patchwork.freedesktop.org/patch/634715/?series=131815&rev=17
> > > and
> > > 
> > > https://patchwork.freedesktop.org/patch/634716/?series=131815&rev=17
> > > 
> > > None of them particularly big considering the amount of doc text.
> > > 
> > > It'd be great if those could have an ack as well so we could
> > > finally
> > > merge this series.
> > > 
> > > Thanks,
> > > Thomas
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > > 
> > > > > ---
> > > > >   drivers/gpu/drm/ttm/ttm_pool.c | 554
> > > > > +++++++++++++++++++++++++++++----
> > > > >   drivers/gpu/drm/ttm/ttm_tt.c   |  54 ++++
> > > > >   include/drm/ttm/ttm_pool.h     |   8 +
> > > > >   include/drm/ttm/ttm_tt.h       |  67 +++-
> > > > >   4 files changed, 629 insertions(+), 54 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > index c9eba76d5143..ffb7abf52bab 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> > > > > @@ -41,6 +41,7 @@
> > > > >   #include <asm/set_memory.h>
> > > > >   #endif
> > > > > 
> > > > > +#include <drm/ttm/ttm_backup.h>
> > > > >   #include <drm/ttm/ttm_pool.h>
> > > > >   #include <drm/ttm/ttm_tt.h>
> > > > >   #include <drm/ttm/ttm_bo.h>
> > > > > @@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {
> > > > >           enum ttm_caching tt_caching;
> > > > >   };
> > > > > 
> > > > > +/**
> > > > > + * struct ttm_pool_tt_restore - State representing restore
> > > > > from
> > > > > backup
> > > > > + * @pool: The pool used for page allocation while restoring.
> > > > > + * @snapshot_alloc: A snapshot of the most recent struct
> > > > > ttm_pool_alloc_state.
> > > > > + * @alloced_page: Pointer to the page most recently
> > > > > allocated
> > > > > from
> > > > > a pool or system.
> > > > > + * @first_dma: The dma address corresponding to
> > > > > @alloced_page if
> > > > > dma_mapping
> > > > > + * is requested.
> > > > > + * @alloced_pages: The number of allocated pages present in
> > > > > the
> > > > > struct ttm_tt
> > > > > + * page vector from this restore session.
> > > > > + * @restored_pages: The number of 4K pages restored for
> > > > > @alloced_page (which
> > > > > + * is typically a multi-order page).
> > > > > + * @page_caching: The struct ttm_tt requested caching
> > > > > + * @order: The order of @alloced_page.
> > > > > + *
> > > > > + * Recovery from backup might fail when we've recovered less
> > > > > than
> > > > > the
> > > > > + * full ttm_tt. In order not to loose any data (yet), keep
> > > > > information
> > > > > + * around that allows us to restart a failed ttm backup
> > > > > recovery.
> > > > > + */
> > > > > +struct ttm_pool_tt_restore {
> > > > > + struct ttm_pool *pool;
> > > > > + struct ttm_pool_alloc_state snapshot_alloc;
> > > > > + struct page *alloced_page;
> > > > > + dma_addr_t first_dma;
> > > > > + pgoff_t alloced_pages;
> > > > > + pgoff_t restored_pages;
> > > > > + enum ttm_caching page_caching;
> > > > > + unsigned int order;
> > > > > +};
> > > > > +
> > > > >   static unsigned long page_pool_size;
> > > > > 
> > > > >   MODULE_PARM_DESC(page_pool_size, "Number of pages in the
> > > > > WC/UC/DMA pool");
> > > > > @@ -199,12 +229,11 @@ static int
> > > > > ttm_pool_apply_caching(struct
> > > > > ttm_pool_alloc_state *alloc)
> > > > >           return 0;
> > > > >   }
> > > > > 
> > > > > -/* Map pages of 1 << order size and fill the DMA address
> > > > > array
> > > > > */
> > > > > +/* DMA Map pages of 1 << order size and return the resulting
> > > > > dma_address. */
> > > > >   static int ttm_pool_map(struct ttm_pool *pool, unsigned int
> > > > > order,
> > > > > -                 struct page *p, dma_addr_t **dma_addr)
> > > > > +                 struct page *p, dma_addr_t *dma_addr)
> > > > >   {
> > > > >           dma_addr_t addr;
> > > > > - unsigned int i;
> > > > > 
> > > > >           if (pool->use_dma_alloc) {
> > > > >                   struct ttm_pool_dma *dma = (void *)p-
> > > > > >private;
> > > > > @@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool
> > > > > *pool,
> > > > > unsigned int order,
> > > > >                           return -EFAULT;
> > > > >           }
> > > > > 
> > > > > - for (i = 1 << order; i ; --i) {
> > > > > -         *(*dma_addr)++ = addr;
> > > > > -         addr += PAGE_SIZE;
> > > > > - }
> > > > > + *dma_addr = addr;
> > > > > 
> > > > >           return 0;
> > > > >   }
> > > > > @@ -371,6 +397,190 @@ static unsigned int
> > > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > > >           return p->private;
> > > > >   }
> > > > > 
> > > > > +/*
> > > > > + * Split larger pages so that we can free each PAGE_SIZE
> > > > > page as
> > > > > soon
> > > > > + * as it has been backed up, in order to avoid memory
> > > > > pressure
> > > > > during
> > > > > + * reclaim.
> > > > > + */
> > > > > +static void ttm_pool_split_for_swap(struct ttm_pool *pool,
> > > > > struct
> > > > > page *p)
> > > > > +{
> > > > > + unsigned int order = ttm_pool_page_order(pool, p);
> > > > > + pgoff_t nr;
> > > > > +
> > > > > + if (!order)
> > > > > +         return;
> > > > > +
> > > > > + split_page(p, order);
> > > > > + nr = 1UL << order;
> > > > > + while (nr--)
> > > > > +         (p++)->private = 0;
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * DOC: Partial backup and restoration of a struct ttm_tt.
> > > > > + *
> > > > > + * Swapout using ttm_backup_backup_page() and swapin using
> > > > > + * ttm_backup_copy_page() may fail.
> > > > > + * The former most likely due to lack of swap-space or
> > > > > memory,
> > > > > the
> > > > > latter due
> > > > > + * to lack of memory or because of signal interruption
> > > > > during
> > > > > waits.
> > > > > + *
> > > > > + * Backup failure is easily handled by using a ttm_tt pages
> > > > > vector
> > > > > that holds
> > > > > + * both backup handles and page pointers. This has to be
> > > > > taken
> > > > > into account when
> > > > > + * restoring such a ttm_tt from backup, and when freeing it
> > > > > while
> > > > > backed up.
> > > > > + * When restoring, for simplicity, new pages are actually
> > > > > allocated from the
> > > > > + * pool and the contents of any old pages are copied in and
> > > > > then
> > > > > the old pages
> > > > > + * are released.
> > > > > + *
> > > > > + * For restoration failures, the struct ttm_pool_tt_restore
> > > > > holds
> > > > > sufficient state
> > > > > + * to be able to resume an interrupted restore, and that
> > > > > structure
> > > > > is freed once
> > > > > + * the restoration is complete. If the struct ttm_tt is
> > > > > destroyed
> > > > > while there
> > > > > + * is a valid struct ttm_pool_tt_restore attached, that is
> > > > > also
> > > > > properly taken
> > > > > + * care of.
> > > > > + */
> > > > > +
> > > > > +/* Is restore ongoing for the currently allocated page? */
> > > > > +static bool ttm_pool_restore_valid(const struct
> > > > > ttm_pool_tt_restore *restore)
> > > > > +{
> > > > > + return restore && restore->restored_pages < (1 <<
> > > > > restore-
> > > > > > order);
> > > > > +}
> > > > > +
> > > > > +/* DMA unmap and free a multi-order page, either to the
> > > > > relevant
> > > > > pool or to system. */
> > > > > +static pgoff_t ttm_pool_unmap_and_free(struct ttm_pool
> > > > > *pool,
> > > > > struct page *page,
> > > > > +                                const dma_addr_t
> > > > > *dma_addr,
> > > > > enum ttm_caching caching)
> > > > > +{
> > > > > + struct ttm_pool_type *pt = NULL;
> > > > > + unsigned int order;
> > > > > + pgoff_t nr;
> > > > > +
> > > > > + if (pool) {
> > > > > +         order = ttm_pool_page_order(pool, page);
> > > > > +         nr = (1UL << order);
> > > > > +         if (dma_addr)
> > > > > +                 ttm_pool_unmap(pool, *dma_addr, nr);
> > > > > +
> > > > > +         pt = ttm_pool_select_type(pool, caching, order);
> > > > > + } else {
> > > > > +         order = page->private;
> > > > > +         nr = (1UL << order);
> > > > > + }
> > > > > +
> > > > > + if (pt)
> > > > > +         ttm_pool_type_give(pt, page);
> > > > > + else
> > > > > +         ttm_pool_free_page(pool, caching, order, page);
> > > > > +
> > > > > + return nr;
> > > > > +}
> > > > > +
> > > > > +/* Populate the page-array using the most recent allocated
> > > > > multi-
> > > > > order page. */
> > > > > +static void ttm_pool_allocated_page_commit(struct page
> > > > > *allocated,
> > > > > +                                    dma_addr_t first_dma,
> > > > > +                                    struct
> > > > > ttm_pool_alloc_state *alloc,
> > > > > +                                    pgoff_t nr)
> > > > > +{
> > > > > + pgoff_t i;
> > > > > +
> > > > > + for (i = 0; i < nr; ++i)
> > > > > +         *alloc->pages++ = allocated++;
> > > > > +
> > > > > + alloc->remaining_pages -= nr;
> > > > > +
> > > > > + if (!alloc->dma_addr)
> > > > > +         return;
> > > > > +
> > > > > + for (i = 0; i < nr; ++i) {
> > > > > +         *alloc->dma_addr++ = first_dma;
> > > > > +         first_dma += PAGE_SIZE;
> > > > > + }
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * When restoring, restore backed-up content to the newly
> > > > > allocated page and
> > > > > + * if successful, populate the page-table and dma-address
> > > > > arrays.
> > > > > + */
> > > > > +static int ttm_pool_restore_commit(struct
> > > > > ttm_pool_tt_restore
> > > > > *restore,
> > > > > +                            struct ttm_backup *backup,
> > > > > +                            const struct
> > > > > ttm_operation_ctx
> > > > > *ctx,
> > > > > +                            struct ttm_pool_alloc_state
> > > > > *alloc)
> > > > > +
> > > > > +{
> > > > > + pgoff_t i, nr = 1UL << restore->order;
> > > > > + struct page **first_page = alloc->pages;
> > > > > + struct page *p;
> > > > > + int ret = 0;
> > > > > +
> > > > > + for (i = restore->restored_pages; i < nr; ++i) {
> > > > > +         p = first_page[i];
> > > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > > +                 unsigned long handle =
> > > > > ttm_backup_page_ptr_to_handle(p);
> > > > > +
> > > > > +                 if (handle == 0) {
> > > > > +                         restore->restored_pages++;
> > > > > +                         continue;
> > > > > +                 }
> > > > > +
> > > > > +                 ret = ttm_backup_copy_page(backup,
> > > > > restore->alloced_page + i,
> > > > > +                                            handle, ctx-
> > > > > > interruptible);
> > > > > +                 if (ret)
> > > > > +                         break;
> > > > > +
> > > > > +                 ttm_backup_drop(backup, handle);
> > > > > +         } else if (p) {
> > > > > +                 /*
> > > > > +                  * We could probably avoid splitting the
> > > > > old page
> > > > > +                  * using clever logic, but ATM we don't
> > > > > care, as
> > > > > +                  * we prioritize releasing memory ASAP.
> > > > > Note that
> > > > > +                  * here, the old retained page is always
> > > > > write-back
> > > > > +                  * cached.
> > > > > +                  */
> > > > > +                 ttm_pool_split_for_swap(restore->pool,
> > > > > p);
> > > > > +                 copy_highpage(restore->alloced_page + i,
> > > > > p);
> > > > > +                 __free_pages(p, 0);
> > > > > +         }
> > > > > +
> > > > > +         restore->restored_pages++;
> > > > > +         first_page[i] =
> > > > > ttm_backup_handle_to_page_ptr(0);
> > > > > + }
> > > > > +
> > > > > + if (ret) {
> > > > > +         if (!restore->restored_pages) {
> > > > > +                 dma_addr_t *dma_addr = alloc->dma_addr ?
> > > > > &restore->first_dma : NULL;
> > > > > +
> > > > > +                 ttm_pool_unmap_and_free(restore->pool,
> > > > > restore->alloced_page,
> > > > > +                                         dma_addr,
> > > > > restore-
> > > > > > page_caching);
> > > > > +                 restore->restored_pages = nr;
> > > > > +         }
> > > > > +         return ret;
> > > > > + }
> > > > > +
> > > > > + ttm_pool_allocated_page_commit(restore->alloced_page,
> > > > > restore->first_dma,
> > > > > +                                alloc, nr);
> > > > > + if (restore->page_caching == alloc->tt_caching ||
> > > > > PageHighMem(restore->alloced_page))
> > > > > +         alloc->caching_divide = alloc->pages;
> > > > > + restore->snapshot_alloc = *alloc;
> > > > > + restore->alloced_pages += nr;
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +
> > > > > +/* If restoring, save information needed for
> > > > > ttm_pool_restore_commit(). */
> > > > > +static void
> > > > > +ttm_pool_page_allocated_restore(struct ttm_pool *pool,
> > > > > unsigned
> > > > > int order,
> > > > > +                         struct page *p,
> > > > > +                         enum ttm_caching page_caching,
> > > > > +                         dma_addr_t first_dma,
> > > > > +                         struct ttm_pool_tt_restore
> > > > > *restore,
> > > > > +                         const struct
> > > > > ttm_pool_alloc_state
> > > > > *alloc)
> > > > > +{
> > > > > + restore->pool = pool;
> > > > > + restore->order = order;
> > > > > + restore->restored_pages = 0;
> > > > > + restore->page_caching = page_caching;
> > > > > + restore->first_dma = first_dma;
> > > > > + restore->alloced_page = p;
> > > > > + restore->snapshot_alloc = *alloc;
> > > > > +}
> > > > > +
> > > > >   /*
> > > > >    * Called when we got a page, either from a pool or newly
> > > > > allocated.
> > > > >    * if needed, dma map the page and populate the dma address
> > > > > array.
> > > > > @@ -380,10 +590,11 @@ static unsigned int
> > > > > ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
> > > > >    */
> > > > >   static int ttm_pool_page_allocated(struct ttm_pool *pool,
> > > > > unsigned int order,
> > > > >                                      struct page *p, enum
> > > > > ttm_caching page_caching,
> > > > > -                            struct ttm_pool_alloc_state
> > > > > *alloc)
> > > > > +                            struct ttm_pool_alloc_state
> > > > > *alloc,
> > > > > +                            struct ttm_pool_tt_restore
> > > > > *restore)
> > > > >   {
> > > > > - pgoff_t i, nr = 1UL << order;
> > > > >           bool caching_consistent;
> > > > > + dma_addr_t first_dma;
> > > > >           int r = 0;
> > > > > 
> > > > >           caching_consistent = (page_caching == alloc-
> > > > > >tt_caching)
> > > > > > > PageHighMem(p);
> > > > > @@ -395,17 +606,20 @@ static int
> > > > > ttm_pool_page_allocated(struct
> > > > > ttm_pool *pool, unsigned int order,
> > > > >           }
> > > > > 
> > > > >           if (alloc->dma_addr) {
> > > > > -         r = ttm_pool_map(pool, order, p, &alloc-
> > > > > > dma_addr);
> > > > > +         r = ttm_pool_map(pool, order, p, &first_dma);
> > > > >                   if (r)
> > > > >                           return r;
> > > > >           }
> > > > > 
> > > > > - alloc->remaining_pages -= nr;
> > > > > - for (i = 0; i < nr; ++i)
> > > > > -         *alloc->pages++ = p++;
> > > > > + if (restore) {
> > > > > +         ttm_pool_page_allocated_restore(pool, order, p,
> > > > > page_caching,
> > > > > +                                         first_dma,
> > > > > restore, alloc);
> > > > > + } else {
> > > > > +         ttm_pool_allocated_page_commit(p, first_dma,
> > > > > alloc, 1UL << order);
> > > > > 
> > > > > - if (caching_consistent)
> > > > > -         alloc->caching_divide = alloc->pages;
> > > > > +         if (caching_consistent)
> > > > > +                 alloc->caching_divide = alloc->pages;
> > > > > + }
> > > > > 
> > > > >           return 0;
> > > > >   }
> > > > > @@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct
> > > > > ttm_pool *pool, struct ttm_tt *tt,
> > > > >                                   pgoff_t start_page, pgoff_t
> > > > > end_page)
> > > > >   {
> > > > >           struct page **pages = &tt->pages[start_page];
> > > > > - unsigned int order;
> > > > > + struct ttm_backup *backup = tt->backup;
> > > > >           pgoff_t i, nr;
> > > > > 
> > > > >           for (i = start_page; i < end_page; i += nr, pages
> > > > > += nr)
> > > > > {
> > > > > -         struct ttm_pool_type *pt = NULL;
> > > > > +         struct page *p = *pages;
> > > > > 
> > > > > -         order = ttm_pool_page_order(pool, *pages);
> > > > > -         nr = (1UL << order);
> > > > > -         if (tt->dma_address)
> > > > > -                 ttm_pool_unmap(pool, tt->dma_address[i],
> > > > > nr);
> > > > > +         nr = 1;
> > > > > +         if (ttm_backup_page_ptr_is_handle(p)) {
> > > > > +                 unsigned long handle =
> > > > > ttm_backup_page_ptr_to_handle(p);
> > > > > 
> > > > > -         pt = ttm_pool_select_type(pool, caching, order);
> > > > > -         if (pt)
> > > > > -                 ttm_pool_type_give(pt, *pages);
> > > > > -         else
> > > > > -                 ttm_pool_free_page(pool, caching, order,
> > > > > *pages);
> > > > > +                 if (handle != 0)
> > > > > +                         ttm_backup_drop(backup, handle);
> > > > > +         } else if (p) {
> > > > > +                 dma_addr_t *dma_addr = tt->dma_address ?
> > > > > +                         tt->dma_address + i : NULL;
> > > > > +
> > > > > +                 nr = ttm_pool_unmap_and_free(pool, p,
> > > > > dma_addr, caching);
> > > > > +         }
> > > > >           }
> > > > >   }
> > > > > 
> > > > > @@ -467,22 +683,11 @@ static unsigned int
> > > > > ttm_pool_alloc_find_order(unsigned int highest,
> > > > >           return min_t(unsigned int, highest, __fls(alloc-
> > > > > > remaining_pages));
> > > > >   }
> > > > > 
> > > > > -/**
> > > > > - * ttm_pool_alloc - Fill a ttm_tt object
> > > > > - *
> > > > > - * @pool: ttm_pool to use
> > > > > - * @tt: ttm_tt object to fill
> > > > > - * @ctx: operation context
> > > > > - *
> > > > > - * Fill the ttm_tt object with pages and also make sure to
> > > > > DMA
> > > > > map
> > > > > them when
> > > > > - * necessary.
> > > > > - *
> > > > > - * Returns: 0 on successe, negative error code otherwise.
> > > > > - */
> > > > > -int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > > > -            struct ttm_operation_ctx *ctx)
> > > > > +static int __ttm_pool_alloc(struct ttm_pool *pool, struct
> > > > > ttm_tt
> > > > > *tt,
> > > > > +                     const struct ttm_operation_ctx *ctx,
> > > > > +                     struct ttm_pool_alloc_state *alloc,
> > > > > +                     struct ttm_pool_tt_restore *restore)
> > > > >   {
> > > > > - struct ttm_pool_alloc_state alloc;
> > > > >           enum ttm_caching page_caching;
> > > > >           gfp_t gfp_flags = GFP_USER;
> > > > >           pgoff_t caching_divide;
> > > > > @@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool
> > > > > *pool,
> > > > > struct ttm_tt *tt,
> > > > >           struct page *p;
> > > > >           int r;
> > > > > 
> > > > > - ttm_pool_alloc_state_init(tt, &alloc);
> > > > > -
> > > > > - WARN_ON(!alloc.remaining_pages ||
> > > > > ttm_tt_is_populated(tt));
> > > > > - WARN_ON(alloc.dma_addr && !pool->dev);
> > > > > + WARN_ON(!alloc->remaining_pages ||
> > > > > ttm_tt_is_populated(tt));
> > > > > + WARN_ON(alloc->dma_addr && !pool->dev);
> > > > > 
> > > > >           if (tt->page_flags & TTM_TT_FLAG_ZERO_ALLOC)
> > > > >                   gfp_flags |= __GFP_ZERO;
> > > > > @@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool,
> > > > > struct ttm_tt *tt,
> > > > > 
> > > > >           page_caching = tt->caching;
> > > > >           allow_pools = true;
> > > > > - for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > > &alloc);
> > > > > -      alloc.remaining_pages;
> > > > > -      order = ttm_pool_alloc_find_order(order, &alloc)) {
> > > > > + for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER,
> > > > > alloc);
> > > > > +      alloc->remaining_pages;
> > > > > +      order = ttm_pool_alloc_find_order(order, alloc)) {
> > > > >                   struct ttm_pool_type *pt;
> > > > > 
> > > > >                   /* First, try to allocate a page from a
> > > > > pool if
> > > > > one exists. */
> > > > > @@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool
> > > > > *pool,
> > > > > struct ttm_tt *tt,
> > > > >                           r = -ENOMEM;
> > > > >                           goto error_free_all;
> > > > >                   }
> > > > > -         r = ttm_pool_page_allocated(pool, order, p,
> > > > > page_caching, &alloc);
> > > > > +         r = ttm_pool_page_allocated(pool, order, p,
> > > > > page_caching, alloc,
> > > > > +                                     restore);
> > > > >                   if (r)
> > > > >                           goto error_free_page;
> > > > > +
> > > > > +         if (ttm_pool_restore_valid(restore)) {
> > > > > +                 r = ttm_pool_restore_commit(restore, tt-
> > > > > > backup, ctx, alloc);
> > > > > +                 if (r)
> > > > > +                         goto error_free_all;
> > > > > +         }
> > > > >           }
> > > > > 
> > > > > - r = ttm_pool_apply_caching(&alloc);
> > > > > + r = ttm_pool_apply_caching(alloc);
> > > > >           if (r)
> > > > >                   goto error_free_all;
> > > > > 
> > > > > + kfree(tt->restore);
> > > > > + tt->restore = NULL;
> > > > > +
> > > > >           return 0;
> > > > > 
> > > > >   error_free_page:
> > > > >           ttm_pool_free_page(pool, page_caching, order, p);
> > > > > 
> > > > >   error_free_all:
> > > > > - caching_divide = alloc.caching_divide - tt->pages;
> > > > > + if (tt->restore)
> > > > > +         return r;
> > > > > +
> > > > > + caching_divide = alloc->caching_divide - tt->pages;
> > > > >           ttm_pool_free_range(pool, tt, tt->caching, 0,
> > > > > caching_divide);
> > > > >           ttm_pool_free_range(pool, tt, ttm_cached,
> > > > > caching_divide,
> > > > > -                     tt->num_pages -
> > > > > alloc.remaining_pages);
> > > > > +                     tt->num_pages - alloc-
> > > > > > remaining_pages);
> > > > > 
> > > > >           return r;
> > > > >   }
> > > > > +
> > > > > +/**
> > > > > + * ttm_pool_alloc - Fill a ttm_tt object
> > > > > + *
> > > > > + * @pool: ttm_pool to use
> > > > > + * @tt: ttm_tt object to fill
> > > > > + * @ctx: operation context
> > > > > + *
> > > > > + * Fill the ttm_tt object with pages and also make sure to
> > > > > DMA
> > > > > map
> > > > > them when
> > > > > + * necessary.
> > > > > + *
> > > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > > + */
> > > > > +int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
> > > > > +            struct ttm_operation_ctx *ctx)
> > > > > +{
> > > > > + struct ttm_pool_alloc_state alloc;
> > > > > +
> > > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > > +         return -EINVAL;
> > > > > +
> > > > > + ttm_pool_alloc_state_init(tt, &alloc);
> > > > > +
> > > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
> > > > > +}
> > > > >   EXPORT_SYMBOL(ttm_pool_alloc);
> > > > > 
> > > > > +/**
> > > > > + * ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring
> > > > > previously backed-up
> > > > > + * content.
> > > > > + *
> > > > > + * @pool: ttm_pool to use
> > > > > + * @tt: ttm_tt object to fill
> > > > > + * @ctx: operation context
> > > > > + *
> > > > > + * Fill the ttm_tt object with pages and also make sure to
> > > > > DMA
> > > > > map
> > > > > them when
> > > > > + * necessary. Read in backed-up content.
> > > > > + *
> > > > > + * Returns: 0 on successe, negative error code otherwise.
> > > > > + */
> > > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > > > ttm_tt *tt,
> > > > > +                        const struct ttm_operation_ctx
> > > > > *ctx)
> > > > > +{
> > > > > + struct ttm_pool_alloc_state alloc;
> > > > > +
> > > > > + if (WARN_ON(!ttm_tt_is_backed_up(tt)))
> > > > > +         return -EINVAL;
> > > > > +
> > > > > + if (!tt->restore) {
> > > > > +         gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
> > > > > +
> > > > > +         ttm_pool_alloc_state_init(tt, &alloc);
> > > > > +         if (ctx->gfp_retry_mayfail)
> > > > > +                 gfp |= __GFP_RETRY_MAYFAIL;
> > > > > +
> > > > > +         tt->restore = kzalloc(sizeof(*tt->restore),
> > > > > gfp);
> > > > > +         if (!tt->restore)
> > > > > +                 return -ENOMEM;
> > > > > +
> > > > > +         tt->restore->snapshot_alloc = alloc;
> > > > > +         tt->restore->pool = pool;
> > > > > +         tt->restore->restored_pages = 1;
> > > > > + } else {
> > > > > +         struct ttm_pool_tt_restore *restore = tt-
> > > > > > restore;
> > > > > +         int ret;
> > > > > +
> > > > > +         alloc = restore->snapshot_alloc;
> > > > > +         if (ttm_pool_restore_valid(tt->restore)) {
> > > > > +                 ret = ttm_pool_restore_commit(restore,
> > > > > tt-
> > > > > > backup, ctx, &alloc);
> > > > > +                 if (ret)
> > > > > +                         return ret;
> > > > > +         }
> > > > > +         if (!alloc.remaining_pages)
> > > > > +                 return 0;
> > > > > + }
> > > > > +
> > > > > + return __ttm_pool_alloc(pool, tt, ctx, &alloc, tt-
> > > > > > restore);
> > > > > +}
> > > > > +
> > > > >   /**
> > > > >    * ttm_pool_free - Free the backing pages from a ttm_tt
> > > > > object
> > > > >    *
> > > > > @@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool
> > > > > *pool,
> > > > > struct ttm_tt *tt)
> > > > >   }
> > > > >   EXPORT_SYMBOL(ttm_pool_free);
> > > > > 
> > > > > +/**
> > > > > + * ttm_pool_drop_backed_up() - Release content of a swapped-
> > > > > out
> > > > > struct ttm_tt
> > > > > + * @tt: The struct ttm_tt.
> > > > > + *
> > > > > + * Release handles with associated content or any remaining
> > > > > pages
> > > > > of
> > > > > + * a backed-up struct ttm_tt.
> > > > > + */
> > > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt)
> > > > > +{
> > > > > + struct ttm_pool_tt_restore *restore;
> > > > > + pgoff_t start_page = 0;
> > > > > +
> > > > > + WARN_ON(!ttm_tt_is_backed_up(tt));
> > > > > +
> > > > > + restore = tt->restore;
> > > > > +
> > > > > + /*
> > > > > +  * Unmap and free any uncommitted restore page.
> > > > > +  * any tt page-array backup entries already read back
> > > > > has
> > > > > +  * been cleared already
> > > > > +  */
> > > > > + if (ttm_pool_restore_valid(restore)) {
> > > > > +         dma_addr_t *dma_addr = tt->dma_address ?
> > > > > &restore-
> > > > > > first_dma : NULL;
> > > > > +
> > > > > +         ttm_pool_unmap_and_free(restore->pool, restore-
> > > > > > alloced_page,
> > > > > +                                 dma_addr, restore-
> > > > > > page_caching);
> > > > > +         restore->restored_pages = 1UL << restore->order;
> > > > > + }
> > > > > +
> > > > > + /*
> > > > > +  * If a restore is ongoing, part of the tt pages may
> > > > > have
> > > > > a
> > > > > +  * caching different than writeback.
> > > > > +  */
> > > > > + if (restore) {
> > > > > +         pgoff_t mid = restore-
> > > > > > snapshot_alloc.caching_divide - tt->pages;
> > > > > +
> > > > > +         start_page = restore->alloced_pages;
> > > > > +         WARN_ON(mid > start_page);
> > > > > +         /* Pages that might be dma-mapped and non-cached
> > > > > */
> > > > > +         ttm_pool_free_range(restore->pool, tt, tt-
> > > > > > caching,
> > > > > +                             0, mid);
> > > > > +         /* Pages that might be dma-mapped but cached */
> > > > > +         ttm_pool_free_range(restore->pool, tt,
> > > > > ttm_cached,
> > > > > +                             mid, restore-
> > > > > > alloced_pages);
> > > > > +         kfree(restore);
> > > > > +         tt->restore = NULL;
> > > > > + }
> > > > > +
> > > > > + ttm_pool_free_range(NULL, tt, ttm_cached, start_page,
> > > > > tt-
> > > > > > num_pages);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * ttm_pool_backup() - Back up or purge a struct ttm_tt
> > > > > + * @pool: The pool used when allocating the struct ttm_tt.
> > > > > + * @tt: The struct ttm_tt.
> > > > > + * @flags: Flags to govern the backup behaviour.
> > > > > + *
> > > > > + * Back up or purge a struct ttm_tt. If @purge is true, then
> > > > > + * all pages will be freed directly to the system rather
> > > > > than to
> > > > > the pool
> > > > > + * they were allocated from, making the function behave
> > > > > similarly
> > > > > to
> > > > > + * ttm_pool_free(). If @purge is false the pages will be
> > > > > backed
> > > > > up
> > > > > instead,
> > > > > + * exchanged for handles.
> > > > > + * A subsequent call to ttm_pool_restore_and_alloc() will
> > > > > then
> > > > > read back the content and
> > > > > + * a subsequent call to ttm_pool_drop_backed_up() will drop
> > > > > it.
> > > > > + * If backup of a page fails for whatever reason, @ttm will
> > > > > still
> > > > > be
> > > > > + * partially backed up, retaining those pages for which
> > > > > backup
> > > > > fails.
> > > > > + * In that case, this function can be retried, possibly
> > > > > after
> > > > > freeing up
> > > > > + * memory resources.
> > > > > + *
> > > > > + * Return: Number of pages actually backed up or freed, or
> > > > > negative
> > > > > + * error code on error.
> > > > > + */
> > > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt
> > > > > *tt,
> > > > > +              const struct ttm_backup_flags *flags)
> > > > > +{
> > > > > + struct ttm_backup *backup = tt->backup;
> > > > > + struct page *page;
> > > > > + unsigned long handle;
> > > > > + gfp_t alloc_gfp;
> > > > > + gfp_t gfp;
> > > > > + int ret = 0;
> > > > > + pgoff_t shrunken = 0;
> > > > > + pgoff_t i, num_pages;
> > > > > +
> > > > > + if (WARN_ON(ttm_tt_is_backed_up(tt)))
> > > > > +         return -EINVAL;
> > > > > +
> > > > > + if ((!ttm_backup_bytes_avail() && !flags->purge) ||
> > > > > +     pool->use_dma_alloc || ttm_tt_is_backed_up(tt))
> > > > > +         return -EBUSY;
> > > > > +
> > > > > +#ifdef CONFIG_X86
> > > > > + /* Anything returned to the system needs to be cached.
> > > > > */
> > > > > + if (tt->caching != ttm_cached)
> > > > > +         set_pages_array_wb(tt->pages, tt->num_pages);
> > > > > +#endif
> > > > > +
> > > > > + if (tt->dma_address || flags->purge) {
> > > > > +         for (i = 0; i < tt->num_pages; i += num_pages) {
> > > > > +                 unsigned int order;
> > > > > +
> > > > > +                 page = tt->pages[i];
> > > > > +                 if (unlikely(!page)) {
> > > > > +                         num_pages = 1;
> > > > > +                         continue;
> > > > > +                 }
> > > > > +
> > > > > +                 order = ttm_pool_page_order(pool, page);
> > > > > +                 num_pages = 1UL << order;
> > > > > +                 if (tt->dma_address)
> > > > > +                         ttm_pool_unmap(pool, tt-
> > > > > > dma_address[i],
> > > > > +                                        num_pages);
> > > > > +                 if (flags->purge) {
> > > > > +                         shrunken += num_pages;
> > > > > +                         page->private = 0;
> > > > > +                         __free_pages(page, order);
> > > > > +                         memset(tt->pages + i, 0,
> > > > > +                                num_pages * sizeof(*tt-
> > > > > > pages));
> > > > > +                 }
> > > > > +         }
> > > > > + }
> > > > > +
> > > > > + if (flags->purge)
> > > > > +         return shrunken;
> > > > > +
> > > > > + if (pool->use_dma32)
> > > > > +         gfp = GFP_DMA32;
> > > > > + else
> > > > > +         gfp = GFP_HIGHUSER;
> > > > > +
> > > > > + alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |
> > > > > __GFP_RETRY_MAYFAIL;
> > > > > +
> > > > > + for (i = 0; i < tt->num_pages; ++i) {
> > > > > +         s64 shandle;
> > > > > +
> > > > > +         page = tt->pages[i];
> > > > > +         if (unlikely(!page))
> > > > > +                 continue;
> > > > > +
> > > > > +         ttm_pool_split_for_swap(pool, page);
> > > > > +
> > > > > +         shandle = ttm_backup_backup_page(backup, page,
> > > > > flags->writeback, i,
> > > > > +                                          gfp,
> > > > > alloc_gfp);
> > > > > +         if (shandle < 0) {
> > > > > +                 /* We allow partially shrunken tts */
> > > > > +                 ret = shandle;
> > > > > +                 break;
> > > > > +         }
> > > > > +         handle = shandle;
> > > > > +         tt->pages[i] =
> > > > > ttm_backup_handle_to_page_ptr(handle);
> > > > > +         put_page(page);
> > > > > +         shrunken++;
> > > > > + }
> > > > > +
> > > > > + return shrunken ? shrunken : ret;
> > > > > +}
> > > > > +
> > > > >   /**
> > > > >    * ttm_pool_init - Initialize a pool
> > > > >    *
> > > > > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > index 3baf215eca23..00b7c28f2329 100644
> > > > > --- a/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> > > > > @@ -40,6 +40,7 @@
> > > > >   #include <drm/drm_cache.h>
> > > > >   #include <drm/drm_device.h>
> > > > >   #include <drm/drm_util.h>
> > > > > +#include <drm/ttm/ttm_backup.h>
> > > > >   #include <drm/ttm/ttm_bo.h>
> > > > >   #include <drm/ttm/ttm_tt.h>
> > > > > 
> > > > > @@ -158,6 +159,8 @@ static void ttm_tt_init_fields(struct
> > > > > ttm_tt
> > > > > *ttm,
> > > > >           ttm->swap_storage = NULL;
> > > > >           ttm->sg = bo->sg;
> > > > >           ttm->caching = caching;
> > > > > + ttm->restore = NULL;
> > > > > + ttm->backup = NULL;
> > > > >   }
> > > > > 
> > > > >   int ttm_tt_init(struct ttm_tt *ttm, struct
> > > > > ttm_buffer_object
> > > > > *bo,
> > > > > @@ -182,6 +185,13 @@ void ttm_tt_fini(struct ttm_tt *ttm)
> > > > >                   fput(ttm->swap_storage);
> > > > >           ttm->swap_storage = NULL;
> > > > > 
> > > > > + if (ttm_tt_is_backed_up(ttm))
> > > > > +         ttm_pool_drop_backed_up(ttm);
> > > > > + if (ttm->backup) {
> > > > > +         ttm_backup_fini(ttm->backup);
> > > > > +         ttm->backup = NULL;
> > > > > + }
> > > > > +
> > > > >           if (ttm->pages)
> > > > >                   kvfree(ttm->pages);
> > > > >           else
> > > > > @@ -253,6 +263,49 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
> > > > >   }
> > > > >   EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
> > > > > 
> > > > > +/**
> > > > > + * ttm_tt_backup() - Helper to back up a struct ttm_tt.
> > > > > + * @bdev: The TTM device.
> > > > > + * @tt: The struct ttm_tt.
> > > > > + * @flags: Flags that govern the backup behaviour.
> > > > > + *
> > > > > + * Update the page accounting and call ttm_pool_shrink_tt to
> > > > > free
> > > > > pages
> > > > > + * or back them up.
> > > > > + *
> > > > > + * Return: Number of pages freed or swapped out, or negative
> > > > > error
> > > > > code on
> > > > > + * error.
> > > > > + */
> > > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt
> > > > > *tt,
> > > > > +            const struct ttm_backup_flags flags)
> > > > > +{
> > > > > + long ret;
> > > > > +
> > > > > + if (WARN_ON(IS_ERR_OR_NULL(tt->backup)))
> > > > > +         return 0;
> > > > > +
> > > > > + ret = ttm_pool_backup(&bdev->pool, tt, &flags);
> > > > > + if (ret > 0) {
> > > > > +         tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
> > > > > +         tt->page_flags |= TTM_TT_FLAG_BACKED_UP;
> > > > > + }
> > > > > +
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt
> > > > > *tt,
> > > > > +            const struct ttm_operation_ctx *ctx)
> > > > > +{
> > > > > + int ret = ttm_pool_restore_and_alloc(&bdev->pool, tt,
> > > > > ctx);
> > > > > +
> > > > > + if (ret)
> > > > > +         return ret;
> > > > > +
> > > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > > +
> > > > > + return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL(ttm_tt_restore);
> > > > > +
> > > > >   /**
> > > > >    * ttm_tt_swapout - swap out tt object
> > > > >    *
> > > > > @@ -348,6 +401,7 @@ int ttm_tt_populate(struct ttm_device
> > > > > *bdev,
> > > > >                   goto error;
> > > > > 
> > > > >           ttm->page_flags |= TTM_TT_FLAG_PRIV_POPULATED;
> > > > > + ttm->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > >           if (unlikely(ttm->page_flags &
> > > > > TTM_TT_FLAG_SWAPPED)) {
> > > > >                   ret = ttm_tt_swapin(ttm);
> > > > >                   if (unlikely(ret != 0)) {
> > > > > diff --git a/include/drm/ttm/ttm_pool.h
> > > > > b/include/drm/ttm/ttm_pool.h
> > > > > index 160d954a261e..54cd34a6e4c0 100644
> > > > > --- a/include/drm/ttm/ttm_pool.h
> > > > > +++ b/include/drm/ttm/ttm_pool.h
> > > > > @@ -33,6 +33,7 @@
> > > > > 
> > > > >   struct device;
> > > > >   struct seq_file;
> > > > > +struct ttm_backup_flags;
> > > > >   struct ttm_operation_ctx;
> > > > >   struct ttm_pool;
> > > > >   struct ttm_tt;
> > > > > @@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);
> > > > > 
> > > > >   int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file
> > > > > *m);
> > > > > 
> > > > > +void ttm_pool_drop_backed_up(struct ttm_tt *tt);
> > > > > +
> > > > > +long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt
> > > > > *ttm,
> > > > > +              const struct ttm_backup_flags *flags);
> > > > > +int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct
> > > > > ttm_tt *tt,
> > > > > +                        const struct ttm_operation_ctx
> > > > > *ctx);
> > > > > +
> > > > >   int ttm_pool_mgr_init(unsigned long num_pages);
> > > > >   void ttm_pool_mgr_fini(void);
> > > > > 
> > > > > diff --git a/include/drm/ttm/ttm_tt.h
> > > > > b/include/drm/ttm/ttm_tt.h
> > > > > index 991edafdb2dd..c736c01ac2ca 100644
> > > > > --- a/include/drm/ttm/ttm_tt.h
> > > > > +++ b/include/drm/ttm/ttm_tt.h
> > > > > @@ -32,11 +32,13 @@
> > > > >   #include <drm/ttm/ttm_caching.h>
> > > > >   #include <drm/ttm/ttm_kmap_iter.h>
> > > > > 
> > > > > +struct ttm_backup;
> > > > >   struct ttm_device;
> > > > >   struct ttm_tt;
> > > > >   struct ttm_resource;
> > > > >   struct ttm_buffer_object;
> > > > >   struct ttm_operation_ctx;
> > > > > +struct ttm_pool_tt_restore;
> > > > > 
> > > > >   /**
> > > > >    * struct ttm_tt - This is a structure holding the pages,
> > > > > caching- and aperture
> > > > > @@ -85,17 +87,22 @@ struct ttm_tt {
> > > > >            * fault handling abuses the DMA api a bit and
> > > > > dma_map_attrs can't be
> > > > >            * used to assure pgprot always matches.
> > > > >            *
> > > > > +  * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set
> > > > > if the
> > > > > +  * struct ttm_tt has been (possibly partially) backed
> > > > > up.
> > > > > +  *
> > > > >            * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only.
> > > > > DO NOT
> > > > > USE. This is
> > > > >            * set by TTM after ttm_tt_populate() has
> > > > > successfully
> > > > > returned, and is
> > > > >            * then unset when TTM calls ttm_tt_unpopulate().
> > > > > +  *
> > > > >            */
> > > > >   #define TTM_TT_FLAG_SWAPPED             BIT(0)
> > > > >   #define TTM_TT_FLAG_ZERO_ALLOC          BIT(1)
> > > > >   #define TTM_TT_FLAG_EXTERNAL            BIT(2)
> > > > >   #define TTM_TT_FLAG_EXTERNAL_MAPPABLE   BIT(3)
> > > > >   #define TTM_TT_FLAG_DECRYPTED           BIT(4)
> > > > > +#define TTM_TT_FLAG_BACKED_UP            BIT(5)
> > > > > 
> > > > > -#define TTM_TT_FLAG_PRIV_POPULATED       BIT(5)
> > > > > +#define TTM_TT_FLAG_PRIV_POPULATED       BIT(6)
> > > > >           uint32_t page_flags;
> > > > >           /** @num_pages: Number of pages in the page array.
> > > > > */
> > > > >           uint32_t num_pages;
> > > > > @@ -105,11 +112,20 @@ struct ttm_tt {
> > > > >           dma_addr_t *dma_address;
> > > > >           /** @swap_storage: Pointer to shmem struct file for
> > > > > swap
> > > > > storage. */
> > > > >           struct file *swap_storage;
> > > > > + /**
> > > > > +  * @backup: Pointer to backup struct for backed up tts.
> > > > > +  * Could be unified with @swap_storage. Meanwhile, the
> > > > > driver's
> > > > > +  * ttm_tt_create() callback is responsible for assigning
> > > > > +  * this field.
> > > > > +  */
> > > > > + struct ttm_backup *backup;
> > > > >           /**
> > > > >            * @caching: The current caching state of the
> > > > > pages, see
> > > > > enum
> > > > >            * ttm_caching.
> > > > >            */
> > > > >           enum ttm_caching caching;
> > > > > + /** @restore: Partial restoration from backup state. TTM
> > > > > private */
> > > > > + struct ttm_pool_tt_restore *restore;
> > > > >   };
> > > > > 
> > > > >   /**
> > > > > @@ -129,9 +145,38 @@ static inline bool
> > > > > ttm_tt_is_populated(struct
> > > > > ttm_tt *tt)
> > > > >           return tt->page_flags & TTM_TT_FLAG_PRIV_POPULATED;
> > > > >   }
> > > > > 
> > > > > +/**
> > > > > + * ttm_tt_is_swapped() - Whether the ttm_tt is swapped out
> > > > > or
> > > > > backed up
> > > > > + * @tt: The struct ttm_tt.
> > > > > + *
> > > > > + * Return: true if swapped or backed up, false otherwise.
> > > > > + */
> > > > >   static inline bool ttm_tt_is_swapped(const struct ttm_tt
> > > > > *tt)
> > > > >   {
> > > > > - return tt->page_flags & TTM_TT_FLAG_SWAPPED;
> > > > > + return tt->page_flags & (TTM_TT_FLAG_SWAPPED |
> > > > > TTM_TT_FLAG_BACKED_UP);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * ttm_tt_is_backed_up() - Whether the ttm_tt backed up
> > > > > + * @tt: The struct ttm_tt.
> > > > > + *
> > > > > + * Return: true if swapped or backed up, false otherwise.
> > > > > + */
> > > > > +static inline bool ttm_tt_is_backed_up(const struct ttm_tt
> > > > > *tt)
> > > > > +{
> > > > > + return tt->page_flags & TTM_TT_FLAG_BACKED_UP;
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * ttm_tt_clear_backed_up() - Clear the ttm_tt backed-up
> > > > > status
> > > > > + * @tt: The struct ttm_tt.
> > > > > + *
> > > > > + * Drivers can use this functionto clear the backed-up
> > > > > status,
> > > > > + * for example before destroying or re-validating a purged
> > > > > tt.
> > > > > + */
> > > > > +static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
> > > > > +{
> > > > > + tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
> > > > >   }
> > > > > 
> > > > >   /**
> > > > > @@ -235,6 +280,24 @@ void ttm_tt_mgr_init(unsigned long
> > > > > num_pages,
> > > > > unsigned long num_dma32_pages);
> > > > >   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct
> > > > > ttm_kmap_iter_tt *iter_tt,
> > > > >                                               struct ttm_tt
> > > > > *tt);
> > > > >   unsigned long ttm_tt_pages_limit(void);
> > > > > +
> > > > > +/**
> > > > > + * struct ttm_backup_flags - Flags to govern backup
> > > > > behaviour.
> > > > > + * @purge: Free pages without backing up. Bypass pools.
> > > > > + * @writeback: Attempt to copy contents directly to swap
> > > > > space,
> > > > > even
> > > > > + * if that means blocking on writes to external memory.
> > > > > + */
> > > > > +struct ttm_backup_flags {
> > > > > + u32 purge : 1;
> > > > > + u32 writeback : 1;
> > > > > +};
> > > > > +
> > > > > +long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt
> > > > > *tt,
> > > > > +            const struct ttm_backup_flags flags);
> > > > > +
> > > > > +int ttm_tt_restore(struct ttm_device *bdev, struct ttm_tt
> > > > > *tt,
> > > > > +            const struct ttm_operation_ctx *ctx);
> > > > > +
> > > > >   #if IS_ENABLED(CONFIG_AGP)
> > > > >   #include <linux/agp_backend.h>
> > > > > 
> > > > 
> > > 
> > 

[PATCH v16 3/7] drm/ttm: Use fault-injection to test error paths

[Thomas Hellström]

Use fault-injection to test partial TTM swapout and interrupted swapin.
Return -EINTR for swapin to test the callers ability to handle and
restart the swapin, and on swapout perform a partial swapout to test that
the swapin and release_shrunken functionality.

v8:
- Use the core fault-injection system.
v9:
- Fix compliation failure for !CONFIG_FAULT_INJECTION

Cc: Christian König <christian.koenig@amd.com>
Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <dri-devel@lists.freedesktop.org>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v7
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index ffb7abf52bab..83b10706ba89 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -48,6 +48,13 @@
 
 #include "ttm_module.h"
 
+#ifdef CONFIG_FAULT_INJECTION
+#include <linux/fault-inject.h>
+static DECLARE_FAULT_ATTR(backup_fault_inject);
+#else
+#define should_fail(...) false
+#endif
+
 /**
  * struct ttm_pool_dma - Helper object for coherent DMA mappings
  *
@@ -514,6 +521,12 @@ static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,
 		if (ttm_backup_page_ptr_is_handle(p)) {
 			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
 
+			if (IS_ENABLED(CONFIG_FAULT_INJECTION) && ctx->interruptible &&
+			    should_fail(&backup_fault_inject, 1)) {
+				ret = -EINTR;
+				break;
+			}
+
 			if (handle == 0) {
 				restore->restored_pages++;
 				continue;
@@ -1007,7 +1020,13 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
 
 	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
 
-	for (i = 0; i < tt->num_pages; ++i) {
+	num_pages = tt->num_pages;
+
+	/* Pretend doing fault injection by shrinking only half of the pages. */
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && should_fail(&backup_fault_inject, 1))
+		num_pages = DIV_ROUND_UP(num_pages, 2);
+
+	for (i = 0; i < num_pages; ++i) {
 		s64 shandle;
 
 		page = tt->pages[i];
@@ -1293,6 +1312,10 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 			    &ttm_pool_debugfs_globals_fops);
 	debugfs_create_file("page_pool_shrink", 0400, ttm_debugfs_root, NULL,
 			    &ttm_pool_debugfs_shrink_fops);
+#ifdef CONFIG_FAULT_INJECTION
+	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
+				  &backup_fault_inject);
+#endif
 #endif
 
 	mm_shrinker = shrinker_alloc(0, "drm-ttm_pool");
-- 
2.48.1

Re: [PATCH v16 3/7] drm/ttm: Use fault-injection to test error paths

[Christian König]

Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> Use fault-injection to test partial TTM swapout and interrupted swapin.
> Return -EINTR for swapin to test the callers ability to handle and
> restart the swapin, and on swapout perform a partial swapout to test that
> the swapin and release_shrunken functionality.
>
> v8:
> - Use the core fault-injection system.
> v9:
> - Fix compliation failure for !CONFIG_FAULT_INJECTION
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: <dri-devel@lists.freedesktop.org>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v7

Reviewed-by: Christian König <christian.koenig@amd.com>

> ---
>   drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
>   1 file changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index ffb7abf52bab..83b10706ba89 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> @@ -48,6 +48,13 @@
>   
>   #include "ttm_module.h"
>   
> +#ifdef CONFIG_FAULT_INJECTION
> +#include <linux/fault-inject.h>
> +static DECLARE_FAULT_ATTR(backup_fault_inject);
> +#else
> +#define should_fail(...) false
> +#endif
> +
>   /**
>    * struct ttm_pool_dma - Helper object for coherent DMA mappings
>    *
> @@ -514,6 +521,12 @@ static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,
>   		if (ttm_backup_page_ptr_is_handle(p)) {
>   			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
>   
> +			if (IS_ENABLED(CONFIG_FAULT_INJECTION) && ctx->interruptible &&
> +			    should_fail(&backup_fault_inject, 1)) {
> +				ret = -EINTR;
> +				break;
> +			}
> +
>   			if (handle == 0) {
>   				restore->restored_pages++;
>   				continue;
> @@ -1007,7 +1020,13 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,
>   
>   	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
>   
> -	for (i = 0; i < tt->num_pages; ++i) {
> +	num_pages = tt->num_pages;
> +
> +	/* Pretend doing fault injection by shrinking only half of the pages. */
> +	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && should_fail(&backup_fault_inject, 1))
> +		num_pages = DIV_ROUND_UP(num_pages, 2);
> +
> +	for (i = 0; i < num_pages; ++i) {
>   		s64 shandle;
>   
>   		page = tt->pages[i];
> @@ -1293,6 +1312,10 @@ int ttm_pool_mgr_init(unsigned long num_pages)
>   			    &ttm_pool_debugfs_globals_fops);
>   	debugfs_create_file("page_pool_shrink", 0400, ttm_debugfs_root, NULL,
>   			    &ttm_pool_debugfs_shrink_fops);
> +#ifdef CONFIG_FAULT_INJECTION
> +	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
> +				  &backup_fault_inject);
> +#endif
>   #endif
>   
>   	mm_shrinker = shrinker_alloc(0, "drm-ttm_pool");

[PATCH v16 6/7] drm/xe: Add a shrinker for xe bos

[Thomas Hellström]

Rather than relying on the TTM watermark accounting add a shrinker
for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup
support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet,
introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active
  device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).
v7:
- s/long/s64/ (Christian König)
- s/sofar/progress/ (Matt Brost)
v8:
- Rebase on Xe KUNIT update.
- Add content verifying to the shrinker kunit test.
- Split out TTM changes to a separate patch.
- Get rid of multiple bool arguments for clarity (Matt Brost)
- Avoid an error pointer dereference (Matt Brost)
- Avoid an integer overflow (Matt Auld)
- Address misc review comments by Matt Brost.
v9:
- Fix a compliation error.
- Rebase.
v10:
- Update to new LRU walk interface.
- Rework ghost-, zombie and purged object shrinking.
- Rebase.
v11:
- Use additional TTM helpers.
- Honor __GFP_FS and __GFP_IO
- Rebase.
v13:
- Use ttm_tt_setup_backup().
v14:
- Don't set up backup on imported bos.
v15:
- Rebase on backup interface changes.

Cc: Christian König <christian.koenig@amd.com>
Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: <dri-devel@lists.freedesktop.org>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/Makefile          |   1 +
 drivers/gpu/drm/xe/tests/xe_bo.c     |   6 +-
 drivers/gpu/drm/xe/xe_bo.c           | 202 +++++++++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h           |  36 ++++
 drivers/gpu/drm/xe/xe_device.c       |   8 +
 drivers/gpu/drm/xe/xe_device_types.h |   2 +
 drivers/gpu/drm/xe/xe_shrinker.c     | 258 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h     |  18 ++
 8 files changed, 513 insertions(+), 18 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h

diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
index 328aff36831b..b3738bcea11e 100644
--- a/drivers/gpu/drm/xe/Makefile
+++ b/drivers/gpu/drm/xe/Makefile
@@ -95,6 +95,7 @@ xe-y += xe_bb.o \
 	xe_ring_ops.o \
 	xe_sa.o \
 	xe_sched_job.o \
+	xe_shrinker.o \
 	xe_step.o \
 	xe_survivability_mode.o \
 	xe_sync.o \
diff --git a/drivers/gpu/drm/xe/tests/xe_bo.c b/drivers/gpu/drm/xe/tests/xe_bo.c
index 6795d1d916e4..9fde67ca989f 100644
--- a/drivers/gpu/drm/xe/tests/xe_bo.c
+++ b/drivers/gpu/drm/xe/tests/xe_bo.c
@@ -514,8 +514,13 @@ static int shrink_test_run_device(struct xe_device *xe)
 		 * other way around, they may not be subject to swapping...
 		 */
 		if (alloced < purgeable) {
+			xe_ttm_tt_account_subtract(&xe_tt->ttm);
 			xe_tt->purgeable = true;
+			xe_ttm_tt_account_add(&xe_tt->ttm);
 			bo->ttm.priority = 0;
+			spin_lock(&bo->ttm.bdev->lru_lock);
+			ttm_bo_move_to_lru_tail(&bo->ttm);
+			spin_unlock(&bo->ttm.bdev->lru_lock);
 		} else {
 			int ret = shrink_test_fill_random(bo, &prng, link);
 
@@ -570,7 +575,6 @@ static int shrink_test_run_device(struct xe_device *xe)
 				if (ret == -EINTR)
 					intr = true;
 			} while (ret == -EINTR && !signal_pending(current));
-
 			if (!ret && !purgeable)
 				failed = shrink_test_verify(test, bo, count, &prng, link);
 
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index c32201123d44..a133d66587b8 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -10,6 +10,7 @@
 #include <drm/drm_drv.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/drm_managed.h>
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
 #include <drm/ttm/ttm_tt.h>
@@ -27,6 +28,7 @@
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
 #include "xe_res_cursor.h"
+#include "xe_shrinker.h"
 #include "xe_trace_bo.h"
 #include "xe_ttm_stolen_mgr.h"
 #include "xe_vm.h"
@@ -283,9 +285,11 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo,
 	}
 }
 
+/* struct xe_ttm_tt - Subclassed ttm_tt for xe */
 struct xe_ttm_tt {
 	struct ttm_tt ttm;
-	struct device *dev;
+	/** @xe - The xe device */
+	struct xe_device *xe;
 	struct sg_table sgt;
 	struct sg_table *sg;
 	/** @purgeable: Whether the content of the pages of @ttm is purgeable. */
@@ -298,7 +302,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
 	unsigned long num_pages = tt->num_pages;
 	int ret;
 
-	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
+	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+		   !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE));
 
 	if (xe_tt->sg)
 		return 0;
@@ -306,13 +311,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
 	ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt->pages,
 						num_pages, 0,
 						(u64)num_pages << PAGE_SHIFT,
-						xe_sg_segment_size(xe_tt->dev),
+						xe_sg_segment_size(xe_tt->xe->drm.dev),
 						GFP_KERNEL);
 	if (ret)
 		return ret;
 
 	xe_tt->sg = &xe_tt->sgt;
-	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg, DMA_BIDIRECTIONAL,
+	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg, DMA_BIDIRECTIONAL,
 			      DMA_ATTR_SKIP_CPU_SYNC);
 	if (ret) {
 		sg_free_table(xe_tt->sg);
@@ -328,7 +333,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
-		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
+		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
 		xe_tt->sg = NULL;
@@ -343,21 +348,47 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
 	return xe_tt->sg;
 }
 
+/*
+ * Account ttm pages against the device shrinker's shrinkable and
+ * purgeable counts.
+ */
+static void xe_ttm_tt_account_add(struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, tt->num_pages);
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, tt->num_pages, 0);
+}
+
+static void xe_ttm_tt_account_subtract(struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, -(long)tt->num_pages);
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, -(long)tt->num_pages, 0);
+}
+
 static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 				       u32 page_flags)
 {
 	struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
 	struct xe_device *xe = xe_bo_device(bo);
-	struct xe_ttm_tt *tt;
+	struct xe_ttm_tt *xe_tt;
+	struct ttm_tt *tt;
 	unsigned long extra_pages;
 	enum ttm_caching caching = ttm_cached;
 	int err;
 
-	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
-	if (!tt)
+	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
+	if (!xe_tt)
 		return NULL;
 
-	tt->dev = xe->drm.dev;
+	tt = &xe_tt->ttm;
+	xe_tt->xe = xe;
 
 	extra_pages = 0;
 	if (xe_bo_needs_ccs_pages(bo))
@@ -403,42 +434,66 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 		caching = ttm_uncached;
 	}
 
-	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching, extra_pages);
+	if (ttm_bo->type != ttm_bo_type_sg)
+		page_flags |= TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE;
+
+	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching, extra_pages);
 	if (err) {
-		kfree(tt);
+		kfree(xe_tt);
 		return NULL;
 	}
 
-	return &tt->ttm;
+	if (ttm_bo->type != ttm_bo_type_sg) {
+		err = ttm_tt_setup_backup(tt);
+		if (err) {
+			ttm_tt_fini(tt);
+			kfree(xe_tt);
+			return NULL;
+		}
+	}
+
+	return tt;
 }
 
 static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 			      struct ttm_operation_ctx *ctx)
 {
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 	int err;
 
 	/*
 	 * dma-bufs are not populated with pages, and the dma-
 	 * addresses are set up when moved to XE_PL_TT.
 	 */
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return 0;
 
-	err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
+	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
+		err = ttm_tt_restore(ttm_dev, tt, ctx);
+	} else {
+		ttm_tt_clear_backed_up(tt);
+		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
+	}
 	if (err)
 		return err;
 
-	return err;
+	xe_tt->purgeable = false;
+	xe_ttm_tt_account_add(tt);
+
+	return 0;
 }
 
 static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return;
 
 	xe_tt_unmap_sg(tt);
 
-	return ttm_pool_free(&ttm_dev->pool, tt);
+	ttm_pool_free(&ttm_dev->pool, tt);
+	xe_ttm_tt_account_subtract(tt);
 }
 
 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)
@@ -888,6 +943,111 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	return ret;
 }
 
+static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
+			       struct ttm_buffer_object *bo,
+			       unsigned long *scanned)
+{
+	long lret;
+
+	/* Fake move to system, without copying data. */
+	if (bo->resource->mem_type != XE_PL_SYSTEM) {
+		struct ttm_resource *new_resource;
+
+		lret = ttm_bo_wait_ctx(bo, ctx);
+		if (lret)
+			return lret;
+
+		lret = ttm_bo_mem_space(bo, &sys_placement, &new_resource, ctx);
+		if (lret)
+			return lret;
+
+		xe_tt_unmap_sg(bo->ttm);
+		ttm_bo_move_null(bo, new_resource);
+	}
+
+	*scanned += bo->ttm->num_pages;
+	lret = ttm_bo_shrink(ctx, bo, (struct ttm_bo_shrink_flags)
+			     {.purge = true,
+			      .writeback = false,
+			      .allow_move = false});
+
+	if (lret > 0)
+		xe_ttm_tt_account_subtract(bo->ttm);
+
+	return lret;
+}
+
+/**
+ * xe_bo_shrink() - Try to shrink an xe bo.
+ * @ctx: The struct ttm_operation_ctx used for shrinking.
+ * @bo: The TTM buffer object whose pages to shrink.
+ * @flags: Flags governing the shrink behaviour.
+ * @scanned: Pointer to a counter of the number of pages
+ * attempted to shrink.
+ *
+ * Try to shrink- or purge a bo, and if it succeeds, unmap dma.
+ * Note that we need to be able to handle also non xe bos
+ * (ghost bos), but only if the struct ttm_tt is embedded in
+ * a struct xe_ttm_tt. When the function attempts to shrink
+ * the pages of a buffer object, The value pointed to by @scanned
+ * is updated.
+ *
+ * Return: The number of pages shrunken or purged, or negative error
+ * code on failure.
+ */
+long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
+		  const struct xe_bo_shrink_flags flags,
+		  unsigned long *scanned)
+{
+	struct ttm_tt *tt = bo->ttm;
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	struct ttm_place place = {.mem_type = bo->resource->mem_type};
+	struct xe_bo *xe_bo = ttm_to_xe_bo(bo);
+	struct xe_device *xe = xe_tt->xe;
+	bool needs_rpm;
+	long lret = 0L;
+
+	if (!(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE) ||
+	    (flags.purge && !xe_tt->purgeable))
+		return -EBUSY;
+
+	if (!ttm_bo_eviction_valuable(bo, &place))
+		return -EBUSY;
+
+	if (!xe_bo_is_xe_bo(bo) || !xe_bo_get_unless_zero(xe_bo))
+		return xe_bo_shrink_purge(ctx, bo, scanned);
+
+	if (xe_tt->purgeable) {
+		if (bo->resource->mem_type != XE_PL_SYSTEM)
+			lret = xe_bo_move_notify(xe_bo, ctx);
+		if (!lret)
+			lret = xe_bo_shrink_purge(ctx, bo, scanned);
+		goto out_unref;
+	}
+
+	/* System CCS needs gpu copy when moving PL_TT -> PL_SYSTEM */
+	needs_rpm = (!IS_DGFX(xe) && bo->resource->mem_type != XE_PL_SYSTEM &&
+		     xe_bo_needs_ccs_pages(xe_bo));
+	if (needs_rpm && !xe_pm_runtime_get_if_active(xe))
+		goto out_unref;
+
+	*scanned += tt->num_pages;
+	lret = ttm_bo_shrink(ctx, bo, (struct ttm_bo_shrink_flags)
+			     {.purge = false,
+			      .writeback = flags.writeback,
+			      .allow_move = true});
+	if (needs_rpm)
+		xe_pm_runtime_put(xe);
+
+	if (lret > 0)
+		xe_ttm_tt_account_subtract(tt);
+
+out_unref:
+	xe_bo_put(xe_bo);
+
+	return lret;
+}
+
 /**
  * xe_bo_evict_pinned() - Evict a pinned VRAM object to system memory
  * @bo: The buffer object to move.
@@ -1905,6 +2065,8 @@ int xe_bo_pin_external(struct xe_bo *bo)
 	}
 
 	ttm_bo_pin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account_subtract(bo->ttm.ttm);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -1964,6 +2126,8 @@ int xe_bo_pin(struct xe_bo *bo)
 	}
 
 	ttm_bo_pin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account_subtract(bo->ttm.ttm);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -1998,6 +2162,8 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	spin_unlock(&xe->pinned.lock);
 
 	ttm_bo_unpin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account_add(bo->ttm.ttm);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -2021,6 +2187,8 @@ void xe_bo_unpin(struct xe_bo *bo)
 		spin_unlock(&xe->pinned.lock);
 	}
 	ttm_bo_unpin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account_add(bo->ttm.ttm);
 }
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 04995c5ced32..9ca87973cff7 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -148,6 +148,28 @@ static inline struct xe_bo *xe_bo_get(struct xe_bo *bo)
 
 void xe_bo_put(struct xe_bo *bo);
 
+/*
+ * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
+ * xe bo
+ * @bo: The bo for which we want to obtain a refcount.
+ *
+ * There is a short window between where the bo's GEM object refcount reaches
+ * zero and where we put the final ttm_bo reference. Code in the eviction- and
+ * shrinking path should therefore attempt to grab a gem object reference before
+ * trying to use members outside of the base class ttm object. This function is
+ * intended for that purpose. On successful return, this function must be paired
+ * with an xe_bo_put().
+ *
+ * Return: @bo on success, NULL on failure.
+ */
+static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)
+{
+	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
+		return NULL;
+
+	return bo;
+}
+
 static inline void __xe_bo_unset_bulk_move(struct xe_bo *bo)
 {
 	if (bo)
@@ -343,6 +365,20 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)
 	return round_down(max / 2, PAGE_SIZE);
 }
 
+/**
+ * struct xe_bo_shrink_flags - flags governing the shrink behaviour.
+ * @purge: Only purging allowed. Don't shrink if bo not purgeable.
+ * @writeback: Attempt to immediately move content to swap.
+ */
+struct xe_bo_shrink_flags {
+	u32 purge : 1;
+	u32 writeback : 1;
+};
+
+long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
+		  const struct xe_bo_shrink_flags flags,
+		  unsigned long *scanned);
+
 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 8fedc72e9db4..c4a6062124b0 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -51,6 +51,7 @@
 #include "xe_pm.h"
 #include "xe_pmu.h"
 #include "xe_query.h"
+#include "xe_shrinker.h"
 #include "xe_sriov.h"
 #include "xe_survivability_mode.h"
 #include "xe_tile.h"
@@ -396,6 +397,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
 	if (xe->unordered_wq)
 		destroy_workqueue(xe->unordered_wq);
 
+	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
+		xe_shrinker_destroy(xe->mem.shrinker);
+
 	if (xe->destroy_wq)
 		destroy_workqueue(xe->destroy_wq);
 
@@ -428,6 +432,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 	if (err)
 		goto err;
 
+	xe->mem.shrinker = xe_shrinker_create(xe);
+	if (IS_ERR(xe->mem.shrinker))
+		return ERR_CAST(xe->mem.shrinker);
+
 	xe->info.devid = pdev->device;
 	xe->info.revid = pdev->revision;
 	xe->info.force_execlist = xe_modparam.force_execlist;
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 89f532b67bc4..7406deac0fec 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -369,6 +369,8 @@ struct xe_device {
 		struct xe_mem_region vram;
 		/** @mem.sys_mgr: system TTM manager */
 		struct ttm_resource_manager sys_mgr;
+		/** @mem.sys_mgr: system memory shrinker. */
+		struct xe_shrinker *shrinker;
 	} mem;
 
 	/** @sriov: device level virtualization data */
diff --git a/drivers/gpu/drm/xe/xe_shrinker.c b/drivers/gpu/drm/xe/xe_shrinker.c
new file mode 100644
index 000000000000..8184390f9c7b
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_shrinker.c
@@ -0,0 +1,258 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <linux/shrinker.h>
+
+#include <drm/ttm/ttm_backup.h>
+#include <drm/ttm/ttm_bo.h>
+#include <drm/ttm/ttm_tt.h>
+
+#include "xe_bo.h"
+#include "xe_pm.h"
+#include "xe_shrinker.h"
+
+/**
+ * struct xe_shrinker - per-device shrinker
+ * @xe: Back pointer to the device.
+ * @lock: Lock protecting accounting.
+ * @shrinkable_pages: Number of pages that are currently shrinkable.
+ * @purgeable_pages: Number of pages that are currently purgeable.
+ * @shrink: Pointer to the mm shrinker.
+ * @pm_worker: Worker to wake up the device if required.
+ */
+struct xe_shrinker {
+	struct xe_device *xe;
+	rwlock_t lock;
+	long shrinkable_pages;
+	long purgeable_pages;
+	struct shrinker *shrink;
+	struct work_struct pm_worker;
+};
+
+static struct xe_shrinker *to_xe_shrinker(struct shrinker *shrink)
+{
+	return shrink->private_data;
+}
+
+/**
+ * xe_shrinker_mod_pages() - Modify shrinker page accounting
+ * @shrinker: Pointer to the struct xe_shrinker.
+ * @shrinkable: Shrinkable pages delta. May be negative.
+ * @purgeable: Purgeable page delta. May be negative.
+ *
+ * Modifies the shrinkable and purgeable pages accounting.
+ */
+void
+xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable)
+{
+	write_lock(&shrinker->lock);
+	shrinker->shrinkable_pages += shrinkable;
+	shrinker->purgeable_pages += purgeable;
+	write_unlock(&shrinker->lock);
+}
+
+static s64 xe_shrinker_walk(struct xe_device *xe,
+			    struct ttm_operation_ctx *ctx,
+			    const struct xe_bo_shrink_flags flags,
+			    unsigned long to_scan, unsigned long *scanned)
+{
+	unsigned int mem_type;
+	s64 freed = 0, lret;
+
+	for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT; ++mem_type) {
+		struct ttm_resource_manager *man = ttm_manager_type(&xe->ttm, mem_type);
+		struct ttm_bo_lru_cursor curs;
+		struct ttm_buffer_object *ttm_bo;
+
+		if (!man || !man->use_tt)
+			continue;
+
+		ttm_bo_lru_for_each_reserved_guarded(&curs, man, ctx, ttm_bo) {
+			if (!ttm_bo_shrink_suitable(ttm_bo, ctx))
+				continue;
+
+			lret = xe_bo_shrink(ctx, ttm_bo, flags, scanned);
+			if (lret < 0)
+				return lret;
+
+			freed += lret;
+			if (*scanned >= to_scan)
+				break;
+		}
+	}
+
+	return freed;
+}
+
+static unsigned long
+xe_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	unsigned long num_pages;
+	bool can_backup = !!(sc->gfp_mask & __GFP_FS);
+
+	num_pages = ttm_backup_bytes_avail() >> PAGE_SHIFT;
+	read_lock(&shrinker->lock);
+
+	if (can_backup)
+		num_pages = min_t(unsigned long, num_pages, shrinker->shrinkable_pages);
+	else
+		num_pages = 0;
+
+	num_pages += shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	return num_pages ? num_pages : SHRINK_EMPTY;
+}
+
+/*
+ * Check if we need runtime pm, and if so try to grab a reference if
+ * already active. If grabbing a reference fails, queue a worker that
+ * does it for us outside of reclaim, but don't wait for it to complete.
+ * If bo shrinking needs an rpm reference and we don't have it (yet),
+ * that bo will be skipped anyway.
+ */
+static bool xe_shrinker_runtime_pm_get(struct xe_shrinker *shrinker, bool force,
+				       unsigned long nr_to_scan, bool can_backup)
+{
+	struct xe_device *xe = shrinker->xe;
+
+	if (IS_DGFX(xe) || !xe_device_has_flat_ccs(xe) ||
+	    !ttm_backup_bytes_avail())
+		return false;
+
+	if (!force) {
+		read_lock(&shrinker->lock);
+		force = (nr_to_scan > shrinker->purgeable_pages && can_backup);
+		read_unlock(&shrinker->lock);
+		if (!force)
+			return false;
+	}
+
+	if (!xe_pm_runtime_get_if_active(xe)) {
+		if (xe_rpm_reclaim_safe(xe) && !ttm_bo_shrink_avoid_wait()) {
+			xe_pm_runtime_get(xe);
+			return true;
+		}
+		queue_work(xe->unordered_wq, &shrinker->pm_worker);
+		return false;
+	}
+
+	return true;
+}
+
+static void xe_shrinker_runtime_pm_put(struct xe_shrinker *shrinker, bool runtime_pm)
+{
+	if (runtime_pm)
+		xe_pm_runtime_put(shrinker->xe);
+}
+
+static unsigned long xe_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	struct ttm_operation_ctx ctx = {
+		.interruptible = false,
+		.no_wait_gpu = ttm_bo_shrink_avoid_wait(),
+	};
+	unsigned long nr_to_scan, nr_scanned = 0, freed = 0;
+	struct xe_bo_shrink_flags shrink_flags = {
+		.purge = true,
+		/* Don't request writeback without __GFP_IO. */
+		.writeback = !ctx.no_wait_gpu && (sc->gfp_mask & __GFP_IO),
+	};
+	bool runtime_pm;
+	bool purgeable;
+	bool can_backup = !!(sc->gfp_mask & __GFP_FS);
+	s64 lret;
+
+	nr_to_scan = sc->nr_to_scan;
+
+	read_lock(&shrinker->lock);
+	purgeable = !!shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	/* Might need runtime PM. Try to wake early if it looks like it. */
+	runtime_pm = xe_shrinker_runtime_pm_get(shrinker, false, nr_to_scan, can_backup);
+
+	if (purgeable && nr_scanned < nr_to_scan) {
+		lret = xe_shrinker_walk(shrinker->xe, &ctx, shrink_flags,
+					nr_to_scan, &nr_scanned);
+		if (lret >= 0)
+			freed += lret;
+	}
+
+	sc->nr_scanned = nr_scanned;
+	if (nr_scanned >= nr_to_scan || !can_backup)
+		goto out;
+
+	/* If we didn't wake before, try to do it now if needed. */
+	if (!runtime_pm)
+		runtime_pm = xe_shrinker_runtime_pm_get(shrinker, true, 0, can_backup);
+
+	shrink_flags.purge = false;
+	lret = xe_shrinker_walk(shrinker->xe, &ctx, shrink_flags,
+				nr_to_scan, &nr_scanned);
+	if (lret >= 0)
+		freed += lret;
+
+	sc->nr_scanned = nr_scanned;
+out:
+	xe_shrinker_runtime_pm_put(shrinker, runtime_pm);
+	return nr_scanned ? freed : SHRINK_STOP;
+}
+
+/* Wake up the device for shrinking. */
+static void xe_shrinker_pm(struct work_struct *work)
+{
+	struct xe_shrinker *shrinker =
+		container_of(work, typeof(*shrinker), pm_worker);
+
+	xe_pm_runtime_get(shrinker->xe);
+	xe_pm_runtime_put(shrinker->xe);
+}
+
+/**
+ * xe_shrinker_create() - Create an xe per-device shrinker
+ * @xe: Pointer to the xe device.
+ *
+ * Returns: A pointer to the created shrinker on success,
+ * Negative error code on failure.
+ */
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe)
+{
+	struct xe_shrinker *shrinker = kzalloc(sizeof(*shrinker), GFP_KERNEL);
+
+	if (!shrinker)
+		return ERR_PTR(-ENOMEM);
+
+	shrinker->shrink = shrinker_alloc(0, "xe system shrinker");
+	if (!shrinker->shrink) {
+		kfree(shrinker);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm);
+	shrinker->xe = xe;
+	rwlock_init(&shrinker->lock);
+	shrinker->shrink->count_objects = xe_shrinker_count;
+	shrinker->shrink->scan_objects = xe_shrinker_scan;
+	shrinker->shrink->private_data = shrinker;
+	shrinker_register(shrinker->shrink);
+
+	return shrinker;
+}
+
+/**
+ * xe_shrinker_destroy() - Destroy an xe per-device shrinker
+ * @shrinker: Pointer to the shrinker to destroy.
+ */
+void xe_shrinker_destroy(struct xe_shrinker *shrinker)
+{
+	xe_assert(shrinker->xe, !shrinker->shrinkable_pages);
+	xe_assert(shrinker->xe, !shrinker->purgeable_pages);
+	shrinker_free(shrinker->shrink);
+	flush_work(&shrinker->pm_worker);
+	kfree(shrinker);
+}
diff --git a/drivers/gpu/drm/xe/xe_shrinker.h b/drivers/gpu/drm/xe/xe_shrinker.h
new file mode 100644
index 000000000000..28a038f4fcbf
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_shrinker.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_SHRINKER_H_
+#define _XE_SHRINKER_H_
+
+struct xe_shrinker;
+struct xe_device;
+
+void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable);
+
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
+
+void xe_shrinker_destroy(struct xe_shrinker *shrinker);
+
+#endif
-- 
2.48.1

Re: [PATCH v16 6/7] drm/xe: Add a shrinker for xe bos

[Christian König]

Am 30.01.25 um 11:13 schrieb Thomas Hellström:
> Rather than relying on the TTM watermark accounting add a shrinker
> for xe_bos in TT or system memory.
>
> Leverage the newly added TTM per-page shrinking and shmem backup
> support.
>
> Although xe doesn't fully support WONTNEED (purgeable) bos yet,
> introduce and add shrinker support for purgeable ttm_tts.
>
> v2:
> - Cleanups bugfixes and a KUNIT shrinker test.
> - Add writeback support, and activate if kswapd.
> v3:
> - Move the try_shrink() helper to core TTM.
> - Minor cleanups.
> v4:
> - Add runtime pm for the shrinker. Shrinking may require an active
>    device for CCS metadata copying.
> v5:
> - Separately purge ghost- and zombie objects in the shrinker.
> - Fix a format specifier - type inconsistency. (Kernel test robot).
> v7:
> - s/long/s64/ (Christian König)
> - s/sofar/progress/ (Matt Brost)
> v8:
> - Rebase on Xe KUNIT update.
> - Add content verifying to the shrinker kunit test.
> - Split out TTM changes to a separate patch.
> - Get rid of multiple bool arguments for clarity (Matt Brost)
> - Avoid an error pointer dereference (Matt Brost)
> - Avoid an integer overflow (Matt Auld)
> - Address misc review comments by Matt Brost.
> v9:
> - Fix a compliation error.
> - Rebase.
> v10:
> - Update to new LRU walk interface.
> - Rework ghost-, zombie and purged object shrinking.
> - Rebase.
> v11:
> - Use additional TTM helpers.
> - Honor __GFP_FS and __GFP_IO
> - Rebase.
> v13:
> - Use ttm_tt_setup_backup().
> v14:
> - Don't set up backup on imported bos.
> v15:
> - Rebase on backup interface changes.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: <dri-devel@lists.freedesktop.org>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Matthew Brost <matthew.brost@intel.com>

Acked-by: Christian König <christian.koenig@amd.com>

@Felix at some point we should do something similar for amdgpu, it 
removes the 50% limit we have on TTMs system memory allocations.

Regards,
Christian.

> ---
>   drivers/gpu/drm/xe/Makefile          |   1 +
>   drivers/gpu/drm/xe/tests/xe_bo.c     |   6 +-
>   drivers/gpu/drm/xe/xe_bo.c           | 202 +++++++++++++++++++--
>   drivers/gpu/drm/xe/xe_bo.h           |  36 ++++
>   drivers/gpu/drm/xe/xe_device.c       |   8 +
>   drivers/gpu/drm/xe/xe_device_types.h |   2 +
>   drivers/gpu/drm/xe/xe_shrinker.c     | 258 +++++++++++++++++++++++++++
>   drivers/gpu/drm/xe/xe_shrinker.h     |  18 ++
>   8 files changed, 513 insertions(+), 18 deletions(-)
>   create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
>   create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
>
> diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
> index 328aff36831b..b3738bcea11e 100644
> --- a/drivers/gpu/drm/xe/Makefile
> +++ b/drivers/gpu/drm/xe/Makefile
> @@ -95,6 +95,7 @@ xe-y += xe_bb.o \
>   	xe_ring_ops.o \
>   	xe_sa.o \
>   	xe_sched_job.o \
> +	xe_shrinker.o \
>   	xe_step.o \
>   	xe_survivability_mode.o \
>   	xe_sync.o \
> diff --git a/drivers/gpu/drm/xe/tests/xe_bo.c b/drivers/gpu/drm/xe/tests/xe_bo.c
> index 6795d1d916e4..9fde67ca989f 100644
> --- a/drivers/gpu/drm/xe/tests/xe_bo.c
> +++ b/drivers/gpu/drm/xe/tests/xe_bo.c
> @@ -514,8 +514,13 @@ static int shrink_test_run_device(struct xe_device *xe)
>   		 * other way around, they may not be subject to swapping...
>   		 */
>   		if (alloced < purgeable) {
> +			xe_ttm_tt_account_subtract(&xe_tt->ttm);
>   			xe_tt->purgeable = true;
> +			xe_ttm_tt_account_add(&xe_tt->ttm);
>   			bo->ttm.priority = 0;
> +			spin_lock(&bo->ttm.bdev->lru_lock);
> +			ttm_bo_move_to_lru_tail(&bo->ttm);
> +			spin_unlock(&bo->ttm.bdev->lru_lock);
>   		} else {
>   			int ret = shrink_test_fill_random(bo, &prng, link);
>   
> @@ -570,7 +575,6 @@ static int shrink_test_run_device(struct xe_device *xe)
>   				if (ret == -EINTR)
>   					intr = true;
>   			} while (ret == -EINTR && !signal_pending(current));
> -
>   			if (!ret && !purgeable)
>   				failed = shrink_test_verify(test, bo, count, &prng, link);
>   
> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index c32201123d44..a133d66587b8 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
> @@ -10,6 +10,7 @@
>   #include <drm/drm_drv.h>
>   #include <drm/drm_gem_ttm_helper.h>
>   #include <drm/drm_managed.h>
> +#include <drm/ttm/ttm_backup.h>
>   #include <drm/ttm/ttm_device.h>
>   #include <drm/ttm/ttm_placement.h>
>   #include <drm/ttm/ttm_tt.h>
> @@ -27,6 +28,7 @@
>   #include "xe_pm.h"
>   #include "xe_preempt_fence.h"
>   #include "xe_res_cursor.h"
> +#include "xe_shrinker.h"
>   #include "xe_trace_bo.h"
>   #include "xe_ttm_stolen_mgr.h"
>   #include "xe_vm.h"
> @@ -283,9 +285,11 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo,
>   	}
>   }
>   
> +/* struct xe_ttm_tt - Subclassed ttm_tt for xe */
>   struct xe_ttm_tt {
>   	struct ttm_tt ttm;
> -	struct device *dev;
> +	/** @xe - The xe device */
> +	struct xe_device *xe;
>   	struct sg_table sgt;
>   	struct sg_table *sg;
>   	/** @purgeable: Whether the content of the pages of @ttm is purgeable. */
> @@ -298,7 +302,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
>   	unsigned long num_pages = tt->num_pages;
>   	int ret;
>   
> -	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
> +	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
> +		   !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE));
>   
>   	if (xe_tt->sg)
>   		return 0;
> @@ -306,13 +311,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
>   	ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt->pages,
>   						num_pages, 0,
>   						(u64)num_pages << PAGE_SHIFT,
> -						xe_sg_segment_size(xe_tt->dev),
> +						xe_sg_segment_size(xe_tt->xe->drm.dev),
>   						GFP_KERNEL);
>   	if (ret)
>   		return ret;
>   
>   	xe_tt->sg = &xe_tt->sgt;
> -	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg, DMA_BIDIRECTIONAL,
> +	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg, DMA_BIDIRECTIONAL,
>   			      DMA_ATTR_SKIP_CPU_SYNC);
>   	if (ret) {
>   		sg_free_table(xe_tt->sg);
> @@ -328,7 +333,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)
>   	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
>   
>   	if (xe_tt->sg) {
> -		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
> +		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,
>   				  DMA_BIDIRECTIONAL, 0);
>   		sg_free_table(xe_tt->sg);
>   		xe_tt->sg = NULL;
> @@ -343,21 +348,47 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
>   	return xe_tt->sg;
>   }
>   
> +/*
> + * Account ttm pages against the device shrinker's shrinkable and
> + * purgeable counts.
> + */
> +static void xe_ttm_tt_account_add(struct ttm_tt *tt)
> +{
> +	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
> +
> +	if (xe_tt->purgeable)
> +		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, tt->num_pages);
> +	else
> +		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, tt->num_pages, 0);
> +}
> +
> +static void xe_ttm_tt_account_subtract(struct ttm_tt *tt)
> +{
> +	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
> +
> +	if (xe_tt->purgeable)
> +		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, -(long)tt->num_pages);
> +	else
> +		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, -(long)tt->num_pages, 0);
> +}
> +
>   static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
>   				       u32 page_flags)
>   {
>   	struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
>   	struct xe_device *xe = xe_bo_device(bo);
> -	struct xe_ttm_tt *tt;
> +	struct xe_ttm_tt *xe_tt;
> +	struct ttm_tt *tt;
>   	unsigned long extra_pages;
>   	enum ttm_caching caching = ttm_cached;
>   	int err;
>   
> -	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
> -	if (!tt)
> +	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
> +	if (!xe_tt)
>   		return NULL;
>   
> -	tt->dev = xe->drm.dev;
> +	tt = &xe_tt->ttm;
> +	xe_tt->xe = xe;
>   
>   	extra_pages = 0;
>   	if (xe_bo_needs_ccs_pages(bo))
> @@ -403,42 +434,66 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
>   		caching = ttm_uncached;
>   	}
>   
> -	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching, extra_pages);
> +	if (ttm_bo->type != ttm_bo_type_sg)
> +		page_flags |= TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE;
> +
> +	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching, extra_pages);
>   	if (err) {
> -		kfree(tt);
> +		kfree(xe_tt);
>   		return NULL;
>   	}
>   
> -	return &tt->ttm;
> +	if (ttm_bo->type != ttm_bo_type_sg) {
> +		err = ttm_tt_setup_backup(tt);
> +		if (err) {
> +			ttm_tt_fini(tt);
> +			kfree(xe_tt);
> +			return NULL;
> +		}
> +	}
> +
> +	return tt;
>   }
>   
>   static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
>   			      struct ttm_operation_ctx *ctx)
>   {
> +	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
>   	int err;
>   
>   	/*
>   	 * dma-bufs are not populated with pages, and the dma-
>   	 * addresses are set up when moved to XE_PL_TT.
>   	 */
> -	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
> +	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
> +	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
>   		return 0;
>   
> -	err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
> +	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
> +		err = ttm_tt_restore(ttm_dev, tt, ctx);
> +	} else {
> +		ttm_tt_clear_backed_up(tt);
> +		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
> +	}
>   	if (err)
>   		return err;
>   
> -	return err;
> +	xe_tt->purgeable = false;
> +	xe_ttm_tt_account_add(tt);
> +
> +	return 0;
>   }
>   
>   static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
>   {
> -	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
> +	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
> +	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
>   		return;
>   
>   	xe_tt_unmap_sg(tt);
>   
> -	return ttm_pool_free(&ttm_dev->pool, tt);
> +	ttm_pool_free(&ttm_dev->pool, tt);
> +	xe_ttm_tt_account_subtract(tt);
>   }
>   
>   static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)
> @@ -888,6 +943,111 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
>   	return ret;
>   }
>   
> +static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
> +			       struct ttm_buffer_object *bo,
> +			       unsigned long *scanned)
> +{
> +	long lret;
> +
> +	/* Fake move to system, without copying data. */
> +	if (bo->resource->mem_type != XE_PL_SYSTEM) {
> +		struct ttm_resource *new_resource;
> +
> +		lret = ttm_bo_wait_ctx(bo, ctx);
> +		if (lret)
> +			return lret;
> +
> +		lret = ttm_bo_mem_space(bo, &sys_placement, &new_resource, ctx);
> +		if (lret)
> +			return lret;
> +
> +		xe_tt_unmap_sg(bo->ttm);
> +		ttm_bo_move_null(bo, new_resource);
> +	}
> +
> +	*scanned += bo->ttm->num_pages;
> +	lret = ttm_bo_shrink(ctx, bo, (struct ttm_bo_shrink_flags)
> +			     {.purge = true,
> +			      .writeback = false,
> +			      .allow_move = false});
> +
> +	if (lret > 0)
> +		xe_ttm_tt_account_subtract(bo->ttm);
> +
> +	return lret;
> +}
> +
> +/**
> + * xe_bo_shrink() - Try to shrink an xe bo.
> + * @ctx: The struct ttm_operation_ctx used for shrinking.
> + * @bo: The TTM buffer object whose pages to shrink.
> + * @flags: Flags governing the shrink behaviour.
> + * @scanned: Pointer to a counter of the number of pages
> + * attempted to shrink.
> + *
> + * Try to shrink- or purge a bo, and if it succeeds, unmap dma.
> + * Note that we need to be able to handle also non xe bos
> + * (ghost bos), but only if the struct ttm_tt is embedded in
> + * a struct xe_ttm_tt. When the function attempts to shrink
> + * the pages of a buffer object, The value pointed to by @scanned
> + * is updated.
> + *
> + * Return: The number of pages shrunken or purged, or negative error
> + * code on failure.
> + */
> +long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
> +		  const struct xe_bo_shrink_flags flags,
> +		  unsigned long *scanned)
> +{
> +	struct ttm_tt *tt = bo->ttm;
> +	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
> +	struct ttm_place place = {.mem_type = bo->resource->mem_type};
> +	struct xe_bo *xe_bo = ttm_to_xe_bo(bo);
> +	struct xe_device *xe = xe_tt->xe;
> +	bool needs_rpm;
> +	long lret = 0L;
> +
> +	if (!(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE) ||
> +	    (flags.purge && !xe_tt->purgeable))
> +		return -EBUSY;
> +
> +	if (!ttm_bo_eviction_valuable(bo, &place))
> +		return -EBUSY;
> +
> +	if (!xe_bo_is_xe_bo(bo) || !xe_bo_get_unless_zero(xe_bo))
> +		return xe_bo_shrink_purge(ctx, bo, scanned);
> +
> +	if (xe_tt->purgeable) {
> +		if (bo->resource->mem_type != XE_PL_SYSTEM)
> +			lret = xe_bo_move_notify(xe_bo, ctx);
> +		if (!lret)
> +			lret = xe_bo_shrink_purge(ctx, bo, scanned);
> +		goto out_unref;
> +	}
> +
> +	/* System CCS needs gpu copy when moving PL_TT -> PL_SYSTEM */
> +	needs_rpm = (!IS_DGFX(xe) && bo->resource->mem_type != XE_PL_SYSTEM &&
> +		     xe_bo_needs_ccs_pages(xe_bo));
> +	if (needs_rpm && !xe_pm_runtime_get_if_active(xe))
> +		goto out_unref;
> +
> +	*scanned += tt->num_pages;
> +	lret = ttm_bo_shrink(ctx, bo, (struct ttm_bo_shrink_flags)
> +			     {.purge = false,
> +			      .writeback = flags.writeback,
> +			      .allow_move = true});
> +	if (needs_rpm)
> +		xe_pm_runtime_put(xe);
> +
> +	if (lret > 0)
> +		xe_ttm_tt_account_subtract(tt);
> +
> +out_unref:
> +	xe_bo_put(xe_bo);
> +
> +	return lret;
> +}
> +
>   /**
>    * xe_bo_evict_pinned() - Evict a pinned VRAM object to system memory
>    * @bo: The buffer object to move.
> @@ -1905,6 +2065,8 @@ int xe_bo_pin_external(struct xe_bo *bo)
>   	}
>   
>   	ttm_bo_pin(&bo->ttm);
> +	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
> +		xe_ttm_tt_account_subtract(bo->ttm.ttm);
>   
>   	/*
>   	 * FIXME: If we always use the reserve / unreserve functions for locking
> @@ -1964,6 +2126,8 @@ int xe_bo_pin(struct xe_bo *bo)
>   	}
>   
>   	ttm_bo_pin(&bo->ttm);
> +	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
> +		xe_ttm_tt_account_subtract(bo->ttm.ttm);
>   
>   	/*
>   	 * FIXME: If we always use the reserve / unreserve functions for locking
> @@ -1998,6 +2162,8 @@ void xe_bo_unpin_external(struct xe_bo *bo)
>   	spin_unlock(&xe->pinned.lock);
>   
>   	ttm_bo_unpin(&bo->ttm);
> +	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
> +		xe_ttm_tt_account_add(bo->ttm.ttm);
>   
>   	/*
>   	 * FIXME: If we always use the reserve / unreserve functions for locking
> @@ -2021,6 +2187,8 @@ void xe_bo_unpin(struct xe_bo *bo)
>   		spin_unlock(&xe->pinned.lock);
>   	}
>   	ttm_bo_unpin(&bo->ttm);
> +	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
> +		xe_ttm_tt_account_add(bo->ttm.ttm);
>   }
>   
>   /**
> diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
> index 04995c5ced32..9ca87973cff7 100644
> --- a/drivers/gpu/drm/xe/xe_bo.h
> +++ b/drivers/gpu/drm/xe/xe_bo.h
> @@ -148,6 +148,28 @@ static inline struct xe_bo *xe_bo_get(struct xe_bo *bo)
>   
>   void xe_bo_put(struct xe_bo *bo);
>   
> +/*
> + * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
> + * xe bo
> + * @bo: The bo for which we want to obtain a refcount.
> + *
> + * There is a short window between where the bo's GEM object refcount reaches
> + * zero and where we put the final ttm_bo reference. Code in the eviction- and
> + * shrinking path should therefore attempt to grab a gem object reference before
> + * trying to use members outside of the base class ttm object. This function is
> + * intended for that purpose. On successful return, this function must be paired
> + * with an xe_bo_put().
> + *
> + * Return: @bo on success, NULL on failure.
> + */
> +static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)
> +{
> +	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
> +		return NULL;
> +
> +	return bo;
> +}
> +
>   static inline void __xe_bo_unset_bulk_move(struct xe_bo *bo)
>   {
>   	if (bo)
> @@ -343,6 +365,20 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)
>   	return round_down(max / 2, PAGE_SIZE);
>   }
>   
> +/**
> + * struct xe_bo_shrink_flags - flags governing the shrink behaviour.
> + * @purge: Only purging allowed. Don't shrink if bo not purgeable.
> + * @writeback: Attempt to immediately move content to swap.
> + */
> +struct xe_bo_shrink_flags {
> +	u32 purge : 1;
> +	u32 writeback : 1;
> +};
> +
> +long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
> +		  const struct xe_bo_shrink_flags flags,
> +		  unsigned long *scanned);
> +
>   #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
>   /**
>    * xe_bo_is_mem_type - Whether the bo currently resides in the given
> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index 8fedc72e9db4..c4a6062124b0 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
> @@ -51,6 +51,7 @@
>   #include "xe_pm.h"
>   #include "xe_pmu.h"
>   #include "xe_query.h"
> +#include "xe_shrinker.h"
>   #include "xe_sriov.h"
>   #include "xe_survivability_mode.h"
>   #include "xe_tile.h"
> @@ -396,6 +397,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
>   	if (xe->unordered_wq)
>   		destroy_workqueue(xe->unordered_wq);
>   
> +	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
> +		xe_shrinker_destroy(xe->mem.shrinker);
> +
>   	if (xe->destroy_wq)
>   		destroy_workqueue(xe->destroy_wq);
>   
> @@ -428,6 +432,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
>   	if (err)
>   		goto err;
>   
> +	xe->mem.shrinker = xe_shrinker_create(xe);
> +	if (IS_ERR(xe->mem.shrinker))
> +		return ERR_CAST(xe->mem.shrinker);
> +
>   	xe->info.devid = pdev->device;
>   	xe->info.revid = pdev->revision;
>   	xe->info.force_execlist = xe_modparam.force_execlist;
> diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
> index 89f532b67bc4..7406deac0fec 100644
> --- a/drivers/gpu/drm/xe/xe_device_types.h
> +++ b/drivers/gpu/drm/xe/xe_device_types.h
> @@ -369,6 +369,8 @@ struct xe_device {
>   		struct xe_mem_region vram;
>   		/** @mem.sys_mgr: system TTM manager */
>   		struct ttm_resource_manager sys_mgr;
> +		/** @mem.sys_mgr: system memory shrinker. */
> +		struct xe_shrinker *shrinker;
>   	} mem;
>   
>   	/** @sriov: device level virtualization data */
> diff --git a/drivers/gpu/drm/xe/xe_shrinker.c b/drivers/gpu/drm/xe/xe_shrinker.c
> new file mode 100644
> index 000000000000..8184390f9c7b
> --- /dev/null
> +++ b/drivers/gpu/drm/xe/xe_shrinker.c
> @@ -0,0 +1,258 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright © 2024 Intel Corporation
> + */
> +
> +#include <linux/shrinker.h>
> +
> +#include <drm/ttm/ttm_backup.h>
> +#include <drm/ttm/ttm_bo.h>
> +#include <drm/ttm/ttm_tt.h>
> +
> +#include "xe_bo.h"
> +#include "xe_pm.h"
> +#include "xe_shrinker.h"
> +
> +/**
> + * struct xe_shrinker - per-device shrinker
> + * @xe: Back pointer to the device.
> + * @lock: Lock protecting accounting.
> + * @shrinkable_pages: Number of pages that are currently shrinkable.
> + * @purgeable_pages: Number of pages that are currently purgeable.
> + * @shrink: Pointer to the mm shrinker.
> + * @pm_worker: Worker to wake up the device if required.
> + */
> +struct xe_shrinker {
> +	struct xe_device *xe;
> +	rwlock_t lock;
> +	long shrinkable_pages;
> +	long purgeable_pages;
> +	struct shrinker *shrink;
> +	struct work_struct pm_worker;
> +};
> +
> +static struct xe_shrinker *to_xe_shrinker(struct shrinker *shrink)
> +{
> +	return shrink->private_data;
> +}
> +
> +/**
> + * xe_shrinker_mod_pages() - Modify shrinker page accounting
> + * @shrinker: Pointer to the struct xe_shrinker.
> + * @shrinkable: Shrinkable pages delta. May be negative.
> + * @purgeable: Purgeable page delta. May be negative.
> + *
> + * Modifies the shrinkable and purgeable pages accounting.
> + */
> +void
> +xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable)
> +{
> +	write_lock(&shrinker->lock);
> +	shrinker->shrinkable_pages += shrinkable;
> +	shrinker->purgeable_pages += purgeable;
> +	write_unlock(&shrinker->lock);
> +}
> +
> +static s64 xe_shrinker_walk(struct xe_device *xe,
> +			    struct ttm_operation_ctx *ctx,
> +			    const struct xe_bo_shrink_flags flags,
> +			    unsigned long to_scan, unsigned long *scanned)
> +{
> +	unsigned int mem_type;
> +	s64 freed = 0, lret;
> +
> +	for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT; ++mem_type) {
> +		struct ttm_resource_manager *man = ttm_manager_type(&xe->ttm, mem_type);
> +		struct ttm_bo_lru_cursor curs;
> +		struct ttm_buffer_object *ttm_bo;
> +
> +		if (!man || !man->use_tt)
> +			continue;
> +
> +		ttm_bo_lru_for_each_reserved_guarded(&curs, man, ctx, ttm_bo) {
> +			if (!ttm_bo_shrink_suitable(ttm_bo, ctx))
> +				continue;
> +
> +			lret = xe_bo_shrink(ctx, ttm_bo, flags, scanned);
> +			if (lret < 0)
> +				return lret;
> +
> +			freed += lret;
> +			if (*scanned >= to_scan)
> +				break;
> +		}
> +	}
> +
> +	return freed;
> +}
> +
> +static unsigned long
> +xe_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
> +	unsigned long num_pages;
> +	bool can_backup = !!(sc->gfp_mask & __GFP_FS);
> +
> +	num_pages = ttm_backup_bytes_avail() >> PAGE_SHIFT;
> +	read_lock(&shrinker->lock);
> +
> +	if (can_backup)
> +		num_pages = min_t(unsigned long, num_pages, shrinker->shrinkable_pages);
> +	else
> +		num_pages = 0;
> +
> +	num_pages += shrinker->purgeable_pages;
> +	read_unlock(&shrinker->lock);
> +
> +	return num_pages ? num_pages : SHRINK_EMPTY;
> +}
> +
> +/*
> + * Check if we need runtime pm, and if so try to grab a reference if
> + * already active. If grabbing a reference fails, queue a worker that
> + * does it for us outside of reclaim, but don't wait for it to complete.
> + * If bo shrinking needs an rpm reference and we don't have it (yet),
> + * that bo will be skipped anyway.
> + */
> +static bool xe_shrinker_runtime_pm_get(struct xe_shrinker *shrinker, bool force,
> +				       unsigned long nr_to_scan, bool can_backup)
> +{
> +	struct xe_device *xe = shrinker->xe;
> +
> +	if (IS_DGFX(xe) || !xe_device_has_flat_ccs(xe) ||
> +	    !ttm_backup_bytes_avail())
> +		return false;
> +
> +	if (!force) {
> +		read_lock(&shrinker->lock);
> +		force = (nr_to_scan > shrinker->purgeable_pages && can_backup);
> +		read_unlock(&shrinker->lock);
> +		if (!force)
> +			return false;
> +	}
> +
> +	if (!xe_pm_runtime_get_if_active(xe)) {
> +		if (xe_rpm_reclaim_safe(xe) && !ttm_bo_shrink_avoid_wait()) {
> +			xe_pm_runtime_get(xe);
> +			return true;
> +		}
> +		queue_work(xe->unordered_wq, &shrinker->pm_worker);
> +		return false;
> +	}
> +
> +	return true;
> +}
> +
> +static void xe_shrinker_runtime_pm_put(struct xe_shrinker *shrinker, bool runtime_pm)
> +{
> +	if (runtime_pm)
> +		xe_pm_runtime_put(shrinker->xe);
> +}
> +
> +static unsigned long xe_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
> +{
> +	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
> +	struct ttm_operation_ctx ctx = {
> +		.interruptible = false,
> +		.no_wait_gpu = ttm_bo_shrink_avoid_wait(),
> +	};
> +	unsigned long nr_to_scan, nr_scanned = 0, freed = 0;
> +	struct xe_bo_shrink_flags shrink_flags = {
> +		.purge = true,
> +		/* Don't request writeback without __GFP_IO. */
> +		.writeback = !ctx.no_wait_gpu && (sc->gfp_mask & __GFP_IO),
> +	};
> +	bool runtime_pm;
> +	bool purgeable;
> +	bool can_backup = !!(sc->gfp_mask & __GFP_FS);
> +	s64 lret;
> +
> +	nr_to_scan = sc->nr_to_scan;
> +
> +	read_lock(&shrinker->lock);
> +	purgeable = !!shrinker->purgeable_pages;
> +	read_unlock(&shrinker->lock);
> +
> +	/* Might need runtime PM. Try to wake early if it looks like it. */
> +	runtime_pm = xe_shrinker_runtime_pm_get(shrinker, false, nr_to_scan, can_backup);
> +
> +	if (purgeable && nr_scanned < nr_to_scan) {
> +		lret = xe_shrinker_walk(shrinker->xe, &ctx, shrink_flags,
> +					nr_to_scan, &nr_scanned);
> +		if (lret >= 0)
> +			freed += lret;
> +	}
> +
> +	sc->nr_scanned = nr_scanned;
> +	if (nr_scanned >= nr_to_scan || !can_backup)
> +		goto out;
> +
> +	/* If we didn't wake before, try to do it now if needed. */
> +	if (!runtime_pm)
> +		runtime_pm = xe_shrinker_runtime_pm_get(shrinker, true, 0, can_backup);
> +
> +	shrink_flags.purge = false;
> +	lret = xe_shrinker_walk(shrinker->xe, &ctx, shrink_flags,
> +				nr_to_scan, &nr_scanned);
> +	if (lret >= 0)
> +		freed += lret;
> +
> +	sc->nr_scanned = nr_scanned;
> +out:
> +	xe_shrinker_runtime_pm_put(shrinker, runtime_pm);
> +	return nr_scanned ? freed : SHRINK_STOP;
> +}
> +
> +/* Wake up the device for shrinking. */
> +static void xe_shrinker_pm(struct work_struct *work)
> +{
> +	struct xe_shrinker *shrinker =
> +		container_of(work, typeof(*shrinker), pm_worker);
> +
> +	xe_pm_runtime_get(shrinker->xe);
> +	xe_pm_runtime_put(shrinker->xe);
> +}
> +
> +/**
> + * xe_shrinker_create() - Create an xe per-device shrinker
> + * @xe: Pointer to the xe device.
> + *
> + * Returns: A pointer to the created shrinker on success,
> + * Negative error code on failure.
> + */
> +struct xe_shrinker *xe_shrinker_create(struct xe_device *xe)
> +{
> +	struct xe_shrinker *shrinker = kzalloc(sizeof(*shrinker), GFP_KERNEL);
> +
> +	if (!shrinker)
> +		return ERR_PTR(-ENOMEM);
> +
> +	shrinker->shrink = shrinker_alloc(0, "xe system shrinker");
> +	if (!shrinker->shrink) {
> +		kfree(shrinker);
> +		return ERR_PTR(-ENOMEM);
> +	}
> +
> +	INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm);
> +	shrinker->xe = xe;
> +	rwlock_init(&shrinker->lock);
> +	shrinker->shrink->count_objects = xe_shrinker_count;
> +	shrinker->shrink->scan_objects = xe_shrinker_scan;
> +	shrinker->shrink->private_data = shrinker;
> +	shrinker_register(shrinker->shrink);
> +
> +	return shrinker;
> +}
> +
> +/**
> + * xe_shrinker_destroy() - Destroy an xe per-device shrinker
> + * @shrinker: Pointer to the shrinker to destroy.
> + */
> +void xe_shrinker_destroy(struct xe_shrinker *shrinker)
> +{
> +	xe_assert(shrinker->xe, !shrinker->shrinkable_pages);
> +	xe_assert(shrinker->xe, !shrinker->purgeable_pages);
> +	shrinker_free(shrinker->shrink);
> +	flush_work(&shrinker->pm_worker);
> +	kfree(shrinker);
> +}
> diff --git a/drivers/gpu/drm/xe/xe_shrinker.h b/drivers/gpu/drm/xe/xe_shrinker.h
> new file mode 100644
> index 000000000000..28a038f4fcbf
> --- /dev/null
> +++ b/drivers/gpu/drm/xe/xe_shrinker.h
> @@ -0,0 +1,18 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2024 Intel Corporation
> + */
> +
> +#ifndef _XE_SHRINKER_H_
> +#define _XE_SHRINKER_H_
> +
> +struct xe_shrinker;
> +struct xe_device;
> +
> +void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable);
> +
> +struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
> +
> +void xe_shrinker_destroy(struct xe_shrinker *shrinker);
> +
> +#endif