Re: [PATCH] drm/xe: Fix unnecessary host-side population of ttm_tt on non-TT resources

["Thomas Hellström" <thomas.hellstrom@linux.intel.com>]

On Thu, 2026-08-20 at 14:56 -0400, Nathan Bourgeois wrote:
> Hi,
> Sorry, I replied only to Matt by mistake earlier and didn't realize
> it
> was text-only. Here's the same reply for the list:
> 
> Hi Matt,
> 
> Thank you so much for reviewing this!
> 
> > Reading the commit which added ttm_bo_setup_export, to me this
> > looks
> > like a bug in ttm_bo_setup_export() or in general TTM.
> 
> > 'This only applies currently to TTM_PL_SYSTEM objects, because
> > GTT objects get populated on first validate, and VRAM doesn't
> > use TT.'
> 
> > I'd probably move this fix into ttm_bo_setup_export() and post it
> > for
> > discussion.
> 
> I agree that this is a much cleaner and smaller fix, I initially
> wanted to keep
> the change scoped only to xe where it was causing an observable
> issue,
> but I will move it into ttm_bo_setup_export() given your
> recommendation.
> 
> > It may also be the case that the ttm_tt preallocated with
> > XE_BO_FLAG_DEFER_BACKING should be dropped once the BO moves to
> > VRAM, as
> > that dangling ttm_tt could cause other issues. I'll need to take a
> > closer look at that angle.
> 
> Yes, considering that this is essentially the source of this
> particular behavior
> it stands to reason that there may be other latent correctness issues
> caused
> by the dangling ttm_tt.
> 
> > This part looks good as different patch from what I'm assuming will
> > be a
> > TTM fix.
> 
> Yes, I'll submit a patch with RFC for discussing the
> ttm_bo_setup_export()
> changes and a second patch for the unit test.
> 
> Best Regards,
> Nathan Bourgeois
> 
> 
> On Thu, Aug 20, 2026 at 2:01 PM Nathan Bourgeois
> <iridescentrosesfall@gmail.com> wrote:
> > 
> > Hi,
> > 
> > Sorry, I replied only to Matt by mistake earlier. Here's the same
> > reply for the list:
> > 
> > Hi Matt, Thank you so much for reviewing this! > Reading the commit
> > which added ttm_bo_setup_export, to me this looks
> > > like a bug in ttm_bo_setup_export() or in general TTM.
> > 
> > > 'This only applies currently to TTM_PL_SYSTEM objects, because
> > > GTT objects get populated on first validate, and VRAM doesn't
> > > use TT.'
> > 
> > > I'd probably move this fix into ttm_bo_setup_export() and post it
> > > for
> > > discussion.
> > 
> > I agree that this is a much cleaner and smaller fix, I initially
> > wanted to keep
> > the change scoped only to xe where it was causing an observable
> > issue,
> > but I will move it into ttm_bo_setup_export() given your
> > recommendation.
> > 
> > > It may also be the case that the ttm_tt preallocated with
> > > XE_BO_FLAG_DEFER_BACKING should be dropped once the BO moves to
> > > VRAM, as
> > > that dangling ttm_tt could cause other issues. I'll need to take
> > > a
> > > closer look at that angle.
> > 
> > Yes, considering that this is essentially the source of this
> > particular behavior
> > it stands to reason that there may be other latent correctness
> > issues caused
> > by the dangling ttm_tt.
> > 
> > > This part looks good as different patch from what I'm assuming
> > > will be a
> > > TTM fix.
> > 
> > Yes, I'll submit a patch with RFC for discussing the
> > ttm_bo_setup_export()
> > changes and a second patch for the unit test.
> > 
> > Best Regards,
> > Nathan Bourgeois

When I read the purpose of the original commit, I wonder if this is the
correct solution.

Regardless of whether the bo will eventually be attached as a VRAM
(p2p) dma-buf or a system dma-buf, we essentially want to charge
upfront.

Shouldn't we just be calling xe_bo_validate() here instead of
ttm_tt_populate? (With the correct xe_validation_guard() wrapping).

 If it's a system bo, ttm_tt_populate would eventually be called. If
it's a VRAM bo, we'd correctly charge dmemcg, and if the bo gets
migrated to system as part of attach, whatever cgroup controller will
be used for system memory will most likely charge the same cgroup that
got charged for the VRAM.

/Thomas

> > 
> > On Thu, Aug 20, 2026 at 1:03 PM Matthew Brost
> > <matthew.brost@intel.com> wrote:
> > > 
> > > On Wed, Aug 19, 2026 at 11:19:01PM -0400, Nathan Bourgeois wrote:
> > > 
> > > Thanks the patch.
> > > 
> > > > When a buffer object (BO) has a created but unpopulated ttm_tt
> > > > and is
> > > > exported, the default behavior populates the ttm_tt, even if
> > > > the manager
> > > > does not require a TT. This unnecessary host-side population
> > > > reduces
> > > > host memory available to the user.
> > > > 
> > > > This occurs when a BO is created with XE_BO_FLAG_DEFER_BACKING
> > > > and
> > > > migration places the BO into a resource whose manager does not
> > > > use the
> > > > TT backing. The retained ttm_tt is not authoritative for
> > > > storage, yet
> > > > the default ttm_bo_setup_export() treats the existence of the
> > > > ttm_tt as
> > > > requiring population.
> > > > 
> > > 
> > > Thanks for explaining this, I see the problem.
> > > 
> > > > The issue was reproduced with vLLM 0.27.1 loading
> > > > 0xSer0/DeepSeek-V4-Flash-180B (d3c704b) on a system with 128 GB
> > > > of RAM
> > > > and six Intel Arc Pro B70 GPUs providing 192 GB of VRAM.
> > > > Loading the
> > > > weights (100.61 GB) triggered host-memory exhaustion and OOM
> > > > events on
> > > > the baseline kernel.
> > > > 
> > > > On Ubuntu 24.04 with 7.0-12-generic, the vLLM service cgroup
> > > > grew by
> > > > 115.84 GB on the instrumented baseline and by 15.34 GB with
> > > > this change,
> > > > a reduction of 100.50 GB. Diagnostic instrumentation counting
> > > > cumulative
> > > > TT pages across all six GPUs recorded 109.22 GB on baseline and
> > > > 8.72 GB
> > > > with this change. The reduction is 0.11 GB less than the model
> > > > weight
> > > > size.
> > > > 
> > > > The change creates an Xe-local export setup helper which skips
> > > > ttm_bo_populate() when the BO has a current resource manager
> > > > and if that
> > > > manager has use_tt == false. If the manager does not use TT,
> > > > then
> > > > population is skipped, while an absent manager conservatively
> > > > calls
> > > > ttm_bo_populate(). Behavior for a manager which requires a TT
> > > > backing
> > > > remains unchanged. Keep this policy local to Xe so other TTM
> > > > drivers
> > > > retain their existing export behavior.
> > > > 
> > > 
> > > Reading the commit which added ttm_bo_setup_export, to me this
> > > looks
> > > like a bug in ttm_bo_setup_export() or in general TTM.
> > > 
> > > git format-patch -1 50243079865ae
> > > 
> > > 'This only applies currently to TTM_PL_SYSTEM objects, because
> > > GTT objects get populated on first validate, and VRAM doesn't
> > > use TT.'
> > > 
> > > I'd probably move this fix into ttm_bo_setup_export() and post it
> > > for
> > > discussion.
> > > 
> > > It may also be the case that the ttm_tt preallocated with
> > > XE_BO_FLAG_DEFER_BACKING should be dropped once the BO moves to
> > > VRAM, as
> > > that dangling ttm_tt could cause other issues. I'll need to take
> > > a
> > > closer look at that angle.
> > > 
> > > > The regression test creates both the control case and the VRAM
> > > > migration
> > > > case. The control case creates the deferred BO in system RAM
> > > > and leaves
> > > > it there, which requires a TT and population when
> > > > xe_gem_prime_export() is
> > > > called. The VRAM migration case creates the deferred BO in
> > > > system RAM
> > > > and then migrates it to VRAM, which does not require a TT and
> > > > thus, when
> > > > xe_gem_prime_export() is called, no population occurs.
> > > > 
> > > 
> > > This part looks good as different patch from what I'm assuming
> > > will be a
> > > TTM fix.
> > > 
> > > Matt
> > > 
> > > > Fixes: 91494dee1091 ("xe: populate buffers before exporting
> > > > them.")
> > > > Cc: stable@vger.kernel.org
> > > > Assisted-by: Codex:gpt-5.6-sol
> > > > Signed-off-by: Nathan Bourgeois <iridescentrosesfall@gmail.com>
> > > > ---
> > > > Newer Testing:
> > > > - Built and booted b4f95affc66ef76342c1f6bf3849f6c8ade6b9d6
> > > > plus this patch on EPYC 7352 with six Intel Arc Pro B70.
> > > > - KUnit xe_live_test: xe_dma_buf_kunit: pass:6 fail:0 skip:0
> > > > total:6 PASS
> > > > - vLLM 0.27.1 loading DeepSeek-V4-Flash-0731 (full model
> > > > compared to REAP) on six Intel Arc Pro B70 GPUs: No OOM error,
> > > > 17.82 GB delta.
> > > >  drivers/gpu/drm/xe/tests/xe_dma_buf.c | 105
> > > > ++++++++++++++++++++++++++
> > > >  drivers/gpu/drm/xe/xe_dma_buf.c       |  28 ++++++-
> > > >  2 files changed, 132 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/gpu/drm/xe/tests/xe_dma_buf.c
> > > > b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
> > > > index 0be8440b3976..bff803ab7a57 100644
> > > > --- a/drivers/gpu/drm/xe/tests/xe_dma_buf.c
> > > > +++ b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
> > > > @@ -260,12 +260,117 @@ static const struct dma_buf_test_params
> > > > test_params[] = {
> > > >       {}
> > > >  };
> > > > 
> > > > +static void xe_test_dmabuf_export_deferred(struct xe_device
> > > > *xe, u32 bo_flags,
> > > > +                                        u32 mem_type, bool
> > > > expect_populated)
> > > > +{
> > > > +     struct drm_exec *exec = XE_VALIDATION_OPT_OUT;
> > > > +     struct kunit *test = kunit_get_current_test();
> > > > +     struct ttm_resource_manager *man;
> > > > +     struct dma_buf *dmabuf;
> > > > +     struct xe_bo *bo;
> > > > +     size_t size = PAGE_SIZE;
> > > > +     int err;
> > > > +
> > > > +     /* No VRAM on device? */
> > > > +     if (!ttm_manager_type(&xe->ttm, mem_type))
> > > > +             return;
> > > > +
> > > > +     if (mem_type == XE_PL_VRAM0 &&
> > > > +         xe->info.vram_flags & XE_VRAM_FLAGS_NEED64K)
> > > > +             size = SZ_64K;
> > > > +
> > > > +     /*
> > > > +      * DEFER_BACKING places the BO in SYSTEM and creates a
> > > > ttm_tt which
> > > > +      * is unpopulated. In the case of VRAM, migrating leaves
> > > > the ttm_tt
> > > > +      * retained and unpopulated while VRAM becomes the real
> > > > backing.
> > > > +      */
> > > > +     bo = xe_bo_create_user(xe, NULL, size,
> > > > DRM_XE_GEM_CPU_CACHING_WC,
> > > > +                            bo_flags |
> > > > XE_BO_FLAG_DEFER_BACKING, NULL);
> > > > +     if (IS_ERR(bo)) {
> > > > +             KUNIT_FAIL(test, "BO creation failed: %pe\n",
> > > > bo);
> > > > +             return;
> > > > +     }
> > > > +
> > > > +     err = xe_bo_lock(bo, false);
> > > > +     if (err) {
> > > > +             KUNIT_FAIL(test, "BO lock failed: %d\n", err);
> > > > +             goto out_put_bo;
> > > > +     }
> > > > +
> > > > +     if (bo->ttm.resource->mem_type != mem_type)
> > > > +             err = xe_bo_migrate(bo, mem_type, NULL, exec);
> > > > +     if (err) {
> > > > +             KUNIT_FAIL(test, "BO migration to %u failed:
> > > > %d\n", mem_type,
> > > > +                        err);
> > > > +             goto out_unlock;
> > > > +     }
> > > > +
> > > > +     man = ttm_manager_type(bo->ttm.bdev, bo->ttm.resource-
> > > > >mem_type);
> > > > +     if (!man || !bo->ttm.ttm) {
> > > > +             KUNIT_FAIL(test, "Expected a retained unpopulated
> > > > TT\n");
> > > > +             goto out_unlock;
> > > > +     }
> > > > +
> > > > +     /* Precondition: ttm_tt starts unpopulated after
> > > > migration */
> > > > +     KUNIT_EXPECT_EQ(test, man->use_tt, expect_populated);
> > > > +     KUNIT_EXPECT_FALSE(test, ttm_tt_is_populated(bo-
> > > > >ttm.ttm));
> > > > +
> > > > +     xe_bo_unlock(bo);
> > > > +
> > > > +     dmabuf = xe_gem_prime_export(&bo->ttm.base, 0);
> > > > +     if (IS_ERR(dmabuf)) {
> > > > +             KUNIT_FAIL(test, "dma-buf export failed: %pe\n",
> > > > dmabuf);
> > > > +             goto out_put_bo;
> > > > +     }
> > > > +
> > > > +     err = xe_bo_lock(bo, false);
> > > > +     if (err) {
> > > > +             KUNIT_FAIL(test, "post-export BO lock failed:
> > > > %d\n", err);
> > > > +             goto out_put_dmabuf;
> > > > +     }
> > > > +
> > > > +     /* Postcondition: if VRAM, ttm_tt remains unpopulated, if
> > > > SYSTEM ttm_tt is populated */
> > > > +     KUNIT_EXPECT_EQ(test, bo->ttm.resource->mem_type,
> > > > mem_type);
> > > > +     KUNIT_EXPECT_NOT_NULL(test, bo->ttm.ttm);
> > > > +     if (bo->ttm.ttm)
> > > > +             KUNIT_EXPECT_EQ(test, ttm_tt_is_populated(bo-
> > > > >ttm.ttm),
> > > > +                             expect_populated);
> > > > +
> > > > +     xe_bo_unlock(bo);
> > > > +     dma_buf_put(dmabuf);
> > > > +     drm_gem_object_put(&bo->ttm.base);
> > > > +     return;
> > > > +
> > > > +out_unlock:
> > > > +     xe_bo_unlock(bo);
> > > > +     goto out_put_bo;
> > > > +out_put_dmabuf:
> > > > +     dma_buf_put(dmabuf);
> > > > +out_put_bo:
> > > > +     drm_gem_object_put(&bo->ttm.base);
> > > > +}
> > > > +
> > > >  static int dma_buf_run_device(struct xe_device *xe)
> > > >  {
> > > >       const struct dma_buf_test_params *params;
> > > >       struct kunit *test = kunit_get_current_test();
> > > > 
> > > >       guard(xe_pm_runtime)(xe);
> > > > +
> > > > +     /*
> > > > +      * A retained TT must not be populated when VRAM is the
> > > > backing
> > > > +      * resource.
> > > > +      */
> > > > +     xe_test_dmabuf_export_deferred(xe, XE_BO_FLAG_VRAM0,
> > > > XE_PL_VRAM0,
> > > > +                                    false);
> > > > +
> > > > +     /*
> > > > +      * Control case: deferred SYSTEM backing must still be
> > > > populated
> > > > +      * before export.
> > > > +      */
> > > > +     xe_test_dmabuf_export_deferred(xe, XE_BO_FLAG_SYSTEM,
> > > > XE_PL_SYSTEM,
> > > > +                                    true);
> > > > +
> > > >       for (params = test_params; params->mem_mask; ++params) {
> > > >               struct dma_buf_test_params p = *params;
> > > > 
> > > > diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c
> > > > b/drivers/gpu/drm/xe/xe_dma_buf.c
> > > > index bf0728838ead..0c4e4e2e1a81 100644
> > > > --- a/drivers/gpu/drm/xe/xe_dma_buf.c
> > > > +++ b/drivers/gpu/drm/xe/xe_dma_buf.c
> > > > @@ -219,6 +219,32 @@ static const struct dma_buf_ops
> > > > xe_dmabuf_ops = {
> > > >       .vunmap = drm_gem_dmabuf_vunmap,
> > > >  };
> > > > 
> > > > +static int xe_dma_bo_setup_export(struct ttm_buffer_object
> > > > *tbo,
> > > > +                               struct ttm_operation_ctx *ctx)
> > > > +{
> > > > +     struct ttm_resource_manager *man = NULL;
> > > > +     int ret;
> > > > +
> > > > +     ret = ttm_bo_reserve(tbo, false, false, NULL);
> > > > +     if (ret)
> > > > +             return ret;
> > > > +
> > > > +     if (tbo->resource)
> > > > +             man = ttm_manager_type(tbo->bdev, tbo->resource-
> > > > >mem_type);
> > > > +
> > > > +     /*
> > > > +      * Do not populate BO-sized system pages when backed by a
> > > > non-TT resource.
> > > > +      * This is Xe-specific; the generic ttm_bo_setup_export()
> > > > always populates.
> > > > +      */
> > > > +     if (man && !man->use_tt)
> > > > +             ret = 0;
> > > > +     else
> > > > +             ret = ttm_bo_populate(tbo, ctx);
> > > > +
> > > > +     ttm_bo_unreserve(tbo);
> > > > +     return ret;
> > > > +}
> > > > +
> > > >  struct dma_buf *xe_gem_prime_export(struct drm_gem_object
> > > > *obj, int flags)
> > > >  {
> > > >       struct xe_bo *bo = gem_to_xe_bo(obj);
> > > > @@ -257,7 +283,7 @@ struct dma_buf *xe_gem_prime_export(struct
> > > > drm_gem_object *obj, int flags)
> > > >       xe_bo_willneed_get_locked(bo);
> > > >       xe_bo_unlock(bo);
> > > > 
> > > > -     ret = ttm_bo_setup_export(&bo->ttm, &ctx);
> > > > +     ret = xe_dma_bo_setup_export(&bo->ttm, &ctx);
> > > >       if (ret)
> > > >               goto out_put;
> > > > 
> > > > 
> > > > base-commit: b4f95affc66ef76342c1f6bf3849f6c8ade6b9d6
> > > > --
> > > > 2.55.0
> > > >