All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@nvidia.com>
To: Daniel Vetter <daniel@ffwll.ch>
Cc: John Hubbard <jhubbard@nvidia.com>,
	Alex Deucher <alexdeucher@gmail.com>,
	Leon Romanovsky <leon@kernel.org>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	Maling list - DRI developers  <dri-devel@lists.freedesktop.org>,
	Doug Ledford <dledford@redhat.com>,
	Daniel Vetter <daniel.vetter@intel.com>,
	"Christian Koenig" <christian.koenig@amd.com>,
	Jianxin Xiong <jianxin.xiong@intel.com>
Subject: Re: [PATCH v16 0/4] RDMA: Add dma-buf support
Date: Fri, 5 Feb 2021 12:00:03 -0400	[thread overview]
Message-ID: <20210205160003.GU4247@nvidia.com> (raw)
In-Reply-To: <YB1p4Bpmz0yFcbEf@phenom.ffwll.local>

On Fri, Feb 05, 2021 at 04:53:04PM +0100, Daniel Vetter wrote:
> On Fri, Feb 05, 2021 at 11:43:19AM -0400, Jason Gunthorpe wrote:
> > On Fri, Feb 05, 2021 at 04:39:47PM +0100, Daniel Vetter wrote:
> > 
> > > > And again, for slightly older hardware, without pinning to VRAM there is
> > > > no way to use this solution here for peer-to-peer. So I'm glad to see that
> > > > so far you're not ruling out the pinning option.
> > > 
> > > Since HMM and ZONE_DEVICE came up, I'm kinda tempted to make ZONE_DEVICE
> > > ZONE_MOVEABLE (at least if you don't have a pinned vram contigent in your
> > > cgroups) or something like that, so we could benefit from the work to make
> > > sure pin_user_pages and all these never end up in there?
> > 
> > ZONE_DEVICE should already not be returned from GUP.
> > 
> > I've understood in the hmm casse the idea was a CPU touch of some
> > ZONE_DEVICE pages would trigger a migration to CPU memory, GUP would
> > want to follow the same logic, presumably it comes for free with the
> > fault handler somehow
> 
> Oh I didn't know this, I thought the proposed p2p direct i/o patches would
> just use the fact that underneath ZONE_DEVICE there's "normal" struct
> pages. 

So, if that every happens, it would be some special FOLL_ALLOW_P2P
flag to get the behavior.

> And so I got worried that maybe also pin_user_pages can creep in.
> But I didn't read the patches in full detail:

And yes, you might want to say that you can't longterm pin certain
kinds of zone_device pages, but if that is the common operating mode
then we'd probably never create a FOLL_ALLOW_P2P

> But if you're saying that this all needs specific code and all the gup/pup
> code we have is excluded, I think we can make sure that we're not ever
> building features that requiring time-unlimited pinning of
> ZONE_DEVICE.

Well, it is certainly a useful idea of some uses of ZONE_DEVICE, GPU
vram is not the whole world.

Jason

WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@nvidia.com>
To: Daniel Vetter <daniel@ffwll.ch>
Cc: Leon Romanovsky <leon@kernel.org>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	John Hubbard <jhubbard@nvidia.com>,
	Maling list - DRI developers <dri-devel@lists.freedesktop.org>,
	Doug Ledford <dledford@redhat.com>,
	Daniel Vetter <daniel.vetter@intel.com>,
	Christian Koenig <christian.koenig@amd.com>,
	Jianxin Xiong <jianxin.xiong@intel.com>
Subject: Re: [PATCH v16 0/4] RDMA: Add dma-buf support
Date: Fri, 5 Feb 2021 12:00:03 -0400	[thread overview]
Message-ID: <20210205160003.GU4247@nvidia.com> (raw)
In-Reply-To: <YB1p4Bpmz0yFcbEf@phenom.ffwll.local>

On Fri, Feb 05, 2021 at 04:53:04PM +0100, Daniel Vetter wrote:
> On Fri, Feb 05, 2021 at 11:43:19AM -0400, Jason Gunthorpe wrote:
> > On Fri, Feb 05, 2021 at 04:39:47PM +0100, Daniel Vetter wrote:
> > 
> > > > And again, for slightly older hardware, without pinning to VRAM there is
> > > > no way to use this solution here for peer-to-peer. So I'm glad to see that
> > > > so far you're not ruling out the pinning option.
> > > 
> > > Since HMM and ZONE_DEVICE came up, I'm kinda tempted to make ZONE_DEVICE
> > > ZONE_MOVEABLE (at least if you don't have a pinned vram contigent in your
> > > cgroups) or something like that, so we could benefit from the work to make
> > > sure pin_user_pages and all these never end up in there?
> > 
> > ZONE_DEVICE should already not be returned from GUP.
> > 
> > I've understood in the hmm casse the idea was a CPU touch of some
> > ZONE_DEVICE pages would trigger a migration to CPU memory, GUP would
> > want to follow the same logic, presumably it comes for free with the
> > fault handler somehow
> 
> Oh I didn't know this, I thought the proposed p2p direct i/o patches would
> just use the fact that underneath ZONE_DEVICE there's "normal" struct
> pages. 

So, if that every happens, it would be some special FOLL_ALLOW_P2P
flag to get the behavior.

> And so I got worried that maybe also pin_user_pages can creep in.
> But I didn't read the patches in full detail:

And yes, you might want to say that you can't longterm pin certain
kinds of zone_device pages, but if that is the common operating mode
then we'd probably never create a FOLL_ALLOW_P2P

> But if you're saying that this all needs specific code and all the gup/pup
> code we have is excluded, I think we can make sure that we're not ever
> building features that requiring time-unlimited pinning of
> ZONE_DEVICE.

Well, it is certainly a useful idea of some uses of ZONE_DEVICE, GPU
vram is not the whole world.

Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

  reply	other threads:[~2021-02-05 22:28 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-15 21:27 [PATCH v16 0/4] RDMA: Add dma-buf support Jianxin Xiong
2020-12-15 21:27 ` Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 1/4] RDMA/umem: Support importing dma-buf as user memory region Jianxin Xiong
2020-12-15 21:27   ` Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 2/4] RDMA/core: Add device method for registering dma-buf based " Jianxin Xiong
2020-12-15 21:27   ` Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 3/4] RDMA/uverbs: Add uverbs command for dma-buf based MR registration Jianxin Xiong
2020-12-15 21:27   ` Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 4/4] RDMA/mlx5: Support dma-buf based userspace memory region Jianxin Xiong
2020-12-15 21:27   ` Jianxin Xiong
2021-01-11 15:24 ` [PATCH v16 0/4] RDMA: Add dma-buf support Xiong, Jianxin
2021-01-11 15:24   ` Xiong, Jianxin
2021-01-11 15:42   ` Jason Gunthorpe
2021-01-11 15:42     ` Jason Gunthorpe
2021-01-11 17:44     ` Xiong, Jianxin
2021-01-11 17:44       ` Xiong, Jianxin
2021-01-11 17:47       ` Alex Deucher
2021-01-11 17:47         ` Alex Deucher
2021-01-11 17:55         ` Xiong, Jianxin
2021-01-11 17:55           ` Xiong, Jianxin
2021-01-12 12:49           ` Yishai Hadas
2021-01-12 12:49             ` Yishai Hadas
2021-01-12 18:11             ` Xiong, Jianxin
2021-01-12 18:11               ` Xiong, Jianxin
2021-01-21 16:59 ` Jason Gunthorpe
2021-01-21 16:59   ` Jason Gunthorpe
2021-02-04  7:48 ` John Hubbard
2021-02-04  7:48   ` John Hubbard
2021-02-04 13:50   ` Alex Deucher
2021-02-04 13:50     ` Alex Deucher
2021-02-04 18:29     ` Jason Gunthorpe
2021-02-04 18:29       ` Jason Gunthorpe
2021-02-04 18:44       ` Alex Deucher
2021-02-04 18:44         ` Alex Deucher
2021-02-04 19:00         ` John Hubbard
2021-02-04 19:00           ` John Hubbard
2021-02-05 15:39           ` Daniel Vetter
2021-02-05 15:39             ` Daniel Vetter
2021-02-05 15:43             ` Jason Gunthorpe
2021-02-05 15:43               ` Jason Gunthorpe
2021-02-05 15:53               ` Daniel Vetter
2021-02-05 15:53                 ` Daniel Vetter
2021-02-05 16:00                 ` Jason Gunthorpe [this message]
2021-02-05 16:00                   ` Jason Gunthorpe
2021-02-05 16:06                   ` Daniel Vetter
2021-02-05 16:06                     ` Daniel Vetter
2021-02-05 20:24                 ` John Hubbard
2021-02-05 20:24                   ` John Hubbard

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210205160003.GU4247@nvidia.com \
    --to=jgg@nvidia.com \
    --cc=alexdeucher@gmail.com \
    --cc=christian.koenig@amd.com \
    --cc=daniel.vetter@intel.com \
    --cc=daniel@ffwll.ch \
    --cc=dledford@redhat.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=jhubbard@nvidia.com \
    --cc=jianxin.xiong@intel.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.