From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E5EEC4332B for ; Fri, 5 Feb 2021 22:23:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0238564F9F for ; Fri, 5 Feb 2021 22:23:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229763AbhBEWWz (ORCPT ); Fri, 5 Feb 2021 17:22:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232631AbhBEO26 (ORCPT ); Fri, 5 Feb 2021 09:28:58 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59035C061797 for ; Fri, 5 Feb 2021 08:06:47 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id v15so8275545wrx.4 for ; Fri, 05 Feb 2021 08:06:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=3OSZRC12f3q/J6xvpmjxkYtxd8Kjl+PfZ12MCmFyNTs=; b=bYXMPhnRMDnWNLg3owU/iy8WI3TbNcpN0GVSP5459+nS10U0XMpOTznWSNv/myvTnO KfxK93fxhG2u7Ura35Me9qKW50VlJ2+swQ01+GIT9Oe8cw0kdtkNOd/LbHWPPDDOtZXf ZKZnHo5aFIIiuH5xeRO5eg9jfpcDgyJe94qos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=3OSZRC12f3q/J6xvpmjxkYtxd8Kjl+PfZ12MCmFyNTs=; b=kFEdmK+P2CDW0UZdhPFXV4IfMQOVbzpS0a9ztn+Zf259AkBK584GYPDmEcMkAXyVzs jC6iqOZLkEr6rgv5QdrcShTGvNfAjCm2w36mKmxfWe240MxIEbhBq+qnGWdg6R6H8k0C ITg9zJKe67aqwXoA9W5t/uLO4WZ+mmvd7JCce5CY1rnWJG9libx+ErwLZLKy8bPpavLM 4G0+CWHZd2efPBV5GsroDfD+0cEZ1Uy+1ez0AopZUR3iX6XBE4pg/gn85wDqye9UFYz1 9+/yzM7AJFQitM7AFoeBK4gw/NV+kPPkVvPrQS7GsPpV+IuL4LdBnVvsT1SBQzkojPi4 xbSA== X-Gm-Message-State: AOAM5306OuJ1CPFzKkw8WgQ0mim5wbRf8gqaQjzhf/OgNlM7L8KjGARL BRZd8uh4udv6UoJsZNz3KAisZw== X-Google-Smtp-Source: ABdhPJxZ2wdJ6quTErZl6d8W+xMqeQz+RNxGJkRMmNGfC01VLq/Z0PeyUo/lOZrBhMLQvh9lWULc3Q== X-Received: by 2002:adf:e7c1:: with SMTP id e1mr5816296wrn.23.1612541206152; Fri, 05 Feb 2021 08:06:46 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id e4sm12452422wrw.96.2021.02.05.08.06.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 08:06:45 -0800 (PST) Date: Fri, 5 Feb 2021 17:06:43 +0100 From: Daniel Vetter To: Jason Gunthorpe Cc: Daniel Vetter , John Hubbard , Alex Deucher , Leon Romanovsky , linux-rdma , Maling list - DRI developers , Doug Ledford , Daniel Vetter , Christian Koenig , Jianxin Xiong Subject: Re: [PATCH v16 0/4] RDMA: Add dma-buf support Message-ID: References: <1608067636-98073-1-git-send-email-jianxin.xiong@intel.com> <5e4ac17d-1654-9abc-9a14-bda223d62866@nvidia.com> <20210204182923.GL4247@nvidia.com> <8e731fce-95c1-4ace-d8bc-dc0df7432d22@nvidia.com> <20210205154319.GT4247@nvidia.com> <20210205160003.GU4247@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210205160003.GU4247@nvidia.com> X-Operating-System: Linux phenom 5.7.0-1-amd64 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Fri, Feb 05, 2021 at 12:00:03PM -0400, Jason Gunthorpe wrote: > On Fri, Feb 05, 2021 at 04:53:04PM +0100, Daniel Vetter wrote: > > On Fri, Feb 05, 2021 at 11:43:19AM -0400, Jason Gunthorpe wrote: > > > On Fri, Feb 05, 2021 at 04:39:47PM +0100, Daniel Vetter wrote: > > > > > > > > And again, for slightly older hardware, without pinning to VRAM there is > > > > > no way to use this solution here for peer-to-peer. So I'm glad to see that > > > > > so far you're not ruling out the pinning option. > > > > > > > > Since HMM and ZONE_DEVICE came up, I'm kinda tempted to make ZONE_DEVICE > > > > ZONE_MOVEABLE (at least if you don't have a pinned vram contigent in your > > > > cgroups) or something like that, so we could benefit from the work to make > > > > sure pin_user_pages and all these never end up in there? > > > > > > ZONE_DEVICE should already not be returned from GUP. > > > > > > I've understood in the hmm casse the idea was a CPU touch of some > > > ZONE_DEVICE pages would trigger a migration to CPU memory, GUP would > > > want to follow the same logic, presumably it comes for free with the > > > fault handler somehow > > > > Oh I didn't know this, I thought the proposed p2p direct i/o patches would > > just use the fact that underneath ZONE_DEVICE there's "normal" struct > > pages. > > So, if that every happens, it would be some special FOLL_ALLOW_P2P > flag to get the behavior. > > > And so I got worried that maybe also pin_user_pages can creep in. > > But I didn't read the patches in full detail: > > And yes, you might want to say that you can't longterm pin certain > kinds of zone_device pages, but if that is the common operating mode > then we'd probably never create a FOLL_ALLOW_P2P > > > But if you're saying that this all needs specific code and all the gup/pup > > code we have is excluded, I think we can make sure that we're not ever > > building features that requiring time-unlimited pinning of > > ZONE_DEVICE. > > Well, it is certainly a useful idea of some uses of ZONE_DEVICE, GPU > vram is not the whole world. Yeah non-volatile RAM can probably pin whatever it wants :-) >From the other thread, I think if we can get some cgroups going for accounting pinned memory, then pinning gpu memory should also not be any real issue. Might be somewhat tricky to glue that into a FOLL_ALLOW_P2P flag, maybe through zone-awareness or something like that. With the right accounting in place I'm happy to let userspace pin whatever they want really. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch