From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f198.google.com (mail-qk0-f198.google.com [209.85.220.198]) by kanga.kvack.org (Postfix) with ESMTP id CC85B6B0038 for ; Fri, 22 Sep 2017 16:32:09 -0400 (EDT) Received: by mail-qk0-f198.google.com with SMTP id i14so2858898qke.6 for ; Fri, 22 Sep 2017 13:32:09 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id d55si586826qtd.113.2017.09.22.13.32.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Sep 2017 13:32:08 -0700 (PDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v8MKVqxx081770 for ; Fri, 22 Sep 2017 16:32:07 -0400 Received: from e19.ny.us.ibm.com (e19.ny.us.ibm.com [129.33.205.209]) by mx0a-001b2d01.pphosted.com with ESMTP id 2d5614hjtq-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 22 Sep 2017 16:32:07 -0400 Received: from localhost by e19.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 22 Sep 2017 16:32:06 -0400 Date: Fri, 22 Sep 2017 15:31:57 -0500 From: Reza Arbab Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> Message-Id: <20170922203157.4txavkwmvyh2ufmd@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Michal Hocko , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Fri, Sep 22, 2017 at 08:13:56PM +0000, Reza Arbab wrote: >The move_pages() syscall can be used to find the numa node where a page >currently resides. This is not working for device public memory pages, >which erroneously report -EFAULT (unmapped or zero page). Argh. Please disregard this patch. My test setup has a chunk of system memory carved out as pretend device public memory, to experiment with. Of course the real thing has no numa node! Apologies all, it's been a long day. -- Reza Arbab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f200.google.com (mail-qt0-f200.google.com [209.85.216.200]) by kanga.kvack.org (Postfix) with ESMTP id 8D79D6B0038 for ; Fri, 22 Sep 2017 17:01:25 -0400 (EDT) Received: by mail-qt0-f200.google.com with SMTP id o3so2391235qte.7 for ; Fri, 22 Sep 2017 14:01:25 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id r34si603405qtd.515.2017.09.22.14.01.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Sep 2017 14:01:24 -0700 (PDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v8ML1C8g040926 for ; Fri, 22 Sep 2017 17:01:23 -0400 Received: from e12.ny.us.ibm.com (e12.ny.us.ibm.com [129.33.205.202]) by mx0a-001b2d01.pphosted.com with ESMTP id 2d542yj3vb-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 22 Sep 2017 17:01:23 -0400 Received: from localhost by e12.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 22 Sep 2017 17:01:22 -0400 Date: Fri, 22 Sep 2017 16:01:13 -0500 From: Reza Arbab Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> <20170922203157.4txavkwmvyh2ufmd@arbab-laptop.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20170922203157.4txavkwmvyh2ufmd@arbab-laptop.localdomain> Message-Id: <20170922210113.c2dn5mjis6zyted7@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Michal Hocko , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Fri, Sep 22, 2017 at 08:31:57PM +0000, Reza Arbab wrote: >On Fri, Sep 22, 2017 at 08:13:56PM +0000, Reza Arbab wrote: >>The move_pages() syscall can be used to find the numa node where a page >>currently resides. This is not working for device public memory pages, >>which erroneously report -EFAULT (unmapped or zero page). > >Argh. Please disregard this patch. > >My test setup has a chunk of system memory carved out as pretend >device public memory, to experiment with. Of course the real thing has >no numa node! On third thought, yes it does! static int hmm_devmem_pages_create(struct hmm_devmem *devmem) { : nid = dev_to_node(device); if (nid < 0) nid = numa_mem_id(); : if (devmem->pagemap.type == MEMORY_DEVICE_PUBLIC) ret = arch_add_memory(nid, align_start, align_size, false); : } So now I think the patch may be right after all. Please un-disregard it. Regard it? Whatever. -- Reza Arbab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f197.google.com (mail-wr0-f197.google.com [209.85.128.197]) by kanga.kvack.org (Postfix) with ESMTP id 8A6856B025F for ; Tue, 26 Sep 2017 09:37:10 -0400 (EDT) Received: by mail-wr0-f197.google.com with SMTP id h16so12663756wrf.0 for ; Tue, 26 Sep 2017 06:37:10 -0700 (PDT) Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id y9si1585163wmd.187.2017.09.26.06.37.09 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 26 Sep 2017 06:37:09 -0700 (PDT) Date: Tue, 26 Sep 2017 15:37:07 +0200 From: Michal Hocko Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory Message-ID: <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> Sender: owner-linux-mm@kvack.org List-ID: To: Reza Arbab Cc: Andrew Morton , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Fri 22-09-17 15:13:56, Reza Arbab wrote: > The move_pages() syscall can be used to find the numa node where a page > currently resides. This is not working for device public memory pages, > which erroneously report -EFAULT (unmapped or zero page). > > Enable by adding a FOLL_DEVICE flag for follow_page(), which > move_pages() will use. This could be done unconditionally, but adding a > flag seems like a safer change. I do not understand purpose of this patch. What is the numa node of a device memory? > Cc: Jerome Glisse > Signed-off-by: Reza Arbab > --- > include/linux/mm.h | 1 + > mm/gup.c | 2 +- > mm/migrate.c | 2 +- > 3 files changed, 3 insertions(+), 2 deletions(-) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index f8c10d3..783cb57 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -2368,6 +2368,7 @@ static inline struct page *follow_page(struct vm_area_struct *vma, > #define FOLL_MLOCK 0x1000 /* lock present pages */ > #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ > #define FOLL_COW 0x4000 /* internal GUP flag */ > +#define FOLL_DEVICE 0x8000 /* return device pages */ > > static inline int vm_fault_to_errno(int vm_fault, int foll_flags) > { > diff --git a/mm/gup.c b/mm/gup.c > index b2b4d42..6fbad70 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -110,7 +110,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, > return NULL; > } > > - page = vm_normal_page(vma, address, pte); > + page = _vm_normal_page(vma, address, pte, flags & FOLL_DEVICE); > if (!page && pte_devmap(pte) && (flags & FOLL_GET)) { > /* > * Only return device mapping pages in the FOLL_GET case since > diff --git a/mm/migrate.c b/mm/migrate.c > index 6954c14..dea0ceb 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1690,7 +1690,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages, > goto set_status; > > /* FOLL_DUMP to ignore special (like zero) pages */ > - page = follow_page(vma, addr, FOLL_DUMP); > + page = follow_page(vma, addr, FOLL_DUMP | FOLL_DEVICE); > > err = PTR_ERR(page); > if (IS_ERR(page)) > -- > 1.8.3.1 > -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f70.google.com (mail-wm0-f70.google.com [74.125.82.70]) by kanga.kvack.org (Postfix) with ESMTP id 5C7476B0069 for ; Tue, 26 Sep 2017 10:47:22 -0400 (EDT) Received: by mail-wm0-f70.google.com with SMTP id u138so12035578wmu.2 for ; Tue, 26 Sep 2017 07:47:22 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id y11si2834489edh.411.2017.09.26.07.47.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Sep 2017 07:47:21 -0700 (PDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v8QEi8w5042077 for ; Tue, 26 Sep 2017 10:47:19 -0400 Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.152]) by mx0b-001b2d01.pphosted.com with ESMTP id 2d7q60gxca-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 26 Sep 2017 10:47:19 -0400 Received: from localhost by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 26 Sep 2017 08:47:18 -0600 Date: Tue, 26 Sep 2017 09:47:10 -0500 From: Reza Arbab Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> Message-Id: <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Andrew Morton , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Sep 26, 2017 at 01:37:07PM +0000, Michal Hocko wrote: >On Fri 22-09-17 15:13:56, Reza Arbab wrote: >> The move_pages() syscall can be used to find the numa node where a page >> currently resides. This is not working for device public memory pages, >> which erroneously report -EFAULT (unmapped or zero page). >> >> Enable by adding a FOLL_DEVICE flag for follow_page(), which >> move_pages() will use. This could be done unconditionally, but adding a >> flag seems like a safer change. > >I do not understand purpose of this patch. What is the numa node of a >device memory? Well, using hmm_devmem_pages_create() it is added to this node: nid = dev_to_node(device); if (nid < 0) nid = numa_mem_id(); I understand it's minimally useful information to userspace, but the memory does have a nid and move_pages() is supposed to be able to return what that is. I ran into this using a testcase which tries to verify that user addresses were correctly migrated to coherent device memory. That said, I'm okay with dropping this if you don't think it's worthwhile. -- Reza Arbab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f197.google.com (mail-qt0-f197.google.com [209.85.216.197]) by kanga.kvack.org (Postfix) with ESMTP id 7F9926B0038 for ; Tue, 26 Sep 2017 12:20:00 -0400 (EDT) Received: by mail-qt0-f197.google.com with SMTP id b1so11675558qtc.4 for ; Tue, 26 Sep 2017 09:20:00 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id a9si768284qtg.553.2017.09.26.09.19.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Sep 2017 09:19:59 -0700 (PDT) Date: Tue, 26 Sep 2017 09:19:42 -0700 From: Jerome Glisse Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory Message-ID: <20170926161941.GB3216@redhat.com> References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Reza Arbab Cc: Michal Hocko , Andrew Morton , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Sep 26, 2017 at 09:47:10AM -0500, Reza Arbab wrote: > On Tue, Sep 26, 2017 at 01:37:07PM +0000, Michal Hocko wrote: > > On Fri 22-09-17 15:13:56, Reza Arbab wrote: > > > The move_pages() syscall can be used to find the numa node where a page > > > currently resides. This is not working for device public memory pages, > > > which erroneously report -EFAULT (unmapped or zero page). > > > > > > Enable by adding a FOLL_DEVICE flag for follow_page(), which > > > move_pages() will use. This could be done unconditionally, but adding a > > > flag seems like a safer change. > > > > I do not understand purpose of this patch. What is the numa node of a > > device memory? > > Well, using hmm_devmem_pages_create() it is added to this node: > > nid = dev_to_node(device); > if (nid < 0) > nid = numa_mem_id(); > > I understand it's minimally useful information to userspace, but the memory > does have a nid and move_pages() is supposed to be able to return what that > is. I ran into this using a testcase which tries to verify that user > addresses were correctly migrated to coherent device memory. > > That said, I'm okay with dropping this if you don't think it's worthwhile. Just to add a data point, PCIE devices are tie to one CPU (architecturaly PCIE lane are connected to CPU at least on x86/ppc AFAIK) and thus to one numa node. Right now i am traveling but i want to check that this patch does not allow user to inadvertaly pin device memory page. I will look into it once i am back. Cheers, Jerome -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f199.google.com (mail-wr0-f199.google.com [209.85.128.199]) by kanga.kvack.org (Postfix) with ESMTP id 2B80A6B0038 for ; Tue, 26 Sep 2017 12:32:45 -0400 (EDT) Received: by mail-wr0-f199.google.com with SMTP id p37so1989171wrb.1 for ; Tue, 26 Sep 2017 09:32:45 -0700 (PDT) Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id s4si7242324wrg.494.2017.09.26.09.32.43 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 26 Sep 2017 09:32:43 -0700 (PDT) Date: Tue, 26 Sep 2017 18:32:41 +0200 From: Michal Hocko Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory Message-ID: <20170926163241.5rd4wyzrzoso4uto@dhcp22.suse.cz> References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Reza Arbab Cc: Andrew Morton , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue 26-09-17 09:47:10, Reza Arbab wrote: > On Tue, Sep 26, 2017 at 01:37:07PM +0000, Michal Hocko wrote: > > On Fri 22-09-17 15:13:56, Reza Arbab wrote: > > > The move_pages() syscall can be used to find the numa node where a page > > > currently resides. This is not working for device public memory pages, > > > which erroneously report -EFAULT (unmapped or zero page). > > > > > > Enable by adding a FOLL_DEVICE flag for follow_page(), which > > > move_pages() will use. This could be done unconditionally, but adding a > > > flag seems like a safer change. > > > > I do not understand purpose of this patch. What is the numa node of a > > device memory? > > Well, using hmm_devmem_pages_create() it is added to this node: > > nid = dev_to_node(device); > if (nid < 0) > nid = numa_mem_id(); OK, but do all the HMM devices have concept of NUMA affinity? From the code you are pasting they do not have to... > I understand it's minimally useful information to userspace, but the memory > does have a nid and move_pages() is supposed to be able to return what that > is. I ran into this using a testcase which tries to verify that user > addresses were correctly migrated to coherent device memory. > > That said, I'm okay with dropping this if you don't think it's worthwhile. I am just worried that we allow information which is not generally sensible and I am also not sure what the userspace can actually do with that information. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f200.google.com (mail-qt0-f200.google.com [209.85.216.200]) by kanga.kvack.org (Postfix) with ESMTP id EAE216B0069 for ; Tue, 26 Sep 2017 14:35:33 -0400 (EDT) Received: by mail-qt0-f200.google.com with SMTP id u48so12109988qtc.3 for ; Tue, 26 Sep 2017 11:35:33 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id i51si2328290qte.371.2017.09.26.11.35.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Sep 2017 11:35:32 -0700 (PDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v8QIYls0043313 for ; Tue, 26 Sep 2017 14:35:31 -0400 Received: from e16.ny.us.ibm.com (e16.ny.us.ibm.com [129.33.205.206]) by mx0a-001b2d01.pphosted.com with ESMTP id 2d7u30va9u-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 26 Sep 2017 14:35:31 -0400 Received: from localhost by e16.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 26 Sep 2017 14:35:30 -0400 Date: Tue, 26 Sep 2017 13:35:22 -0500 From: Reza Arbab Subject: Re: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory References: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> <20170926133707.wquyw3ic5nbmfjuo@dhcp22.suse.cz> <20170926144710.zepvnyktqjomnx2n@arbab-laptop.localdomain> <20170926163241.5rd4wyzrzoso4uto@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20170926163241.5rd4wyzrzoso4uto@dhcp22.suse.cz> Message-Id: <20170926183522.zeky6yfjdbuistso@arbab-laptop.localdomain> Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Andrew Morton , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?iso-8859-1?B?Suly9G1l?= Glisse , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Sep 26, 2017 at 04:32:41PM +0000, Michal Hocko wrote: >On Tue 26-09-17 09:47:10, Reza Arbab wrote: >> On Tue, Sep 26, 2017 at 01:37:07PM +0000, Michal Hocko wrote: >> > On Fri 22-09-17 15:13:56, Reza Arbab wrote: >> > > The move_pages() syscall can be used to find the numa node where a page >> > > currently resides. This is not working for device public memory pages, >> > > which erroneously report -EFAULT (unmapped or zero page). >> > > >> > > Enable by adding a FOLL_DEVICE flag for follow_page(), which >> > > move_pages() will use. This could be done unconditionally, but adding a >> > > flag seems like a safer change. >> > >> > I do not understand purpose of this patch. What is the numa node of a >> > device memory? >> >> Well, using hmm_devmem_pages_create() it is added to this node: >> >> nid = dev_to_node(device); >> if (nid < 0) >> nid = numa_mem_id(); > >OK, but do all the HMM devices have concept of NUMA affinity? From the >code you are pasting they do not have to... I don't know the definitive answer here, but as Jerome said PCIE devices should, and we are heading that way with NVLink/CAPI as well. It seems the default is just the nearest node. >> I understand it's minimally useful information to userspace, but the memory >> does have a nid and move_pages() is supposed to be able to return what that >> is. I ran into this using a testcase which tries to verify that user >> addresses were correctly migrated to coherent device memory. >> >> That said, I'm okay with dropping this if you don't think it's worthwhile. > >I am just worried that we allow information which is not generally >sensible and I am also not sure what the userspace can actually do with >that information. As mentioned, it is minimally useful, e.g. for verifying migration, so returning the nid seems sensible to me. Alternatively, we might at least change the documentation to say -EFAULT This is a zero page, a device page, or the memory area is not mapped by the process. ^^^^^^^^^^^^^ -- Reza Arbab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Reza Arbab Subject: [PATCH] mm/device-public-memory: Enable move_pages() to stat device memory Date: Fri, 22 Sep 2017 15:13:56 -0500 Message-ID: <1506111236-28975-1-git-send-email-arbab@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Return-path: Sender: linux-kernel-owner@vger.kernel.org To: Andrew Morton Cc: Michal Hocko , Jan Kara , Ross Zwisler , "Kirill A. Shutemov" , Lorenzo Stoakes , Dave Jiang , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Matthew Wilcox , Hugh Dickins , Huang Ying , Ingo Molnar , "Aneesh Kumar K.V" , James Morse , Naoya Horiguchi , Minchan Kim , Johannes Weiner , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org List-Id: linux-mm.kvack.org The move_pages() syscall can be used to find the numa node where a page currently resides. This is not working for device public memory pages, which erroneously report -EFAULT (unmapped or zero page). Enable by adding a FOLL_DEVICE flag for follow_page(), which move_pages() will use. This could be done unconditionally, but adding a flag seems like a safer change. Cc: Jérôme Glisse Signed-off-by: Reza Arbab --- include/linux/mm.h | 1 + mm/gup.c | 2 +- mm/migrate.c | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f8c10d3..783cb57 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2368,6 +2368,7 @@ static inline struct page *follow_page(struct vm_area_struct *vma, #define FOLL_MLOCK 0x1000 /* lock present pages */ #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000 /* internal GUP flag */ +#define FOLL_DEVICE 0x8000 /* return device pages */ static inline int vm_fault_to_errno(int vm_fault, int foll_flags) { diff --git a/mm/gup.c b/mm/gup.c index b2b4d42..6fbad70 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -110,7 +110,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, return NULL; } - page = vm_normal_page(vma, address, pte); + page = _vm_normal_page(vma, address, pte, flags & FOLL_DEVICE); if (!page && pte_devmap(pte) && (flags & FOLL_GET)) { /* * Only return device mapping pages in the FOLL_GET case since diff --git a/mm/migrate.c b/mm/migrate.c index 6954c14..dea0ceb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1690,7 +1690,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages, goto set_status; /* FOLL_DUMP to ignore special (like zero) pages */ - page = follow_page(vma, addr, FOLL_DUMP); + page = follow_page(vma, addr, FOLL_DUMP | FOLL_DEVICE); err = PTR_ERR(page); if (IS_ERR(page)) -- 1.8.3.1