Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[Michal Hocko]

On Tue 15-10-19 14:51:41, Anshuman Khandual wrote:
[...]
> +/**
> + * alloc_gigantic_page_order() -- tries to allocate given order of pages
> + * @order:	allocation order (greater than MAX_ORDER)
> + * @gfp_mask:	GFP mask to use during compaction
> + * @nid:	allocation node
> + * @nodemask:	allocation nodemask
> + *
> + * This routine is an wrapper around alloc_contig_range() which scans over
> + * all zones on an applicable zonelist to find a contiguous pfn range which
> + * can the be allocated with alloc_contig_range(). This routine is intended
> + * to be used for allocations greater than MAX_ORDER.
> + *
> + * Return: page on success or NULL on failure. On success a memory block
> + * of 'order' starting with 'page' has been allocated successfully. Memory
> + * allocated here needs to be freed with free_contig_range().
> + */
> +struct page *alloc_gigantic_page_order(unsigned int order, gfp_t gfp_mask,
> +				       int nid, nodemask_t *nodemask)

One of the objections when Mike has proposed a similar thing last year
was that the interface shouldn't be order bases
http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz

Order based API makes sense for the buddy allocator but why should we
restrict sizes like that for an allocator that is capable to allocate
arbitrary page sized requests?
-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[Anshuman Khandual]

On 10/15/2019 04:15 PM, Michal Hocko wrote:
> On Tue 15-10-19 14:51:41, Anshuman Khandual wrote:
> [...]
>> +/**
>> + * alloc_gigantic_page_order() -- tries to allocate given order of pages
>> + * @order:	allocation order (greater than MAX_ORDER)
>> + * @gfp_mask:	GFP mask to use during compaction
>> + * @nid:	allocation node
>> + * @nodemask:	allocation nodemask
>> + *
>> + * This routine is an wrapper around alloc_contig_range() which scans over
>> + * all zones on an applicable zonelist to find a contiguous pfn range which
>> + * can the be allocated with alloc_contig_range(). This routine is intended
>> + * to be used for allocations greater than MAX_ORDER.
>> + *
>> + * Return: page on success or NULL on failure. On success a memory block
>> + * of 'order' starting with 'page' has been allocated successfully. Memory
>> + * allocated here needs to be freed with free_contig_range().
>> + */
>> +struct page *alloc_gigantic_page_order(unsigned int order, gfp_t gfp_mask,
>> +				       int nid, nodemask_t *nodemask)
> 
> One of the objections when Mike has proposed a similar thing last year
> was that the interface shouldn't be order bases
> http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz
> 
> Order based API makes sense for the buddy allocator but why should we
> restrict sizes like that for an allocator that is capable to allocate
> arbitrary page sized requests?

Fair enough, will change it. Anyways we calculate nr_pages from the order 
argument at the very beginning.

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[David Hildenbrand]

On 15.10.19 11:21, Anshuman Khandual wrote:
> alloc_gigantic_page() implements an allocation method where it scans over
> various zones looking for a large contiguous memory block which could not
> have been allocated through the buddy allocator. A subsequent patch which
> tests arch page table helpers needs such a method to allocate PUD_SIZE
> sized memory block. In the future such methods might have other use cases
> as well. So alloc_gigantic_page() has been split carving out actual memory
> allocation method and made available via new alloc_gigantic_page_order()
> which is wrapped under CONFIG_CONTIG_ALLOC.
> 
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
> Cc: Mike Kravetz <mike.kravetz@oracle.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Mark Brown <broonie@kernel.org>
> Cc: Steven Price <Steven.Price@arm.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> Cc: Kees Cook <keescook@chromium.org>
> Cc: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Sri Krishna chowdary <schowdary@nvidia.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Cc: Russell King - ARM Linux <linux@armlinux.org.uk>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Vineet Gupta <vgupta@synopsys.com>
> Cc: James Hogan <jhogan@kernel.org>
> Cc: Paul Burton <paul.burton@mips.com>
> Cc: Ralf Baechle <ralf@linux-mips.org>
> Cc: Kirill A. Shutemov <kirill@shutemov.name>
> Cc: Gerald Schaefer <gerald.schaefer@de.ibm.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: David Rientjes <rientjes@google.com>
> Cc: Andrea Arcangeli <aarcange@redhat.com>
> Cc: Oscar Salvador <osalvador@suse.de>
> Cc: Mel Gorman <mgorman@techsingularity.net>
> Cc: linux-snps-arc@lists.infradead.org
> Cc: linux-mips@vger.kernel.org
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-ia64@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-s390@vger.kernel.org
> Cc: linux-sh@vger.kernel.org
> Cc: sparclinux@vger.kernel.org
> Cc: x86@kernel.org
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
>   include/linux/gfp.h |  3 ++
>   mm/hugetlb.c        | 76 +----------------------------------
>   mm/page_alloc.c     | 98 +++++++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 102 insertions(+), 75 deletions(-)
> 
> diff --git a/include/linux/gfp.h b/include/linux/gfp.h
> index fb07b503dc45..379ad23437d1 100644
> --- a/include/linux/gfp.h
> +++ b/include/linux/gfp.h
> @@ -589,6 +589,9 @@ static inline bool pm_suspended_storage(void)
>   /* The below functions must be run on a range from a single zone. */
>   extern int alloc_contig_range(unsigned long start, unsigned long end,
>   			      unsigned migratetype, gfp_t gfp_mask);
> +extern struct page *alloc_gigantic_page_order(unsigned int order,
> +					      gfp_t gfp_mask, int nid,
> +					      nodemask_t *nodemask);
>   #endif
>   void free_contig_range(unsigned long pfn, unsigned int nr_pages);
>   
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 977f9a323a7a..d199556a4a2c 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1023,86 +1023,12 @@ static void free_gigantic_page(struct page *page, unsigned int order)
>   }
>   
>   #ifdef CONFIG_CONTIG_ALLOC
> -static int __alloc_gigantic_page(unsigned long start_pfn,
> -				unsigned long nr_pages, gfp_t gfp_mask)
> -{
> -	unsigned long end_pfn = start_pfn + nr_pages;
> -	return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE,
> -				  gfp_mask);
> -}
> -
> -static bool pfn_range_valid_gigantic(struct zone *z,
> -			unsigned long start_pfn, unsigned long nr_pages)
> -{
> -	unsigned long i, end_pfn = start_pfn + nr_pages;
> -	struct page *page;
> -
> -	for (i = start_pfn; i < end_pfn; i++) {
> -		if (!pfn_valid(i))
> -			return false;
> -
> -		page = pfn_to_page(i);

Am I missing something or should here really be a pfn_to_online_page() 
here instead of a pfn_valid() ?


-- 

Thanks,

David / dhildenb

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 2/2] mm/debug: Add tests validating architecture page table helpers

[Michal Hocko]

On Tue 15-10-19 14:51:42, Anshuman Khandual wrote:
> This adds tests which will validate architecture page table helpers and
> other accessors in their compliance with expected generic MM semantics.
> This will help various architectures in validating changes to existing
> page table helpers or addition of new ones.
> 
> Test page table and memory pages creating it's entries at various level are
> all allocated from system memory with required size and alignments. But if
> memory pages with required size and alignment could not be allocated, then
> all depending individual tests are just skipped afterwards. This test gets
> called right after init_mm_internals() required for alloc_contig_range() to
> work correctly.
> 
> This gets build and run when CONFIG_DEBUG_VM_PGTABLE is selected along with
> CONFIG_VM_DEBUG. Architectures willing to subscribe this test also need to
> select CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE which for now is limited to x86 and
> arm64. Going forward, other architectures too can enable this after fixing
> build or runtime problems (if any) with their page table helpers.

A highlevel description of tests and what they are testing for would be
really appreciated. Who wants to run these tests and why/when? What kind
of bugs would get detected? In short why do we really need/want this
code in the tree?

-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[Michal Hocko]

On Tue 15-10-19 13:42:03, David Hildenbrand wrote:
[...]
> > -static bool pfn_range_valid_gigantic(struct zone *z,
> > -			unsigned long start_pfn, unsigned long nr_pages)
> > -{
> > -	unsigned long i, end_pfn = start_pfn + nr_pages;
> > -	struct page *page;
> > -
> > -	for (i = start_pfn; i < end_pfn; i++) {
> > -		if (!pfn_valid(i))
> > -			return false;
> > -
> > -		page = pfn_to_page(i);
> 
> Am I missing something or should here really be a pfn_to_online_page() here
> instead of a pfn_valid() ?

http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz

-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[David Hildenbrand]

On 15.10.19 13:47, Michal Hocko wrote:
> On Tue 15-10-19 13:42:03, David Hildenbrand wrote:
> [...]
>>> -static bool pfn_range_valid_gigantic(struct zone *z,
>>> -			unsigned long start_pfn, unsigned long nr_pages)
>>> -{
>>> -	unsigned long i, end_pfn = start_pfn + nr_pages;
>>> -	struct page *page;
>>> -
>>> -	for (i = start_pfn; i < end_pfn; i++) {
>>> -		if (!pfn_valid(i))
>>> -			return false;
>>> -
>>> -		page = pfn_to_page(i);
>>
>> Am I missing something or should here really be a pfn_to_online_page() here
>> instead of a pfn_valid() ?
> 
> http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz
> 

So we managed to add PageReserved(page) but not pfn_to_online_page(). 
But it is the right thing to do? (or am I missing something?)

-- 

Thanks,

David / dhildenb

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[David Hildenbrand]

On 15.10.19 13:50, David Hildenbrand wrote:
> On 15.10.19 13:47, Michal Hocko wrote:
>> On Tue 15-10-19 13:42:03, David Hildenbrand wrote:
>> [...]
>>>> -static bool pfn_range_valid_gigantic(struct zone *z,
>>>> -			unsigned long start_pfn, unsigned long nr_pages)
>>>> -{
>>>> -	unsigned long i, end_pfn = start_pfn + nr_pages;
>>>> -	struct page *page;
>>>> -
>>>> -	for (i = start_pfn; i < end_pfn; i++) {
>>>> -		if (!pfn_valid(i))
>>>> -			return false;
>>>> -
>>>> -		page = pfn_to_page(i);
>>>
>>> Am I missing something or should here really be a pfn_to_online_page() here
>>> instead of a pfn_valid() ?
>>
>> http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz
>>
> 
> So we managed to add PageReserved(page) but not pfn_to_online_page().
> But it is the right thing to do? (or am I missing something?)
> 

Will send a patch.

-- 

Thanks,

David / dhildenb

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[Michal Hocko]

On Tue 15-10-19 13:50:02, David Hildenbrand wrote:
> On 15.10.19 13:47, Michal Hocko wrote:
> > On Tue 15-10-19 13:42:03, David Hildenbrand wrote:
> > [...]
> > > > -static bool pfn_range_valid_gigantic(struct zone *z,
> > > > -			unsigned long start_pfn, unsigned long nr_pages)
> > > > -{
> > > > -	unsigned long i, end_pfn = start_pfn + nr_pages;
> > > > -	struct page *page;
> > > > -
> > > > -	for (i = start_pfn; i < end_pfn; i++) {
> > > > -		if (!pfn_valid(i))
> > > > -			return false;
> > > > -
> > > > -		page = pfn_to_page(i);
> > > 
> > > Am I missing something or should here really be a pfn_to_online_page() here
> > > instead of a pfn_valid() ?
> > 
> > http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz
> > 
> 
> So we managed to add PageReserved(page) but not pfn_to_online_page(). But it
> is the right thing to do? (or am I missing something?)

Yeah, pfn_to_online_page is better. But please note that this is an
optimistic check. The real check has to be done when isolating the
pageblock because things might change in the meantime.
-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 1/2] mm/page_alloc: Make alloc_gigantic_page() available for general use

[Michal Hocko]

On Tue 15-10-19 14:09:56, Michal Hocko wrote:
> On Tue 15-10-19 13:50:02, David Hildenbrand wrote:
> > On 15.10.19 13:47, Michal Hocko wrote:
> > > On Tue 15-10-19 13:42:03, David Hildenbrand wrote:
> > > [...]
> > > > > -static bool pfn_range_valid_gigantic(struct zone *z,
> > > > > -			unsigned long start_pfn, unsigned long nr_pages)
> > > > > -{
> > > > > -	unsigned long i, end_pfn = start_pfn + nr_pages;
> > > > > -	struct page *page;
> > > > > -
> > > > > -	for (i = start_pfn; i < end_pfn; i++) {
> > > > > -		if (!pfn_valid(i))
> > > > > -			return false;
> > > > > -
> > > > > -		page = pfn_to_page(i);
> > > > 
> > > > Am I missing something or should here really be a pfn_to_online_page() here
> > > > instead of a pfn_valid() ?
> > > 
> > > http://lkml.kernel.org/r/20180423000943.GO17484@dhcp22.suse.cz
> > > 
> > 
> > So we managed to add PageReserved(page) but not pfn_to_online_page(). But it
> > is the right thing to do? (or am I missing something?)
> 
> Yeah, pfn_to_online_page is better. But please note that this is an
> optimistic check. The real check has to be done when isolating the
> pageblock because things might change in the meantime.

Except I have missed that we do get zone from the page and other
undefined state. Scratch my above comment.
-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 2/2] mm/debug: Add tests validating architecture page table helpers

[Anshuman Khandual]

On 10/15/2019 05:16 PM, Michal Hocko wrote:
> On Tue 15-10-19 14:51:42, Anshuman Khandual wrote:
>> This adds tests which will validate architecture page table helpers and
>> other accessors in their compliance with expected generic MM semantics.
>> This will help various architectures in validating changes to existing
>> page table helpers or addition of new ones.
>>
>> Test page table and memory pages creating it's entries at various level are
>> all allocated from system memory with required size and alignments. But if
>> memory pages with required size and alignment could not be allocated, then
>> all depending individual tests are just skipped afterwards. This test gets
>> called right after init_mm_internals() required for alloc_contig_range() to
>> work correctly.
>>
>> This gets build and run when CONFIG_DEBUG_VM_PGTABLE is selected along with
>> CONFIG_VM_DEBUG. Architectures willing to subscribe this test also need to
>> select CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE which for now is limited to x86 and
>> arm64. Going forward, other architectures too can enable this after fixing
>> build or runtime problems (if any) with their page table helpers.
> 
> A highlevel description of tests and what they are testing for would be
> really appreciated. Who wants to run these tests and why/when? What kind
> of bugs would get detected? In short why do we really need/want this
> code in the tree?

Sure, will do.

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 0/2] mm/debug: Add tests validating architecture page table helpers

[Qian Cai]

The x86 will crash with linux-next during boot due to this series (v5) with the
config below plus CONFIG_DEBUG_VM_PGTABLE=y. I am not sure if v6 would address
it.

https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

[   33.862600][    T1] page:ffffea0009000000 is uninitialized and poisoned
[   33.862608][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff
ffffff871140][    T1]  ? _raw_spin_unlock_irq+0x27/0x40
[   33.871140][    T1]  ? rest_init+0x307/0x307
[   33.871140][    T1]  kernel_init+0x11/0x139
[   33.871140][    T1]  ? rest_init+0x307/0x307
[   33.871140][    T1]  ret_from_fork+0x27/0x50
[   33.871140][    T1] Modules linked in:
[   33.871140][    T1] ---[ end trace e99d392b0f7befbd ]---
[   33.871140][    T1] RIP: 0010:alloc_gigantic_page_order+0x3fe/0x490

On Tue, 2019-10-15 at 14:51 +0530, Anshuman Khandual wrote:
> This series adds a test validation for architecture exported page table
> helpers. Patch in the series adds basic transformation tests at various
> levels of the page table. Before that it exports gigantic page allocation
> function from HugeTLB.
> 
> This test was originally suggested by Catalin during arm64 THP migration
> RFC discussion earlier. Going forward it can include more specific tests
> with respect to various generic MM functions like THP, HugeTLB etc and
> platform specific tests.
> 
> https://lore.kernel.org/linux-mm/20190628102003.GA56463@arrakis.emea.arm.com/
> 
> Changes in V6:
> 
> - Moved alloc_gigantic_page_order() into mm/page_alloc.c per Michal
> - Moved alloc_gigantic_page_order() within CONFIG_CONTIG_ALLOC in the test
> - Folded Andrew's include/asm-generic/pgtable.h fix into the test patch 2/2
> 
> Changes in V5: (https://patchwork.kernel.org/project/linux-mm/list/?series=185991)
> 
> - Redefined and moved X86 mm_p4d_folded() into a different header per Kirill/Ingo
> - Updated the config option comment per Ingo and dropped 'kernel module' reference
> - Updated the commit message and dropped 'kernel module' reference
> - Changed DEBUG_ARCH_PGTABLE_TEST into DEBUG_VM_PGTABLE per Ingo
> - Moved config option from mm/Kconfig.debug into lib/Kconfig.debug
> - Renamed core test function arch_pgtable_tests() as debug_vm_pgtable()
> - Renamed mm/arch_pgtable_test.c as mm/debug_vm_pgtable.c
> - debug_vm_pgtable() gets called from kernel_init_freeable() after init_mm_internals()
> - Added an entry in Documentation/features/debug/ per Ingo
> - Enabled the test on arm64 and x86 platforms for now
> 
> Changes in V4: (https://patchwork.kernel.org/project/linux-mm/list/?series=183465)
> 
> - Disable DEBUG_ARCH_PGTABLE_TEST for ARM and IA64 platforms
> 
> Changes in V3: (https://lore.kernel.org/patchwork/project/lkml/list/?series=411216)
> 
> - Changed test trigger from module format into late_initcall()
> - Marked all functions with __init to be freed after completion
> - Changed all __PGTABLE_PXX_FOLDED checks as mm_pxx_folded()
> - Folded in PPC32 fixes from Christophe
> 
> Changes in V2:
> 
> https://lore.kernel.org/linux-mm/1568268173-31302-1-git-send-email-anshuman.khandual@arm.com/T/#t
> 
> - Fixed small typo error in MODULE_DESCRIPTION()
> - Fixed m64k build problems for lvalue concerns in pmd_xxx_tests()
> - Fixed dynamic page table level folding problems on x86 as per Kirril
> - Fixed second pointers during pxx_populate_tests() per Kirill and Gerald
> - Allocate and free pte table with pte_alloc_one/pte_free per Kirill
> - Modified pxx_clear_tests() to accommodate s390 lower 12 bits situation
> - Changed RANDOM_NZVALUE value from 0xbe to 0xff
> - Changed allocation, usage, free sequence for saved_ptep
> - Renamed VMA_FLAGS as VMFLAGS
> - Implemented a new method for random vaddr generation
> - Implemented some other cleanups
> - Dropped extern reference to mm_alloc()
> - Created and exported new alloc_gigantic_page_order()
> - Dropped the custom allocator and used new alloc_gigantic_page_order()
> 
> Changes in V1:
> 
> https://lore.kernel.org/linux-mm/1567497706-8649-1-git-send-email-anshuman.khandual@arm.com/
> 
> - Added fallback mechanism for PMD aligned memory allocation failure
> 
> Changes in RFC V2:
> 
> https://lore.kernel.org/linux-mm/1565335998-22553-1-git-send-email-anshuman.khandual@arm.com/T/#u
> 
> - Moved test module and it's config from lib/ to mm/
> - Renamed config TEST_ARCH_PGTABLE as DEBUG_ARCH_PGTABLE_TEST
> - Renamed file from test_arch_pgtable.c to arch_pgtable_test.c
> - Added relevant MODULE_DESCRIPTION() and MODULE_AUTHOR() details
> - Dropped loadable module config option
> - Basic tests now use memory blocks with required size and alignment
> - PUD aligned memory block gets allocated with alloc_contig_range()
> - If PUD aligned memory could not be allocated it falls back on PMD aligned
>   memory block from page allocator and pud_* tests are skipped
> - Clear and populate tests now operate on real in memory page table entries
> - Dummy mm_struct gets allocated with mm_alloc()
> - Dummy page table entries get allocated with [pud|pmd|pte]_alloc_[map]()
> - Simplified [p4d|pgd]_basic_tests(), now has random values in the entries
> 
> Original RFC V1:
> 
> https://lore.kernel.org/linux-mm/1564037723-26676-1-git-send-email-anshuman.khandual@arm.com/
> 
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Mark Brown <broonie@kernel.org>
> Cc: Steven Price <Steven.Price@arm.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> Cc: Kees Cook <keescook@chromium.org>
> Cc: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Sri Krishna chowdary <schowdary@nvidia.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Cc: Russell King - ARM Linux <linux@armlinux.org.uk>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Vineet Gupta <vgupta@synopsys.com>
> Cc: James Hogan <jhogan@kernel.org>
> Cc: Paul Burton <paul.burton@mips.com>
> Cc: Ralf Baechle <ralf@linux-mips.org>
> Cc: Kirill A. Shutemov <kirill@shutemov.name>
> Cc: Gerald Schaefer <gerald.schaefer@de.ibm.com>
> Cc: Christophe Leroy <christophe.leroy@c-s.fr>
> Cc: Mike Kravetz <mike.kravetz@oracle.com>
> Cc: linux-snps-arc@lists.infradead.org
> Cc: linux-mips@vger.kernel.org
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-ia64@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-s390@vger.kernel.org
> Cc: linux-sh@vger.kernel.org
> Cc: sparclinux@vger.kernel.org
> Cc: x86@kernel.org
> Cc: linux-kernel@vger.kernel.org
> 
> 
> Anshuman Khandual (2):
>   mm/page_alloc: Make alloc_gigantic_page() available for general use
>   mm/debug: Add tests validating architecture page table helpers
> 
>  .../debug/debug-vm-pgtable/arch-support.txt   |  34 ++
>  arch/arm64/Kconfig                            |   1 +
>  arch/x86/Kconfig                              |   1 +
>  arch/x86/include/asm/pgtable_64.h             |   6 +
>  include/asm-generic/pgtable.h                 |   6 +
>  include/linux/gfp.h                           |   3 +
>  init/main.c                                   |   1 +
>  lib/Kconfig.debug                             |  21 +
>  mm/Makefile                                   |   1 +
>  mm/debug_vm_pgtable.c                         | 450 ++++++++++++++++++
>  mm/hugetlb.c                                  |  76 +--
>  mm/page_alloc.c                               |  98 ++++
>  12 files changed, 623 insertions(+), 75 deletions(-)
>  create mode 100644 Documentation/features/debug/debug-vm-pgtable/arch-support.txt
>  create mode 100644 mm/debug_vm_pgtable.c
> 

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 0/2] mm/debug: Add tests validating architecture page table helpers

[Anshuman Khandual]

On 10/15/2019 08:11 PM, Qian Cai wrote:
> The x86 will crash with linux-next during boot due to this series (v5) with the
> config below plus CONFIG_DEBUG_VM_PGTABLE=y. I am not sure if v6 would address
> it.
> 
> https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
> 
> [   33.862600][    T1] page:ffffea0009000000 is uninitialized and poisoned
> [   33.862608][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff
> ffffff871140][    T1]  ? _raw_spin_unlock_irq+0x27/0x40
> [   33.871140][    T1]  ? rest_init+0x307/0x307
> [   33.871140][    T1]  kernel_init+0x11/0x139
> [   33.871140][    T1]  ? rest_init+0x307/0x307
> [   33.871140][    T1]  ret_from_fork+0x27/0x50
> [   33.871140][    T1] Modules linked in:
> [   33.871140][    T1] ---[ end trace e99d392b0f7befbd ]---
> [   33.871140][    T1] RIP: 0010:alloc_gigantic_page_order+0x3fe/0x490

Hmm, with defconfig (DEBUG_VM=y and DEBUG_VM_PGTABLE=y) it does not crash but
with the config above, it does. Just wondering if it is possible that these
pages might not been initialized yet because DEFERRED_STRUCT_PAGE_INIT=y ?

[   13.898549][    T1] page:ffffea0005000000 is uninitialized and poisoned
[   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
[   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
[   13.898549][    T1] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
[   13.898549][    T1] ------------[ cut here ]------------
[   13.898549][    T1] kernel BUG at ./include/linux/mm.h:1107!
[   13.898549][    T1] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
[   13.898549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-rc3-next-20191015+ #

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 2/2] mm/debug: Add tests validating architecture page table helpers

[Qian Cai]

On Tue, 2019-10-15 at 14:51 +0530, Anshuman Khandual wrote:
> +static unsigned long __init get_random_vaddr(void)
> +{
> +	unsigned long random_vaddr, random_pages, total_user_pages;
> +
> +	total_user_pages = (TASK_SIZE - FIRST_USER_ADDRESS) / PAGE_SIZE;
> +
> +	random_pages = get_random_long() % total_user_pages;
> +	random_vaddr = FIRST_USER_ADDRESS + random_pages * PAGE_SIZE;
> +
> +	WARN_ON(random_vaddr > TASK_SIZE);
> +	WARN_ON(random_vaddr < FIRST_USER_ADDRESS);

It would be nice if this patch does not introduce a new W=1 GCC warning here on
x86 because FIRST_USER_ADDRESS is 0, and GCC think the code is dumb because
"random_vaddr" is unsigned,

In file included from ./arch/x86/include/asm/bug.h:83,
                 from ./include/linux/bug.h:5,
                 from ./include/linux/mmdebug.h:5,
                 from ./include/linux/gfp.h:5,
                 from mm/debug_vm_pgtable.c:13:
mm/debug_vm_pgtable.c: In function ‘get_random_vaddr’:
mm/debug_vm_pgtable.c:359:23: warning: comparison of unsigned expression < 0 is
always false [-Wtype-limits]
  WARN_ON(random_vaddr < FIRST_USER_ADDRESS);
                       ^
./include/asm-generic/bug.h:113:25: note: in definition of macro ‘WARN_ON’
  int __ret_warn_on = !!(condition);    \
                         ^~~~~~~~~

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 0/2] mm/debug: Add tests validating architecture page table helpers

[Qian Cai]

On Tue, 2019-10-15 at 20:51 +0530, Anshuman Khandual wrote:
> 
> On 10/15/2019 08:11 PM, Qian Cai wrote:
> > The x86 will crash with linux-next during boot due to this series (v5) with the
> > config below plus CONFIG_DEBUG_VM_PGTABLE=y. I am not sure if v6 would address
> > it.
> > 
> > https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
> > 
> > [   33.862600][    T1] page:ffffea0009000000 is uninitialized and poisoned
> > [   33.862608][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff
> > ffffff871140][    T1]  ? _raw_spin_unlock_irq+0x27/0x40
> > [   33.871140][    T1]  ? rest_init+0x307/0x307
> > [   33.871140][    T1]  kernel_init+0x11/0x139
> > [   33.871140][    T1]  ? rest_init+0x307/0x307
> > [   33.871140][    T1]  ret_from_fork+0x27/0x50
> > [   33.871140][    T1] Modules linked in:
> > [   33.871140][    T1] ---[ end trace e99d392b0f7befbd ]---
> > [   33.871140][    T1] RIP: 0010:alloc_gigantic_page_order+0x3fe/0x490
> 
> Hmm, with defconfig (DEBUG_VM=y and DEBUG_VM_PGTABLE=y) it does not crash but
> with the config above, it does. Just wondering if it is possible that these
> pages might not been initialized yet because DEFERRED_STRUCT_PAGE_INIT=y ?

Yes, this patch works fine.

diff --git a/init/main.c b/init/main.c
index 676d8020dd29..591be8f9e8e0 100644
--- a/init/main.c
+++ b/init/main.c
@@ -1177,7 +1177,6 @@ static noinline void __init kernel_init_freeable(void)
        workqueue_init();
 
        init_mm_internals();
-       debug_vm_pgtable();
 
        do_pre_smp_initcalls();
        lockup_detector_init();
@@ -1186,6 +1185,8 @@ static noinline void __init kernel_init_freeable(void)
        sched_init_smp();
 
        page_alloc_init_late();
+       debug_vm_pgtable();
+
        /* Initialize page ext after all struct pages are initialized. */
        page_ext_init();

> 
> [   13.898549][    T1] page:ffffea0005000000 is uninitialized and poisoned
> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
> [   13.898549][    T1] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
> [   13.898549][    T1] ------------[ cut here ]------------
> [   13.898549][    T1] kernel BUG at ./include/linux/mm.h:1107!
> [   13.898549][    T1] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
> [   13.898549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-rc3-next-20191015+ #

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 0/2] mm/debug: Add tests validating architecture page table helpers

[Michal Hocko]

On Tue 15-10-19 20:51:11, Anshuman Khandual wrote:
> 
> 
> On 10/15/2019 08:11 PM, Qian Cai wrote:
> > The x86 will crash with linux-next during boot due to this series (v5) with the
> > config below plus CONFIG_DEBUG_VM_PGTABLE=y. I am not sure if v6 would address
> > it.
> > 
> > https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
> > 
> > [   33.862600][    T1] page:ffffea0009000000 is uninitialized and poisoned
> > [   33.862608][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff
> > ffffff871140][    T1]  ? _raw_spin_unlock_irq+0x27/0x40
> > [   33.871140][    T1]  ? rest_init+0x307/0x307
> > [   33.871140][    T1]  kernel_init+0x11/0x139
> > [   33.871140][    T1]  ? rest_init+0x307/0x307
> > [   33.871140][    T1]  ret_from_fork+0x27/0x50
> > [   33.871140][    T1] Modules linked in:
> > [   33.871140][    T1] ---[ end trace e99d392b0f7befbd ]---
> > [   33.871140][    T1] RIP: 0010:alloc_gigantic_page_order+0x3fe/0x490
> 
> Hmm, with defconfig (DEBUG_VM=y and DEBUG_VM_PGTABLE=y) it does not crash but
> with the config above, it does. Just wondering if it is possible that these
> pages might not been initialized yet because DEFERRED_STRUCT_PAGE_INIT=y ?

Quite likely. You need to wait for page_alloc_init_late to finish.
> 
> [   13.898549][    T1] page:ffffea0005000000 is uninitialized and poisoned
> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
> [   13.898549][    T1] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
> [   13.898549][    T1] ------------[ cut here ]------------
> [   13.898549][    T1] kernel BUG at ./include/linux/mm.h:1107!
> [   13.898549][    T1] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
> [   13.898549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-rc3-next-20191015+ #

-- 
Michal Hocko
SUSE Labs

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 0/2] mm/debug: Add tests validating architecture page table helpers

[Anshuman Khandual]

On 10/16/2019 12:12 AM, Qian Cai wrote:
> On Tue, 2019-10-15 at 20:51 +0530, Anshuman Khandual wrote:
>>
>> On 10/15/2019 08:11 PM, Qian Cai wrote:
>>> The x86 will crash with linux-next during boot due to this series (v5) with the
>>> config below plus CONFIG_DEBUG_VM_PGTABLE=y. I am not sure if v6 would address
>>> it.
>>>
>>> https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
>>>
>>> [   33.862600][    T1] page:ffffea0009000000 is uninitialized and poisoned
>>> [   33.862608][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff
>>> ffffff871140][    T1]  ? _raw_spin_unlock_irq+0x27/0x40
>>> [   33.871140][    T1]  ? rest_init+0x307/0x307
>>> [   33.871140][    T1]  kernel_init+0x11/0x139
>>> [   33.871140][    T1]  ? rest_init+0x307/0x307
>>> [   33.871140][    T1]  ret_from_fork+0x27/0x50
>>> [   33.871140][    T1] Modules linked in:
>>> [   33.871140][    T1] ---[ end trace e99d392b0f7befbd ]---
>>> [   33.871140][    T1] RIP: 0010:alloc_gigantic_page_order+0x3fe/0x490
>>
>> Hmm, with defconfig (DEBUG_VM=y and DEBUG_VM_PGTABLE=y) it does not crash but
>> with the config above, it does. Just wondering if it is possible that these
>> pages might not been initialized yet because DEFERRED_STRUCT_PAGE_INIT=y ?
> 
> Yes, this patch works fine.
> 
> diff --git a/init/main.c b/init/main.c
> index 676d8020dd29..591be8f9e8e0 100644
> --- a/init/main.c
> +++ b/init/main.c
> @@ -1177,7 +1177,6 @@ static noinline void __init kernel_init_freeable(void)
>         workqueue_init();
>  
>         init_mm_internals();
> -       debug_vm_pgtable();
>  
>         do_pre_smp_initcalls();
>         lockup_detector_init();
> @@ -1186,6 +1185,8 @@ static noinline void __init kernel_init_freeable(void)
>         sched_init_smp();
>  
>         page_alloc_init_late();
> +       debug_vm_pgtable();
> +
>         /* Initialize page ext after all struct pages are initialized. */
>         page_ext_init();
> 

Sure, will keep this in mind if we at all end up with memory allocation approach
for this test.

>>
>> [   13.898549][    T1] page:ffffea0005000000 is uninitialized and poisoned
>> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
>> [   13.898549][    T1] raw: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
>> [   13.898549][    T1] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
>> [   13.898549][    T1] ------------[ cut here ]------------
>> [   13.898549][    T1] kernel BUG at ./include/linux/mm.h:1107!
>> [   13.898549][    T1] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
>> [   13.898549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-rc3-next-20191015+ #
> 

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc

Re: [PATCH V6 2/2] mm/debug: Add tests validating architecture page table helpers

[Anshuman Khandual]

On 10/15/2019 11:39 PM, Qian Cai wrote:
> On Tue, 2019-10-15 at 14:51 +0530, Anshuman Khandual wrote:
>> +static unsigned long __init get_random_vaddr(void)
>> +{
>> +	unsigned long random_vaddr, random_pages, total_user_pages;
>> +
>> +	total_user_pages = (TASK_SIZE - FIRST_USER_ADDRESS) / PAGE_SIZE;
>> +
>> +	random_pages = get_random_long() % total_user_pages;
>> +	random_vaddr = FIRST_USER_ADDRESS + random_pages * PAGE_SIZE;
>> +
>> +	WARN_ON(random_vaddr > TASK_SIZE);
>> +	WARN_ON(random_vaddr < FIRST_USER_ADDRESS);
> 
> It would be nice if this patch does not introduce a new W=1 GCC warning here on
> x86 because FIRST_USER_ADDRESS is 0, and GCC think the code is dumb because
> "random_vaddr" is unsigned,
> 
> In file included from ./arch/x86/include/asm/bug.h:83,
>                  from ./include/linux/bug.h:5,
>                  from ./include/linux/mmdebug.h:5,
>                  from ./include/linux/gfp.h:5,
>                  from mm/debug_vm_pgtable.c:13:
> mm/debug_vm_pgtable.c: In function ‘get_random_vaddr’:
> mm/debug_vm_pgtable.c:359:23: warning: comparison of unsigned expression < 0 is
> always false [-Wtype-limits]
>   WARN_ON(random_vaddr < FIRST_USER_ADDRESS);
>                        ^
> ./include/asm-generic/bug.h:113:25: note: in definition of macro ‘WARN_ON’
>   int __ret_warn_on = !!(condition);    \
>                          ^~~~~~~~~

The test checks against an erroneous unsigned long overflow when
FIRST_USER_ADDRESS is not 0 but a positive number. Wondering if
the compiler will still complain if we merge both the WARN_ON()
checks as || on a single statement.

_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc