* [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups
@ 2013-03-20 18:03 Johannes Weiner
2013-03-20 18:03 ` [patch 1/5] mm: Try harder to allocate vmemmap blocks Johannes Weiner
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw)
To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel
Hotplug can happen at times when the memory situation is less than
perfect to allocate huge pages for the vmemmap. This series makes the
allocation try harder in patch #1. The remaining patches allow x86-64
to fall back to regular pages as a last resort before the hotplug
event fails completely. As a prerequisite to this, the arch interface
to sparse is cleaned up a little, which should also enable other
architectures to easily mix huge and regular pages in the vmemmap.
arch/arm64/mm/mmu.c | 13 +++++--------
arch/ia64/mm/discontig.c | 7 +++----
arch/powerpc/mm/init_64.c | 11 +++--------
arch/s390/mm/vmem.c | 13 +++++--------
arch/sparc/mm/init_64.c | 7 +++----
arch/x86/mm/init_64.c | 68 ++++++++++++++++++++++++++++++++------------------------------------
include/linux/mm.h | 8 ++++----
mm/sparse-vmemmap.c | 27 +++++++++++++++++----------
mm/sparse.c | 10 ++++++++--
9 files changed, 80 insertions(+), 84 deletions(-)
^ permalink raw reply [flat|nested] 12+ messages in thread* [patch 1/5] mm: Try harder to allocate vmemmap blocks 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes Johannes Weiner ` (3 subsequent siblings) 4 siblings, 1 reply; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel From: Ben Hutchings <ben@decadent.org.uk> Hot-adding memory on x86_64 normally requires huge page allocation. When this is done to a VM guest, it's usually because the system is already tight on memory, so the request tends to fail. Try to avoid this by adding __GFP_REPEAT to the allocation flags. Reported-and-tested-by: Bernhard Schmidt <Bernhard.Schmidt@lrz.de> Reference: http://bugs.debian.org/699913 Signed-off-by: Ben Hutchings <ben@decadent.org.uk> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- mm/sparse-vmemmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 1b7e22a..22b7e18 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -53,10 +53,12 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) struct page *page; if (node_state(node, N_HIGH_MEMORY)) - page = alloc_pages_node(node, - GFP_KERNEL | __GFP_ZERO, get_order(size)); + page = alloc_pages_node( + node, GFP_KERNEL | __GFP_ZERO | __GFP_REPEAT, + get_order(size)); else - page = alloc_pages(GFP_KERNEL | __GFP_ZERO, + page = alloc_pages( + GFP_KERNEL | __GFP_ZERO | __GFP_REPEAT, get_order(size)); if (page) return page_address(page); -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 1/5] mm: Try harder to allocate vmemmap blocks 2013-03-20 18:03 ` [patch 1/5] mm: Try harder to allocate vmemmap blocks Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 0 siblings, 0 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel From: Ben Hutchings <ben@decadent.org.uk> Hot-adding memory on x86_64 normally requires huge page allocation. When this is done to a VM guest, it's usually because the system is already tight on memory, so the request tends to fail. Try to avoid this by adding __GFP_REPEAT to the allocation flags. Reported-and-tested-by: Bernhard Schmidt <Bernhard.Schmidt@lrz.de> Reference: http://bugs.debian.org/699913 Signed-off-by: Ben Hutchings <ben@decadent.org.uk> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- mm/sparse-vmemmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 1b7e22a..22b7e18 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -53,10 +53,12 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) struct page *page; if (node_state(node, N_HIGH_MEMORY)) - page = alloc_pages_node(node, - GFP_KERNEL | __GFP_ZERO, get_order(size)); + page = alloc_pages_node( + node, GFP_KERNEL | __GFP_ZERO | __GFP_REPEAT, + get_order(size)); else - page = alloc_pages(GFP_KERNEL | __GFP_ZERO, + page = alloc_pages( + GFP_KERNEL | __GFP_ZERO | __GFP_REPEAT, get_order(size)); if (page) return page_address(page); -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner 2013-03-20 18:03 ` [patch 1/5] mm: Try harder to allocate vmemmap blocks Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:43 ` David Miller 2013-03-20 18:03 ` [patch 3/5] x86-64: remove dead debugging code for !pse setups Johannes Weiner ` (2 subsequent siblings) 4 siblings, 2 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel The sparse code, when asking the architecture to populate the vmemmap, specifies the section range as a starting page and a number of pages. This is an awkward interface, because none of the arch-specific code actually thinks of the range in terms of 'struct page' units and always translates it to bytes first. In addition, later patches mix huge page and regular page backing for the vmemmap. For this, they need to call vmemmap_populate_basepages() on sub-section ranges with PAGE_SIZE and PMD_SIZE in mind. But these are not necessarily multiples of the 'struct page' size and so this unit is too coarse. Just translate the section range into bytes once in the generic sparse code, then pass byte ranges down the stack. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/arm64/mm/mmu.c | 13 +++++-------- arch/ia64/mm/discontig.c | 7 +++---- arch/powerpc/mm/init_64.c | 11 +++-------- arch/s390/mm/vmem.c | 13 +++++-------- arch/sparc/mm/init_64.c | 7 +++---- arch/x86/mm/init_64.c | 15 +++++---------- include/linux/mm.h | 8 ++++---- mm/sparse-vmemmap.c | 19 ++++++++++++------- mm/sparse.c | 10 ++++++++-- 9 files changed, 48 insertions(+), 55 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 224b44a..369fbe1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -391,17 +391,14 @@ int kern_addr_valid(unsigned long addr) } #ifdef CONFIG_SPARSEMEM_VMEMMAP #ifdef CONFIG_ARM64_64K_PAGES -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - return vmemmap_populate_basepages(start_page, size, node); + return vmemmap_populate_basepages(start, end, node); } #else /* !CONFIG_ARM64_64K_PAGES */ -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr = start; unsigned long next; pgd_t *pgd; pud_t *pud; @@ -434,7 +431,7 @@ int __meminit vmemmap_populate(struct page *start_page, return 0; } #endif /* CONFIG_ARM64_64K_PAGES */ -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } #endif /* CONFIG_SPARSEMEM_VMEMMAP */ diff --git a/arch/ia64/mm/discontig.c b/arch/ia64/mm/discontig.c index c2e955e..1627c77 100644 --- a/arch/ia64/mm/discontig.c +++ b/arch/ia64/mm/discontig.c @@ -817,13 +817,12 @@ void arch_refresh_nodedata(int update_node, pg_data_t *update_pgdat) #endif #ifdef CONFIG_SPARSEMEM_VMEMMAP -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - return vmemmap_populate_basepages(start_page, size, node); + return vmemmap_populate_basepages(start, end, node); } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } #endif diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index 7e2246f..5a535b7 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -263,19 +263,14 @@ static __meminit void vmemmap_list_populate(unsigned long phys, vmemmap_list = vmem_back; } -int __meminit vmemmap_populate(struct page *start_page, - unsigned long nr_pages, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long start = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + nr_pages); unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; /* Align to the page size of the linear mapping. */ start = _ALIGN_DOWN(start, page_size); - pr_debug("vmemmap_populate page %p, %ld pages, node %d\n", - start_page, nr_pages, node); - pr_debug(" -> map %lx..%lx\n", start, end); + pr_debug("vmemmap_populate %lx..%lx, node %d\n", start, end, node); for (; start < end; start += page_size) { void *p; @@ -298,7 +293,7 @@ int __meminit vmemmap_populate(struct page *start_page, return 0; } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c index ffab84d..74609e4 100644 --- a/arch/s390/mm/vmem.c +++ b/arch/s390/mm/vmem.c @@ -191,19 +191,16 @@ static void vmem_remove_range(unsigned long start, unsigned long size) /* * Add a backed mem_map array to the virtual mem_map array. */ -int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long address, start_addr, end_addr; + unsigned long address = start; pgd_t *pg_dir; pud_t *pu_dir; pmd_t *pm_dir; pte_t *pt_dir; int ret = -ENOMEM; - start_addr = (unsigned long) start; - end_addr = (unsigned long) (start + nr); - - for (address = start_addr; address < end_addr;) { + for (address = start; address < end;) { pg_dir = pgd_offset_k(address); if (pgd_none(*pg_dir)) { pu_dir = vmem_pud_alloc(); @@ -265,11 +262,11 @@ int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) memset(start, 0, nr * sizeof(struct page)); ret = 0; out: - flush_tlb_kernel_range(start_addr, end_addr); + flush_tlb_kernel_range(start, end); return ret; } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 1588d33..6ac99d6 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2181,10 +2181,9 @@ unsigned long vmemmap_table[VMEMMAP_SIZE]; static long __meminitdata addr_start, addr_end; static int __meminitdata node_start; -int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) +int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend, + int node) { - unsigned long vstart = (unsigned long) start; - unsigned long vend = (unsigned long) (start + nr); unsigned long phys_start = (vstart - VMEMMAP_BASE); unsigned long phys_end = (vend - VMEMMAP_BASE); unsigned long addr = phys_start & VMEMMAP_CHUNK_MASK; @@ -2236,7 +2235,7 @@ void __meminit vmemmap_populate_print_last(void) } } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 474e28f..7dd132c 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1011,11 +1011,8 @@ remove_pagetable(unsigned long start, unsigned long end, bool direct) flush_tlb_all(); } -void __ref vmemmap_free(struct page *memmap, unsigned long nr_pages) +void __ref vmemmap_free(unsigned long start, unsigned long end) { - unsigned long start = (unsigned long)memmap; - unsigned long end = (unsigned long)(memmap + nr_pages); - remove_pagetable(start, end, false); } @@ -1285,17 +1282,15 @@ static long __meminitdata addr_start, addr_end; static void __meminitdata *p_start, *p_end; static int __meminitdata node_start; -int __meminit -vmemmap_populate(struct page *start_page, unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr; unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (; addr < end; addr = next) { + for (addr = start; addr < end; addr = next) { void *p = NULL; pgd = vmemmap_pgd_populate(addr, node); @@ -1352,7 +1347,7 @@ vmemmap_populate(struct page *start_page, unsigned long size, int node) } } - sync_global_pgds((unsigned long)start_page, end - 1); + sync_global_pgds(start, end - 1); return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 7acc9dc..2c6fc30 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1703,12 +1703,12 @@ pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node); void *vmemmap_alloc_block(unsigned long size, int node); void *vmemmap_alloc_block_buf(unsigned long size, int node); void vmemmap_verify(pte_t *, int, unsigned long, unsigned long); -int vmemmap_populate_basepages(struct page *start_page, - unsigned long pages, int node); -int vmemmap_populate(struct page *start_page, unsigned long pages, int node); +int vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node); +int vmemmap_populate(unsigned long start, unsigned long end, int node); void vmemmap_populate_print_last(void); #ifdef CONFIG_MEMORY_HOTPLUG -void vmemmap_free(struct page *memmap, unsigned long nr_pages); +void vmemmap_free(unsigned long start, unsigned long end); #endif void register_page_bootmem_memmap(unsigned long section_nr, struct page *map, unsigned long size); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 22b7e18..27eeab3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -147,11 +147,10 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate_basepages(unsigned long start, + unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr = start; pgd_t *pgd; pud_t *pud; pmd_t *pmd; @@ -178,9 +177,15 @@ int __meminit vmemmap_populate_basepages(struct page *start_page, struct page * __meminit sparse_mem_map_populate(unsigned long pnum, int nid) { - struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION); - int error = vmemmap_populate(map, PAGES_PER_SECTION, nid); - if (error) + unsigned long start; + unsigned long end; + struct page *map; + + map = pfn_to_page(pnum * PAGES_PER_SECTION); + start = (unsigned long)map; + end = (unsigned long)(map + PAGES_PER_SECTION); + + if (vmemmap_populate(start, end, nid)) return NULL; return map; diff --git a/mm/sparse.c b/mm/sparse.c index 7ca6dc8..a37be5f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -615,11 +615,17 @@ static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid, } static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages) { - vmemmap_free(memmap, nr_pages); + unsigned long start = (unsigned long)memmap; + unsigned long end = (unsigned long)(memmap + nr_pages); + + vmemmap_free(start, end); } static void free_map_bootmem(struct page *memmap, unsigned long nr_pages) { - vmemmap_free(memmap, nr_pages); + unsigned long start = (unsigned long)memmap; + unsigned long end = (unsigned long)(memmap + nr_pages); + + vmemmap_free(start, end); } #else static struct page *__kmalloc_section_memmap(unsigned long nr_pages) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes 2013-03-20 18:03 ` [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:43 ` David Miller 1 sibling, 0 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel The sparse code, when asking the architecture to populate the vmemmap, specifies the section range as a starting page and a number of pages. This is an awkward interface, because none of the arch-specific code actually thinks of the range in terms of 'struct page' units and always translates it to bytes first. In addition, later patches mix huge page and regular page backing for the vmemmap. For this, they need to call vmemmap_populate_basepages() on sub-section ranges with PAGE_SIZE and PMD_SIZE in mind. But these are not necessarily multiples of the 'struct page' size and so this unit is too coarse. Just translate the section range into bytes once in the generic sparse code, then pass byte ranges down the stack. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/arm64/mm/mmu.c | 13 +++++-------- arch/ia64/mm/discontig.c | 7 +++---- arch/powerpc/mm/init_64.c | 11 +++-------- arch/s390/mm/vmem.c | 13 +++++-------- arch/sparc/mm/init_64.c | 7 +++---- arch/x86/mm/init_64.c | 15 +++++---------- include/linux/mm.h | 8 ++++---- mm/sparse-vmemmap.c | 19 ++++++++++++------- mm/sparse.c | 10 ++++++++-- 9 files changed, 48 insertions(+), 55 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 224b44a..369fbe1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -391,17 +391,14 @@ int kern_addr_valid(unsigned long addr) } #ifdef CONFIG_SPARSEMEM_VMEMMAP #ifdef CONFIG_ARM64_64K_PAGES -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - return vmemmap_populate_basepages(start_page, size, node); + return vmemmap_populate_basepages(start, end, node); } #else /* !CONFIG_ARM64_64K_PAGES */ -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr = start; unsigned long next; pgd_t *pgd; pud_t *pud; @@ -434,7 +431,7 @@ int __meminit vmemmap_populate(struct page *start_page, return 0; } #endif /* CONFIG_ARM64_64K_PAGES */ -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } #endif /* CONFIG_SPARSEMEM_VMEMMAP */ diff --git a/arch/ia64/mm/discontig.c b/arch/ia64/mm/discontig.c index c2e955e..1627c77 100644 --- a/arch/ia64/mm/discontig.c +++ b/arch/ia64/mm/discontig.c @@ -817,13 +817,12 @@ void arch_refresh_nodedata(int update_node, pg_data_t *update_pgdat) #endif #ifdef CONFIG_SPARSEMEM_VMEMMAP -int __meminit vmemmap_populate(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - return vmemmap_populate_basepages(start_page, size, node); + return vmemmap_populate_basepages(start, end, node); } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } #endif diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index 7e2246f..5a535b7 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -263,19 +263,14 @@ static __meminit void vmemmap_list_populate(unsigned long phys, vmemmap_list = vmem_back; } -int __meminit vmemmap_populate(struct page *start_page, - unsigned long nr_pages, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long start = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + nr_pages); unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; /* Align to the page size of the linear mapping. */ start = _ALIGN_DOWN(start, page_size); - pr_debug("vmemmap_populate page %p, %ld pages, node %d\n", - start_page, nr_pages, node); - pr_debug(" -> map %lx..%lx\n", start, end); + pr_debug("vmemmap_populate %lx..%lx, node %d\n", start, end, node); for (; start < end; start += page_size) { void *p; @@ -298,7 +293,7 @@ int __meminit vmemmap_populate(struct page *start_page, return 0; } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c index ffab84d..74609e4 100644 --- a/arch/s390/mm/vmem.c +++ b/arch/s390/mm/vmem.c @@ -191,19 +191,16 @@ static void vmem_remove_range(unsigned long start, unsigned long size) /* * Add a backed mem_map array to the virtual mem_map array. */ -int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long address, start_addr, end_addr; + unsigned long address = start; pgd_t *pg_dir; pud_t *pu_dir; pmd_t *pm_dir; pte_t *pt_dir; int ret = -ENOMEM; - start_addr = (unsigned long) start; - end_addr = (unsigned long) (start + nr); - - for (address = start_addr; address < end_addr;) { + for (address = start; address < end;) { pg_dir = pgd_offset_k(address); if (pgd_none(*pg_dir)) { pu_dir = vmem_pud_alloc(); @@ -265,11 +262,11 @@ int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) memset(start, 0, nr * sizeof(struct page)); ret = 0; out: - flush_tlb_kernel_range(start_addr, end_addr); + flush_tlb_kernel_range(start, end); return ret; } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 1588d33..6ac99d6 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2181,10 +2181,9 @@ unsigned long vmemmap_table[VMEMMAP_SIZE]; static long __meminitdata addr_start, addr_end; static int __meminitdata node_start; -int __meminit vmemmap_populate(struct page *start, unsigned long nr, int node) +int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend, + int node) { - unsigned long vstart = (unsigned long) start; - unsigned long vend = (unsigned long) (start + nr); unsigned long phys_start = (vstart - VMEMMAP_BASE); unsigned long phys_end = (vend - VMEMMAP_BASE); unsigned long addr = phys_start & VMEMMAP_CHUNK_MASK; @@ -2236,7 +2235,7 @@ void __meminit vmemmap_populate_print_last(void) } } -void vmemmap_free(struct page *memmap, unsigned long nr_pages) +void vmemmap_free(unsigned long start, unsigned long end) { } diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 474e28f..7dd132c 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1011,11 +1011,8 @@ remove_pagetable(unsigned long start, unsigned long end, bool direct) flush_tlb_all(); } -void __ref vmemmap_free(struct page *memmap, unsigned long nr_pages) +void __ref vmemmap_free(unsigned long start, unsigned long end) { - unsigned long start = (unsigned long)memmap; - unsigned long end = (unsigned long)(memmap + nr_pages); - remove_pagetable(start, end, false); } @@ -1285,17 +1282,15 @@ static long __meminitdata addr_start, addr_end; static void __meminitdata *p_start, *p_end; static int __meminitdata node_start; -int __meminit -vmemmap_populate(struct page *start_page, unsigned long size, int node) +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr; unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (; addr < end; addr = next) { + for (addr = start; addr < end; addr = next) { void *p = NULL; pgd = vmemmap_pgd_populate(addr, node); @@ -1352,7 +1347,7 @@ vmemmap_populate(struct page *start_page, unsigned long size, int node) } } - sync_global_pgds((unsigned long)start_page, end - 1); + sync_global_pgds(start, end - 1); return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 7acc9dc..2c6fc30 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1703,12 +1703,12 @@ pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node); void *vmemmap_alloc_block(unsigned long size, int node); void *vmemmap_alloc_block_buf(unsigned long size, int node); void vmemmap_verify(pte_t *, int, unsigned long, unsigned long); -int vmemmap_populate_basepages(struct page *start_page, - unsigned long pages, int node); -int vmemmap_populate(struct page *start_page, unsigned long pages, int node); +int vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node); +int vmemmap_populate(unsigned long start, unsigned long end, int node); void vmemmap_populate_print_last(void); #ifdef CONFIG_MEMORY_HOTPLUG -void vmemmap_free(struct page *memmap, unsigned long nr_pages); +void vmemmap_free(unsigned long start, unsigned long end); #endif void register_page_bootmem_memmap(unsigned long section_nr, struct page *map, unsigned long size); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 22b7e18..27eeab3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -147,11 +147,10 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(struct page *start_page, - unsigned long size, int node) +int __meminit vmemmap_populate_basepages(unsigned long start, + unsigned long end, int node) { - unsigned long addr = (unsigned long)start_page; - unsigned long end = (unsigned long)(start_page + size); + unsigned long addr = start; pgd_t *pgd; pud_t *pud; pmd_t *pmd; @@ -178,9 +177,15 @@ int __meminit vmemmap_populate_basepages(struct page *start_page, struct page * __meminit sparse_mem_map_populate(unsigned long pnum, int nid) { - struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION); - int error = vmemmap_populate(map, PAGES_PER_SECTION, nid); - if (error) + unsigned long start; + unsigned long end; + struct page *map; + + map = pfn_to_page(pnum * PAGES_PER_SECTION); + start = (unsigned long)map; + end = (unsigned long)(map + PAGES_PER_SECTION); + + if (vmemmap_populate(start, end, nid)) return NULL; return map; diff --git a/mm/sparse.c b/mm/sparse.c index 7ca6dc8..a37be5f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -615,11 +615,17 @@ static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid, } static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages) { - vmemmap_free(memmap, nr_pages); + unsigned long start = (unsigned long)memmap; + unsigned long end = (unsigned long)(memmap + nr_pages); + + vmemmap_free(start, end); } static void free_map_bootmem(struct page *memmap, unsigned long nr_pages) { - vmemmap_free(memmap, nr_pages); + unsigned long start = (unsigned long)memmap; + unsigned long end = (unsigned long)(memmap + nr_pages); + + vmemmap_free(start, end); } #else static struct page *__kmalloc_section_memmap(unsigned long nr_pages) -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes 2013-03-20 18:03 ` [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner @ 2013-03-20 18:43 ` David Miller 1 sibling, 0 replies; 12+ messages in thread From: David Miller @ 2013-03-20 18:43 UTC (permalink / raw) To: hannes; +Cc: x86, akpm, ben, linux-mm, linux-arch, linux-kernel From: Johannes Weiner <hannes@cmpxchg.org> Date: Wed, 20 Mar 2013 14:03:29 -0400 > The sparse code, when asking the architecture to populate the vmemmap, > specifies the section range as a starting page and a number of pages. > > This is an awkward interface, because none of the arch-specific code > actually thinks of the range in terms of 'struct page' units and > always translates it to bytes first. > > In addition, later patches mix huge page and regular page backing for > the vmemmap. For this, they need to call vmemmap_populate_basepages() > on sub-section ranges with PAGE_SIZE and PMD_SIZE in mind. But these > are not necessarily multiples of the 'struct page' size and so this > unit is too coarse. > > Just translate the section range into bytes once in the generic sparse > code, then pass byte ranges down the stack. > > Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Boot tested on sparc64: Acked-by: David S. Miller <davem@davemloft.net> ^ permalink raw reply [flat|nested] 12+ messages in thread
* [patch 3/5] x86-64: remove dead debugging code for !pse setups 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner 2013-03-20 18:03 ` [patch 1/5] mm: Try harder to allocate vmemmap blocks Johannes Weiner 2013-03-20 18:03 ` [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 4/5] x86-64: use vmemmap_populate_basepages() " Johannes Weiner 2013-03-20 18:03 ` [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure Johannes Weiner 4 siblings, 1 reply; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel No need to maintain addr_end and p_end when they are never actually read anywhere on !pse setups. Remove the dead code. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 7dd132c..1acba7e 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1312,9 +1312,6 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) if (!p) return -ENOMEM; - - addr_end = addr + PAGE_SIZE; - p_end = p + PAGE_SIZE; } else { next = pmd_addr_end(addr, end); -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 3/5] x86-64: remove dead debugging code for !pse setups 2013-03-20 18:03 ` [patch 3/5] x86-64: remove dead debugging code for !pse setups Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 0 siblings, 0 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel No need to maintain addr_end and p_end when they are never actually read anywhere on !pse setups. Remove the dead code. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 7dd132c..1acba7e 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1312,9 +1312,6 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) if (!p) return -ENOMEM; - - addr_end = addr + PAGE_SIZE; - p_end = p + PAGE_SIZE; } else { next = pmd_addr_end(addr, end); -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 4/5] x86-64: use vmemmap_populate_basepages() for !pse setups 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner ` (2 preceding siblings ...) 2013-03-20 18:03 ` [patch 3/5] x86-64: remove dead debugging code for !pse setups Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure Johannes Weiner 4 siblings, 1 reply; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel We already have generic code to allocate vmemmap with regular pages, use it. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 81 ++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 43 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 1acba7e..134c85d 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1282,17 +1282,15 @@ static long __meminitdata addr_start, addr_end; static void __meminitdata *p_start, *p_end; static int __meminitdata node_start; -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +static int __meminit vmemmap_populate_hugepages(unsigned long start, + unsigned long end, int node) { unsigned long addr; - unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (addr = start; addr < end; addr = next) { - void *p = NULL; - + for (addr = start; addr < end; addr += PMD_SIZE) { pgd = vmemmap_pgd_populate(addr, node); if (!pgd) return -ENOMEM; @@ -1301,53 +1299,50 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) if (!pud) return -ENOMEM; - if (!cpu_has_pse) { - next = (addr + PAGE_SIZE) & PAGE_MASK; - pmd = vmemmap_pmd_populate(pud, addr, node); - - if (!pmd) - return -ENOMEM; - - p = vmemmap_pte_populate(pmd, addr, node); + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) { + pte_t entry; + void *p; + p = vmemmap_alloc_block_buf(PMD_SIZE, node); if (!p) return -ENOMEM; - } else { - next = pmd_addr_end(addr, end); - pmd = pmd_offset(pud, addr); - if (pmd_none(*pmd)) { - pte_t entry; - - p = vmemmap_alloc_block_buf(PMD_SIZE, node); - if (!p) - return -ENOMEM; - - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, - PAGE_KERNEL_LARGE); - set_pmd(pmd, __pmd(pte_val(entry))); - - /* check to see if we have contiguous blocks */ - if (p_end != p || node_start != node) { - if (p_start) - printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", - addr_start, addr_end-1, p_start, p_end-1, node_start); - addr_start = addr; - node_start = node; - p_start = p; - } - - addr_end = addr + PMD_SIZE; - p_end = p + PMD_SIZE; - } else - vmemmap_verify((pte_t *)pmd, node, addr, next); - } + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, + PAGE_KERNEL_LARGE); + set_pmd(pmd, __pmd(pte_val(entry))); + + /* check to see if we have contiguous blocks */ + if (p_end != p || node_start != node) { + if (p_start) + printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", + addr_start, addr_end-1, p_start, p_end-1, node_start); + addr_start = addr; + node_start = node; + p_start = p; + } + addr_end = addr + PMD_SIZE; + p_end = p + PMD_SIZE; + } else + vmemmap_verify((pte_t *)pmd, node, addr, next); } - sync_global_pgds(start, end - 1); return 0; } +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +{ + int err; + + if (cpu_has_pse) + err = vmemmap_populate_hugepages(start, end, node); + else + err = vmemmap_populate_basepages(start, end, node); + if (!err) + sync_global_pgds(start, end - 1); + return err; +} + #if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_HAVE_BOOTMEM_INFO_NODE) void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 4/5] x86-64: use vmemmap_populate_basepages() for !pse setups 2013-03-20 18:03 ` [patch 4/5] x86-64: use vmemmap_populate_basepages() " Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 0 siblings, 0 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel We already have generic code to allocate vmemmap with regular pages, use it. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 81 ++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 43 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 1acba7e..134c85d 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1282,17 +1282,15 @@ static long __meminitdata addr_start, addr_end; static void __meminitdata *p_start, *p_end; static int __meminitdata node_start; -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +static int __meminit vmemmap_populate_hugepages(unsigned long start, + unsigned long end, int node) { unsigned long addr; - unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (addr = start; addr < end; addr = next) { - void *p = NULL; - + for (addr = start; addr < end; addr += PMD_SIZE) { pgd = vmemmap_pgd_populate(addr, node); if (!pgd) return -ENOMEM; @@ -1301,53 +1299,50 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) if (!pud) return -ENOMEM; - if (!cpu_has_pse) { - next = (addr + PAGE_SIZE) & PAGE_MASK; - pmd = vmemmap_pmd_populate(pud, addr, node); - - if (!pmd) - return -ENOMEM; - - p = vmemmap_pte_populate(pmd, addr, node); + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) { + pte_t entry; + void *p; + p = vmemmap_alloc_block_buf(PMD_SIZE, node); if (!p) return -ENOMEM; - } else { - next = pmd_addr_end(addr, end); - pmd = pmd_offset(pud, addr); - if (pmd_none(*pmd)) { - pte_t entry; - - p = vmemmap_alloc_block_buf(PMD_SIZE, node); - if (!p) - return -ENOMEM; - - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, - PAGE_KERNEL_LARGE); - set_pmd(pmd, __pmd(pte_val(entry))); - - /* check to see if we have contiguous blocks */ - if (p_end != p || node_start != node) { - if (p_start) - printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", - addr_start, addr_end-1, p_start, p_end-1, node_start); - addr_start = addr; - node_start = node; - p_start = p; - } - - addr_end = addr + PMD_SIZE; - p_end = p + PMD_SIZE; - } else - vmemmap_verify((pte_t *)pmd, node, addr, next); - } + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, + PAGE_KERNEL_LARGE); + set_pmd(pmd, __pmd(pte_val(entry))); + + /* check to see if we have contiguous blocks */ + if (p_end != p || node_start != node) { + if (p_start) + printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", + addr_start, addr_end-1, p_start, p_end-1, node_start); + addr_start = addr; + node_start = node; + p_start = p; + } + addr_end = addr + PMD_SIZE; + p_end = p + PMD_SIZE; + } else + vmemmap_verify((pte_t *)pmd, node, addr, next); } - sync_global_pgds(start, end - 1); return 0; } +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node) +{ + int err; + + if (cpu_has_pse) + err = vmemmap_populate_hugepages(start, end, node); + else + err = vmemmap_populate_basepages(start, end, node); + if (!err) + sync_global_pgds(start, end - 1); + return err; +} + #if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_HAVE_BOOTMEM_INFO_NODE) void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner ` (3 preceding siblings ...) 2013-03-20 18:03 ` [patch 4/5] x86-64: use vmemmap_populate_basepages() " Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 4 siblings, 1 reply; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel Memory hotplug can happen on a machine under load, memory shortness and fragmentation, so huge page allocations for the vmemmap are not guaranteed to succeed. Try to fall back to regular pages before failing the hotplug event completely. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 51 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 134c85d..e2e7afb 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1286,11 +1286,14 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, int node) { unsigned long addr; + unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (addr = start; addr < end; addr += PMD_SIZE) { + for (addr = start; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + pgd = vmemmap_pgd_populate(addr, node); if (!pgd) return -ENOMEM; @@ -1301,31 +1304,37 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start, pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - pte_t entry; void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node); - if (!p) - return -ENOMEM; - - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, - PAGE_KERNEL_LARGE); - set_pmd(pmd, __pmd(pte_val(entry))); - - /* check to see if we have contiguous blocks */ - if (p_end != p || node_start != node) { - if (p_start) - printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", - addr_start, addr_end-1, p_start, p_end-1, node_start); - addr_start = addr; - node_start = node; - p_start = p; - } + if (p) { + pte_t entry; + + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, + PAGE_KERNEL_LARGE); + set_pmd(pmd, __pmd(pte_val(entry))); + + /* check to see if we have contiguous blocks */ + if (p_end != p || node_start != node) { + if (p_start) + printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", + addr_start, addr_end-1, p_start, p_end-1, node_start); + addr_start = addr; + node_start = node; + p_start = p; + } - addr_end = addr + PMD_SIZE; - p_end = p + PMD_SIZE; - } else + addr_end = addr + PMD_SIZE; + p_end = p + PMD_SIZE; + continue; + } + } else if (pmd_large(*pmd)) { vmemmap_verify((pte_t *)pmd, node, addr, next); + continue; + } + pr_warn_once("vmemmap: falling back to regular page backing\n"); + if (vmemmap_populate_basepages(addr, next, node)) + return -ENOMEM; } return 0; } -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure 2013-03-20 18:03 ` [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure Johannes Weiner @ 2013-03-20 18:03 ` Johannes Weiner 0 siblings, 0 replies; 12+ messages in thread From: Johannes Weiner @ 2013-03-20 18:03 UTC (permalink / raw) To: x86, Andrew Morton; +Cc: Ben Hutchings, linux-mm, linux-arch, linux-kernel Memory hotplug can happen on a machine under load, memory shortness and fragmentation, so huge page allocations for the vmemmap are not guaranteed to succeed. Try to fall back to regular pages before failing the hotplug event completely. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- arch/x86/mm/init_64.c | 51 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 134c85d..e2e7afb 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1286,11 +1286,14 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, int node) { unsigned long addr; + unsigned long next; pgd_t *pgd; pud_t *pud; pmd_t *pmd; - for (addr = start; addr < end; addr += PMD_SIZE) { + for (addr = start; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + pgd = vmemmap_pgd_populate(addr, node); if (!pgd) return -ENOMEM; @@ -1301,31 +1304,37 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start, pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - pte_t entry; void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node); - if (!p) - return -ENOMEM; - - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, - PAGE_KERNEL_LARGE); - set_pmd(pmd, __pmd(pte_val(entry))); - - /* check to see if we have contiguous blocks */ - if (p_end != p || node_start != node) { - if (p_start) - printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", - addr_start, addr_end-1, p_start, p_end-1, node_start); - addr_start = addr; - node_start = node; - p_start = p; - } + if (p) { + pte_t entry; + + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, + PAGE_KERNEL_LARGE); + set_pmd(pmd, __pmd(pte_val(entry))); + + /* check to see if we have contiguous blocks */ + if (p_end != p || node_start != node) { + if (p_start) + printk(KERN_DEBUG " [%lx-%lx] PMD -> [%p-%p] on node %d\n", + addr_start, addr_end-1, p_start, p_end-1, node_start); + addr_start = addr; + node_start = node; + p_start = p; + } - addr_end = addr + PMD_SIZE; - p_end = p + PMD_SIZE; - } else + addr_end = addr + PMD_SIZE; + p_end = p + PMD_SIZE; + continue; + } + } else if (pmd_large(*pmd)) { vmemmap_verify((pte_t *)pmd, node, addr, next); + continue; + } + pr_warn_once("vmemmap: falling back to regular page backing\n"); + if (vmemmap_populate_basepages(addr, next, node)) + return -ENOMEM; } return 0; } -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-03-20 18:43 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-03-20 18:03 [patch 0/5] sparse-vmemmap: hotplug fixes & cleanups Johannes Weiner 2013-03-20 18:03 ` [patch 1/5] mm: Try harder to allocate vmemmap blocks Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 2/5] sparse-vmemmap: specify vmemmap population range in bytes Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:43 ` David Miller 2013-03-20 18:03 ` [patch 3/5] x86-64: remove dead debugging code for !pse setups Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 4/5] x86-64: use vmemmap_populate_basepages() " Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner 2013-03-20 18:03 ` [patch 5/5] x86-64: fall back to regular page vmemmap on allocation failure Johannes Weiner 2013-03-20 18:03 ` Johannes Weiner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).