Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Maxim Levitsky]

Hi!

My system has 64G of ram and I want to create 32 1G huge pages to use
in KVM virtualization,
on demand, only when VM is running.

So I booted the kernel with
'hugepagesz=1G hugepages=0 default_hugepagesz=1G movablecore=40G'

However I still can't allocate the pages reliably.
For instance this simple script is enough to make it not possible to
even allocate one 1G huge page after few dozens of iterations:

while true ; do
    sudo hugeadm  --enable-zone-movable  --pool-pages-min 1G:0G
    sudo hugeadm  --enable-zone-movable  --pool-pages-min 1G:60G
done


I disabled mlock systemwide (now ulimit -l shows 0), I still see 8
pages mlocked in  zone 'Movable' but this is not enough to explain
this
nr_mlock     8

I do have around 64GB of swap too, but I see no even an attempt to use it.

# free
              total        used        free      shared  buff/cache   available
Mem:       65887928     1748344    62640276       61688     1499308    62053832
Swap:      67108860           0    67108860

Any idea about what is going on?

This was tested on 4.14.0-rc5 (my custom compiled) and on several
older kernels (4.10,4.12,4.13) from ubuntu repositories.

Disabling/enabling transparent huge pages in the kernel config didn't
make a difference.

VT-d was enabled during the tests (intel_iommu=on,igfx_off) if that
would make any difference, but no VM was started when I run the above
script, in fact I run it just after the system booted.

Best regards,
          Maxim Levitsky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Sat 04-11-17 11:55:14, Maxim Levitsky wrote:
> Hi!
> 
> My system has 64G of ram and I want to create 32 1G huge pages to use
> in KVM virtualization,
> on demand, only when VM is running.
> 
> So I booted the kernel with
> 'hugepagesz=1G hugepages=0 default_hugepagesz=1G movablecore=40G'

Why do you think movablecore will help you? Giga pages are not
migrateable and as such they do not end up on movable zones.

I have recently changed the code to reflect that reality because
allowing giga pages to consume movable zone simply breaks memory
hotplug.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Maxim Levitsky]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

I am fully aware of this.
This is why we have /proc/vm/treat_hugepages_as_moveable which I did set.
Did you remove this option?

I don't need/have memory hotplug so I am ok with huge pages beeing not
movable in the movable zone.
The idea here is that other pages in that zone should be moveable so I
should be able to move all of them outside and replace them with hugepages.
This clearly doesn't work here so thats why I am asking my question

Best regards,
    Maxim Levitsky

[-- Attachment #2: Type: text/html, Size: 769 bytes --]

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Mon 06-11-17 19:03:08, Maxim Levitsky wrote:
> I am fully aware of this.
> This is why we have /proc/vm/treat_hugepages_as_moveable which I did set.
> Did you remove this option?

Yes http://lkml.kernel.org/r/20171003072619.8654-1-mhocko@kernel.org

> I don't need/have memory hotplug so I am ok with huge pages beeing not
> movable in the movable zone.
> The idea here is that other pages in that zone should be moveable so I
> should be able to move all of them outside and replace them with hugepages.
> This clearly doesn't work here so thats why I am asking my question

This is an abuse of the zone movable. If we really want gigapages
movable then we should implement that. Maybe it would be as simple as
updating hugepage_migration_supported to support PUD pages. But this
requires some testing.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Maxim Levitsky]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

Isn't this a non backward compatible change? Why to remove an optional non
default option for use cases like mine.
I won't argue with you on this, but my question was different, and was why
the kernel can't move other pages from moveable zone in my case.

PS: removed LKML from the CC because I am on mobile and that shit gmail
client sends only html mail. Sorry for that.

Best regards,
      Maxim Levitsky

[-- Attachment #2: Type: text/html, Size: 581 bytes --]

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Mon 06-11-17 19:36:38, Maxim Levitsky wrote:
> Isn't this a non backward compatible change? Why to remove an optional non
> default option for use cases like mine.

Well, strictly speaking it is. The reality is that with the current
implementation the option breaks the hotplug usecase. I can see your
argument about the opt in and we might need to hold on this patch for
merging uut it seems that http://lkml.kernel.org/r/20171003072619.8654-1-mhocko@kernel.org
is not the problem you are seeing.

> I won't argue with you on this, but my question was different, and was why
> the kernel can't move other pages from moveable zone in my case.

OK, I have re-read your original report where you say
: This was tested on 4.14.0-rc5 (my custom compiled) and on several
: older kernels (4.10,4.12,4.13) from ubuntu repositories.

Does that mean that this a new regression in 4.14-rc5 or you see the
problem in other kernels too?

If this a new rc5 thing then 79b63f12abcb ("mm, hugetlb: do not
allocate non-migrateable gigantic pages from movable zones") might be
related. Although it shouldn't if hugepages_treat_as_movable is enabled.

I wouldn't be all that surprised if this was an older issue, though. If
I look at pfn_range_valid_gigantic it seems that the page count check
makes it just too easy to fail even on migratable memory. To be honest
I consider the giga pages runtime support rather fragile and that is
why I wasn't very much afraid to remove hacks that allow breaking other
usecases.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Maxim Levitsky]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Yes, I tested git head from mainline and few kernels from ubuntu repos
since I was lazy to compile them too.

Do you have an idea what can I do about this issue? Do you think its
feasable to fix this?

And if not using moveable zone, how would it even be possible to have
guaranreed allocation of 1g pages

I do know some kernel programming (I contributed some drivers for my
laptop) so I could help if you have a direction for me to take.

Best regards,
      Maxim Levitsky

[-- Attachment #2: Type: text/html, Size: 966 bytes --]

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Mon 06-11-17 20:13:36, Maxim Levitsky wrote:
> Yes, I tested git head from mainline and few kernels from ubuntu repos
> since I was lazy to compile them too.

OK, so this hasn't worked realiably as I've suspected.

> Do you have an idea what can I do about this issue? Do you think its
> feasable to fix this?

Well, I think that giga pages need quite some love to be usable
reliably. The current implementation is more towards "make it work if
there is enough unused memory".

> And if not using moveable zone, how would it even be possible to have
> guaranreed allocation of 1g pages

Having a guaranteed giga pages is something the kernel is not yet ready
to offer.  Abusing zone movable might look like the right direction
but that is not really the case until we make sure those pages are
migratable.

There has been a simple patch which makes PUD (1GB) pages migrateable
http://lkml.kernel.org/r/20170913101047.GA13026@gmail.com but I've had
concerns that it really didn't consider the migration path much
http://lkml.kernel.org/r/20171003073301.hydw7jf2wztsx2om%40dhcp22.suse.cz
I still believe the patch is not complete but maybe it is not that far
away from being so. E.g. the said pfn_range_valid_gigantic can be
enhanced to make the migration much more reliable or get rid of it
altogether because the pfn based allocator already knows how to do
migration and other stuff.

I can help some with that.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Mon 06-11-17 19:32:37, Michal Hocko wrote:
> On Mon 06-11-17 20:13:36, Maxim Levitsky wrote:
> > Yes, I tested git head from mainline and few kernels from ubuntu repos
> > since I was lazy to compile them too.
> 
> OK, so this hasn't worked realiably as I've suspected.
> 
> > Do you have an idea what can I do about this issue? Do you think its
> > feasable to fix this?
> 
> Well, I think that giga pages need quite some love to be usable
> reliably. The current implementation is more towards "make it work if
> there is enough unused memory".
> 
> > And if not using moveable zone, how would it even be possible to have
> > guaranreed allocation of 1g pages
> 
> Having a guaranteed giga pages is something the kernel is not yet ready
> to offer.  Abusing zone movable might look like the right direction
> but that is not really the case until we make sure those pages are
> migratable.
> 
> There has been a simple patch which makes PUD (1GB) pages migrateable
> http://lkml.kernel.org/r/20170913101047.GA13026@gmail.com but I've had
> concerns that it really didn't consider the migration path much
> http://lkml.kernel.org/r/20171003073301.hydw7jf2wztsx2om%40dhcp22.suse.cz
> I still believe the patch is not complete but maybe it is not that far
> away from being so. E.g. the said pfn_range_valid_gigantic can be
> enhanced to make the migration much more reliable or get rid of it
> altogether because the pfn based allocator already knows how to do
> migration and other stuff.

Here is the first shot on the weird pfn_range_valid_gigantic. This is
completely (even compile) untested. It should just give an idea. I will
think about this some more later. If you have a scratch system that you
are not afraid to play with I would appreciate if you could give it a
try.
---
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 5ab12fda8ed5..17ca753560b7 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1071,34 +1071,6 @@ static int __alloc_gigantic_page(unsigned long start_pfn,
 				  gfp_mask);
 }
 
-static bool pfn_range_valid_gigantic(struct zone *z,
-			unsigned long start_pfn, unsigned long nr_pages)
-{
-	unsigned long i, end_pfn = start_pfn + nr_pages;
-	struct page *page;
-
-	for (i = start_pfn; i < end_pfn; i++) {
-		if (!pfn_valid(i))
-			return false;
-
-		page = pfn_to_page(i);
-
-		if (page_zone(page) != z)
-			return false;
-
-		if (PageReserved(page))
-			return false;
-
-		if (page_count(page) > 0)
-			return false;
-
-		if (PageHuge(page))
-			return false;
-	}
-
-	return true;
-}
-
 static bool zone_spans_last_pfn(const struct zone *zone,
 			unsigned long start_pfn, unsigned long nr_pages)
 {
@@ -1110,7 +1082,7 @@ static struct page *alloc_gigantic_page(int nid, struct hstate *h)
 {
 	unsigned int order = huge_page_order(h);
 	unsigned long nr_pages = 1 << order;
-	unsigned long ret, pfn, flags;
+	unsigned long ret, pfn;
 	struct zonelist *zonelist;
 	struct zone *zone;
 	struct zoneref *z;
@@ -1119,28 +1091,29 @@ static struct page *alloc_gigantic_page(int nid, struct hstate *h)
 	gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
 	zonelist = node_zonelist(nid, gfp_mask);
 	for_each_zone_zonelist_nodemask(zone, z, zonelist, gfp_zone(gfp_mask), NULL) {
-		spin_lock_irqsave(&zone->lock, flags);
 
 		pfn = ALIGN(zone->zone_start_pfn, nr_pages);
 		while (zone_spans_last_pfn(zone, pfn, nr_pages)) {
-			if (pfn_range_valid_gigantic(zone, pfn, nr_pages)) {
-				/*
-				 * We release the zone lock here because
-				 * alloc_contig_range() will also lock the zone
-				 * at some point. If there's an allocation
-				 * spinning on this lock, it may win the race
-				 * and cause alloc_contig_range() to fail...
-				 */
-				spin_unlock_irqrestore(&zone->lock, flags);
-				ret = __alloc_gigantic_page(pfn, nr_pages, gfp_mask);
-				if (!ret)
-					return pfn_to_page(pfn);
-				spin_lock_irqsave(&zone->lock, flags);
+			struct page *page = pfn_to_online_page(pfn);
+
+			/*
+			 * be careful about offline pageblocks and interleaving
+			 * zones
+			 */
+			if (!page || page_zone(page) != zone) {
+				pfn += pageblock_nr_pages;
+				continue;
 			}
+			if (PageReserved(page)) {
+				pfn++;
+				continue;
+			}
+
+			ret = __alloc_gigantic_page(pfn, nr_pages, gfp_mask);
+			if (!ret)
+				return pfn_to_page(pfn);
 			pfn += nr_pages;
 		}
-
-		spin_unlock_irqrestore(&zone->lock, flags);
 	}
 
 	return NULL;
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Maxim Levitsky]

I'll try tomorrow. Thanks!
Best regards,
           Maxim Levitsky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Vlastimil Babka]

On 11/06/2017 07:32 PM, Michal Hocko wrote:
> On Mon 06-11-17 20:13:36, Maxim Levitsky wrote:
>> Yes, I tested git head from mainline and few kernels from ubuntu repos
>> since I was lazy to compile them too.
> 
> OK, so this hasn't worked realiably as I've suspected.
> 
>> Do you have an idea what can I do about this issue? Do you think its
>> feasable to fix this?
> 
> Well, I think that giga pages need quite some love to be usable
> reliably. The current implementation is more towards "make it work if
> there is enough unused memory".
> 
>> And if not using moveable zone, how would it even be possible to have
>> guaranreed allocation of 1g pages
> 
> Having a guaranteed giga pages is something the kernel is not yet ready
> to offer.  Abusing zone movable might look like the right direction
> but that is not really the case until we make sure those pages are
> migratable.

Migratable where? It's very unlikely you will be able to migrate them
away from a movable zone to a normal zone. So the use case is migration
between hotplugable nodes, so one of them can be removed? That would
probably be an improvement (even if you could not guarantee to offline
all hotplugable nodes at once without admin intervention removing those
giga pages). Right now the only scenario where giga pages are compatible
with hot-remove is to put them on the already-limited non-removable node
0...

> There has been a simple patch which makes PUD (1GB) pages migrateable
> http://lkml.kernel.org/r/20170913101047.GA13026@gmail.com but I've had
> concerns that it really didn't consider the migration path much
> http://lkml.kernel.org/r/20171003073301.hydw7jf2wztsx2om%40dhcp22.suse.cz
> I still believe the patch is not complete but maybe it is not that far
> away from being so. E.g. the said pfn_range_valid_gigantic can be
> enhanced to make the migration much more reliable or get rid of it
> altogether because the pfn based allocator already knows how to do
> migration and other stuff.
> 
> I can help some with that.
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Guaranteed allocation of huge pages (1G) using movablecore=N doesn't seem to work at all

[Michal Hocko]

On Tue 07-11-17 09:20:47, Vlastimil Babka wrote:
> On 11/06/2017 07:32 PM, Michal Hocko wrote:
> > On Mon 06-11-17 20:13:36, Maxim Levitsky wrote:
> >> Yes, I tested git head from mainline and few kernels from ubuntu repos
> >> since I was lazy to compile them too.
> > 
> > OK, so this hasn't worked realiably as I've suspected.
> > 
> >> Do you have an idea what can I do about this issue? Do you think its
> >> feasable to fix this?
> > 
> > Well, I think that giga pages need quite some love to be usable
> > reliably. The current implementation is more towards "make it work if
> > there is enough unused memory".
> > 
> >> And if not using moveable zone, how would it even be possible to have
> >> guaranreed allocation of 1g pages
> > 
> > Having a guaranteed giga pages is something the kernel is not yet ready
> > to offer.  Abusing zone movable might look like the right direction
> > but that is not really the case until we make sure those pages are
> > migratable.
> 
> Migratable where? It's very unlikely you will be able to migrate them
> away from a movable zone to a normal zone. So the use case is migration
> between hotplugable nodes, so one of them can be removed?

Yes, basically what we do for hugetlb pages normally. Smaller hugetlb
pages are more likely to succeed, though.

> That would
> probably be an improvement (even if you could not guarantee to offline
> all hotplugable nodes at once without admin intervention removing those
> giga pages). Right now the only scenario where giga pages are compatible
> with hot-remove is to put them on the already-limited non-removable node
> 0...

Yes, we will never be perfect, but I can see why people want to allocate
from movable zones so we definitely should work on making giga pages
more robust. The current state makes cry...
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>