Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Wei Wang]

On 09/14/2019 02:36 AM, Tyler Sanderson wrote:
> Hello, I'm curious about the intent of VIRTIO_BALLOON_F_FREE_PAGE_HINT 
> (commit 
> <https://github.com/torvalds/linux/commit/86a559787e6f5cf662c081363f64a20cad654195#diff-fd202acf694d9eba19c8c64da3e480c9>). 
>
>
> My understanding is that this mechanism works similarly to the 
> existing inflate/deflate queues. Pages are allocated by the guest and 
> then reported on VQ_FREE_PAGE.
>
> Question: Is there a limit to how many pages will be allocated? What 
> controls the amount of memory pressure applied?

No control for the limit currently. The implementation reports all the 
guest free pages to host.
The main usage for this feature so far is to have guest skip sending 
those guest free pages
(the more, the better) during live migration.


>
> In my experience with virtio balloon there are problems with the 
> mechanisms that are supposed to deflate the balloon in response to 
> memory pressure (e.g. OOM notifier).

What problem did you see? We've also changed balloon to use memory shrinker,
did you see the problem with shrinker as well?

>
> It seems an ideal balloon interface would allow the guest to round 
> robin through free guest physical pages, allowing the host to unback 
> them, but never having more than a few pages allocated to the balloon 
> at any one time. For example:
> 1. Guest allocates 1 page and notifies balloon device of this page's 
> address.
> 2. Host debacks the received page.
> 3. Guest frees the page.
> 4. Repeat at #1, but ensure that different pages are allocated each time.

Probably you need a mechanism to "ensure" different pages to be allocated.
The current implementation (having balloon hold the allocated pages) could
be thought of as one mechanism (it is simple).

>
> This way the "balloon size" is never more than a few pages and does 
> not create memory pressure. However the difficulty is in ensuring each 
> set of sent pages is disjoint from previously sent pages. Is there a 
> mechanism to round-robin allocations through all of guest physical 
> memory? Does VIRTIO_BALLOON_F_FREE_PAGE_HINT enable this?
>

AFAIK, no such round-robin allocation so far. This may need the page 
allocation to have states recording
the allocation history.

Best,
Wei

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[David Hildenbrand]

On 16.09.19 03:41, Wei Wang wrote:
> On 09/14/2019 02:36 AM, Tyler Sanderson wrote:
>> Hello, I'm curious about the intent of VIRTIO_BALLOON_F_FREE_PAGE_HINT 
>> (commit 
>> <https://github.com/torvalds/linux/commit/86a559787e6f5cf662c081363f64a20cad654195#diff-fd202acf694d9eba19c8c64da3e480c9>). 
>>
>>
>> My understanding is that this mechanism works similarly to the 
>> existing inflate/deflate queues. Pages are allocated by the guest and 
>> then reported on VQ_FREE_PAGE.
>>
>> Question: Is there a limit to how many pages will be allocated? What 
>> controls the amount of memory pressure applied?
> 
> No control for the limit currently. The implementation reports all the 
> guest free pages to host.
> The main usage for this feature so far is to have guest skip sending 
> those guest free pages
> (the more, the better) during live migration.
> 
> 
>>
>> In my experience with virtio balloon there are problems with the 
>> mechanisms that are supposed to deflate the balloon in response to 
>> memory pressure (e.g. OOM notifier).
> 
> What problem did you see? We've also changed balloon to use memory shrinker,
> did you see the problem with shrinker as well?
> 
>>
>> It seems an ideal balloon interface would allow the guest to round 
>> robin through free guest physical pages, allowing the host to unback 
>> them, but never having more than a few pages allocated to the balloon 
>> at any one time. For example:
>> 1. Guest allocates 1 page and notifies balloon device of this page's 
>> address.
>> 2. Host debacks the received page.
>> 3. Guest frees the page.
>> 4. Repeat at #1, but ensure that different pages are allocated each time.
> 
> Probably you need a mechanism to "ensure" different pages to be allocated.
> The current implementation (having balloon hold the allocated pages) could
> be thought of as one mechanism (it is simple).
> 
>>
>> This way the "balloon size" is never more than a few pages and does 
>> not create memory pressure. However the difficulty is in ensuring each 
>> set of sent pages is disjoint from previously sent pages. Is there a 
>> mechanism to round-robin allocations through all of guest physical 
>> memory? Does VIRTIO_BALLOON_F_FREE_PAGE_HINT enable this?

There are use cases where you really want memory pressure (page cache is
the prime example). Anyhow, I think the use case you want the
"round-robin allocations" for is better tackled by "free page reporting"
(used to be called "free page hinting") currently discussed on various
lists.

"allowing the host to unback them, but never having more than a few
pages allocated to the balloon at any one time." is similar to what
"free page reporting" does. We decided to only report bigger pages
(avoid splitting up THP in the hypervisor, overhead) and only
temporarily pull out a fixed amount of pages (16) from the page
allocator to avoid false-OOM. Guaranteeing forward progress (similar to
what you describe) is one important key concept.

-- 

Thanks,

David / dhildenb

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Michael S. Tsirkin]

On Thu, Oct 03, 2019 at 11:27:46AM -0700, Tyler Sanderson wrote:
> Sorry for the slow reply, I did some verification on my end. See responses
> inline.
> 
> On Mon, Sep 16, 2019 at 12:26 AM David Hildenbrand <david@redhat.com> wrote:
> 
>     On 16.09.19 03:41, Wei Wang wrote:
>     > On 09/14/2019 02:36 AM, Tyler Sanderson wrote:
>     >> Hello, I'm curious about the intent of VIRTIO_BALLOON_F_FREE_PAGE_HINT
>     >> (commit
>     >> <https://github.com/torvalds/linux/commit/
>     86a559787e6f5cf662c081363f64a20cad654195#
>     diff-fd202acf694d9eba19c8c64da3e480c9>).
>     >>
>     >>
>     >> My understanding is that this mechanism works similarly to the
>     >> existing inflate/deflate queues. Pages are allocated by the guest and
>     >> then reported on VQ_FREE_PAGE.
>     >>
>     >> Question: Is there a limit to how many pages will be allocated? What
>     >> controls the amount of memory pressure applied?
>     >
>     > No control for the limit currently. The implementation reports all the
>     > guest free pages to host.
>     > The main usage for this feature so far is to have guest skip sending
>     > those guest free pages
>     > (the more, the better) during live migration.
> 
> How does this differ from the regular inflate/deflate queue?
> Also, couldn't you simply skip sending pages that do not have host pages
> backing them (assuming pages added to the balloon are unbacked to reclaim the
> memory)?

Yes but putting most guest memory into the balloon would
slow the guest down significantly.


> 
>     >
>     >
>     >>
>     >> In my experience with virtio balloon there are problems with the
>     >> mechanisms that are supposed to deflate the balloon in response to
>     >> memory pressure (e.g. OOM notifier).
>     >
>     > What problem did you see? We've also changed balloon to use memory
>     shrinker,
>     > did you see the problem with shrinker as well?
> 
> Yes, I've observed problems both before and after the shrinker change (although
> different problems).
> Before the shrinker change, the overcommit accounting feature gets in the way
> and prevents allocations, even when the balloon could be deflated. The OOM
> notifier is never invoked so the balloon driver's hook into the OOM notifier is
> useless.
> After the shrinker change the overcommit accounting problem is fixed, but I
> have still found that forcibly deflating the balloon under memory pressure is
> slow enough that random allocations can still fail (is there a timeout for
> allocations?).
> For example, I've seen:
> tysand@vm ~ $ fallocate -l 5G d/foo    // d is tmpfs mount. This command causes
> balloon to require deflation.
> tysand@vm grep Mem /proc/meminfo
> MemTotal:        8172852 kB
> MemFree:          138932 kB
> MemAvailable:      83428 kB
> tysand@vm ~ $ grep Mem /proc/meminfo
> free(): invalid pointer
> -bash: wait_for: No record of process 5415
> free(): invalid pointer
> 
> Or similarly, I've seen SSH terminate with:
> tysand@vm:~$ grep Mem /proc/meminfo
> *** stack smashing detected ***: <unknown> terminated
> 
> Presumably the stack smashing and "free(): invalid pointer" are caused by
> malloc returning null in those programs and the programs not handling it
> correctly.
> 
> Notably I don't see the fallocate command fail. Usually only other processes.
> 
> 
>     >
>     >>
>     >> It seems an ideal balloon interface would allow the guest to round
>     >> robin through free guest physical pages, allowing the host to unback
>     >> them, but never having more than a few pages allocated to the balloon
>     >> at any one time. For example:
>     >> 1. Guest allocates 1 page and notifies balloon device of this page's
>     >> address.
>     >> 2. Host debacks the received page.
>     >> 3. Guest frees the page.
>     >> 4. Repeat at #1, but ensure that different pages are allocated each
>     time.
>     >
>     > Probably you need a mechanism to "ensure" different pages to be
>     allocated.
>     > The current implementation (having balloon hold the allocated pages)
>     could
>     > be thought of as one mechanism (it is simple).
>     >
>     >>
>     >> This way the "balloon size" is never more than a few pages and does
>     >> not create memory pressure. However the difficulty is in ensuring each
>     >> set of sent pages is disjoint from previously sent pages. Is there a
>     >> mechanism to round-robin allocations through all of guest physical
>     >> memory? Does VIRTIO_BALLOON_F_FREE_PAGE_HINT enable this?
> 
>     There are use cases where you really want memory pressure (page cache is
>     the prime example). Anyhow, I think the use case you want the
>     "round-robin allocations" for is better tackled by "free page reporting"
>     (used to be called "free page hinting") currently discussed on various
>     lists.
> 
>     "allowing the host to unback them, but never having more than a few
>     pages allocated to the balloon at any one time." is similar to what
>     "free page reporting" does. We decided to only report bigger pages
>     (avoid splitting up THP in the hypervisor, overhead) and only
>     temporarily pull out a fixed amount of pages (16) from the page
>     allocator to avoid false-OOM. Guaranteeing forward progress (similar to
>     what you describe) is one important key concept.
> 
> 
> I'm really excited to see this being pursued! It looks like things are actively
> moving forward.
> 
> 
> 
>     --
> 
>     Thanks,
> 
>     David / dhildenb
> 

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[David Hildenbrand]

On 04.10.19 01:15, Tyler Sanderson wrote:
> I was mistaken, the problem with overcommit accounting is not fixed by
> the change to shrinker interface.
> This means that large allocations are stopped even if they could succeed
> by deflating the balloon.

Please note that some people use the balloon for actual memory unplug -
so initiating to deflate the balloon under any circumstances is
undesired. It's different with "VIRTIO_BALLOON_F_DEFLATE_ON_OOM" being
set - however that is barely the case (at least in the setups I know :) ).

So yes, free page reporting is a different thing, because it really is
used to "hint" and not to "agree to unplug" in any scenario.

-- 

Thanks,

David / dhildenb

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Michael S. Tsirkin]

On Fri, Oct 04, 2019 at 10:06:03AM +0200, David Hildenbrand wrote:
> On 04.10.19 01:15, Tyler Sanderson wrote:
> > I was mistaken, the problem with overcommit accounting is not fixed by
> > the change to shrinker interface.
> > This means that large allocations are stopped even if they could succeed
> > by deflating the balloon.
> 
> Please note that some people use the balloon for actual memory unplug -
> so initiating to deflate the balloon under any circumstances is
> undesired. It's different with "VIRTIO_BALLOON_F_DEFLATE_ON_OOM" being
> set - however that is barely the case (at least in the setups I know :) ).
> 
> So yes, free page reporting is a different thing, because it really is
> used to "hint" and not to "agree to unplug" in any scenario.
> 
> -- 
> 
> Thanks,
> 


VIRTIO_BALLOON_F_DEFLATE_ON_OOM isn't really well thought through
at the spec level either. For example, when will we inflate again?
Current code does this at the next interrupt, which requires
host to somehow know it's time to inflate.

-- 
MST

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[David Hildenbrand]

On 04.10.19 10:35, Michael S. Tsirkin wrote:
> On Fri, Oct 04, 2019 at 10:06:03AM +0200, David Hildenbrand wrote:
>> On 04.10.19 01:15, Tyler Sanderson wrote:
>>> I was mistaken, the problem with overcommit accounting is not fixed by
>>> the change to shrinker interface.
>>> This means that large allocations are stopped even if they could succeed
>>> by deflating the balloon.
>>
>> Please note that some people use the balloon for actual memory unplug -
>> so initiating to deflate the balloon under any circumstances is
>> undesired. It's different with "VIRTIO_BALLOON_F_DEFLATE_ON_OOM" being
>> set - however that is barely the case (at least in the setups I know :) ).
>>
>> So yes, free page reporting is a different thing, because it really is
>> used to "hint" and not to "agree to unplug" in any scenario.
>>
>> -- 
>>
>> Thanks,
>>
> 
> 
> VIRTIO_BALLOON_F_DEFLATE_ON_OOM isn't really well thought through
> at the spec level either. For example, when will we inflate again?
> Current code does this at the next interrupt, which requires
> host to somehow know it's time to inflate.
> 

The host has access to memory stats of the guest, so it could come up
with some heuristics - but I do agree that is not well thought through -
one reason why it is barely used :)

-- 

Thanks,

David / dhildenb

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Michael S. Tsirkin]

On Fri, Oct 04, 2019 at 12:03:43PM -0700, Tyler Sanderson wrote:
> I think DEFLATE_ON_OOM makes sense conceptually, it's just that the
> implementation doesn't play well with the rest of memory management under
> memory pressure.
> It could probably be fixed with enough effort, but IMO free page hinting gets
> 90% of the benefit without poking the dark corners of memory management and so
> is a net win.
> 
> The obvious place where free page hinting falls short (as David pointed out
> above) is that it can't pressure the page cache.
> Would it be possible to add a mechanism that explicitly causes page cache to
> shrink without requiring the system to be under memory pressure?

Which API would you call to shrink it?


> On Fri, Oct 4, 2019 at 1:56 AM David Hildenbrand <david@redhat.com> wrote:
> 
>     On 04.10.19 10:35, Michael S. Tsirkin wrote:
>     > On Fri, Oct 04, 2019 at 10:06:03AM +0200, David Hildenbrand wrote:
>     >> On 04.10.19 01:15, Tyler Sanderson wrote:
>     >>> I was mistaken, the problem with overcommit accounting is not fixed by
>     >>> the change to shrinker interface.
>     >>> This means that large allocations are stopped even if they could
>     succeed
>     >>> by deflating the balloon.
>     >>
>     >> Please note that some people use the balloon for actual memory unplug -
>     >> so initiating to deflate the balloon under any circumstances is
>     >> undesired. It's different with "VIRTIO_BALLOON_F_DEFLATE_ON_OOM" being
>     >> set - however that is barely the case (at least in the setups I know :)
>     ).
>     >>
>     >> So yes, free page reporting is a different thing, because it really is
>     >> used to "hint" and not to "agree to unplug" in any scenario.
>     >>
>     >> --
>     >>
>     >> Thanks,
>     >>
>     >
>     >
>     > VIRTIO_BALLOON_F_DEFLATE_ON_OOM isn't really well thought through
>     > at the spec level either. For example, when will we inflate again?
>     > Current code does this at the next interrupt, which requires
>     > host to somehow know it's time to inflate.
>     >
> 
>     The host has access to memory stats of the guest, so it could come up
>     with some heuristics - but I do agree that is not well thought through -
>     one reason why it is barely used :)
> 
>     --
> 
>     Thanks,
> 
>     David / dhildenb
> 

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[David Hildenbrand]

On 04.10.19 21:03, Tyler Sanderson wrote:
> I think DEFLATE_ON_OOM makes sense conceptually, it's just that the
> implementation doesn't play well with the rest of memory management
> under memory pressure.
> It could probably be fixed with enough effort, but IMO free page hinting
> gets 90% of the benefit without poking the dark corners of memory
> management and so is a net win.
> 
> The obvious place where free page hinting falls short (as David pointed
> out above) is that it can't pressure the page cache.

One solution is to move the page cache to the hypervisor, e.g., using
emulated NVDIMMs or virtio-pmem.

> Would it be possible to add a mechanism that explicitly causes page
> cache to shrink without requiring the system to be under memory pressure?
> 

We do have a sysctl "drop_caches" which calls
iterate_supers(drop_pagecache_sb, NULL) and drop_slab().

doc/Documentation/sysctl/vm.txt:
==============================================================

drop_caches

Writing to this will cause the kernel to drop clean caches, as well as
reclaimable slab objects like dentries and inodes.  Once dropped, their
memory becomes free.

To free pagecache:
	echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
	echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
	echo 3 > /proc/sys/vm/drop_caches

This is a non-destructive operation and will not free any dirty objects.
To increase the number of objects freed by this operation, the user may run
`sync' prior to writing to /proc/sys/vm/drop_caches.  This will minimize the
number of dirty objects on the system and create more candidates to be
dropped.

This file is not a means to control the growth of the various kernel caches
(inodes, dentries, pagecache, etc...)  These objects are automatically
reclaimed by the kernel when memory is needed elsewhere on the system.

Use of this file can cause performance problems.  Since it discards cached
objects, it may cost a significant amount of I/O and CPU to recreate the
dropped objects, especially if they were under heavy use.  Because of this,
use outside of a testing or debugging environment is not recommended.

You may see informational messages in your kernel log when this file is
used:

	cat (1234): drop_caches: 3

These are informational only.  They do not mean that anything is wrong
with your system.  To disable them, echo 4 (bit 2) into drop_caches.

==============================================================

Please note the "use outside of a testing or debugging environment is
not recommended". Usually you want a "soft" version of this, e.g., via
the OOM handler (so only drop parts of the cache, not all).

-- 

Thanks,

David / dhildenb
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Michael S. Tsirkin]

On Sun, Oct 06, 2019 at 10:30:40AM +0200, David Hildenbrand wrote:
> Please note the "use outside of a testing or debugging environment is
> not recommended". Usually you want a "soft" version of this, e.g., via
> the OOM handler (so only drop parts of the cache, not all).

Right. We'll need something softer I guess. By how much, I don't know.

-- 
MST

Re: VIRTIO_BALLOON_F_FREE_PAGE_HINT

[Tyler Sanderson via Virtualization]

[-- Attachment #1.1: Type: text/plain, Size: 989 bytes --]

On Sun, Oct 6, 2019 at 4:48 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> On Sun, Oct 06, 2019 at 10:30:40AM +0200, David Hildenbrand wrote:
> > Please note the "use outside of a testing or debugging environment is
> > not recommended". Usually you want a "soft" version of this, e.g., via
> > the OOM handler (so only drop parts of the cache, not all).
>
> Right. We'll need something softer I guess. By how much, I don't know.
>

It should be left up to the guest system administrator to configure the
trade off between performance and cache memory reclaimed.
And the primary performance metric would be cache hit rate.
So maybe some configuration option that lets you drop cache pages that have
a hit rate less than X per minute.
Or more simply: Pages that haven't been hit in the last X seconds.

The other way to do it would be to set a strict cache size limit, but that
would be workload dependent and toilsome to tune correctly, so it seems
like a worse choice.


>
> --
> MST
>

[-- Attachment #1.2: Type: text/html, Size: 1580 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization