* [PATCH net-next V2 0/8] Packed virtqueue support for vhost
From: Jason Wang @ 2018-07-16 3:28 UTC (permalink / raw)
To: mst, jasowang
Cc: kvm, netdev, linux-kernel, virtualization, maxime.coquelin, wexu
Hi all:
This series implements packed virtqueues. The code were tested with
Tiwei's guest driver series at https://patchwork.ozlabs.org/cover/942297/
Pktgen test for both RX and TX does not show obvious difference with
split virtqueues. The main bottleneck is the guest Linux driver, since
it can not stress vhost for a 100% CPU utilization. A full TCP
benchmark is ongoing. Will test virtio-net pmd as well when it was
ready.
Notes:
- This version depends on Tiwei's series at https://patchwork.ozlabs.org/cover/942297/
This version were tested with:
- Zerocopy (Out of Order) support
- vIOMMU support
- mergeable buffer on/off
- busy polling on/off
- vsock (nc-vsock)
Changes from V1:
- drop uapi patch and use Tiwei's
- split the enablement of packed virtqueue into a separate patch
Changes from RFC V5:
- save unnecessary barriers during vhost_add_used_packed_n()
- more compact math for event idx
- fix failure of SET_VRING_BASE when avail_wrap_counter is true
- fix not copy avail_wrap_counter during GET_VRING_BASE
- introduce SET_VRING_USED_BASE/GET_VRING_USED_BASE for syncing last_used_idx
- rename used_wrap_counter to last_used_wrap_counter
- rebase to net-next
Changes from RFC V4:
- fix signalled_used index recording
- track avail index correctly
- various minor fixes
Changes from RFC V3:
- Fix math on event idx checking
- Sync last avail wrap counter through GET/SET_VRING_BASE
- remove desc_event prefix in the driver/device structure
Changes from RFC V2:
- do not use & in checking desc_event_flags
- off should be most significant bit
- remove the workaround of mergeable buffer for dpdk prototype
- id should be in the last descriptor in the chain
- keep _F_WRITE for write descriptor when adding used
- device flags updating should use ADDR_USED type
- return error on unexpected unavail descriptor in a chain
- return false in vhost_ve_avail_empty is descriptor is available
- track last seen avail_wrap_counter
- correctly examine available descriptor in get_indirect_packed()
- vhost_idx_diff should return u16 instead of bool
Changes from RFC V1:
- Refactor vhost used elem code to avoid open coding on used elem
- Event suppression support (compile test only).
- Indirect descriptor support (compile test only).
- Zerocopy support.
- vIOMMU support.
- SCSI/VSOCK support (compile test only).
- Fix several bugs
Jason Wang (8):
vhost: move get_rx_bufs to vhost.c
vhost: hide used ring layout from device
vhost: do not use vring_used_elem
vhost_net: do not explicitly manipulate vhost_used_elem
vhost: vhost_put_user() can accept metadata type
vhost: packed ring support
vhost: event suppression for packed ring
vhost: enable packed virtqueues
drivers/vhost/net.c | 143 ++-----
drivers/vhost/scsi.c | 62 +--
drivers/vhost/vhost.c | 994 ++++++++++++++++++++++++++++++++++++++++-----
drivers/vhost/vhost.h | 55 ++-
drivers/vhost/vsock.c | 42 +-
include/uapi/linux/vhost.h | 7 +
6 files changed, 1035 insertions(+), 268 deletions(-)
--
2.7.4
^ permalink raw reply
* [kbuild ack?] Re: [PATCH v6 0/9] x86: macrofying inline asm for better compilation
From: Ingo Molnar @ 2018-07-15 21:54 UTC (permalink / raw)
To: Nadav Amit
Cc: Juergen Gross, Kate Stewart, Kees Cook, Josh Poimboeuf,
Peter Zijlstra, Greg Kroah-Hartman, Christopher Li, X86 ML,
Linux Kernel Mailing List, Philippe Ombredanne,
virtualization@lists.linux-foundation.org, Masahiro Yamada,
linux-sparse@vger.kernel.org, Ingo Molnar, Jan Beulich,
H. Peter Anvin, Linus Torvalds, Thomas Gleixner, Sam Ravnborg,
Alok Kataria
In-Reply-To: <BA4C0931-A51D-41AE-B918-C37EFFF35840@vmware.com>
* Nadav Amit <namit@vmware.com> wrote:
> > I ran some limited number of benchmarks, and in general the performance
> > impact is not very notable. You can still see >10 cycles shaved off some
> > syscalls that manipulate page-tables (e.g., mprotect()), in which
> > paravirt caused many functions not to be inlined. In addition this
> > patch-set can prevent issues such as [1], and improves code readability
> > and maintainability.
Ok, that's good enough as a benefit, I suppose.
> > Nadav Amit (9):
> > Makefile: Prepare for using macros for inline asm
This non-trivial kbuild patch needs an Acked-by from Masahiro.
Thanks,
Ingo
^ permalink raw reply
* RE: [PATCH v6 3/3] x86: paravirt: make native_save_fl extern inline
From: David Laight @ 2018-07-13 10:16 UTC (permalink / raw)
To: 'Nick Desaulniers', mingo@redhat.com, tglx@linutronix.de
Cc: kstewart@linuxfoundation.org, andrea.parri@amarulasolutions.com,
linux-efi@vger.kernel.org, brijesh.singh@amd.com,
jan.kiszka@siemens.com, jpoimboe@redhat.com, will.deacon@arm.com,
jarkko.sakkinen@linux.intel.com,
virtualization@lists.linux-foundation.org,
yamada.masahiro@socionext.com, manojgupta@google.com,
hpa@zytor.com, akataria@vmware.com, tweek@google.com,
mawilcox@microsoft.com, x86@kernel.org, ghackmann
In-Reply-To: <20180621162324.36656-4-ndesaulniers@google.com>
[-- Attachment #1.1: Type: text/plain, Size: 206 bytes --]
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Please consider the environment and don't print this e-mail unless you really need to
[-- Attachment #1.2: Type: text/html, Size: 1534 bytes --]
[-- Attachment #2: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH net-next v2 0/5] virtio: support packed ring
From: Michael S. Tsirkin @ 2018-07-13 3:26 UTC (permalink / raw)
To: David Miller; +Cc: virtio-dev, netdev, linux-kernel, virtualization, wexu
In-Reply-To: <20180712.144458.2076041018423659380.davem@davemloft.net>
On Thu, Jul 12, 2018 at 02:44:58PM -0700, David Miller wrote:
> From: Tiwei Bie <tiwei.bie@intel.com>
> Date: Wed, 11 Jul 2018 10:27:06 +0800
>
> > Hello everyone,
> >
> > This patch set implements packed ring support in virtio driver.
> >
> > Some functional tests have been done with Jason's
> > packed ring implementation in vhost:
> >
> > https://lkml.org/lkml/2018/7/3/33
> >
> > Both of ping and netperf worked as expected.
>
> Michael and Jason, where are we with this series?
I'm at netdev, won't be able to review before Monday.
--
MST
^ permalink raw reply
* Re: [PATCH net-next v2 0/5] virtio: support packed ring
From: Jason Wang @ 2018-07-13 0:52 UTC (permalink / raw)
To: David Miller, tiwei.bie
Cc: virtio-dev, mst, netdev, linux-kernel, virtualization, wexu
In-Reply-To: <20180712.144458.2076041018423659380.davem@davemloft.net>
On 2018年07月13日 05:44, David Miller wrote:
> From: Tiwei Bie <tiwei.bie@intel.com>
> Date: Wed, 11 Jul 2018 10:27:06 +0800
>
>> Hello everyone,
>>
>> This patch set implements packed ring support in virtio driver.
>>
>> Some functional tests have been done with Jason's
>> packed ring implementation in vhost:
>>
>> https://lkml.org/lkml/2018/7/3/33
>>
>> Both of ping and netperf worked as expected.
> Michael and Jason, where are we with this series?
For the series:
Acked-by: Jason Wang <jasowang@redhat.com>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Wei Wang @ 2018-07-13 0:33 UTC (permalink / raw)
To: Michal Hocko
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal,
Linus Torvalds
In-Reply-To: <20180712114946.GI32648@dhcp22.suse.cz>
On 07/12/2018 07:49 PM, Michal Hocko wrote:
> On Thu 12-07-18 19:34:16, Wei Wang wrote:
>> On 07/12/2018 04:13 PM, Michal Hocko wrote:
>>> On Thu 12-07-18 10:52:08, Wei Wang wrote:
>>>> On 07/12/2018 10:30 AM, Linus Torvalds wrote:
>>>>> On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
>>>>>> Would it be better to remove __GFP_THISNODE? We actually want to get all
>>>>>> the guest free pages (from all the nodes).
>>>>> Maybe. Or maybe it would be better to have the memory balloon logic be
>>>>> per-node? Maybe you don't want to remove too much memory from one
>>>>> node? I think it's one of those "play with it" things.
>>>>>
>>>>> I don't think that's the big issue, actually. I think the real issue
>>>>> is how to react quickly and gracefully to "oops, I'm trying to give
>>>>> memory away, but now the guest wants it back" while you're in the
>>>>> middle of trying to create that 2TB list of pages.
>>>> OK. virtio-balloon has already registered an oom notifier
>>>> (virtballoon_oom_notify). I plan to add some control there. If oom happens,
>>>> - stop the page allocation;
>>>> - immediately give back the allocated pages to mm.
>>> Please don't. Oom notifier is an absolutely hideous interface which
>>> should go away sooner or later (I would much rather like the former) so
>>> do not build a new logic on top of it. I would appreciate if you
>>> actually remove the notifier much more.
>>>
>>> You can give memory back from the standard shrinker interface. If we are
>>> reaching low reclaim priorities then we are struggling to reclaim memory
>>> and then you can start returning pages back.
>> OK. Just curious why oom notifier is thought to be hideous, and has it been
>> a consensus?
> Because it is a completely non-transparent callout from the OOM context
> which is really subtle on its own. It is just too easy to end up in
> weird corner cases. We really have to be careful and be as swift as
> possible. Any potential sleep would make the OOM situation much worse
> because nobody would be able to make a forward progress or (in)direct
> dependency on MM subsystem can easily deadlock. Those are really hard
> to track down and defining the notifier as blockable by design which
> just asks for bad implementations because most people simply do not
> realize how subtle the oom context is.
>
> Another thing is that it happens way too late when we have basically
> reclaimed the world and didn't get out of the memory pressure so you can
> expect any workload is suffering already. Anybody sitting on a large
> amount of reclaimable memory should have released that memory by that
> time. Proportionally to the reclaim pressure ideally.
>
> The notifier API is completely unaware of oom constrains. Just imagine
> you are OOM in a subset of numa nodes. Callback doesn't have any idea
> about that.
>
> Moreover we do have proper reclaim mechanism that has a feedback
> loop and that should be always preferable to an abrupt reclaim.
Sounds very reasonable, thanks for the elaboration. I'll try with shrinker.
Best,
Wei
^ permalink raw reply
* Re: [PATCH net-next v2 0/5] virtio: support packed ring
From: David Miller @ 2018-07-12 21:44 UTC (permalink / raw)
To: tiwei.bie; +Cc: virtio-dev, mst, netdev, linux-kernel, virtualization, wexu
In-Reply-To: <20180711022711.7090-1-tiwei.bie@intel.com>
From: Tiwei Bie <tiwei.bie@intel.com>
Date: Wed, 11 Jul 2018 10:27:06 +0800
> Hello everyone,
>
> This patch set implements packed ring support in virtio driver.
>
> Some functional tests have been done with Jason's
> packed ring implementation in vhost:
>
> https://lkml.org/lkml/2018/7/3/33
>
> Both of ping and netperf worked as expected.
Michael and Jason, where are we with this series?
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michal Hocko @ 2018-07-12 13:12 UTC (permalink / raw)
To: Linus Torvalds
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal
In-Reply-To: <CA+55aFwku2tDH4+rfaC67xc4-cEwSrXgnQaci=e2id5ZCRE9JQ@mail.gmail.com>
[Hmm this one somehow got stuck in my outgoing emails]
On Wed 11-07-18 09:23:54, Linus Torvalds wrote:
[...]
> So I'm open to new interfaces. I just want those new interfaces to
> make sense, and be low latency and simple for the VM to do. I'm
> objecting to the incredibly baroque and heavy-weight one that can
> return near-infinite amounts of memory.
Mel was suggesting a bulk page allocator a year ago [1]. I can see only
slab bulk api so I am not sure what happened with that work. Anyway
I think that starting with what we have right now is much more
appropriate than over design this thing from the early beginning.
[1] http://lkml.kernel.org/r/20170109163518.6001-5-mgorman@techsingularity.net
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michal Hocko @ 2018-07-12 11:49 UTC (permalink / raw)
To: Wei Wang
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal,
Linus Torvalds
In-Reply-To: <5B473CB8.1050306@intel.com>
On Thu 12-07-18 19:34:16, Wei Wang wrote:
> On 07/12/2018 04:13 PM, Michal Hocko wrote:
> > On Thu 12-07-18 10:52:08, Wei Wang wrote:
> > > On 07/12/2018 10:30 AM, Linus Torvalds wrote:
> > > > On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
> > > > > Would it be better to remove __GFP_THISNODE? We actually want to get all
> > > > > the guest free pages (from all the nodes).
> > > > Maybe. Or maybe it would be better to have the memory balloon logic be
> > > > per-node? Maybe you don't want to remove too much memory from one
> > > > node? I think it's one of those "play with it" things.
> > > >
> > > > I don't think that's the big issue, actually. I think the real issue
> > > > is how to react quickly and gracefully to "oops, I'm trying to give
> > > > memory away, but now the guest wants it back" while you're in the
> > > > middle of trying to create that 2TB list of pages.
> > > OK. virtio-balloon has already registered an oom notifier
> > > (virtballoon_oom_notify). I plan to add some control there. If oom happens,
> > > - stop the page allocation;
> > > - immediately give back the allocated pages to mm.
> > Please don't. Oom notifier is an absolutely hideous interface which
> > should go away sooner or later (I would much rather like the former) so
> > do not build a new logic on top of it. I would appreciate if you
> > actually remove the notifier much more.
> >
> > You can give memory back from the standard shrinker interface. If we are
> > reaching low reclaim priorities then we are struggling to reclaim memory
> > and then you can start returning pages back.
>
> OK. Just curious why oom notifier is thought to be hideous, and has it been
> a consensus?
Because it is a completely non-transparent callout from the OOM context
which is really subtle on its own. It is just too easy to end up in
weird corner cases. We really have to be careful and be as swift as
possible. Any potential sleep would make the OOM situation much worse
because nobody would be able to make a forward progress or (in)direct
dependency on MM subsystem can easily deadlock. Those are really hard
to track down and defining the notifier as blockable by design which
just asks for bad implementations because most people simply do not
realize how subtle the oom context is.
Another thing is that it happens way too late when we have basically
reclaimed the world and didn't get out of the memory pressure so you can
expect any workload is suffering already. Anybody sitting on a large
amount of reclaimable memory should have released that memory by that
time. Proportionally to the reclaim pressure ideally.
The notifier API is completely unaware of oom constrains. Just imagine
you are OOM in a subset of numa nodes. Callback doesn't have any idea
about that.
Moreover we do have proper reclaim mechanism that has a feedback
loop and that should be always preferable to an abrupt reclaim.
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Wei Wang @ 2018-07-12 11:34 UTC (permalink / raw)
To: Michal Hocko
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal,
Linus Torvalds
In-Reply-To: <20180712081317.GD32648@dhcp22.suse.cz>
On 07/12/2018 04:13 PM, Michal Hocko wrote:
> On Thu 12-07-18 10:52:08, Wei Wang wrote:
>> On 07/12/2018 10:30 AM, Linus Torvalds wrote:
>>> On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
>>>> Would it be better to remove __GFP_THISNODE? We actually want to get all
>>>> the guest free pages (from all the nodes).
>>> Maybe. Or maybe it would be better to have the memory balloon logic be
>>> per-node? Maybe you don't want to remove too much memory from one
>>> node? I think it's one of those "play with it" things.
>>>
>>> I don't think that's the big issue, actually. I think the real issue
>>> is how to react quickly and gracefully to "oops, I'm trying to give
>>> memory away, but now the guest wants it back" while you're in the
>>> middle of trying to create that 2TB list of pages.
>> OK. virtio-balloon has already registered an oom notifier
>> (virtballoon_oom_notify). I plan to add some control there. If oom happens,
>> - stop the page allocation;
>> - immediately give back the allocated pages to mm.
> Please don't. Oom notifier is an absolutely hideous interface which
> should go away sooner or later (I would much rather like the former) so
> do not build a new logic on top of it. I would appreciate if you
> actually remove the notifier much more.
>
> You can give memory back from the standard shrinker interface. If we are
> reaching low reclaim priorities then we are struggling to reclaim memory
> and then you can start returning pages back.
OK. Just curious why oom notifier is thought to be hideous, and has it
been a consensus?
Best,
Wei
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michal Hocko @ 2018-07-12 8:13 UTC (permalink / raw)
To: Wei Wang
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal,
Linus Torvalds
In-Reply-To: <5B46C258.40601@intel.com>
On Thu 12-07-18 10:52:08, Wei Wang wrote:
> On 07/12/2018 10:30 AM, Linus Torvalds wrote:
> > On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
> > > Would it be better to remove __GFP_THISNODE? We actually want to get all
> > > the guest free pages (from all the nodes).
> > Maybe. Or maybe it would be better to have the memory balloon logic be
> > per-node? Maybe you don't want to remove too much memory from one
> > node? I think it's one of those "play with it" things.
> >
> > I don't think that's the big issue, actually. I think the real issue
> > is how to react quickly and gracefully to "oops, I'm trying to give
> > memory away, but now the guest wants it back" while you're in the
> > middle of trying to create that 2TB list of pages.
>
> OK. virtio-balloon has already registered an oom notifier
> (virtballoon_oom_notify). I plan to add some control there. If oom happens,
> - stop the page allocation;
> - immediately give back the allocated pages to mm.
Please don't. Oom notifier is an absolutely hideous interface which
should go away sooner or later (I would much rather like the former) so
do not build a new logic on top of it. I would appreciate if you
actually remove the notifier much more.
You can give memory back from the standard shrinker interface. If we are
reaching low reclaim priorities then we are struggling to reclaim memory
and then you can start returning pages back.
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Jason Wang @ 2018-07-12 5:51 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <20180712082324-mutt-send-email-mst@kernel.org>
On 2018年07月12日 13:24, Michael S. Tsirkin wrote:
> On Thu, Jul 12, 2018 at 01:21:03PM +0800, Jason Wang wrote:
>>
>> On 2018年07月12日 11:34, Michael S. Tsirkin wrote:
>>> On Thu, Jul 12, 2018 at 11:26:12AM +0800, Jason Wang wrote:
>>>> On 2018年07月11日 19:59, Michael S. Tsirkin wrote:
>>>>> On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
>>>>>> On 2018年07月11日 11:49, Tonghao Zhang wrote:
>>>>>>> On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
>>>>>>>> On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
>>>>>>>>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>>>>>>>>>
>>>>>>>>> This patches improve the guest receive and transmit performance.
>>>>>>>>> On the handle_tx side, we poll the sock receive queue at the same time.
>>>>>>>>> handle_rx do that in the same way.
>>>>>>>>>
>>>>>>>>> For more performance report, see patch 4.
>>>>>>>>>
>>>>>>>>> v4 -> v5:
>>>>>>>>> fix some issues
>>>>>>>>>
>>>>>>>>> v3 -> v4:
>>>>>>>>> fix some issues
>>>>>>>>>
>>>>>>>>> v2 -> v3:
>>>>>>>>> This patches are splited from previous big patch:
>>>>>>>>> http://patchwork.ozlabs.org/patch/934673/
>>>>>>>>>
>>>>>>>>> Tonghao Zhang (4):
>>>>>>>>> vhost: lock the vqs one by one
>>>>>>>>> net: vhost: replace magic number of lock annotation
>>>>>>>>> net: vhost: factor out busy polling logic to vhost_net_busy_poll()
>>>>>>>>> net: vhost: add rx busy polling in tx path
>>>>>>>>>
>>>>>>>>> drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
>>>>>>>>> drivers/vhost/vhost.c | 24 ++++-------
>>>>>>>>> 2 files changed, 67 insertions(+), 65 deletions(-)
>>>>>>>>>
>>>>>>>> Hi, any progress on the new version?
>>>>>>>>
>>>>>>>> I plan to send a new series of packed virtqueue support of vhost. If you
>>>>>>>> plan to send it soon, I can wait. Otherwise, I will send my series.
>>>>>>> I rebase the codes. and find there is no improvement anymore, the
>>>>>>> patches of makita may solve the problem. jason you may send your
>>>>>>> patches, and I will do some research on busypoll.
>>>>>> I see. Maybe you can try some bi-directional traffic.
>>>>>>
>>>>>> Btw, lots of optimizations could be done for busy polling. E.g integrating
>>>>>> with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
>>>>>> to work or propose new ideas.
>>>>>>
>>>>>> Thanks
>>>>> It seems clear we do need adaptive polling.
>>>> Yes.
>>>>
>>>>> The difficulty with NAPI
>>>>> polling is it can't access guest memory easily. But maybe
>>>>> get_user_pages on the polled memory+NAPI polling can work.
>>>> You mean something like zerocopy? Looks like we can do busy polling without
>>>> it. I mean something like https://patchwork.kernel.org/patch/8707511/.
>>>>
>>>> Thanks
>>> How does this patch work? vhost_vq_avail_empty can sleep,
>>> you are calling it within an rcu read side critical section.
>> Ok, I get your meaning. I have patches to access vring through
>> get_user_pages + vmap() which should help here. (And it increase PPS about
>> 10%-20%).
> Remember you must mark it as dirty on unpin too ...
Ok.
>
>
>>> That's not the only problem btw, another one is that the
>>> CPU time spent polling isn't accounted with the VM.
>>
>> Yes, but it's not the 'issue' of this patch.
> Yes it is. polling within thread context accounts CPU correctly.
>
>> And I believe cgroup can help?
>>
>> Thanks
>
> cgroups are what's broken by polling in irq context.
But I think the NAPI busy polling is still done in process context.
Thanks
>
>>>>>>>> Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Michael S. Tsirkin @ 2018-07-12 5:24 UTC (permalink / raw)
To: Jason Wang; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <2898b8aa-8a7c-06ad-9dc0-0e97e2f8e417@redhat.com>
On Thu, Jul 12, 2018 at 01:21:03PM +0800, Jason Wang wrote:
>
>
> On 2018年07月12日 11:34, Michael S. Tsirkin wrote:
> > On Thu, Jul 12, 2018 at 11:26:12AM +0800, Jason Wang wrote:
> > >
> > > On 2018年07月11日 19:59, Michael S. Tsirkin wrote:
> > > > On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
> > > > > On 2018年07月11日 11:49, Tonghao Zhang wrote:
> > > > > > On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > > > > On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
> > > > > > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > > > > >
> > > > > > > > This patches improve the guest receive and transmit performance.
> > > > > > > > On the handle_tx side, we poll the sock receive queue at the same time.
> > > > > > > > handle_rx do that in the same way.
> > > > > > > >
> > > > > > > > For more performance report, see patch 4.
> > > > > > > >
> > > > > > > > v4 -> v5:
> > > > > > > > fix some issues
> > > > > > > >
> > > > > > > > v3 -> v4:
> > > > > > > > fix some issues
> > > > > > > >
> > > > > > > > v2 -> v3:
> > > > > > > > This patches are splited from previous big patch:
> > > > > > > > http://patchwork.ozlabs.org/patch/934673/
> > > > > > > >
> > > > > > > > Tonghao Zhang (4):
> > > > > > > > vhost: lock the vqs one by one
> > > > > > > > net: vhost: replace magic number of lock annotation
> > > > > > > > net: vhost: factor out busy polling logic to vhost_net_busy_poll()
> > > > > > > > net: vhost: add rx busy polling in tx path
> > > > > > > >
> > > > > > > > drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
> > > > > > > > drivers/vhost/vhost.c | 24 ++++-------
> > > > > > > > 2 files changed, 67 insertions(+), 65 deletions(-)
> > > > > > > >
> > > > > > > Hi, any progress on the new version?
> > > > > > >
> > > > > > > I plan to send a new series of packed virtqueue support of vhost. If you
> > > > > > > plan to send it soon, I can wait. Otherwise, I will send my series.
> > > > > > I rebase the codes. and find there is no improvement anymore, the
> > > > > > patches of makita may solve the problem. jason you may send your
> > > > > > patches, and I will do some research on busypoll.
> > > > > I see. Maybe you can try some bi-directional traffic.
> > > > >
> > > > > Btw, lots of optimizations could be done for busy polling. E.g integrating
> > > > > with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
> > > > > to work or propose new ideas.
> > > > >
> > > > > Thanks
> > > > It seems clear we do need adaptive polling.
> > > Yes.
> > >
> > > > The difficulty with NAPI
> > > > polling is it can't access guest memory easily. But maybe
> > > > get_user_pages on the polled memory+NAPI polling can work.
> > > You mean something like zerocopy? Looks like we can do busy polling without
> > > it. I mean something like https://patchwork.kernel.org/patch/8707511/.
> > >
> > > Thanks
> > How does this patch work? vhost_vq_avail_empty can sleep,
> > you are calling it within an rcu read side critical section.
>
> Ok, I get your meaning. I have patches to access vring through
> get_user_pages + vmap() which should help here. (And it increase PPS about
> 10%-20%).
Remember you must mark it as dirty on unpin too ...
> >
> > That's not the only problem btw, another one is that the
> > CPU time spent polling isn't accounted with the VM.
>
>
> Yes, but it's not the 'issue' of this patch.
Yes it is. polling within thread context accounts CPU correctly.
> And I believe cgroup can help?
>
> Thanks
cgroups are what's broken by polling in irq context.
> >
> > > > > > > Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Jason Wang @ 2018-07-12 5:21 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <20180712062756-mutt-send-email-mst@kernel.org>
On 2018年07月12日 11:34, Michael S. Tsirkin wrote:
> On Thu, Jul 12, 2018 at 11:26:12AM +0800, Jason Wang wrote:
>>
>> On 2018年07月11日 19:59, Michael S. Tsirkin wrote:
>>> On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
>>>> On 2018年07月11日 11:49, Tonghao Zhang wrote:
>>>>> On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
>>>>>> On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
>>>>>>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>>>>>>>
>>>>>>> This patches improve the guest receive and transmit performance.
>>>>>>> On the handle_tx side, we poll the sock receive queue at the same time.
>>>>>>> handle_rx do that in the same way.
>>>>>>>
>>>>>>> For more performance report, see patch 4.
>>>>>>>
>>>>>>> v4 -> v5:
>>>>>>> fix some issues
>>>>>>>
>>>>>>> v3 -> v4:
>>>>>>> fix some issues
>>>>>>>
>>>>>>> v2 -> v3:
>>>>>>> This patches are splited from previous big patch:
>>>>>>> http://patchwork.ozlabs.org/patch/934673/
>>>>>>>
>>>>>>> Tonghao Zhang (4):
>>>>>>> vhost: lock the vqs one by one
>>>>>>> net: vhost: replace magic number of lock annotation
>>>>>>> net: vhost: factor out busy polling logic to vhost_net_busy_poll()
>>>>>>> net: vhost: add rx busy polling in tx path
>>>>>>>
>>>>>>> drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
>>>>>>> drivers/vhost/vhost.c | 24 ++++-------
>>>>>>> 2 files changed, 67 insertions(+), 65 deletions(-)
>>>>>>>
>>>>>> Hi, any progress on the new version?
>>>>>>
>>>>>> I plan to send a new series of packed virtqueue support of vhost. If you
>>>>>> plan to send it soon, I can wait. Otherwise, I will send my series.
>>>>> I rebase the codes. and find there is no improvement anymore, the
>>>>> patches of makita may solve the problem. jason you may send your
>>>>> patches, and I will do some research on busypoll.
>>>> I see. Maybe you can try some bi-directional traffic.
>>>>
>>>> Btw, lots of optimizations could be done for busy polling. E.g integrating
>>>> with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
>>>> to work or propose new ideas.
>>>>
>>>> Thanks
>>> It seems clear we do need adaptive polling.
>> Yes.
>>
>>> The difficulty with NAPI
>>> polling is it can't access guest memory easily. But maybe
>>> get_user_pages on the polled memory+NAPI polling can work.
>> You mean something like zerocopy? Looks like we can do busy polling without
>> it. I mean something like https://patchwork.kernel.org/patch/8707511/.
>>
>> Thanks
> How does this patch work? vhost_vq_avail_empty can sleep,
> you are calling it within an rcu read side critical section.
Ok, I get your meaning. I have patches to access vring through
get_user_pages + vmap() which should help here. (And it increase PPS
about 10%-20%).
>
> That's not the only problem btw, another one is that the
> CPU time spent polling isn't accounted with the VM.
Yes, but it's not the 'issue' of this patch. And I believe cgroup can help?
Thanks
>
>>>>>> Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Michael S. Tsirkin @ 2018-07-12 3:34 UTC (permalink / raw)
To: Jason Wang; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <0dcaf8f1-01b5-417d-420b-d5b716a82a8a@redhat.com>
On Thu, Jul 12, 2018 at 11:26:12AM +0800, Jason Wang wrote:
>
>
> On 2018年07月11日 19:59, Michael S. Tsirkin wrote:
> > On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
> > >
> > > On 2018年07月11日 11:49, Tonghao Zhang wrote:
> > > > On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > >
> > > > > On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
> > > > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > > > >
> > > > > > This patches improve the guest receive and transmit performance.
> > > > > > On the handle_tx side, we poll the sock receive queue at the same time.
> > > > > > handle_rx do that in the same way.
> > > > > >
> > > > > > For more performance report, see patch 4.
> > > > > >
> > > > > > v4 -> v5:
> > > > > > fix some issues
> > > > > >
> > > > > > v3 -> v4:
> > > > > > fix some issues
> > > > > >
> > > > > > v2 -> v3:
> > > > > > This patches are splited from previous big patch:
> > > > > > http://patchwork.ozlabs.org/patch/934673/
> > > > > >
> > > > > > Tonghao Zhang (4):
> > > > > > vhost: lock the vqs one by one
> > > > > > net: vhost: replace magic number of lock annotation
> > > > > > net: vhost: factor out busy polling logic to vhost_net_busy_poll()
> > > > > > net: vhost: add rx busy polling in tx path
> > > > > >
> > > > > > drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
> > > > > > drivers/vhost/vhost.c | 24 ++++-------
> > > > > > 2 files changed, 67 insertions(+), 65 deletions(-)
> > > > > >
> > > > > Hi, any progress on the new version?
> > > > >
> > > > > I plan to send a new series of packed virtqueue support of vhost. If you
> > > > > plan to send it soon, I can wait. Otherwise, I will send my series.
> > > > I rebase the codes. and find there is no improvement anymore, the
> > > > patches of makita may solve the problem. jason you may send your
> > > > patches, and I will do some research on busypoll.
> > > I see. Maybe you can try some bi-directional traffic.
> > >
> > > Btw, lots of optimizations could be done for busy polling. E.g integrating
> > > with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
> > > to work or propose new ideas.
> > >
> > > Thanks
> > It seems clear we do need adaptive polling.
>
> Yes.
>
> > The difficulty with NAPI
> > polling is it can't access guest memory easily. But maybe
> > get_user_pages on the polled memory+NAPI polling can work.
>
> You mean something like zerocopy? Looks like we can do busy polling without
> it. I mean something like https://patchwork.kernel.org/patch/8707511/.
>
> Thanks
How does this patch work? vhost_vq_avail_empty can sleep,
you are calling it within an rcu read side critical section.
That's not the only problem btw, another one is that the
CPU time spent polling isn't accounted with the VM.
> >
> > > > > Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Jason Wang @ 2018-07-12 3:26 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <20180711145421-mutt-send-email-mst@kernel.org>
On 2018年07月11日 19:59, Michael S. Tsirkin wrote:
> On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
>>
>> On 2018年07月11日 11:49, Tonghao Zhang wrote:
>>> On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
>>>>
>>>> On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
>>>>> From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
>>>>>
>>>>> This patches improve the guest receive and transmit performance.
>>>>> On the handle_tx side, we poll the sock receive queue at the same time.
>>>>> handle_rx do that in the same way.
>>>>>
>>>>> For more performance report, see patch 4.
>>>>>
>>>>> v4 -> v5:
>>>>> fix some issues
>>>>>
>>>>> v3 -> v4:
>>>>> fix some issues
>>>>>
>>>>> v2 -> v3:
>>>>> This patches are splited from previous big patch:
>>>>> http://patchwork.ozlabs.org/patch/934673/
>>>>>
>>>>> Tonghao Zhang (4):
>>>>> vhost: lock the vqs one by one
>>>>> net: vhost: replace magic number of lock annotation
>>>>> net: vhost: factor out busy polling logic to vhost_net_busy_poll()
>>>>> net: vhost: add rx busy polling in tx path
>>>>>
>>>>> drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
>>>>> drivers/vhost/vhost.c | 24 ++++-------
>>>>> 2 files changed, 67 insertions(+), 65 deletions(-)
>>>>>
>>>> Hi, any progress on the new version?
>>>>
>>>> I plan to send a new series of packed virtqueue support of vhost. If you
>>>> plan to send it soon, I can wait. Otherwise, I will send my series.
>>> I rebase the codes. and find there is no improvement anymore, the
>>> patches of makita may solve the problem. jason you may send your
>>> patches, and I will do some research on busypoll.
>> I see. Maybe you can try some bi-directional traffic.
>>
>> Btw, lots of optimizations could be done for busy polling. E.g integrating
>> with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
>> to work or propose new ideas.
>>
>> Thanks
> It seems clear we do need adaptive polling.
Yes.
> The difficulty with NAPI
> polling is it can't access guest memory easily. But maybe
> get_user_pages on the polled memory+NAPI polling can work.
You mean something like zerocopy? Looks like we can do busy polling
without it. I mean something like
https://patchwork.kernel.org/patch/8707511/.
Thanks
>
>>>> Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Wei Wang @ 2018-07-12 2:52 UTC (permalink / raw)
To: Linus Torvalds
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, nilal, liliang.opensource,
Linux Kernel Mailing List, virtualization, linux-mm,
Paolo Bonzini, Andrew Morton, Michal Hocko
In-Reply-To: <CA+55aFxyv=EUAJFUSio=k+pm3ddteojshP7Radjia5ZRgm53zQ@mail.gmail.com>
On 07/12/2018 10:30 AM, Linus Torvalds wrote:
> On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
>> Would it be better to remove __GFP_THISNODE? We actually want to get all
>> the guest free pages (from all the nodes).
> Maybe. Or maybe it would be better to have the memory balloon logic be
> per-node? Maybe you don't want to remove too much memory from one
> node? I think it's one of those "play with it" things.
>
> I don't think that's the big issue, actually. I think the real issue
> is how to react quickly and gracefully to "oops, I'm trying to give
> memory away, but now the guest wants it back" while you're in the
> middle of trying to create that 2TB list of pages.
OK. virtio-balloon has already registered an oom notifier
(virtballoon_oom_notify). I plan to add some control there. If oom happens,
- stop the page allocation;
- immediately give back the allocated pages to mm.
Best,
Wei
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Linus Torvalds @ 2018-07-12 2:30 UTC (permalink / raw)
To: wei.w.wang
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, nilal, liliang.opensource,
Linux Kernel Mailing List, virtualization, linux-mm,
Paolo Bonzini, Andrew Morton, Michal Hocko
In-Reply-To: <5B46BB46.2080802@intel.com>
On Wed, Jul 11, 2018 at 7:17 PM Wei Wang <wei.w.wang@intel.com> wrote:
>
> Would it be better to remove __GFP_THISNODE? We actually want to get all
> the guest free pages (from all the nodes).
Maybe. Or maybe it would be better to have the memory balloon logic be
per-node? Maybe you don't want to remove too much memory from one
node? I think it's one of those "play with it" things.
I don't think that's the big issue, actually. I think the real issue
is how to react quickly and gracefully to "oops, I'm trying to give
memory away, but now the guest wants it back" while you're in the
middle of trying to create that 2TB list of pages.
IOW, I think the real work is in whatever tuning for the righ
tbehavior. But I'm just guessing.
Linus
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Wei Wang @ 2018-07-12 2:21 UTC (permalink / raw)
To: Linus Torvalds, Michal Hocko
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal
In-Reply-To: <CA+55aFwku2tDH4+rfaC67xc4-cEwSrXgnQaci=e2id5ZCRE9JQ@mail.gmail.com>
On 07/12/2018 12:23 AM, Linus Torvalds wrote:
> On Wed, Jul 11, 2018 at 2:21 AM Michal Hocko <mhocko@kernel.org> wrote:
>> We already have an interface for that. alloc_pages(GFP_NOWAIT, MAX_ORDER -1).
>> So why do we need any array based interface?
> That was actually my original argument in the original thread - that
> the only new interface people might want is one that just tells how
> many of those MAX_ORDER-1 pages there are.
>
> See the thread in v33 with the subject
>
> "[PATCH v33 1/4] mm: add a function to get free page blocks"
>
> and look for me suggesting just using
>
> #define GFP_MINFLAGS (__GFP_NORETRY | __GFP_NOWARN |
> __GFP_THISNODE | __GFP_NOMEMALLOC)
Would it be better to remove __GFP_THISNODE? We actually want to get all
the guest free pages (from all the nodes).
Best,
Wei
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michael S. Tsirkin @ 2018-07-11 19:36 UTC (permalink / raw)
To: Michal Hocko
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
liliang.opensource, Linux Kernel Mailing List, virtualization,
linux-mm, Paolo Bonzini, Andrew Morton, nilal, Linus Torvalds
In-Reply-To: <20180711110949.GJ20050@dhcp22.suse.cz>
On Wed, Jul 11, 2018 at 01:09:49PM +0200, Michal Hocko wrote:
> But let me note that I am not really convinced how this (or previous)
> approach will really work in most workloads. We tend to cache heavily so
> there is rarely any memory free.
It might be that it's worth flushing the cache when VM is
migrating. Or maybe we should implement virtio-tmem or add
transcendent memory support to the balloon.
--
MST
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Linus Torvalds @ 2018-07-11 16:23 UTC (permalink / raw)
To: Michal Hocko
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal
In-Reply-To: <20180711092152.GE20050@dhcp22.suse.cz>
On Wed, Jul 11, 2018 at 2:21 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> We already have an interface for that. alloc_pages(GFP_NOWAIT, MAX_ORDER -1).
> So why do we need any array based interface?
That was actually my original argument in the original thread - that
the only new interface people might want is one that just tells how
many of those MAX_ORDER-1 pages there are.
See the thread in v33 with the subject
"[PATCH v33 1/4] mm: add a function to get free page blocks"
and look for me suggesting just using
#define GFP_MINFLAGS (__GFP_NORETRY | __GFP_NOWARN |
__GFP_THISNODE | __GFP_NOMEMALLOC)
struct page *page = alloc_pages(GFP_MINFLAGS, MAX_ORDER-1);
for this all.
But I could also see an argument for "allocate N pages of size
MAX_ORDER-1", with some small N, simply because I can see the
advantage of not taking and releasing the locking and looking up the
zone individually N times.
If you want to get gigabytes of memory (or terabytes), doing it in
bigger chunks than one single maximum-sized page sounds fairly
reasonable.
I just don't think that "thousands of pages" is reasonable. But "tens
of max-sized pages" sounds fair enough to me, and it would certainly
not be a pain for the VM.
So I'm open to new interfaces. I just want those new interfaces to
make sense, and be low latency and simple for the VM to do. I'm
objecting to the incredibly baroque and heavy-weight one that can
return near-infinite amounts of memory.
The real advantage of jjuist the existing "alloc_pages()" model is
that I think the ballooning people can use that to *test* things out.
If it turns out that taking and releasing the VM locks is a big cost,
we can see if a batch interface that allows you to get tens of pages
at the same time is worth it.
So yes, I'd suggest starting with just the existing alloc_pages. Maybe
it's not enough, but it should be good enough for testing.
Linus
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michal Hocko @ 2018-07-11 14:38 UTC (permalink / raw)
To: Wang, Wei W
Cc: yang.zhang.wz@gmail.com, virtio-dev@lists.oasis-open.org,
Rik van Riel, quan.xu0@gmail.com, KVM list, Michael S. Tsirkin,
liliang.opensource@gmail.com, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton,
nilal@redhat.com, Linus Torvalds
In-Reply-To: <286AC319A985734F985F78AFA26841F7396EEFD8@SHSMSX101.ccr.corp.intel.com>
On Wed 11-07-18 13:55:15, Wang, Wei W wrote:
> On Wednesday, July 11, 2018 7:10 PM, Michal Hocko wrote:
> > On Wed 11-07-18 18:52:45, Wei Wang wrote:
> > > On 07/11/2018 05:21 PM, Michal Hocko wrote:
> > > > On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> > > > [...]
> > > > > That was what I tried to encourage with actually removing the
> > > > > pages form the page list. That would be an _incremental_
> > > > > interface. You can remove MAX_ORDER-1 pages one by one (or a
> > > > > hundred at a time), and mark them free for ballooning that way.
> > > > > And if you still feel you have tons of free memory, just continue
> > removing more pages from the free list.
> > > > We already have an interface for that. alloc_pages(GFP_NOWAIT,
> > MAX_ORDER -1).
> > > > So why do we need any array based interface?
> > >
> > > Yes, I'm trying to get free pages directly via alloc_pages, so there
> > > will be no new mm APIs.
> >
> > OK. The above was just a rough example. In fact you would need a more
> > complex gfp mask. I assume you only want to balloon only memory directly
> > usable by the kernel so it will be
> > (GFP_KERNEL | __GFP_NOWARN) & ~__GFP_RECLAIM
>
> Sounds good to me, thanks.
>
> >
> > > I plan to let free page allocation stop when the remaining system free
> > > memory becomes close to min_free_kbytes (prevent swapping).
> >
> > ~__GFP_RECLAIM will make sure you are allocate as long as there is any
> > memory without reclaim. It will not even poke the kswapd to do the
> > background work. So I do not think you would need much more than that.
>
> "close to min_free_kbytes" - I meant when doing the allocations, we
> intentionally reserve some small amount of memory, e.g. 2 free page
> blocks of "MAX_ORDER - 1". So when other applications happen to do
> some allocation, they may easily get some from the reserved memory
> left on the free list. Without that reserved memory, other allocation
> may cause the system free memory below the WMARK[MIN], and kswapd
> would start to do swapping. This is actually just a small optimization
> to reduce the probability of causing swapping (nice to have, but not
> mandatary because we will allocate free page blocks one by one).
I really have hard time to follow you here. Nothing outside of the core
MM proper should play with watermarks.
> > But let me note that I am not really convinced how this (or previous)
> > approach will really work in most workloads. We tend to cache heavily so
> > there is rarely any memory free.
>
> With less free memory, the improvement becomes less, but should be
> nicer than no optimization. For example, the Linux build workload
> would cause 4~5 GB (out of 8GB) memory to be used as page cache at the
> final stage, there is still ~44% live migration time reduction.
But most systems will stay somewhere around the high watermark if there
is any page cache activity. Especially after a longer uptime.
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* RE: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Wang, Wei W @ 2018-07-11 13:55 UTC (permalink / raw)
To: Michal Hocko
Cc: yang.zhang.wz@gmail.com, virtio-dev@lists.oasis-open.org,
Rik van Riel, quan.xu0@gmail.com, KVM list, Michael S. Tsirkin,
liliang.opensource@gmail.com, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton,
nilal@redhat.com, Linus Torvalds
In-Reply-To: <20180711110949.GJ20050@dhcp22.suse.cz>
On Wednesday, July 11, 2018 7:10 PM, Michal Hocko wrote:
> On Wed 11-07-18 18:52:45, Wei Wang wrote:
> > On 07/11/2018 05:21 PM, Michal Hocko wrote:
> > > On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> > > [...]
> > > > That was what I tried to encourage with actually removing the
> > > > pages form the page list. That would be an _incremental_
> > > > interface. You can remove MAX_ORDER-1 pages one by one (or a
> > > > hundred at a time), and mark them free for ballooning that way.
> > > > And if you still feel you have tons of free memory, just continue
> removing more pages from the free list.
> > > We already have an interface for that. alloc_pages(GFP_NOWAIT,
> MAX_ORDER -1).
> > > So why do we need any array based interface?
> >
> > Yes, I'm trying to get free pages directly via alloc_pages, so there
> > will be no new mm APIs.
>
> OK. The above was just a rough example. In fact you would need a more
> complex gfp mask. I assume you only want to balloon only memory directly
> usable by the kernel so it will be
> (GFP_KERNEL | __GFP_NOWARN) & ~__GFP_RECLAIM
Sounds good to me, thanks.
>
> > I plan to let free page allocation stop when the remaining system free
> > memory becomes close to min_free_kbytes (prevent swapping).
>
> ~__GFP_RECLAIM will make sure you are allocate as long as there is any
> memory without reclaim. It will not even poke the kswapd to do the
> background work. So I do not think you would need much more than that.
"close to min_free_kbytes" - I meant when doing the allocations, we intentionally reserve some small amount of memory, e.g. 2 free page blocks of "MAX_ORDER - 1". So when other applications happen to do some allocation, they may easily get some from the reserved memory left on the free list. Without that reserved memory, other allocation may cause the system free memory below the WMARK[MIN], and kswapd would start to do swapping. This is actually just a small optimization to reduce the probability of causing swapping (nice to have, but not mandatary because we will allocate free page blocks one by one).
> But let me note that I am not really convinced how this (or previous)
> approach will really work in most workloads. We tend to cache heavily so
> there is rarely any memory free.
With less free memory, the improvement becomes less, but should be nicer than no optimization. For example, the Linux build workload would cause 4~5 GB (out of 8GB) memory to be used as page cache at the final stage, there is still ~44% live migration time reduction.
Since we have many cloud customers interested in this feature, I think we can let them test the usefulness.
Best,
Wei
^ permalink raw reply
* Re: [PATCH net-next v5 0/4] net: vhost: improve performance when enable busyloop
From: Michael S. Tsirkin @ 2018-07-11 11:59 UTC (permalink / raw)
To: Jason Wang; +Cc: Linux Kernel Network Developers, virtualization
In-Reply-To: <4901593f-1d9e-2c47-b963-5fc1f48a1bfe@redhat.com>
On Wed, Jul 11, 2018 at 01:12:59PM +0800, Jason Wang wrote:
>
>
> On 2018年07月11日 11:49, Tonghao Zhang wrote:
> > On Wed, Jul 11, 2018 at 10:56 AM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > >
> > > On 2018年07月04日 12:31, xiangxia.m.yue@gmail.com wrote:
> > > > From: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > >
> > > > This patches improve the guest receive and transmit performance.
> > > > On the handle_tx side, we poll the sock receive queue at the same time.
> > > > handle_rx do that in the same way.
> > > >
> > > > For more performance report, see patch 4.
> > > >
> > > > v4 -> v5:
> > > > fix some issues
> > > >
> > > > v3 -> v4:
> > > > fix some issues
> > > >
> > > > v2 -> v3:
> > > > This patches are splited from previous big patch:
> > > > http://patchwork.ozlabs.org/patch/934673/
> > > >
> > > > Tonghao Zhang (4):
> > > > vhost: lock the vqs one by one
> > > > net: vhost: replace magic number of lock annotation
> > > > net: vhost: factor out busy polling logic to vhost_net_busy_poll()
> > > > net: vhost: add rx busy polling in tx path
> > > >
> > > > drivers/vhost/net.c | 108 ++++++++++++++++++++++++++++----------------------
> > > > drivers/vhost/vhost.c | 24 ++++-------
> > > > 2 files changed, 67 insertions(+), 65 deletions(-)
> > > >
> > > Hi, any progress on the new version?
> > >
> > > I plan to send a new series of packed virtqueue support of vhost. If you
> > > plan to send it soon, I can wait. Otherwise, I will send my series.
> > I rebase the codes. and find there is no improvement anymore, the
> > patches of makita may solve the problem. jason you may send your
> > patches, and I will do some research on busypoll.
>
> I see. Maybe you can try some bi-directional traffic.
>
> Btw, lots of optimizations could be done for busy polling. E.g integrating
> with host NAPI busy polling or a 100% busy polling vhost_net. You're welcome
> to work or propose new ideas.
>
> Thanks
It seems clear we do need adaptive polling. The difficulty with NAPI
polling is it can't access guest memory easily. But maybe
get_user_pages on the polled memory+NAPI polling can work.
> >
> > > Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v35 1/5] mm: support to get hints of free page blocks
From: Michal Hocko @ 2018-07-11 11:09 UTC (permalink / raw)
To: Wei Wang
Cc: yang.zhang.wz, virtio-dev, Rik van Riel, quan.xu0, KVM list,
Michael S. Tsirkin, liliang.opensource, Linux Kernel Mailing List,
virtualization, linux-mm, Paolo Bonzini, Andrew Morton, nilal,
Linus Torvalds
In-Reply-To: <5B45E17D.2090205@intel.com>
On Wed 11-07-18 18:52:45, Wei Wang wrote:
> On 07/11/2018 05:21 PM, Michal Hocko wrote:
> > On Tue 10-07-18 18:44:34, Linus Torvalds wrote:
> > [...]
> > > That was what I tried to encourage with actually removing the pages
> > > form the page list. That would be an _incremental_ interface. You can
> > > remove MAX_ORDER-1 pages one by one (or a hundred at a time), and mark
> > > them free for ballooning that way. And if you still feel you have tons
> > > of free memory, just continue removing more pages from the free list.
> > We already have an interface for that. alloc_pages(GFP_NOWAIT, MAX_ORDER -1).
> > So why do we need any array based interface?
>
> Yes, I'm trying to get free pages directly via alloc_pages, so there will be
> no new mm APIs.
OK. The above was just a rough example. In fact you would need a more
complex gfp mask. I assume you only want to balloon only memory directly
usable by the kernel so it will be
(GFP_KERNEL | __GFP_NOWARN) & ~__GFP_RECLAIM
> I plan to let free page allocation stop when the remaining system free
> memory becomes close to min_free_kbytes (prevent swapping).
~__GFP_RECLAIM will make sure you are allocate as long as there is any
memory without reclaim. It will not even poke the kswapd to do the
background work. So I do not think you would need much more than that.
But let me note that I am not really convinced how this (or previous)
approach will really work in most workloads. We tend to cache heavily so
there is rarely any memory free.
--
Michal Hocko
SUSE Labs
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox