* Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
From: Jason Wang @ 2019-01-07 6:50 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Andrea Parri, linux-doc, Peter Zijlstra, Akira Yokosawa,
Will Deacon, virtualization, David Howells, linux-arch,
Jonathan Corbet, linux-sparse, Alan Stern, Matt Turner,
Paul E. McKenney, Daniel Lustig, Arnd Bergmann, Boqun Feng,
Nicholas Piggin, Ivan Kokshaysky, Luc Maranget, Richard Henderson,
Jade Alglave, netdev, linux-kernel, linux-alpha
In-Reply-To: <20190106231756-mutt-send-email-mst@kernel.org>
On 2019/1/7 下午12:23, Michael S. Tsirkin wrote:
> On Mon, Jan 07, 2019 at 11:58:23AM +0800, Jason Wang wrote:
>> On 2019/1/3 上午4:57, Michael S. Tsirkin wrote:
>>> It's not uncommon to have two access two unrelated memory locations in a
>>> specific order. At the moment one has to use a memory barrier for this.
>>>
>>> However, if the first access was a read and the second used an address
>>> depending on the first one we would have a data dependency and no
>>> barrier would be necessary.
>>>
>>> This adds a new interface: dependent_ptr_mb which does exactly this: it
>>> returns a pointer with a data dependency on the supplied value.
>>>
>>> Signed-off-by: Michael S. Tsirkin<mst@redhat.com>
>>> ---
>>> Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
>>> arch/alpha/include/asm/barrier.h | 1 +
>>> include/asm-generic/barrier.h | 18 ++++++++++++++++++
>>> include/linux/compiler.h | 4 ++++
>>> 4 files changed, 43 insertions(+)
>>>
>>> diff --git a/Documentation/memory-barriers.txt b/Documentation/memory-barriers.txt
>>> index c1d913944ad8..9dbaa2e1dbf6 100644
>>> --- a/Documentation/memory-barriers.txt
>>> +++ b/Documentation/memory-barriers.txt
>>> @@ -691,6 +691,18 @@ case what's actually required is:
>>> p = READ_ONCE(b);
>>> }
>>> +Alternatively, a control dependency can be converted to a data dependency,
>>> +e.g.:
>>> +
>>> + q = READ_ONCE(a);
>>> + if (q) {
>>> + b = dependent_ptr_mb(b, q);
>>> + p = READ_ONCE(b);
>>> + }
>>> +
>>> +Note how the result of dependent_ptr_mb must be used with the following
>>> +accesses in order to have an effect.
>>> +
>>> However, stores are not speculated. This means that ordering -is- provided
>>> for load-store control dependencies, as in the following example:
>>> @@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force
>>> the compiler to actually emit code for a given load, it does not force
>>> the compiler to use the results.
>>> +Converting to a data dependency helps with this too:
>>> +
>>> + q = READ_ONCE(a);
>>> + b = dependent_ptr_mb(b, q);
>>> + WRITE_ONCE(b, 1);
>>> +
>>> In addition, control dependencies apply only to the then-clause and
>>> else-clause of the if-statement in question. In particular, it does
>>> not necessarily apply to code following the if-statement:
>>> @@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity"
>>> for more information.
>>> +
>>> +
>>> In summary:
>>> (*) Control dependencies can order prior loads against later stores.
>>> diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
>>> index 92ec486a4f9e..b4934e8c551b 100644
>>> --- a/arch/alpha/include/asm/barrier.h
>>> +++ b/arch/alpha/include/asm/barrier.h
>>> @@ -59,6 +59,7 @@
>>> * as Alpha, "y" could be set to 3 and "x" to 0. Use rmb()
>>> * in cases like this where there are no data dependencies.
>>> */
>>> +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
>>> #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
>>> #ifdef CONFIG_SMP
>>> diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
>>> index 2cafdbb9ae4c..fa2e2ef72b68 100644
>>> --- a/include/asm-generic/barrier.h
>>> +++ b/include/asm-generic/barrier.h
>>> @@ -70,6 +70,24 @@
>>> #define __smp_read_barrier_depends() read_barrier_depends()
>>> #endif
>>> +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
>>> + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
>>> +
>>> +#define dependent_ptr_mb(ptr, val) ({ \
>>> + long dependent_ptr_mb_val = (long)(val); \
>>> + long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \
>>> + \
>>> + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \
>>> + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \
>>> + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \
>>> +})
>>> +
>>> +#else
>>> +
>>> +#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })
>> So for the example of patch 4, we'd better fall back to rmb() or need a
>> dependent_ptr_rmb()?
>>
>> Thanks
> You mean for strongly ordered architectures like Intel?
> Yes, maybe it makes sense to have dependent_ptr_smp_rmb,
> dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.
>
> mb variant is unused right now so I'll remove it.
>
>
Yes.
Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Jason Wang @ 2019-01-07 6:50 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: kvm, netdev, linux-kernel, virtualization, Dan Williams, davem
In-Reply-To: <20190106230224-mutt-send-email-mst@kernel.org>
On 2019/1/7 下午12:17, Michael S. Tsirkin wrote:
> On Mon, Jan 07, 2019 at 11:53:41AM +0800, Jason Wang wrote:
>> On 2019/1/7 上午11:28, Michael S. Tsirkin wrote:
>>> On Mon, Jan 07, 2019 at 10:19:03AM +0800, Jason Wang wrote:
>>>> On 2019/1/3 上午4:47, Michael S. Tsirkin wrote:
>>>>> On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
>>>>>> This series tries to access virtqueue metadata through kernel virtual
>>>>>> address instead of copy_user() friends since they had too much
>>>>>> overheads like checks, spec barriers or even hardware feature
>>>>>> toggling.
>>>>> Will review, thanks!
>>>>> One questions that comes to mind is whether it's all about bypassing
>>>>> stac/clac. Could you please include a performance comparison with
>>>>> nosmap?
>>>>>
>>>> On machine without SMAP (Sandy Bridge):
>>>>
>>>> Before: 4.8Mpps
>>>>
>>>> After: 5.2Mpps
>>> OK so would you say it's really unsafe versus safe accesses?
>>> Or would you say it's just a better written code?
>>
>> It's the effect of removing speculation barrier.
>
> You mean __uaccess_begin_nospec introduced by
> commit 304ec1b050310548db33063e567123fae8fd0301
> ?
Yes.
>
> So fundamentally we do access_ok checks when supplying
> the memory table to the kernel thread, and we should
> do the spec barrier there.
>
> Then we can just create and use a variant of uaccess macros that does
> not include the barrier?
The unsafe ones?
>
> Or, how about moving the barrier into access_ok?
> This way repeated accesses with a single access_ok get a bit faster.
> CC Dan Williams on this idea.
The problem is, e.g for vhost control path. During mem table validation,
we don't even want to access them there. So the spec barrier is not needed.
>
>
>>>> On machine with SMAP (Broadwell):
>>>>
>>>> Before: 5.0Mpps
>>>>
>>>> After: 6.1Mpps
>>>>
>>>> No smap: 7.5Mpps
>>>>
>>>>
>>>> Thanks
>>> no smap being before or after?
>>>
>> Let me clarify:
>>
>>
>> Before (SMAP on): 5.0Mpps
>>
>> Before (SMAP off): 7.5Mpps
>>
>> After (SMAP on): 6.1Mpps
>>
>>
>> Thanks
> How about after + smap off?
After (SMAP off): 8.0Mpps
>
> And maybe we want a module option just for the vhost thread to keep smap
> off generally since almost all it does is copy stuff from userspace into
> kernel anyway. Because what above numbers should is that we really
> really want a solution that isn't limited to just meta-data access,
> and I really do not see how any such solution can not also be
> used to make meta-data access fast.
As we've discussed in another thread of previous version. This requires
lots of changes, the main issues is SMAP state was not saved/restored on
explicit schedule(). Even if it did, since vhost will call lots of
net/block codes, any kind of uaccess in those codes needs understand
this special request from vhost e.g you provably need to invent a new
kinds of iov iterator that does not touch SMAP at all. And I'm not sure
this is the only thing we need to deal with.
So I still prefer to:
1) speedup the metadata access through vmap + MMU notifier
2) speedup the datacopy with batched copy (unsafe ones or other new
interfaces)
Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 1/2] virtio-net: bql support
From: Jason Wang @ 2019-01-07 6:31 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: netdev, linux-kernel, virtualization, maxime.coquelin, wexu,
David S. Miller
In-Reply-To: <20190106225951-mutt-send-email-mst@kernel.org>
On 2019/1/7 下午12:01, Michael S. Tsirkin wrote:
> On Mon, Jan 07, 2019 at 11:51:55AM +0800, Jason Wang wrote:
>> On 2019/1/7 上午11:17, Michael S. Tsirkin wrote:
>>> On Mon, Jan 07, 2019 at 10:14:37AM +0800, Jason Wang wrote:
>>>> On 2019/1/2 下午9:59, Michael S. Tsirkin wrote:
>>>>> On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote:
>>>>>> On 2018/12/31 上午2:45, Michael S. Tsirkin wrote:
>>>>>>> On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote:
>>>>>>>> On 2018/12/26 下午11:19, Michael S. Tsirkin wrote:
>>>>>>>>> On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote:
>>>>>>>>>> On 2018/12/6 上午6:54, Michael S. Tsirkin wrote:
>>>>>>>>>>> When use_napi is set, let's enable BQLs. Note: some of the issues are
>>>>>>>>>>> similar to wifi. It's worth considering whether something similar to
>>>>>>>>>>> commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be
>>>>>>>>>>> benefitial.
>>>>>>>>>> I've played a similar patch several days before. The tricky part is the mode
>>>>>>>>>> switching between napi and no napi. We should make sure when the packet is
>>>>>>>>>> sent and trakced by BQL, it should be consumed by BQL as well. I did it by
>>>>>>>>>> tracking it through skb->cb. And deal with the freeze by reset the BQL
>>>>>>>>>> status. Patch attached.
>>>>>>>>>>
>>>>>>>>>> But when testing with vhost-net, I don't very a stable performance,
>>>>>>>>> So how about increasing TSQ pacing shift then?
>>>>>>>> I can test this. But changing default TCP value is much more than a
>>>>>>>> virtio-net specific thing.
>>>>>>> Well same logic as wifi applies. Unpredictable latencies related
>>>>>>> to radio in one case, to host scheduler in the other.
>>>>>>>
>>>>>>>>>> it was
>>>>>>>>>> probably because we batch the used ring updating so tx interrupt may come
>>>>>>>>>> randomly. We probably need to implement time bounded coalescing mechanism
>>>>>>>>>> which could be configured from userspace.
>>>>>>>>> I don't think it's reasonable to expect userspace to be that smart ...
>>>>>>>>> Why do we need time bounded? used ring is always updated when ring
>>>>>>>>> becomes empty.
>>>>>>>> We don't add used when means BQL may not see the consumed packet in time.
>>>>>>>> And the delay varies based on the workload since we count packets not bytes
>>>>>>>> or time before doing the batched updating.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>> Sorry I still don't get it.
>>>>>>> When nothing is outstanding then we do update the used.
>>>>>>> So if BQL stops userspace from sending packets then
>>>>>>> we get an interrupt and packets start flowing again.
>>>>>> Yes, but how about the cases of multiple flows. That's where I see unstable
>>>>>> results.
>>>>>>
>>>>>>
>>>>>>> It might be suboptimal, we might need to tune it but I doubt running
>>>>>>> timers is a solution, timer interrupts cause VM exits.
>>>>>> Probably not a timer but a time counter (or event byte counter) in vhost to
>>>>>> add used and signal guest if it exceeds a value instead of waiting the
>>>>>> number of packets.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>> Well we already have VHOST_NET_WEIGHT - is it too big then?
>>>> I'm not sure, it might be too big.
>>>>
>>>>
>>>>> And maybe we should expose the "MORE" flag in the descriptor -
>>>>> do you think that will help?
>>>>>
>>>> I don't know. But how a "more" flag can help here?
>>>>
>>>> Thanks
>>> It sounds like we should be a bit more aggressive in updating used ring.
>>> But if we just do it naively we will harm performance for sure as that
>>> is how we are doing batching right now.
>>
>> I agree but the problem is to balance the PPS and throughput. More batching
>> helps for PPS but may damage TCP throughput.
> That is what more flag is supposed to be I think - it is only set if
> there's a socket that actually needs the skb freed in order to go on.
I'm not quite sure I get, but is this something similar to what you want?
https://lists.linuxfoundation.org/pipermail/virtualization/2014-October/027667.html
Which enables tx interrupt for TCP packets, and you want to add used
more aggressively for those sockets?
Thanks
>>> Instead we could make guest
>>> control batching using the more flag - if that's not set we write out
>>> the used ring.
>>
>> It's under the control of guest, so I'm afraid we still need some more guard
>> (e.g time/bytes counters) on host.
>>
>> Thanks
> Point is if guest does not care about the skb being freed, then there is no
> rush host side to mark buffer used.
>
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
From: Michael S. Tsirkin @ 2019-01-07 4:23 UTC (permalink / raw)
To: Jason Wang
Cc: Andrea Parri, linux-doc, Peter Zijlstra, Akira Yokosawa,
Will Deacon, virtualization, David Howells, linux-arch,
Jonathan Corbet, linux-sparse, Alan Stern, Matt Turner,
Paul E. McKenney, Daniel Lustig, Arnd Bergmann, Boqun Feng,
Nicholas Piggin, Ivan Kokshaysky, Luc Maranget, Richard Henderson,
Jade Alglave, netdev, linux-kernel, linux-alpha
In-Reply-To: <86023cbe-d1ae-a0d6-7b75-26556f1a0c1f@redhat.com>
On Mon, Jan 07, 2019 at 11:58:23AM +0800, Jason Wang wrote:
>
> On 2019/1/3 上午4:57, Michael S. Tsirkin wrote:
> > It's not uncommon to have two access two unrelated memory locations in a
> > specific order. At the moment one has to use a memory barrier for this.
> >
> > However, if the first access was a read and the second used an address
> > depending on the first one we would have a data dependency and no
> > barrier would be necessary.
> >
> > This adds a new interface: dependent_ptr_mb which does exactly this: it
> > returns a pointer with a data dependency on the supplied value.
> >
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> > Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
> > arch/alpha/include/asm/barrier.h | 1 +
> > include/asm-generic/barrier.h | 18 ++++++++++++++++++
> > include/linux/compiler.h | 4 ++++
> > 4 files changed, 43 insertions(+)
> >
> > diff --git a/Documentation/memory-barriers.txt b/Documentation/memory-barriers.txt
> > index c1d913944ad8..9dbaa2e1dbf6 100644
> > --- a/Documentation/memory-barriers.txt
> > +++ b/Documentation/memory-barriers.txt
> > @@ -691,6 +691,18 @@ case what's actually required is:
> > p = READ_ONCE(b);
> > }
> > +Alternatively, a control dependency can be converted to a data dependency,
> > +e.g.:
> > +
> > + q = READ_ONCE(a);
> > + if (q) {
> > + b = dependent_ptr_mb(b, q);
> > + p = READ_ONCE(b);
> > + }
> > +
> > +Note how the result of dependent_ptr_mb must be used with the following
> > +accesses in order to have an effect.
> > +
> > However, stores are not speculated. This means that ordering -is- provided
> > for load-store control dependencies, as in the following example:
> > @@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force
> > the compiler to actually emit code for a given load, it does not force
> > the compiler to use the results.
> > +Converting to a data dependency helps with this too:
> > +
> > + q = READ_ONCE(a);
> > + b = dependent_ptr_mb(b, q);
> > + WRITE_ONCE(b, 1);
> > +
> > In addition, control dependencies apply only to the then-clause and
> > else-clause of the if-statement in question. In particular, it does
> > not necessarily apply to code following the if-statement:
> > @@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity"
> > for more information.
> > +
> > +
> > In summary:
> > (*) Control dependencies can order prior loads against later stores.
> > diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
> > index 92ec486a4f9e..b4934e8c551b 100644
> > --- a/arch/alpha/include/asm/barrier.h
> > +++ b/arch/alpha/include/asm/barrier.h
> > @@ -59,6 +59,7 @@
> > * as Alpha, "y" could be set to 3 and "x" to 0. Use rmb()
> > * in cases like this where there are no data dependencies.
> > */
> > +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
> > #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
> > #ifdef CONFIG_SMP
> > diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
> > index 2cafdbb9ae4c..fa2e2ef72b68 100644
> > --- a/include/asm-generic/barrier.h
> > +++ b/include/asm-generic/barrier.h
> > @@ -70,6 +70,24 @@
> > #define __smp_read_barrier_depends() read_barrier_depends()
> > #endif
> > +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
> > + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
> > +
> > +#define dependent_ptr_mb(ptr, val) ({ \
> > + long dependent_ptr_mb_val = (long)(val); \
> > + long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \
> > + \
> > + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \
> > + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \
> > + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \
> > +})
> > +
> > +#else
> > +
> > +#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })
>
>
> So for the example of patch 4, we'd better fall back to rmb() or need a
> dependent_ptr_rmb()?
>
> Thanks
You mean for strongly ordered architectures like Intel?
Yes, maybe it makes sense to have dependent_ptr_smp_rmb,
dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.
mb variant is unused right now so I'll remove it.
>
> > +
> > +#endif
> > +
> > #ifdef CONFIG_SMP
> > #ifndef smp_mb
> > diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> > index 6601d39e8c48..f599c30f1b28 100644
> > --- a/include/linux/compiler.h
> > +++ b/include/linux/compiler.h
> > @@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
> > #endif
> > #ifndef OPTIMIZER_HIDE_VAR
> > +
> > /* Make the optimizer believe the variable can be manipulated arbitrarily. */
> > #define OPTIMIZER_HIDE_VAR(var) \
> > __asm__ ("" : "=rm" (var) : "0" (var))
> > +
> > +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
> > +
> > #endif
> > /* Not-quite-unique ID. */
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Michael S. Tsirkin @ 2019-01-07 4:17 UTC (permalink / raw)
To: Jason Wang; +Cc: kvm, netdev, linux-kernel, virtualization, Dan Williams, davem
In-Reply-To: <c15a17b3-b604-a105-af44-a7e6dfd79801@redhat.com>
On Mon, Jan 07, 2019 at 11:53:41AM +0800, Jason Wang wrote:
>
> On 2019/1/7 上午11:28, Michael S. Tsirkin wrote:
> > On Mon, Jan 07, 2019 at 10:19:03AM +0800, Jason Wang wrote:
> > > On 2019/1/3 上午4:47, Michael S. Tsirkin wrote:
> > > > On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
> > > > > This series tries to access virtqueue metadata through kernel virtual
> > > > > address instead of copy_user() friends since they had too much
> > > > > overheads like checks, spec barriers or even hardware feature
> > > > > toggling.
> > > > Will review, thanks!
> > > > One questions that comes to mind is whether it's all about bypassing
> > > > stac/clac. Could you please include a performance comparison with
> > > > nosmap?
> > > >
> > > On machine without SMAP (Sandy Bridge):
> > >
> > > Before: 4.8Mpps
> > >
> > > After: 5.2Mpps
> > OK so would you say it's really unsafe versus safe accesses?
> > Or would you say it's just a better written code?
>
>
> It's the effect of removing speculation barrier.
You mean __uaccess_begin_nospec introduced by
commit 304ec1b050310548db33063e567123fae8fd0301
?
So fundamentally we do access_ok checks when supplying
the memory table to the kernel thread, and we should
do the spec barrier there.
Then we can just create and use a variant of uaccess macros that does
not include the barrier?
Or, how about moving the barrier into access_ok?
This way repeated accesses with a single access_ok get a bit faster.
CC Dan Williams on this idea.
>
> >
> > > On machine with SMAP (Broadwell):
> > >
> > > Before: 5.0Mpps
> > >
> > > After: 6.1Mpps
> > >
> > > No smap: 7.5Mpps
> > >
> > >
> > > Thanks
> >
> > no smap being before or after?
> >
>
> Let me clarify:
>
>
> Before (SMAP on): 5.0Mpps
>
> Before (SMAP off): 7.5Mpps
>
> After (SMAP on): 6.1Mpps
>
>
> Thanks
How about after + smap off?
And maybe we want a module option just for the vhost thread to keep smap
off generally since almost all it does is copy stuff from userspace into
kernel anyway. Because what above numbers should is that we really
really want a solution that isn't limited to just meta-data access,
and I really do not see how any such solution can not also be
used to make meta-data access fast.
--
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 1/2] virtio-net: bql support
From: Michael S. Tsirkin @ 2019-01-07 4:01 UTC (permalink / raw)
To: Jason Wang
Cc: netdev, linux-kernel, virtualization, maxime.coquelin, wexu,
David S. Miller
In-Reply-To: <aea2fd16-ec5b-64b5-2095-9a37044223f6@redhat.com>
On Mon, Jan 07, 2019 at 11:51:55AM +0800, Jason Wang wrote:
>
> On 2019/1/7 上午11:17, Michael S. Tsirkin wrote:
> > On Mon, Jan 07, 2019 at 10:14:37AM +0800, Jason Wang wrote:
> > > On 2019/1/2 下午9:59, Michael S. Tsirkin wrote:
> > > > On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote:
> > > > > On 2018/12/31 上午2:45, Michael S. Tsirkin wrote:
> > > > > > On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote:
> > > > > > > On 2018/12/26 下午11:19, Michael S. Tsirkin wrote:
> > > > > > > > On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote:
> > > > > > > > > On 2018/12/6 上午6:54, Michael S. Tsirkin wrote:
> > > > > > > > > > When use_napi is set, let's enable BQLs. Note: some of the issues are
> > > > > > > > > > similar to wifi. It's worth considering whether something similar to
> > > > > > > > > > commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be
> > > > > > > > > > benefitial.
> > > > > > > > > I've played a similar patch several days before. The tricky part is the mode
> > > > > > > > > switching between napi and no napi. We should make sure when the packet is
> > > > > > > > > sent and trakced by BQL, it should be consumed by BQL as well. I did it by
> > > > > > > > > tracking it through skb->cb. And deal with the freeze by reset the BQL
> > > > > > > > > status. Patch attached.
> > > > > > > > >
> > > > > > > > > But when testing with vhost-net, I don't very a stable performance,
> > > > > > > > So how about increasing TSQ pacing shift then?
> > > > > > > I can test this. But changing default TCP value is much more than a
> > > > > > > virtio-net specific thing.
> > > > > > Well same logic as wifi applies. Unpredictable latencies related
> > > > > > to radio in one case, to host scheduler in the other.
> > > > > >
> > > > > > > > > it was
> > > > > > > > > probably because we batch the used ring updating so tx interrupt may come
> > > > > > > > > randomly. We probably need to implement time bounded coalescing mechanism
> > > > > > > > > which could be configured from userspace.
> > > > > > > > I don't think it's reasonable to expect userspace to be that smart ...
> > > > > > > > Why do we need time bounded? used ring is always updated when ring
> > > > > > > > becomes empty.
> > > > > > > We don't add used when means BQL may not see the consumed packet in time.
> > > > > > > And the delay varies based on the workload since we count packets not bytes
> > > > > > > or time before doing the batched updating.
> > > > > > >
> > > > > > > Thanks
> > > > > > Sorry I still don't get it.
> > > > > > When nothing is outstanding then we do update the used.
> > > > > > So if BQL stops userspace from sending packets then
> > > > > > we get an interrupt and packets start flowing again.
> > > > > Yes, but how about the cases of multiple flows. That's where I see unstable
> > > > > results.
> > > > >
> > > > >
> > > > > > It might be suboptimal, we might need to tune it but I doubt running
> > > > > > timers is a solution, timer interrupts cause VM exits.
> > > > > Probably not a timer but a time counter (or event byte counter) in vhost to
> > > > > add used and signal guest if it exceeds a value instead of waiting the
> > > > > number of packets.
> > > > >
> > > > >
> > > > > Thanks
> > > > Well we already have VHOST_NET_WEIGHT - is it too big then?
> > >
> > > I'm not sure, it might be too big.
> > >
> > >
> > > > And maybe we should expose the "MORE" flag in the descriptor -
> > > > do you think that will help?
> > > >
> > > I don't know. But how a "more" flag can help here?
> > >
> > > Thanks
> > It sounds like we should be a bit more aggressive in updating used ring.
> > But if we just do it naively we will harm performance for sure as that
> > is how we are doing batching right now.
>
>
> I agree but the problem is to balance the PPS and throughput. More batching
> helps for PPS but may damage TCP throughput.
That is what more flag is supposed to be I think - it is only set if
there's a socket that actually needs the skb freed in order to go on.
>
> > Instead we could make guest
> > control batching using the more flag - if that's not set we write out
> > the used ring.
>
>
> It's under the control of guest, so I'm afraid we still need some more guard
> (e.g time/bytes counters) on host.
>
> Thanks
Point is if guest does not care about the skb being freed, then there is no
rush host side to mark buffer used.
>
> >
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
From: Jason Wang @ 2019-01-07 3:58 UTC (permalink / raw)
To: Michael S. Tsirkin, linux-kernel
Cc: Andrea Parri, linux-doc, Peter Zijlstra, Akira Yokosawa,
Will Deacon, virtualization, David Howells, linux-arch,
Jonathan Corbet, linux-sparse, Alan Stern, Matt Turner,
Paul E. McKenney, Daniel Lustig, Arnd Bergmann, Boqun Feng,
Nicholas Piggin, Ivan Kokshaysky, Luc Maranget, Richard Henderson,
Jade Alglave, netdev, linux-alpha, Luc Van Oostenryck
In-Reply-To: <20190102205715.14054-4-mst@redhat.com>
On 2019/1/3 上午4:57, Michael S. Tsirkin wrote:
> It's not uncommon to have two access two unrelated memory locations in a
> specific order. At the moment one has to use a memory barrier for this.
>
> However, if the first access was a read and the second used an address
> depending on the first one we would have a data dependency and no
> barrier would be necessary.
>
> This adds a new interface: dependent_ptr_mb which does exactly this: it
> returns a pointer with a data dependency on the supplied value.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
> Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
> arch/alpha/include/asm/barrier.h | 1 +
> include/asm-generic/barrier.h | 18 ++++++++++++++++++
> include/linux/compiler.h | 4 ++++
> 4 files changed, 43 insertions(+)
>
> diff --git a/Documentation/memory-barriers.txt b/Documentation/memory-barriers.txt
> index c1d913944ad8..9dbaa2e1dbf6 100644
> --- a/Documentation/memory-barriers.txt
> +++ b/Documentation/memory-barriers.txt
> @@ -691,6 +691,18 @@ case what's actually required is:
> p = READ_ONCE(b);
> }
>
> +Alternatively, a control dependency can be converted to a data dependency,
> +e.g.:
> +
> + q = READ_ONCE(a);
> + if (q) {
> + b = dependent_ptr_mb(b, q);
> + p = READ_ONCE(b);
> + }
> +
> +Note how the result of dependent_ptr_mb must be used with the following
> +accesses in order to have an effect.
> +
> However, stores are not speculated. This means that ordering -is- provided
> for load-store control dependencies, as in the following example:
>
> @@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force
> the compiler to actually emit code for a given load, it does not force
> the compiler to use the results.
>
> +Converting to a data dependency helps with this too:
> +
> + q = READ_ONCE(a);
> + b = dependent_ptr_mb(b, q);
> + WRITE_ONCE(b, 1);
> +
> In addition, control dependencies apply only to the then-clause and
> else-clause of the if-statement in question. In particular, it does
> not necessarily apply to code following the if-statement:
> @@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity"
> for more information.
>
>
> +
> +
> In summary:
>
> (*) Control dependencies can order prior loads against later stores.
> diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
> index 92ec486a4f9e..b4934e8c551b 100644
> --- a/arch/alpha/include/asm/barrier.h
> +++ b/arch/alpha/include/asm/barrier.h
> @@ -59,6 +59,7 @@
> * as Alpha, "y" could be set to 3 and "x" to 0. Use rmb()
> * in cases like this where there are no data dependencies.
> */
> +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
> #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
>
> #ifdef CONFIG_SMP
> diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
> index 2cafdbb9ae4c..fa2e2ef72b68 100644
> --- a/include/asm-generic/barrier.h
> +++ b/include/asm-generic/barrier.h
> @@ -70,6 +70,24 @@
> #define __smp_read_barrier_depends() read_barrier_depends()
> #endif
>
> +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
> + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
> +
> +#define dependent_ptr_mb(ptr, val) ({ \
> + long dependent_ptr_mb_val = (long)(val); \
> + long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \
> + \
> + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \
> + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \
> + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \
> +})
> +
> +#else
> +
> +#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })
So for the example of patch 4, we'd better fall back to rmb() or need a
dependent_ptr_rmb()?
Thanks
> +
> +#endif
> +
> #ifdef CONFIG_SMP
>
> #ifndef smp_mb
> diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> index 6601d39e8c48..f599c30f1b28 100644
> --- a/include/linux/compiler.h
> +++ b/include/linux/compiler.h
> @@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
> #endif
>
> #ifndef OPTIMIZER_HIDE_VAR
> +
> /* Make the optimizer believe the variable can be manipulated arbitrarily. */
> #define OPTIMIZER_HIDE_VAR(var) \
> __asm__ ("" : "=rm" (var) : "0" (var))
> +
> +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
> +
> #endif
>
> /* Not-quite-unique ID. */
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Jason Wang @ 2019-01-07 3:53 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <20190106221832-mutt-send-email-mst@kernel.org>
On 2019/1/7 上午11:28, Michael S. Tsirkin wrote:
> On Mon, Jan 07, 2019 at 10:19:03AM +0800, Jason Wang wrote:
>> On 2019/1/3 上午4:47, Michael S. Tsirkin wrote:
>>> On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
>>>> This series tries to access virtqueue metadata through kernel virtual
>>>> address instead of copy_user() friends since they had too much
>>>> overheads like checks, spec barriers or even hardware feature
>>>> toggling.
>>> Will review, thanks!
>>> One questions that comes to mind is whether it's all about bypassing
>>> stac/clac. Could you please include a performance comparison with
>>> nosmap?
>>>
>> On machine without SMAP (Sandy Bridge):
>>
>> Before: 4.8Mpps
>>
>> After: 5.2Mpps
> OK so would you say it's really unsafe versus safe accesses?
> Or would you say it's just a better written code?
It's the effect of removing speculation barrier.
>
>> On machine with SMAP (Broadwell):
>>
>> Before: 5.0Mpps
>>
>> After: 6.1Mpps
>>
>> No smap: 7.5Mpps
>>
>>
>> Thanks
>
> no smap being before or after?
>
Let me clarify:
Before (SMAP on): 5.0Mpps
Before (SMAP off): 7.5Mpps
After (SMAP on): 6.1Mpps
Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 1/2] virtio-net: bql support
From: Jason Wang @ 2019-01-07 3:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: netdev, linux-kernel, virtualization, maxime.coquelin, wexu,
David S. Miller
In-Reply-To: <20190106221506-mutt-send-email-mst@kernel.org>
On 2019/1/7 上午11:17, Michael S. Tsirkin wrote:
> On Mon, Jan 07, 2019 at 10:14:37AM +0800, Jason Wang wrote:
>> On 2019/1/2 下午9:59, Michael S. Tsirkin wrote:
>>> On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote:
>>>> On 2018/12/31 上午2:45, Michael S. Tsirkin wrote:
>>>>> On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote:
>>>>>> On 2018/12/26 下午11:19, Michael S. Tsirkin wrote:
>>>>>>> On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote:
>>>>>>>> On 2018/12/6 上午6:54, Michael S. Tsirkin wrote:
>>>>>>>>> When use_napi is set, let's enable BQLs. Note: some of the issues are
>>>>>>>>> similar to wifi. It's worth considering whether something similar to
>>>>>>>>> commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be
>>>>>>>>> benefitial.
>>>>>>>> I've played a similar patch several days before. The tricky part is the mode
>>>>>>>> switching between napi and no napi. We should make sure when the packet is
>>>>>>>> sent and trakced by BQL, it should be consumed by BQL as well. I did it by
>>>>>>>> tracking it through skb->cb. And deal with the freeze by reset the BQL
>>>>>>>> status. Patch attached.
>>>>>>>>
>>>>>>>> But when testing with vhost-net, I don't very a stable performance,
>>>>>>> So how about increasing TSQ pacing shift then?
>>>>>> I can test this. But changing default TCP value is much more than a
>>>>>> virtio-net specific thing.
>>>>> Well same logic as wifi applies. Unpredictable latencies related
>>>>> to radio in one case, to host scheduler in the other.
>>>>>
>>>>>>>> it was
>>>>>>>> probably because we batch the used ring updating so tx interrupt may come
>>>>>>>> randomly. We probably need to implement time bounded coalescing mechanism
>>>>>>>> which could be configured from userspace.
>>>>>>> I don't think it's reasonable to expect userspace to be that smart ...
>>>>>>> Why do we need time bounded? used ring is always updated when ring
>>>>>>> becomes empty.
>>>>>> We don't add used when means BQL may not see the consumed packet in time.
>>>>>> And the delay varies based on the workload since we count packets not bytes
>>>>>> or time before doing the batched updating.
>>>>>>
>>>>>> Thanks
>>>>> Sorry I still don't get it.
>>>>> When nothing is outstanding then we do update the used.
>>>>> So if BQL stops userspace from sending packets then
>>>>> we get an interrupt and packets start flowing again.
>>>> Yes, but how about the cases of multiple flows. That's where I see unstable
>>>> results.
>>>>
>>>>
>>>>> It might be suboptimal, we might need to tune it but I doubt running
>>>>> timers is a solution, timer interrupts cause VM exits.
>>>> Probably not a timer but a time counter (or event byte counter) in vhost to
>>>> add used and signal guest if it exceeds a value instead of waiting the
>>>> number of packets.
>>>>
>>>>
>>>> Thanks
>>> Well we already have VHOST_NET_WEIGHT - is it too big then?
>>
>> I'm not sure, it might be too big.
>>
>>
>>> And maybe we should expose the "MORE" flag in the descriptor -
>>> do you think that will help?
>>>
>> I don't know. But how a "more" flag can help here?
>>
>> Thanks
> It sounds like we should be a bit more aggressive in updating used ring.
> But if we just do it naively we will harm performance for sure as that
> is how we are doing batching right now.
I agree but the problem is to balance the PPS and throughput. More
batching helps for PPS but may damage TCP throughput.
> Instead we could make guest
> control batching using the more flag - if that's not set we write out
> the used ring.
It's under the control of guest, so I'm afraid we still need some more
guard (e.g time/bytes counters) on host.
Thanks
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Michael S. Tsirkin @ 2019-01-07 3:28 UTC (permalink / raw)
To: Jason Wang; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <0efd115a-a7fb-54bf-5376-59d047a15fd3@redhat.com>
On Mon, Jan 07, 2019 at 10:19:03AM +0800, Jason Wang wrote:
>
> On 2019/1/3 上午4:47, Michael S. Tsirkin wrote:
> > On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
> > > This series tries to access virtqueue metadata through kernel virtual
> > > address instead of copy_user() friends since they had too much
> > > overheads like checks, spec barriers or even hardware feature
> > > toggling.
> > Will review, thanks!
> > One questions that comes to mind is whether it's all about bypassing
> > stac/clac. Could you please include a performance comparison with
> > nosmap?
> >
>
> On machine without SMAP (Sandy Bridge):
>
> Before: 4.8Mpps
>
> After: 5.2Mpps
OK so would you say it's really unsafe versus safe accesses?
Or would you say it's just a better written code?
> On machine with SMAP (Broadwell):
>
> Before: 5.0Mpps
>
> After: 6.1Mpps
>
> No smap: 7.5Mpps
>
>
> Thanks
no smap being before or after?
--
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 1/2] virtio-net: bql support
From: Michael S. Tsirkin @ 2019-01-07 3:17 UTC (permalink / raw)
To: Jason Wang
Cc: netdev, linux-kernel, virtualization, maxime.coquelin, wexu,
David S. Miller
In-Reply-To: <17d2ab21-1c9a-2bb9-166f-2863d019cb0b@redhat.com>
On Mon, Jan 07, 2019 at 10:14:37AM +0800, Jason Wang wrote:
>
> On 2019/1/2 下午9:59, Michael S. Tsirkin wrote:
> > On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote:
> > > On 2018/12/31 上午2:45, Michael S. Tsirkin wrote:
> > > > On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote:
> > > > > On 2018/12/26 下午11:19, Michael S. Tsirkin wrote:
> > > > > > On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote:
> > > > > > > On 2018/12/6 上午6:54, Michael S. Tsirkin wrote:
> > > > > > > > When use_napi is set, let's enable BQLs. Note: some of the issues are
> > > > > > > > similar to wifi. It's worth considering whether something similar to
> > > > > > > > commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be
> > > > > > > > benefitial.
> > > > > > > I've played a similar patch several days before. The tricky part is the mode
> > > > > > > switching between napi and no napi. We should make sure when the packet is
> > > > > > > sent and trakced by BQL, it should be consumed by BQL as well. I did it by
> > > > > > > tracking it through skb->cb. And deal with the freeze by reset the BQL
> > > > > > > status. Patch attached.
> > > > > > >
> > > > > > > But when testing with vhost-net, I don't very a stable performance,
> > > > > > So how about increasing TSQ pacing shift then?
> > > > > I can test this. But changing default TCP value is much more than a
> > > > > virtio-net specific thing.
> > > > Well same logic as wifi applies. Unpredictable latencies related
> > > > to radio in one case, to host scheduler in the other.
> > > >
> > > > > > > it was
> > > > > > > probably because we batch the used ring updating so tx interrupt may come
> > > > > > > randomly. We probably need to implement time bounded coalescing mechanism
> > > > > > > which could be configured from userspace.
> > > > > > I don't think it's reasonable to expect userspace to be that smart ...
> > > > > > Why do we need time bounded? used ring is always updated when ring
> > > > > > becomes empty.
> > > > > We don't add used when means BQL may not see the consumed packet in time.
> > > > > And the delay varies based on the workload since we count packets not bytes
> > > > > or time before doing the batched updating.
> > > > >
> > > > > Thanks
> > > > Sorry I still don't get it.
> > > > When nothing is outstanding then we do update the used.
> > > > So if BQL stops userspace from sending packets then
> > > > we get an interrupt and packets start flowing again.
> > > Yes, but how about the cases of multiple flows. That's where I see unstable
> > > results.
> > >
> > >
> > > > It might be suboptimal, we might need to tune it but I doubt running
> > > > timers is a solution, timer interrupts cause VM exits.
> > > Probably not a timer but a time counter (or event byte counter) in vhost to
> > > add used and signal guest if it exceeds a value instead of waiting the
> > > number of packets.
> > >
> > >
> > > Thanks
> > Well we already have VHOST_NET_WEIGHT - is it too big then?
>
>
> I'm not sure, it might be too big.
>
>
> >
> > And maybe we should expose the "MORE" flag in the descriptor -
> > do you think that will help?
> >
>
> I don't know. But how a "more" flag can help here?
>
> Thanks
It sounds like we should be a bit more aggressive in updating used ring.
But if we just do it naively we will harm performance for sure as that
is how we are doing batching right now. Instead we could make guest
control batching using the more flag - if that's not set we write out
the used ring.
>
> >
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Jason Wang @ 2019-01-07 2:19 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <20190102154038-mutt-send-email-mst@kernel.org>
On 2019/1/3 上午4:47, Michael S. Tsirkin wrote:
> On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
>> This series tries to access virtqueue metadata through kernel virtual
>> address instead of copy_user() friends since they had too much
>> overheads like checks, spec barriers or even hardware feature
>> toggling.
> Will review, thanks!
> One questions that comes to mind is whether it's all about bypassing
> stac/clac. Could you please include a performance comparison with
> nosmap?
>
On machine without SMAP (Sandy Bridge):
Before: 4.8Mpps
After: 5.2Mpps
On machine with SMAP (Broadwell):
Before: 5.0Mpps
After: 6.1Mpps
No smap: 7.5Mpps
Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH RFC 1/2] virtio-net: bql support
From: Jason Wang @ 2019-01-07 2:14 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: netdev, linux-kernel, virtualization, maxime.coquelin, wexu,
David S. Miller
In-Reply-To: <20190102085457-mutt-send-email-mst@kernel.org>
On 2019/1/2 下午9:59, Michael S. Tsirkin wrote:
> On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote:
>> On 2018/12/31 上午2:45, Michael S. Tsirkin wrote:
>>> On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote:
>>>> On 2018/12/26 下午11:19, Michael S. Tsirkin wrote:
>>>>> On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote:
>>>>>> On 2018/12/6 上午6:54, Michael S. Tsirkin wrote:
>>>>>>> When use_napi is set, let's enable BQLs. Note: some of the issues are
>>>>>>> similar to wifi. It's worth considering whether something similar to
>>>>>>> commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be
>>>>>>> benefitial.
>>>>>> I've played a similar patch several days before. The tricky part is the mode
>>>>>> switching between napi and no napi. We should make sure when the packet is
>>>>>> sent and trakced by BQL, it should be consumed by BQL as well. I did it by
>>>>>> tracking it through skb->cb. And deal with the freeze by reset the BQL
>>>>>> status. Patch attached.
>>>>>>
>>>>>> But when testing with vhost-net, I don't very a stable performance,
>>>>> So how about increasing TSQ pacing shift then?
>>>> I can test this. But changing default TCP value is much more than a
>>>> virtio-net specific thing.
>>> Well same logic as wifi applies. Unpredictable latencies related
>>> to radio in one case, to host scheduler in the other.
>>>
>>>>>> it was
>>>>>> probably because we batch the used ring updating so tx interrupt may come
>>>>>> randomly. We probably need to implement time bounded coalescing mechanism
>>>>>> which could be configured from userspace.
>>>>> I don't think it's reasonable to expect userspace to be that smart ...
>>>>> Why do we need time bounded? used ring is always updated when ring
>>>>> becomes empty.
>>>> We don't add used when means BQL may not see the consumed packet in time.
>>>> And the delay varies based on the workload since we count packets not bytes
>>>> or time before doing the batched updating.
>>>>
>>>> Thanks
>>> Sorry I still don't get it.
>>> When nothing is outstanding then we do update the used.
>>> So if BQL stops userspace from sending packets then
>>> we get an interrupt and packets start flowing again.
>> Yes, but how about the cases of multiple flows. That's where I see unstable
>> results.
>>
>>
>>> It might be suboptimal, we might need to tune it but I doubt running
>>> timers is a solution, timer interrupts cause VM exits.
>> Probably not a timer but a time counter (or event byte counter) in vhost to
>> add used and signal guest if it exceeds a value instead of waiting the
>> number of packets.
>>
>>
>> Thanks
> Well we already have VHOST_NET_WEIGHT - is it too big then?
I'm not sure, it might be too big.
>
> And maybe we should expose the "MORE" flag in the descriptor -
> do you think that will help?
>
I don't know. But how a "more" flag can help here?
Thanks
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* CISTI'2019 - Doctoral Symposium | Coimbra, Portugal
From: Maria Lemos @ 2019-01-06 18:58 UTC (permalink / raw)
To: virtualization
[-- Attachment #1.1: Type: text/plain, Size: 5358 bytes --]
* Published in IEEE Xplore and indexed by ISI, Scopus, etc.
---------------------------------------------------------------------------------------------------------------------------
Doctoral Symposium of CISTI'2019 - 14th Iberian Conference on Information Systems and Technologies
Coimbra, Portugal, 19 - 22 June 2019
http://www.cisti.eu/ <http://www.cisti.eu/>
------------------------------------------------------------------------------------------------------------------------------------
The purpose of CISTI'2019’s Doctoral Symposium is to provide graduate students a setting where they can, informally, expose and discuss their work, collecting valuable expert opinions and sharing new ideas, methods and applications. The Doctoral Symposium is an excellent opportunity for PhD students to present and discuss their work in a Workshop format. Each presentation will be evaluated by a panel composed by at least three Information Systems and Technologies experts.
Contributions Submission
The Doctoral Symposium is opened to PhD students whose research area includes the themes proposed for this Conference. Submissions must include an extended abstract (maximum 4 pages), following the Conference style guide <http://cisti.eu/2017/images/templates.zip>. All selected contributions will be handed out along with the Conference Proceedings, in CD with an ISBN. These contributions will be available in the IEEE Xplore <https://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=8390719> Digital Library and will be sent for indexing in ISI, Scopus, EI-Compendex, INSPEC and Google Scholar.
Submissions must include the field, the PhD institution and the number of months devoted to the development of the work. Additionally, they should include in a clear and succinct manner:
• The problem approached and its significance or relevance
• The research objectives and related investigation topics
• A brief display of what is already known
• A proposed solution methodology for the problem
• Expected results
Important Dates
Paper submission: February 10, 2019
Notification of acceptance: March 17, 2019
Submission of accepted papers: March 31, 2019
Payment of registration, to ensure the inclusion of an accepted paper in the conference proceedings: April 1, 2019
Organizing Committee
Álvaro Rocha, Universidade de Coimbra
Manuel Pérez Cota, Universidad de Vigo
Scientific Committee
Manuel Pérez Cota, Universidad de Vigo (Chair)
A. Augusto Sousa, FEUP, Universidade do Porto
Adolfo Lozano Tello, Universidad de Extremadura
Alma María Gómez Rodríguez, Universidade de Vigo
Álvaro Rocha, Universidade de Coimbra
Ana Amélia Carvalho, Universidade de Coimbra
Ana Maria Ramalho Correia, NOVA Information Management School
António Coelho, FEUP, Universidade do Porto
Antonio Garcia-Loureiro, Universidad de Santiago de Compostela
Arnaldo Martins, Universidade de Aveiro
Arturo Méndez Penín, Universidade de Vigo
Bráulio Alturas, ISCTE - Insituto Universitário de Lisboa
Carlos Costa, ISEG, Universidade de Lisboa
Carlos Ferrás Sexto, Universidad de Santiago de Compostela
David Fonseca, La Salle, Universitat Ramon Llull
Ernest Redondo, Universidad Politécnica de Catalunya
Fernando Moreira, Universidade Portucalense
Fernando Ramos, Universidade de Aveiro
Francisco Restivo, Universidade Católica Portuguesa
Gonçalo Paiva Dias, Universidade de Aveiro
Gonzalo Cuevas Agustin, Universidad Politécnica de Madrid
Guilhermina Maria Lobato de Miranda, IE, Universidade de Lisboa
João Costa, Universidade de Coimbra
João Manuel R.S. Tavares, FEUP, Universidade do Porto
José Antonio Calvo-Manzano Villalón, Universidad Politécnica de Madrid
José Borbinha, IST, Universidade de Lisboa
José Machado, Universidade do Minho
José Martins, Universidade de Trás-os-Montes e Alto Douro
Juan de Dios Murillo, Universidad Nacional de Costa Rica
Leandro Rodríguez Linares, Universidade de Vigo
Luciano Boquete, Universidad de Alcalá
Luis Camarinha Matos, Universidade Nova de Lisboa
Luis Macedo, Universidade de Coimbra
Luís Paulo Reis, FEUP, Universidade do Porto
Marco Painho, NOVA Information Management School
Mareca López María Pilar, Universidad Politécnica de Madrid
María José Lado Touriño, Universidade de Vigo
Mário Piattini, Universidad de Castilla-La Mancha
Mário Rela, Universidade de Coimbra
Martin Llamas-Nistal, Universidad de Vigo
Miguel Ramón González Castro, Ence, Energía y Celulosa
Nelson Rocha, Universidade de Aveiro
Paulo Pinto, Univesidade Nova de Lisboa
Óscar Mealha, Universidade de Aveiro
Ramiro Gonçalves, Universidade de Trás-os-Montes e Alto Douro
Vitor Santos, NOVA Information Management School
Yolanda García Vázquez, Universidad de Santiago de Compostela
Doctoral Symposium webpage: https://goo.gl/JTrcLB
<https://goo.gl/JTrcLB>
Kind regards,
CISTI'2019 Team
http://www.cisti.eu/ <http://www.cisti.eu/>
---
This email has been checked for viruses by AVG.
https://www.avg.com
[-- Attachment #1.2: Type: text/html, Size: 9341 bytes --]
[-- Attachment #2: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v2 1/2] virtio-balloon: tweak config_changed implementation
From: Michael S. Tsirkin @ 2019-01-06 0:31 UTC (permalink / raw)
To: Wang, Wei W
Cc: virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org,
cohuck@redhat.com, linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, pasic@linux.ibm.com,
pbonzini@redhat.com, dgilbert@redhat.com
In-Reply-To: <286AC319A985734F985F78AFA26841F73DEF57A3@shsmsx102.ccr.corp.intel.com>
On Sat, Jan 05, 2019 at 03:32:44AM +0000, Wang, Wei W wrote:
> On Friday, January 4, 2019 11:45 PM, Michael S. Tsirkin wrote:
> > > struct virtio_balloon {
> > > struct virtio_device *vdev;
> > > struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq;
> > > @@ -77,6 +81,8 @@ struct virtio_balloon {
> > > /* Prevent updating balloon when it is being canceled. */
> > > spinlock_t stop_update_lock;
> > > bool stop_update;
> > > + /* Bitmap to indicate if reading the related config fields are needed
> > */
> > > + unsigned long config_read_bitmap;
> > >
> > > /* The list of allocated free pages, waiting to be given back to mm */
> > > struct list_head free_page_list;
> >
> > It seems that you never initialize this bitmap. Probably harmless here but
> > generally using uninitialized memory isn't good.
>
> We've used kzalloc to allocate the vb struct, so it's already 0-initialized :)
Ah ok. We do explicitly init other fields like stop_update so
it seems cleaner to init this one too though.
> > > +
> > > static int send_free_pages(struct virtio_balloon *vb) {
> > > int err;
> > > @@ -620,6 +630,7 @@ static int send_free_pages(struct virtio_balloon *vb)
> > > * stop the reporting.
> > > */
> > > cmd_id_active = virtio32_to_cpu(vb->vdev, vb-
> > >cmd_id_active);
> > > + virtio_balloon_read_cmd_id_received(vb);
>
>
> [1]
>
>
> > > if (cmd_id_active != vb->cmd_id_received)
> > > break;
> > >
> > > @@ -637,11 +648,9 @@ static int send_free_pages(struct virtio_balloon
> > *vb)
> > > return 0;
> > > }
> > >
> > > -static void report_free_page_func(struct work_struct *work)
> > > +static void virtio_balloon_report_free_page(struct virtio_balloon
> > > +*vb)
> > > {
> > > int err;
> > > - struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> > > - report_free_page_work);
> > > struct device *dev = &vb->vdev->dev;
> > >
> > > /* Start by sending the received cmd id to host with an outbuf. */
> > > @@ -659,6 +668,22 @@ static void report_free_page_func(struct
> > work_struct *work)
> > > dev_err(dev, "Failed to send a stop id, err = %d\n", err); }
> > >
> > > +static void report_free_page_func(struct work_struct *work) {
> > > + struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> > > + report_free_page_work);
> > > +
> > > + virtio_balloon_read_cmd_id_received(vb);
> >
> > This will not achieve what you are trying to do, which is cancel reporting if it's
> > in progress.
> >
> > You need to re-read each time you compare to cmd_id_active.
>
> Yes, already did, please see [1] above
Oh right. Sorry.
>
> > An API similar to
> > u32 virtio_balloon_cmd_id_received(vb)
> > seems to be called for, and I would rename cmd_id_received to
> > cmd_id_received_cache to make sure we caught all users.
> >
>
> I'm not sure about adding "cache" here, cmd_id_received refers to the cmd id
> that the driver just received from the device. There is one called "cmd_id_active" which
> is the one that the driver is actively using.
> So cmd_id_received is already a "cache" to " cmd_id_active " in some sense.
>
> Best,
> Wei
The point is that any access to cmd_id_received should first call
virtio_balloon_read_cmd_id_received. So a better API is
to have virtio_balloon_read_cmd_id_received return the value
instead of poking at cmd_id_received afterwards.
Renaming cmd_id_received will help make sure all no
call sites were missed and help stress it's never used directly.
^ permalink raw reply
* RE: [PATCH v2 1/2] virtio-balloon: tweak config_changed implementation
From: Wang, Wei W @ 2019-01-05 3:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org,
cohuck@redhat.com, linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, pasic@linux.ibm.com,
pbonzini@redhat.com, dgilbert@redhat.com
In-Reply-To: <20190104103213-mutt-send-email-mst@kernel.org>
On Friday, January 4, 2019 11:45 PM, Michael S. Tsirkin wrote:
> > struct virtio_balloon {
> > struct virtio_device *vdev;
> > struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq;
> > @@ -77,6 +81,8 @@ struct virtio_balloon {
> > /* Prevent updating balloon when it is being canceled. */
> > spinlock_t stop_update_lock;
> > bool stop_update;
> > + /* Bitmap to indicate if reading the related config fields are needed
> */
> > + unsigned long config_read_bitmap;
> >
> > /* The list of allocated free pages, waiting to be given back to mm */
> > struct list_head free_page_list;
>
> It seems that you never initialize this bitmap. Probably harmless here but
> generally using uninitialized memory isn't good.
We've used kzalloc to allocate the vb struct, so it's already 0-initialized :)
> > +
> > static int send_free_pages(struct virtio_balloon *vb) {
> > int err;
> > @@ -620,6 +630,7 @@ static int send_free_pages(struct virtio_balloon *vb)
> > * stop the reporting.
> > */
> > cmd_id_active = virtio32_to_cpu(vb->vdev, vb-
> >cmd_id_active);
> > + virtio_balloon_read_cmd_id_received(vb);
[1]
> > if (cmd_id_active != vb->cmd_id_received)
> > break;
> >
> > @@ -637,11 +648,9 @@ static int send_free_pages(struct virtio_balloon
> *vb)
> > return 0;
> > }
> >
> > -static void report_free_page_func(struct work_struct *work)
> > +static void virtio_balloon_report_free_page(struct virtio_balloon
> > +*vb)
> > {
> > int err;
> > - struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> > - report_free_page_work);
> > struct device *dev = &vb->vdev->dev;
> >
> > /* Start by sending the received cmd id to host with an outbuf. */
> > @@ -659,6 +668,22 @@ static void report_free_page_func(struct
> work_struct *work)
> > dev_err(dev, "Failed to send a stop id, err = %d\n", err); }
> >
> > +static void report_free_page_func(struct work_struct *work) {
> > + struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> > + report_free_page_work);
> > +
> > + virtio_balloon_read_cmd_id_received(vb);
>
> This will not achieve what you are trying to do, which is cancel reporting if it's
> in progress.
>
> You need to re-read each time you compare to cmd_id_active.
Yes, already did, please see [1] above
> An API similar to
> u32 virtio_balloon_cmd_id_received(vb)
> seems to be called for, and I would rename cmd_id_received to
> cmd_id_received_cache to make sure we caught all users.
>
I'm not sure about adding "cache" here, cmd_id_received refers to the cmd id
that the driver just received from the device. There is one called "cmd_id_active" which
is the one that the driver is actively using.
So cmd_id_received is already a "cache" to " cmd_id_active " in some sense.
Best,
Wei
^ permalink raw reply
* Re: [RFC PATCH V3 0/5] Hi:
From: Michael S. Tsirkin @ 2019-01-04 21:41 UTC (permalink / raw)
To: Jason Wang; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <20181229124656.3900-1-jasowang@redhat.com>
On Sat, Dec 29, 2018 at 08:46:51PM +0800, Jason Wang wrote:
> This series tries to access virtqueue metadata through kernel virtual
> address instead of copy_user() friends since they had too much
> overheads like checks, spec barriers or even hardware feature
> toggling.
I think it's a reasonable approach.
However I need to look at whether and which mmu notifiers are invoked before
writeback. Do you know?
> Test shows about 24% improvement on TX PPS. It should benefit other
> cases as well.
>
> Changes from V2:
> - fix buggy range overlapping check
> - tear down MMU notifier during vhost ioctl to make sure invalidation
> request can read metadata userspace address and vq size without
> holding vq mutex.
> Changes from V1:
> - instead of pinning pages, use MMU notifier to invalidate vmaps and
> remap duing metadata prefetch
> - fix build warning on MIPS
>
> Jason Wang (5):
> vhost: generalize adding used elem
> vhost: fine grain userspace memory accessors
> vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch()
> vhost: introduce helpers to get the size of metadata area
> vhost: access vq metadata through kernel virtual address
>
> drivers/vhost/net.c | 4 +-
> drivers/vhost/vhost.c | 416 +++++++++++++++++++++++++++++++++++++-----
> drivers/vhost/vhost.h | 15 +-
> 3 files changed, 384 insertions(+), 51 deletions(-)
>
> --
> 2.17.1
^ permalink raw reply
* Re: [RFC PATCH V3 5/5] vhost: access vq metadata through kernel virtual address
From: Michael S. Tsirkin @ 2019-01-04 21:34 UTC (permalink / raw)
To: Jason Wang; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <20181229124656.3900-6-jasowang@redhat.com>
On Sat, Dec 29, 2018 at 08:46:56PM +0800, Jason Wang wrote:
> It was noticed that the copy_user() friends that was used to access
> virtqueue metdata tends to be very expensive for dataplane
> implementation like vhost since it involves lots of software checks,
> speculation barrier, hardware feature toggling (e.g SMAP). The
> extra cost will be more obvious when transferring small packets since
> the time spent on metadata accessing become significant..
>
> This patch tries to eliminate those overhead by accessing them through
> kernel virtual address by vmap(). To make the pages can be migrated,
> instead of pinning them through GUP, we use mmu notifiers to
> invalidate vmaps and re-establish vmaps during each round of metadata
> prefetching in necessary. For devices that doesn't use metadata
> prefetching, the memory acessors fallback to normal copy_user()
> implementation gracefully. The invalidation was synchronized with
> datapath through vq mutex, and in order to avoid hold vq mutex during
> range checking, MMU notifier was teared down when trying to modify vq
> metadata.
>
> Note that this was only done when device IOTLB is not enabled. We
> could use similar method to optimize it in the future.
>
> Tests shows about ~24% improvement on TX PPS when using virtio-user +
> vhost_net + xdp1 on TAP:
>
> Before: ~5.0Mpps
> After: ~6.1Mpps
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
> drivers/vhost/vhost.c | 263 +++++++++++++++++++++++++++++++++++++++++-
> drivers/vhost/vhost.h | 13 +++
> 2 files changed, 274 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 54b43feef8d9..e1ecb8acf8a3 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -440,6 +440,9 @@ void vhost_dev_init(struct vhost_dev *dev,
> vq->indirect = NULL;
> vq->heads = NULL;
> vq->dev = dev;
> + memset(&vq->avail_ring, 0, sizeof(vq->avail_ring));
> + memset(&vq->used_ring, 0, sizeof(vq->used_ring));
> + memset(&vq->desc_ring, 0, sizeof(vq->desc_ring));
> mutex_init(&vq->mutex);
> vhost_vq_reset(dev, vq);
> if (vq->handle_kick)
> @@ -510,6 +513,73 @@ static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, int num)
> return sizeof(*vq->desc) * num;
> }
>
> +static void vhost_uninit_vmap(struct vhost_vmap *map)
> +{
> + if (map->addr)
> + vunmap(map->unmap_addr);
> +
> + map->addr = NULL;
> + map->unmap_addr = NULL;
> +}
> +
> +static int vhost_invalidate_vmap(struct vhost_virtqueue *vq,
> + struct vhost_vmap *map,
> + unsigned long ustart,
> + size_t size,
> + unsigned long start,
> + unsigned long end,
> + bool blockable)
> +{
> + if (end < ustart || start > ustart - 1 + size)
> + return 0;
> +
> + if (!blockable)
> + return -EAGAIN;
> +
> + mutex_lock(&vq->mutex);
> + vhost_uninit_vmap(map);
> + mutex_unlock(&vq->mutex);
> +
> + return 0;
> +}
> +
> +static int vhost_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
> + struct mm_struct *mm,
> + unsigned long start,
> + unsigned long end,
> + bool blockable)
> +{
> + struct vhost_dev *dev = container_of(mn, struct vhost_dev,
> + mmu_notifier);
> + int i;
> +
> + for (i = 0; i < dev->nvqs; i++) {
> + struct vhost_virtqueue *vq = dev->vqs[i];
> +
> + if (vhost_invalidate_vmap(vq, &vq->avail_ring,
> + (unsigned long)vq->avail,
> + vhost_get_avail_size(vq, vq->num),
> + start, end, blockable))
> + return -EAGAIN;
> + if (vhost_invalidate_vmap(vq, &vq->desc_ring,
> + (unsigned long)vq->desc,
> + vhost_get_desc_size(vq, vq->num),
> + start, end, blockable))
> + return -EAGAIN;
> + if (vhost_invalidate_vmap(vq, &vq->used_ring,
> + (unsigned long)vq->used,
> + vhost_get_used_size(vq, vq->num),
> + start, end, blockable))
> + return -EAGAIN;
> + }
> +
> + return 0;
> +}
> +
> +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = {
> + .invalidate_range_start = vhost_mmu_notifier_invalidate_range_start,
> +};
> +
> /* Caller should have device mutex */
> long vhost_dev_set_owner(struct vhost_dev *dev)
> {
> @@ -541,7 +611,14 @@ long vhost_dev_set_owner(struct vhost_dev *dev)
> if (err)
> goto err_cgroup;
>
> + dev->mmu_notifier.ops = &vhost_mmu_notifier_ops;
> + err = mmu_notifier_register(&dev->mmu_notifier, dev->mm);
> + if (err)
> + goto err_mmu_notifier;
> +
> return 0;
> +err_mmu_notifier:
> + vhost_dev_free_iovecs(dev);
> err_cgroup:
> kthread_stop(worker);
> dev->worker = NULL;
> @@ -632,6 +709,72 @@ static void vhost_clear_msg(struct vhost_dev *dev)
> spin_unlock(&dev->iotlb_lock);
> }
>
> +static int vhost_init_vmap(struct vhost_vmap *map, unsigned long uaddr,
> + size_t size, int write)
> +{
> + struct page **pages;
> + int npages = DIV_ROUND_UP(size, PAGE_SIZE);
> + int npinned;
> + void *vaddr;
> + int err = 0;
> +
> + pages = kmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> + if (!pages)
> + return -ENOMEM;
> +
> + npinned = get_user_pages_fast(uaddr, npages, write, pages);
> + if (npinned != npages) {
> + err = -EFAULT;
> + goto err;
> + }
> +
> + vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL);
> + if (!vaddr) {
> + err = EFAULT;
> + goto err;
> + }
> +
> + map->addr = vaddr + (uaddr & (PAGE_SIZE - 1));
> + map->unmap_addr = vaddr;
> +
> +err:
> + /* Don't pin pages, mmu notifier will notify us about page
> + * migration.
> + */
> + if (npinned > 0)
> + release_pages(pages, npinned);
> + kfree(pages);
> + return err;
> +}
> +
> +static void vhost_clean_vmaps(struct vhost_virtqueue *vq)
> +{
> + vhost_uninit_vmap(&vq->avail_ring);
> + vhost_uninit_vmap(&vq->desc_ring);
> + vhost_uninit_vmap(&vq->used_ring);
> +}
> +
> +static int vhost_setup_avail_vmap(struct vhost_virtqueue *vq,
> + unsigned long avail)
> +{
> + return vhost_init_vmap(&vq->avail_ring, avail,
> + vhost_get_avail_size(vq, vq->num), false);
> +}
> +
> +static int vhost_setup_desc_vmap(struct vhost_virtqueue *vq,
> + unsigned long desc)
> +{
> + return vhost_init_vmap(&vq->desc_ring, desc,
> + vhost_get_desc_size(vq, vq->num), false);
> +}
> +
> +static int vhost_setup_used_vmap(struct vhost_virtqueue *vq,
> + unsigned long used)
> +{
> + return vhost_init_vmap(&vq->used_ring, used,
> + vhost_get_used_size(vq, vq->num), true);
> +}
> +
> void vhost_dev_cleanup(struct vhost_dev *dev)
> {
> int i;
> @@ -661,8 +804,12 @@ void vhost_dev_cleanup(struct vhost_dev *dev)
> kthread_stop(dev->worker);
> dev->worker = NULL;
> }
> - if (dev->mm)
> + if (dev->mm) {
> + mmu_notifier_unregister(&dev->mmu_notifier, dev->mm);
> mmput(dev->mm);
> + }
> + for (i = 0; i < dev->nvqs; i++)
> + vhost_clean_vmaps(dev->vqs[i]);
> dev->mm = NULL;
> }
> EXPORT_SYMBOL_GPL(vhost_dev_cleanup);
> @@ -891,6 +1038,16 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq,
>
> static inline int vhost_put_avail_event(struct vhost_virtqueue *vq)
> {
> + if (!vq->iotlb) {
Do we have to limit this to !iotlb?
> + struct vring_used *used = vq->used_ring.addr;
> +
> + if (likely(used)) {
> + *((__virtio16 *)&used->ring[vq->num]) =
> + cpu_to_vhost16(vq, vq->avail_idx);
So here we are modifying userspace memory without marking it dirty.
Is this OK? And why?
> + return 0;
> + }
> + }
> +
> return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx),
> vhost_avail_event(vq));
> }
> @@ -899,6 +1056,16 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq,
> struct vring_used_elem *head, int idx,
> int count)
> {
> + if (!vq->iotlb) {
> + struct vring_used *used = vq->used_ring.addr;
> +
> + if (likely(used)) {
> + memcpy(used->ring + idx, head,
> + count * sizeof(*head));
> + return 0;
Same here.
> + }
> + }
> +
> return vhost_copy_to_user(vq, vq->used->ring + idx, head,
> count * sizeof(*head));
> }
> @@ -906,6 +1073,15 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq,
> static inline int vhost_put_used_flags(struct vhost_virtqueue *vq)
>
> {
> + if (!vq->iotlb) {
> + struct vring_used *used = vq->used_ring.addr;
> +
> + if (likely(used)) {
> + used->flags = cpu_to_vhost16(vq, vq->used_flags);
> + return 0;
> + }
> + }
> +
> return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags),
> &vq->used->flags);
> }
> @@ -913,6 +1089,15 @@ static inline int vhost_put_used_flags(struct vhost_virtqueue *vq)
> static inline int vhost_put_used_idx(struct vhost_virtqueue *vq)
>
> {
> + if (!vq->iotlb) {
> + struct vring_used *used = vq->used_ring.addr;
> +
> + if (likely(used)) {
> + used->idx = cpu_to_vhost16(vq, vq->last_used_idx);
> + return 0;
> + }
> + }
> +
> return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx),
> &vq->used->idx);
> }
> @@ -958,12 +1143,30 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d)
> static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq,
> __virtio16 *idx)
> {
> + if (!vq->iotlb) {
> + struct vring_avail *avail = vq->avail_ring.addr;
> +
> + if (likely(avail)) {
> + *idx = avail->idx;
> + return 0;
> + }
> + }
> +
> return vhost_get_avail(vq, *idx, &vq->avail->idx);
> }
>
> static inline int vhost_get_avail_head(struct vhost_virtqueue *vq,
> __virtio16 *head, int idx)
> {
> + if (!vq->iotlb) {
> + struct vring_avail *avail = vq->avail_ring.addr;
> +
> + if (likely(avail)) {
> + *head = avail->ring[idx & (vq->num - 1)];
> + return 0;
> + }
> + }
> +
> return vhost_get_avail(vq, *head,
> &vq->avail->ring[idx & (vq->num - 1)]);
> }
> @@ -971,24 +1174,60 @@ static inline int vhost_get_avail_head(struct vhost_virtqueue *vq,
> static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq,
> __virtio16 *flags)
> {
> + if (!vq->iotlb) {
> + struct vring_avail *avail = vq->avail_ring.addr;
> +
> + if (likely(avail)) {
> + *flags = avail->flags;
> + return 0;
> + }
> + }
> +
> return vhost_get_avail(vq, *flags, &vq->avail->flags);
> }
>
> static inline int vhost_get_used_event(struct vhost_virtqueue *vq,
> __virtio16 *event)
> {
> + if (!vq->iotlb) {
> + struct vring_avail *avail = vq->avail_ring.addr;
> +
> + if (likely(avail)) {
> + *event = (__virtio16)avail->ring[vq->num];
> + return 0;
> + }
> + }
> +
> return vhost_get_avail(vq, *event, vhost_used_event(vq));
> }
>
> static inline int vhost_get_used_idx(struct vhost_virtqueue *vq,
> __virtio16 *idx)
> {
> + if (!vq->iotlb) {
> + struct vring_used *used = vq->used_ring.addr;
> +
> + if (likely(used)) {
> + *idx = used->idx;
> + return 0;
> + }
> + }
> +
> return vhost_get_used(vq, *idx, &vq->used->idx);
> }
>
> static inline int vhost_get_desc(struct vhost_virtqueue *vq,
> struct vring_desc *desc, int idx)
> {
> + if (!vq->iotlb) {
> + struct vring_desc *d = vq->desc_ring.addr;
> +
> + if (likely(d)) {
> + *desc = *(d + idx);
> + return 0;
> + }
> + }
> +
> return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc));
> }
>
> @@ -1325,8 +1564,16 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq)
> {
> unsigned int num = vq->num;
>
> - if (!vq->iotlb)
> + if (!vq->iotlb) {
> + if (unlikely(!vq->avail_ring.addr))
> + vhost_setup_avail_vmap(vq, (unsigned long)vq->avail);
> + if (unlikely(!vq->desc_ring.addr))
> + vhost_setup_desc_vmap(vq, (unsigned long)vq->desc);
> + if (unlikely(!vq->used_ring.addr))
> + vhost_setup_used_vmap(vq, (unsigned long)vq->used);
> +
> return 1;
> + }
>
> return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc,
> vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) &&
> @@ -1478,6 +1725,13 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>
> mutex_lock(&vq->mutex);
>
> + /* Unregister MMU notifer to allow invalidation callback
> + * can access vq->avail, vq->desc , vq->used and vq->num
> + * without holding vq->mutex.
> + */
> + if (d->mm)
> + mmu_notifier_unregister(&d->mmu_notifier, d->mm);
> +
> switch (ioctl) {
> case VHOST_SET_VRING_NUM:
> /* Resizing ring with an active backend?
> @@ -1494,6 +1748,7 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> r = -EINVAL;
> break;
> }
> + vhost_clean_vmaps(vq);
> vq->num = s.num;
> break;
> case VHOST_SET_VRING_BASE:
> @@ -1571,6 +1826,8 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> }
> }
>
> + vhost_clean_vmaps(vq);
> +
> vq->log_used = !!(a.flags & (0x1 << VHOST_VRING_F_LOG));
> vq->desc = (void __user *)(unsigned long)a.desc_user_addr;
> vq->avail = (void __user *)(unsigned long)a.avail_user_addr;
> @@ -1651,6 +1908,8 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> if (pollstart && vq->handle_kick)
> r = vhost_poll_start(&vq->poll, vq->kick);
>
> + if (d->mm)
> + mmu_notifier_register(&d->mmu_notifier, d->mm);
> mutex_unlock(&vq->mutex);
>
> if (pollstop && vq->handle_kick)
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 0d1ff977a43e..00f016a4f198 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -12,6 +12,8 @@
> #include <linux/virtio_config.h>
> #include <linux/virtio_ring.h>
> #include <linux/atomic.h>
> +#include <linux/pagemap.h>
> +#include <linux/mmu_notifier.h>
>
> struct vhost_work;
> typedef void (*vhost_work_fn_t)(struct vhost_work *work);
> @@ -80,6 +82,11 @@ enum vhost_uaddr_type {
> VHOST_NUM_ADDRS = 3,
> };
>
> +struct vhost_vmap {
> + void *addr;
> + void *unmap_addr;
> +};
> +
How about using actual types like struct vring_used etc so we get type
safety and do not need to cast on access?
> /* The virtqueue structure describes a queue attached to a device. */
> struct vhost_virtqueue {
> struct vhost_dev *dev;
> @@ -90,6 +97,11 @@ struct vhost_virtqueue {
> struct vring_desc __user *desc;
> struct vring_avail __user *avail;
> struct vring_used __user *used;
> +
> + struct vhost_vmap avail_ring;
> + struct vhost_vmap desc_ring;
> + struct vhost_vmap used_ring;
> +
> const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
> struct file *kick;
> struct eventfd_ctx *call_ctx;
> @@ -158,6 +170,7 @@ struct vhost_msg_node {
>
> struct vhost_dev {
> struct mm_struct *mm;
> + struct mmu_notifier mmu_notifier;
> struct mutex mutex;
> struct vhost_virtqueue **vqs;
> int nvqs;
> --
> 2.17.1
^ permalink raw reply
* Re: [RFC PATCH V3 1/5] vhost: generalize adding used elem
From: Michael S. Tsirkin @ 2019-01-04 21:29 UTC (permalink / raw)
To: Jason Wang; +Cc: netdev, davem, linux-kernel, kvm, virtualization
In-Reply-To: <20181229124656.3900-2-jasowang@redhat.com>
On Sat, Dec 29, 2018 at 08:46:52PM +0800, Jason Wang wrote:
> Use one generic vhost_copy_to_user() instead of two dedicated
> accessor. This will simplify the conversion to fine grain
> accessors. About 2% improvement of PPS were seen during vitio-user
> txonly test.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
I don't hve a problem with this patch but do you have
any idea how come removing what's supposed to be
an optimization speeds things up?
> ---
> drivers/vhost/vhost.c | 11 +----------
> 1 file changed, 1 insertion(+), 10 deletions(-)
>
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 55e5aa662ad5..f179b5ee14c4 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2174,16 +2174,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq,
>
> start = vq->last_used_idx & (vq->num - 1);
> used = vq->used->ring + start;
> - if (count == 1) {
> - if (vhost_put_user(vq, heads[0].id, &used->id)) {
> - vq_err(vq, "Failed to write used id");
> - return -EFAULT;
> - }
> - if (vhost_put_user(vq, heads[0].len, &used->len)) {
> - vq_err(vq, "Failed to write used len");
> - return -EFAULT;
> - }
> - } else if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) {
> + if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) {
> vq_err(vq, "Failed to write used");
> return -EFAULT;
> }
> --
> 2.17.1
^ permalink raw reply
* Re: [PATCH v2 1/2] virtio-balloon: tweak config_changed implementation
From: Michael S. Tsirkin @ 2019-01-04 15:44 UTC (permalink / raw)
To: Wei Wang
Cc: virtio-dev, kvm, cohuck, linux-kernel, virtualization, pasic,
pbonzini, dgilbert
In-Reply-To: <1546585913-34804-2-git-send-email-wei.w.wang@intel.com>
On Fri, Jan 04, 2019 at 03:11:52PM +0800, Wei Wang wrote:
> virtio-ccw has deadlock issues with reading the config space inside the
> interrupt context, so we tweak the virtballoon_changed implementation
> by moving the config read operations into the related workqueue contexts.
> The config_read_bitmap is used as a flag to the workqueue callbacks
> about the related config fields that need to be read.
>
> Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Signed-off-by: Wei Wang <wei.w.wang@intel.com>
> ---
> drivers/virtio/virtio_balloon.c | 81 +++++++++++++++++++++++++++--------------
> 1 file changed, 53 insertions(+), 28 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 728ecd1..35ee762 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -61,6 +61,10 @@ enum virtio_balloon_vq {
> VIRTIO_BALLOON_VQ_MAX
> };
>
> +enum virtio_balloon_config_read {
> + VIRTIO_BALLOON_CONFIG_READ_CMD_ID = 0,
> +};
> +
> struct virtio_balloon {
> struct virtio_device *vdev;
> struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq;
> @@ -77,6 +81,8 @@ struct virtio_balloon {
> /* Prevent updating balloon when it is being canceled. */
> spinlock_t stop_update_lock;
> bool stop_update;
> + /* Bitmap to indicate if reading the related config fields are needed */
> + unsigned long config_read_bitmap;
>
> /* The list of allocated free pages, waiting to be given back to mm */
> struct list_head free_page_list;
It seems that you never initialize this bitmap. Probably harmless here
but generally using uninitialized memory isn't good.
> @@ -390,37 +396,31 @@ static unsigned long return_free_pages_to_mm(struct virtio_balloon *vb,
> return num_returned;
> }
>
> +static void virtio_balloon_queue_free_page_work(struct virtio_balloon *vb)
> +{
> + if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
> + return;
> +
> + /* No need to queue the work if the bit was already set. */
> + if (test_and_set_bit(VIRTIO_BALLOON_CONFIG_READ_CMD_ID,
> + &vb->config_read_bitmap))
> + return;
> +
> + queue_work(vb->balloon_wq, &vb->report_free_page_work);
> +}
> +
> static void virtballoon_changed(struct virtio_device *vdev)
> {
> struct virtio_balloon *vb = vdev->priv;
> unsigned long flags;
> - s64 diff = towards_target(vb);
> -
> - if (diff) {
> - spin_lock_irqsave(&vb->stop_update_lock, flags);
> - if (!vb->stop_update)
> - queue_work(system_freezable_wq,
> - &vb->update_balloon_size_work);
> - spin_unlock_irqrestore(&vb->stop_update_lock, flags);
> - }
>
> - if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) {
> - virtio_cread(vdev, struct virtio_balloon_config,
> - free_page_report_cmd_id, &vb->cmd_id_received);
> - if (vb->cmd_id_received == VIRTIO_BALLOON_CMD_ID_DONE) {
> - /* Pass ULONG_MAX to give back all the free pages */
> - return_free_pages_to_mm(vb, ULONG_MAX);
> - } else if (vb->cmd_id_received != VIRTIO_BALLOON_CMD_ID_STOP &&
> - vb->cmd_id_received !=
> - virtio32_to_cpu(vdev, vb->cmd_id_active)) {
> - spin_lock_irqsave(&vb->stop_update_lock, flags);
> - if (!vb->stop_update) {
> - queue_work(vb->balloon_wq,
> - &vb->report_free_page_work);
> - }
> - spin_unlock_irqrestore(&vb->stop_update_lock, flags);
> - }
> + spin_lock_irqsave(&vb->stop_update_lock, flags);
> + if (!vb->stop_update) {
> + queue_work(system_freezable_wq,
> + &vb->update_balloon_size_work);
> + virtio_balloon_queue_free_page_work(vb);
> }
> + spin_unlock_irqrestore(&vb->stop_update_lock, flags);
> }
>
> static void update_balloon_size(struct virtio_balloon *vb)
> @@ -609,6 +609,16 @@ static int get_free_page_and_send(struct virtio_balloon *vb)
> return 0;
> }
>
> +static void virtio_balloon_read_cmd_id_received(struct virtio_balloon *vb)
> +{
> + if (!test_and_clear_bit(VIRTIO_BALLOON_CONFIG_READ_CMD_ID,
> + &vb->config_read_bitmap))
> + return;
> +
> + virtio_cread(vb->vdev, struct virtio_balloon_config,
> + free_page_report_cmd_id, &vb->cmd_id_received);
> +}
> +
> static int send_free_pages(struct virtio_balloon *vb)
> {
> int err;
> @@ -620,6 +630,7 @@ static int send_free_pages(struct virtio_balloon *vb)
> * stop the reporting.
> */
> cmd_id_active = virtio32_to_cpu(vb->vdev, vb->cmd_id_active);
> + virtio_balloon_read_cmd_id_received(vb);
> if (cmd_id_active != vb->cmd_id_received)
> break;
>
> @@ -637,11 +648,9 @@ static int send_free_pages(struct virtio_balloon *vb)
> return 0;
> }
>
> -static void report_free_page_func(struct work_struct *work)
> +static void virtio_balloon_report_free_page(struct virtio_balloon *vb)
> {
> int err;
> - struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> - report_free_page_work);
> struct device *dev = &vb->vdev->dev;
>
> /* Start by sending the received cmd id to host with an outbuf. */
> @@ -659,6 +668,22 @@ static void report_free_page_func(struct work_struct *work)
> dev_err(dev, "Failed to send a stop id, err = %d\n", err);
> }
>
> +static void report_free_page_func(struct work_struct *work)
> +{
> + struct virtio_balloon *vb = container_of(work, struct virtio_balloon,
> + report_free_page_work);
> +
> + virtio_balloon_read_cmd_id_received(vb);
This will not achieve what you are trying to do,
which is cancel reporting if it's in progress.
You need to re-read each time you compare to cmd_id_active.
An API similar to
u32 virtio_balloon_cmd_id_received(vb)
seems to be called for, and I would rename cmd_id_received to
cmd_id_received_cache to make sure we caught all users.
> + if (vb->cmd_id_received == VIRTIO_BALLOON_CMD_ID_DONE) {
> + /* Pass ULONG_MAX to give back all the free pages */
> + return_free_pages_to_mm(vb, ULONG_MAX);
> + } else if (vb->cmd_id_received != VIRTIO_BALLOON_CMD_ID_STOP &&
> + vb->cmd_id_received !=
> + virtio32_to_cpu(vb->vdev, vb->cmd_id_active)) {
> + virtio_balloon_report_free_page(vb);
> + }
> +}
> +
> #ifdef CONFIG_BALLOON_COMPACTION
> /*
> * virtballoon_migratepage - perform the balloon page migration on behalf of
> --
> 2.7.4
^ permalink raw reply
* Re: [RFC PATCH 1/1] s390/virtio: handle find on invalid queue gracefully
From: Cornelia Huck @ 2019-01-04 14:01 UTC (permalink / raw)
To: Halil Pasic; +Cc: linux-s390, kvm, virtualization
In-Reply-To: <20190103140010.191ce605@oc2783563651>
On Thu, 3 Jan 2019 14:00:10 +0100
Halil Pasic <pasic@linux.ibm.com> wrote:
> On Wed, 2 Jan 2019 19:25:49 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
>
> > On Wed, 2 Jan 2019 18:50:20 +0100
> > Halil Pasic <pasic@linux.ibm.com> wrote:
> >
> > > A queue with a capacity of zero is clearly not a valid virtio queue.
> > > Some emulators report zero queue size if queried with an invalid queue
> > > index. Instead of crashing in this case let us just return -EINVAL. To
> > > make that work properly, let us fix the notifier cleanup logic as well.
> > >
> > > Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> > > ---
> > >
> > > This patch is motivated by commit 86a5597 "virtio-balloon:
> > > VIRTIO_BALLOON_F_FREE_PAGE_HINT" (Wei Wang, 2018-08-27) which triggered
> > > the described scenario. The emulator in question is the current QEMU.
> > > The problem we run into is the underflow in the following loop
> > > in __vring_new_virtqueue():
> > > for (i = 0; i < vring.num-1; i++)
> > > vq->vring.desc[i].next = cpu_to_virtio16(vdev, i + 1)
> > > Namely vring.num is an unsigned int.
> > >
> > > RFC because I'm not sure about -EINVAL being a good choice, and about
> > > us caring about what happens if a virtio driver misbehaves like described.
> >
> > For virtio-pci, the spec says that a zero queue size means that the
> > queue is unavailable. I don't think we have specified that explicitly
> > for virtio-ccw, but it does make sense.
> >
> > virtio-pci returns -ENOENT in that case, which might be a good choice
> > here as well.
>
> virtio-mmio does the same. I will change it to -ENOENT. Maybe also do
> something like
> int virtio_ccw_read_vq_conf() {
> [..]
> return vcdev->config_block->num ?: -ENOENT;
> }
>
> instead of the extra if statement, or?
Yes, why not.
>
> >
> > >
> > > ---
> > > drivers/s390/virtio/virtio_ccw.c | 6 ++++++
> > > 1 file changed, 6 insertions(+)
> > >
> > > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> > > index fc9dbad476c0..147927ed4fca 100644
> > > --- a/drivers/s390/virtio/virtio_ccw.c
> > > +++ b/drivers/s390/virtio/virtio_ccw.c
> > > @@ -272,6 +272,8 @@ static void virtio_ccw_drop_indicators(struct virtio_ccw_device *vcdev)
> > > {
> > > struct virtio_ccw_vq_info *info;
> > >
> > > + if (!vcdev->airq_info)
> > > + return;
> >
> > Which case is this guarding against? names[i] was NULL for every index?
> >
>
> Consider:
> static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs,
> struct virtqueue *vqs[],
> vq_callback_t *callbacks[],
> const char * const names[],
> const bool *ctx,
> struct irq_affinity *desc)
> {
> struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> unsigned long *indicatorp = NULL;
> int ret, i;
> struct ccw1 *ccw;
>
> ccw = kzalloc(sizeof(*ccw), GFP_DMA | GFP_KERNEL);
> if (!ccw)
> return -ENOMEM;
>
> for (i = 0; i < nvqs; ++i) {
> vqs[i] = virtio_ccw_setup_vq(vdev, i, callbacks[i], names[i],
> ctx ? ctx[i] : false, ccw);
> if (IS_ERR(vqs[i])) {
> ret = PTR_ERR(vqs[i]);
> vqs[i] = NULL;
> goto out;
> }
> }
> [..]
> if (vcdev->is_thinint) {
> ret = virtio_ccw_register_adapter_ind(vcdev, vqs, nvqs, ccw);
> if (ret)
> /* no error, just fall back to legacy interrupts */
> vcdev->is_thinint = false;
> }
> [..]
> out:
> kfree(indicatorp);
> kfree(ccw);
> virtio_ccw_del_vqs(vdev);
> return ret;
> }
> when the loop that calls virtio_ccw_setup_vq() fails after a couple
> of iterations. We end up with some queues in vcdev->virtqueues but
> with virtio_ccw_register_adapter_ind() never called and thus with
> vcdev->airq_info never set. So when virtio_ccw_del_vqs() tries to clean
> up we get an invalid pointer dereference.
>
> Does that answer your question?
Yes, thanks.
>
> I don't quite get your comments about names[i] == NULL.
I was looking at a tree with "virtio: don't allocate vqs when names[i]
= NULL" applied :)
^ permalink raw reply
* Re: [PATCH 2/2] virtio: document virtio_config_ops restrictions
From: Cornelia Huck @ 2019-01-04 12:48 UTC (permalink / raw)
To: Halil Pasic; +Cc: virtio-dev, Michael S . Tsirkin, linux-kernel, virtualization
In-Reply-To: <20190103182849.7b0b0f0b@oc2783563651>
On Thu, 3 Jan 2019 18:28:49 +0100
Halil Pasic <pasic@linux.ibm.com> wrote:
> On Thu, 3 Jan 2019 17:08:04 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
>
> > Some transports (e.g. virtio-ccw) implement virtio operations that
> > seem to be a simple read/write as something more involved that
> > cannot be done from an atomic context.
> >
> > Give at least a hint about that.
> >
> > Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> > ---
> > include/linux/virtio_config.h | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> > index 7087ef946ba7..987b6491b946 100644
> > --- a/include/linux/virtio_config.h
> > +++ b/include/linux/virtio_config.h
> > @@ -12,6 +12,11 @@ struct irq_affinity;
> >
> > /**
> > * virtio_config_ops - operations for configuring a virtio device
> > + * Note: Do not assume that a transport implements all of the operations
> > + * getting/setting a value as a simple read/write! Generally speaking,
> > + * any of @get/@set, @get_status/@set_status, or @get_features/
> > + * @finalize_features are NOT safe to be called from an atomic
> > + * context.
>
> I think the only exception is @bus_name (and maybe @generation, I don't
> know) because it does not have to 'speak' with the hypervisor. If a
> transport operation has to 'speak' with the hypervisor, we do it by
> making it interpret a channel program. That means not safe to be called
> form atomic context. Or am I missing something?
I explicitly singled out the listed callbacks because they read/write a
value and there might be more to them than meets the eye. I would
assume that nobody expects calling e.g. find_vqs (allocating queues)
from an atomic context to be a good idea :) Maybe I should do
s/Generally speaking/In particular/ ?
That said, it's only a hint; we should add might_sleep as well to
interfaces where it makes sense.
^ permalink raw reply
* Re: [PATCH 2/2] virtio: document virtio_config_ops restrictions
From: Cornelia Huck @ 2019-01-04 12:39 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: virtio-dev, linux-kernel, virtualization, Halil Pasic
In-Reply-To: <20190103112627-mutt-send-email-mst@kernel.org>
On Thu, 3 Jan 2019 11:28:28 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jan 03, 2019 at 05:08:04PM +0100, Cornelia Huck wrote:
> > Some transports (e.g. virtio-ccw) implement virtio operations that
> > seem to be a simple read/write as something more involved that
> > cannot be done from an atomic context.
> >
> > Give at least a hint about that.
> >
> > Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> > ---
> > include/linux/virtio_config.h | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> > index 7087ef946ba7..987b6491b946 100644
> > --- a/include/linux/virtio_config.h
> > +++ b/include/linux/virtio_config.h
> > @@ -12,6 +12,11 @@ struct irq_affinity;
> >
> > /**
> > * virtio_config_ops - operations for configuring a virtio device
> > + * Note: Do not assume that a transport implements all of the operations
> > + * getting/setting a value as a simple read/write! Generally speaking,
> > + * any of @get/@set, @get_status/@set_status, or @get_features/
> > + * @finalize_features are NOT safe to be called from an atomic
> > + * context.
> > * @get: read the value of a configuration field
> > * vdev: the virtio_device
> > * offset: the offset of the configuration field
>
> Then might_sleep in virtio_cread/virtio_cwrite and
> friends would be appropriate? I guess we'll need to fix
> balloon first.
Yes, it makes sense to go over the code and add might_sleep to
functions where it makes sense after the balloon changes have been
merged.
^ permalink raw reply
* Re: [PATCH v2 1/2] virtio-balloon: tweak config_changed implementation
From: Cornelia Huck @ 2019-01-04 11:02 UTC (permalink / raw)
To: Wei Wang
Cc: virtio-dev, kvm, mst, linux-kernel, virtualization, pasic,
pbonzini, dgilbert
In-Reply-To: <1546585913-34804-2-git-send-email-wei.w.wang@intel.com>
On Fri, 4 Jan 2019 15:11:52 +0800
Wei Wang <wei.w.wang@intel.com> wrote:
> virtio-ccw has deadlock issues with reading the config space inside the
> interrupt context, so we tweak the virtballoon_changed implementation
> by moving the config read operations into the related workqueue contexts.
> The config_read_bitmap is used as a flag to the workqueue callbacks
> about the related config fields that need to be read.
>
> Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Signed-off-by: Wei Wang <wei.w.wang@intel.com>
> ---
> drivers/virtio/virtio_balloon.c | 81 +++++++++++++++++++++++++++--------------
> 1 file changed, 53 insertions(+), 28 deletions(-)
>
(...)
> @@ -77,6 +81,8 @@ struct virtio_balloon {
> /* Prevent updating balloon when it is being canceled. */
> spinlock_t stop_update_lock;
> bool stop_update;
> + /* Bitmap to indicate if reading the related config fields are needed */
s/are/is/
> + unsigned long config_read_bitmap;
>
> /* The list of allocated free pages, waiting to be given back to mm */
> struct list_head free_page_list;
(...)
Bitmap handling looks sane to me.
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply
* Re: [PULL 0/1] virtio-ccw: one more patch for 4.21
From: Cornelia Huck @ 2019-01-04 9:56 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Halil Pasic, linux-s390, kvm, virtualization
In-Reply-To: <20190103115056-mutt-send-email-mst@kernel.org>
On Thu, 3 Jan 2019 11:51:45 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jan 03, 2019 at 11:50:22AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Jan 03, 2019 at 09:55:07AM +0100, Cornelia Huck wrote:
> > > On Wed, 19 Dec 2018 11:14:13 +0100
> > > Cornelia Huck <cohuck@redhat.com> wrote:
> > >
> > > > The following changes since commit 37d1246af2d530710cf5822d2845774f01c03b22:
> > > >
> > > > virtio_net: bulk free tx skbs (2018-12-06 14:28:39 -0500)
> > > >
> > > > are available in the Git repository at:
> > > >
> > > > git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git tags/virtio-ccw-20181219
> > > >
> > > > for you to fetch changes up to 6fb95ef30707f1b2fcaea8d6dc873055e0460b1a:
> > > >
> > > > virtio-ccw: diag 500 may return a negative cookie (2018-12-19 11:01:57 +0100)
> > > >
> > > > ----------------------------------------------------------------
> > > > One small documentation fix for 4.21.
> > > >
> > > > ----------------------------------------------------------------
> > > >
> > > > Cornelia Huck (1):
> > > > virtio-ccw: diag 500 may return a negative cookie
> > > >
> > > > Documentation/virtual/kvm/s390-diag.txt | 3 ++-
> > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > >
> > > ping?
> >
> > Sorry holidays :)
> >
> > I'll pick this up for the next pull.
>
> On a related note I'm not yet set up to handle pull requests,
> I'd prefer just patches for now.
Thanks for letting me know; I presume you can simply pick up the patch
as I sent it along in the pull request as well.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox