[Qemu-devel] Any topics for today's MTTCG sync-up call?

[Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

Hi,

It's been a while since we've actually held the call. Here are some
things that might be worth discussing:

Soft Freeze
===========

This cycles soft-freeze is upon us. QHT has already been merged this
cycle which is a win. I've taken some of the lock contention patches
from my base enabling patches which improve the situation for linux-user
mode which I hope to get accepted this cycle (as they have been on the
list before and just need minor tweaks).

Are there any other patches that meet the soft-freeze criteria worth
trying to merge this cycle?

Quiescent Work
==============

Sergey has posted a series of patches that attempt to draw together all
the requirements of the various safe work patches into a common feature
that can be meet all the various use cases we have. It comes with a
thread safe tb_flush implementation although obviously there are other
uses in LL/SC and system emulation tasks. Is everyone happy with the
features it provides?

Atomics
=======

Emilio has posted his set of patches for implementing atomics in a
thread safe manner. So far I've only had a chance to glance over it but
it certainly has some interesting numbers. I suspect the next step will
be some serious benmarking between this and Alvise's upcoming re-base to
solve the race between an initial LL and the flushing of the TLB
entries.

As I understand it the LL/SC work will require some quiescent work to
complete before things can continue in a race free manner. The question
is if this cost is too high compared to Emilio's solution which suffers
from ABA but for practical purposes should be fine.

My intention is to build trees of both solutions on top of the base
patches and compare their performances on both real-world and artificial
work-loads.

Memory Ordering
===============

Pranith has posted a series of patches to the list the implement basic
memory ordering TCGOps. The current discussions centre mainly around the
best way to represent Acq/Rel semantics in an efficient manner.

Base-enabling patches
=====================

Since the posting of v3 I've had some feedback so I'll be update v4
while the other trees get reviewed before trying to build another
complete series (probably ARM based) and doing some benchmarking of the
system performance.

Any other topics that need discussion?

--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Hi,
>
> It's been a while since we've actually held the call. Here are some
> things that might be worth discussing:
<snip>
>
> Any other topics that need discussion?

I'll take that as a no then. Feel free to add any updates to this thread.

--
Alex Bennée

[Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

Hi,

We missed the last call (sorry I was travelling). Have we any topics we
would like to cover this week?

--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[alvise rigo]

Hi Alex,

I'm looking into the worries that Sergey issued in his review of the
last LL/SC series. The target is to reduce the TLB flushes by using an
exclusive history of dynamic length. I don't have anything ready yet
though.

Best regards,
alvise

On Mon, Jun 20, 2016 at 1:57 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Hi,
>
> We missed the last call (sorry I was travelling). Have we any topics we
> would like to cover this week?
>
> --
> Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

alvise rigo <a.rigo@virtualopensystems.com> writes:

> Hi Alex,
>
> I'm looking into the worries that Sergey issued in his review of the
> last LL/SC series. The target is to reduce the TLB flushes by using an
> exclusive history of dynamic length. I don't have anything ready yet
> though.

Are you also tackling the race condition and ensuring all flushes are
done before the critical work?

Sergey has posted an RFC for his "quiescent work" solution which is
worth looking at.

>
> Best regards,
> alvise
>
> On Mon, Jun 20, 2016 at 1:57 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Hi,
>>
>> We missed the last call (sorry I was travelling). Have we any topics we
>> would like to cover this week?
>>
>> --
>> Alex Bennée


--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[alvise rigo]

On Mon, Jun 20, 2016 at 4:12 PM, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> alvise rigo <a.rigo@virtualopensystems.com> writes:
>
> > Hi Alex,
> >
> > I'm looking into the worries that Sergey issued in his review of the
> > last LL/SC series. The target is to reduce the TLB flushes by using an
> > exclusive history of dynamic length. I don't have anything ready yet
> > though.
>
> Are you also tackling the race condition and ensuring all flushes are
> done before the critical work?
>

Yes, I'm am.


>
> Sergey has posted an RFC for his "quiescent work" solution which is
> worth looking at.
>

I will see if it fits well with my use case and let him know.

Regards,
alvise


>
> >
> > Best regards,
> > alvise
> >
> > On Mon, Jun 20, 2016 at 1:57 PM, Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >>
> >> Hi,
> >>
> >> We missed the last call (sorry I was travelling). Have we any topics we
> >> would like to cover this week?
> >>
> >> --
> >> Alex Bennée
>
>
> --
> Alex Bennée
>

[Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

Hi,

It's been a while since the last sync-up call. Have we got any topics to
discuss today?

Sergey and I (with a little Paolo) have spent some of last week delving
into the locking hierarchy w.r.t to tb_lock vs mmap_lock to see if there
is any simplification to be had. I'm not sure if this is a topic
conducive to a phone call instead of the mailing list but if others want
to discuss it we can add it as an agenda item.

We also have a new member of the team. Pranith has joined as a GSoC
student. He'll be looking at memory ordering with his first pass at the
problem looking to solve the store-after-load issues which do show up on
ARM-on-x86 (see my testcase).

Alvise, is there any help you need with the LL/SC stuff? The MTTCG aware
version has been taking some time so would it be worth sharing the
issues you have hit with the group?

Emilio, is there anything you want to add? I've been following the QHT
stuff which is a really positive addition which my v3 base patches is
based upon (making the hot-path non lock contended). Do you have
anything in the works above that?

Cheers,

--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[alvise rigo]

Hi Alex,

I finally solved the issue I had, the branch is working well as far as I
can say.
The work I will share, in addition to making the LL/SC work mttcg-aware,
extends the various TLB flushes calls with the query-based mechanism: the
requesting CPU queries the flushes to the target CPUs and wait them for
completion.

Sorry for the delay, I have been quite busy. I just need to polish some
commits, than (this week) I will share the branch.

Best regards,
alvise

On Mon, May 23, 2016 at 12:57 PM, Alex Bennée <alex.bennee@linaro.org>
wrote:

> Hi,
>
> It's been a while since the last sync-up call. Have we got any topics to
> discuss today?
>
> Sergey and I (with a little Paolo) have spent some of last week delving
> into the locking hierarchy w.r.t to tb_lock vs mmap_lock to see if there
> is any simplification to be had. I'm not sure if this is a topic
> conducive to a phone call instead of the mailing list but if others want
> to discuss it we can add it as an agenda item.
>
> We also have a new member of the team. Pranith has joined as a GSoC
> student. He'll be looking at memory ordering with his first pass at the
> problem looking to solve the store-after-load issues which do show up on
> ARM-on-x86 (see my testcase).
>
> Alvise, is there any help you need with the LL/SC stuff? The MTTCG aware
> version has been taking some time so would it be worth sharing the
> issues you have hit with the group?
>
> Emilio, is there anything you want to add? I've been following the QHT
> stuff which is a really positive addition which my v3 base patches is
> based upon (making the hot-path non lock contended). Do you have
> anything in the works above that?
>
> Cheers,
>
> --
> Alex Bennée
>

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

alvise rigo <a.rigo@virtualopensystems.com> writes:

> Hi Alex,
>
> I finally solved the issue I had, the branch is working well as far as I
> can say.
> The work I will share, in addition to making the LL/SC work mttcg-aware,
> extends the various TLB flushes calls with the query-based mechanism: the
> requesting CPU queries the flushes to the target CPUs and wait them for
> completion.
>
> Sorry for the delay, I have been quite busy. I just need to polish some
> commits, than (this week) I will share the branch.

Thanks for the update. It sounds like we don't need to add to that on
the call.

Anyone else have something they want to discuss?

>
> Best regards,
> alvise
>
> On Mon, May 23, 2016 at 12:57 PM, Alex Bennée <alex.bennee@linaro.org>
> wrote:
>
>> Hi,
>>
>> It's been a while since the last sync-up call. Have we got any topics to
>> discuss today?
>>
>> Sergey and I (with a little Paolo) have spent some of last week delving
>> into the locking hierarchy w.r.t to tb_lock vs mmap_lock to see if there
>> is any simplification to be had. I'm not sure if this is a topic
>> conducive to a phone call instead of the mailing list but if others want
>> to discuss it we can add it as an agenda item.
>>
>> We also have a new member of the team. Pranith has joined as a GSoC
>> student. He'll be looking at memory ordering with his first pass at the
>> problem looking to solve the store-after-load issues which do show up on
>> ARM-on-x86 (see my testcase).
>>
>> Alvise, is there any help you need with the LL/SC stuff? The MTTCG aware
>> version has been taking some time so would it be worth sharing the
>> issues you have hit with the group?
>>
>> Emilio, is there anything you want to add? I've been following the QHT
>> stuff which is a really positive addition which my v3 base patches is
>> based upon (making the hot-path non lock contended). Do you have
>> anything in the works above that?
>>
>> Cheers,
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Claudio Fontana]

On 23.05.2016 14:47, Alex Bennée wrote:
> 
> alvise rigo <a.rigo@virtualopensystems.com> writes:
> 
>> Hi Alex,
>>
>> I finally solved the issue I had, the branch is working well as far as I
>> can say.
>> The work I will share, in addition to making the LL/SC work mttcg-aware,
>> extends the various TLB flushes calls with the query-based mechanism: the
>> requesting CPU queries the flushes to the target CPUs and wait them for
>> completion.
>>
>> Sorry for the delay, I have been quite busy. I just need to polish some
>> commits, than (this week) I will share the branch.
> 
> Thanks for the update. It sounds like we don't need to add to that on
> the call.
> 
> Anyone else have something they want to discuss?

Hi, at some point in the past there was a set of performance benchmarks which were showing the improvements using mttcg,
is there some update on that?

Thanks,

Claudio

> 
>>
>> Best regards,
>> alvise
>>
>> On Mon, May 23, 2016 at 12:57 PM, Alex Bennée <alex.bennee@linaro.org>
>> wrote:
>>
>>> Hi,
>>>
>>> It's been a while since the last sync-up call. Have we got any topics to
>>> discuss today?
>>>
>>> Sergey and I (with a little Paolo) have spent some of last week delving
>>> into the locking hierarchy w.r.t to tb_lock vs mmap_lock to see if there
>>> is any simplification to be had. I'm not sure if this is a topic
>>> conducive to a phone call instead of the mailing list but if others want
>>> to discuss it we can add it as an agenda item.
>>>
>>> We also have a new member of the team. Pranith has joined as a GSoC
>>> student. He'll be looking at memory ordering with his first pass at the
>>> problem looking to solve the store-after-load issues which do show up on
>>> ARM-on-x86 (see my testcase).
>>>
>>> Alvise, is there any help you need with the LL/SC stuff? The MTTCG aware
>>> version has been taking some time so would it be worth sharing the
>>> issues you have hit with the group?
>>>
>>> Emilio, is there anything you want to add? I've been following the QHT
>>> stuff which is a really positive addition which my v3 base patches is
>>> based upon (making the hot-path non lock contended). Do you have
>>> anything in the works above that?
>>>
>>> Cheers,
>>>
>>> --
>>> Alex Bennée
>>>
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Alex Bennée]

Claudio Fontana <claudio.fontana@huawei.com> writes:

> On 23.05.2016 14:47, Alex Bennée wrote:
>>
>> alvise rigo <a.rigo@virtualopensystems.com> writes:
>>
>>> Hi Alex,
>>>
>>> I finally solved the issue I had, the branch is working well as far as I
>>> can say.
>>> The work I will share, in addition to making the LL/SC work mttcg-aware,
>>> extends the various TLB flushes calls with the query-based mechanism: the
>>> requesting CPU queries the flushes to the target CPUs and wait them for
>>> completion.
>>>
>>> Sorry for the delay, I have been quite busy. I just need to polish some
>>> commits, than (this week) I will share the branch.
>>
>> Thanks for the update. It sounds like we don't need to add to that on
>> the call.
>>
>> Anyone else have something they want to discuss?
>
> Hi, at some point in the past there was a set of performance benchmarks which were showing the improvements using mttcg,
> is there some update on that?

I did some basic benchmarks on my last set of enabling patches that
showed a regression in performance for the MTTCG case. This was mostly
due to lock contention in the hot-path (holding lock although not
generating or patching TBs). This had been skipped in Fred's tree
because of the performance impact but wasn't "safe" hence dropped from
the last version.

Since then Emilio's QHT work has made having an un-locked hot-path both
safe and efficient and we are back to having a positive impact from
enabling additional threads. I have been building up a suite of
benchmark tests but I haven't done a full benchmark run on the latest
code. I will do a run when I'm ready to post my next iteration of the
patches.

For a very rough and ready comparison:

14:11 alex@zen/x86_64  [kvm-unit-tests-32bit.git/mttcg/current-tests-v5] >time ./arm/run ./arm/locking-test.flat -smp 4 -tcg mttcg=off
/home/alex/lsrc/qemu/qemu.git/arm-softmmu/qemu-system-arm -machine virt,accel=tcg -cpu cortex-a15 -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -display none -serial stdio -kernel ./arm/locking-test.flat -smp 4 -tcg mttcg=off
CPU1: online and ++ing
CPU2: online and ++ing
CPU3: online and ++ing
CPU0: online and ++ing
CPU1: Done, 10000000 incs
CPU2: Done, 10000000 incs
CPU3: Done, 10000000 incs
CPU0: Done, 10000000 incs
XPASS: total incs 40000000

SUMMARY: 1 tests, 1 unexpected failures

real    0m14.399s
user    0m14.368s
sys     0m0.020s
14:14 alex@zen/x86_64  [kvm-unit-tests-32bit.git/mttcg/current-tests-v5] >time ./arm/run ./arm/locking-test.flat -smp 4 -tcg mttcg=on
/home/alex/lsrc/qemu/qemu.git/arm-softmmu/qemu-system-arm -machine virt,accel=tcg -cpu cortex-a15 -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -display none -serial stdio -kernel ./arm/locking-test.flat -smp 4 -tcg mttcg=on
CPU1: online and ++ing
CPU2: online and ++ing
CPU3: online and ++ing
CPU0: online and ++ing
CPU2: Done, 10000000 incs
CPU0: Done, 10000000 incs
CPU3: Done, 10000000 incs
CPU1: Done, 10000000 incs
XFAIL: total incs 34052657

SUMMARY: 1 tests, 0 unexpected failures, 1 expected failures

real    0m5.089s
user    0m19.712s
sys     0m0.028s

Note the second test is expected to fail as there is no locking on the
increment. It only flukes it in -tcg mttcg=off mode because of the round
robin scheduling ;-)

>
> Thanks,
>
> Claudio
>
>>
>>>
>>> Best regards,
>>> alvise
>>>
>>> On Mon, May 23, 2016 at 12:57 PM, Alex Bennée <alex.bennee@linaro.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> It's been a while since the last sync-up call. Have we got any topics to
>>>> discuss today?
>>>>
>>>> Sergey and I (with a little Paolo) have spent some of last week delving
>>>> into the locking hierarchy w.r.t to tb_lock vs mmap_lock to see if there
>>>> is any simplification to be had. I'm not sure if this is a topic
>>>> conducive to a phone call instead of the mailing list but if others want
>>>> to discuss it we can add it as an agenda item.
>>>>
>>>> We also have a new member of the team. Pranith has joined as a GSoC
>>>> student. He'll be looking at memory ordering with his first pass at the
>>>> problem looking to solve the store-after-load issues which do show up on
>>>> ARM-on-x86 (see my testcase).
>>>>
>>>> Alvise, is there any help you need with the LL/SC stuff? The MTTCG aware
>>>> version has been taking some time so would it be worth sharing the
>>>> issues you have hit with the group?
>>>>
>>>> Emilio, is there anything you want to add? I've been following the QHT
>>>> stuff which is a really positive addition which my v3 base patches is
>>>> based upon (making the hot-path non lock contended). Do you have
>>>> anything in the works above that?
>>>>
>>>> Cheers,
>>>>
>>>> --
>>>> Alex Bennée
>>>>
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Emilio G. Cota]

On Mon, May 23, 2016 at 14:57:12 +0200, Claudio Fontana wrote:
> Hi, at some point in the past there was a set of performance benchmarks
> which were showing the improvements using mttcg, is there some update
> on that?

Any scalable parallel workload should do.

I've used the C/C++ benchmarks in synchrobench[1] extensively for
scalability benchmarking. It's not cross-compile friendly though, so for
ARM I only managed to compile a subset of its benchmarks.

[1] https://github.com/gramoli/synchrobench

		Emilio

Re: [Qemu-devel] Any topics for today's MTTCG sync-up call?

[Emilio G. Cota]

On Mon, May 23, 2016 at 11:57:26 +0100, Alex Bennée wrote:
> Emilio, is there anything you want to add? I've been following the QHT
> stuff which is a really positive addition which my v3 base patches is
> based upon (making the hot-path non lock contended). Do you have
> anything in the works above that?

- I'll submit a new v6 of the qht patches once v5 gets more reviews.
  The changes so far are very minor so I take that as a sign that the
  patchset will be merged soon.

- I'm also preparing a patch to leverage the host's cmpxchg for atomic
  instruction emulation, starting with i386 guests.

		Emilio