Ask help for litmus tests

Ask help for litmus tests

[Junchang Wang]

Hi Akira,

Thanks for your email about litmus tests a few weeks ago. I can
successfully run litmus tests on all three servers I can touch (The
configurations of the servers are listed below). Everything goes well
except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
never trigger on all of my servers. I understand that these assertions
trigger only if the speeds of propagating writing x to different CPU
cores varies a lot. So I have tried different litmus thread placement
strategies by adjusting the affinity setting of litmus7. For example:
$ litmus7 -affinity incr1 ... ...
$ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6

But it seems my effort doesn't work; the exist assertions never
trigger. Did I miss something? What's the right/possible configuration
to trigger these assertions. I do know you might be very busy. But if
possible, please give me some hints. These two tests seems very
interesting :-). Thanks in advance.

Regards,
--Junchang

===
Three servers I can touch:
  1. One PPC server which consists of 8 cores. I don's know hardware
details (e.g. which cores share the same LLC or write buffer) because
tools such as dmidecode don't work.
  2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
  3. One Intel server with two Xeon E5-2630 CPUs, each of which
consists of 12 cores (with Hyper-Threading).

Re: Ask help for litmus tests

[Akira Yokosawa]

On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> Hi Akira,
> 
> Thanks for your email about litmus tests a few weeks ago. I can
> successfully run litmus tests on all three servers I can touch (The
> configurations of the servers are listed below).

Hi Junchang,

Glad to know it helped you!

>                                                  Everything goes well
> except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> never trigger on all of my servers. I understand that these assertions
> trigger only if the speeds of propagating writing x to different CPU
> cores varies a lot. So I have tried different litmus thread placement
> strategies by adjusting the affinity setting of litmus7. For example:
> $ litmus7 -affinity incr1 ... ...
> $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6

By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.

One of the results reads:

----------------------------------------------------------
Test C-WRC+o+o-data-o+o-rmb-o Allowed
Histogram (6 states)
60327103:>1:r1=0; 2:r2=0; 2:r3=0;
22740026:>1:r1=1; 2:r2=0; 2:r3=0;
5       *>1:r1=1; 2:r2=1; 2:r3=0;
51117838:>1:r1=0; 2:r2=0; 2:r3=1;
55552319:>1:r1=1; 2:r2=0; 2:r3=1;
10262709:>1:r1=1; 2:r2=1; 2:r3=1;
Ok

Witnesses
Positive: 5, Negative: 199999995
Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
Hash=43057833028631b2f87eb65fe95c0ba2
Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
Time C-WRC+o+o-data-o+o-rmb-o 68.34
----------------------------------------------------------

In theory, litmus7 can also trigger the "exists" clause, but it would
require a very long runtime.  Also please refer to the Answer to
Quick Quiz 15.24. (You will find my name is mentioned there.)

As Armv8 architecture is other-multicopy-atomic, this "exists" clause
should never trigger, I suppose.

As for C-CCIRIW+o+o+o-o+o-o, herd7 says

----------------------------------------------------------
Test C-CCIRIW+o+o+o-o+o-o Allowed
States 47
2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
[...]
2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
No
Witnesses
Positive: 0 Negative: 72
Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
Time C-CCIRIW+o+o+o-o+o-o 0.06
Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
----------------------------------------------------------

It says "Never", so if you could ever trigger the condition on a
real machine, it would've meant something wrong in the implementation
on the platform side (cache coherency).

Hope this helps.

       Thanks, Akira

> 
> But it seems my effort doesn't work; the exist assertions never
> trigger. Did I miss something? What's the right/possible configuration
> to trigger these assertions. I do know you might be very busy. But if
> possible, please give me some hints. These two tests seems very
> interesting :-). Thanks in advance.
> 
> Regards,
> --Junchang
> 
> ===
> Three servers I can touch:
>   1. One PPC server which consists of 8 cores. I don's know hardware
> details (e.g. which cores share the same LLC or write buffer) because
> tools such as dmidecode don't work.
>   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
>   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> consists of 12 cores (with Hyper-Threading).
> 

Re: Ask help for litmus tests

[Junchang Wang]

On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
>
> On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> > Hi Akira,
> >
> > Thanks for your email about litmus tests a few weeks ago. I can
> > successfully run litmus tests on all three servers I can touch (The
> > configurations of the servers are listed below).
>
> Hi Junchang,
>
> Glad to know it helped you!
>
> >                                                  Everything goes well
> > except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> > C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> > never trigger on all of my servers. I understand that these assertions
> > trigger only if the speeds of propagating writing x to different CPU
> > cores varies a lot. So I have tried different litmus thread placement
> > strategies by adjusting the affinity setting of litmus7. For example:
> > $ litmus7 -affinity incr1 ... ...
> > $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
>
> By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
>
> One of the results reads:
>
> ----------------------------------------------------------
> Test C-WRC+o+o-data-o+o-rmb-o Allowed
> Histogram (6 states)
> 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> Ok
>
> Witnesses
> Positive: 5, Negative: 199999995
> Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> Hash=43057833028631b2f87eb65fe95c0ba2
> Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> Time C-WRC+o+o-data-o+o-rmb-o 68.34
> ----------------------------------------------------------
>
> In theory, litmus7 can also trigger the "exists" clause, but it would
> require a very long runtime.  Also please refer to the Answer to
> Quick Quiz 15.24. (You will find my name is mentioned there.)
>

Hi Akira,

I will try a long runtime. If that doesn't work, I suppose I need to
wait until I get another PPC server which I can physically touch.
Thanks a lot.

> As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> should never trigger, I suppose.
>

I'm a bit confused here. Perfbook says "Therefore, a
non-multicopy-atomic platform can have a store reach different threads
at different times." (Page 277 the first paragraph). So my
understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
message of writing y could reach P2 before the message of writing x
could reach there, and hence the "exists" clause could trigger even if
on an ARM server. Is that correct? Or did I misunderstand something?

Perfbook says "Most CPU vendors interested in providing multicopy
atomicity have therefore instead provided the slightly weaker
other-multicopy atomicity".(Page 277 the first paragraph). My
understanding is that "Most CPU vendors" here implies PPC, ARM, and
x86. In other words, all of the servers I can touch adopt
other-multicopy-atomic? Is that correct?


Thanks,
--Junchang

> As for C-CCIRIW+o+o+o-o+o-o, herd7 says
>
> ----------------------------------------------------------
> Test C-CCIRIW+o+o+o-o+o-o Allowed
> States 47
> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> [...]
> 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> No
> Witnesses
> Positive: 0 Negative: 72
> Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> Time C-CCIRIW+o+o+o-o+o-o 0.06
> Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> ----------------------------------------------------------
>
> It says "Never", so if you could ever trigger the condition on a
> real machine, it would've meant something wrong in the implementation
> on the platform side (cache coherency).
>
> Hope this helps.
>
>        Thanks, Akira
>
> >
> > But it seems my effort doesn't work; the exist assertions never
> > trigger. Did I miss something? What's the right/possible configuration
> > to trigger these assertions. I do know you might be very busy. But if
> > possible, please give me some hints. These two tests seems very
> > interesting :-). Thanks in advance.
> >
> > Regards,
> > --Junchang
> >
> > ===
> > Three servers I can touch:
> >   1. One PPC server which consists of 8 cores. I don's know hardware
> > details (e.g. which cores share the same LLC or write buffer) because
> > tools such as dmidecode don't work.
> >   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> >   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> > consists of 12 cores (with Hyper-Threading).
> >
>

Re: Ask help for litmus tests

[Paul E. McKenney]

On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
> On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
> >
> > On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> > > Hi Akira,
> > >
> > > Thanks for your email about litmus tests a few weeks ago. I can
> > > successfully run litmus tests on all three servers I can touch (The
> > > configurations of the servers are listed below).
> >
> > Hi Junchang,
> >
> > Glad to know it helped you!
> >
> > >                                                  Everything goes well
> > > except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> > > C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> > > never trigger on all of my servers. I understand that these assertions
> > > trigger only if the speeds of propagating writing x to different CPU
> > > cores varies a lot. So I have tried different litmus thread placement
> > > strategies by adjusting the affinity setting of litmus7. For example:
> > > $ litmus7 -affinity incr1 ... ...
> > > $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
> >
> > By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
> >
> > One of the results reads:
> >
> > ----------------------------------------------------------
> > Test C-WRC+o+o-data-o+o-rmb-o Allowed
> > Histogram (6 states)
> > 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> > 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> > 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> > 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> > 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> > 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> > Ok
> >
> > Witnesses
> > Positive: 5, Negative: 199999995
> > Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> > Hash=43057833028631b2f87eb65fe95c0ba2
> > Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> > Time C-WRC+o+o-data-o+o-rmb-o 68.34
> > ----------------------------------------------------------
> >
> > In theory, litmus7 can also trigger the "exists" clause, but it would
> > require a very long runtime.  Also please refer to the Answer to
> > Quick Quiz 15.24. (You will find my name is mentioned there.)
> >
> 
> Hi Akira,
> 
> I will try a long runtime. If that doesn't work, I suppose I need to
> wait until I get another PPC server which I can physically touch.
> Thanks a lot.
> 
> > As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> > should never trigger, I suppose.
> >
> 
> I'm a bit confused here. Perfbook says "Therefore, a
> non-multicopy-atomic platform can have a store reach different threads
> at different times." (Page 277 the first paragraph). So my
> understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
> message of writing y could reach P2 before the message of writing x
> could reach there, and hence the "exists" clause could trigger even if
> on an ARM server. Is that correct? Or did I misunderstand something?
> 
> Perfbook says "Most CPU vendors interested in providing multicopy
> atomicity have therefore instead provided the slightly weaker
> other-multicopy atomicity".(Page 277 the first paragraph). My
> understanding is that "Most CPU vendors" here implies PPC, ARM, and
> x86. In other words, all of the servers I can touch adopt
> other-multicopy-atomic? Is that correct?

ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
at all.

							Thanx, Paul

> Thanks,
> --Junchang
> 
> > As for C-CCIRIW+o+o+o-o+o-o, herd7 says
> >
> > ----------------------------------------------------------
> > Test C-CCIRIW+o+o+o-o+o-o Allowed
> > States 47
> > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> > 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> > [...]
> > 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> > No
> > Witnesses
> > Positive: 0 Negative: 72
> > Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> > Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> > Time C-CCIRIW+o+o+o-o+o-o 0.06
> > Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> > ----------------------------------------------------------
> >
> > It says "Never", so if you could ever trigger the condition on a
> > real machine, it would've meant something wrong in the implementation
> > on the platform side (cache coherency).
> >
> > Hope this helps.
> >
> >        Thanks, Akira
> >
> > >
> > > But it seems my effort doesn't work; the exist assertions never
> > > trigger. Did I miss something? What's the right/possible configuration
> > > to trigger these assertions. I do know you might be very busy. But if
> > > possible, please give me some hints. These two tests seems very
> > > interesting :-). Thanks in advance.
> > >
> > > Regards,
> > > --Junchang
> > >
> > > ===
> > > Three servers I can touch:
> > >   1. One PPC server which consists of 8 cores. I don's know hardware
> > > details (e.g. which cores share the same LLC or write buffer) because
> > > tools such as dmidecode don't work.
> > >   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> > >   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> > > consists of 12 cores (with Hyper-Threading).
> > >
> >
> 

Re: Ask help for litmus tests

[Junchang Wang]

On Mon, Nov 5, 2018 at 11:41 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
>
> On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
> > On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
> > >
> > > On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> > > > Hi Akira,
> > > >
> > > > Thanks for your email about litmus tests a few weeks ago. I can
> > > > successfully run litmus tests on all three servers I can touch (The
> > > > configurations of the servers are listed below).
> > >
> > > Hi Junchang,
> > >
> > > Glad to know it helped you!
> > >
> > > >                                                  Everything goes well
> > > > except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> > > > C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> > > > never trigger on all of my servers. I understand that these assertions
> > > > trigger only if the speeds of propagating writing x to different CPU
> > > > cores varies a lot. So I have tried different litmus thread placement
> > > > strategies by adjusting the affinity setting of litmus7. For example:
> > > > $ litmus7 -affinity incr1 ... ...
> > > > $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
> > >
> > > By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
> > >
> > > One of the results reads:
> > >
> > > ----------------------------------------------------------
> > > Test C-WRC+o+o-data-o+o-rmb-o Allowed
> > > Histogram (6 states)
> > > 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> > > 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> > > 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> > > 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> > > 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> > > 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> > > Ok
> > >
> > > Witnesses
> > > Positive: 5, Negative: 199999995
> > > Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> > > Hash=43057833028631b2f87eb65fe95c0ba2
> > > Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> > > Time C-WRC+o+o-data-o+o-rmb-o 68.34
> > > ----------------------------------------------------------
> > >
> > > In theory, litmus7 can also trigger the "exists" clause, but it would
> > > require a very long runtime.  Also please refer to the Answer to
> > > Quick Quiz 15.24. (You will find my name is mentioned there.)
> > >
> >
> > Hi Akira,
> >
> > I will try a long runtime. If that doesn't work, I suppose I need to
> > wait until I get another PPC server which I can physically touch.
> > Thanks a lot.
> >
> > > As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> > > should never trigger, I suppose.
> > >
> >
> > I'm a bit confused here. Perfbook says "Therefore, a
> > non-multicopy-atomic platform can have a store reach different threads
> > at different times." (Page 277 the first paragraph). So my
> > understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
> > message of writing y could reach P2 before the message of writing x
> > could reach there, and hence the "exists" clause could trigger even if
> > on an ARM server. Is that correct? Or did I misunderstand something?
> >
> > Perfbook says "Most CPU vendors interested in providing multicopy
> > atomicity have therefore instead provided the slightly weaker
> > other-multicopy atomicity".(Page 277 the first paragraph). My
> > understanding is that "Most CPU vendors" here implies PPC, ARM, and
> > x86. In other words, all of the servers I can touch adopt
> > other-multicopy-atomic? Is that correct?
>
> ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
> fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
> at all.
>

Thanks a lot!!! This is the first time I know about this. How about we
add this as a footnote or something to help readers easier to get a
sense of the use of multicopy atomicity in practice? Does the
following look OK?

===

From 8e07ecd836e5b581d2953938a65a395208fe0af3 Mon Sep 17 00:00:00 2001
From: Junchang Wang <junchangwang@gmail.com>
Date: Tue, 6 Nov 2018 15:00:31 +0800
Subject: [PATCH] memorder: Clarify implementation details of multicopy
 atomicity in mainstream architectures

Reported-by: Junchang Wang <junchangwang@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
---
 memorder/memorder.tex | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/memorder/memorder.tex b/memorder/memorder.tex
index 5e74a91..5db4971 100644
--- a/memorder/memorder.tex
+++ b/memorder/memorder.tex
@@ -1901,7 +1901,9 @@ Most CPU vendors interested in providing
multicopy atomicity have therefore
 instead provided the slightly weaker
 \emph{other-multicopy atomicity}~\cite[Section B2.3]{ARMv8A:2017},
 which excludes the CPU doing a given store from the requirement that all
-CPUs agree on the order of all stores.
+CPUs agree on the order of all stores \footnote{As of late 2018, ARM and x86
+provide other-multicopy atomicity, IBM mainframe provides fully multicopy
+atomicity, and PPC does not provide multicopy atomicity at all.}.
 This means that if only a subset of CPUs are doing stores, the
 other CPUs will agree on the order of stores, hence the ``other''
 in ``other-multicopy atomicity''.
-- 
2.7.4

Thanks,
--Junchang


>                                                         Thanx, Paul
>
> > Thanks,
> > --Junchang
> >
> > > As for C-CCIRIW+o+o+o-o+o-o, herd7 says
> > >
> > > ----------------------------------------------------------
> > > Test C-CCIRIW+o+o+o-o+o-o Allowed
> > > States 47
> > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> > > 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> > > [...]
> > > 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> > > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> > > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> > > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> > > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> > > No
> > > Witnesses
> > > Positive: 0 Negative: 72
> > > Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> > > Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> > > Time C-CCIRIW+o+o+o-o+o-o 0.06
> > > Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> > > ----------------------------------------------------------
> > >
> > > It says "Never", so if you could ever trigger the condition on a
> > > real machine, it would've meant something wrong in the implementation
> > > on the platform side (cache coherency).
> > >
> > > Hope this helps.
> > >
> > >        Thanks, Akira
> > >
> > > >
> > > > But it seems my effort doesn't work; the exist assertions never
> > > > trigger. Did I miss something? What's the right/possible configuration
> > > > to trigger these assertions. I do know you might be very busy. But if
> > > > possible, please give me some hints. These two tests seems very
> > > > interesting :-). Thanks in advance.
> > > >
> > > > Regards,
> > > > --Junchang
> > > >
> > > > ===
> > > > Three servers I can touch:
> > > >   1. One PPC server which consists of 8 cores. I don's know hardware
> > > > details (e.g. which cores share the same LLC or write buffer) because
> > > > tools such as dmidecode don't work.
> > > >   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> > > >   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> > > > consists of 12 cores (with Hyper-Threading).
> > > >
> > >
> >
>

Re: Ask help for litmus tests

[Paul E. McKenney]

On Tue, Nov 06, 2018 at 03:12:26PM +0800, Junchang Wang wrote:
> On Mon, Nov 5, 2018 at 11:41 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> >
> > On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
> > > On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
> > > >
> > > > On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> > > > > Hi Akira,
> > > > >
> > > > > Thanks for your email about litmus tests a few weeks ago. I can
> > > > > successfully run litmus tests on all three servers I can touch (The
> > > > > configurations of the servers are listed below).
> > > >
> > > > Hi Junchang,
> > > >
> > > > Glad to know it helped you!
> > > >
> > > > >                                                  Everything goes well
> > > > > except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> > > > > C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> > > > > never trigger on all of my servers. I understand that these assertions
> > > > > trigger only if the speeds of propagating writing x to different CPU
> > > > > cores varies a lot. So I have tried different litmus thread placement
> > > > > strategies by adjusting the affinity setting of litmus7. For example:
> > > > > $ litmus7 -affinity incr1 ... ...
> > > > > $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
> > > >
> > > > By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
> > > >
> > > > One of the results reads:
> > > >
> > > > ----------------------------------------------------------
> > > > Test C-WRC+o+o-data-o+o-rmb-o Allowed
> > > > Histogram (6 states)
> > > > 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> > > > 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> > > > 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> > > > 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> > > > 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> > > > 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> > > > Ok
> > > >
> > > > Witnesses
> > > > Positive: 5, Negative: 199999995
> > > > Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> > > > Hash=43057833028631b2f87eb65fe95c0ba2
> > > > Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> > > > Time C-WRC+o+o-data-o+o-rmb-o 68.34
> > > > ----------------------------------------------------------
> > > >
> > > > In theory, litmus7 can also trigger the "exists" clause, but it would
> > > > require a very long runtime.  Also please refer to the Answer to
> > > > Quick Quiz 15.24. (You will find my name is mentioned there.)
> > > >
> > >
> > > Hi Akira,
> > >
> > > I will try a long runtime. If that doesn't work, I suppose I need to
> > > wait until I get another PPC server which I can physically touch.
> > > Thanks a lot.
> > >
> > > > As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> > > > should never trigger, I suppose.
> > > >
> > >
> > > I'm a bit confused here. Perfbook says "Therefore, a
> > > non-multicopy-atomic platform can have a store reach different threads
> > > at different times." (Page 277 the first paragraph). So my
> > > understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
> > > message of writing y could reach P2 before the message of writing x
> > > could reach there, and hence the "exists" clause could trigger even if
> > > on an ARM server. Is that correct? Or did I misunderstand something?
> > >
> > > Perfbook says "Most CPU vendors interested in providing multicopy
> > > atomicity have therefore instead provided the slightly weaker
> > > other-multicopy atomicity".(Page 277 the first paragraph). My
> > > understanding is that "Most CPU vendors" here implies PPC, ARM, and
> > > x86. In other words, all of the servers I can touch adopt
> > > other-multicopy-atomic? Is that correct?
> >
> > ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
> > fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
> > at all.
> >
> 
> Thanks a lot!!! This is the first time I know about this. How about we
> add this as a footnote or something to help readers easier to get a
> sense of the use of multicopy atomicity in practice? Does the
> following look OK?

It does look like a good addition, but it needs a couple of changes,
please see below.

> ===
> 
> >From 8e07ecd836e5b581d2953938a65a395208fe0af3 Mon Sep 17 00:00:00 2001
> From: Junchang Wang <junchangwang@gmail.com>
> Date: Tue, 6 Nov 2018 15:00:31 +0800
> Subject: [PATCH] memorder: Clarify implementation details of multicopy
>  atomicity in mainstream architectures
> 
> Reported-by: Junchang Wang <junchangwang@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>

This needs to be just:

Signed-off-by: Junchang Wang <junchangwang@gmail.com>

It is your patch, so it has your signed-off-by.

> ---
>  memorder/memorder.tex | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/memorder/memorder.tex b/memorder/memorder.tex
> index 5e74a91..5db4971 100644
> --- a/memorder/memorder.tex
> +++ b/memorder/memorder.tex
> @@ -1901,7 +1901,9 @@ Most CPU vendors interested in providing
> multicopy atomicity have therefore
>  instead provided the slightly weaker
>  \emph{other-multicopy atomicity}~\cite[Section B2.3]{ARMv8A:2017},
>  which excludes the CPU doing a given store from the requirement that all
> -CPUs agree on the order of all stores.
> +CPUs agree on the order of all stores \footnote{As of late 2018, ARM and x86
> +provide other-multicopy atomicity, IBM mainframe provides fully multicopy
> +atomicity, and PPC does not provide multicopy atomicity at all.}.

Please indent the footnote text so that it stands out more and please
remove the space character just before the "\footnote":

CPUs agree on the order of all stores\footnote{
	As of late 2018, ARM and x86 provide other-multicopy atomicity,
	IBM mainframe provides fully multicopy atomicity, and PPC does
	not provide multicopy atomicity at all.}.

Could you please also add a reference to:

	Table~\ref{\label{tab:memorder:Summary of Memory Ordering}?

If you update the patch as suggested and send it as a standalone patch,
I will accept it.

							Thanx, Paul

>  This means that if only a subset of CPUs are doing stores, the
>  other CPUs will agree on the order of stores, hence the ``other''
>  in ``other-multicopy atomicity''.
> -- 
> 2.7.4
> 
> Thanks,
> --Junchang
> 
> 
> >                                                         Thanx, Paul
> >
> > > Thanks,
> > > --Junchang
> > >
> > > > As for C-CCIRIW+o+o+o-o+o-o, herd7 says
> > > >
> > > > ----------------------------------------------------------
> > > > Test C-CCIRIW+o+o+o-o+o-o Allowed
> > > > States 47
> > > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> > > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> > > > 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> > > > 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> > > > [...]
> > > > 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> > > > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> > > > 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> > > > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> > > > 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> > > > No
> > > > Witnesses
> > > > Positive: 0 Negative: 72
> > > > Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> > > > Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> > > > Time C-CCIRIW+o+o+o-o+o-o 0.06
> > > > Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> > > > ----------------------------------------------------------
> > > >
> > > > It says "Never", so if you could ever trigger the condition on a
> > > > real machine, it would've meant something wrong in the implementation
> > > > on the platform side (cache coherency).
> > > >
> > > > Hope this helps.
> > > >
> > > >        Thanks, Akira
> > > >
> > > > >
> > > > > But it seems my effort doesn't work; the exist assertions never
> > > > > trigger. Did I miss something? What's the right/possible configuration
> > > > > to trigger these assertions. I do know you might be very busy. But if
> > > > > possible, please give me some hints. These two tests seems very
> > > > > interesting :-). Thanks in advance.
> > > > >
> > > > > Regards,
> > > > > --Junchang
> > > > >
> > > > > ===
> > > > > Three servers I can touch:
> > > > >   1. One PPC server which consists of 8 cores. I don's know hardware
> > > > > details (e.g. which cores share the same LLC or write buffer) because
> > > > > tools such as dmidecode don't work.
> > > > >   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> > > > >   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> > > > > consists of 12 cores (with Hyper-Threading).
> > > > >
> > > >
> > >
> >
> 

Re: Ask help for litmus tests

[Akira Yokosawa]

On 2018/11/06 06:36:10 -0800, Paul E. McKenney wrote:
> On Tue, Nov 06, 2018 at 03:12:26PM +0800, Junchang Wang wrote:
>> On Mon, Nov 5, 2018 at 11:41 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
>>>
>>> On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
>>>> On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
>>>>>
>>>>> On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
>>>>>> Hi Akira,
>>>>>>
>>>>>> Thanks for your email about litmus tests a few weeks ago. I can
>>>>>> successfully run litmus tests on all three servers I can touch (The
>>>>>> configurations of the servers are listed below).
>>>>>
>>>>> Hi Junchang,
>>>>>
>>>>> Glad to know it helped you!
>>>>>
>>>>>>                                                  Everything goes well
>>>>>> except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
>>>>>> C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
>>>>>> never trigger on all of my servers. I understand that these assertions
>>>>>> trigger only if the speeds of propagating writing x to different CPU
>>>>>> cores varies a lot. So I have tried different litmus thread placement
>>>>>> strategies by adjusting the affinity setting of litmus7. For example:
>>>>>> $ litmus7 -affinity incr1 ... ...
>>>>>> $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
>>>>>
>>>>> By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
>>>>>
>>>>> One of the results reads:
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Test C-WRC+o+o-data-o+o-rmb-o Allowed
>>>>> Histogram (6 states)
>>>>> 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
>>>>> 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
>>>>> 5       *>1:r1=1; 2:r2=1; 2:r3=0;
>>>>> 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
>>>>> 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
>>>>> 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
>>>>> Ok
>>>>>
>>>>> Witnesses
>>>>> Positive: 5, Negative: 199999995
>>>>> Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
>>>>> Hash=43057833028631b2f87eb65fe95c0ba2
>>>>> Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
>>>>> Time C-WRC+o+o-data-o+o-rmb-o 68.34
>>>>> ----------------------------------------------------------
>>>>>
>>>>> In theory, litmus7 can also trigger the "exists" clause, but it would
>>>>> require a very long runtime.  Also please refer to the Answer to
>>>>> Quick Quiz 15.24. (You will find my name is mentioned there.)
>>>>>
>>>>
>>>> Hi Akira,
>>>>
>>>> I will try a long runtime. If that doesn't work, I suppose I need to
>>>> wait until I get another PPC server which I can physically touch.
>>>> Thanks a lot.
>>>>
>>>>> As Armv8 architecture is other-multicopy-atomic, this "exists" clause
>>>>> should never trigger, I suppose.
>>>>>
>>>>
>>>> I'm a bit confused here. Perfbook says "Therefore, a
>>>> non-multicopy-atomic platform can have a store reach different threads
>>>> at different times." (Page 277 the first paragraph). So my
>>>> understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
>>>> message of writing y could reach P2 before the message of writing x
>>>> could reach there, and hence the "exists" clause could trigger even if
>>>> on an ARM server. Is that correct? Or did I misunderstand something?
>>>>
>>>> Perfbook says "Most CPU vendors interested in providing multicopy
>>>> atomicity have therefore instead provided the slightly weaker
>>>> other-multicopy atomicity".(Page 277 the first paragraph). My
>>>> understanding is that "Most CPU vendors" here implies PPC, ARM, and
>>>> x86. In other words, all of the servers I can touch adopt
>>>> other-multicopy-atomic? Is that correct?
>>>
>>> ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
>>> fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
>>> at all.
>>>
>>
>> Thanks a lot!!! This is the first time I know about this. How about we
>> add this as a footnote or something to help readers easier to get a
>> sense of the use of multicopy atomicity in practice? Does the
>> following look OK?
> 
> It does look like a good addition, but it needs a couple of changes,
> please see below.
> 
>> ===
>>
>> >From 8e07ecd836e5b581d2953938a65a395208fe0af3 Mon Sep 17 00:00:00 2001
>> From: Junchang Wang <junchangwang@gmail.com>
>> Date: Tue, 6 Nov 2018 15:00:31 +0800
>> Subject: [PATCH] memorder: Clarify implementation details of multicopy
>>  atomicity in mainstream architectures
>>
>> Reported-by: Junchang Wang <junchangwang@gmail.com>
>> Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
> 
> This needs to be just:
> 
> Signed-off-by: Junchang Wang <junchangwang@gmail.com>
> 
> It is your patch, so it has your signed-off-by.

But it came from Paul's reply. It might be a good idea to cite
the message at

	https://www.spinics.net/lists/perfbook/msg01952.html

in the commit log.

> 
>> ---
>>  memorder/memorder.tex | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/memorder/memorder.tex b/memorder/memorder.tex
>> index 5e74a91..5db4971 100644
>> --- a/memorder/memorder.tex
>> +++ b/memorder/memorder.tex
>> @@ -1901,7 +1901,9 @@ Most CPU vendors interested in providing
>> multicopy atomicity have therefore
>>  instead provided the slightly weaker
>>  \emph{other-multicopy atomicity}~\cite[Section B2.3]{ARMv8A:2017},
>>  which excludes the CPU doing a given store from the requirement that all
>> -CPUs agree on the order of all stores.
>> +CPUs agree on the order of all stores \footnote{As of late 2018, ARM and x86
>> +provide other-multicopy atomicity, IBM mainframe provides fully multicopy
>> +atomicity, and PPC does not provide multicopy atomicity at all.}.
> 
> Please indent the footnote text so that it stands out more and please
> remove the space character just before the "\footnote":
> 
> CPUs agree on the order of all stores\footnote{
> 	As of late 2018, ARM and x86 provide other-multicopy atomicity,
> 	IBM mainframe provides fully multicopy atomicity, and PPC does
> 	not provide multicopy atomicity at all.}.

I think the footnote should be just after the punctuation character, i.e.,

CPUs agree on the order of all stores.\footnote{
	As of late 2018, ARM and x86 provide other-multicopy atomicity,
	IBM mainframe provides fully multicopy atomicity, and PPC does
	not provide multicopy atomicity at all.}

> 
> Could you please also add a reference to:
> 
> 	Table~\ref{\label{tab:memorder:Summary of Memory Ordering}?

Did you mean:

 	Table~\ref{tab:memorder:Summary of Memory Ordering}?

Thanks, Akira

> 
> If you update the patch as suggested and send it as a standalone patch,
> I will accept it.
> 
> 							Thanx, Paul
> 
>>  This means that if only a subset of CPUs are doing stores, the
>>  other CPUs will agree on the order of stores, hence the ``other''
>>  in ``other-multicopy atomicity''.
>> -- 
>> 2.7.4
>>
>> Thanks,
>> --Junchang
>>
>>
>>>                                                         Thanx, Paul
>>>
>>>> Thanks,
>>>> --Junchang
>>>>
>>>>> As for C-CCIRIW+o+o+o-o+o-o, herd7 says
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Test C-CCIRIW+o+o+o-o+o-o Allowed
>>>>> States 47
>>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
>>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
>>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
>>>>> 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
>>>>> [...]
>>>>> 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
>>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
>>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
>>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
>>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
>>>>> No
>>>>> Witnesses
>>>>> Positive: 0 Negative: 72
>>>>> Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
>>>>> Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
>>>>> Time C-CCIRIW+o+o+o-o+o-o 0.06
>>>>> Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
>>>>> ----------------------------------------------------------
>>>>>
>>>>> It says "Never", so if you could ever trigger the condition on a
>>>>> real machine, it would've meant something wrong in the implementation
>>>>> on the platform side (cache coherency).
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>>        Thanks, Akira
>>>>>
>>>>>>
>>>>>> But it seems my effort doesn't work; the exist assertions never
>>>>>> trigger. Did I miss something? What's the right/possible configuration
>>>>>> to trigger these assertions. I do know you might be very busy. But if
>>>>>> possible, please give me some hints. These two tests seems very
>>>>>> interesting :-). Thanks in advance.
>>>>>>
>>>>>> Regards,
>>>>>> --Junchang
>>>>>>
>>>>>> ===
>>>>>> Three servers I can touch:
>>>>>>   1. One PPC server which consists of 8 cores. I don's know hardware
>>>>>> details (e.g. which cores share the same LLC or write buffer) because
>>>>>> tools such as dmidecode don't work.
>>>>>>   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
>>>>>>   3. One Intel server with two Xeon E5-2630 CPUs, each of which
>>>>>> consists of 12 cores (with Hyper-Threading).
>>>>>>
>>>>>
>>>>
>>>
>>
> 

Re: Ask help for litmus tests

[Paul E. McKenney]

On Wed, Nov 07, 2018 at 12:02:59AM +0900, Akira Yokosawa wrote:
> On 2018/11/06 06:36:10 -0800, Paul E. McKenney wrote:
> > On Tue, Nov 06, 2018 at 03:12:26PM +0800, Junchang Wang wrote:
> >> On Mon, Nov 5, 2018 at 11:41 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> >>>
> >>> On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
> >>>> On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
> >>>>>
> >>>>> On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> >>>>>> Hi Akira,
> >>>>>>
> >>>>>> Thanks for your email about litmus tests a few weeks ago. I can
> >>>>>> successfully run litmus tests on all three servers I can touch (The
> >>>>>> configurations of the servers are listed below).
> >>>>>
> >>>>> Hi Junchang,
> >>>>>
> >>>>> Glad to know it helped you!
> >>>>>
> >>>>>>                                                  Everything goes well
> >>>>>> except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> >>>>>> C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> >>>>>> never trigger on all of my servers. I understand that these assertions
> >>>>>> trigger only if the speeds of propagating writing x to different CPU
> >>>>>> cores varies a lot. So I have tried different litmus thread placement
> >>>>>> strategies by adjusting the affinity setting of litmus7. For example:
> >>>>>> $ litmus7 -affinity incr1 ... ...
> >>>>>> $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
> >>>>>
> >>>>> By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
> >>>>>
> >>>>> One of the results reads:
> >>>>>
> >>>>> ----------------------------------------------------------
> >>>>> Test C-WRC+o+o-data-o+o-rmb-o Allowed
> >>>>> Histogram (6 states)
> >>>>> 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> >>>>> 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> >>>>> 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> >>>>> 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> >>>>> 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> >>>>> 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> >>>>> Ok
> >>>>>
> >>>>> Witnesses
> >>>>> Positive: 5, Negative: 199999995
> >>>>> Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> >>>>> Hash=43057833028631b2f87eb65fe95c0ba2
> >>>>> Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> >>>>> Time C-WRC+o+o-data-o+o-rmb-o 68.34
> >>>>> ----------------------------------------------------------
> >>>>>
> >>>>> In theory, litmus7 can also trigger the "exists" clause, but it would
> >>>>> require a very long runtime.  Also please refer to the Answer to
> >>>>> Quick Quiz 15.24. (You will find my name is mentioned there.)
> >>>>>
> >>>>
> >>>> Hi Akira,
> >>>>
> >>>> I will try a long runtime. If that doesn't work, I suppose I need to
> >>>> wait until I get another PPC server which I can physically touch.
> >>>> Thanks a lot.
> >>>>
> >>>>> As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> >>>>> should never trigger, I suppose.
> >>>>>
> >>>>
> >>>> I'm a bit confused here. Perfbook says "Therefore, a
> >>>> non-multicopy-atomic platform can have a store reach different threads
> >>>> at different times." (Page 277 the first paragraph). So my
> >>>> understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
> >>>> message of writing y could reach P2 before the message of writing x
> >>>> could reach there, and hence the "exists" clause could trigger even if
> >>>> on an ARM server. Is that correct? Or did I misunderstand something?
> >>>>
> >>>> Perfbook says "Most CPU vendors interested in providing multicopy
> >>>> atomicity have therefore instead provided the slightly weaker
> >>>> other-multicopy atomicity".(Page 277 the first paragraph). My
> >>>> understanding is that "Most CPU vendors" here implies PPC, ARM, and
> >>>> x86. In other words, all of the servers I can touch adopt
> >>>> other-multicopy-atomic? Is that correct?
> >>>
> >>> ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
> >>> fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
> >>> at all.
> >>>
> >>
> >> Thanks a lot!!! This is the first time I know about this. How about we
> >> add this as a footnote or something to help readers easier to get a
> >> sense of the use of multicopy atomicity in practice? Does the
> >> following look OK?
> > 
> > It does look like a good addition, but it needs a couple of changes,
> > please see below.
> > 
> >> ===
> >>
> >> >From 8e07ecd836e5b581d2953938a65a395208fe0af3 Mon Sep 17 00:00:00 2001
> >> From: Junchang Wang <junchangwang@gmail.com>
> >> Date: Tue, 6 Nov 2018 15:00:31 +0800
> >> Subject: [PATCH] memorder: Clarify implementation details of multicopy
> >>  atomicity in mainstream architectures
> >>
> >> Reported-by: Junchang Wang <junchangwang@gmail.com>
> >> Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
> > 
> > This needs to be just:
> > 
> > Signed-off-by: Junchang Wang <junchangwang@gmail.com>
> > 
> > It is your patch, so it has your signed-off-by.
> 
> But it came from Paul's reply. It might be a good idea to cite
> the message at
> 
> 	https://www.spinics.net/lists/perfbook/msg01952.html
> 
> in the commit log.

That would be a nice addition, but not essential in this case.

> >> ---
> >>  memorder/memorder.tex | 4 +++-
> >>  1 file changed, 3 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/memorder/memorder.tex b/memorder/memorder.tex
> >> index 5e74a91..5db4971 100644
> >> --- a/memorder/memorder.tex
> >> +++ b/memorder/memorder.tex
> >> @@ -1901,7 +1901,9 @@ Most CPU vendors interested in providing
> >> multicopy atomicity have therefore
> >>  instead provided the slightly weaker
> >>  \emph{other-multicopy atomicity}~\cite[Section B2.3]{ARMv8A:2017},
> >>  which excludes the CPU doing a given store from the requirement that all
> >> -CPUs agree on the order of all stores.
> >> +CPUs agree on the order of all stores \footnote{As of late 2018, ARM and x86
> >> +provide other-multicopy atomicity, IBM mainframe provides fully multicopy
> >> +atomicity, and PPC does not provide multicopy atomicity at all.}.
> > 
> > Please indent the footnote text so that it stands out more and please
> > remove the space character just before the "\footnote":
> > 
> > CPUs agree on the order of all stores\footnote{
> > 	As of late 2018, ARM and x86 provide other-multicopy atomicity,
> > 	IBM mainframe provides fully multicopy atomicity, and PPC does
> > 	not provide multicopy atomicity at all.}.
> 
> I think the footnote should be just after the punctuation character, i.e.,
> 
> CPUs agree on the order of all stores.\footnote{
> 	As of late 2018, ARM and x86 provide other-multicopy atomicity,
> 	IBM mainframe provides fully multicopy atomicity, and PPC does
> 	not provide multicopy atomicity at all.}

You are right, I did miss that.

> > Could you please also add a reference to:
> > 
> > 	Table~\ref{\label{tab:memorder:Summary of Memory Ordering}?
> 
> Did you mean:
> 
>  	Table~\ref{tab:memorder:Summary of Memory Ordering}?

Indeed I did, thank you!

							Thanx, Paul

> Thanks, Akira
> 
> > 
> > If you update the patch as suggested and send it as a standalone patch,
> > I will accept it.
> > 
> > 							Thanx, Paul
> > 
> >>  This means that if only a subset of CPUs are doing stores, the
> >>  other CPUs will agree on the order of stores, hence the ``other''
> >>  in ``other-multicopy atomicity''.
> >> -- 
> >> 2.7.4
> >>
> >> Thanks,
> >> --Junchang
> >>
> >>
> >>>                                                         Thanx, Paul
> >>>
> >>>> Thanks,
> >>>> --Junchang
> >>>>
> >>>>> As for C-CCIRIW+o+o+o-o+o-o, herd7 says
> >>>>>
> >>>>> ----------------------------------------------------------
> >>>>> Test C-CCIRIW+o+o+o-o+o-o Allowed
> >>>>> States 47
> >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> >>>>> 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> >>>>> [...]
> >>>>> 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> >>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> >>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> >>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> >>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> >>>>> No
> >>>>> Witnesses
> >>>>> Positive: 0 Negative: 72
> >>>>> Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> >>>>> Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> >>>>> Time C-CCIRIW+o+o+o-o+o-o 0.06
> >>>>> Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> >>>>> ----------------------------------------------------------
> >>>>>
> >>>>> It says "Never", so if you could ever trigger the condition on a
> >>>>> real machine, it would've meant something wrong in the implementation
> >>>>> on the platform side (cache coherency).
> >>>>>
> >>>>> Hope this helps.
> >>>>>
> >>>>>        Thanks, Akira
> >>>>>
> >>>>>>
> >>>>>> But it seems my effort doesn't work; the exist assertions never
> >>>>>> trigger. Did I miss something? What's the right/possible configuration
> >>>>>> to trigger these assertions. I do know you might be very busy. But if
> >>>>>> possible, please give me some hints. These two tests seems very
> >>>>>> interesting :-). Thanks in advance.
> >>>>>>
> >>>>>> Regards,
> >>>>>> --Junchang
> >>>>>>
> >>>>>> ===
> >>>>>> Three servers I can touch:
> >>>>>>   1. One PPC server which consists of 8 cores. I don's know hardware
> >>>>>> details (e.g. which cores share the same LLC or write buffer) because
> >>>>>> tools such as dmidecode don't work.
> >>>>>>   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> >>>>>>   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> >>>>>> consists of 12 cores (with Hyper-Threading).
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> > 
> 

Re: Ask help for litmus tests

[Junchang Wang]

On Tue, Nov 6, 2018 at 11:13 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
>
> On Wed, Nov 07, 2018 at 12:02:59AM +0900, Akira Yokosawa wrote:
> > On 2018/11/06 06:36:10 -0800, Paul E. McKenney wrote:
> > > On Tue, Nov 06, 2018 at 03:12:26PM +0800, Junchang Wang wrote:
> > >> On Mon, Nov 5, 2018 at 11:41 AM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> > >>>
> > >>> On Mon, Nov 05, 2018 at 11:08:15AM +0800, Junchang Wang wrote:
> > >>>> On Sun, Nov 4, 2018 at 10:21 PM Akira Yokosawa <akiyks@gmail.com> wrote:
> > >>>>>
> > >>>>> On 2018/11/04 20:14:50 +0800, Junchang Wang wrote:
> > >>>>>> Hi Akira,
> > >>>>>>
> > >>>>>> Thanks for your email about litmus tests a few weeks ago. I can
> > >>>>>> successfully run litmus tests on all three servers I can touch (The
> > >>>>>> configurations of the servers are listed below).
> > >>>>>
> > >>>>> Hi Junchang,
> > >>>>>
> > >>>>> Glad to know it helped you!
> > >>>>>
> > >>>>>>                                                  Everything goes well
> > >>>>>> except litmus tests C-CCIRIW+o+o+o-o+o-o (Listing 15.15) and
> > >>>>>> C-WRC+o+o-data-o+o-rmb-o (Listing 15.16). The "exists" assertions
> > >>>>>> never trigger on all of my servers. I understand that these assertions
> > >>>>>> trigger only if the speeds of propagating writing x to different CPU
> > >>>>>> cores varies a lot. So I have tried different litmus thread placement
> > >>>>>> strategies by adjusting the affinity setting of litmus7. For example:
> > >>>>>> $ litmus7 -affinity incr1 ... ...
> > >>>>>> $ ./C-WRC+o+o-data-o+o-rmb-o.exe +ra -p 0,1,6
> > >>>>>
> > >>>>> By klitmus7, I have seen a low probability "Sometimes" on POWER8 system.
> > >>>>>
> > >>>>> One of the results reads:
> > >>>>>
> > >>>>> ----------------------------------------------------------
> > >>>>> Test C-WRC+o+o-data-o+o-rmb-o Allowed
> > >>>>> Histogram (6 states)
> > >>>>> 60327103:>1:r1=0; 2:r2=0; 2:r3=0;
> > >>>>> 22740026:>1:r1=1; 2:r2=0; 2:r3=0;
> > >>>>> 5       *>1:r1=1; 2:r2=1; 2:r3=0;
> > >>>>> 51117838:>1:r1=0; 2:r2=0; 2:r3=1;
> > >>>>> 55552319:>1:r1=1; 2:r2=0; 2:r3=1;
> > >>>>> 10262709:>1:r1=1; 2:r2=1; 2:r3=1;
> > >>>>> Ok
> > >>>>>
> > >>>>> Witnesses
> > >>>>> Positive: 5, Negative: 199999995
> > >>>>> Condition exists (1:r1=1 /\ 2:r2=1 /\ 2:r3=0) is validated
> > >>>>> Hash=43057833028631b2f87eb65fe95c0ba2
> > >>>>> Observation C-WRC+o+o-data-o+o-rmb-o Sometimes 5 199999995
> > >>>>> Time C-WRC+o+o-data-o+o-rmb-o 68.34
> > >>>>> ----------------------------------------------------------
> > >>>>>
> > >>>>> In theory, litmus7 can also trigger the "exists" clause, but it would
> > >>>>> require a very long runtime.  Also please refer to the Answer to
> > >>>>> Quick Quiz 15.24. (You will find my name is mentioned there.)
> > >>>>>
> > >>>>
> > >>>> Hi Akira,
> > >>>>
> > >>>> I will try a long runtime. If that doesn't work, I suppose I need to
> > >>>> wait until I get another PPC server which I can physically touch.
> > >>>> Thanks a lot.
> > >>>>
> > >>>>> As Armv8 architecture is other-multicopy-atomic, this "exists" clause
> > >>>>> should never trigger, I suppose.
> > >>>>>
> > >>>>
> > >>>> I'm a bit confused here. Perfbook says "Therefore, a
> > >>>> non-multicopy-atomic platform can have a store reach different threads
> > >>>> at different times." (Page 277 the first paragraph). So my
> > >>>> understanding is that for Listing C-WRC+o+o-data-o+o-rmb-o, the
> > >>>> message of writing y could reach P2 before the message of writing x
> > >>>> could reach there, and hence the "exists" clause could trigger even if
> > >>>> on an ARM server. Is that correct? Or did I misunderstand something?
> > >>>>
> > >>>> Perfbook says "Most CPU vendors interested in providing multicopy
> > >>>> atomicity have therefore instead provided the slightly weaker
> > >>>> other-multicopy atomicity".(Page 277 the first paragraph). My
> > >>>> understanding is that "Most CPU vendors" here implies PPC, ARM, and
> > >>>> x86. In other words, all of the servers I can touch adopt
> > >>>> other-multicopy-atomic? Is that correct?
> > >>>
> > >>> ARM and x86 provide other-multicopy atomicity.  IBM mainframe provides
> > >>> fully multicopy atomicity.  PPC doesn't provide multicopy atomicity
> > >>> at all.
> > >>>
> > >>
> > >> Thanks a lot!!! This is the first time I know about this. How about we
> > >> add this as a footnote or something to help readers easier to get a
> > >> sense of the use of multicopy atomicity in practice? Does the
> > >> following look OK?
> > >
> > > It does look like a good addition, but it needs a couple of changes,
> > > please see below.
> > >
> > >> ===
> > >>
> > >> >From 8e07ecd836e5b581d2953938a65a395208fe0af3 Mon Sep 17 00:00:00 2001
> > >> From: Junchang Wang <junchangwang@gmail.com>
> > >> Date: Tue, 6 Nov 2018 15:00:31 +0800
> > >> Subject: [PATCH] memorder: Clarify implementation details of multicopy
> > >>  atomicity in mainstream architectures
> > >>
> > >> Reported-by: Junchang Wang <junchangwang@gmail.com>
> > >> Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
> > >
> > > This needs to be just:
> > >
> > > Signed-off-by: Junchang Wang <junchangwang@gmail.com>
> > >
> > > It is your patch, so it has your signed-off-by.
> >
> > But it came from Paul's reply. It might be a good idea to cite
> > the message at
> >
> >       https://www.spinics.net/lists/perfbook/msg01952.html
> >
> > in the commit log.
>
> That would be a nice addition, but not essential in this case.
>
> > >> ---
> > >>  memorder/memorder.tex | 4 +++-
> > >>  1 file changed, 3 insertions(+), 1 deletion(-)
> > >>
> > >> diff --git a/memorder/memorder.tex b/memorder/memorder.tex
> > >> index 5e74a91..5db4971 100644
> > >> --- a/memorder/memorder.tex
> > >> +++ b/memorder/memorder.tex
> > >> @@ -1901,7 +1901,9 @@ Most CPU vendors interested in providing
> > >> multicopy atomicity have therefore
> > >>  instead provided the slightly weaker
> > >>  \emph{other-multicopy atomicity}~\cite[Section B2.3]{ARMv8A:2017},
> > >>  which excludes the CPU doing a given store from the requirement that all
> > >> -CPUs agree on the order of all stores.
> > >> +CPUs agree on the order of all stores \footnote{As of late 2018, ARM and x86
> > >> +provide other-multicopy atomicity, IBM mainframe provides fully multicopy
> > >> +atomicity, and PPC does not provide multicopy atomicity at all.}.
> > >
> > > Please indent the footnote text so that it stands out more and please
> > > remove the space character just before the "\footnote":
> > >
> > > CPUs agree on the order of all stores\footnote{
> > >     As of late 2018, ARM and x86 provide other-multicopy atomicity,
> > >     IBM mainframe provides fully multicopy atomicity, and PPC does
> > >     not provide multicopy atomicity at all.}.
> >
> > I think the footnote should be just after the punctuation character, i.e.,
> >
> > CPUs agree on the order of all stores.\footnote{
> >       As of late 2018, ARM and x86 provide other-multicopy atomicity,
> >       IBM mainframe provides fully multicopy atomicity, and PPC does
> >       not provide multicopy atomicity at all.}
>
> You are right, I did miss that.
>
> > > Could you please also add a reference to:
> > >
> > >     Table~\ref{\label{tab:memorder:Summary of Memory Ordering}?
> >
> > Did you mean:
> >
> >       Table~\ref{tab:memorder:Summary of Memory Ordering}?
>
> Indeed I did, thank you!

Hi Akira and Paul,

Thank you so much for the great suggestions. I just submitted a
standalone patch regarding this. Please take a look and let me know if
it looks OK.

Thanks,
--Junchang


>
>                                                         Thanx, Paul
>
> > Thanks, Akira
> >
> > >
> > > If you update the patch as suggested and send it as a standalone patch,
> > > I will accept it.
> > >
> > >                                                     Thanx, Paul
> > >
> > >>  This means that if only a subset of CPUs are doing stores, the
> > >>  other CPUs will agree on the order of stores, hence the ``other''
> > >>  in ``other-multicopy atomicity''.
> > >> --
> > >> 2.7.4
> > >>
> > >> Thanks,
> > >> --Junchang
> > >>
> > >>
> > >>>                                                         Thanx, Paul
> > >>>
> > >>>> Thanks,
> > >>>> --Junchang
> > >>>>
> > >>>>> As for C-CCIRIW+o+o+o-o+o-o, herd7 says
> > >>>>>
> > >>>>> ----------------------------------------------------------
> > >>>>> Test C-CCIRIW+o+o+o-o+o-o Allowed
> > >>>>> States 47
> > >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=0;
> > >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=1;
> > >>>>> 2:r1=0; 2:r2=0; 3:r3=0; 3:r4=2;
> > >>>>> 2:r1=0; 2:r2=0; 3:r3=1; 3:r4=1;
> > >>>>> [...]
> > >>>>> 2:r1=2; 2:r2=2; 3:r3=0; 3:r4=2;
> > >>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=1;
> > >>>>> 2:r1=2; 2:r2=2; 3:r3=1; 3:r4=2;
> > >>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=1;
> > >>>>> 2:r1=2; 2:r2=2; 3:r3=2; 3:r4=2;
> > >>>>> No
> > >>>>> Witnesses
> > >>>>> Positive: 0 Negative: 72
> > >>>>> Condition exists (2:r1=1 /\ 2:r2=2 /\ 3:r3=2 /\ 3:r4=1)
> > >>>>> Observation C-CCIRIW+o+o+o-o+o-o Never 0 72
> > >>>>> Time C-CCIRIW+o+o+o-o+o-o 0.06
> > >>>>> Hash=8e54976d74e1bc1ec5b6ce10eda3cb12
> > >>>>> ----------------------------------------------------------
> > >>>>>
> > >>>>> It says "Never", so if you could ever trigger the condition on a
> > >>>>> real machine, it would've meant something wrong in the implementation
> > >>>>> on the platform side (cache coherency).
> > >>>>>
> > >>>>> Hope this helps.
> > >>>>>
> > >>>>>        Thanks, Akira
> > >>>>>
> > >>>>>>
> > >>>>>> But it seems my effort doesn't work; the exist assertions never
> > >>>>>> trigger. Did I miss something? What's the right/possible configuration
> > >>>>>> to trigger these assertions. I do know you might be very busy. But if
> > >>>>>> possible, please give me some hints. These two tests seems very
> > >>>>>> interesting :-). Thanks in advance.
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> --Junchang
> > >>>>>>
> > >>>>>> ===
> > >>>>>> Three servers I can touch:
> > >>>>>>   1. One PPC server which consists of 8 cores. I don's know hardware
> > >>>>>> details (e.g. which cores share the same LLC or write buffer) because
> > >>>>>> tools such as dmidecode don't work.
> > >>>>>>   2. One ARM server with two Cavium CPU. 96 cores in total (no Hyper-threading).
> > >>>>>>   3. One Intel server with two Xeon E5-2630 CPUs, each of which
> > >>>>>> consists of 12 cores (with Hyper-Threading).
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>