Re: [PATCH][16/26] IB/mthca: mem-free doorbell record writing

[Roland Dreier <roland@topspin.com>]

    Jeff> Well, we don't just add code to "hope and pray" for an event
    Jeff> that nobody is sure can even occur...

The hardware requires that if the record is written in two 32-bit
chunks, then they must be written in order.  Of course the hardware
probably won't be reading just as we're writing, so almost all of the
time we won't notice the problem.

It feels more like "hope and pray" to me to leave the barrier out and
assume that every possible implementation of every architecture will
always write them in order.

    Jeff> Does someone have a concrete case where this could happen? ever?

I don't see how you can rule it out on out-of-order architectures.  If
the second word becomes ready before the first, then the CPU may
execute the second write before the first.

It's not precisely the same situation, but if you look at mthca_eq.c
you'll see an rmb() in mthca_eq_int().  That's there because on ppc64,
I really saw a situation where code like:

	while (foo->x) {
		switch (foo->y) {

was behaving as if foo->y was being read before foo->x.  Even though
both foo->x and foo->y are in the same cache line, and foo->x was
written by the hardware after foo->y.

 - R.