Re: [RFC] MMIO accessors & barriers documentation

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

> where next_eqe_sw() checks a "valid" bit of a 32-byte event queue
> entry that is DMA-ed into memory by the device.  The device is careful
> to write the valid bit (byte actually) last, but on PowerPC 970
> without the rmb(), we actually saw the CPU reordering the read of
> eqe->type (which is another field of the EQ entry written by the
> device) so it happened before the entry was valid, but then executing
> the check of the valid bit far enough into the future so that the
> entry tested as valid.

Yes, the CPU can perfectly load it before the previous load, indeed. I'm
sure that wouldn't be powerpc specific. In this case, it would be a
speculative load (since there is a data dependency, thus you would think
it's ok, but it's not on CPUs that do speculative execution).

> This isn't that surprising: if you had two CPUs, with one CPU writing
> into a queue and the other CPU polling the queue, you would obviously
> need smp_rmb() on the CPU doing the reading.  But somehow it's not
> quite as obvious when a device plays the role of one of the CPUs.
> 
> Of course there's no MMIO anywhere in sight here, so this isn't
> directly applicable I guess.

It's a "normal" case memory barrier in this case. Same as for SMP. Yup. 

Ben.