Re: [RFC] MMIO accessors & barriers documentation

[Roland Dreier <rdreier@cisco.com>]

    Benjamin> and rmb is heavy handed for a compiler barrier :) what
    Benjamin> you might need on some platforms is an rmb between the
    Benjamin> MMIO read of whatever status/index register and the
    Benjamin> following memory reads of descriptors, and you may want
    Benjamin> an rmb in case where it matters if the chip has been
    Benjamin> changing a value behind your back (which it generally
    Benjamin> doesn't) but that's pretty much it....

In drivers/infiniband/hw/mthca/mthca_eq.c, there is:

	while ((eqe = next_eqe_sw(eq))) {
		/*
		 * Make sure we read EQ entry contents after we've
		 * checked the ownership bit.
		 */
		rmb();

		switch (eqe->type) {

where next_eqe_sw() checks a "valid" bit of a 32-byte event queue
entry that is DMA-ed into memory by the device.  The device is careful
to write the valid bit (byte actually) last, but on PowerPC 970
without the rmb(), we actually saw the CPU reordering the read of
eqe->type (which is another field of the EQ entry written by the
device) so it happened before the entry was valid, but then executing
the check of the valid bit far enough into the future so that the
entry tested as valid.

This isn't that surprising: if you had two CPUs, with one CPU writing
into a queue and the other CPU polling the queue, you would obviously
need smp_rmb() on the CPU doing the reading.  But somehow it's not
quite as obvious when a device plays the role of one of the CPUs.

Of course there's no MMIO anywhere in sight here, so this isn't
directly applicable I guess.

 - R.