Re: [RFC] MMIO accessors & barriers documentation

Roland Dreier
2006-09-11T23:08:29Z

Benjamin> and rmb is heavy handed for a compiler barrier :) what
Benjamin> you might need on some platforms is an rmb between the
Benjamin> MMIO read of whatever status/index register and the
Benjamin> following memory reads of descriptors, and you may want
Benjamin> an rmb in case where it matters if the chip has been
Benjamin> changing a value behind your back (which it generally
Benjamin> doesn't) but that's pretty much it....

In drivers/infiniband/hw/mthca/mthca_eq.c, there is:

while ((eqe = next_eqe_sw(eq))) {
/*
* Make sure we read EQ entry contents after we've
* checked the ownership bit.
*/
rmb();

switch (eqe->type) {

where next_eqe_sw() checks a "valid" bit of a 32-byte event queue
entry that is DMA-ed into memory by the device. The device is careful
to write the valid bit (byte actually) last, but on PowerPC 970
without the rmb(), we actually saw the CPU reordering the read of
eqe->type (which is another field of the EQ entry written by the
device) so it happened before the entry was valid, but then executing
the check of the valid bit far enough into the future so that the
entry tested as valid.

This isn't that surprising: if you had two CPUs, with one CPU writing
into a queue and the other CPU polling the queue, you would obviously
need smp_rmb() on the CPU doing the reading. But somehow it's not
quite as obvious when a device plays the role of one of the CPUs.

Of course there's no MMIO anywhere in sight here, so this isn't
directly applicable I guess.

- R.