Re: [PATCH 0/11] RFC: PCI using capabilitities

[Avi Kivity <avi@redhat.com>]

On 12/08/2011 05:37 PM, Sasha Levin wrote:
> On Thu, 2011-12-08 at 20:52 +1030, Rusty Russell wrote:
> > Here's the patch series I ended up with.  I haven't coded up the QEMU
> > side yet, so no idea if the new driver works.
> > 
> > Questions:
> > (1) Do we win from separating ISR, NOTIFY and COMMON?
> > (2) I used a "u8 bar"; should I use a bir and pack it instead?  BIR
> >     seems a little obscure (noone else in the kernel source seems to
> >     refer to it).
>
> I started implementing it for KVM tools, when I noticed a strange thing:
> my vq creating was failing because the driver was reading a value other
> than 0 from the address field of a new vq, and failing.
>
> I've added simple prints in the usermode code, and saw the following
> ordering:
>
> 1. queue select vq 0
> 2. queue read address (returns 0 - new vq)
> 3. queue write address (good address of vq)
> 4. queue read address (returns !=0, fails)
> 4. queue select vq 1
>
> From that I understood that the ordering is wrong, the driver was trying
> to read address before selecting the correct vq.
>
> At that point, I've added simple prints to the driver. Initially it
> looked as follows:
>
> 	iowrite16(index, &vp_dev->common->queue_select);
>
> 	switch (ioread64(&vp_dev->common->queue_address)) {
> 		[...]
> 	};
>
> So I added prints before the iowrite16() and after the ioread64(), and
> saw that while the driver prints were ordered, the device ones weren't:
>
> 	[    1.264052] before iowrite index=1
> 	kvmtool: net returning pfn (vq=0): 310706176
> 	kvmtool: queue selected: 1
> 	[    1.264890] after ioread index=1
>
> Suspecting that something was wrong with ordering, I've added a print
> between the iowrite and the ioread, and it finally started working well.
>
> Which leads me to the question: Are MMIO vs MMIO reads/writes not
> ordered?

mmios are strictly ordered.

Perhaps your printfs are reordered by buffering?  Are they from
different threads?  Are you using coalesced mmio (which is still
strictly ordered, if used correctly)?

-- 
error compiling committee.c: too many arguments to function