Re: [PATCH rdma-core 06/14] i40iw: Get rid of unique barrier macros

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Mar 01, 2017 at 04:14:20PM -0600, Shiraz Saleem wrote:
> > Is there DMA occuring to shadow_area?
> 
> The shadow area contains status variables which are read by SW and 
> updated by PCI device.

So the device is DMA'ing to it, and the driver is reading DMA memory..

> > What can wrong if it executes like this?
> > 
> > get_64bit_val(qp->shadow_area, I40IW_BYTE_0, &temp);
> > udma_to_device_barrier(); /* make sure WQE is populated before valid bit is set */
> > set_64bit_val(wqe, I40IW_BYTE_24, header);
> > udma_to_device_barrier();
> 
> We need strict ordering that ensures write of the WQE completes before 
> read of the shadow area.

> This ensures the value read from the shadow can be used to determine
> if a DB ring is needed. If the shadow area is read first, the
> algorithm, in certain cases, would not ring the DB when it should
> and the HW may go idle with work requests posted.

This still is not making a lot of sense to me.. I really need to see a
ladder diagram to understand your case.

Here is an example, I think what you are saying is: The HW could have
fetched valid = 0 and stopped the queue and the driver needs to
doorbell it to wake it up again. However, the driver optimizes away
the doorbell rings in certain cases based on reading a DMA result.

So here is a possible ladder diagram:

CPU                         DMA DEVICE
                            Issue READ#1 of valid bit
 Respond to READ#1
 SFENCE
 set_valid_bit
 MEFENCE
 read_tail
                            Receive READ#1 response with valid bit unset
			    Issue DMA WRITE to shadow_area indicating STOPPED
 DMA WRITE arrives

And the version where the DMA is seen:

CPU                         DMA DEVICE
                            Issue READ#1 of valid bit
 SFENCE
 Respond to READ#1
 set_valid_bit
 MEFENCE
                            Receive READ#1 response with valid bit unset
			    Issue DMA WRITE to shadow_area indicating STOPPED
 DMA WRITE arrives
 read_tail

These diagrams attempt to show that the DMA device reads the valid bit
then DMA's back to the shadow_area depending on what it read.

Given the semantics of MFENCE, both are possible, so I still don't
understand what the MFENCE is supposed to help with.

I get the feeling this approach requires MFENCE to do something it
doesn't...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html