HELP: Is writeq an atomic operation??

HELP: Is writeq an atomic operation??

[Moore, Eric]

Is a 64bit write to MMIO registers an atomic operation when using the
writeq API?  

My concern is when I send 64bit data via writeq, will it be sent out as
two 32 bit writes?  If so, is it possible that another CPU be sending
the data at the same time.  Meaning can I write the 1st 32bit data from
CPU-A, meanwhile CPU-B is writing his 32bit data at the same time, and
CPU-A didn't complete the full 64bit in one shot.  If this could occur,
is there an API that I can use to make sure the entire data sent in one
atomic operation?


Here is a trace from pci express analyzer.   I'm sending
0x0800010000000000 to the adress DD1400C0 using writeq.   Notice that in
the TLP header it sent a 32bit Memory write with data length of two.

Trace follows:

Link Tra(597) Downstream 2.5(x1) TLP(1992) Mem MWr(32)(10:00000) TC(0)
TD(0) 
_______| EP(0) Attributes(01) Length(2) RequesterID(000:02:0) Tag(8) 
_______| Address(DD1400C0) 1st BE(1111) Last BE(1111) Data(08000100
00000000) 
_______| VC ID(0) Explicit ACK(Packet #1195) Metrics # Packets(2) 
_______| Time Stamp(0003 . 120 181 840 s) 

Re: HELP: Is writeq an atomic operation??

[David Miller]

From: "Moore, Eric" <Eric.Moore@lsi.com>
Date: Fri, 2 May 2008 16:19:49 -0600

> Is a 64bit write to MMIO registers an atomic operation when using the
> writeq API?  

The answer to this question this is platform dependent.

On most 64-bit platforms, it is.  On some 32-bit ones, it is not.

This is not a SCSI layer question, so belongs minimally on
linux-kernel which I've CC:'d.

Re: HELP: Is writeq an atomic operation??

[Roland Dreier]

 > > Is a 64bit write to MMIO registers an atomic operation when using the
 > > writeq API?  
 > 
 > The answer to this question this is platform dependent.
 > 
 > On most 64-bit platforms, it is.  On some 32-bit ones, it is not.

Are there any 32-bit platforms with writeq()?  A quick grep suggests not.

Are there any 64-bit platforms where writeq() allows the MMIO to be
split into multiple cycles from the target device's view?  I've been
coding assuming that at least no other MMIO writes will reach the device
in the middle of a writeq().

 - R.

Re: HELP: Is writeq an atomic operation??

[David Miller]

From: Roland Dreier <rdreier@cisco.com>
Date: Fri, 02 May 2008 15:43:32 -0700

> Are there any 32-bit platforms with writeq()?  A quick grep suggests not.

Right, I guess there aren't, but what drivers do currently is roll
their own 64-bit MMIO for such cases.

I noticed this when writing drivers/net/niu.c

I suppose this is on purpose, so the driver can setup any
such protection and handling, as needed.

RE: HELP: Is writeq an atomic operation??

[Moore, Eric]

>  > 
>  > The answer to this question this is platform dependent.
>  > 
>  > On most 64-bit platforms, it is.  On some 32-bit ones, it is not.
> 
> Are there any 32-bit platforms with writeq()?  A quick grep 
> suggests not.

I think writeq defined in include/asm-x86/io_64.h

> 
> Are there any 64-bit platforms where writeq() allows the MMIO to be
> split into multiple cycles from the target device's view?  I've been
> coding assuming that at least no other MMIO writes will reach 
> the device
> in the middle of a writeq().
> 

I hope that is the case.

Re: HELP: Is writeq an atomic operation??

[Roland Dreier]

 > > Are there any 32-bit platforms with writeq()?  A quick grep 
 > > suggests not.
 > 
 > I think writeq defined in include/asm-x86/io_64.h

Umm... io_64.h is 64-bit only (look at asm-x86/io.h if you don't believe me ;)

 - R.

RE: HELP: Is writeq an atomic operation??

[Moore, Eric]

On Friday, May 02, 2008 4:53 PM, Roland Dreier wrote: 
> 
>  > > Are there any 32-bit platforms with writeq()?  A quick grep 
>  > > suggests not.
>  > 
>  > I think writeq defined in include/asm-x86/io_64.h
> 
> Umm... io_64.h is 64-bit only (look at asm-x86/io.h if you 
> don't believe me ;)
> 

Yeah,  I forgot I have a #ifndef writeq, then defined the x86_64 version
of that.   I've not tested on x86, so I'm not sure whether it works.
How are you handling writeq when its not defined, as the case in x86?

Eric

Re: HELP: Is writeq an atomic operation??

[Roland Dreier]

 > Yeah,  I forgot I have a #ifndef writeq, then defined the x86_64 version
 > of that.   I've not tested on x86, so I'm not sure whether it works.
 > How are you handling writeq when its not defined, as the case in x86?

Write two writel() inside a spinlock to avoid any transactions in the
middle (the HW I'm dealing with can deal with two 32-bit transactions,
as long as nothing comes in the middle).  If your hardware demands a
single 64-bit transaction, you may be in trouble, because I'm not sure
all 32-bit systems can generate such a PCIe transaction.

You can see include/linux/mlx4/doorbell.h for exactly what I did.

 - R.

RE: HELP: Is writeq an atomic operation??

[Moore, Eric]

On Friday, May 02, 2008 5:22 PM,  Roland Dreier wrote:
>  > Yeah,  I forgot I have a #ifndef writeq, then defined the 
> x86_64 version
>  > of that.   I've not tested on x86, so I'm not sure whether 
> it works.
>  > How are you handling writeq when its not defined, as the 
> case in x86?
> 
> Write two writel() inside a spinlock to avoid any transactions in the
> middle (the HW I'm dealing with can deal with two 32-bit transactions,
> as long as nothing comes in the middle).  If your hardware demands a
> single 64-bit transaction, you may be in trouble, because I'm not sure
> all 32-bit systems can generate such a PCIe transaction.
> 
> You can see include/linux/mlx4/doorbell.h for exactly what I did.
> 

Thanks for the code sample.   Yes, I need to send a single atomic 64-bit
transaction.

Eric