Flushing writes to PCI devices

Flushing writes to PCI devices

[Alan Stern]

I've heard that to insure proper synchronization it's necessary to flush
MMIO writes (writel, writew, writeb) to PCI devices by reading from the
same area.  Is this equally true for I/O-space writes (inl, inw, inb)?  
What about configuration space writes (pci_write_config_dword etc.)?

Alan Stern

Re: Flushing writes to PCI devices

[linux-os (Dick Johnson)]

On Wed, 20 Sep 2006, Alan Stern wrote:

> I've heard that to insure proper synchronization it's necessary to flush
> MMIO writes (writel, writew, writeb) to PCI devices by reading from the
> same area.  Is this equally true for I/O-space writes (inl, inw, inb)?
> What about configuration space writes (pci_write_config_dword etc.)?
>
> Alan Stern

Writes to I/O space are not queued through a FIFO so there is
no need to flush the FIFO. Configuration space uses special
configuration cycles which are handshakes with the devices. They
cannot be queued, therefore don't need to be flushed either.

Flushing PCI space writes shouldn't be done until you want
whatever you've been planning to happen __now__. Otherwise
the advantages of queued writes go away. In other words, one
should NOT attach a read to every PCI space write! Typically
use of the flushing read might be in the case of setting up
hardware for a DMA transfer. You write all the data, source
address, destination address, byte-count, DMA type, etc., then
after the last instruction, the one should should start the DMA,
you issue a read.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.66 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Flushing writes to PCI devices

[Bill Waddington]

On Wed, 20 Sep 2006 19:10:55 UTC, in fa.linux.kernel you wrote:

>
>On Wed, 20 Sep 2006, Alan Stern wrote:
>
>> I've heard that to insure proper synchronization it's necessary to flush
>> MMIO writes (writel, writew, writeb) to PCI devices by reading from the
>> same area.  Is this equally true for I/O-space writes (inl, inw, inb)?
>> What about configuration space writes (pci_write_config_dword etc.)?
>>
>> Alan Stern
>
>Writes to I/O space are not queued through a FIFO so there is
>no need to flush the FIFO. Configuration space uses special
>configuration cycles which are handshakes with the devices. They
>cannot be queued, therefore don't need to be flushed either.
>
>Flushing PCI space writes shouldn't be done until you want
>whatever you've been planning to happen __now__. Otherwise
>the advantages of queued writes go away. In other words, one
>should NOT attach a read to every PCI space write! Typically
>use of the flushing read might be in the case of setting up
>hardware for a DMA transfer. You write all the data, source
>address, destination address, byte-count, DMA type, etc., then
>after the last instruction, the one should should start the DMA,
>you issue a read.

Are there ever any issues with out-of-order writes from the posting
buffer on supported architectures?  I can (barely) imagine a device
that has the register with the start bit lower in the register address
space than the count/address registers.  Out-of-order writes from
the cache/non-fifo/posting buffer (maybe to assemble a burst) could
make the occasional mess.

Just wondering,
Bill
-- 
William D Waddington
william.waddington@beezmo.com
"Even bugs...are unexpected signposts on
the long road of creativity..." - Ken Burtch

Re: Flushing writes to PCI devices

[linux-os (Dick Johnson)]

On Wed, 20 Sep 2006, Bill Waddington wrote:

> On Wed, 20 Sep 2006 19:10:55 UTC, in fa.linux.kernel you wrote:
>
>>
>> On Wed, 20 Sep 2006, Alan Stern wrote:
>>
>>> I've heard that to insure proper synchronization it's necessary to flush
>>> MMIO writes (writel, writew, writeb) to PCI devices by reading from the
>>> same area.  Is this equally true for I/O-space writes (inl, inw, inb)?
>>> What about configuration space writes (pci_write_config_dword etc.)?
>>>
>>> Alan Stern
>>
>> Writes to I/O space are not queued through a FIFO so there is
>> no need to flush the FIFO. Configuration space uses special
>> configuration cycles which are handshakes with the devices. They
>> cannot be queued, therefore don't need to be flushed either.
>>
>> Flushing PCI space writes shouldn't be done until you want
>> whatever you've been planning to happen __now__. Otherwise
>> the advantages of queued writes go away. In other words, one
>> should NOT attach a read to every PCI space write! Typically
>> use of the flushing read might be in the case of setting up
>> hardware for a DMA transfer. You write all the data, source
>> address, destination address, byte-count, DMA type, etc., then
>> after the last instruction, the one should should start the DMA,
>> you issue a read.
>
> Are there ever any issues with out-of-order writes from the posting
> buffer on supported architectures?  I can (barely) imagine a device
> that has the register with the start bit lower in the register address
> space than the count/address registers.  Out-of-order writes from
> the cache/non-fifo/posting buffer (maybe to assemble a burst) could
> make the occasional mess.
>
> Just wondering,
> Bill
> --
> William D Waddington
> william.waddington@beezmo.com
> "Even bugs...are unexpected signposts on
> the long road of creativity..." - Ken Burtch
>

People who make PCI interface chips like PLX would never make
devices that couldn't work on the PCI bus. However, it is possible
for somebody to make byte-wide or word-wide setup registers on
their local bus, accessed through an interface chip such as the PLX,
that might suffer because of write-combining. However, such defective
designs shouldn't make it to production.

Also, the PLX and other such interface chips don't rely upon
initialization order (like the PC 8237A DMA chips) when setting
up for a DMA transfer. They have a specific instruction to start
the transfer.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.66 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Flushing writes to PCI devices

[Bryan O'Sullivan]

On Wed, 2006-09-20 at 12:41 -0700, Bill Waddington wrote:

> Are there ever any issues with out-of-order writes from the posting
> buffer on supported architectures?

Yes.  If your device requires that writes to some locations in MMIO
space be performed in a specific order, you must explicitly do this in
your driver.  Intel CPUs will flush posted writes out of order, for
example.

	<b

Re: Flushing writes to PCI devices

[Alan Cox]

Ar Iau, 2006-09-21 am 11:14 -0700, ysgrifennodd Bryan O'Sullivan:
> Yes.  If your device requires that writes to some locations in MMIO
> space be performed in a specific order, you must explicitly do this in
> your driver.  Intel CPUs will flush posted writes out of order, for
> example.

According to the docs I have here if the pci target area is
prefetchable/postwritable or has MTRRs set specifically to do this (eg
video ram).

There is no ordering guarantee between PCI and main memory however.

Alan

Re: Flushing writes to PCI devices

[Roland Dreier]

    Bryan> Yes.  If your device requires that writes to some locations
    Bryan> in MMIO space be performed in a specific order, you must
    Bryan> explicitly do this in your driver.  Intel CPUs will flush
    Bryan> posted writes out of order, for example.

Really?  Just normal posted PCI writes without using MTRRs or
write-combining or anything like that?

That doesn't seem right to me, and I would expect all sorts of things
to break if it were true.

 - R.

Re: Flushing writes to PCI devices

[Bryan O'Sullivan]

On Thu, 2006-09-21 at 13:56 -0700, Roland Dreier wrote:
>     Bryan> Yes.  If your device requires that writes to some locations
>     Bryan> in MMIO space be performed in a specific order, you must
>     Bryan> explicitly do this in your driver.  Intel CPUs will flush
>     Bryan> posted writes out of order, for example.
> 
> Really?  Just normal posted PCI writes without using MTRRs or
> write-combining or anything like that?

No, not normal writes, but if you're writing to an area where you've
enabled write combining, it will happen (but not IIRC on AMD x86_64
CPUs).  Sorry for the lack of clarity.

	<b

Re: Flushing writes to PCI devices

[Alan Cox]

Ar Mer, 2006-09-20 am 14:28 -0400, ysgrifennodd Alan Stern:
> I've heard that to insure proper synchronization it's necessary to flush
> MMIO writes (writel, writew, writeb) to PCI devices by reading from the
> same area.  Is this equally true for I/O-space writes (inl, inw, inb)? 

writel/writew/writeb may be posted and delayed until a read from that
device. The outb/outw/outl operations are not posted but are
synchronous.

There may be posting in action both ways (your read may force not only
the write to complete but pending writes the other end). That usually
doesn't matter but very occasionally requires care.

> What about configuration space writes (pci_write_config_dword etc.)?

Usually this is entirely synchronous to PCI bus activity and main
memory. Occasionally chipsets get it slightly wrong

Re: Flushing writes to PCI devices

[Robert Hancock]

Alan Stern wrote:
> I've heard that to insure proper synchronization it's necessary to flush
> MMIO writes (writel, writew, writeb) to PCI devices by reading from the
> same area.  Is this equally true for I/O-space writes (inl, inw, inb)?  
> What about configuration space writes (pci_write_config_dword etc.)?

Technically, according to the PCI specification, I/O space writes may be 
posted in the host bus bridge (though unlike MMIO space, not at any 
other PCI bridges) and would require a read in order to flush them. 
However, there seem to be few if any chipsets ever made which actually 
do this for I/O writes, and most code in the kernel seems to just assume 
it won't happen, so it's probably safe to assume the same.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/