question on memory barrier

question on memory barrier

[moreau francis]

Hi,

I'm currently trying to write a USB driver for Linux. The device must be
configured by writing some values into the same register but I want to be
sure that the writing order is respected by either the compiler and the cpu.

For example, here is a bit of driver's code:

"""
#include <asm/io.h>

static inline void dev_out(u32 *reg, u32 value)
{
        writel(value, regs);
}

void config_dev(void)
{
        dev_out(reg_a, 0x0); /* first io */
        dev_out(reg_a, 0xA); /* second io */
}

void config_dev_fixed(void)
{
        dev_out(reg_a, 0x0); /* first io */
        wmb();
        dev_out(reg_a, 0xA); /* second io */
        wmb();
}
"""

In this case, am I sure that the order will be respected ? can gcc remove
the first io while optimizing...If so, does "config_dev_fixed" fix it ?

thanks for your answers,

            Francis


	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
Téléchargez cette version sur http://fr.messenger.yahoo.com

Re: question on memory barrier

[linux-os (Dick Johnson)]

On Wed, 24 Aug 2005, moreau francis wrote:

> Hi,
>
> I'm currently trying to write a USB driver for Linux. The device must be
> configured by writing some values into the same register but I want to be
> sure that the writing order is respected by either the compiler and the cpu.
>
> For example, here is a bit of driver's code:
>
> """
> #include <asm/io.h>
>
> static inline void dev_out(u32 *reg, u32 value)
> {
>        writel(value, regs);
> }
>
> void config_dev(void)
> {
>        dev_out(reg_a, 0x0); /* first io */
>        dev_out(reg_a, 0xA); /* second io */
> }
>

This should be fine. The effect of the first bit of code
plus all side-effects (if any) should be complete at the
first effective sequence-point (;) but you need to
change the procedure, dev_out, to:

static inline void dev_out(volatile u32 *regs, u32 value)
                            ^^^^^^^^

... or ... just use writel() directly. wmb() just tells
the compiler that memory has been changed. It already knows
that but you must use the keyword, "volatile" so the compiler
doesn't use something stale that's cached in registers.

> void config_dev_fixed(void)
> {
>        dev_out(reg_a, 0x0); /* first io */
>        wmb();
>        dev_out(reg_a, 0xA); /* second io */
>        wmb();
> }
> """
>
> In this case, am I sure that the order will be respected ? can gcc remove
> the first io while optimizing...If so, does "config_dev_fixed" fix it ?
>
> thanks for your answers,
>
>            Francis
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12.5 on an i686 machine (5537.79 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: question on memory barrier

[moreau francis]

--- "linux-os (Dick Johnson)" <linux-os@analogic.com> a écrit :

> 
> On Wed, 24 Aug 2005, moreau francis wrote:
> 
> > Hi,
> >
> > I'm currently trying to write a USB driver for Linux. The device must be
> > configured by writing some values into the same register but I want to be
> > sure that the writing order is respected by either the compiler and the
> cpu.
> >
> > For example, here is a bit of driver's code:
> >
> > """
> > #include <asm/io.h>
> >
> > static inline void dev_out(u32 *reg, u32 value)
> > {
> >        writel(value, regs);
> > }
> >
> > void config_dev(void)
> > {
> >        dev_out(reg_a, 0x0); /* first io */
> >        dev_out(reg_a, 0xA); /* second io */
> > }
> >
> 
> This should be fine. The effect of the first bit of code
> plus all side-effects (if any) should be complete at the
> first effective sequence-point (;) but you need to

sorry but I'm not sure to understand you...Do you mean that the first write
into reg_a pointer will be completed before the second write because they're
separated by a (;) ?
Or because writes are encapsulated inside an inline function, therefore
compiler
must execute every single writes before returning from the inline function ?

Thanks.

            Francis


	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
Téléchargez cette version sur http://fr.messenger.yahoo.com

Re: question on memory barrier

[linux-os (Dick Johnson)]

On Wed, 24 Aug 2005, moreau francis wrote:

>
> --- "linux-os (Dick Johnson)" <linux-os@analogic.com> a écrit :
>
>>
>> On Wed, 24 Aug 2005, moreau francis wrote:
>>
>>> Hi,
>>>
>>> I'm currently trying to write a USB driver for Linux. The device must be
>>> configured by writing some values into the same register but I want to be
>>> sure that the writing order is respected by either the compiler and the
>> cpu.
>>>
>>> For example, here is a bit of driver's code:
>>>
>>> """
>>> #include <asm/io.h>
>>>
>>> static inline void dev_out(u32 *reg, u32 value)
>>> {
>>>        writel(value, regs);
>>> }
>>>
>>> void config_dev(void)
>>> {
>>>        dev_out(reg_a, 0x0); /* first io */
>>>        dev_out(reg_a, 0xA); /* second io */
>>> }
>>>
>>
>> This should be fine. The effect of the first bit of code
>> plus all side-effects (if any) should be complete at the
>> first effective sequence-point (;) but you need to
>
> sorry but I'm not sure to understand you...Do you mean that the first write
> into reg_a pointer will be completed before the second write because they're
> separated by a (;) ?

Yes. The compiler must make sure that every effect of all previous
code and all side-effects are complete at a "sequence-point". There
are several sequence-points and the most obvious is a ";".

The compiler is free to reorder anything in a compound statement
as long as it complies with presidence rules, but it can't re-order
the statements themselves.

In other words:

volatile unsigned int *hardware = virtual(MY_DEVICE);

    *hardware = 1;
    *hardware = 2;
    *hardware = 4;
    *hardware = 8;

.. happens exactly as shown above. If it didn't, you couldn't
write device drivers. An example of the code above:

 	.file	"xxx.c"
 	.text
.globl foo
 	.type	foo, @function
foo:
 	pushl	%ebp
 	movl	%esp, %ebp
 	subl	$4, %esp
 	movl	$305419896, -4(%ebp)	// init the pointer
 	movl	-4(%ebp), %eax		// Get pointer
 	movl	$1, (%eax)		// *hardware = 1;
 	movl	-4(%ebp), %eax		// Get pointer
 	movl	$2, (%eax)		// *hardware = 2;
 	movl	-4(%ebp), %eax		// Get pointer
 	movl	$4, (%eax)		// *hardware = 4;
 	movl	-4(%ebp), %eax		// Get pointer
 	movl	$8, (%eax)		// *hardware = 8;
 	movl	$0, %eax
 	leave
 	ret
 	.size	foo, .-foo
 	.section	.note.GNU-stack,"",@progbits
 	.ident	"GCC: (GNU) 3.3.3 20040412 (Red Hat Linux 3.3.3-7)"

Because of the necessary 'volatile' keyword, even the pointer is
obtained over again each time. This wastes CPU cycles. One
could do:

 	movl	$1, (%eax)
 	movl	$2, (%eax)
 	movl	$4, (%eax)
 	movl	$8, (%eax)

.. after the pointer is loaded. Unfortunately, 'C' doesn't have
a keyword that would accommodate that.

When communicating across a PCI/Bus, the writes are cached.
They don't necessarily occur __now__. It will take a (perhaps
dummy) read of the PCI/Bus to make them get to the hardware
now. Even then, they will still get there in order, all 4 writes,
because the interface is a FIFO. A read on the PCI/Bus will force
all pending writes to be written).

Some arcitectures do write-combining which means that, for instance
on an Intel Pentium P6, it's possible for all writes of adjacent
words may be condensed into a single quadword write. This may not be
what you want. If you have such a situation, one would execute
WBINVD after the critical writes. No not do this just to "make sure".
It wastes a lot of bandwidth.

See http://developer.intel.com/design/PentiumII/applnots/24442201.pdf



> Or because writes are encapsulated inside an inline function, therefore
> compiler
> must execute every single writes before returning from the inline function ?
>

In-line doesn't care. It's not as complicated as many expect.

> Thanks.
>
>            Francis
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12.5 on an i686 machine (5537.79 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: question on memory barrier

[Oliver Neukum]

Am Mittwoch, 24. August 2005 20:22 schrieb linux-os (Dick Johnson):
> > sorry but I'm not sure to understand you...Do you mean that the first write
> > into reg_a pointer will be completed before the second write because they're
> > separated by a (;) ?
> 
> Yes. The compiler must make sure that every effect of all previous
> code and all side-effects are complete at a "sequence-point". There
> are several sequence-points and the most obvious is a ";".

What the compiler may or may not generate is a little beside the point.
You have no guarantee that the CPU will execute these instructions in
the order given. If you need ordered writes use the appropriate barriers,
eg. wmb();
If this is PCI you also need to worry about the bridge caching. You
need to do dummy reads.

	Regards
		Oliver

Re: question on memory barrier

[linux-os (Dick Johnson)]

On Wed, 24 Aug 2005, Oliver Neukum wrote:

> Am Mittwoch, 24. August 2005 20:22 schrieb linux-os (Dick Johnson):
>>> sorry but I'm not sure to understand you...Do you mean that the first write
>>> into reg_a pointer will be completed before the second write because they're
>>> separated by a (;) ?
>>
>> Yes. The compiler must make sure that every effect of all previous
>> code and all side-effects are complete at a "sequence-point". There
>> are several sequence-points and the most obvious is a ";".
>
> What the compiler may or may not generate is a little beside the point.
> You have no guarantee that the CPU will execute these instructions in
> the order given. If you need ordered writes use the appropriate barriers,
> eg. wmb();

You mean these??

system.h: * For now, "wmb()" doesn't actually do anything, as all
system.h: * Some non intel clones support out of order store. wmb() ceases to be a
system.h:#define wmb() alternative("lock; addl $0,0(%%esp)", "sfence", X86_FEATURE_XMM)
system.h:#define wmb()	__asm__ __volatile__ ("": : :"memory")
system.h:#define smp_wmb()	wmb()
system.h:#define smp_wmb()	barrier()
system.h:#define set_wmb(var, value) do { var = value; wmb(); } while (0)

The "future" black magic that currently does nothing?????

> If this is PCI you also need to worry about the bridge caching. You
> need to do dummy reads.
>

And you never even bothered to read what I said about that???
The write ORDER will NOT change. Period. It's a FIFO. Writes
may be cached for awhile. You can force the writes, IN ORDER,
by doing a dummy read.


Too many people like to pretend that there is some black magic.
It works like it's supposed to. If it doesn't work, you have a bug.


> 	Regards
> 		Oliver
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12.5 on an i686 machine (5537.79 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: question on memory barrier

[Oliver Neukum]

> And you never even bothered to read what I said about that???
> The write ORDER will NOT change. Period. It's a FIFO. Writes

On current implementations of i386. That is not good enough.
We want code that works everywhere. You are wrong.
If you need to have ordered writes to a bus, use wmb().

	Regards
		Oliver

Re: question on memory barrier

[Andy Isaacson]

On Wed, Aug 24, 2005 at 07:31:31PM +0200, moreau francis wrote:
> --- "linux-os (Dick Johnson)" <linux-os@analogic.com> a écrit :
> > On Wed, 24 Aug 2005, moreau francis wrote:
> > > I'm currently trying to write a USB driver for Linux. The device must be
> > > configured by writing some values into the same register but I want to be
> > > sure that the writing order is respected by either the compiler and the
> > > cpu.
[snip]
> > > static inline void dev_out(u32 *reg, u32 value)
> > > {
> > >        writel(value, regs);
> > > }
> > >
> > > void config_dev(void)
> > > {
> > >        dev_out(reg_a, 0x0); /* first io */
> > >        dev_out(reg_a, 0xA); /* second io */
> > > }
> > >
> > 
> > This should be fine. The effect of the first bit of code
> > plus all side-effects (if any) should be complete at the
> > first effective sequence-point (;) but you need to

While sequence points are relevant for purposes of reasoning about the
effects of C code on the abstract state machine as defined by the C
standard, they are irrelevant when talking about IO.

> sorry but I'm not sure to understand you...Do you mean that the first write
> into reg_a pointer will be completed before the second write because they're
> separated by a (;) ?
> Or because writes are encapsulated inside an inline function,
> therefore compiler must execute every single writes before returning
> from the inline function ?

The first register write will be completed before the second register
write because you use writel, which is defined to have the semantics you
want.  (It uses a platform-specific method to guarantee this, possibly
"volatile" or "asm("eieio")" or whatever method your platform requires.)

The sequence points, by themselves, do not make any requirements on the
externally visible behavior of the program.  Nor does the fact that
there's an inline function involved.  It's just the writel() that
contains the magic to force in-order execution.

You might benefit by running your source code through gcc -E and seeing
what the writel() expands to.  (I do something like "rm drivers/mydev.o;
make V=1" and then copy-n-paste the gcc line, replacing the "-c -o mydev.o"
options with -E.)

The sequence point argument is obviously wrong, BTW - if it were the
case that a mere sequence point required the compiler to have completed
all externally-visible side effects, then almost every optimization that
gcc does with -O2 would be impossible.  CSE, loop splitting, etc.

-andy

Re: question on memory barrier

[Jesse Barnes]

On Wednesday, August 24, 2005 12:48 pm, Andy Isaacson wrote:
> The first register write will be completed before the second register
> write because you use writel, which is defined to have the semantics
> you want.  (It uses a platform-specific method to guarantee this,
> possibly "volatile" or "asm("eieio")" or whatever method your platform
> requires.)

writel() ensures ordering?  Only from one CPU, another CPU issuing a 
write at some later time may have its write arrive first.  See 
Documentation/io_ordering.txt for some documentation I put together on 
this issue.

Jesse

Re: question on memory barrier

[Alan Cox]

On Mer, 2005-08-24 at 12:53 -0700, Jesse Barnes wrote:
> writel() ensures ordering?  Only from one CPU, another CPU issuing a 
> write at some later time may have its write arrive first.  See 
> Documentation/io_ordering.txt for some documentation I put together on 
> this issue.

And in more detail from the deviceiobook..

      <para>
        In addition to write posting, on some large multiprocessing
systems
        (e.g. SGI Challenge, Origin and Altix machines) posted writes
won't
        be strongly ordered coming from different CPUs.  Thus it's
important
        to properly protect parts of your driver that do memory-mapped
writes
        with locks and use the <function>mmiowb</function> to make sure
they
        arrive in the order intended.  Issuing a regular <function>readX
        </function> will also ensure write ordering, but should only be
used
        when the driver has to be sure that the write has actually
arrived
        at the device (not that it's simply ordered with respect to
other
        writes), since a full <function>readX</function> is a relatively
        expensive operation.
      </para>

Re: question on memory barrier

[Jesse Barnes]

On Wednesday, August 24, 2005 2:45 pm, Alan Cox wrote:
> And in more detail from the deviceiobook..
>
>       <para>
>         In addition to write posting, on some large multiprocessing
> systems
>         (e.g. SGI Challenge, Origin and Altix machines) posted writes
> won't
>         be strongly ordered coming from different CPUs.  Thus it's
> important
>         to properly protect parts of your driver that do memory-mapped
> writes
>         with locks and use the <function>mmiowb</function> to make
> sure they
>         arrive in the order intended.  Issuing a regular
> <function>readX </function> will also ensure write ordering, but
> should only be used
>         when the driver has to be sure that the write has actually
> arrived
>         at the device (not that it's simply ordered with respect to
> other
>         writes), since a full <function>readX</function> is a
> relatively expensive operation.
>       </para>

Yeah, wrote that too :).  io_ordering.txt should probably just get folded 
into deviceiobook at some point...  Or we could just replace both with 
URL pointers to LDD vol. 3. ;)

Jesse

Re: question on memory barrier

[linux-os (Dick Johnson)]

On Wed, 24 Aug 2005, Andy Isaacson wrote:

> On Wed, Aug 24, 2005 at 07:31:31PM +0200, moreau francis wrote:
>> --- "linux-os (Dick Johnson)" <linux-os@analogic.com> a écrit :
>>> On Wed, 24 Aug 2005, moreau francis wrote:
>>>> I'm currently trying to write a USB driver for Linux. The device must be
>>>> configured by writing some values into the same register but I want to be
>>>> sure that the writing order is respected by either the compiler and the
>>>> cpu.
> [snip]
>>>> static inline void dev_out(u32 *reg, u32 value)
>>>> {
>>>>        writel(value, regs);
>>>> }
>>>>
>>>> void config_dev(void)
>>>> {
>>>>        dev_out(reg_a, 0x0); /* first io */
>>>>        dev_out(reg_a, 0xA); /* second io */
>>>> }
>>>>
>>>
>>> This should be fine. The effect of the first bit of code
>>> plus all side-effects (if any) should be complete at the
>>> first effective sequence-point (;) but you need to
>
> While sequence points are relevant for purposes of reasoning about the
> effects of C code on the abstract state machine as defined by the C
> standard, they are irrelevant when talking about IO.
>
>> sorry but I'm not sure to understand you...Do you mean that the first write
>> into reg_a pointer will be completed before the second write because they're
>> separated by a (;) ?
>> Or because writes are encapsulated inside an inline function,
>> therefore compiler must execute every single writes before returning
>> from the inline function ?
>
> The first register write will be completed before the second register
> write because you use writel, which is defined to have the semantics you
> want.  (It uses a platform-specific method to guarantee this, possibly
> "volatile" or "asm("eieio")" or whatever method your platform requires.)
>
> The sequence points, by themselves, do not make any requirements on the
> externally visible behavior of the program.  Nor does the fact that
> there's an inline function involved.  It's just the writel() that
> contains the magic to force in-order execution.
>
> You might benefit by running your source code through gcc -E and seeing
> what the writel() expands to.  (I do something like "rm drivers/mydev.o;
> make V=1" and then copy-n-paste the gcc line, replacing the "-c -o mydev.o"
> options with -E.)
>
> The sequence point argument is obviously wrong, BTW - if it were the
> case that a mere sequence point required the compiler to have completed
> all externally-visible side effects, then almost every optimization that
> gcc does with -O2 would be impossible.  CSE, loop splitting, etc.
>
> -andy


Wrong. Reference:

http://www.phy.duke.edu/~rgb/General/c_book/c_book/chapter8/sequence_points.html

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12.5 on an i686 machine (5537.79 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: question on memory barrier

[Oliver Neukum]

> > The sequence point argument is obviously wrong, BTW - if it were the
> > case that a mere sequence point required the compiler to have completed
> > all externally-visible side effects, then almost every optimization that
> > gcc does with -O2 would be impossible.  CSE, loop splitting, etc.
> >
> > -andy
> 
> 
> Wrong. Reference:
> 
> http://www.phy.duke.edu/~rgb/General/c_book/c_book/chapter8/sequence_points.html

This refers to externally visible in the view of _one_ cpu , not other
CPUs or other hardware on a bus.
Externally visible means by references to other routines.
In practice function calls.

m = a * b;
c += m;
x(c);
d -= m;
y(d);

is better than

c += (a * b);
x(c);
d -= (a * b);
y(d);

But this doesn't tell much more. The compiler is free to generate code that
acts like all values hit main memory at a ;. Only where it can't tell, it really
needs to write it out. Writes to memory may or may not be of that category,
it depends on how far aliasing can be computed.
In any case nothing of that applies to the order data goes onto the bus.

	Regards
		Oliver

RE: question on memory barrier

[David Schwartz]

> Wrong. Reference:
>
> http://www.phy.duke.edu/~rgb/General/c_book/c_book/chapter8/sequen
> ce_points.html
>
> Cheers,
> Dick Johnson

	That discussion is wrong. The form of the argument is simply invalid.

	Just because an optimization could break things in some cases doesn’t mean
the compiler can’t ever make the optimization. It just can’t make the
optimization in the case that breaks things. And by “things” I mean things
that are defined in the standard that would be broken, not things outside of
it.

	DS

Re: question on memory barrier

[Denis Vlasenko]

> You might benefit by running your source code through gcc -E and seeing
> what the writel() expands to.  (I do something like "rm drivers/mydev.o;
> make V=1" and then copy-n-paste the gcc line, replacing the "-c -o mydev.o"
> options with -E.)

Just use "make drivers/mydev.i"
--
vda

Re: question on memory barrier

[moreau francis]

--- Andy Isaacson <adi@hexapodia.org> a écrit :

> 
> The first register write will be completed before the second register
> write because you use writel, which is defined to have the semantics you
> want.  (It uses a platform-specific method to guarantee this, possibly
> "volatile" or "asm("eieio")" or whatever method your platform requires.)

I'm compiling Linux kernel for a MIPS32 cpu. On my platform, writel seems
expand to:

    static inline writel(u32 val, volatile void __iomem *mem)
    {
            volatile u32 *__mem;
            u32 __val;

            __mem = (void *)((unsigned long)(mem));
            __val = val;

            *__mem = __val;
    }

I don't see the magic in it since "volatile" keyword do not handle memory
ordering constraints...Linus wrote on volatile keyword, see
http://www.ussg.iu.edu/hypermail/linux/kernel/0401.0/1387.html

> 
> The sequence points, by themselves, do not make any requirements on the
> externally visible behavior of the program.  Nor does the fact that
> there's an inline function involved.  It's just the writel() that
> contains the magic to force in-order execution.
> 
> You might benefit by running your source code through gcc -E and seeing
> what the writel() expands to.  (I do something like "rm drivers/mydev.o;
> make V=1" and then copy-n-paste the gcc line, replacing the "-c -o mydev.o"
> options with -E.)

make drivers/mydev.i should do the job but preprocessor doesn't expand inline
functions. So I won't be able to see the expanded writel function.

> 
> The sequence point argument is obviously wrong, BTW - if it were the
> case that a mere sequence point required the compiler to have completed
> all externally-visible side effects, then almost every optimization that
> gcc does with -O2 would be impossible.  CSE, loop splitting, etc.
> 
> -andy
> 



	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
Téléchargez cette version sur http://fr.messenger.yahoo.com

Re: question on memory barrier

[Alan Cox]

On Iau, 2005-08-25 at 11:14 +0200, moreau francis wrote:
> I'm compiling Linux kernel for a MIPS32 cpu. On my platform, writel seems
> expand to:
> 
>     static inline writel(u32 val, volatile void __iomem *mem)
>     {
>             volatile u32 *__mem;
>             u32 __val;
> 
>             __mem = (void *)((unsigned long)(mem));
>             __val = val;
> 
>             *__mem = __val;
>     }
> 
> I don't see the magic in it since "volatile" keyword do not handle memory
> ordering constraints...Linus wrote on volatile keyword, see

For the case and the platform the volatile is sufficient to force
ordering. That or the arch code author made a mistake. But I think its
sufifcient for MIPS. The volatile prevents

	*foo = 1; 
	*foo = 2;

or even

	*foo;

from being collapsed together or eliminated

Re: question on memory barrier

[Andy Isaacson]

On Thu, Aug 25, 2005 at 11:14:03AM +0200, moreau francis wrote:
> --- Andy Isaacson <adi@hexapodia.org> a écrit :
> > The first register write will be completed before the second register
> > write because you use writel, which is defined to have the semantics you
> > want.  (It uses a platform-specific method to guarantee this, possibly
> > "volatile" or "asm("eieio")" or whatever method your platform requires.)
> 
> I'm compiling Linux kernel for a MIPS32 cpu.

Funny, me too.  (Well, mostly MIPS64, but some MIPS32.)

> On my platform, writel seems
> expand to:
> 
>     static inline writel(u32 val, volatile void __iomem *mem)
>     {
>             volatile u32 *__mem;
>             u32 __val;
> 
>             __mem = (void *)((unsigned long)(mem));
>             __val = val;
> 
>             *__mem = __val;
>     }
> 
> I don't see the magic in it since "volatile" keyword do not handle memory
> ordering constraints...Linus wrote on volatile keyword, see
> http://www.ussg.iu.edu/hypermail/linux/kernel/0401.0/1387.html

Did you *read* the post?

# The _only_ acceptable use of "volatile" is basically:
# 
# - in _code_ (not data structures), where we might mark a place as making
#   a special type of access. For example, in the PCI MMIO read functions,
#   rather than use inline assembly to force the particular access (which
#   would work equally well), we can force the pointer to a volatile type.

That's *exactly* what the writel you quote above does!

The thing that Linus is railing against is stupidity like

struct my_dev_regs {
	volatile u8 reg0;
};

That "volatile" does not do what you might think it would do.

To return to the point directly at hand - on MIPS architectures to date,
simply doing your memory access through a "volatile u32 *" is sufficient
to ensure that the IO hits the bus (assuming that your pointer points to
kseg1, not kseg0, or is otherwise uncached), because 'volatile' forces
gcc to generate a "sw" for each store, and all MIPS so far have been
designed so that multiple uncached writes to mmio locations do generate
multiple bus transactions.

I'm not an architect, but I think it would be possible to build a MIPS
where this was not the case, and require additional contortions from
users.  Such a MIPS would suck to program and would probably fail in the
marketplace, and there's no compelling benefit to doing so; ergo, I
would expect "volatile" to continue to be sufficient on MIPS.

> make drivers/mydev.i should do the job

Thanks for the pointer!  

> but preprocessor doesn't expand inline
> functions. So I won't be able to see the expanded writel function.

But you can simply search backwards in the .i file!  The whole point of
inline functions is that they're present in the postprocessed text.  The
vim command "#" is useful for this.

-andy

Re: question on memory barrier

[moreau francis]

--- Andy Isaacson <adi@hexapodia.org> a écrit :
> 
> Did you *read* the post?
> 
> # The _only_ acceptable use of "volatile" is basically:
> # 
> # - in _code_ (not data structures), where we might mark a place as making
> #   a special type of access. For example, in the PCI MMIO read functions,
> #   rather than use inline assembly to force the particular access (which
> #   would work equally well), we can force the pointer to a volatile type.
> 
> That's *exactly* what the writel you quote above does!

OK but he speaks about special type of access, no ordering constraint.
I don't think that MIPS cpu reorder memory access, but gcc can ! And I
don't think that the use of 'volatile' can prevent it to do that.


> To return to the point directly at hand - on MIPS architectures to date,
> simply doing your memory access through a "volatile u32 *" is sufficient
> to ensure that the IO hits the bus (assuming that your pointer points to
> kseg1, not kseg0, or is otherwise uncached), because 'volatile' forces
> gcc to generate a "sw" for each store, and all MIPS so far have been
> designed so that multiple uncached writes to mmio locations do generate
> multiple bus transactions.

ok thanks for this, but once again, there's no ordering constraint garantuee.

> 
> I'm not an architect, but I think it would be possible to build a MIPS
> where this was not the case, and require additional contortions from
> users.  Such a MIPS would suck to program and would probably fail in the
> marketplace, and there's no compelling benefit to doing so; ergo, I
> would expect "volatile" to continue to be sufficient on MIPS.
> 

I hope so...It's hard to find out an answer to such questions (maybe
it's the case only for MIPS arch) although it's an important point.

Thanks.
             Francis


	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
Téléchargez cette version sur http://fr.messenger.yahoo.com

Re: question on memory barrier

[Maciej W. Rozycki]

On Fri, 26 Aug 2005, moreau francis wrote:

> I don't think that MIPS cpu reorder memory access, but gcc can ! And I
> don't think that the use of 'volatile' can prevent it to do that.

 Well, certain MIPS implementations may merge multiple uncached writes in 
the writeback buffer, e.g. writes to different bytes within a single 
aligned word.  This is true for consecutive writes; I'm not sure this 
permits jumping the writeback queue, though.

> > To return to the point directly at hand - on MIPS architectures to date,
> > simply doing your memory access through a "volatile u32 *" is sufficient
> > to ensure that the IO hits the bus (assuming that your pointer points to
> > kseg1, not kseg0, or is otherwise uncached), because 'volatile' forces
> > gcc to generate a "sw" for each store, and all MIPS so far have been
> > designed so that multiple uncached writes to mmio locations do generate
> > multiple bus transactions.

 Unfortunately this is not true -- see above.  This is why even wmb() 
isn't a no-op on MIPS.

  Maciej

Re: question on memory barrier

[Maciej W. Rozycki]

On Wed, 24 Aug 2005, Andy Isaacson wrote:

> You might benefit by running your source code through gcc -E and seeing
> what the writel() expands to.  (I do something like "rm drivers/mydev.o;
> make V=1" and then copy-n-paste the gcc line, replacing the "-c -o mydev.o"
> options with -E.)

 Well, `make drivers/mydev.i' does the same and is simpler. ;-)

  Maciej

Re: question on memory barrier

[Alan Cox]

On Mer, 2005-08-24 at 14:43 +0200, moreau francis wrote:
> I'm currently trying to write a USB driver for Linux. The device must be
> configured by writing some values into the same register but I want to be
> sure that the writing order is respected by either the compiler and the cpu.

The Linux kernel defines writel() in such a way that for each platform a
series of writel() calls are ordered with respect to the processor. In
other words if on one processor you issue 

	writel(0, foo);
	writel(1, foo);
	writel(2, foo);

the hardware will see 0, 1, 2. writel does not guarantee that the write
occurs immediately so while you know the writes are ordered you don't
know the write has "arrived" at the device when the writel() call
returns. That isn't usually a problem except when delays are required.
Then you need to avoid PCI posting and do

	writel(0, foo);
	readl(foo);
	udelay(50);
	writel(1, foo);

The only other complication is multiprocessor systems - if you have
multiple places that may issue these I/O's you may need a lock to
protect them from both processors configuring at the same time, and in
theory an mmiowb() call to ensure the first processor has finished its
I/O before the second starts - ie

	spin_lock(&conf_lock);
	writel(0, foo);
	writel(1, foo);
	mmiowb();
	spin_unlock(&conf_lock);


The "wmb/rmb" are barriers to memory not to device I/O. The locking
functions (spin_unlock etc) are implicit memory barriers, but atomic
operations are not.

Generally speaking if you use writel the right semantics just happen,
and that is why the writel definition is the way it is.