Accessing flash directly from User Space

Accessing flash directly from User Space

[Jonathan Haws]

I know this is probably a really dumb question, but a wise man once said th=
at the only stupid question is the one that is not asked.

So, I have written a flash driver in VxWorks that simply addresses the flas=
h directly and handles all the hardware accesses just fine.  I am porting t=
hat to Linux and need it to run in user space (mainly to simplify the inter=
face with the user - I want to keep it the same as in VxWorks).  Here is a =
snippet of what my question is:

static uint8_t bflashEraseSector(int sa, int verbose)
{
	uint16_t * flash =3D (uint16_t *) NOR_FLASH_BASE_ADRS;
	uint32_t offset;

	...

	/* We divide by 2 here to adjust for the 16-bit offset into the address */
	offset =3D sa * NOR_FLASH_SECTOR_SIZE / 2;
	flash[BFLASH_SECTOR_ERASE_ADDR1] =3D BFLASH_SECTOR_ERASE_BYTE1;

	...

}

I am trying to get a pointer to NOR_FLASH_BASE_ADRS which is defined to be =
0xFC000000.  I then dereference that directly to write to the flash.

How can I get that pointer?  Unfortunately I cannot simply use the address =
of the flash.  Is there some magical function call that gives me access to =
that portion of the memory space?

Thanks for the help!

Jonathan

PS - I know that I could simply use the MTD driver provided by the kernel, =
but I need to be able to keep the interface the same so we can use previous=
ly written code.

Re: Accessing flash directly from User Space

[Bill Gatliff]

Jonathan Haws wrote:
>
> How can I get that pointer?  Unfortunately I cannot simply use the address of the flash.  Is there some magical function call that gives me access to that portion of the memory space?
>   

$ man 2 mmap

You want MAP_SHARED and O_SYNC.


b.g.

-- 
Bill Gatliff
bgat@billgatliff.com

RE: Accessing flash directly from User Space

[Jonathan Haws]

> > How can I get that pointer?  Unfortunately I cannot simply use the
> address of the flash.  Is there some magical function call that
> gives me access to that portion of the memory space?
> >
>=20
> $ man 2 mmap
>=20
> You want MAP_SHARED and O_SYNC.


To use that I need to have a file descriptor to a device, do I not?  Howeve=
r, I do not have a base flash driver to give me that file descriptor.  Am I=
 missing something with that call?

Thanks!

Re: Accessing flash directly from User Space

[Bill Gatliff]

Jonathan Haws wrote:
>>> How can I get that pointer?  Unfortunately I cannot simply use the
>>>       
>> address of the flash.  Is there some magical function call that
>> gives me access to that portion of the memory space?
>>     
>> $ man 2 mmap
>>
>> You want MAP_SHARED and O_SYNC.
>>     
>
>
> To use that I need to have a file descriptor to a device, do I not?  However, I do not have a base flash driver to give me that file descriptor.  Am I missing something with that call?
>   

/dev/mem


b.g.

-- 
Bill Gatliff
bgat@billgatliff.com

RE: Accessing flash directly from User Space

[Jonathan Haws]

> Jonathan Haws wrote:
> >>> How can I get that pointer?  Unfortunately I cannot simply use
> the
> >>>
> >> address of the flash.  Is there some magical function call that
> >> gives me access to that portion of the memory space?
> >>
> >> $ man 2 mmap
> >>
> >> You want MAP_SHARED and O_SYNC.
> >>
> >
> >
> > To use that I need to have a file descriptor to a device, do I
> not?  However, I do not have a base flash driver to give me that
> file descriptor.  Am I missing something with that call?
> >
>=20
> /dev/mem


Ah, yes.  I told you this was going to be a dumb question.

Thanks, Bill.

Jonathan

RE: Accessing flash directly from User Space

[Jonathan Haws]

> >>> How can I get that pointer?  Unfortunately I cannot simply use
> the
> >>>
> >> address of the flash.  Is there some magical function call that
> >> gives me access to that portion of the memory space?
> >>
> >> $ man 2 mmap
> >>
> >> You want MAP_SHARED and O_SYNC.
> >>
> >
> >
> > To use that I need to have a file descriptor to a device, do I
> not?  However, I do not have a base flash driver to give me that
> file descriptor.  Am I missing something with that call?
> >
>=20
> /dev/mem
>=20
Okay, I now have access to the flash memory, however when I write to it the=
 writes do not take.  I have tried calling msync() on the mapping to no ava=
il.  I have opened the fd with O_SYNC, but cannot get things to work right.

Here are the calls:

	int fd =3D open("/dev/mem", O_SYNC | O_RDWR);
	uint16_t * flash =3D (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
			NOR_FLASH_BASE_ADRS);

When I do flash[0] =3D 0x1234, and then check the value, they do not match.

	flash[0] =3D 0x1234;
	msync(flash, NOR_FLASH_SIZE, MS_SYNC | MS_INVALIDATE);
	printf("flash[0] =3D %#04x\n", flash[0]);

That prints flash[0] =3D 0x7f45.  I have verified that I am reading the cor=
rect values.  I can display the flash contents in U-Boot and 7f45 is what i=
s in the first 16 bits of flash.

Why can I not write to flash?  What am I doing wrong?

Thanks!

Jonathan

Re: Accessing flash directly from User Space

[Scott Wood]

Jonathan Haws wrote:
> Okay, I now have access to the flash memory, however when I write to it the writes do not take.  I have tried calling msync() on the mapping to no avail.  I have opened the fd with O_SYNC, but cannot get things to work right.
> 
> Here are the calls:
> 
> 	int fd = open("/dev/mem", O_SYNC | O_RDWR);
> 	uint16_t * flash = (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> 			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> 			NOR_FLASH_BASE_ADRS);
> 
> When I do flash[0] = 0x1234, and then check the value, they do not match.
> 
> 	flash[0] = 0x1234;
> 	msync(flash, NOR_FLASH_SIZE, MS_SYNC | MS_INVALIDATE);
> 	printf("flash[0] = %#04x\n", flash[0]);
> 
> That prints flash[0] = 0x7f45.  I have verified that I am reading the correct values.  I can display the flash contents in U-Boot and 7f45 is what is in the first 16 bits of flash.
> 
> Why can I not write to flash?  What am I doing wrong?

Flash does not work that way -- you must send it commands to erase a 
block, and then further commands to program new data.

It sounds like what you really want is the /dev/mtd or /dev/mtdblock 
interface, not raw access to the flash chip.

-Scott

Re: Accessing flash directly from User Space

[Alessandro Rubini]

> Okay, I now have access to the flash memory, however when I write to
> it the writes do not take.


> 			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,

MAP_SHARED. Bill told you.  With MAP_PRIVATE you write to a local
in-ram copy of the data, not to the original one.

/alessandro

RE: Accessing flash directly from User Space

[Jonathan Haws]

> > Okay, I now have access to the flash memory, however when I write
> to
> > it the writes do not take.
>=20
>=20
> > 			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
>=20
> MAP_SHARED. Bill told you.  With MAP_PRIVATE you write to a local
> in-ram copy of the data, not to the original one.

I apologize, that MAP_PRIVATE was leftover from me trying to get it to work=
.  With MAP_SHARED I am having the problem.

Sorry for the confusion.

RE: Accessing flash directly from User Space

[Jonathan Haws]

> > Okay, I now have access to the flash memory, however when I write
> to it the writes do not take.  I have tried calling msync() on the
> mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> get things to work right.
> >
> > Here are the calls:
> >
> > 	int fd =3D open("/dev/mem", O_SYNC | O_RDWR);
> > 	uint16_t * flash =3D (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> > 			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> > 			NOR_FLASH_BASE_ADRS);
> >
> > When I do flash[0] =3D 0x1234, and then check the value, they do not
> match.
> >
> > 	flash[0] =3D 0x1234;
> > 	msync(flash, NOR_FLASH_SIZE, MS_SYNC | MS_INVALIDATE);
> > 	printf("flash[0] =3D %#04x\n", flash[0]);
> >
> > That prints flash[0] =3D 0x7f45.  I have verified that I am reading
> the correct values.  I can display the flash contents in U-Boot and
> 7f45 is what is in the first 16 bits of flash.
> >
> > Why can I not write to flash?  What am I doing wrong?
>=20
> Flash does not work that way -- you must send it commands to erase a
> block, and then further commands to program new data.

I realize that.  I have a driver written that does exactly that.  However, =
I need to be able to write to certain registers to setup the erasure.  The =
driver works perfectly in VxWorks, now I am porting it to Linux.

> It sounds like what you really want is the /dev/mtd or /dev/mtdblock
> interface, not raw access to the flash chip.

As mentioned in my initial post, I need to use my custom driver to maintain=
 the interface to the application that uses the flash for data storage.

I had thought about using MTD, but decided against it because with previous=
 benchmarking that we did with MTD and our custom driver, we found that our=
 custom driver was about 10x faster.

Re: Accessing flash directly from User Space

[Alessandro Rubini]

> I realize that.  I have a driver written that does exactly that.
> However, I need to be able to write to certain registers to setup
> the erasure.  The driver works perfectly in VxWorks, now I am
> porting it to Linux.

Most likey you are hit by the compiler reordering the accesses and
caching data in registers. You should try to use volatile pointers
(but also read the "volatile considered harmful" file in the kernel
sources -- anyways you are doing bad things nonetheless), or add
memory barriers (copy the mb() and wmb() macros defined for kernel
space).

Then, the first-word-access you show is wrong, as Scott Wood told you,
so the fact that _that_ access is not working even across msync
that's another matter.

I usually have no serious problems with driving hardware from
/dev/mem, although I've never done flash specifically as mtd is there.

hope this helps
/alessandro

Re: Accessing flash directly from User Space

[Scott Wood]

Jonathan Haws wrote:
>>> 	flash[0] = 0x1234;
>>> 	msync(flash, NOR_FLASH_SIZE, MS_SYNC | MS_INVALIDATE);
>>> 	printf("flash[0] = %#04x\n", flash[0]);
>>>
>>> That prints flash[0] = 0x7f45.  I have verified that I am reading
>> the correct values.  I can display the flash contents in U-Boot and
>> 7f45 is what is in the first 16 bits of flash.
>>> Why can I not write to flash?  What am I doing wrong?
>> Flash does not work that way -- you must send it commands to erase a
>> block, and then further commands to program new data.
> 
> I realize that.  I have a driver written that does exactly that.
> However, I need to be able to write to certain registers to setup the
> erasure. 

Will the device respond to 0x1234 being written at offset zero?  You 
generally have to poke these things pretty specifically in order to get 
them to go into command mode.

> The driver works perfectly in VxWorks,

Including the 0x1234 thing?

>> It sounds like what you really want is the /dev/mtd or /dev/mtdblock
>> interface, not raw access to the flash chip.
> 
> As mentioned in my initial post, I need to use my custom driver to maintain the interface to the application that uses the flash for data storage.
> 
> I had thought about using MTD, but decided against it because with
> previous benchmarking that we did with MTD and our custom driver, we
> found that our custom driver was about 10x faster.

Ouch.  Any idea where the slowdown is coming from?

-Scott

RE: Accessing flash directly from User Space

[Jonathan Haws]

> Jonathan Haws wrote:
> >>> 	flash[0] =3D 0x1234;
> >>> 	msync(flash, NOR_FLASH_SIZE, MS_SYNC | MS_INVALIDATE);
> >>> 	printf("flash[0] =3D %#04x\n", flash[0]);
> >>>
> >>> That prints flash[0] =3D 0x7f45.  I have verified that I am
> reading
> >> the correct values.  I can display the flash contents in U-Boot
> and
> >> 7f45 is what is in the first 16 bits of flash.
> >>> Why can I not write to flash?  What am I doing wrong?
> >> Flash does not work that way -- you must send it commands to
> erase a
> >> block, and then further commands to program new data.
> >
> > I realize that.  I have a driver written that does exactly that.
> > However, I need to be able to write to certain registers to setup
> the
> > erasure.
>=20
> Will the device respond to 0x1234 being written at offset zero?  You
> generally have to poke these things pretty specifically in order to
> get
> them to go into command mode.
>=20

It should because that is the first data location in flash.  Also, just to =
be sure I am telling the truth, I tried writing to one of the registers to =
setup an erase and got the same results - the value did not get written.

> > The driver works perfectly in VxWorks,
>=20
> Including the 0x1234 thing?

Actually, I have not tried that - I have not had to since the driver worked=
.

> >> It sounds like what you really want is the /dev/mtd or
> /dev/mtdblock
> >> interface, not raw access to the flash chip.
> >
> > As mentioned in my initial post, I need to use my custom driver to
> maintain the interface to the application that uses the flash for
> data storage.
> >
> > I had thought about using MTD, but decided against it because with
> > previous benchmarking that we did with MTD and our custom driver,
> we
> > found that our custom driver was about 10x faster.
>=20
> Ouch.  Any idea where the slowdown is coming from?

>From what I remember (I would have to dig up notes to make sure) it is some=
thing to do with MTD looking for a signal to go high that is processed a bu=
nch before MTD even sees it.  Our flash produces a hardware ready signal th=
at we are triggering off of to move on.  MTD took much longer to report to =
us that the hardware was ready.

Thanks

RE: Accessing flash directly from User Space

[Kenneth Johansson]

On Tue, 2009-10-27 at 16:52 -0600, Jonathan Haws wrote:
> > Jonathan Haws wrote:
> > > I had thought about using MTD, but decided against it because with
> > > previous benchmarking that we did with MTD and our custom driver,
> > we
> > > found that our custom driver was about 10x faster.
> > 
> > Ouch.  Any idea where the slowdown is coming from?
> 
> From what I remember (I would have to dig up notes to make sure) it is something to do with MTD looking for a signal to go high that is processed a bunch before MTD even sees it.  Our flash produces a hardware ready signal that we are triggering off of to move on.  MTD took much longer to report to us that the hardware was ready.
> 
> Thanks
> 

It would be interesting to know in more detail what is was. If we have a
10x performance increase hiding from for us I would be very interested
in knowing where it is.

Are you using some custom command to the flash that the generic chip
drivers in Linux is not yet supporting ?

Re: Accessing flash directly from User Space

[Josh Boyer]

On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
>> >>> How can I get that pointer?  Unfortunately I cannot simply use
>> the
>> >>>
>> >> address of the flash.  Is there some magical function call that
>> >> gives me access to that portion of the memory space?
>> >>
>> >> $ man 2 mmap
>> >>
>> >> You want MAP_SHARED and O_SYNC.
>> >>
>> >
>> >
>> > To use that I need to have a file descriptor to a device, do I
>> not?  However, I do not have a base flash driver to give me that
>> file descriptor.  Am I missing something with that call?
>> >
>> 
>> /dev/mem
>> 
>Okay, I now have access to the flash memory, however when I write to it the writes do not take.  I have tried calling msync() on the mapping to no avail.  I have opened the fd with O_SYNC, but cannot get things to work right.
>
>Here are the calls:
>
>	int fd = open("/dev/mem", O_SYNC | O_RDWR);
>	uint16_t * flash = (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
>			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
>			NOR_FLASH_BASE_ADRS);

What board and CPU are you using?  Is your flash really at 0xFC800000, or is
that the virtual address that VxWorks puts it at?

josh

RE: Accessing flash directly from User Space

[Jonathan Haws]

> On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
> >> >>> How can I get that pointer?  Unfortunately I cannot simply
> use
> >> the
> >> >>>
> >> >> address of the flash.  Is there some magical function call
> that
> >> >> gives me access to that portion of the memory space?
> >> >>
> >> >> $ man 2 mmap
> >> >>
> >> >> You want MAP_SHARED and O_SYNC.
> >> >>
> >> >
> >> >
> >> > To use that I need to have a file descriptor to a device, do I
> >> not?  However, I do not have a base flash driver to give me that
> >> file descriptor.  Am I missing something with that call?
> >> >
> >>
> >> /dev/mem
> >>
> >Okay, I now have access to the flash memory, however when I write
> to it the writes do not take.  I have tried calling msync() on the
> mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> get things to work right.
> >
> >Here are the calls:
> >
> >	int fd =3D open("/dev/mem", O_SYNC | O_RDWR);
> >	uint16_t * flash =3D (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> >			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> >			NOR_FLASH_BASE_ADRS);
>=20
> What board and CPU are you using?  Is your flash really at
> 0xFC800000, or is
> that the virtual address that VxWorks puts it at?

I am using a custom board based on the AMCC Kilauea development board.  It =
uses a 405EX CPU.  Yes, the flash is really at 0xFC000000.

RE: Accessing flash directly from User Space

[Jonathan Haws]

> On Tue, 2009-10-27 at 16:52 -0600, Jonathan Haws wrote:
> > > Jonathan Haws wrote:
> > > > I had thought about using MTD, but decided against it because
> with
> > > > previous benchmarking that we did with MTD and our custom
> driver,
> > > we
> > > > found that our custom driver was about 10x faster.
> > >
> > > Ouch.  Any idea where the slowdown is coming from?
> >
> > From what I remember (I would have to dig up notes to make sure)
> it is something to do with MTD looking for a signal to go high that
> is processed a bunch before MTD even sees it.  Our flash produces a
> hardware ready signal that we are triggering off of to move on.  MTD
> took much longer to report to us that the hardware was ready.
> >
> > Thanks
> >
>=20
> It would be interesting to know in more detail what is was. If we
> have a
> 10x performance increase hiding from for us I would be very
> interested
> in knowing where it is.
>=20
> Are you using some custom command to the flash that the generic chip
> drivers in Linux is not yet supporting ?

Looking through our notes and talking with the engineer who was performing =
the tests, it was exactly that - MTD was waiting for a signal that was prod=
uced differently than the hardware ready signal.  By simply polling the fla=
sh until the hardware ready signal toggled we were able to get a much faste=
r read and write speed.  Granted, most of our signals are being sent throug=
h a CPLD, so that may be why MTD did not work as well.

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> > On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
> > >> >>> How can I get that pointer?  Unfortunately I cannot simply
> > use
> > >> the
> > >> >>>
> > >> >> address of the flash.  Is there some magical function call
> > that
> > >> >> gives me access to that portion of the memory space?
> > >> >>
> > >> >> $ man 2 mmap
> > >> >>
> > >> >> You want MAP_SHARED and O_SYNC.
> > >> >>
> > >> >
> > >> >
> > >> > To use that I need to have a file descriptor to a device, do
> I
> > >> not?  However, I do not have a base flash driver to give me
> that
> > >> file descriptor.  Am I missing something with that call?
> > >> >
> > >>
> > >> /dev/mem
> > >>
> > >Okay, I now have access to the flash memory, however when I write
> > to it the writes do not take.  I have tried calling msync() on the
> > mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> > get things to work right.
> > >
> > >Here are the calls:
> > >
> > >	int fd =3D open("/dev/mem", O_SYNC | O_RDWR);
> > >	uint16_t * flash =3D (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> > >			(PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> > >			NOR_FLASH_BASE_ADRS);
> >
> > What board and CPU are you using?  Is your flash really at
> > 0xFC800000, or is
> > that the virtual address that VxWorks puts it at?
>=20
> I am using a custom board based on the AMCC Kilauea development
> board.  It uses a 405EX CPU.  Yes, the flash is really at
> 0xFC000000.

I have found the problem.  It occurred to me in the shower (okay not really=
, but most good ideas happen there).

What was happening is that I was in fact able to write to the correct regis=
ters.  However, I would try and write to them in a batch.  But the way mmap=
 works (at least according to the man page) with MAP_SHARED is that the fil=
e may not be updated until msync() is called.  Now, I thought that O_SYNC w=
ould take care of that when I open /dev/mem, but that was not the case.

Anyway, to make a long story short, I inserted an msync() after each assign=
ment to the flash.  This resolved my problem and I can now program my flash=
.

Thanks everyone for your help!

Jonathan

RE: Accessing flash directly from User Space [SOLVED]

[Joakim Tjernlund]

>
> > > On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
> > > >> >>> How can I get that pointer?  Unfortunately I cannot simply
> > > use
> > > >> the
> > > >> >>>
> > > >> >> address of the flash.  Is there some magical function call
> > > that
> > > >> >> gives me access to that portion of the memory space?
> > > >> >>
> > > >> >> $ man 2 mmap
> > > >> >>
> > > >> >> You want MAP_SHARED and O_SYNC.
> > > >> >>
> > > >> >
> > > >> >
> > > >> > To use that I need to have a file descriptor to a device, do
> > I
> > > >> not?  However, I do not have a base flash driver to give me
> > that
> > > >> file descriptor.  Am I missing something with that call?
> > > >> >
> > > >>
> > > >> /dev/mem
> > > >>
> > > >Okay, I now have access to the flash memory, however when I write
> > > to it the writes do not take.  I have tried calling msync() on the
> > > mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> > > get things to work right.
> > > >
> > > >Here are the calls:
> > > >
> > > >   int fd = open("/dev/mem", O_SYNC | O_RDWR);
> > > >   uint16_t * flash = (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> > > >         (PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> > > >         NOR_FLASH_BASE_ADRS);
> > >
> > > What board and CPU are you using?  Is your flash really at
> > > 0xFC800000, or is
> > > that the virtual address that VxWorks puts it at?
> >
> > I am using a custom board based on the AMCC Kilauea development
> > board.  It uses a 405EX CPU.  Yes, the flash is really at
> > 0xFC000000.
>
> I have found the problem.  It occurred to me in the shower (okay not really,
> but most good ideas happen there).
>
> What was happening is that I was in fact able to write to the correct
> registers.  However, I would try and write to them in a batch.  But the way
> mmap works (at least according to the man page) with MAP_SHARED is that the
> file may not be updated until msync() is called.  Now, I thought that O_SYNC
> would take care of that when I open /dev/mem, but that was not the case.
>
> Anyway, to make a long story short, I inserted an msync() after each
> assignment to the flash.  This resolved my problem and I can now program my flash.

Ouch, this was news to me too. Calling msync() after every write kills performance.
We use mmap(/dev/mem) to access HW and havn't seen any issues yet. Is this
perhaps a new behaviour for mmap(/dev/mem) and is there a way
to avoid calling msync()?

     Jocke

RE: Accessing flash directly from User Space [SOLVED]

[Joakim Tjernlund]

>
> >
> > > > On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
> > > > >> >>> How can I get that pointer?  Unfortunately I cannot simply
> > > > use
> > > > >> the
> > > > >> >>>
> > > > >> >> address of the flash.  Is there some magical function call
> > > > that
> > > > >> >> gives me access to that portion of the memory space?
> > > > >> >>
> > > > >> >> $ man 2 mmap
> > > > >> >>
> > > > >> >> You want MAP_SHARED and O_SYNC.
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >> > To use that I need to have a file descriptor to a device, do
> > > I
> > > > >> not?  However, I do not have a base flash driver to give me
> > > that
> > > > >> file descriptor.  Am I missing something with that call?
> > > > >> >
> > > > >>
> > > > >> /dev/mem
> > > > >>
> > > > >Okay, I now have access to the flash memory, however when I write
> > > > to it the writes do not take.  I have tried calling msync() on the
> > > > mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> > > > get things to work right.
> > > > >
> > > > >Here are the calls:
> > > > >
> > > > >   int fd = open("/dev/mem", O_SYNC | O_RDWR);
> > > > >   uint16_t * flash = (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> > > > >         (PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> > > > >         NOR_FLASH_BASE_ADRS);
> > > >
> > > > What board and CPU are you using?  Is your flash really at
> > > > 0xFC800000, or is
> > > > that the virtual address that VxWorks puts it at?
> > >
> > > I am using a custom board based on the AMCC Kilauea development
> > > board.  It uses a 405EX CPU.  Yes, the flash is really at
> > > 0xFC000000.
> >
> > I have found the problem.  It occurred to me in the shower (okay not really,
> > but most good ideas happen there).
> >
> > What was happening is that I was in fact able to write to the correct
> > registers.  However, I would try and write to them in a batch.  But the way
> > mmap works (at least according to the man page) with MAP_SHARED is that the
> > file may not be updated until msync() is called.  Now, I thought that O_SYNC
> > would take care of that when I open /dev/mem, but that was not the case.
> >
> > Anyway, to make a long story short, I inserted an msync() after each
> > assignment to the flash.  This resolved my problem and I can now program my flash.
>
> Ouch, this was news to me too. Calling msync() after every write kills performance.
> We use mmap(/dev/mem) to access HW and havn't seen any issues yet. Is this
> perhaps a new behaviour for mmap(/dev/mem) and is there a way
> to avoid calling msync()?

Does O_DIRECT help? (you may need to define _GNU_SOURCE before #include)

 Jocke

RE: Accessing flash directly from User Space [SOLVED]

[Kenneth Johansson]

On Thu, 2009-10-29 at 10:00 +0100, Joakim Tjernlund wrote:
> >
> > > > On Tue, Oct 27, 2009 at 04:24:53PM -0600, Jonathan Haws wrote:
> > > > >> >>> How can I get that pointer?  Unfortunately I cannot simply
> > > > use
> > > > >> the
> > > > >> >>>
> > > > >> >> address of the flash.  Is there some magical function call
> > > > that
> > > > >> >> gives me access to that portion of the memory space?
> > > > >> >>
> > > > >> >> $ man 2 mmap
> > > > >> >>
> > > > >> >> You want MAP_SHARED and O_SYNC.
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >> > To use that I need to have a file descriptor to a device, do
> > > I
> > > > >> not?  However, I do not have a base flash driver to give me
> > > that
> > > > >> file descriptor.  Am I missing something with that call?
> > > > >> >
> > > > >>
> > > > >> /dev/mem
> > > > >>
> > > > >Okay, I now have access to the flash memory, however when I write
> > > > to it the writes do not take.  I have tried calling msync() on the
> > > > mapping to no avail.  I have opened the fd with O_SYNC, but cannot
> > > > get things to work right.
> > > > >
> > > > >Here are the calls:
> > > > >
> > > > >   int fd = open("/dev/mem", O_SYNC | O_RDWR);
> > > > >   uint16_t * flash = (uint16_t *)mmap(NULL, NOR_FLASH_SIZE,
> > > > >         (PROT_READ | PROT_WRITE), MAP_PRIVATE, fd,
> > > > >         NOR_FLASH_BASE_ADRS);
> > > >
> > > > What board and CPU are you using?  Is your flash really at
> > > > 0xFC800000, or is
> > > > that the virtual address that VxWorks puts it at?
> > >
> > > I am using a custom board based on the AMCC Kilauea development
> > > board.  It uses a 405EX CPU.  Yes, the flash is really at
> > > 0xFC000000.
> >
> > I have found the problem.  It occurred to me in the shower (okay not really,
> > but most good ideas happen there).
> >
> > What was happening is that I was in fact able to write to the correct
> > registers.  However, I would try and write to them in a batch.  But the way
> > mmap works (at least according to the man page) with MAP_SHARED is that the
> > file may not be updated until msync() is called.  Now, I thought that O_SYNC
> > would take care of that when I open /dev/mem, but that was not the case.
> >
> > Anyway, to make a long story short, I inserted an msync() after each
> > assignment to the flash.  This resolved my problem and I can now program my flash.
> 
> Ouch, this was news to me too. Calling msync() after every write kills performance.
> We use mmap(/dev/mem) to access HW and havn't seen any issues yet. Is this
> perhaps a new behaviour for mmap(/dev/mem) and is there a way
> to avoid calling msync()?

The address range should be outside the dram and thus uncached. Any
write to any address in the range mmaped should go directly to the NOR
flash. Any other behavior is a bug. It's not mapping an actual file
here.

Re: Accessing flash directly from User Space

[Scott Wood]

On Tue, Oct 27, 2009 at 04:52:40PM -0600, Jonathan Haws wrote:
> > Will the device respond to 0x1234 being written at offset zero?  You
> > generally have to poke these things pretty specifically in order to
> > get
> > them to go into command mode.
> > 
> 
> It should because that is the first data location in flash. 

I don't follow.  Even if you have an Intel command set flash (and thus don't
need unlock writes), 0x1234 isn't a valid command that I know of.  The flash
doesn't behave as a register that you can read back; it just responds in a
certain way based on what you write to it.

> Also, just to be sure I am telling the truth, I tried writing to one of
> the registers to setup an erase and got the same results - the value did
> not get written.

Following the exact sequence that the driver uses?  What did you write, what
did you expect (you're generally not going to get the same thing back that
you wrote), and what did you get?  What kind of command set, bus width, and
interleaving do you have?

If you manually do the same exact accesses from a firmware prompt, external
debugger, etc. does it work?

> > > The driver works perfectly in VxWorks,

On this exact hardware?

> > Including the 0x1234 thing?
> 
> Actually, I have not tried that - I have not had to since the driver worked.

What happens without the 0x1234?

-Scott

RE: Accessing flash directly from User Space

[Jonathan Haws]

> On Tue, Oct 27, 2009 at 04:52:40PM -0600, Jonathan Haws wrote:
> > > Will the device respond to 0x1234 being written at offset zero?
> You
> > > generally have to poke these things pretty specifically in order
> to
> > > get
> > > them to go into command mode.
> > >
> >
> > It should because that is the first data location in flash.
>=20
> I don't follow.  Even if you have an Intel command set flash (and
> thus don't
> need unlock writes), 0x1234 isn't a valid command that I know of.
> The flash
> doesn't behave as a register that you can read back; it just
> responds in a
> certain way based on what you write to it.
>=20
> > Also, just to be sure I am telling the truth, I tried writing to
> one of
> > the registers to setup an erase and got the same results - the
> value did
> > not get written.
>=20
> Following the exact sequence that the driver uses?  What did you
> write, what
> did you expect (you're generally not going to get the same thing
> back that
> you wrote), and what did you get?  What kind of command set, bus
> width, and
> interleaving do you have?

I used the erase pattern, then write pattern for my flash device.  When I t=
ried to read back the value that should have been stored, it was what it wa=
s previously.

>=20
> If you manually do the same exact accesses from a firmware prompt,
> external
> debugger, etc. does it work?
>=20
> > > > The driver works perfectly in VxWorks,
>=20
> On this exact hardware?

Yes.

> > > Including the 0x1234 thing?
> >
> > Actually, I have not tried that - I have not had to since the
> driver worked.
>=20
> What happens without the 0x1234?

Have not bothered to try it.  My guess, after finding out what the problem =
is that it would not read back 0x1234.  In the test I performed, I intended=
 to erase the sector, prep it for write, then write out 0x1234 to the first=
 two bytes in flash.  However, I failed in include the code to erase and pr=
ep the sector for writing in my rush to find out what the heck was going on=
.

As I mentioned previously, I was just not allowing the correct sequence of =
operations to take place to erase the sector (that is where my problem bega=
n) because when I setup the sector for erasure, the sequencing did not take=
 place correctly because what I would assign to flash was not committed imm=
ediately.

I hope that makes sense.

Thanks,

Jonathan

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> Does O_DIRECT help? (you may need to define _GNU_SOURCE before
> #include)


Nope, O_DIRECT did not help - in fact it caused the application to crash.  =
Why that is I am not sure, but it crashed.

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> > > Anyway, to make a long story short, I inserted an msync() after
> each
> > > assignment to the flash.  This resolved my problem and I can now
> program my flash.
> >
> > Ouch, this was news to me too. Calling msync() after every write
> kills performance.
> > We use mmap(/dev/mem) to access HW and havn't seen any issues yet.
> Is this
> > perhaps a new behaviour for mmap(/dev/mem) and is there a way
> > to avoid calling msync()?
>=20
> The address range should be outside the dram and thus uncached. Any
> write to any address in the range mmaped should go directly to the
> NOR
> flash. Any other behavior is a bug. It's not mapping an actual file
> here.

That is what I was thinking.  But I have a working driver and the extra del=
ay of writing to flash caused by the msync() calls I can deal with.  I only=
 ever write data to flash when I need to update one of my boot images.

Thanks,

Jonathan

Re: Accessing flash directly from User Space

[Joerg Albert]

Jonathan,

On 10/28/2009 03:45 PM, Jonathan Haws wrote:
> Looking through our notes and talking with the engineer 
> who was performing the tests, it was exactly that - MTD was waiting
> for a signal that was produced differently than the hardware 
> ready signal.  By simply polling the flash until the hardware
> ready signal toggled we were able to get a much faster read and write speed.
> Granted, most of our signals are being sent through a CPLD,
> so that may be why MTD did not work as well.

even if your problem is solved I'd like to understand this performance issue.
I had a look into the datasheet of the S29GL Mirrorbit flash by Spansion as an example. They provide a dedicated pin RY/BY#, which signals the end of an embedded algorithm (erase or programming). While figure 11.9 shows no timing advance of RY/BY# against Dout on the data line, figure 11.12 has one of unspecified length between RY/BY# and the end of data toggling.

If you had a 10-fold slowdown with MTD, either the CPLD really slows down the read access to the flash or maybe your custom driver uses some acceleration (write buffer programming,
unlock bypass, accelerated program with 12V on the WP#/ACC pin) while MTD does not.

Which kernel version and flash device did you use in this comparsion?


Regards,
Jörg.

RE: Accessing flash directly from User Space

[Jonathan Haws]

> Looking through our notes and talking with the engineer=20
> who was performing the tests, it was exactly that - MTD was waiting
> for a signal that was produced differently than the hardware=20
> ready signal.  By simply polling the flash until the hardware
> ready signal toggled we were able to get a much faster read and write spe=
ed.
> Granted, most of our signals are being sent through a CPLD,
> so that may be why MTD did not work as well.

even if your problem is solved I'd like to understand this performance issu=
e.
I had a look into the datasheet of the S29GL Mirrorbit flash by Spansion as=
 an example. They provide a dedicated pin RY/BY#, which signals the end of =
an embedded algorithm (erase or programming). While figure 11.9 shows no ti=
ming advance of RY/BY# against Dout on the data line, figure 11.12 has one =
of unspecified length between RY/BY# and the end of data toggling.

If you had a 10-fold slowdown with MTD, either the CPLD really slows down t=
he read access to the flash or maybe your custom driver uses some accelerat=
ion (write buffer programming,
unlock bypass, accelerated program with 12V on the WP#/ACC pin) while MTD d=
oes not.

Which kernel version and flash device did you use in this comparsion?

We were using VxWorks when we did the comparison, so there may be a problem=
 in their driver.  We are using unlock bypass by the way.  Our flash chip i=
s from Spansion.  I do not have the datasheet right with me, so I do not ha=
ve the part number.

Re: Accessing flash directly from User Space [SOLVED]

[Scott Wood]

On Thu, Oct 29, 2009 at 10:00:28AM +0100, Joakim Tjernlund wrote:
> > I have found the problem.  It occurred to me in the shower (okay not really,
> > but most good ideas happen there).
> >
> > What was happening is that I was in fact able to write to the correct
> > registers.  However, I would try and write to them in a batch.  But the way
> > mmap works (at least according to the man page) with MAP_SHARED is that the
> > file may not be updated until msync() is called.  Now, I thought that O_SYNC
> > would take care of that when I open /dev/mem, but that was not the case.
> >
> > Anyway, to make a long story short, I inserted an msync() after each
> > assignment to the flash.  This resolved my problem and I can now program my flash.
> 
> Ouch, this was news to me too. Calling msync() after every write kills performance.
> We use mmap(/dev/mem) to access HW and havn't seen any issues yet. Is this
> perhaps a new behaviour for mmap(/dev/mem) and is there a way
> to avoid calling msync()?

I suspect that the msync() was merely serving as a very heavyweight
memory barrier.

-Scott

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> On Thu, Oct 29, 2009 at 10:00:28AM +0100, Joakim Tjernlund wrote:
> > > I have found the problem.  It occurred to me in the shower (okay
> not really,
> > > but most good ideas happen there).
> > >
> > > What was happening is that I was in fact able to write to the
> correct
> > > registers.  However, I would try and write to them in a batch.
> But the way
> > > mmap works (at least according to the man page) with MAP_SHARED
> is that the
> > > file may not be updated until msync() is called.  Now, I thought
> that O_SYNC
> > > would take care of that when I open /dev/mem, but that was not
> the case.
> > >
> > > Anyway, to make a long story short, I inserted an msync() after
> each
> > > assignment to the flash.  This resolved my problem and I can now
> program my flash.
> >
> > Ouch, this was news to me too. Calling msync() after every write
> kills performance.
> > We use mmap(/dev/mem) to access HW and havn't seen any issues yet.
> Is this
> > perhaps a new behaviour for mmap(/dev/mem) and is there a way
> > to avoid calling msync()?
>=20
> I suspect that the msync() was merely serving as a very heavyweight
> memory barrier.

I did try hacking the mb() calls from the kernel source to use them in user=
 space, but they had no effect.  I still had to include the calls to msync(=
).

Re: Accessing flash directly from User Space [SOLVED]

[Michael Buesch]

On Friday 30 October 2009 15:50:22 Jonathan Haws wrote:
> > I suspect that the msync() was merely serving as a very heavyweight
> > memory barrier.
> 
> I did try hacking the mb() calls from the kernel source to use them in user space, but they had no effect.  I still had to include the calls to msync().

What did the resulting mb() that you used look like?

-- 
Greetings, Michael.

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> On Friday 30 October 2009 15:50:22 Jonathan Haws wrote:
> > > I suspect that the msync() was merely serving as a very
> heavyweight
> > > memory barrier.
> >
> > I did try hacking the mb() calls from the kernel source to use
> them in user space, but they had no effect.  I still had to include
> the calls to msync().
>=20
> What did the resulting mb() that you used look like?

asm("eieio; sync");

Re: Accessing flash directly from User Space [SOLVED]

[Alessandro Rubini]

> asm("eieio; sync");

Hmm...
	: : : "memory"

And, doesn't ";" start a comment in assembly? (no, not on powerpc it seems)

Just to make sure, if you replace msync() with another system call,
like "kill(1, 0);" you can verify wether it's really useful or if it's
only acting as a barrier.

/alessandro

Re: Accessing flash directly from User Space [SOLVED]

[Michael Buesch]

On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > asm("eieio; sync");
> 
> Hmm...
> 	: : : "memory"
> 
> And, doesn't ";" start a comment in assembly? (no, not on powerpc it seems)

Yes, I think the barrier is wrong.
Please try with

#define mb()	__asm__ __volatile__("eieio\n sync\n" : : : "memory")

-- 
Greetings, Michael.

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > asm("eieio; sync");
> >
> > Hmm...
> > 	: : : "memory"
> >
> > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> it seems)
>=20
> Yes, I think the barrier is wrong.
> Please try with
>=20
> #define mb()	__asm__ __volatile__("eieio\n sync\n" : : :
> "memory")

That definition worked great.  I must have missed the : : : "memory" bit wh=
en I was digging through code.

Thanks, that gives me about a 2x speedup over the msync() calls.

Thanks for the help!

Re: Accessing flash directly from User Space [SOLVED]

[Micha Nelissen]

Michael Buesch wrote:
> Yes, I think the barrier is wrong.
> Please try with
> 
> #define mb()	__asm__ __volatile__("eieio\n sync\n" : : : "memory")


For uncached memory, eieio should be enough.

Micha

Re: Accessing flash directly from User Space [SOLVED]

[Scott Wood]

On Fri, Oct 30, 2009 at 04:08:55PM +0100, Alessandro Rubini wrote:
> > asm("eieio; sync");
> 
> Hmm...
> 	: : : "memory"
> 
> And, doesn't ";" start a comment in assembly? (no, not on powerpc it seems)

';' is an instruction separator on all GNU as targets that I'm familiar
with (though a quick manual search shows z80 as an exception).

-Scott

RE: Accessing flash directly from User Space [SOLVED]

[Jonathan Haws]

> Michael Buesch wrote:
> > Yes, I think the barrier is wrong.
> > Please try with
> >
> > #define mb()	__asm__ __volatile__("eieio\n sync\n" : : :
> "memory")
>=20
>=20
> For uncached memory, eieio should be enough.

I tried eieio alone and it did not work.  It needed the above command to ta=
ke care of it all.

Re: Accessing flash directly from User Space [SOLVED]

[Gabriel Paubert]

On Fri, Oct 30, 2009 at 10:46:46AM -0600, Jonathan Haws wrote:
> > Michael Buesch wrote:
> > > Yes, I think the barrier is wrong.
> > > Please try with
> > >
> > > #define mb()	__asm__ __volatile__("eieio\n sync\n" : : :
> > "memory")
> > 
> > 
> > For uncached memory, eieio should be enough.
> 
> I tried eieio alone and it did not work.  It needed the above command to take care of it all.

Surprising. But sync is a superset of eieio, no?

In this case a sync alone should be sufficient (although an eieio
followed by a sync should not be much slower than a sync alone).

	Gabriel

RE: Accessing flash directly from User Space [SOLVED]

[Joakim Tjernlund]

>
> > On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > > asm("eieio; sync");
> > >
> > > Hmm...
> > >    : : : "memory"
> > >
> > > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> > it seems)
> >
> > Yes, I think the barrier is wrong.
> > Please try with
> >
> > #define mb()   __asm__ __volatile__("eieio\n sync\n" : : :
> > "memory")
>
> That definition worked great.  I must have missed the : : : "memory" bit when
> I was digging through code.
>
> Thanks, that gives me about a 2x speedup over the msync() calls.

Exactly when should you use the barrier? At every access,
every read or when changing from write to read?

 Jocke

Re: Accessing flash directly from User Space [SOLVED]

[Michael Buesch]

On Saturday 31 October 2009 14:26:48 Joakim Tjernlund wrote:
> >
> > > On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > > > asm("eieio; sync");
> > > >
> > > > Hmm...
> > > >    : : : "memory"
> > > >
> > > > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> > > it seems)
> > >
> > > Yes, I think the barrier is wrong.
> > > Please try with
> > >
> > > #define mb()   __asm__ __volatile__("eieio\n sync\n" : : :
> > > "memory")
> >
> > That definition worked great.  I must have missed the : : : "memory" bit when
> > I was digging through code.
> >
> > Thanks, that gives me about a 2x speedup over the msync() calls.
> 
> Exactly when should you use the barrier? At every access,
> every read or when changing from write to read?

Well, it depends on the device you are accessing. I'll give you a small pseudo example.

mmio[0] = address;
mmio[1] = data;
mb();
mmio[3] |= 0x01; /* This triggers an operation -> address=data */
/* probably also need an mb() here, if the following code
 * depends on the operation to be triggered. */

-- 
Greetings, Michael.

Re: Accessing flash directly from User Space [SOLVED]

[Joakim Tjernlund]

Michael Buesch <mb@bu3sch.de> wrote on 31/10/2009 17:42:54:
>
> On Saturday 31 October 2009 14:26:48 Joakim Tjernlund wrote:
> > >
> > > > On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > > > > asm("eieio; sync");
> > > > >
> > > > > Hmm...
> > > > >    : : : "memory"
> > > > >
> > > > > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> > > > it seems)
> > > >
> > > > Yes, I think the barrier is wrong.
> > > > Please try with
> > > >
> > > > #define mb()   __asm__ __volatile__("eieio\n sync\n" : : :
> > > > "memory")
> > >
> > > That definition worked great.  I must have missed the : : : "memory" bit when
> > > I was digging through code.
> > >
> > > Thanks, that gives me about a 2x speedup over the msync() calls.
> >
> > Exactly when should you use the barrier? At every access,
> > every read or when changing from write to read?
>
> Well, it depends on the device you are accessing. I'll give you a small pseudo example.
>
> mmio[0] = address;
> mmio[1] = data;
> mb();
> mmio[3] |= 0x01; /* This triggers an operation -> address=data */
> /* probably also need an mb() here, if the following code
>  * depends on the operation to be triggered. */

So anything that depends on the previous accesses needs a mb()
hmm, the mmio[0] and mmio[1] are written in order I hope?

 Jocke

Re: Accessing flash directly from User Space [SOLVED]

[Michael Buesch]

On Saturday 31 October 2009 21:14:07 Joakim Tjernlund wrote:
> Michael Buesch <mb@bu3sch.de> wrote on 31/10/2009 17:42:54:
> >
> > On Saturday 31 October 2009 14:26:48 Joakim Tjernlund wrote:
> > > >
> > > > > On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > > > > > asm("eieio; sync");
> > > > > >
> > > > > > Hmm...
> > > > > >    : : : "memory"
> > > > > >
> > > > > > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> > > > > it seems)
> > > > >
> > > > > Yes, I think the barrier is wrong.
> > > > > Please try with
> > > > >
> > > > > #define mb()   __asm__ __volatile__("eieio\n sync\n" : : :
> > > > > "memory")
> > > >
> > > > That definition worked great.  I must have missed the : : : "memory" bit when
> > > > I was digging through code.
> > > >
> > > > Thanks, that gives me about a 2x speedup over the msync() calls.
> > >
> > > Exactly when should you use the barrier? At every access,
> > > every read or when changing from write to read?
> >
> > Well, it depends on the device you are accessing. I'll give you a small pseudo example.
> >
> > mmio[0] = address;
> > mmio[1] = data;
> > mb();
> > mmio[3] |= 0x01; /* This triggers an operation -> address=data */
> > /* probably also need an mb() here, if the following code
> >  * depends on the operation to be triggered. */
> 
> So anything that depends on the previous accesses needs a mb()

No, not really. I would always put an mb() where the comment is.
Imagine you have two instances of the above (probably in a loop):

mmio[0] = address;
mmio[1] = data;
mb();
mmio[3] |= 0x01; /* This triggers an operation -> address=data */
mb();

mmio[0] = address;
mmio[1] = data;
mb();
mmio[3] |= 0x01; /* This triggers an operation -> address=data */
mb();

> hmm, the mmio[0] and mmio[1] are written in order I hope?

We do not care in this example, as the write to [3] does trigger
the device operation. We do only care that [0] and [1] are set
when [3] is written. We do not care in what order [0] and [1] are written.

This is just an artificial example, but I think you get my point.
My point just being is that an mb() does only belong where it matters.
And what matters is defined by your device specifications.

-- 
Greetings, Michael.

Re: Accessing flash directly from User Space [SOLVED]

[Joakim Tjernlund]

Michael Buesch <mb@bu3sch.de> wrote on 31/10/2009 21:35:31:
>
> On Saturday 31 October 2009 21:14:07 Joakim Tjernlund wrote:
> > Michael Buesch <mb@bu3sch.de> wrote on 31/10/2009 17:42:54:
> > >
> > > On Saturday 31 October 2009 14:26:48 Joakim Tjernlund wrote:
> > > > >
> > > > > > On Friday 30 October 2009 16:08:55 Alessandro Rubini wrote:
> > > > > > > > asm("eieio; sync");
> > > > > > >
> > > > > > > Hmm...
> > > > > > >    : : : "memory"
> > > > > > >
> > > > > > > And, doesn't ";" start a comment in assembly? (no, not on powerpc
> > > > > > it seems)
> > > > > >
> > > > > > Yes, I think the barrier is wrong.
> > > > > > Please try with
> > > > > >
> > > > > > #define mb()   __asm__ __volatile__("eieio\n sync\n" : : :
> > > > > > "memory")
> > > > >
> > > > > That definition worked great.  I must have missed the : : : "memory" bit when
> > > > > I was digging through code.
> > > > >
> > > > > Thanks, that gives me about a 2x speedup over the msync() calls.
> > > >
> > > > Exactly when should you use the barrier? At every access,
> > > > every read or when changing from write to read?
> > >
> > > Well, it depends on the device you are accessing. I'll give you a small
> pseudo example.
> > >
> > > mmio[0] = address;
> > > mmio[1] = data;
> > > mb();
> > > mmio[3] |= 0x01; /* This triggers an operation -> address=data */
> > > /* probably also need an mb() here, if the following code
> > >  * depends on the operation to be triggered. */
> >
> > So anything that depends on the previous accesses needs a mb()
>
> No, not really. I would always put an mb() where the comment is.
> Imagine you have two instances of the above (probably in a loop):
>
> mmio[0] = address;
> mmio[1] = data;
> mb();
> mmio[3] |= 0x01; /* This triggers an operation -> address=data */
> mb();
>
> mmio[0] = address;
> mmio[1] = data;
> mb();
> mmio[3] |= 0x01; /* This triggers an operation -> address=data */
> mb();
>
> > hmm, the mmio[0] and mmio[1] are written in order I hope?
>
> We do not care in this example, as the write to [3] does trigger
> the device operation. We do only care that [0] and [1] are set
> when [3] is written. We do not care in what order [0] and [1] are written.

In this example yes, I was wondering in general.

>
> This is just an artificial example, but I think you get my point.
> My point just being is that an mb() does only belong where it matters.
> And what matters is defined by your device specifications.

So what does guarded memory mapping on ppc mean really? If I need
that much mb(), guarded does not seem to do much.

 Jocke

Re: Accessing flash directly from User Space [SOLVED]

[Segher Boessenkool]

>>>> mmio[0] = address;
>>>> mmio[1] = data;
>>>> mb();

eieio is enough here.

>>>> mmio[3] |= 0x01; /* This triggers an operation -> address=data */
>>>> /* probably also need an mb() here, if the following code
>>>>  * depends on the operation to be triggered. */

No, a sync does not guarantee the device has seen the store yet;
you need something specific to the device to guarantee this.
Usually a load (from the same register!) followed by code that
makes sure the load has finished is sufficient (and necessary).

>>> hmm, the mmio[0] and mmio[1] are written in order I hope?
>>
>> We do not care in this example, as the write to [3] does trigger
>> the device operation. We do only care that [0] and [1] are set
>> when [3] is written. We do not care in what order [0] and [1] are  
>> written.
>
> In this example yes, I was wondering in general.

The writes will not be reordered, but they can be combined,
unless you put an eieio (or sync) inbetween.

> So what does guarded memory mapping on ppc mean really? If I need
> that much mb(), guarded does not seem to do much.

Loosely speaking, guarded means no prefetch is done.


Segher