mmap question

mmap question

[Frederic Gobry]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

Hi,

I need to port an in-memory database so that it can use mtd
devices. The program currently uses mmap in order to access a file as
permanent storage. I discovered that using this capability is not
possible on jffs2, as mmap does not accept MAP_SHARED.

In my framework, write operations are performed by explicit calls, but
the data must be readable as if it were in a direct-access
memory. Would it be possible to do that on jffs2 (for instance,
open/write/close sessions to modify the data, but with immediate
update on the mmapped version of the file)

Alternatively, would it possible to use a raw MTD device to provide
the equivalent service ? I don't need filesystem semantic, as the
program already considers the data as a sequence of pages on a
flash-like device.

Thanks for any suggestion,

Frédéric

-- 
Frédéric Gobry    SMARTDATA    	  
      ---         http://www.smartdata.ch
Software Engineer Lausanne - Switzerland
                  +41 21 693 84 98

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: mmap question

[Frederic Gobry]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

I follow-up to my own question: I've managed to use mmap with jffs2
quite successfully, but now I would like to directly use MTD (so that
I can handle the way sectors are erased / written to in order to
ensure good atomicity).

I've seen that the MTD char driver does not provide mmap support. In
order to implement it, I would like to know if I can use the 'point'
function of the mtd_info structure to get the physical address of the
flash device. From the documentation

  "For devices which are entirely memory-mapped and which can be
   mapped directly into user-space page tables, we may support
   execute-in-place (XIP) mapping of data on the flash. The precise
   semantics of this are yet to be defined, so it's probably best just
   not to either implement or attempt to use these two functions right
   at the moment."

So, is it the correct way to go ?

Thanks,

Frédéric

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: mmap question

[Frederic Gobry]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hi,

I don't despair having an answer to at least one of my questions :-)

I implemented mmap on a mtd device (via a simple remap_page_range). I
have a userland program that opens and mmap a mtd device with success,
and can read-access it correctly. But once I perform a MEMERASE ioctl
or a simple write, the process only reads series of 0x80 bytes. If I
restart the process, it continues reading this value (even before any
erase/write operation) unless I reboot.

Do anybody have a clue about this behavior ?

My setup: ARM process (OMAP1510 board),  MTD device uses cfi_probe 

Thanks in advance,
Frédéric

-- 
Frédéric Gobry    SMARTDATA    	  
      ---         http://www.smartdata.ch
Software Engineer Lausanne - Switzerland
                  +41 21 693 84 98
				  

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: mmap question

[Jörn Engel]

Hi!

> I don't despair having an answer to at least one of my questions :-)

Hm. Maybe I should read/reread them. :-)

> I implemented mmap on a mtd device (via a simple remap_page_range). I
> have a userland program that opens and mmap a mtd device with success,
> and can read-access it correctly. But once I perform a MEMERASE ioctl
> or a simple write, the process only reads series of 0x80 bytes. If I
> restart the process, it continues reading this value (even before any
> erase/write operation) unless I reboot.
> 
> Do anybody have a clue about this behavior ?

0x80 is the status register, you are reading. The command set (Intel
or Amd, I suppose) can read it, handle any errors (0x80 mean no errors
on intel) and return to read mode by writing 0xff to the flash.

My guess is that you command set has a bug. Fix it or provide a spec
and some beer and pizza. ;-)

Jörn

-- 
Often, the most striking and innovative solutions come from realizing
that your concept of the problem was wrong.
-- Eric Raymond in "The cathedral and the bazaar"

Re: mmap question

[Frederic Gobry]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

> 0x80 is the status register, you are reading. The command set (Intel
> or Amd, I suppose) can read it, handle any errors (0x80 mean no errors
> on intel) and return to read mode by writing 0xff to the flash.

Mmmm, this enlightens the situation. I was fearing some cache problem...

> My guess is that you command set has a bug. Fix it or provide a spec
> and some beer and pizza. ;-)

The base code uses cfi_probe: it seemed to me the command set is
determined and provided automatically, isn't it ?

I think it's more because I interfere with the "canonical" way MTD
handles the flash: I access directly the memory after the erase or
write operation (via mmap, *not* via a read system call), but from the
code in do_read_onechip (cfi_cmdset_0001.c), it seems the flash is set
to read mode not when the erase or write operation is finished, but
before an actual read must be performed (which makes sense).

I can work around this for my program (I just tested, performing a
simple read just after the write restores what is seen in the mmapped
memory), but I still would like to know if the MTD API could be
augmented in order to handle read-only memory mapped areas (when
available) in a cleaner way as what I currently do?  I don't think
this would imply lots of changes:

      - a flag indicating if the flash can be mmapped

      - the implementation  of mmap (which probably would  need a call
        similar  to  'point'  but  with an  explicit  semantic  toward
        mmapping.

      - possibly another  flag to indicate  that an area  is currently
        mmapped, so that the erase / write operations set the flash in
        read mode once they have finished their duty

Frédéric

PS : where can I send the beer and pizza ? :-)

-- 
Frédéric Gobry    SMARTDATA    	  
      ---         http://www.smartdata.ch
Software Engineer Lausanne - Switzerland
                  +41 21 693 84 98
				  

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: mmap question

[Jörn Engel]

On Fri, 12 July 2002 09:57:16 +0200, Frederic Gobry wrote:
> The base code uses cfi_probe: it seemed to me the command set is
> determined and provided automatically, isn't it ?

Nyes, kind of. You configure the kernel to use one or more command
sets and the flash probing code picks the correct one out of those, if
present.

> I think it's more because I interfere with the "canonical" way MTD
> handles the flash: I access directly the memory after the erase or
> write operation (via mmap, *not* via a read system call), but from the
> code in do_read_onechip (cfi_cmdset_0001.c), it seems the flash is set
> to read mode not when the erase or write operation is finished, but
> before an actual read must be performed (which makes sense).

This helps performance a little, somewhere in the range of 5% for
large writes on the flashes I know. I have seen other code though,
that writes a sequence of 0xff 0x50 0xff into the flash before and
after every write operation, just to be sure the flash is in a sane
state. Slow, but effective.

> I can work around this for my program (I just tested, performing a
> simple read just after the write restores what is seen in the mmapped
> memory), but I still would like to know if the MTD API could be
> augmented in order to handle read-only memory mapped areas (when
> available) in a cleaner way as what I currently do?  I don't think
> this would imply lots of changes:
> 
>       - a flag indicating if the flash can be mmapped
> 
>       - the implementation  of mmap (which probably would  need a call
>         similar  to  'point'  but  with an  explicit  semantic  toward
>         mmapping.
> 
>       - possibly another  flag to indicate  that an area  is currently
>         mmapped, so that the erase / write operations set the flash in
>         read mode once they have finished their duty

BTW: Did you already try to mmap /dev/mtd* or /dev/mtdblock* ?

> PS : where can I send the beer and pizza ? :-)

>From Thursday on, I will be in Wedel, writing exams:
Joern Engel
Tinsdaler Weg 125
22880 Wedel
Germany

One possibility to order the pizza is Joeys:
http://www.joeys.de/joeys/65/index.htm

And if you actually do it, remember to cc David Woodhouse.

Jörn

-- 
"Translations are and will always be problematic. They inflict violence 
upon two languages." (translation from German)

Re: mmap question

[David Woodhouse]

frederic.gobry@smartdata.ch said:
>  I can work around this for my program (I just tested, performing a
> simple read just after the write restores what is seen in the mmapped
> memory), but I still would like to know if the MTD API could be
> augmented in order to handle read-only memory mapped areas (when
> available) in a cleaner way as what I currently do?  I don't think
> this would imply lots of changes:

>       - a flag indicating if the flash can be mmapped

>       - the implementation  of mmap (which probably would  need a call
>         similar  to  'point'  but  with an  explicit  semantic  toward
>         mmapping.

Er, ditch the crap 'point' semantics and implement something sane for 
mmapping, because that's about the only sane use of point() anyway.

>       - possibly another  flag to indicate  that an area  is currently
>         mmapped, so that the erase / write operations set the flash in
>         read mode once they have finished their duty 

That's not sufficient. If you let the erase or write operations happen 
while it's mapped, what prevents your app from reading from the flash 
_during_ the erase/write operation? You need to either prevent the erases/
writes from happening while the flash is mapped (bad) or arrange for the 
pages mapped to userspace to be marked invalid during the erase/write 
operation, and faults on those pages have to wait until the flash is back 
in read mode again.

We want some kind of mtd_remap() call which has its own struct vm_operations,
with a nopage() which sets a flag in the state of the underlying mtd_info, 
and ensure that the page can be found again and made invalid before any 
other MTD operations. With the rmap VM the latter is quite simple, without 
it we'd need to keep track of all vmas ourselves.

This can be used from userspace via the mtdchar device's mmap() method, and 
also from kernelspace so that things like jffs2 can just remap the flash 
and read from it directly instead of having to make repeated mtd->read() 
calls. 

--
dwmw2

Re: mmap question

[Frederic Gobry]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

Jörn said:
> BTW: Did you already try to mmap /dev/mtd* or /dev/mtdblock* ?

Sure, but neither does implement the mmap method.

> One possibility to order the pizza is Joeys:
> http://www.joeys.de/joeys/65/index.htm

Gargl, I'll need to find my french/german dictionnary first (I'm in
the french speaking part of switzerland !) Should I drop you an email
before, or is a pizza always welcome ?

David said:
> Er, ditch the crap 'point' semantics and implement something sane for 
> mmapping, because that's about the only sane use of point() anyway.

Ok. What was the idea of unpoint, BTW ?

> That's not sufficient. If you let the erase or write operations happen 
> while it's mapped, what prevents your app from reading from the flash 
> _during_ the erase/write operation? You need to either prevent the erases/
> writes from happening while the flash is mapped (bad) or arrange for the 
> pages mapped to userspace to be marked invalid during the erase/write 
> operation, and faults on those pages have to wait until the flash is back 
> in read mode again.

I admit I looked at it from the narrow point of view of my single
process which is safe by construction... I do agree that the only
acceptable solution is the second (disallow page access during write /
erase).

> We want some kind of mtd_remap() call which has its own struct vm_operations,
> with a nopage() which sets a flag in the state of the underlying mtd_info, 
> and ensure that the page can be found again and made invalid before any 
> other MTD operations. With the rmap VM the latter is quite simple, without 
> it we'd need to keep track of all vmas ourselves.

I'm quite a newcomer to VM management (my only experience comes from
"Linux Device Drivers" and what I tested for this driver...). What is
the rmap VM ?

Frédéric

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]