mmap returns incorrect data

mmap returns incorrect data

[marcus hall]

I have a system based on 2.5.59 kernel for arm.  This system was previously
running OK with all filesystems on a stratoflash chip, but I have recently
moved the filesystems to a compact flash chip.  Some filesystems are
cramfs, and some have changed from jffs2 to ext3 along with the move.

I am having trouble with ldconfig not recognizing some files as being valid.
The files that cause the problem are files that are on an ext3 filesystem
on the CF.  Other files on a cramfs filesystem seem to work reliably.
It seems that ldconfig mmap()s the file into its address space, and what
it sees doesn't match what is in the file.  If I pre-read the file before
running ldconfig, so that the contents get into the cache, then ldconfig
proceeds normally.

Working through various layers, I see that the generic_file_mmap() gets
called and sets things up OK, then when ldconfig references the file, the
page fault correctly invokes filemap_nopage(), which calls
page_cache_readaround().  By the time page_cache_readaround() returns,
the page has been added to mapping->page_tree, and the page is added to
ldconfig's address space.

I can see where page_cache_readaround() gets to ext3_readpages() and then
to ext3_get_block() to put the requests (page_cache_readahead() asks for
16 pages) onto the queue, then __do_page_cache_readahead() calls
blk_run_queues() to unplug the queues, but it isn't clear (yet!) where
we wait until the blocks are read in before returning from
page_cache_readaround().

The CF interface has a real delay between starting a request and getting
an interrupt when the card is ready, which the mtd driver I was using before
doesn't do, so I am guessing that this may be the source of my problems.
But it isn't clear why the cramfs files don't suffer from this problem (since
they are also on the CF).

Is there any known issue in this area?  Where should I be focusing my
attention?

Thanks for any suggestions!

Marcus Hall
marcus@tuells.org

Re: mmap returns incorrect data

[Russell King]

On Thu, Apr 29, 2004 at 05:12:43PM -0600, marcus hall wrote:
> I have a system based on 2.5.59 kernel for arm.  This system was previously
> running OK with all filesystems on a stratoflash chip, but I have recently
> moved the filesystems to a compact flash chip.  Some filesystems are
> cramfs, and some have changed from jffs2 to ext3 along with the move.

You _really_ don't want to use ext3 on CF unless you want to wear the
flash quickly.  Just because a filesystem has journaling does _not_
mean that its suitable for flash (in fact, it may be far worse for
flash than its non-journaled counterpart, as in this case.)

> I am having trouble with ldconfig not recognizing some files as being valid.
> The files that cause the problem are files that are on an ext3 filesystem
> on the CF.  Other files on a cramfs filesystem seem to work reliably.
> It seems that ldconfig mmap()s the file into its address space, and what
> it sees doesn't match what is in the file.  If I pre-read the file before
> running ldconfig, so that the contents get into the cache, then ldconfig
> proceeds normally.

This is an IDE driver bug - it performs PIO but does not ensure that the
individual pages are properly flushed to RAM before telling the kernel
that the IO is complete.  (If someone wants to disagree, look at how
rd.c does flush_dcache_page, and see previous discussions on lkml
concerning this.)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: mmap returns incorrect data

[Paulo Marques]

Russell King wrote:

> ....
> This is an IDE driver bug - it performs PIO but does not ensure that the
> individual pages are properly flushed to RAM before telling the kernel
> that the IO is complete.  (If someone wants to disagree, look at how
> rd.c does flush_dcache_page, and see previous discussions on lkml
> concerning this.)

 From what I could google from previous discussions, this bug affects ARM 
architectures but not x86. Am I correct in assuming this?

Please note that I'm not implying that since it doesn't affect x86, then it's 
ok. I've already tried linux on an iPaq 3970 and it runs great! I really feel 
that one of the Linux's strenghts is that it can run almost anywhere :)

It is just that I'm responsible for a project where we use a compact flash as 
storing device on an x86 system, and I would sleep better if this bug doesn't 
affect our system directly.

-- 
Paulo Marques - www.grupopie.com
"In a world without walls and fences who needs windows and gates?"

Re: mmap returns incorrect data

[marcus hall]

On Thu Apr 29 2004 - 18:48:22 EST, Russell King wrote:
>You _really_ don't want to use ext3 on CF unless you want to wear the
>flash quickly. Just because a filesystem has journaling does _not_
>mean that its suitable for flash (in fact, it may be far worse for
>flash than its non-journaled counterpart, as in this case.)

I was concerned about that in the beginning, and was told that CF devices
perform wear-leveling internally.  In fact, we have had units in the field
for a couple of years without failure, so my objections carried little
weight.  In practice, the filesystem isn't often written.  Since the jffs2
filesystem seems to be intertwined with the mtd drivers, there don't seem
to be any other good failure-tolerant solutions.  But, that isn't the
point for now..

>This is an IDE driver bug - it performs PIO but does not ensure that the
>individual pages are properly flushed to RAM before telling the kernel
>that the IO is complete. (If someone wants to disagree, look at how
>rd.c does flush_dcache_page, and see previous discussions on lkml
>concerning this.)

Yes, this was in fact where the problem was.  In the 2.5.59 kernel, in
mm/filemap.c there were still two calls to flush_page_to_ram(), but in
include/asm-arm/proc-armv/cache.h this was defined to be a nop.  I
replaced the flush_page_to_ram() calls with flush_dcache_page() calls
and everything started working properly.

In later versions, the flush_page_to_ram() calls have been removed from
filemap.c, and I do not see any replacement of the functionality (at least
nothing that seems clear to me!)

I would like to eventually move forward to a 2.6 kernel, so I am interested
in how this is addressed there.

I also didn't seem to find (or recognize) the earlier discussion, so perhaps
it is covered there..

Anyhow, thanks for the help.  Things are rolling again!

Marcus Hall
marcus@tuells.org