How to Sync properly ?

How to Sync properly ?

[Patrick Allaire]

Hi,

I am having some problems with the syncing of data to flash. Last week I
have updated to 2.4.17 kernel and with cvs MTD drivers ... because I was
having the same kinds of problems with my previous version. 

If I want to move a file from a ram drive to the flash, I will issue a 'mv'
command followed by a 'sync' command. Sometime it will be updated but
sometime it wont !!! 'sync' should not return until the flash is updated ?
no ? 

To see what was happening I added some printk in the driver, in the delete
and write ! I saw some stranges behaviour. After a 'mv', sometimes the sync
command does not do any things, sometime I see somes delteting and writing
of buffers but not all of the blocks ... the remaining block will be written
to the flash in a delay from 1 to 5 mintutes later ... and sometime the sync
work fine !!!

It is really important for me, because my system has a hard power ... so if
I tell the user the system is ready to shut down ... there wont be a halt
command issued. For me the only way to do things, is to send a sync command,
when this command return, we can shut down.

I am using ST Flash, it is working on Intel flash driver(CFI) on char and
block device, with some modifications (ST cannot write partial block)

thank you

Patrick Allaire
mailto:pallaire@gameloft.com
If you can see it, but it's not there, it's virtual. 
If you can't see it, but it is there, it's hidden. 
It you can't see it and it isn't there, it's gone.

Re: How to Sync properly ?

[David Woodhouse]

pallaire@gameloft.com said:
>  If I want to move a file from a ram drive to the flash, I will issue
> a 'mv' command followed by a 'sync' command. Sometime it will be
> updated but sometime it wont !!! 'sync' should not return until the
> flash is updated ? no ? 

> To see what was happening I added some printk in the driver, in the
> delete and write ! I saw some stranges behaviour. After a 'mv',
> sometimes the sync command does not do any things, sometime I see
> somes delteting and writing of buffers but not all of the blocks ...
> the remaining block will be written to the flash in a delay from 1 to
> 5 mintutes later ... and sometime the sync work fine !!! 

What file system? If you're using a 'normal' filesystem on the mtdblock 
device, you shouldn't expect it to be reliable. 

Linux stupidly just assumes that block devices never do any write caching, 
and a sync request is therefore never passed to the block device driver. 

--
dwmw2

RE: How to Sync properly ?

[Patrick Allaire]

Ok, I am using Minix FS ... to minimized space used by the meta data (short
file name ...)

What would be the FS suggested to minimize space used ?

thank


Patrick Allaire
mailto:pallaire@gameloft.com
If you can see it, but it's not there, it's virtual. 
If you can't see it, but it is there, it's hidden. 
It you can't see it and it isn't there, it's gone.



> -----Original Message-----
> From: David Woodhouse [mailto:dwmw2@infradead.org]
> Sent: January 7, 2002 8:59 AM
> To: Patrick Allaire
> Cc: linux-mtd@lists.infradead.org
> Subject: Re: How to Sync properly ? 
> 
> 
> 
> pallaire@gameloft.com said:
> >  If I want to move a file from a ram drive to the flash, I 
> will issue
> > a 'mv' command followed by a 'sync' command. Sometime it will be
> > updated but sometime it wont !!! 'sync' should not return until the
> > flash is updated ? no ? 
> 
> > To see what was happening I added some printk in the driver, in the
> > delete and write ! I saw some stranges behaviour. After a 'mv',
> > sometimes the sync command does not do any things, sometime I see
> > somes delteting and writing of buffers but not all of the blocks ...
> > the remaining block will be written to the flash in a delay 
> from 1 to
> > 5 mintutes later ... and sometime the sync work fine !!! 
> 
> What file system? If you're using a 'normal' filesystem on 
> the mtdblock 
> device, you shouldn't expect it to be reliable. 
> 
> Linux stupidly just assumes that block devices never do any 
> write caching, 
> and a sync request is therefore never passed to the block 
> device driver. 
> 
> --
> dwmw2
> 
> 

Re: How to Sync properly ?

[David Woodhouse]

pallaire@gameloft.com said:
>  Ok, I am using Minix FS ... to minimized space used by the meta data
> (short file name ...)

> What would be the FS suggested to minimize space used ? 

I'd be inclined to modify JFFS2 so that it aligns writes to 8 bytes and 
doesn't mark nodes obsolete. It shouldn't be more more than a one-liner for 
each change, I believe. 

You'll need to make sure your flash chip driver implements writev(). 

--
dwmw2

RE: How to Sync properly ?

[Patrick Allaire]

Do you know any of the standard FS that will support sync ??

because for the moment the dirver for my ST falsh will only write 128k at a
time ... I would have to change my hardware to use Intel Flash, to be able
to use JFFS 1 or 2.



Patrick Allaire
mailto:pallaire@gameloft.com
If you can see it, but it's not there, it's virtual. 
If you can't see it, but it is there, it's hidden. 
It you can't see it and it isn't there, it's gone.



> -----Original Message-----
> From: David Woodhouse [mailto:dwmw2@infradead.org]
> Sent: January 7, 2002 9:35 AM
> To: Patrick Allaire
> Cc: linux-mtd@lists.infradead.org
> Subject: Re: How to Sync properly ? 
> 
> 
> 
> pallaire@gameloft.com said:
> >  Ok, I am using Minix FS ... to minimized space used by the 
> meta data
> > (short file name ...)
> 
> > What would be the FS suggested to minimize space used ? 
> 
> I'd be inclined to modify JFFS2 so that it aligns writes to 8 
> bytes and 
> doesn't mark nodes obsolete. It shouldn't be more more than a 
> one-liner for 
> each change, I believe. 
> 
> You'll need to make sure your flash chip driver implements writev(). 
> 
> --
> dwmw2
> 
> 

Re: How to Sync properly ?

[David Woodhouse]

pallaire@gameloft.com said:
> Do you know any of the standard FS that will support sync ??

Linux has no way to signal to the block device driver that it should flush 
the write cache - so _no_ Linux filesystem can get it right, even on 
disc drives with write caches. It's just more obvious with the large write 
cache on the mtdblock device, that's all. 

Linux is broken. Unless Linux gets unbroken and starts to deal with block 
devices having internal write caches, the only option you have is to 
disable the caching completely.

> because for the moment the dirver for my ST falsh will only write 128k
> at a time ... I would have to change my hardware to use Intel Flash,
> to be able to use JFFS 1 or 2.

I thought you could write the ST flash in 8-byte chunks?



--
dwmw2

RE: How to Sync properly ?

[Patrick Allaire]

Yes we can, but my driver cant for the moment, Since there is no driver for
ST flash in your cvs, We have to port the one from INTEL to ST ... the guy
who did the first draft cant work on it for the moment. 

I have been told :

>>> You can write less than 128k at a time, but there is an architectural
feature of the ST multilevel flash which causes a little issue, there is no
word write command, only a buffer write command - the write buffer is 32
bytes long, and this is broken into 4 8-byte pages. You have to write a
complete 8-byte page at a time. You cannot write (for example 2 bytes of a
page and then write the other 6-bytes later) - this hangs up the state
machine. The reason for this is that they have included an ECC engine in the
device, and each 8-byte page has ECC calculated automatically as you write -
this is obviously stored in flash somewhere, so when you try to write to the
same page again, when it tries to write the ECC codes, it gets into a
screw-up - the issue is, if you want to re-write a page, you need to erase
the entire erase block (128k). <<<

Is this true ? is there a work around to use properly ST flash ? because
INTEL flash cost us the double !

thank you

Patrick Allaire
mailto:pallaire@gameloft.com
If you can see it, but it's not there, it's virtual. 
If you can't see it, but it is there, it's hidden. 
It you can't see it and it isn't there, it's gone.



> -----Original Message-----
> From: David Woodhouse [mailto:dwmw2@infradead.org]
> Sent: January 8, 2002 4:08 AM
> To: Patrick Allaire
> Cc: linux-mtd@lists.infradead.org
> Subject: Re: How to Sync properly ? 
> 
> 
> 
> pallaire@gameloft.com said:
> > Do you know any of the standard FS that will support sync ??
> 
> Linux has no way to signal to the block device driver that it 
> should flush 
> the write cache - so _no_ Linux filesystem can get it right, even on 
> disc drives with write caches. It's just more obvious with 
> the large write 
> cache on the mtdblock device, that's all. 
> 
> Linux is broken. Unless Linux gets unbroken and starts to 
> deal with block 
> devices having internal write caches, the only option you have is to 
> disable the caching completely.
> 
> > because for the moment the dirver for my ST falsh will only 
> write 128k
> > at a time ... I would have to change my hardware to use Intel Flash,
> > to be able to use JFFS 1 or 2.
> 
> I thought you could write the ST flash in 8-byte chunks?
> 
> 
> 
> --
> dwmw2
> 
> 

Re: How to Sync properly ?

[David Woodhouse]

pallaire@gameloft.com said:
> I have been told :

> >>> [...] You have to write a complete 8-byte page at a time. You cannot
> write (for example 2 bytes of a page and then write the other 6-bytes
> later) - this hangs up the state machine. [...] <<<

> Is this true ? is there a work around to use properly ST flash ?
> because INTEL flash cost us the double ! 

I don't know - I haven't seen any information about these chips. It 
wouldn't surprise me. It's a shame it's 8 bytes not 4 - 4 would be fine for 
JFFS2. A page size of 8 bytes means you have to change the padding to 8 
bytes, or do the write-behind buffering that we're planning for NAND 
support anyway.

--
dwmw2