JFFS2 Garbage collection issue

JFFS2 Garbage collection issue

[Albrecht, Harald]

Hi all,

we are currently facing a problem with the Garbage-collection on a JFFS2 
file system.

We have an embedded linux system based on an ARM9 AT91RM9200 processor 
with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
The Flash holds the U-Boot loader and its environment and in the main 
part (30MB) of the chip a JFFS2-based root-file-system.
Our Linux kernel version is 2.6.24.3

Some minutes after startup of the system, the garbage collector (gc.c) 
does a scan of the file system, gathering information for the garbage 
collection procedures that may be initiated based upon that data.
On some file systems, especially such with long runtime and therefore 
many file operations, this scan consumes a lot of time. On some systems 
it stays in that scan for more that 10 minutes. The problem with this 
behaviour is the fact, that during the scan the file system is locked, 
and therefore all file operations are suspended. This leads to a crash 
of the system, when the application program hangs for a long time, 
waiting for a file operation to return from a system call, and the 
watchdog is not triggered.

In a test system without watchdog, after the scan, the system runs normally.
The flash chip itself shows no errors, so it seems to be a problem of 
the file system.

Any idea or suggestion to solve the problem would be very welcome!

Harald Albrecht

Re: JFFS2 Garbage collection issue

[Joakim Tjernlund]

"linux-mtd" <linux-mtd-bounces@lists.infradead.org> wrote on 2013/11/27 
11:11:04:
> 
> Hi all,
> 
> we are currently facing a problem with the Garbage-collection on a JFFS2 

> file system.
> 
> We have an embedded linux system based on an ARM9 AT91RM9200 processor 
> with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
> The Flash holds the U-Boot loader and its environment and in the main 
> part (30MB) of the chip a JFFS2-based root-file-system.
> Our Linux kernel version is 2.6.24.3
> 
> Some minutes after startup of the system, the garbage collector (gc.c) 
> does a scan of the file system, gathering information for the garbage 
> collection procedures that may be initiated based upon that data.
> On some file systems, especially such with long runtime and therefore 
> many file operations, this scan consumes a lot of time. On some systems 
> it stays in that scan for more that 10 minutes. The problem with this 
> behaviour is the fact, that during the scan the file system is locked, 
> and therefore all file operations are suspended. This leads to a crash 
> of the system, when the application program hangs for a long time, 
> waiting for a file operation to return from a system call, and the 
> watchdog is not triggered.
> 
> In a test system without watchdog, after the scan, the system runs 
normally.
> The flash chip itself shows no errors, so it seems to be a problem of 
> the file system.
> 
> Any idea or suggestion to solve the problem would be very welcome!

The simple fix is to send a signal(SIGCONT?) to the GC process regularly 
to do GC
over time. That way you don't get all GC at boot time.

 Jocke
 

Re: JFFS2 Garbage collection issue

[Joakim Tjernlund]

"linux-mtd" <linux-mtd-bounces@lists.infradead.org> wrote on 2013/11/27 
11:23:25:
> 
> "linux-mtd" <linux-mtd-bounces@lists.infradead.org> wrote on 2013/11/27 
> 11:11:04:
> > 
> > Hi all,
> > 
> > we are currently facing a problem with the Garbage-collection on a 
JFFS2 
> 
> > file system.
> > 
> > We have an embedded linux system based on an ARM9 AT91RM9200 processor 

> > with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
> > The Flash holds the U-Boot loader and its environment and in the main 
> > part (30MB) of the chip a JFFS2-based root-file-system.
> > Our Linux kernel version is 2.6.24.3
> > 
> > Some minutes after startup of the system, the garbage collector (gc.c) 

> > does a scan of the file system, gathering information for the garbage 
> > collection procedures that may be initiated based upon that data.
> > On some file systems, especially such with long runtime and therefore 
> > many file operations, this scan consumes a lot of time. On some 
systems 
> > it stays in that scan for more that 10 minutes. The problem with this 
> > behaviour is the fact, that during the scan the file system is locked, 

> > and therefore all file operations are suspended. This leads to a crash 

> > of the system, when the application program hangs for a long time, 
> > waiting for a file operation to return from a system call, and the 
> > watchdog is not triggered.
> > 
> > In a test system without watchdog, after the scan, the system runs 
> normally.
> > The flash chip itself shows no errors, so it seems to be a problem of 
> > the file system.
> > 
> > Any idea or suggestion to solve the problem would be very welcome!
> 
> The simple fix is to send a signal(SIGCONT?) to the GC process regularly 

> to do GC
> over time. That way you don't get all GC at boot time.
> 
>  Jocke

BTW, I guess you got logging to file such as syslog on your system?
JFFS2 really don't do well with small incremental writes. I recommend
you turn such logging off.

Re: JFFS2 Garbage collection issue

[Albrecht Dreß]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

Hi Jocke:

Am 27.11.13 11:30 schrieb(en) Joakim Tjernlund:
> BTW, I guess you got logging to file such as syslog on your system? JFFS2 really don't do well with small incremental writes. I recommend you turn such logging off.

Does the delay by the gc process happen only when the jffs2 fs is mounted, or also during the normal operation when many updates occur?

Background: I had a similar issue as Harald with a NVRAM which is used to hold quickly changing event buffers and logs.  My solution was to wait for the gc process during the startup of the system, before the "main" application is launched.  However, if this could also happen *after* the initial scan has finished, this would be bad news for me, as I then probably would need the SIG(STOP|CONT) approach...

Thanks,
Albrecht.

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: JFFS2 Garbage collection issue

[Joakim Tjernlund]

Albrecht Dreß <albrecht.dress@arcor.de> wrote on 2013/11/27 20:34:06:
> 
> Hi Jocke:
> 
> Am 27.11.13 11:30 schrieb(en) Joakim Tjernlund:
> > BTW, I guess you got logging to file such as syslog on your system? 
JFFS2 really don't do well with small incremental writes. I recommend you 
turn such logging off.
> 
> Does the delay by the gc process happen only when the jffs2 fs is 
mounted, or also during the normal operation when many updates occur?

Only at mount AFAIK. During initial scanning lots of GC can be triggered 
but during normal
operation not so much. The GC algo. is a bit naive and you have to help it 
if you got
lots of small writes. By sending the correct signal to the GC thread you 
trigger one GC.


> 
> Background: I had a similar issue as Harald with a NVRAM which is used 
to hold quickly changing event buffers and logs.  My solution was to wait 
for the gc process during the startup of the system, before the "main" 
application is launched.  However, if this could also happen *after* the 
initial scan has finished, this would be bad news for me, as I then 
probably would need the SIG(STOP|CONT) approach...
> 
> Thanks,
> Albrecht.[attachment "attq2f2p.dat" deleted by Joakim 
Tjernlund/Transmode] 

Re: JFFS2 Garbage collection issue

[Albrecht, Harald]

Hi All,

thank you for the fast reply.
What we figured out is that the jffs2_garbage_collect_pass loops for 
several minutes in the first for(;;) loop. It always went into the if 
statement after jffs2_get_ino_cache. In this situation it's not 
interruptable by any signals.

Regards
   Harald


"linux-mtd" <linux-mtd-bounces@lists.infradead.org> wrote on 2013/11/27 
11:11:04:
>> Hi all,
>>
>> we are currently facing a problem with the Garbage-collection on a JFFS2
>> file system.
>>
>> We have an embedded linux system based on an ARM9 AT91RM9200 processor
>> with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
>> The Flash holds the U-Boot loader and its environment and in the main
>> part (30MB) of the chip a JFFS2-based root-file-system.
>> Our Linux kernel version is 2.6.24.3
>>
>> Some minutes after startup of the system, the garbage collector (gc.c)
>> does a scan of the file system, gathering information for the garbage
>> collection procedures that may be initiated based upon that data.
>> On some file systems, especially such with long runtime and therefore
>> many file operations, this scan consumes a lot of time. On some systems
>> it stays in that scan for more that 10 minutes. The problem with this
>> behaviour is the fact, that during the scan the file system is locked,
>> and therefore all file operations are suspended. This leads to a crash
>> of the system, when the application program hangs for a long time,
>> waiting for a file operation to return from a system call, and the
>> watchdog is not triggered.
>>
>> In a test system without watchdog, after the scan, the system runs
> normally.
>> The flash chip itself shows no errors, so it seems to be a problem of
>> the file system.
>>
>> Any idea or suggestion to solve the problem would be very welcome!
> The simple fix is to send a signal(SIGCONT?) to the GC process regularly
> to do GC
> over time. That way you don't get all GC at boot time.
>
>   Jocke
>   
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: JFFS2 Garbage collection issue

[Joakim Tjernlund]

> 
> Hi All,
> 
> thank you for the fast reply.
> What we figured out is that the jffs2_garbage_collect_pass loops for 
> several minutes in the first for(;;) loop. It always went into the if 
> statement after jffs2_get_ino_cache. In this situation it's not 
> interruptable by any signals.

Yes, by then it is too late. You need to force GC during normal operation
so that when you reboot there isn't much GC left to do.
Try sending the right signal to the GC thread, say every minute(just to 
test something)

> 
> Regards
>    Harald
> 
> 
> "linux-mtd" <linux-mtd-bounces@lists.infradead.org> wrote on 2013/11/27 
> 11:11:04:
> >> Hi all,
> >>
> >> we are currently facing a problem with the Garbage-collection on a 
JFFS2
> >> file system.
> >>
> >> We have an embedded linux system based on an ARM9 AT91RM9200 
processor
> >> with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
> >> The Flash holds the U-Boot loader and its environment and in the main
> >> part (30MB) of the chip a JFFS2-based root-file-system.
> >> Our Linux kernel version is 2.6.24.3
> >>
> >> Some minutes after startup of the system, the garbage collector 
(gc.c)
> >> does a scan of the file system, gathering information for the garbage
> >> collection procedures that may be initiated based upon that data.
> >> On some file systems, especially such with long runtime and therefore
> >> many file operations, this scan consumes a lot of time. On some 
systems
> >> it stays in that scan for more that 10 minutes. The problem with this
> >> behaviour is the fact, that during the scan the file system is 
locked,
> >> and therefore all file operations are suspended. This leads to a 
crash
> >> of the system, when the application program hangs for a long time,
> >> waiting for a file operation to return from a system call, and the
> >> watchdog is not triggered.
> >>
> >> In a test system without watchdog, after the scan, the system runs
> > normally.
> >> The flash chip itself shows no errors, so it seems to be a problem of
> >> the file system.
> >>
> >> Any idea or suggestion to solve the problem would be very welcome!
> > The simple fix is to send a signal(SIGCONT?) to the GC process 
regularly
> > to do GC
> > over time. That way you don't get all GC at boot time.
> >
> >   Jocke
> > 
> >
> >
> > ______________________________________________________
> > Linux MTD discussion mailing list
> > http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: JFFS2 Garbage collection issue

[Albrecht Dreß]

[-- Attachment #1: Type: text/plain, Size: 2874 bytes --]

Hi Harald:

I had *exactly* the same issue with a board based on a MPC5200 (PPC).  I use kernel 3.2, and the problematic file system is a jffs2 on a NVRAM which is used to hold quickly changing logs and event buffers (before they are shifted to flash or a CF card).  Apparently the scan process is triggered immediately when the NVRAM is mounted by the startup script.  Apart from that I had the same picture - the scan eats so much time that the hw wdt is triggered... (see my original post <http://lists.infradead.org/pipermail/linux-mtd/2013-April/046632.html> - I unfortunately never got an answer to it).

My solution was to add a task to the startup script which monitors the scan process (jffs2_gcd_mtd6 in my case) and waits until it apparently finished.  The main application (which in turn activates the hardware wdt) is launched only *afterwards*; then everything seems to run just fine, even if there is much "traffic" on the nvram.

The monitor application basically identifies the pid of the jffs2_gcd_mtd6, and then repeatedly reads /proc/<pid>/stat as long as the sum of the utime, stime, cutime and cstime is still growing (of course with a maximum loop count, so the system doesn't deadlock).

Hope this helps,
Albrecht.


Am 27.11.13 11:11 schrieb(en) Albrecht, Harald:
> Hi all,
> 
> we are currently facing a problem with the Garbage-collection on a JFFS2 file system.
> 
> We have an embedded linux system based on an ARM9 AT91RM9200 processor with an Intel / Micron JS28F256 NOR-flash of 32MB in size.
> The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system.
> Our Linux kernel version is 2.6.24.3
> 
> Some minutes after startup of the system, the garbage collector (gc.c) does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data.
> On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some systems it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked, and therefore all file operations are suspended. This leads to a crash of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.
> 
> In a test system without watchdog, after the scan, the system runs normally.
> The flash chip itself shows no errors, so it seems to be a problem of the file system.
> 
> Any idea or suggestion to solve the problem would be very welcome!
> 
> Harald Albrecht
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

JFFS2 Garbage Collection Issue

[suresh]

Hi,

I am running a stress test which keeps writing to the flash on every  
reboot (around 15 MB each time). After running the test for 2 days, I  
see that the garbage collection takes a long time to complete (almost  
65 seconds).

I disabled the garbage collection from running during bootup (and the  
stress test is stopped). When I see the free space on the JFFS2  
partition its much more than what it should be (we expect it to be  
around 17 MB, but its 35 MB). When the JFFS2 garbage collection is  
started again it comes down to 17MB. Again after a reboot its back at  
35 MB (garbage collection is stopped at reboot and stress test is now  
not running).

Why does the JFFS2 NOT remember the garbage collection it did the  
previous time? This is affecting the application start up time in out  
case.

I am using JFFS2 in 2.6.16-rc3 on OMAP5912

Thanks in advance,

Best Regards,
Suresh