JFFS2 behaviour when large number of files are created.

JFFS2 behaviour when large number of files are created.

[David Ho]

Hi all,

I have a snapshot of the CVS MTD/JFFS2 code from Sept 22 patched into a
2.4.24 kernel.  In the process of running a varied set of applications on
the system, a program was creating a large number of small files each less
than 1KB.  The Linux card is powered up for a short period of time to dump
data to persistent storage, then it is powered down for until the next data
dump.  As the sytem is run, the mount is becoming slower to a point that
the watchdog is reset the card.

It appears that JFFS2 is bad for certain usage patterns.  Creating large
number of files is one.  What other usage patterns should I avoid?

Creating a large file?
Removing and updating a large file?

Also , the kernel is crashing at one instance in gc.c.  Is this a known
problem in 2.4?

Checked all inodes but still 0x406868 bytes of unchecked space?
kernel BUG at gc.c:139!
Oops: Kernel Mode Software FPU Emulation, sig: 8
NIP: C009DBF4 XER: 00000000 LR: C009DBF4 SP: C7DADEF0 REGS: c7dade40 TRAP:
1000
   Not tainted
MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c7dac000[8] 'jffs2_gcd_mtd2' Last syscall: -1
last math 00000000 last altivec 00000000
GPR00: C009DBF4 C7DADEF0 C7DAC000 00000018 00001032 00000001 00000000
C01C0000
GPR08: 000010F4 C0191590 00000000 C7DADE10 42008024 1001E76C 07FFC000
007FFF16
GPR16: 00000000 00000001 007FFF00 07FFABA0 00000000 FFFFFFFF 07FBADE8
00000002
GPR24: 00000000 C7FA5918 00000000 C7FA5928 C7DADFB8 C7DB32C0 C7FA58F4
C7FA5928
Call backtrace:
C009DBF4 C00A0E20 C0006D00

Regards,
David Ho

Re: JFFS2 behaviour when large number of files are created.

[Pantelis Antoniou]

David Ho wrote:

>
>
>Hi all,
>
>I have a snapshot of the CVS MTD/JFFS2 code from Sept 22 patched into a
>2.4.24 kernel.  In the process of running a varied set of applications on
>the system, a program was creating a large number of small files each less
>than 1KB.  The Linux card is powered up for a short period of time to dump
>data to persistent storage, then it is powered down for until the next data
>dump.  As the sytem is run, the mount is becoming slower to a point that
>the watchdog is reset the card.
>
>It appears that JFFS2 is bad for certain usage patterns.  Creating large
>number of files is one.  What other usage patterns should I avoid?
>
>Creating a large file?
>Removing and updating a large file?
>
>Also , the kernel is crashing at one instance in gc.c.  Is this a known
>problem in 2.4?
>
>Checked all inodes but still 0x406868 bytes of unchecked space?
>kernel BUG at gc.c:139!
>Oops: Kernel Mode Software FPU Emulation, sig: 8
>NIP: C009DBF4 XER: 00000000 LR: C009DBF4 SP: C7DADEF0 REGS: c7dade40 TRAP:
>1000
>   Not tainted
>MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
>TASK = c7dac000[8] 'jffs2_gcd_mtd2' Last syscall: -1
>last math 00000000 last altivec 00000000
>GPR00: C009DBF4 C7DADEF0 C7DAC000 00000018 00001032 00000001 00000000
>C01C0000
>GPR08: 000010F4 C0191590 00000000 C7DADE10 42008024 1001E76C 07FFC000
>007FFF16
>GPR16: 00000000 00000001 007FFF00 07FFABA0 00000000 FFFFFFFF 07FBADE8
>00000002
>GPR24: 00000000 C7FA5918 00000000 C7FA5928 C7DADFB8 C7DB32C0 C7FA58F4
>C7FA5928
>Call backtrace:
>C009DBF4 C00A0E20 C0006D00
>
>Regards,
>David Ho
>
>
>______________________________________________________
>Linux MTD discussion mailing list
>http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
>
>  
>
I can confirm the same behaviour on 2.4.28 + latest snapshot.

Now what?

Any ideas for a workaround?

Regards

Pantelis

Re: JFFS2 behaviour when large number of files are created.

[Jan Vestby]

On Monday 18 October 2004 18:28, David Ho wrote:
> Hi all,
>
> I have a snapshot of the CVS MTD/JFFS2 code from Sept 22 patched into a
> 2.4.24 kernel.  In the process of running a varied set of applications on
> the system, a program was creating a large number of small files each less
> than 1KB.  The Linux card is powered up for a short period of time to dump
> data to persistent storage, then it is powered down for until the next data
> dump.  As the sytem is run, the mount is becoming slower to a point that
> the watchdog is reset the card.
>
> It appears that JFFS2 is bad for certain usage patterns.  Creating large
> number of files is one.  What other usage patterns should I avoid?
>
I have seen startup time is quite nonlinear with respect to the number of 
files per directory. You could try to restrict the size of directories.

regards
jan vestby 

Re: JFFS2 behaviour when large number of files are created.

[David Woodhouse]

On Tue, 2004-10-19 at 20:16 +0200, Jan Vestby wrote:
> I have seen startup time is quite nonlinear with respect to the number of 
> files per directory. You could try to restrict the size of directories.

Hm. Should we be using rbtrees for dirents too? Or will the snapshots
fix this anyway?
-- 
dwmw2

Re: JFFS2 behaviour when large number of files are created.

[Artem B. Bityuckiy]

I also don't see any other reason why the mount is much slower if there 
are directories with many files in the file system. The only reason I 
see is (what David said) that direntries are stored (in RAM) in the 
lists, and it is relatively slow operation to find/insert/delete 
direntries from the list. Instead, nodes are stored in RB-tees. So, in 
case of slow CPU, there may be significant difference.

David, If you mean inode checkpoints saying "snapshots", I don't think 
they help to increase the mount time (But! They will increase the first 
directory access delay!). If you mean what Ferenc is implementing, this 
will increase the mount speed, but file system with "large" directories 
will be mounted slower anyway ( IMHO, of course :-) ).


David Woodhouse wrote:
> On Tue, 2004-10-19 at 20:16 +0200, Jan Vestby wrote:
> 
>>I have seen startup time is quite nonlinear with respect to the number of 
>>files per directory. You could try to restrict the size of directories.
> 
> 
> Hm. Should we be using rbtrees for dirents too? Or will the snapshots
> fix this anyway?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.