JFFS2 Stability problem!

JFFS2 Stability problem!

[Frederic Giasson]

Hi all,

	I found a stability problem that looks like a flash space leak while
running intense test case with JFFS2.  Someone else with a diffenrent
platform with different flash driver at a different company told me he
encountered the same problem, so I think am not dreaming.  Here is what I do
and a description of the problem:

1- I erase my flash.
2- I mount JFFS2.  The empty file system reports a 22% disk usage when I
call "df" system command.
3- I start a script that does the following loop:
	a. Copy many files of various size and types to JFFS2 file system
using "cp" command.
	b. Call "df" system command after the transfer of the files.  After
the first transfer is completed, it reports a 88% 	   disk usage.
	c. Call "rm" command to erase all files on JFFS2 file system.

	The set of files copyied is always the same.  Logging the output of
my terminal, I noticed that after about 200 loops, the disk usage at step
3-b begins to grow slowly.  After about 500 loops, the disk usage as grown
so much that there is no longer enough room on the file system to store all
the same files that at first took 88% disk usage.  When my test case script
cannot copy all files, it stops.

	After the test case failure, I manually erased all files (using "rm"
again).  Then, calling "df" I got a 30% disk usage.  I ran the test case
many times on different machines (of the same types, though), and I obtained
the same failure after about the same number of test case loops.

Description of my platform:

Embedded MPC860T
2MB Atmel AT49BV16x4 single flash memory chip
16MB RAM
Linux Kernel 2.4.4

	Since my flash chip was not supported by MTD, I wrote my own driver
but I doubt greatly that the problem could come from it.  I tested it not
using JFFS2 and I did not get any error.  As I said, another developper ran
my test case and got the same problem and he have a different system with
differents drivers.

	Could it be a garbage collector problem?  Anyone who have a clue
about it is welcome to tell me even the slightyiest details, since resolving
this problem is very important to me.



Frédéric Giasson

Re: JFFS2 Stability problem!

[David Woodhouse]

fgiasson@mediatrix.com said:
> 	I found a stability problem that looks like a flash space leak while
> running intense test case with JFFS2. 

Definitely looks like a leak. Does the space come back after an unmount and 
remount?

--
dwmw2

RE: JFFS2 Stability problem!

[Frederic Giasson]

Here are Vipin's debug terminal log:

<snip>

Message from syslogd@elan at Sun Jul 15 12:27:26 2001 ...
elan kernel: 4c6d10 is obsolete... skipping

Message from syslogd@elan at Sun Jul 15 12:27:56 2001 ...
elan kernel: 00000, bad_size 0x00000000 (total 0x00800000 of 0x00800000)

Message from syslogd@elan at Sun Jul 15 12:28:17 2001 ...
elan kernel: 29298, erasing_size 0x00000000, bad_size 0x00000000 (total 
0x00800000 of 0x00800000)

Message from syslogd@elan at Sun Jul 15 12:29:59 2001 ...
elan kernel: 31ba0c is obsolete... skipping

Message from syslogd@elan at Sun Jul 15 12:30:30 2001 ...
elan kernel:  nextblock->free_size 0x0003b150

<snip>

Message from syslogd@elan at Sun Jul 15 15:08:44 2001 ...
elan kernel: 006c0000, nextblock->free_size 0x000366b8

Message from syslogd@elan at Sun Jul 15 15:08:44 2001 ...
elan kernel: 000000, bad_size 0x00000000 (total 0x00800000 of 0x00800000)

Message from syslogd@elan at Sun Jul 15 15:10:07 2001 ...
elan kernel: 00, bad_size 0x00000000 (total 0x00800000 of 0x00800000)

Message from syslogd@elan at Sun Jul 15 15:10:12 2001 ...
elan kernel: 00000, bad_size 0x00000000 (total 0x00800000 of 0x00800000)


P.S. I did not get those errors on my terminal, though.

Frédéric Giasson





-----Original Message-----
From: Vipin Malik [mailto:vipin.malik@daniel.com]
Sent: Monday, July 16, 2001 10:37 AM
To: David Woodhouse
Cc: Frederic Giasson; linux-mtd@lists.infradead.org; jffs-dev@axis.com;
Jean-Denis Boyer
Subject: Re: JFFS2 Stability problem!


David Woodhouse wrote:

> fgiasson@mediatrix.com said:
> >       I found a stability problem that looks like a flash space leak
while
> > running intense test case with JFFS2.
>
> Definitely looks like a leak. Does the space come back after an unmount
and
> remount?

[root@elan /root]# rm -rf /mnt/jffs2/*

[root@elan /root]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock1            8192      2544      5648  31% /mnt/jffs2

[root@elan /root]# umount /mnt/jffs2/
[root@elan /root]#
[root@elan /root]# mount -t jffs2 /dev/mtdblock1 /mnt/jffs2/
[root@elan /root]#
[root@elan /root]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock1            8192      1284      6908  16% /mnt/jffs2

[root@elan /root]#

Hence Yes!

Vipin
P.S. I also got some strange (debug type) messages from JFFS2 on the
terminal I
was running the program at, but NOT on the serial console (why?)
I sent those to Frederic. I don't have access to those here, but maybe
Frederic
can forward them to you. It may help decipher what is going on.

Re: JFFS2 Stability problem!

[Vipin Malik]

David Woodhouse wrote:

> fgiasson@mediatrix.com said:
> >       I found a stability problem that looks like a flash space leak while
> > running intense test case with JFFS2.
>
> Definitely looks like a leak. Does the space come back after an unmount and
> remount?

[root@elan /root]# rm -rf /mnt/jffs2/*

[root@elan /root]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock1            8192      2544      5648  31% /mnt/jffs2

[root@elan /root]# umount /mnt/jffs2/
[root@elan /root]#
[root@elan /root]# mount -t jffs2 /dev/mtdblock1 /mnt/jffs2/
[root@elan /root]#
[root@elan /root]# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock1            8192      1284      6908  16% /mnt/jffs2

[root@elan /root]#

Hence Yes!

Vipin
P.S. I also got some strange (debug type) messages from JFFS2 on the terminal I
was running the program at, but NOT on the serial console (why?)
I sent those to Frederic. I don't have access to those here, but maybe Frederic
can forward them to you. It may help decipher what is going on.

Re: JFFS2 Stability problem!

[David Woodhouse]

fgiasson@mediatrix.com said:
> Here are Vipin's debug terminal log:
> <snip> 

That's just klogd/syslogd failing to keep up with the spew of normal 
debugging messages. 

--
dwmw2

Re: JFFS2 Stability problem!

[Vipin Malik]

David Woodhouse wrote:

> fgiasson@mediatrix.com said:
> > Here are Vipin's debug terminal log:
> > <snip>
>
> That's just klogd/syslogd failing to keep up with the spew of normal
> debugging messages.
>

Well, actually that's why I called them "strange" as these did not show
up on my serial console (I was logging my serial console messages to a
file on another computer)!.

Plus, they were not a "spew"- rather one in a few minutes type
frequency. Do they tell you anything?

Vipin

Re: JFFS2 Stability problem!

[David Woodhouse]

vipin.malik@daniel.com said:
>  Well, actually that's why I called them "strange" as these did not
> show up on my serial console (I was logging my serial console messages
> to a file on another computer)!.

> Plus, they were not a "spew"- rather one in a few minutes type
> frequency. Do they tell you anything? 

KERN_DEBUG messages don't go to the console by default. But there were 
enough of them to confuse syslogd.


--
dwmw2

RE: JFFS2 Stability problem!

[Frederic Giasson]

Do those KERN_DEBUG messages tell you anything about this?  Do you have an
idea on how fixing this problem?

Frédéric Giasson




-----Original Message-----
From: David Woodhouse [mailto:dwmw2@infradead.org]
Sent: Tuesday, July 17, 2001 3:11 AM
To: Vipin Malik
Cc: Frederic Giasson; linux-mtd@lists.infradead.org; jffs-dev@axis.com;
Jean-Denis Boyer
Subject: Re: JFFS2 Stability problem! 



vipin.malik@daniel.com said:
>  Well, actually that's why I called them "strange" as these did not
> show up on my serial console (I was logging my serial console messages
> to a file on another computer)!.

> Plus, they were not a "spew"- rather one in a few minutes type
> frequency. Do they tell you anything? 

KERN_DEBUG messages don't go to the console by default. But there were 
enough of them to confuse syslogd.


--
dwmw2

RE: JFFS2 Stability problem!

[Doron Sandroy]

 Hello All.
 
 Please can you send my exactly instructions how to work & install Jffs on
 the asabet board.
 
 Best regards
 
 Doron Sandroy

jffs & amd_flash.c

[Kugan]

Hi All
I am new to this mailing list.
I have uClinux 2.0.38 kernel running on Atmel EB63 Evaluation board, there I
have ATMEL AT491604 90 TC flash.  I have partitioned the flash with pnc2000.
after mounting the jffs when i try to create a file on the partitione it is
going into this function and adding the task to waiq and calling and
schedule and seems it is struck there,
Where does this waitq consumed and what could be the mistake????
Thanx in Advance

Kugan

tatic int write_one_word(struct map_info *map, struct flchip *chip,
     unsigned long adr, __u32 datum)
{
 unsigned long timeo = jiffies + HZ;
 struct amd_flash_private *private = map->fldrv_priv;
 DECLARE_WAITQUEUE(wait, current);
 int ret = 0;
 int times_left;

retry:
 spin_lock_bh(chip->mutex);

 if (chip->state != FL_READY){
  printk("%s: waiting for chip to write, state = %d\n",
         map->name, chip->state);
  set_current_state(TASK_UNINTERRUPTIBLE);
  add_wait_queue(&chip->wq, &wait);

  spin_unlock_bh(chip->mutex);

  schedule();
  remove_wait_queue(&chip->wq, &wait);
  printk(KERN_INFO "%s: woke up to write\n", map->name);
  if(signal_pending(current))
   return -EINTR;

  timeo = jiffies + HZ;

  goto retry;
 }

 chip->state = FL_WRITING;

 adr += chip->start;
 ENABLE_VPP(map);
 send_cmd(map, chip->start, CMD_PROGRAM_UNLOCK_DATA);
 wide_write(map, datum, adr);

 times_left = 500000;
 while (times_left-- && flash_is_busy(map, adr, private->interleave))

  if (current->need_resched) {
   spin_unlock_bh(chip->mutex);
   schedule();
   spin_lock_bh(chip->mutex);
  }
 }

 if (!times_left) {
  printk(KERN_WARNING "%s: write to 0x%lx timed out!\n",
         map->name, adr);
  ret = -EIO;
 } else {
  __u32 verify;
  if ((verify = wide_read(map, adr)) != datum) {
   printk(KERN_WARNING "%s: write to 0x%lx failed. "
          "datum = %x, verify = %x\n",
          map->name, adr, datum, verify);
   ret = -EIO;
  }
 }

 DISABLE_VPP(map);
 chip->state = FL_READY;
 wake_up(&chip->wq);
 spin_unlock_bh(chip->mutex);

 return ret;
}

Re: jffs & amd_flash.c

[Jarkko Lavinen]

> have ATMEL AT491604 90 TC flash.  I have partitioned the flash with pnc2000.
> after mounting the jffs when i try to create a file on the partitione it is
> going into this function and adding the task to waiq and calling and

I am using happily the amd_flash.c driver with AMD 64mbit non-CFI
flash chip.  I haven't experienced this problem.  I am using 
arm-linux kernel, which is based on Alan Cox's 2.4.8-ac8.

Your problem seems not to be failing erase or programming, or is it?
Surely the debugging message would nag about timeout errors?

There were problems in sector erase and word writing code in amd_flash.c. 
The problem was that the operation status query was not made form the 
same bank and sector as being erased or written into. This was an issue 
only with chips supporting simultaneous read & write on separate banks.
I've committed a fixes for these problems into CVS.

The Amd driver works nicely for me and I although I had initially
hardware configuration related speed problems they have been now
resolved. Using the command

  dd if=/dev/zero of=/dev/mtd0 bs=1k

to test the writing speed, I get an average speed of 24 us / word.
The chip datasheet claims 12 us typical. My initial writing speed
was 210 us/word until I found out the proper hardware configuration.

Jarkko Lavinen

Re: jffs & amd_flash.c

[Kugan]

>
> I am using happily the amd_flash.c driver with AMD 64mbit non-CFI
> flash chip.  I haven't experienced this problem.  I am using
> arm-linux kernel, which is based on Alan Cox's 2.4.8-ac8.

I am using 2.0.38 uClinux kernel on ARM7TDMI

>
> Your problem seems not to be failing erase or programming, or is it?
> Surely the debugging message would nag about timeout errors?

I have a dual plane flash , I am running the kernel from the flash(XIP) and
I bott from romfs and mount a jffs partitione in the other flash plane
respect to kernel. when I mount I can see some data being written and read
back to verify, some time it if failing,

> There were problems in sector erase and word writing code in amd_flash.c.
> The problem was that the operation status query was not made form the
> same bank and sector as being erased or written into. This was an issue
> only with chips supporting simultaneous read & write on separate banks.
> I've committed a fixes for these problems into CVS.

The flash chip I have also support simultaneous read and write on different
plane simultaneously. Is your patch in latest CVS or do i have to get and
apply???

>
> The Amd driver works nicely for me and I although I had initially
> hardware configuration related speed problems they have been now
> resolved. Using the command
>
>   dd if=/dev/zero of=/dev/mtd0 bs=1k
I will try this

Thanx and regards
Kugan

Re: jffs & amd_flash.c

[Jarkko Lavinen]

> The flash chip I have also support simultaneous read and write on different
> plane simultaneously. Is your patch in latest CVS or do i have to get and
> apply???

It is in the MTD CVS, but the changes are so small it may not be worth the 
trouble to take fresh CVS image. flash_is_busy() is called only in two places.
The change applies to its second argument. Previously there was "chip->start".
Now there should be "adr".

For example in write_one_word() there should read:

891:  while (times_left-- && flash_is_busy(map, adr, private->interleave)) {

This cured all my writing problems. You may already have this fixed in
erase_one_block(), since I committed that already many weeks ago and the
write_one_word() change only last weekend.

Regards,
Jarkko Lavinen