Problem: 51MB partition max size on jffs2

Problem: 51MB partition max size on jffs2

[Ken Offer]

Hello all,

I've run into a strange problem and I wonder if any of you have seen 
this and perhaps understand it better.

I've got a Toshiba 128MB Smart Card in my embedded system and I wrote a 
custom nand driver to support it.  I have discovered that I cannot 
mount a jffs2 partition larger than 51MB on the card (so at this point 
I have two 51MB partitions and 1 26MB partition).  Trying to mount a 
larger partition yields a less than helpful error message from mount.  
I've tried adding printk statements to the jffs2 code, but none of them 
get hit during the errant mount process.

So... is 51MB a valid jffs2 limitation?  If so, why?

+-----------------------------------------------------------+
|  Ken Offer        koffer@arlut.utexas.edu         ARL:UT  |
|  Office: 512-835-3859                  Fax: 512-835-3259  |
+-----------------------------------------------------------+

Re: Problem: 51MB partition max size on jffs2

[Thomas Gleixner]

On Wednesday 04 December 2002 17:54, Ken Offer wrote:
> Hello all,
>
> I've run into a strange problem and I wonder if any of you have seen
> this and perhaps understand it better.
>
> I've got a Toshiba 128MB Smart Card in my embedded system and I wrote a
> custom nand driver to support it.  I have discovered that I cannot
> mount a jffs2 partition larger than 51MB on the card (so at this point
> I have two 51MB partitions and 1 26MB partition).  Trying to mount a
> larger partition yields a less than helpful error message from mount.
> I've tried adding printk statements to the jffs2 code, but none of them
> get hit during the errant mount process.
>
> So... is 51MB a valid jffs2 limitation?  If so, why?
Latest CVS code ?
-- 
Thomas
____________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: Problem: 51MB partition max size on jffs2

[David Woodhouse]

koffer@arlut.utexas.edu said:
>   Trying to mount a  larger partition yields a less than helpful error
> message from mount.   I've tried adding printk statements to the jffs2
> code, but none of them  get hit during the errant mount process.

Set CONFIG_JFFS2_FS_DEBUG=1, show the output.

--
dwmw2

RE: Problem: 51MB partition max size on jffs2

[Dave Ellis]

On Wednesday 04 December 2002 17:54, Ken Offer wrote:
> I've got a Toshiba 128MB Smart Card in my embedded system and I wrote a 
> custom nand driver to support it.  I have discovered that I cannot 
> mount a jffs2 partition larger than 51MB on the card (so at this point 
> I have two 51MB partitions and 1 26MB partition).  Trying to mount a 
> larger partition yields a less than helpful error message from mount.  

I had the same problem a few weeks ago, and this patch seems
to solve it:

--- dist/fs/jffs2/build.c	Mon Sep  9 12:29:08 2002
+++ curr/fs/jffs2/build.c	Mon Nov 18 14:18:29 2002
@@ -211,7 +211,8 @@
 
 	c->free_size = c->flash_size;
 	c->nr_blocks = c->flash_size / c->sector_size;
-	c->blocks = kmalloc(sizeof(struct jffs2_eraseblock) * c->nr_blocks,
GFP_KERNEL);
+	/* kmalloc fails for large NAND since can't get more than 128K */
+	c->blocks = vmalloc(sizeof(struct jffs2_eraseblock) * c->nr_blocks);
 	if (!c->blocks)
 		return -ENOMEM;
 	for (i=0; i<c->nr_blocks; i++) {

kmalloc() is limited to 128K and the 128MB Flash needs more than twice that,

so I tried vmalloc() and it seems to be OK (at least with PPC 2.4.18 
patched for the new MTD code). Does anyone know if this change is safe?

Dave

Re: Problem: 51MB partition max size on jffs2

[David Woodhouse]

DGE@sixnetio.com said:
>  kmalloc() is limited to 128K and the 128MB Flash needs more than
> twice that,

> so I tried vmalloc() and it seems to be OK (at least with PPC 2.4.18
> patched for the new MTD code). Does anyone know if this change is
> safe? 

You need to change the corresponding kfree() to vfree().

--
dwmw2

Re: Problem: 51MB partition max size on jffs2

[Thomas Gleixner]

On Wednesday 04 December 2002 19:46, David Woodhouse wrote:
> DGE@sixnetio.com said:
> >  kmalloc() is limited to 128K and the 128MB Flash needs more than
> > twice that,
> >
> > so I tried vmalloc() and it seems to be OK (at least with PPC 2.4.18
> > patched for the new MTD code). Does anyone know if this change is
> > safe?
>
> You need to change the corresponding kfree() to vfree().

That's right. In actual CVS-code this problem is solved by building virtual 
blocks of 2,4 or 8 physical blocks. 

-- 
Thomas
____________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: Problem: 51MB partition max size on jffs2

[Charles Manning]

On Thu, 05 Dec 2002 08:01, Thomas Gleixner wrote:
> On Wednesday 04 December 2002 19:46, David Woodhouse wrote:
> > DGE@sixnetio.com said:
> > >  kmalloc() is limited to 128K and the 128MB Flash needs more than
> > > twice that,
> > >
> > > so I tried vmalloc() and it seems to be OK (at least with PPC 2.4.18
> > > patched for the new MTD code). Does anyone know if this change is
> > > safe?
> >
> > You need to change the corresponding kfree() to vfree().
>
> That's right. In actual CVS-code this problem is solved by building virtual
> blocks of 2,4 or 8 physical blocks.

Does this have an impact on performance?

Re: Problem: 51MB partition max size on jffs2

[Thomas Gleixner]

On Wednesday 04 December 2002 20:04, Charles Manning wrote:
> On Thu, 05 Dec 2002 08:01, Thomas Gleixner wrote:
> > On Wednesday 04 December 2002 19:46, David Woodhouse wrote:
> > > DGE@sixnetio.com said:
> > > >  kmalloc() is limited to 128K and the 128MB Flash needs more than
> > > > twice that,
> > > >
> > > > so I tried vmalloc() and it seems to be OK (at least with PPC 2.4.18
> > > > patched for the new MTD code). Does anyone know if this change is
> > > > safe?
> > >
> > > You need to change the corresponding kfree() to vfree().
> >
> > That's right. In actual CVS-code this problem is solved by building
> > virtual blocks of 2,4 or 8 physical blocks.
>
> Does this have an impact on performance?
Not at all. It just consumes less memory. Maybe garbage collect is a little 
bit slower, but I did not notice a remarkable change.

-- 
Thomas
____________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

RE: Problem: 51MB partition max size on jffs2

[Dave Ellis]

Thomas Gleixner wrote: 
> ... In actual CVS-code this problem is solved by building virtual 
> blocks of 2,4 or 8 physical blocks. 

Thanks! I hadn't checked CVS for a few weeks, so I didn't
realize there was a better solution. I'll drop the vmalloc()
change and use the CVS-code.

Dave
 

Re: Problem: 51MB partition max size on jffs2

[Ken Offer]

On Wednesday, December 4, 2002, at 11:47  AM, Thomas Gleixner wrote:

> Latest CVS code ?

I tried it with an older version (circa 20020819) and the latest code 
(20021202).  Same result.  I am going to try the change Dave Ellis 
<DGE@sixnetio.com> suggested.

+-----------------------------------------------------------+
|  Ken Offer        koffer@arlut.utexas.edu         ARL:UT  |
|  Office: 512-835-3859                  Fax: 512-835-3259  |
+-----------------------------------------------------------+

Re: Problem: 51MB partition max size on jffs2

[Thomas Gleixner]

On Wednesday 04 December 2002 23:00, Ken Offer wrote:
> On Wednesday, December 4, 2002, at 11:47  AM, Thomas Gleixner wrote:
> > Latest CVS code ?
>
> I tried it with an older version (circa 20020819) and the latest code
> (20021202).  Same result.  I am going to try the change Dave Ellis
> <DGE@sixnetio.com> suggested.
Strange. Could you please tell me, what output the latest CVS code produces

-- 
Thomas
____________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: Problem: 51MB partition max size on jffs2

[Ken Offer]

On Wednesday, December 4, 2002, at 04:02  PM, Thomas Gleixner wrote:

> On Wednesday 04 December 2002 23:00, Ken Offer wrote:
>> On Wednesday, December 4, 2002, at 11:47  AM, Thomas Gleixner wrote:
>>> Latest CVS code ?
>>
>> I tried it with an older version (circa 20020819) and the latest code
>> (20021202).  Same result.  I am going to try the change Dave Ellis
>> <DGE@sixnetio.com> suggested.
> Strange. Could you please tell me, what output the latest CVS code 
> produces

Well, I was getting a mount failure.  There wasn't much info other than 
that.  If you look in the latest (20021202) jffs2 build.c file, you can 
see if the kmalloc of c->blocks fails in jffs2_do_mount_fs, no debug 
message is printed.  Only a non-mem error is returned.  That is 
consistent with what I was seeing, even with the highest verbosity 
debugging on.

I made the change to use vmalloc/vfree for c->blocks and that seemd to 
fix the problem.  kmalloc is indeed limited to 128KB.  Of course, I 
made the change to the 20020819 code.  I'll try the vmalloc change with 
the latest CVS code as well and see how that holds up.

+-----------------------------------------------------------+
|  Ken Offer        koffer@arlut.utexas.edu         ARL:UT  |
|  Office: 512-835-3859                  Fax: 512-835-3259  |
+-----------------------------------------------------------+

Re: Problem: 51MB partition max size on jffs2

[Thomas Gleixner]

On Thursday 05 December 2002 00:08, Ken Offer wrote:
> On Wednesday, December 4, 2002, at 04:02  PM, Thomas Gleixner wrote:
> > On Wednesday 04 December 2002 23:00, Ken Offer wrote:
> >> On Wednesday, December 4, 2002, at 11:47  AM, Thomas Gleixner wrote:
> >>> Latest CVS code ?
> >>
> >> I tried it with an older version (circa 20020819) and the latest code
> >> (20021202).  Same result.  I am going to try the change Dave Ellis
> >> <DGE@sixnetio.com> suggested.
> >
> > Strange. Could you please tell me, what output the latest CVS code
> > produces
>
> Well, I was getting a mount failure.  There wasn't much info other than
> that.  If you look in the latest (20021202) jffs2 build.c file, you can
> see if the kmalloc of c->blocks fails in jffs2_do_mount_fs, no debug
> message is printed.  Only a non-mem error is returned.  That is
> consistent with what I was seeing, even with the highest verbosity
> debugging on.
>
> I made the change to use vmalloc/vfree for c->blocks and that seemd to
> fix the problem.  kmalloc is indeed limited to 128KB.  Of course, I
> made the change to the 20020819 code.  I'll try the vmalloc change with
> the latest CVS code as well and see how that holds up.
Please wait. 
Could you please verify, if the code in fs. c does, what it should do ?

int jffs2_do_fill_super(struct super_block *sb, void *data, int silent)
{
SNIP
	c->flash_size = c->mtd->size;
	/* 
	 * Check, if we have to concatenate physical blocks to larger virtual blocks
	 * to reduce the memorysize for c->blocks. (kmalloc allows max. 128K 
allocation)
	 */
	blocks = c->flash_size / c->mtd->erasesize;
	while ((blocks * sizeof (struct jffs2_eraseblock)) > (128 * 1024))
		blocks >>= 1;
	
	c->sector_size = c->flash_size / blocks;
	if (c->sector_size != c->mtd->erasesize)
		printk(KERN_INFO "jffs2: Erase block size too small (%dKiB). Using virtual 
blocks size (%dKiB) instead\n", 
			c->mtd->erasesize / 1024, c->sector_size / 1024);

This is processed before calling jffs2_do_mount_fs. There is calculated:

	c->nr_blocks = c->flash_size / c->sector_size;
	c->blocks = kmalloc(sizeof(struct jffs2_eraseblock) * c->nr_blocks, 
GFP_KERNEL);
	if (!c->blocks)
		return -ENOMEM;

So it should rely on the previous adjusted sectorsize.

-- 
Thomas
____________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: Problem: 51MB partition max size on jffs2

[Ken Offer]

On Wednesday, December 4, 2002, at 05:20  PM, Thomas Gleixner wrote:

> On Thursday 05 December 2002 00:08, Ken Offer wrote:
>> On Wednesday, December 4, 2002, at 04:02  PM, Thomas Gleixner wrote:
>>> On Wednesday 04 December 2002 23:00, Ken Offer wrote:
>>>> On Wednesday, December 4, 2002, at 11:47  AM, Thomas Gleixner wrote:
>>>>> Latest CVS code ?
>>>>
>>>> I tried it with an older version (circa 20020819) and the latest 
>>>> code
>>>> (20021202).  Same result.  I am going to try the change Dave Ellis
>>>> <DGE@sixnetio.com> suggested.
>>>
>>> Strange. Could you please tell me, what output the latest CVS code
>>> produces
>>
>> Well, I was getting a mount failure.  There wasn't much info other 
>> than
>> that.  If you look in the latest (20021202) jffs2 build.c file, you 
>> can
>> see if the kmalloc of c->blocks fails in jffs2_do_mount_fs, no debug
>> message is printed.  Only a non-mem error is returned.  That is
>> consistent with what I was seeing, even with the highest verbosity
>> debugging on.
>>
>> I made the change to use vmalloc/vfree for c->blocks and that seemd to
>> fix the problem.  kmalloc is indeed limited to 128KB.  Of course, I
>> made the change to the 20020819 code.  I'll try the vmalloc change 
>> with
>> the latest CVS code as well and see how that holds up.
> Please wait.
> Could you please verify, if the code in fs. c does, what it should do ?
>
> int jffs2_do_fill_super(struct super_block *sb, void *data, int silent)
> {
> SNIP
> 	c->flash_size = c->mtd->size;
> 	/*
> 	 * Check, if we have to concatenate physical blocks to larger virtual 
> blocks
> 	 * to reduce the memorysize for c->blocks. (kmalloc allows max. 128K
> allocation)
> 	 */
> 	blocks = c->flash_size / c->mtd->erasesize;
> 	while ((blocks * sizeof (struct jffs2_eraseblock)) > (128 * 1024))
> 		blocks >>= 1;
> 	
> 	c->sector_size = c->flash_size / blocks;
> 	if (c->sector_size != c->mtd->erasesize)
> 		printk(KERN_INFO "jffs2: Erase block size too small (%dKiB). Using 
> virtual
> blocks size (%dKiB) instead\n",
> 			c->mtd->erasesize / 1024, c->sector_size / 1024);
>
> This is processed before calling jffs2_do_mount_fs. There is 
> calculated:
>
> 	c->nr_blocks = c->flash_size / c->sector_size;
> 	c->blocks = kmalloc(sizeof(struct jffs2_eraseblock) * c->nr_blocks,
> GFP_KERNEL);
> 	if (!c->blocks)
> 		return -ENOMEM;
>
> So it should rely on the previous adjusted sectorsize.

Ok, I switched back to the latest CVS and tried to mount the partition. 
  No luck, even with a 32MB partition.  I get:

mount -t jffs2 /dev/mtdblock1 /mnt/jffs
mount: /dev/mtdblock1 is not a valid block device

I have tried the latest CVS with both the "vmalloc c->blocks" change 
and without.  In both cases mount fails the same way.  I turned up 
debugging all the way for jffs2 and mtd but no messages were printed to 
/var/log/messages.  Guess I need to scour the jffs2 code and add some 
more printks...

So... I don't know yet if the above code is working or not.  I need to 
find out why mount doesn't like /dev/mtdblock1 with the latest CVS code 
(but DOES like it for the 20020819 code).

+-----------------------------------------------------------+
|  Ken Offer        koffer@arlut.utexas.edu         ARL:UT  |
|  Office: 512-835-3859                  Fax: 512-835-3259  |
+-----------------------------------------------------------+

RE: Problem: 51MB partition max size on jffs2

[Dave Ellis]

Ken Offer wrote: 
> Ok, I switched back to the latest CVS and tried to mount the 
> partition. 
>   No luck, even with a 32MB partition.  I get:
> 
> mount -t jffs2 /dev/mtdblock1 /mnt/jffs
> mount: /dev/mtdblock1 is not a valid block device
> 
> I have tried the latest CVS with both the "vmalloc c->blocks" change 
> and without.  In both cases mount fails the same way.  I turned up 
> debugging all the way for jffs2 and mtd but no messages were 
> printed to 
> /var/log/messages.  Guess I need to scour the jffs2 code and add some 
> more printks...

The latest CVS (05-Dec-2002) is working for me with kmalloc()
for c->blocks. 128MB Nand flash, 8MB Nor Flash and slram running
on battery-backed RAM all work. I have only tried copying some
files to them, no extensive testing, but they all mount OK. This
is with PPC 2.4.18.

Dave