[PATCH] mtd: Expand the ecc placement locations to 1216

[PATCH] mtd: Expand the ecc placement locations to 1216

[Punnaiah Choudary Kalluri]

Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
So, increasing the maximum ecc placement locations to 1216

Signed-off-by: Punnaiah Choudary Kalluri <punnaia@xilinx.com>
---
 include/linux/mtd/mtd.h    | 2 +-
 include/uapi/mtd/mtd-abi.h | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
index f17fa75..1fd3cc6 100644
--- a/include/linux/mtd/mtd.h
+++ b/include/linux/mtd/mtd.h
@@ -95,7 +95,7 @@ struct mtd_oob_ops {
 };
 
 #define MTD_MAX_OOBFREE_ENTRIES_LARGE	32
-#define MTD_MAX_ECCPOS_ENTRIES_LARGE	640
+#define MTD_MAX_ECCPOS_ENTRIES_LARGE	1216
 /*
  * Internal ECC layout control structure. For historical reasons, there is a
  * similar, smaller struct nand_ecclayout_user (in mtd-abi.h) that is retained
diff --git a/include/uapi/mtd/mtd-abi.h b/include/uapi/mtd/mtd-abi.h
index 763bb69..c4d592c 100644
--- a/include/uapi/mtd/mtd-abi.h
+++ b/include/uapi/mtd/mtd-abi.h
@@ -220,8 +220,8 @@ struct nand_oobfree {
 	__u32 length;
 };
 
-#define MTD_MAX_OOBFREE_ENTRIES	8
-#define MTD_MAX_ECCPOS_ENTRIES	64
+#define MTD_MAX_OOBFREE_ENTRIES	32
+#define MTD_MAX_ECCPOS_ENTRIES	1216
 /*
  * OBSOLETE: ECC layout control structure. Exported to user-space via ioctl
  * ECCGETLAYOUT for backwards compatbility and should not be mistaken as a
-- 
2.1.2

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[Brian Norris]

On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
> Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
> So, increasing the maximum ecc placement locations to 1216

I'd really prefer not increasing the size of the internal arrays any
more. The structures should be rewritten to be dynamic.

> Signed-off-by: Punnaiah Choudary Kalluri <punnaia@xilinx.com>
> ---
>  include/linux/mtd/mtd.h    | 2 +-
>  include/uapi/mtd/mtd-abi.h | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
> index f17fa75..1fd3cc6 100644
> --- a/include/linux/mtd/mtd.h
> +++ b/include/linux/mtd/mtd.h
> @@ -95,7 +95,7 @@ struct mtd_oob_ops {
>  };
>  
>  #define MTD_MAX_OOBFREE_ENTRIES_LARGE	32
> -#define MTD_MAX_ECCPOS_ENTRIES_LARGE	640
> +#define MTD_MAX_ECCPOS_ENTRIES_LARGE	1216
>  /*
>   * Internal ECC layout control structure. For historical reasons, there is a
>   * similar, smaller struct nand_ecclayout_user (in mtd-abi.h) that is retained
> diff --git a/include/uapi/mtd/mtd-abi.h b/include/uapi/mtd/mtd-abi.h
> index 763bb69..c4d592c 100644
> --- a/include/uapi/mtd/mtd-abi.h
> +++ b/include/uapi/mtd/mtd-abi.h
> @@ -220,8 +220,8 @@ struct nand_oobfree {
>  	__u32 length;
>  };
>  
> -#define MTD_MAX_OOBFREE_ENTRIES	8
> -#define MTD_MAX_ECCPOS_ENTRIES	64
> +#define MTD_MAX_OOBFREE_ENTRIES	32
> +#define MTD_MAX_ECCPOS_ENTRIES	1216

NAK. This is part of the ABI, and changing this will break user space.
If you actually bothered to read code, you might understand why there
are now internal and external versions of this struct. The external
(user space) version cannot be changed. The internal version can be
refactored, as long as the external version still maintains some sanity,
at least for old/small devices.

>  /*
>   * OBSOLETE: ECC layout control structure. Exported to user-space via ioctl
>   * ECCGETLAYOUT for backwards compatbility and should not be mistaken as a

Brian

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[punnaiah choudary kalluri]

On Sat, Nov 21, 2015 at 1:03 AM, Brian Norris
<computersforpeace@gmail.com> wrote:
> On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
>> Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
>> So, increasing the maximum ecc placement locations to 1216
>
> I'd really prefer not increasing the size of the internal arrays any
> more. The structures should be rewritten to be dynamic.

Ok. I have seen the comment "nand_ecclayout should be expandable in the
future simply by the above macros" and so increased the macro sizes.
As you said, it is ideal to have memory allocated dynamically for eccpos and
oobfree fields.

>
>> Signed-off-by: Punnaiah Choudary Kalluri <punnaia@xilinx.com>
>> ---
>>  include/linux/mtd/mtd.h    | 2 +-
>>  include/uapi/mtd/mtd-abi.h | 4 ++--
>>  2 files changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
>> index f17fa75..1fd3cc6 100644
>> --- a/include/linux/mtd/mtd.h
>> +++ b/include/linux/mtd/mtd.h
>> @@ -95,7 +95,7 @@ struct mtd_oob_ops {
>>  };
>>
>>  #define MTD_MAX_OOBFREE_ENTRIES_LARGE        32
>> -#define MTD_MAX_ECCPOS_ENTRIES_LARGE 640
>> +#define MTD_MAX_ECCPOS_ENTRIES_LARGE 1216
>>  /*
>>   * Internal ECC layout control structure. For historical reasons, there is a
>>   * similar, smaller struct nand_ecclayout_user (in mtd-abi.h) that is retained
>> diff --git a/include/uapi/mtd/mtd-abi.h b/include/uapi/mtd/mtd-abi.h
>> index 763bb69..c4d592c 100644
>> --- a/include/uapi/mtd/mtd-abi.h
>> +++ b/include/uapi/mtd/mtd-abi.h
>> @@ -220,8 +220,8 @@ struct nand_oobfree {
>>       __u32 length;
>>  };
>>
>> -#define MTD_MAX_OOBFREE_ENTRIES      8
>> -#define MTD_MAX_ECCPOS_ENTRIES       64
>> +#define MTD_MAX_OOBFREE_ENTRIES      32
>> +#define MTD_MAX_ECCPOS_ENTRIES       1216
>
> NAK. This is part of the ABI, and changing this will break user space.
> If you actually bothered to read code, you might understand why there
> are now internal and external versions of this struct. The external
> (user space) version cannot be changed. The internal version can be
> refactored, as long as the external version still maintains some sanity,
> at least for old/small devices.

Sorry, I am confused here.
Arasan nand controller supports 24 bit ecc and uses BCH algorithm. So,
it requires
672 ecc positions. in this case, how the user space know which are the free
slots available in spare area?
because as per the above definitions only 64 positions can be defined.

Regards,
Punnaiah

>
>>  /*
>>   * OBSOLETE: ECC layout control structure. Exported to user-space via ioctl
>>   * ECCGETLAYOUT for backwards compatbility and should not be mistaken as a
>
> Brian

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[Brian Norris]

Hi Punnaiah,

On Sat, Nov 21, 2015 at 08:59:15PM +0530, punnaiah choudary kalluri wrote:
> On Sat, Nov 21, 2015 at 1:03 AM, Brian Norris
> <computersforpeace@gmail.com> wrote:
> > On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
> >> Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
> >> So, increasing the maximum ecc placement locations to 1216
> >
> > I'd really prefer not increasing the size of the internal arrays any
> > more. The structures should be rewritten to be dynamic.
> 
> Ok. I have seen the comment "nand_ecclayout should be expandable in the
> future simply by the above macros" and so increased the macro sizes.
> As you said, it is ideal to have memory allocated dynamically for eccpos and
> oobfree fields.

Yes, it *can* be done. Previously, we couldn't do this, because any
change there would also change the ABI, which is off limits, but I fixed
that by separating the 'nand_ecclayout' and 'nand_ecclayout_user'
structs. That still doesn't make it preferable. These structs are
getting pretty big, and they hold a very low density of information...

> >
> >> Signed-off-by: Punnaiah Choudary Kalluri <punnaia@xilinx.com>
> >> ---
> >>  include/linux/mtd/mtd.h    | 2 +-
> >>  include/uapi/mtd/mtd-abi.h | 4 ++--
> >>  2 files changed, 3 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
> >> index f17fa75..1fd3cc6 100644
> >> --- a/include/linux/mtd/mtd.h
> >> +++ b/include/linux/mtd/mtd.h
> >> @@ -95,7 +95,7 @@ struct mtd_oob_ops {
> >>  };
> >>
> >>  #define MTD_MAX_OOBFREE_ENTRIES_LARGE        32
> >> -#define MTD_MAX_ECCPOS_ENTRIES_LARGE 640
> >> +#define MTD_MAX_ECCPOS_ENTRIES_LARGE 1216
> >>  /*
> >>   * Internal ECC layout control structure. For historical reasons, there is a
> >>   * similar, smaller struct nand_ecclayout_user (in mtd-abi.h) that is retained
> >> diff --git a/include/uapi/mtd/mtd-abi.h b/include/uapi/mtd/mtd-abi.h
> >> index 763bb69..c4d592c 100644
> >> --- a/include/uapi/mtd/mtd-abi.h
> >> +++ b/include/uapi/mtd/mtd-abi.h
> >> @@ -220,8 +220,8 @@ struct nand_oobfree {
> >>       __u32 length;
> >>  };
> >>
> >> -#define MTD_MAX_OOBFREE_ENTRIES      8
> >> -#define MTD_MAX_ECCPOS_ENTRIES       64
> >> +#define MTD_MAX_OOBFREE_ENTRIES      32
> >> +#define MTD_MAX_ECCPOS_ENTRIES       1216
> >
> > NAK. This is part of the ABI, and changing this will break user space.
> > If you actually bothered to read code, you might understand why there
> > are now internal and external versions of this struct. The external
> > (user space) version cannot be changed. The internal version can be
> > refactored, as long as the external version still maintains some sanity,
> > at least for old/small devices.
> 
> Sorry, I am confused here.
> Arasan nand controller supports 24 bit ecc and uses BCH algorithm. So,
> it requires
> 672 ecc positions. in this case, how the user space know which are the free
> slots available in spare area?
> because as per the above definitions only 64 positions can be defined.

If you actually need to know this info from user space, then you need to
define a new interface to expose that. The existing one is legacy, and
it is insufficient for most modern use cases. It won't work for you.

Other mechanisms can already help you though. For one, auto-placed OOB
(e.g., 'nandwrite --autoplace') can let the MTD layer automatically put
your data into the available spare area.

Why do you need to access the spare area anyway? Modern NAND filesystems
shouldn't really be storing anything there anyway.

Brian

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[punnaiah choudary kalluri]

Hi Brian,

On Tue, Dec 1, 2015 at 2:41 AM, Brian Norris
<computersforpeace@gmail.com> wrote:
> Hi Punnaiah,
>
> On Sat, Nov 21, 2015 at 08:59:15PM +0530, punnaiah choudary kalluri wrote:
>> On Sat, Nov 21, 2015 at 1:03 AM, Brian Norris
>> <computersforpeace@gmail.com> wrote:
>> > On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
>> >> Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
>> >> So, increasing the maximum ecc placement locations to 1216
>> >
>> > I'd really prefer not increasing the size of the internal arrays any
>> > more. The structures should be rewritten to be dynamic.
>>
>> Ok. I have seen the comment "nand_ecclayout should be expandable in the
>> future simply by the above macros" and so increased the macro sizes.
>> As you said, it is ideal to have memory allocated dynamically for eccpos and
>> oobfree fields.
>
> Yes, it *can* be done. Previously, we couldn't do this, because any
> change there would also change the ABI, which is off limits, but I fixed
> that by separating the 'nand_ecclayout' and 'nand_ecclayout_user'
> structs. That still doesn't make it preferable. These structs are
> getting pretty big, and they hold a very low density of information...
>
>> >
>> >> Signed-off-by: Punnaiah Choudary Kalluri <punnaia@xilinx.com>
>> >> ---
>> >>  include/linux/mtd/mtd.h    | 2 +-
>> >>  include/uapi/mtd/mtd-abi.h | 4 ++--
>> >>  2 files changed, 3 insertions(+), 3 deletions(-)
>> >>
>> >> diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
>> >> index f17fa75..1fd3cc6 100644
>> >> --- a/include/linux/mtd/mtd.h
>> >> +++ b/include/linux/mtd/mtd.h
>> >> @@ -95,7 +95,7 @@ struct mtd_oob_ops {
>> >>  };
>> >>
>> >>  #define MTD_MAX_OOBFREE_ENTRIES_LARGE        32
>> >> -#define MTD_MAX_ECCPOS_ENTRIES_LARGE 640
>> >> +#define MTD_MAX_ECCPOS_ENTRIES_LARGE 1216
>> >>  /*
>> >>   * Internal ECC layout control structure. For historical reasons, there is a
>> >>   * similar, smaller struct nand_ecclayout_user (in mtd-abi.h) that is retained
>> >> diff --git a/include/uapi/mtd/mtd-abi.h b/include/uapi/mtd/mtd-abi.h
>> >> index 763bb69..c4d592c 100644
>> >> --- a/include/uapi/mtd/mtd-abi.h
>> >> +++ b/include/uapi/mtd/mtd-abi.h
>> >> @@ -220,8 +220,8 @@ struct nand_oobfree {
>> >>       __u32 length;
>> >>  };
>> >>
>> >> -#define MTD_MAX_OOBFREE_ENTRIES      8
>> >> -#define MTD_MAX_ECCPOS_ENTRIES       64
>> >> +#define MTD_MAX_OOBFREE_ENTRIES      32
>> >> +#define MTD_MAX_ECCPOS_ENTRIES       1216
>> >
>> > NAK. This is part of the ABI, and changing this will break user space.
>> > If you actually bothered to read code, you might understand why there
>> > are now internal and external versions of this struct. The external
>> > (user space) version cannot be changed. The internal version can be
>> > refactored, as long as the external version still maintains some sanity,
>> > at least for old/small devices.
>>
>> Sorry, I am confused here.
>> Arasan nand controller supports 24 bit ecc and uses BCH algorithm. So,
>> it requires
>> 672 ecc positions. in this case, how the user space know which are the free
>> slots available in spare area?
>> because as per the above definitions only 64 positions can be defined.
>
> If you actually need to know this info from user space, then you need to
> define a new interface to expose that. The existing one is legacy, and
> it is insufficient for most modern use cases. It won't work for you.
>
> Other mechanisms can already help you though. For one, auto-placed OOB
> (e.g., 'nandwrite --autoplace') can let the MTD layer automatically put
> your data into the available spare area.
>
> Why do you need to access the spare area anyway? Modern NAND filesystems
> shouldn't really be storing anything there anyway.
>

I have tested jffs2 and ubifs file system using arasan nand driver but
observed issues
with jffs2 filesystem. While debugging further found that the jffs2
file system is using
the oob area for storing the clear markers.

Not only for file system, probably one can implement application for
accessing  the
nand in raw mode and may utilize the oob area for other purposes like
storing the
encryption/decryption  key associated with the page data.

not related to this patch, request your time and help for reviewing
the arasan nand driver patch.
http://permalink.gmane.org/gmane.linux.kernel/2090815


Regards,
Punnaiah
> Brian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[Boris Brezillon]

Brian, Punnaiah,

On Fri, 20 Nov 2015 11:33:42 -0800
Brian Norris <computersforpeace@gmail.com> wrote:

> On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
> > Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
> > So, increasing the maximum ecc placement locations to 1216
> 
> I'd really prefer not increasing the size of the internal arrays any
> more. The structures should be rewritten to be dynamic.

I already started this migration (you can find the code here [1]). I
was planning to rework this series to make the eccpos definition mimic
what's done for oobfree (as suggested by Brian) before sending it, but
I'm not sure when I'll be able to do that, so if you really need that
quickly I suggest you submit those patches instead of reimplementing
it yourself.

Best Regards,

Boris

[1]https://github.com/bbrezillon/linux-sunxi/commits/nand/ecclayout

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: [PATCH] mtd: Expand the ecc placement locations to 1216

[punnaiah choudary kalluri]

Hi Boris,

On Tue, Dec 1, 2015 at 4:45 PM, Boris Brezillon
<boris.brezillon@free-electrons.com> wrote:
> Brian, Punnaiah,
>
> On Fri, 20 Nov 2015 11:33:42 -0800
> Brian Norris <computersforpeace@gmail.com> wrote:
>
>> On Thu, Nov 05, 2015 at 10:28:55PM +0530, Punnaiah Choudary Kalluri wrote:
>> > Device like MT29F32G08ABCDBJ4 have a writesize/oobsize of 16K/1216 Bytes.
>> > So, increasing the maximum ecc placement locations to 1216
>>
>> I'd really prefer not increasing the size of the internal arrays any
>> more. The structures should be rewritten to be dynamic.
>
> I already started this migration (you can find the code here [1]). I
> was planning to rework this series to make the eccpos definition mimic
> what's done for oobfree (as suggested by Brian) before sending it, but
> I'm not sure when I'll be able to do that, so if you really need that
> quickly I suggest you submit those patches instead of reimplementing
> it yourself.
>

Sure. let me check your implementation and then send these patches for review
with your sign off.

Thanks,
Punnaiah

> Best Regards,
>
> Boris
>
> [1]https://github.com/bbrezillon/linux-sunxi/commits/nand/ecclayout
>
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com