[ANNOUNCE] MLC support for UBI/UBIFS

[ANNOUNCE] MLC support for UBI/UBIFS

[Richard Weinberger]

Dear MTD folks,

As some of you might have noticed, the last few months Boris and I have been
heavily working on MLC support. Our first approaches targeted UBIFS since
we thought MLC NAND's most annoying feature, namely the paired pages constraint,
can be best addressed there. It turned out that UBIFS is the wrong layer and given
the massive changes required to make it work, we had to go back to the drawing board,
multiple times.
Finally we came up with an approach which is a) rather simple and b) working.
This approach operates on the UBI level and we call it "LEB consolidation",
it is also known as "write lower and merge".

The main problem with paired pages is that an interrupted write to a high page
will also damage the content of its partner, the low page. Some MLC NANDs
offer a safe SLC mode where only low pages are exposed and prevent data loss
due to interrupted writes can happen. The obvious down side is that you lose 50%
of your NAND's storage capacity.
While SLC mode is clearly not an option for us, the "LEB consolidation" approach
makes partial use of it. On MLC, UBI will operate on PEBs in two modes,
software emulated SLC and raw mode.
By default, LEBs are written in software emulated SLC mode. Hence, only lower
pages are written, such that interrupted writes won't damage already written
data. If UBI notices that it has no longer enough writable LEBs, it tries
to consolidate two full LEBs. Full means that the last page has been written
and therefore no further writes are expected. This works since you can program
an erase block only in linear manner and on MLC the number of programs (NOP)
is in general 1. This also means that there are no subpages.
LEB consolidation works more or less like UBI's atomic LEB change feature.
But instead of atomically exchanging the data of a LEB, it atomically moves
the data of two LEBs into another LEB in raw mode.
The pages of the source LEBs are read in SLC mode, on the consolidated LEB both
low and high pages are used since raw mode is used.
Consolidated LEBs are read-only by design. Especially as we merge only full LEBs.
This also implies that a PEB can host one or two LEBs.
If one LEB of a consolidated LEB pair is unmapped the other LEB is marked as
candidate for consolidation again.
If a PEB hosts consolidated LEBs it will carry two successive volume headers
right after the erase counter header. Currently the relationship between PEBs
and LEBs is 1:1, with MLC it is 1:2. Which means that UBI will expose two
times as many LEBs as PEBs. This complicates UBI a bit and userspace interfacing
needs some massaging since in many places the number of LEBs and PEBs were considered
to be the same, and the two different concepts where kind of interchangeable.
With MLC this matters and makes a difference.
As UBI is only allowed to consolidate full LEBs, a user on top of UBI should try
to fill LEBs as much as possible before mapping a new LEB. If half of all
available LEBs are mapped but not filled UBI will refuse to map more since all
available PEBs are full and there is no empty PEB to atomically consolidate two LEBs
into. Luckily, UBIFS behaves sane and does not map new LEBs before previously mapped
LEBs aren't full.

Adding MLC support in UBI requires changes in the on-flash layout, as now two
volume headers can exist. The original plan was just raising UBI version
from 1 to 2 but this turned out as almost impossible as UBI userspace
tools are not designed for backwards compatibility.
They refuse to work when UBI version is not 1. Instead of doing a flag day
release we stay with version 1 (for ever) and add a new feature flag to
UBI which indicates which extra features the implementation supports.
0x1 will be "basic UBI version 1" support, and 0x2 "LEB consolidation".

Our implementation is being tested on nandsim (with some hacks to have paired pages)
and real MLC NAND from Hynix, Micron and Toshiba.
So far it seems to work and no major issues have been spotted.
Of course it survives UBI tests too.

We'll soon start sending patches. Some patches are already on
the list and/or need an update.
For full MLC NAND support patch series for NAND core,
UBI and mtd-utils will be sent.

The current (not ready for mainline) code can be found here:
git://git.infradead.org/users/rw/ubi-mlc.git wip

Feedback and tests reports are very welcome!

We'd like to thank Next Thing co. for funding this work! :-)

Thanks,
//richard

RE: [ANNOUNCE] MLC support for UBI/UBIFS

[Bean Huo 霍斌斌 (beanhuo)]

Hi, Richard 
Do you need our Micron 70s and 80s MLC NAND sample?
Current code only enable Hynix MLC NAND, can I send patch about this our MLC
NAND paired page distance to you? I already tested them on your codes.

Yours Faithfully
BeanHuo
Cell :+86 183 2156 9399

> -----Original Message-----
> From: Richard Weinberger [mailto:richard@nod.at]
> Sent: Thursday, April 28, 2016 10:29 PM
> To: linux-mtd@lists.infradead.org
> Cc: Alexander Kaplan <alex@nextthing.co>; Boris Brezillon
> <boris.brezillon@free-electrons.com>; David Gstir <david@sigma-star.at>;
> David Oberhollenzer <goliath@sigma-star.at>; Bean Huo 霍斌斌 (beanhuo)
> <beanhuo@micron.com>; Artem Bityutskiy <dedekind1@gmail.com>
> Subject: [ANNOUNCE] MLC support for UBI/UBIFS
> 
> Dear MTD folks,
> 
> As some of you might have noticed, the last few months Boris and I have
> been heavily working on MLC support. Our first approaches targeted UBIFS
> since we thought MLC NAND's most annoying feature, namely the paired
> pages constraint, can be best addressed there. It turned out that UBIFS is the
> wrong layer and given the massive changes required to make it work, we had
> to go back to the drawing board, multiple times.
> Finally we came up with an approach which is a) rather simple and b) working.
> This approach operates on the UBI level and we call it "LEB consolidation", it
> is also known as "write lower and merge".
> 
> The main problem with paired pages is that an interrupted write to a high
> page will also damage the content of its partner, the low page. Some MLC
> NANDs offer a safe SLC mode where only low pages are exposed and prevent
> data loss due to interrupted writes can happen. The obvious down side is that
> you lose 50% of your NAND's storage capacity.
> While SLC mode is clearly not an option for us, the "LEB consolidation"
> approach makes partial use of it. On MLC, UBI will operate on PEBs in two
> modes, software emulated SLC and raw mode.
> By default, LEBs are written in software emulated SLC mode. Hence, only
> lower pages are written, such that interrupted writes won't damage already
> written data. If UBI notices that it has no longer enough writable LEBs, it tries
> to consolidate two full LEBs. Full means that the last page has been written
> and therefore no further writes are expected. This works since you can
> program an erase block only in linear manner and on MLC the number of
> programs (NOP) is in general 1. This also means that there are no subpages.
> LEB consolidation works more or less like UBI's atomic LEB change feature.
> But instead of atomically exchanging the data of a LEB, it atomically moves
> the data of two LEBs into another LEB in raw mode.
> The pages of the source LEBs are read in SLC mode, on the consolidated LEB
> both low and high pages are used since raw mode is used.
> Consolidated LEBs are read-only by design. Especially as we merge only full
> LEBs.
> This also implies that a PEB can host one or two LEBs.
> If one LEB of a consolidated LEB pair is unmapped the other LEB is marked as
> candidate for consolidation again.
> If a PEB hosts consolidated LEBs it will carry two successive volume headers
> right after the erase counter header. Currently the relationship between PEBs
> and LEBs is 1:1, with MLC it is 1:2. Which means that UBI will expose two
> times as many LEBs as PEBs. This complicates UBI a bit and userspace
> interfacing needs some massaging since in many places the number of LEBs
> and PEBs were considered to be the same, and the two different concepts
> where kind of interchangeable.
> With MLC this matters and makes a difference.
> As UBI is only allowed to consolidate full LEBs, a user on top of UBI should try
> to fill LEBs as much as possible before mapping a new LEB. If half of all
> available LEBs are mapped but not filled UBI will refuse to map more since all
> available PEBs are full and there is no empty PEB to atomically consolidate
> two LEBs into. Luckily, UBIFS behaves sane and does not map new LEBs
> before previously mapped LEBs aren't full.
> 
> Adding MLC support in UBI requires changes in the on-flash layout, as now
> two volume headers can exist. The original plan was just raising UBI version
> from 1 to 2 but this turned out as almost impossible as UBI userspace tools
> are not designed for backwards compatibility.
> They refuse to work when UBI version is not 1. Instead of doing a flag day
> release we stay with version 1 (for ever) and add a new feature flag to UBI
> which indicates which extra features the implementation supports.
> 0x1 will be "basic UBI version 1" support, and 0x2 "LEB consolidation".
> 
> Our implementation is being tested on nandsim (with some hacks to have
> paired pages) and real MLC NAND from Hynix, Micron and Toshiba.
> So far it seems to work and no major issues have been spotted.
> Of course it survives UBI tests too.
> 
> We'll soon start sending patches. Some patches are already on the list and/or
> need an update.
> For full MLC NAND support patch series for NAND core, UBI and mtd-utils will
> be sent.
> 
> The current (not ready for mainline) code can be found here:
> git://git.infradead.org/users/rw/ubi-mlc.git wip
> 
> Feedback and tests reports are very welcome!
> 
> We'd like to thank Next Thing co. for funding this work! :-)
> 
> Thanks,
> //richard

Re: [ANNOUNCE] MLC support for UBI/UBIFS

[Richard Weinberger]

Bean,

Am 04.05.2016 um 09:40 schrieb Bean Huo 霍斌斌 (beanhuo):
> Hi, Richard 
> Do you need our Micron 70s and 80s MLC NAND sample?
> Current code only enable Hynix MLC NAND, can I send patch about this our MLC
> NAND paired page distance to you? I already tested them on your codes.

Patches are always welcome. :-)

Does your MLC have a mapping other than dist3 or dist6
as implemented in Boris' "[PATCH 2/4] mtd: nand: implement two pairing scheme"
patch?

Thanks,
//richard

RE: [ANNOUNCE] MLC support for UBI/UBIFS

[Bean Huo 霍斌斌 (beanhuo)]

Hi, Richard
Yes , different with "dis6 and dis3", but I think this is not a perfect 
Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.


Yours Faithfully
BeanHuo
Cell :+86 183 2156 9399


> -----Original Message-----
> From: Richard Weinberger [mailto:richard@nod.at]
> Sent: Wednesday, May 04, 2016 4:27 PM
> To: Bean Huo 霍斌斌 (beanhuo) <beanhuo@micron.com>;
> linux-mtd@lists.infradead.org
> Cc: Alexander Kaplan <alex@nextthing.co>; Boris Brezillon
> <boris.brezillon@free-electrons.com>; David Gstir <david@sigma-star.at>;
> David Oberhollenzer <goliath@sigma-star.at>; Artem Bityutskiy
> <dedekind1@gmail.com>
> Subject: Re: [ANNOUNCE] MLC support for UBI/UBIFS
> 
> Bean,
> 
> Am 04.05.2016 um 09:40 schrieb Bean Huo 霍斌斌 (beanhuo):
> > Hi, Richard
> > Do you need our Micron 70s and 80s MLC NAND sample?
> > Current code only enable Hynix MLC NAND, can I send patch about this
> > our MLC NAND paired page distance to you? I already tested them on your
> codes.
> 
> Patches are always welcome. :-)
> 
> Does your MLC have a mapping other than dist3 or dist6 as implemented in
> Boris' "[PATCH 2/4] mtd: nand: implement two pairing scheme"
> patch?
> 
> Thanks,
> //richard

Re: [ANNOUNCE] MLC support for UBI/UBIFS

[Boris Brezillon]

On Wed, 4 May 2016 08:55:12 +0000
Bean Huo 霍斌斌 (beanhuo) <beanhuo@micron.com> wrote:

> Hi, Richard
> Yes , different with "dis6 and dis3", but I think this is not a perfect 
> Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.

Sorry but I'm clearly opposed to that: the NAND infrastructure provides
a way to uniquely identify the part number, let's not add extra
information in the DT if it's not needed.
Just giving a simple use case where putting NAND chip details in the DT
is a bad idea: some board manufacturers choose 2 or 3 equivalent NANDs
coming from different vendors (and those NANDs may have different
pairing scheme). With your solution this means having one DT per NAND
chip, which is not easy to deal with.

For the "the table will keep growing and become unmaintainable"
argument, I agree. My plan is to provide a per-vendor ->init() hook
so that each vendor can implement a simpler solution to detect/assign
the pairing scheme (I'm pretty sure this is correlated to the NAND
technology...), or other vendor specific stuff (read-retry, HW SLC
mode, ...).

Regarding your initial statement, are you sure your NAND chip does not
use one of the already defined scheme. I had a look at several NAND
datasheets (including Micron ones), and all of them were using either
distance 3 or distance 6.
Can you share more information about those different pairing scheme?

Thanks,

Boris


-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: [ANNOUNCE] MLC support for UBI/UBIFS

[Richard Weinberger]

Bean,

Am 04.05.2016 um 10:55 schrieb Bean Huo 霍斌斌 (beanhuo):
> Hi, Richard
> Yes , different with "dis6 and dis3", but I think this is not a perfect 
> Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.

Not sure if I understand you correctly.
Is it not possible to describe your pairing scheme in an algorithmic way?
Did you check Boris' paired pages patches for MTD?
Please try to implement it.
Having the pairing table in DT is IMHO not a good thing.

Thanks,
//richard

Re: [ANNOUNCE] MLC support for UBI/UBIFS

[Boris Brezillon]

On Wed, 4 May 2016 11:17:35 +0200
Boris Brezillon <boris.brezillon@free-electrons.com> wrote:

> On Wed, 4 May 2016 08:55:12 +0000
> Bean Huo 霍斌斌 (beanhuo) <beanhuo@micron.com> wrote:
> 
> > Hi, Richard
> > Yes , different with "dis6 and dis3", but I think this is not a perfect 
> > Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.  
> 
> Sorry but I'm clearly opposed to that: the NAND infrastructure provides
> a way to uniquely identify the part number, let's not add extra
> information in the DT if it's not needed.
> Just giving a simple use case where putting NAND chip details in the DT
> is a bad idea: some board manufacturers choose 2 or 3 equivalent NANDs
> coming from different vendors (and those NANDs may have different
> pairing scheme). With your solution this means having one DT per NAND
> chip, which is not easy to deal with.
> 
> For the "the table will keep growing and become unmaintainable"
> argument, I agree. My plan is to provide a per-vendor ->init() hook
> so that each vendor can implement a simpler solution to detect/assign
> the pairing scheme (I'm pretty sure this is correlated to the NAND
> technology...), or other vendor specific stuff (read-retry, HW SLC
> mode, ...).
> 
> Regarding your initial statement, are you sure your NAND chip does not
> use one of the already defined scheme. I had a look at several NAND
> datasheets (including Micron ones), and all of them were using either
> distance 3 or distance 6.
> Can you share more information about those different pairing scheme?

Okay, I had a closer look at your "driver:mtd:ubi:add new bakvol module
in ubi layer" patch and some Micron datasheet, and it seems your L7X
NANDs are using the "standard" distance 6 scheme, but L8X are actually
using a different scheme.
This being said, I don't see any problem implementing this new scheme
(the question is, is it really used by other vendors or not).


-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com