Eraseblocks torture: OneNAND results

Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 2851 bytes --]

Hello Kyungmin,

We have a test board with KFN2G16Q2M 256MiB OneNAND. We decided to write
a torture test which will torture few eraseblocks just in order to see
what happens.

We wrote a small test program which basically erases several eraseblocks
in a cycle till it gets an error. The test program is attached
(torture.git.tar.bz2), and it is also available at
git://git.infradead.org/~dedekind/torture.git.

By default the program does the following:

1. Erases an eraseblock. Reads it back and makes sure there are only
0xFF bytes.
2. Writes 0x55/0xAA pattern. In case of NAND we store 0x55 at one page,
0xAA at the next and so on. Each next erase we switch 0x55 and 0xAA
bytes.
3. Read the eraseblock back and make sure we read the same data.

And so on till an error occurs. Of course we check return codes.

The reason for this test is just because we are curious how our OneNAND
setup behaves in case of worn-out eraseblocks.

We have got kind of strange result. What we have is that after several
million erase cycles we start reading incorrect data back. Sometimes
there are one-bit errors, sometimes many-byte errors. We do not get any
error code from mtd->read(). We do not see single-bit errors corrected.
mtd->write() and mtd->erase() functions do not return any error as well.

Kyungmin, did you do any kind of tests like this? I offer you to try our
test too.

Other people may also try to wear-out few eraseblocks on their devices
and see what happens. Then for example, mount JFFS2 and see what it
says/does.

But please, beware, the test may damage your system so run it only if
you know exactly what you do. Authors are not responsible for any
damaged caused by this test.

----------------------------------------------------------------
[54592.767700] EB torture: Page 0 has 4 bytes/16 bits failing verify,
starting at offset 0x1c
[54592.776214] Offset           Read                   Written
[54592.781860] 0x018:  aa aa aa aa 00 00 00 00  ***   aa aa aa aa aa aa
aa aa
[54592.789123] 0x020:  aa aa aa aa aa aa aa aa        aa aa aa aa aa aa
aa aa
----------------------------------------------------------------
[90073.926055] EB torture: Page 0 has 20 bytes/160 bits failing verify,
starting at offset 0xc
[90073.934661] Offset           Read                   Written
[90073.940490] 0x008:  55 55 55 55 aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.947784] 0x010:  aa aa aa aa aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.954895] 0x018:  aa aa aa aa aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.962158] 0x020:  55 55 55 55 55 55 55 55        55 55 55 55 55 55
55 55
----------------------------------------------------------------


Below go some examples of test failures.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

[-- Attachment #2: torture.git.tar.bz2 --]
[-- Type: application/x-bzip-compressed-tar, Size: 13108 bytes --]

Eraseblocks torture: OneNAND results

[Kyungmin Park]

Hi Artem,

Okay, I also try to test attached program for this weekend.

However, I have a question
There's some strange pattern in log.
In any case. it can't occur from 0xaa(0b1010) to 0x55(0b0101) since it's impossible to change from 0 to 1 even though it's possible from 1 to 0

Anyway, I also ask the hardware team to check this problem.

And also could you send the chip dump data to me? since it takes a long time to worn-out. I first analyze the worn-out chip data.

If you have any issues or updated news. please let me know. 

Thank you,
Kyungmin Park

------- Original Message -------
Sender : Artem Bityutskiy<dedekind@infradead.org> 
Date   : Dec 07, 2006 23:30
Title  : Eraseblocks torture: OneNAND results

Hello Kyungmin,

We have a test board with KFN2G16Q2M 256MiB OneNAND. We decided to write
a torture test which will torture few eraseblocks just in order to see
what happens.

We wrote a small test program which basically erases several eraseblocks
in a cycle till it gets an error. The test program is attached
(torture.git.tar.bz2), and it is also available at
git://git.infradead.org/~dedekind/torture.git.

By default the program does the following:

1. Erases an eraseblock. Reads it back and makes sure there are only
0xFF bytes.
2. Writes 0x55/0xAA pattern. In case of NAND we store 0x55 at one page,
0xAA at the next and so on. Each next erase we switch 0x55 and 0xAA
bytes.
3. Read the eraseblock back and make sure we read the same data.

And so on till an error occurs. Of course we check return codes.

The reason for this test is just because we are curious how our OneNAND
setup behaves in case of worn-out eraseblocks.

We have got kind of strange result. What we have is that after several
million erase cycles we start reading incorrect data back. Sometimes
there are one-bit errors, sometimes many-byte errors. We do not get any
error code from mtd->read(). We do not see single-bit errors corrected.
mtd->write() and mtd->erase() functions do not return any error as well.

Kyungmin, did you do any kind of tests like this? I offer you to try our
test too.

Other people may also try to wear-out few eraseblocks on their devices
and see what happens. Then for example, mount JFFS2 and see what it
says/does.

But please, beware, the test may damage your system so run it only if
you know exactly what you do. Authors are not responsible for any
damaged caused by this test.

----------------------------------------------------------------
[54592.767700] EB torture: Page 0 has 4 bytes/16 bits failing verify,
starting at offset 0x1c
[54592.776214] Offset           Read                   Written
[54592.781860] 0x018:  aa aa aa aa 00 00 00 00  ***   aa aa aa aa aa aa
aa aa
[54592.789123] 0x020:  aa aa aa aa aa aa aa aa        aa aa aa aa aa aa
aa aa
----------------------------------------------------------------
[90073.926055] EB torture: Page 0 has 20 bytes/160 bits failing verify,
starting at offset 0xc
[90073.934661] Offset           Read                   Written
[90073.940490] 0x008:  55 55 55 55 aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.947784] 0x010:  aa aa aa aa aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.954895] 0x018:  aa aa aa aa aa aa aa aa  ***   55 55 55 55 55 55
55 55
[90073.962158] 0x020:  55 55 55 55 55 55 55 55        55 55 55 55 55 55
55 55
----------------------------------------------------------------


Below go some examples of test failures.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

Kyungmin,

On Fri, 2006-12-08 at 02:00 +0000, Kyungmin Park wrote:
> Okay, I also try to test attached program for this weekend.
If you use it (which would _very_ be nice!), please, take the latest
version from git. Also, take into account the following notes:

1. Although the documented eraseblock lifetime is about 100000, we
started getting errors after about 6 million erase cycles. May be
because our test is not torturous enough.

2. There is a "check" module option which is enabled by default. It
slows the test down considerably. So I recommend to disable checking at
first, run the test for, say 4 million erase cycles, then re-run it with
checking enabled. So that you first screw up the eraseblocks, then you
start checking data. There is a handy "cycles_count" option.

3. By default the test tortures 32 eraseblocks. You may configure this
via a module parameter. Just glance inside of the torture.c.

> However, I have a question
> There's some strange pattern in log.
> In any case. it can't occur from 0xaa(0b1010) to 0x55(0b0101) since it's impossible to change from 0 to 1 even though it's possible from 1 to 0
Yeah, no idea. We didn't check that eraseblock contains all 0xFF bytes
after erase. May be erase operation left some garbage? In the newer
version we do check this.

> Anyway, I also ask the hardware team to check this problem.
Would be cool!

> And also could you send the chip dump data to me? since it takes a long time to worn-out. I first analyze the worn-out chip data.
I will send you more data.

> If you have any issues or updated news. please let me know. 
Sure.

P.S.: Test git: git://git.infradead.org/~dedekind/torture.git
Web snout for the Git tree: 
http://git.infradead.org/?p=users/dedekind/torture.git;a=summary

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Eraseblocks torture: OneNAND results

[Kyungmin Park]

Hi Artem,

> 2. There is a "check" module option which is enabled by default. It
> slows the test down considerably. So I recommend to disable checking at
> first, run the test for, say 4 million erase cycles, then re-run it with
> checking enabled. So that you first screw up the eraseblocks, then you
> start checking data. There is a handy "cycles_count" option.
>
> 3. By default the test tortures 32 eraseblocks. You may configure this
> via a module parameter. Just glance inside of the torture.c.

Yes I already modified the source for my environment. and also add check initial bad block.

in tort_init()

       while (1) {
                int i;

                for(i = eb; i < eb + ebcnt; i++) {
                        err = ebtest(i);
                        /* Skip initial bad block */
                        if (err == -EFAULT)
                                continue;
                        if (err)
                                break;
                }

in ebtest()

        err = mtd->erase(mtd, &ei);
        if (unlikely(err)) {
                printk(PRINT_PREF "error %d while erasing EB %d\n", err, ebnum);
                /* Initial bad block case */
                if (err == -EIO)
                        err = -EFAULT;
                return err;
        }

> P.S.: Test git: git://git.infradead.org/~dedekind/torture.git

I already downloaded it.

After weekend test. I will send the results.


Thank you,
Kyungmin Park

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

Kyungmin,

On Fri, 2006-12-08 at 07:42 +0000, Kyungmin Park wrote:
> Yes I already modified the source for my environment. and also add check initial bad block.

Ah, indeed, thanks. I will add bad block checking. Also, I am
re-structuring the test now to make it baster and more torturous. I am
going to do erase/write/operations with the whole group, not with each
eraseblock separately. HW guys think it should cause more trouble for
flash. I will notify you when to update.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

Kyungmin,

On Fri, 2006-12-08 at 10:08 +0200, Artem Bityutskiy wrote:
> Ah, indeed, thanks. I will add bad block checking. Also, I am
> re-structuring the test now to make it baster and more torturous. I am
> going to do erase/write/operations with the whole group, not with each
> eraseblock separately. HW guys think it should cause more trouble for
> flash. I will notify you when to update.

I have re-worked the test and made it aware of bad eraseblock. The
updated version is at git.

I also got one more error, the log is attached. I do not know how it
happens that we write 0xAA but read 0x55. But what I can say is that the
first errors we saw were one-bit errors. Now we mostly get massive
corruptions. This looks strange. May be we should leave these (20-51)
eraseblocks alone and start torturing new set of eraseblocks.

Anyways, I hope you will run the test and see what you have on your
system.

I have attach one more failure log. The symptoms look the same. But now
we verify eraseblocks after erase, so we can be sure erase does not
leave garbage in eraseblocks.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

[-- Attachment #2: 2nd_g_torture_01.txt --]
[-- Type: text/plain, Size: 21923 bytes --]

insmod torture.ko dev=4
[12016.978698] 
[12016.980529] =========================================================================================
[12016.990051] EB torture: Torture 32 eraseblocks (20-52) of mtd4
[12016.996093] EB torture: Write verify enabled
[13670.757598] - 00001000 erase cycles done
[15319.406188] - 00002000 erase cycles done
[15416.399993] EB torture: read wrong data from EB 43 after
[15416.412719] EB torture: Verify fails on 1 pages, 2016 bytes/16128 bits
[15416.419525] EB torture: The following is a list of all differences between what was read from flash and what was expected
[15416.431579] -----------------------------------------------------------------------------------------
[15416.441101] EB torture: Page 0 has 2016 bytes/16128 bits failing verify, starting at offset 0x0
[15416.450225] Offset       Read                          Written
[15416.456115] 0x00000000:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.463745] 0x00000008:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.471374] 0x00000010:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.479003] 0x00000018:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.486633] 0x00000020:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.494262] 0x00000028:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.501892] 0x00000030:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.509521] 0x00000038:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.517150] 0x00000040:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.524780] 0x00000048:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.532409] 0x00000050:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.540039] 0x00000058:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.547668] 0x00000060:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.555297] 0x00000068:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.562927] 0x00000070:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.570556] 0x00000078:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.578186] 0x00000080:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.585845] 0x00000088:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.593475] 0x00000090:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.601104] 0x00000098:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.608734] 0x000000a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.616363] 0x000000a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.623992] 0x000000b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.631622] 0x000000b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.639251] 0x000000c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.646881] 0x000000c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.654510] 0x000000d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.662170] 0x000000d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.669799] 0x000000e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.677429] 0x000000e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.685058] 0x000000f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.692687] 0x000000f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.700317] 0x00000100:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.707946] 0x00000108:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.715576] 0x00000110:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.723205] 0x00000118:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.730834] 0x00000120:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.738464] 0x00000128:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.746093] 0x00000130:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.753540] 0x00000138:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.761169] 0x00000140:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.768798] 0x00000148:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.776428] 0x00000150:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.784057] 0x00000158:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.791687] 0x00000160:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.799316] 0x00000168:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.806945] 0x00000170:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.814575] 0x00000178:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.822204] 0x00000180:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.829833] 0x00000188:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.837463] 0x00000190:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.845092] 0x00000198:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.852722] 0x000001a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.860351] 0x000001a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.867980] 0x000001b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.875610] 0x000001b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.883239] 0x000001c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.890869] 0x000001c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.898498] 0x000001d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.906158] 0x000001d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.913787] 0x000001e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.921417] 0x000001e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.929046] 0x000001f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.936676] 0x000001f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.944305] 0x00000200:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.951934] 0x00000208:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.959564] 0x00000210:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.967193] 0x00000218:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.974853] 0x00000220:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.982482] 0x00000228:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.990112] 0x00000230:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15416.997741] 0x00000238:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.005371] 0x00000240:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.013000] 0x00000248:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.020629] 0x00000250:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.028259] 0x00000258:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.035888] 0x00000260:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.043518] 0x00000268:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.051147] 0x00000270:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.058776] 0x00000278:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.066406] 0x00000280:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.073852] 0x00000288:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.081481] 0x00000290:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.089111] 0x00000298:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.096740] 0x000002a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.104370] 0x000002a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.111999] 0x000002b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.119628] 0x000002b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.127258] 0x000002c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.134887] 0x000002c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.142517] 0x000002d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.150146] 0x000002d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.157775] 0x000002e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.165405] 0x000002e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.173034] 0x000002f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.180664] 0x000002f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.188293] 0x00000300:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.195922] 0x00000308:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.203552] 0x00000310:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.211181] 0x00000318:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.218811] 0x00000320:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.226470] 0x00000328:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.234100] 0x00000330:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.241729] 0x00000338:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.249359] 0x00000340:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.256988] 0x00000348:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.264617] 0x00000350:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.272247] 0x00000358:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.279876] 0x00000360:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.287506] 0x00000368:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.295135] 0x00000370:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.302795] 0x00000378:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.310424] 0x00000380:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.318054] 0x00000388:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.325683] 0x00000390:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.333312] 0x00000398:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.340942] 0x000003a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.348571] 0x000003a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.356201] 0x000003b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.363830] 0x000003b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.371459] 0x000003c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.379089] 0x000003c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.386718] 0x000003d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.394165] 0x000003d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.401794] 0x000003e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.409423] 0x000003e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.417053] 0x000003f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.424682] 0x000003f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.432312] 0x00000400:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.439941] 0x00000408:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.447570] 0x00000410:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.455200] 0x00000418:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.462829] 0x00000420:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.470458] 0x00000428:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.478088] 0x00000430:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.485717] 0x00000438:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.493347] 0x00000440:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.500976] 0x00000448:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.508605] 0x00000450:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.516235] 0x00000458:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.523864] 0x00000460:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.531494] 0x00000468:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.539123] 0x00000470:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.546752] 0x00000478:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.554412] 0x00000480:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.562042] 0x00000488:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.569671] 0x00000490:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.577301] 0x00000498:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.584930] 0x000004a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.592559] 0x000004a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.600189] 0x000004b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.607818] 0x000004b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.615447] 0x000004c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.623107] 0x000004c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.630737] 0x000004d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.638366] 0x000004d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.645996] 0x000004e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.653625] 0x000004e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.661254] 0x000004f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.668884] 0x000004f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.676513] 0x00000500:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.684143] 0x00000508:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.691772] 0x00000510:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.699401] 0x00000518:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.707031] 0x00000520:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.714477] 0x00000528:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.722106] 0x00000530:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.729736] 0x00000538:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.737365] 0x00000540:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.744995] 0x00000548:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.752624] 0x00000550:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.760253] 0x00000558:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.767883] 0x00000560:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.775512] 0x00000568:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.783142] 0x00000570:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.790771] 0x00000578:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.798400] 0x00000580:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.806030] 0x00000588:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.813659] 0x00000590:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.821289] 0x00000598:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.828918] 0x000005a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.836547] 0x000005a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.844177] 0x000005b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.851806] 0x000005b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.859436] 0x000005c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.867065] 0x000005c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.874725] 0x000005d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.882354] 0x000005d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.889984] 0x000005e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.897613] 0x000005e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.905242] 0x000005f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.912872] 0x000005f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.920501] 0x00000600:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.928131] 0x00000608:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.935760] 0x00000610:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.943420] 0x00000618:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.951049] 0x00000620:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.958679] 0x00000628:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.966308] 0x00000630:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.973937] 0x00000638:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.981567] 0x00000640:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.989196] 0x00000648:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15417.996826] 0x00000650:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.004455] 0x00000658:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.012084] 0x00000660:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.019714] 0x00000668:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.027343] 0x00000670:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.034790] 0x00000678:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.042419] 0x00000680:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.050048] 0x00000688:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.057678] 0x00000690:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.065307] 0x00000698:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.072937] 0x000006a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.080566] 0x000006a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.088195] 0x000006b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.095825] 0x000006b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.103454] 0x000006c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.111083] 0x000006c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.118713] 0x000006d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.126342] 0x000006d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.133972] 0x000006e0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.141601] 0x000006e8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.149230] 0x000006f0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.156860] 0x000006f8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.164489] 0x00000700:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.172119] 0x00000708:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.179748] 0x00000710:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.187377] 0x00000718:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.195037] 0x00000720:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.202667] 0x00000728:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.210296] 0x00000730:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.217926] 0x00000738:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.225555] 0x00000740:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.233184] 0x00000748:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.240814] 0x00000750:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.248443] 0x00000758:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.256072] 0x00000760:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.263732] 0x00000768:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.271362] 0x00000770:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.278991] 0x00000778:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.286621] 0x00000780:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.294250] 0x00000788:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.301879] 0x00000790:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.309509] 0x00000798:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.317138] 0x000007a0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.324768] 0x000007a8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.332397] 0x000007b0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.340026] 0x000007b8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.347656] 0x000007c0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.355102] 0x000007c8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.362731] 0x000007d0:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.370361] 0x000007d8:  55 55 55 55 55 55 55 55  ***  aa aa aa aa aa aa aa aa
[15418.377990] 0x000007e0:  aa aa aa aa aa aa aa aa       aa aa aa aa aa aa aa aa
[15418.391937] EB torture: verify failed for 0xAA55AA... pattern
[15418.397949] EB torture: torture finished after 2058 erase cycles
[15418.404235] EB torture: error -22 occurred during torturing
[15418.409851] =========================================================================================

Re: Eraseblocks torture: OneNAND results

[Ricard Wanderlof]

On Fri, 8 Dec 2006, Artem Bityutskiy wrote:

> 1. Although the documented eraseblock lifetime is about 100000, we 
> started getting errors after about 6 million erase cycles. May be 
> because our test is not torturous enough.

Isn't 100000 a figure quoted by the flash manufacturer to be the _minimum_ 
number of erase cycles, a specification which all devices must meet, even 
at extreme operating conditions (e.g. high temperature and extreme supply 
voltages)? That would imply that during more ordinary conditions, a random 
sample of flash chip would very likely be erasable many more times.

Another thing: Why implement it as a kernel module? Speed I would assume?

/Ricard

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

Hello Ricard,

On Fri, 2006-12-08 at 14:43 +0100, Ricard Wanderlof wrote:
> Isn't 100000 a figure quoted by the flash manufacturer to be the _minimum_ 
> number of erase cycles, a specification which all devices must meet, even 
> at extreme operating conditions (e.g. high temperature and extreme supply 
> voltages)? That would imply that during more ordinary conditions, a random 
> sample of flash chip would very likely be erasable many more times.

Probably. Not sure. Need to carefully look to the manual. Do not have it
now handy. But probably you are right. Also HW guys say that such a very
frequent erase may matter.

> Another thing: Why implement it as a kernel module? Speed I would assume?
Yeah, wanted to make it quicker.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Eraseblocks torture: OneNAND results

[Kyungmin Park]

Hi,

For now, there's no special things.

Currently I used the latest torture test file.

I think I need the more time to make a block worn-out.

Thank you,
Kyungmin Park

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

Kyungmin,

On Mon, 2006-12-11 at 08:31 +0000, Kyungmin Park wrote:
> I think I need the more time to make a block worn-out.

FYI: now I see that the tortured eraseblocks do not contain all 0xFFs
after erase which is strange - the driver must have returned an error.
But mtd->erase is totally silent about this. Most probably it is a bug
in the OneNAND driver.

May you please take a look at onenand_wait() from
drivers/mtd/onenand/onenand_base.c in mtd-2.6.git. I see the following
code there:

-----------------------------------------------------------------------
ctrl = this->read_word(this->base + ONENAND_REG_CTRL_STATUS);

if (ctrl & ONENAND_CTRL_ERROR) {
    /* It maybe occur at initial bad block */
    DEBUG(MTD_DEBUG_LEVEL0, "onenand_wait: controller error = 0x%04x\n",
ctrl);
    /* Clear other interrupt bits for preventing ECC error */
    interrupt &= ONENAND_INT_MASTER;
}

...

return 0;
-----------------------------------------------------------------------

AFAIU, this is exactly the place when we should catch erase errors. But
what we do - we only change local 'interrupt' variable and later return
0. So we do not report about errors. This looks suspiciously. May you
comment on this?

Thanks.

Below is the log of the torture test error when we read garbage instead
of 0xFFs just after erase.

[89707.861785] EB torture: read wrong data from EB 20 after
[89707.874450] EB torture: Verify fails on 1 pages, 2048 bytes/8192 bits
[89707.881195] EB torture: The following is a list of all differences between what was read from flash and what was expected
[89707.893920] -----------------------------------------------------------------------------------------
[89707.903442] EB torture: Page 7 has 2048 bytes/8192 bits failing verify, starting at offset 0x3800
[89707.912567] Offset       Read                          Written
[89707.918640] 0x00003800:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.926300] 0x00003808:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.933929] 0x00003810:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.941558] 0x00003818:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.949188] 0x00003820:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.956634] 0x00003828:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.964263] 0x00003830:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.971893] 0x00003838:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.979553] 0x00003840:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.987182] 0x00003848:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89707.994812] 0x00003850:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.002441] 0x00003858:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.010070] 0x00003860:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.017700] 0x00003868:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.025329] 0x00003870:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.032958] 0x00003878:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.040588] 0x00003880:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.048217] 0x00003888:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.055847] 0x00003890:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.063476] 0x00003898:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.071105] 0x000038a0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.078735] 0x000038a8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.086364] 0x000038b0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.093994] 0x000038b8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.101623] 0x000038c0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.109252] 0x000038c8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.116912] 0x000038d0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.124542] 0x000038d8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.132171] 0x000038e0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.139801] 0x000038e8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.147430] 0x000038f0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.155059] 0x000038f8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.162689] 0x00003900:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.170318] 0x00003908:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.177978] 0x00003910:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.185607] 0x00003918:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.193237] 0x00003920:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.200866] 0x00003928:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.208496] 0x00003930:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.216125] 0x00003938:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.223754] 0x00003940:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.231384] 0x00003948:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.239013] 0x00003950:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.246643] 0x00003958:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.254272] 0x00003960:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.261901] 0x00003968:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.269561] 0x00003970:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.277008] 0x00003978:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.284729] 0x00003980:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.292358] 0x00003988:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.299987] 0x00003990:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.307617] 0x00003998:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.315246] 0x000039a0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.322875] 0x000039a8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.330505] 0x000039b0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.338134] 0x000039b8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.345764] 0x000039c0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.353393] 0x000039c8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.361022] 0x000039d0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.368652] 0x000039d8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.376281] 0x000039e0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.383911] 0x000039e8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.391540] 0x000039f0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.399169] 0x000039f8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.406799] 0x00003a00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.414428] 0x00003a08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.422058] 0x00003a10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.429687] 0x00003a18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.437347] 0x00003a20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.444976] 0x00003a28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.452606] 0x00003a30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.460235] 0x00003a38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.467864] 0x00003a40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.475494] 0x00003a48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.483123] 0x00003a50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.490753] 0x00003a58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.498382] 0x00003a60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.506011] 0x00003a68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.513671] 0x00003a70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.521301] 0x00003a78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.528930] 0x00003a80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.536560] 0x00003a88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.544189] 0x00003a90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.551818] 0x00003a98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.559448] 0x00003aa0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.567077] 0x00003aa8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.574707] 0x00003ab0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.582336] 0x00003ab8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.590026] 0x00003ac0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.597656] 0x00003ac8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.605102] 0x00003ad0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.612731] 0x00003ad8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.620361] 0x00003ae0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.627990] 0x00003ae8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.635620] 0x00003af0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.643249] 0x00003af8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.650878] 0x00003b00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.658508] 0x00003b08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.666137] 0x00003b10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.673767] 0x00003b18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.681396] 0x00003b20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.689025] 0x00003b28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.696655] 0x00003b30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.704650] 0x00003b38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.712310] 0x00003b40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.719970] 0x00003b48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.727630] 0x00003b50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.735260] 0x00003b58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.742889] 0x00003b60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.750518] 0x00003b68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.758148] 0x00003b70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.765777] 0x00003b78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.773437] 0x00003b80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.781066] 0x00003b88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.788696] 0x00003b90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.796325] 0x00003b98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.803955] 0x00003ba0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.811584] 0x00003ba8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.819213] 0x00003bb0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.826843] 0x00003bb8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.834472] 0x00003bc0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.842102] 0x00003bc8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.849731] 0x00003bd0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.857391] 0x00003bd8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.865020] 0x00003be0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.872650] 0x00003be8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.880279] 0x00003bf0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.887908] 0x00003bf8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.895538] 0x00003c00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.903167] 0x00003c08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.910797] 0x00003c10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.918426] 0x00003c18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.926055] 0x00003c20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.933685] 0x00003c28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.941131] 0x00003c30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.948760] 0x00003c38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.956390] 0x00003c40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.964019] 0x00003c48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.971649] 0x00003c50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.979278] 0x00003c58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.986907] 0x00003c60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89708.994537] 0x00003c68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.002166] 0x00003c70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.009796] 0x00003c78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.017425] 0x00003c80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.025054] 0x00003c88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.032684] 0x00003c90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.040313] 0x00003c98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.047943] 0x00003ca0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.055572] 0x00003ca8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.063201] 0x00003cb0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.070831] 0x00003cb8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.078491] 0x00003cc0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.086090] 0x00003cc8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.093750] 0x00003cd0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.101379] 0x00003cd8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.109008] 0x00003ce0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.116638] 0x00003ce8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.124267] 0x00003cf0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.131896] 0x00003cf8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.139526] 0x00003d00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.147155] 0x00003d08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.154785] 0x00003d10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.162414] 0x00003d18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.170043] 0x00003d20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.177703] 0x00003d28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.185333] 0x00003d30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.192962] 0x00003d38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.200592] 0x00003d40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.208221] 0x00003d48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.215850] 0x00003d50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.223480] 0x00003d58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.231109] 0x00003d60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.238739] 0x00003d68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.246368] 0x00003d70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.253997] 0x00003d78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.261444] 0x00003d80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.269073] 0x00003d88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.276733] 0x00003d90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.284545] 0x00003d98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.292205] 0x00003da0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.299865] 0x00003da8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.307525] 0x00003db0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.315155] 0x00003db8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.322784] 0x00003dc0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.330413] 0x00003dc8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.338043] 0x00003dd0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.345672] 0x00003dd8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.353302] 0x00003de0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.360931] 0x00003de8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.368560] 0x00003df0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.376190] 0x00003df8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.383819] 0x00003e00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.391448] 0x00003e08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.399078] 0x00003e10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.406707] 0x00003e18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.414337] 0x00003e20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.421966] 0x00003e28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.429626] 0x00003e30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.437255] 0x00003e38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.444885] 0x00003e40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.452514] 0x00003e48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.460144] 0x00003e50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.467773] 0x00003e58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.475433] 0x00003e60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.483032] 0x00003e68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.490661] 0x00003e70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.498291] 0x00003e78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.505920] 0x00003e80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.513580] 0x00003e88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.521209] 0x00003e90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.528839] 0x00003e98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.536468] 0x00003ea0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.544097] 0x00003ea8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.551727] 0x00003eb0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.559356] 0x00003eb8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.566986] 0x00003ec0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.574615] 0x00003ec8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.582244] 0x00003ed0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.589874] 0x00003ed8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.597320] 0x00003ee0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.604949] 0x00003ee8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.612579] 0x00003ef0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.620208] 0x00003ef8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.627838] 0x00003f00:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.635467] 0x00003f08:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.643096] 0x00003f10:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.650726] 0x00003f18:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.658355] 0x00003f20:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.665985] 0x00003f28:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.673614] 0x00003f30:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.681243] 0x00003f38:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.688873] 0x00003f40:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.696502] 0x00003f48:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.704132] 0x00003f50:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.711761] 0x00003f58:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.719390] 0x00003f60:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.727050] 0x00003f68:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.734710] 0x00003f70:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.742340] 0x00003f78:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.750000] 0x00003f80:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.757629] 0x00003f88:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.765258] 0x00003f90:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.772888] 0x00003f98:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.780548] 0x00003fa0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.788208] 0x00003fa8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.795837] 0x00003fb0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.803466] 0x00003fb8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.811096] 0x00003fc0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.818725] 0x00003fc8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.826354] 0x00003fd0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.834014] 0x00003fd8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.841644] 0x00003fe0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.849273] 0x00003fe8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.856903] 0x00003ff0:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.864532] 0x00003ff8:  aa aa aa aa aa aa aa aa  ***  ff ff ff ff ff ff ff ff
[89709.872161] 0x00004000:  ff ff ff ff ff ff ff ff       ff ff ff ff ff ff ff ff
[89709.885223] EB torture: verify failed for 0xFF... pattern
[89709.890899] EB torture: torture finished after 10796 erase cycles
[89709.897247] EB torture: error -22 occurred during torturing
[89709.903045] =========================================================================================

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Eraseblocks torture: OneNAND results

[Kyungmin Park]

Hi, Artem

> FYI: now I see that the tortured eraseblocks do not contain all 0xFFs
> after erase which is strange - the driver must have returned an error.
> But mtd->erase is totally silent about this. Most probably it is a bug
> in the OneNAND driver.
>
> May you please take a look at onenand_wait() from
> drivers/mtd/onenand/onenand_base.c in mtd-2.6.git. I see the following
> code there:
>
> -----------------------------------------------------------------------
> ctrl = this->read_word(this->base + ONENAND_REG_CTRL_STATUS);
> 
> if (ctrl & ONENAND_CTRL_ERROR) {
>     /* It maybe occur at initial bad block */
>     DEBUG(MTD_DEBUG_LEVEL0, "onenand_wait: controller error = 0x%04x\n",
> ctrl);
>     /* Clear other interrupt bits for preventing ECC error */
>     interrupt &= ONENAND_INT_MASTER;
> }

> AFAIU, this is exactly the place when we should catch erase errors. But
> what we do - we only change local 'interrupt' variable and later return
> 0. So we do not report about errors. This looks suspiciously. May you
> comment on this?

Yes, you're right. onenand_wait has a bug. It don't report the any error. it's my falut.
And also it don't check the locked block error.

The below patch fix the onenand_wait bug. (This is temporary one. I also try to fix another things)

please test this one.
(You may have some parts already. please ignore it)

In my opition, if the block goes worn-out. it occurs as following.
First, 2-bit ecc read error occurs. (bit error)
Second, Write failed. (page error)
Finally, Erase failed. (block error)

Thank you,
Kyungmin Park

P.S., In target environment, it still can't report any error. I'm also surpise with the OneNAND which has good erase guarantee.


--

Index: drivers/mtd/onenand/onenand_base.c
===================================================================
RCS file: /cvsroot/linux-2.6.18-omap/drivers/mtd/onenand/onenand_base.c,v
retrieving revision 1.2
diff -u -p -r1.2 onenand_base.c
--- drivers/mtd/onenand/onenand_base.c	12 Oct 2006 06:59:27 -0000	1.2
+++ drivers/mtd/onenand/onenand_base.c	15 Dec 2006 04:36:02 -0000
@@ -316,22 +316,20 @@ static int onenand_wait(struct mtd_info 
 	ctrl = this->read_word(this->base + ONENAND_REG_CTRL_STATUS);
 
 	if (ctrl & ONENAND_CTRL_ERROR) {
-		/* It maybe occur at initial bad block */
 		DEBUG(MTD_DEBUG_LEVEL0, "onenand_wait: controller error = 0x%04x\n", ctrl);
-		/* Clear other interrupt bits for preventing ECC error */
-		interrupt &= ONENAND_INT_MASTER;
-	}
-
-	if (ctrl & ONENAND_CTRL_LOCK) {
-		DEBUG(MTD_DEBUG_LEVEL0, "onenand_wait: it's locked error = 0x%04x\n", ctrl);
-		return -EACCES;
+		if (ctrl & ONENAND_CTRL_LOCK)
+			DEBUG(MTD_DEBUG_LEVEL0, "onenand_erase: Device is write protected!!!\n");
+		return ctrl;
 	}
 
 	if (interrupt & ONENAND_INT_READ) {
 		ecc = this->read_word(this->base + ONENAND_REG_ECC_STATUS);
-		if (ecc & ONENAND_ECC_2BIT_ALL) {
+		if (ecc) {
 			DEBUG(MTD_DEBUG_LEVEL0, "onenand_wait: ECC error = 0x%04x\n", ecc);
-			return -EBADMSG;
+			if (ecc & ONENAND_ECC_2BIT_ALL)
+				mtd->ecc_stats.failed++;
+			else if (ecc & ONENAND_ECC_1BIT_ALL)
+				mtd->ecc_stats.corrected++;
 		}
 	}
 
@@ -608,6 +606,7 @@ static int onenand_read(struct mtd_info 
 	size_t *retlen, u_char *buf)
 {
 	struct onenand_chip *this = mtd->priv;
+	struct mtd_ecc_stats stats;
 	int read = 0, column;
 	int thislen;
 	int ret = 0;
@@ -626,6 +625,7 @@ static int onenand_read(struct mtd_info 
 
 	/* TODO handling oob */
 
+	stats = mtd->ecc_stats;
 	while (read < len) {
 		thislen = min_t(int, mtd->writesize, len - read);
 
@@ -643,16 +643,16 @@ static int onenand_read(struct mtd_info 
 
 		this->read_bufferram(mtd, ONENAND_DATARAM, buf, column, thislen);
 
-		read += thislen;
-
-		if (read == len)
-			break;
-
 		if (ret) {
 			DEBUG(MTD_DEBUG_LEVEL0, "onenand_read: read failed = %d\n", ret);
 			goto out;
 		}
 
+		read += thislen;
+
+		if (read == len)
+			break;
+
 		from += thislen;
 		buf += thislen;
 	}
@@ -667,7 +667,10 @@ out:
 	 * retlen == desired len and result == -EBADMSG
 	 */
 	*retlen = read;
-	return ret;
+	if (mtd->ecc_stats.failed - stats.failed)
+		return -EBADMSG;
+
+	return mtd->ecc_stats.corrected - stats.corrected ? -EUCLEAN : 0;
 }
 
 /**
@@ -716,15 +719,16 @@ int onenand_do_read_oob(struct mtd_info 
 
 		this->read_bufferram(mtd, ONENAND_SPARERAM, buf, column, thislen);
 
+		if (ret) {
+			DEBUG(MTD_DEBUG_LEVEL0, "onenand_read_oob: read failed = 0x%x\n", ret);
+			goto out;
+		}
+
 		read += thislen;
 
 		if (read == len)
 			break;
 
-		if (ret) {
-			DEBUG(MTD_DEBUG_LEVEL0, "onenand_read_oob: read failed = %d\n", ret);
-			goto out;
-		}
 
 		buf += thislen;
 
@@ -1083,10 +1087,7 @@ static int onenand_erase(struct mtd_info
 		ret = this->wait(mtd, FL_ERASING);
 		/* Check, if it is write protected */
 		if (ret) {
-			if (ret == -EPERM)
-				DEBUG(MTD_DEBUG_LEVEL0, "onenand_erase: Device is write protected!!!\n");
-			else
-				DEBUG(MTD_DEBUG_LEVEL0, "onenand_erase: Failed erase, block %d\n", (unsigned) (addr >> this->erase_shift));
+			DEBUG(MTD_DEBUG_LEVEL0, "onenand_erase: Failed erase, block %d\n", (unsigned) (addr >> this->erase_shift));
 			instr->state = MTD_ERASE_FAILED;
 			instr->fail_addr = addr;
 			goto erase_exit;
Index: drivers/mtd/onenand/onenand_bbt.c
===================================================================
RCS file: /cvsroot/linux-2.6.18-omap/drivers/mtd/onenand/onenand_bbt.c,v
retrieving revision 1.1.1.1
diff -u -p -r1.1.1.1 onenand_bbt.c
--- drivers/mtd/onenand/onenand_bbt.c	12 Oct 2006 05:49:24 -0000	1.1.1.1
+++ drivers/mtd/onenand/onenand_bbt.c	15 Dec 2006 04:36:02 -0000
@@ -93,7 +93,8 @@ static int create_bbt(struct mtd_info *m
 			ret = onenand_do_read_oob(mtd, from + j * mtd->writesize + bd->offs,
 						  readlen, &retlen, &buf[0]);
 
-			if (ret)
+			/* Handle initial bad block */
+			if (ret && !(ret & ONENAND_CTRL_LOAD))
 				return ret;
 
 			if (check_short_pattern(&buf[j * scanlen], scanlen, mtd->writesize, bd)) {

Re: Eraseblocks torture: OneNAND results

[Enrico Migliore]

>> FYI: now I see that the tortured eraseblocks do not contain all 0xFFs
>> after erase which is strange - the driver must have returned an error.
>> But mtd->erase is totally silent about this. Most probably it is a bug
>> in the OneNAND driver.
>>
>> May you please take a look at onenand_wait() from
>> drivers/mtd/onenand/onenand_base.c in mtd-2.6.git. I see the following
>> code there:
>>
>>     
Hi,

my name is Enrico, I'm new to this list and I've been reading the 
messages since the end of November.

I've recently read  the Micron's "TN-29-16 Boot from NAND..."  technical 
note and got puzzled by the following:

"Avoid excessive reads to the area of the NAND Flash where code is 
stored. When
repeated accesses are required, the code should be copied to the DRAM. 
This minimizes
the probability of read-disturb errors in the NAND Flash device."

Does that mean that excessive readings might compromise the integrity of 
NAND cells?

Enrico

Re: Eraseblocks torture: OneNAND results

[Ricard Wanderlof]

On Fri, 15 Dec 2006, Enrico Migliore wrote:

> I've recently read  the Micron's "TN-29-16 Boot from NAND..."  technical
> note and got puzzled by the following:
>
> "Avoid excessive reads to the area of the NAND Flash where code is
> stored. When
> repeated accesses are required, the code should be copied to the DRAM.
> This minimizes
> the probability of read-disturb errors in the NAND Flash device."
>
> Does that mean that excessive readings might compromise the integrity of
> NAND cells?

That's the way I would interpret it too. (BTW, the paragraph wasn't there 
in the first edition of the document, rev.A from March this year...).

Although I'm also somewhat puzzled at the conclusion. I always thought 
read errors could occur at any time, the probability of which could be 
increased by reading or even more writing to the device. But not that it 
was localized to the area which you are reading. Although it makes sense I 
guess; accessing a bit cell requires some activity in the vicinity of the 
cell in order to read the bit, which I guess would increase the 
probability of failure.

/Ricard
--
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Eraseblocks torture: OneNAND results

[Jarkko Lavinen]

Hi Kyungmin

I have tried your patch on an Omap 2420 base test board with OneNand
and it seems to work.

I had previously run an earlier version of torture test and had some
worn out erase blocks available. When I try to erase them I see often
controller error occuring and it is caught and returned correctly.

But I have also encounted cases where there is no erase error but yet
erase verify fails. I then tried a retry aften the failed verify and 
on the second read the erase block is blanj, all FF, as it should.

Jarkko Lavinen

Eraseblocks torture: OneNAND results

[Kyungmin Park]

Hi Jarkko Lavinen

> I have tried your patch on an Omap 2420 base test board with OneNand
> and it seems to work.

Oh it's good news, In our environment. it's hard to produce the problem.
I erase it more than 1,000k (2 weeks). but it's still working.

> I had previously run an earlier version of torture test and had some
> worn out erase blocks available. When I try to erase them I see often
> controller error occuring and it is caught and returned correctly.

I commited the patch in OneNAND MTD git. and it will be merged into mtd.

> But I have also encounted cases where there is no erase error but yet
> erase verify fails. I then tried a retry aften the failed verify and 
> on the second read the erase block is blank, all FF, as it should.

As you know. If the block is worn-out. we don't sure its behavior.

Actually I don't have any idea. How can we handle this one?

Thank you,
Kyungmin Park

Re: Eraseblocks torture: OneNAND results

[Artem Bityutskiy]

On Fri, 2006-12-22 at 07:58 +0000, Kyungmin Park wrote:
> Oh it's good news, In our environment. it's hard to produce the problem.
> I erase it more than 1,000k (2 weeks). but it's still working.

Disable checking and run it for a million cycles. Then enable checking
again.

modprobe torture.ko check=0 cycles_count=1000000

after this the eraseblocks will be more worn-out. Then you could run it
with checking.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)