Emulated write failures cause block marking as bad

Emulated write failures cause block marking as bad

[Marek Skuczynski]

Hello,
   Today, I was trying to use debug option MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES.
   What I have found is that occasionally,  the emulated write error
happens while
   executing sync_erase() function. In result of this, the part that
is calling sync_erase()
   mark the block as bad (see log below).
   After a few minutes I had many bad blocks. Using this option to
volume update test
   makes no sense.

   I am using kernel 2.6.23 with updated UBI  from 2.6.29.
   Have you experienced this problem already ? if so, is this has been fixed ?

   Regards,
     Marek

   static int sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e,
                      int torture)
   {
        /* ... */
        dbg_wl("erase PEB %d, old EC %llu", e->pnum, ec);

        /* ... */

        err = ubi_io_sync_erase(ubi, e->pnum, torture);
        if (err < 0)
                goto out_free;

        /* ... */

        dbg_wl("erased PEB %d, new EC %llu", e->pnum, ec);

        /* ... */

        err = ubi_io_write_ec_hdr(ubi, e->pnum, ec_hdr);
        if (err) {
                goto out_free;
        }

        /* ... */

out_free:
        /* ... */
        return err;
}

[---CUT---]

UBI DBG (pid 1837): schedule_erase: schedule erasure of PEB 721, EC 3517, tortu0
UBI DBG (pid 458): erase_worker: erase PEB 721 EC 3517
UBI DBG (pid 458): sync_erase: erase PEB 721, old EC 3517
UBI DBG (pid 458): do_sync_erase: erase PEB 721
UBI DBG (pid 458): sync_erase: erased PEB 721, new EC 3518
UBI DBG (pid 458): ubi_io_write_ec_hdr: write EC header to PEB 721
UBI DBG (pid 458): ubi_io_write: write 512 bytes to PEB 721:0
UBI error: ubi_io_write: cannot write 512 bytes to PEB 721:0 (emulated)
Stack: (0x8ca59ec8 to 0x8ca5a000)
9ec0:                   809751ea 00000000 fffffffb 80816cd8 80975950 8ca58000
9ee0: 00000001 80816cd8 000002d1 80b37800 8cdb4600 80b37c7c 00000200 80976cac
9f00: 8ca30770 00000000 00000dbe be0d0000 00000dbd 80a0eb08 ffffff0f 8751ada0
9f20: 00000000 8ca4ca60 80b37800 000002d1 8cdb4600 80976508 00000000 00000000
9f40: 80b37800 80b37c04 80b37800 <7>UBI DBG (pid 1837): ubi_wl_get_peb: PEB 6264
8ca4ca60 8ca58000 809766f6 80b37c3c ffffff0f
9f60: 8ca58000 00000000 80b37c40 80b37c2c 8082819c 00000000 00000000 00000000
9f80: 00000000 80976618 80b37800 <7>UBI DBG (pid 1837): ubi_io_write_vid_hdr: w6
8ca58000 80803664 00000000 00000000 00000000
9fa0: 00000000 00000000 00000000 00000000 8ccbbdf0 80828168 00000000 00000000
9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 <7>UBI DBG8
8ca59fa0
9fe0: 80803660 00000000 40008000 00000000 00000000 00000000 00000000 00000160

Call trace:
[<80816cd8>] printk+0x0/0x28
[<80975950>] ubi_io_write_ec_hdr+0xe0/0x114
[<80816cd8>] printk+0x0/0x28
UBI DBG (pid 1837): ubi_io_write: write 22528 bytes to PEB 626:4096
[<80976cac>] erase_worker+0x164/0x500
[<80a0eb08>] __switch_to_begin+0x24/0x36
[<80976508>] do_work+0x9c/0xf8
[<809766f6>] ubi_thread+0xde/0x1a4
[<8082819c>] kthread+0x34/0x64
[<80976618>] ubi_thread+0x0/0x1a4
[<80803664>] kernel_thread_helper+0x4/0x10
[<80828168>] kthread+0x0/0x64
[<80803660>] kernel_thread_helper+0x0/0x10

UBI error: erase_worker: failed to erase PEB 721, error -5
UBI: reserve more 1 PEBs
UBI DBG (pid 1837): ubi_wl_put_peb: PEB 680
UBI: mark PEB 721 as bad
Bad block table written to 0x07fe0000, version 0x22
Bad block table written to 0x07fc0000, version 0x22
[---CUT---]

Re: Emulated write failures cause block marking as bad

[Artem Bityutskiy]

Hi,

On Thu, 2010-02-04 at 18:02 +0100, Marek Skuczynski wrote:
> Hello,
>    Today, I was trying to use debug option MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES.
>    What I have found is that occasionally,  the emulated write error
> happens while
>    executing sync_erase() function. In result of this, the part that
> is calling sync_erase()
>    mark the block as bad (see log below).
>    After a few minutes I had many bad blocks.

This is true.

>  Using this option to
> volume update test
>    makes no sense.

This is too strong statement. It does make sens - you verify that the
error handling functionality works.

You can say that this option is not good enough for you, this will be
more fair statement.

>    I am using kernel 2.6.23 with updated UBI  from 2.6.29.
>    Have you experienced this problem already ? if so, is this has been fixed ?

Yes, I saw it. This is purely a debugging feature, and it was enough for
me.

You can easilly develop it a bit more, and make it stop returning erase
errors when the amount of bad eraseblocks has reached some level.

Just amend ubi_dbg_is_erase_failure()

You might want to do the same for 'ubi_dbg_is_write_failure()', for the
same reasons, basically.

HTH.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: Emulated write failures cause block marking as bad

[Marek Skuczynski]

Hello Artem,
> Using this option to
>> volume update test
>>    makes no sense.
>>     
> This is too strong statement. It does make sens - you verify that the
> error handling functionality works.
>
> You can say that this option is not good enough for you, this will be
> more fair statement.
>   

Okay. You are right. This option is not implemented as I expect.

>>    I am using kernel 2.6.23 with updated UBI  from 2.6.29.
>>    Have you experienced this problem already ? if so, is this has been fixed ?
>>     
> Yes, I saw it. This is purely a debugging feature, and it was enough for
> me.
>
> You can easilly develop it a bit more, and make it stop returning erase
> errors when the amount of bad eraseblocks has reached some level.
>
> Just amend ubi_dbg_is_erase_failure()
>
> You might want to do the same for 'ubi_dbg_is_write_failure()', for the
> same reasons, basically.
>
>   
Yes, of course I can modify the number of errors that will be reported.
However, I would expect that the "emulated write errors" will be
reported out of sync_erase() function,
because most of these errors are reported during writing EC header (in
my case) just after flash block
is erased. As result of this, using the "emulated write errors" I have
almost the same behavior that
I would get with "emulated erase errors" enabled.

Okay, maybe my case is isolated because of tested system configuration,
I will try manage this case myself.

Regards,
  Marek

Re: Emulated write failures cause block marking as bad

[Marek Skuczynski]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

> Okay, maybe my case is isolated because of tested system configuration,
> I will try manage this case myself.
>
I have prepared a patch, maybe someone will find it useful.

[-- Attachment #2: safe_erase.patch --]
[-- Type: text/x-patch, Size: 4305 bytes --]

diff --git a/drivers/mtd/ubi/Kconfig.debug b/drivers/mtd/ubi/Kconfig.debug
index 1e2ee22..fa1a92c 100644
--- a/drivers/mtd/ubi/Kconfig.debug
+++ b/drivers/mtd/ubi/Kconfig.debug
@@ -59,6 +59,14 @@ config MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES
 	  This option emulates write failures with probability 1/100. Useful for
 	  debugging and testing how UBI handlines errors.
 
+config MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES_SAFE_ERASE
+	bool "Prevent write failure during the sync erase"
+	depends on MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES
+	default n
+	help
+	  This option prevents write failures during the sync erase that may
+	  cause marking the block as bad.
+
 config MTD_UBI_DEBUG_EMULATE_ERASE_FAILURES
 	bool "Emulate flash erase failures"
 	depends on MTD_UBI_DEBUG
diff --git a/drivers/mtd/ubi/debug.c b/drivers/mtd/ubi/debug.c
index c949cb4..8ff0d6a 100644
--- a/drivers/mtd/ubi/debug.c
+++ b/drivers/mtd/ubi/debug.c
@@ -194,6 +194,23 @@ void ubi_dbg_dump_mkvol_req(const struct ubi_mkvol_req *req)
 	printk(KERN_DEBUG "\t1st 16 characters of name: %s\n", nm);
 }
 
+#ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES
+/**
+ * ubi_dbg_is_write_failure - if it is time to emulate a write failure.
+ *
+ * Returns non-zero if a write failure should be emulated, otherwise returns
+ * zero.
+ */
+int ubi_dbg_is_write_failure(struct ubi_device *ubi)
+{
+#ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES_SAFE_ERASE
+	return (!ubi->write_failure_masked && !(random32() % 500));
+#else
+	return !(random32() % 500);
+#endif
+}
+#endif
+
 #endif /* CONFIG_MTD_UBI_DEBUG */
 
 /*
diff --git a/drivers/mtd/ubi/debug.h b/drivers/mtd/ubi/debug.h
index fa4cffe..ff77a26 100644
--- a/drivers/mtd/ubi/debug.h
+++ b/drivers/mtd/ubi/debug.h
@@ -114,18 +114,9 @@ static inline int ubi_dbg_is_bitflip(void)
 #endif
 
 #ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES
-/**
- * ubi_dbg_is_write_failure - if it is time to emulate a write failure.
- *
- * Returns non-zero if a write failure should be emulated, otherwise returns
- * zero.
- */
-static inline int ubi_dbg_is_write_failure(void)
-{
-	return !(random32() % 500);
-}
+int ubi_dbg_is_write_failure(struct ubi_device *ubi);
 #else
-#define ubi_dbg_is_write_failure() 0
+#define ubi_dbg_is_write_failure(ubi) 0
 #endif
 
 #ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_ERASE_FAILURES
@@ -165,7 +156,7 @@ static inline int ubi_dbg_is_erase_failure(void)
 #define UBI_IO_DEBUG               0
 #define DBG_DISABLE_BGT            0
 #define ubi_dbg_is_bitflip()       0
-#define ubi_dbg_is_write_failure() 0
+#define ubi_dbg_is_write_failure(ubi) 0
 #define ubi_dbg_is_erase_failure() 0
 
 #endif /* !CONFIG_MTD_UBI_DEBUG */
diff --git a/drivers/mtd/ubi/io.c b/drivers/mtd/ubi/io.c
index 2904991..c164299 100644
--- a/drivers/mtd/ubi/io.c
+++ b/drivers/mtd/ubi/io.c
@@ -259,7 +259,7 @@ int ubi_io_write(struct ubi_device *ubi, const void *buf, int pnum, int offset,
 			return err > 0 ? -EINVAL : err;
 	}
 
-	if (ubi_dbg_is_write_failure()) {
+	if (ubi_dbg_is_write_failure(ubi)) {
 		dbg_err("cannot write %d bytes to PEB %d:%d "
 			"(emulated)", len, pnum, offset);
 		ubi_dbg_dump_stack();
diff --git a/drivers/mtd/ubi/ubi.h b/drivers/mtd/ubi/ubi.h
index d8f5915..1194549 100644
--- a/drivers/mtd/ubi/ubi.h
+++ b/drivers/mtd/ubi/ubi.h
@@ -438,6 +438,9 @@ struct ubi_device {
 #ifdef CONFIG_MTD_UBI_DEBUG
 	void *dbg_peb_buf;
 	struct mutex dbg_buf_mutex;
+#ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES_SAFE_ERASE
+	int  write_failure_masked;
+#endif
 #endif
 };
 
diff --git a/drivers/mtd/ubi/wl.c b/drivers/mtd/ubi/wl.c
index dc38fc5..6799c25 100644
--- a/drivers/mtd/ubi/wl.c
+++ b/drivers/mtd/ubi/wl.c
@@ -615,10 +615,15 @@ static int sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e,
 
 	ec_hdr->ec = cpu_to_be64(ec);
 
+#ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES_SAFE_ERASE
+	ubi->write_failure_masked = 1;
+#endif
+
 	err = ubi_io_write_ec_hdr(ubi, e->pnum, ec_hdr);
 	if (err)
 		goto out_free;
 
+
 	e->ec = ec;
 	spin_lock(&ubi->wl_lock);
 	if (e->ec > ubi->max_ec)
@@ -626,6 +631,9 @@ static int sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e,
 	spin_unlock(&ubi->wl_lock);
 
 out_free:
+#ifdef CONFIG_MTD_UBI_DEBUG_EMULATE_WRITE_FAILURES_SAFE_ERASE
+	ubi->write_failure_masked = 0;
+#endif
 	kfree(ec_hdr);
 	return err;
 }

Re: Emulated write failures cause block marking as bad

[Artem Bityutskiy]

On Fri, 2010-02-05 at 10:23 +0100, Marek Skuczynski wrote:
> > Okay, maybe my case is isolated because of tested system configuration,
> > I will try manage this case myself.
> >
> I have prepared a patch, maybe someone will find it useful.

Something like this is useful, but I disapprove of adding another
configuration option. Instead, I think you should just improve the
current debugging code.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)