Input/Output errors

Input/Output errors

[Kenny MacDermid]

I'm running btrfs on DM-Crypt Luks running on LVM.

Occasionally I get files that are unreadable for some period of time.
Attempting to read from them results in an 

Input/output error

Sometimes they'll come back on their own, and sometimes a scrub seems to
help, but sometimes I just have to delete them.

Nothing shows up in dmesg when these occur, and I can't predict which
files it will be, or what causes it.

It's currently happening running 4.4.1-2-ARCH, but I've seen the same
thing for many previous kernel versions.

Does anyone have any suggestions?

Re: Input/Output errors

[Marc MERLIN]

On Tue, Feb 23, 2016 at 08:40:46PM -0400, Kenny MacDermid wrote:
> I'm running btrfs on DM-Crypt Luks running on LVM.
> 
> Occasionally I get files that are unreadable for some period of time.
> Attempting to read from them results in an 
> 
> Input/output error
> 
> Sometimes they'll come back on their own, and sometimes a scrub seems to
> help, but sometimes I just have to delete them.
> 
> Nothing shows up in dmesg when these occur, and I can't predict which
> files it will be, or what causes it.
> 
> It's currently happening running 4.4.1-2-ARCH, but I've seen the same
> thing for many previous kernel versions.
> 
> Does anyone have any suggestions?

That's weird to say the least, you should at least get *something* in
dmesg.
And you are getting other error messages and btrfs kernel messages in
your logs?

When whatever app you have that's trying to read them fails, I assume
they also fail with cat or less?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Input/Output errors

[Kenny MacDermid]

On Tue, Feb 23, 2016 at 05:56:58PM -0800, Marc MERLIN wrote:
> On Tue, Feb 23, 2016 at 08:40:46PM -0400, Kenny MacDermid wrote:
> > I'm running btrfs on DM-Crypt Luks running on LVM.
> > 
> > Occasionally I get files that are unreadable for some period of time.
> > Attempting to read from them results in an 
> > 
> > Input/output error
> > 
> > Sometimes they'll come back on their own, and sometimes a scrub seems to
> > help, but sometimes I just have to delete them.
> > 
> > Nothing shows up in dmesg when these occur, and I can't predict which
> > files it will be, or what causes it.
> > 
> > It's currently happening running 4.4.1-2-ARCH, but I've seen the same
> > thing for many previous kernel versions.
> > 
> > Does anyone have any suggestions?
> 
> That's weird to say the least, you should at least get *something* in
> dmesg.
> And you are getting other error messages and btrfs kernel messages in
> your logs?
> 
> When whatever app you have that's trying to read them fails, I assume
> they also fail with cat or less?

I am getting other, normal btrfs messages. I'll include them at the end.
When it happens I get nothing at all in dmesg/logs.

And yes, cat will fail. I can move the file to another name though,
which I often do to get it out of the way.

They're mounted with:

rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

issue_discards=1 is in lvm.conf, and discard in /etc/crypttab. (I'm now
reading that I probably shouldn't have it in fstab though and just run
fstrim.)

I don't know if this is related yet at all, but it /seems/ more likely
to happen after I delete a bunch of data. That could be a red herring
though.

When the file becomes readable again it's perfectly fine. Scrub never
finds any errors.

$ dmesg | grep -i btrfs
[   11.837137] Btrfs loaded
[   11.837403] BTRFS: device label root devid 1 transid 508963 /dev/dm-3
[   11.856203] BTRFS info (device dm-3): disk space caching is enabled
[   11.879366] BTRFS: detected SSD devices, enabling SSD mode
[   12.160267] BTRFS info (device dm-3): turning on discard
[   12.160272] BTRFS info (device dm-3): enabling auto defrag
[   12.160275] BTRFS info (device dm-3): enabling inode map caching
[   12.160277] BTRFS info (device dm-3): disk space caching is enabled
[   14.979093] BTRFS: device label home devid 1 transid 705779 /dev/dm-5
[   15.013978] BTRFS info (device dm-5): use ssd allocation scheme
[   15.013983] BTRFS info (device dm-5): turning on discard
[   15.013987] BTRFS info (device dm-5): enabling auto defrag
[   15.013989] BTRFS info (device dm-5): enabling inode map caching
[   15.013991] BTRFS info (device dm-5): disk space caching is enabled
[   15.100779] BTRFS error (device dm-5): could not find root 8
[   15.102889] BTRFS error (device dm-5): could not find root 8
[   15.105833] BTRFS error (device dm-3): could not find root 8
[   15.105838] BTRFS error (device dm-3): could not find root 8

Re: Input/Output errors

[Chris Murphy]

On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
<kenny.macdermid@gmail.com> wrote:

>
> rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

It sounds like an ssd trim bug. I'd check the firmware for updates. If
it's up to date, I'd drop discard mount option first and try to
reproduce. Or just use the default mount options and try to reproduce,
then add them back one at a time until you discover the culprit.

Also, how many files/directories are there? inode_cache isn't
recommended for most use cases. And space_cache is the default so it
doesn't need to be listed.



-- 
Chris Murphy

Re: Input/Output errors

[Szalma László]

2016-02-24 05:37 keltezéssel, Chris Murphy írta:
> On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
> <kenny.macdermid@gmail.com> wrote:
>
>> rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
> It sounds like an ssd trim bug. I'd check the firmware for updates. If
> it's up to date, I'd drop discard mount option first and try to
> reproduce. Or just use the default mount options and try to reproduce,
> then add them back one at a time until you discover the culprit.
>
> Also, how many files/directories are there? inode_cache isn't
> recommended for most use cases. And space_cache is the default so it
> doesn't need to be listed.
>
>
>

As i wrote to the list a few weeks ago, this problem seems to be the 
same I have.

The difference:

- i use mount options: noatime,compress,nossd
- I don't use dm-crypt, but these machines are Xen pvms (there is a 
virtualization layer between btrfs and lvm)

The same:

- io error without any error or message in the dmesg.
- umount / mount always fixes the problem (for some time)

More info:

- these files are usually smalls (mysql myisam files, 10-20-50 kbyte 
size, without heave fragmentation)
- defrag don't help
- scrub always works (no errors) but not fix the errors
- no hw or hw read error on the block device (in any layer)
- get this problem with 4.4.1 kernel too (seems to be somewhat less 
frequent than before, but the problem happened with 3.18 and on)
- echo 3 > /proc/sys/vm/drop_caches    sometimes fixes the problem, but 
not every time
- the problem is happening rarely, sometimes there are days without error
- the problem is not for specific hardware or virtual machine

I can try any debug option or patch if needed.

László Szalma

Re: Input/Output errors

[Kenny MacDermid]

On Wed, Feb 24, 2016 at 09:58:08AM +0100, Szalma László wrote:
> - echo 3 > /proc/sys/vm/drop_caches    sometimes fixes the problem, but not
> every time

Interesting. I just had the issue happen with one of my files, tried
this, and it's now readable again.

Re: Input/Output errors

[Kenny MacDermid]

On Tue, Feb 23, 2016 at 09:37:03PM -0700, Chris Murphy wrote:
> On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
> <kenny.macdermid@gmail.com> wrote:
> 
> >
> > rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
> 
> It sounds like an ssd trim bug. I'd check the firmware for updates. If
> it's up to date, I'd drop discard mount option first and try to
> reproduce. Or just use the default mount options and try to reproduce,
> then add them back one at a time until you discover the culprit.

Thanks Chris,

The disk is a SAMSUNG MZMTE512HMHP-000L1 running f/w: EXT42L0Q. I didn't
find any updates in my searches.

I did drop the discard option, and the inode_cache and space_cache ones
as well. I'm not sure I originally got them from.

Even if it's a disk firmware bug, doesn't it seem like some layer should
report something about it? Is there any extra logging I should enable in
case it happens again?

As I don't have a way to reliable reproduce the error, I'll have to wait
and see if this helps.

input/output errors

[Dmitry Skorinko]

Hi all..
It's my attempt number...

kernel 2.6.8.1

#modprobe diskonchip

#dmesg
--skip--
DiskOnChip found at 0xe0000
Detected 2 chips per floor.
NAND device: Manufacturer ID: 0xec, Chip ID: 0x79 (Samsung NAND 128MiB 3,3V 
8-bit)
2 NAND chips detected
Bad block table found at page 524256, version 0x55
Bad block table not found for chip 0
nand_read_bbt: Reserved block at 0x00000000
nand_read_bbt: Bad block at 0x00080000
nand_read_bbt: Bad block at 0x000c4000
nand_read_bbt: Bad block at 0x000d4000
nand_read_bbt: Bad block at 0x08080000
nand_read_bbt: Bad block at 0x08124000
nand_read_bbt: Bad block at 0x08134000
nand_read_bbt: Bad block at 0x09600000
nand_read_bbt: Reserved block at 0x0fffc000
ECC error scanning DOC at 0x0
DiskOnChip BNAND Media Header not found.
Found alias of DOC at 0xe0000 to 0xe2000
Found alias of DOC at 0xe0000 to 0xe4000
Found alias of DOC at 0xe0000 to 0xe6000

# lsmod
Module                  Size  Used by
mtdchar                 6784  0
diskonchip             12184  0
nand                   26500  1 diskonchip
nand_ids                4224  1 nand
nand_ecc                2816  1 nand
docecc                  4608  1 diskonchip
mtdpart                 9600  2 diskonchip,nand
mtdcore                 7496  6 mtdchar,diskonchip,nand,mtdpart
--skip--

#cat /proc/mtd
dev:    size   erasesize  name
mtd0: 10000000 00004000 "DiskOnChip 2000 (INFTL Model)"

#mtd_debug info /dev/mtd0
mtd.type = MTD_NANDFLASH
mtd.flags = MTD_CLEAR_BITS | MTD_ERASEABLE | MTD_OOB | MTD_ECC
mtd.size = 268435456 (256M)
mtd.erasesize = 16384 (16K)
mtd.oobblock = 512
mtd.oobsize = 16
mtd.ecctype = (unknown ECC type - new MTD API maybe?)
regions = 0

"new MTD API" ?? Is it dangerous for drivers work?

#flash_info /dev/mtd0
Device /dev/mtd0 has 0 erase regions

#flash_eraseall /dev/mtd0
Erasing 16 Kibyte @ 0 --  0 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 80000 --  0 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ c4000 --  0 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ d4000 --  0 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 5e14000 --  4 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 8080000 --  2 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 8124000 --  2 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 8134000 --  2 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ 9600000 -- 10 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
Erasing 16 Kibyte @ fffc000 --  3 % complete.
./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error

What is it? I don't understand, did programm finish her work?

#nftl_format /dev/mtd0
$Id: nftl_format.c,v 1.22 2004/05/05 15:19:57 dwmw2 Exp $
Phase 1. Checking and erasing Erase Zones from 0x00000000 to 0x07800000
         Erasing Zone #0 @ 0x0: Erase failed (Input/output error)
Skipping bad zone (factory marked) #32 @ 0x80000
Skipping bad zone (factory marked) #49 @ 0xc4000
Skipping bad zone (factory marked) #53 @ 0xd4000
        Checking Zone #3014 @ 0x2f18000: Block not zero after clearing
Skipping bad zone (RWE test failed) #3014 @ 0x2f18000
        Checking Zone #4234 @ 0x4228000: Block not zero after clearing
Skipping bad zone (RWE test failed) #4234 @ 0x4228000
         Erasing Zone #6021 @ 0x5e14000: Erase failed (Input/output error)
        Checking Zone #7679 @ 0x77fc000
Phase 2.a Writing NFTL Media Header and Bad Unit Table
Phase 2.b Writing Spare NFTL Media Header and Spare Bad Unit Table
Phase 3. Writing Unit Control Information to each Erase UnitINFTL: inftlcore.c 

#modprobe inftl
#dmesg
--skip--
INFTL: inftlcore.c $Revision: 1.17 $, inftlmount.c $Revision: 1.14 $
INFTL: could not find valid boot record?
INFTL: could not mount device

#rmmod inftl
#modprobe nftl
Segmentation fault
#dmesg
--skip--
NFTL driver: nftlcore.c $Revision: 1.96 $, nftlmount.c $Revision: 1.36 $
Formatting block 3014
Unable to handle kernel NULL pointer dereference at virtual address 00000100
 printing eip:
c802a711
*pde = 00000000
Oops: 0000 [#1]
PREEMPT
Modules linked in: nftl mtd_blkdevs mtdchar diskonchip nand nand_ids nand_ecc 
docecc mtdpart mtdcore ipv6 af_packet dm_mod 8139too mii crc32 usbkbd usbcore 
rtc ext3 jbd ide_generic sis5513 ide_disk ide_core unix
CPU:    0
EIP:    0060:[<c802a711>]    Not tainted
EFLAGS: 00010246   (2.6.8patch26)
EIP is at mtd_erase_callback+0x6/0x3b [mtdpart]
eax: 00004000   ebx: c63a3568   ecx: c67f3e78   edx: 00000000
esi: 000178e0   edi: c67f3e78   ebp: 00000000   esp: c5c9beb8
ds: 007b   es: 007b   ss: 0068
Process modprobe (pid: 859, threadinfo=c5c9a000 task=c61ca130)
Stack: c810a10e c67f3e78 00000000 00000020 c67f3e78 c67f3e00 c67f3ea8 00000bc6
       c8109f69 c63a3400 c67f3e78 00000000 c80cf0ea c63a3400 c67f3e78 00000008
       00000000 3c693c69 00000bc6 c67f3e00 c67f3e00 00000bc6 c80cf571 c67f3e00
Call Trace:
 [<c810a10e>] nand_erase_nand+0x1a1/0x1b9 [nand]
 [<c8109f69>] nand_erase+0xf/0x13 [nand]
 [<c80cf0ea>] NFTL_formatblock+0x88/0x112 [nftl]
 [<c80cf571>] NFTL_mount+0x139/0x3bc [nftl]
 [<c80ce0bb>] nftl_add_mtd+0xbb/0x1b0 [nftl]
 [<c80ac96f>] register_mtd_blktrans+0x191/0x1b0 [mtd_blkdevs]
 [<c802e019>] init_module+0x19/0x1d [nftl]
 [<c012af8c>] sys_init_module+0xe3/0x1d4
 [<c0105f97>] syscall_call+0x7/0xb
Code: 81 ba 00 01 00 00 cf a6 02 c8 75 1d 8b 41 0c 83 f8 ff 74 09



If I use last cvs - I have not even diskonchip.....  see previous my post


help

Re: input/output errors

[Thomas Gleixner]

On Mon, 2004-11-01 at 12:18 +0300, Dmitry Skorinko wrote:
> DiskOnChip found at 0xe0000
> Detected 2 chips per floor.
> NAND device: Manufacturer ID: 0xec, Chip ID: 0x79 (Samsung NAND 128MiB 3,3V 
> 8-bit)
> 2 NAND chips detected
> Bad block table found at page 524256, version 0x55
> Bad block table not found for chip 0
> nand_read_bbt: Reserved block at 0x00000000
> nand_read_bbt: Bad block at 0x00080000
> nand_read_bbt: Bad block at 0x000c4000
> nand_read_bbt: Bad block at 0x000d4000
> nand_read_bbt: Bad block at 0x08080000
> nand_read_bbt: Bad block at 0x08124000
> nand_read_bbt: Bad block at 0x08134000
> nand_read_bbt: Bad block at 0x09600000
> nand_read_bbt: Reserved block at 0x0fffc000
> ECC error scanning DOC at 0x0
> DiskOnChip BNAND Media Header not found.
> Found alias of DOC at 0xe0000 to 0xe2000
> Found alias of DOC at 0xe0000 to 0xe4000
> Found alias of DOC at 0xe0000 to 0xe6000
> 

> mtd.oobsize = 16
> mtd.ecctype = (unknown ECC type - new MTD API maybe?)
> regions = 0
> "new MTD API" ?? Is it dangerous for drivers work?

mtd_debug is outdated. Nothing to worry about.

> Erasing 16 Kibyte @ fffc000 --  3 % complete.
> ./flash_eraseall: /dev/mtd0: MTD Erase failure: Input/output error
> 
> What is it? I don't understand, did programm finish her work?

Look at the bad block output above. Bad blocks cannot be erased.

> #nftl_format /dev/mtd0
> $Id: nftl_format.c,v 1.22 2004/05/05 15:19:57 dwmw2 Exp $

I'm not sure whether nftl_format will work, as I assume, that it must
access at least block 0, which is marked bad to protect the information
there.

> #modprobe inftl
> #dmesg
> --skip--
> INFTL: inftlcore.c $Revision: 1.17 $, inftlmount.c $Revision: 1.14 $
> INFTL: could not find valid boot record?
> INFTL: could not mount device

Can you format the chip from DOS with the M-Sys tools and try again ?

> If I use last cvs - I have not even diskonchip.....  see previous my
> post

That previous post tells me not much. It's not really helpful to mixup
information.
What happens with current CVS code, if you modprobe diskonchip ?

tglx

Input/Output Errors

[qwerty t]

I am getting some very strange results I would like to
share.

I have a 16 M DoC 2000. I have formated it as an ext2
partition, and have copied 10M in various directories.

During boot MTD support says that it found unformated
sectors and formats them. This doesn't happen all the
time. Sometimes after one of these unformated sections
is formatted the result is I/O errors when I try to
read a file or directory.

Has anyone seen this before? Has anyone used ext2 as
filesystem.

Bill


__________________________________________________
Do You Yahoo!?
Spot the hottest trends in music, movies, and more.
http://buzz.yahoo.com/

Re: Input/Output Errors

[Stephen Herzog]

Greetings,

I tried using ext2 on an 8M DoC 2000  however had problems
related to corrupting the entire file system if the partition
was not unmounted correctly from read/write mode. 

We switched to a minix file system from ext2 to save space 
and make things a bit more stable, however the best thing
to do is keep the partition mounted read-only except 
when doing os updates.  

The problems we were having were not related to the MTD 
layer however.  

Steve


on Friday June 15, qwerty172@yahoo.com wrote:
> I have a 16 M DoC 2000. I have formated it as an ext2
> partition, and have copied 10M in various directories.
> During boot MTD support says that it found unformated
> sectors and formats them. 
> ...
> Has anyone seen this before? Has anyone used ext2 as
> filesystem.

Re: Input/Output Errors

[Ollie Lho]

qwerty t wrote:
> 
> I am getting some very strange results I would like to
> share.
> 
> I have a 16 M DoC 2000. I have formated it as an ext2
> partition, and have copied 10M in various directories.
> 
> During boot MTD support says that it found unformated
> sectors and formats them. This doesn't happen all the
> time. Sometimes after one of these unformated sections
> is formatted the result is I/O errors when I try to
> read a file or directory.
> 
> Has anyone seen this before? Has anyone used ext2 as
> filesystem.
> 


This seems to be a bug in the NFTL code. Somtimes the NFTL layer is
skewed up with the underlay DoC and does not work correctly on the 
NFTL metadata. Please send me the kernel message (the reformating
process) and nftldump output of the victim DoC.

Ollie