Input/Output errors

Input/Output errors

[Kenny MacDermid]

I'm running btrfs on DM-Crypt Luks running on LVM.

Occasionally I get files that are unreadable for some period of time.
Attempting to read from them results in an 

Input/output error

Sometimes they'll come back on their own, and sometimes a scrub seems to
help, but sometimes I just have to delete them.

Nothing shows up in dmesg when these occur, and I can't predict which
files it will be, or what causes it.

It's currently happening running 4.4.1-2-ARCH, but I've seen the same
thing for many previous kernel versions.

Does anyone have any suggestions?

Re: Input/Output errors

[Marc MERLIN]

On Tue, Feb 23, 2016 at 08:40:46PM -0400, Kenny MacDermid wrote:
> I'm running btrfs on DM-Crypt Luks running on LVM.
> 
> Occasionally I get files that are unreadable for some period of time.
> Attempting to read from them results in an 
> 
> Input/output error
> 
> Sometimes they'll come back on their own, and sometimes a scrub seems to
> help, but sometimes I just have to delete them.
> 
> Nothing shows up in dmesg when these occur, and I can't predict which
> files it will be, or what causes it.
> 
> It's currently happening running 4.4.1-2-ARCH, but I've seen the same
> thing for many previous kernel versions.
> 
> Does anyone have any suggestions?

That's weird to say the least, you should at least get *something* in
dmesg.
And you are getting other error messages and btrfs kernel messages in
your logs?

When whatever app you have that's trying to read them fails, I assume
they also fail with cat or less?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Input/Output errors

[Kenny MacDermid]

On Tue, Feb 23, 2016 at 05:56:58PM -0800, Marc MERLIN wrote:
> On Tue, Feb 23, 2016 at 08:40:46PM -0400, Kenny MacDermid wrote:
> > I'm running btrfs on DM-Crypt Luks running on LVM.
> > 
> > Occasionally I get files that are unreadable for some period of time.
> > Attempting to read from them results in an 
> > 
> > Input/output error
> > 
> > Sometimes they'll come back on their own, and sometimes a scrub seems to
> > help, but sometimes I just have to delete them.
> > 
> > Nothing shows up in dmesg when these occur, and I can't predict which
> > files it will be, or what causes it.
> > 
> > It's currently happening running 4.4.1-2-ARCH, but I've seen the same
> > thing for many previous kernel versions.
> > 
> > Does anyone have any suggestions?
> 
> That's weird to say the least, you should at least get *something* in
> dmesg.
> And you are getting other error messages and btrfs kernel messages in
> your logs?
> 
> When whatever app you have that's trying to read them fails, I assume
> they also fail with cat or less?

I am getting other, normal btrfs messages. I'll include them at the end.
When it happens I get nothing at all in dmesg/logs.

And yes, cat will fail. I can move the file to another name though,
which I often do to get it out of the way.

They're mounted with:

rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

issue_discards=1 is in lvm.conf, and discard in /etc/crypttab. (I'm now
reading that I probably shouldn't have it in fstab though and just run
fstrim.)

I don't know if this is related yet at all, but it /seems/ more likely
to happen after I delete a bunch of data. That could be a red herring
though.

When the file becomes readable again it's perfectly fine. Scrub never
finds any errors.

$ dmesg | grep -i btrfs
[   11.837137] Btrfs loaded
[   11.837403] BTRFS: device label root devid 1 transid 508963 /dev/dm-3
[   11.856203] BTRFS info (device dm-3): disk space caching is enabled
[   11.879366] BTRFS: detected SSD devices, enabling SSD mode
[   12.160267] BTRFS info (device dm-3): turning on discard
[   12.160272] BTRFS info (device dm-3): enabling auto defrag
[   12.160275] BTRFS info (device dm-3): enabling inode map caching
[   12.160277] BTRFS info (device dm-3): disk space caching is enabled
[   14.979093] BTRFS: device label home devid 1 transid 705779 /dev/dm-5
[   15.013978] BTRFS info (device dm-5): use ssd allocation scheme
[   15.013983] BTRFS info (device dm-5): turning on discard
[   15.013987] BTRFS info (device dm-5): enabling auto defrag
[   15.013989] BTRFS info (device dm-5): enabling inode map caching
[   15.013991] BTRFS info (device dm-5): disk space caching is enabled
[   15.100779] BTRFS error (device dm-5): could not find root 8
[   15.102889] BTRFS error (device dm-5): could not find root 8
[   15.105833] BTRFS error (device dm-3): could not find root 8
[   15.105838] BTRFS error (device dm-3): could not find root 8

Re: Input/Output errors

[Chris Murphy]

On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
<kenny.macdermid@gmail.com> wrote:

>
> rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

It sounds like an ssd trim bug. I'd check the firmware for updates. If
it's up to date, I'd drop discard mount option first and try to
reproduce. Or just use the default mount options and try to reproduce,
then add them back one at a time until you discover the culprit.

Also, how many files/directories are there? inode_cache isn't
recommended for most use cases. And space_cache is the default so it
doesn't need to be listed.



-- 
Chris Murphy

Re: Input/Output errors

[Szalma László]

2016-02-24 05:37 keltezéssel, Chris Murphy írta:
> On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
> <kenny.macdermid@gmail.com> wrote:
>
>> rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
> It sounds like an ssd trim bug. I'd check the firmware for updates. If
> it's up to date, I'd drop discard mount option first and try to
> reproduce. Or just use the default mount options and try to reproduce,
> then add them back one at a time until you discover the culprit.
>
> Also, how many files/directories are there? inode_cache isn't
> recommended for most use cases. And space_cache is the default so it
> doesn't need to be listed.
>
>
>

As i wrote to the list a few weeks ago, this problem seems to be the 
same I have.

The difference:

- i use mount options: noatime,compress,nossd
- I don't use dm-crypt, but these machines are Xen pvms (there is a 
virtualization layer between btrfs and lvm)

The same:

- io error without any error or message in the dmesg.
- umount / mount always fixes the problem (for some time)

More info:

- these files are usually smalls (mysql myisam files, 10-20-50 kbyte 
size, without heave fragmentation)
- defrag don't help
- scrub always works (no errors) but not fix the errors
- no hw or hw read error on the block device (in any layer)
- get this problem with 4.4.1 kernel too (seems to be somewhat less 
frequent than before, but the problem happened with 3.18 and on)
- echo 3 > /proc/sys/vm/drop_caches    sometimes fixes the problem, but 
not every time
- the problem is happening rarely, sometimes there are days without error
- the problem is not for specific hardware or virtual machine

I can try any debug option or patch if needed.

László Szalma

Re: Input/Output errors

[Kenny MacDermid]

On Wed, Feb 24, 2016 at 09:58:08AM +0100, Szalma László wrote:
> - echo 3 > /proc/sys/vm/drop_caches    sometimes fixes the problem, but not
> every time

Interesting. I just had the issue happen with one of my files, tried
this, and it's now readable again.

Re: Input/Output errors

[Kenny MacDermid]

On Tue, Feb 23, 2016 at 09:37:03PM -0700, Chris Murphy wrote:
> On Tue, Feb 23, 2016 at 8:02 PM, Kenny MacDermid
> <kenny.macdermid@gmail.com> wrote:
> 
> >
> > rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
> 
> It sounds like an ssd trim bug. I'd check the firmware for updates. If
> it's up to date, I'd drop discard mount option first and try to
> reproduce. Or just use the default mount options and try to reproduce,
> then add them back one at a time until you discover the culprit.

Thanks Chris,

The disk is a SAMSUNG MZMTE512HMHP-000L1 running f/w: EXT42L0Q. I didn't
find any updates in my searches.

I did drop the discard option, and the inode_cache and space_cache ones
as well. I'm not sure I originally got them from.

Even if it's a disk firmware bug, doesn't it seem like some layer should
report something about it? Is there any extra logging I should enable in
case it happens again?

As I don't have a way to reliable reproduce the error, I'll have to wait
and see if this helps.