Input/output error on newly created file

Input/output error on newly created file

[Nikolaus Rath]

Hello,

I just created an innocent 10 MB on a btrfs file system, yet my attempt
to read it a few seconds later (and ever since), just gives:

$ ls -l in-progress/mysterious-io-error 
-rw-rw-r-- 1 nikratio nikratio 10485760 May  8 14:41 in-progress/mysterious-io-error
$ cat in-progress/mysterious-io-error 
cat: in-progress/mysterious-io-error: Input/output error

I was hoping the kernel might tell me a little more about this, but:

$ dmesg | tail -3
[   48.982851] wlan0: RX AssocResp from 00:1f:33:be:9a:c4 (capab=0x411 status=0 aid=3)
[   48.983757] wlan0: associated
[ 2770.464505] kvm: zapping shadow pages for mmio generation wraparound


How can I find out the reason for the error? Is it a bug in BTRFS,
or somewhere else in the kernel ? Is my hardware broken?

Kernel is Debian's 4.5.0-0.bpo.1-amd64. This is a btrfs file system on a
LVM LV backed by a bcache array with two backing devices and one caching
device.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Input/output error on newly created file

[Filipe Manana]

On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
> Hello,
>
> I just created an innocent 10 MB on a btrfs file system, yet my attempt
> to read it a few seconds later (and ever since), just gives:
>
> $ ls -l in-progress/mysterious-io-error
> -rw-rw-r-- 1 nikratio nikratio 10485760 May  8 14:41 in-progress/mysterious-io-error
> $ cat in-progress/mysterious-io-error
> cat: in-progress/mysterious-io-error: Input/output error

If you unmount and mount again the filesystem, does it happen again?
How did you create the file? Does fsck reports any issues?

>
> I was hoping the kernel might tell me a little more about this, but:
>
> $ dmesg | tail -3
> [   48.982851] wlan0: RX AssocResp from 00:1f:33:be:9a:c4 (capab=0x411 status=0 aid=3)
> [   48.983757] wlan0: associated
> [ 2770.464505] kvm: zapping shadow pages for mmio generation wraparound
>
>
> How can I find out the reason for the error? Is it a bug in BTRFS,
> or somewhere else in the kernel ? Is my hardware broken?
>
> Kernel is Debian's 4.5.0-0.bpo.1-amd64. This is a btrfs file system on a
> LVM LV backed by a bcache array with two backing devices and one caching
> device.
>
>
> Best,
> -Nikolaus
>
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
>              »Time flies like an arrow, fruit flies like a Banana.«
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: Input/output error on newly created file

[Nikolaus Rath]

On May 09 2016, Filipe Manana <fdmanana@gmail.com> wrote:
> On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
>> Hello,
>>
>> I just created an innocent 10 MB on a btrfs file system, yet my attempt
>> to read it a few seconds later (and ever since), just gives:
>>
>> $ ls -l in-progress/mysterious-io-error
>> -rw-rw-r-- 1 nikratio nikratio 10485760 May  8 14:41 in-progress/mysterious-io-error
>> $ cat in-progress/mysterious-io-error
>> cat: in-progress/mysterious-io-error: Input/output error
>
> If you unmount and mount again the filesystem, does it happen again?

After rebooting, the previously unaccessible file can now be read. But I
cannot tell if it contains the right data.

However, I just encountered the same problem with another, freshly
created file.

> How did you create the file?

In Python 3. The equivalent code is more or less:

with open('file.dat', 'wb+') as fh:
     for buf in generate_data():
         fh.write(buf) # bufsize is about 128 kB


However, I should note that there is a lot of activity in this
file system (it contains my home directory), so the above alone will
probably not reproduce the problem.

That said, so far both the problematic files were created by the same
application (S3QL, of which luckily I am also the maintainer).


> Does fsck reports any issues?

Do you mean btrfsck? It actually has a lot to say:

checking extents
checking free space cache
checking fs roots
root 5 inode 3149867 errors 400, nbytes wrong
root 5 inode 3150237 errors 400, nbytes wrong
root 5 inode 3150238 errors 400, nbytes wrong
root 5 inode 3150242 errors 400, nbytes wrong
root 5 inode 3150260 errors 400, nbytes wrong
[...]
root 5 inode 15595011 errors 400, nbytes wrong
root 5 inode 15595016 errors 400, nbytes wrong
Checking filesystem on /dev/mapper/vg0-nikratio_crypt
UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38
found 263648960636 bytes used err is 1
total csum bytes: 395314372
total tree bytes: 908644352
total fs tree bytes: 352735232
total extent tree bytes: 95039488
btree space waste bytes: 156301160
file data blocks allocated: 675209801728
 referenced 410351722496
Btrfs v3.17

However, the inode of the problematic file (16186241) is not
mentioned. But I guess this is not surprising, because also for this
file, I can read the contents after remounting.


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Input/output error on newly created file

[Nikolaus Rath]

*ping*

Anyone any idea?

Best,
-Nikolaus

On May 09 2016, Nikolaus Rath <Nikolaus@rath.org> wrote:
> On May 09 2016, Filipe Manana <fdmanana@gmail.com> wrote:
>> On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
>>> Hello,
>>>
>>> I just created an innocent 10 MB on a btrfs file system, yet my attempt
>>> to read it a few seconds later (and ever since), just gives:
>>>
>>> $ ls -l in-progress/mysterious-io-error
>>> -rw-rw-r-- 1 nikratio nikratio 10485760 May  8 14:41 in-progress/mysterious-io-error
>>> $ cat in-progress/mysterious-io-error
>>> cat: in-progress/mysterious-io-error: Input/output error
>>
>> If you unmount and mount again the filesystem, does it happen again?
>
> After rebooting, the previously unaccessible file can now be read. But I
> cannot tell if it contains the right data.
>
> However, I just encountered the same problem with another, freshly
> created file.
>
>> How did you create the file?
>
> In Python 3. The equivalent code is more or less:
>
> with open('file.dat', 'wb+') as fh:
>      for buf in generate_data():
>          fh.write(buf) # bufsize is about 128 kB
>
>
> However, I should note that there is a lot of activity in this
> file system (it contains my home directory), so the above alone will
> probably not reproduce the problem.
>
> That said, so far both the problematic files were created by the same
> application (S3QL, of which luckily I am also the maintainer).
>
>
>> Does fsck reports any issues?
>
> Do you mean btrfsck? It actually has a lot to say:
>
> checking extents
> checking free space cache
> checking fs roots
> root 5 inode 3149867 errors 400, nbytes wrong
> root 5 inode 3150237 errors 400, nbytes wrong
> root 5 inode 3150238 errors 400, nbytes wrong
> root 5 inode 3150242 errors 400, nbytes wrong
> root 5 inode 3150260 errors 400, nbytes wrong
> [...]
> root 5 inode 15595011 errors 400, nbytes wrong
> root 5 inode 15595016 errors 400, nbytes wrong
> Checking filesystem on /dev/mapper/vg0-nikratio_crypt
> UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38
> found 263648960636 bytes used err is 1
> total csum bytes: 395314372
> total tree bytes: 908644352
> total fs tree bytes: 352735232
> total extent tree bytes: 95039488
> btree space waste bytes: 156301160
> file data blocks allocated: 675209801728
>  referenced 410351722496
> Btrfs v3.17
>
> However, the inode of the problematic file (16186241) is not
> mentioned. But I guess this is not surprising, because also for this
> file, I can read the contents after remounting.
>
>
> Best,
> -Nikolaus
>
>
> -- 
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
>              »Time flies like an arrow, fruit flies like a Banana.«
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Input/output error on newly created file

[Diego Calleja]

El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
> *ping*
> 
> Anyone any idea?

All I can say is that I've had the same problem in the past. In my
case, the problematic files where active torrents. The interesting
thing is that I was able to read them correctly up to a point, then
I would get the same error as you. No messages in dmesg. The amount
of data I was able to read from them was not random, it was
something multiple of 4K. After reboot the problems went away and
I wasn't able to reproduce it.

There has been reports of similiar issues in the past:
http://www.spinics.net/lists/linux-btrfs/msg52371.html

Re: Input/output error on newly created file

[Nikolaus Rath]

On May 12 2016, Diego Calleja <diegocg@gmail.com> wrote:
> El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
>> *ping*
>> 
>> Anyone any idea?
>
> All I can say is that I've had the same problem in the past. In my
> case, the problematic files where active torrents. The interesting
> thing is that I was able to read them correctly up to a point, then
> I would get the same error as you. No messages in dmesg. The amount
> of data I was able to read from them was not random, it was
> something multiple of 4K. After reboot the problems went away and
> I wasn't able to reproduce it.
>
> There has been reports of similiar issues in the past:
> http://www.spinics.net/lists/linux-btrfs/msg52371.html

Thanks for the pointer. So according to these reports, the IO errors
happen with:


1. dm-crypt, LVM and mount options
   rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
2. dm-crypt, no LVM, and mount options
   noatime,compress,nossd

I can add the following data point:

3. dm-crypt on LVM, and mount options
   relatime,compress=lzo


(Just to preserve this for the future)


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Input/output error on newly created file

[Szalma László]

2016-05-12 19:41 keltezéssel, Nikolaus Rath írta:
> On May 12 2016, Diego Calleja <diegocg@gmail.com> wrote:
>> El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
>>> *ping*
>>>
>>> Anyone any idea?
>> All I can say is that I've had the same problem in the past. In my
>> case, the problematic files where active torrents. The interesting
>> thing is that I was able to read them correctly up to a point, then
>> I would get the same error as you. No messages in dmesg. The amount
>> of data I was able to read from them was not random, it was
>> something multiple of 4K. After reboot the problems went away and
>> I wasn't able to reproduce it.
>>
>> There has been reports of similiar issues in the past:
>> http://www.spinics.net/lists/linux-btrfs/msg52371.html
> Thanks for the pointer. So according to these reports, the IO errors
> happen with:
>
>
> 1. dm-crypt, LVM and mount options
>     rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
> 2. dm-crypt, no LVM, and mount options
>     noatime,compress,nossd
>
> I can add the following data point:
>
> 3. dm-crypt on LVM, and mount options
>     relatime,compress=lzo
>
>
> (Just to preserve this for the future)

I don't want to repeat my e-mails in the past, but I think I have a 
similar issue. In short: xen virtual machine. The files that rarely 
become unreadable (I/O error but no error in dmesg or anywhere) are 
mysql MyIsam database files, and they are always small. Like 16kbyte for 
example, or smaller. Sometimes dropping the fs cache fixes the problem, 
sometimes not. Umount / mount always fixes the problem. Scrub says the 
filesystem is OK (when the file is unreadable).

I experienced similar problem with log files (smtp or apache log files), 
but it is rare. It happens 1-2 times in a month on a heavy loaded 
mail/web server. (the log file io error is not a real problem for me, 
mysql is.). The problem present from 3.18 to 4.5.

I started to migrate mysql table files to innodb, but i think this is 
only a workaround.

László Szalma


>
>
> Best,
> -Nikolaus

Re: Input/output error on newly created file

[Duncan]

Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted:

> The files that rarely become unreadable (I/O error but no error in dmesg
> or anywhere) are mysql MyIsam database files, and they are always small.
> Like 16kbyte for example, or smaller. Sometimes dropping the fs cache
> fixes the problem, sometimes not. Umount / mount always fixes the
> problem. Scrub says the filesystem is OK (when the file is unreadable).

Is it possible the files are always under 4 KiB?

While there's a few variables as to max size, with 4 KiB being the 
practical max, btrfs will inline really small files into their metadata 
node instead of writing them out as 4 KiB data block extents.  Since 
that's obviously a different code-path, if it's only 4 KiB and smaller 
files, it's possible there's a race in that code-path that when lost, 
results in the file "disappearing" without a dmesg trace, only to 
reappear after reboot.

Actually, now that I think about it, if the files are all OVER say 2 KiB 
but still quite small, say under 16 KiB, and if they had just recently 
grown, it's possible that the race is in the transition from the inline 
metadata state to the multiples of 4 KiB block data extent state.

And if all the files had just shrunk, say from compaction (if done in-
place, not with a copy and rename), perhaps it's the reverse, the 
transition from written data blocks to inline metadata state.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Input/output error on newly created file

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

Am Friday 13 May 2016
schrieb Duncan <1i5t5.duncan@cox.net>

>Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted:
>> The files that rarely become unreadable (I/O error but no error in dmesg
>> or anywhere) are mysql MyIsam database files, and they are always small.
>> Like 16kbyte for example, or smaller. Sometimes dropping the fs cache
>> fixes the problem, sometimes not. Umount / mount always fixes the
>> problem. Scrub says the filesystem is OK (when the file is unreadable).
>
>Is it possible the files are always under 4 KiB?

For the record, I was seeing a similar error with dovecot *.index.log files 
(see the ML thread started by Szalma László) .  In my case they are *not* all 
under 4 KiB.  Looking at some of the affected files, one of them is 25K, and 
another is 6.6K.  However, perhaps they compress to under 4K?  But compressing 
the 25K one with lzop only goes down to 5.6K with -9 :-/ .

>While there's a few variables as to max size, with 4 KiB being the
>practical max, btrfs will inline really small files into their metadata
>node instead of writing them out as 4 KiB data block extents.  Since
>that's obviously a different code-path, if it's only 4 KiB and smaller
>files, it's possible there's a race in that code-path that when lost,
>results in the file "disappearing" without a dmesg trace, only to
>reappear after reboot.
>
>Actually, now that I think about it, if the files are all OVER say 2 KiB
>but still quite small, say under 16 KiB, and if they had just recently
>grown, it's possible that the race is in the transition from the inline
>metadata state to the multiples of 4 KiB block data extent state.
>
>And if all the files had just shrunk, say from compaction (if done in-
>place, not with a copy and rename), perhaps it's the reverse, the
>transition from written data blocks to inline metadata state.

I'm glad somebody is (publicly) thinking about this :-) !

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]