Error during balancing '/': Input/output error

public inbox for linux-btrfs@vger.kernel.org
 help / color / mirror / Atom feed

* Error during balancing '/': Input/output error
@ 2019-12-27 12:39 Michael Ruiz
  2019-12-27 13:21 ` Qu Wenruo
  2019-12-28  9:53 ` Michael Ruiz
  0 siblings, 2 replies; 9+ messages in thread
From: Michael Ruiz @ 2019-12-27 12:39 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

Hi,

I recently did a btrfs-convert from ext4 to btrfs and it went well, so I 
deleted the ext2 'home-backup' or whatever subvolume it created and I 
proceeded to balance the disk. However, I keep getting this error when I try 
to run this command (see bottom of message for debug info.) This might be a 
simple question, but I am worried that my root drive is corrupt and if I 
restart it might not boot so I haven't tried rebooting and mounting the disk 
from a live usb. Any suggestions pointing me in the right direction are most 
appreciated.


Thanks,
Michael

``` BEGIN DEBUG INFO ```
[michael@linux /]$ sudo btrfs balance /
ERROR: error during balancing '/': Input/output error
There may be more info in syslog - try dmesg | tail

[michael@linux /]$ dmesg | tail
[ 1235.433416] BTRFS info (device dm-1): relocating block group 265709158400 
flags metadata
[ 1375.701515] BTRFS info (device dm-1): found 65034 extents
[ 1381.270271] BTRFS info (device dm-1): relocating block group 264635416576 
flags metadata
[ 1483.628290] BTRFS info (device dm-1): found 64995 extents
[ 1485.160174] BTRFS info (device dm-1): relocating block group 253562454016 
flags data
[ 1485.352336] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 
1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
[ 1485.353008] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 
1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
[ 1485.375271] BTRFS info (device dm-1): balance: ended with status: -5
[ 1485.544838] audit: type=1106 audit(1577448716.970:152): pid=4178 uid=0 
auid=1000 ses=1 msg='op=PAM:session_close 
grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" 
hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 1485.545220] audit: type=1104 audit(1577448716.970:153): pid=4178 uid=0 
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env 
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 
res=success'
``` END DEBUG INFO ```


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-27 12:39 Error during balancing '/': Input/output error Michael Ruiz
@ 2019-12-27 13:21 ` Qu Wenruo
       [not found]   ` <5555870.lOV4Wx5bFT@archlinux>
  2019-12-28  9:53 ` Michael Ruiz
  1 sibling, 1 reply; 9+ messages in thread
From: Qu Wenruo @ 2019-12-27 13:21 UTC (permalink / raw)
  To: Michael Ruiz, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 2572 bytes --]



On 2019/12/27 下午8:39, Michael Ruiz wrote:
> Hi,
> 
> I recently did a btrfs-convert from ext4 to btrfs and it went well, so I 
> deleted the ext2 'home-backup' or whatever subvolume it created and I 
> proceeded to balance the disk. However, I keep getting this error when I try 
> to run this command (see bottom of message for debug info.) This might be a 
> simple question, but I am worried that my root drive is corrupt and if I 
> restart it might not boot so I haven't tried rebooting and mounting the disk 
> from a live usb. Any suggestions pointing me in the right direction are most 
> appreciated.

There are quite some reports about this problem. So far no real
corrupted confirmed.

You can confirm it by scrub.

I'm afraid there is some bug in data balance code which leads to this.
We will dig further to pin down the cause.

BTW, does that problem always show up at block group 253562454016? Or it
randomly shows up at different block groups?

Thanks,
Qu

> 
> 
> Thanks,
> Michael
> 
> ``` BEGIN DEBUG INFO ```
> [michael@linux /]$ sudo btrfs balance /
> ERROR: error during balancing '/': Input/output error
> There may be more info in syslog - try dmesg | tail
> 
> [michael@linux /]$ dmesg | tail
> [ 1235.433416] BTRFS info (device dm-1): relocating block group 265709158400 
> flags metadata
> [ 1375.701515] BTRFS info (device dm-1): found 65034 extents
> [ 1381.270271] BTRFS info (device dm-1): relocating block group 264635416576 
> flags metadata
> [ 1483.628290] BTRFS info (device dm-1): found 64995 extents
> [ 1485.160174] BTRFS info (device dm-1): relocating block group 253562454016 
> flags data
> [ 1485.352336] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 
> 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
> [ 1485.353008] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 
> 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
> [ 1485.375271] BTRFS info (device dm-1): balance: ended with status: -5
> [ 1485.544838] audit: type=1106 audit(1577448716.970:152): pid=4178 uid=0 
> auid=1000 ses=1 msg='op=PAM:session_close 
> grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" 
> hostname=? addr=? terminal=/dev/pts/2 res=success'
> [ 1485.545220] audit: type=1104 audit(1577448716.970:153): pid=4178 uid=0 
> auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env 
> acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 
> res=success'
> ``` END DEBUG INFO ```
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

[parent not found: <5555870.lOV4Wx5bFT@archlinux>]

* Re: Error during balancing '/': Input/output error
       [not found]   ` <5555870.lOV4Wx5bFT@archlinux>
@ 2019-12-28 10:09     ` Qu Wenruo
  2019-12-28 11:26       ` Michael Ruiz
  0 siblings, 1 reply; 9+ messages in thread
From: Qu Wenruo @ 2019-12-28 10:09 UTC (permalink / raw)
  To: Michael Ruiz, linux-btrfs@vger.kernel.org


[-- Attachment #1.1: Type: text/plain, Size: 2800 bytes --]



On 2019/12/28 下午5:49, Michael Ruiz wrote:
>> There are quite some reports about this problem. So far no real
>> corrupted confirmed.
>>
>> You can confirm it by scrub.
>>
>> I'm afraid there is some bug in data balance code which leads to this.
>> We will dig further to pin down the cause.
>>
>> BTW, does that problem always show up at block group 253562454016? Or it
>> randomly shows up at different block groups?
>>
>> Thanks,
>> Qu
> 
> Hi Qu,
> It appears to be the same data blocks giving me this issue. I have tried it 
> while mounting from a bootable usb and in my arch linux install both with 
> similar results.

So it looks like you will hit the same -EIO even just relocating that
specific block group using vrange filter.

> I should also mention I have a btrfs root partition within a 
> luks lvm partition becuase btrfs provides no encryption. I'm not sure this 
> makes a difference.

I don't believe it would cause any difference.

> Would you suggest running somekind of disk check or by 
> running scrub? Please let me know what I should try to resolve this.

Running a scrub is always preferred.
An `btrfs check` run would be even better.

If all above reports no error, we need more info to dig. (See below).

> 谢谢
> 
> ``` BEGIN DEBUG INFO ```
> ERROR: error during balancing '/': Input/output error
> [  465.880807] BTRFS info (device dm-1): found 25 extents
> [  470.501319] BTRFS info (device dm-1): found 25 extents
> [  471.713073] BTRFS info (device dm-1): relocating block group 253562454016 
> flags data
> [  472.251356] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 
> 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1

Another question, is the reported csum (both have and expected) always
the same?

> [  472.254390] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 
> 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1

This line means two things:
- We have extra inodes in data reloc tree
  Normally data reloc only has one inode, 256.
  It's possible we have some extra orphan inodes.
  Would you like to do the following dump?

  # btrfs ins dump-tree -t data_reloc /dev/dm-1

  Although the result is not something representing the content when we
  hit the bug, it may still provide some clue.

- The original data is from logical 253563502592
  Would you like to check which file(s) owns that logical?
  It can be done by the following command:

  # btrfs ins logical-resolve 253563502592 <mnt>

  Unlike previous command, it needs to be executed on a mounted btrfs
  other than the block device.

  And the info about that inode is also helpful, like its attr
  (nodatacow?).

Thanks,
Qu

> ``` END DEBUG INFO ```
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-28 10:09     ` Qu Wenruo
@ 2019-12-28 11:26       ` Michael Ruiz
  2019-12-28 11:38         ` Qu Wenruo
  0 siblings, 1 reply; 9+ messages in thread
From: Michael Ruiz @ 2019-12-28 11:26 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs, linux-btrfs; +Cc: linux-btrfs, Qu Wenruo, Qu Wenruo

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

Hi Qu,
Thanks for the quick reply. This is what I could find. 

[michael@linux /]$ sudo btrfs ins dump-tree -t data_reloc /dev/dm-1
btrfs-progs v5.4 
data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 2) 
leaf 286941609984 items 2 free space 16061 generation 1186381 owner 
DATA_RELOC_TREE
leaf 286941609984 flags 0x1(WRITTEN) backref revision 1
fs uuid <redacted>
chunk uuid <redacted>
        item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
                generation 2 transid 0 size 0 nbytes 16384
                block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
                sequence 0 flags 0x0(none)
                atime 1577186710.0 (2019-12-24 03:25:10)
                ctime 1577186710.0 (2019-12-24 03:25:10)
                mtime 1577186710.0 (2019-12-24 03:25:10)
                otime 1577186710.0 (2019-12-24 03:25:10)
        item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
                index 0 namelen 2 name: ..
total bytes 255307284480
bytes used 94295339008
uuid <redacted>

[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 /
//home/michael/.mozilla/firefox/default/storage/default/https++
+www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688-
ab3c5fd49010}.final



``` BTRFS SCRUB ```
sudo btrfs scrub status /
UUID:             <redacted>
Scrub started:    Sat Dec 28 03:17:49 2019
Status:           finished
Duration:         0:05:09
Total to scrub:   87.82GiB
Rate:             291.04MiB/s
Error summary:    csum=16
  Corrected:      0
  Uncorrectable:  16
  Unverified:     0
```

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-28 11:26       ` Michael Ruiz
@ 2019-12-28 11:38         ` Qu Wenruo
  2019-12-28 11:45           ` Michael Ruiz
  0 siblings, 1 reply; 9+ messages in thread
From: Qu Wenruo @ 2019-12-28 11:38 UTC (permalink / raw)
  To: Michael Ruiz, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 1814 bytes --]



On 2019/12/28 下午7:26, Michael Ruiz wrote:
> Hi Qu,
> Thanks for the quick reply. This is what I could find. 
> 
> [michael@linux /]$ sudo btrfs ins dump-tree -t data_reloc /dev/dm-1
> btrfs-progs v5.4 
> data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 2) 
> leaf 286941609984 items 2 free space 16061 generation 1186381 owner 
> DATA_RELOC_TREE
> leaf 286941609984 flags 0x1(WRITTEN) backref revision 1
> fs uuid <redacted>
> chunk uuid <redacted>
>         item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
>                 generation 2 transid 0 size 0 nbytes 16384
>                 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>                 sequence 0 flags 0x0(none)
>                 atime 1577186710.0 (2019-12-24 03:25:10)
>                 ctime 1577186710.0 (2019-12-24 03:25:10)
>                 mtime 1577186710.0 (2019-12-24 03:25:10)
>                 otime 1577186710.0 (2019-12-24 03:25:10)
>         item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
>                 index 0 namelen 2 name: ..
> total bytes 255307284480
> bytes used 94295339008
> uuid <redacted>
> 
> [michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 /
> //home/michael/.mozilla/firefox/default/storage/default/https++
> +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688-
> ab3c5fd49010}.final
> 
> 
> 
> ``` BTRFS SCRUB ```
> sudo btrfs scrub status /
> UUID:             <redacted>
> Scrub started:    Sat Dec 28 03:17:49 2019
> Status:           finished
> Duration:         0:05:09
> Total to scrub:   87.82GiB
> Rate:             291.04MiB/s
> Error summary:    csum=16
>   Corrected:      0
>   Uncorrectable:  16
>   Unverified:     0
> ```
Dmesg please, as that shows which file(s) is involved.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-28 11:38         ` Qu Wenruo
@ 2019-12-28 11:45           ` Michael Ruiz
  2019-12-28 12:05             ` Qu Wenruo
  0 siblings, 1 reply; 9+ messages in thread
From: Michael Ruiz @ 2019-12-28 11:45 UTC (permalink / raw)
  To: linux-btrfs; +Cc: quwenruo.btrfs

[-- Attachment #1: Type: text/plain, Size: 6668 bytes --]

On Saturday, December 28, 2019 3:38:48 AM PST you wrote:
> 
> Dmesg please, as that shows which file(s) is involved.
> 
> Thanks,
> Qu

Sure thing, it appears they are all from firefox cache.. Perhaps they are 
failing checksum because they were modified since I converted the partition 
earlier this week? I'm just guessing. Would you reccomend deleting these files 
manually or running btrfs check?
Thanks

``` BEGIN DMESG OUTPUT ```
[ 5638.996798] BTRFS warning (device dm-1): checksum error at logical 
253563510784 on dev /dev/mapper/volgroup0-lv_root, physical 85319491584, root 
5, inode 6477931, offset 8192, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.996805] BTRFS warning (device dm-1): checksum error at logical 
253563502592 on dev /dev/mapper/volgroup0-lv_root, physical 85319483392, root 
5, inode 6477931, offset 0, length 4096, links 1 (path: home/michael/.mozilla/
firefox/default/storage/default/https+++www.pinterest.com/cache/morgue/16/
{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.996808] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 17, gen 0
[ 5638.996812] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563510784 on dev /dev/mapper/volgroup0-lv_root
[ 5638.996815] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 18, gen 0
[ 5638.996818] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563502592 on dev /dev/mapper/volgroup0-lv_root
[ 5638.997338] BTRFS warning (device dm-1): checksum error at logical 
253563506688 on dev /dev/mapper/volgroup0-lv_root, physical 85319487488, root 
5, inode 6477931, offset 4096, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.997344] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 19, gen 0
[ 5638.997348] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563506688 on dev /dev/mapper/volgroup0-lv_root
[ 5638.997526] BTRFS warning (device dm-1): checksum error at logical 
253563514880 on dev /dev/mapper/volgroup0-lv_root, physical 85319495680, root 
5, inode 6477931, offset 12288, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.997531] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
[ 5638.997533] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563514880 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998086] BTRFS warning (device dm-1): checksum error at logical 
253563518976 on dev /dev/mapper/volgroup0-lv_root, physical 85319499776, root 
5, inode 6477931, offset 16384, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998097] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 21, gen 0
[ 5638.998101] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563518976 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998542] BTRFS warning (device dm-1): checksum error at logical 
253563523072 on dev /dev/mapper/volgroup0-lv_root, physical 85319503872, root 
5, inode 6477931, offset 20480, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998551] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 22, gen 0
[ 5638.998554] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563523072 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998975] BTRFS warning (device dm-1): checksum error at logical 
253563527168 on dev /dev/mapper/volgroup0-lv_root, physical 85319507968, root 
5, inode 6477931, offset 24576, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998981] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 23, gen 0
[ 5638.998984] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563527168 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999356] BTRFS warning (device dm-1): checksum error at logical 
253563531264 on dev /dev/mapper/volgroup0-lv_root, physical 85319512064, root 
5, inode 6477931, offset 28672, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999362] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 24, gen 0
[ 5638.999366] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563531264 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999592] BTRFS warning (device dm-1): checksum error at logical 
253563535360 on dev /dev/mapper/volgroup0-lv_root, physical 85319516160, root 
5, inode 6477931, offset 32768, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999596] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 25, gen 0
[ 5638.999598] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563535360 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999861] BTRFS warning (device dm-1): checksum error at logical 
253563539456 on dev /dev/mapper/volgroup0-lv_root, physical 85319520256, root 
5, inode 6477931, offset 36864, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999866] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
errs: wr 0, rd 0, flush 0, corrupt 26, gen 0
[ 5638.999869] BTRFS error (device dm-1): unable to fixup (regular) error at 
logical 253563539456 on dev /dev/mapper/volgroup0-lv_root
[ 5692.239308] BTRFS info (device dm-1): scrub: finished on devid 1 with 
status: 0
``` END DMESG OUTPUT ```

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-28 11:45           ` Michael Ruiz
@ 2019-12-28 12:05             ` Qu Wenruo
  2019-12-28 12:28               ` Michael Ruiz
  0 siblings, 1 reply; 9+ messages in thread
From: Qu Wenruo @ 2019-12-28 12:05 UTC (permalink / raw)
  To: Michael Ruiz, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 7819 bytes --]



On 2019/12/28 下午7:45, Michael Ruiz wrote:
> On Saturday, December 28, 2019 3:38:48 AM PST you wrote:
>>
>> Dmesg please, as that shows which file(s) is involved.
>>
>> Thanks,
>> Qu
> 
> Sure thing, it appears they are all from firefox cache.. Perhaps they are 
> failing checksum because they were modified since I converted the partition 
> earlier this week? I'm just guessing. Would you reccomend deleting these files 
> manually or running btrfs check?
> Thanks
> 
> ``` BEGIN DMESG OUTPUT ```
> [ 5638.996798] BTRFS warning (device dm-1): checksum error at logical 
> 253563510784 on dev /dev/mapper/volgroup0-lv_root, physical 85319491584, root 
> 5, inode 6477931, offset 8192, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)

Doesn't this filename look familiar with previous logical-resolve output? :)

[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 /
//home/michael/.mozilla/firefox/default/storage/default/https++
+www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688-
ab3c5fd49010}.final

The URL, the UUID, they all matches!

So yep, the file itself doesn't match its csum in the first place.

It looks like balance doesn't go regular file csum checking, but copy
all data and all csum without verifying them, then check the csum of
data reloc tree instead.

Thus this causes this hard-to-locate corrupted files.
We should make it more user-friendly I guess.


And obviously, just remove the offending files should allow you continue.

Thanks,
Qu

> [ 5638.996805] BTRFS warning (device dm-1): checksum error at logical 
> 253563502592 on dev /dev/mapper/volgroup0-lv_root, physical 85319483392, root 
> 5, inode 6477931, offset 0, length 4096, links 1 (path: home/michael/.mozilla/
> firefox/default/storage/default/https+++www.pinterest.com/cache/morgue/16/
> {b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.996808] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 17, gen 0
> [ 5638.996812] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563510784 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.996815] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 18, gen 0
> [ 5638.996818] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563502592 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.997338] BTRFS warning (device dm-1): checksum error at logical 
> 253563506688 on dev /dev/mapper/volgroup0-lv_root, physical 85319487488, root 
> 5, inode 6477931, offset 4096, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.997344] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 19, gen 0
> [ 5638.997348] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563506688 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.997526] BTRFS warning (device dm-1): checksum error at logical 
> 253563514880 on dev /dev/mapper/volgroup0-lv_root, physical 85319495680, root 
> 5, inode 6477931, offset 12288, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.997531] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
> [ 5638.997533] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563514880 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.998086] BTRFS warning (device dm-1): checksum error at logical 
> 253563518976 on dev /dev/mapper/volgroup0-lv_root, physical 85319499776, root 
> 5, inode 6477931, offset 16384, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.998097] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 21, gen 0
> [ 5638.998101] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563518976 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.998542] BTRFS warning (device dm-1): checksum error at logical 
> 253563523072 on dev /dev/mapper/volgroup0-lv_root, physical 85319503872, root 
> 5, inode 6477931, offset 20480, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.998551] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 22, gen 0
> [ 5638.998554] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563523072 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.998975] BTRFS warning (device dm-1): checksum error at logical 
> 253563527168 on dev /dev/mapper/volgroup0-lv_root, physical 85319507968, root 
> 5, inode 6477931, offset 24576, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.998981] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 23, gen 0
> [ 5638.998984] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563527168 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.999356] BTRFS warning (device dm-1): checksum error at logical 
> 253563531264 on dev /dev/mapper/volgroup0-lv_root, physical 85319512064, root 
> 5, inode 6477931, offset 28672, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.999362] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 24, gen 0
> [ 5638.999366] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563531264 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.999592] BTRFS warning (device dm-1): checksum error at logical 
> 253563535360 on dev /dev/mapper/volgroup0-lv_root, physical 85319516160, root 
> 5, inode 6477931, offset 32768, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.999596] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 25, gen 0
> [ 5638.999598] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563535360 on dev /dev/mapper/volgroup0-lv_root
> [ 5638.999861] BTRFS warning (device dm-1): checksum error at logical 
> 253563539456 on dev /dev/mapper/volgroup0-lv_root, physical 85319520256, root 
> 5, inode 6477931, offset 36864, length 4096, links 1 (path: home/
> michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
> cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
> [ 5638.999866] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root 
> errs: wr 0, rd 0, flush 0, corrupt 26, gen 0
> [ 5638.999869] BTRFS error (device dm-1): unable to fixup (regular) error at 
> logical 253563539456 on dev /dev/mapper/volgroup0-lv_root
> [ 5692.239308] BTRFS info (device dm-1): scrub: finished on devid 1 with 
> status: 0
> ``` END DMESG OUTPUT ```
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-28 12:05             ` Qu Wenruo
@ 2019-12-28 12:28               ` Michael Ruiz
  0 siblings, 0 replies; 9+ messages in thread
From: Michael Ruiz @ 2019-12-28 12:28 UTC (permalink / raw)
  To: linux-btrfs; +Cc: quwenruo.btrfs

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

On Saturday, December 28, 2019 4:05:54 AM PST you wrote:
> Doesn't this filename look familiar with previous logical-resolve output? :)
> 
> [michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 /
> //home/michael/.mozilla/firefox/default/storage/default/https++
> +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688-
> ab3c5fd49010}.final
> 
> The URL, the UUID, they all matches!
> 
> So yep, the file itself doesn't match its csum in the first place.
> 
> It looks like balance doesn't go regular file csum checking, but copy
> all data and all csum without verifying them, then check the csum of
> data reloc tree instead.
> 
> Thus this causes this hard-to-locate corrupted files.
> We should make it more user-friendly I guess.
> 
> 
> And obviously, just remove the offending files should allow you continue.
> 
> Thanks,
> Qu

Wow, you are right. I was worried my file system was corrupted and I'm not 
sure the exact cause leading up to this error. If there is any more debug info 
I can give you so that you can get a better idea of why this is happening, 
please let me know. 

Currently I am running balance, but I predict it will finally be able to 
complete sucessfully. However, if there are any issues I will send a reply 
here. (If I don't figure it out myself using what you just taught me)
Thanks a lot for your help. I am new to using mailing lists such as these, so 
hopefully I did everything by the rules here.
Thanks again Qu!

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Error during balancing '/': Input/output error
  2019-12-27 12:39 Error during balancing '/': Input/output error Michael Ruiz
  2019-12-27 13:21 ` Qu Wenruo
@ 2019-12-28  9:53 ` Michael Ruiz
  1 sibling, 0 replies; 9+ messages in thread
From: Michael Ruiz @ 2019-12-28  9:53 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 2494 bytes --]

> There are quite some reports about this problem. So far no real
> corrupted confirmed.
>
> You can confirm it by scrub.
>
> I'm afraid there is some bug in data balance code which leads to this.
> We will dig further to pin down the cause.
>
> BTW, does that problem always show up at block group 253562454016? Or it
> randomly shows up at different block groups?
>
> Thanks,
> Qu

Hi Qu,
It appears to be the same data blocks giving me this issue. I have tried it 
while mounting from a bootable usb and in my arch linux install both with 
similar results. I should also mention I have a btrfs root partition within a 
luks lvm partition becuase btrfs provides no encryption. I'm not sure this 
makes a difference. Would you suggest running somekind of disk check or by 
running scrub? Please let me know what I should try to resolve this.
谢谢

``` BEGIN DEBUG INFO ```
ERROR: error during balancing '/': Input/output error
[michael@linux /]$ dmesg | tail
[  462.460626] audit: type=1104 audit(1577525576.905:124): pid=3415 uid=0 
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env 
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 
res=success'
[  465.880807] BTRFS info (device dm-1): found 25 extents
[  470.501319] BTRFS info (device dm-1): found 25 extents
[  471.713073] BTRFS info (device dm-1): relocating block group 253562454016 
flags data
[  472.251356] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 
1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
[  472.254390] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 
1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1
[  472.716215] BTRFS info (device dm-1): balance: ended with status: -5
[  473.122721] audit: type=1106 audit(1577525587.565:125): pid=2775 uid=0 
auid=1000 ses=1 msg='op=PAM:session_close 
grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" 
hostname=? addr=? terminal=/dev/pts/3 res=success'
[  473.122840] audit: type=1104 audit(1577525587.565:126): pid=2775 uid=0 
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit acct="root" 
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[  629.160619] audit: type=1131 audit(1577525743.603:127): pid=1 uid=0 
auid=4294967295 ses=4294967295 msg='unit=packagekit comm="systemd" exe="/usr/
lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
``` END DEBUG INFO ```

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2019-12-28 12:28 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-12-27 12:39 Error during balancing '/': Input/output error Michael Ruiz
2019-12-27 13:21 ` Qu Wenruo
     [not found]   ` <5555870.lOV4Wx5bFT@archlinux>
2019-12-28 10:09     ` Qu Wenruo
2019-12-28 11:26       ` Michael Ruiz
2019-12-28 11:38         ` Qu Wenruo
2019-12-28 11:45           ` Michael Ruiz
2019-12-28 12:05             ` Qu Wenruo
2019-12-28 12:28               ` Michael Ruiz
2019-12-28  9:53 ` Michael Ruiz

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox