* Kernel bug during RAID1 replace @ 2016-06-27 21:36 Saint Germain 2016-06-27 21:42 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-27 21:36 UTC (permalink / raw) To: linux-btrfs Hello, I am on Debian Jessie with a kernel from backports: 4.6.0-0.bpo.1-amd64 I am also using btrfs-tools 4.4.1-1.1~bpo8+1 When trying to replace a RAID1 drive (with btrfs replace start -f /dev/sda1 /dev/sdd1), the operation is cancelled after completing only 5%. I got this error in the /var/log/syslog: ------------[ cut here ]------------ WARNING: CPU: 2 PID: 2617 at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/dev-replace.c:430 btrfs_dev_replace_start+0x2be/0x400 [btrfs] Modules linked in: uas(E) usb_storage(E) bnep(E) ftdi_sio(E) usbserial(E) snd_hda_codec_hdmi(E) nls_utf8(E) nls_cp437(E) vfat(E) fat(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) irqbypass(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) hmac(E) drbg(E) ansi_cprng(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) wl(POE) btusb(E) btrtl(E) btbcm(E) btintel(E) cfg80211(E) bluetooth(E) efi_pstore(E) snd_hda_codec_realtek(E) evdev(E) crc16(E) serio_raw(E) pcspkr(E) efivars(E) joydev(E) snd_hda_codec_generic(E) rfkill(E) snd_hda_intel(E) nuvoton_cir(E) rc_core(E) snd_hda_codec(E) i915(E) battery(E) snd_hda_core(E) snd_hwdep(E) soc_button_array(E) tpm_tis(E) drm_kms_helper(E) intel_smartconnect(E) snd_pcm(E) tpm(E) video(E) i2c_i801(E) snd_timer(E) drm(E) snd(E) lpc_ich(E) i2c_algo_bit(E) soundcore(E) mfd_core(E) mei_me(E) processor(E) button(E) mei(E) shpchp(E) fuse(E) autofs4(E) hid_logitech_hidpp(E) btrfs(E) hid_logitech_dj(E) usbhid(E) hid(E) xor(E) raid6_pq(E) sg(E) sr_mod(E) cdrom(E) sd_mod(E) crc32c_intel(E) ahci(E) libahci(E) libata(E) psmouse(E) scsi_mod(E) xhci_pci(E) ehci_pci(E) xhci_hcd(E) ehci_hcd(E) e1000e(E) usbcore(E) ptp(E) pps_core(E) usb_common(E) fjes(E) CPU: 2 PID: 2617 Comm: btrfs Tainted: P OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87E-ITX, BIOS P2.10 10/04/2013 0000000000000286 00000000f0ba7fe7 ffffffff813123c5 0000000000000000 0000000000000000 ffffffff8107af94 ffff880186caf000 00000000fffffffb ffff8800c76b0800 ffff8800cae70000 ffff8800cae70ee0 00007ffdd5397d98 Call Trace: [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 [<ffffffff8107af94>] ? __warn+0xc4/0xe0 [<ffffffffc030990e>] ? btrfs_dev_replace_start+0x2be/0x400 [btrfs] [<ffffffffc02d24f2>] ? btrfs_ioctl+0x1d42/0x2190 [btrfs] [<ffffffff811a8e2d>] ? handle_mm_fault+0x154d/0x1cb0 [<ffffffff81205379>] ? do_vfs_ioctl+0x99/0x5d0 [<ffffffff81205926>] ? SyS_ioctl+0x76/0x90 [<ffffffff815c83b6>] ? system_call_fast_compare_end+0xc/0x96 ---[ end trace 9fbfaa137cc5a72a ]--- What should I do to replace correctly my drive ? Thanks in advance, ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 21:36 Kernel bug during RAID1 replace Saint Germain @ 2016-06-27 21:42 ` Chris Murphy 2016-06-27 22:26 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-27 21:42 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Mon, Jun 27, 2016 at 3:36 PM, Saint Germain <saintger@gmail.com> wrote: > Hello, > > I am on Debian Jessie with a kernel from backports: > 4.6.0-0.bpo.1-amd64 > > I am also using btrfs-tools 4.4.1-1.1~bpo8+1 > > When trying to replace a RAID1 drive (with btrfs replace start > -f /dev/sda1 /dev/sdd1), the operation is cancelled after completing > only 5%. > > I got this error in the /var/log/syslog: > ------------[ cut here ]------------ > WARNING: CPU: 2 PID: 2617 at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/dev-replace.c:430 btrfs_dev_replace_start+0x2be/0x400 [btrfs] > Modules linked in: uas(E) usb_storage(E) bnep(E) ftdi_sio(E) usbserial(E) snd_hda_codec_hdmi(E) nls_utf8(E) nls_cp437(E) vfat(E) fat(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) irqbypass(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) hmac(E) drbg(E) ansi_cprng(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) wl(POE) btusb(E) btrtl(E) btbcm(E) btintel(E) cfg80211(E) bluetooth(E) efi_pstore(E) snd_hda_codec_realtek(E) evdev(E) crc16(E) serio_raw(E) pcspkr(E) efivars(E) joydev(E) snd_hda_codec_generic(E) rfkill(E) snd_hda_intel(E) nuvoton_cir(E) rc_core(E) snd_hda_codec(E) i915(E) battery(E) snd_hda_core(E) snd_hwdep(E) soc_button_array(E) tpm_tis(E) drm_kms_helper(E) intel_smartconnect(E) snd_pcm(E) tpm(E) video(E) i2c_i801(E) snd_timer(E) drm(E) snd(E) lpc_ich(E) i2c_algo_bit(E) soundcore(E) mfd_core(E) mei_me(E) processor(E) button(E) mei(E) shpchp(E) fuse(E) autofs4(E) hid_logitech_hidpp(E) btrfs(E) hid_logitech_dj(E) usbhid(E) hid(E) xor(E) raid6_pq(E) sg(E) sr_mod(E) cdrom(E) sd_mod(E) crc32c_intel(E) ahci(E) libahci(E) libata(E) psmouse(E) scsi_mod(E) xhci_pci(E) ehci_pci(E) xhci_hcd(E) ehci_hcd(E) e1000e(E) usbcore(E) ptp(E) pps_core(E) usb_common(E) fjes(E) > CPU: 2 PID: 2617 Comm: btrfs Tainted: P OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 > Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87E-ITX, BIOS P2.10 10/04/2013 > 0000000000000286 00000000f0ba7fe7 ffffffff813123c5 0000000000000000 > 0000000000000000 ffffffff8107af94 ffff880186caf000 00000000fffffffb > ffff8800c76b0800 ffff8800cae70000 ffff8800cae70ee0 00007ffdd5397d98 > Call Trace: > [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 > [<ffffffff8107af94>] ? __warn+0xc4/0xe0 > [<ffffffffc030990e>] ? btrfs_dev_replace_start+0x2be/0x400 [btrfs] > [<ffffffffc02d24f2>] ? btrfs_ioctl+0x1d42/0x2190 [btrfs] > [<ffffffff811a8e2d>] ? handle_mm_fault+0x154d/0x1cb0 > [<ffffffff81205379>] ? do_vfs_ioctl+0x99/0x5d0 > [<ffffffff81205926>] ? SyS_ioctl+0x76/0x90 > [<ffffffff815c83b6>] ? system_call_fast_compare_end+0xc/0x96 > ---[ end trace 9fbfaa137cc5a72a ]--- > > > > What should I do to replace correctly my drive ? I don't often see handle_mm_fault with btrfs problems, maybe the entire dmesg from mounting the fs and including btrfs replace would reveal a related problem that instigates the failure? If the device being replaced is acting unreliably, then you'd want to use -r with replace to ignore that device unless it's absolutely necessary to read from it. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 21:42 ` Chris Murphy @ 2016-06-27 22:26 ` Saint Germain 2016-06-27 22:55 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-27 22:26 UTC (permalink / raw) To: Btrfs BTRFS On Mon, 27 Jun 2016 15:42:42 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Mon, Jun 27, 2016 at 3:36 PM, Saint Germain <saintger@gmail.com> > wrote: > > Hello, > > > > I am on Debian Jessie with a kernel from backports: > > 4.6.0-0.bpo.1-amd64 > > > > I am also using btrfs-tools 4.4.1-1.1~bpo8+1 > > > > When trying to replace a RAID1 drive (with btrfs replace start > > -f /dev/sda1 /dev/sdd1), the operation is cancelled after completing > > only 5%. > > > > I got this error in the /var/log/syslog: > > ------------[ cut here ]------------ > > WARNING: CPU: 2 PID: 2617 > > at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/dev-replace.c:430 > > btrfs_dev_replace_start+0x2be/0x400 [btrfs] Modules linked in: > > uas(E) usb_storage(E) bnep(E) ftdi_sio(E) usbserial(E) > > snd_hda_codec_hdmi(E) nls_utf8(E) nls_cp437(E) vfat(E) fat(E) > > intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) > > coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) irqbypass(E) > > iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) > > ghash_clmulni_intel(E) hmac(E) drbg(E) ansi_cprng(E) aesni_intel(E) > > aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) > > cryptd(E) wl(POE) btusb(E) btrtl(E) btbcm(E) btintel(E) cfg80211(E) > > bluetooth(E) efi_pstore(E) snd_hda_codec_realtek(E) evdev(E) > > crc16(E) serio_raw(E) pcspkr(E) efivars(E) joydev(E) > > snd_hda_codec_generic(E) rfkill(E) snd_hda_intel(E) nuvoton_cir(E) > > rc_core(E) snd_hda_codec(E) i915(E) battery(E) snd_hda_core(E) > > snd_hwdep(E) soc_button_array(E) tpm_tis(E) drm_kms_helper(E) > > intel_smartconnect(E) snd_pcm(E) tpm(E) video(E) i2c_i801(E) > > snd_timer(E) drm(E) snd(E) lpc_ich(E) i2c_algo_bit(E) soundcore(E) > > mfd_core(E) mei_me(E) processor(E) button(E) mei(E) shpchp(E) > > fuse(E) autofs4(E) hid_logitech_hidpp(E) btrfs(E) > > hid_logitech_dj(E) usbhid(E) hid(E) xor(E) raid6_pq(E) sg(E) > > sr_mod(E) cdrom(E) sd_mod(E) crc32c_intel(E) ahci(E) libahci(E) > > libata(E) psmouse(E) scsi_mod(E) xhci_pci(E) ehci_pci(E) > > xhci_hcd(E) ehci_hcd(E) e1000e(E) usbcore(E) ptp(E) pps_core(E) > > usb_common(E) fjes(E) CPU: 2 PID: 2617 Comm: btrfs Tainted: > > P OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 > > Hardware name: To Be Filled By O.E.M. To Be Filled By > > O.E.M./Z87E-ITX, BIOS P2.10 10/04/2013 0000000000000286 > > 00000000f0ba7fe7 ffffffff813123c5 0000000000000000 0000000000000000 > > ffffffff8107af94 ffff880186caf000 00000000fffffffb ffff8800c76b0800 > > ffff8800cae70000 ffff8800cae70ee0 00007ffdd5397d98 Call Trace: > > [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 [<ffffffff8107af94>] ? > > __warn+0xc4/0xe0 [<ffffffffc030990e>] ? > > btrfs_dev_replace_start+0x2be/0x400 [btrfs] [<ffffffffc02d24f2>] ? > > btrfs_ioctl+0x1d42/0x2190 [btrfs] [<ffffffff811a8e2d>] ? > > handle_mm_fault+0x154d/0x1cb0 [<ffffffff81205379>] ? > > do_vfs_ioctl+0x99/0x5d0 [<ffffffff81205926>] ? SyS_ioctl+0x76/0x90 > > [<ffffffff815c83b6>] ? system_call_fast_compare_end+0xc/0x96 > > ---[ end trace 9fbfaa137cc5a72a ]--- > > > > > > > > What should I do to replace correctly my drive ? > > I don't often see handle_mm_fault with btrfs problems, maybe the > entire dmesg from mounting the fs and including btrfs replace would > reveal a related problem that instigates the failure? > > If the device being replaced is acting unreliably, then you'd want to > use -r with replace to ignore that device unless it's absolutely > necessary to read from it. > Thanks for your help. Ok here is the log from the mounting, and including btrfs replace (btrfs replace start -f /dev/sda1 /dev/sdd1 /home): BTRFS info (device sdb1): disk space caching is enabled BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705631, gen 1335 BTRFS info (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14220, gen 24 BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) to /dev/sdd1 started scrub_handle_errored_block: 166 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) btrfs_dev_stat_print_on_error: 166 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14221, gen 24 scrub_handle_errored_block: 166 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sda1 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14222, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445259264 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349810688 on dev /dev/sda1, sector 140429952, root 5, inode 4265283, offset 0, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14223, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349810688 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349814784 on dev /dev/sda1, sector 140429960, root 5, inode 4265283, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14224, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349814784 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349818880 on dev /dev/sda1, sector 140429968, root 5, inode 4265283, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14225, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349818880 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349822976 on dev /dev/sda1, sector 140429976, root 5, inode 4265283, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14226, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349822976 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349827072 on dev /dev/sda1, sector 140429984, root 5, inode 4265283, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14227, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349827072 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349831168 on dev /dev/sda1, sector 140429992, root 5, inode 4265283, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14228, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349831168 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349835264 on dev /dev/sda1, sector 140430000, root 5, inode 4265283, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14229, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349835264 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349839360 on dev /dev/sda1, sector 140430008, root 5, inode 4265283, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14230, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349839360 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349843456 on dev /dev/sda1, sector 140430016, root 5, inode 4265283, offset 32768, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14231, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349843456 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349847552 on dev /dev/sda1, sector 140430024, root 5, inode 4265283, offset 36864, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14232, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349847552 on dev /dev/sda1 scrub_handle_errored_block: 164 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 139677270016 on dev /dev/sda1, sector 146928896, root 5, inode 4265287, offset 0, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) btrfs_dev_stat_print_on_error: 164 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14397, gen 24 scrub_handle_errored_block: 164 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677270016 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677274112 on dev /dev/sda1, sector 146928904, root 5, inode 4265287, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14398, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677274112 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677278208 on dev /dev/sda1, sector 146928912, root 5, inode 4265287, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14399, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677278208 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677282304 on dev /dev/sda1, sector 146928920, root 5, inode 4265287, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14400, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677282304 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677286400 on dev /dev/sda1, sector 146928928, root 5, inode 4265287, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14401, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677286400 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677290496 on dev /dev/sda1, sector 146928936, root 5, inode 4265287, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14402, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677290496 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677294592 on dev /dev/sda1, sector 146928944, root 5, inode 4265287, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14403, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677294592 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677298688 on dev /dev/sda1, sector 146928952, root 5, inode 4265287, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14404, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677298688 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677302784 on dev /dev/sda1, sector 146928960, root 5, inode 4265287, offset 32768, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14405, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677302784 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677306880 on dev /dev/sda1, sector 146928968, root 5, inode 4265287, offset 36864, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14406, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677306880 on dev /dev/sda1 scrub_handle_errored_block: 434 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 140475064320 on dev /dev/sda1, sector 148487088, root 5, inode 4265291, offset 0, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) btrfs_dev_stat_print_on_error: 434 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14841, gen 24 scrub_handle_errored_block: 434 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475064320 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475068416 on dev /dev/sda1, sector 148487096, root 5, inode 4265291, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14842, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475068416 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475072512 on dev /dev/sda1, sector 148487104, root 5, inode 4265291, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14843, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475072512 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475076608 on dev /dev/sda1, sector 148487112, root 5, inode 4265291, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14844, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475076608 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475080704 on dev /dev/sda1, sector 148487120, root 5, inode 4265291, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14845, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475080704 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475084800 on dev /dev/sda1, sector 148487128, root 5, inode 4265291, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14846, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475084800 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475088896 on dev /dev/sda1, sector 148487136, root 5, inode 4265291, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14847, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475088896 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475092992 on dev /dev/sda1, sector 148487144, root 5, inode 4265291, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14848, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475092992 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475097088 on dev /dev/sda1, sector 148487152, root 5, inode 4265291, offset 32768, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14849, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475097088 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475101184 on dev /dev/sda1, sector 148487160, root 5, inode 4265291, offset 36864, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14850, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475101184 on dev /dev/sda1 scrub_handle_errored_block: 310 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 162995269632 on dev /dev/sda1, sector 192471864, root 5, inode 4265299, offset 0, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) btrfs_dev_stat_print_on_error: 310 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15161, gen 24 scrub_handle_errored_block: 310 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995269632 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995273728 on dev /dev/sda1, sector 192471872, root 5, inode 4265299, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15162, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995273728 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995277824 on dev /dev/sda1, sector 192471880, root 5, inode 4265299, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15163, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995277824 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995281920 on dev /dev/sda1, sector 192471888, root 5, inode 4265299, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15164, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995281920 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995286016 on dev /dev/sda1, sector 192471896, root 5, inode 4265299, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15165, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995286016 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995290112 on dev /dev/sda1, sector 192471904, root 5, inode 4265299, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15166, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995290112 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995294208 on dev /dev/sda1, sector 192471912, root 5, inode 4265299, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15167, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995294208 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995298304 on dev /dev/sda1, sector 192471920, root 5, inode 4265299, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15168, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995298304 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995302400 on dev /dev/sda1, sector 192471928, root 5, inode 4265299, offset 32768, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15169, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995302400 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995306496 on dev /dev/sda1, sector 192471936, root 5, inode 4265299, offset 36864, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15170, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995306496 on dev /dev/sda1 scrub_handle_errored_block: 434 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 170607226880 on dev /dev/sda1, sector 207338968, root 5, inode 4265298, offset 450560, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) btrfs_dev_stat_print_on_error: 434 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15605, gen 24 scrub_handle_errored_block: 434 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607226880 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607230976 on dev /dev/sda1, sector 207338976, root 5, inode 4265298, offset 454656, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15606, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607230976 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607235072 on dev /dev/sda1, sector 207338984, root 5, inode 4265298, offset 458752, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15607, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607235072 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607239168 on dev /dev/sda1, sector 207338992, root 5, inode 4265298, offset 462848, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15608, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607239168 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607243264 on dev /dev/sda1, sector 207339000, root 5, inode 4265298, offset 466944, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15609, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607243264 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607247360 on dev /dev/sda1, sector 207339008, root 5, inode 4265298, offset 471040, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15610, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607247360 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607251456 on dev /dev/sda1, sector 207339016, root 5, inode 4265298, offset 475136, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15611, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607251456 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607255552 on dev /dev/sda1, sector 207339024, root 5, inode 4265298, offset 479232, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15612, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607255552 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607259648 on dev /dev/sda1, sector 207339032, root 5, inode 4265298, offset 483328, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15613, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607259648 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170607263744 on dev /dev/sda1, sector 207339040, root 5, inode 4265298, offset 487424, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15614, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607263744 on dev /dev/sda1 scrub_handle_errored_block: 281 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 171216498688 on dev /dev/sda1, sector 208528952, root 5, inode 4265266, offset 655360, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) btrfs_dev_stat_print_on_error: 281 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15896, gen 24 scrub_handle_errored_block: 281 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216498688 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216502784 on dev /dev/sda1, sector 208528960, root 5, inode 4265266, offset 659456, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15897, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216502784 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216506880 on dev /dev/sda1, sector 208528968, root 5, inode 4265266, offset 663552, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15898, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216506880 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216510976 on dev /dev/sda1, sector 208528976, root 5, inode 4265266, offset 667648, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15899, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216510976 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216515072 on dev /dev/sda1, sector 208528984, root 5, inode 4265266, offset 671744, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15900, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216515072 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216519168 on dev /dev/sda1, sector 208528992, root 5, inode 4265266, offset 675840, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15901, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216519168 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216523264 on dev /dev/sda1, sector 208529000, root 5, inode 4265266, offset 679936, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15902, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216523264 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216527360 on dev /dev/sda1, sector 208529008, root 5, inode 4265266, offset 684032, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15903, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216527360 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216531456 on dev /dev/sda1, sector 208529016, root 5, inode 4265266, offset 688128, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15904, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216531456 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171216535552 on dev /dev/sda1, sector 208529024, root 5, inode 4265266, offset 692224, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 15905, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216535552 on dev /dev/sda1 scrub_handle_errored_block: 311 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 182510542848 on dev /dev/sdb1, sector 228484336, root 5, inode 4265300, offset 0, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) btrfs_dev_stat_print_on_error: 311 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705632, gen 1335 scrub_handle_errored_block: 311 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510542848 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510546944 on dev /dev/sdb1, sector 228484344, root 5, inode 4265300, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705633, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510546944 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510551040 on dev /dev/sdb1, sector 228484352, root 5, inode 4265300, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705634, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510551040 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510555136 on dev /dev/sdb1, sector 228484360, root 5, inode 4265300, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705635, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510555136 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510559232 on dev /dev/sdb1, sector 228484368, root 5, inode 4265300, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705636, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510559232 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510563328 on dev /dev/sdb1, sector 228484376, root 5, inode 4265300, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705637, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510563328 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510567424 on dev /dev/sdb1, sector 228484384, root 5, inode 4265300, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705638, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510567424 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510571520 on dev /dev/sdb1, sector 228484392, root 5, inode 4265300, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705639, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510571520 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510575616 on dev /dev/sdb1, sector 228484400, root 5, inode 4265300, offset 32768, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705640, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510575616 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510579712 on dev /dev/sdb1, sector 228484408, root 5, inode 4265300, offset 36864, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705641, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510579712 on dev /dev/sdb1 scrub_handle_errored_block: 345 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 198160945152 on dev /dev/sda1, sector 261154824, root 5, inode 4265307, offset 2592768, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) btrfs_dev_stat_print_on_error: 345 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16217, gen 24 ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0 ata2.00: irq_stat 0x40000008 ata2.00: failed command: READ FPDMA QUEUED ata2.00: cmd 60/08:70:08:d8:70/00:00:0f:00:00/40 tag 14 ncq 4096 in res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> ata2.00: status: { DRDY ERR } ata2.00: error: { UNC } ata2.00: configured for UDMA/133 sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] [descriptor] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 blk_update_request: I/O error, dev sdb, sector 259053576 scrub_handle_errored_block: 345 callbacks suppressed ata2: EH complete BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160945152 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160949248 on dev /dev/sda1, sector 261154832, root 5, inode 4265307, offset 2596864, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16218, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160949248 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160953344 on dev /dev/sda1, sector 261154840, root 5, inode 4265307, offset 2600960, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16219, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160953344 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160957440 on dev /dev/sda1, sector 261154848, root 5, inode 4265307, offset 2605056, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16220, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160957440 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160961536 on dev /dev/sda1, sector 261154856, root 5, inode 4265307, offset 2609152, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16221, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160961536 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160965632 on dev /dev/sda1, sector 261154864, root 5, inode 4265307, offset 2613248, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16222, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160965632 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160969728 on dev /dev/sda1, sector 261154872, root 5, inode 4265307, offset 2617344, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16223, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160969728 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160973824 on dev /dev/sda1, sector 261154880, root 5, inode 4265307, offset 2621440, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16224, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160973824 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160977920 on dev /dev/sda1, sector 261154888, root 5, inode 4265307, offset 2625536, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16225, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160977920 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160982016 on dev /dev/sda1, sector 261154896, root 5, inode 4265307, offset 2629632, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16226, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160982016 on dev /dev/sda1 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 BTRFS error (device sdb1): btrfs_scrub_dev(/dev/sda1, 1, /dev/sdd1) failed -5 ------------[ cut here ]------------ WARNING: CPU: 0 PID: 1143 at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/dev-replace.c:430 btrfs_dev_replace_start+0x2be/0x400 [btrfs] Modules linked in: uas(E) usb_storage(E) bnep(E) ftdi_sio(E) usbserial(E) snd_hda_codec_hdmi(E) nls_utf8(E) nls_cp437(E) vfat(E) fat(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) irqbypass(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) hmac(E) drbg(E) ansi_cprng(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) wl(POE) btusb(E) btrtl(E) btbcm(E) btintel(E) cfg80211(E) bluetooth(E) efi_pstore(E) snd_hda_codec_realtek(E) evdev(E) crc16(E) serio_raw(E) pcspkr(E) efivars(E) joydev(E) snd_hda_codec_generic(E) rfkill(E) snd_hda_intel(E) nuvoton_cir(E) rc_core(E) snd_hda_codec(E) i915(E) battery(E) snd_hda_core(E) snd_hwdep(E) soc_button_array(E) tpm_tis(E) drm_kms_helper(E) intel_smartconnect(E) snd_pcm(E) tpm(E) video(E) i2c_i801(E) snd_timer(E) drm(E) snd(E) lpc_ich(E) i2c_algo_bit(E) soundcore(E) mfd_core(E) mei_me(E) processor(E) button(E) mei(E) shpchp(E) fuse(E) autofs4(E) hid_logitech_hidpp(E) btrfs(E) hid_logitech_dj(E) usbhid(E) hid(E) xor(E) raid6_pq(E) sg(E) sr_mod(E) cdrom(E) sd_mod(E) crc32c_intel(E) ahci(E) libahci(E) libata(E) psmouse(E) scsi_mod(E) xhci_pci(E) ehci_pci(E) xhci_hcd(E) ehci_hcd(E) e1000e(E) usbcore(E) ptp(E) pps_core(E) usb_common(E) fjes(E) CPU: 0 PID: 1143 Comm: btrfs Tainted: P W OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87E-ITX, BIOS P2.10 10/04/2013 0000000000000286 000000002f7c2f6a ffffffff813123c5 0000000000000000 0000000000000000 ffffffff8107af94 ffff880176f5d000 00000000fffffffb ffff8801937a6000 ffff880176f10000 ffff880176f10ee0 00007ffe88dfa8f8 Call Trace: [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 [<ffffffff8107af94>] ? __warn+0xc4/0xe0 [<ffffffffc030990e>] ? btrfs_dev_replace_start+0x2be/0x400 [btrfs] [<ffffffffc02d24f2>] ? btrfs_ioctl+0x1d42/0x2190 [btrfs] [<ffffffff811a8e2d>] ? handle_mm_fault+0x154d/0x1cb0 [<ffffffff81205379>] ? do_vfs_ioctl+0x99/0x5d0 [<ffffffff81205926>] ? SyS_ioctl+0x76/0x90 [<ffffffff815c83b6>] ? system_call_fast_compare_end+0xc/0x96 ---[ end trace 9fbfaa137cc5a72b ]--- ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 22:26 ` Saint Germain @ 2016-06-27 22:55 ` Chris Murphy 2016-06-27 22:58 ` Chris Murphy 2016-06-27 23:03 ` Saint Germain 0 siblings, 2 replies; 27+ messages in thread From: Chris Murphy @ 2016-06-27 22:55 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Mon, Jun 27, 2016 at 4:26 PM, Saint Germain <saintger@gmail.com> wrote: >> > > Thanks for your help. > > Ok here is the log from the mounting, and including btrfs replace > (btrfs replace start -f /dev/sda1 /dev/sdd1 /home): > > BTRFS info (device sdb1): disk space caching is enabled > BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, flush 7928, corrupt 1705631, gen 1335 > BTRFS info (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14220, gen 24 Eek. So sdb has 11+ million write errors, flush errors, read errors, and over 1 million corruptions. It's dying or dead. And sda has a dozen thousand+ corruptions. This isn't a good combination, as you have two devices with problems and raid5 only protects you from one device with problems. You were in the process of replacing sda, which is good, but it may not be enough... > BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) to /dev/sdd1 started > scrub_handle_errored_block: 166 callbacks suppressed > BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) > btrfs_dev_stat_print_on_error: 166 callbacks suppressed > BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14221, gen 24 > scrub_handle_errored_block: 166 callbacks suppressed > BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sda1 Shoot. You have a lot of these. It looks suspiciously like you're hitting a case list regulars are only just starting to understand (somewhat) where it's possible to have a legit corrupt sector that Btrfs detects during scrub as wrong, fixes it from parity, but then occasionally wrongly overwrites the parity with bad parity. This doesn't cause an immediately recognizable problem. But if the volume becomes degraded later, Btrfs must use parity to reconstruct on-the-fly and if it hits one of these bad parities, the reconstruction is bad, and ends up causing lots of these checksum errors. We can tell it's not metadata corruption because a.) there's a file listed as being affected and b.) the file system doesn't fail and go read only. But still it means those files are likely toast... [...snip many instances of checksum errors...] > BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 16217, gen 24 > ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0 > ata2.00: irq_stat 0x40000008 > ata2.00: failed command: READ FPDMA QUEUED > ata2.00: cmd 60/08:70:08:d8:70/00:00:0f:00:00/40 tag 14 ncq 4096 in > res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> > ata2.00: status: { DRDY ERR } > ata2.00: error: { UNC } > ata2.00: configured for UDMA/133 > sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] [descriptor] > sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed > sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 > blk_update_request: I/O error, dev sdb, sector 259053576 OK yeah so bad sector on sdb. So you have two failures because sda is already giving you trouble while being replaced and on top of it you now get a 2nd (partial) failure via bad sectors. So rather urgently I think you need to copy things off this volume if you don't already have a backup so you can save as much as possible. Don't write to the drives. You might even consider 'mount -o remount,ro' to avoid anything writing to the volume. Copy the most important data first, triage time. While that happens you can safely collect some more information: btrfs fi us <mp> smartctl -x <dev> ## for both drives -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 22:55 ` Chris Murphy @ 2016-06-27 22:58 ` Chris Murphy 2016-06-27 23:06 ` Saint Germain 2016-06-27 23:03 ` Saint Germain 1 sibling, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-27 22:58 UTC (permalink / raw) To: Chris Murphy; +Cc: Saint Germain, Btrfs BTRFS On Mon, Jun 27, 2016 at 4:55 PM, Chris Murphy <lists@colorremedies.com> wrote: >> BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) to /dev/sdd1 started >> scrub_handle_errored_block: 166 callbacks suppressed >> BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) >> btrfs_dev_stat_print_on_error: 166 callbacks suppressed >> BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14221, gen 24 >> scrub_handle_errored_block: 166 callbacks suppressed >> BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sda1 > > Shoot. You have a lot of these. It looks suspiciously like you're > hitting a case list regulars are only just starting to understand Forget this part completely. It doesn't affect raid1. I just re-read that your setup is not raid1, I don't know why I thought it was raid5. The likely issue here is that you've got legit corruptions on sda (mix of slow and flat out bad sectors), as well as a failing drive. This is also safe to issue: smartctl -l scterc /dev/sda smartctl -l scterc /dev/sdb cat /sys/block/sda/device/timeout cat /sys/block/sdb/device/timeout -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 22:58 ` Chris Murphy @ 2016-06-27 23:06 ` Saint Germain 2016-06-28 0:00 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-27 23:06 UTC (permalink / raw) To: Btrfs BTRFS On Mon, 27 Jun 2016 16:58:37 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Mon, Jun 27, 2016 at 4:55 PM, Chris Murphy > <lists@colorremedies.com> wrote: > > >> BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) > >> to /dev/sdd1 started scrub_handle_errored_block: 166 callbacks > >> suppressed BTRFS warning (device sdb1): checksum error at logical > >> 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode > >> 3434831, offset 479232, length 4096, links 1 (path: > >> user/.local/share/zeitgeist/activity.sqlite-wal) > >> btrfs_dev_stat_print_on_error: 166 callbacks suppressed BTRFS > >> error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, > >> corrupt 14221, gen 24 scrub_handle_errored_block: 166 callbacks > >> suppressed BTRFS error (device sdb1): unable to fixup (regular) > >> error at logical 93445255168 on dev /dev/sda1 > > > > Shoot. You have a lot of these. It looks suspiciously like you're > > hitting a case list regulars are only just starting to understand > > Forget this part completely. It doesn't affect raid1. I just re-read > that your setup is not raid1, I don't know why I thought it was raid5. > > The likely issue here is that you've got legit corruptions on sda (mix > of slow and flat out bad sectors), as well as a failing drive. > > This is also safe to issue: > > smartctl -l scterc /dev/sda > smartctl -l scterc /dev/sdb > cat /sys/block/sda/device/timeout > cat /sys/block/sdb/device/timeout > My setup is indeed RAID1 (and not RAID5) root@system:/# smartctl -l scterc /dev/sda smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: Disabled Write: Disabled root@system:/# smartctl -l scterc /dev/sdb smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: Disabled Write: Disabled root@system:/# cat /sys/block/sda/device/timeout 30 root@system:/# cat /sys/block/sdb/device/timeout 30 ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 23:06 ` Saint Germain @ 2016-06-28 0:00 ` Chris Murphy 2016-06-28 0:10 ` Chris Murphy 2016-06-28 0:49 ` Saint Germain 0 siblings, 2 replies; 27+ messages in thread From: Chris Murphy @ 2016-06-28 0:00 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Mon, Jun 27, 2016 at 5:06 PM, Saint Germain <saintger@gmail.com> wrote: > On Mon, 27 Jun 2016 16:58:37 -0600, Chris Murphy > <lists@colorremedies.com> wrote : > >> On Mon, Jun 27, 2016 at 4:55 PM, Chris Murphy >> <lists@colorremedies.com> wrote: >> >> >> BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) >> >> to /dev/sdd1 started scrub_handle_errored_block: 166 callbacks >> >> suppressed BTRFS warning (device sdb1): checksum error at logical >> >> 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode >> >> 3434831, offset 479232, length 4096, links 1 (path: >> >> user/.local/share/zeitgeist/activity.sqlite-wal) >> >> btrfs_dev_stat_print_on_error: 166 callbacks suppressed BTRFS >> >> error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, >> >> corrupt 14221, gen 24 scrub_handle_errored_block: 166 callbacks >> >> suppressed BTRFS error (device sdb1): unable to fixup (regular) >> >> error at logical 93445255168 on dev /dev/sda1 >> > >> > Shoot. You have a lot of these. It looks suspiciously like you're >> > hitting a case list regulars are only just starting to understand >> >> Forget this part completely. It doesn't affect raid1. I just re-read >> that your setup is not raid1, I don't know why I thought it was raid5. >> >> The likely issue here is that you've got legit corruptions on sda (mix >> of slow and flat out bad sectors), as well as a failing drive. >> >> This is also safe to issue: >> >> smartctl -l scterc /dev/sda >> smartctl -l scterc /dev/sdb >> cat /sys/block/sda/device/timeout >> cat /sys/block/sdb/device/timeout >> > > My setup is indeed RAID1 (and not RAID5) > > root@system:/# smartctl -l scterc /dev/sda > smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local > build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, > www.smartmontools.org > > SCT Error Recovery Control: > Read: Disabled > Write: Disabled > > root@system:/# smartctl -l scterc /dev/sdb > smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local > build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, > www.smartmontools.org > > SCT Error Recovery Control: > Read: Disabled > Write: Disabled > > root@system:/# cat /sys/block/sda/device/timeout > 30 > root@system:/# cat /sys/block/sdb/device/timeout > 30 Good news and bad news. The bad news is this is a significant misconfiguration, it's very common, and it means that any bad sectors that don't result in read errors before 30 seconds will mean they don't get fixed by Btrfs (or even mdadm or LVM raid). So they can accumulate. There are two options since your drives support SCT ERC. 1. smartctl -l scterc,70,70 /dev/sdX ## done for both drives That will make sure the drive reports a read error in 7 seconds, well under the kernel's command timer of 7 seconds. This is how your drives should normally be configured for RAID usage. 2. echo 180 > /sys/block/sda/device/timeout echo 180 > /sys/block/sdb/device/timeout This *might* actually work better in your case. If you permit the drives to have really long error recovery, it might actually allow the data to be returned to Btrfs and then it can start fixing problems. Maybe. It's a long shot. And there will be upwards of 3 minute hangs. I would give this a shot first. You can issue these commands safely at any time, no umount is needed or anything like that. I would do this even before using cp/rsync or ddrescue because it increases the chance the drive can recover data from these bad sectors and fix the other drive. These settings are not persistent across a reboot unless you set a udev rule or equivalent. On one of my drives that supports SCT ERC it only accepts the smartctl -l command to set the timeout once. I can't change it without power cycling the drive or it just crashes (yay firmware bugs). Just FYI it's possible to run into other weirdness. Last, I have no idea if the massive Btrfs write errors on sda are from an earlier problem where the drive data or power cable got jiggled or was otherwise absent temporarily? So depending on how the block timeout change affects your data recovery, you might end up needing to do a reboot to get back to a more stable state for all of this? It really should be able to fix things *if* at least one copy can be read and then written to the other drive. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-28 0:00 ` Chris Murphy @ 2016-06-28 0:10 ` Chris Murphy 2016-06-28 0:49 ` Saint Germain 1 sibling, 0 replies; 27+ messages in thread From: Chris Murphy @ 2016-06-28 0:10 UTC (permalink / raw) To: Chris Murphy; +Cc: Saint Germain, Btrfs BTRFS On Mon, Jun 27, 2016 at 6:00 PM, Chris Murphy <lists@colorremedies.com> wrote: > There are two options since your drives support SCT ERC. > > 1. > smartctl -l scterc,70,70 /dev/sdX ## done for both drives > > That will make sure the drive reports a read error in 7 seconds, well > under the kernel's command timer of 7 seconds. correction: "well under the kernel's command timer default of 30 seconds" -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-28 0:00 ` Chris Murphy 2016-06-28 0:10 ` Chris Murphy @ 2016-06-28 0:49 ` Saint Germain 2016-06-28 2:14 ` Chris Murphy 1 sibling, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-28 0:49 UTC (permalink / raw) Cc: Btrfs BTRFS On Mon, 27 Jun 2016 18:00:34 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Mon, Jun 27, 2016 at 5:06 PM, Saint Germain <saintger@gmail.com> > wrote: > > On Mon, 27 Jun 2016 16:58:37 -0600, Chris Murphy > > <lists@colorremedies.com> wrote : > > > >> On Mon, Jun 27, 2016 at 4:55 PM, Chris Murphy > >> <lists@colorremedies.com> wrote: > >> > >> >> BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) > >> >> to /dev/sdd1 started scrub_handle_errored_block: 166 callbacks > >> >> suppressed BTRFS warning (device sdb1): checksum error at > >> >> logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, > >> >> inode 3434831, offset 479232, length 4096, links 1 (path: > >> >> user/.local/share/zeitgeist/activity.sqlite-wal) > >> >> btrfs_dev_stat_print_on_error: 166 callbacks suppressed BTRFS > >> >> error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, > >> >> corrupt 14221, gen 24 scrub_handle_errored_block: 166 callbacks > >> >> suppressed BTRFS error (device sdb1): unable to fixup (regular) > >> >> error at logical 93445255168 on dev /dev/sda1 > >> > > >> > Shoot. You have a lot of these. It looks suspiciously like you're > >> > hitting a case list regulars are only just starting to understand > >> > >> Forget this part completely. It doesn't affect raid1. I just > >> re-read that your setup is not raid1, I don't know why I thought > >> it was raid5. > >> > >> The likely issue here is that you've got legit corruptions on sda > >> (mix of slow and flat out bad sectors), as well as a failing drive. > >> > >> This is also safe to issue: > >> > >> smartctl -l scterc /dev/sda > >> smartctl -l scterc /dev/sdb > >> cat /sys/block/sda/device/timeout > >> cat /sys/block/sdb/device/timeout > >> > > > > My setup is indeed RAID1 (and not RAID5) > > > > root@system:/# smartctl -l scterc /dev/sda > > smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] > > (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, > > www.smartmontools.org > > > > SCT Error Recovery Control: > > Read: Disabled > > Write: Disabled > > > > root@system:/# smartctl -l scterc /dev/sdb > > smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] > > (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, > > www.smartmontools.org > > > > SCT Error Recovery Control: > > Read: Disabled > > Write: Disabled > > > > root@system:/# cat /sys/block/sda/device/timeout > > 30 > > root@system:/# cat /sys/block/sdb/device/timeout > > 30 > > Good news and bad news. The bad news is this is a significant > misconfiguration, it's very common, and it means that any bad sectors > that don't result in read errors before 30 seconds will mean they > don't get fixed by Btrfs (or even mdadm or LVM raid). So they can > accumulate. > > There are two options since your drives support SCT ERC. > > 1. > smartctl -l scterc,70,70 /dev/sdX ## done for both drives > > That will make sure the drive reports a read error in 7 seconds, well > under the kernel's command timer of 7 seconds. This is how your drives > should normally be configured for RAID usage. > > 2. > echo 180 > /sys/block/sda/device/timeout > echo 180 > /sys/block/sdb/device/timeout > > This *might* actually work better in your case. If you permit the > drives to have really long error recovery, it might actually allow the > data to be returned to Btrfs and then it can start fixing problems. > Maybe. It's a long shot. And there will be upwards of 3 minute hangs. > > I would give this a shot first. You can issue these commands safely at > any time, no umount is needed or anything like that. I would do this > even before using cp/rsync or ddrescue because it increases the chance > the drive can recover data from these bad sectors and fix the other > drive. > > These settings are not persistent across a reboot unless you set a > udev rule or equivalent. > > On one of my drives that supports SCT ERC it only accepts the smartctl > -l command to set the timeout once. I can't change it without power > cycling the drive or it just crashes (yay firmware bugs). Just FYI > it's possible to run into other weirdness. > I've tried both option and launched a replace, but I got the same error (replace is cancelled, jernel bug). I will let these options on and attempt a ddrescue on /dev/sda to /dev/sdd. Then I will disconnect /dev/sda and reboot and see if it works better. > Last, I have no idea if the massive Btrfs write errors on sda are from > an earlier problem where the drive data or power cable got jiggled or > was otherwise absent temporarily? So depending on how the block > timeout change affects your data recovery, you might end up needing to > do a reboot to get back to a more stable state for all of this? It > really should be able to fix things *if* at least one copy can be read > and then written to the other drive. > I have also no idea why is sda behaving like this. I haven't done anything particular on these drives. Thanks for your help ! ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-28 0:49 ` Saint Germain @ 2016-06-28 2:14 ` Chris Murphy 2016-06-28 22:52 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-28 2:14 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Mon, Jun 27, 2016 at 6:49 PM, Saint Germain <saintger@gmail.com> wrote: > > I've tried both option and launched a replace, but I got the same error > (replace is cancelled, jernel bug). > I will let these options on and attempt a ddrescue on /dev/sda > to /dev/sdd. > Then I will disconnect /dev/sda and reboot and see if it works better. Sounds reasonable. Just make sure the file system is already unmounted when you use ddrescue because otherwise you're block copying it while it could be modified while rw mounted (generation number tends to get incremented while rw mounted). >> Last, I have no idea if the massive Btrfs write errors on sda are from >> an earlier problem where the drive data or power cable got jiggled or >> was otherwise absent temporarily? So depending on how the block >> timeout change affects your data recovery, you might end up needing to >> do a reboot to get back to a more stable state for all of this? It >> really should be able to fix things *if* at least one copy can be read >> and then written to the other drive. >> > > I have also no idea why is sda behaving like this. I haven't done > anything particular on these drives. Yeah pretty weird. At some point once things are stable, if this file system survives, you might want to use btrfs dev stat -z to wipe out those stats. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-28 2:14 ` Chris Murphy @ 2016-06-28 22:52 ` Saint Germain 2016-06-29 4:25 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-28 22:52 UTC (permalink / raw) To: Btrfs BTRFS On Mon, 27 Jun 2016 20:14:58 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Mon, Jun 27, 2016 at 6:49 PM, Saint Germain <saintger@gmail.com> > wrote: > > > > > I've tried both option and launched a replace, but I got the same > > error (replace is cancelled, jernel bug). > > I will let these options on and attempt a ddrescue on /dev/sda > > to /dev/sdd. > > Then I will disconnect /dev/sda and reboot and see if it works > > better. > > Sounds reasonable. Just make sure the file system is already unmounted > when you use ddrescue because otherwise you're block copying it while > it could be modified while rw mounted (generation number tends to get > incremented while rw mounted). > > Well I made a ddrescue image of both drives (only one error on sdb during ddrescue copy) and started the computer again (after disconnecting the old drives). However the errors remains there, and I still cannot scrub (scrub is aborted), nor delete the file which have errors (drive is remounted read-only if I try to delete the files). I don't know if I should continue trying to repair this RAID1 or if I should just cp/rsync to a new BTRFS volume and get done with it. On the other hand it seems interesting to repair instead of just giving up. It gives a good look at BTRFS resiliency/reliability. Here is the log from the mount to the scrub aborting and the result from smartctl. Thanks for your precious help so far. BTRFS error (device sdb1): cleaner transaction attach returned -30 BTRFS info (device sdb1): disk space caching is enabled BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714507, gen 1335 BTRFS info (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21622, gen 24 scrub_handle_errored_block: 164 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sdb1, sector 54528696, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) btrfs_dev_stat_print_on_error: 164 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714508, gen 1335 scrub_handle_errored_block: 164 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sdb1 BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714509, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445259264 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21623, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sda1 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21624, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445259264 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349810688 on dev /dev/sda1, sector 140429952, root 5, inode 4265283, offset 0, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21625, gen 24 BTRFS warning (device sdb1): checksum error at logical 136349929472 on dev /dev/sda1, sector 140430184, root 5, inode 4265283, offset 118784, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS warning (device sdb1): checksum error at logical 136350060544 on dev /dev/sda1, sector 140430440, root 5, inode 4265283, offset 249856, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21626, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21627, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349810688 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136350060544 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349929472 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349814784 on dev /dev/sda1, sector 140429960, root 5, inode 4265283, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21628, gen 24 BTRFS warning (device sdb1): checksum error at logical 136350064640 on dev /dev/sda1, sector 140430448, root 5, inode 4265283, offset 253952, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21629, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349814784 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349933568 on dev /dev/sda1, sector 140430192, root 5, inode 4265283, offset 122880, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21630, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136350064640 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349933568 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349818880 on dev /dev/sda1, sector 140429968, root 5, inode 4265283, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21631, gen 24 BTRFS warning (device sdb1): checksum error at logical 136350068736 on dev /dev/sda1, sector 140430456, root 5, inode 4265283, offset 258048, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21632, gen 24 BTRFS warning (device sdb1): checksum error at logical 136349937664 on dev /dev/sda1, sector 140430200, root 5, inode 4265283, offset 126976, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21633, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349818880 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136350068736 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349937664 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 136349822976 on dev /dev/sda1, sector 140429976, root 5, inode 4265283, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21634, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349822976 on dev /dev/sda1 scrub_handle_errored_block: 164 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 136349810688 on dev /dev/sdb1, sector 138326656, root 5, inode 4265283, offset 0, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) btrfs_dev_stat_print_on_error: 164 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714510, gen 1335 BTRFS warning (device sdb1): checksum error at logical 136349929472 on dev /dev/sdb1, sector 138326888, root 5, inode 4265283, offset 118784, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714511, gen 1335 BTRFS warning (device sdb1): checksum error at logical 136350060544 on dev /dev/sdb1, sector 138327144, root 5, inode 4265283, offset 249856, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714512, gen 1335 scrub_handle_errored_block: 164 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349810688 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349814784 on dev /dev/sdb1, sector 138326664, root 5, inode 4265283, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714513, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349814784 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349818880 on dev /dev/sdb1, sector 138326672, root 5, inode 4265283, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714514, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349818880 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349822976 on dev /dev/sdb1, sector 138326680, root 5, inode 4265283, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714515, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349822976 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349827072 on dev /dev/sdb1, sector 138326688, root 5, inode 4265283, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714516, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349827072 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349831168 on dev /dev/sdb1, sector 138326696, root 5, inode 4265283, offset 20480, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714517, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349831168 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349835264 on dev /dev/sdb1, sector 138326704, root 5, inode 4265283, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714518, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349835264 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 136349839360 on dev /dev/sdb1, sector 138326712, root 5, inode 4265283, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-42-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714519, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349839360 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349843456 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 136349847552 on dev /dev/sdb1 scrub_handle_errored_block: 164 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 139677478912 on dev /dev/sda1, sector 146929304, root 5, inode 4265287, offset 208896, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) btrfs_dev_stat_print_on_error: 164 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21799, gen 24 BTRFS warning (device sdb1): checksum error at logical 139677347840 on dev /dev/sda1, sector 146929048, root 5, inode 4265287, offset 77824, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21800, gen 24 BTRFS warning (device sdb1): checksum error at logical 139677609984 on dev /dev/sda1, sector 146929560, root 5, inode 4265287, offset 339968, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21801, gen 24 BTRFS warning (device sdb1): checksum error at logical 139677270016 on dev /dev/sda1, sector 146928896, root 5, inode 4265287, offset 0, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21802, gen 24 scrub_handle_errored_block: 164 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677478912 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677483008 on dev /dev/sda1, sector 146929312, root 5, inode 4265287, offset 212992, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21803, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677347840 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677609984 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677483008 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677270016 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677351936 on dev /dev/sda1, sector 146929056, root 5, inode 4265287, offset 81920, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21804, gen 24 BTRFS warning (device sdb1): checksum error at logical 139677614080 on dev /dev/sda1, sector 146929568, root 5, inode 4265287, offset 344064, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21805, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677351936 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677487104 on dev /dev/sda1, sector 146929320, root 5, inode 4265287, offset 217088, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21806, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677614080 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677274112 on dev /dev/sda1, sector 146928904, root 5, inode 4265287, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21807, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677487104 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 139677356032 on dev /dev/sda1, sector 146929064, root 5, inode 4265287, offset 86016, length 4096, links 1 (path: user/Pictures/Picture-51-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21808, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677356032 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 139677274112 on dev /dev/sda1 scrub_handle_errored_block: 434 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 140475179008 on dev /dev/sda1, sector 148487312, root 5, inode 4265291, offset 114688, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) btrfs_dev_stat_print_on_error: 434 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22243, gen 24 BTRFS warning (device sdb1): checksum error at logical 140475064320 on dev /dev/sda1, sector 148487088, root 5, inode 4265291, offset 0, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22244, gen 24 BTRFS warning (device sdb1): checksum error at logical 140475310080 on dev /dev/sda1, sector 148487568, root 5, inode 4265291, offset 245760, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22245, gen 24 BTRFS warning (device sdb1): checksum error at logical 140475441152 on dev /dev/sda1, sector 148487824, root 5, inode 4265291, offset 376832, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22246, gen 24 scrub_handle_errored_block: 434 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475179008 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475064320 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475310080 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475441152 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475183104 on dev /dev/sda1, sector 148487320, root 5, inode 4265291, offset 118784, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22247, gen 24 BTRFS warning (device sdb1): checksum error at logical 140475068416 on dev /dev/sda1, sector 148487096, root 5, inode 4265291, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22248, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475183104 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475314176 on dev /dev/sda1, sector 148487576, root 5, inode 4265291, offset 249856, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22249, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475068416 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475445248 on dev /dev/sda1, sector 148487832, root 5, inode 4265291, offset 380928, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22250, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475314176 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475187200 on dev /dev/sda1, sector 148487328, root 5, inode 4265291, offset 122880, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22251, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475445248 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 140475072512 on dev /dev/sda1, sector 148487104, root 5, inode 4265291, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-56-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22252, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475187200 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140475072512 on dev /dev/sda1 scrub_handle_errored_block: 649 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 140157034496 on dev /dev/sdb1, sector 145762640, root 5, inode 4265289, offset 0, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) btrfs_dev_stat_print_on_error: 649 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715023, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157091840 on dev /dev/sdb1, sector 145762752, root 5, inode 4265289, offset 57344, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715024, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157222912 on dev /dev/sdb1, sector 145763008, root 5, inode 4265289, offset 188416, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715025, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157353984 on dev /dev/sdb1, sector 145763264, root 5, inode 4265289, offset 319488, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715026, gen 1335 scrub_handle_errored_block: 649 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157034496 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157091840 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 140157038592 on dev /dev/sdb1, sector 145762648, root 5, inode 4265289, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715027, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157095936 on dev /dev/sdb1, sector 145762760, root 5, inode 4265289, offset 61440, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715028, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157038592 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157095936 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 140157042688 on dev /dev/sdb1, sector 145762656, root 5, inode 4265289, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715029, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157100032 on dev /dev/sdb1, sector 145762768, root 5, inode 4265289, offset 65536, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715030, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157042688 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157100032 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 140157046784 on dev /dev/sdb1, sector 145762664, root 5, inode 4265289, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715031, gen 1335 BTRFS warning (device sdb1): checksum error at logical 140157104128 on dev /dev/sdb1, sector 145762776, root 5, inode 4265289, offset 69632, length 4096, links 1 (path: user/Pictures/Picture-54-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715032, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157046784 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157104128 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157050880 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 140157108224 on dev /dev/sdb1 scrub_handle_errored_block: 415 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 162995294208 on dev /dev/sda1, sector 192471912, root 5, inode 4265299, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS warning (device sdb1): checksum error at logical 162995556352 on dev /dev/sda1, sector 192472424, root 5, inode 4265299, offset 286720, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) btrfs_dev_stat_print_on_error: 415 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22564, gen 24 BTRFS warning (device sdb1): checksum error at logical 162995269632 on dev /dev/sda1, sector 192471864, root 5, inode 4265299, offset 0, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS warning (device sdb1): checksum error at logical 162995425280 on dev /dev/sda1, sector 192472168, root 5, inode 4265299, offset 155648, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22565, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22566, gen 24 BTRFS warning (device sdb1): checksum error at logical 162995818496 on dev /dev/sda1, sector 192472936, root 5, inode 4265299, offset 548864, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS warning (device sdb1): checksum error at logical 162995687424 on dev /dev/sda1, sector 192472680, root 5, inode 4265299, offset 417792, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22567, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22568, gen 24 scrub_handle_errored_block: 415 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995294208 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995556352 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 162995298304 on dev /dev/sda1, sector 192471920, root 5, inode 4265299, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22569, gen 24 BTRFS warning (device sdb1): checksum error at logical 162995429376 on dev /dev/sda1, sector 192472176, root 5, inode 4265299, offset 159744, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22570, gen 24 BTRFS warning (device sdb1): checksum error at logical 162995560448 on dev /dev/sda1, sector 192472432, root 5, inode 4265299, offset 290816, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22571, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995818496 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995687424 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995298304 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995429376 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995560448 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995269632 on dev /dev/sda1 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22572, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 22573, gen 24 BTRFS warning (device sdb1): checksum error at logical 162995822592 on dev /dev/sda1, sector 192472944, root 5, inode 4265299, offset 552960, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995691520 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995822592 on dev /dev/sda1 scrub_handle_errored_block: 434 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 162995294208 on dev /dev/sdb1, sector 190368616, root 5, inode 4265299, offset 24576, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS warning (device sdb1): checksum error at logical 162995425280 on dev /dev/sdb1, sector 190368872, root 5, inode 4265299, offset 155648, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) btrfs_dev_stat_print_on_error: 433 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715449, gen 1335 BTRFS warning (device sdb1): checksum error at logical 162995556352 on dev /dev/sdb1, sector 190369128, root 5, inode 4265299, offset 286720, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715450, gen 1335 BTRFS warning (device sdb1): checksum error at logical 162995687424 on dev /dev/sdb1, sector 190369384, root 5, inode 4265299, offset 417792, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715451, gen 1335 BTRFS warning (device sdb1): checksum error at logical 162995269632 on dev /dev/sdb1, sector 190368568, root 5, inode 4265299, offset 0, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715452, gen 1335 BTRFS warning (device sdb1): checksum error at logical 162995818496 on dev /dev/sdb1, sector 190369640, root 5, inode 4265299, offset 548864, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715453, gen 1335 scrub_handle_errored_block: 433 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995294208 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 162995298304 on dev /dev/sdb1, sector 190368624, root 5, inode 4265299, offset 28672, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715454, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995425280 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995556352 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995687424 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995269632 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 162995429376 on dev /dev/sdb1, sector 190368880, root 5, inode 4265299, offset 159744, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715455, gen 1335 BTRFS warning (device sdb1): checksum error at logical 162995560448 on dev /dev/sdb1, sector 190369136, root 5, inode 4265299, offset 290816, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715456, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995429376 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 162995691520 on dev /dev/sdb1, sector 190369392, root 5, inode 4265299, offset 421888, length 4096, links 1 (path: user/Pictures/Picture-22.jpg) BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995818496 on dev /dev/sdb1 BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715457, gen 1335 BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1715458, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995298304 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995560448 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 162995691520 on dev /dev/sdb1 scrub_handle_errored_block: 434 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 170606833664 on dev /dev/sda1, sector 207338200, root 5, inode 4265298, offset 57344, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) btrfs_dev_stat_print_on_error: 433 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23007, gen 24 BTRFS warning (device sdb1): checksum error at logical 170606776320 on dev /dev/sda1, sector 207338088, root 5, inode 4265298, offset 0, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23008, gen 24 BTRFS warning (device sdb1): checksum error at logical 170606964736 on dev /dev/sda1, sector 207338456, root 5, inode 4265298, offset 188416, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23009, gen 24 BTRFS warning (device sdb1): checksum error at logical 170607095808 on dev /dev/sda1, sector 207338712, root 5, inode 4265298, offset 319488, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23010, gen 24 BTRFS warning (device sdb1): checksum error at logical 170607226880 on dev /dev/sda1, sector 207338968, root 5, inode 4265298, offset 450560, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23011, gen 24 BTRFS warning (device sdb1): checksum error at logical 170607357952 on dev /dev/sda1, sector 207339224, root 5, inode 4265298, offset 581632, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23012, gen 24 scrub_handle_errored_block: 434 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606833664 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606776320 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606964736 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607095808 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607226880 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607357952 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 170606837760 on dev /dev/sda1, sector 207338208, root 5, inode 4265298, offset 61440, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23013, gen 24 BTRFS warning (device sdb1): checksum error at logical 170606780416 on dev /dev/sda1, sector 207338096, root 5, inode 4265298, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23014, gen 24 BTRFS warning (device sdb1): checksum error at logical 170606968832 on dev /dev/sda1, sector 207338464, root 5, inode 4265298, offset 192512, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23015, gen 24 BTRFS warning (device sdb1): checksum error at logical 170607099904 on dev /dev/sda1, sector 207338720, root 5, inode 4265298, offset 323584, length 4096, links 1 (path: user/Pictures/Picture-22-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23016, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606837760 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606780416 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170606968832 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 170607099904 on dev /dev/sda1 scrub_handle_errored_block: 281 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 171215843328 on dev /dev/sda1, sector 208527672, root 5, inode 4265266, offset 0, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS warning (device sdb1): checksum error at logical 171216105472 on dev /dev/sda1, sector 208528184, root 5, inode 4265266, offset 262144, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) btrfs_dev_stat_print_on_error: 281 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23299, gen 24 BTRFS warning (device sdb1): checksum error at logical 171215974400 on dev /dev/sda1, sector 208527928, root 5, inode 4265266, offset 131072, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23300, gen 24 BTRFS warning (device sdb1): checksum error at logical 171216236544 on dev /dev/sda1, sector 208528440, root 5, inode 4265266, offset 393216, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23301, gen 24 BTRFS warning (device sdb1): checksum error at logical 171216498688 on dev /dev/sda1, sector 208528952, root 5, inode 4265266, offset 655360, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS warning (device sdb1): checksum error at logical 171216367616 on dev /dev/sda1, sector 208528696, root 5, inode 4265266, offset 524288, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23302, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23303, gen 24 scrub_handle_errored_block: 281 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215843328 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216105472 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215974400 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216236544 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 171215847424 on dev /dev/sda1, sector 208527680, root 5, inode 4265266, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23304, gen 24 BTRFS warning (device sdb1): checksum error at logical 171216109568 on dev /dev/sda1, sector 208528192, root 5, inode 4265266, offset 266240, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23305, gen 24 BTRFS warning (device sdb1): checksum error at logical 171215978496 on dev /dev/sda1, sector 208527936, root 5, inode 4265266, offset 135168, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23306, gen 24 BTRFS warning (device sdb1): checksum error at logical 171216240640 on dev /dev/sda1, sector 208528448, root 5, inode 4265266, offset 397312, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23307, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216367616 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216498688 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216109568 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215847424 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215978496 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171216240640 on dev /dev/sda1 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23308, gen 24 scrub_handle_errored_block: 602 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 171215843328 on dev /dev/sdb1, sector 206424376, root 5, inode 4265266, offset 0, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) btrfs_dev_stat_print_on_error: 601 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716183, gen 1335 BTRFS warning (device sdb1): checksum error at logical 171215974400 on dev /dev/sdb1, sector 206424632, root 5, inode 4265266, offset 131072, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716184, gen 1335 BTRFS warning (device sdb1): checksum error at logical 171216105472 on dev /dev/sdb1, sector 206424888, root 5, inode 4265266, offset 262144, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716185, gen 1335 BTRFS warning (device sdb1): checksum error at logical 171216236544 on dev /dev/sdb1, sector 206425144, root 5, inode 4265266, offset 393216, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716186, gen 1335 BTRFS warning (device sdb1): checksum error at logical 171216498688 on dev /dev/sdb1, sector 206425656, root 5, inode 4265266, offset 655360, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716187, gen 1335 BTRFS warning (device sdb1): checksum error at logical 171216367616 on dev /dev/sdb1, sector 206425400, root 5, inode 4265266, offset 524288, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716188, gen 1335 scrub_handle_errored_block: 602 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215843328 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 171215847424 on dev /dev/sdb1, sector 206424384, root 5, inode 4265266, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716189, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215847424 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 171215851520 on dev /dev/sdb1, sector 206424392, root 5, inode 4265266, offset 8192, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716190, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215851520 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 171215855616 on dev /dev/sdb1, sector 206424400, root 5, inode 4265266, offset 12288, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716191, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215855616 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 171215859712 on dev /dev/sdb1, sector 206424408, root 5, inode 4265266, offset 16384, length 4096, links 1 (path: user/Pictures/Picture-33.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716192, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215859712 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215863808 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215867904 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215872000 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215876096 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 171215880192 on dev /dev/sdb1 scrub_handle_errored_block: 311 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 182510542848 on dev /dev/sda1, sector 230587632, root 5, inode 4265300, offset 0, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) btrfs_dev_stat_print_on_error: 311 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23619, gen 24 BTRFS warning (device sdb1): checksum error at logical 182510669824 on dev /dev/sda1, sector 230587880, root 5, inode 4265300, offset 126976, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23620, gen 24 BTRFS warning (device sdb1): checksum error at logical 182510800896 on dev /dev/sda1, sector 230588136, root 5, inode 4265300, offset 258048, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23621, gen 24 BTRFS warning (device sdb1): checksum error at logical 182510931968 on dev /dev/sda1, sector 230588392, root 5, inode 4265300, offset 389120, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23622, gen 24 BTRFS warning (device sdb1): checksum error at logical 182511063040 on dev /dev/sda1, sector 230588648, root 5, inode 4265300, offset 520192, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23623, gen 24 BTRFS warning (device sdb1): checksum error at logical 182511194112 on dev /dev/sda1, sector 230588904, root 5, inode 4265300, offset 651264, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23624, gen 24 scrub_handle_errored_block: 311 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510542848 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 182510546944 on dev /dev/sda1, sector 230587640, root 5, inode 4265300, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23625, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510669824 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510800896 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510931968 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182511063040 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 182510673920 on dev /dev/sda1, sector 230587888, root 5, inode 4265300, offset 131072, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23626, gen 24 BTRFS warning (device sdb1): checksum error at logical 182510804992 on dev /dev/sda1, sector 230588144, root 5, inode 4265300, offset 262144, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23627, gen 24 BTRFS warning (device sdb1): checksum error at logical 182510936064 on dev /dev/sda1, sector 230588400, root 5, inode 4265300, offset 393216, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23628, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182511194112 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510546944 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510673920 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510804992 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510936064 on dev /dev/sda1 scrub_handle_errored_block: 345 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 182510542848 on dev /dev/sdb1, sector 228484336, root 5, inode 4265300, offset 0, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) btrfs_dev_stat_print_on_error: 345 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716504, gen 1335 BTRFS warning (device sdb1): checksum error at logical 182510669824 on dev /dev/sdb1, sector 228484584, root 5, inode 4265300, offset 126976, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716505, gen 1335 BTRFS warning (device sdb1): checksum error at logical 182510800896 on dev /dev/sdb1, sector 228484840, root 5, inode 4265300, offset 258048, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716506, gen 1335 BTRFS warning (device sdb1): checksum error at logical 182510931968 on dev /dev/sdb1, sector 228485096, root 5, inode 4265300, offset 389120, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716507, gen 1335 BTRFS warning (device sdb1): checksum error at logical 182511063040 on dev /dev/sdb1, sector 228485352, root 5, inode 4265300, offset 520192, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716508, gen 1335 BTRFS warning (device sdb1): checksum error at logical 182511194112 on dev /dev/sdb1, sector 228485608, root 5, inode 4265300, offset 651264, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716509, gen 1335 scrub_handle_errored_block: 345 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510542848 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510546944 on dev /dev/sdb1, sector 228484344, root 5, inode 4265300, offset 4096, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716510, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510800896 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510804992 on dev /dev/sdb1, sector 228484848, root 5, inode 4265300, offset 262144, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716511, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510931968 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182510936064 on dev /dev/sdb1, sector 228485104, root 5, inode 4265300, offset 393216, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716512, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182511063040 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 182511067136 on dev /dev/sdb1, sector 228485360, root 5, inode 4265300, offset 524288, length 4096, links 1 (path: user/Pictures/Picture-24-2.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716513, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182511194112 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510669824 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182511198208 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510546944 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510804992 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 182510936064 on dev /dev/sdb1 scrub_handle_errored_block: 345 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 198160961536 on dev /dev/sda1, sector 261154856, root 5, inode 4265307, offset 2609152, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) btrfs_dev_stat_print_on_error: 345 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 198161223680 on dev /dev/sda1, sector 261155368, root 5, inode 4265307, offset 2871296, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS warning (device sdb1): checksum error at logical 198160945152 on dev /dev/sda1, sector 261154824, root 5, inode 4265307, offset 2592768, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23976, gen 24 BTRFS warning (device sdb1): checksum error at logical 198161092608 on dev /dev/sda1, sector 261155112, root 5, inode 4265307, offset 2740224, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS warning (device sdb1): checksum error at logical 198161485824 on dev /dev/sda1, sector 261155880, root 5, inode 4265307, offset 3133440, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS warning (device sdb1): checksum error at logical 198161354752 on dev /dev/sda1, sector 261155624, root 5, inode 4265307, offset 3002368, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23977, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23978, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23979, gen 24 scrub_handle_errored_block: 345 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161223680 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198161227776 on dev /dev/sda1, sector 261155376, root 5, inode 4265307, offset 2875392, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23980, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160945152 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160961536 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161485824 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161354752 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161092608 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161227776 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198160949248 on dev /dev/sda1, sector 261154832, root 5, inode 4265307, offset 2596864, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23981, gen 24 BTRFS warning (device sdb1): checksum error at logical 198160965632 on dev /dev/sda1, sector 261154864, root 5, inode 4265307, offset 2613248, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23982, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160949248 on dev /dev/sda1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160965632 on dev /dev/sda1 BTRFS warning (device sdb1): checksum error at logical 198161489920 on dev /dev/sda1, sector 261155888, root 5, inode 4265307, offset 3137536, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23983, gen 24 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23984, gen 24 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161489920 on dev /dev/sda1 BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 23985, gen 24 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 scrub_handle_errored_block: 166 callbacks suppressed BTRFS warning (device sdb1): checksum error at logical 198160945152 on dev /dev/sdb1, sector 259051528, root 5, inode 4265307, offset 2592768, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) btrfs_dev_stat_print_on_error: 164 callbacks suppressed BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716859, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198160961536 on dev /dev/sdb1, sector 259051560, root 5, inode 4265307, offset 2609152, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716860, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198161092608 on dev /dev/sdb1, sector 259051816, root 5, inode 4265307, offset 2740224, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716861, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198161223680 on dev /dev/sdb1, sector 259052072, root 5, inode 4265307, offset 2871296, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716862, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198161354752 on dev /dev/sdb1, sector 259052328, root 5, inode 4265307, offset 3002368, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716863, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198161485824 on dev /dev/sdb1, sector 259052584, root 5, inode 4265307, offset 3133440, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716864, gen 1335 scrub_handle_errored_block: 166 callbacks suppressed BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160945152 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160961536 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161092608 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161223680 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161354752 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 198160949248 on dev /dev/sdb1, sector 259051536, root 5, inode 4265307, offset 2596864, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716865, gen 1335 BTRFS warning (device sdb1): checksum error at logical 198160965632 on dev /dev/sdb1, sector 259051568, root 5, inode 4265307, offset 2613248, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716866, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161485824 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 198161096704 on dev /dev/sdb1, sector 259051824, root 5, inode 4265307, offset 2744320, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716867, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198160949248 on dev /dev/sdb1 BTRFS warning (device sdb1): checksum error at logical 198161358848 on dev /dev/sdb1, sector 259052336, root 5, inode 4265307, offset 3006464, length 4096, links 1 (path: user/Pictures/Picture-38.jpg) BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1716868, gen 1335 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161096704 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161227776 on dev /dev/sdb1 BTRFS error (device sdb1): unable to fixup (regular) error at logical 198161358848 on dev /dev/sdb1 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 BTRFS error (device sdb1): parent transid verify failed on 1894596354048 wanted 246893 found 240551 root@system:/# smartctl -x /dev/sda smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST2000LM003 HN-M201RAD Serial Number: S377J9CH203273 LU WWN Device Id: 5 0004cf 211e05d6a Firmware Version: 2BE10001 User Capacity: 2 000 398 934 016 bytes [2,00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Jun 29 00:46:44 2016 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM level is: 254 (maximum performance), recommended: 254 APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 38) The self-test routine was interrupted by the host with a hard or soft reset. Total time to complete Offline data collection: (21420) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 357) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 0 2 Throughput_Performance -OS--K 252 252 000 - 0 3 Spin_Up_Time PO---K 090 089 025 - 3170 4 Start_Stop_Count -O--CK 100 100 000 - 7 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 8 Seek_Time_Performance --S--K 252 252 015 - 0 9 Power_On_Hours -O--CK 100 100 000 - 20 10 Spin_Retry_Count -O--CK 252 252 051 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 19 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 194 Temperature_Celsius -O---- 064 058 000 - 35 (Min/Max 24/45) 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 197 Current_Pending_Sector -O--CK 252 252 000 - 0 198 Offline_Uncorrectable ----CK 252 252 000 - 0 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 0 223 Load_Retry_Count -O--CK 252 252 000 - 0 225 Load_Cycle_Count -O--CK 100 100 000 - 250 241 Total_LBAs_Written -O--CK 096 094 000 - 6054713 242 Total_LBAs_Read -O--CK 094 094 000 - 8176813 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 2 Comprehensive SMART error log 0x03 GPL R/O 2 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 2 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xc0-0xdf GPL,SL VS 16 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (2 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (2 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Offline Interrupted (host reset) 60% 20 - # 2 Short captive Self-test routine in progress 60% 20 - SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Interrupted [60% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 2 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: Active (0) Current Temperature: 35 Celsius Power Cycle Min/Max Temperature: 30/40 Celsius Lifetime Min/Max Temperature: 24/45 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 5 minutes Temperature Logging Interval: 5 minutes Min/Max recommended Temperature: -5/80 Celsius Min/Max Temperature Limit: -10/85 Celsius Temperature History Size (Index): 128 (50) Index Estimated Time Temperature Celsius 51 2016-06-28 14:10 35 **************** 52 2016-06-28 14:15 44 ************************* 53 2016-06-28 14:20 44 ************************* 54 2016-06-28 14:25 44 ************************* 55 2016-06-28 14:30 42 *********************** 56 2016-06-28 14:35 43 ************************ 57 2016-06-28 14:40 44 ************************* ... ..( 2 skipped). .. ************************* 60 2016-06-28 14:55 44 ************************* 61 2016-06-28 15:00 45 ************************** ... ..( 16 skipped). .. ************************** 78 2016-06-28 16:25 45 ************************** 79 2016-06-28 16:30 44 ************************* ... ..( 29 skipped). .. ************************* 109 2016-06-28 19:00 44 ************************* 110 2016-06-28 19:05 43 ************************ 111 2016-06-28 19:10 44 ************************* 112 2016-06-28 19:15 44 ************************* 113 2016-06-28 19:20 43 ************************ ... ..( 19 skipped). .. ************************ 5 2016-06-28 21:00 43 ************************ 6 2016-06-28 21:05 42 *********************** ... ..( 4 skipped). .. *********************** 11 2016-06-28 21:30 42 *********************** 12 2016-06-28 21:35 43 ************************ 13 2016-06-28 21:40 41 ********************** 14 2016-06-28 21:45 33 ************** 15 2016-06-28 21:50 36 ***************** 16 2016-06-28 21:55 37 ****************** 17 2016-06-28 22:00 37 ****************** 18 2016-06-28 22:05 34 *************** 19 2016-06-28 22:10 33 ************** 20 2016-06-28 22:15 33 ************** 21 2016-06-28 22:20 32 ************* 22 2016-06-28 22:25 32 ************* 23 2016-06-28 22:30 33 ************** ... ..( 2 skipped). .. ************** 26 2016-06-28 22:45 33 ************** 27 2016-06-28 22:50 34 *************** 28 2016-06-28 22:55 34 *************** 29 2016-06-28 23:00 34 *************** 30 2016-06-28 23:05 35 **************** ... ..( 7 skipped). .. **************** 38 2016-06-28 23:45 35 **************** 39 2016-06-28 23:50 34 *************** 40 2016-06-28 23:55 37 ****************** 41 2016-06-29 00:00 39 ******************** 42 2016-06-29 00:05 38 ******************* 43 2016-06-29 00:10 38 ******************* 44 2016-06-29 00:15 39 ******************** 45 2016-06-29 00:20 40 ********************* 46 2016-06-29 00:25 40 ********************* 47 2016-06-29 00:30 38 ******************* 48 2016-06-29 00:35 37 ****************** 49 2016-06-29 00:40 37 ****************** 50 2016-06-29 00:45 36 ***************** SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP/SMART Log 0x04) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 4 0 Command failed due to ICRC error 0x0002 4 0 R_ERR response for data FIS 0x0003 4 0 R_ERR response for device-to-host data FIS 0x0004 4 0 R_ERR response for host-to-device data FIS 0x0005 4 0 R_ERR response for non-data FIS 0x0006 4 0 R_ERR response for device-to-host non-data FIS 0x0007 4 0 R_ERR response for host-to-device non-data FIS 0x0008 4 0 Device-to-host non-data FIS retries 0x0009 4 7 Transition from drive PhyRdy to drive PhyNRdy 0x000a 4 6 Device-to-host register FISes sent due to a COMRESET 0x000b 4 0 CRC errors within host-to-device FIS 0x000d 4 0 Non-CRC errors within host-to-device FIS 0x000f 4 0 R_ERR response for host-to-device data FIS, CRC 0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC 0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC 0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC 0x8e00 4 0 Vendor specific 0x8e01 4 0 Vendor specific 0x8e02 4 0 Vendor specific 0x8e03 4 0 Vendor specific 0x8e04 4 0 Vendor specific 0x8e05 4 0 Vendor specific 0x8e06 4 0 Vendor specific 0x8e07 4 0 Vendor specific 0x8e08 4 0 Vendor specific 0x8e09 4 0 Vendor specific 0x8e0a 4 0 Vendor specific 0x8e0b 4 0 Vendor specific 0x8e0c 4 0 Vendor specific 0x8e0d 4 0 Vendor specific 0x8e0e 4 0 Vendor specific 0x8e0f 4 0 Vendor specific 0x8e10 4 0 Vendor specific 0x8e11 4 0 Vendor specific root@system:/# smartctl -x /dev/sdb smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST2000LM003 HN-M201RAD Serial Number: S377J9CH203271 LU WWN Device Id: 5 0004cf 211e05d48 Firmware Version: 2BE10001 User Capacity: 2 000 398 934 016 bytes [2,00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Jun 29 00:46:46 2016 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM level is: 254 (maximum performance), recommended: 254 APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 38) The self-test routine was interrupted by the host with a hard or soft reset. Total time to complete Offline data collection: (23220) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 387) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 0 2 Throughput_Performance -OS--K 252 252 000 - 0 3 Spin_Up_Time PO---K 090 090 025 - 3121 4 Start_Stop_Count -O--CK 100 100 000 - 5 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 8 Seek_Time_Performance --S--K 252 252 015 - 0 9 Power_On_Hours -O--CK 100 100 000 - 12 10 Spin_Retry_Count -O--CK 252 252 051 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 10 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 194 Temperature_Celsius -O---- 064 061 000 - 34 (Min/Max 26/45) 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 197 Current_Pending_Sector -O--CK 252 252 000 - 0 198 Offline_Uncorrectable ----CK 252 252 000 - 0 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 1 223 Load_Retry_Count -O--CK 252 252 000 - 0 225 Load_Cycle_Count -O--CK 100 100 000 - 114 241 Total_LBAs_Written -O--CK 095 095 000 - 7238560 242 Total_LBAs_Read -O--CK 097 094 000 - 4420088 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 2 Comprehensive SMART error log 0x03 GPL R/O 2 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 2 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xc0-0xdf GPL,SL VS 16 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (2 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (2 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Offline Interrupted (host reset) 60% 12 - # 2 Short captive Self-test routine in progress 60% 12 - SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Interrupted [60% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 2 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: Active (0) Current Temperature: 33 Celsius Power Cycle Min/Max Temperature: 29/39 Celsius Lifetime Min/Max Temperature: 24/45 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 5 minutes Temperature Logging Interval: 5 minutes Min/Max recommended Temperature: -5/80 Celsius Min/Max Temperature Limit: -10/85 Celsius Temperature History Size (Index): 128 (116) Index Estimated Time Temperature Celsius 117 2016-06-28 14:10 33 ************** 118 2016-06-28 14:15 ? - ... ..( 8 skipped). .. - 127 2016-06-28 15:00 ? - 0 2016-06-28 15:05 33 ************** 1 2016-06-28 15:10 36 ***************** 2 2016-06-28 15:15 39 ******************** 3 2016-06-28 15:20 40 ********************* 4 2016-06-28 15:25 40 ********************* 5 2016-06-28 15:30 41 ********************** 6 2016-06-28 15:35 42 *********************** 7 2016-06-28 15:40 43 ************************ 8 2016-06-28 15:45 43 ************************ 9 2016-06-28 15:50 44 ************************* 10 2016-06-28 15:55 44 ************************* 11 2016-06-28 16:00 44 ************************* 12 2016-06-28 16:05 45 ************************** ... ..( 29 skipped). .. ************************** 42 2016-06-28 18:35 45 ************************** 43 2016-06-28 18:40 44 ************************* ... ..( 28 skipped). .. ************************* 72 2016-06-28 21:05 44 ************************* 73 2016-06-28 21:10 43 ************************ ... ..( 8 skipped). .. ************************ 82 2016-06-28 21:55 43 ************************ 83 2016-06-28 22:00 32 ************* 84 2016-06-28 22:05 35 **************** 85 2016-06-28 22:10 36 ***************** 86 2016-06-28 22:15 33 ************** 87 2016-06-28 22:20 32 ************* 88 2016-06-28 22:25 31 ************ ... ..( 2 skipped). .. ************ 91 2016-06-28 22:40 31 ************ 92 2016-06-28 22:45 32 ************* 93 2016-06-28 22:50 32 ************* 94 2016-06-28 22:55 32 ************* 95 2016-06-28 23:00 33 ************** ... ..( 2 skipped). .. ************** 98 2016-06-28 23:15 33 ************** 99 2016-06-28 23:20 34 *************** ... ..( 4 skipped). .. *************** 104 2016-06-28 23:45 34 *************** 105 2016-06-28 23:50 33 ************** 106 2016-06-28 23:55 33 ************** 107 2016-06-29 00:00 36 ***************** 108 2016-06-29 00:05 38 ******************* 109 2016-06-29 00:10 36 ***************** 110 2016-06-29 00:15 36 ***************** 111 2016-06-29 00:20 38 ******************* 112 2016-06-29 00:25 39 ******************** 113 2016-06-29 00:30 37 ****************** 114 2016-06-29 00:35 35 **************** 115 2016-06-29 00:40 36 ***************** 116 2016-06-29 00:45 34 *************** SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP/SMART Log 0x04) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 4 0 Command failed due to ICRC error 0x0002 4 0 R_ERR response for data FIS 0x0003 4 0 R_ERR response for device-to-host data FIS 0x0004 4 0 R_ERR response for host-to-device data FIS 0x0005 4 0 R_ERR response for non-data FIS 0x0006 4 0 R_ERR response for device-to-host non-data FIS 0x0007 4 0 R_ERR response for host-to-device non-data FIS 0x0008 4 0 Device-to-host non-data FIS retries 0x0009 4 6 Transition from drive PhyRdy to drive PhyNRdy 0x000a 4 6 Device-to-host register FISes sent due to a COMRESET 0x000b 4 0 CRC errors within host-to-device FIS 0x000d 4 0 Non-CRC errors within host-to-device FIS 0x000f 4 0 R_ERR response for host-to-device data FIS, CRC 0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC 0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC 0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC 0x8e00 4 0 Vendor specific 0x8e01 4 0 Vendor specific 0x8e02 4 0 Vendor specific 0x8e03 4 0 Vendor specific 0x8e04 4 0 Vendor specific 0x8e05 4 0 Vendor specific 0x8e06 4 0 Vendor specific 0x8e07 4 0 Vendor specific 0x8e08 4 0 Vendor specific 0x8e09 4 0 Vendor specific 0x8e0a 4 0 Vendor specific 0x8e0b 4 0 Vendor specific 0x8e0c 4 0 Vendor specific 0x8e0d 4 0 Vendor specific 0x8e0e 4 0 Vendor specific 0x8e0f 4 0 Vendor specific 0x8e10 4 0 Vendor specific 0x8e11 4 0 Vendor specific ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-28 22:52 ` Saint Germain @ 2016-06-29 4:25 ` Chris Murphy 2016-06-29 9:50 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-29 4:25 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Tue, Jun 28, 2016 at 4:52 PM, Saint Germain <saintger@gmail.com> wrote: > Well I made a ddrescue image of both drives (only one error on sdb > during ddrescue copy) and started the computer again (after > disconnecting the old drives). What was the error? Any kernel message at the time of this error? > I don't know if I should continue trying to repair this RAID1 or if I > should just cp/rsync to a new BTRFS volume and get done with it. Well for sure already you should prepare to lose this volume, so whatever backup you need, do that yesterday. > On the other hand it seems interesting to repair instead of just giving > up. It gives a good look at BTRFS resiliency/reliability. On the one hand Btrfs shouldn't become inconsistent in the first place, that's the design goal. On the other hand, I'm finding from the problems reported on the list that Btrfs increasingly mounts at least read only and allows getting data off, even when the file system isn't fully functional or repairable. In your case, once there are metadata problems even with raid 1, it's difficult at best. But once you have the backup you could try some other things once it's certain the hardware isn't adding to the problems, which I'm still not yet certain of. > > Here is the log from the mount to the scrub aborting and the result > from smartctl. > > Thanks for your precious help so far. > > > BTRFS error (device sdb1): cleaner transaction attach returned -30 Not sure what this is. The Btrfs cleaner is used to remove snapshots, decrement extent reference count, and if the count is 0, then free up that space. So, why is it running? I don't know what -30 means. > BTRFS info (device sdb1): disk space caching is enabled > BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714507, gen 1335 > BTRFS info (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21622, gen 24 I missed something the first time around in these messages: the generation error. Both drives have generation errors. A generation error on a single drive means that drive was not successfully being written to or was missing. For it to happen on both drives is bad. If it happens to just one drive, once it's reappears it will be passively caught up to the other one as reads happen, but best practice for now requires the user to run scrub or balance. If that doesn't happen and a 2nd drive vanishes or has write errors that cause generation mismatches, now both drives are simultaneously behind and ahead of each other. Some commits went to one drive, some went to the other. And right now Btrfs totally flips out and will irreparably get corrupted. So I have to ask if this volume was ever mounted degraded? If not you really need to look at logs and find out why the drives weren't being written to. sdb show lots of write, flush, corruption and generation errors, so it seems like it was having a hardware issue. But then sda has only corruptions and generation problems, as if it wasn't even connected or powered on. OR another possibility is one of the drives was previously cloned (block copied), or snapshot via LVM and you ran into the block level copies gotcha: https://btrfs.wiki.kernel.org/index.php/Gotchas > BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sdb1, sector 54528696, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) Some extent data and its checksum don't match, on sdb. So this file is considered corrupt. Maybe the data is OK and the checksum is wrong? > btrfs_dev_stat_print_on_error: 164 callbacks suppressed > BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714508, gen 1335 > scrub_handle_errored_block: 164 callbacks suppressed > BTRFS error (device sdb1): unable to fixup (regular) error at logical 93445255168 on dev /dev/sdb1 And it can't be fixed, because... > BTRFS warning (device sdb1): checksum error at logical 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset 479232, length 4096, links 1 (path: user/.local/share/zeitgeist/activity.sqlite-wal) The same block on sda also doesn't match checksum. So either both checksums are wrong, or both datas are wrong. You can make these errors "go away" by using btrfs check --repair --init-csum-tree but what this does it it will totally paper over any real corruptions. You will have no idea if they're really corrupt or not without checking them. Looks like most of the messages have to do with files, not metadata although I didn't look at every single line. I think the generations between the two drives is too far off for them to be put back together again. But if the --init-csum-tree starts to clean up the data related errors, you could use rsync -c to compare the files to a backup and see if they are the same and further inspect to see if they're corrupt or not. You definitely don't want corrupt files propagating into your future backups. That's bad news. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 4:25 ` Chris Murphy @ 2016-06-29 9:50 ` Saint Germain 2016-06-29 17:28 ` Chris Murphy 2016-06-29 17:41 ` Saint Germain 0 siblings, 2 replies; 27+ messages in thread From: Saint Germain @ 2016-06-29 9:50 UTC (permalink / raw) Cc: Btrfs BTRFS On Tue, 28 Jun 2016 22:25:32 -0600, Chris Murphy <lists@colorremedies.com> wrote : > > Well I made a ddrescue image of both drives (only one error on sdb > > during ddrescue copy) and started the computer again (after > > disconnecting the old drives). > > What was the error? Any kernel message at the time of this error? > ddrescue reported an error during operation ("error: 1" displayed). Dump of /var/log/syslog during the ddrescue operation is appended at the end of this email. > > > I don't know if I should continue trying to repair this RAID1 or if > > I should just cp/rsync to a new BTRFS volume and get done with it. > > Well for sure already you should prepare to lose this volume, so > whatever backup you need, do that yesterday. Already got a backup. I just really want to try to repair it (in order to test BTRFS). > > On the other hand it seems interesting to repair instead of just > > giving up. It gives a good look at BTRFS resiliency/reliability. > > On the one hand Btrfs shouldn't become inconsistent in the first > place, that's the design goal. On the other hand, I'm finding from the > problems reported on the list that Btrfs increasingly mounts at least > read only and allows getting data off, even when the file system isn't > fully functional or repairable. > > In your case, once there are metadata problems even with raid 1, it's > difficult at best. But once you have the backup you could try some > other things once it's certain the hardware isn't adding to the > problems, which I'm still not yet certain of. > I'm ready to try anything. Let's experiment. > > > > Here is the log from the mount to the scrub aborting and the result > > from smartctl. > > > > Thanks for your precious help so far. > > > > > > BTRFS error (device sdb1): cleaner transaction attach returned -30 > > Not sure what this is. The Btrfs cleaner is used to remove snapshots, > decrement extent reference count, and if the count is 0, then free up > that space. So, why is it running? I don't know what -30 means. > > > > BTRFS info (device sdb1): disk space caching is enabled > > BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, > > flush 7928, corrupt 1714507, gen 1335 BTRFS info (device sdb1): > > bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 21622, gen 24 > > I missed something the first time around in these messages: the > generation error. Both drives have generation errors. A generation > error on a single drive means that drive was not successfully being > written to or was missing. For it to happen on both drives is bad. If > it happens to just one drive, once it's reappears it will be passively > caught up to the other one as reads happen, but best practice for now > requires the user to run scrub or balance. If that doesn't happen and > a 2nd drive vanishes or has write errors that cause generation > mismatches, now both drives are simultaneously behind and ahead of > each other. Some commits went to one drive, some went to the other. > And right now Btrfs totally flips out and will irreparably get > corrupted. > > So I have to ask if this volume was ever mounted degraded? If not you > really need to look at logs and find out why the drives weren't being > written to. sdb show lots of write, flush, corruption and generation > errors, so it seems like it was having a hardware issue. But then sda > has only corruptions and generation problems, as if it wasn't even > connected or powered on. > > OR another possibility is one of the drives was previously cloned > (block copied), or snapshot via LVM and you ran into the block level > copies gotcha: > https://btrfs.wiki.kernel.org/index.php/Gotchas > I got some errors on sdb 2 months ago (I noticed it because it was suddenly mounted read-only). I ran a scrub and a check --repair, and a lot of errors were corrected. I deleted the files which were not repairable and everything was running smoothly since. I ran a scrub a few weeks ago and everything was fine. I never mounted in degraded mode or made a snapshot via LVM (I only upgraded both drives through "replace" 6 months ago). > > > BTRFS warning (device sdb1): checksum error at logical 93445255168 > > on dev /dev/sdb1, sector 54528696, root 5, inode 3434831, offset > > 479232, length 4096, links 1 (path: > > user/.local/share/zeitgeist/activity.sqlite-wal) > > Some extent data and its checksum don't match, on sdb. So this file is > considered corrupt. Maybe the data is OK and the checksum is wrong? > > > btrfs_dev_stat_print_on_error: 164 callbacks suppressed > > BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, > > flush 7928, corrupt 1714508, gen 1335 scrub_handle_errored_block: > > 164 callbacks suppressed BTRFS error (device sdb1): unable to fixup > > (regular) error at logical 93445255168 on dev /dev/sdb1 > > And it can't be fixed, because... > > > BTRFS warning (device sdb1): checksum error at logical 93445255168 > > on dev /dev/sda1, sector 77669048, root 5, inode 3434831, offset > > 479232, length 4096, links 1 (path: > > user/.local/share/zeitgeist/activity.sqlite-wal) > > > The same block on sda also doesn't match checksum. So either both > checksums are wrong, or both datas are wrong. > > You can make these errors "go away" by using btrfs check --repair > --init-csum-tree but what this does it it will totally paper over any > real corruptions. You will have no idea if they're really corrupt or > not without checking them. Looks like most of the messages have to do > with files, not metadata although I didn't look at every single line. > All these files are expendables (it seems to be files which were written only recently). So I can safely delete them if necessary. > I think the generations between the two drives is too far off for them > to be put back together again. But if the --init-csum-tree starts to > clean up the data related errors, you could use rsync -c to compare > the files to a backup and see if they are the same and further inspect > to see if they're corrupt or not. > > You definitely don't want corrupt files propagating into your future > backups. That's bad news. > So if I understand correctly, you advise to use check --repair --init-csum-tree and delete the files which were reported as having checksum error ? After that I can compare the important files to a backup, but there is always the non-important files which are not backuped. Is there anyway I can be sure afterwards that the volume is indeed completely correct and reliable ? If there is no way to be sure, I think it is better that I cp/rsync all data to a new BTRFS volume. Thanks for yout help ! Jun 28 11:34:43 system kernel: [49846.737005] perf: interrupt took too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 79500 Jun 28 11:35:23 system kernel: [49887.681350] ata2.00: exception Emask 0x0 SAct 0x600 SErr 0x0 action 0x6 frozen Jun 28 11:35:23 system kernel: [49887.685258] ata2.00: failed command: READ FPDMA QUEUED Jun 28 11:35:23 system kernel: [49887.689251] ata2.00: cmd 60/00:48:80:d7:70/01:00:0f:00:00/40 tag 9 ncq 131072 in Jun 28 11:35:23 system kernel: [49887.689251] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jun 28 11:35:23 system kernel: [49887.696884] ata2.00: status: { DRDY } Jun 28 11:35:23 system kernel: [49887.700353] ata2.00: failed command: READ FPDMA QUEUED Jun 28 11:35:23 system kernel: [49887.703749] ata2.00: cmd 60/00:50:80:d8:70/01:00:0f:00:00/40 tag 10 ncq 131072 in Jun 28 11:35:23 system kernel: [49887.703749] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jun 28 11:35:23 system kernel: [49887.710243] ata2.00: status: { DRDY } Jun 28 11:35:23 system kernel: [49887.713250] ata2: hard resetting link Jun 28 11:35:24 system kernel: [49888.023941] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Jun 28 11:35:24 system kernel: [49888.037132] ata2.00: configured for UDMA/133 Jun 28 11:35:24 system kernel: [49888.037143] ata2: EH complete Jun 28 11:35:27 system kernel: [49890.782261] ata2.00: exception Emask 0x0 SAct 0x2000 SErr 0x0 action 0x0 Jun 28 11:35:27 system kernel: [49890.783864] ata2.00: irq_stat 0x40000008 Jun 28 11:35:27 system kernel: [49890.785437] ata2.00: failed command: READ FPDMA QUEUED Jun 28 11:35:27 system kernel: [49890.787034] ata2.00: cmd 60/00:68:80:d7:70/01:00:0f:00:00/40 tag 13 ncq 131072 in Jun 28 11:35:27 system kernel: [49890.787034] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 11:35:27 system kernel: [49890.794700] ata2.00: status: { DRDY ERR } Jun 28 11:35:27 system kernel: [49890.796272] ata2.00: error: { UNC } Jun 28 11:35:27 system kernel: [49890.811742] ata2.00: configured for UDMA/133 Jun 28 11:35:27 system kernel: [49890.811755] sd 1:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 11:35:27 system kernel: [49890.811758] sd 1:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] [descriptor] Jun 28 11:35:27 system kernel: [49890.811760] sd 1:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 11:35:27 system kernel: [49890.811763] sd 1:0:0:0: [sdb] tag#13 CDB: Read(10) 28 00 0f 70 d7 80 00 01 00 00 Jun 28 11:35:27 system kernel: [49890.811765] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 11:35:27 system kernel: [49890.813241] ata2: EH complete Jun 28 11:35:29 system kernel: [49893.533391] ata2.00: exception Emask 0x0 SAct 0x4000000 SErr 0x0 action 0x0 Jun 28 11:35:29 system kernel: [49893.534241] ata2.00: irq_stat 0x40000008 Jun 28 11:35:29 system kernel: [49893.535065] ata2.00: failed command: READ FPDMA QUEUED Jun 28 11:35:29 system kernel: [49893.535883] ata2.00: cmd 60/08:d0:08:d8:70/00:00:0f:00:00/40 tag 26 ncq 4096 in Jun 28 11:35:29 system kernel: [49893.535883] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 11:35:29 system kernel: [49893.537511] ata2.00: status: { DRDY ERR } Jun 28 11:35:29 system kernel: [49893.538319] ata2.00: error: { UNC } Jun 28 11:35:29 system kernel: [49893.553353] ata2.00: configured for UDMA/133 Jun 28 11:35:29 system kernel: [49893.553360] sd 1:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 11:35:29 system kernel: [49893.553362] sd 1:0:0:0: [sdb] tag#26 Sense Key : Medium Error [current] [descriptor] Jun 28 11:35:29 system kernel: [49893.553364] sd 1:0:0:0: [sdb] tag#26 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 11:35:29 system kernel: [49893.553365] sd 1:0:0:0: [sdb] tag#26 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 Jun 28 11:35:29 system kernel: [49893.553367] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 11:35:29 system kernel: [49893.554172] Buffer I/O error on dev sdb, logical block 32381697, async page read Jun 28 11:35:29 system kernel: [49893.555125] ata2: EH complete Jun 28 11:35:32 system kernel: [49896.284548] ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 Jun 28 11:35:32 system kernel: [49896.288067] ata2.00: irq_stat 0x40000008 Jun 28 11:35:32 system kernel: [49896.291516] ata2.00: failed command: READ FPDMA QUEUED Jun 28 11:35:32 system kernel: [49896.295130] ata2.00: cmd 60/08:00:08:d8:70/00:00:0f:00:00/40 tag 0 ncq 4096 in Jun 28 11:35:32 system kernel: [49896.295130] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 11:35:32 system kernel: [49896.301235] ata2.00: status: { DRDY ERR } Jun 28 11:35:32 system kernel: [49896.304039] ata2.00: error: { UNC } Jun 28 11:35:32 system kernel: [49896.320080] ata2.00: configured for UDMA/133 Jun 28 11:35:32 system kernel: [49896.320097] sd 1:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 11:35:32 system kernel: [49896.320102] sd 1:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current] [descriptor] Jun 28 11:35:32 system kernel: [49896.320106] sd 1:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 11:35:32 system kernel: [49896.320110] sd 1:0:0:0: [sdb] tag#0 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 Jun 28 11:35:32 system kernel: [49896.320113] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 11:35:32 system kernel: [49896.322349] Buffer I/O error on dev sdb, logical block 32381697, async page read Jun 28 11:35:32 system kernel: [49896.325007] ata2: EH complete Jun 28 11:44:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 11:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 61 to 57 Jun 28 11:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 99 to 97 Jun 28 11:44:13 system smartd[783]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 51 to 53 Jun 28 12:14:12 system smartd[783]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 59 to 56 Jun 28 12:14:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 12:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 57 to 55 Jun 28 12:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 97 to 99 Jun 28 12:14:12 system smartd[783]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 53 to 56 Jun 28 12:44:12 system smartd[783]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 56 to 55 Jun 28 12:44:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 12:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 55 to 54 Jun 28 12:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 99 to 95 Jun 28 13:14:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 13:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 95 to 97 Jun 28 13:44:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 13:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 54 to 55 Jun 28 13:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 97 to 98 Jun 28 14:14:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 14:44:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 15:14:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 15:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 98 to 97 Jun 28 15:44:12 system smartd[783]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 55 to 56 Jun 28 15:44:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 15:44:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 97 to 95 Jun 28 16:03:34 system dhclient: DHCPREQUEST on eth0 to 192.168.1.1 port 67 Jun 28 16:03:34 system dhclient: DHCPACK from 192.168.1.1 Jun 28 16:03:34 system NetworkManager[774]: <info> (eth0): DHCPv4 state changed renew -> renew Jun 28 16:03:34 system NetworkManager[774]: <info> address 192.168.1.200 Jun 28 16:03:34 system NetworkManager[774]: <info> plen 24 (255.255.255.0) Jun 28 16:03:34 system NetworkManager[774]: <info> gateway 192.168.1.1 Jun 28 16:03:34 system NetworkManager[774]: <info> server identifier 192.168.1.1 Jun 28 16:03:34 system NetworkManager[774]: <info> lease time 86400 Jun 28 16:03:34 system NetworkManager[774]: <info> hostname 'system_ethernet' Jun 28 16:03:34 system NetworkManager[774]: <info> nameserver '192.168.1.1' Jun 28 16:03:34 system dbus[801]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' Jun 28 16:03:34 system dhclient: bound to 192.168.1.200 -- renewal in 32692 seconds. Jun 28 16:03:34 system dbus[801]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Jun 28 16:03:34 system nm-dispatcher: Dispatching action 'dhcp4-change' for eth0 Jun 28 16:14:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 16:14:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 55 to 56 Jun 28 16:14:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 95 to 99 Jun 28 16:14:13 system smartd[783]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 56 to 55 Jun 28 16:16:13 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 16:21:50 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 16:38:20 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 16:44:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 16:44:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 99 to 96 Jun 28 16:50:21 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:02:21 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:14:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 17:14:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 96 to 100 Jun 28 17:14:21 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:21:24 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:28:25 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:40:34 system avahi-daemon[800]: Invalid response packet from host 192.168.1.32. Jun 28 17:44:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 17:44:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 100 to 99 Jun 28 18:14:12 system smartd[783]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 56 to 57 Jun 28 18:14:12 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 18:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 56 to 57 Jun 28 18:14:12 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 99 to 100 Jun 28 18:14:12 system smartd[783]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 55 to 54 Jun 28 18:44:13 system smartd[783]: Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Jun 28 18:44:13 system smartd[783]: Device: /dev/sdb [SAT], SMART Usage Attribute: 242 Total_LBAs_Read changed from 100 to 96 Jun 28 18:45:44 system kernel: [75710.247647] ata2.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x0 Jun 28 18:45:44 system kernel: [75710.249226] ata2.00: irq_stat 0x40000008 Jun 28 18:45:44 system kernel: [75710.250775] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:45:44 system kernel: [75710.254223] ata2.00: cmd 60/08:80:08:d8:70/00:00:0f:00:00/40 tag 16 ncq 4096 in Jun 28 18:45:44 system kernel: [75710.254223] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:45:44 system kernel: [75710.260891] ata2.00: status: { DRDY ERR } Jun 28 18:45:44 system kernel: [75710.262803] ata2.00: error: { UNC } Jun 28 18:45:44 system kernel: [75710.277857] ata2.00: configured for UDMA/133 Jun 28 18:45:44 system kernel: [75710.277868] sd 1:0:0:0: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:45:44 system kernel: [75710.277871] sd 1:0:0:0: [sdb] tag#16 Sense Key : Medium Error [current] [descriptor] Jun 28 18:45:44 system kernel: [75710.277874] sd 1:0:0:0: [sdb] tag#16 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:45:44 system kernel: [75710.277877] sd 1:0:0:0: [sdb] tag#16 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 Jun 28 18:45:44 system kernel: [75710.277879] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:45:44 system kernel: [75710.279274] ata2: EH complete Jun 28 18:45:47 system kernel: [75713.009880] ata2.00: exception Emask 0x0 SAct 0x200000 SErr 0x0 action 0x0 Jun 28 18:45:47 system kernel: [75713.012309] ata2.00: irq_stat 0x40000008 Jun 28 18:45:47 system kernel: [75713.014680] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:45:47 system kernel: [75713.017163] ata2.00: cmd 60/08:a8:08:d8:70/00:00:0f:00:00/40 tag 21 ncq 4096 in Jun 28 18:45:47 system kernel: [75713.017163] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:45:47 system kernel: [75713.021895] ata2.00: status: { DRDY ERR } Jun 28 18:45:47 system kernel: [75713.024094] ata2.00: error: { UNC } Jun 28 18:45:47 system kernel: [75713.039089] ata2.00: configured for UDMA/133 Jun 28 18:45:47 system kernel: [75713.039102] sd 1:0:0:0: [sdb] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:45:47 system kernel: [75713.039106] sd 1:0:0:0: [sdb] tag#21 Sense Key : Medium Error [current] [descriptor] Jun 28 18:45:47 system kernel: [75713.039109] sd 1:0:0:0: [sdb] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:45:47 system kernel: [75713.039112] sd 1:0:0:0: [sdb] tag#21 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 Jun 28 18:45:47 system kernel: [75713.039114] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:45:47 system kernel: [75713.040604] Buffer I/O error on dev sdb, logical block 32381697, async page read Jun 28 18:45:47 system kernel: [75713.042117] ata2: EH complete Jun 28 18:45:50 system kernel: [75715.816494] ata2.00: exception Emask 0x0 SAct 0x20000000 SErr 0x0 action 0x0 Jun 28 18:45:50 system kernel: [75715.819694] ata2.00: irq_stat 0x40000008 Jun 28 18:45:50 system kernel: [75715.822831] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:45:50 system kernel: [75715.826094] ata2.00: cmd 60/08:e8:08:d8:70/00:00:0f:00:00/40 tag 29 ncq 4096 in Jun 28 18:45:50 system kernel: [75715.826094] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:45:50 system kernel: [75715.832324] ata2.00: status: { DRDY ERR } Jun 28 18:45:50 system kernel: [75715.835140] ata2.00: error: { UNC } Jun 28 18:45:50 system kernel: [75715.852239] ata2.00: configured for UDMA/133 Jun 28 18:45:50 system kernel: [75715.852263] sd 1:0:0:0: [sdb] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:45:50 system kernel: [75715.852271] sd 1:0:0:0: [sdb] tag#29 Sense Key : Medium Error [current] [descriptor] Jun 28 18:45:50 system kernel: [75715.852277] sd 1:0:0:0: [sdb] tag#29 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:45:50 system kernel: [75715.852285] sd 1:0:0:0: [sdb] tag#29 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 Jun 28 18:45:50 system kernel: [75715.852290] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:45:50 system kernel: [75715.856216] Buffer I/O error on dev sdb, logical block 32381697, async page read Jun 28 18:45:50 system kernel: [75715.859438] ata2: EH complete Jun 28 18:45:53 system kernel: [75718.634863] sdd: sdd1 sdd2 Jun 28 18:47:07 system kernel: [75793.402923] ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0 Jun 28 18:47:07 system kernel: [75793.404224] ata2.00: irq_stat 0x40000008 Jun 28 18:47:07 system kernel: [75793.405487] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:07 system kernel: [75793.406743] ata2.00: cmd 60/01:70:09:d8:70/00:00:0f:00:00/40 tag 14 ncq 512 in Jun 28 18:47:07 system kernel: [75793.406743] res 41/40:00:09:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:07 system kernel: [75793.409295] ata2.00: status: { DRDY ERR } Jun 28 18:47:07 system kernel: [75793.412577] ata2.00: error: { UNC } Jun 28 18:47:07 system kernel: [75793.428802] ata2.00: configured for UDMA/133 Jun 28 18:47:07 system kernel: [75793.428819] sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:07 system kernel: [75793.428824] sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:07 system kernel: [75793.428828] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:07 system kernel: [75793.428832] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 0f 70 d8 09 00 00 01 00 Jun 28 18:47:07 system kernel: [75793.428835] blk_update_request: I/O error, dev sdb, sector 259053577 Jun 28 18:47:07 system kernel: [75793.431406] ata2: EH complete Jun 28 18:47:10 system kernel: [75796.154073] ata2.00: exception Emask 0x0 SAct 0x80000 SErr 0x0 action 0x0 Jun 28 18:47:10 system kernel: [75796.157105] ata2.00: irq_stat 0x40000008 Jun 28 18:47:10 system kernel: [75796.160078] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:10 system kernel: [75796.163212] ata2.00: cmd 60/01:98:0a:d8:70/00:00:0f:00:00/40 tag 19 ncq 512 in Jun 28 18:47:10 system kernel: [75796.163212] res 41/40:00:0a:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:10 system kernel: [75796.169209] ata2.00: status: { DRDY ERR } Jun 28 18:47:10 system kernel: [75796.171919] ata2.00: error: { UNC } Jun 28 18:47:10 system kernel: [75796.187795] ata2.00: configured for UDMA/133 Jun 28 18:47:10 system kernel: [75796.187813] sd 1:0:0:0: [sdb] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:10 system kernel: [75796.187819] sd 1:0:0:0: [sdb] tag#19 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:10 system kernel: [75796.187823] sd 1:0:0:0: [sdb] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:10 system kernel: [75796.187827] sd 1:0:0:0: [sdb] tag#19 CDB: Read(10) 28 00 0f 70 d8 0a 00 00 01 00 Jun 28 18:47:10 system kernel: [75796.187831] blk_update_request: I/O error, dev sdb, sector 259053578 Jun 28 18:47:10 system kernel: [75796.190051] ata2: EH complete Jun 28 18:47:13 system kernel: [75798.916308] ata2.00: exception Emask 0x0 SAct 0x4 SErr 0x0 action 0x0 Jun 28 18:47:13 system kernel: [75798.919315] ata2.00: irq_stat 0x40000008 Jun 28 18:47:13 system kernel: [75798.922296] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:13 system kernel: [75798.926192] ata2.00: cmd 60/01:10:0b:d8:70/00:00:0f:00:00/40 tag 2 ncq 512 in Jun 28 18:47:13 system kernel: [75798.926192] res 41/40:00:0b:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:13 system kernel: [75798.932238] ata2.00: status: { DRDY ERR } Jun 28 18:47:13 system kernel: [75798.934953] ata2.00: error: { UNC } Jun 28 18:47:13 system kernel: [75798.950911] ata2.00: configured for UDMA/133 Jun 28 18:47:13 system kernel: [75798.950932] sd 1:0:0:0: [sdb] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:13 system kernel: [75798.950937] sd 1:0:0:0: [sdb] tag#2 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:13 system kernel: [75798.950942] sd 1:0:0:0: [sdb] tag#2 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:13 system kernel: [75798.950947] sd 1:0:0:0: [sdb] tag#2 CDB: Read(10) 28 00 0f 70 d8 0b 00 00 01 00 Jun 28 18:47:13 system kernel: [75798.950951] blk_update_request: I/O error, dev sdb, sector 259053579 Jun 28 18:47:13 system kernel: [75798.953386] ata2: EH complete Jun 28 18:47:16 system kernel: [75801.678506] ata2.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x0 Jun 28 18:47:16 system kernel: [75801.679200] ata2.00: irq_stat 0x40000008 Jun 28 18:47:16 system kernel: [75801.679875] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:16 system kernel: [75801.680558] ata2.00: cmd 60/01:90:0c:d8:70/00:00:0f:00:00/40 tag 18 ncq 512 in Jun 28 18:47:16 system kernel: [75801.680558] res 41/40:00:0c:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:16 system kernel: [75801.683620] ata2.00: status: { DRDY ERR } Jun 28 18:47:16 system kernel: [75801.685104] ata2.00: error: { UNC } Jun 28 18:47:16 system kernel: [75801.699814] ata2.00: configured for UDMA/133 Jun 28 18:47:16 system kernel: [75801.699822] sd 1:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:16 system kernel: [75801.699824] sd 1:0:0:0: [sdb] tag#18 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:16 system kernel: [75801.699825] sd 1:0:0:0: [sdb] tag#18 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:16 system kernel: [75801.699827] sd 1:0:0:0: [sdb] tag#18 CDB: Read(10) 28 00 0f 70 d8 0c 00 00 01 00 Jun 28 18:47:16 system kernel: [75801.699828] blk_update_request: I/O error, dev sdb, sector 259053580 Jun 28 18:47:16 system kernel: [75801.700513] ata2: EH complete Jun 28 18:47:18 system kernel: [75804.429671] ata2.00: exception Emask 0x0 SAct 0x4 SErr 0x0 action 0x0 Jun 28 18:47:18 system kernel: [75804.432653] ata2.00: irq_stat 0x40000008 Jun 28 18:47:18 system kernel: [75804.435586] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:18 system kernel: [75804.438532] ata2.00: cmd 60/01:10:0d:d8:70/00:00:0f:00:00/40 tag 2 ncq 512 in Jun 28 18:47:18 system kernel: [75804.438532] res 41/40:00:0d:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:18 system kernel: [75804.444640] ata2.00: status: { DRDY ERR } Jun 28 18:47:18 system kernel: [75804.447298] ata2.00: error: { UNC } Jun 28 18:47:18 system kernel: [75804.463171] ata2.00: configured for UDMA/133 Jun 28 18:47:18 system kernel: [75804.463191] sd 1:0:0:0: [sdb] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:18 system kernel: [75804.463197] sd 1:0:0:0: [sdb] tag#2 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:18 system kernel: [75804.463201] sd 1:0:0:0: [sdb] tag#2 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:18 system kernel: [75804.463206] sd 1:0:0:0: [sdb] tag#2 CDB: Read(10) 28 00 0f 70 d8 0d 00 00 01 00 Jun 28 18:47:18 system kernel: [75804.463209] blk_update_request: I/O error, dev sdb, sector 259053581 Jun 28 18:47:18 system kernel: [75804.465591] ata2: EH complete Jun 28 18:47:21 system kernel: [75807.191900] ata2.00: exception Emask 0x0 SAct 0x20000 SErr 0x0 action 0x0 Jun 28 18:47:21 system kernel: [75807.194879] ata2.00: irq_stat 0x40000008 Jun 28 18:47:21 system kernel: [75807.197469] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:21 system kernel: [75807.200204] ata2.00: cmd 60/01:88:0e:d8:70/00:00:0f:00:00/40 tag 17 ncq 512 in Jun 28 18:47:21 system kernel: [75807.200204] res 41/40:00:0e:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:21 system kernel: [75807.205435] ata2.00: status: { DRDY ERR } Jun 28 18:47:21 system kernel: [75807.207823] ata2.00: error: { UNC } Jun 28 18:47:21 system kernel: [75807.223069] ata2.00: configured for UDMA/133 Jun 28 18:47:21 system kernel: [75807.223087] sd 1:0:0:0: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:21 system kernel: [75807.223092] sd 1:0:0:0: [sdb] tag#17 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:21 system kernel: [75807.223096] sd 1:0:0:0: [sdb] tag#17 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:21 system kernel: [75807.223100] sd 1:0:0:0: [sdb] tag#17 CDB: Read(10) 28 00 0f 70 d8 0e 00 00 01 00 Jun 28 18:47:21 system kernel: [75807.223103] blk_update_request: I/O error, dev sdb, sector 259053582 Jun 28 18:47:21 system kernel: [75807.225080] ata2: EH complete Jun 28 18:47:24 system kernel: [75809.954101] ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 Jun 28 18:47:24 system kernel: [75809.954785] ata2.00: irq_stat 0x40000008 Jun 28 18:47:24 system kernel: [75809.955447] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:24 system kernel: [75809.956111] ata2.00: cmd 60/01:08:08:d8:70/00:00:0f:00:00/40 tag 1 ncq 512 in Jun 28 18:47:24 system kernel: [75809.956111] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:24 system kernel: [75809.957465] ata2.00: status: { DRDY ERR } Jun 28 18:47:24 system kernel: [75809.958127] ata2.00: error: { UNC } Jun 28 18:47:24 system kernel: [75809.971936] ata2.00: configured for UDMA/133 Jun 28 18:47:24 system kernel: [75809.971944] sd 1:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:24 system kernel: [75809.971946] sd 1:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:24 system kernel: [75809.971947] sd 1:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:24 system kernel: [75809.971949] sd 1:0:0:0: [sdb] tag#1 CDB: Read(10) 28 00 0f 70 d8 08 00 00 01 00 Jun 28 18:47:24 system kernel: [75809.971950] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:47:24 system kernel: [75809.972624] ata2: EH complete Jun 28 18:47:27 system kernel: [75812.705263] ata2.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x0 Jun 28 18:47:27 system kernel: [75812.708197] ata2.00: irq_stat 0x40000008 Jun 28 18:47:27 system kernel: [75812.711087] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:27 system kernel: [75812.714039] ata2.00: cmd 60/01:80:09:d8:70/00:00:0f:00:00/40 tag 16 ncq 512 in Jun 28 18:47:27 system kernel: [75812.714039] res 41/40:00:09:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:27 system kernel: [75812.719889] ata2.00: status: { DRDY ERR } Jun 28 18:47:27 system kernel: [75812.722480] ata2.00: error: { UNC } Jun 28 18:47:27 system kernel: [75812.738338] ata2.00: configured for UDMA/133 Jun 28 18:47:27 system kernel: [75812.738358] sd 1:0:0:0: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:27 system kernel: [75812.738364] sd 1:0:0:0: [sdb] tag#16 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:27 system kernel: [75812.738369] sd 1:0:0:0: [sdb] tag#16 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:27 system kernel: [75812.738374] sd 1:0:0:0: [sdb] tag#16 CDB: Read(10) 28 00 0f 70 d8 09 00 00 01 00 Jun 28 18:47:27 system kernel: [75812.738377] blk_update_request: I/O error, dev sdb, sector 259053577 Jun 28 18:47:27 system kernel: [75812.740705] ata2: EH complete Jun 28 18:47:29 system kernel: [75815.467499] ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 Jun 28 18:47:29 system kernel: [75815.470432] ata2.00: irq_stat 0x40000008 Jun 28 18:47:29 system kernel: [75815.473288] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:29 system kernel: [75815.476150] ata2.00: cmd 60/01:00:0a:d8:70/00:00:0f:00:00/40 tag 0 ncq 512 in Jun 28 18:47:29 system kernel: [75815.476150] res 41/40:00:0a:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:29 system kernel: [75815.482017] ata2.00: status: { DRDY ERR } Jun 28 18:47:29 system kernel: [75815.484556] ata2.00: error: { UNC } Jun 28 18:47:29 system kernel: [75815.500453] ata2.00: configured for UDMA/133 Jun 28 18:47:29 system kernel: [75815.500472] sd 1:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:29 system kernel: [75815.500477] sd 1:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:29 system kernel: [75815.500481] sd 1:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:29 system kernel: [75815.500486] sd 1:0:0:0: [sdb] tag#0 CDB: Read(10) 28 00 0f 70 d8 0a 00 00 01 00 Jun 28 18:47:29 system kernel: [75815.500489] blk_update_request: I/O error, dev sdb, sector 259053578 Jun 28 18:47:29 system kernel: [75815.502596] ata2: EH complete Jun 28 18:47:32 system kernel: [75818.229731] ata2.00: exception Emask 0x0 SAct 0x8000 SErr 0x0 action 0x0 Jun 28 18:47:32 system kernel: [75818.232312] ata2.00: irq_stat 0x40000008 Jun 28 18:47:32 system kernel: [75818.234886] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:32 system kernel: [75818.237860] ata2.00: cmd 60/01:78:0b:d8:70/00:00:0f:00:00/40 tag 15 ncq 512 in Jun 28 18:47:32 system kernel: [75818.237860] res 41/40:00:0b:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:32 system kernel: [75818.243629] ata2.00: status: { DRDY ERR } Jun 28 18:47:32 system kernel: [75818.245984] ata2.00: error: { UNC } Jun 28 18:47:32 system kernel: [75818.261562] ata2.00: configured for UDMA/133 Jun 28 18:47:32 system kernel: [75818.261581] sd 1:0:0:0: [sdb] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:32 system kernel: [75818.261586] sd 1:0:0:0: [sdb] tag#15 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:32 system kernel: [75818.261590] sd 1:0:0:0: [sdb] tag#15 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:32 system kernel: [75818.261595] sd 1:0:0:0: [sdb] tag#15 CDB: Read(10) 28 00 0f 70 d8 0b 00 00 01 00 Jun 28 18:47:32 system kernel: [75818.261598] blk_update_request: I/O error, dev sdb, sector 259053579 Jun 28 18:47:32 system kernel: [75818.263682] ata2: EH complete Jun 28 18:47:35 system kernel: [75821.003002] ata2.00: exception Emask 0x0 SAct 0x40000000 SErr 0x0 action 0x0 Jun 28 18:47:35 system kernel: [75821.003677] ata2.00: irq_stat 0x40000008 Jun 28 18:47:35 system kernel: [75821.004332] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:35 system kernel: [75821.004985] ata2.00: cmd 60/01:f0:0c:d8:70/00:00:0f:00:00/40 tag 30 ncq 512 in Jun 28 18:47:35 system kernel: [75821.004985] res 41/40:00:0c:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:35 system kernel: [75821.006304] ata2.00: status: { DRDY ERR } Jun 28 18:47:35 system kernel: [75821.006996] ata2.00: error: { UNC } Jun 28 18:47:35 system kernel: [75821.020725] ata2.00: configured for UDMA/133 Jun 28 18:47:35 system kernel: [75821.020732] sd 1:0:0:0: [sdb] tag#30 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:35 system kernel: [75821.020734] sd 1:0:0:0: [sdb] tag#30 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:35 system kernel: [75821.020736] sd 1:0:0:0: [sdb] tag#30 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:35 system kernel: [75821.020737] sd 1:0:0:0: [sdb] tag#30 CDB: Read(10) 28 00 0f 70 d8 0c 00 00 01 00 Jun 28 18:47:35 system kernel: [75821.020739] blk_update_request: I/O error, dev sdb, sector 259053580 Jun 28 18:47:35 system kernel: [75821.021400] ata2: EH complete Jun 28 18:47:38 system kernel: [75823.743062] ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0 Jun 28 18:47:38 system kernel: [75823.743872] ata2.00: irq_stat 0x40000008 Jun 28 18:47:38 system kernel: [75823.744661] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:38 system kernel: [75823.745446] ata2.00: cmd 60/01:70:0d:d8:70/00:00:0f:00:00/40 tag 14 ncq 512 in Jun 28 18:47:38 system kernel: [75823.745446] res 41/40:00:0d:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:38 system kernel: [75823.747035] ata2.00: status: { DRDY ERR } Jun 28 18:47:38 system kernel: [75823.749511] ata2.00: error: { UNC } Jun 28 18:47:38 system kernel: [75823.764986] ata2.00: configured for UDMA/133 Jun 28 18:47:38 system kernel: [75823.764997] sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:38 system kernel: [75823.765000] sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:38 system kernel: [75823.765002] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:38 system kernel: [75823.765005] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 0f 70 d8 0d 00 00 01 00 Jun 28 18:47:38 system kernel: [75823.765007] blk_update_request: I/O error, dev sdb, sector 259053581 Jun 28 18:47:38 system kernel: [75823.766111] ata2: EH complete Jun 28 18:47:40 system kernel: [75826.494226] ata2.00: exception Emask 0x0 SAct 0x20000000 SErr 0x0 action 0x0 Jun 28 18:47:40 system kernel: [75826.497139] ata2.00: irq_stat 0x40000008 Jun 28 18:47:40 system kernel: [75826.500020] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:40 system kernel: [75826.503200] ata2.00: cmd 60/01:e8:0e:d8:70/00:00:0f:00:00/40 tag 29 ncq 512 in Jun 28 18:47:40 system kernel: [75826.503200] res 41/40:00:0e:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:40 system kernel: [75826.509006] ata2.00: status: { DRDY ERR } Jun 28 18:47:40 system kernel: [75826.511670] ata2.00: error: { UNC } Jun 28 18:47:40 system kernel: [75826.527534] ata2.00: configured for UDMA/133 Jun 28 18:47:40 system kernel: [75826.527554] sd 1:0:0:0: [sdb] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:40 system kernel: [75826.527560] sd 1:0:0:0: [sdb] tag#29 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:40 system kernel: [75826.527564] sd 1:0:0:0: [sdb] tag#29 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:40 system kernel: [75826.527569] sd 1:0:0:0: [sdb] tag#29 CDB: Read(10) 28 00 0f 70 d8 0e 00 00 01 00 Jun 28 18:47:40 system kernel: [75826.527573] blk_update_request: I/O error, dev sdb, sector 259053582 Jun 28 18:47:40 system kernel: [75826.529882] ata2: EH complete Jun 28 18:47:43 system kernel: [75829.256439] ata2.00: exception Emask 0x0 SAct 0x2000 SErr 0x0 action 0x0 Jun 28 18:47:43 system kernel: [75829.257615] ata2.00: irq_stat 0x40000008 Jun 28 18:47:43 system kernel: [75829.258774] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:43 system kernel: [75829.261778] ata2.00: cmd 60/01:68:0f:d8:70/00:00:0f:00:00/40 tag 13 ncq 512 in Jun 28 18:47:43 system kernel: [75829.261778] res 41/40:00:0f:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:43 system kernel: [75829.267570] ata2.00: status: { DRDY ERR } Jun 28 18:47:43 system kernel: [75829.269152] ata2.00: error: { UNC } Jun 28 18:47:43 system kernel: [75829.283951] ata2.00: configured for UDMA/133 Jun 28 18:47:43 system kernel: [75829.283966] sd 1:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:43 system kernel: [75829.283969] sd 1:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:43 system kernel: [75829.283972] sd 1:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:43 system kernel: [75829.283976] sd 1:0:0:0: [sdb] tag#13 CDB: Read(10) 28 00 0f 70 d8 0f 00 00 01 00 Jun 28 18:47:43 system kernel: [75829.283978] blk_update_request: I/O error, dev sdb, sector 259053583 Jun 28 18:47:43 system kernel: [75829.285422] ata2: EH complete Jun 28 18:47:46 system kernel: [75832.007561] ata2.00: exception Emask 0x0 SAct 0x10000000 SErr 0x0 action 0x0 Jun 28 18:47:46 system kernel: [75832.008238] ata2.00: irq_stat 0x40000008 Jun 28 18:47:46 system kernel: [75832.008896] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:46 system kernel: [75832.009551] ata2.00: cmd 60/01:e0:0f:d8:70/00:00:0f:00:00/40 tag 28 ncq 512 in Jun 28 18:47:46 system kernel: [75832.009551] res 41/40:00:0f:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:46 system kernel: [75832.010882] ata2.00: status: { DRDY ERR } Jun 28 18:47:46 system kernel: [75832.012345] ata2.00: error: { UNC } Jun 28 18:47:46 system kernel: [75832.026973] ata2.00: configured for UDMA/133 Jun 28 18:47:46 system kernel: [75832.026981] sd 1:0:0:0: [sdb] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:46 system kernel: [75832.026983] sd 1:0:0:0: [sdb] tag#28 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:46 system kernel: [75832.026985] sd 1:0:0:0: [sdb] tag#28 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:46 system kernel: [75832.026987] sd 1:0:0:0: [sdb] tag#28 CDB: Read(10) 28 00 0f 70 d8 0f 00 00 01 00 Jun 28 18:47:46 system kernel: [75832.026988] blk_update_request: I/O error, dev sdb, sector 259053583 Jun 28 18:47:46 system kernel: [75832.027654] ata2: EH complete Jun 28 18:47:49 system kernel: [75834.758706] ata2.00: exception Emask 0x0 SAct 0x1000 SErr 0x0 action 0x0 Jun 28 18:47:49 system kernel: [75834.759523] ata2.00: irq_stat 0x40000008 Jun 28 18:47:49 system kernel: [75834.760319] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:49 system kernel: [75834.761111] ata2.00: cmd 60/01:60:0e:d8:70/00:00:0f:00:00/40 tag 12 ncq 512 in Jun 28 18:47:49 system kernel: [75834.761111] res 41/40:00:0e:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:49 system kernel: [75834.762712] ata2.00: status: { DRDY ERR } Jun 28 18:47:49 system kernel: [75834.765211] ata2.00: error: { UNC } Jun 28 18:47:49 system kernel: [75834.780699] ata2.00: configured for UDMA/133 Jun 28 18:47:49 system kernel: [75834.780707] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:49 system kernel: [75834.780709] sd 1:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:49 system kernel: [75834.780711] sd 1:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:49 system kernel: [75834.780712] sd 1:0:0:0: [sdb] tag#12 CDB: Read(10) 28 00 0f 70 d8 0e 00 00 01 00 Jun 28 18:47:49 system kernel: [75834.780713] blk_update_request: I/O error, dev sdb, sector 259053582 Jun 28 18:47:49 system kernel: [75834.781377] ata2: EH complete Jun 28 18:47:51 system kernel: [75837.509851] ata2.00: exception Emask 0x0 SAct 0x8000000 SErr 0x0 action 0x0 Jun 28 18:47:51 system kernel: [75837.511330] ata2.00: irq_stat 0x40000008 Jun 28 18:47:51 system kernel: [75837.512771] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:51 system kernel: [75837.514209] ata2.00: cmd 60/01:d8:0d:d8:70/00:00:0f:00:00/40 tag 27 ncq 512 in Jun 28 18:47:51 system kernel: [75837.514209] res 41/40:00:0d:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:51 system kernel: [75837.517114] ata2.00: status: { DRDY ERR } Jun 28 18:47:51 system kernel: [75837.520241] ata2.00: error: { UNC } Jun 28 18:47:51 system kernel: [75837.534463] ata2.00: configured for UDMA/133 Jun 28 18:47:51 system kernel: [75837.534472] sd 1:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:51 system kernel: [75837.534474] sd 1:0:0:0: [sdb] tag#27 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:51 system kernel: [75837.534477] sd 1:0:0:0: [sdb] tag#27 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:51 system kernel: [75837.534479] sd 1:0:0:0: [sdb] tag#27 CDB: Read(10) 28 00 0f 70 d8 0d 00 00 01 00 Jun 28 18:47:51 system kernel: [75837.534480] blk_update_request: I/O error, dev sdb, sector 259053581 Jun 28 18:47:51 system kernel: [75837.535423] ata2: EH complete Jun 28 18:47:54 system kernel: [75840.261004] ata2.00: exception Emask 0x0 SAct 0x800 SErr 0x0 action 0x0 Jun 28 18:47:54 system kernel: [75840.263953] ata2.00: irq_stat 0x40000008 Jun 28 18:47:54 system kernel: [75840.266842] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:54 system kernel: [75840.269741] ata2.00: cmd 60/01:58:0c:d8:70/00:00:0f:00:00/40 tag 11 ncq 512 in Jun 28 18:47:54 system kernel: [75840.269741] res 41/40:00:0c:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:54 system kernel: [75840.275705] ata2.00: status: { DRDY ERR } Jun 28 18:47:54 system kernel: [75840.278279] ata2.00: error: { UNC } Jun 28 18:47:54 system kernel: [75840.294204] ata2.00: configured for UDMA/133 Jun 28 18:47:54 system kernel: [75840.294225] sd 1:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:54 system kernel: [75840.294230] sd 1:0:0:0: [sdb] tag#11 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:54 system kernel: [75840.294235] sd 1:0:0:0: [sdb] tag#11 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:54 system kernel: [75840.294240] sd 1:0:0:0: [sdb] tag#11 CDB: Read(10) 28 00 0f 70 d8 0c 00 00 01 00 Jun 28 18:47:54 system kernel: [75840.294244] blk_update_request: I/O error, dev sdb, sector 259053580 Jun 28 18:47:54 system kernel: [75840.296588] ata2: EH complete Jun 28 18:47:57 system kernel: [75843.023208] ata2.00: exception Emask 0x0 SAct 0x4000000 SErr 0x0 action 0x0 Jun 28 18:47:57 system kernel: [75843.024342] ata2.00: irq_stat 0x40000008 Jun 28 18:47:57 system kernel: [75843.025445] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:47:57 system kernel: [75843.026543] ata2.00: cmd 60/01:d0:0b:d8:70/00:00:0f:00:00/40 tag 26 ncq 512 in Jun 28 18:47:57 system kernel: [75843.026543] res 41/40:00:0b:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:47:57 system kernel: [75843.028757] ata2.00: status: { DRDY ERR } Jun 28 18:47:57 system kernel: [75843.031683] ata2.00: error: { UNC } Jun 28 18:47:57 system kernel: [75843.047781] ata2.00: configured for UDMA/133 Jun 28 18:47:57 system kernel: [75843.047798] sd 1:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:47:57 system kernel: [75843.047803] sd 1:0:0:0: [sdb] tag#26 Sense Key : Medium Error [current] [descriptor] Jun 28 18:47:57 system kernel: [75843.047807] sd 1:0:0:0: [sdb] tag#26 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:47:57 system kernel: [75843.047812] sd 1:0:0:0: [sdb] tag#26 CDB: Read(10) 28 00 0f 70 d8 0b 00 00 01 00 Jun 28 18:47:57 system kernel: [75843.047815] blk_update_request: I/O error, dev sdb, sector 259053579 Jun 28 18:47:57 system kernel: [75843.049799] ata2: EH complete Jun 28 18:48:00 system kernel: [75845.785467] ata2.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 0x0 Jun 28 18:48:00 system kernel: [75845.788096] ata2.00: irq_stat 0x40000008 Jun 28 18:48:00 system kernel: [75845.790665] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:00 system kernel: [75845.793238] ata2.00: cmd 60/01:50:0a:d8:70/00:00:0f:00:00/40 tag 10 ncq 512 in Jun 28 18:48:00 system kernel: [75845.793238] res 41/40:00:0a:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:00 system kernel: [75845.799173] ata2.00: status: { DRDY ERR } Jun 28 18:48:00 system kernel: [75845.801521] ata2.00: error: { UNC } Jun 28 18:48:00 system kernel: [75845.817189] ata2.00: configured for UDMA/133 Jun 28 18:48:00 system kernel: [75845.817207] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:00 system kernel: [75845.817212] sd 1:0:0:0: [sdb] tag#10 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:00 system kernel: [75845.817216] sd 1:0:0:0: [sdb] tag#10 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:00 system kernel: [75845.817220] sd 1:0:0:0: [sdb] tag#10 CDB: Read(10) 28 00 0f 70 d8 0a 00 00 01 00 Jun 28 18:48:00 system kernel: [75845.817223] blk_update_request: I/O error, dev sdb, sector 259053578 Jun 28 18:48:00 system kernel: [75845.819164] ata2: EH complete Jun 28 18:48:02 system kernel: [75848.547661] ata2.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0 Jun 28 18:48:02 system kernel: [75848.548345] ata2.00: irq_stat 0x40000008 Jun 28 18:48:02 system kernel: [75848.549008] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:02 system kernel: [75848.549668] ata2.00: cmd 60/01:c8:09:d8:70/00:00:0f:00:00/40 tag 25 ncq 512 in Jun 28 18:48:02 system kernel: [75848.549668] res 41/40:00:09:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:02 system kernel: [75848.550997] ata2.00: status: { DRDY ERR } Jun 28 18:48:02 system kernel: [75848.551663] ata2.00: error: { UNC } Jun 28 18:48:02 system kernel: [75848.566255] ata2.00: configured for UDMA/133 Jun 28 18:48:02 system kernel: [75848.566263] sd 1:0:0:0: [sdb] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:02 system kernel: [75848.566265] sd 1:0:0:0: [sdb] tag#25 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:02 system kernel: [75848.566267] sd 1:0:0:0: [sdb] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:02 system kernel: [75848.566269] sd 1:0:0:0: [sdb] tag#25 CDB: Read(10) 28 00 0f 70 d8 09 00 00 01 00 Jun 28 18:48:02 system kernel: [75848.566270] blk_update_request: I/O error, dev sdb, sector 259053577 Jun 28 18:48:02 system kernel: [75848.566938] ata2: EH complete Jun 28 18:48:05 system kernel: [75851.298823] ata2.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0 Jun 28 18:48:05 system kernel: [75851.301447] ata2.00: irq_stat 0x40000008 Jun 28 18:48:05 system kernel: [75851.304035] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:05 system kernel: [75851.307092] ata2.00: cmd 60/01:48:08:d8:70/00:00:0f:00:00/40 tag 9 ncq 512 in Jun 28 18:48:05 system kernel: [75851.307092] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:05 system kernel: [75851.312926] ata2.00: status: { DRDY ERR } Jun 28 18:48:05 system kernel: [75851.315313] ata2.00: error: { UNC } Jun 28 18:48:05 system kernel: [75851.330867] ata2.00: configured for UDMA/133 Jun 28 18:48:05 system kernel: [75851.330884] sd 1:0:0:0: [sdb] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:05 system kernel: [75851.330889] sd 1:0:0:0: [sdb] tag#9 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:05 system kernel: [75851.330893] sd 1:0:0:0: [sdb] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:05 system kernel: [75851.330897] sd 1:0:0:0: [sdb] tag#9 CDB: Read(10) 28 00 0f 70 d8 08 00 00 01 00 Jun 28 18:48:05 system kernel: [75851.330900] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:48:05 system kernel: [75851.332856] ata2: EH complete Jun 28 18:48:08 system kernel: [75854.061061] ata2.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0 Jun 28 18:48:08 system kernel: [75854.064004] ata2.00: irq_stat 0x40000008 Jun 28 18:48:08 system kernel: [75854.066928] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:08 system kernel: [75854.069862] ata2.00: cmd 60/01:c0:08:d8:70/00:00:0f:00:00/40 tag 24 ncq 512 in Jun 28 18:48:08 system kernel: [75854.069862] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:08 system kernel: [75854.075700] ata2.00: status: { DRDY ERR } Jun 28 18:48:08 system kernel: [75854.078200] ata2.00: error: { UNC } Jun 28 18:48:08 system kernel: [75854.093765] ata2.00: configured for UDMA/133 Jun 28 18:48:08 system kernel: [75854.093784] sd 1:0:0:0: [sdb] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:08 system kernel: [75854.093790] sd 1:0:0:0: [sdb] tag#24 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:08 system kernel: [75854.093794] sd 1:0:0:0: [sdb] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:08 system kernel: [75854.093798] sd 1:0:0:0: [sdb] tag#24 CDB: Read(10) 28 00 0f 70 d8 08 00 00 01 00 Jun 28 18:48:08 system kernel: [75854.093802] blk_update_request: I/O error, dev sdb, sector 259053576 Jun 28 18:48:08 system kernel: [75854.095918] ata2: EH complete Jun 28 18:48:11 system kernel: [75856.823270] ata2.00: exception Emask 0x0 SAct 0x100 SErr 0x0 action 0x0 Jun 28 18:48:11 system kernel: [75856.828337] ata2.00: irq_stat 0x40000008 Jun 28 18:48:11 system kernel: [75856.833178] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:11 system kernel: [75856.837483] ata2.00: cmd 60/01:40:09:d8:70/00:00:0f:00:00/40 tag 8 ncq 512 in Jun 28 18:48:11 system kernel: [75856.837483] res 41/40:00:09:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:11 system kernel: [75856.845437] ata2.00: status: { DRDY ERR } Jun 28 18:48:11 system kernel: [75856.849104] ata2.00: error: { UNC } Jun 28 18:48:11 system kernel: [75856.865558] ata2.00: configured for UDMA/133 Jun 28 18:48:11 system kernel: [75856.865580] sd 1:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:11 system kernel: [75856.865587] sd 1:0:0:0: [sdb] tag#8 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:11 system kernel: [75856.865593] sd 1:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:11 system kernel: [75856.865599] sd 1:0:0:0: [sdb] tag#8 CDB: Read(10) 28 00 0f 70 d8 09 00 00 01 00 Jun 28 18:48:11 system kernel: [75856.865603] blk_update_request: I/O error, dev sdb, sector 259053577 Jun 28 18:48:11 system kernel: [75856.868467] ata2: EH complete Jun 28 18:48:14 system kernel: [75859.596611] ata2.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0 Jun 28 18:48:14 system kernel: [75859.599558] ata2.00: irq_stat 0x40000008 Jun 28 18:48:14 system kernel: [75859.602487] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:14 system kernel: [75859.605430] ata2.00: cmd 60/01:b8:0a:d8:70/00:00:0f:00:00/40 tag 23 ncq 512 in Jun 28 18:48:14 system kernel: [75859.605430] res 41/40:00:0a:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:14 system kernel: [75859.611286] ata2.00: status: { DRDY ERR } Jun 28 18:48:14 system kernel: [75859.613896] ata2.00: error: { UNC } Jun 28 18:48:14 system kernel: [75859.629738] ata2.00: configured for UDMA/133 Jun 28 18:48:14 system kernel: [75859.629760] sd 1:0:0:0: [sdb] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:14 system kernel: [75859.629765] sd 1:0:0:0: [sdb] tag#23 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:14 system kernel: [75859.629770] sd 1:0:0:0: [sdb] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:14 system kernel: [75859.629775] sd 1:0:0:0: [sdb] tag#23 CDB: Read(10) 28 00 0f 70 d8 0a 00 00 01 00 Jun 28 18:48:14 system kernel: [75859.629779] blk_update_request: I/O error, dev sdb, sector 259053578 Jun 28 18:48:14 system kernel: [75859.632107] ata2: EH complete Jun 28 18:48:16 system kernel: [75862.358828] ata2.00: exception Emask 0x0 SAct 0x80 SErr 0x0 action 0x0 Jun 28 18:48:16 system kernel: [75862.360301] ata2.00: irq_stat 0x40000008 Jun 28 18:48:16 system kernel: [75862.361754] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:16 system kernel: [75862.364673] ata2.00: cmd 60/01:38:0b:d8:70/00:00:0f:00:00/40 tag 7 ncq 512 in Jun 28 18:48:16 system kernel: [75862.364673] res 41/40:00:0b:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:16 system kernel: [75862.370474] ata2.00: status: { DRDY ERR } Jun 28 18:48:16 system kernel: [75862.372162] ata2.00: error: { UNC } Jun 28 18:48:16 system kernel: [75862.387053] ata2.00: configured for UDMA/133 Jun 28 18:48:16 system kernel: [75862.387063] sd 1:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:16 system kernel: [75862.387066] sd 1:0:0:0: [sdb] tag#7 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:16 system kernel: [75862.387068] sd 1:0:0:0: [sdb] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:16 system kernel: [75862.387070] sd 1:0:0:0: [sdb] tag#7 CDB: Read(10) 28 00 0f 70 d8 0b 00 00 01 00 Jun 28 18:48:16 system kernel: [75862.387071] blk_update_request: I/O error, dev sdb, sector 259053579 Jun 28 18:48:16 system kernel: [75862.388038] ata2: EH complete Jun 28 18:48:19 system kernel: [75865.109980] ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x0 action 0x0 Jun 28 18:48:19 system kernel: [75865.112951] ata2.00: irq_stat 0x40000008 Jun 28 18:48:19 system kernel: [75865.115836] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:19 system kernel: [75865.118735] ata2.00: cmd 60/01:b0:0c:d8:70/00:00:0f:00:00/40 tag 22 ncq 512 in Jun 28 18:48:19 system kernel: [75865.118735] res 41/40:00:0c:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:19 system kernel: [75865.124025] ata2.00: status: { DRDY ERR } Jun 28 18:48:19 system kernel: [75865.126375] ata2.00: error: { UNC } Jun 28 18:48:19 system kernel: [75865.141986] ata2.00: configured for UDMA/133 Jun 28 18:48:19 system kernel: [75865.142001] sd 1:0:0:0: [sdb] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:19 system kernel: [75865.142005] sd 1:0:0:0: [sdb] tag#22 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:19 system kernel: [75865.142009] sd 1:0:0:0: [sdb] tag#22 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:19 system kernel: [75865.142013] sd 1:0:0:0: [sdb] tag#22 CDB: Read(10) 28 00 0f 70 d8 0c 00 00 01 00 Jun 28 18:48:19 system kernel: [75865.142016] blk_update_request: I/O error, dev sdb, sector 259053580 Jun 28 18:48:19 system kernel: [75865.143675] ata2: EH complete Jun 28 18:48:22 system kernel: [75867.872211] ata2.00: exception Emask 0x0 SAct 0x40 SErr 0x0 action 0x0 Jun 28 18:48:22 system kernel: [75867.875153] ata2.00: irq_stat 0x40000008 Jun 28 18:48:22 system kernel: [75867.878041] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:22 system kernel: [75867.881003] ata2.00: cmd 60/01:30:0d:d8:70/00:00:0f:00:00/40 tag 6 ncq 512 in Jun 28 18:48:22 system kernel: [75867.881003] res 41/40:00:0d:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:22 system kernel: [75867.886814] ata2.00: status: { DRDY ERR } Jun 28 18:48:22 system kernel: [75867.889458] ata2.00: error: { UNC } Jun 28 18:48:22 system kernel: [75867.905133] ata2.00: configured for UDMA/133 Jun 28 18:48:22 system kernel: [75867.905151] sd 1:0:0:0: [sdb] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:22 system kernel: [75867.905157] sd 1:0:0:0: [sdb] tag#6 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:22 system kernel: [75867.905161] sd 1:0:0:0: [sdb] tag#6 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:22 system kernel: [75867.905166] sd 1:0:0:0: [sdb] tag#6 CDB: Read(10) 28 00 0f 70 d8 0d 00 00 01 00 Jun 28 18:48:22 system kernel: [75867.905169] blk_update_request: I/O error, dev sdb, sector 259053581 Jun 28 18:48:22 system kernel: [75867.907275] ata2: EH complete Jun 28 18:48:25 system kernel: [75870.634424] ata2.00: exception Emask 0x0 SAct 0x200000 SErr 0x0 action 0x0 Jun 28 18:48:25 system kernel: [75870.635895] ata2.00: irq_stat 0x40000008 Jun 28 18:48:25 system kernel: [75870.637347] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:25 system kernel: [75870.640275] ata2.00: cmd 60/01:a8:0e:d8:70/00:00:0f:00:00/40 tag 21 ncq 512 in Jun 28 18:48:25 system kernel: [75870.640275] res 41/40:00:0e:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:25 system kernel: [75870.646083] ata2.00: status: { DRDY ERR } Jun 28 18:48:25 system kernel: [75870.647747] ata2.00: error: { UNC } Jun 28 18:48:25 system kernel: [75870.662682] ata2.00: configured for UDMA/133 Jun 28 18:48:25 system kernel: [75870.662696] sd 1:0:0:0: [sdb] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:25 system kernel: [75870.662700] sd 1:0:0:0: [sdb] tag#21 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:25 system kernel: [75870.662704] sd 1:0:0:0: [sdb] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:25 system kernel: [75870.662707] sd 1:0:0:0: [sdb] tag#21 CDB: Read(10) 28 00 0f 70 d8 0e 00 00 01 00 Jun 28 18:48:25 system kernel: [75870.662709] blk_update_request: I/O error, dev sdb, sector 259053582 Jun 28 18:48:25 system kernel: [75870.664257] ata2: EH complete Jun 28 18:48:27 system kernel: [75873.407746] ata2.00: exception Emask 0x0 SAct 0x20 SErr 0x0 action 0x0 Jun 28 18:48:27 system kernel: [75873.409213] ata2.00: irq_stat 0x40000008 Jun 28 18:48:27 system kernel: [75873.410663] ata2.00: failed command: READ FPDMA QUEUED Jun 28 18:48:27 system kernel: [75873.413638] ata2.00: cmd 60/01:28:0f:d8:70/00:00:0f:00:00/40 tag 5 ncq 512 in Jun 28 18:48:27 system kernel: [75873.413638] res 41/40:00:0f:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> Jun 28 18:48:27 system kernel: [75873.419444] ata2.00: status: { DRDY ERR } Jun 28 18:48:27 system kernel: [75873.421118] ata2.00: error: { UNC } Jun 28 18:48:27 system kernel: [75873.436057] ata2.00: configured for UDMA/133 Jun 28 18:48:27 system kernel: [75873.436071] sd 1:0:0:0: [sdb] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Jun 28 18:48:27 system kernel: [75873.436075] sd 1:0:0:0: [sdb] tag#5 Sense Key : Medium Error [current] [descriptor] Jun 28 18:48:27 system kernel: [75873.436079] sd 1:0:0:0: [sdb] tag#5 Add. Sense: Unrecovered read error - auto reallocate failed Jun 28 18:48:27 system kernel: [75873.436082] sd 1:0:0:0: [sdb] tag#5 CDB: Read(10) 28 00 0f 70 d8 0f 00 00 01 00 Jun 28 18:48:27 system kernel: [75873.436085] blk_update_request: I/O error, dev sdb, sector 259053583 Jun 28 18:48:27 system kernel: [75873.437641] ata2: EH complete ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 9:50 ` Saint Germain @ 2016-06-29 17:28 ` Chris Murphy 2016-06-29 18:12 ` Saint Germain 2016-06-29 17:41 ` Saint Germain 1 sibling, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-29 17:28 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Wed, Jun 29, 2016 at 3:50 AM, Saint Germain <saintger@gmail.com> wrote: > Already got a backup. I just really want to try to repair it (in order > to test BTRFS). I don't know that this is a good test because I think the file system has already been sufficient corrupted that it can't be fixed. Part of the problem is that Btrfs isn't aware of faulty drives like mdadm or lvm yet, so it looks like it'll try to write to all devices and it's possible for significant confusion to happen if they're each getting different generation writes. Significant as in, currently beyond repair. > >> > On the other hand it seems interesting to repair instead of just >> > giving up. It gives a good look at BTRFS resiliency/reliability. >> >> On the one hand Btrfs shouldn't become inconsistent in the first >> place, that's the design goal. On the other hand, I'm finding from the >> problems reported on the list that Btrfs increasingly mounts at least >> read only and allows getting data off, even when the file system isn't >> fully functional or repairable. >> >> In your case, once there are metadata problems even with raid 1, it's >> difficult at best. But once you have the backup you could try some >> other things once it's certain the hardware isn't adding to the >> problems, which I'm still not yet certain of. >> > > I'm ready to try anything. Let's experiment. I kinda think it's a waste of time. Someone else maybe has a better idea? I think your time is better spent finding out when and why the device with all of these write errors happened. It must have gone missing for a while, and you need to find out why that happened and prevent it; OR you have to be really vigilent at every mount time to make sure both devices have the same transid (generation). In my case when I tried to sabotage this, being of by a generation of 1 wasn't a problem for Btrfs to automatically fix up but I suspect it was only a generation mismatch in the superblock. > I got some errors on sdb 2 months ago (I noticed it because it was > suddenly mounted read-only). I ran a scrub and a check --repair, and > a lot of errors were corrected. I deleted the files which were not > repairable and everything was running smoothly since. I ran a scrub a > few weeks ago and everything was fine. I'm not sure what to tell you. Maybe look at logs for the last three weeks for Btrfs and libata related messages to see if something else happened since that good scrub that you didn't notice. Also look at the mount messages and see if there are incrementing error counts in messages like this > BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 14, flush 7928, corrupt 1714507, gen 1335 Maybe you can ignore corrupt value because once there are checksum problems that number increments everytime the same corruption is found, it looks like. But if in the week after the scrub these numbers are changing there is a hardware problem somewhere and you need to find it. Btrfs can't be expected to work around every possible hardware problem. > So if I understand correctly, you advise to use check --repair > --init-csum-tree and delete the files which were reported as having > checksum error ? Yes but all this does is mask the problem with the files. It might clean up the error messages do you can find out if the file system itself is bad, including making btrfs check more legible for file system problems rather than file extent corruption. > Is there anyway I can be sure afterwards that the volume is indeed > completely correct and reliable ? Unknown. In theory if it passes scrub and btrfs check and mounts read write then it's a good file system. But there are bugs so it's possible it comes up healthy but then in a week or month, *shrug*, it's a test. > If there is no way to be sure, I think it is better that I cp/rsync all > data to a new BTRFS volume. I personally would probably just obliterate it and start over, *and* also be more vigilant at mount time about generation numbers, and also check the logs periodically for device problems. This one *especially* is not good: > Jun 28 11:35:23 system kernel: [49887.681350] ata2.00: exception Emask 0x0 SAct 0x600 SErr 0x0 action 0x6 frozen > Jun 28 11:35:23 system kernel: [49887.685258] ata2.00: failed command: READ FPDMA QUEUED > Jun 28 11:35:23 system kernel: [49887.689251] ata2.00: cmd 60/00:48:80:d7:70/01:00:0f:00:00/40 tag 9 ncq 131072 in > Jun 28 11:35:23 system kernel: [49887.689251] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) > Jun 28 11:35:23 system kernel: [49887.696884] ata2.00: status: { DRDY } > Jun 28 11:35:23 system kernel: [49887.700353] ata2.00: failed command: READ FPDMA QUEUED > Jun 28 11:35:23 system kernel: [49887.703749] ata2.00: cmd 60/00:50:80:d8:70/01:00:0f:00:00/40 tag 10 ncq 131072 in > Jun 28 11:35:23 system kernel: [49887.703749] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) > Jun 28 11:35:23 system kernel: [49887.710243] ata2.00: status: { DRDY } > Jun 28 11:35:23 system kernel: [49887.713250] ata2: hard resetting link This is a hung read command. The drive hasn't responded within the time set in /sys/block/sdX/device/timeout and like I mentioned in a previous message that's 30 seconds by default. When recovering from a bad or unknown state it's probably better to set this to 180 for each drive. In normal usage, it's better to set the SCT ERC for each drive to 70 deciseconds so errors happens faster and Btrfs can fix it sooner. Reset means we don't really know what went wrong. > Jun 28 11:35:24 system kernel: [49888.023941] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) > Jun 28 11:35:24 system kernel: [49888.037132] ata2.00: configured for UDMA/133 > Jun 28 11:35:24 system kernel: [49888.037143] ata2: EH complete > Jun 28 11:35:27 system kernel: [49890.782261] ata2.00: exception Emask 0x0 SAct 0x2000 SErr 0x0 action 0x0 > Jun 28 11:35:27 system kernel: [49890.783864] ata2.00: irq_stat 0x40000008 > Jun 28 11:35:27 system kernel: [49890.785437] ata2.00: failed command: READ FPDMA QUEUED > Jun 28 11:35:27 system kernel: [49890.787034] ata2.00: cmd 60/00:68:80:d7:70/01:00:0f:00:00/40 tag 13 ncq 131072 in > Jun 28 11:35:27 system kernel: [49890.787034] res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media error) <F> > Jun 28 11:35:27 system kernel: [49890.794700] ata2.00: status: { DRDY ERR } > Jun 28 11:35:27 system kernel: [49890.796272] ata2.00: error: { UNC } > Jun 28 11:35:27 system kernel: [49890.811742] ata2.00: configured for UDMA/133 > Jun 28 11:35:27 system kernel: [49890.811755] sd 1:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > Jun 28 11:35:27 system kernel: [49890.811758] sd 1:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] [descriptor] > Jun 28 11:35:27 system kernel: [49890.811760] sd 1:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed > Jun 28 11:35:27 system kernel: [49890.811763] sd 1:0:0:0: [sdb] tag#13 CDB: Read(10) 28 00 0f 70 d7 80 00 01 00 00 > Jun 28 11:35:27 system kernel: [49890.811765] blk_update_request: I/O error, dev sdb, sector 259053576 > Jun 28 11:35:27 system kernel: [49890.813241] ata2: EH complete This is semi-bad in that it means there's a sector that can't be read. Its data is lost. Btrfs has to use a mirror copy but if that mirror copy is corrupt / doesn't mass checksum, then Btrfs will not do a fixup and you're stuck. It's good in that there's an explicit read error due to media defect and the drive supplied the LBA that's affected. Some of these are OK as long as they're being fixed through redundancy. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 17:28 ` Chris Murphy @ 2016-06-29 18:12 ` Saint Germain 2016-06-29 18:19 ` Austin S. Hemmelgarn 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-29 18:12 UTC (permalink / raw) To: Btrfs BTRFS On Wed, 29 Jun 2016 11:28:24 -0600, Chris Murphy <lists@colorremedies.com> wrote : > > Already got a backup. I just really want to try to repair it (in > > order to test BTRFS). > > I don't know that this is a good test because I think the file system > has already been sufficient corrupted that it can't be fixed. Part of > the problem is that Btrfs isn't aware of faulty drives like mdadm or > lvm yet, so it looks like it'll try to write to all devices and it's > possible for significant confusion to happen if they're each getting > different generation writes. Significant as in, currently beyond > repair. > > >> > On the other hand it seems interesting to repair instead of just > >> > giving up. It gives a good look at BTRFS resiliency/reliability. > >> > >> On the one hand Btrfs shouldn't become inconsistent in the first > >> place, that's the design goal. On the other hand, I'm finding from > >> the problems reported on the list that Btrfs increasingly mounts > >> at least read only and allows getting data off, even when the file > >> system isn't fully functional or repairable. > >> > >> In your case, once there are metadata problems even with raid 1, > >> it's difficult at best. But once you have the backup you could try > >> some other things once it's certain the hardware isn't adding to > >> the problems, which I'm still not yet certain of. > >> > > > > I'm ready to try anything. Let's experiment. > > I kinda think it's a waste of time. Someone else maybe has a better > idea? > > I think your time is better spent finding out when and why the device > with all of these write errors happened. It must have gone missing for > a while, and you need to find out why that happened and prevent it; OR > you have to be really vigilent at every mount time to make sure both > devices have the same transid (generation). In my case when I tried to > sabotage this, being of by a generation of 1 wasn't a problem for > Btrfs to automatically fix up but I suspect it was only a generation > mismatch in the superblock. > Ok I will follow your advice and start over with a fresh BTRFS volume. As explained on another email, rsync doesn't support reflink, so do you think it is worth trying with BTRFS send instead ? Is it safe to copy this way or rsync is more reliable in case of faulty BTRFS volume ? Many thanks ! ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 18:12 ` Saint Germain @ 2016-06-29 18:19 ` Austin S. Hemmelgarn 2016-06-29 19:02 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Austin S. Hemmelgarn @ 2016-06-29 18:19 UTC (permalink / raw) To: Btrfs BTRFS On 2016-06-29 14:12, Saint Germain wrote: > On Wed, 29 Jun 2016 11:28:24 -0600, Chris Murphy > <lists@colorremedies.com> wrote : > >>> Already got a backup. I just really want to try to repair it (in >>> order to test BTRFS). >> >> I don't know that this is a good test because I think the file system >> has already been sufficient corrupted that it can't be fixed. Part of >> the problem is that Btrfs isn't aware of faulty drives like mdadm or >> lvm yet, so it looks like it'll try to write to all devices and it's >> possible for significant confusion to happen if they're each getting >> different generation writes. Significant as in, currently beyond >> repair. >> >>>>> On the other hand it seems interesting to repair instead of just >>>>> giving up. It gives a good look at BTRFS resiliency/reliability. >>>> >>>> On the one hand Btrfs shouldn't become inconsistent in the first >>>> place, that's the design goal. On the other hand, I'm finding from >>>> the problems reported on the list that Btrfs increasingly mounts >>>> at least read only and allows getting data off, even when the file >>>> system isn't fully functional or repairable. >>>> >>>> In your case, once there are metadata problems even with raid 1, >>>> it's difficult at best. But once you have the backup you could try >>>> some other things once it's certain the hardware isn't adding to >>>> the problems, which I'm still not yet certain of. >>>> >>> >>> I'm ready to try anything. Let's experiment. >> >> I kinda think it's a waste of time. Someone else maybe has a better >> idea? >> >> I think your time is better spent finding out when and why the device >> with all of these write errors happened. It must have gone missing for >> a while, and you need to find out why that happened and prevent it; OR >> you have to be really vigilent at every mount time to make sure both >> devices have the same transid (generation). In my case when I tried to >> sabotage this, being of by a generation of 1 wasn't a problem for >> Btrfs to automatically fix up but I suspect it was only a generation >> mismatch in the superblock. >> > > Ok I will follow your advice and start over with a fresh BTRFS volume. > As explained on another email, rsync doesn't support reflink, so do you > think it is worth trying with BTRFS send instead ? > Is it safe to copy this way or rsync is more reliable in case of faulty > BTRFS volume ? > If you have the space, btrfs restore would probably be the best option. It's not likely, but using send has a risk of contaminating the new filesystem as well. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 18:19 ` Austin S. Hemmelgarn @ 2016-06-29 19:02 ` Saint Germain 2016-06-29 19:08 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-29 19:02 UTC (permalink / raw) To: Btrfs BTRFS On Wed, 29 Jun 2016 14:19:23 -0400, "Austin S. Hemmelgarn" <ahferroin7@gmail.com> wrote : > >>> Already got a backup. I just really want to try to repair it (in > >>> order to test BTRFS). > >> > >> I don't know that this is a good test because I think the file > >> system has already been sufficient corrupted that it can't be > >> fixed. Part of the problem is that Btrfs isn't aware of faulty > >> drives like mdadm or lvm yet, so it looks like it'll try to write > >> to all devices and it's possible for significant confusion to > >> happen if they're each getting different generation writes. > >> Significant as in, currently beyond repair. > >> > >>>>> On the other hand it seems interesting to repair instead of just > >>>>> giving up. It gives a good look at BTRFS resiliency/reliability. > >>>> > >>>> On the one hand Btrfs shouldn't become inconsistent in the first > >>>> place, that's the design goal. On the other hand, I'm finding > >>>> from the problems reported on the list that Btrfs increasingly > >>>> mounts at least read only and allows getting data off, even when > >>>> the file system isn't fully functional or repairable. > >>>> > >>>> In your case, once there are metadata problems even with raid 1, > >>>> it's difficult at best. But once you have the backup you could > >>>> try some other things once it's certain the hardware isn't > >>>> adding to the problems, which I'm still not yet certain of. > >>>> > >>> > >>> I'm ready to try anything. Let's experiment. > >> > >> I kinda think it's a waste of time. Someone else maybe has a better > >> idea? > >> > >> I think your time is better spent finding out when and why the > >> device with all of these write errors happened. It must have gone > >> missing for a while, and you need to find out why that happened > >> and prevent it; OR you have to be really vigilent at every mount > >> time to make sure both devices have the same transid (generation). > >> In my case when I tried to sabotage this, being of by a generation > >> of 1 wasn't a problem for Btrfs to automatically fix up but I > >> suspect it was only a generation mismatch in the superblock. > >> > > > > Ok I will follow your advice and start over with a fresh BTRFS > > volume. As explained on another email, rsync doesn't support > > reflink, so do you think it is worth trying with BTRFS send > > instead ? Is it safe to copy this way or rsync is more reliable in > > case of faulty BTRFS volume ? > > > If you have the space, btrfs restore would probably be the best > option. It's not likely, but using send has a risk of contaminating > the new filesystem as well. > I have to copy through the network (I am running out of disks...) so btrfs restore is unfortunately not an option. I didn't know that btrfs send could contaminate the target disk as well ? Ok rsync it is then. Thanks ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 19:02 ` Saint Germain @ 2016-06-29 19:08 ` Chris Murphy 2016-06-29 19:16 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-29 19:08 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Wed, Jun 29, 2016 at 1:02 PM, Saint Germain <saintger@gmail.com> wrote: > > On Wed, 29 Jun 2016 14:19:23 -0400, "Austin S. Hemmelgarn" > <ahferroin7@gmail.com> wrote : > >> >>> Already got a backup. I just really want to try to repair it (in >> >>> order to test BTRFS). >> >> >> >> I don't know that this is a good test because I think the file >> >> system has already been sufficient corrupted that it can't be >> >> fixed. Part of the problem is that Btrfs isn't aware of faulty >> >> drives like mdadm or lvm yet, so it looks like it'll try to write >> >> to all devices and it's possible for significant confusion to >> >> happen if they're each getting different generation writes. >> >> Significant as in, currently beyond repair. >> >> >> >>>>> On the other hand it seems interesting to repair instead of just >> >>>>> giving up. It gives a good look at BTRFS resiliency/reliability. >> >>>> >> >>>> On the one hand Btrfs shouldn't become inconsistent in the first >> >>>> place, that's the design goal. On the other hand, I'm finding >> >>>> from the problems reported on the list that Btrfs increasingly >> >>>> mounts at least read only and allows getting data off, even when >> >>>> the file system isn't fully functional or repairable. >> >>>> >> >>>> In your case, once there are metadata problems even with raid 1, >> >>>> it's difficult at best. But once you have the backup you could >> >>>> try some other things once it's certain the hardware isn't >> >>>> adding to the problems, which I'm still not yet certain of. >> >>>> >> >>> >> >>> I'm ready to try anything. Let's experiment. >> >> >> >> I kinda think it's a waste of time. Someone else maybe has a better >> >> idea? >> >> >> >> I think your time is better spent finding out when and why the >> >> device with all of these write errors happened. It must have gone >> >> missing for a while, and you need to find out why that happened >> >> and prevent it; OR you have to be really vigilent at every mount >> >> time to make sure both devices have the same transid (generation). >> >> In my case when I tried to sabotage this, being of by a generation >> >> of 1 wasn't a problem for Btrfs to automatically fix up but I >> >> suspect it was only a generation mismatch in the superblock. >> >> >> > >> > Ok I will follow your advice and start over with a fresh BTRFS >> > volume. As explained on another email, rsync doesn't support >> > reflink, so do you think it is worth trying with BTRFS send >> > instead ? Is it safe to copy this way or rsync is more reliable in >> > case of faulty BTRFS volume ? >> > >> If you have the space, btrfs restore would probably be the best >> option. It's not likely, but using send has a risk of contaminating >> the new filesystem as well. >> > > I have to copy through the network (I am running out of disks...) so > btrfs restore is unfortunately not an option. > I didn't know that btrfs send could contaminate the target disk as > well ? > Ok rsync it is then. restore will let you extract files despite csum errors. I don't think send will, and using cp or rsync Btrfs definitely won't hand over the file. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 19:08 ` Chris Murphy @ 2016-06-29 19:16 ` Saint Germain 2016-06-29 19:23 ` Hugo Mills 2016-06-30 0:19 ` Chris Murphy 0 siblings, 2 replies; 27+ messages in thread From: Saint Germain @ 2016-06-29 19:16 UTC (permalink / raw) To: Btrfs BTRFS On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy <lists@colorremedies.com> wrote : > >> > Ok I will follow your advice and start over with a fresh BTRFS > >> > volume. As explained on another email, rsync doesn't support > >> > reflink, so do you think it is worth trying with BTRFS send > >> > instead ? Is it safe to copy this way or rsync is more reliable > >> > in case of faulty BTRFS volume ? > >> > > >> If you have the space, btrfs restore would probably be the best > >> option. It's not likely, but using send has a risk of contaminating > >> the new filesystem as well. > >> > > > > I have to copy through the network (I am running out of disks...) so > > btrfs restore is unfortunately not an option. > > I didn't know that btrfs send could contaminate the target disk as > > well ? > > Ok rsync it is then. > > restore will let you extract files despite csum errors. I don't think > send will, and using cp or rsync Btrfs definitely won't hand over the > file. > That's Ok I'd prefer to avoid copying files with csum errors anyway (I can restore them from backups). However will btrfs send abort the whole operation as soon as it finds a csum error ? And will I have the risk to "contaminate" the target BTRFS volume by using BTRFS send ? Thanks ! ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 19:16 ` Saint Germain @ 2016-06-29 19:23 ` Hugo Mills 2016-06-29 23:51 ` Saint Germain 2016-06-30 0:19 ` Chris Murphy 1 sibling, 1 reply; 27+ messages in thread From: Hugo Mills @ 2016-06-29 19:23 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 1918 bytes --] On Wed, Jun 29, 2016 at 09:16:13PM +0200, Saint Germain wrote: > On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy > <lists@colorremedies.com> wrote : > > > >> > Ok I will follow your advice and start over with a fresh BTRFS > > >> > volume. As explained on another email, rsync doesn't support > > >> > reflink, so do you think it is worth trying with BTRFS send > > >> > instead ? Is it safe to copy this way or rsync is more reliable > > >> > in case of faulty BTRFS volume ? > > >> > > > >> If you have the space, btrfs restore would probably be the best > > >> option. It's not likely, but using send has a risk of contaminating > > >> the new filesystem as well. > > >> > > > > > > I have to copy through the network (I am running out of disks...) so > > > btrfs restore is unfortunately not an option. > > > I didn't know that btrfs send could contaminate the target disk as > > > well ? > > > Ok rsync it is then. > > > > restore will let you extract files despite csum errors. I don't think > > send will, and using cp or rsync Btrfs definitely won't hand over the > > file. > > > > That's Ok I'd prefer to avoid copying files with csum errors anyway (I > can restore them from backups). > However will btrfs send abort the whole operation as soon as it finds a > csum error ? > And will I have the risk to "contaminate" the target BTRFS volume by > using BTRFS send ? A send stream is effectively just a sequence of filesystem commands (mv, cp, cp --reflink, rm, dd). So any damage that it can do when replayed by receive is limited to what you can do with the basic shell commands (plus cloning extents). If you have metadata breakage in your source filesystem, this won't cause the same metadata breakage to show up in the target filesystem. Hugo. -- Hugo Mills | Quidquid latine dictum sit, altum videtur hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 19:23 ` Hugo Mills @ 2016-06-29 23:51 ` Saint Germain 2016-06-30 0:24 ` Chris Murphy 0 siblings, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-29 23:51 UTC (permalink / raw) To: Btrfs BTRFS On Wed, 29 Jun 2016 19:23:57 +0000, Hugo Mills <hugo@carfax.org.uk> wrote : > On Wed, Jun 29, 2016 at 09:16:13PM +0200, Saint Germain wrote: > > On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy > > <lists@colorremedies.com> wrote : > > > > > >> > Ok I will follow your advice and start over with a fresh > > > >> > BTRFS volume. As explained on another email, rsync doesn't > > > >> > support reflink, so do you think it is worth trying with > > > >> > BTRFS send instead ? Is it safe to copy this way or rsync is > > > >> > more reliable in case of faulty BTRFS volume ? > > > >> > > > > >> If you have the space, btrfs restore would probably be the best > > > >> option. It's not likely, but using send has a risk of > > > >> contaminating the new filesystem as well. > > > >> > > > > > > > > I have to copy through the network (I am running out of > > > > disks...) so btrfs restore is unfortunately not an option. > > > > I didn't know that btrfs send could contaminate the target disk > > > > as well ? > > > > Ok rsync it is then. > > > > > > restore will let you extract files despite csum errors. I don't > > > think send will, and using cp or rsync Btrfs definitely won't > > > hand over the file. > > > > > > > That's Ok I'd prefer to avoid copying files with csum errors anyway > > (I can restore them from backups). > > However will btrfs send abort the whole operation as soon as it > > finds a csum error ? > > And will I have the risk to "contaminate" the target BTRFS volume by > > using BTRFS send ? > > A send stream is effectively just a sequence of filesystem commands > (mv, cp, cp --reflink, rm, dd). So any damage that it can do when > replayed by receive is limited to what you can do with the basic shell > commands (plus cloning extents). If you have metadata breakage in your > source filesystem, this won't cause the same metadata breakage to show > up in the target filesystem. > Well after 300GB copied through "btrfs send", the process is aborted with the following error: ERROR: send ioctl failed with -5: Input/output error ERROR: unexpected EOF in stream. /var/log/syslog relevant lines are appended at the end of this email. So it seems that I will have to go with rsync then. WARNING: CPU: 3 PID: 1779 at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/extent-tree.c:6608 __btrfs_free_extent.isra.67+0x152/0xdc0 [btrfs] BTRFS: Transaction aborted (error -5) Modules linked in: bnep(E) snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) fscache(E) sunrpc(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) nls_utf8(E) hmac(E) nls_cp437(E) drbg(E) ansi_cprng(E) vfat(E) fat(E) wl(POE) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) cfg80211(E) pcspkr(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) evdev(E) btusb(E) efi_pstore(E) joydev(E) btrtl(E) serio_raw(E) snd_pcm(E) snd_timer(E) snd(E) soundcore(E) shpchp(E) efivars(E) i2c_i801(E) hci_uart(E) btbcm(E) btqca(E) btintel(E) bluetooth(E) rfkill(E) i915(E) battery(E) crc16(E) video(E) intel_lpss_acpi(E) drm_kms_helper(E) intel_lpss(E) mfd_core(E) tpm_tis(E) acpi_pad(E) tpm(E) drm(E) acpi_als(E) kfifo_buf(E) i2c_algo_bit(E) mei_me(E) button(E) processor(E) industrialio(E) mei(E) fuse(E) autofs4(E) btrfs(E) xor(E) raid6_pq(E) sg(E) sd_mod(E) hid_logitech_hidpp(E) hid_logitech_dj(E) usbhid(E) ahci(E) libahci(E) crc32c_intel(E) e1000e(E) xhci_pci(E) xhci_hcd(E) ptp(E) psmouse(E) libata(E) pps_core(E) scsi_mod(E) usbcore(E) usb_common(E) i2c_hid(E) hid(E) fjes(E) CPU: 3 PID: 1779 Comm: btrfs-transacti Tainted: P OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170 Gaming-ITX/ac, BIOS P2.10 04/13/2016 0000000000000286 000000003e3b5862 ffffffff813123c5 ffff880067783b68 0000000000000000 ffffffff8107af94 000001b8f6065000 ffff880067783bc0 ffff88006b328000 ffff880165fc6000 0000000000000000 ffff880105a21150 Call Trace: [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 [<ffffffff8107af94>] ? __warn+0xc4/0xe0 [<ffffffff8107b00f>] ? warn_slowpath_fmt+0x5f/0x80 [<ffffffffc0246cf2>] ? __btrfs_free_extent.isra.67+0x152/0xdc0 [btrfs] [<ffffffffc02b131c>] ? btrfs_merge_delayed_refs+0x6c/0x610 [btrfs] [<ffffffffc024b7cd>] ? __btrfs_run_delayed_refs+0x9ad/0x1210 [btrfs] [<ffffffffc024edde>] ? btrfs_run_delayed_refs+0x8e/0x2b0 [btrfs] [<ffffffffc0264853>] ? btrfs_commit_transaction+0x4a3/0xa30 [btrfs] [<ffffffffc0264e76>] ? start_transaction+0x96/0x4d0 [btrfs] [<ffffffffc025f76e>] ? transaction_kthread+0x1ce/0x1f0 [btrfs] [<ffffffffc025f5a0>] ? btrfs_cleanup_transaction+0x590/0x590 [btrfs] [<ffffffff81099ecf>] ? kthread+0xdf/0x100 [<ffffffff815c85f2>] ? ret_from_fork+0x22/0x40 [<ffffffff81099df0>] ? kthread_park+0x50/0x50 ---[ end trace 7c9951bcf0da5e1b ]--- BTRFS: error (device sdb1) in __btrfs_free_extent:6608: errno=-5 IO failure BTRFS info (device sdb1): forced readonly BTRFS: error (device sdb1) in btrfs_run_delayed_refs:2946: errno=-5 IO failure BTRFS warning (device sdb1): Skipping commit of aborted transaction. BTRFS: error (device sdb1) in cleanup_transaction:1771: errno=-5 IO failure ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 23:51 ` Saint Germain @ 2016-06-30 0:24 ` Chris Murphy 2016-06-30 21:02 ` Saint Germain 0 siblings, 1 reply; 27+ messages in thread From: Chris Murphy @ 2016-06-30 0:24 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Wed, Jun 29, 2016 at 5:51 PM, Saint Germain <saintger@gmail.com> wrote: > On Wed, 29 Jun 2016 19:23:57 +0000, Hugo Mills <hugo@carfax.org.uk> > wrote : > >> On Wed, Jun 29, 2016 at 09:16:13PM +0200, Saint Germain wrote: >> > On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy >> > <lists@colorremedies.com> wrote : >> > >> > > >> > Ok I will follow your advice and start over with a fresh >> > > >> > BTRFS volume. As explained on another email, rsync doesn't >> > > >> > support reflink, so do you think it is worth trying with >> > > >> > BTRFS send instead ? Is it safe to copy this way or rsync is >> > > >> > more reliable in case of faulty BTRFS volume ? >> > > >> > >> > > >> If you have the space, btrfs restore would probably be the best >> > > >> option. It's not likely, but using send has a risk of >> > > >> contaminating the new filesystem as well. >> > > >> >> > > > >> > > > I have to copy through the network (I am running out of >> > > > disks...) so btrfs restore is unfortunately not an option. >> > > > I didn't know that btrfs send could contaminate the target disk >> > > > as well ? >> > > > Ok rsync it is then. >> > > >> > > restore will let you extract files despite csum errors. I don't >> > > think send will, and using cp or rsync Btrfs definitely won't >> > > hand over the file. >> > > >> > >> > That's Ok I'd prefer to avoid copying files with csum errors anyway >> > (I can restore them from backups). >> > However will btrfs send abort the whole operation as soon as it >> > finds a csum error ? >> > And will I have the risk to "contaminate" the target BTRFS volume by >> > using BTRFS send ? >> >> A send stream is effectively just a sequence of filesystem commands >> (mv, cp, cp --reflink, rm, dd). So any damage that it can do when >> replayed by receive is limited to what you can do with the basic shell >> commands (plus cloning extents). If you have metadata breakage in your >> source filesystem, this won't cause the same metadata breakage to show >> up in the target filesystem. >> > > Well after 300GB copied through "btrfs send", the process is aborted > with the following error: > ERROR: send ioctl failed with -5: Input/output error > ERROR: unexpected EOF in stream. > > /var/log/syslog relevant lines are appended at the end of this email. > > So it seems that I will have to go with rsync then. You'll likely hit the same bad file and get EIO, is my guess. What you can do is mount it ro from the get go, and do btrfs send receive again and maybe then it won't hit this sequence where it's finding some need to clean up a transaction and free an extent. Maybe you still get some failure to send whatever file is using that extent, but I think receive will tolerate it. > > WARNING: CPU: 3 PID: 1779 at /build/linux-9LouV5/linux-4.6.1/fs/btrfs/extent-tree.c:6608 __btrfs_free_extent.isra.67+0x152/0xdc0 [btrfs] > BTRFS: Transaction aborted (error -5) > Modules linked in: bnep(E) snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) fscache(E) sunrpc(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) nls_utf8(E) hmac(E) nls_cp437(E) drbg(E) ansi_cprng(E) vfat(E) fat(E) wl(POE) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) ablk_helper(E) cryptd(E) cfg80211(E) pcspkr(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) evdev(E) btusb(E) efi_pstore(E) joydev(E) btrtl(E) serio_raw(E) snd_pcm(E) snd_timer(E) snd(E) soundcore(E) shpchp(E) efivars(E) i2c_i801(E) hci_uart(E) btbcm(E) btqca(E) btintel(E) bluetooth(E) rfkill(E) i915(E) battery(E) crc16(E) video(E) intel_lpss_acpi(E) drm_kms_helper(E) intel_lpss(E) mfd_core(E) tpm_tis(E) acpi_pad(E) tpm(E) drm(E) acpi_als(E) kfifo_buf(E) i2c_algo_bit(E) mei_me(E) button(E) processor(E) industrialio(E) mei(E) fuse(E) autofs4(E) btrfs(E) xor(E) raid6_pq(E) sg(E) sd_mod(E) hid_logitech_hidpp(E) hid_logitech_dj(E) usbhid(E) ahci(E) libahci(E) crc32c_intel(E) e1000e(E) xhci_pci(E) xhci_hcd(E) ptp(E) psmouse(E) libata(E) pps_core(E) scsi_mod(E) usbcore(E) usb_common(E) i2c_hid(E) hid(E) fjes(E) > CPU: 3 PID: 1779 Comm: btrfs-transacti Tainted: P OE 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.1-1~bpo8+1 > Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170 Gaming-ITX/ac, BIOS P2.10 04/13/2016 > 0000000000000286 000000003e3b5862 ffffffff813123c5 ffff880067783b68 > 0000000000000000 ffffffff8107af94 000001b8f6065000 ffff880067783bc0 > ffff88006b328000 ffff880165fc6000 0000000000000000 ffff880105a21150 > Call Trace: > [<ffffffff813123c5>] ? dump_stack+0x5c/0x77 > [<ffffffff8107af94>] ? __warn+0xc4/0xe0 > [<ffffffff8107b00f>] ? warn_slowpath_fmt+0x5f/0x80 > [<ffffffffc0246cf2>] ? __btrfs_free_extent.isra.67+0x152/0xdc0 [btrfs] > [<ffffffffc02b131c>] ? btrfs_merge_delayed_refs+0x6c/0x610 [btrfs] > [<ffffffffc024b7cd>] ? __btrfs_run_delayed_refs+0x9ad/0x1210 [btrfs] > [<ffffffffc024edde>] ? btrfs_run_delayed_refs+0x8e/0x2b0 [btrfs] > [<ffffffffc0264853>] ? btrfs_commit_transaction+0x4a3/0xa30 [btrfs] > [<ffffffffc0264e76>] ? start_transaction+0x96/0x4d0 [btrfs] > [<ffffffffc025f76e>] ? transaction_kthread+0x1ce/0x1f0 [btrfs] > [<ffffffffc025f5a0>] ? btrfs_cleanup_transaction+0x590/0x590 [btrfs] > [<ffffffff81099ecf>] ? kthread+0xdf/0x100 > [<ffffffff815c85f2>] ? ret_from_fork+0x22/0x40 > [<ffffffff81099df0>] ? kthread_park+0x50/0x50 > ---[ end trace 7c9951bcf0da5e1b ]--- > BTRFS: error (device sdb1) in __btrfs_free_extent:6608: errno=-5 IO failure > BTRFS info (device sdb1): forced readonly > BTRFS: error (device sdb1) in btrfs_run_delayed_refs:2946: errno=-5 IO failure > BTRFS warning (device sdb1): Skipping commit of aborted transaction. > BTRFS: error (device sdb1) in cleanup_transaction:1771: errno=-5 IO > failure > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-30 0:24 ` Chris Murphy @ 2016-06-30 21:02 ` Saint Germain 0 siblings, 0 replies; 27+ messages in thread From: Saint Germain @ 2016-06-30 21:02 UTC (permalink / raw) To: Btrfs BTRFS On Wed, 29 Jun 2016 18:24:07 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Wed, Jun 29, 2016 at 5:51 PM, Saint Germain <saintger@gmail.com> > wrote: > > On Wed, 29 Jun 2016 19:23:57 +0000, Hugo Mills <hugo@carfax.org.uk> > > wrote : > > > >> On Wed, Jun 29, 2016 at 09:16:13PM +0200, Saint Germain wrote: > >> > On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy > >> > <lists@colorremedies.com> wrote : > >> > > >> > > >> > Ok I will follow your advice and start over with a fresh > >> > > >> > BTRFS volume. As explained on another email, rsync doesn't > >> > > >> > support reflink, so do you think it is worth trying with > >> > > >> > BTRFS send instead ? Is it safe to copy this way or rsync > >> > > >> > is more reliable in case of faulty BTRFS volume ? > >> > > >> > > >> > > >> If you have the space, btrfs restore would probably be the > >> > > >> best option. It's not likely, but using send has a risk of > >> > > >> contaminating the new filesystem as well. > >> > > >> > >> > > > > >> > > > I have to copy through the network (I am running out of > >> > > > disks...) so btrfs restore is unfortunately not an option. > >> > > > I didn't know that btrfs send could contaminate the target > >> > > > disk as well ? > >> > > > Ok rsync it is then. > >> > > > >> > > restore will let you extract files despite csum errors. I don't > >> > > think send will, and using cp or rsync Btrfs definitely won't > >> > > hand over the file. > >> > > > >> > > >> > That's Ok I'd prefer to avoid copying files with csum errors > >> > anyway (I can restore them from backups). > >> > However will btrfs send abort the whole operation as soon as it > >> > finds a csum error ? > >> > And will I have the risk to "contaminate" the target BTRFS > >> > volume by using BTRFS send ? > >> > >> A send stream is effectively just a sequence of filesystem > >> commands (mv, cp, cp --reflink, rm, dd). So any damage that it can > >> do when replayed by receive is limited to what you can do with the > >> basic shell commands (plus cloning extents). If you have metadata > >> breakage in your source filesystem, this won't cause the same > >> metadata breakage to show up in the target filesystem. > >> > > > > Well after 300GB copied through "btrfs send", the process is aborted > > with the following error: > > ERROR: send ioctl failed with -5: Input/output error > > ERROR: unexpected EOF in stream. > > > > /var/log/syslog relevant lines are appended at the end of this > > email. > > > > So it seems that I will have to go with rsync then. > > You'll likely hit the same bad file and get EIO, is my guess. What you > can do is mount it ro from the get go, and do btrfs send receive again > and maybe then it won't hit this sequence where it's finding some need > to clean up a transaction and free an extent. Maybe you still get some > failure to send whatever file is using that extent, but I think > receive will tolerate it. > Well I tried "btrfs send" and the process stalled at 300 GB (on a total of 2 TB) with a never ending stream of: "ERROR: unexpected EOF in stream." I gave up and launched a rsync which is about to be finished. Now I have some work to make sure that all rsynced files are consistent (I have to compared them to the backuped ones). Thanks for your help, I learned a bit more about BTRFS this way. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 19:16 ` Saint Germain 2016-06-29 19:23 ` Hugo Mills @ 2016-06-30 0:19 ` Chris Murphy 1 sibling, 0 replies; 27+ messages in thread From: Chris Murphy @ 2016-06-30 0:19 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Wed, Jun 29, 2016 at 1:16 PM, Saint Germain <saintger@gmail.com> wrote: > On Wed, 29 Jun 2016 13:08:30 -0600, Chris Murphy > <lists@colorremedies.com> wrote : > >> >> > Ok I will follow your advice and start over with a fresh BTRFS >> >> > volume. As explained on another email, rsync doesn't support >> >> > reflink, so do you think it is worth trying with BTRFS send >> >> > instead ? Is it safe to copy this way or rsync is more reliable >> >> > in case of faulty BTRFS volume ? >> >> > >> >> If you have the space, btrfs restore would probably be the best >> >> option. It's not likely, but using send has a risk of contaminating >> >> the new filesystem as well. >> >> >> > >> > I have to copy through the network (I am running out of disks...) so >> > btrfs restore is unfortunately not an option. >> > I didn't know that btrfs send could contaminate the target disk as >> > well ? >> > Ok rsync it is then. >> >> restore will let you extract files despite csum errors. I don't think >> send will, and using cp or rsync Btrfs definitely won't hand over the >> file. >> > > That's Ok I'd prefer to avoid copying files with csum errors anyway (I > can restore them from backups). > However will btrfs send abort the whole operation as soon as it finds a > csum error ? By default yes, but it's the receive side that aborts. So you use --max-errors 0 to let it tolerate unlimited errors, and use -v for both send and receive to track what errors there are. > And will I have the risk to "contaminate" the target BTRFS volume by > using BTRFS send ? No. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-29 9:50 ` Saint Germain 2016-06-29 17:28 ` Chris Murphy @ 2016-06-29 17:41 ` Saint Germain 1 sibling, 0 replies; 27+ messages in thread From: Saint Germain @ 2016-06-29 17:41 UTC (permalink / raw) Cc: Btrfs BTRFS On Wed, 29 Jun 2016 11:50:55 +0200, Saint Germain <saintger@gmail.com> wrote : > So if I understand correctly, you advise to use check --repair > --init-csum-tree and delete the files which were reported as having > checksum error ? > After that I can compare the important files to a backup, but there is > always the non-important files which are not backuped. > > Is there anyway I can be sure afterwards that the volume is indeed > completely correct and reliable ? > If there is no way to be sure, I think it is better that I cp/rsync > all data to a new BTRFS volume. > Oh and I forgot to add that rsync doesn't support reflink yet, so I am bit reluctant to rsync all data to a new volume instead of repairing the existing BTRFS volume. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 22:55 ` Chris Murphy 2016-06-27 22:58 ` Chris Murphy @ 2016-06-27 23:03 ` Saint Germain 2016-06-27 23:49 ` Chris Murphy 1 sibling, 1 reply; 27+ messages in thread From: Saint Germain @ 2016-06-27 23:03 UTC (permalink / raw) To: Btrfs BTRFS On Mon, 27 Jun 2016 16:55:07 -0600, Chris Murphy <lists@colorremedies.com> wrote : > On Mon, Jun 27, 2016 at 4:26 PM, Saint Germain <saintger@gmail.com> > wrote: > > >> > > > > Thanks for your help. > > > > Ok here is the log from the mounting, and including btrfs replace > > (btrfs replace start -f /dev/sda1 /dev/sdd1 /home): > > > > BTRFS info (device sdb1): disk space caching is enabled > > BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 11881695, rd 12, > > flush 7928, corrupt 1705631, gen 1335 BTRFS info (device sdb1): > > bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 14220, gen 24 > > Eek. So sdb has 11+ million write errors, flush errors, read errors, > and over 1 million corruptions. It's dying or dead. > > And sda has a dozen thousand+ corruptions. This isn't a good > combination, as you have two devices with problems and raid5 only > protects you from one device with problems. > > You were in the process of replacing sda, which is good, but it may > not be enough... > > > > BTRFS info (device sdb1): dev_replace from /dev/sda1 (devid 1) > > to /dev/sdd1 started scrub_handle_errored_block: 166 callbacks > > suppressed BTRFS warning (device sdb1): checksum error at logical > > 93445255168 on dev /dev/sda1, sector 77669048, root 5, inode > > 3434831, offset 479232, length 4096, links 1 (path: > > user/.local/share/zeitgeist/activity.sqlite-wal) > > btrfs_dev_stat_print_on_error: 166 callbacks suppressed BTRFS error > > (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt > > 14221, gen 24 scrub_handle_errored_block: 166 callbacks suppressed > > BTRFS error (device sdb1): unable to fixup (regular) error at > > logical 93445255168 on dev /dev/sda1 > > Shoot. You have a lot of these. It looks suspiciously like you're > hitting a case list regulars are only just starting to understand > (somewhat) where it's possible to have a legit corrupt sector that > Btrfs detects during scrub as wrong, fixes it from parity, but then > occasionally wrongly overwrites the parity with bad parity. This > doesn't cause an immediately recognizable problem. But if the volume > becomes degraded later, Btrfs must use parity to reconstruct > on-the-fly and if it hits one of these bad parities, the > reconstruction is bad, and ends up causing lots of these checksum > errors. We can tell it's not metadata corruption because a.) there's a > file listed as being affected and b.) the file system doesn't fail and > go read only. But still it means those files are likely toast... > > > [...snip many instances of checksum errors...] > > > BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush > > 0, corrupt 16217, gen 24 ata2.00: exception Emask 0x0 SAct 0x4000 > > SErr 0x0 action 0x0 ata2.00: irq_stat 0x40000008 > > ata2.00: failed command: READ FPDMA QUEUED > > ata2.00: cmd 60/08:70:08:d8:70/00:00:0f:00:00/40 tag 14 ncq 4096 in > > res 41/40:00:08:d8:70/00:00:0f:00:00/40 Emask 0x409 (media > > error) <F> ata2.00: status: { DRDY ERR } > > ata2.00: error: { UNC } > > ata2.00: configured for UDMA/133 > > sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK > > driverbyte=DRIVER_SENSE sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium > > Error [current] [descriptor] sd 1:0:0:0: [sdb] tag#14 Add. Sense: > > Unrecovered read error - auto reallocate failed sd 1:0:0:0: [sdb] > > tag#14 CDB: Read(10) 28 00 0f 70 d8 08 00 00 08 00 > > blk_update_request: I/O error, dev sdb, sector 259053576 > > OK yeah so bad sector on sdb. So you have two failures because sda is > already giving you trouble while being replaced and on top of it you > now get a 2nd (partial) failure via bad sectors. > > So rather urgently I think you need to copy things off this volume if > you don't already have a backup so you can save as much as possible. > Don't write to the drives. You might even consider 'mount -o > remount,ro' to avoid anything writing to the volume. Copy the most > important data first, triage time. > > While that happens you can safely collect some more information: > > btrfs fi us <mp> > smartctl -x <dev> ## for both drives > Ok thanks I will begin to make an image with dd. Do you recommend to use sda or sdb ? In the meantime here are the info requested: btrfs fi us /home Overall: Device size: 3.63TiB Device allocated: 2.76TiB Device unallocated: 888.51GiB Device missing: 0.00B Used: 2.62TiB Free (estimated): 517.56GiB (min: 517.56GiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,RAID1: Size:1.38TiB, Used:1.31TiB /dev/sda1 1.38TiB /dev/sdb1 1.38TiB Metadata,RAID1: Size:5.00GiB, Used:3.15GiB /dev/sda1 5.00GiB /dev/sdb1 5.00GiB System,RAID1: Size:64.00MiB, Used:216.00KiB /dev/sda1 64.00MiB /dev/sdb1 64.00MiB Unallocated: /dev/sda1 444.26GiB /dev/sdb1 444.26GiB root@system:/# smartctl -x /dev/sda smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST2000LM003 HN-M201RAD Serial Number: S377J9DGB01499 LU WWN Device Id: 5 0004cf 2111fa028 Firmware Version: 2BE10001 User Capacity: 2 000 398 934 016 bytes [2,00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Jun 28 00:59:36 2016 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM level is: 254 (maximum performance), recommended: 254 APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 38) The self-test routine was interrupted by the host with a hard or soft reset. Total time to complete Offline data collection: (23100) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 385) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 0 2 Throughput_Performance -OS--K 252 252 000 - 0 3 Spin_Up_Time PO---K 091 090 025 - 2993 4 Start_Stop_Count -O--CK 100 100 000 - 661 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 8 Seek_Time_Performance --S--K 252 252 015 - 0 9 Power_On_Hours -O--CK 100 100 000 - 1379 10 Spin_Retry_Count -O--CK 252 252 051 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 349 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 194 Temperature_Celsius -O---- 060 047 000 - 40 (Min/Max 18/53) 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 197 Current_Pending_Sector -O--CK 252 252 000 - 0 198 Offline_Uncorrectable ----CK 252 252 000 - 0 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 2 223 Load_Retry_Count -O--CK 100 100 000 - 1 225 Load_Cycle_Count -O--CK 099 099 000 - 10744 241 Total_LBAs_Written -O--CK 095 094 000 - 7981553 242 Total_LBAs_Read -O--CK 098 094 000 - 4015781 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 2 Comprehensive SMART error log 0x03 GPL R/O 2 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 2 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xc0-0xdf GPL,SL VS 16 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (2 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (2 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Offline Interrupted (host reset) 60% 1372 - # 2 Short captive Self-test routine in progress 60% 1372 - # 3 Offline Interrupted (host reset) 60% 1372 - # 4 Short captive Self-test routine in progress 60% 1372 - SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Interrupted [60% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 2 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: Active (0) Current Temperature: 40 Celsius Power Cycle Min/Max Temperature: 23/45 Celsius Lifetime Min/Max Temperature: 18/53 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 5 minutes Temperature Logging Interval: 5 minutes Min/Max recommended Temperature: -5/80 Celsius Min/Max Temperature Limit: -10/85 Celsius Temperature History Size (Index): 128 (60) Index Estimated Time Temperature Celsius 61 2016-06-27 14:20 40 ********************* 62 2016-06-27 14:25 37 ****************** 63 2016-06-27 14:30 37 ****************** 64 2016-06-27 14:35 37 ****************** 65 2016-06-27 14:40 38 ******************* 66 2016-06-27 14:45 38 ******************* 67 2016-06-27 14:50 39 ******************** 68 2016-06-27 14:55 40 ********************* 69 2016-06-27 15:00 40 ********************* 70 2016-06-27 15:05 41 ********************** 71 2016-06-27 15:10 41 ********************** 72 2016-06-27 15:15 26 ******* 73 2016-06-27 15:20 28 ********* 74 2016-06-27 15:25 30 *********** 75 2016-06-27 15:30 32 ************* 76 2016-06-27 15:35 34 *************** 77 2016-06-27 15:40 34 *************** 78 2016-06-27 15:45 35 **************** 79 2016-06-27 15:50 36 ***************** 80 2016-06-27 15:55 37 ****************** 81 2016-06-27 16:00 38 ******************* 82 2016-06-27 16:05 39 ******************** 83 2016-06-27 16:10 26 ******* 84 2016-06-27 16:15 29 ********** 85 2016-06-27 16:20 30 *********** 86 2016-06-27 16:25 32 ************* 87 2016-06-27 16:30 35 **************** 88 2016-06-27 16:35 37 ****************** 89 2016-06-27 16:40 39 ******************** 90 2016-06-27 16:45 41 ********************** 91 2016-06-27 16:50 43 ************************ 92 2016-06-27 16:55 45 ************************** 93 2016-06-27 17:00 46 *************************** 94 2016-06-27 17:05 47 **************************** 95 2016-06-27 17:10 47 **************************** 96 2016-06-27 17:15 48 ***************************** 97 2016-06-27 17:20 48 ***************************** 98 2016-06-27 17:25 48 ***************************** 99 2016-06-27 17:30 49 ****************************** ... ..( 3 skipped). .. ****************************** 103 2016-06-27 17:50 49 ****************************** 104 2016-06-27 17:55 44 ************************* 105 2016-06-27 18:00 43 ************************ 106 2016-06-27 18:05 42 *********************** 107 2016-06-27 18:10 44 ************************* 108 2016-06-27 18:15 45 ************************** 109 2016-06-27 18:20 45 ************************** 110 2016-06-27 18:25 44 ************************* 111 2016-06-27 18:30 41 ********************** 112 2016-06-27 18:35 40 ********************* 113 2016-06-27 18:40 39 ******************** 114 2016-06-27 18:45 39 ******************** 115 2016-06-27 18:50 38 ******************* ... ..( 2 skipped). .. ******************* 118 2016-06-27 19:05 38 ******************* 119 2016-06-27 19:10 41 ********************** 120 2016-06-27 19:15 43 ************************ 121 2016-06-27 19:20 43 ************************ 122 2016-06-27 19:25 40 ********************* 123 2016-06-27 19:30 39 ******************** 124 2016-06-27 19:35 39 ******************** 125 2016-06-27 19:40 39 ******************** 126 2016-06-27 19:45 40 ********************* ... ..( 5 skipped). .. ********************* 4 2016-06-27 20:15 40 ********************* 5 2016-06-27 20:20 39 ******************** ... ..( 3 skipped). .. ******************** 9 2016-06-27 20:40 39 ******************** 10 2016-06-27 20:45 38 ******************* 11 2016-06-27 20:50 38 ******************* 12 2016-06-27 20:55 41 ********************** 13 2016-06-27 21:00 44 ************************* 14 2016-06-27 21:05 44 ************************* 15 2016-06-27 21:10 41 ********************** 16 2016-06-27 21:15 40 ********************* 17 2016-06-27 21:20 40 ********************* 18 2016-06-27 21:25 43 ************************ 19 2016-06-27 21:30 43 ************************ 20 2016-06-27 21:35 27 ******** 21 2016-06-27 21:40 29 ********** 22 2016-06-27 21:45 30 *********** 23 2016-06-27 21:50 34 *************** 24 2016-06-27 21:55 35 **************** 25 2016-06-27 22:00 36 ***************** 26 2016-06-27 22:05 36 ***************** 27 2016-06-27 22:10 37 ****************** ... ..( 3 skipped). .. ****************** 31 2016-06-27 22:30 37 ****************** 32 2016-06-27 22:35 38 ******************* 33 2016-06-27 22:40 38 ******************* 34 2016-06-27 22:45 41 ********************** 35 2016-06-27 22:50 43 ************************ 36 2016-06-27 22:55 41 ********************** 37 2016-06-27 23:00 39 ******************** 38 2016-06-27 23:05 39 ******************** 39 2016-06-27 23:10 39 ******************** 40 2016-06-27 23:15 40 ********************* 41 2016-06-27 23:20 43 ************************ 42 2016-06-27 23:25 44 ************************* 43 2016-06-27 23:30 45 ************************** 44 2016-06-27 23:35 42 *********************** 45 2016-06-27 23:40 40 ********************* 46 2016-06-27 23:45 39 ******************** 47 2016-06-27 23:50 39 ******************** 48 2016-06-27 23:55 38 ******************* ... ..( 2 skipped). .. ******************* 51 2016-06-28 00:10 38 ******************* 52 2016-06-28 00:15 41 ********************** 53 2016-06-28 00:20 43 ************************ 54 2016-06-28 00:25 44 ************************* 55 2016-06-28 00:30 41 ********************** 56 2016-06-28 00:35 40 ********************* ... ..( 3 skipped). .. ********************* 60 2016-06-28 00:55 40 ********************* SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP/SMART Log 0x04) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 4 0 Command failed due to ICRC error 0x0002 4 0 R_ERR response for data FIS 0x0003 4 0 R_ERR response for device-to-host data FIS 0x0004 4 0 R_ERR response for host-to-device data FIS 0x0005 4 0 R_ERR response for non-data FIS 0x0006 4 0 R_ERR response for device-to-host non-data FIS 0x0007 4 0 R_ERR response for host-to-device non-data FIS 0x0008 4 0 Device-to-host non-data FIS retries 0x0009 4 5 Transition from drive PhyRdy to drive PhyNRdy 0x000a 4 5 Device-to-host register FISes sent due to a COMRESET 0x000b 4 0 CRC errors within host-to-device FIS 0x000d 4 0 Non-CRC errors within host-to-device FIS 0x000f 4 0 R_ERR response for host-to-device data FIS, CRC 0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC 0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC 0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC 0x8e00 4 0 Vendor specific 0x8e01 4 0 Vendor specific 0x8e02 4 0 Vendor specific 0x8e03 4 0 Vendor specific 0x8e04 4 0 Vendor specific 0x8e05 4 0 Vendor specific 0x8e06 4 0 Vendor specific 0x8e07 4 0 Vendor specific 0x8e08 4 0 Vendor specific 0x8e09 4 0 Vendor specific 0x8e0a 4 0 Vendor specific 0x8e0b 4 0 Vendor specific 0x8e0c 4 0 Vendor specific 0x8e0d 4 0 Vendor specific 0x8e0e 4 0 Vendor specific 0x8e0f 4 0 Vendor specific 0x8e10 4 0 Vendor specific 0x8e11 4 0 Vendor specific root@system:/# smartctl -x /dev/sdb smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.6.0-0.bpo.1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST2000LM003 HN-M201RAD Serial Number: S377J9CGB01582 LU WWN Device Id: 5 0004cf 211210d5b Firmware Version: 2BE10001 User Capacity: 2 000 398 934 016 bytes [2,00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Jun 28 00:59:37 2016 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM level is: 254 (maximum performance), recommended: 254 APM level is: 254 (maximum performance) Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (21420) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 357) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 28 2 Throughput_Performance -OS--K 252 252 000 - 0 3 Spin_Up_Time PO---K 092 083 025 - 2678 4 Start_Stop_Count -O--CK 100 100 000 - 575 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 8 Seek_Time_Performance --S--K 252 252 015 - 0 9 Power_On_Hours -O--CK 100 100 000 - 1391 10 Spin_Retry_Count -O--CK 252 252 051 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 371 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 194 Temperature_Celsius -O---- 061 047 000 - 39 (Min/Max 19/53) 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 197 Current_Pending_Sector -O--CK 100 100 000 - 1 198 Offline_Uncorrectable ----CK 252 252 000 - 0 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 3 223 Load_Retry_Count -O--CK 100 100 000 - 1 225 Load_Cycle_Count -O--CK 099 099 000 - 13957 241 Total_LBAs_Written -O--CK 096 094 000 - 6153920 242 Total_LBAs_Read -O--CK 097 094 000 - 4873960 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 2 Comprehensive SMART error log 0x03 GPL R/O 2 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 2 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xc0-0xdf GPL,SL VS 16 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (2 sectors) Device Error Count: 28 (device log contains only the most recent 8 errors) CR = Command Register FEATR = Features Register COUNT = Count (was: Sector Count) Register LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8 LH = LBA High (was: Cylinder High) Register ] LBA LM = LBA Mid (was: Cylinder Low) Register ] Register LL = LBA Low (was: Sector Number) Register ] DV = Device (was: Device/Head) Register DC = Device Control Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 28 [3] occurred at disk power-on lifetime: 1390 hours (57 days + 22 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:21.275 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d8 08 40 08 00:00:21.279 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d8 00 40 08 00:00:21.279 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d7 f8 40 08 00:00:21.279 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d7 f0 40 08 00:00:21.279 READ FPDMA QUEUED Error 27 [2] occurred at disk power-on lifetime: 1390 hours (57 days + 22 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 05 80 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 06 60 00 00 0f 70 d7 28 40 08 00:00:21.275 READ FPDMA QUEUED 60 00 00 06 60 00 00 0f 70 d7 28 40 08 00:00:21.276 READ FPDMA QUEUED 60 00 00 07 00 00 00 0f 70 d0 28 40 08 00:00:21.276 READ FPDMA QUEUED 60 00 00 0a 00 00 00 0f 70 c6 28 40 08 00:00:21.276 READ FPDMA QUEUED 60 00 00 0a 00 00 00 0f 70 bc 28 40 08 00:00:21.276 READ FPDMA QUEUED Error 26 [1] occurred at disk power-on lifetime: 1390 hours (57 days + 22 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:19.046 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:19.051 READ FPDMA QUEUED ea 00 00 00 00 00 00 00 00 00 00 e0 08 00:00:19.051 FLUSH CACHE EXT 61 00 00 00 08 00 00 20 00 08 00 40 08 00:00:19.051 WRITE FPDMA QUEUED 61 00 00 00 08 00 00 00 02 08 00 40 08 00:00:19.051 WRITE FPDMA QUEUED Error 25 [0] occurred at disk power-on lifetime: 1388 hours (57 days + 20 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:11.810 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:11.816 READ FPDMA QUEUED 60 00 00 00 08 00 00 07 9c d9 20 40 08 00:00:11.811 READ FPDMA QUEUED 60 00 00 00 08 00 00 3e 60 1a 90 40 08 00:00:11.811 READ FPDMA QUEUED 60 00 00 00 08 00 00 07 9c 68 f0 40 08 00:00:11.811 READ FPDMA QUEUED Error 24 [7] occurred at disk power-on lifetime: 1387 hours (57 days + 19 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:09.010 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:09.066 READ FPDMA QUEUED ea 00 00 00 00 00 00 00 00 00 00 e0 08 00:00:09.051 FLUSH CACHE EXT 61 00 00 00 08 00 00 20 00 08 00 40 08 00:00:09.051 WRITE FPDMA QUEUED 61 00 00 00 08 00 00 00 02 08 00 40 08 00:00:09.051 WRITE FPDMA QUEUED Error 23 [6] occurred at disk power-on lifetime: 1385 hours (57 days + 17 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:01.685 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d8 08 40 08 00:00:01.688 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d8 00 40 08 00:00:01.688 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d7 f8 40 08 00:00:01.688 READ FPDMA QUEUED 60 00 10 00 08 00 00 0f 70 d7 f0 40 08 00:00:01.688 READ FPDMA QUEUED Error 22 [5] occurred at disk power-on lifetime: 1385 hours (57 days + 17 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 05 80 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 06 60 00 00 0f 70 d7 28 40 08 00:00:01.685 READ FPDMA QUEUED 60 00 00 06 60 00 00 0f 70 d7 28 40 08 00:00:01.685 READ FPDMA QUEUED 60 00 00 08 00 00 00 0f 70 cf 28 40 08 00:00:01.685 READ FPDMA QUEUED 60 00 00 05 00 00 00 0f 70 ca 28 40 08 00:00:01.685 READ FPDMA QUEUED 60 00 00 07 00 00 00 0f 70 c3 28 40 08 00:00:01.685 READ FPDMA QUEUED Error 21 [4] occurred at disk power-on lifetime: 1384 hours (57 days + 16 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 60 00 00 00 08 00 00 0f 70 d8 08 40 08 00:00:23.754 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 db 30 40 08 00:00:23.774 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 d8 30 40 08 00:00:23.774 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 d9 30 40 08 00:00:23.774 READ FPDMA QUEUED 60 00 00 00 08 00 00 0f 70 da 30 40 08 00:00:23.774 READ FPDMA QUEUED SMART Extended Self-test Log Version: 1 (2 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short captive Completed: read failure 90% 1384 259053576 # 2 Short captive Completed: read failure 90% 1384 259053576 SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Completed_read_failure [90% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 2 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: Active (0) Current Temperature: 38 Celsius Power Cycle Min/Max Temperature: 23/42 Celsius Lifetime Min/Max Temperature: 19/53 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 5 minutes Temperature Logging Interval: 5 minutes Min/Max recommended Temperature: -5/80 Celsius Min/Max Temperature Limit: -10/85 Celsius Temperature History Size (Index): 128 (17) Index Estimated Time Temperature Celsius 18 2016-06-27 14:20 38 ******************* 19 2016-06-27 14:25 36 ***************** ... ..( 17 skipped). .. ***************** 37 2016-06-27 15:55 36 ***************** 38 2016-06-27 16:00 35 **************** 39 2016-06-27 16:05 36 ***************** 40 2016-06-27 16:10 35 **************** 41 2016-06-27 16:15 35 **************** 42 2016-06-27 16:20 32 ************* 43 2016-06-27 16:25 33 ************** 44 2016-06-27 16:30 34 *************** 45 2016-06-27 16:35 34 *************** 46 2016-06-27 16:40 35 **************** 47 2016-06-27 16:45 27 ******** 48 2016-06-27 16:50 29 ********** 49 2016-06-27 16:55 30 *********** 50 2016-06-27 17:00 34 *************** 51 2016-06-27 17:05 37 ****************** 52 2016-06-27 17:10 39 ******************** 53 2016-06-27 17:15 40 ********************* 54 2016-06-27 17:20 41 ********************** 55 2016-06-27 17:25 42 *********************** 56 2016-06-27 17:30 42 *********************** 57 2016-06-27 17:35 43 ************************ 58 2016-06-27 17:40 43 ************************ 59 2016-06-27 17:45 44 ************************* 60 2016-06-27 17:50 44 ************************* 61 2016-06-27 17:55 44 ************************* 62 2016-06-27 18:00 41 ********************** 63 2016-06-27 18:05 40 ********************* 64 2016-06-27 18:10 39 ******************** 65 2016-06-27 18:15 41 ********************** 66 2016-06-27 18:20 42 *********************** 67 2016-06-27 18:25 43 ************************ 68 2016-06-27 18:30 43 ************************ 69 2016-06-27 18:35 38 ******************* 70 2016-06-27 18:40 37 ****************** 71 2016-06-27 18:45 36 ***************** ... ..( 2 skipped). .. ***************** 74 2016-06-27 19:00 36 ***************** 75 2016-06-27 19:05 35 **************** 76 2016-06-27 19:10 36 ***************** 77 2016-06-27 19:15 40 ********************* 78 2016-06-27 19:20 41 ********************** 79 2016-06-27 19:25 41 ********************** 80 2016-06-27 19:30 38 ******************* 81 2016-06-27 19:35 37 ****************** 82 2016-06-27 19:40 37 ****************** 83 2016-06-27 19:45 37 ****************** 84 2016-06-27 19:50 38 ******************* ... ..( 4 skipped). .. ******************* 89 2016-06-27 20:15 38 ******************* 90 2016-06-27 20:20 37 ****************** 91 2016-06-27 20:25 38 ******************* 92 2016-06-27 20:30 37 ****************** ... ..( 2 skipped). .. ****************** 95 2016-06-27 20:45 37 ****************** 96 2016-06-27 20:50 36 ***************** 97 2016-06-27 20:55 39 ******************** 98 2016-06-27 21:00 41 ********************** 99 2016-06-27 21:05 42 *********************** 100 2016-06-27 21:10 42 *********************** 101 2016-06-27 21:15 39 ******************** 102 2016-06-27 21:20 38 ******************* 103 2016-06-27 21:25 39 ******************** 104 2016-06-27 21:30 41 ********************** 105 2016-06-27 21:35 39 ******************** 106 2016-06-27 21:40 27 ******** 107 2016-06-27 21:45 36 ***************** 108 2016-06-27 21:50 35 **************** 109 2016-06-27 21:55 34 *************** 110 2016-06-27 22:00 34 *************** 111 2016-06-27 22:05 35 **************** ... ..( 2 skipped). .. **************** 114 2016-06-27 22:20 35 **************** 115 2016-06-27 22:25 36 ***************** ... ..( 4 skipped). .. ***************** 120 2016-06-27 22:50 36 ***************** 121 2016-06-27 22:55 37 ****************** 122 2016-06-27 23:00 39 ******************** 123 2016-06-27 23:05 37 ****************** 124 2016-06-27 23:10 37 ****************** 125 2016-06-27 23:15 38 ******************* ... ..( 2 skipped). .. ******************* 0 2016-06-27 23:30 38 ******************* 1 2016-06-27 23:35 39 ******************** 2 2016-06-27 23:40 39 ******************** 3 2016-06-27 23:45 38 ******************* 4 2016-06-27 23:50 37 ****************** 5 2016-06-27 23:55 37 ****************** 6 2016-06-28 00:00 37 ****************** 7 2016-06-28 00:05 36 ***************** 8 2016-06-28 00:10 36 ***************** 9 2016-06-28 00:15 36 ***************** 10 2016-06-28 00:20 37 ****************** 11 2016-06-28 00:25 37 ****************** 12 2016-06-28 00:30 38 ******************* 13 2016-06-28 00:35 38 ******************* 14 2016-06-28 00:40 37 ****************** 15 2016-06-28 00:45 42 *********************** 16 2016-06-28 00:50 42 *********************** 17 2016-06-28 00:55 39 ******************** SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP/SMART Log 0x04) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 4 0 Command failed due to ICRC error 0x0002 4 0 R_ERR response for data FIS 0x0003 4 0 R_ERR response for device-to-host data FIS 0x0004 4 0 R_ERR response for host-to-device data FIS 0x0005 4 0 R_ERR response for non-data FIS 0x0006 4 0 R_ERR response for device-to-host non-data FIS 0x0007 4 0 R_ERR response for host-to-device non-data FIS 0x0008 4 0 Device-to-host non-data FIS retries 0x0009 4 6 Transition from drive PhyRdy to drive PhyNRdy 0x000a 4 5 Device-to-host register FISes sent due to a COMRESET 0x000b 4 0 CRC errors within host-to-device FIS 0x000d 4 0 Non-CRC errors within host-to-device FIS 0x000f 4 0 R_ERR response for host-to-device data FIS, CRC 0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC 0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC 0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC 0x8e00 4 0 Vendor specific 0x8e01 4 0 Vendor specific 0x8e02 4 0 Vendor specific 0x8e03 4 0 Vendor specific 0x8e04 4 0 Vendor specific 0x8e05 4 0 Vendor specific 0x8e06 4 82711 Vendor specific 0x8e07 4 36347773 Vendor specific 0x8e08 4 144 Vendor specific 0x8e09 4 43703 Vendor specific 0x8e0a 4 2848493 Vendor specific 0x8e0b 4 193920 Vendor specific 0x8e0c 4 10284463 Vendor specific 0x8e0d 4 74795 Vendor specific 0x8e0e 4 935421 Vendor specific 0x8e0f 4 131112960 Vendor specific 0x8e10 4 3741684 Vendor specific 0x8e11 4 7 Vendor specific ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Kernel bug during RAID1 replace 2016-06-27 23:03 ` Saint Germain @ 2016-06-27 23:49 ` Chris Murphy 0 siblings, 0 replies; 27+ messages in thread From: Chris Murphy @ 2016-06-27 23:49 UTC (permalink / raw) To: Saint Germain; +Cc: Btrfs BTRFS On Mon, Jun 27, 2016 at 5:03 PM, Saint Germain <saintger@gmail.com> wrote: >> > > Ok thanks I will begin to make an image with dd. > Do you recommend to use sda or sdb ? Well at the moment you're kinda stuck. I'd leave them together and just get the data off the drive normally with cp -a (or just -r if you don't care about permissions and other metadata like time stamps and xattr) or rsync -a. Certainly the dying drive is being really pissy but if you get a bad read off one drive *maybe* it can correct off the other drive. But that's not possible if you pull one of those drives. Also as for imaging the drive, you probably need to use ddrescue instead of dd. Be warned that there's a gotcha where you can corrupt Btrfs volumes where multiple instances of the same fs uuid and dev uuid appear at the same time to the kernel. So once you've cloned in this manner, don't mount the volume until you hide (as in remove) one of the copies. See block level copies: https://btrfs.wiki.kernel.org/index.php/Gotchas > root@system:/# smartctl -x /dev/sda > ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE > 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 0 > 2 Throughput_Performance -OS--K 252 252 000 - 0 > 3 Spin_Up_Time PO---K 091 090 025 - 2993 > 4 Start_Stop_Count -O--CK 100 100 000 - 661 > 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 > 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 > 8 Seek_Time_Performance --S--K 252 252 015 - 0 > 9 Power_On_Hours -O--CK 100 100 000 - 1379 > 10 Spin_Retry_Count -O--CK 252 252 051 - 0 > 12 Power_Cycle_Count -O--CK 100 100 000 - 349 > 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 > 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 > 194 Temperature_Celsius -O---- 060 047 000 - 40 (Min/Max 18/53) > 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 > 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 > 197 Current_Pending_Sector -O--CK 252 252 000 - 0 > 198 Offline_Uncorrectable ----CK 252 252 000 - 0 > 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 > 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 2 > 223 Load_Retry_Count -O--CK 100 100 000 - 1 > 225 Load_Cycle_Count -O--CK 099 099 000 - 10744 > 241 Total_LBAs_Written -O--CK 095 094 000 - 7981553 > 242 Total_LBAs_Read -O--CK 098 094 000 - 4015781 No current pending, reallocated, or uncorrected sectors. Interesting. But this drive has piles of write errors. Why? Bad cable? That should result in UDMA CRC errors, lots of them. > SATA Phy Event Counters (GP Log 0x11) No significant problems. > root@system:/# smartctl -x /dev/sdb > > SMART Attributes Data Structure revision number: 16 > Vendor Specific SMART Attributes with Thresholds: > ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE > 1 Raw_Read_Error_Rate POSR-K 100 100 051 - 28 > 2 Throughput_Performance -OS--K 252 252 000 - 0 > 3 Spin_Up_Time PO---K 092 083 025 - 2678 > 4 Start_Stop_Count -O--CK 100 100 000 - 575 > 5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0 > 7 Seek_Error_Rate -OSR-K 252 252 051 - 0 > 8 Seek_Time_Performance --S--K 252 252 015 - 0 > 9 Power_On_Hours -O--CK 100 100 000 - 1391 > 10 Spin_Retry_Count -O--CK 252 252 051 - 0 > 12 Power_Cycle_Count -O--CK 100 100 000 - 371 > 191 G-Sense_Error_Rate -O---K 252 252 000 - 0 > 192 Power-Off_Retract_Count -O---K 252 252 000 - 0 > 194 Temperature_Celsius -O---- 061 047 000 - 39 (Min/Max 19/53) > 195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0 > 196 Reallocated_Event_Count -O--CK 252 252 000 - 0 > 197 Current_Pending_Sector -O--CK 100 100 000 - 1 > 198 Offline_Uncorrectable ----CK 252 252 000 - 0 > 199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0 > 200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 3 > 223 Load_Retry_Count -O--CK 100 100 000 - 1 > 225 Load_Cycle_Count -O--CK 099 099 000 - 13957 > 241 Total_LBAs_Written -O--CK 096 094 000 - 6153920 > 242 Total_LBAs_Read -O--CK 097 094 000 - 4873960 One pending sector. Enough for a dozen scary warnings or so, but not enough to account for as many as you have. Pretty curious. > > Error 28 [3] occurred at disk power-on lifetime: 1390 hours (57 days + 22 hours) > When the command that caused the error occurred, the device was active or idle. > > After command completion occurred, registers were: > ER -- ST COUNT LBA_48 LH LM LL DV DC > -- -- -- == -- == == == -- -- -- -- -- > 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 > 40 -- 41 05 80 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 > 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 > 40 -- 41 00 08 00 00 0f 70 d8 08 40 00 Error: UNC at LBA = 0x0f70d808 = 259053576 [..snip extras of these..] Consistent. > SMART Extended Self-test Log Version: 1 (2 sectors) > Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error > # 1 Short captive Completed: read failure 90% 1384 259053576 > # 2 Short captive Completed: read failure 90% 1384 259053576 Also consistent. For whatever reason it's not being overwritten... I guess the copy on dev/sda is bad or unavailable. > > SATA Phy Event Counters (GP Log 0x11) The vendor specific ones have a massive pile of noise in them compared to the other drive. But inconclusive because they aren't defined. -- Chris Murphy ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2016-06-30 21:02 UTC | newest] Thread overview: 27+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-06-27 21:36 Kernel bug during RAID1 replace Saint Germain 2016-06-27 21:42 ` Chris Murphy 2016-06-27 22:26 ` Saint Germain 2016-06-27 22:55 ` Chris Murphy 2016-06-27 22:58 ` Chris Murphy 2016-06-27 23:06 ` Saint Germain 2016-06-28 0:00 ` Chris Murphy 2016-06-28 0:10 ` Chris Murphy 2016-06-28 0:49 ` Saint Germain 2016-06-28 2:14 ` Chris Murphy 2016-06-28 22:52 ` Saint Germain 2016-06-29 4:25 ` Chris Murphy 2016-06-29 9:50 ` Saint Germain 2016-06-29 17:28 ` Chris Murphy 2016-06-29 18:12 ` Saint Germain 2016-06-29 18:19 ` Austin S. Hemmelgarn 2016-06-29 19:02 ` Saint Germain 2016-06-29 19:08 ` Chris Murphy 2016-06-29 19:16 ` Saint Germain 2016-06-29 19:23 ` Hugo Mills 2016-06-29 23:51 ` Saint Germain 2016-06-30 0:24 ` Chris Murphy 2016-06-30 21:02 ` Saint Germain 2016-06-30 0:19 ` Chris Murphy 2016-06-29 17:41 ` Saint Germain 2016-06-27 23:03 ` Saint Germain 2016-06-27 23:49 ` Chris Murphy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).