Kernel backtrace and hang while replacing dead drive

[Hugo Mills <hugo@carfax.org.uk>]

   On Saturday morning, one SSD in a 4-drive RAID-1 array failed (the
failed drive is /dev/sdb, devid 5). The replacement arrived today. I
physically removed the dead drive, and added the new one to the
machine. This was /dev/sdj.

   I ran "btrfs replace start -r 5 /dev/sdj1 /". At 12% done,
everything hung. I was able to get the log info below before the whole
machine locked up (the array in question contains the system
installation, so when the array locked up, the machine became
unresponsive).

   I'm going to reboot and try again. I'm leaving this report here in
case anyone's interested in following up on why it happened, or wants
to suggest an alternative way forward. (I'll be on the freenode IRC
channel, too, as darkling).

   Thanks,
   Hugo.

Nov  3 15:56:14 s_src@amelia kernel: BTRFS error (device sdd1): bdev /dev/sdb1 errs: wr 50488570, rd 36558204, f
lush 24486, corrupt 0, gen 0
Nov  3 15:56:16 s_src@amelia kernel: scrub_handle_errored_block: 18199 callbacks suppressed
Nov  3 15:56:16 s_src@amelia kernel: _btrfs_printk: 163 callbacks suppressed
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236658860032 on dev /dev
/sdb1, physical 61027368960: metadata leaf (level 0) in tree 2284479184896
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236658860032 on dev /dev
/sdb1, physical 61027368960: metadata leaf (level 0) in tree 2284479184896
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236658909184 on dev /dev
/sdb1, physical 61027418112: metadata leaf (level 0) in tree 257
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236658909184 on dev /dev
/sdb1, physical 61027418112: metadata leaf (level 0) in tree 257
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236659089408 on dev /dev
/sdb1, physical 61027598336: metadata leaf (level 0) in tree 7
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236659089408 on dev /dev
/sdb1, physical 61027598336: metadata leaf (level 0) in tree 7
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): i/o error at logical 2236659204096 on dev /dev
/sdb1, physical 61027713024: metadata leaf (level -1) in tree 18446744073709551615
Nov  3 15:56:16 s_src@amelia kernel: BTRFS warning (device sdd1): bad eb member end: ptr 0x4000 start 2237987028
992 member offset 16384 size 1
Nov  3 15:56:16 s_src@amelia kernel: general protection fault, probably for non-canonical address 0xda7b8c000000
0: 0000 [#1] PREEMPT SMP NOPTI
Nov  3 15:56:16 s_src@amelia kernel: CPU: 5 PID: 1939101 Comm: kworker/u16:8 Not tainted 5.19.0-2-amd64 #1  Debi
an 5.19.11-1
Nov  3 15:56:16 s_src@amelia kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS
3P, BIOS FD 02/26/2016
Nov  3 15:56:16 s_src@amelia kernel: Workqueue: btrfs-scrub scrub_bio_end_io_worker [btrfs]
Nov  3 15:56:16 s_src@amelia kernel: RIP: 0010:btrfs_get_8+0x57/0x70 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel: Code: e5 ff 0f 00 00 48 c1 fb 06 48 c1 e3 0c 48 03 1d 4f 83 12 ec 49 39 d0 
0f 87 22 2f 09 00 49 8d 48 01 48 39 ca 0f 82 36 2f 09 00 <0f> b6 04 2b 5b 5d e9 ce 4e b4 eb 66 66 2e 0f 1f 84 00
 00 00 00 00
Nov  3 15:56:16 s_src@amelia kernel: RSP: 0018:ffffa3cf8d6abc50 EFLAGS: 00010296
Nov  3 15:56:16 s_src@amelia kernel: RAX: 0000000000000000 RBX: 000da7b8c0000000 RCX: 0000000000000000
Nov  3 15:56:16 s_src@amelia kernel: RDX: 0000000000000000 RSI: ffffffffac76ee29 RDI: 00000000ffffffff
Nov  3 15:56:16 s_src@amelia kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000104903c77
Nov  3 15:56:16 s_src@amelia kernel: R10: ffffa3cf8d6aba60 R11: ffffffffacecca48 R12: 0000000000003fd6
Nov  3 15:56:16 s_src@amelia kernel: R13: 0000000000004000 R14: 0000000000003fee R15: ffff94c9d6cfc400
Nov  3 15:56:16 s_src@amelia kernel: FS:  0000000000000000(0000) GS:ffff94cceed40000(0000) knlGS:000000000000000
0
Nov  3 15:56:16 s_src@amelia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  3 15:56:16 s_src@amelia kernel: CR2: 00007fc2cba0a360 CR3: 000000010b8a6000 CR4: 00000000000406e0
Nov  3 15:56:16 s_src@amelia kernel: Call Trace:
Nov  3 15:56:16 s_src@amelia kernel:  <TASK>
Nov  3 15:56:16 s_src@amelia kernel:  btrfs_get_extent_inline_ref_type+0x1a/0x110 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel:  tree_backref_for_extent+0x96/0x1a0 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel:  scrub_print_warning+0x1b1/0x1d0 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel:  scrub_handle_errored_block.isra.0+0x100f/0x1110 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel:  scrub_bio_end_io_worker+0xa6/0x230 [btrfs]
Nov  3 15:56:16 s_src@amelia kernel:  process_one_work+0x1e5/0x3b0
Nov  3 15:56:16 s_src@amelia kernel:  worker_thread+0x50/0x3a0
Nov  3 15:56:16 s_src@amelia kernel:  ? rescuer_thread+0x390/0x390
Nov  3 15:56:16 s_src@amelia kernel:  kthread+0xe8/0x110
Nov  3 15:56:16 s_src@amelia kernel:  ? kthread_complete_and_exit+0x20/0x20
Nov  3 15:56:16 s_src@amelia kernel:  ret_from_fork+0x22/0x30
Nov  3 15:56:16 s_src@amelia kernel:  </TASK>
Nov  3 15:56:16 s_src@amelia kernel: Modules linked in: tls unix_diag fuse udf crc_itu_t nfsd auth_rpcgss nfs_acl lockd grace sunrpc bridge stp llc it87 hwmon_vid parport_pc ppdev lp parport dm_crypt dm_mod ses enclosure scsi_transport_sas amdgpu gpu_sched radeon amd64_edac edac_mce_amd drm_display_helper kvm_amd cec ccp rc_core snd_hda_codec_realtek drm_ttm_helper ttm rng_core r8169 snd_hda_codec_generic drm_kms_helper kvm snd_hda_codec_hdmi ledtrig_audio irqbypass realtek xhci_pci crc32_pclmul snd_hda_intel sr_mod snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core mdio_devres ohci_pci ghash_clmulni_intel cdrom drm megaraid_sas snd_hwdep pcspkr efi_pstore k10temp fam15h_power libphy xhci_hcd sg ohci_hcd ehci_pci i2c_algo_bit snd_pcm sp5100_tco ehci_hcd watchdog snd_timer usbcore snd i2c_piix4 usb_common soundcore button acpi_cpufreq btrfs blake2b_generic zstd_compress efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c crc32c_generic xor raid6_pq raid1 raid0
Nov  3 15:56:16 s_src@amelia kernel:  multipath linear md_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci sata_sil24 libata aesni_intel scsi_mod crypto_simd cryptd evdev serio_raw scsi_common
Nov  3 15:56:16 s_src@amelia kernel: ---[ end trace 0000000000000000 ]---
Nov  3 15:56:17 s_src@amelia kernel: RIP: 0010:btrfs_get_8+0x57/0x70 [btrfs]
Nov  3 15:56:17 s_src@amelia kernel: Code: e5 ff 0f 00 00 48 c1 fb 06 48 c1 e3 0c 48 03 1d 4f 83 12 ec 49 39 d0 0f 87 22 2f 09 00 49 8d 48 01 48 39 ca 0f 82 36 2f 09 00 <0f> b6 04 2b 5b 5d e9 ce 4e b4 eb 66 66 2e 0f 1f 84 00 00 00 00 00
Nov  3 15:56:17 s_src@amelia kernel: RSP: 0018:ffffa3cf8d6abc50 EFLAGS: 00010296
Nov  3 15:56:17 s_src@amelia kernel: RAX: 0000000000000000 RBX: 000da7b8c0000000 RCX: 0000000000000000
Nov  3 15:56:17 s_src@amelia kernel: RDX: 0000000000000000 RSI: ffffffffac76ee29 RDI: 00000000ffffffff
Nov  3 15:56:17 s_src@amelia kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000104903c77
Nov  3 15:56:17 s_src@amelia kernel: R10: ffffa3cf8d6aba60 R11: ffffffffacecca48 R12: 0000000000003fd6
Nov  3 15:56:17 s_src@amelia kernel: R13: 0000000000004000 R14: 0000000000003fee R15: ffff94c9d6cfc400
Nov  3 15:56:17 s_src@amelia kernel: FS:  0000000000000000(0000) GS:ffff94cceed40000(0000) knlGS:0000000000000000
Nov  3 15:56:17 s_src@amelia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  3 15:56:17 s_src@amelia kernel: CR2: 00007fc2cba0a360 CR3: 000000010b8a6000 CR4: 00000000000406e0
Nov  3 15:56:23 s_src@amelia kernel: btrfs_dev_stat_print_on_error: 7137 callbacks suppressed
Nov  3 15:56:23 s_src@amelia kernel: BTRFS error (device sdd1): bdev /dev/sdb1 errs: wr 50488571, rd 36565341, flush 24486, corrupt 0, gen 0
Nov  3 15:56:24 s_src@amelia kernel: BTRFS error (device sdd1): bdev /dev/sdb1 errs: wr 50488571, rd 36565341, flush 24487, corrupt 0, gen 0


-- 
Hugo Mills             | "You know, the British have always been nice to mad
hugo@... carfax.org.uk | people."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                         Laura Jesson, Brief Encounter