* Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
@ 2014-04-08 15:36 Marc MERLIN
2014-04-08 22:09 ` Marc MERLIN
0 siblings, 1 reply; 10+ messages in thread
From: Marc MERLIN @ 2014-04-08 15:36 UTC (permalink / raw)
To: linux-btrfs
Soon after upgrading my server from 3.14rc5 to 3.14.0, my server went
into a crash loop.
Unfortunately because I used btrfs on my root filesystem, the problem
didn't get logged because btrfs failing on a separate array prevented
another thread to use btrfs to write on a healthy /var filesystem.
So I found my system in a reboot loop, and after commenting out the
devices in /etc/crypttab, prevented btrfs from mounting dm-5, which has
restored my system to sanity for now, but with a missing filesystem.
dm-5 was a 2 device raid0.
I'm afraid of mounting it, because I think it'll crash my system again,
but I'm happy to run commands against it.
Question #1:
Will mounting with read-only stop the cleaner process?
Question #2:
Is mounting with recovery,ro and copying the data out if I can the best
way to recover?
(and destroying, recreating the filesystem after that)
Question #3:
Any idea what happened and what bug I hit?
Thanks,
Marc
Logs below:
Will now mount local filesystems:[ 63.516591] EXT4-fs (md11): mounted filesystem with ordered data mode. Opts: (null)
[ 63.540841] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
[ 63.692581] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
[ 63.836698] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
[ 63.946790] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
[ 64.119580] BTRFS: device label btrfs_raid0 devid 2 transid 50560 /dev/mapper/raid0d2
[ 64.145339] BTRFS info (device dm-5): disk space caching is enabled
[ 71.139007] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
[ 71.337153] BTRFS: device label btrfs_raid0 devid 2 transid 50560 /dev/mapper/raid0d2
[ 71.395269] BTRFS: device label btrfs_pool1 devid 1 transid 43963 /dev/mapper/dshelf1
[ 71.420616] BTRFS info (device dm-1): disk space caching is enabled
[ 96.523664] BTRFS: device label btrfs_pool2 devid 1 transid 42458 /dev/mapper/dshelf2
[ 96.549209] BTRFS info (device dm-2): disk space caching is enabled
[ 96.568833] BTRFS: has skinny extents
[ 102.679436] ------------[ cut here ]------------
[ 102.694202] WARNING: CPU: 2 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[ 102.723152] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
[ 103.011058] CPU: 2 PID: 3105 Comm: btrfs-cleaner Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[ 103.038925] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[ 103.067351] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
[ 103.090778] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
[ 103.114210] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
[ 103.137678] Call Trace:
[ 103.146122] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[ 103.162653] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[ 103.181877] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[ 103.201778] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[ 103.220393] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[ 103.239793] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[ 103.258428] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[ 103.278897] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[ 103.299623] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[ 103.319852] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[ 103.339823] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[ 103.357452] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[ 103.378698] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[ 103.398131] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[ 103.420997] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[ 103.439399] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[ 103.458068] [<ffffffff8106bc62>] kthread+0xae/0xb6
[ 103.473886] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 103.492615] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[ 103.509950] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 103.528605] ---[ end trace a51c57d8cb84bfb4 ]---
[ 103.543602] BTRFS info (device dm-5): leaf 264060928 total ptrs 186 free space 4751
[ 103.568045] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[ 103.590080] extent refs 1 gen 12770 flags 1
[ 103.604291] shared data backref parent 223898337280 count 1
[ 103.622671] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[ 103.644693] extent refs 1 gen 12770 flags 1
[ 103.658909] shared data backref parent 223898337280 count 1
[ 103.677261] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
[ 103.699293] extent refs 1 gen 12770 flags 1
[ 103.713432] shared data backref parent 223898337280 count 1
[ 103.731703] item 3 key (1125738274816 168 40960) itemoff 16135 itemsize 37
[ 103.753599] extent refs 1 gen 12770 flags 1
[ 103.767668] shared data backref parent 223898337280 count 1
[ 103.785898] item 4 key (1125740027904 168 40960) itemoff 16098 itemsize 37
[ 103.807734] extent refs 1 gen 12770 flags 1
[ 103.821722] shared data backref parent 223898337280 count 1
[ 103.839879] item 5 key (1125741158400 168 40960) itemoff 16061 itemsize 37
[ 103.861682] extent refs 1 gen 12770 flags 1
[ 103.875633] shared data backref parent 223898337280 count 1
[ 103.893775] item 6 key (1125742604288 168 40960) itemoff 16024 itemsize 37
[ 103.915557] extent refs 1 gen 12770 flags 1
[ 103.929519] shared data backref parent 223898468352 count 1
[ 103.947641] item 7 key (1125745111040 168 40960) itemoff 15987 itemsize 37
[ 103.969435] extent refs 1 gen 12770 flags 1
[ 103.983401] shared data backref parent 223898468352 count 1
[ 104.001546] item 8 key (1125746237440 168 40960) itemoff 15950 itemsize 37
[ 104.023342] extent refs 1 gen 12770 flags 1
[ 104.037334] shared data backref parent 223898468352 count 1
[ 104.055509] item 9 key (1125749014528 168 40960) itemoff 15913 itemsize 37
[ 104.077335] extent refs 1 gen 12770 flags 1
[ 104.091347] shared data backref parent 223898468352 count 1
[ 104.109514] item 10 key (1125751156736 168 40960) itemoff 15876 itemsize 37
[ 104.131628] extent refs 1 gen 12770 flags 1
[ 104.145784] shared data backref parent 223898468352 count 1
[ 104.163977] item 11 key (1125752602624 168 40960) itemoff 15839 itemsize 37
[ 104.186107] extent refs 1 gen 12770 flags 1
[ 104.200151] shared data backref parent 223898468352 count 1
[ 104.218373] item 12 key (1125755486208 168 40960) itemoff 15802 itemsize 37
[ 104.240572] extent refs 1 gen 12770 flags 1
[ 104.254636] shared data backref parent 223898468352 count 1
[ 104.272895] item 13 key (1125756932096 168 40960) itemoff 15765 itemsize 37
[ 104.295062] extent refs 1 gen 12770 flags 1
[ 104.309145] shared data backref parent 223898468352 count 1
[ 104.327391] item 14 key (1125758607360 168 40960) itemoff 15728 itemsize 37
[ 104.349541] extent refs 1 gen 12770 flags 1
[ 104.363630] shared data backref parent 223898468352 count 1
[ 104.381880] item 15 key (1125759422464 168 40960) itemoff 15691 itemsize 37
[ 104.404193] extent refs 1 gen 12770 flags 1
[ 104.418281] shared data backref parent 223898468352 count 1
[ 104.436527] item 16 key (1125762203648 168 40960) itemoff 15654 itemsize 37
[ 104.458707] extent refs 1 gen 12770 flags 1
[ 104.472797] shared data backref parent 223898468352 count 1
[ 104.491049] item 17 key (1125763727360 168 40960) itemoff 15617 itemsize 37
[ 104.513226] extent refs 1 gen 12770 flags 1
[ 104.527314] shared data backref parent 223898468352 count 1
[ 104.545561] item 18 key (1125765218304 168 40960) itemoff 15580 itemsize 37
[ 104.567749] extent refs 1 gen 12770 flags 1
[ 104.581835] shared data backref parent 223898468352 count 1
[ 104.600110] item 19 key (1125765726208 168 40960) itemoff 15543 itemsize 37
[ 104.622290] extent refs 1 gen 12770 flags 1
[ 104.636377] shared data backref parent 223898468352 count 1
[ 104.654657] item 20 key (1125766475776 168 40960) itemoff 15506 itemsize 37
[ 104.676833] extent refs 1 gen 12770 flags 1
[ 104.690918] shared data backref parent 223898468352 count 1
[ 104.709197] item 21 key (1125766668288 168 40960) itemoff 15469 itemsize 37
[ 104.731375] extent refs 1 gen 12770 flags 1
[ 104.745463] shared data backref parent 223898468352 count 1
[ 104.763745] item 22 key (1125767483392 168 40960) itemoff 15432 itemsize 37
[ 104.785917] extent refs 1 gen 12770 flags 1
[ 104.800064] shared data backref parent 223898468352 count 1
[ 104.818330] item 23 key (1125767868416 168 40960) itemoff 15395 itemsize 37
[ 104.840515] extent refs 1 gen 12770 flags 1
[ 104.854637] shared data backref parent 223898468352 count 1
[ 104.872905] item 24 key (1125771141120 168 40960) itemoff 15358 itemsize 37
[ 104.895080] extent refs 1 gen 12770 flags 1
[ 104.909198] shared data backref parent 223898468352 count 1
[ 104.927474] item 25 key (1125771886592 168 40960) itemoff 15321 itemsize 37
[ 104.949648] extent refs 1 gen 12770 flags 1
[ 104.963760] shared data backref parent 223898468352 count 1
[ 104.982034] item 26 key (1125773021184 168 40960) itemoff 15284 itemsize 37
[ 105.004216] extent refs 1 gen 12770 flags 1
[ 105.018331] shared data backref parent 223898468352 count 1
[ 105.036604] item 27 key (1125774393344 168 40960) itemoff 15247 itemsize 37
[ 105.058784] extent refs 1 gen 12770 flags 1
[ 105.072898] shared data backref parent 223898468352 count 1
[ 105.091171] item 28 key (1125779865600 168 40960) itemoff 15210 itemsize 37
[ 105.113350] extent refs 1 gen 12770 flags 1
[ 105.127464] shared data backref parent 223898468352 count 1
[ 105.145740] item 29 key (1125781073920 168 40960) itemoff 15173 itemsize 37
[ 105.167919] extent refs 1 gen 12770 flags 1
[ 105.182036] shared data backref parent 223898468352 count 1
[ 105.200307] item 30 key (1125782454272 168 40960) itemoff 15136 itemsize 37
[ 105.222488] extent refs 1 gen 12770 flags 1
[ 105.236605] shared data backref parent 223898468352 count 1
[ 105.254879] item 31 key (1125785354240 168 40960) itemoff 15099 itemsize 37
[ 105.277055] extent refs 1 gen 12770 flags 1
[ 105.291166] shared data backref parent 223898468352 count 1
[ 105.309445] item 32 key (1125785784320 168 40960) itemoff 15062 itemsize 37
[ 105.331622] extent refs 1 gen 12770 flags 1
[ 105.345737] shared data backref parent 223898468352 count 1
[ 105.364011] item 33 key (1125786988544 168 40960) itemoff 15025 itemsize 37
[ 105.386193] extent refs 1 gen 12770 flags 1
[ 105.400302] shared data backref parent 223898468352 count 1
[ 105.418585] item 34 key (1125790654464 168 40960) itemoff 14988 itemsize 37
[ 105.440785] extent refs 1 gen 12770 flags 1
[ 105.454897] shared data backref parent 223898468352 count 1
[ 105.473181] item 35 key (1125791793152 168 40960) itemoff 14951 itemsize 37
[ 105.495381] extent refs 1 gen 12770 flags 1
[ 105.509493] shared data backref parent 223898468352 count 1
[ 105.527774] item 36 key (1125793554432 168 40960) itemoff 14914 itemsize 37
[ 105.550052] extent refs 1 gen 12770 flags 1
[ 105.564169] shared data backref parent 223898468352 count 1
[ 105.582441] item 37 key (1125794611200 168 40960) itemoff 14877 itemsize 37
[ 105.604643] extent refs 1 gen 12770 flags 1
[ 105.618762] shared data backref parent 223898468352 count 1
[ 105.637094] item 38 key (1125797896192 168 40960) itemoff 14840 itemsize 37
[ 105.659379] extent refs 1 gen 12770 flags 1
[ 105.673483] shared data backref parent 223898468352 count 1
[ 105.691765] item 39 key (1125801062400 168 40960) itemoff 14803 itemsize 37
[ 105.714015] extent refs 1 gen 12770 flags 1
[ 105.728135] shared data backref parent 223898468352 count 1
[ 105.746428] item 40 key (1125802123264 168 40960) itemoff 14766 itemsize 37
[ 105.768688] extent refs 1 gen 12770 flags 1
[ 105.782796] shared data backref parent 223898468352 count 1
[ 105.801145] item 41 key (1125803565056 168 40960) itemoff 14729 itemsize 37
[ 105.823332] extent refs 1 gen 12770 flags 1
[ 105.837445] shared data backref parent 223898468352 count 1
[ 105.855720] item 42 key (1125804216320 168 40960) itemoff 14692 itemsize 37
[ 105.877921] extent refs 1 gen 12770 flags 1
[ 105.892040] shared data backref parent 223898468352 count 1
[ 105.910310] item 43 key (1125804675072 168 40960) itemoff 14655 itemsize 37
[ 105.932487] extent refs 1 gen 12770 flags 1
[ 105.946599] shared data backref parent 223898468352 count 1
[ 105.964877] item 44 key (1125804867584 168 40960) itemoff 14618 itemsize 37
[ 105.987054] extent refs 1 gen 12770 flags 1
[ 106.001172] shared data backref parent 223898468352 count 1
[ 106.019443] item 45 key (1125806006272 168 40960) itemoff 14581 itemsize 37
[ 106.041617] extent refs 1 gen 12770 flags 1
[ 106.055743] shared data backref parent 223898468352 count 1
[ 106.074014] item 46 key (1125807456256 168 40960) itemoff 14544 itemsize 37
[ 106.096229] extent refs 1 gen 12770 flags 1
[ 106.110327] shared data backref parent 223898468352 count 1
[ 106.128616] item 47 key (1125810675712 168 40960) itemoff 14507 itemsize 37
[ 106.150814] extent refs 1 gen 12770 flags 1
[ 106.164927] shared data backref parent 223898468352 count 1
[ 106.183256] item 48 key (1125812047872 168 40960) itemoff 14470 itemsize 37
[ 106.205454] extent refs 1 gen 12770 flags 1
[ 106.219629] shared data backref parent 223898468352 count 1
[ 106.237898] item 49 key (1125813186560 168 40960) itemoff 14433 itemsize 37
[ 106.260080] extent refs 1 gen 12770 flags 1
[ 106.274206] shared data backref parent 223898468352 count 1
[ 106.292494] item 50 key (1125814251520 168 40960) itemoff 14396 itemsize 37
[ 106.314695] extent refs 1 gen 12770 flags 1
(...)
[ 113.651650] item 185 key (1126124646400 168 45056) itemoff 9401 itemsize 37
[ 113.673859] extent refs 1 gen 12537 flags 1
[ 113.687970] shared data backref parent 63894044672 count 1
[ 113.706013] BTRFS error (device dm-5): unable to find ref byte nr 1125950996480 parent 63893716992 root 0 owner 7977109 offset 0
[ 113.788672] ------------[ cut here ]------------
[ 113.803626] WARNING: CPU: 2 PID: 3105 at fs/btrfs/free-space-cache.c:1183 tree_insert_offset+0x81/0xc5()
[ 113.833793] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
[ 114.125608] CPU: 2 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
[ 114.156659] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[ 114.185373] 0000000000000000 ffff880213f63928 ffffffff8160a06d 0000000000000000
[ 114.209071] ffff880213f63960 ffffffff81050025 ffffffff8125e81a 00000000ffffffef
[ 114.232737] ffff88020ce2a1b0 0000000000000000 ffff8802125f8800 ffff880213f63970
[ 114.256422] Call Trace:
[ 114.265075] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[ 114.281817] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[ 114.301150] [<ffffffff8125e81a>] ? tree_insert_offset+0x81/0xc5
[ 114.320518] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[ 114.339318] [<ffffffff8125e81a>] tree_insert_offset+0x81/0xc5
[ 114.358249] [<ffffffff8125e88a>] link_free_space+0x2c/0x41
[ 114.376238] [<ffffffff81260f1b>] __load_free_space_cache+0x304/0x56c
[ 114.396812] [<ffffffff81261269>] load_free_space_cache+0xe6/0x199
[ 114.416619] [<ffffffff81141648>] ? kmem_cache_alloc_trace+0x11b/0x12b
[ 114.437456] [<ffffffff81216354>] cache_block_group+0x1b2/0x33d
[ 114.456465] [<ffffffff81085116>] ? finish_wait+0x65/0x65
[ 114.473916] [<ffffffff81216c49>] update_block_group.isra.38+0xaa/0x1fe
[ 114.495009] [<ffffffff81217449>] __btrfs_free_extent+0x6ac/0x712
[ 114.514526] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[ 114.533262] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[ 114.553842] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[ 114.574789] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[ 114.595070] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[ 114.615064] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[ 114.632726] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[ 114.654014] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[ 114.673530] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[ 114.696360] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[ 114.714783] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[ 114.733435] [<ffffffff8106bc62>] kthread+0xae/0xb6
[ 114.749169] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 114.767775] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[ 114.785052] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 114.803608] ---[ end trace a51c57d8cb84bfb5 ]---
[ 114.818501] BTRFS error (device dm-5): Duplicate entries in free space cache, dumping
[ 114.843818] BTRFS error (device dm-5): failed to load free space cache for block group 1125310791680
[ 114.873135] ------------[ cut here ]------------
[ 114.888456] WARNING: CPU: 0 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[ 114.917736] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
[ 115.207261] CPU: 0 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
[ 115.238090] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[ 115.266591] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
[ 115.290066] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
[ 115.313567] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
[ 115.337073] Call Trace:
[ 115.345558] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[ 115.362133] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[ 115.381302] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[ 115.401258] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[ 115.419913] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[ 115.439328] [<ffffffff81610200>] ? retint_signal+0x8c/0x8c
[ 115.457198] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[ 115.475833] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[ 115.496285] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[ 115.516999] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[ 115.537191] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[ 115.557128] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[ 115.574739] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[ 115.595976] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[ 115.615413] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[ 115.638217] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[ 115.656608] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[ 115.675260] [<ffffffff8106bc62>] kthread+0xae/0xb6
[ 115.691028] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 115.709660] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[ 115.726989] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 115.745622] ---[ end trace a51c57d8cb84bfb6 ]---
[ 115.760616] BTRFS info (device dm-5): leaf 264060928 total ptrs 185 free space 4813
[ 115.784754] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[ 115.806782] extent refs 1 gen 12770 flags 1
[ 115.821011] shared data backref parent 223898337280 count 1
[ 115.839401] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[ 115.861406] extent refs 1 gen 12770 flags 1
[ 115.875593] shared data backref parent 223898337280 count 1
[ 115.893927] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
[ 115.915880] extent refs 1 gen 12770 flags 1
[ 115.929977] shared data backref parent 223898337280 count 1
(...)
[ 125.804965] item 184 key (1126124646400 168 45056) itemoff 9438 itemsize 37
[ 125.827174] extent refs 1 gen 12537 flags 1
[ 125.841290] shared data backref parent 63894044672 count 1
[ 125.859328] BTRFS error (device dm-5): unable to find ref byte nr 1125993029632 parent 63893716992 root 0 owner 7976978 offset 0
[ 125.896481] ------------[ cut here ]------------
[ 125.912031] WARNING: CPU: 2 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[ 125.941657] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
[ 126.233519] CPU: 2 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
[ 126.264560] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[ 126.293279] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
[ 126.316958] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
[ 126.340621] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
[ 126.364309] Call Trace:
[ 126.372966] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[ 126.389719] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[ 126.409086] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[ 126.429198] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[ 126.447995] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[ 126.467566] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[ 126.486341] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[ 126.507009] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[ 126.527865] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[ 126.548231] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[ 126.568283] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[ 126.586109] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[ 126.607470] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[ 126.626998] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[ 126.649898] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[ 126.668364] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[ 126.687087] [<ffffffff8106bc62>] kthread+0xae/0xb6
[ 126.702931] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 126.721652] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[ 126.739054] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[ 126.757757] ---[ end trace a51c57d8cb84bfb7 ]---
[ 126.772784] BTRFS info (device dm-5): leaf 264060928 total ptrs 184 free space 4875
[ 126.796952] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[ 126.819017] extent refs 1 gen 12770 flags 1
[ 126.833247] shared data backref parent 223898337280 count 1
[ 126.851622] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[ 126.873650] extent refs 1 gen 12770 flags 1
[ 126.887857] shared data backref parent 223898337280 count 1
[ 126.906246] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
[ 126.928237] extent refs 1 gen 12770 flags 1
[ 126.942386] shared data backref parent 223898337280 count 1
(...)
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-08 15:36 Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 ) Marc MERLIN
@ 2014-04-08 22:09 ` Marc MERLIN
2014-04-08 23:49 ` Chris Mason
0 siblings, 1 reply; 10+ messages in thread
From: Marc MERLIN @ 2014-04-08 22:09 UTC (permalink / raw)
To: linux-btrfs
I forgot to add that while I'm not sure if anyone ended up looking at the
last image I made regarding
https://bugzilla.kernel.org/show_bug.cgi?id=72801
I can generate a an image of that filesystem if that helps, or try other
commands which hopefully won't crash my running server :)
(filesystem is almost 2TB, so the image will again be big)
Marc
On Tue, Apr 08, 2014 at 08:36:09AM -0700, Marc MERLIN wrote:
> Soon after upgrading my server from 3.14rc5 to 3.14.0, my server went
> into a crash loop.
>
> Unfortunately because I used btrfs on my root filesystem, the problem
> didn't get logged because btrfs failing on a separate array prevented
> another thread to use btrfs to write on a healthy /var filesystem.
>
> So I found my system in a reboot loop, and after commenting out the
> devices in /etc/crypttab, prevented btrfs from mounting dm-5, which has
> restored my system to sanity for now, but with a missing filesystem.
>
> dm-5 was a 2 device raid0.
> I'm afraid of mounting it, because I think it'll crash my system again,
> but I'm happy to run commands against it.
>
> Question #1:
> Will mounting with read-only stop the cleaner process?
>
> Question #2:
> Is mounting with recovery,ro and copying the data out if I can the best
> way to recover?
> (and destroying, recreating the filesystem after that)
>
> Question #3:
> Any idea what happened and what bug I hit?
>
> Thanks,
> Marc
>
> Logs below:
>
> Will now mount local filesystems:[ 63.516591] EXT4-fs (md11): mounted filesystem with ordered data mode. Opts: (null)
> [ 63.540841] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
> [ 63.692581] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
> [ 63.836698] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
> [ 63.946790] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
> [ 64.119580] BTRFS: device label btrfs_raid0 devid 2 transid 50560 /dev/mapper/raid0d2
> [ 64.145339] BTRFS info (device dm-5): disk space caching is enabled
> [ 71.139007] BTRFS: device label btrfs_boot devid 1 transid 130715 /dev/mapper/cryptroot
> [ 71.337153] BTRFS: device label btrfs_raid0 devid 2 transid 50560 /dev/mapper/raid0d2
> [ 71.395269] BTRFS: device label btrfs_pool1 devid 1 transid 43963 /dev/mapper/dshelf1
> [ 71.420616] BTRFS info (device dm-1): disk space caching is enabled
> [ 96.523664] BTRFS: device label btrfs_pool2 devid 1 transid 42458 /dev/mapper/dshelf2
> [ 96.549209] BTRFS info (device dm-2): disk space caching is enabled
> [ 96.568833] BTRFS: has skinny extents
> [ 102.679436] ------------[ cut here ]------------
> [ 102.694202] WARNING: CPU: 2 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
> [ 102.723152] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
> [ 103.011058] CPU: 2 PID: 3105 Comm: btrfs-cleaner Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
> [ 103.038925] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
> [ 103.067351] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
> [ 103.090778] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
> [ 103.114210] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
> [ 103.137678] Call Trace:
> [ 103.146122] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
> [ 103.162653] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
> [ 103.181877] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
> [ 103.201778] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
> [ 103.220393] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
> [ 103.239793] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
> [ 103.258428] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
> [ 103.278897] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
> [ 103.299623] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
> [ 103.319852] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
> [ 103.339823] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
> [ 103.357452] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
> [ 103.378698] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
> [ 103.398131] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
> [ 103.420997] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
> [ 103.439399] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
> [ 103.458068] [<ffffffff8106bc62>] kthread+0xae/0xb6
> [ 103.473886] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 103.492615] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
> [ 103.509950] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 103.528605] ---[ end trace a51c57d8cb84bfb4 ]---
> [ 103.543602] BTRFS info (device dm-5): leaf 264060928 total ptrs 186 free space 4751
> [ 103.568045] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
> [ 103.590080] extent refs 1 gen 12770 flags 1
> [ 103.604291] shared data backref parent 223898337280 count 1
> [ 103.622671] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
> [ 103.644693] extent refs 1 gen 12770 flags 1
> [ 103.658909] shared data backref parent 223898337280 count 1
> [ 103.677261] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
> [ 103.699293] extent refs 1 gen 12770 flags 1
> [ 103.713432] shared data backref parent 223898337280 count 1
> [ 103.731703] item 3 key (1125738274816 168 40960) itemoff 16135 itemsize 37
> [ 103.753599] extent refs 1 gen 12770 flags 1
> [ 103.767668] shared data backref parent 223898337280 count 1
> [ 103.785898] item 4 key (1125740027904 168 40960) itemoff 16098 itemsize 37
> [ 103.807734] extent refs 1 gen 12770 flags 1
> [ 103.821722] shared data backref parent 223898337280 count 1
> [ 103.839879] item 5 key (1125741158400 168 40960) itemoff 16061 itemsize 37
> [ 103.861682] extent refs 1 gen 12770 flags 1
> [ 103.875633] shared data backref parent 223898337280 count 1
> [ 103.893775] item 6 key (1125742604288 168 40960) itemoff 16024 itemsize 37
> [ 103.915557] extent refs 1 gen 12770 flags 1
> [ 103.929519] shared data backref parent 223898468352 count 1
> [ 103.947641] item 7 key (1125745111040 168 40960) itemoff 15987 itemsize 37
> [ 103.969435] extent refs 1 gen 12770 flags 1
> [ 103.983401] shared data backref parent 223898468352 count 1
> [ 104.001546] item 8 key (1125746237440 168 40960) itemoff 15950 itemsize 37
> [ 104.023342] extent refs 1 gen 12770 flags 1
> [ 104.037334] shared data backref parent 223898468352 count 1
> [ 104.055509] item 9 key (1125749014528 168 40960) itemoff 15913 itemsize 37
> [ 104.077335] extent refs 1 gen 12770 flags 1
> [ 104.091347] shared data backref parent 223898468352 count 1
> [ 104.109514] item 10 key (1125751156736 168 40960) itemoff 15876 itemsize 37
> [ 104.131628] extent refs 1 gen 12770 flags 1
> [ 104.145784] shared data backref parent 223898468352 count 1
> [ 104.163977] item 11 key (1125752602624 168 40960) itemoff 15839 itemsize 37
> [ 104.186107] extent refs 1 gen 12770 flags 1
> [ 104.200151] shared data backref parent 223898468352 count 1
> [ 104.218373] item 12 key (1125755486208 168 40960) itemoff 15802 itemsize 37
> [ 104.240572] extent refs 1 gen 12770 flags 1
> [ 104.254636] shared data backref parent 223898468352 count 1
> [ 104.272895] item 13 key (1125756932096 168 40960) itemoff 15765 itemsize 37
> [ 104.295062] extent refs 1 gen 12770 flags 1
> [ 104.309145] shared data backref parent 223898468352 count 1
> [ 104.327391] item 14 key (1125758607360 168 40960) itemoff 15728 itemsize 37
> [ 104.349541] extent refs 1 gen 12770 flags 1
> [ 104.363630] shared data backref parent 223898468352 count 1
> [ 104.381880] item 15 key (1125759422464 168 40960) itemoff 15691 itemsize 37
> [ 104.404193] extent refs 1 gen 12770 flags 1
> [ 104.418281] shared data backref parent 223898468352 count 1
> [ 104.436527] item 16 key (1125762203648 168 40960) itemoff 15654 itemsize 37
> [ 104.458707] extent refs 1 gen 12770 flags 1
> [ 104.472797] shared data backref parent 223898468352 count 1
> [ 104.491049] item 17 key (1125763727360 168 40960) itemoff 15617 itemsize 37
> [ 104.513226] extent refs 1 gen 12770 flags 1
> [ 104.527314] shared data backref parent 223898468352 count 1
> [ 104.545561] item 18 key (1125765218304 168 40960) itemoff 15580 itemsize 37
> [ 104.567749] extent refs 1 gen 12770 flags 1
> [ 104.581835] shared data backref parent 223898468352 count 1
> [ 104.600110] item 19 key (1125765726208 168 40960) itemoff 15543 itemsize 37
> [ 104.622290] extent refs 1 gen 12770 flags 1
> [ 104.636377] shared data backref parent 223898468352 count 1
> [ 104.654657] item 20 key (1125766475776 168 40960) itemoff 15506 itemsize 37
> [ 104.676833] extent refs 1 gen 12770 flags 1
> [ 104.690918] shared data backref parent 223898468352 count 1
> [ 104.709197] item 21 key (1125766668288 168 40960) itemoff 15469 itemsize 37
> [ 104.731375] extent refs 1 gen 12770 flags 1
> [ 104.745463] shared data backref parent 223898468352 count 1
> [ 104.763745] item 22 key (1125767483392 168 40960) itemoff 15432 itemsize 37
> [ 104.785917] extent refs 1 gen 12770 flags 1
> [ 104.800064] shared data backref parent 223898468352 count 1
> [ 104.818330] item 23 key (1125767868416 168 40960) itemoff 15395 itemsize 37
> [ 104.840515] extent refs 1 gen 12770 flags 1
> [ 104.854637] shared data backref parent 223898468352 count 1
> [ 104.872905] item 24 key (1125771141120 168 40960) itemoff 15358 itemsize 37
> [ 104.895080] extent refs 1 gen 12770 flags 1
> [ 104.909198] shared data backref parent 223898468352 count 1
> [ 104.927474] item 25 key (1125771886592 168 40960) itemoff 15321 itemsize 37
> [ 104.949648] extent refs 1 gen 12770 flags 1
> [ 104.963760] shared data backref parent 223898468352 count 1
> [ 104.982034] item 26 key (1125773021184 168 40960) itemoff 15284 itemsize 37
> [ 105.004216] extent refs 1 gen 12770 flags 1
> [ 105.018331] shared data backref parent 223898468352 count 1
> [ 105.036604] item 27 key (1125774393344 168 40960) itemoff 15247 itemsize 37
> [ 105.058784] extent refs 1 gen 12770 flags 1
> [ 105.072898] shared data backref parent 223898468352 count 1
> [ 105.091171] item 28 key (1125779865600 168 40960) itemoff 15210 itemsize 37
> [ 105.113350] extent refs 1 gen 12770 flags 1
> [ 105.127464] shared data backref parent 223898468352 count 1
> [ 105.145740] item 29 key (1125781073920 168 40960) itemoff 15173 itemsize 37
> [ 105.167919] extent refs 1 gen 12770 flags 1
> [ 105.182036] shared data backref parent 223898468352 count 1
> [ 105.200307] item 30 key (1125782454272 168 40960) itemoff 15136 itemsize 37
> [ 105.222488] extent refs 1 gen 12770 flags 1
> [ 105.236605] shared data backref parent 223898468352 count 1
> [ 105.254879] item 31 key (1125785354240 168 40960) itemoff 15099 itemsize 37
> [ 105.277055] extent refs 1 gen 12770 flags 1
> [ 105.291166] shared data backref parent 223898468352 count 1
> [ 105.309445] item 32 key (1125785784320 168 40960) itemoff 15062 itemsize 37
> [ 105.331622] extent refs 1 gen 12770 flags 1
> [ 105.345737] shared data backref parent 223898468352 count 1
> [ 105.364011] item 33 key (1125786988544 168 40960) itemoff 15025 itemsize 37
> [ 105.386193] extent refs 1 gen 12770 flags 1
> [ 105.400302] shared data backref parent 223898468352 count 1
> [ 105.418585] item 34 key (1125790654464 168 40960) itemoff 14988 itemsize 37
> [ 105.440785] extent refs 1 gen 12770 flags 1
> [ 105.454897] shared data backref parent 223898468352 count 1
> [ 105.473181] item 35 key (1125791793152 168 40960) itemoff 14951 itemsize 37
> [ 105.495381] extent refs 1 gen 12770 flags 1
> [ 105.509493] shared data backref parent 223898468352 count 1
> [ 105.527774] item 36 key (1125793554432 168 40960) itemoff 14914 itemsize 37
> [ 105.550052] extent refs 1 gen 12770 flags 1
> [ 105.564169] shared data backref parent 223898468352 count 1
> [ 105.582441] item 37 key (1125794611200 168 40960) itemoff 14877 itemsize 37
> [ 105.604643] extent refs 1 gen 12770 flags 1
> [ 105.618762] shared data backref parent 223898468352 count 1
> [ 105.637094] item 38 key (1125797896192 168 40960) itemoff 14840 itemsize 37
> [ 105.659379] extent refs 1 gen 12770 flags 1
> [ 105.673483] shared data backref parent 223898468352 count 1
> [ 105.691765] item 39 key (1125801062400 168 40960) itemoff 14803 itemsize 37
> [ 105.714015] extent refs 1 gen 12770 flags 1
> [ 105.728135] shared data backref parent 223898468352 count 1
> [ 105.746428] item 40 key (1125802123264 168 40960) itemoff 14766 itemsize 37
> [ 105.768688] extent refs 1 gen 12770 flags 1
> [ 105.782796] shared data backref parent 223898468352 count 1
> [ 105.801145] item 41 key (1125803565056 168 40960) itemoff 14729 itemsize 37
> [ 105.823332] extent refs 1 gen 12770 flags 1
> [ 105.837445] shared data backref parent 223898468352 count 1
> [ 105.855720] item 42 key (1125804216320 168 40960) itemoff 14692 itemsize 37
> [ 105.877921] extent refs 1 gen 12770 flags 1
> [ 105.892040] shared data backref parent 223898468352 count 1
> [ 105.910310] item 43 key (1125804675072 168 40960) itemoff 14655 itemsize 37
> [ 105.932487] extent refs 1 gen 12770 flags 1
> [ 105.946599] shared data backref parent 223898468352 count 1
> [ 105.964877] item 44 key (1125804867584 168 40960) itemoff 14618 itemsize 37
> [ 105.987054] extent refs 1 gen 12770 flags 1
> [ 106.001172] shared data backref parent 223898468352 count 1
> [ 106.019443] item 45 key (1125806006272 168 40960) itemoff 14581 itemsize 37
> [ 106.041617] extent refs 1 gen 12770 flags 1
> [ 106.055743] shared data backref parent 223898468352 count 1
> [ 106.074014] item 46 key (1125807456256 168 40960) itemoff 14544 itemsize 37
> [ 106.096229] extent refs 1 gen 12770 flags 1
> [ 106.110327] shared data backref parent 223898468352 count 1
> [ 106.128616] item 47 key (1125810675712 168 40960) itemoff 14507 itemsize 37
> [ 106.150814] extent refs 1 gen 12770 flags 1
> [ 106.164927] shared data backref parent 223898468352 count 1
> [ 106.183256] item 48 key (1125812047872 168 40960) itemoff 14470 itemsize 37
> [ 106.205454] extent refs 1 gen 12770 flags 1
> [ 106.219629] shared data backref parent 223898468352 count 1
> [ 106.237898] item 49 key (1125813186560 168 40960) itemoff 14433 itemsize 37
> [ 106.260080] extent refs 1 gen 12770 flags 1
> [ 106.274206] shared data backref parent 223898468352 count 1
> [ 106.292494] item 50 key (1125814251520 168 40960) itemoff 14396 itemsize 37
> [ 106.314695] extent refs 1 gen 12770 flags 1
> (...)
> [ 113.651650] item 185 key (1126124646400 168 45056) itemoff 9401 itemsize 37
> [ 113.673859] extent refs 1 gen 12537 flags 1
> [ 113.687970] shared data backref parent 63894044672 count 1
> [ 113.706013] BTRFS error (device dm-5): unable to find ref byte nr 1125950996480 parent 63893716992 root 0 owner 7977109 offset 0
> [ 113.788672] ------------[ cut here ]------------
> [ 113.803626] WARNING: CPU: 2 PID: 3105 at fs/btrfs/free-space-cache.c:1183 tree_insert_offset+0x81/0xc5()
> [ 113.833793] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
> [ 114.125608] CPU: 2 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
> [ 114.156659] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
> [ 114.185373] 0000000000000000 ffff880213f63928 ffffffff8160a06d 0000000000000000
> [ 114.209071] ffff880213f63960 ffffffff81050025 ffffffff8125e81a 00000000ffffffef
> [ 114.232737] ffff88020ce2a1b0 0000000000000000 ffff8802125f8800 ffff880213f63970
> [ 114.256422] Call Trace:
> [ 114.265075] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
> [ 114.281817] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
> [ 114.301150] [<ffffffff8125e81a>] ? tree_insert_offset+0x81/0xc5
> [ 114.320518] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
> [ 114.339318] [<ffffffff8125e81a>] tree_insert_offset+0x81/0xc5
> [ 114.358249] [<ffffffff8125e88a>] link_free_space+0x2c/0x41
> [ 114.376238] [<ffffffff81260f1b>] __load_free_space_cache+0x304/0x56c
> [ 114.396812] [<ffffffff81261269>] load_free_space_cache+0xe6/0x199
> [ 114.416619] [<ffffffff81141648>] ? kmem_cache_alloc_trace+0x11b/0x12b
> [ 114.437456] [<ffffffff81216354>] cache_block_group+0x1b2/0x33d
> [ 114.456465] [<ffffffff81085116>] ? finish_wait+0x65/0x65
> [ 114.473916] [<ffffffff81216c49>] update_block_group.isra.38+0xaa/0x1fe
> [ 114.495009] [<ffffffff81217449>] __btrfs_free_extent+0x6ac/0x712
> [ 114.514526] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
> [ 114.533262] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
> [ 114.553842] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
> [ 114.574789] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
> [ 114.595070] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
> [ 114.615064] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
> [ 114.632726] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
> [ 114.654014] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
> [ 114.673530] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
> [ 114.696360] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
> [ 114.714783] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
> [ 114.733435] [<ffffffff8106bc62>] kthread+0xae/0xb6
> [ 114.749169] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 114.767775] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
> [ 114.785052] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 114.803608] ---[ end trace a51c57d8cb84bfb5 ]---
> [ 114.818501] BTRFS error (device dm-5): Duplicate entries in free space cache, dumping
> [ 114.843818] BTRFS error (device dm-5): failed to load free space cache for block group 1125310791680
> [ 114.873135] ------------[ cut here ]------------
> [ 114.888456] WARNING: CPU: 0 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
> [ 114.917736] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
> [ 115.207261] CPU: 0 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
> [ 115.238090] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
> [ 115.266591] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
> [ 115.290066] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
> [ 115.313567] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
> [ 115.337073] Call Trace:
> [ 115.345558] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
> [ 115.362133] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
> [ 115.381302] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
> [ 115.401258] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
> [ 115.419913] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
> [ 115.439328] [<ffffffff81610200>] ? retint_signal+0x8c/0x8c
> [ 115.457198] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
> [ 115.475833] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
> [ 115.496285] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
> [ 115.516999] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
> [ 115.537191] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
> [ 115.557128] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
> [ 115.574739] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
> [ 115.595976] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
> [ 115.615413] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
> [ 115.638217] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
> [ 115.656608] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
> [ 115.675260] [<ffffffff8106bc62>] kthread+0xae/0xb6
> [ 115.691028] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 115.709660] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
> [ 115.726989] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 115.745622] ---[ end trace a51c57d8cb84bfb6 ]---
> [ 115.760616] BTRFS info (device dm-5): leaf 264060928 total ptrs 185 free space 4813
> [ 115.784754] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
> [ 115.806782] extent refs 1 gen 12770 flags 1
> [ 115.821011] shared data backref parent 223898337280 count 1
> [ 115.839401] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
> [ 115.861406] extent refs 1 gen 12770 flags 1
> [ 115.875593] shared data backref parent 223898337280 count 1
> [ 115.893927] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
> [ 115.915880] extent refs 1 gen 12770 flags 1
> [ 115.929977] shared data backref parent 223898337280 count 1
> (...)
> [ 125.804965] item 184 key (1126124646400 168 45056) itemoff 9438 itemsize 37
> [ 125.827174] extent refs 1 gen 12537 flags 1
> [ 125.841290] shared data backref parent 63894044672 count 1
> [ 125.859328] BTRFS error (device dm-5): unable to find ref byte nr 1125993029632 parent 63893716992 root 0 owner 7976978 offset 0
> [ 125.896481] ------------[ cut here ]------------
> [ 125.912031] WARNING: CPU: 2 PID: 3105 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
> [ 125.941657] Modules linked in: aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse snd_hda_codec_realtek microcode snd_hda_codec_generic snd_cmipci kvm_intel gameport kvm snd_hda_intel snd_hda_codec snd_opl3_lib eeepc_wmi snd_mpu401_uart snd_seq_midi snd_seq_midi_event snd_seq asus_wmi tpm_infineon snd_hwdep sparse_keymap snd_pcm snd_rawmidi coretemp rfkill battery snd_timer snd_seq_device intel_rapl rc_ati_x10 snd tpm_tis ati_remote processor x86_pkg_temp_thermal wmi tpm asix pl2303 lpc_ich i2c_i801 soundcore libphy parport_pc intel_powerclamp ftdi_sio keyspan parport usbnet pcspkr evdev rc_core xhci_hcd usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci ehci_hcd crct10dif_pclmul sata_sil24 usbcore ghash_clmulni_intel cryptd fan thermal r8169 mii usb_common sata_mv
> [ 126.233519] CPU: 2 PID: 3105 Comm: btrfs-cleaner Tainted: G W 3.14.0-amd64-i915-preempt-20140216 #2
> [ 126.264560] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
> [ 126.293279] 0000000000000000 ffff880213f63b38 ffffffff8160a06d 0000000000000000
> [ 126.316958] ffff880213f63b70 ffffffff81050025 ffffffff812170f6 ffff880211a58f10
> [ 126.340621] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff880213f63b80
> [ 126.364309] Call Trace:
> [ 126.372966] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
> [ 126.389719] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
> [ 126.409086] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
> [ 126.429198] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
> [ 126.447995] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
> [ 126.467566] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
> [ 126.486341] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
> [ 126.507009] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
> [ 126.527865] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
> [ 126.548231] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
> [ 126.568283] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
> [ 126.586109] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
> [ 126.607470] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
> [ 126.626998] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
> [ 126.649898] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
> [ 126.668364] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
> [ 126.687087] [<ffffffff8106bc62>] kthread+0xae/0xb6
> [ 126.702931] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 126.721652] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
> [ 126.739054] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
> [ 126.757757] ---[ end trace a51c57d8cb84bfb7 ]---
> [ 126.772784] BTRFS info (device dm-5): leaf 264060928 total ptrs 184 free space 4875
> [ 126.796952] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
> [ 126.819017] extent refs 1 gen 12770 flags 1
> [ 126.833247] shared data backref parent 223898337280 count 1
> [ 126.851622] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
> [ 126.873650] extent refs 1 gen 12770 flags 1
> [ 126.887857] shared data backref parent 223898337280 count 1
> [ 126.906246] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
> [ 126.928237] extent refs 1 gen 12770 flags 1
> [ 126.942386] shared data backref parent 223898337280 count 1
> (...)
>
>
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
> .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-08 22:09 ` Marc MERLIN
@ 2014-04-08 23:49 ` Chris Mason
2014-04-09 4:31 ` Marc MERLIN
0 siblings, 1 reply; 10+ messages in thread
From: Chris Mason @ 2014-04-08 23:49 UTC (permalink / raw)
To: Marc MERLIN, linux-btrfs
On 04/08/2014 06:09 PM, Marc MERLIN wrote:
> I forgot to add that while I'm not sure if anyone ended up looking at the
> last image I made regarding
> https://bugzilla.kernel.org/show_bug.cgi?id=72801
>
> I can generate a an image of that filesystem if that helps, or try other
> commands which hopefully won't crash my running server :)
> (filesystem is almost 2TB, so the image will again be big)
>
Hi Marc,
So from the messages it looks like your space cache is corrupted. Lets
start with clearing the space cache and running fsck and seeing exactly
what is wrong.
An image will definitely help if you have a pipe big enough to upload it.
-chris
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-08 23:49 ` Chris Mason
@ 2014-04-09 4:31 ` Marc MERLIN
2014-04-09 5:31 ` Marc MERLIN
0 siblings, 1 reply; 10+ messages in thread
From: Marc MERLIN @ 2014-04-09 4:31 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
On Tue, Apr 08, 2014 at 07:49:14PM -0400, Chris Mason wrote:
>
>
> On 04/08/2014 06:09 PM, Marc MERLIN wrote:
> >I forgot to add that while I'm not sure if anyone ended up looking at the
> >last image I made regarding
> >https://bugzilla.kernel.org/show_bug.cgi?id=72801
> >
> >I can generate a an image of that filesystem if that helps, or try other
> >commands which hopefully won't crash my running server :)
> >(filesystem is almost 2TB, so the image will again be big)
> >
>
> Hi Marc,
>
> So from the messages it looks like your space cache is corrupted. Lets
> start with clearing the space cache and running fsck and seeing exactly
> what is wrong.
gargamel:~# mount -o clear_cache /dev/dm-4 /mnt/mnt
[48132.661274] BTRFS: device label btrfs_raid0 devid 1 transid 50567 /dev/mapper/raid0d1
[48132.703063] BTRFS info (device dm-5): force clearing of disk cache
[48132.724780] BTRFS info (device dm-5): disk space caching is enabled
Mmmh, I've never had much luck with btrfsck
> An image will definitely help if you have a pipe big enough to upload it.
Mmmh, I guess I should have taken this before the clear_cache mount, but
even after that, I still got the crash, so let me take an image first
Then, what fsck options do you recommend?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 4:31 ` Marc MERLIN
@ 2014-04-09 5:31 ` Marc MERLIN
2014-04-09 15:42 ` Marc MERLIN
0 siblings, 1 reply; 10+ messages in thread
From: Marc MERLIN @ 2014-04-09 5:31 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
On Tue, Apr 08, 2014 at 09:31:25PM -0700, Marc MERLIN wrote:
> On Tue, Apr 08, 2014 at 07:49:14PM -0400, Chris Mason wrote:
> >
> >
> > On 04/08/2014 06:09 PM, Marc MERLIN wrote:
> > >I forgot to add that while I'm not sure if anyone ended up looking at the
> > >last image I made regarding
> > >https://bugzilla.kernel.org/show_bug.cgi?id=72801
> > >
> > >I can generate a an image of that filesystem if that helps, or try other
> > >commands which hopefully won't crash my running server :)
> > >(filesystem is almost 2TB, so the image will again be big)
> > >
> >
> > Hi Marc,
> >
> > So from the messages it looks like your space cache is corrupted. Lets
> > start with clearing the space cache and running fsck and seeing exactly
> > what is wrong.
>
> gargamel:~# mount -o clear_cache /dev/dm-4 /mnt/mnt
> [48132.661274] BTRFS: device label btrfs_raid0 devid 1 transid 50567 /dev/mapper/raid0d1
> [48132.703063] BTRFS info (device dm-5): force clearing of disk cache
> [48132.724780] BTRFS info (device dm-5): disk space caching is enabled
>
> Mmmh, I've never had much luck with btrfsck
>
> > An image will definitely help if you have a pipe big enough to upload it.
>
> Mmmh, I guess I should have taken this before the clear_cache mount, but
> even after that, I still got the crash, so let me take an image first
>
> Then, what fsck options do you recommend?
Here is the image (1.5GB):
http://marc.merlins.org/tmp/btrfs-raid0-image
Please let me know when I can remove it.
It was taken right after I tried a mount again, and soon after got:
[48297.721718] BTRFS: device label btrfs_raid0 devid 2 transid 50567 /dev/mapper/raid0d2
[48317.369527] ------------[ cut here ]------------
[48317.385559] WARNING: CPU: 3 PID: 13019 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[48317.415941] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables tun ppdev lp autofs4 binfmt_misc kl5kusb105 deflate ctr twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_generic twofish_common camellia_x86_64 camellia_generic serpent_sse2_x86_64 serpent_avx_x86_64 glue_helper lrw serpent_generic blowfish_x86_64 blowfish_generic blowfish_common cast5_avx_x86_64 ablk_helper cast5_generic cast_common des_generic cmac xcbc rmd160 sha512_ssse3 sha512_generic ftdi_sio crypto_null keyspan af_key xfrm_algo dm_mirror dm_region_hash dm_log nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc ipt_REJECT xt_conntrack xt_nat xt_tcpudp xt_LOG iptable_mangle iptable_filter aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse eeepc_wmi microcode asus_wmi kvm_intel sparse_keymap snd_hda_codec_realtek rfkill snd_hda_codec_generic kvm snd_cmipci gameport snd_hda_intel snd_opl3_lib snd_mpu401_uart snd_seq_midi snd_hda_codec snd_seq_midi_event snd_seq snd_rawmidi snd_hwdep asix snd_pcm snd_timer battery tpm_infineon snd_seq_device coretemp libphy processor wmi pl2303 pcspkr snd parport_pc intel_rapl i2c_i801 usbnet rc_ati_x10 parport lpc_ich xhci_hcd tpm_tis ati_remote x86_pkg_temp_thermal intel_powerclamp evdev tpm rc_core soundcore usbserial ezusb xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core ehci_pci ehci_hcd crc32_pclmul crc32c_intel sata_sil24 crct10dif_pclmul ghash_clmulni_intel thermal cryptd fan r8169 mii usbcore usb_common sata_mv [last unloaded: kl5kusb105]
[48317.913132] CPU: 3 PID: 13019 Comm: btrfs-cleaner Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[48317.943303] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[48317.973797] 0000000000000000 ffff8801bbfd5b38 ffffffff8160a06d 0000000000000000
[48317.998522] ffff8801bbfd5b70 ffffffff81050025 ffffffff812170f6 ffff880211af3970
[48318.022993] 00000000fffffffe 0000000ee05cc000 0000000000000000 ffff8801bbfd5b80
[48318.047511] Call Trace:
[48318.057221] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[48318.074556] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[48318.095323] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[48318.116854] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[48318.136119] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[48318.156327] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[48318.175732] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[48318.196996] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[48318.218542] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[48318.239536] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[48318.260623] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[48318.279115] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[48318.301178] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[48318.321381] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[48318.344943] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[48318.363935] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[48318.383334] [<ffffffff8106bc62>] kthread+0xae/0xb6
[48318.399802] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[48318.419029] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[48318.436931] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[48318.456118] ---[ end trace 88f99f6aed83f7e4 ]---
[48318.471916] BTRFS info (device dm-5): leaf 300367872 total ptrs 186 free space 4751
[48318.496465] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[48318.518933] extent refs 1 gen 12770 flags 1
[48318.533698] shared data backref parent 223898337280 count 1
[48318.552449] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[48318.574986] extent refs 1 gen 12770 flags 1
[48318.589596] shared data backref parent 223898337280 count 1
[48318.608403] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
[48318.630736] extent refs 1 gen 12770 flags 1
[48318.645735] shared data backref parent 223898337280 count 1
[48318.664534] item 3 key (1125738274816 168 40960) itemoff 16135 itemsize 37
(...)
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 5:31 ` Marc MERLIN
@ 2014-04-09 15:42 ` Marc MERLIN
2014-04-09 15:46 ` Chris Mason
0 siblings, 1 reply; 10+ messages in thread
From: Marc MERLIN @ 2014-04-09 15:42 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
On Tue, Apr 08, 2014 at 10:31:39PM -0700, Marc MERLIN wrote:
> On Tue, Apr 08, 2014 at 09:31:25PM -0700, Marc MERLIN wrote:
> > On Tue, Apr 08, 2014 at 07:49:14PM -0400, Chris Mason wrote:
> > >
> > >
> > > On 04/08/2014 06:09 PM, Marc MERLIN wrote:
> > > >I forgot to add that while I'm not sure if anyone ended up looking at the
> > > >last image I made regarding
> > > >https://bugzilla.kernel.org/show_bug.cgi?id=72801
> > > >
> > > >I can generate a an image of that filesystem if that helps, or try other
> > > >commands which hopefully won't crash my running server :)
> > > >(filesystem is almost 2TB, so the image will again be big)
> > > >
> > >
> > > Hi Marc,
> > >
> > > So from the messages it looks like your space cache is corrupted. Lets
> > > start with clearing the space cache and running fsck and seeing exactly
> > > what is wrong.
> >
> > gargamel:~# mount -o clear_cache /dev/dm-4 /mnt/mnt
> > [48132.661274] BTRFS: device label btrfs_raid0 devid 1 transid 50567 /dev/mapper/raid0d1
> > [48132.703063] BTRFS info (device dm-5): force clearing of disk cache
> > [48132.724780] BTRFS info (device dm-5): disk space caching is enabled
So, I tried again this morning, mounted with clear_cache, let the clearer
process work a bit:
root 25187 0.0 0.0 0 0 ? S 07:56 0:00 [btrfs-freespace]
but even though I did not have the FS mounted, after just one minute, the
kernel went into that death loop again.
Then (2nd log below), I tried mounting with -o clear_cache,nospace_cache and
had the same problem too.
I'll wait on your next suggestion, with maybe how you'd like me to run btrfsck
Thanks,
Marc
[37652.548583] BTRFS: device label btrfs_raid0 devid 2 transid 50571 /dev/mapper/raid0d2
[37652.757397] BTRFS info (device dm-5): force clearing of disk cache
[37652.779375] BTRFS info (device dm-5): disk space caching is enabled
[37842.582194] WARNING: CPU: 2 PID: 25231 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[37842.613790] Modules linked in: udp_diag tcp_diag inet_diag ip6table_filter ip6_tables ebtable_nat ebtables tun ppdev lp autofs4 binfmt_misc kl5kusb105 deflate ctr twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_generic twofish_common camellia_x86_64 camellia_generic serpent_sse2_x86_64 serpent_avx_x86_64 glue_helper lrw serpent_generic blowfish_x86_64 blowfish_generic blowfish_common cast5_avx_x86_64 ablk_helper cast5_generic cast_common des_generic cmac xcbc rmd160 sha512_ssse3 sha512_generic ftdi_sio crypto_null keyspan af_key xfrm_algo dm_mirror dm_region_hash dm_log nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc ipt_REJECT xt_conntrack xt_nat xt_tcpudp xt_LOG iptable_mangle iptable_filter aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse microcode snd_hda_codec_realtek snd_cmipci snd_hda_codec_generic kvm_intel gameport kvm eeepc_wmi snd_hda_intel asus_wmi sparse_keymap snd_opl3_lib snd_mpu401_uart snd_seq_midi snd_hda_codec rfkill snd_seq_midi_event snd_seq snd_rawmidi snd_hwdep snd_pcm snd_timer tpm_infineon battery snd_seq_device wmi coretemp processor rc_ati_x10 pl2303 pcspkr snd intel_rapl asix tpm_tis x86_pkg_temp_thermal parport_pc ati_remote libphy ezusb soundcore i2c_i801 parport tpm intel_powerclamp rc_core lpc_ich xhci_hcd usbnet usbserial evdev xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core crc32_pclmul crc32c_intel ehci_pci sata_sil24 r8169 ehci_hcd thermal mii crct10dif_pclmul fan sata_mv ghash_clmulni_intel cryptd usbcore usb_common [last unloaded: kl5kusb105]
[37843.113872] CPU: 2 PID: 25231 Comm: btrfs-cleaner Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[37843.143161] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[37843.143161] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[37843.172720] 0000000000000000 ffff880076d15b38 ffffffff8160a06d 0000000000000000
[37843.197245] ffff880076d15b70 ffffffff81050025 ffffffff812170f6 ffff8801bbdf3580
[37843.221785] 00000000fffffffe 0000000ee0610000 0000000000000000 ffff880076d15b80
[37843.246199] Call Trace:
[37843.255694] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[37843.273251] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[37843.293448] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[37843.314212] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[37843.334376] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[37843.354692] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[37843.374076] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[37843.395273] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[37843.417102] [<ffffffff8113fcf3>] ? __cache_free.isra.39+0x1b4/0x1c3
[37843.437969] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[37843.458651] [<ffffffff8121ac3a>] ? walk_up_tree+0x72/0xf9
[37843.476870] [<ffffffff8122af08>] btrfs_should_end_transaction+0x52/0x5b
[37843.498825] [<ffffffff8121cba9>] btrfs_drop_snapshot+0x36f/0x610
[37843.518879] [<ffffffff8122c73d>] btrfs_clean_one_deleted_snapshot+0x103/0x10f
[37843.542271] [<ffffffff81224f09>] cleaner_kthread+0x103/0x136
[37843.561253] [<ffffffff81224e06>] ? btrfs_alloc_root+0x26/0x26
[37843.580455] [<ffffffff8106bc62>] kthread+0xae/0xb6
[37843.597117] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[37843.616153] [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[37843.633875] [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[37843.652855] ---[ end trace 12ad5103b5a879ce ]---
[37843.668162] BTRFS info (device dm-5): leaf 307314688 total ptrs 134 free space 7975
[37843.692552] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[37843.714875] extent refs 1 gen 12770 flags 1
[37843.729365] shared data backref parent 223898337280 count 1
[37843.748012] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[37843.770332] extent refs 1 gen 12770 flags 1
[37843.784809] shared data backref parent 223898337280 count 1
For fun, I tried this:
gargamel:~# mount -o clear_cache,nospace_cache LABEL=btrfs_raid0 /mnt/btrfs_raid0
and then looked at the mounted filesystem. It very quickly crashed the kernel with a looping of:
gargamel login: [ 505.952090] BTRFS: device label btrfs_raid0 devid 2 transid 50573 /dev/mapper/raid0d2
[ 505.990811] BTRFS info (device dm-5): force clearing of disk cache
[ 506.009489] BTRFS info (device dm-5): disabling disk space caching
[ 541.494536] ------------[ cut here ]------------
[ 541.508444] WARNING: CPU: 2 PID: 16979 at fs/btrfs/extent-tree.c:5748 __btrfs_free_extent+0x359/0x712()
[ 541.536672] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables tun ppdev lp autofs4 binfmt_misc kl5kusb105 ftdi_sio keyspan deflate ctr twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_generic twofish_common camellia_x86_64 camellia_generic serpent_sse2_x86_64 serpent_avx_x86_64 glue_helper lrw serpent_generic blowfish_x86_64 blowfish_generic blowfish_common cast5_avx_x86_64 ablk_helper cast5_generic cast_common des_generic cmac xcbc rmd160 sha512_ssse3 sha512_generic crypto_null af_key xfrm_algo dm_mirror dm_region_hash dm_log nfsd nfs_acl auth_rpcgss nfs fscache lockd sunrpc ipt_REJECT xt_conntrack xt_nat xt_tcpudp xt_LOG iptable_mangle iptable_filter aes_x86_64 lm85 hwmon_vid dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat x_tables nf_conntrack sg st snd_pcm_oss snd_mixer_oss fuse microcode snd_hda_codec_realtek snd_hda_codec_generic eeepc_wmi kvm_intel asus_wmi snd_cmipci kvm gameport sparse_keymap rfkill snd_hda_intel snd_opl3_lib snd_mpu401_uart snd_hda_codec coretemp snd_seq_midi snd_seq_midi_event rc_ati_x10 snd_hwdep snd_seq asix snd_pcm ati_remote intel_rapl battery pcspkr evdev wmi tpm_infineon snd_rawmidi processor libphy snd_timer snd_seq_device lpc_ich parport_pc rc_core i2c_i801 pl2303 parport usbnet x86_pkg_temp_thermal intel_powerclamp ezusb xhci_hcd tpm_tis usbserial snd tpm soundcore xts gf128mul dm_crypt dm_mod raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx e1000e ptp pps_core ehci_pci ehci_hcd sata_sil24 crc32_pclmul crc32c_intel sata_mv crct10dif_pclmul usbcore thermal r8169 mii fan ghash_clmulni_intel cryptd usb_common [last unloaded: kl5kusb105]
[ 541.995046] CPU: 2 PID: 16979 Comm: find Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[ 542.019859] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[ 542.047285] 0000000000000000 ffff8800a44fdb58 ffffffff8160a06d 0000000000000000
[ 542.069689] ffff8800a44fdb90 ffffffff81050025 ffffffff812170f6 ffff88020787e7c0
[ 542.092095] 00000000fffffffe 0000000ee0610000 0000000000000000 ffff8800a44fdba0
[ 542.114496] Call Trace:
[ 542.121862] [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[ 542.137306] [<ffffffff81050025>] warn_slowpath_common+0x7f/0x98
[ 542.155348] [<ffffffff812170f6>] ? __btrfs_free_extent+0x359/0x712
[ 542.174188] [<ffffffff810500ec>] warn_slowpath_null+0x1a/0x1c
[ 542.191733] [<ffffffff812170f6>] __btrfs_free_extent+0x359/0x712
[ 542.210059] [<ffffffff8107c300>] ? account_entity_enqueue+0xd/0x8b
[ 542.228885] [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[ 542.246418] [<ffffffff8126518b>] ? btrfs_check_delayed_seq+0x84/0x90
[ 542.265754] [<ffffffff8121c262>] __btrfs_run_delayed_refs+0xa94/0xbdf
[ 542.286760] [<ffffffff81218c3d>] ? reserve_metadata_bytes+0x1b2/0x723
[ 542.307720] [<ffffffff8121df46>] btrfs_run_delayed_refs+0x81/0x18f
[ 542.327888] [<ffffffff8122c13f>] __btrfs_end_transaction+0xe1/0x2c6
[ 542.348690] [<ffffffff8122c334>] btrfs_end_transaction+0x10/0x12
[ 542.368326] [<ffffffff812306c1>] btrfs_dirty_inode+0x8f/0xac
[ 542.386876] [<ffffffff8123075c>] btrfs_update_time+0x7e/0x8c
[ 542.405559] [<ffffffff81169daf>] update_time+0x25/0xb4
[ 542.422629] [<ffffffff8116a528>] touch_atime+0xe8/0x121
[ 542.440005] [<ffffffff81164148>] iterate_dir+0x84/0xa6
[ 542.457091] [<ffffffff8119516b>] compat_sys_getdents64+0x7d/0xd9
[ 542.476786] [<ffffffff81193acb>] ? compat_filldir+0xf8/0xf8
[ 542.495114] [<ffffffff810a13f3>] ? current_kernel_time+0xe/0x32
[ 542.514653] [<ffffffff8161812c>] sysenter_dispatch+0x7/0x21
[ 542.533315] ---[ end trace ada256a83cb53c26 ]---
[ 542.548714] BTRFS info (device dm-5): leaf 123420672 total ptrs 134 free space 7975
[ 542.573374] item 0 key (1125730791424 168 40960) itemoff 16246 itemsize 37
[ 542.595682] extent refs 1 gen 12770 flags 1
[ 542.610165] shared data backref parent 223898337280 count 1
[ 542.628775] item 1 key (1125732311040 168 40960) itemoff 16209 itemsize 37
[ 542.651009] extent refs 1 gen 12770 flags 1
[ 542.665505] shared data backref parent 223898337280 count 1
[ 542.684047] item 2 key (1125733752832 168 40960) itemoff 16172 itemsize 37
[ 542.706425] extent refs 1 gen 12770 flags 1
[ 542.720730] shared data backref parent 223898337280 count 1
[ 542.739209] item 3 key (1125738274816 168 40960) itemoff 16135 itemsize 37
[ 542.761287] extent refs 1 gen 12770 flags 1
[ 542.775553] shared data backref parent 223898337280 count 1
[ 542.793946] item 4 key (1125740027904 168 40960) itemoff 16098 itemsize 37
[ 542.815973] extent refs 1 gen 12770 flags 1
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 15:42 ` Marc MERLIN
@ 2014-04-09 15:46 ` Chris Mason
2014-04-09 16:51 ` Marc MERLIN
0 siblings, 1 reply; 10+ messages in thread
From: Chris Mason @ 2014-04-09 15:46 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On 04/09/2014 11:42 AM, Marc MERLIN wrote:
> On Tue, Apr 08, 2014 at 10:31:39PM -0700, Marc MERLIN wrote:
>> On Tue, Apr 08, 2014 at 09:31:25PM -0700, Marc MERLIN wrote:
>>> On Tue, Apr 08, 2014 at 07:49:14PM -0400, Chris Mason wrote:
>>>>
>>>>
>>>> On 04/08/2014 06:09 PM, Marc MERLIN wrote:
>>>>> I forgot to add that while I'm not sure if anyone ended up looking at the
>>>>> last image I made regarding
>>>>> https://urldefense.proofpoint.com/v1/url?u=https://bugzilla.kernel.org/show_bug.cgi?id%3D72801&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2BQyA%3D%3D%0A&m=dQHW2ddzMXNlRthusH4o6nZVMltGycqZ8zO5AgwPphE%3D%0A&s=783fd1cd39566becdfb62904889cd7459c81dd2793d19538b2ab093bb8d06f88
>>>>>
>>>>> I can generate a an image of that filesystem if that helps, or try other
>>>>> commands which hopefully won't crash my running server :)
>>>>> (filesystem is almost 2TB, so the image will again be big)
>>>>>
>>>>
>>>> Hi Marc,
>>>>
>>>> So from the messages it looks like your space cache is corrupted. Lets
>>>> start with clearing the space cache and running fsck and seeing exactly
>>>> what is wrong.
>>>
>>> gargamel:~# mount -o clear_cache /dev/dm-4 /mnt/mnt
>>> [48132.661274] BTRFS: device label btrfs_raid0 devid 1 transid 50567 /dev/mapper/raid0d1
>>> [48132.703063] BTRFS info (device dm-5): force clearing of disk cache
>>> [48132.724780] BTRFS info (device dm-5): disk space caching is enabled
>
> So, I tried again this morning, mounted with clear_cache, let the clearer
> process work a bit:
> root 25187 0.0 0.0 0 0 ? S 07:56 0:00 [btrfs-freespace]
> but even though I did not have the FS mounted, after just one minute, the
> kernel went into that death loop again.
>
> Then (2nd log below), I tried mounting with -o clear_cache,nospace_cache and
> had the same problem too.
>
> I'll wait on your next suggestion, with maybe how you'd like me to run btrfsck
Downloading the image now. I'd just run a readonly btrfsck /dev/xxx
-chris
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 15:46 ` Chris Mason
@ 2014-04-09 16:51 ` Marc MERLIN
2014-04-09 18:54 ` Chris Mason
2014-04-09 19:24 ` Duncan
0 siblings, 2 replies; 10+ messages in thread
From: Marc MERLIN @ 2014-04-09 16:51 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
On Wed, Apr 09, 2014 at 11:46:13AM -0400, Chris Mason wrote:
> Downloading the image now. I'd just run a readonly btrfsck /dev/xxx
http://marc.merlins.org/tmp/btrfs-raid0-image-fsck.txt (6MB)
I admit to not knowing how to read that output, I've only ever seen
thousands of lines of output from it on any filesystem., but hopefully you
know how to grep out expected noise.
But since we're talking about this, is btrfsck ever supposed to return clean
on a clean filesystem?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 16:51 ` Marc MERLIN
@ 2014-04-09 18:54 ` Chris Mason
2014-04-09 19:24 ` Duncan
1 sibling, 0 replies; 10+ messages in thread
From: Chris Mason @ 2014-04-09 18:54 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On 04/09/2014 12:51 PM, Marc MERLIN wrote:
> On Wed, Apr 09, 2014 at 11:46:13AM -0400, Chris Mason wrote:
>> Downloading the image now. I'd just run a readonly btrfsck /dev/xxx
>
> https://urldefense.proofpoint.com/v1/url?u=http://marc.merlins.org/tmp/btrfs-raid0-image-fsck.txt&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2BQyA%3D%3D%0A&m=FvUwVb5mtQKfcHDJf0YzDhzyfAaFwrR9BXQbyCmT0No%3D%0A&s=633624e090ad2c187c1c5d62169bc0a8470ff2560049f83f195642638bff4b91 (6MB)
>
> I admit to not knowing how to read that output, I've only ever seen
> thousands of lines of output from it on any filesystem., but hopefully you
> know how to grep out expected noise.
>
> But since we're talking about this, is btrfsck ever supposed to return clean
> on a clean filesystem?
Looks like I'm getting different results from btrfsck on the image.
Still a ton of corruptions but complaints about different blocks.
Can you please use btrfs-map-logical to dump both copies of block
245432320 and send me the results?
-chris
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 )
2014-04-09 16:51 ` Marc MERLIN
2014-04-09 18:54 ` Chris Mason
@ 2014-04-09 19:24 ` Duncan
1 sibling, 0 replies; 10+ messages in thread
From: Duncan @ 2014-04-09 19:24 UTC (permalink / raw)
To: linux-btrfs
Marc MERLIN posted on Wed, 09 Apr 2014 09:51:34 -0700 as excerpted:
> But since we're talking about this, is btrfsck ever supposed to return
> clean on a clean filesystem?
FWIW, it seems to return clean here, on everything I've tried it on.
But I run relatively small partitions (the biggest is I believe 40 gig,
my media partitions are still reiserfs on spinning rust, while all my
btrfs partitions are on SSD and most are raid1 both data/metadata, with
the exceptions (my normal /boot and the backup /boot on the other ssd in
the pair that's btrfs raid1 for most partitions) being tiny mixed-data/
metadata dup), and keep them pretty clean, running balance and scrub when
needed.
I had seen some scrub recoveries back when I was doing suspend-to-ram and
the system wasn't reliably resuming, I've quit doing that and recently
did a new mkfs.btrfs and restored from backup on the affected filesystems
in ordered to take advantage of newer features like 16k metadata nodes,
so in fact have never personally seen an unclean output of any type from
btrfs check.
Tho I don't run btrfs check regularly as in normal mode it's read-only
anyway, and I know it can make some problems worse instead of fixing them
in repair mode, so my normal idea is why run it and see stuff that might
make me worried if I can't really do much about it, and I prefer balance
and scrub instead if there's problems. But I have run it a few times as
I was curious just what it /would/ output, and everything came up clean
on the filesystems I ran it on.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-04-09 19:24 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-08 15:36 Upgrade to 3.14.0 messed up raid0 array (btrfs cleaner crashes in fs/btrfs/extent-tree.c:5748 and fs/btrfs/free-space-cache.c:1183 ) Marc MERLIN
2014-04-08 22:09 ` Marc MERLIN
2014-04-08 23:49 ` Chris Mason
2014-04-09 4:31 ` Marc MERLIN
2014-04-09 5:31 ` Marc MERLIN
2014-04-09 15:42 ` Marc MERLIN
2014-04-09 15:46 ` Chris Mason
2014-04-09 16:51 ` Marc MERLIN
2014-04-09 18:54 ` Chris Mason
2014-04-09 19:24 ` Duncan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).