help: btrfs bad tree block start

help: btrfs bad tree block start

[Andrea Gelmini]

Hi everybody,
   and thanks a lot for your work.

   I have this problem.
   With latest 3.9 kernel and btrfs-next I was close to full my home.
   The system went read only mode while I was copying in some files.

   Everything was good. I can't write, but I can read and copy last files.

   Then I rebooted, but since that moment I cannot mount.

   The kernel complain is:

[ 1026.226883] device label 16K devid 1 transid 66906 /dev/mapper/glen-home
[ 1026.227452] btrfs: disk space caching is enabled
[ 1026.311650] Btrfs detected SSD devices, enabling SSD mode
[ 1026.311825] btrfs bad tree block start 4262492213953725727 591596257280
[ 1026.312000] btrfs bad tree block start 4262492213953725727 591596257280
[ 1026.312010] btrfs: failed to read log tree
[ 1026.332708] btrfs: open_ctree failed

   If I use btrfsck (Joseph one) I got this:

root@glen:/home/gelma/dev/prg/btrfs_josef# ./btrfsck --repair
/dev/mapper/glen-home enabling repair mode
Check tree block failed, want=591596257280, have=4262492213953725727
Check tree block failed, want=591596257280, have=4262492213953725727
Check tree block failed, want=591596257280, have=4262492213953725727
Check tree block failed, want=591596257280, have=4262492213953725727
Check tree block failed, want=591596257280, have=4262492213953725727
read block failed check_tree_block
Checking filesystem on /dev/mapper/glen-home
UUID: 920ae0b9-55da-4606-af40-b58493c7882b
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 127643191167 bytes used err is 0
total csum bytes: 355191196
total tree bytes: 2457419776
total fs tree bytes: 1832763392
total extent tree bytes: 195133440
btree space waste bytes: 472140292
file data blocks allocated: 1525241155584
 referenced 396208013312
Btrfs v0.20-rc1

   The home partition is 352,70 GiB.

   I've got backup, but I would like to recover it.

Thanks a lot,
Gelma

Re: help: btrfs bad tree block start

[Andrea Gelmini]

2013/6/23 Andrea Gelmini <andrea.gelmini@gmail.com>:
>    The system went read only mode while I was copying in some files.

I reply to myself to add the kernel complain when BTFS switched to ro mode:

Jun 23 05:01:06 glen kernel: [46511.108103] device label GelmaWdUsb2T
devid 1 transid 1479 /dev/mapper/toshi
Jun 23 05:01:06 glen kernel: [46511.111118] btrfs: use zlib compression
Jun 23 05:01:06 glen kernel: [46511.111122] btrfs: turning off barriers
Jun 23 05:01:06 glen kernel: [46511.111125] btrfs: disk space caching is enabled
Jun 23 05:01:59 glen kernel: [46564.107708] ------------[ cut here ]------------
Jun 23 05:01:59 glen kernel: [46564.107718] WARNING: at
fs/btrfs/super.c:253 __btrfs_abort_transaction+0x11d/0x130()
Jun 23 05:01:59 glen kernel: [46564.107720] Hardware name: 239238G
Jun 23 05:01:59 glen kernel: [46564.107721] btrfs: Transaction aborted
(error -75)
Jun 23 05:01:59 glen kernel: [46564.107723] Modules linked in:
nls_iso8859_1 nls_cp850 vfat fat btusb pci_stub vboxpci(O)
vboxnetadp(O) vboxnetflt(O) joydev vboxdrv(O) cpufreq_userspace
cpufreq_powersave cpufreq_ondemand iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi intel_powerclamp coretemp uvcvideo
videobuf2_vmalloc videobuf2_memops videobuf2_core videodev media
microcode snd_hda_codec_hdmi psmouse serio_raw snd_hda_codec_realtek
thinkpad_acpi nvram snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_pcm
snd_page_alloc iwldvm snd_seq_midi snd_seq_midi_event mac80211
snd_rawmidi snd_seq iwlwifi lpc_ich tpm_tis cfg80211 tpm
snd_seq_device tpm_bios snd_timer snd soundcore mei bnep rfcomm
bluetooth binfmt_misc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc
fscache raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx raid1 raid0 multipath hid_generic usbhid hid
crc32_pclmul usb_storage ghash_clmulni_intel wmi sdhci_pci sdhci
mmc_core e1000e ptp pps_core i915 video i2c_algo_bit drm_kms_helper
drm
Jun 23 05:01:59 glen kernel:
Jun 23 05:01:59 glen kernel: [46564.107784] Pid: 766, comm:
btrfs-transacti Tainted: G           O 3.9.7+ #3
Jun 23 05:01:59 glen kernel: [46564.107786] Call Trace:
Jun 23 05:01:59 glen kernel: [46564.107791]  [<ffffffff810894b0>]
warn_slowpath_common+0x70/0xa0
Jun 23 05:01:59 glen kernel: [46564.107793]  [<ffffffff8108952c>]
warn_slowpath_fmt+0x4c/0x50
Jun 23 05:01:59 glen kernel: [46564.107796]  [<ffffffff8128589d>]
__btrfs_abort_transaction+0x11d/0x130
Jun 23 05:01:59 glen kernel: [46564.107800]  [<ffffffff812a3e1f>]
btrfs_del_csums+0x2bf/0x2d0
Jun 23 05:01:59 glen kernel: [46564.107802]  [<ffffffff812963b8>]
__btrfs_free_extent+0x7e8/0xa60
Jun 23 05:01:59 glen kernel: [46564.107806]  [<ffffffff8129a102>]
run_clustered_refs+0x372/0xd90
Jun 23 05:01:59 glen kernel: [46564.107810]  [<ffffffff810b86d5>] ?
check_preempt_curr+0x85/0xa0
Jun 23 05:01:59 glen kernel: [46564.107812]  [<ffffffff810b871c>] ?
ttwu_do_wakeup+0x2c/0xe0
Jun 23 05:01:59 glen kernel: [46564.107815]  [<ffffffff8129eec0>]
btrfs_run_delayed_refs+0xe0/0x530
Jun 23 05:01:59 glen kernel: [46564.107818]  [<ffffffff812aed0e>]
btrfs_commit_transaction+0x4e/0x960
Jun 23 05:01:59 glen kernel: [46564.107821]  [<ffffffff812a68e5>]
transaction_kthread+0x195/0x230
Jun 23 05:01:59 glen kernel: [46564.107824]  [<ffffffff812a6750>] ?
free_fs_root+0xa0/0xa0
Jun 23 05:01:59 glen kernel: [46564.107828]  [<ffffffff810abdb0>]
kthread+0xc0/0xd0
Jun 23 05:01:59 glen kernel: [46564.107831]  [<ffffffff810abcf0>] ?
kthread_create_on_node+0x120/0x120
Jun 23 05:01:59 glen kernel: [46564.107835]  [<ffffffff8166125c>]
ret_from_fork+0x7c/0xb0
Jun 23 05:01:59 glen kernel: [46564.107838]  [<ffffffff810abcf0>] ?
kthread_create_on_node+0x120/0x120
Jun 23 05:01:59 glen kernel: [46564.107840] ---[ end trace 61ae1f206606f04b ]---
Jun 23 05:01:59 glen kernel: [46564.107842] BTRFS error (device dm-3)
in btrfs_del_csums:657: errno=-75 unknown
Jun 23 05:01:59 glen kernel: [46564.107844] BTRFS info (device dm-3):
forced readonly
Jun 23 05:01:59 glen kernel: [46564.107846] BTRFS error (device dm-3)
in __btrfs_free_extent:5801: errno=-75 unknown
Jun 23 05:01:59 glen kernel: [46564.107847] BTRFS debug (device dm-3):
run_one_delayed_ref returned -75
Jun 23 05:01:59 glen kernel: [46564.107850] BTRFS error (device dm-3)
in btrfs_run_delayed_refs:2677: errno=-75 unknown

Thanks a lot for your time,
Gelma

Re: help: btrfs bad tree block start

[Josef Bacik]

On Sun, Jun 23, 2013 at 05:33:59AM +0200, Andrea Gelmini wrote:
> Hi everybody,
>    and thanks a lot for your work.
> 
>    I have this problem.
>    With latest 3.9 kernel and btrfs-next I was close to full my home.
>    The system went read only mode while I was copying in some files.
> 
>    Everything was good. I can't write, but I can read and copy last files.
> 
>    Then I rebooted, but since that moment I cannot mount.
> 
>    The kernel complain is:
> 
> [ 1026.226883] device label 16K devid 1 transid 66906 /dev/mapper/glen-home
> [ 1026.227452] btrfs: disk space caching is enabled
> [ 1026.311650] Btrfs detected SSD devices, enabling SSD mode
> [ 1026.311825] btrfs bad tree block start 4262492213953725727 591596257280
> [ 1026.312000] btrfs bad tree block start 4262492213953725727 591596257280
> [ 1026.312010] btrfs: failed to read log tree
> [ 1026.332708] btrfs: open_ctree failed
> 
>    If I use btrfsck (Joseph one) I got this:
> 
> root@glen:/home/gelma/dev/prg/btrfs_josef# ./btrfsck --repair
> /dev/mapper/glen-home enabling repair mode
> Check tree block failed, want=591596257280, have=4262492213953725727
> Check tree block failed, want=591596257280, have=4262492213953725727
> Check tree block failed, want=591596257280, have=4262492213953725727
> Check tree block failed, want=591596257280, have=4262492213953725727
> Check tree block failed, want=591596257280, have=4262492213953725727
> read block failed check_tree_block
> Checking filesystem on /dev/mapper/glen-home
> UUID: 920ae0b9-55da-4606-af40-b58493c7882b
> checking extents
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 127643191167 bytes used err is 0
> total csum bytes: 355191196
> total tree bytes: 2457419776
> total fs tree bytes: 1832763392
> total extent tree bytes: 195133440
> btree space waste bytes: 472140292
> file data blocks allocated: 1525241155584
>  referenced 396208013312
> Btrfs v0.20-rc1
> 
>    The home partition is 352,70 GiB.
> 
>    I've got backup, but I would like to recover it.
> 

So it looks like you just have a few transid mismatches but we manage to find
something that works out just fine.  Could you make an image of this fs for me
and upload it somewhere so I can pull it down and run some stuff on it?  I've
been toying with making fsck just re-write transid mismatched blocks if we
manage to put together the file system properly and I'd like to experiment with
this on your fs so maybe we can put it back together without you having to
restore from backups.  Thanks,

Josef

Re: help: btrfs bad tree block start

[Andrea Gelmini]

2013/6/24 Josef Bacik <jbacik@fusionio.com>:
> On Sun, Jun 23, 2013 at 05:33:59AM +0200, Andrea Gelmini wrote:
> So it looks like you just have a few transid mismatches but we manage to find
> something that works out just fine.  Could you make an image of this fs for me
> and upload it somewhere so I can pull it down and run some stuff on it?  I've

Hi Josef,
   and thanks a lot for your usual precious help.
   I've already restored the backup, but I made a complete dd of the
logical volume after the crash.
   So, now I'm flying home, in a few days I'll give you what you asked for.

Thanks a lot again,
Andrea

n.b.: I've got a lot of sensible/private data in my home, but maybe we
could find a way to give all to you. I'm not so afraid to trust you.

Re: help: btrfs bad tree block start

[Duncan]

Andrea Gelmini posted on Mon, 24 Jun 2013 18:45:39 +0200 as excerpted:

> 2013/6/24 Josef Bacik <jbacik@fusionio.com>:
>> Could you make an image of this fs for me and upload it somewhere so I
>> can pull it down and run some stuff on it?  I've
> 
> n.b.: I've got a lot of sensible/private data in my home, but maybe we
> could find a way to give all to you. I'm not so afraid to trust you.

FWIW, I just updated btrfs-progs from git, and one of the new commits 
allows anonymizing filenames (actual file data/content hasn't been in the 
image, which I believe is meta-data only, either ever or at least for 
some time, but filenames can be sensitive too).  There's even two 
different modes that can be used to do it, a fast "random name of the 
same length" mode, and a much, MUCH slower "find a crc32c hash collision 
of the same length and use it" mode.  (The commit noted that on new Intel 
hardware with hardware crc accel, it took two plus minutes to find a 
collision for a single text string, three-plus minutes without that 
hardware accel, so doing the math, on a filesystem with even moderate 
thousands of files, the long one's likely to take days to generate.  
Obviously this will only be used on relatively small filesystems 
troubleshooting problems where the fast method doesn't give a useful 
result.) 

So if you can build a current git btrfs-progs, you should be able to try 
at least the fast version, and reasonably anonymize even the filenames. 
=:^)

If filenames aren't a problem, then certainly anything even reasonably 
current should deliver an image with in-the-clear filenames, but no 
content to worry about.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman