Re: how to repair a damaged filesystem with btrfs raid5

[Alexander Fieroch <alexander.fieroch@mpi-dortmund.mpg.de>]

[-- Attachment #1.1: Type: text/plain, Size: 20326 bytes --]

Am 03.02.2015 um 01:24 schrieb Tobias Holst:
> Hi.

Hi,

> There is a known bug when you re-plug in a missing hdd of a btrfs raid
> without wiping the device before. In worst case this results in a
> totally corrupted filesystem as it did sometimes during my tests of
> the raid6 implementation. With raid1 it may just "go back in time" to
> the point when you unplugged the device. Which is also bad but still
> no complete data loss - but in raid6 sometimes it was worse.
> 
> Sounds like you did that (plug in the missing device without wiping)?

yes I just unplugged the device for a few seconds and re-plugged it in again.

> 
> Next thing is, that scrub and filesystem-check of raid5/6 is not
> implemented/completed (yet) as Duncan said. It will be (mostly)
> included in 3.19, but maybe with bugs.

Unfortunately a filesystem check is still not working with kernel 3.19rc6 and btrfs 3.18.1:

# btrfs check --repair /dev/mapper/crypt-3 
enabling repair mode
Checking filesystem on /dev/mapper/crypt-3
UUID: 504c2850-3977-4340-8849-18dd3ac2e5e4
checking extents
Check tree block failed, want=165396480, have=5385177728513973313
Check tree block failed, want=165396480, have=5385177728513973313
Check tree block failed, want=165396480, have=65536
Check tree block failed, want=165396480, have=5385177728513973313
Check tree block failed, want=165396480, have=5385177728513973313
read block failed check_tree_block
Check tree block failed, want=165740544, have=6895225932619678086
Check tree block failed, want=165740544, have=6895225932619678086
Check tree block failed, want=165740544, have=65536
Check tree block failed, want=165740544, have=6895225932619678086
Check tree block failed, want=165740544, have=6895225932619678086
read block failed check_tree_block
Check tree block failed, want=165756928, have=13399486021073017810
Check tree block failed, want=165756928, have=13399486021073017810
Check tree block failed, want=165756928, have=65536
Check tree block failed, want=165756928, have=13399486021073017810
Check tree block failed, want=165756928, have=13399486021073017810
read block failed check_tree_block
Check tree block failed, want=165773312, have=12571697019259051064
Check tree block failed, want=165773312, have=12571697019259051064
Check tree block failed, want=165773312, have=65536
Check tree block failed, want=165773312, have=12571697019259051064
Check tree block failed, want=165773312, have=12571697019259051064
read block failed check_tree_block
Check tree block failed, want=165789696, have=4069002570438424782
Check tree block failed, want=165789696, have=4069002570438424782
Check tree block failed, want=165789696, have=65536
Check tree block failed, want=165789696, have=4069002570438424782
Check tree block failed, want=165789696, have=4069002570438424782
read block failed check_tree_block
Check tree block failed, want=165838848, have=9612508092910615774
Check tree block failed, want=165838848, have=9612508092910615774
Check tree block failed, want=165838848, have=65536
Check tree block failed, want=165838848, have=9612508092910615774
Check tree block failed, want=165838848, have=9612508092910615774
read block failed check_tree_block
ref mismatch on [99516416 16384] extent item 1, found 0
failed to repair damaged filesystem, aborting

> You may try to do a balance instead of a scrub as this should read and
> check your data and then write it back. This worked for me most of the
> time during my personal raid6 stability and stress tests.

I tried a balance and got a btrfs error trace in syslog as well as several sata errors (complete syslog is attached):

[ 1835.914430] ------------[ cut here ]------------
[ 1835.914468] WARNING: CPU: 3 PID: 15995 at /home/kernel/COD/linux/fs/btrfs/inode.c:923 cow_file_range+0x44c/0x460 [btrfs]()
[ 1835.914470] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c snd_hda_codec_analog snd_hda_codec_generic snd_hda_intel coretemp ppdev snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer kvm_intel kvm snd soundcore dcdbas mei_me mei serio_raw mac_hid parport_pc lpc_ich 8250_fintek shpchp parport btrfs xor raid6_pq xts gf128mul dm_crypt amdkfd amd_iommu_v2 radeon hid_generic i2c_algo_bit e1000e ttm uas drm_kms_helper usbhid ahci sata_sil usb_storage psmouse ptp hid libahci drm pps_core pata_acpi
[ 1835.914511] CPU: 3 PID: 15995 Comm: btrfs Not tainted 3.19.0-031900rc6-generic #201501261152
[ 1835.914513] Hardware name: Dell Inc. OptiPlex 755                 /0GM819, BIOS A22 06/11/2012
[ 1835.914514]  000000000000039b ffff8800a784b2f8 ffffffff817c4584 0000000000000007
[ 1835.914517]  0000000000000000 ffff8800a784b338 ffffffff81076df7 0000000000000004
[ 1835.914520]  00000000ffffffea ffff8800b056c6d0 0000000000000000 0000000000080000
[ 1835.914523] Call Trace:
[ 1835.914530]  [<ffffffff817c4584>] dump_stack+0x45/0x57
[ 1835.914535]  [<ffffffff81076df7>] warn_slowpath_common+0x97/0xe0
[ 1835.914538]  [<ffffffff81076e5a>] warn_slowpath_null+0x1a/0x20
[ 1835.914554]  [<ffffffffc073e5ec>] cow_file_range+0x44c/0x460 [btrfs]
[ 1835.914572]  [<ffffffffc0758d35>] ? free_extent_buffer+0x35/0x40 [btrfs]
[ 1835.914588]  [<ffffffffc073ea9b>] run_delalloc_nocow+0x49b/0xae0 [btrfs]
[ 1835.914605]  [<ffffffffc073f26e>] run_delalloc_range+0x18e/0x1b0 [btrfs]
[ 1835.914622]  [<ffffffffc07550a4>] writepage_delalloc.isra.33+0xf4/0x170 [btrfs]
[ 1835.914639]  [<ffffffffc07576ff>] __extent_writepage+0xcf/0x280 [btrfs]
[ 1835.914643]  [<ffffffff811b6f70>] ? SyS_msync+0x230/0x230
[ 1835.914661]  [<ffffffffc0757b6a>] extent_write_cache_pages.isra.26.constprop.39+0x2ba/0x420 [btrfs]
[ 1835.914679]  [<ffffffffc07581ce>] extent_writepages+0x4e/0x70 [btrfs]
[ 1835.914694]  [<ffffffffc073c400>] ? btrfs_submit_direct+0x1b0/0x1b0 [btrfs]
[ 1835.914710]  [<ffffffffc0738758>] btrfs_writepages+0x28/0x30 [btrfs]
[ 1835.914714]  [<ffffffff81188a00>] do_writepages+0x20/0x40
[ 1835.914717]  [<ffffffff8117dd99>] __filemap_fdatawrite_range+0x59/0x60
[ 1835.914719]  [<ffffffff8117e5c3>] filemap_fdatawrite_range+0x13/0x20
[ 1835.914735]  [<ffffffffc074bedb>] btrfs_fdatawrite_range+0x2b/0x70 [btrfs]
[ 1835.914753]  [<ffffffffc07517ac>] btrfs_wait_ordered_range+0x4c/0x150 [btrfs]
[ 1835.914770]  [<ffffffffc077c70c>] __btrfs_write_out_cache+0x39c/0x4b0 [btrfs]
[ 1835.914788]  [<ffffffffc077cb5f>] btrfs_write_out_cache+0x9f/0x100 [btrfs]
[ 1835.914802]  [<ffffffffc0725492>] btrfs_write_dirty_block_groups+0x252/0x290 [btrfs]
[ 1835.914817]  [<ffffffffc07af4ad>] update_cowonly_root+0x42/0xa8 [btrfs]
[ 1835.914832]  [<ffffffffc07af666>] commit_cowonly_roots+0x153/0x1b5 [btrfs]
[ 1835.914847]  [<ffffffffc073625d>] btrfs_commit_transaction+0x4bd/0xa70 [btrfs]
[ 1835.914865]  [<ffffffffc078a6d5>] prepare_to_merge+0x1f5/0x230 [btrfs]
[ 1835.914882]  [<ffffffffc078a9fa>] relocate_block_group+0x2ea/0x510 [btrfs]
[ 1835.914899]  [<ffffffffc078add0>] btrfs_relocate_block_group+0x1b0/0x2d0 [btrfs]
[ 1835.914917]  [<ffffffffc0760b1b>] btrfs_relocate_chunk.isra.75+0x4b/0xd0 [btrfs]
[ 1835.914934]  [<ffffffffc0761f58>] __btrfs_balance+0x348/0x460 [btrfs]
[ 1835.914951]  [<ffffffffc0762425>] btrfs_balance+0x3b5/0x5d0 [btrfs]
[ 1835.914969]  [<ffffffffc076e36c>] btrfs_ioctl_balance+0x1cc/0x530 [btrfs]
[ 1835.914986]  [<ffffffffc077033e>] btrfs_ioctl+0x69e/0xb20 [btrfs]
[ 1835.914989]  [<ffffffff811b3848>] ? do_brk+0x258/0x330
[ 1835.914993]  [<ffffffff81207475>] do_vfs_ioctl+0x75/0x320
[ 1835.914996]  [<ffffffff812077b1>] SyS_ioctl+0x91/0xb0
[ 1835.915000]  [<ffffffff817d18ad>] system_call_fastpath+0x16/0x1b
[ 1835.915002] ---[ end trace 42ee65a1e517ce93 ]---

After this I cannot umount the filesystem anymore. I get new errors in syslog:

[ 3000.196039] INFO: task kworker/u16:0:15873 blocked for more than 120 seconds.
[ 3000.198122]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.200199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.202298] kworker/u16:0   D ffff880213cbbcc8     0 15873      2 0x00000000
[ 3000.202341] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.202344]  ffff880213cbbcc8 ffff880213cbbc78 ffff880213cbbfd8 00000000000141c0
[ 3000.202349]  ffff8800b98e3100 ffff88022992e220 ffff880214f744b0 ffff880213cbbcd8
[ 3000.202353]  ffff8801f0c6b850 ffff8801f0c6b8d8 0000000000000000 ffff8800365f9df0
[ 3000.202357] Call Trace:
[ 3000.202366]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.202392]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.202399]  [<ffffffff810a068c>] ? ttwu_do_wakeup+0x2c/0x100
[ 3000.202403]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.202429]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.202455]  [<ffffffffc076720e>] normal_work_helper+0x7e/0x1b0 [btrfs]
[ 3000.202481]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.202486]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.202490]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.202495]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.202499]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.202503]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.202508]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.202512]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.202516] INFO: task kworker/u16:2:15945 blocked for more than 120 seconds.
[ 3000.204622]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.206720] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.208825] kworker/u16:2   D ffff8800adf13cc8     0 15945      2 0x00000000
[ 3000.208856] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.208859]  ffff8800adf13cc8 ffff8800adf13c78 ffff8800adf13fd8 00000000000141c0
[ 3000.208863]  ffff8800b98e3100 ffff8800bad88000 ffff880229929d70 ffff8800adf13cd8
[ 3000.208867]  ffff8801f0c6b9f8 ffff8801f0c6ba80 0000000000000000 ffff8800365f9df0
[ 3000.208871] Call Trace:
[ 3000.208876]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.208901]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.208914]  [<ffffffff810a068c>] ? ttwu_do_wakeup+0x2c/0x100
[ 3000.208920]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.208945]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.208971]  [<ffffffffc076720e>] normal_work_helper+0x7e/0x1b0 [btrfs]
[ 3000.208996]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.209001]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.209005]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.209009]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.209013]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.209017]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.209022]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.209026]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.209029] INFO: task kworker/u16:6:15951 blocked for more than 120 seconds.
[ 3000.211143]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.213270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.215403] kworker/u16:6   D ffff8800acdcbcc8     0 15951      2 0x00000000
[ 3000.215431] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.215434]  ffff8800acdcbcc8 ffff8800acdcbc78 ffff8800acdcbfd8 00000000000141c0
[ 3000.215438]  ffff8800b98e3100 ffff880214f744b0 ffff8800bad8e220 ffff8800acdcbcd8
[ 3000.215442]  ffff88009b6d6470 ffff88009b6d64f8 0000000000000000 ffff8800365f9df0
[ 3000.215446] Call Trace:
[ 3000.215450]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.215475]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.215480]  [<ffffffff810a068c>] ? ttwu_do_wakeup+0x2c/0x100
[ 3000.215484]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.215509]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.215534]  [<ffffffffc076720e>] normal_work_helper+0x7e/0x1b0 [btrfs]
[ 3000.215560]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.215564]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.215569]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.215573]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.215577]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.215581]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.215586]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.215590]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.215593] INFO: task kworker/u16:7:15952 blocked for more than 120 seconds.
[ 3000.217734]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.219857] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.221989] kworker/u16:7   D ffff8800997bfcc8     0 15952      2 0x00000000
[ 3000.222019] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.222021]  ffff8800997bfcc8 ffff8800997bfc78 ffff8800997bffd8 00000000000141c0
[ 3000.222025]  ffff8800b98e3100 ffff880229929d70 ffff88022992e220 ffff8800997bfcd8
[ 3000.222029]  ffff8801f0c6bd48 ffff8801f0c6bdd0 0000000000000000 ffff8800365f9df0
[ 3000.222033] Call Trace:
[ 3000.222038]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.222070]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.222076]  [<ffffffff810a068c>] ? ttwu_do_wakeup+0x2c/0x100
[ 3000.222079]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.222105]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.222130]  [<ffffffffc076720e>] normal_work_helper+0x7e/0x1b0 [btrfs]
[ 3000.222156]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.222160]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.222164]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.222169]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.222172]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.222177]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.222181]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.222185]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.222191] INFO: task kworker/u16:1:16059 blocked for more than 120 seconds.
[ 3000.224322]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.226463] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.228612] kworker/u16:1   D ffff8801c903fcc8     0 16059      2 0x00000000
[ 3000.228643] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.228646]  ffff8801c903fcc8 ffff8801c903fc78 ffff8801c903ffd8 00000000000141c0
[ 3000.228650]  ffff8800b98e3100 ffff88022d253110 ffff8800bad88000 ffff8801c903fcd8
[ 3000.228654]  ffff8801f0c6bba0 ffff8801f0c6bc28 0000000000000000 ffff8800365f9df0
[ 3000.228658] Call Trace:
[ 3000.228663]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.228688]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.228692]  [<ffffffff810a068c>] ? ttwu_do_wakeup+0x2c/0x100
[ 3000.228696]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.228721]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.228754]  [<ffffffffc076720e>] normal_work_helper+0x7e/0x1b0 [btrfs]
[ 3000.228784]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.228789]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.228793]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.228797]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.228801]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.228805]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.228810]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.228814]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.228818] INFO: task kworker/u16:3:16076 blocked for more than 120 seconds.
[ 3000.230960]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.233105] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.235269] kworker/u16:3   D ffff8801d48d7cc8     0 16076      2 0x00000000
[ 3000.235297] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper [btrfs]
[ 3000.235299]  ffff8801d48d7cc8 ffff8801d48d7c78 ffff8801d48d7fd8 00000000000141c0
[ 3000.235303]  ffff8800b98e3100 ffff88022e6209d0 ffff88022d253110 ffff8801d48d7cd8
[ 3000.235307]  ffff8801f0c6b500 ffff8801f0c6b588 0000000000000000 ffff8800365f9df0
[ 3000.235311] Call Trace:
[ 3000.235316]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.235341]  [<ffffffffc0751305>] btrfs_start_ordered_extent+0x115/0x140 [btrfs]
[ 3000.235346]  [<ffffffff810a4007>] ? wake_up_process+0x27/0x50
[ 3000.235350]  [<ffffffff810b7440>] ? prepare_to_wait_event+0x100/0x100
[ 3000.235375]  [<ffffffffc0751359>] btrfs_run_ordered_extent_work+0x29/0x40 [btrfs]
[ 3000.235401]  [<ffffffffc07672d2>] normal_work_helper+0x142/0x1b0 [btrfs]
[ 3000.235426]  [<ffffffffc0767392>] btrfs_flush_delalloc_helper+0x12/0x20 [btrfs]
[ 3000.235431]  [<ffffffff8108f6dd>] process_one_work+0x14d/0x460
[ 3000.235435]  [<ffffffff810900bb>] worker_thread+0x11b/0x3f0
[ 3000.235439]  [<ffffffff8108ffa0>] ? create_worker+0x1e0/0x1e0
[ 3000.235443]  [<ffffffff81095cc9>] kthread+0xc9/0xe0
[ 3000.235447]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.235452]  [<ffffffff817d17fc>] ret_from_fork+0x7c/0xb0
[ 3000.235456]  [<ffffffff81095c00>] ? flush_kthread_worker+0x90/0x90
[ 3000.235459] INFO: task umount:16078 blocked for more than 120 seconds.
[ 3000.237630]       Tainted: G        W      3.19.0-031900rc6-generic #201501261152
[ 3000.239788] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3000.241955] umount          D ffff8801beac3b58     0 16078   1623 0x00000000
[ 3000.241962]  ffff8801beac3b58 ffff8801beac3b28 ffff8801beac3fd8 00000000000141c0
[ 3000.241966]  ffff8800b98e3100 ffff88022e620000 ffff88022d2513a0 ffff8801beac3b48
[ 3000.241970]  ffff88009b6d6590 7fffffffffffffff 7fffffffffffffff ffff88022d2513a0
[ 3000.241974] Call Trace:
[ 3000.241979]  [<ffffffff817cd6b9>] schedule+0x29/0x70
[ 3000.241983]  [<ffffffff817d0445>] schedule_timeout+0x1b5/0x210
[ 3000.241987]  [<ffffffff810a4007>] ? wake_up_process+0x27/0x50
[ 3000.241991]  [<ffffffff817cebc7>] wait_for_completion+0xa7/0x160
[ 3000.241996]  [<ffffffff810a3fa0>] ? try_to_wake_up+0x2a0/0x2a0
[ 3000.242021]  [<ffffffffc0750fca>] btrfs_wait_ordered_extents+0x20a/0x250 [btrfs]
[ 3000.242055]  [<ffffffffc0751138>] btrfs_wait_ordered_roots+0x128/0x1e0 [btrfs]
[ 3000.242076]  [<ffffffffc0703e91>] btrfs_sync_fs+0x51/0x150 [btrfs]
[ 3000.242082]  [<ffffffff81224e70>] __sync_filesystem+0x30/0x60
[ 3000.242086]  [<ffffffff81224f1b>] sync_filesystem+0x4b/0x70
[ 3000.242091]  [<ffffffff811f6dcb>] generic_shutdown_super+0x3b/0x110
[ 3000.242096]  [<ffffffff8118d88d>] ? unregister_shrinker+0x5d/0x70
[ 3000.242100]  [<ffffffff811f6f36>] kill_anon_super+0x16/0x30
[ 3000.242118]  [<ffffffffc070710e>] btrfs_kill_super+0x1e/0x130 [btrfs]
[ 3000.242123]  [<ffffffff811f7129>] deactivate_locked_super+0x59/0x80
[ 3000.242127]  [<ffffffff811f7dee>] deactivate_super+0x4e/0x70
[ 3000.242132]  [<ffffffff812138d3>] cleanup_mnt+0x43/0x90
[ 3000.242136]  [<ffffffff81213972>] __cleanup_mnt+0x12/0x20
[ 3000.242139]  [<ffffffff81093dff>] task_work_run+0xaf/0xf0
[ 3000.242145]  [<ffffffff81015077>] do_notify_resume+0xc7/0xd0
[ 3000.242149]  [<ffffffff817d1b4f>] int_signal+0x12/0x17

After a reboot the filesystem is still damaged, not repairable with a filesystem check but accessible.

> But maybe
> your filesystem has already been corrupted...
> Give it a try :)

Yea, no problem. It's just to test the current stability of raid5 and maybe to find bugs for you developers.

Regards,
Alexander

> Regards
> Tobias


[-- Attachment #1.2: dmesg.log.gz --]
[-- Type: application/gzip, Size: 23072 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4860 bytes --]