* BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
@ 2016-08-09 23:15 Matt McKinnon
2016-08-10 0:01 ` Chris Murphy
0 siblings, 1 reply; 12+ messages in thread
From: Matt McKinnon @ 2016-08-09 23:15 UTC (permalink / raw)
To: linux-btrfs
Hello,
Our server recently crashed and was rebooted. When it returned our
BTRFS volume is mounting read-only:
[ 142.395093] BTRFS: error (device sda1) in
btrfs_run_delayed_refs:2963: errno=-17 Object already exists
[ 142.404418] BTRFS info (device sda1): forced readonly
I tried upgrading the kernel from 4.3 to 4.7. Upgraded btrfs-progs to
v4.7 as well.
# uname -a
Linux hostname 4.7.0-custom #1 SMP Tue Aug 9 11:16:28 EDT 2016 x86_64
x86_64 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v4.7
# btrfs fi show
Label: none uuid: 33f9089e-acc7-4a39-8b83-b18bb182faaf
Total devices 1 FS bytes used 14.95TiB
devid 1 size 50.93TiB used 22.67TiB path /dev/sda1
# btrfs fi df /export/
Data, single: total=22.53TiB, used=14.89TiB
System, DUP: total=40.00MiB, used=2.39MiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=70.50GiB, used=60.21GiB
Metadata, single: total=1.51GiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B
# dmesg
[ 142.394841] ------------[ cut here ]------------
[ 142.394874] WARNING: CPU: 6 PID: 269 at fs/btrfs/extent-tree.c:2963
btrfs_run_delayed_refs+0x292/0x2d0 [btrfs]
[ 142.394876] BTRFS: Transaction aborted (error -17)
[ 142.394878] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack
iptable_filter ip_tables x_tables ipmi_devintf nfsd auth_rpcgss nfs_acl
nfs lockd grace sunrpc fscache sb_edac edac_core x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
glue_helper ablk_helper cryptd dm_multipath joydev lpc_ich mei_me mei
ioatdma wmi ipmi_si ipmi_msghandler shpchp mac_hid btrfs lp parport ses
enclosure scsi_transport_sas raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor igb raid6_pq libcrc32c
i2c_algo_bit raid1 hid_generic dca usbhid raid0 ptp hid ahci
megaraid_sas multipath libahci pps_core linear dm_mirror dm_region_hash
dm_log
[ 142.394942] CPU: 6 PID: 269 Comm: kworker/u18:5 Not tainted
4.7.0-custom #1
[ 142.394944] Hardware name: Supermicro
X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[ 142.394966] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs]
[ 142.394969] 0000000000000000 ffff88086a057ca0 ffffffff813b816c
ffff88086a057cf0
[ 142.394972] 0000000000000000 ffff88086a057ce0 ffffffff8107a321
00000b9325288170
[ 142.394975] ffff8808519eb000 ffff880825288170 ffff88086b2c1000
0000000000000020
[ 142.394978] Call Trace:
[ 142.394987] [<ffffffff813b816c>] dump_stack+0x63/0x87
[ 142.394993] [<ffffffff8107a321>] __warn+0xd1/0xf0
[ 142.394996] [<ffffffff8107a38f>] warn_slowpath_fmt+0x4f/0x60
[ 142.395012] [<ffffffffc01c6e52>] btrfs_run_delayed_refs+0x292/0x2d0
[btrfs]
[ 142.395025] [<ffffffffc01c6f24>] delayed_ref_async_start+0x94/0xb0
[btrfs]
[ 142.395044] [<ffffffffc020f780>] normal_work_helper+0xc0/0x2d0 [btrfs]
[ 142.395050] [<ffffffff81091082>] ? pwq_activate_delayed_work+0x42/0xb0
[ 142.395066] [<ffffffffc020fbc2>] btrfs_extent_refs_helper+0x12/0x20
[btrfs]
[ 142.395070] [<ffffffff81093173>] process_one_work+0x153/0x3f0
[ 142.395073] [<ffffffff8109392b>] worker_thread+0x12b/0x4b0
[ 142.395076] [<ffffffff81093800>] ? rescuer_thread+0x340/0x340
[ 142.395079] [<ffffffff81099109>] kthread+0xc9/0xe0
[ 142.395085] [<ffffffff817db85f>] ret_from_fork+0x1f/0x40
[ 142.395088] [<ffffffff81099040>] ? kthread_park+0x60/0x60
[ 142.395090] ---[ end trace e2b0b8dc37502011 ]---
[ 142.395093] BTRFS: error (device sda1) in
btrfs_run_delayed_refs:2963: errno=-17 Object already exists
[ 142.404418] BTRFS info (device sda1): forced readonly
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-09 23:15 BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists Matt McKinnon
@ 2016-08-10 0:01 ` Chris Murphy
2016-08-10 0:06 ` Chris Murphy
2016-08-10 0:17 ` Matt McKinnon
0 siblings, 2 replies; 12+ messages in thread
From: Chris Murphy @ 2016-08-10 0:01 UTC (permalink / raw)
To: Matt McKinnon; +Cc: Btrfs BTRFS
On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com> wrote:
> Hello,
>
> Our server recently crashed and was rebooted. When it returned our BTRFS
> volume is mounting read-only:
What happens when you try mounting with -o usebackuproot ?
If that fails, what output do you get for 'btrfs check' (without
--repair)? If you only get some "errors 400, nbytes wrong" then
--repair should fix the problem.
--
Chris Murphy
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:01 ` Chris Murphy
@ 2016-08-10 0:06 ` Chris Murphy
2016-08-10 0:29 ` Matt McKinnon
2016-08-10 0:56 ` Matt McKinnon
2016-08-10 0:17 ` Matt McKinnon
1 sibling, 2 replies; 12+ messages in thread
From: Chris Murphy @ 2016-08-10 0:06 UTC (permalink / raw)
To: Chris Murphy; +Cc: Matt McKinnon, Btrfs BTRFS
On Tue, Aug 9, 2016 at 6:01 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com> wrote:
>> Hello,
>>
>> Our server recently crashed and was rebooted. When it returned our BTRFS
>> volume is mounting read-only:
>
> What happens when you try mounting with -o usebackuproot ?
>
> If that fails, what output do you get for 'btrfs check' (without
> --repair)? If you only get some "errors 400, nbytes wrong" then
> --repair should fix the problem.
This could also be a regression somewhere...
https://bugzilla.kernel.org/show_bug.cgi?id=60522
--
Chris Murphy
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:06 ` Chris Murphy
@ 2016-08-10 0:29 ` Matt McKinnon
2016-08-10 2:25 ` Chris Murphy
2016-08-10 0:56 ` Matt McKinnon
1 sibling, 1 reply; 12+ messages in thread
From: Matt McKinnon @ 2016-08-10 0:29 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
Spoke too soon. Do I need to continue to run with that mount option in
place?
[ 83.775984] BTRFS warning (device sda1): block group 25741009879040
has wrong amount of free space
[ 83.775989] BTRFS warning (device sda1): failed to load free space
cache for block group 25741009879040, rebuilding it now
[ 85.231748] BTRFS warning (device sda1): block group 25737721544704
has wrong amount of free space
[ 85.231752] BTRFS warning (device sda1): failed to load free space
cache for block group 25737721544704, rebuilding it now
[ 98.913796] BTRFS info (device sda1): disk space caching is enabled
[ 98.913803] BTRFS info (device sda1): has skinny extents
[ 179.564408] BTRFS warning (device sda1): block group 78412513280 has
wrong amount of free space
[ 179.564414] BTRFS warning (device sda1): failed to load free space
cache for block group 78412513280, rebuilding it now
[ 667.106718] ------------[ cut here ]------------
[ 667.106772] WARNING: CPU: 0 PID: 2726 at fs/btrfs/extent-tree.c:2963
btrfs_run_delayed_refs+0x292/0x2d0 [btrfs]
[ 667.106775] BTRFS: Transaction aborted (error -17)
[ 667.106777] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack
iptable_filter ip_tables x_tables ipmi_devintf sb_edac edac_core
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel btrfs kvm
irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd dm_multipath
joydev lpc_ich mei_me mei wmi ipmi_si ipmi_msghandler nfsd auth_rpcgss
nfs_acl nfs lockd grace ioatdma sunrpc shpchp mac_hid fscache lp parport
ses enclosure scsi_transport_sas raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c
hid_generic igb raid1 usbhid i2c_algo_bit ahci raid0 dca multipath ptp
hid megaraid_sas libahci linear pps_core dm_mirror dm_region_hash dm_log
[ 667.106859] CPU: 0 PID: 2726 Comm: btrfs-transacti Not tainted
4.7.0-custom #1
[ 667.106861] Hardware name: Supermicro
X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[ 667.106864] 0000000000000000 ffff880464e73c08 ffffffff813b816c
ffff880464e73c58
[ 667.106869] 0000000000000000 ffff880464e73c48 ffffffff8107a321
00000b936c3cc170
[ 667.106873] ffff880443191130 ffff88046c3cc170 ffff88046b43f000
ffffffffffffffff
[ 667.106878] Call Trace:
[ 667.106889] [<ffffffff813b816c>] dump_stack+0x63/0x87
[ 667.106896] [<ffffffff8107a321>] __warn+0xd1/0xf0
[ 667.106901] [<ffffffff8107a38f>] warn_slowpath_fmt+0x4f/0x60
[ 667.106925] [<ffffffffc06a2e52>] btrfs_run_delayed_refs+0x292/0x2d0
[btrfs]
[ 667.106947] [<ffffffffc06a3be8>]
btrfs_write_dirty_block_groups+0x178/0x3b0 [btrfs]
[ 667.106974] [<ffffffffc06b609c>] commit_cowonly_roots+0x23c/0x2e0
[btrfs]
[ 667.106999] [<ffffffffc06b8d0b>]
btrfs_commit_transaction+0x4fb/0xa80 [btrfs]
[ 667.107021] [<ffffffffc06b32b2>] transaction_kthread+0x1d2/0x200 [btrfs]
[ 667.107042] [<ffffffffc06b30e0>] ?
btrfs_cleanup_transaction+0x580/0x580 [btrfs]
[ 667.107047] [<ffffffff81099109>] kthread+0xc9/0xe0
[ 667.107053] [<ffffffff817db85f>] ret_from_fork+0x1f/0x40
[ 667.107056] [<ffffffff81099040>] ? kthread_park+0x60/0x60
[ 667.107060] ---[ end trace 336c80ba4db66e78 ]---
[ 667.107065] BTRFS: error (device sda1) in
btrfs_run_delayed_refs:2963: errno=-17 Object already exists
[ 667.116389] BTRFS info (device sda1): forced readonly
[ 667.117081] BTRFS warning (device sda1): Skipping commit of aborted
transaction.
[ 667.117086] BTRFS: error (device sda1) in cleanup_transaction:1853:
errno=-17 Object already exists
On 08/09/2016 08:06 PM, Chris Murphy wrote:
> On Tue, Aug 9, 2016 at 6:01 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com> wrote:
>>> Hello,
>>>
>>> Our server recently crashed and was rebooted. When it returned our BTRFS
>>> volume is mounting read-only:
>>
>> What happens when you try mounting with -o usebackuproot ?
>>
>> If that fails, what output do you get for 'btrfs check' (without
>> --repair)? If you only get some "errors 400, nbytes wrong" then
>> --repair should fix the problem.
>
> This could also be a regression somewhere...
> https://bugzilla.kernel.org/show_bug.cgi?id=60522
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:29 ` Matt McKinnon
@ 2016-08-10 2:25 ` Chris Murphy
2016-08-10 6:41 ` Duncan
2016-08-11 1:51 ` Matt McKinnon
0 siblings, 2 replies; 12+ messages in thread
From: Chris Murphy @ 2016-08-10 2:25 UTC (permalink / raw)
To: Matt McKinnon; +Cc: Btrfs BTRFS
On Tue, Aug 9, 2016 at 6:29 PM, Matt McKinnon <matt@techsquare.com> wrote:
> Spoke too soon. Do I need to continue to run with that mount option in
> place?
It shouldn't be necessary. Something's still wrong for some reason,
even with DUP metadata being CoW'd so someone else is going to have to
speak up what the problem is. And that btrfs check not only doesn't
come up clean but crashes suggests some confluence of things in kernel
4.3 and your hardware conspired to make the file system inconsistent
in a way that isn't immediately recovering the usual way. That is,
usebackuproots working suggests that there's a bug elsewhere in the
storage stack because normally that shouldn't be necessary -
something's happened out of order.
1 size 50.93TiB used 22.67TiB path /dev/sda1
What is the exact nature of this block device?
If getting this back up and running is urgent I suggest inquiring on
IRC what the next steps are.
In the meantime I'd get a btrfs-image (which is probably going to be
quite large given metadata is 60GiB), if that pukes then see if 'btrfs
inspect-internal dump-tree /dev/sda1 > dumptree.log' which may also
fail but before it fails might contain something useful. Obviously
btrfs check shouldn't crash so that's a bug already. What do you get
for free -m? It's known that btrfs check needs a lot of memory and
pretty much all the metadata needs to be read in, so... if you have an
SSD available it might make sense to setup a huge pile of swap on that
SSD and rerun btrfs check.
--
Chris Murphy
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 2:25 ` Chris Murphy
@ 2016-08-10 6:41 ` Duncan
2016-08-11 1:51 ` Matt McKinnon
1 sibling, 0 replies; 12+ messages in thread
From: Duncan @ 2016-08-10 6:41 UTC (permalink / raw)
To: linux-btrfs
Chris Murphy posted on Tue, 09 Aug 2016 20:25:39 -0600 as excerpted:
> 1 size 50.93TiB used 22.67TiB path /dev/sda1
>
> What is the exact nature of this block device?
>
> If getting this back up and running is urgent I suggest inquiring on IRC
> what the next steps are.
>
> In the meantime I'd get a btrfs-image (which is probably going to be
> quite large given metadata is 60GiB), if that pukes then see if 'btrfs
> inspect-internal dump-tree /dev/sda1 > dumptree.log' which may also fail
> but before it fails might contain something useful. Obviously btrfs
> check shouldn't crash so that's a bug already.
I'd suggest that at 50+ TB size, 22+ TB used, given btrfs check's known
memory usage, the crash may well simply be due to lack of memory. It's
very likely that will require well over the 16 GiB memory that's
reasonably common these days, and if there's no swap to fill in the gap...
There's a low memory mode check under development right now, but it's not
complete yet.
(FWIW my usage is at the other extreme, quarter TB SSDs and partitioned
down beyond that, so the largest btrfs is well under 50 GiB usable, tho
it's pair-device raid1 so about 50 GiB total. That's too small to be
practical for many, but at least until the low-memory btrfs check or
until 32 and 64 GiB RAM becomes common, single filesystems in the double-
digit TB may arguably be too big to be practical, at least for other than
the lucky few with the sort of memory required, as well.)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 2:25 ` Chris Murphy
2016-08-10 6:41 ` Duncan
@ 2016-08-11 1:51 ` Matt McKinnon
1 sibling, 0 replies; 12+ messages in thread
From: Matt McKinnon @ 2016-08-11 1:51 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
This block device is a volume which is on hardware RAID. We have a few
machines like this running without issue. Something odd hit this one.
# free -m
total used free shared buffers cached
Mem: 32143 1258 30885 1 52 973
-/+ buffers/cache: 232 31910
Swap: 32733 0 32733
I'll gather up some more logs, in the meantime it crashed again:
[81348.134017] ------------[ cut here ]------------
[81348.138635] kernel BUG at fs/btrfs/ctree.c:3187!
[81348.143253] invalid opcode: 0000 [#1] SMP
[81348.147267] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack
iptable_filter ip_tables x_tables ipmi_devintf sb_edac edac_core
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel btrfs kvm
irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd dm_multipath
joydev lpc_ich mei_me mei wmi ipmi_si ipmi_msghandler nfsd auth_rpcgss
nfs_acl nfs lockd grace ioatdma sunrpc shpchp mac_hid fscache lp parport
ses enclosure scsi_transport_sas raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c
hid_generic igb raid1 usbhid i2c_algo_bit ahci raid0 dca multipath ptp
hid megaraid_sas libahci linear pps_core dm_mirror dm_region_hash dm_log
[81348.219932] CPU: 3 PID: 9218 Comm: nfsd Tainted: G W
4.7.0-custom #1
[81348.227409] Hardware name: Supermicro
X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[81348.236441] task: ffff88046bd28e40 ti: ffff880248b6c000 task.ti:
ffff880248b6c000
[81348.243921] RIP: 0010:[<ffffffffc06906c4>] [<ffffffffc06906c4>]
btrfs_set_item_key_safe+0x154/0x160 [btrfs]
[81348.253799] RSP: 0018:ffff880248b6f578 EFLAGS: 00010246
[81348.259110] RAX: 0000000000000000 RBX: 0000000000000017 RCX:
000000000000d000
[81348.266233] RDX: 0000000000000000 RSI: ffff880248b6f67e RDI:
ffff880248b6f597
[81348.273357] RBP: ffff880248b6f5d0 R08: 0000000000004000 R09:
ffff880248b6f598
[81348.280479] R10: 0000000000000000 R11: 0000000000000003 R12:
ffff880248b6f586
[81348.287604] R13: ffff8800394c9770 R14: ffff880248b6f67e R15:
ffff8802b8bb4f50
[81348.294729] FS: 0000000000000000(0000) GS:ffff88046fd80000(0000)
knlGS:0000000000000000
[81348.302813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[81348.308550] CR2: 00007fa0c244a000 CR3: 0000000001c06000 CR4:
00000000001406e0
[81348.315674] Stack:
[81348.317687] ffff88046b402000 335effffc06d30b9 006c0000000001ea
5e000000000000d0
[81348.325145] 6c0000000001ea33 000000000000d000 ffff8800394c9770
0000000000003a6b
[81348.332598] 0000000000000000 000000000000e000 ffff8802b8bb4f50
ffff880248b6f6e0
[81348.340051] Call Trace:
[81348.342517] [<ffffffffc06ce023>] __btrfs_drop_extents+0x503/0xd30
[btrfs]
[81348.349403] [<ffffffffc06d1200>] ? free_extent_map+0x40/0xa0 [btrfs]
[81348.355850] [<ffffffffc07347fb>]
btrfs_log_changed_extents+0x2cc/0x5b1 [btrfs]
[81348.363163] [<ffffffffc0736583>] btrfs_log_inode+0xce4/0xe99 [btrfs]
[81348.369612] [<ffffffffc06da690>] ?
extent_write_cache_pages.isra.37.constprop.60+0x3a0/0x450 [btrfs]
[81348.378821] [<ffffffff810af555>] ? put_prev_entity+0x35/0x730
[81348.384653] [<ffffffff8102b6ff>] ? __switch_to+0x24f/0x590
[81348.390237] [<ffffffffc06faf37>] btrfs_log_inode_parent+0x227/0x970
[btrfs]
[81348.397290] [<ffffffffc06b6c62>] ?
wait_current_trans.isra.24+0x22/0x110 [btrfs]
[81348.404764] [<ffffffff811dc5d6>] ? kmem_cache_alloc+0x1d6/0x1f0
[81348.410782] [<ffffffffc06b93bf>] ? start_transaction+0x12f/0x4d0 [btrfs]
[81348.417579] [<ffffffffc06fc542>] btrfs_log_dentry_safe+0x62/0x80 [btrfs]
[81348.424372] [<ffffffffc06d05b3>] btrfs_sync_file+0x283/0x3b0 [btrfs]
[81348.430809] [<ffffffff8123236d>] vfs_fsync_range+0x3d/0xb0
[81348.436390] [<ffffffffc06d0aca>] btrfs_file_write_iter+0x3ea/0x4c0
[btrfs]
[81348.443348] [<ffffffff811ffa6a>] do_iter_readv_writev+0x9a/0xf0
[81348.449351] [<ffffffff81200438>] do_readv_writev+0x178/0x210
[81348.455107] [<ffffffffc06d06e0>] ? btrfs_sync_file+0x3b0/0x3b0 [btrfs]
[81348.461718] [<ffffffff811fd623>] ? do_dentry_open+0x273/0x320
[81348.467550] [<ffffffff811fcd60>] ? finish_no_open+0x20/0x20
[81348.473208] [<ffffffff811fe8d5>] ? vfs_open+0x55/0x80
[81348.478347] [<ffffffff812006ec>] vfs_writev+0x3c/0x50
[81348.483488] [<ffffffffc032ce3e>] nfsd_vfs_write+0xbe/0x330 [nfsd]
[81348.489672] [<ffffffffc032f5b0>] nfsd_write+0x120/0x2f0 [nfsd]
[81348.495591] [<ffffffffc0335571>] nfsd3_proc_write+0xb1/0x130 [nfsd]
[81348.501942] [<ffffffffc0328278>] nfsd_dispatch+0xb8/0x1f0 [nfsd]
[81348.508047] [<ffffffffc01db25c>] svc_process_common+0x40c/0x6a0 [sunrpc]
[81348.514842] [<ffffffffc01dc655>] svc_process+0x105/0x1c0 [sunrpc]
[81348.521023] [<ffffffffc0327cd0>] nfsd+0xf0/0x160 [nfsd]
[81348.526337] [<ffffffffc0327be0>] ? nfsd_destroy+0x60/0x60 [nfsd]
[81348.532426] [<ffffffff81099109>] kthread+0xc9/0xe0
[81348.537306] [<ffffffff817db85f>] ret_from_fork+0x1f/0x40
[81348.542702] [<ffffffff81099040>] ? kthread_park+0x60/0x60
[81348.548188] Code: 48 8b 45 bf 48 8d 7d c7 4c 89 f6 48 89 45 d0 0f b6
45 be 88 45 cf 48 8b 45 b6 48 89 45 c7 e8 74 f3 ff ff 85 c0 0f 8f 55 ff
ff ff <0f> 0b 0f 0b 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 b8 00
[81348.568144] RIP [<ffffffffc06906c4>]
btrfs_set_item_key_safe+0x154/0x160 [btrfs]
[81348.575649] RSP <ffff880248b6f578>
[81348.625322] ---[ end trace 336c80ba4db66e7a ]---
Mounting will crash again with this error:
[ 79.921962] ------------[ cut here ]------------
[ 79.922000] WARNING: CPU: 6 PID: 2632 at fs/btrfs/file.c:546
btrfs_drop_extent_cache+0x3e8/0x400 [btrfs]
[ 79.922002] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack
iptable_filter ip_tables x_tables ipmi_devintf sb_edac edac_core
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel btrfs aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd dm_multipath nfsd
auth_rpcgss joydev nfs_acl mei_me nfs lpc_ich mei lockd wmi grace
ipmi_si sunrpc ipmi_msghandler fscache shpchp ioatdma mac_hid lp parport
ses enclosure scsi_transport_sas raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor hid_generic igb raid6_pq
i2c_algo_bit libcrc32c dca usbhid raid1 ahci raid0 ptp megaraid_sas
multipath hid libahci pps_core linear dm_mirror dm_region_hash dm_log
[ 79.922063] CPU: 6 PID: 2632 Comm: mount Not tainted 4.7.0-custom #1
[ 79.922065] Hardware name: Supermicro
X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[ 79.922067] 0000000000000000 ffff88046ca1f538 ffffffff813b816c
0000000000000000
[ 79.922071] 0000000000000000 ffff88046ca1f578 ffffffff8107a321
000002226ca1f5e0
[ 79.922074] ffff880841d19460 000000000000e000 ffff880841e21290
ffff880841e210c0
[ 79.922077] Call Trace:
[ 79.922089] [<ffffffff813b816c>] dump_stack+0x63/0x87
[ 79.922096] [<ffffffff8107a321>] __warn+0xd1/0xf0
[ 79.922099] [<ffffffff8107a40d>] warn_slowpath_null+0x1d/0x20
[ 79.922117] [<ffffffffc05b17c8>] btrfs_drop_extent_cache+0x3e8/0x400
[btrfs]
[ 79.922133] [<ffffffffc05b20d2>] __btrfs_drop_extents+0x5b2/0xd30
[btrfs]
[ 79.922147] [<ffffffffc0573b05>] ?
generic_bin_search.constprop.36+0x85/0x190 [btrfs]
[ 79.922160] [<ffffffffc05715c6>] ? btrfs_set_path_blocking+0x36/0x70
[btrfs]
[ 79.922173] [<ffffffffc0576468>] ? btrfs_search_slot+0x438/0x970 [btrfs]
[ 79.922178] [<ffffffff811dc5d6>] ? kmem_cache_alloc+0x1d6/0x1f0
[ 79.922190] [<ffffffffc057158a>] ? btrfs_alloc_path+0x1a/0x20 [btrfs]
[ 79.922205] [<ffffffffc05b3839>] btrfs_drop_extents+0x79/0xa0 [btrfs]
[ 79.922224] [<ffffffffc05de179>] replay_one_extent+0x419/0x750 [btrfs]
[ 79.922241] [<ffffffffc05de98b>] replay_one_buffer+0x4db/0x7d0 [btrfs]
[ 79.922258] [<ffffffffc05ba6bf>] ?
mark_extent_buffer_accessed+0x4f/0x70 [btrfs]
[ 79.922274] [<ffffffffc05dae0c>] walk_down_log_tree+0x1cc/0x3d0 [btrfs]
[ 79.922289] [<ffffffffc05db2ea>] walk_log_tree+0xba/0x1a0 [btrfs]
[ 79.922304] [<ffffffffc05e0773>] btrfs_recover_log_trees+0x213/0x470
[btrfs]
[ 79.922318] [<ffffffffc05de4b0>] ? replay_one_extent+0x750/0x750 [btrfs]
[ 79.922335] [<ffffffffc0599c6d>] open_ctree+0x264d/0x2760 [btrfs]
[ 79.922348] [<ffffffffc056e8f4>] btrfs_mount+0xc94/0xeb0 [btrfs]
[ 79.922353] [<ffffffff813cd67e>] ? find_next_zero_bit+0x1e/0x20
[ 79.922358] [<ffffffff8119fbbe>] ? pcpu_next_unpop+0x3e/0x50
[ 79.922362] [<ffffffff813cd659>] ? find_next_bit+0x19/0x20
[ 79.922368] [<ffffffff81204179>] mount_fs+0x39/0x160
[ 79.922371] [<ffffffff811a0df5>] ? __alloc_percpu+0x15/0x20
[ 79.922375] [<ffffffff8121fda7>] vfs_kern_mount+0x67/0x110
[ 79.922387] [<ffffffffc056ddeb>] btrfs_mount+0x18b/0xeb0 [btrfs]
[ 79.922390] [<ffffffff813cd67e>] ? find_next_zero_bit+0x1e/0x20
[ 79.922394] [<ffffffff81204179>] mount_fs+0x39/0x160
[ 79.922397] [<ffffffff811a0df5>] ? __alloc_percpu+0x15/0x20
[ 79.922399] [<ffffffff8121fda7>] vfs_kern_mount+0x67/0x110
[ 79.922402] [<ffffffff8122236a>] do_mount+0x22a/0xd90
[ 79.922406] [<ffffffff811dfbcf>] ? __kmalloc_track_caller+0x1af/0x250
[ 79.922408] [<ffffffff8119b831>] ? strndup_user+0x41/0x80
[ 79.922411] [<ffffffff8119b752>] ? memdup_user+0x42/0x70
[ 79.922413] [<ffffffff812231b3>] SyS_mount+0x83/0xd0
[ 79.922418] [<ffffffff817db636>] entry_SYSCALL_64_fastpath+0x1e/0xa8
[ 79.922436] ---[ end trace 0db3466cdad31dcf ]---
On 08/09/2016 10:25 PM, Chris Murphy wrote:
> On Tue, Aug 9, 2016 at 6:29 PM, Matt McKinnon <matt@techsquare.com> wrote:
>> Spoke too soon. Do I need to continue to run with that mount option in
>> place?
>
> It shouldn't be necessary. Something's still wrong for some reason,
> even with DUP metadata being CoW'd so someone else is going to have to
> speak up what the problem is. And that btrfs check not only doesn't
> come up clean but crashes suggests some confluence of things in kernel
> 4.3 and your hardware conspired to make the file system inconsistent
> in a way that isn't immediately recovering the usual way. That is,
> usebackuproots working suggests that there's a bug elsewhere in the
> storage stack because normally that shouldn't be necessary -
> something's happened out of order.
>
> 1 size 50.93TiB used 22.67TiB path /dev/sda1
>
> What is the exact nature of this block device?
>
> If getting this back up and running is urgent I suggest inquiring on
> IRC what the next steps are.
>
> In the meantime I'd get a btrfs-image (which is probably going to be
> quite large given metadata is 60GiB), if that pukes then see if 'btrfs
> inspect-internal dump-tree /dev/sda1 > dumptree.log' which may also
> fail but before it fails might contain something useful. Obviously
> btrfs check shouldn't crash so that's a bug already. What do you get
> for free -m? It's known that btrfs check needs a lot of memory and
> pretty much all the metadata needs to be read in, so... if you have an
> SSD available it might make sense to setup a huge pile of swap on that
> SSD and rerun btrfs check.
>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:06 ` Chris Murphy
2016-08-10 0:29 ` Matt McKinnon
@ 2016-08-10 0:56 ` Matt McKinnon
1 sibling, 0 replies; 12+ messages in thread
From: Matt McKinnon @ 2016-08-10 0:56 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
# btrfs check /dev/sda1
Checking filesystem on /dev/sda1
UUID: 33f9089e-acc7-4a39-8b83-b18bb182faaf
checking extents
ref mismatch on [958277767168 5894144] extent item 0, found 1
Backref 958277767168 root 257 owner 15799573 offset 750342144 num_refs 0
not found in extent tree
Incorrect local backref count on 958277767168 root 257 owner 15799573
offset 750342144 found 1 wanted 0 back 0x15d380f90
backpointer mismatch on [958277767168 5894144]
ref mismatch on [958298935296 9666560] extent item 0, found 2
Backref 958298935296 root 257 owner 15799573 offset 559185920 num_refs 0
not found in extent tree
Incorrect local backref count on 958298935296 root 257 owner 15799573
offset 559185920 found 2 wanted 0 back 0x15d3809a0
backpointer mismatch on [958298935296 9666560]
about 859 of those ...
Then:
owner ref check failed [25737445867520 16384]
checking free space cache
There is no free space entry for 109105479680-109105496064
There is no free space entry for 109105479680-109551026176
cache appears valid but isn't 109014155264
There is no free space entry for 139709693952-139709710336
There is no free space entry for 139709693952-140152668160
cache appears valid but isn't 139615797248
Wanted offset 171291525120, found 171291426816
Wanted offset 171291525120, found 171291426816
cache appears valid but isn't 171291181056
Wanted offset 220146597888, found 220146532352
Wanted offset 220146597888, found 220146532352
cache appears valid but isn't 220146434048
btrfs: unable to add free space :-17
free-space-cache.c:824: btrfs_add_free_space: Assertion `ret == -EEXIST`
failed.
btrfs[0x464af9]
btrfs(btrfs_add_free_space+0x154)[0x46531f]
btrfs(load_free_space_cache+0xab7)[0x465e36]
btrfs(cmd_check+0x22c7)[0x42db0e]
btrfs(main+0x155)[0x40a4fd]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7faad34cdf45]
btrfs[0x40a0f9]
and we crashed out of the check there.
-Matt
On 08/09/2016 08:06 PM, Chris Murphy wrote:
> On Tue, Aug 9, 2016 at 6:01 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com> wrote:
>>> Hello,
>>>
>>> Our server recently crashed and was rebooted. When it returned our BTRFS
>>> volume is mounting read-only:
>>
>> What happens when you try mounting with -o usebackuproot ?
>>
>> If that fails, what output do you get for 'btrfs check' (without
>> --repair)? If you only get some "errors 400, nbytes wrong" then
>> --repair should fix the problem.
>
> This could also be a regression somewhere...
> https://bugzilla.kernel.org/show_bug.cgi?id=60522
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:01 ` Chris Murphy
2016-08-10 0:06 ` Chris Murphy
@ 2016-08-10 0:17 ` Matt McKinnon
2016-08-10 7:09 ` g6094199
1 sibling, 1 reply; 12+ messages in thread
From: Matt McKinnon @ 2016-08-10 0:17 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
-o usebackuproot worked well.
after the file system settled, performing a sync and a clean umount, a
normal mount works now as well.
Anything I should be doing going forward?
Thanks,
Matt
On 08/09/2016 08:01 PM, Chris Murphy wrote:
> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com> wrote:
>> Hello,
>>
>> Our server recently crashed and was rebooted. When it returned our BTRFS
>> volume is mounting read-only:
>
> What happens when you try mounting with -o usebackuproot ?
>
> If that fails, what output do you get for 'btrfs check' (without
> --repair)? If you only get some "errors 400, nbytes wrong" then
> --repair should fix the problem.
>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 0:17 ` Matt McKinnon
@ 2016-08-10 7:09 ` g6094199
2016-08-10 11:16 ` Matt McKinnon
0 siblings, 1 reply; 12+ messages in thread
From: g6094199 @ 2016-08-10 7:09 UTC (permalink / raw)
To: Matt McKinnon, Btrfs BTRFS
Hi,
from what i see you have a non finished balance ongoing, since you have
system and metadata DUP and single information on disk.
so you should (re)run a balance for this data.
sash
Am 10.08.2016 um 02:17 schrieb Matt McKinnon:
> -o usebackuproot worked well.
>
> after the file system settled, performing a sync and a clean umount, a
> normal mount works now as well.
>
> Anything I should be doing going forward?
>
> Thanks,
> Matt
>
> On 08/09/2016 08:01 PM, Chris Murphy wrote:
>> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com>
>> wrote:
>>> Hello,
>>>
>>> Our server recently crashed and was rebooted. When it returned our
>>> BTRFS
>>> volume is mounting read-only:
>>
>> What happens when you try mounting with -o usebackuproot ?
>>
>> If that fails, what output do you get for 'btrfs check' (without
>> --repair)? If you only get some "errors 400, nbytes wrong" then
>> --repair should fix the problem.
>>
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 7:09 ` g6094199
@ 2016-08-10 11:16 ` Matt McKinnon
2016-08-10 20:43 ` g6094199
0 siblings, 1 reply; 12+ messages in thread
From: Matt McKinnon @ 2016-08-10 11:16 UTC (permalink / raw)
To: g6094199, Btrfs BTRFS
I performed a quick balance which gave me:
[39020.030638] BTRFS info (device sda1): relocating block group
25428383236096 flags 1
[39020.206097] BTRFS warning (device sda1): block group 23113395863552
has wrong amount of free space
[39020.206101] BTRFS warning (device sda1): failed to load free space
cache for block group 23113395863552, rebuilding it now
then a crash dump.
Remounted with -o clear_cache,nospace_cache and the balance completed.
Running a larger balance now.
Will umount, and remount with default options to see if that works.
-Matt
On 08/10/2016 03:09 AM, g6094199@freenet.de wrote:
> Hi,
>
> from what i see you have a non finished balance ongoing, since you have
> system and metadata DUP and single information on disk.
>
> so you should (re)run a balance for this data.
>
>
> sash
>
>
> Am 10.08.2016 um 02:17 schrieb Matt McKinnon:
>> -o usebackuproot worked well.
>>
>> after the file system settled, performing a sync and a clean umount, a
>> normal mount works now as well.
>>
>> Anything I should be doing going forward?
>>
>> Thanks,
>> Matt
>>
>> On 08/09/2016 08:01 PM, Chris Murphy wrote:
>>> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com>
>>> wrote:
>>>> Hello,
>>>>
>>>> Our server recently crashed and was rebooted. When it returned our
>>>> BTRFS
>>>> volume is mounting read-only:
>>>
>>> What happens when you try mounting with -o usebackuproot ?
>>>
>>> If that fails, what output do you get for 'btrfs check' (without
>>> --repair)? If you only get some "errors 400, nbytes wrong" then
>>> --repair should fix the problem.
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-10 11:16 ` Matt McKinnon
@ 2016-08-10 20:43 ` g6094199
0 siblings, 0 replies; 12+ messages in thread
From: g6094199 @ 2016-08-10 20:43 UTC (permalink / raw)
To: Matt McKinnon, linux-btrfs
Am 10.08.2016 um 13:16 schrieb Matt McKinnon:
> I performed a quick balance which gave me:
>
> [39020.030638] BTRFS info (device sda1): relocating block group
> 25428383236096 flags 1
> [39020.206097] BTRFS warning (device sda1): block group 23113395863552
> has wrong amount of free space
> [39020.206101] BTRFS warning (device sda1): failed to load free space
> cache for block group 23113395863552, rebuilding it now
>
> then a crash dump.
>
> Remounted with -o clear_cache,nospace_cache and the balance completed.
> Running a larger balance now.
>
> Will umount, and remount with default options to see if that works.
>
> -Matt
i guess the crash info would be very interesing for some devs here!
sash
>
> On 08/10/2016 03:09 AM, g6094199@freenet.de wrote:
>> Hi,
>>
>> from what i see you have a non finished balance ongoing, since you have
>> system and metadata DUP and single information on disk.
>>
>> so you should (re)run a balance for this data.
>>
>>
>> sash
>>
>>
>> Am 10.08.2016 um 02:17 schrieb Matt McKinnon:
>>> -o usebackuproot worked well.
>>>
>>> after the file system settled, performing a sync and a clean umount, a
>>> normal mount works now as well.
>>>
>>> Anything I should be doing going forward?
>>>
>>> Thanks,
>>> Matt
>>>
>>> On 08/09/2016 08:01 PM, Chris Murphy wrote:
>>>> On Tue, Aug 9, 2016 at 5:15 PM, Matt McKinnon <matt@techsquare.com>
>>>> wrote:
>>>>> Hello,
>>>>>
>>>>> Our server recently crashed and was rebooted. When it returned our
>>>>> BTRFS
>>>>> volume is mounting read-only:
>>>>
>>>> What happens when you try mounting with -o usebackuproot ?
>>>>
>>>> If that fails, what output do you get for 'btrfs check' (without
>>>> --repair)? If you only get some "errors 400, nbytes wrong" then
>>>> --repair should fix the problem.
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-08-11 1:51 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-08-09 23:15 BTRFS: error (device sda1) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists Matt McKinnon
2016-08-10 0:01 ` Chris Murphy
2016-08-10 0:06 ` Chris Murphy
2016-08-10 0:29 ` Matt McKinnon
2016-08-10 2:25 ` Chris Murphy
2016-08-10 6:41 ` Duncan
2016-08-11 1:51 ` Matt McKinnon
2016-08-10 0:56 ` Matt McKinnon
2016-08-10 0:17 ` Matt McKinnon
2016-08-10 7:09 ` g6094199
2016-08-10 11:16 ` Matt McKinnon
2016-08-10 20:43 ` g6094199
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.