* block rsv returned -28
@ 2012-07-07 12:23 Jan Engelhardt
0 siblings, 0 replies; 3+ messages in thread
From: Jan Engelhardt @ 2012-07-07 12:23 UTC (permalink / raw)
To: linux-btrfs
- RAID10 btrfs volume consisting of 4 disks.
- One disk failed, was replaced, resync started
(`btrfs dev add /dev/sdf /srv; btrfs dev del missing /srv`)
- Another disk failed before resync was done.
Disk was replaced, resync restarted.
(`btrfs dev add /dev/sdc /srv; btrfs dev del missing /srv`)
Naturally I don't expect it to recover from 2 failures, but
doing the attempt was worth a try, for this throws kernel warnings
(below).
Can I somehow just have all inodes unlinked from the filesystem that are
not recoverable? That would be preferable to re-mkfs.
The current status per fi show:
Label: 'srv' uuid: 88300cd5-dbcb-4147-9ee4-c65a1c895e1d
Total devices 6 FS bytes used 1.71TiB
devid 7 size 1.36TiB used 29.50GiB path /dev/sdc
devid 2 size 1.36TiB used 1.13TiB path /dev/sdd
devid 3 size 1.36TiB used 1.13TiB path /dev/sde
devid 6 size 1.36TiB used 377.00GiB path /dev/sdf
*** Some devices missing
[ 1013.381148] btrfs: found 5451 extents
[ 1105.177601] btrfs: block rsv returned -28
[ 1105.177608] ------------[ cut here ]------------
[ 1105.177684] WARNING: at /home/abuild/rpmbuild/BUILD/kernel-default-3.4.4~jng6/linux-3.4/fs/btrfs/extent-tree.c:6219 use_block_rsv+0xb8/0x104 [btrfs]()
[ 1105.177692] Hardware name: RX945GC
[ 1105.177696] Modules linked in: f71882fg nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc bonding mperf btrfs zlib_deflate crc32c libcrc32c snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer i915 snd drm_kms_helper drm coretemp sg intel_agp led_class soundcore snd_page_alloc r8169 iTCO_wdt iTCO_vendor_support microcode i2c_i801 i2c_algo_bit intel_gtt agpgart i2c_core button pcspkr mii rng_core video edd autofs4 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid0 raid1 md_mod ata_piix uhci_hcd ehci_hcd usbcore usb_common processor thermal_sys hwmon
[ 1105.177810] Pid: 2912, comm: btrfs Not tainted 3.4.4-jng6-default #1
[ 1105.177815] Call Trace:
[ 1105.177842] [<ffffffff81003b0e>] dump_trace+0x89/0x1e9
[ 1105.177858] [<ffffffff813bdd45>] dump_stack+0x69/0x6f
[ 1105.177871] [<ffffffff8102d9d4>] warn_slowpath_common+0x78/0x8c
[ 1105.177937] [<ffffffffa02fcaf6>] use_block_rsv+0xb8/0x104 [btrfs]
[ 1105.178092] [<ffffffffa02fef88>] btrfs_alloc_free_block+0x29/0x1bd [btrfs]
[ 1105.178236] [<ffffffffa02f17fe>] __btrfs_cow_block+0x10e/0x350 [btrfs]
[ 1105.178338] [<ffffffffa02f1b34>] btrfs_cow_block+0xf4/0x13f [btrfs]
[ 1105.178441] [<ffffffffa02f4753>] btrfs_search_slot+0x230/0x531 [btrfs]
[ 1105.178581] [<ffffffffa0342990>] do_relocation+0x104/0x3fe [btrfs]
[ 1105.179050] [<ffffffffa0342d29>] finish_pending_nodes+0x9f/0x103 [btrfs]
[ 1105.179518] [<ffffffffa03456b5>] relocate_tree_blocks+0x124/0x14a [btrfs]
[ 1105.179994] [<ffffffffa034624e>] relocate_block_group+0x21a/0x439 [btrfs]
[ 1105.180774] [<ffffffffa03465ac>] btrfs_relocate_block_group+0x13f/0x267 [btrfs]
[ 1105.181471] [<ffffffffa032b849>] btrfs_relocate_chunk.isra.51+0x4b/0x2ce [btrfs]
[ 1105.182037] [<ffffffffa032e532>] btrfs_shrink_device+0x1fa/0x34d [btrfs]
[ 1105.182623] [<ffffffffa033011f>] btrfs_rm_device+0x1f4/0x4a6 [btrfs]
[ 1105.183206] [<ffffffffa0331ad2>] btrfs_ioctl_rm_dev+0x95/0xbe [btrfs]
[ 1105.183797] [<ffffffffa0335632>] btrfs_ioctl+0x3b4/0x6d6 [btrfs]
[ 1105.184369] [<ffffffff8110f9af>] do_vfs_ioctl+0x240/0x254
[ 1105.184390] [<ffffffff8110fa0e>] sys_ioctl+0x4b/0x71
[ 1105.184412] [<ffffffff813d8b79>] system_call_fastpath+0x16/0x1b
[ 1105.184455] [<00007fb47191a837>] 0x7fb47191a836
[ 1105.184464] ---[ end trace 7a7259f915d317e1 ]---
(many of these as it attempted to resync/rebalance)
^ permalink raw reply [flat|nested] 3+ messages in thread
* block rsv returned -28
@ 2013-01-06 10:28 TM
2013-01-06 13:02 ` TM
0 siblings, 1 reply; 3+ messages in thread
From: TM @ 2013-01-06 10:28 UTC (permalink / raw)
To: linux-btrfs
Hi all,
in a newly created btrfs filesystem, just after two days and ~4K dirs, 40K
files, performance has degraded very bad.
I did a chmod / chown the files once (that might have implications on the
filesystem) but this is casual/expected use.
Only notable thing in dmesg is [9 times]
btrfs: block rsv returned -28
------------[ cut here ]------------
WARNING: at fs/btrfs/extent-tree.c:6297 use_block_rsv+0x192/0x1a0 [btrfs]()
Hardware name: ProLiant MicroServer
Modules linked in: bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt
vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) 8021q vboxdrv(OF) garp stp llc sunrpc
ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack
ip6table_filter ip6_tables ipv6 btrfs zlib_deflate libcrc32c kvm_amd kvm
microcode pcspkr k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 tg3 sg
shpchp ext4 mbcache jbd2 sd_mod crc_t10dif ahci libahci radeon ttm
drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: nf_defrag_ipv4]
Pid: 1859, comm: btrfs-transacti Tainted: GF W O 3.7.1 #1
Call Trace:
[<ffffffff8105556f>] warn_slowpath_common+0x7f/0xc0
[<ffffffff810555ca>] warn_slowpath_null+0x1a/0x20
[<ffffffffa0406e02>] use_block_rsv+0x192/0x1a0 [btrfs]
[<ffffffffa040bb9d>] btrfs_alloc_free_block+0x3d/0x210 [btrfs]
[<ffffffffa0432ab1>] ? read_extent_buffer+0xd1/0x130 [btrfs]
[<ffffffffa03f6eb0>] __btrfs_cow_block+0x130/0x560 [btrfs]
[<ffffffffa03f7942>] btrfs_cow_block+0x102/0x210 [btrfs]
[<ffffffffa03fad11>] btrfs_search_slot+0x391/0x810 [btrfs]
[<ffffffffa04577f7>] __btrfs_write_out_cache+0x757/0x960 [btrfs]
[<ffffffffa045b3ae>] ? btrfs_find_ref_cluster+0x5e/0x160 [btrfs]
[<ffffffffa0457b72>] btrfs_write_out_cache+0xb2/0xf0 [btrfs]
[<ffffffffa0409fe8>] btrfs_write_dirty_block_groups+0x238/0x270 [btrfs]
[<ffffffffa041a2c1>] commit_cowonly_roots+0x171/0x250 [btrfs]
[<ffffffffa041b120>] btrfs_commit_transaction+0x570/0xa20 [btrfs]
[<ffffffffa041ba84>] ? start_transaction+0x94/0x430 [btrfs]
[<ffffffff81079cb0>] ? wake_up_bit+0x40/0x40
[<ffffffffa0415d26>] transaction_kthread+0x1a6/0x220 [btrfs]
[<ffffffffa0415b80>] ? btree_readpage_end_io_hook+0x290/0x290 [btrfs]
[<ffffffffa0415b80>] ? btree_readpage_end_io_hook+0x290/0x290 [btrfs]
[<ffffffff8107942e>] kthread+0xce/0xe0
[<ffffffff81079360>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff8153e6ac>] ret_from_fork+0x7c/0xb0
[<ffffffff81079360>] ? kthread_freezable_should_stop+0x70/0x70
---[ end trace 592323d6a331318d ]---
Kernel:
Linux microserver 3.7.1 #1 SMP Sun Dec 30 21:34:59 EET 2012 x86_64 x86_64 x86_64
GNU/Linux
Toos:
Btrfs v0.20-rc1-37-g91d9eec
# btrfs fi show
Label: 'tm_0' uuid: f2866a33-fe53-4fc0-98bf-52d347b43824
Total devices 1 FS bytes used 712.21GB
devid 1 size 1.82TB used 714.04GB path /dev/sdb1
# btrfs fi df /mnt/dls
Data: total=712.01GB, used=711.24GB
System, DUP: total=8.00MB, used=80.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=998.53MB
Metadata: total=8.00MB, used=0.00
Performance is horrible for the last 12 hours at least
I tried remounting with noatime nothing changed
I tried rebooting, stopping any services like smb etc.
I tried defragmenting nothing changed
Metadata seems full
Darksatanic suggested That's ENOSPC
(Thanks to the guys at irc chanel for their support :)
But how can I regain space on Metadata?
Any suggestions?
TM
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: block rsv returned -28
2013-01-06 10:28 block rsv returned -28 TM
@ 2013-01-06 13:02 ` TM
0 siblings, 0 replies; 3+ messages in thread
From: TM @ 2013-01-06 13:02 UTC (permalink / raw)
To: linux-btrfs
TM <tmjuju <at> yahoo.com> writes:
# btrfs fi df /mnt/dls
Data: total=712.01GB, used=711.24GB
System, DUP: total=8.00MB, used=80.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=998.53MB
Metadata: total=8.00MB, used=0.00
btrfs fi show
Label: 'tm_0' uuid: f2866a33-fe53-4fc0-98bf-52d347b43824
Total devices 1 FS bytes used 712.21GB
devid 1 size 1.82TB used 714.04GB path /dev/sdb1
...
[root@microserver mnt]# btrfs balance start -dusage=5 /mnt/dls/
Done, had to relocate 0 out of 716 chunks
[root@microserver mnt]# btrfs balance start -dusage=10 /mnt/dls/
Done, had to relocate 0 out of 716 chunks
[root@microserver mnt]# btrfs balance start -dusage=15 /mnt/dls/
Done, had to relocate 0 out of 716 chunks
[root@microserver mnt]# btrfs balance start -dusage=25 /mnt/dls/
Done, had to relocate 0 out of 716 chunks
[root@microserver mnt]# btrfs balance start -dusage=35 /mnt/dls/
Done, had to relocate 1 out of 716 chunks
...
Meanwhile situation got worst
..
[root@microserver ~]# btrfs fi df /mnt/dls/
Data: total=713.01GB, used=712.23GB
System, DUP: total=8.00MB, used=80.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=999.31MB
Metadata: total=8.00MB, used=0.00
and I kept getting more messages
dmesg |grep "block rsv returned -28" |wc -l
60
(I had stared with only 9 messages just a few hours ago)
So I decided to remove files
I started to remove (rm -rfv) thousands of files and directories
At first the rate was about 1 filedelete per second...
Always monitoring usage while deleteing
Utilll I got down to:
# btrfs fi df /mnt/dls/
Data: total=712.01GB, used=638.04GB
System, DUP: total=8.00MB, used=80.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=895.24MB
Metadata: total=8.00MB, used=0.00
And then the filesystem became responsive again
Back to 100MBps range from ~1MBps
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-01-06 13:02 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-06 10:28 block rsv returned -28 TM
2013-01-06 13:02 ` TM
-- strict thread matches above, loose matches on Subject: below --
2012-07-07 12:23 Jan Engelhardt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox