From: jb@capsec.org
To: linux-btrfs@vger.kernel.org
Subject: Very slow balance / btrfs-transaction
Date: Fri, 03 Feb 2017 22:13:31 +0000 [thread overview]
Message-ID: <80d3e5ce55ddc7e454cce96e67e2ea64@88cbed2449cf> (raw)
Hi,
I'm currently running a balance (without any filters) on a 4 drives raid1 filesystem. The array contains 3 3TB drives and one 6TB drive; I'm running the rebalance because the 6TB drive recently replaced a 2TB drive.
I know that balance is not supposed to be a fast operation, but this one is now running for ~6 days and it managed to balance ~18% (754 out of about 4250 chunks balanced (755 considered), 82% left) -- so I expect it to take another ~4 weeks.
That seems excessively slow for ~8TiB of data.
Is this expected behavior? In case it's not: Is there anything I can do to help debug it?
The 4 individual devices are bcache devices with currently no ssd cache partition attached; the bcache backing devices sit ontop of luks encrypted devices. Maybe a few words about the history of this fs: It used to be a 1 drive btrfs ontop of a bcache partition with a 30GiB SSD cache (actively used for >1 year). During the last month, I gradually added devices (always with active bcaches). At some point, after adding the 4th device, I deactivated (detached) the bcache caching device and instead activated raid1 for data and metadata and ran a rebalance (which was reasonably fast -- I don't remember how fast exactly, but probably <24h). The finaly steps that lead to the current situation: I activated "nossd" and replaced the smallest device with "btrfs dev replace" (which was also reasonabley fast, <12h).
Best & thanks,
j
--
[joerg@dorsal ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 110.8G 0 part
└─crypted 254:0 0 110.8G 0 crypt
├─ssd-root 254:1 0 72.8G 0 lvm /
├─ssd-swap 254:2 0 8G 0 lvm [SWAP]
└─ssd-cache 254:3 0 30G 0 lvm
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
└─crypted-sdb 254:7 0 2.7T 0 crypt
└─bcache2 253:2 0 2.7T 0 disk
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 2.7T 0 part
└─crypted-sdc 254:4 0 2.7T 0 crypt
└─bcache1 253:1 0 2.7T 0 disk
sdd 8:48 0 2.7T 0 disk
└─sdd1 8:49 0 2.7T 0 part
└─crypted-sdd 254:6 0 2.7T 0 crypt
└─bcache0 253:0 0 2.7T 0 disk
sde 8:64 0 5.5T 0 disk
└─sde1 8:65 0 5.5T 0 part
└─crypted-sde 254:5 0 5.5T 0 crypt
└─bcache3 253:3 0 5.5T 0 disk /storage
--
joerg@dorsal ~]$ sudo btrfs fi usage -h /storage/
Overall:
Device size: 13.64TiB
Device allocated: 8.35TiB
Device unallocated: 5.29TiB
Device missing: 0.00B
Used: 8.34TiB
Free (estimated): 2.65TiB (min: 2.65TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 15.77MiB)
Data,RAID1: Size:4.17TiB, Used:4.16TiB
/dev/bcache0 2.38TiB
/dev/bcache1 2.37TiB
/dev/bcache2 2.38TiB
/dev/bcache3 1.20TiB
Metadata,RAID1: Size:9.00GiB, Used:7.49GiB
/dev/bcache1 8.00GiB
/dev/bcache2 1.00GiB
/dev/bcache3 9.00GiB
System,RAID1: Size:32.00MiB, Used:624.00KiB
/dev/bcache1 32.00MiB
/dev/bcache3 32.00MiB
Unallocated:
/dev/bcache0 355.52GiB
/dev/bcache1 356.49GiB
/dev/bcache2 355.52GiB
/dev/bcache3 4.25TiB
--
[joerg@dorsal ~]$ ps -xal | grep btrfs
1 0 227 2 0 -20 0 0 - S< ? 0:00 [btrfs-worker]
1 0 229 2 0 -20 0 0 - S< ? 0:00 [btrfs-worker-hi]
1 0 230 2 0 -20 0 0 - S< ? 0:00 [btrfs-delalloc]
1 0 231 2 0 -20 0 0 - S< ? 0:00 [btrfs-flush_del]
1 0 232 2 0 -20 0 0 - S< ? 0:00 [btrfs-cache]
1 0 233 2 0 -20 0 0 - S< ? 0:00 [btrfs-submit]
1 0 234 2 0 -20 0 0 - S< ? 0:00 [btrfs-fixup]
1 0 235 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio]
1 0 236 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-met]
1 0 237 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-met]
1 0 238 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-rai]
1 0 239 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-rep]
1 0 240 2 0 -20 0 0 - S< ? 0:00 [btrfs-rmw]
1 0 241 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-wri]
1 0 242 2 0 -20 0 0 - S< ? 0:00 [btrfs-freespace]
1 0 243 2 0 -20 0 0 - S< ? 0:00 [btrfs-delayed-m]
1 0 244 2 0 -20 0 0 - S< ? 0:00 [btrfs-readahead]
1 0 245 2 0 -20 0 0 - S< ? 0:00 [btrfs-qgroup-re]
1 0 246 2 0 -20 0 0 - S< ? 0:00 [btrfs-extent-re]
1 0 247 2 20 0 0 0 - S ? 0:00 [btrfs-cleaner]
1 0 248 2 20 0 0 0 - S ? 0:30 [btrfs-transacti]
1 0 2283 2 0 -20 0 0 - S< ? 0:00 [btrfs-worker]
1 0 2285 2 0 -20 0 0 - S< ? 0:00 [btrfs-worker-hi]
1 0 2286 2 0 -20 0 0 - S< ? 0:00 [btrfs-delalloc]
1 0 2287 2 0 -20 0 0 - S< ? 0:00 [btrfs-flush_del]
1 0 2288 2 0 -20 0 0 - S< ? 0:00 [btrfs-cache]
1 0 2289 2 0 -20 0 0 - S< ? 0:00 [btrfs-submit]
1 0 2290 2 0 -20 0 0 - S< ? 0:00 [btrfs-fixup]
1 0 2291 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio]
1 0 2292 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-met]
1 0 2293 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-met]
1 0 2294 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-rai]
1 0 2295 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-rep]
1 0 2296 2 0 -20 0 0 - S< ? 0:00 [btrfs-rmw]
1 0 2297 2 0 -20 0 0 - S< ? 0:00 [btrfs-endio-wri]
1 0 2298 2 0 -20 0 0 - S< ? 0:00 [btrfs-freespace]
1 0 2299 2 0 -20 0 0 - S< ? 0:00 [btrfs-delayed-m]
1 0 2300 2 0 -20 0 0 - S< ? 0:00 [btrfs-readahead]
1 0 2301 2 0 -20 0 0 - S< ? 0:00 [btrfs-qgroup-re]
1 0 2302 2 0 -20 0 0 - S< ? 0:00 [btrfs-extent-re]
1 0 2303 2 20 0 0 0 - D ? 0:10 [btrfs-cleaner]
1 0 2304 2 20 0 0 0 - D ? 3247:49 [btrfs-transacti]
4 0 10316 9321 20 0 83604 4868 - S+ pts/1 0:00 sudo btrfs balance start /storage/
4 0 10317 10316 20 0 15788 1044 - R+ pts/1 5352:22 btrfs balance start /storage/
0 1000 17901 13293 20 0 12288 2136 - R+ pts/5 0:00 grep --color=auto btrfs
next reply other threads:[~2017-02-03 22:20 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-03 22:13 jb [this message]
2017-02-03 23:25 ` Very slow balance / btrfs-transaction Goldwyn Rodrigues
2017-02-04 0:30 ` Jorg Bornschein
2017-02-04 1:07 ` Goldwyn Rodrigues
2017-02-04 1:47 ` Jorg Bornschein
2017-02-04 2:55 ` Lakshmipathi.G
2017-02-04 8:22 ` Duncan
2017-02-06 1:45 ` Qu Wenruo
2017-02-06 16:09 ` Goldwyn Rodrigues
2017-02-07 0:22 ` Qu Wenruo
2017-02-07 15:55 ` Filipe Manana
2017-02-08 0:39 ` Qu Wenruo
2017-02-08 13:56 ` Filipe Manana
2017-02-09 1:13 ` Qu Wenruo
2017-02-06 9:14 ` Jorg Bornschein
2017-02-06 9:29 ` Qu Wenruo
2017-02-04 20:50 ` Jorg Bornschein
2017-02-04 21:10 ` Kai Krakow
2017-02-06 13:19 ` Austin S. Hemmelgarn
2017-02-07 19:47 ` Kai Krakow
2017-02-07 19:58 ` Austin S. Hemmelgarn
-- strict thread matches above, loose matches on Subject: below --
2017-07-01 14:24 Sidney San Martín
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=80d3e5ce55ddc7e454cce96e67e2ea64@88cbed2449cf \
--to=jb@capsec.org \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).