From: Phil Karn <karn@ka9q.net>
To: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Cc: Chris Murphy <lists@colorremedies.com>,
Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: Extremely slow device removals
Date: Thu, 30 Apr 2020 21:48:05 -0700 [thread overview]
Message-ID: <b2cd0c70-b955-197c-d68b-cf77e102690c@ka9q.net> (raw)
In-Reply-To: <20200501024753.GE10769@hungrycats.org>
On 4/30/20 19:47, Zygo Blaxell wrote:
>
> If it keeps repeating "found 1115 extents" over and over (say 5 or
> more times) then you're hitting the balance looping bug in kernel 5.1
> and later. Every N block groups (N seems to vary by user, I've heard
> reports from 3 to over 6000) the kernel will get stuck in a loop and
> will need to reboot to recover. Even if you cancel the balance, it will
> just loop again until rebooted, and there's no cancel for device delete
> so if you start looping there you can just skip directly to the reboot.
> For a non-trivial filesystem the probability of successfully deleting
> or resizing a device is more or less zero.
This does not seem to be happening. Each message is for a different
block group with a different number of clusters. The device remove *is*
making progress, just very very slowly. I'm almost down to just 2TB
left. Woot!
If I ever have to do this again, I'll insert bcache and a big SSD
between btrfs and my devices. The slowness here has to be due to the
(spinning) disk I/O being highly fragmented and random. I've checked,
and none of my drives (despite their large sizes) are shingled, so
that's not it. The 6 TB units have 128 MB caches and the 16 TB have 256
MB caches.
I've never understood *exactly* what a hard drive internal cache does. I
see little sense in a LRU cache just like the host's own buffer cache
since the host has far more RAM. I do know they're used to reorder
operations to reduce seek latency, though this can be limited by the
need to fence writes to protect against a crash. I've wondered if
they're also used on reads to reduce rotational latency by prospectively
grabbing data as soon as the heads land on a cylinder. How big is a
"cylinder'' anyway? The inner workings of hard drives have become
steadily more opaque over the years, which makes it difficult to
optimize their use. Kinda like CPUs, actually. Last time I really tuned
up some tight code, I found that using vector instructions and avoiding
branch mispredictions made a big difference but nothing else seemed to
matter at all.
>
> There is no fix for that regression yet. Kernel 4.19 doesn't have the
> regression and does have other relevant bug fixes for balance, so it
> can be used as a workaround.
I'm running 4.19.0-8-rt-amd64, the current real-time kernel in Debian
'stable'.
Phil
next prev parent reply other threads:[~2020-05-01 4:48 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-28 7:22 Extremely slow device removals Phil Karn
2020-04-30 17:31 ` Phil Karn
2020-04-30 18:13 ` Jean-Denis Girard
2020-05-01 8:05 ` Phil Karn
2020-05-02 3:35 ` Zygo Blaxell
[not found] ` <CAMwB8mjUw+KV8mxg8ynPsv0sj5vSpwG7_khw=oP5n+SnPYzumQ@mail.gmail.com>
2020-05-02 4:31 ` Zygo Blaxell
2020-05-02 4:48 ` Paul Jones
2020-05-02 5:25 ` Phil Karn
2020-05-02 6:04 ` Remi Gauvin
2020-05-02 7:20 ` Zygo Blaxell
2020-05-02 7:27 ` Phil Karn
2020-05-02 7:52 ` Zygo Blaxell
2020-05-02 6:00 ` Zygo Blaxell
2020-05-02 6:23 ` Paul Jones
2020-05-02 7:20 ` Phil Karn
2020-05-02 7:42 ` Zygo Blaxell
2020-05-02 8:22 ` Phil Karn
2020-05-02 8:24 ` Phil Karn
2020-05-02 9:09 ` Zygo Blaxell
2020-05-02 17:48 ` Chris Murphy
2020-05-03 5:26 ` Zygo Blaxell
2020-05-03 5:39 ` Chris Murphy
2020-05-03 6:05 ` Chris Murphy
2020-05-04 2:09 ` Phil Karn
2020-05-02 7:43 ` Jukka Larja
2020-05-02 4:49 ` Phil Karn
2020-04-30 18:40 ` Chris Murphy
2020-04-30 19:59 ` Phil Karn
2020-04-30 20:27 ` Alexandru Dordea
2020-04-30 20:58 ` Phil Karn
2020-05-01 2:47 ` Zygo Blaxell
2020-05-01 4:48 ` Phil Karn [this message]
2020-05-01 6:05 ` Alexandru Dordea
2020-05-01 7:29 ` Phil Karn
2020-05-02 4:18 ` Zygo Blaxell
2020-05-02 4:48 ` Phil Karn
2020-05-02 5:00 ` Phil Karn
2020-05-03 2:28 ` Phil Karn
2020-05-04 7:39 ` Phil Karn
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b2cd0c70-b955-197c-d68b-cf77e102690c@ka9q.net \
--to=karn@ka9q.net \
--cc=ce3g8jdj@umail.furryterror.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=lists@colorremedies.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).