Re: BTRFS free space handling still needs more work: Hangs again (no complete lockups, "just" tasks stuck for some time)

[Martin Steigerwald <Martin@lichtvoll.de>]

[-- Attachment #1: Type: text/plain, Size: 5782 bytes --]

Am Samstag, 27. Dezember 2014, 18:40:17 schrieb Hugo Mills:
> On Sat, Dec 27, 2014 at 01:28:46PM -0500, Zygo Blaxell wrote:
> > On Sat, Dec 27, 2014 at 09:30:43AM +0000, Hugo Mills wrote:
> > > On Sat, Dec 27, 2014 at 10:01:17AM +0100, Martin Steigerwald wrote:
> > > > Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:
> > > > > On 12/26/2014 05:37 AM, Martin Steigerwald wrote:
> > >    Now, since you're seeing lockups when the space on your disks is
> > > all allocated I'd say that's a bug. However, you're the *only* person
> > > who's reported this as a regular occurrence. Does this happen with all
> > > filesystems you have, or just this one?
> > 
> > I do see something similar, but there are so many problems going on I
> > have no idea which ones to report, and which ones are my own doing.  :-P
> > 
> > I see lots of CPU being burned when all the disk space is allocated
> > to chunks, but there is still lots of space free (multiple GB) inside
> > the chunks.
> > 
> > iotop shows a crapton of disk writes (1-5MB/sec) from one kworker.
> > There are maybe a few kB/sec of writes through the filesystem at the time.
> > 
> > The filesystem where I see this most is on a laptop, so the disk writes
> > also hit the CPU again for encryption.  There's so much CPU usage it's
> > worth mentioning twice.  :-(
> > 
> > 'watch cat /proc/12345/stack' on the active processes shows the kernel
> > fairly often in that new chunk deallocator function whose name escapes
> > me at the moment.
> > 
> > Deleting a bunch of data then running balance helps return to sane CPU
> > usage...for a while (maybe a week?).
> > 
> > It's not technically "locked up" per se, but when a 5KB download takes
> > a minute or more, most users won't wait around to see the difference.
> > 
> > Kernel versions I'm using are 3.17.7 and 3.18.1.
> 
>    OK, so I'd like to change my statement above.
> 
>    When I first read Martin's problem, I thought that he was referring
> to a complete, hit-the-power-button kind of lock-up. Given that
> (erroneous) assumption, I stand by my (now pointless) statement. :)
> 
>    I realised during a brief conversation on IRC that Martin was
> actually referring to long but temporary periods where the machine is
> unusable by any process requiring disk activity. There's clearly a
> number of people seeing that.
> 
>    It doesn't stop it being a major problem, but it does change the
> interpretation considerably.

Ah, then my bet was right with whom I talked there. :)

Yeah, it does not seem to be a complete hang, I though so initially, cause
honestly after waiting several minutes for my Plasma desktop to come back
I just gave up. Maybe it would have returned at some time. I just didn´t
have the patience to wait.

It now did at my last testing where I continued on tty1 (had all the testing
in a screen) as the desktop session locked up. After some time after the
test completed I was able to use that desktop again and I am still using it.

So the issue I see is: One kworker uses 100% of one core for minutes and
while doing so processes that do I/O to the BTRFS that I test (/home) in my
case seem to be stuck in uninteruptible sleep ("D" process state). While I
see this there is no huge load on the SSDs so… it seems to be something
CPU bound. I didn´t yet use a strace on the kworker process – or at the
allocation time on the fio process –, Robert, thats a good suggestion. From
a gut feeling I wouldn´t be surprised if I see *nothing* in strace as my bet
is that the kworker thread deals with finding free space inside the chunks
and deals with some data structures while doing so. But that is really just
a gut feeling and so an strace would be nice.

I made a backup yesterday, so I think I can try the strace. But I also spend
a considerable amount of time of reproducing it and digging deeper into it
so likely not this weekend anymore although this even makes some fun. But
I see myself neglecting other stuff thats important to me as well, so…

My simple test case didn´t trigger it, and I so not have another twice 160
GiB available on this SSDs available to try with a copy of my home
filesystem. Then I could safely test without bringing the desktop session to
an halt. Maybe someone has an idea on how to "enhance" my test case in
order to reliably trigger the issue.

It may be challenging tough. My /home is quite a filesystem. It has a maildir
with at least one million of files (yeah, I am performance testing KMail and
Akonadi as well to the limit!), and it has git repos and this one VM image,
and the desktop search and the Akonadi database. In other words: It has
been hit nicely with various mostly random I think workloads over the last
about six months. I bet its not that easy to simulate that. Maybe some runs
of compilebench to age the filesystem before the fio test?

That said, BTRFS performs a lot better. The complete lockups without any
CPU usage of 3.15 and 3.16 have gone for sure. Thats wonderful. But there
is this kworker issue now. I noticed it that gravely just while trying to
complete this tax returns stuff with the Windows XP VM. Otherwise it may
have happened, I have seen some backtraces in kern.log, but it didn´t last
for minutes. So this indeed is of less severity than the full lockups with
3.15 and 3.16.

Zygo, was is the characteristics of your filesystem. Do you use
compress=lzo and skinny metadata as well? How are the chunks allocated?
What kind of data you have on it?

Well now off to some dancing event. Thats just right now :)

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 181 bytes --]