unclean shutdown and space cache rebuild

unclean shutdown and space cache rebuild

[Shridhar Daithankar]

Hello,

I have 3 partitions with btrfs(/, /home and /data). All of them have following 
mount options

noatime,space_cache,inode_cache,compress=lzo,defaults

Whenever there is a unclean shutdown(which happens a lot in my case), the next 
reboot, system comes up relatively at the same speed but as systemd is 
starting up daemons, the disk is continuously ( and unusally long) grinding.

This causes random delays with various daemons such as postgresql failing to 
start, kdm timing out of xorg servers etc and I have to reboot after the dust 
settles to bring back the system to the normal. At one time, even 
keyboad/mouse were not responding as some debus service timed out..

I think it is rebuilding the space cache because I saw similar long disk 
activity when I activated it first.

How can I confirm that it is the space cache rebuild thats taking time?

if space cache rebuild is the reason, is there any way to improve it? 

I am running archlinux/systemd/kde setup with two 7200 RPM seagate sata 
disks(no RAID, one 80 GB for / and /home, other 500GB for data). The kernel is 
3.9.8 x86_64.

Thanks.

-- 
Regards
 Shridhar

Re: unclean shutdown and space cache rebuild

[Garry T. Williams]

On 6-30-13 19:26:16 Shridhar Daithankar wrote:
> Whenever there is a unclean shutdown(which happens a lot in my
> case), the next reboot, system comes up relatively at the same speed
> but as systemd is starting up daemons, the disk is continuously (and
> unusally long) grinding.

[snip]

> How can I confirm that it is the space cache rebuild thats taking
> time?
> 
> if space cache rebuild is the reason, is there any way to improve
> it?
> 
> I am running archlinux/systemd/kde

I suspect this is, at least in part, related to severe fragmentation
in /home.

There are large files in these directories that are updated frequently
by various components of KDE and the Chrome browser.  (Firefox has its
own databases that are frequently updated, too.)

    ~/.local/share/akonadi
    ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend
    ~/.cache/chromium/Default/Cache
    ~/.cache/chromium/Default/Media\ Cache

I improved performance dramatically (orders of magnitude) by copying
the database files into an empty file that was modified with:

    chattr -C

and renaming to make the files no COW.  (Note that this is the only
way to change an existing file to no COW.)  I also set the same
attribute on the owning directories so that all new files inherit the
no COW attribute.

I suspect there are other files that fragment badly since I see
periods of high disk activity coming back slowly over a few weeks of
use after making the modifications above.  I intend to track them down
and do the same.

Also, see these:

    https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work 
    https://btrfs.wiki.kernel.org/index.php/UseCases#How_do_I_defragment_many_files.3F 

    $ uname -r
    3.9.6-200.fc18.x86_64
    $

-- 
Garry T. Williams

Re: unclean shutdown and space cache rebuild

[Pete]

On 06/30/2013 06:53 PM, Garry T. Williams wrote:
> On 6-30-13 19:26:16 Shridhar Daithankar wrote:
>> Whenever there is a unclean shutdown(which happens a lot in my
>> case), the next reboot, system comes up relatively at the same speed
>> but as systemd is starting up daemons, the disk is continuously (and
>> unusally long) grinding.
>

>> I am running archlinux/systemd/kde
>
> I suspect this is, at least in part, related to severe fragmentation
> in /home.
>

I'm wondering if this is affecting myself.  I have a big issue with my 
data drive slowing down and there being near long periods of high disk 
IO that prevent me doing anything else.  I've noticed from iotop various 
btrfs processes hogging the IO for long periods, e.g. 
btrfs-transacti... & btrfs-submit

I've been running kde which has got unusable (not from reboot, but in 
general).  xfce is less hampered but IO still seems like an issue at 
times.  Of course, xfce hits different files.  I've been using this file 
system a couple of months and not defragged before.  I started 
defragging the various subvolumes a week or two ago - but I did not 
realise this was not recursive until this weekend.  I've got a python 
script running defrag on various files and folders - I can better track 
what it is defragging.  But it is _slow_ many many minutes for a rarely 
accessed folder with little content.  Is this normal?

I too had an issue with unclean shutdowns.  I, relatively infrequently, 
get lockups.  However, I had a spate last week which I have yet to 
resolve.  I wonder if that is related.

I wonder, if I defrag everything on say a weekly basis then will these 
performance issues go away?  Running a 3.9.3 kernel.

Pete



> There are large files in these directories that are updated frequently
> by various components of KDE and the Chrome browser.  (Firefox has its
> own databases that are frequently updated, too.)
>
>      ~/.local/share/akonadi
>      ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend
>      ~/.cache/chromium/Default/Cache
>      ~/.cache/chromium/Default/Media\ Cache
>
> I improved performance dramatically (orders of magnitude) by copying
> the database files into an empty file that was modified with:
>
>      chattr -C
>
> and renaming to make the files no COW.  (Note that this is the only
> way to change an existing file to no COW.)  I also set the same
> attribute on the owning directories so that all new files inherit the
> no COW attribute.
>
> I suspect there are other files that fragment badly since I see
> periods of high disk activity coming back slowly over a few weeks of
> use after making the modifications above.  I intend to track them down
> and do the same.
>
> Also, see these:
>
>      https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work
>      https://btrfs.wiki.kernel.org/index.php/UseCases#How_do_I_defragment_many_files.3F
>
>      $ uname -r
>      3.9.6-200.fc18.x86_64
>      $
>

Re: unclean shutdown and space cache rebuild

[Clemens Eisserer]

Hi,

> I suspect this is, at least in part, related to severe fragmentation
> in /home.

In his cause those issues are only present after an unclean shutdown -
whereas fragmentation would show its effect after every reboot.

Regards, Clemens

Re: unclean shutdown and space cache rebuild

[Duncan]

Garry T. Williams posted on Sun, 30 Jun 2013 13:53:48 -0400 as excerpted:

> I suspect this is, at least in part, related to severe fragmentation in
> /home.
> 
> There are large files in these directories that are updated frequently
> by various components of KDE and the Chrome browser.  (Firefox has its
> own databases that are frequently updated, too.)
> 
>     ~/.local/share/akonadi
>     ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend
>     ~/.cache/chromium/Default/Cache ~/.cache/chromium/Default/Media\
>     Cache
> 
> I improved performance dramatically (orders of magnitude) by copying the
> database files into an empty file that was modified with:
> 
>     chattr -C
> 
> and renaming to make the files no COW.  (Note that this is the only way
> to change an existing file to no COW.)  I also set the same attribute on
> the owning directories so that all new files inherit the no COW
> attribute.
> 
> I suspect there are other files that fragment badly since I see periods
> of high disk activity coming back slowly over a few weeks of use after
> making the modifications above.  I intend to track them down and do the
> same.

This definitely won't be practical for everyone, but...

1) I run kde here, but switched away from kmail, akregator, basically 
anything kdepim related, when that akonadified.  I had been using kmail 
for nearly a decade, and it had converted MSOE mail in it from before the 
turn of the century (!!), but one day when akonadi simply lost an email 
for the Nth time in so many days, the question occurred to me, why do I 
put up with this when there's so many sane alternatives?  Yes, I could 
have probably recovered that mail as I had others by redoing the akonadi 
resources or whatever, but the question was, why should I *HAVE* to, 
again, when there's all sorts of saner alternatives that, like kmail 
before the akonadi insanity, didn't lose mail in the first place?

So I switched, choosing claws-mail to replace both kmail and akregator 
here, FWIW, but there's other alternatives for those who don't like claws-
mail.

And when I switched that off, I began wondering about semantic-desktop at 
all, even tho it was run-time switched off.  So being a gentooer who had 
the option, I set USE=-semantic-desktop and a few other flags and rebuilt 
the affected bits of kde.  Now no more semantic-desktop AT ALL! =:^)  
(Unfortunately, the gentoo/kde folks decided to kill the option and hard-
enable semantic-desktop for the coming 4.11, which I'm running the betas 
of ATM, but using the diffs between the 4.10 ebuilds with the option and 
4.11 builds without, I was able to patch out the the support, so now run 
with semantic-desktop build-time hard-disabled (instead of the normal 
gentoo 4.11 hard-enabled) here.

So no gigabytes of nepomuk and akonadi files doing nothing but create 
problems for me, here!  I do run firefox, but haven't seen a problem with 
it, either, possibly due to #2...

2) Throw hardware at the problem.  About a month ago I finally bit the 
financial bullet and upgraded to (mostly) SSD.  My media partition and 
backups are still on spinning rust (on reiserfs since btrfs is still 
experimental), but the main system and /home are now on dual (fairly 
fast, Corsair Neutron) SSD, on btrfs in raid1 mode (both data/metadata).

That's actually why I'm running btrfs here, as my old standby, reiserfs, 
while highly reliable on spinning rust (yes, I know the rumors, but after 
a decade on reiserfs on spinning rust, it has been /extremely/ reliable 
for me, at least it has been since the data=ordered switch in kernel 
2.6.16 IIRC, even thru various hardware issues!), isn't particularly 
appropriate for SSD.

So I run btrfs, which detects my SSDs and activates SSD mode 
automatically, here.  I use dual SSDs in btrfs raid1 mode, in ordered to 
take advantage of btrfs data integrity with the checksumming.

And with the SSDs, there's no mechanical seek latency and the IOPS are 
high enough that, at least with the btrfs autodefrag mount option 
activated from the beginning, I've seen no noticeable fragmentation 
slowdowns either.  (It may also help that I'm close to 100% 
overprovisioned on the SSDs as my effectively write-once-read-many media 
and backups remain on reiserfs on the spinning rust, so even without the 
trim option, the SSDs have plenty of room to do their write-management.)

What I've noticed most is that the stuff that small, often written files 
that caused fragmentation issues on reiserfs on spinning rust, generally 
aren't an issue at all on btrfs on SSD.  This includes my main gentoo 
package tree and overlays, my git kernel checkout, and pan's nntp message 
cache (by default 10 MB with two-week header expiry, but I'm running 
nearly a gig of text messages with no expiry, with messages on some gmane 
mailing-list groups I follow going back to 2002).  All these are an order 
of magnitude at least faster on btrfs on the SSDs than they were on 
reiserfs on spinning rust, without the slowdowns I saw due to 
fragmentation on spinning rust, either.

So that has been my answer to the fragmentation issue.  The space-cache 
issue of the OP may be different.  I had some problems with that and 
unclean shutdown when I first experimented with btrfs a bit over a year 
ago (still on spinning rust at the time), but I decided it wasn't ready 
for me then and waited a year, and I haven't had problems this go-round.  
I've only had a couple unclean shutdowns, however, but the system did 
seem to come right back up afterward.  The SSDs may be playing some role 
in that too, tho.  But I've really not had enough unclean shutdowns to be 
sure, yet.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: unclean shutdown and space cache rebuild

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/06/13 10:53, Garry T. Williams wrote:
> ~/.cache/chromium/Default/Cache ~/.cache/chromium/Default/Media\ Cache

I've taken to making ~/.cache be tmpfs and all the apps have been fine
with that.  It also meant I didn't have to worry about my btrfs snapshots
being full of transient web junk.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlHQu0wACgkQmOOfHg372QRTMACg1YQx1B6liiLnVpOZLxnoHC+W
5ewAn1Z40V/52dongHBpg6OUdprUVqwo
=601F
-----END PGP SIGNATURE-----

Re: unclean shutdown and space cache rebuild

[Shridhar Daithankar]

On Sunday, June 30, 2013 01:53:48 PM Garry T. Williams wrote:
> I suspect this is, at least in part, related to severe fragmentation
> in /home.

I don't think so. The problem I have described occur only before anybody logs 
in to the system and /home being a separate partition, it is not the problem 
in this case.
w
> 
> There are large files in these directories that are updated frequently
> by various components of KDE and the Chrome browser.  (Firefox has its
> own databases that are frequently updated, too.)
> 
>     ~/.local/share/akonadi

Thats 3.9MB in my case since I point the akonadi db to a systemwide postgresql 
instance. of course, it will just shift defragmentation there.

>     ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend

damn!

# filefrag soprano-virtuoso.db                                                                                                             
soprano-virtuoso.db: 10518 extents found
                                                                                                                               
# btrfs fi defrag soprano-virtuoso.db

# filefrag soprano-virtuoso.db
soprano-virtuoso.db: 957 extents found

How much is an extend anyways?  is it a page or 256M?


>     ~/.cache/chromium/Default/Cache
>     ~/.cache/chromium/Default/Media\ Cache

I don't use chromium. but I get the idea.

But in general, how to find out most fragmented files and folders? mouting 
with autodefrag is a serious degradation..

> I improved performance dramatically (orders of magnitude) by copying
> the database files into an empty file that was modified with:
> 
>     chattr -C
> 
> and renaming to make the files no COW.  (Note that this is the only
> way to change an existing file to no COW.)  I also set the same
> attribute on the owning directories so that all new files inherit the
> no COW attribute.
> 
> I suspect there are other files that fragment badly since I see
> periods of high disk activity coming back slowly over a few weeks of
> use after making the modifications above.  I intend to track them down
> and do the same.

hmm.. a trick to find most badly fragmented files/directories and defragment 
them should do too I think.


-- 
Regards
 Shridhar

Re: unclean shutdown and space cache rebuild

[Duncan]

Shridhar Daithankar posted on Mon, 01 Jul 2013 08:20:19 +0530 as
excerpted:

> On Sunday, June 30, 2013 01:53:48 PM Garry T. Williams wrote:

[discussing fragmentation]
> 
>> ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend
> 
> damn!
> 
> # filefrag soprano-virtuoso.db soprano-virtuoso.db: 10518 extents found
>                                                                                                                                
> # btrfs fi defrag soprano-virtuoso.db
> 
> # filefrag soprano-virtuoso.db soprano-virtuoso.db: 957 extents found

While you evidently had quite some fragmentation as the number of extents 
dropped considerably, if you're running btrfs compression, it's worth 
noting that (based on earlier posts here) filefrag always counts 
compressed files as fragmented, even if they're not.  So a sufficiently 
sized file will almost certainly show fragmentation via filefrag if it's 
compressed, and all you can do is use filefrag as a hint in that case; 
defrag may well not do anything if it's not actually fragmented.

> How much is an extend anyways?  is it a page or 256M?

I don't know...
 
> But in general, how to find out most fragmented files and folders?
> mouting with autodefrag is a serious degradation..

It is?  AFAIK, all the autodefrag mount option does is scan files for 
fragmentation as they are written and queue any fragmentation-detected 
files for background defrag by the defrag thread.

I had expected, particularly on spinning rust, that the benefits of 
autodefrag to far exceed the costs, so your performance drag claim is 
interesting to me indeed.  If my expectation is wrong, which it could be, 
I'd love to know why, and see some numbers.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: unclean shutdown and space cache rebuild

[Shridhar Daithankar]

On Monday, July 01, 2013 09:10:41 AM Duncan wrote:
> > But in general, how to find out most fragmented files and folders?
> > mouting with autodefrag is a serious degradation..
> 
> It is?  AFAIK, all the autodefrag mount option does is scan files for
> fragmentation as they are written and queue any fragmentation-detected
> files for background defrag by the defrag thread.
> 
> I had expected, particularly on spinning rust, that the benefits of
> autodefrag to far exceed the costs, so your performance drag claim is
> interesting to me indeed.  If my expectation is wrong, which it could be,
> I'd love to know why, and see some numbers.

while I don't have numbers, I enabled autodefrag on all the partitions and 
rebooted(twice, just to confirm) and its slow.. 

everything has a 10 second tail of disk activity and has quite some visible 
latency. Moving mouse, switching windows, starting new programs, everything 
has visible latency thats unusable.

It seems autodefrag is being too aggressive for its own good.. 

I am sticking with defragging folders individually. /var, /home and a 1GB 
squid cache is what I have narrowed down and things are reasonably fast.
-- 
Regards
 Shridhar

Re: unclean shutdown and space cache rebuild

[Duncan]

Shridhar Daithankar posted on Mon, 01 Jul 2013 21:49:16 +0530 as
excerpted:

> On Monday, July 01, 2013 09:10:41 AM Duncan wrote:
>> 
>>> mouting with autodefrag is a serious degradation..
>> 
>> It is?  AFAIK, all the autodefrag mount option does is scan files for
>> fragmentation as they are written and queue any fragmentation-detected
>> files for background defrag by the defrag thread.
>> 
>> I had expected, particularly on spinning rust, that the benefits of
>> autodefrag to far exceed the costs, so your performance drag claim is
>> interesting to me indeed.  If my expectation is wrong, which it could
>> be, I'd love to know why, and see some numbers.
> 
> while I don't have numbers, I enabled autodefrag on all the partitions
> and rebooted(twice, just to confirm) and its slow..
> 
> everything has a 10 second tail of disk activity and has quite some
> visible latency. Moving mouse, switching windows, starting new programs,
> everything has visible latency thats unusable.
> 
> It seems autodefrag is being too aggressive for its own good..

Just to be clear, your system, your call.  I'd never /dream/ of 
interfering with that due to the implications for my own system (which is 
certainly highly customized even matched against a peer-group of other 
gentoo installs =:^).  That said...

I'm guessing what you experiences with the autodefrag mount option was 
because you were not in stable-state yet.  The original btrfs filesystem 
setup and fill was very likely without the flag on[2], so there's quite a 
lot of existing fragmentation what would have to be worked thru before 
the filesystem gets defragged and you reach stable-state, at which I'd 
expect the autodefrag mount option to have little overhead.

Tho if what you're saying is correct[1] then it may be that the 
background defrag thread isn't (io-)niced as I would have expected it to 
be.

But I'd still expect there to be some better performance steady state 
after a few mounts gets the basic filesystem defragged.  Tho if the 
fileystem is heavily fragmented[2], in practice it may well be easier to 
backup the filesystem content, do a clean mkfs and mount with autodefrag 
and restore from backup, thus ensuring autodefrag is on while filling the 
filesystem in the first place, than to wait for autodefrag to reach a 
stable system state in normal operation over many mounts.

All that stated, you've definitely demonstrated that I hadn't put enough 
thought into my initial general-case assumptions, which now come with far 
more qualifiers than they did before this subthread.  Thanks. =:^)

[1] I simply don't know from personal experience as I (1) ensured I 
enabled autodefrag on the empty filesystem before I started filling it, 
and (2) I'm on fast ssd, an entirely different world from spinning rust.

[2] Various comments I've read seem to hint that, surprisingly, certain 
distro installers leave a brand new install in a heavily fragmented 
state, as they apparently install without autodefrag on during the 
install and also apparently do heavy rewriting into existing files 
thereby triggering heavy fragmentation without the flag during that 
install.  No, I've not bothered to track which distros, I simply ensured 
autodefrag was on, here, before filling my filesystems in the first place.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: unclean shutdown and space cache rebuild

[Shridhar Daithankar]

On Tuesday, July 02, 2013 01:00:29 PM Duncan wrote:
> Just to be clear, your system, your call.  I'd never /dream/ of
> interfering with that due to the implications for my own system (which is
> certainly highly customized even matched against a peer-group of other
> gentoo installs =:^).  That said...
> 
> I'm guessing what you experiences with the autodefrag mount option was
> because you were not in stable-state yet.  The original btrfs filesystem
> setup and fill was very likely without the flag on[2], so there's quite a
> lot of existing fragmentation what would have to be worked thru before
> the filesystem gets defragged and you reach stable-state, at which I'd
> expect the autodefrag mount option to have little overhead.

Yes, I suspect as much. One of the data parition I have is over an year old 
and never defragged, /home about 3 months old.

and I can see the defragmentation working when run by hand. The initial run on 
/var(only directories, to defrag metadata) was 6 min. Now that I am running it 
daily by hand, entire /var(including files, that includes 1.8G pacman cache, 
few tiny postgresql databases and /var/tmp where kde generates tons of IO) 
takes about 6-7 min.

Despite of my experience with autodefrag, I want to use it because thats the 
best solution(kinda like autovacuum in postgresql) that keeps things clean, 
all the time.

> 
> Tho if what you're saying is correct[1] then it may be that the
> background defrag thread isn't (io-)niced as I would have expected it to
> be.
> 
> But I'd still expect there to be some better performance steady state
> after a few mounts gets the basic filesystem defragged.  Tho if the
> fileystem is heavily fragmented[2], in practice it may well be easier to
> backup the filesystem content, do a clean mkfs and mount with autodefrag
> and restore from backup, thus ensuring autodefrag is on while filling the
> filesystem in the first place, than to wait for autodefrag to reach a
> stable system state in normal operation over many mounts.

well, I think I will bite the bullet and defrag entire / overnight and repeat 
the autodefrag mount option. That should work too.

Thanks

-- 
Regards
 Shridhar

Re: unclean shutdown and space cache rebuild

[Shridhar Daithankar]

On Tuesday, July 02, 2013 09:19:07 PM Shridhar Daithankar wrote:
> On Tuesday, July 02, 2013 01:00:29 PM Duncan wrote:
> > But I'd still expect there to be some better performance steady state
> > after a few mounts gets the basic filesystem defragged.  Tho if the
> > fileystem is heavily fragmented[2], in practice it may well be easier to
> > backup the filesystem content, do a clean mkfs and mount with autodefrag
> > and restore from backup, thus ensuring autodefrag is on while filling the
> > filesystem in the first place, than to wait for autodefrag to reach a
> > stable system state in normal operation over many mounts.
> 
> well, I think I will bite the bullet and defrag entire / overnight and
> repeat the autodefrag mount option. That should work too.

and that worked. defragged all the mount points including files and dirs, 
enabled autodefrag and reboot. Took about 2 hour to defrag the existing files.

but the filesystem is now extremely smooth. faster than ext4 I might say. sure 
there are occasional stalls but they are more of noticable than annoyance and 
thats pretty much compensated by the significant improvement in latency in 
everything.

fun fact, pg_test_fsync now reports 44 fsyncs per seconds instead of earlier 
20. I don't know if that down to defragmentation or compression.

Also another bigger disk of 500GB, the score is around 24 fsyncs per second. 
So I suspect it has to do with tree size.

anyways, good things overall

thanks for the help and suggestions.


-- 
Regards
 Shridhar