Btrfs might be gradually slowing the boot process

Btrfs might be gradually slowing the boot process

[yzb3]

Hello,

I recently noticed that my boot has become slower - it took around 29s, while at the beginning it was ~6s. I thought it was an issue with systemd, because it failed to properly indicate at which stage the slowdown occurred and how long it took. I rolled back to a pretty fresh root subvolume and the boot was fast again. However, after several reboots it started lagging again (10s? preposterous!). I decided to try the *clear_cache* mount option; it was only a guess, but it did speed the boot to 
the proper ~6s.

The question is - why? My filesystem is really small - I see no reason for a 5x boot slowdown. Is this a known issue? I will be glad to give any additional details about my fs.

Re: Btrfs might be gradually slowing the boot process

[Shridhar Daithankar]

On Friday, November 08, 2013 07:35:18 PM yzb3@wp.pl wrote:
> Hello,
> 
> I recently noticed that my boot has become slower - it took around 29s,
> while at the beginning it was ~6s. I thought it was an issue with systemd,
> because it failed to properly indicate at which stage the slowdown occurred
> and how long it took. I rolled back to a pretty fresh root subvolume and
> the boot was fast again. However, after several reboots it started lagging
> again (10s? preposterous!). I decided to try the *clear_cache* mount
> option; it was only a guess, but it did speed the boot to the proper ~6s.
> 
> The question is - why? My filesystem is really small - I see no reason for a
> 5x boot slowdown. Is this a known issue? I will be glad to give any
> additional details about my fs.

Have you tried defragmenting everything, including directory objects? despite 
of small amount of data, it could be massively fragmented, slowing down 
everything.

HTH
-- 
Regards
 Shridhar

Re: Btrfs might be gradually slowing the boot process

[Kai Krakow]

yzb3@wp.pl <yzb3@wp.pl> schrieb:

> I recently noticed that my boot has become slower - it took around 29s,
> while at the beginning it was ~6s. I thought it was an issue with systemd,
> because it failed to properly indicate at which stage the slowdown
> occurred and how long it took. I rolled back to a pretty fresh root
> subvolume and the boot was fast again. However, after several reboots it
> started lagging again (10s? preposterous!). I decided to try the
> *clear_cache* mount option; it was only a guess, but it did speed the boot
> to the proper ~6s.
> 
> The question is - why? My filesystem is really small - I see no reason for
> a 5x boot slowdown. Is this a known issue? I will be glad to give any
> additional details about my fs.

I'm using a similar setup (although my btrfs rootfs is a 3 disk array and is 
more or less huge) and I didn't see such a vast boot slowdown. Did you check 
that systemd is actually using the readahead services which will relocate 
and defragment your boot files automatically?

HTH
Kai

Re: Btrfs might be gradually slowing the boot process

[Chris Murphy]

What's leafsize was used when making the file system? The default is now (as of yesterday) 16KB to avoid metadata fragmentation.


Chris Murphy

Re: Btrfs might be gradually slowing the boot process

[Duncan]

Chris Murphy posted on Sat, 09 Nov 2013 21:53:10 -0700 as excerpted:

> What's leafsize was used when making the file system? The default is now
> (as of yesterday) 16KB to avoid metadata fragmentation.

I saw the discussion of that, but two questions it didn't answer -- (1) 
what was the default size before, and (2) how do I check what I actually 
have?

The wiki mkfs.btrfs page answers the first question: 4KiB, standard page-
size.  (Of course that needs updated now.)

The same page says nodesize defaults to the same as leafsize.  Is that 
still true, or does it remain 4KiB?  Also, sectorsize defaults to 4KiB, 
but the wiki doesn't note whether it always defaults to that, or whether 
it too defaults to the same as leafsize.

And nowhere on the wiki do I see a sysadmin's level guide to how 
leafsize, nodesize and sectorsize relate to each other, nor how they 
relate to chunks or how btrfs manages chunksize.

Josef Bacik's article, "Btrfs - The Swiss Army Knife of Storage"[1], 
linked below and in the wiki articles section, does mention that nodes 
and leaves are b-tree terms, where nodes contain keys and links to the 
next level nodes or leaves, and leaves contain actual data, which helps 
somewhat.  But where do sectors fit in to this?

The same article says data chunk sizes are 1GiB by default, metadata 
256MiB, but when space starts getting tight (and how tight is tight, with 
the last full gig be allocated to a full gig data chunk or will it go 
smaller before it reserves the last fill gig?  how much before, 10 gig, 2 
gig?), does it shrink by orders of two or does it jump down from a gig to 
say 128 meg in one jump, and do data and metadata follow the same jump 
rules or not?

And hardly any of this is actually on the wiki, at least the sysadmin's 
docs (the dev docs might cover it, I'm not a dev and haven't looked that 
closely at them, but that still leaves sysadmins).

[1] http://static.usenix.org/publications/login/2012-02/openpdfs/Bacik.pdf

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Btrfs might be gradually slowing the boot process

[Russell Coker]

[-- Attachment #1: Type: Text/Plain, Size: 1339 bytes --]

On Sun, 10 Nov 2013, Duncan <1i5t5.duncan@cox.net> wrote:
> Chris Murphy posted on Sat, 09 Nov 2013 21:53:10 -0700 as excerpted:
> > What's leafsize was used when making the file system? The default is now
> > (as of yesterday) 16KB to avoid metadata fragmentation.
> 
> I saw the discussion of that, but two questions it didn't answer -- (1) 
> what was the default size before, and (2) how do I check what I actually 
> have?

I've attached the Debian patch to the magic database for file(1) to display 
BTRFS data, it gives output such as the following.  The magic database 
displayed the leafsize before this patch, but the patch displays the UUID 
(which you REALLY want to know) and all the other information I could figure 
out.

# file -s /dev/dm-0
/dev/dm-0: BTRFS Filesystem sectorsize 4096, nodesize 4096, leafsize 4096, 
UUID=586e6f48-2985-4115-9f89-f844b319c7c0, 82545332224/115536781312 bytes 
used, 1 devices

I might file another bug report with a new version to display error counts 
etc.  I think that all data which we can extract should be displayed to the 
user.  As an aside, I don't know if the bytes used field is going to be very 
useful on a RAID-0 filesystem, but it's there and I think we might as well 
show it.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

[-- Attachment #2: 0011-btrfs.patch --]
[-- Type: text/x-patch, Size: 1307 bytes --]

Author: Russell Coker <russell@coker.com.au>
Description: Adding updated magics for BTRFS (Closes: #663454).

diff -Naurp file.orig/magic/Magdir/filesystems file/magic/Magdir/filesystems
--- file.orig/magic/Magdir/filesystems	2013-03-13 16:54:43.588452886 +0100
+++ file/magic/Magdir/filesystems	2013-03-13 17:15:59.659044842 +0100
@@ -1894,12 +1894,21 @@
 >>0x10060        string          >\0             lockproto %s)
 
 # BTRFS
-0x10040         string          _BHRfS_M        BTRFS Filesystem
->0x1012b        string          >\0             (label "%s",
->0x10090        lelong          x               sectorsize %d,
->0x10094        lelong          x               nodesize %d,
->0x10098        lelong          x               leafsize %d)
-
+# Russell Coker <russell@coker.com.au>
+0x10040	string	_BHRfS_M	BTRFS Filesystem
+>0x1012b	string	>\0		label "%s",
+>0x10090	lelong	x		sectorsize %d,
+>0x10094	lelong	x		nodesize %d,
+>0x10098	lelong	x		leafsize %d,
+>0x10020	belong	x		UUID=%8x-
+>0x10024	beshort	x		\b%4x-
+>0x10026	beshort	x		\b%4x-
+>0x10028	beshort	x		\b%4x-
+>0x1002a	beshort	x		\b%4x
+>0x1002c	belong	x		\b%8x,
+>0x10078	lequad	x		%lld/
+>0x10070	lequad	x		\b%lld bytes used,
+>0x10088	lequad	x		%lld devices
 
 # dvdisaster's .ecc
 # From: "Nelson A. de Oliveira" <naoliv@gmail.com>

Re: Btrfs might be gradually slowing the boot process

[Kai Krakow]

Hi!

> What's leafsize was used when making the file system? The default is now
> (as of yesterday) 16KB to avoid metadata fragmentation.

Since my btrfs is about 2 years old I suppose I'm still using 4kB leafsize. 
Is there a way to change it without recreating the whole filesystem from 
scratch?

Regards,
Kai

Re: Btrfs might be gradually slowing the boot process

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Sun, Nov 10, 2013 at 05:07:23PM +0100, Kai Krakow wrote:
> Hi!
> 
> > What's leafsize was used when making the file system? The default is now
> > (as of yesterday) 16KB to avoid metadata fragmentation.
> 
> Since my btrfs is about 2 years old I suppose I'm still using 4kB leafsize. 
> Is there a way to change it without recreating the whole filesystem from 
> scratch?

   No, it's one of the few options that's not changable after mkfs.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- If the first-ever performance is the première,  is the ---     
                  last-ever performance the derrière?                   

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]