"/tmp/mnt.", and not honouring compression

"/tmp/mnt.", and not honouring compression

[Chris Murray]

Hi,

I'm trying to troubleshoot a ceph cluster which doesn't seem to be
honouring BTRFS compression on some OSDs. Can anyone offer some help? Is
it likely to be a ceph issue or a BTRFS one? Or something else? I've
asked on ceph-users already, but not received a response yet.

Config is set to mount with "noatime,nodiratime,compress-force=lzo"

Some OSDs have been getting much more full than others though, which I
think is something to do with these 'tmp' mounts e.g. below:

/dev/sdc1 on /var/lib/ceph/tmp/mnt.AywYKY type btrfs
(rw,noatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/)
/dev/sdd1 on /var/lib/ceph/osd/ceph-16 type btrfs
(rw,noatime,nodiratime,compress-force=lzo,space_cache,subvolid=5,subvol=
/)
/dev/sdc1 on /var/lib/ceph/osd/ceph-15 type btrfs
(rw,noatime,nodiratime,space_cache,user_subvol_rm_allowed,subvolid=5,sub
vol=/)
/dev/sdb1 on /var/lib/ceph/osd/ceph-20 type btrfs
(rw,noatime,nodiratime,compress-force=lzo,space_cache,subvolid=5,subvol=
/)

After a reboot, it's moved to another drive:

/dev/sdd1 on /var/lib/ceph/tmp/mnt.kWh2NA type btrfs
(rw,noatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/)
/dev/sdd1 on /var/lib/ceph/osd/ceph-16 type btrfs
(rw,noatime,nodiratime,space_cache,user_subvol_rm_allowed,subvolid=5,sub
vol=/)
/dev/sdc1 on /var/lib/ceph/osd/ceph-15 type btrfs
(rw,noatime,nodiratime,compress-force=lzo,space_cache,subvolid=5,subvol=
/)
/dev/sdb1 on /var/lib/ceph/osd/ceph-20 type btrfs
(rw,noatime,nodiratime,compress-force=lzo,space_cache,subvolid=5,subvol=
/)

I'm using Proxmox, based on Debian. Kernel version 4.2.8-1-pve. Btrfs
v3.17.

Thank you,
Chris

Re: "/tmp/mnt.", and not honouring compression

[Duncan]

Chris Murray posted on Thu, 31 Mar 2016 21:49:29 +0100 as excerpted:

> I'm using Proxmox, based on Debian. Kernel version 4.2.8-1-pve. Btrfs
> v3.17.

The problem itself is beyond my level, but aiming for the obvious low-
hanging fruit...

On this list, which is forward looking as btrfs remains stabilizing, not 
yet fully stable and mature, kernel support comes in four tracks, 
mainstream and btrfs development trees, mainstream current, mainstream 
lts, and everything else.

Mainstream and btrfs development trees should be obvious.  It covers 
mainstream current git and rc kernels as well as btrfs-integration and 
linux-next.  Generally only recommended for bleeding edge testers willing 
to lose what they're testing.

Mainstream current follows mainstream latest releases, with generally the 
latest two kernel series being best supported.  With 4.5 out, that's 4.5 
and 4.4.

Mainstream LTS follows mainstream LTS series, and until recently, again 
the latest two were best supported.  That's the 4.4 and 4.1 LTS series.  
However, as btrfs has matured, the previous LTS series, 3.18, hasn't 
turned out so bad and remains reasonably well supported as well, tho 
depending on the issue, you may still be asked to upgrade and see if it's 
still there in 4.1 or 4.4.

Then there's "everything else", which is where a 4.2 kernel such as 
you're running comes in.  These kernels are either long ago history 
(pre-3.18 LTS, for instance) in btrfs terms, or out of their mainstream 
kernel support windows, which is where 4.2 is.  While we recognize that 
various distros claiming btrfs support may still be using these kernels, 
because we're mainline focused we don't track what patches they may or 
may not have backported, and thus aren't in a particularly good position 
to support them.  If you're relying on your distro's support in such a 
case, that's where you need to look, as they know what they've backported 
and what they haven't and are thus in a far better position to provide 
support.

As for the list, we still do the best we can with these "everything else" 
kernels, but unless it's a known problem recognized on-sight, that's most 
often simply to recommend upgrading to something that's better supported 
and trying to duplicate the problem there.

Meanwhile, for long-term enterprise level stability, btrfs isn't likely 
to be a good choice in any case, as it really is still stabilizing and 
the expectation is that people running it will be upgrading to get the 
newer patches.  If that's not feasible, as it may not be for the 
enterprise-stability-level use-case, then it's very likely that btrfs 
isn't a good match for the use-case anyway, as it's simply not to that 
level of stability yet.  A more mature filesystem such as ext4, ext3, the 
old reiserfs which I still use on some spinning rust here (all my btrfs 
are on ssd), xfs, etc, is very likely to be a more appropriate choice for 
that use-case.

For kernel 4.2, that leaves you with a few choices:

1) Ask your distro for btrfs support if they offer it on the out-of-
mainline-support kernels which they've obviously chosen to use instead of 
the LTS series that /are/ still mainline supported.

2) Upgrade to the supported 4.4 LTS kernel series.

3) Downgrade to the older supported 4.1 LTS kernel series.

4) Decide btrfs is inappropriate for your use-case and switch to a fully 
stable and mature filesystem.

5) Continue with 4.2 and muddle thru, using our "best effort" help where 
you can and doing without or getting it elsewhere if the opportunity 
presents itself or you have money to buy it from a qualified provider.


Personally I'd choose option 2, upgrading to 4.4, but that's just me.  
The other choices may work better for you.


As for btrfs-progs userspace, when the filesystem is working it's not as 
critical, since other than filesystem creation with mkfs.btrfs, most 
operational commands simply invoke kernel code to do the real work.  
However, once problems appear, a newer version can be critical as patches 
to deal with newly discovered problems continue to be added to tools such 
as btrfs check (for detecting and repairing problems) and btrfs restore 
(for recovery of files off an unmountable filesystem).  And newer 
userspace is designed to work with older kernels, so newer isn't a 
problem in that regard.

As a result, to keep userspace from getting /too/ far behind and because 
userspace release version numbers are synced with kernel version, a good 
rule of thumb is to run a userspace version similar to that of your 
kernel, or newer.  Assuming you're already following the current or LTS 
track kernel recommendations, that will keep you reasonably current, and 
you can always upgrade to the newest available if you're trying to fix 
otherwise unfixable problems.

Unfortunately your userspace falls well outside that recommendation as 
well, with 3.17 userspace being before the earliest supported 3.18 LTS 
kernel, let alone comparable to your current 4.2 kernel or the 4.4 LTS 
upgrade or 4.1 LTS downgrade that would be recommended on the LTS track, 
or 4.4 or 4.5 on the current track.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: "/tmp/mnt.", and not honouring compression

[Chris Murray]

On Thu, 2016-03-31 at 23:43 +0100, Duncan wrote:
> Chris Murray posted on Thu, 31 Mar 2016 21:49:29 +0100 as excerpted:
> 
> > I'm using Proxmox, based on Debian. Kernel version 4.2.8-1-pve.
> Btrfs
> > v3.17.
> 
> The problem itself is beyond my level, but aiming for the obvious low-
> hanging fruit...
> 
> On this list, which is forward looking as btrfs remains stabilizing,
> not
> yet fully stable and mature, kernel support comes in four tracks,
> mainstream and btrfs development trees, mainstream current, mainstream
> lts, and everything else.
> 
> Mainstream and btrfs development trees should be obvious.  It covers
> mainstream current git and rc kernels as well as btrfs-integration and
> linux-next.  Generally only recommended for bleeding edge testers
> willing
> to lose what they're testing.
> 
> Mainstream current follows mainstream latest releases, with generally
> the
> latest two kernel series being best supported.  With 4.5 out, that's
> 4.5
> and 4.4.
> 
> Mainstream LTS follows mainstream LTS series, and until recently,
> again
> the latest two were best supported.  That's the 4.4 and 4.1 LTS
> series. 
> However, as btrfs has matured, the previous LTS series, 3.18, hasn't
> turned out so bad and remains reasonably well supported as well, tho
> depending on the issue, you may still be asked to upgrade and see if
> it's
> still there in 4.1 or 4.4.
> 
> Then there's "everything else", which is where a 4.2 kernel such as
> you're running comes in.  These kernels are either long ago history
> (pre-3.18 LTS, for instance) in btrfs terms, or out of their
> mainstream
> kernel support windows, which is where 4.2 is.  While we recognize
> that
> various distros claiming btrfs support may still be using these
> kernels,
> because we're mainline focused we don't track what patches they may or
> may not have backported, and thus aren't in a particularly good
> position
> to support them.  If you're relying on your distro's support in such a
> case, that's where you need to look, as they know what they've
> backported
> and what they haven't and are thus in a far better position to provide
> support.
> 
> As for the list, we still do the best we can with these "everything
> else"
> kernels, but unless it's a known problem recognized on-sight, that's
> most
> often simply to recommend upgrading to something that's better
> supported
> and trying to duplicate the problem there.
> 
> Meanwhile, for long-term enterprise level stability, btrfs isn't
> likely
> to be a good choice in any case, as it really is still stabilizing and
> the expectation is that people running it will be upgrading to get the
> newer patches.  If that's not feasible, as it may not be for the
> enterprise-stability-level use-case, then it's very likely that btrfs
> isn't a good match for the use-case anyway, as it's simply not to that
> level of stability yet.  A more mature filesystem such as ext4, ext3,
> the
> old reiserfs which I still use on some spinning rust here (all my
> btrfs
> are on ssd), xfs, etc, is very likely to be a more appropriate choice
> for
> that use-case.
> 
> For kernel 4.2, that leaves you with a few choices:
> 
> 1) Ask your distro for btrfs support if they offer it on the out-of-
> mainline-support kernels which they've obviously chosen to use instead
> of
> the LTS series that /are/ still mainline supported.
> 
> 2) Upgrade to the supported 4.4 LTS kernel series.
> 
> 3) Downgrade to the older supported 4.1 LTS kernel series.
> 
> 4) Decide btrfs is inappropriate for your use-case and switch to a
> fully
> stable and mature filesystem.
> 
> 5) Continue with 4.2 and muddle thru, using our "best effort" help
> where
> you can and doing without or getting it elsewhere if the opportunity
> presents itself or you have money to buy it from a qualified provider.
> 
> 
> Personally I'd choose option 2, upgrading to 4.4, but that's just me. 
> The other choices may work better for you.
> 
> 
> As for btrfs-progs userspace, when the filesystem is working it's not
> as
> critical, since other than filesystem creation with mkfs.btrfs, most
> operational commands simply invoke kernel code to do the real work. 
> However, once problems appear, a newer version can be critical as
> patches
> to deal with newly discovered problems continue to be added to tools
> such
> as btrfs check (for detecting and repairing problems) and btrfs
> restore
> (for recovery of files off an unmountable filesystem).  And newer
> userspace is designed to work with older kernels, so newer isn't a
> problem in that regard.
> 
> As a result, to keep userspace from getting /too/ far behind and
> because
> userspace release version numbers are synced with kernel version, a
> good
> rule of thumb is to run a userspace version similar to that of your
> kernel, or newer.  Assuming you're already following the current or
> LTS
> track kernel recommendations, that will keep you reasonably current,
> and
> you can always upgrade to the newest available if you're trying to fix
> otherwise unfixable problems.
> 
> Unfortunately your userspace falls well outside that recommendation as
> well, with 3.17 userspace being before the earliest supported 3.18 LTS
> kernel, let alone comparable to your current 4.2 kernel or the 4.4 LTS
> upgrade or 4.1 LTS downgrade that would be recommended on the LTS
> track,
> or 4.4 or 4.5 on the current track.
> 
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Duncan, many thanks for that comprehensive response.

I was hoping for an "ah-ha, I've seen that!" from someone, but can now
see the position I'm in and the options available. I see there's
probably little to be gained in trying to support my ageing kernel and
btrfs-progs, so I'm happy to leave it at that and not waste anyone's
time.

My workaround for now is to reweight CEPH OSDs to cater for the
'missing' compression on these "/tmp/mnt." mounts - some disks will be
compressed, and some won't, but overall there are at least *some*
savings.

Thanks again,
Chris

Re: "/tmp/mnt.", and not honouring compression

[Lionel Bouton]

Le 31/03/2016 22:49, Chris Murray a écrit :
> Hi,
>
> I'm trying to troubleshoot a ceph cluster which doesn't seem to be
> honouring BTRFS compression on some OSDs. Can anyone offer some help? Is
> it likely to be a ceph issue or a BTRFS one? Or something else? I've
> asked on ceph-users already, but not received a response yet.
>
> Config is set to mount with "noatime,nodiratime,compress-force=lzo"
>
> Some OSDs have been getting much more full than others though, which I
> think is something to do with these 'tmp' mounts e.g. below:

Note that there are other reasons for unbalanced storage on Ceph OSD.
The main reason is too few PGs (there's a calculator on ceph.com, google
for it).
These tmp mounts aren't normal, you should find out what is causing them.

So it might be a Ceph issue (too few PGs) or a system issue (some
component trying to use your filesystems for its own purposes).
You might have more luck on the ceph-users list (post your Ceph version,
the result of ceph osd tree, df on all OSDs and hunt for the process
creating theses mounts on your systems).

It's probably not a Btrfs issue (I run a Ceph on Btrfs cluster in
production and I've never seen this kind of problem).

Lionel