Setting options permanently?

Setting options permanently?

[Hadmut Danisch]

Hi,

just a question:

Mounting options for file systems are usually given on the command line
or in /etc/fstab.

Both do not work with mobile storage devices like usb hard disks, since
they are either mounted automatically through the desktop, or manually
by a user who does not know or remember the mount options for this file
system. At least it's error prone.



What happens if a btrfs is mounted sometimes with and sometimes without
the compression option? I'd guess it generates a mixture of compressed
and uncompressed files, what is not exactly smart.

Is there a way to set options like compression on a btrfs permanently to
activate them even when mounted automatically by the desktop or manually
by a third person?


regards
Hadmut

Re: Setting options permanently?

[Chester]

On Fri, Jan 27, 2012 at 9:03 AM, Hadmut Danisch <hadmut@danisch.de> wro=
te:
> Hi,
>
> just a question:
>
> Mounting options for file systems are usually given on the command li=
ne
> or in /etc/fstab.
>
> Both do not work with mobile storage devices like usb hard disks, sin=
ce
> they are either mounted automatically through the desktop, or manuall=
y
> by a user who does not know or remember the mount options for this fi=
le
> system. At least it's error prone.
>
>
>
> What happens if a btrfs is mounted sometimes with and sometimes witho=
ut
> the compression option? I'd guess it generates a mixture of compresse=
d
> and uncompressed files, what is not exactly smart.
>

It should be okay to mount with compress or without compress. Even if
you mount a volume with compressed data without '-o compress' you will
still be able to correctly read the data (but newly written data will
not be compressed)

> Is there a way to set options like compression on a btrfs permanently=
 to
> activate them even when mounted automatically by the desktop or manua=
lly
> by a third person?
>
>
> regards
> Hadmut
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Setting options permanently?

[Hadmut Danisch]

Am 28.01.2012 00:20, schrieb Chester:
> It should be okay to mount with compress or without compress. Even if
> you mount a volume with compressed data without '-o compress' you wil=
l
> still be able to correctly read the data (but newly written data will
> not be compressed)

But having both compressed and uncompressed files in the filesystem is
exactly what I want to avoid. Not because of reading problems, but to
avoid wasting disk space. I don't have a reading problem. I have a
writing problem.

I want to pack some data onto USB hard disks, which exceed the plain
disk capacity. With btrfs it works exactly the way as I need it, except
for the fact that once the compress option was missed the file system i=
s
=84tainted=94 with uncompressed files, thus wasting disk capacity and m=
aybe
cause the files to not fit on the disk anymore.

I currently don't see how to repair this afterwards without removing th=
e
uncompressed files and writing new ones, which on the other hand spoils
hte memory saving effect of using snapshots instead of copies.


It would be much better if there was a way to set options like the
compression option permanently when initially creating the file system,
or later with the btrfs tools.



regards
Hadmut

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Setting options permanently?

[Duncan]

Hadmut Danisch posted on Sat, 28 Jan 2012 01:49:26 +0100 as excerpted:

> Am 28.01.2012 00:20, schrieb Chester:
>> It should be okay to mount with compress or without compress. Even i=
f
>> you mount a volume with compressed data without '-o compress' you wi=
ll
>> still be able to correctly read the data (but newly written data wil=
l
>> not be compressed)
>=20
> But having both compressed and uncompressed files in the filesystem i=
s
> exactly what I want to avoid. Not because of reading problems, but to
> avoid wasting disk space. I don't have a reading problem. I have a
> writing problem.

Having a compression flag in the superblock (probably more than one, th=
us=20
allowing indication of compression type, etc), which is I guess what=20
you're suggesting, sounds useful to me too.  Perhaps it'll get it, once=
=20
development settles down a bit and they know enough about what's still=20
missing in the existing superblock to be able to intelligently allocate=
=20
flags, etc, on a priority basis based on what they then know they need.

At this point I can certainly see a reluctance to make that change, tho=
,=20
since it's a rather small change to make on its own, and with continuin=
g=20
tweaks to the compression types and etc, it's worth holding off=20
committing now to what might look like a bad implementation down the=20
line, when it has to continue to be supported.

> I want to pack some data onto USB hard disks, which exceed the plain
> disk capacity. With btrfs it works exactly the way as I need it, exce=
pt
> for the fact that once the compress option was missed the file system=
 is
> =E2=80=9Etainted=E2=80=9D with uncompressed files, thus wasting disk =
capacity and maybe
> cause the files to not fit on the disk anymore.

I'd be a bit leary of your whole idea of using btrfs on external drives=
=20
attached to multiple systems, at this point.  There's still enough chan=
ge=20
going into btrfs that there's a chance that mounting on a system with a=
=20
newer kernel will prevent mounting on a system with an older kernel.

And if you have enough control over kernel versions on all potentially=20
mounting systems to deal with that, then pretty much by definition, you=
=20
also have enough control over them to hack up a solution that, for=20
instance, looks for a file in specific location on all newly mounted=20
partitions (probably using udev/udisks event hooks), and if found, uses=
=20
the content of said file as a cue to remount the filesystem with the=20
options found in said file.

> I currently don't see how to repair this afterwards without removing =
the
> uncompressed files and writing new ones, which on the other hand spoi=
ls
> hte memory saving effect of using snapshots instead of copies.

A rebalance is the usual way of handling such things, since that rewrit=
es=20
every chunk, taking account of current mount options, etc.  If I've bee=
n=20
reading correctly, a defrag should handle it now as well, altho that us=
ed=20
to break snapshots/cow/reflink-copies, but I /think/ I read that with 3=
=2E2=20
(or was it 3.3-rc1) it doesn't, any more.

> It would be much better if there was a way to set options like the
> compression option permanently when initially creating the file syste=
m,
> or later with the btrfs tools.

As hinted above, I'd /guess/ the chances of something like that=20
eventually occurring are reasonably high, but there's just too much goi=
ng=20
on still, to make superblock changes for every little new feature. =20
Better to save them up for one final change before the caution about th=
e=20
on-disk format changes comes off, prioritize the changes then, and do=20
them all at once.  Then test the new format for a kernel cycle or two=20
just to be sure, and remove the format change boilerplate the cycle aft=
er=20
that.

--=20
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Setting options permanently?

[Ben Klein]

On 28 January 2012 14:32, Duncan <1i5t5.duncan@cox.net> wrote:
> Hadmut Danisch posted on Sat, 28 Jan 2012 01:49:26 +0100 as excerpted=
:
>
>> Am 28.01.2012 00:20, schrieb Chester:
>>> It should be okay to mount with compress or without compress. Even =
if
>>> you mount a volume with compressed data without '-o compress' you w=
ill
>>> still be able to correctly read the data (but newly written data wi=
ll
>>> not be compressed)
>>
>> But having both compressed and uncompressed files in the filesystem =
is
>> exactly what I want to avoid. Not because of reading problems, but t=
o
>> avoid wasting disk space. I don't have a reading problem. I have a
>> writing problem.
>
> Having a compression flag in the superblock (probably more than one, =
thus
> allowing indication of compression type, etc), which is I guess what
> you're suggesting, sounds useful to me too. =C2=A0Perhaps it'll get i=
t, once
> development settles down a bit and they know enough about what's stil=
l
> missing in the existing superblock to be able to intelligently alloca=
te
> flags, etc, on a priority basis based on what they then know they nee=
d.

mkfs -O some_feature comes to mind. I know mke2fs has that, but I
haven't yet actually used btrfs; I'm waiting for fsck to be able to
repair a damaged filesystem.

> At this point I can certainly see a reluctance to make that change, t=
ho,
> since it's a rather small change to make on its own, and with continu=
ing
> tweaks to the compression types and etc, it's worth holding off
> committing now to what might look like a bad implementation down the
> line, when it has to continue to be supported.

This would affect not just a compression-preference option, but
existing filesystems with compression. If the compression routine
changes in a later kernel/filesystem revision, then you could end up
with some files using one routine and others using another. Does btrfs
currently have an "algorithm" specifier for compressed files?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Setting options permanently?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Sat, 28 Jan 2012 17:05:41 +1100
Ben Klein <shacklein@gmail.com> wrote:

> If the compression routine changes in a later kernel/filesystem revision,

There are already two different algorithms, zlib and lzo, and a third one -
snappy - planned for inclusion.

> then you could end up with some files using one routine and others using another.

Which is absolutely not a problem even right now.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Setting options permanently?

[Ben Klein]

On 28 January 2012 19:23, Roman Mamedov <rm@romanrm.ru> wrote:
> On Sat, 28 Jan 2012 17:05:41 +1100
> Ben Klein <shacklein@gmail.com> wrote:
>
>> If the compression routine changes in a later kernel/filesystem revision,
>
> There are already two different algorithms, zlib and lzo, and a third one -
> snappy - planned for inclusion.
>
>> then you could end up with some files using one routine and others using another.
>
> Which is absolutely not a problem even right now.

My bad :)

> --
> With respect,
> Roman
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> "Stallman had a printer,
> with code he could not see.
> So he began to tinker,
> and set the software free."

Re: Setting options permanently?

[Duncan]

Roman Mamedov posted on Sat, 28 Jan 2012 14:23:44 +0600 as excerpted:

> On Sat, 28 Jan 2012 17:05:41 +1100 Ben Klein <shacklein@gmail.com>
> wrote:
> 
>> If the compression routine changes in a later kernel/filesystem
>> revision,
> 
> There are already two different algorithms, zlib and lzo, and a third
> one - snappy - planned for inclusion.
> 
>> then you could end up with some files using one routine and others
>> using another.
> 
> Which is absolutely not a problem even right now.

Reading them isn't a problem, sure, but the thread indicates a desire to 
set the write-compression characteristics.

But that's why I specifically mentioned more than just a compression 
flag.  Presumably it'd be several bits (four bits, 16 values, would leave 
enough room for additional compression types in the future, tho three or 
even two would do it with the current three choices plus uncompressed), 
with one value indicating a no-compression default, and the others 
various choices as the default.  The mount option could then be used to 
override the default, just as it is now, altho the only choice for 
default currently is no compression.

Actually, given that btrfs is in line to be the successor to the ext* 
series, I'd say it should be close to a given that many mount options 
will have settable defaults recorded in the superblocks at mkfs time, and 
updatable using some btrfs parallel to tune2fs (tunebtrfs? tunebfs?).  
However, as I said earlier, committing to specifics with incremental on-
disk including superblock format changes doesn't really cut it.  If a big 
format change is needed due to continuing development, so it can be 
tested and gotchas worked out, yeah, do it now, and hopefully there's 
some space in the superblock(s) reserved for more trivial stuff like 
mount-option defaults already, but especially for mount options since 
they can be tested without committing to the superblock spec changes 
necessary to nail it down there, just testing them using the mount 
options is fine at this point, and the actual recording of the default 
choices can be nailed down all at once in one final change, just before 
the "be prepared for on-disk-format-changes" label comes off.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Setting options permanently?

[Li Zefan]

>> I currently don't see how to repair this afterwards without removing the
>> uncompressed files and writing new ones, which on the other hand spoils
>> hte memory saving effect of using snapshots instead of copies.
> 
> A rebalance is the usual way of handling such things, since that rewrites 
> every chunk, taking account of current mount options, etc.  If I've been 
> reading correctly, a defrag should handle it now as well, altho that used 
> to break snapshots/cow/reflink-copies, but I /think/ I read that with 3.2 
> (or was it 3.3-rc1) it doesn't, any more.
> 

Rebalance won't do the trick, but defrag can. And defrag still breaks
snapshots, which I'm working on.

Re: Setting options permanently?

[Fajar A. Nugraha]

On Sat, Jan 28, 2012 at 7:49 AM, Hadmut Danisch <hadmut@danisch.de> wrote:
> Am 28.01.2012 00:20, schrieb Chester:
>> It should be okay to mount with compress or without compress. Even if
>> you mount a volume with compressed data without '-o compress' you will
>> still be able to correctly read the data (but newly written data will
>> not be compressed)
>
> But having both compressed and uncompressed files in the filesystem is
> exactly what I want to avoid. Not because of reading problems, but to
> avoid wasting disk space. I don't have a reading problem. I have a
> writing problem.

If you've been using -o compress, then you should know that even then
not ALL data is compressed. If btrfs predicts that a data will be
unable to benefit frmo compression, it will store it uncompressed. The
problem is the prediction is not always right. Which is why there's -o
compress-force.

Anyway, for removable media case, there's a workaround that you can
use (at least it works with gnome). Put the entry for the usb block
device (e.g. /dev/sdb1) in fstab, with appropriate mount option, and
the option will be used when you mount it using nautilus.

-- 
Fajar

Re: Setting options permanently?

[Sander]

Hadmut Danisch wrote (ao):
> I currently don't see how to repair this afterwards without removing the
> uncompressed files and writing new ones, which on the other hand spoils
> hte memory saving effect of using snapshots instead of copies.

As also mentioned by Li Zefan, you can use defrag. But this will indeed
not work nicely with snapshots. And you need more free space than the
largest file on the filesystem.

find / -xdev -execdir btrfs filesystem defrag -czlib {} +

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Setting options permanently?

[Li Zefan]

> Is there a way to set options like compression on a btrfs permanently to
> activate them even when mounted automatically by the desktop or manually
> by a third person?

Actually there is.

Btrfs supports per file compression flag, and if this flag is set to a
directory, all files in that directory will inherit the flag.

So you can mount the filesystem for the first time and set the flag for
the root dir, and that's it. There's a flaw that the compression method
is zlib, which should be changed to lzo (or lz4, or snappy..) in the
future.

chattr is used to set file flags, and there was a patchset to enable
compression flag for chattr, but was never merged:

http://www.spinics.net/lists/linux-btrfs/msg09604.html
http://www.spinics.net/lists/linux-btrfs/msg09605.html