property designating root subvolumes

property designating root subvolumes

[Eric Levy]

The file system allows one subvolume per partition to be designated as 
the default, and no more than one would be sensible. Generally, for 
partitions organized with a base file hierarchy constituted of multiple 
subvolumes, the one representing the root mount would be designated as 
the default. Although this association is not required, it is a 
reasonable assumption, enough so that some tools depend on it for 
certain features. For example, the rEFInd bootloader scans a BTRFS file 
system for a Linux-based OS by attempting to identify a root file 
hierarchy on the default subvolume. However, in such usage, the 
constraint of one subvolume designated per partition is limiting. In 
principle, a bootloader might support multiple operating systems 
installed on the same partition, as long as each root partition may be 
separately indicated. To support such usage, it might be helpful if a 
property, separate from the designation of a default subvolume, was 
supported. As a property, it would be allowed to be assigned 
arbitrarily to any number of subvolumes.

Presently, rEFInd supports multiple operating systems on different 
subvolumes of the same partition only by static configuration. Such a 
constraint is particularly cumbersome because any operation for 
installing the bootloader utilizes configuration only on the active 
operating system.

In principle, the broader concept might be extended further, adding 
even more properties, for supporting yet further use cases. As one 
example, a subvolume might be selected as containing configuration 
information applicable to the bootloader, regardless of the active 
operating system. Such a feature would facilitate synchronization of 
bootloader settings for installation tools across all operating systems 
on a partition. Yet, even the single new property would support cleaner 
semantics for greater flexibility of usage.

Note that for such a feature to work properly, the file system would 
need to enforce that the property not be inherited by child volumes, 
that is, snapshots derived from a subvolume with the property enabled. 
Furthermore, some thought must be given to the case of the user 
enabling the property for a subvolume having an ancestor with the 
property already enabled. In such a case, it most likely is desired 
that the property would be disabled for the ancestor.

Re: property designating root subvolumes

[Andrei Borzenkov]

On 15.11.2022 02:23, Eric Levy wrote:
> The file system allows one subvolume per partition to be designated as
> the default, and no more than one would be sensible. Generally, for

It is not partition, it is filesystem.

> partitions organized with a base file hierarchy constituted of multiple
> subvolumes, the one representing the root mount would be designated as
> the default. Although this association is not required, it is a
> reasonable assumption, enough so that some tools depend on it for
> certain features. For example, the rEFInd bootloader scans a BTRFS file
> system for a Linux-based OS by attempting to identify a root file
> hierarchy on the default subvolume. However, in such usage, the
> constraint of one subvolume designated per partition is limiting. In

So far the only usage proposed by you is to scan btrfs subvolumes to 
apparently present user multiple choices during boot. Any bootloader 
that understands btrfs enough to read subvolume properties can also read 
information inside subvolume to achieve the same effect. You need to 
prepare subvolume anyway.

If you suggest some other usage I am afraid I failed to understand it.

> principle, a bootloader might support multiple operating systems
> installed on the same partition, as long as each root partition may be
> separately indicated. To support such usage, it might be helpful if a
> property, separate from the designation of a default subvolume, was
> supported. As a property, it would be allowed to be assigned
> arbitrarily to any number of subvolumes.
> 
> Presently, rEFInd supports multiple operating systems on different
> subvolumes of the same partition only by static configuration. Such a
> constraint is particularly cumbersome because any operation for
> installing the bootloader utilizes configuration only on the active
> operating system.
> 
> In principle, the broader concept might be extended further, adding
> even more properties, for supporting yet further use cases. As one
> example, a subvolume might be selected as containing configuration
> information applicable to the bootloader, regardless of the active
> operating system. Such a feature would facilitate synchronization of
> bootloader settings for installation tools across all operating systems
> on a partition. Yet, even the single new property would support cleaner
> semantics for greater flexibility of usage.
> 
> Note that for such a feature to work properly, the file system would
> need to enforce that the property not be inherited by child volumes,
> that is, snapshots derived from a subvolume with the property enabled.
> Furthermore, some thought must be given to the case of the user
> enabling the property for a subvolume having an ancestor with the
> property already enabled. In such a case, it most likely is desired
> that the property would be disabled for the ancestor.
> 
> 

Re: property designating root subvolumes

[Eric Levy]

On Tue, Nov 15 2022 at 08:50:50 PM +0300, Andrei Borzenkov 
<arvidjaar@gmail.com> wrote:
> On 15.11.2022 02:23, Eric Levy wrote:
>> The file system allows one subvolume per partition to be designated 
>> as
>> the default, and no more than one would be sensible. Generally, for
> 
> It is not partition, it is filesystem.

I'm sorry, but I fail to understand any distinction from your comment 
that clarifies a genuine confusion, within the present context, more 
than a semantic nuance.


>> So far the only usage proposed by you is to scan btrfs subvolumes to 
>> apparently present user multiple choices during boot. Any bootloader 
>> that understands btrfs enough to read subvolume properties can also 
>> read information inside subvolume to achieve the same effect. You 
>> need to prepare subvolume anyway.

A bootloader may not be the best location to include logic for 
descending into a subvolume, to resolve whether it contains a root file 
hierarchy. Meanwhile, some subvolumes, such a snapshots, may appear 
internally as root subvolumes, though not being desired for entries on 
a boot menu. Presently, it is worth considering the historic precedent 
that the default subvolume is being used to identify a location for a 
root file hierarchy. The association often holds, but is not the basis 
of any fully robust assumption.


> If you suggest some other usage I am afraid I failed to understand it.

I will let others express whether they understand more clearly.

Re: property designating root subvolumes

[Goffredo Baroncelli]

On 15/11/2022 00.23, Eric Levy wrote:
> The file system allows one subvolume per partition to be designated as the default, and no more than one would be sensible. Generally, for partitions organized with a base file hierarchy constituted of multiple subvolumes, the one representing the root mount would be designated as the default. Although this association is not required, it is a reasonable assumption, enough so that some tools depend on it for certain features. For example, the rEFInd bootloader scans a BTRFS file system for a Linux-based OS by attempting to identify a root file hierarchy on the default subvolume. However, in such usage, the constraint of one subvolume designated per partition is limiting. In principle, a bootloader might support multiple operating systems installed on the same partition, as long as each root partition may be separately indicated. To support such usage, it might be helpful if a property, separate from the designation of a default subvolume, was supported. As a property, it 
> would be allowed to be assigned arbitrarily to any number of subvolumes.


I don't know rEFInd very well, but I don't think that it is job of the bootloader to do the automatic OS discovery.

If you want to perform an OS discovery, at first it would be enough to check the presence of "/bin/init" (or /init or ...) for linux and the equivalent for Windows and OS-X.
But this will not give the information like:
	- os name
	- initrd
	- the linux boot parameter...

Some (most) setup have different boot entries for the same filesystem (e.g. the standard one, and the emergency one).

I prefer the other way that it is used in the linux world: it is responsibility of the os to inform the bootloader about its existence.
Look at the BLS specification (even not so widespread adopted).




> 
> Presently, rEFInd supports multiple operating systems on different subvolumes of the same partition only by static configuration. Such a constraint is particularly cumbersome because any operation for installing the bootloader utilizes configuration only on the active operating system.
> 
> In principle, the broader concept might be extended further, adding even more properties, for supporting yet further use cases. As one example, a subvolume might be selected as containing configuration information applicable to the bootloader, regardless of the active operating system. Such a feature would facilitate synchronization of bootloader settings for installation tools across all operating systems on a partition. Yet, even the single new property would support cleaner semantics for greater flexibility of usage.
> 
> Note that for such a feature to work properly, the file system would need to enforce that the property not be inherited by child volumes, that is, snapshots derived from a subvolume with the property enabled. Furthermore, some thought must be given to the case of the user enabling the property for a subvolume having an ancestor with the property already enabled. In such a case, it most likely is desired that the property would be disabled for the ancestor.
> 
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: property designating root subvolumes

[Eric Levy]

On Tue, Nov 15 2022 at 07:23:21 PM +0100, Goffredo Baroncelli 
<kreijack@libero.it> wrote:
> I don't know rEFInd very well, but I don't think that it is job of 
> the bootloader to do the automatic OS discovery.
> 
> If you want to perform an OS discovery, at first it would be enough 
> to check the presence of "/bin/init" (or /init or ...) for linux and 
> the equivalent for Windows and OS-X.
> But this will not give the information like:
> 	- os name
> 	- initrd
> 	- the linux boot parameter...
> 
> Some (most) setup have different boot entries for the same filesystem 
> (e.g. the standard one, and the emergency one).
> 
> I prefer the other way that it is used in the linux world: it is 
> responsibility of the os to inform the bootloader about its existence.
> Look at the BLS specification (even not so widespread adopted).

Yes, I respect the preference you have provided, but in a sense, merely 
affirming the preference is begging the larger question.

The intention of rEFInd is as a bootloader, though less advanced 
overall than Grub, that is more user friendly, supporting autodiscovery 
of resident operating systems without needing to be preconfigured by 
tools installed on one of those operating systems. rEFInd does support 
some static configuration, similar to the Grub configuration system, 
but it is not generally required, and supported primarily as an 
afterthought, for advanced use cases not currently possible through the 
main method of on-the-fly autodetection.

Grub is planned primarily with the assumption that a single operating 
system, of some Linux variety, is the one dominantly used on a system, 
and all others, whether also Linux or not, are in some sense 
subordinate, installed only for occasional use. At times, such an 
assumption may be agreeable, but not always.

I think there is a place in the discussion for preferences about the 
design of the bootloader. However, I also think the discussion should 
not reject the historical observation, that due to demand for the 
feature, the default subvolume selector is being used, in a sense 
incorrectly, for detecting a root file system, because no better 
approach has been identified. Someone familiar with the nuances might 
try to discuss with the developer whether the suggested logic for 
detecting the root tree is a helpful approach, more so than using the 
default selector. If, from the standpoint of the bootloader, the 
approach of analyzing the file tree is more agreeable than using 
features of the file system, then perhaps the current request is not 
needed.

However, even so, I have concerns about desired handling of snapshots, 
as well as about the other reasons that a subvolume may appear 
internally as hosting a root tree, but not being desired for showing as 
an item in the boot menu.

Regardless, I suggest that the status quo deserves some inspection.

Re: property designating root subvolumes

[Goffredo Baroncelli]

On 15/11/2022 19.45, Eric Levy wrote:
> 
> On Tue, Nov 15 2022 at 07:23:21 PM +0100, Goffredo Baroncelli <kreijack@libero.it> wrote:
>> I don't know rEFInd very well, but I don't think that it is job of the bootloader to do the automatic OS discovery.
>>
>> If you want to perform an OS discovery, at first it would be enough to check the presence of "/bin/init" (or /init or ...) for linux and the equivalent for Windows and OS-X.
>> But this will not give the information like:
>>     - os name
>>     - initrd
>>     - the linux boot parameter...
>>
>> Some (most) setup have different boot entries for the same filesystem (e.g. the standard one, and the emergency one).
>>
>> I prefer the other way that it is used in the linux world: it is responsibility of the os to inform the bootloader about its existence.
>> Look at the BLS specification (even not so widespread adopted).
> 
> Yes, I respect the preference you have provided, but in a sense, merely affirming the preference is begging the larger question.
> 
> The intention of rEFInd is as a bootloader, though less advanced overall than Grub, that is more user friendly, supporting autodiscovery of resident operating systems without needing to be preconfigured by tools installed on one of those operating systems. rEFInd does support some static configuration, similar to the Grub configuration system, but it is not generally required, and supported primarily as an afterthought, for advanced use cases not currently possible through the main method of on-the-fly autodetection.
> 

My point is that the "on fly autodetection", is more complex than find the root of the filesystem, which is indeed quite trivial (e.g. looking ad /bin/init or /init or /sbin/init); the snapshot is a bit more challenging, but due the fact that these have a pointer to the parent (PARENT_UUID) is still doable.

What is not easily doable (and definitely is not a filesystem job) is discover the boot parameter; for example in my system I have the following cmdline:

$ cat /proc/cmdline
initrd=\e84907d099904117b355a99c98378dca\6.0.8\initrd.img-6.0.8 root=UUID=d7a06504-cc14-435d-a5df-674da09c2894 ro rootflags=subvol=@rootfs btrfsrollback=@rollback boot=btrfs quiet splash loglevel=0 vt.global_cursor_default=0 systemd.machine_id=e84907d099904117b355a99c98378dca

Also I have to point out again that some distribution have multiple pair of kernel/image for the same root filesystem (e.g. the fedora rescue images).

> Grub is planned primarily with the assumption that a single operating system, of some Linux variety, is the one dominantly used on a system, and all others, whether also Linux or not, are in some sense subordinate, installed only for occasional use. At times, such an assumption may be agreeable, but not always.
> 
> I think there is a place in the discussion for preferences about the design of the bootloader. However, I also think the discussion should not reject the historical observation, that due to demand for the feature, the default subvolume selector is being used, in a sense incorrectly, for detecting a root file system, because no better approach has been identified.

The concept of "default subvolume" doesn't mix well with "multiple os on the same filesystem"; in my setups  I prefer to pass the root subvolume in the commandline (e.g. 'rootflags=subvol=@rootfs'); recently I discovered that Debian does the same thing.


[...]
> 
> However, even so, I have concerns about desired handling of snapshots, as well as about the other reasons that a subvolume may appear internally as hosting a root tree, but not being desired for showing as an item in the boot menu.
> 
About this point I agree more, but from a different point of view:
- the subvolume is the unit of snapshot
- a filesystem may be composed by different subvolumes (e.g. @boot, @log, @portables mounted over @rootfs)
- each subvolume may have different snapshot policy
- assuming that the place of a subvolume doesn't change over the time in the filesystem, I would like to find a filesystem layout where it is possible to mount a full filesystem or its snapshot with only a command.

I am trying this layout:

In the root of the btrfs filesystem there are the different subvolumes and theirs snapshots:

/@rootfs
/@rootfs-20221017
/@rootfs-20220917
/@rootfs-20220817
/@home-foo
/@home-foo-20221001
/@home-foo-20220901
/@home-foo-20220801
/@boot
/@log
/@portable

When I "assemble" the filesystem I would like to have:
@rootfs -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo -> /home/foo


But if want to look at the past I would have also:

[20221017]
@rootfs-20221017 -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo-20221001 -> /home/foo

[20221001]
@rootfs-20220917 -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo-20221001 -> /home/foo

[20220901]
@rootfs-20220817 -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo-20220901 -> /home/foo


But also have the possibility to have also

[snapshot of the root filesystem only 20220901]
@rootfs-20220817 -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo -> /home/foo

[snapshot of the home only 20220901]
@rootfs -> /
@boot -> /boot
@log -> /var/log
@portable -> /var/lib/portable
@home-foo-20220901 -> /home/foo

Of course the key would be to have a mount.btrfs command that does the mounts on the basis of a config file


> Regardless, I suggest that the status quo deserves some inspection.
> 
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: property designating root subvolumes

[Eric Levy]

On Wed, Nov 16 2022 at 08:09:05 PM +0100, Goffredo Baroncelli 
<kreijack@libero.it> wrote:
>> The intention of rEFInd is as a bootloader, though less advanced 
>> overall than Grub, that is more user friendly, supporting 
>> autodiscovery of resident operating systems without needing to be 
>> preconfigured by tools installed on one of those operating systems. 
>> rEFInd does support some static configuration, similar to the Grub 
>> configuration system, but it is not generally required, and 
>> supported primarily as an afterthought, for advanced use cases not 
>> currently possible through the main method of on-the-fly 
>> autodetection.
>> 
> 
> My point is that the "on fly autodetection", is more complex than 
> find the root of the filesystem, which is indeed quite trivial (e.g. 
> looking ad /bin/init or /init or /sbin/init); the snapshot is a bit 
> more challenging, but due the fact that these have a pointer to the 
> parent (PARENT_UUID) is still doable.

The way I was framing the conversation is that inspecting the files in 
a subvolume is one way to achieve "on-the-fly autodetection", in 
contrast to other approaches, current or hypothetical. It tends to 
loose the point of my concerns to suggest that the bootloader such as 
rEFInd ought to use an approach more similar to the one used by Grub. 
Being oriented toward autodetection is the overarching design choice of 
rEFInd, and this broad distinction gives it a reason to exist, instead 
of simply being a direct competitor or clone of Grub.

> The concept of "default subvolume" doesn't mix well with "multiple os 
> on the same filesystem"; in my setups  I prefer to pass the root 
> subvolume in the commandline (e.g. 'rootflags=subvol=@rootfs'); 
> recently I discovered that Debian does the same thing.

Yes, but again, isn't it missing the point, that the default subvolume 
may be used as an alternative to the subvolume argument, but not so in 
a way that generalizes well for multiple resident OSs?


>> 
>> However, even so, I have concerns about desired handling of 
>> snapshots, as well as about the other reasons that a subvolume may 
>> appear internally as hosting a root tree, but not being desired for 
>> showing as an item in the boot menu.
>> 
> About this point I agree more, but from a different point of view:
> - the subvolume is the unit of snapshot
> - a filesystem may be composed by different subvolumes (e.g. @boot, 
> @log, @portables mounted over @rootfs)
> - each subvolume may have different snapshot policy
> - assuming that the place of a subvolume doesn't change over the time 
> in the filesystem, I would like to find a filesystem layout where it 
> is possible to mount a full filesystem or its snapshot with only a 
> command.

I believe that presently, it is possible to revert to a previous 
snapshot of the root subvolume in any of three ways, 1) changing the 
location of the snapshot to replace the main subvolume, 2) changing the 
bootloader or system configurations to select the correct subvolume, 
and 3) changing the subvolume selected for default mounting, assuming 
none is configured explicitly.

The first two are generally drastic, whereas the the latter is trivial. 
It is, however, limited, due to requiring that one operating system 
"take over" the entire file system, in the sense of using a resource of 
which the entire file system provides only one, the selection of the 
default subvolume.

The main topic is how to designate certain subvolumes as bootable, if 
the count of such subvolumes is desired as being more than one, but not 
necessarily as many as the total count of snapshots, in a way that is 
nearly as straightforward as changing the default subvolume.
> 

Re: property designating root subvolumes

[Graham Cobb]

On 16/11/2022 21:08, Eric Levy wrote:
> 
> 
> On Wed, Nov 16 2022 at 08:09:05 PM +0100, Goffredo Baroncelli
> <kreijack@libero.it> wrote:
>>> The intention of rEFInd is as a bootloader, though less advanced
>>> overall than Grub, that is more user friendly, supporting
>>> autodiscovery of resident operating systems without needing to be
>>> preconfigured by tools installed on one of those operating systems.
>>> rEFInd does support some static configuration, similar to the Grub
>>> configuration system, but it is not generally required, and supported
>>> primarily as an afterthought, for advanced use cases not currently
>>> possible through the main method of on-the-fly autodetection.
>>>
>>
>> My point is that the "on fly autodetection", is more complex than find
>> the root of the filesystem, which is indeed quite trivial (e.g.
>> looking ad /bin/init or /init or /sbin/init); the snapshot is a bit
>> more challenging, but due the fact that these have a pointer to the
>> parent (PARENT_UUID) is still doable.
> 
> The way I was framing the conversation is that inspecting the files in a
> subvolume is one way to achieve "on-the-fly autodetection", in contrast
> to other approaches, current or hypothetical. It tends to loose the
> point of my concerns to suggest that the bootloader such as rEFInd ought
> to use an approach more similar to the one used by Grub. Being oriented
> toward autodetection is the overarching design choice of rEFInd, and
> this broad distinction gives it a reason to exist, instead of simply
> being a direct competitor or clone of Grub.
> 
>> The concept of "default subvolume" doesn't mix well with "multiple os
>> on the same filesystem"; in my setups  I prefer to pass the root
>> subvolume in the commandline (e.g. 'rootflags=subvol=@rootfs');
>> recently I discovered that Debian does the same thing.
> 
> Yes, but again, isn't it missing the point, that the default subvolume
> may be used as an alternative to the subvolume argument, but not so in a
> way that generalizes well for multiple resident OSs?

So, what you are asking for is the ability to add some information to a
btrfs filesystem listing a set of possible "alternate default"
subvolumes. How any user, OS, application or boot manager might use that
information is unspecified and up to them. It would have no impact on
btrfs behaviour - just provide a useful place to store some hints for
alternative subvolumes users or applications might choose to try to
mount as the filesystem root subvolume if they want. It wouldn't
interact with the existing concept of setting a default subvolume - it
would just be hint to users that here are some other subvolumes they
might want to try mounting.

rEFInd could, if it wished, look that list of alternates and create
choices for trying to boot using each of those subvolumes as the mount
point. What happens then would depend on what the user had put in those
subvolumes.

Graham

Re: property designating root subvolumes

[Eric Levy]

On Wed, Nov 16 2022 at 09:37:49 PM +0000, Graham Cobb 
<g.btrfs@cobb.uk.net> wrote:
>> So, what you are asking for is the ability to add some information 
>> to a
> btrfs filesystem listing a set of possible "alternate default"
> subvolumes. How any user, OS, application or boot manager might use 
> that
> information is unspecified and up to them. It would have no impact on
> btrfs behaviour - just provide a useful place to store some hints for
> alternative subvolumes users or applications might choose to try to
> mount as the filesystem root subvolume if they want. It wouldn't
> interact with the existing concept of setting a default subvolume - it
> would just be hint to users that here are some other subvolumes they
> might want to try mounting.

Not quite. It would make no difference whether there would one bootable 
subvolume or many, and none would be alternative less so than any 
other. The property simply would mark a subvolume as bootable, so it 
would not be necessary to utilize the default selector for such a 
purpose, with all of its various constraints or side effects. The 
property may be assigned to the default subvolume, or not, without 
either affecting the other.

> rEFInd could, if it wished, look that list of alternates and create
> choices for trying to boot using each of those subvolumes as the mount
> point. What happens then would depend on what the user had put in 
> those
> subvolumes.

Would you please elaborate? I'm not sure what you are suggesting beyond 
what has already been discussed.