is it possible to add a new filter to detect unusable partition types

is it possible to add a new filter to detect unusable partition types

[Heming Zhao]

Hi LVM2 maintainers,

One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.

The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.

On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.

Best regards,
Heming

Re: is it possible to add a new filter to detect unusable partition types

[Glass Su]

> On Dec 17, 2024, at 16:34, Heming Zhao <heming.zhao@suse.com> wrote:
> 
> Hi LVM2 maintainers,
> 
> One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.
> 
> The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.
> 
> On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.
> 
> Best regards,
> Heming

Also Cc util-linux@vger.kernel.org and grub-devel@gnu.org as it’s not an issue with lvm but also other fs progs.
It would be great if we can enhance libblkid to avoid data loss even caused by user mistakes.

— 
Su

Re: is it possible to add a new filter to detect unusable partition types

[Zdenek Kabelac]

Dne 17. 12. 24 v 10:13 Glass Su napsal(a):
> 
>> On Dec 17, 2024, at 16:34, Heming Zhao <heming.zhao@suse.com> wrote:
>>
>> Hi LVM2 maintainers,
>>
>> One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.
>>
>> The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.
>>
>> On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.

Hi

lvm2 is using  blkid to detect 'present' signature on a block device - and 
normally prompt to confirm wiping such signature.

We may possibly add similar logic for 'partition signatures'.

However there is still the plain fact that lvm2  with  --force  or even just 
'--yes' option is assumed to simply proceed  and clean&clear such conflicting 
signatures and simply makes the block device to be a PV.

All that said IMHO primary bug here is within  'grub2-install'  which simply 
should not be blindingly overwriting  block device which is in use - this 
should be fixed ASAP as there is the biggest risk of data loss, although I 
guess everyone is using  'grub2-install --force'  - as without this option 
(even in my personal experience) is typically refusing to do any work....

And same applies to most UI tools I've seen that use lvm2 - all seem to be 
pushing  '--force & --yes' with each it emitted lvm2 command...

Regards

Zdenek

Re: is it possible to add a new filter to detect unusable partition types

[Michael Chang]

On Tue, Dec 17, 2024 at 11:21:26AM +0100, Zdenek Kabelac wrote:
> Dne 17. 12. 24 v 10:13 Glass Su napsal(a):
> > 
> > > On Dec 17, 2024, at 16:34, Heming Zhao <heming.zhao@suse.com> wrote:
> > > 
> > > Hi LVM2 maintainers,
> > > 
> > > One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.
> > > 
> > > The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.
> > > 
> > > On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.
> 
> Hi
> 
> lvm2 is using  blkid to detect 'present' signature on a block device - and
> normally prompt to confirm wiping such signature.
> 
> We may possibly add similar logic for 'partition signatures'.
> 
> However there is still the plain fact that lvm2  with  --force  or even just
> '--yes' option is assumed to simply proceed  and clean&clear such
> conflicting signatures and simply makes the block device to be a PV.
> 
> All that said IMHO primary bug here is within  'grub2-install'  which simply
> should not be blindingly overwriting  block device which is in use - this
> should be fixed ASAP as there is the biggest risk of data loss, although I
> guess everyone is using  'grub2-install --force'  - as without this option
> (even in my personal experience) is typically refusing to do any work....

IMHO, the BIOS Boot partition is dedicated to grub boot code and cannot
be shared with other software. Any attempt other than grub writing to
this area should be prohibited, it should not be the other way around.
Furthermore, adding such check could lead to unexpected failures if the
data is a leftover.

Grub does not write blindly, it checks that the partition is indeed a
BIOS Boot partition before writing to it, as the user is required to
explicitly set the partition type.

For LVM root with legacy BIOS boot, having a BIOS Boot partition is
mandatory, otherwise grub won't have usable space to embed the boot code
in the GPT partition layout, and you won't be able to boot or access a
functional system in the first place. That said, the BIOS Boot partition
is in use by grub before it is mistakenly used to create a PV and extend
the LVM root onto it. It is unlikely that GRUB is overwriting it. In
such cases, it's more likely the other way around.

Thanks,
Michael

> 
> And same applies to most UI tools I've seen that use lvm2 - all seem to be
> pushing  '--force & --yes' with each it emitted lvm2 command...
> 
> Regards
> 
> Zdenek
> 

Re: is it possible to add a new filter to detect unusable partition types

[Demi Marie Obenour]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

On Tue, Dec 17, 2024 at 11:21:26AM +0100, Zdenek Kabelac wrote:
> Dne 17. 12. 24 v 10:13 Glass Su napsal(a):
> > 
> > > On Dec 17, 2024, at 16:34, Heming Zhao <heming.zhao@suse.com> wrote:
> > > 
> > > Hi LVM2 maintainers,
> > > 
> > > One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.
> > > 
> > > The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.
> > > 
> > > On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.
> 
> Hi
> 
> lvm2 is using  blkid to detect 'present' signature on a block device - and
> normally prompt to confirm wiping such signature.
> 
> We may possibly add similar logic for 'partition signatures'.
> 
> However there is still the plain fact that lvm2  with  --force  or even just
> '--yes' option is assumed to simply proceed  and clean&clear such
> conflicting signatures and simply makes the block device to be a PV.
> 
> All that said IMHO primary bug here is within  'grub2-install'  which simply
> should not be blindingly overwriting  block device which is in use - this
> should be fixed ASAP as there is the biggest risk of data loss, although I
> guess everyone is using  'grub2-install --force'  - as without this option
> (even in my personal experience) is typically refusing to do any work....
> 
> And same applies to most UI tools I've seen that use lvm2 - all seem to be
> pushing  '--force & --yes' with each it emitted lvm2 command...

If prompts were in a machine-parsable format, tools that used lvm2 could
differentiate between ones that should automatically be responded to
with "yes" and ones that should not.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: is it possible to add a new filter to detect unusable partition types

[Zdenek Kabelac]

Dne 17. 12. 24 v 13:45 Michael Chang napsal(a):
> On Tue, Dec 17, 2024 at 11:21:26AM +0100, Zdenek Kabelac wrote:
>> Dne 17. 12. 24 v 10:13 Glass Su napsal(a):
>>>
>>>> On Dec 17, 2024, at 16:34, Heming Zhao <heming.zhao@suse.com> wrote:
>>>>
>>>> Hi LVM2 maintainers,
>>>>
>>>> One of SUSE's customers encountered an issue with LVM2. The user created several partitions, one of which was marked as "BIOS boot" (4) instead of "LINUX LVM" (8E). Subsequently, the user ran pvcreate/vgcreate/lvcreate on this partition. During a system update, grub2-install installed GRUB2 in the "BIOS boot" partition, resulting in LVM2 metadata corruption.
>>>>
>>>> The root cause of this issue is that grub2-install targets the "BIOS boot" partition when this lvm2 device is specified for installation. If the user had initially marked the partition as "LINUX LVM", grub2-install would not have chosen this partition.
>>>>
>>>> On the other hand, it would be beneficial if LVM2 could implement a new filter or a filter function to detect and exclude the "BIOS boot" partition from being considered a valid target for LVM2 device creation. This could involve issuing a warning or error message to alert the user of the potential conflict. This may also help user to notice the issue more easily.
>>
>> Hi
>>
>> lvm2 is using  blkid to detect 'present' signature on a block device - and
>> normally prompt to confirm wiping such signature.
>>
>> We may possibly add similar logic for 'partition signatures'.
>>
>> However there is still the plain fact that lvm2  with  --force  or even just
>> '--yes' option is assumed to simply proceed  and clean&clear such
>> conflicting signatures and simply makes the block device to be a PV.
>>
>> All that said IMHO primary bug here is within  'grub2-install'  which simply
>> should not be blindingly overwriting  block device which is in use - this
>> should be fixed ASAP as there is the biggest risk of data loss, although I
>> guess everyone is using  'grub2-install --force'  - as without this option
>> (even in my personal experience) is typically refusing to do any work....
> 
> IMHO, the BIOS Boot partition is dedicated to grub boot code and cannot
> be shared with other software. Any attempt other than grub writing to

Hi

Sorry to say this, but the fact the 'someone' has created 'GUID' for GPT with 
the name 'BIOS boot' doesn't really make anything in the Linux world - so far 
I was not even aware such partition type exists (not using this myself).

It's never even been submitted to lvm2 as something to be understood by tool 
till this thread.

There are over 220 types shown by 'cfdisk' just for GPT and there is a 
completely different set for DOS partition types...

So how should we know which type is lvm2 allowed to 'use' freely ?

Should we now store somewhere those 'hundreds' GUID where there is something 
with Linux in its name ?

I don't think this is a practical thing to do in lvm2 nor in many other 
userland tools that are doing something with block devices.

There should likely be something in blkid telling other Linux tools  'don't 
touch this device unless you are XYZ' eventually you use some --force override 
option.


> For LVM root with legacy BIOS boot, having a BIOS Boot partition is
> mandatory, otherwise grub won't have usable space to embed the boot code
> in the GPT partition layout, and you won't be able to boot or access a
> functional system in the first place. That said, the BIOS Boot partition
> is in use by grub before it is mistakenly used to create a PV and extend
> the LVM root onto it. It is unlikely that GRUB is overwriting it. In
> such cases, it's more likely the other way around.

Well protection needs to be from all sides here - otherwise it makes no sense. 
  When the grub sees some signature, it must be telling to a user and not just 
let user to loose his data blindly.

And in the same way blkid should expose installed grub loader - currently the 
partition with installed grub looks 'empty' with blkid....


Regards

Zdenek

Re: is it possible to add a new filter to detect unusable partition types

[Karel Zak]

On Wed, Dec 18, 2024 at 11:12:59AM GMT, Zdenek Kabelac wrote:
> Sorry to say this, but the fact the 'someone' has created 'GUID' for GPT
> with the name 'BIOS boot' doesn't really make anything in the Linux world -
> so far I was not even aware such partition type exists (not using this
> myself).

Yes, partition types are a legacy from the previous century. They have
very limited relevance in today's world. They may make sense for
things like firmwares or Systemd Discoverable Partitions, but in most
cases, it is only the device content that matters. It is important to
note that we have no way of synchronizing device content and device
types. Additionally, for Linux, device types have had no meaning since
the beginning.

> Well protection needs to be from all sides here - otherwise it makes no
> sense.  When the grub sees some signature, it must be telling to a user and
> not just let user to loose his data blindly.

Yes, this behavior should be standard for all mkfs-like and
partitioning tools. The use of --force should be required in order to
perform any potentially risky actions.

> And in the same way blkid should expose installed grub loader - currently
> the partition with installed grub looks 'empty' with blkid....

The issue I see is that boot loaders can coexist with filesystems on
the same device. This can lead to unexpected warnings when attempting
to view the contents of the device using mkfs tools.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: is it possible to add a new filter to detect unusable partition types

[Thomas Weißschuh]

On 2024-12-18 15:44:45+0100, Karel Zak wrote:
> On Wed, Dec 18, 2024 at 11:12:59AM GMT, Zdenek Kabelac wrote:

[..]

> > And in the same way blkid should expose installed grub loader - currently
> > the partition with installed grub looks 'empty' with blkid....
> 
> The issue I see is that boot loaders can coexist with filesystems on
> the same device. This can lead to unexpected warnings when attempting
> to view the contents of the device using mkfs tools.

Isn't this specifically about the grub second stage on GPT systems
inside a dedicated partition?
There should be no valid coexistence with a filesystem.

So having a probe in blkid looks reasonable to me.

Not that it helps in the specific case mentioned above, where everybody
is using --force anyways.


Thomas

Re: is it possible to add a new filter to detect unusable partition types

[Michael Chang]

On Wed, Dec 18, 2024 at 06:05:54PM +0100, Thomas Weißschuh wrote:
> On 2024-12-18 15:44:45+0100, Karel Zak wrote:
> > On Wed, Dec 18, 2024 at 11:12:59AM GMT, Zdenek Kabelac wrote:
> 
> [..]
> 
> > > And in the same way blkid should expose installed grub loader - currently
> > > the partition with installed grub looks 'empty' with blkid....
> > 
> > The issue I see is that boot loaders can coexist with filesystems on
> > the same device. This can lead to unexpected warnings when attempting
> > to view the contents of the device using mkfs tools.
> 
> Isn't this specifically about the grub second stage on GPT systems
> inside a dedicated partition?

Yes, GPT has no unallocated space similar to the MBR gap in the MSDOS
partition table that can be repurposed for grub second stage, therefore
a dedicated partition has to be defined and allocated. A similar scheme
is also used in PowerPC, where a dedicated firmware PReP boot partition
must be allocated for the boot code.

See Also:

https://www.gnu.org/software/grub/manual/grub/grub.html#BIOS-installation

> There should be no valid coexistence with a filesystem.
> 
> So having a probe in blkid looks reasonable to me.
> 
> Not that it helps in the specific case mentioned above, where everybody
> is using --force anyways.

That's the reason I think adding such a check in grub-install doesn't
help at all. After adding the check, I believe the tools managing the
bootloader installation will start to use wipefs or enforce --force to
grub-install to make sure no leftover can get in the way. In that sense,
it seems like unnecessary breaking change to the toolings.

Thanks,
Michael

> 
> 
> Thomas

Re: is it possible to add a new filter to detect unusable partition types

[Zdenek Kabelac]

Dne 19. 12. 24 v 5:48 Michael Chang napsal(a):
> On Wed, Dec 18, 2024 at 06:05:54PM +0100, Thomas Weißschuh wrote:
>> On 2024-12-18 15:44:45+0100, Karel Zak wrote:
>>> On Wed, Dec 18, 2024 at 11:12:59AM GMT, Zdenek Kabelac wrote:
>>
>> [..]
>>
>>>> And in the same way blkid should expose installed grub loader - currently
>>>> the partition with installed grub looks 'empty' with blkid....
>>>
>>> The issue I see is that boot loaders can coexist with filesystems on
>>> the same device. This can lead to unexpected warnings when attempting
>>> to view the contents of the device using mkfs tools.
>>
>> Isn't this specifically about the grub second stage on GPT systems
>> inside a dedicated partition?
> 
> Yes, GPT has no unallocated space similar to the MBR gap in the MSDOS
> partition table that can be repurposed for grub second stage, therefore
> a dedicated partition has to be defined and allocated. A similar scheme
> is also used in PowerPC, where a dedicated firmware PReP boot partition
> must be allocated for the boot code.

Hi

Yep - it's obvious grub needs to have a space to store its data.
In fact if a device would be 'just & only' a PV, such a PV actually has 
prepared empty space to be consumed by grub - (see  'pvcreate 
--bootloaderareasize)  - which probably never reached its audience - so when 
the user is using PV lvm2 he should not need a special dedicated partition 
(theoretically).

But all that is said here is that choosing some 'random' GUID GPT partition 
type really doesn't change anything in Linux - all tools in Linux are scanning 
content of device - checking for  'partition type' is highly unusual and 
pretty much undefined.

So the focus should go to blkid being able to report that device is occupied 
by some content.

> 
>> There should be no valid coexistence with a filesystem.
>>
>> So having a probe in blkid looks reasonable to me.

Speaking of this - there was use in old ages (and I believe it's still support 
by lvm2) the usage of a PV & MBR at the same time (it's also the reason why 
the PV header is storing it's LABELONE on 2nd. sector (512b)

This has also caused some troubles in past to properly identify device content.

Also blkid already can identify multiple signatures on the same device so it's 
just about the priority which one will be then shown by 'udev' as primary.
lvm2 also checks and clears all signatures one-by-one...

>>
>> Not that it helps in the specific case mentioned above, where everybody
>> is using --force anyways.
> 
> That's the reason I think adding such a check in grub-install doesn't
> help at all. After adding the check, I believe the tools managing the
> bootloader installation will start to use wipefs or enforce --force to
> grub-install to make sure no leftover can get in the way. In that sense,
> it seems like unnecessary breaking change to the toolings.

I guess we may not move forward with this logic...
(aka it's ok change lvm2 to not wipe, but it's fine grub will overwrite anything)

lvm2 is for long time trying to advocate against using '--force' regularly.
In some cases we've introduced even 2nd. --force required to be entered if the 
compatibility usage would be broken otherwise.

Thus the proper logic should be that some 'operations' that currently do need 
--force  - may have it's own dedicated option - i.e. in my case  grub usually 
doesn't really like to store it's data on the partition in use - so maybe 
there can be an option just for this aka --in-use-is-ok y|n

Similarly lvm2 has  'pvcreate --zero y'  to clear device content 
unconditionally - so there is no need to use --force for such case - although 
it takes time to teach other tools to use options the right way....

Regard

Zdenek