[RFC] determining if the underlying storage device of blockdev is "real", the Linux modern way

[David Timber <dxdt@dev.snart.me>]

Hello maintainers,

https://github.com/exfatprogs/exfatprogs/pull/351

I'm currently implementing a feature in exfatprogs to fix a long
standing compatibility issue with MS Windows systems. mkfs.exfat was
missing a crucial feature and that lead to compatibility issues reported
by users.

*Background*

There exist two different authoritative exFAT specifications by two
different organisations: one being Microsoft and the other being the SD
Association. As Brauner said years ago when the exFAT implementation has
been introduced to the mainline, Microsoft's exFAT specs remain largely
incomplete and my humble personal view is that the specs by the SD
Association is more mature. However, the SD specs are still locked
behind the paywall to this day so it's hard for the file system devs to
implement to the spec. Nonetheless, the fact that there exist two
different specs lead opensource implementations of exFAT to become a
kind of amalgamation of the two and exfatprogs and the in-kernel exFAT
fs were no exception.

Let me bring your attention to dosfstools, which has played an important
role for supporting UEFI on Linux therefore has been shipped with many
distros as long as the Linux community can remember.

MKFS.FAT(8):
...
>        --mbr[=y|yes|n|no|a|auto]
>            Fill (fake) MBR table with disk signature one partition
> which starts at sector 0 (includes MBR itself) and spans whole disk
> device.  It is needed only for non-removable disks used on Microsoft
> Windows  systems  and  only  when formatting  whole  unpartitioned
> disk.  Location of the disk signature and partition table overlaps
> with the end of the first FAT sector (boot code location), therefore
> there is no additional space usage.  Default is auto mode in which
> mkfs.fat put MBR table only for non-removable disks when formatting
> whole unpartitioned disk.
...

It is evident from the manpage that MS Window systems had issues
recognising "removable" disks formatted without MBR. From the personal
observations I made, this still remains true in recent releases of
Windows don't seem to recognise "fixed"
devices(/sys/dev/block/*/removable reads 0) formatted without MBR.

https://github.com/dosfstools/dosfstools/commit/5199d6847a5e183800fc356abcb2e2ea5fb8fa01
https://github.com/dosfstools/dosfstools/blob/289a48b9cb5b3c589391d28aa2515c325c932c7a/src/device_info.c#L109

Diving into the source code, we can see that mkfs.fat does its best to
figure out if the device associated with a "real" physical device and if
the device is NOT removable, mkfs.fat deems that the device needs the
MBR region to be filled so that the volume can be recognised and used on
MS Windows systems.

https://aussiestorageblog.wordpress.com/2013/11/11/your-sandisk-usb-stick-is-no-longer-removable-and-its-microsofts-fault/

For your information, the "removable" attribute comes from the device
itself via SCSI RMB bit(UAS uses as its underlying transport protocol).
So, whether the storage is actually removable or not has nothing to do
with the actual type of the storage. If the USB stick advertises itself
as "removable", then it is removable. If it doesn't, it's not. For
example, the MMC interface doesn't have the corresponding attribute so
"removable" in sysfs always reads as 0 although you'd think that MMC
media are removable.

The problem is that MS Windows kernel treats "removable" and "fixed"
storage devices differently. If the device is removable, it's not
required to have MBR. MS Windows will detect the volume with or without
MBR. But in case of "fixed"(RMB unset), MS Windows kernel will treat the
device uninitialised if it doesn't contain an MBR. We can all agree that
this behaviour has some serious consequences, but the problem remains
corrected in recent release of Windows.

Ironically, despite the quirk of their own implementation, MS's version
of exFAT specs do not mandate the use of MBR, which lead to
compatibility issues with mkfs.exfat. On the other hand, SD
associations' specs clearly recommends the use of MBR regardless of the
type of device.

The recursive MBR partition entry approach has its own potential
issues(parted actually can't handle recursive partitions other than that
of vFAT and nukes the entries of exFAT boot sector without any prompts).
So when it's not required, it's best not to put one to be on the safe
side. Besides, Linux kernel partscanning the MBR and a partition being
create after runinng mkfs.fat is definitely confusing for many users as
well as other fs utils.

*Question*

The existing dosfstools implementation juggles through the sysfs tree to
see if the device has "children" or "slaves", or if the device is
dm(RAID/dm-crypt ...) or loopdev. If any of the condition is true, the
device is considered "virtual" and no MBR partition entries will be
written for the new volume.

https://github.com/exfatprogs/exfatprogs/pull/351/changes/25bea15e3e229bf2661392d0f960c28ab3badf0e

I believe that this is mostly redundant and the code required to do this
is not so straightforwrad. Using the fairly modern kernel(post-2.6), I
think it can be simplified to:

  * has /sys/dev/block/*/partition and reads integer value > 0: then
    it's a partition so MBR not required
  * has /sys/dev/block/*/device symlink: then it's definitely a real
    physical device
  * but if there's entries in /sys/dev/block/*/slaves: then it's
    "virtual" device

And maybe add

  * /sys/dev/block/*/start reads > 0: then it's a partition or a some
    sort of slave

So symlink dereferencing, no visiting parent directories and so on.

The whole point of doing this is to write MBR only to "real" storage
devices that may be connected to MS Windows systems later. More
specifically: when running mkfs.exfat on a HDD/SDD/SATA/NVME/USB(UAS) drive.

What did I miss? Would this be good enough heuristics? Or any better way
to achieve this?

Thanks,
Davo