Re: [RFC][PATCH V4] btrfs: preferred_metadata: preferred device for metadata

[Goffredo Baroncelli <kreijack@libero.it>]

On 5/28/20 11:59 PM, Hans van Kranenburg wrote:
> Hi!
> 
> On 5/28/20 8:34 PM, Goffredo Baroncelli wrote:
>>
>> [the previous patches sets called this mode ssd_metadata]
>>
>> Hi all,
>>
>> This is an RFC; I wrote this patch because I find the idea interesting
>> even though it adds more complication to the chunk allocator.
> 
> Thanks for working on this. This is an often discussed feature request.
> So, taking it to the next level by actually writing PoC code helps a lot
> I guess.
> 
>> The initial idea was to store the metadata on the ssd and to leave the data
>> on the rotational disks. The kind of disk was determined from the rotational
>> flag. However looking only at the rotational flags is not flexible enough. So
>> I added a device property called "preferred_metadata" to mark a device
>> as preferred for metadata.
>>
>> A separate patches set is sent to extend the "btrfs property" command
>> for supporting the preferred_metadata device flag. The basic usage is:
>>
>>      $ # set a new value
>>      $ sudo btrfs property set /dev/vde preferred_metadata 1
>>      
>>      $ # get the current value
>>      $ sudo btrfs property get /dev/vde preferred_metadata
>>      devid=4, path=/dev/vde: dedicated_metadata=1
>>
>> This new mode is enabled passing the option "preferred_metadata" at mount time.
>> This policy of allocation is the default one. However if this doesn't permit
>> a chunk allocation, the "classic" one is used.
>>
>> Some examples: (/dev/sd[abc] are marked as preferred_metadata,
>> and /dev/sd[ef] are not)
>>
>> Non striped profile: metadata->raid1, data->raid1
>> The data is stored on /dev/sd[ef], metadata is stored on /dev/sd[abc].
>> When /dev/sd[ef] are full, then the data chunk is allocated also on
>> /dev/sd[abc].
>>
>> Striped profile: metadata->raid6, data->raid6
>> raid6 requires 3 disks at minimum, so /dev/sd[ef] are not enough for a
>> data profile raid6. To allow a data chunk allocation, the data profile raid6
>> will be stored on all the disks /dev/sd[abcdef].
>> Instead the metadata profile raid6 will be allocated on /dev/sd[abc],
>> because these are enough to host this chunk.
>>
>> The patches set is composed by four patches:
>>
>> - The first patch adds the ioctl to update the btrfs_dev_item.type field.
>> The ioctl is generic to handle more fields, however now only the "type"
>> field is supported.
> 
> What are your thoughts about the chicken/egg situation of changing these
> properties only when the filesystem is mounted?

The logic is related only to a chunk allocation. I.e. if you have a not
empty filesystem, after enabling the preferred_metadata "mode", in order
to get the benefit a full balance is required.

> 
> E.g. mkfs puts metadata on the wrong disk, and then only after actually
> mounting, I have to find out how to find out where metadata is actually
> placed, and then play around with btrfs balance options until I get
> everything moved to my preferred disks. Do you have any ideas about
> improving the out of the box usability of this?

In order to figure out where the (meta)data are placed, "btrfs fi us"
is your friend.
Of course setting this at mkfs.btrfs time is a good suggestion.

> 
>> - The second patch adds the flag BTRFS_DEV_PREFERRED_METADATA which is
>> used to mark a device as "preferred_metadata"
>>
>> - The third patch exports the btrfs_dev_item.type field via sysfs files
>> /sys/fs/btrfs/<UUID>/devinfo/<devid>/type
>>
>> It is possible only to read the value. It is not implemented the updated
>> of the value because in btrfs/stsfs.c there is a comment that states:
>> "We don't want to do full transaction commit from inside sysfs".
>>
>> - The fourth patch implements this new mode
>>
>> Changelog:
>> v4: - renamed ssd_metadata to preferred_metadata
>>      - add the device property "preferred_metadata"
>>      - add the ioctl BTRFS_IOC_DEV_PROPERTIES
>>      - export the btrfs_dev_item.type values via sysfs
>> v3: - correct the collision between BTRFS_MOUNT_DISCARD_ASYNC and
>>        BTRFS_MOUNT_SSD_METADATA.
>> v2: - rebased to v5.6.2
>>      - correct the comparison about the rotational disks (>= instead of >)
>>      - add the flag rotational to the struct btrfs_device_info to
>>        simplify the comparison function (btrfs_cmp_device_info*() )
>> v1: - first issue
>>
>> [...]
> Another question: what is your opinion about device replace? Should it
> leave properties of the destination device alone, or should it copy the
> bit over?
>
> If I'm replacing my ssd with metadata with a larger one, then what
> should I expect to happen by default as user (already having forgotten
> about that property command that I had to use to actually make it work
> months ago)?

In the previous attempt I rtried  to detect automatically which disk is faster
looking at the rotation flag. However someone pointed me that even from
a sata ssd and a pci nvme there is an huge speed differences (even tough
the latency is more important).
This to say that an automatic logic is not the best possible choice for all
the cases .
Then the next step was to add a flag to mark explicitly the devices for
metadata.

I think that "replacing" and "adding" doesn't have a "sane" default. There will
be always a case where an user replace an ssd with an mechanical hdd or
a case where an ssd is added where there is already an pci nvme.

What would make sense is an additional option to btrfs add/replace
that allows to specify if the disk should be preferred for metadata or not.
> 
> Thanks,
> Hans
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5