md-linear accidental(?) removal, removed significant(?) use case?

md-linear accidental(?) removal, removed significant(?) use case?

[Allen Toureg]

Preamble: I have extensive arrays of drives and due to their irregular
sizes, I was using md-linear as a convenient way to manually
concatenate arrays of underlying MD (raid5/etc) to manually deal with
redundancy.

I have probably a few thousand TB in total raw space, and hundreds of
TB of actual data and files attached to singular systems.

In a recent OS update, I discovered my larger volumes no longer mount
because md-linear has been removed entirely from the kernel as of 6.8.

I am trying to do rationale archaeology. I sent a note to Mr. Neil
Brown who was responsible for the earliest change I found related to
this and he suggested I email regressions and linux-raid along with
the current maintainers about it.

What I've been able to find:

In 2604b703b6b3db80e3c75ce472a54dfd0b7bf9f4 (2006) Neil Brown marked a
MODULE_ALIAS entry for md-personality-1 as deprecated but it appears
the reason was because the form of the personality was changed (not
that the underlying md-linear itself was deprecated.)

d9d166c2a9d5d01af34396793950aa695883eed4 (2006) reinforced this change
via a diff algorithm that overzealously included that line in a diff
chunk but which makes annotating prior to it a more manual process.

608f52e30aae7dc8da836e5b7b112d50a2d00e43 (2021) marked md-linear as
deprecated in Kconfig, using the rationale that md-linear was
deprecated in MODULE_ALIAS—but again which doesn't explain why the
*module* was deprecated and appears to me at least to accidentally
misconstrue the original reason for the deprecation comment.

849d18e27be9a1253f2318cb4549cc857219d991 (2023) eliminated md-linear
entirely, again mostly self-referencing a deprecation notice which was
there in actuality for basically multiple decades and seems to have
referenced something else entirely.

I was hoping you could help me understand why this module was removed?
I have found others who also are running into this. Functionality they
relied on has disappeared, as per the existence of the following:

https://github.com/dougvj/md-linear

https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/34
https://bbs.archlinux.org/viewtopic.php?id=294005
(etc)

So, it looks like there are many of us who were still using mdadm to
manage sub-device concatenation, again in my case for 100s of TB of
admittedly casual data storage, and I can't currently find what the
actual actual rationale was for removing it. :(

For utility's sake, I would like to suggest that linear volumes lessen
problems like substriping. I do not think for many of us that
shuffling around a few hundred TB is easy to do at any rate. Currently
I'm manually re-compiling a fairly heavily-modified md-linear as a
user-built module and it seems to work okay. I am definitely not the
only one doing this.

Please consider resurrecting md-linear. :-)

Thank you,
Sincerely,
at

Re: md-linear accidental(?) removal, removed significant(?) use case?

[Yu Kuai]

Hi,

在 2025/01/02 12:14, Allen Toureg 写道:
> Preamble: I have extensive arrays of drives and due to their irregular
> sizes, I was using md-linear as a convenient way to manually
> concatenate arrays of underlying MD (raid5/etc) to manually deal with
> redundancy.
> 
> I have probably a few thousand TB in total raw space, and hundreds of
> TB of actual data and files attached to singular systems.
> 
> In a recent OS update, I discovered my larger volumes no longer mount
> because md-linear has been removed entirely from the kernel as of 6.8.
> 
> I am trying to do rationale archaeology. I sent a note to Mr. Neil
> Brown who was responsible for the earliest change I found related to
> this and he suggested I email regressions and linux-raid along with
> the current maintainers about it.
> 
> What I've been able to find:
> 
> In 2604b703b6b3db80e3c75ce472a54dfd0b7bf9f4 (2006) Neil Brown marked a
> MODULE_ALIAS entry for md-personality-1 as deprecated but it appears
> the reason was because the form of the personality was changed (not
> that the underlying md-linear itself was deprecated.)
> 
> d9d166c2a9d5d01af34396793950aa695883eed4 (2006) reinforced this change
> via a diff algorithm that overzealously included that line in a diff
> chunk but which makes annotating prior to it a more manual process.
> 
> 608f52e30aae7dc8da836e5b7b112d50a2d00e43 (2021) marked md-linear as
> deprecated in Kconfig, using the rationale that md-linear was
> deprecated in MODULE_ALIAS—but again which doesn't explain why the
> *module* was deprecated and appears to me at least to accidentally
> misconstrue the original reason for the deprecation comment.
> 
> 849d18e27be9a1253f2318cb4549cc857219d991 (2023) eliminated md-linear
> entirely, again mostly self-referencing a deprecation notice which was
> there in actuality for basically multiple decades and seems to have
> referenced something else entirely.
> 
> I was hoping you could help me understand why this module was removed?
> I have found others who also are running into this. Functionality they
> relied on has disappeared, as per the existence of the following:
> 
> https://github.com/dougvj/md-linear
> 
> https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/34
> https://bbs.archlinux.org/viewtopic.php?id=294005
> (etc)
> 
> So, it looks like there are many of us who were still using mdadm to
> manage sub-device concatenation, again in my case for 100s of TB of
> admittedly casual data storage, and I can't currently find what the
> actual actual rationale was for removing it. :(

Thanks for the report! I believe the reason md-linear is removed is that
we don't recognize real users, and it's deprecated for a long time.
> 
> For utility's sake, I would like to suggest that linear volumes lessen
> problems like substriping. I do not think for many of us that
> shuffling around a few hundred TB is easy to do at any rate. Currently
> I'm manually re-compiling a fairly heavily-modified md-linear as a
> user-built module and it seems to work okay. I am definitely not the
> only one doing this.
> 
> Please consider resurrecting md-linear. :-)

I think it's good to introduce md-linear back, I'll send a patch and see
how other people thinks. And perhaps can you give the new md-linear a
test?

Thanks,
Kuai

> 
> Thank you,
> Sincerely,
> at
> .
> 

Re: md-linear accidental(?) removal, removed significant(?) use case?

[Roman Mamedov]

On Wed, 1 Jan 2025 20:14:12 -0800
Allen Toureg <thetanix@gmail.com> wrote:

> Preamble: I have extensive arrays of drives and due to their irregular
> sizes, I was using md-linear as a convenient way to manually
> concatenate arrays of underlying MD (raid5/etc) to manually deal with
> redundancy.
> 
> I have probably a few thousand TB in total raw space, and hundreds of
> TB of actual data and files attached to singular systems.
> 
> In a recent OS update, I discovered my larger volumes no longer mount
> because md-linear has been removed entirely from the kernel as of 6.8.
> 
> I am trying to do rationale archaeology. I sent a note to Mr. Neil
> Brown who was responsible for the earliest change I found related to
> this and he suggested I email regressions and linux-raid along with
> the current maintainers about it.
> 
> What I've been able to find:
> 
> In 2604b703b6b3db80e3c75ce472a54dfd0b7bf9f4 (2006) Neil Brown marked a
> MODULE_ALIAS entry for md-personality-1 as deprecated but it appears
> the reason was because the form of the personality was changed (not
> that the underlying md-linear itself was deprecated.)
> 
> d9d166c2a9d5d01af34396793950aa695883eed4 (2006) reinforced this change
> via a diff algorithm that overzealously included that line in a diff
> chunk but which makes annotating prior to it a more manual process.
> 
> 608f52e30aae7dc8da836e5b7b112d50a2d00e43 (2021) marked md-linear as
> deprecated in Kconfig, using the rationale that md-linear was
> deprecated in MODULE_ALIAS—but again which doesn't explain why the
> *module* was deprecated and appears to me at least to accidentally
> misconstrue the original reason for the deprecation comment.
> 
> 849d18e27be9a1253f2318cb4549cc857219d991 (2023) eliminated md-linear
> entirely, again mostly self-referencing a deprecation notice which was
> there in actuality for basically multiple decades and seems to have
> referenced something else entirely.
> 
> I was hoping you could help me understand why this module was removed?
> I have found others who also are running into this. Functionality they
> relied on has disappeared, as per the existence of the following:
> 
> https://github.com/dougvj/md-linear
> 
> https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/34
> https://bbs.archlinux.org/viewtopic.php?id=294005
> (etc)
> 
> So, it looks like there are many of us who were still using mdadm to
> manage sub-device concatenation, again in my case for 100s of TB of
> admittedly casual data storage, and I can't currently find what the
> actual actual rationale was for removing it. :(
> 
> For utility's sake, I would like to suggest that linear volumes lessen
> problems like substriping. I do not think for many of us that
> shuffling around a few hundred TB is easy to do at any rate. Currently
> I'm manually re-compiling a fairly heavily-modified md-linear as a
> user-built module and it seems to work okay. I am definitely not the
> only one doing this.
> 
> Please consider resurrecting md-linear. :-)

I fully support keeping md-linear for users with existing deployments.

Wanted to only ask out of curiosity, did you try using md-raid0 for the same
scenario?

It can use different sized devices in RAID0. In case of two disks it will
stripe between them over the matching portion of the sizes, and then the tail
of the larger device will be accessed in a linear fashion. Not sure it can
handle 3 or more in this manner, will there be multiple steps of the striping,
each time with a smaller number of the remaining larger devices (but would not
be surprised if yes).

Given that a loss of one device in md-linear likely means complete data loss
anyway (relying on picking up pieces with data recovery tools is not a good
plan), seems like using md-raid0 here instead would have no downsides but
likely improve performance by a lot.

Aside from all that, the "industry" way to do the task of md-linear currently
would be a large LVM LV over a set of multiple PVs in a volume group.

-- 
With respect,
Roman

Re: md-linear accidental(?) removal, removed significant(?) use case?

[Yu Kuai]

Hi,

在 2025/01/02 20:16, Roman Mamedov 写道:
> On Wed, 1 Jan 2025 20:14:12 -0800
> Allen Toureg <thetanix@gmail.com> wrote:
> 
>> Preamble: I have extensive arrays of drives and due to their irregular
>> sizes, I was using md-linear as a convenient way to manually
>> concatenate arrays of underlying MD (raid5/etc) to manually deal with
>> redundancy.
>>
>> I have probably a few thousand TB in total raw space, and hundreds of
>> TB of actual data and files attached to singular systems.
>>
>> In a recent OS update, I discovered my larger volumes no longer mount
>> because md-linear has been removed entirely from the kernel as of 6.8.
>>
>> I am trying to do rationale archaeology. I sent a note to Mr. Neil
>> Brown who was responsible for the earliest change I found related to
>> this and he suggested I email regressions and linux-raid along with
>> the current maintainers about it.
>>
>> What I've been able to find:
>>
>> In 2604b703b6b3db80e3c75ce472a54dfd0b7bf9f4 (2006) Neil Brown marked a
>> MODULE_ALIAS entry for md-personality-1 as deprecated but it appears
>> the reason was because the form of the personality was changed (not
>> that the underlying md-linear itself was deprecated.)
>>
>> d9d166c2a9d5d01af34396793950aa695883eed4 (2006) reinforced this change
>> via a diff algorithm that overzealously included that line in a diff
>> chunk but which makes annotating prior to it a more manual process.
>>
>> 608f52e30aae7dc8da836e5b7b112d50a2d00e43 (2021) marked md-linear as
>> deprecated in Kconfig, using the rationale that md-linear was
>> deprecated in MODULE_ALIAS—but again which doesn't explain why the
>> *module* was deprecated and appears to me at least to accidentally
>> misconstrue the original reason for the deprecation comment.
>>
>> 849d18e27be9a1253f2318cb4549cc857219d991 (2023) eliminated md-linear
>> entirely, again mostly self-referencing a deprecation notice which was
>> there in actuality for basically multiple decades and seems to have
>> referenced something else entirely.
>>
>> I was hoping you could help me understand why this module was removed?
>> I have found others who also are running into this. Functionality they
>> relied on has disappeared, as per the existence of the following:
>>
>> https://github.com/dougvj/md-linear
>>
>> https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/34
>> https://bbs.archlinux.org/viewtopic.php?id=294005
>> (etc)
>>
>> So, it looks like there are many of us who were still using mdadm to
>> manage sub-device concatenation, again in my case for 100s of TB of
>> admittedly casual data storage, and I can't currently find what the
>> actual actual rationale was for removing it. :(
>>
>> For utility's sake, I would like to suggest that linear volumes lessen
>> problems like substriping. I do not think for many of us that
>> shuffling around a few hundred TB is easy to do at any rate. Currently
>> I'm manually re-compiling a fairly heavily-modified md-linear as a
>> user-built module and it seems to work okay. I am definitely not the
>> only one doing this.
>>
>> Please consider resurrecting md-linear. :-)
> 
> I fully support keeping md-linear for users with existing deployments.
> 
> Wanted to only ask out of curiosity, did you try using md-raid0 for the same
> scenario?
> 
> It can use different sized devices in RAID0. In case of two disks it will
> stripe between them over the matching portion of the sizes, and then the tail
> of the larger device will be accessed in a linear fashion. Not sure it can
> handle 3 or more in this manner, will there be multiple steps of the striping,
> each time with a smaller number of the remaining larger devices (but would not
> be surprised if yes).
> 
> Given that a loss of one device in md-linear likely means complete data loss
> anyway (relying on picking up pieces with data recovery tools is not a good
> plan), seems like using md-raid0 here instead would have no downsides but
> likely improve performance by a lot.
> 
> Aside from all that, the "industry" way to do the task of md-linear currently
> would be a large LVM LV over a set of multiple PVs in a volume group.

First of all, an exist md-linear can't switch to raid0, data will be
lost.

Then, for the case that disk with different size, for example:

sda 10T, sdb and sdc 12T.

Currently, in order to
most home NAS solution will create partition for sdb and sdc,
split it to 10T + 2T, then assemble two raid first:

md0:	sda + sdb1 + sdc1
md1:	sdb2 + sdc2

md0 can be 20T raid5, and md1 can be 2T raid1, and finially, assemble
new md-linear:

md2: md0 + md1

I agree that md-linear is better than raid0 in this case, user will
prefer to use the md0 first, and use md1 when md0 is exhausted.

Thanks,
Kuai

> 

Re: md-linear accidental(?) removal, removed significant(?) use case?

[Roman Mamedov]

On Thu, 2 Jan 2025 20:58:36 +0800
Yu Kuai <yukuai1@huaweicloud.com> wrote:

> First of all, an exist md-linear can't switch to raid0, data will be
> lost.

Yes, I did not mean to switch in place, but try out for new setups.

> Then, for the case that disk with different size, for example:
> 
> sda 10T, sdb and sdc 12T.
> 
> Currently, in order to
> most home NAS solution will create partition for sdb and sdc,
> split it to 10T + 2T, then assemble two raid first:
> 
> md0:	sda + sdb1 + sdc1
> md1:	sdb2 + sdc2
> 
> md0 can be 20T raid5, and md1 can be 2T raid1, and finially, assemble
> new md-linear:
> 
> md2: md0 + md1
> 
> I agree that md-linear is better than raid0 in this case, user will
> prefer to use the md0 first, and use md1 when md0 is exhausted.

Oh yes, in case this is made of the same physical devices under the hood,
raid0 of their partitions will bring only harm, not benefits.

-- 
With respect,
Roman

Re: md-linear accidental(?) removal, removed significant(?) use case?

[Pascal Hambourg]

On 02/01/2025 at 13:16, Roman Mamedov wrote:
> 
> I fully support keeping md-linear for users with existing deployments.

Of course. Breaking existing setups is bad.

> Wanted to only ask out of curiosity, did you try using md-raid0 for the same
> scenario?
> 
> It can use different sized devices in RAID0. In case of two disks it will
> stripe between them over the matching portion of the sizes, and then the tail
> of the larger device will be accessed in a linear fashion. Not sure it can
> handle 3 or more in this manner, will there be multiple steps of the striping,
> each time with a smaller number of the remaining larger devices (but would not
> be surprised if yes).

Yes. If I remember correctly, md-raid0 divides disks in as many zones as 
different disk sizes. The first zone contains the area equal the size of 
the smallest disk(s) on all disks, and the last zone contains the 
remaining area on the biggest disk(s).

> Given that a loss of one device in md-linear likely means complete data loss
> anyway (relying on picking up pieces with data recovery tools is not a good
> plan), seems like using md-raid0 here instead would have no downsides but
> likely improve performance by a lot.

A downside is that adding a disk to a RAID0 array requires a reshape.

> Aside from all that, the "industry" way to do the task of md-linear currently
> would be a large LVM LV over a set of multiple PVs in a volume group.

I fully agree. LVM adds some complexity but also provides much more 
flexibility. You cannot hot-swap, resize or remove a disk in a md-linear 
array.