* Re: Linear device of two arrays
From: Andreas Klauer @ 2017-07-07 22:33 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Veljko, linux-raid
In-Reply-To: <20170708025318.216210e7@natsu>
On Sat, Jul 08, 2017 at 02:53:18AM +0500, Roman Mamedov wrote:
> however it requires a shrinkable filesystem, and XFS cannot be shrunk.
And an easy way to adapt may be thus:
Just backup the last 100M of the filesystem. Then you can consider
the filesystem to be shrunk by 100M. Not really, but it covers
the space that will be lost in the LVM conversion (two extents).
When the conversion is done, you grow the LV back to original size,
(now that's using two extents on another PV you have to provide)
and restore the 100M you backed up to the LV.
The conversion is an offline operation anyways so it doesn't matter
whether you really shrank the filesystem or not, as long as you can
provide enough space for an LV of the original partition size.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Linear device of two arrays
From: Stan Hoeppner @ 2017-07-07 22:52 UTC (permalink / raw)
To: veljko3, linux-raid
In-Reply-To: <CALee7hZqmLK19tSEEFBgSWMKm9TRoUqLBCxUmbNJpwmM45BjTQ@mail.gmail.com>
On 07/07/2017 03:26 PM, Veljko wrote:
> I just noticed that I replied to Wol insted to list.
>
> On Wed, Jul 5, 2017 at 8:07 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 05/07/17 17:42, Roman Mamedov wrote:
>>> On Wed, 5 Jul 2017 17:34:09 +0200
>>> Veljko <veljko3@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have a RAID10 device which I have formated using the mkfs.xfs
>>>> defaults (Stan helped me with this few years back). I reached 88%
>>>> capacity and it is time to expand it. I bought 4 more drives to create
>>>> another RIAD10 array. I would like to create linear device out of
>>>> those two and grow XFS across the 2nd device. How can this be done
>>>> without loosing the existing device's data? I would also like to add a
>>>> spare HDD. Do I have to have a separate spare HDD for each array or
>>>> one can be used by both of them?
>>> Why make another RAID10? With modern versions of mdadm and kernel you should
>>> be able to simply reshape the current RAID10 to increase the number of
>>> devices used from 4 to 8.
>>>
>>>
>> I was thinking of replying, but isn't that not possible for some
>> versions of RAID-10?
>>
>> My feeling was, if you can't just add drives to the existing raid 10,
>> create a new one which you can expand, migrate the fs across (btrfs
>> would let you do that live, I believe, so xfs probably can too), then
>> you can scrap the old raid-10 and add the drives into the new one.
>>
>> Cheers,
>> Wol
>
> Thanks for your input, Roman and Wol.
>
> Expanding existing RAID is one of the options, but I was advised by
> Stan Hoeppner to do it this way and I tend to believe him on this
> subject. With my metadata heavy backup workload, this will provide
> better performance.
>
> So my question is still, how can an existing array be added to linear
> device, and it's file system expanded over the second array.
For this to work the existing RAID10 array must already be a member of a
linear device with one component device. If this linear array already
exists then you could add another RAID10 array to the linear device. If
you currently have an XFS filesystem sitting atop a 'bare' RAID10 then I
don't believe the linear option will work for you. Thus I'd tend to
agree with others that reshaping your current RAID10 is the best option.
My apologies if I wasn't clear in my previous advice.
Stan
> And there is also question of spare drive.
>
> Regards,
> Veljko
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 0:41 UTC (permalink / raw)
To: Linux Raid
Hi,
I asked in other linux forums and did not get enough info. So I come
here even though this is not mdadm/RAID issue.
I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
I had 32bit version of ext4 on it which can only be grown to 16TB. Any
one had this issue before and any _/inplace/_ good solution to grow ext4
to full 24TB? It is unlikely that I will be able to back up and recreate
file system.
My filesystem is on md0 drive so I guess partitioning and making into 2x
12TB ext4 will not work. I am not even sure if we can partition md like
any other disk.
On the web, I only found one solution that required upgrading kernel to
some very recent one (not in my distro) and getting the bleeding edge
resize2fs. This makes me nervous. Is there a solution that avoids this.
Ramesh
^ permalink raw reply
* [GIT PULL] MD update for 4.13
From: Shaohua Li @ 2017-07-08 1:09 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid, neilb
Hi,
Please pull MD update for 4.13. This update only includes several bug fixes:
- Neil Brown fixes a deadlock in MD suspend and a potential bug in bio allocation.
- Mikulas Patocka fixes a signal issue
- Guoqing Jiang fixes a typo in FailFast test
- Other trival fixes
Please note, there is a merge conflict with block tree and causes build error,
which could be fixed by this one:
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c4d41b03324b..8cdca0296749 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5450,7 +5450,7 @@ int md_run(struct mddev *mddev)
return -ENOMEM;
}
if (mddev->sync_set == NULL) {
- mddev->sync_set = bioset_create(BIO_POOL_SIZE, 0);
+ mddev->sync_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (!mddev->sync_set)
return -ENOMEM;
}
Thanks,
Shaohua
The following changes since commit 63f700aab4c11d46626de3cd051dae56cf7e9056:
Merge tag 'xtensa-20170612' of git://github.com/jcmvbkbc/linux-xtensa (2017-06-13 15:09:10 +0900)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git for-next
for you to fetch changes up to 7184ef8bab0cb865c3cea9dd1a675771145df0af:
MD: fix sleep in atomic (2017-07-03 14:38:59 -0700)
----------------------------------------------------------------
Guoqing Jiang (2):
md/raid10: fix FailFast test for wrong device
md/raid1: remove unused bio in sync_request_write
Lidong Zhong (1):
md: change the initialization value for a spare device spot to MD_DISK_ROLE_SPARE
Mikulas Patocka (1):
md: don't use flush_signals in userspace processes
NeilBrown (2):
md: fix deadlock between mddev_suspend() and md_write_start()
md: use a separate bio_set for synchronous IO.
Shaohua Li (2):
MD: fix a null dereference
MD: fix sleep in atomic
drivers/md/faulty.c | 5 +++--
drivers/md/linear.c | 7 ++++---
drivers/md/md.c | 47 ++++++++++++++++++++++++++++++++++++++---------
drivers/md/md.h | 7 +++++--
drivers/md/multipath.c | 8 ++++----
drivers/md/raid0.c | 7 ++++---
drivers/md/raid1.c | 20 ++++++++++++--------
drivers/md/raid10.c | 16 +++++++++-------
drivers/md/raid5.c | 22 +++++++++++++---------
9 files changed, 92 insertions(+), 47 deletions(-)
^ permalink raw reply related
* Re: My MD is too big to resize ext4.
From: Andreas Klauer @ 2017-07-08 6:50 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <6c827d07-19d8-017b-ca95-5e6f84b7821a@gmail.com>
On Fri, Jul 07, 2017 at 07:41:13PM -0500, Ram Ramesh wrote:
> On the web, I only found one solution that required upgrading kernel to
> some very recent one (not in my distro) and getting the bleeding edge
> resize2fs. This makes me nervous. Is there a solution that avoids this.
Well, if that works nowadays, then just go for it (LiveCD).
For details you should ask the ext4 mailing list I guess.
You can assemble your RAID in read-only mode and then use
overlays for a non-destructive test run. To make really extra sure
you could even export it via NBD/KVM to see if the old kernel
is able to mount it.
IIRC, btrfs also has a migration path (ext4 to btrfs converter)
and then you could grow that. But then you're stuck with btrfs.
> My filesystem is on md0 drive so I guess partitioning and making into 2x
> 12TB ext4 will not work. I am not even sure if we can partition md like
> any other disk.
Adding a partition table or LVM header would shift the start position.
So it would involve moving/shifting all your data in-place or using
a conversion to LVM. (We just had that in the thread above yours,
subject "linear device of two arrays")
If your RAID has a large enough data offset, you could shift that
to add some extra space at the start of the RAID, but it has to
be done with great care (must be aligned to whatever your RAID
layout is so data will be intact and not garbled).
All in-place operations are kind of hackish/dangerous.
I actually prefer using several smaller filesystems over one gigantic one.
It's all great until your one and only filesystem goes corrupt.
Regards
Andreas Klauer
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Brad Campbell @ 2017-07-08 7:34 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <6c827d07-19d8-017b-ca95-5e6f84b7821a@gmail.com>
On 08/07/17 08:41, Ram Ramesh wrote:
> Hi,
>
> I asked in other linux forums and did not get enough info. So I come
> here even though this is not mdadm/RAID issue.
>
> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
> one had this issue before and any _/inplace/_ good solution to grow ext4
> to full 24TB? It is unlikely that I will be able to back up and recreate
> file system.
Ignore this first suggestion if you are absolutely reliant on data
integrity and don't have backups of any critical data.
Take 2 of the 6TB drives, create a 12TB RAID0 or Linear and copy all
your data to that.
Create the 6 drive RAID6 with 2 disks missing, copy all the data back,
then add the disks you used in the first copy back to the array.
I had this issue a few years ago and decided buying some extra disks to
use as scratch was a better move for me. In your shoes I would have
found 6 extra SATA ports on a machine (any machine), created the new
filesystem and rsynced the data across, then swapped the disks into the
production machine.
In fact I've just done that migrating from 14x2TB drives to 8x6TB
drives, but I keep a staging machine around for testing that has 15
drive bays for just this sort of migration.
This is the 4th time I've done it in 13 years. 15x250GB -> 8x1TB ->
14x2TB -> 8x6TB. The advantage is you never lose data integrity or
redundancy, and you can keep the old drives around in a box until you
are sure the new build is reliable.
Alternatively, find a machine with 6 slots, build a RAID6 from the
drives you already have. Copy the data across, re-format the new array
and copy the data back. A bit of double handling, but no loss of
redundancy or integrity. Run md5 or something over all the files to make
sure it's all good, or use rsync with the "always checksum" option,
although this is slow.
Brad
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Reindl Harald @ 2017-07-08 7:40 UTC (permalink / raw)
To: Brad Campbell, Ram Ramesh, Linux Raid
In-Reply-To: <02786811-5d00-af90-adfd-82ae3f9f0781@fnarfbargle.com>
Am 08.07.2017 um 09:34 schrieb Brad Campbell:
> Alternatively, find a machine with 6 slots, build a RAID6 from the
> drives you already have. Copy the data across, re-format the new array
> and copy the data back. A bit of double handling, but no loss of
> redundancy or integrity. Run md5 or something over all the files to make
> sure it's all good, or use rsync with the "always checksum" option,
> although this is slow
--checksum
If the file size and time match, it will do a checksum at both ends to
see if the files are really identical
that won't do much when the target is empty
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Brad Campbell @ 2017-07-08 8:12 UTC (permalink / raw)
To: Reindl Harald, Ram Ramesh, Linux Raid
In-Reply-To: <9dee7eb4-795e-a30e-8cd6-d38d0ad9891b@thelounge.net>
On 08/07/17 15:40, Reindl Harald wrote:
>
>
> Am 08.07.2017 um 09:34 schrieb Brad Campbell:
>> Alternatively, find a machine with 6 slots, build a RAID6 from the
>> drives you already have. Copy the data across, re-format the new array
>> and copy the data back. A bit of double handling, but no loss of
>> redundancy or integrity. Run md5 or something over all the files to
>> make sure it's all good, or use rsync with the "always checksum"
>> option, although this is slow
>
> --checksum
> If the file size and time match, it will do a checksum at both ends to
> see if the files are really identical
>
> that won't do much when the target is empty
>
Sorry, if you read the man page I thought it'd be pretty obvious you
need to run it twice and I didn't need to explicitly spell it out.
Brad
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Wols Lists @ 2017-07-08 8:30 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <6c827d07-19d8-017b-ca95-5e6f84b7821a@gmail.com>
On 08/07/17 01:41, Ram Ramesh wrote:
> Hi,
>
> I asked in other linux forums and did not get enough info. So I come
> here even though this is not mdadm/RAID issue.
>
> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
> one had this issue before and any _/inplace/_ good solution to grow ext4
> to full 24TB? It is unlikely that I will be able to back up and recreate
> file system.
>
> My filesystem is on md0 drive so I guess partitioning and making into 2x
> 12TB ext4 will not work. I am not even sure if we can partition md like
> any other disk.
You can partition an array and create partitions in it - caveat I
haven't done it, and I've never seen anyone here mention that they've
done it - lvm seems to be the more popular version.
>
> On the web, I only found one solution that required upgrading kernel to
> some very recent one (not in my distro) and getting the bleeding edge
> resize2fs. This makes me nervous. Is there a solution that avoids this.
>
resize2fs is bleeding edge? I suspect it's v0.99 quality, ie nobody has
the nerve to upgrade it to v1, despite it being rock solid. I know I've
used it without trouble. Most of these utilities are pretty solid
(unless the underlying filesystem itself is experimental ...)
Cheers,
Wol
^ permalink raw reply
* Re: Linear device of two arrays
From: Veljko @ 2017-07-08 10:26 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <df053262-9334-1070-c24a-5d2f91402716@hardwarefreak.org>
On Sat, Jul 8, 2017 at 12:52 AM, Stan Hoeppner <stan@hardwarefreak.org> wrote:
> On 07/07/2017 03:26 PM, Veljko wrote:
>>
>> I just noticed that I replied to Wol insted to list.
>>
>> On Wed, Jul 5, 2017 at 8:07 PM, Wols Lists <antlists@youngman.org.uk>
>> wrote:
>>>
>>> On 05/07/17 17:42, Roman Mamedov wrote:
>>>>
>>>> On Wed, 5 Jul 2017 17:34:09 +0200
>>>> Veljko <veljko3@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have a RAID10 device which I have formated using the mkfs.xfs
>>>>> defaults (Stan helped me with this few years back). I reached 88%
>>>>> capacity and it is time to expand it. I bought 4 more drives to create
>>>>> another RIAD10 array. I would like to create linear device out of
>>>>> those two and grow XFS across the 2nd device. How can this be done
>>>>> without loosing the existing device's data? I would also like to add a
>>>>> spare HDD. Do I have to have a separate spare HDD for each array or
>>>>> one can be used by both of them?
>>>>
>>>> Why make another RAID10? With modern versions of mdadm and kernel you
>>>> should
>>>> be able to simply reshape the current RAID10 to increase the number of
>>>> devices used from 4 to 8.
>>>>
>>>>
>>> I was thinking of replying, but isn't that not possible for some
>>> versions of RAID-10?
>>>
>>> My feeling was, if you can't just add drives to the existing raid 10,
>>> create a new one which you can expand, migrate the fs across (btrfs
>>> would let you do that live, I believe, so xfs probably can too), then
>>> you can scrap the old raid-10 and add the drives into the new one.
>>>
>>> Cheers,
>>> Wol
>>
>>
>> Thanks for your input, Roman and Wol.
>>
>> Expanding existing RAID is one of the options, but I was advised by
>> Stan Hoeppner to do it this way and I tend to believe him on this
>> subject. With my metadata heavy backup workload, this will provide
>> better performance.
>>
>> So my question is still, how can an existing array be added to linear
>> device, and it's file system expanded over the second array.
>
> For this to work the existing RAID10 array must already be a member of a
> linear device with one component device. If this linear array already
> exists then you could add another RAID10 array to the linear device. If you
> currently have an XFS filesystem sitting atop a 'bare' RAID10 then I don't
> believe the linear option will work for you. Thus I'd tend to agree with
> others that reshaping your current RAID10 is the best option.
>
> My apologies if I wasn't clear in my previous advice.
>
> Stan
Here is your previous advice:
"Do not use LVM. Directly format the RAID10 device using the mkfs.xfs
defaults. mkfs.xfs will read the md configuration and automatically
align the filesystem to the stripe width.
When the filesystem reaches 85% capacity, add 4 more drives and create
another RAID10 array. At that point we'll teach you how to create a
linear device of the two arrays and grow XFS across the 2nd array."
From this I concluded that it is possible to create linear device
using existing array, but since it is not the case, I'll just have to
create new array, move data to it, than add first array to a new
linear device (1 member), copy my data to it, and than join second
device to it. I wanted to avoid all this copying but will have to do
it. Than I will add one drive to spare-group both arrays will be a
members of, as advised by Andreas. Does this sound OK?
Regards,
Veljko
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Roman Mamedov @ 2017-07-08 11:15 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <6c827d07-19d8-017b-ca95-5e6f84b7821a@gmail.com>
On Fri, 7 Jul 2017 19:41:13 -0500
Ram Ramesh <rramesh2400@gmail.com> wrote:
> On the web, I only found one solution that required upgrading kernel to
> some very recent one (not in my distro) and getting the bleeding edge
> resize2fs. This makes me nervous. Is there a solution that avoids this.
Considering the other possible options that have been mentioned, using the
Ext4 built-in larger devices support (which has been implemented recently)
seems to be your best bet.
https://askubuntu.com/questions/779754/how-do-i-resize-an-ext4-partition-beyond-the-16tb-limit
The required e2fsprogs version 1.43 is included in Ubuntu 16.10 by now, so you
don't even need to build it from the source.
https://packages.ubuntu.com/yakkety/e2fsprogs
But really, if you have everything on one array with a single huge filesystem
and no backups, that's just asking for trouble and a complete data loss.
--
With respect,
Roman
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 18:12 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Linux Raid
In-Reply-To: <20170708065040.GA3682@metamorpher.de>
On 07/08/2017 01:50 AM, Andreas Klauer wrote:
> On Fri, Jul 07, 2017 at 07:41:13PM -0500, Ram Ramesh wrote:
>> On the web, I only found one solution that required upgrading kernel to
>> some very recent one (not in my distro) and getting the bleeding edge
>> resize2fs. This makes me nervous. Is there a solution that avoids this.
> Well, if that works nowadays, then just go for it (LiveCD).
> For details you should ask the ext4 mailing list I guess.
>
> You can assemble your RAID in read-only mode and then use
> overlays for a non-destructive test run. To make really extra sure
> you could even export it via NBD/KVM to see if the old kernel
> is able to mount it.
>
> IIRC, btrfs also has a migration path (ext4 to btrfs converter)
> and then you could grow that. But then you're stuck with btrfs.
>
>> My filesystem is on md0 drive so I guess partitioning and making into 2x
>> 12TB ext4 will not work. I am not even sure if we can partition md like
>> any other disk.
> Adding a partition table or LVM header would shift the start position.
> So it would involve moving/shifting all your data in-place or using
> a conversion to LVM. (We just had that in the thread above yours,
> subject "linear device of two arrays")
>
> If your RAID has a large enough data offset, you could shift that
> to add some extra space at the start of the RAID, but it has to
> be done with great care (must be aligned to whatever your RAID
> layout is so data will be intact and not garbled).
>
> All in-place operations are kind of hackish/dangerous.
>
> I actually prefer using several smaller filesystems over one gigantic one.
> It's all great until your one and only filesystem goes corrupt.
>
> Regards
> Andreas Klauer
Thanks. After thinking a bit more, I also like several smaller ext4/mds
than large one. Since this is a mythtv machine, it can share multiple
mount point and distribute data smartly (in its own sense).
In order to create several md devices out of the 6x6TB drives I have, I
have to do the following. Let me know if this sounds possible.
1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
I can create partitions on the drive instead of on the MD.
2. This means I need to shrink my current md device to smaller size
(say 12TB-14TB) - need to check my current
active ext4 data size. It is definitely less than 16TB.
3. Repartition the disks to create sd{b-g}2 for the reamining unused
6xnTB area.
4. Created md1 with sd{b-g]2 to get md1.
5. Mount and use md1.
Do you see every step here reasonably safe to try on a
unmounted/readonly md disks.
Ramesh
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 18:19 UTC (permalink / raw)
To: Brad Campbell, Linux Raid
In-Reply-To: <02786811-5d00-af90-adfd-82ae3f9f0781@fnarfbargle.com>
On 07/08/2017 02:34 AM, Brad Campbell wrote:
> On 08/07/17 08:41, Ram Ramesh wrote:
>> Hi,
>>
>> I asked in other linux forums and did not get enough info. So I come
>> here even though this is not mdadm/RAID issue.
>>
>> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
>> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
>> one had this issue before and any _/inplace/_ good solution to grow ext4
>> to full 24TB? It is unlikely that I will be able to back up and recreate
>> file system.
>
> Ignore this first suggestion if you are absolutely reliant on data
> integrity and don't have backups of any critical data.
>
> Take 2 of the 6TB drives, create a 12TB RAID0 or Linear and copy all
> your data to that.
>
> Create the 6 drive RAID6 with 2 disks missing, copy all the data back,
> then add the disks you used in the first copy back to the array.
>
>
> I had this issue a few years ago and decided buying some extra disks
> to use as scratch was a better move for me. In your shoes I would have
> found 6 extra SATA ports on a machine (any machine), created the new
> filesystem and rsynced the data across, then swapped the disks into
> the production machine.
>
> In fact I've just done that migrating from 14x2TB drives to 8x6TB
> drives, but I keep a staging machine around for testing that has 15
> drive bays for just this sort of migration.
>
> This is the 4th time I've done it in 13 years. 15x250GB -> 8x1TB ->
> 14x2TB -> 8x6TB. The advantage is you never lose data integrity or
> redundancy, and you can keep the old drives around in a box until you
> are sure the new build is reliable.
>
> Alternatively, find a machine with 6 slots, build a RAID6 from the
> drives you already have. Copy the data across, re-format the new array
> and copy the data back. A bit of double handling, but no loss of
> redundancy or integrity. Run md5 or something over all the files to
> make sure it's all good, or use rsync with the "always checksum"
> option, although this is slow.
>
> Brad
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks for the details. I think I have 2x4TB and 2x3TB spares. This is
enough for the data I have and I have scripted a way to distribute files
across those to make a mock backup/snapshot. They are old drives and I
do not trust their shelf life. So, I consider that there is no backup.
However, for short/week exercise, it may be considered as a safe data.
So, I am not that paranoid about doing inplace changes. I just don't
like to choose known risky paths - a more tested approach is preferred
even if it has nonzero propability of failure. That is all.
Ramesh
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 18:28 UTC (permalink / raw)
To: Wols Lists, Linux Raid
In-Reply-To: <5960981C.9040906@youngman.org.uk>
On 07/08/2017 03:30 AM, Wols Lists wrote:resize2fs is bleeding edge? I
suspect it's v0.99 quality, ie nobody has
> the nerve to upgrade it to v1, despite it being rock solid. I know I've
> used it without trouble. Most of these utilities are pretty solid
> (unless the underlying filesystem itself is experimental ...)
>
> Cheers,
> Wol
>
The web solution talks about not yet released resize2fs source and
compiling in latest kernel etc. That is bleeding edge to me. I like to
stick to stable versions of distro.
My days of young and adventurous are gone. I have been with Linus/linux
when he released some mock alpha kernlel that would only print A/B to
show context switching. I remember the lengthy conversation/argument
that he had with Prof. Tannenbaum (sp?) on monolithic vs. micro kernel.
I enjoyed compiling kernel and reading code to figure out why some thing
did not work. However that is a long time ago. I just have aged, become
less current and rusty with tricks. I use stable distros and stick the
trusted kernel/packages in them now. So, in that context, would you not
say resize2fs approach is bleeding edge :-)
Ramesh
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 18:37 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux Raid
In-Reply-To: <20170708161513.37b09dbc@natsu>
On 07/08/2017 06:15 AM, Roman Mamedov wrote:
> On Fri, 7 Jul 2017 19:41:13 -0500
> Ram Ramesh <rramesh2400@gmail.com> wrote:
>
>> On the web, I only found one solution that required upgrading kernel to
>> some very recent one (not in my distro) and getting the bleeding edge
>> resize2fs. This makes me nervous. Is there a solution that avoids this.
> Considering the other possible options that have been mentioned, using the
> Ext4 built-in larger devices support (which has been implemented recently)
> seems to be your best bet.
> https://askubuntu.com/questions/779754/how-do-i-resize-an-ext4-partition-beyond-the-16tb-limit
>
> The required e2fsprogs version 1.43 is included in Ubuntu 16.10 by now, so you
> don't even need to build it from the source.
> https://packages.ubuntu.com/yakkety/e2fsprogs
>
> But really, if you have everything on one array with a single huge filesystem
> and no backups, that's just asking for trouble and a complete data loss.
>
Thanks. This is exactly the one that I was talking about as a web
solution. Did not know it is available in 16.10.
Besides the above, I hear your last caution well. I do have backup by
distributing files using rsync over multiple (old) disks (4+4+3+3TB). I
just don't trust my back up and thus pretend that I do not have one.
However, for a short one week trial, I think, I am ok.
Going forward, I will not be able to backup 24TB volume (if I succeed in
building one big MD) as I will not have enough drives to distribute
data. So, there is going to be a problem. So, I am thinking seriously
building several smaller MD devices out of my 6x6TB drives. Look at my
reply to Andreas. Let me know your thoughts on that. If I build smaller
MDs each can be temporarily backed up for updates etc. So, I will still
be ok with data as long as I work only on one MD at a time that fits my
set of backup disks.
Ramesh
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 18:48 UTC (permalink / raw)
To: Brad Campbell, Linux Raid
In-Reply-To: <f4e57040-177d-39b9-ee30-95b82b1ac15a@gmail.com>
>> I had this issue a few years ago and decided buying some extra disks
>> to use as scratch was a better move for me. In your shoes I would
>> have found 6 extra SATA ports on a machine (any machine), created the
>> new filesystem and rsynced the data across, then swapped the disks
>> into the production machine.
Brad,
I try to keep disks from different production batch in my MD as
proabilistically speaking, this is safe. This means I buy one disk every
year and replace the oldest one. This time it is special case that My
last 3TB got replaced with new 6TB and now I can actually expand the
array. So, the above method may not always work for me. However, I see
your point and therefore see merit to it. In any case, I am taking
Andreas (and few others) advice and try to build several smaller MD
devices instead of one large one. I responded to him on this idea. Let
me see if any one sees issues with that.
Ramesh
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Andreas Klauer @ 2017-07-08 19:44 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <ae47b321-07d1-2157-c217-844447e51ce2@gmail.com>
On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
> 1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
> I can create partitions on the drive instead of on the MD.
> 2. This means I need to shrink my current md device to smaller size
> (say 12TB-14TB) - need to check my current
> active ext4 data size. It is definitely less than 16TB.
> 3. Repartition the disks to create sd{b-g}2 for the reamining unused
> 6xnTB area.
> 4. Created md1 with sd{b-g]2 to get md1.
> 5. Mount and use md1.
Should work; you have to shrink the filesystem first, (or stick to
whatever size it has now), then the md, then the partition.
What makes this risky is that you have to pick the correct sizes.
When growing you can't go wrong. When shrinking, you have to be
careful not to shrink too much. Filesystems don't like it at all
if their end is missing, and md doesn't like it if the block device
is smaller than what it says in the metadata.
So you have to determine the exact filesystem size (tune2fs -l),
and take mdadm data offsets into account.
Not necessary if you know what you're doing but if in doubt,
you can leave a safety margin with each of these steps.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Linear device of two arrays
From: Stan Hoeppner @ 2017-07-08 21:24 UTC (permalink / raw)
To: veljko3, linux-raid, neilb
In-Reply-To: <CALee7haowPD0DKPtEQUN8M8LDCNOq62SDrk5VNMO5C+5iUqtrQ@mail.gmail.com>
On 07/08/2017 05:26 AM, Veljko wrote:
> On Sat, Jul 8, 2017 at 12:52 AM, Stan Hoeppner <stan@hardwarefreak.org> wrote:
>> On 07/07/2017 03:26 PM, Veljko wrote:
>>> I just noticed that I replied to Wol insted to list.
>>>
>>> On Wed, Jul 5, 2017 at 8:07 PM, Wols Lists <antlists@youngman.org.uk>
>>> wrote:
>>>> On 05/07/17 17:42, Roman Mamedov wrote:
>>>>> On Wed, 5 Jul 2017 17:34:09 +0200
>>>>> Veljko <veljko3@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have a RAID10 device which I have formated using the mkfs.xfs
>>>>>> defaults (Stan helped me with this few years back). I reached 88%
>>>>>> capacity and it is time to expand it. I bought 4 more drives to create
>>>>>> another RIAD10 array. I would like to create linear device out of
>>>>>> those two and grow XFS across the 2nd device. How can this be done
>>>>>> without loosing the existing device's data? I would also like to add a
>>>>>> spare HDD. Do I have to have a separate spare HDD for each array or
>>>>>> one can be used by both of them?
>>>>> Why make another RAID10? With modern versions of mdadm and kernel you
>>>>> should
>>>>> be able to simply reshape the current RAID10 to increase the number of
>>>>> devices used from 4 to 8.
>>>>>
>>>>>
>>>> I was thinking of replying, but isn't that not possible for some
>>>> versions of RAID-10?
>>>>
>>>> My feeling was, if you can't just add drives to the existing raid 10,
>>>> create a new one which you can expand, migrate the fs across (btrfs
>>>> would let you do that live, I believe, so xfs probably can too), then
>>>> you can scrap the old raid-10 and add the drives into the new one.
>>>>
>>>> Cheers,
>>>> Wol
>>>
>>> Thanks for your input, Roman and Wol.
>>>
>>> Expanding existing RAID is one of the options, but I was advised by
>>> Stan Hoeppner to do it this way and I tend to believe him on this
>>> subject. With my metadata heavy backup workload, this will provide
>>> better performance.
>>>
>>> So my question is still, how can an existing array be added to linear
>>> device, and it's file system expanded over the second array.
>> For this to work the existing RAID10 array must already be a member of a
>> linear device with one component device. If this linear array already
>> exists then you could add another RAID10 array to the linear device. If you
>> currently have an XFS filesystem sitting atop a 'bare' RAID10 then I don't
>> believe the linear option will work for you. Thus I'd tend to agree with
>> others that reshaping your current RAID10 is the best option.
>>
>> My apologies if I wasn't clear in my previous advice.
>>
>> Stan
> Here is your previous advice:
> "Do not use LVM. Directly format the RAID10 device using the mkfs.xfs
> defaults. mkfs.xfs will read the md configuration and automatically
> align the filesystem to the stripe width.
>
> When the filesystem reaches 85% capacity, add 4 more drives and create
> another RAID10 array. At that point we'll teach you how to create a
> linear device of the two arrays and grow XFS across the 2nd array."
>
> From this I concluded that it is possible to create linear device
> using existing array, but since it is not the case, I'll just have to
> create new array, move data to it, than add first array to a new
> linear device (1 member), copy my data to it, and than join second
> device to it. I wanted to avoid all this copying but will have to do
> it. Than I will add one drive to spare-group both arrays will be a
> members of, as advised by Andreas. Does this sound OK?
Yes. Although, I was hoping Neil Brown would chime in here. I'm not
entirely sure that you can't do this the way I originally mentioned.
> Regards,
> Veljko
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-08 22:22 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Linux Raid
In-Reply-To: <20170708194427.GA15208@metamorpher.de>
On 07/08/2017 02:44 PM, Andreas Klauer wrote:
> On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
>> 1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
>> I can create partitions on the drive instead of on the MD.
>> 2. This means I need to shrink my current md device to smaller size
>> (say 12TB-14TB) - need to check my current
>> active ext4 data size. It is definitely less than 16TB.
>> 3. Repartition the disks to create sd{b-g}2 for the reamining unused
>> 6xnTB area.
>> 4. Created md1 with sd{b-g]2 to get md1.
>> 5. Mount and use md1.
> Should work; you have to shrink the filesystem first, (or stick to
> whatever size it has now), then the md, then the partition.
>
> What makes this risky is that you have to pick the correct sizes.
> When growing you can't go wrong. When shrinking, you have to be
> careful not to shrink too much. Filesystems don't like it at all
> if their end is missing, and md doesn't like it if the block device
> is smaller than what it says in the metadata.
>
> So you have to determine the exact filesystem size (tune2fs -l),
> and take mdadm data offsets into account.
>
> Not necessary if you know what you're doing but if in doubt,
> you can leave a safety margin with each of these steps.
>
> Regards
> Andreas Klauer
My df shows that I use 10431258240 blocks (1K) in my ext4 on the md.
This is less than 10T. So, resize2fs to 11T should be fine. I will
shrink md to 11.5T and partition to 12T. I will use 2^30 to mean TB. Is
0.5T enough margin in each case?
I can also stretch the limit to 15TB for ext4 and 15.5 for md and 16TB
for partiton. Is this better because it maxes out the 32bit
ext4? Somehow something tells me pushing the limit is always a bad idea
as mistakes generally happen on checking boundary conditions.
Alternatively, I can do resize2fs with -M to minimal size, but the man
says calculations are not perfect for 1/2K block sizes. However, I see
that dumpe2fs says I have 4K blocks.
Ramesh
^ permalink raw reply
* Re: Linear device of two arrays
From: NeilBrown @ 2017-07-09 22:37 UTC (permalink / raw)
To: Stan Hoeppner, veljko3, linux-raid
In-Reply-To: <5afd5cea-06b8-8524-e4f5-8534f23bd714@hardwarefreak.org>
[-- Attachment #1: Type: text/plain, Size: 4714 bytes --]
On Sat, Jul 08 2017, Stan Hoeppner wrote:
> On 07/08/2017 05:26 AM, Veljko wrote:
>> On Sat, Jul 8, 2017 at 12:52 AM, Stan Hoeppner <stan@hardwarefreak.org> wrote:
>>> On 07/07/2017 03:26 PM, Veljko wrote:
>>>> I just noticed that I replied to Wol insted to list.
>>>>
>>>> On Wed, Jul 5, 2017 at 8:07 PM, Wols Lists <antlists@youngman.org.uk>
>>>> wrote:
>>>>> On 05/07/17 17:42, Roman Mamedov wrote:
>>>>>> On Wed, 5 Jul 2017 17:34:09 +0200
>>>>>> Veljko <veljko3@gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have a RAID10 device which I have formated using the mkfs.xfs
>>>>>>> defaults (Stan helped me with this few years back). I reached 88%
>>>>>>> capacity and it is time to expand it. I bought 4 more drives to create
>>>>>>> another RIAD10 array. I would like to create linear device out of
>>>>>>> those two and grow XFS across the 2nd device. How can this be done
>>>>>>> without loosing the existing device's data? I would also like to add a
>>>>>>> spare HDD. Do I have to have a separate spare HDD for each array or
>>>>>>> one can be used by both of them?
>>>>>> Why make another RAID10? With modern versions of mdadm and kernel you
>>>>>> should
>>>>>> be able to simply reshape the current RAID10 to increase the number of
>>>>>> devices used from 4 to 8.
>>>>>>
>>>>>>
>>>>> I was thinking of replying, but isn't that not possible for some
>>>>> versions of RAID-10?
>>>>>
>>>>> My feeling was, if you can't just add drives to the existing raid 10,
>>>>> create a new one which you can expand, migrate the fs across (btrfs
>>>>> would let you do that live, I believe, so xfs probably can too), then
>>>>> you can scrap the old raid-10 and add the drives into the new one.
>>>>>
>>>>> Cheers,
>>>>> Wol
>>>>
>>>> Thanks for your input, Roman and Wol.
>>>>
>>>> Expanding existing RAID is one of the options, but I was advised by
>>>> Stan Hoeppner to do it this way and I tend to believe him on this
>>>> subject. With my metadata heavy backup workload, this will provide
>>>> better performance.
>>>>
>>>> So my question is still, how can an existing array be added to linear
>>>> device, and it's file system expanded over the second array.
>>> For this to work the existing RAID10 array must already be a member of a
>>> linear device with one component device. If this linear array already
>>> exists then you could add another RAID10 array to the linear device. If you
>>> currently have an XFS filesystem sitting atop a 'bare' RAID10 then I don't
>>> believe the linear option will work for you. Thus I'd tend to agree with
>>> others that reshaping your current RAID10 is the best option.
>>>
>>> My apologies if I wasn't clear in my previous advice.
>>>
>>> Stan
>> Here is your previous advice:
>> "Do not use LVM. Directly format the RAID10 device using the mkfs.xfs
>> defaults. mkfs.xfs will read the md configuration and automatically
>> align the filesystem to the stripe width.
>>
>> When the filesystem reaches 85% capacity, add 4 more drives and create
>> another RAID10 array. At that point we'll teach you how to create a
>> linear device of the two arrays and grow XFS across the 2nd array."
>>
>> From this I concluded that it is possible to create linear device
>> using existing array, but since it is not the case, I'll just have to
>> create new array, move data to it, than add first array to a new
>> linear device (1 member), copy my data to it, and than join second
>> device to it. I wanted to avoid all this copying but will have to do
>> it. Than I will add one drive to spare-group both arrays will be a
>> members of, as advised by Andreas. Does this sound OK?
>
> Yes. Although, I was hoping Neil Brown would chime in here. I'm not
> entirely sure that you can't do this the way I originally mentioned.
I wasn't clear to me that I needed to chime in.. and the complete lack
of details (not even an "mdadm --examine" output), meant I could only
answer in vague generalizations.
However, seeing you asked.
If you really want to have a 'linear' of 2 RAID10s, then
0/ unmount the xfs filesystem
1/ backup the last few megabytes of the device
dd if=/dev/mdXX of=/safe/place/backup bs=1M skip=$BIGNUM
2/ create a linear array of the two RAID10s, ensuring the
metadata is v1.0, and the dataoffset is zero (should be default with
1.0)
mdadm -C /dev/mdZZ -l linear -n 2 -e 1.0 --data-offset=0 /dev/mdXX /dev/mdYY
3/ restore the saved data
dd of=/dev/mdZZ if=/safe/place/backup bs=1M seek=$BIGNUM
4/ grow the xfs filesystem
5/ be happy.
I cannot comment on the values of "few" and "$BUGNUM" without seeing
specifics.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: [PATCH v3 05/14] md: raid1: don't use bio's vec table to manage resync pages
From: NeilBrown @ 2017-07-09 23:09 UTC (permalink / raw)
To: Shaohua Li, Jens Axboe, linux-raid, linux-block,
Christoph Hellwig; +Cc: Ming Lei
In-Reply-To: <20170316161235.27110-6-tom.leiming@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3988 bytes --]
On Fri, Mar 17 2017, Ming Lei wrote:
> Now we allocate one page array for managing resync pages, instead
> of using bio's vec table to do that, and the old way is very hacky
> and won't work any more if multipage bvec is enabled.
>
> The introduced cost is that we need to allocate (128 + 16) * raid_disks
> bytes per r1_bio, and it is fine because the inflight r1_bio for
> resync shouldn't be much, as pointed by Shaohua.
>
> Also the bio_reset() in raid1_sync_request() is removed because
> all bios are freshly new now and not necessary to reset any more.
>
> This patch can be thought as a cleanup too
>
> Suggested-by: Shaohua Li <shli@kernel.org>
> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> ---
> drivers/md/raid1.c | 94 +++++++++++++++++++++++++++++++++++++-----------------
> 1 file changed, 64 insertions(+), 30 deletions(-)
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index e30d89690109..0e64beb60e4d 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -80,6 +80,24 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
> #define raid1_log(md, fmt, args...) \
> do { if ((md)->queue) blk_add_trace_msg((md)->queue, "raid1 " fmt, ##args); } while (0)
>
> +/*
> + * 'strct resync_pages' stores actual pages used for doing the resync
> + * IO, and it is per-bio, so make .bi_private points to it.
> + */
> +static inline struct resync_pages *get_resync_pages(struct bio *bio)
> +{
> + return bio->bi_private;
> +}
> +
> +/*
> + * for resync bio, r1bio pointer can be retrieved from the per-bio
> + * 'struct resync_pages'.
> + */
> +static inline struct r1bio *get_resync_r1bio(struct bio *bio)
> +{
> + return get_resync_pages(bio)->raid_bio;
> +}
> +
> static void * r1bio_pool_alloc(gfp_t gfp_flags, void *data)
> {
> struct pool_info *pi = data;
> @@ -107,12 +125,18 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
> struct r1bio *r1_bio;
> struct bio *bio;
> int need_pages;
> - int i, j;
> + int j;
> + struct resync_pages *rps;
>
> r1_bio = r1bio_pool_alloc(gfp_flags, pi);
> if (!r1_bio)
> return NULL;
>
> + rps = kmalloc(sizeof(struct resync_pages) * pi->raid_disks,
> + gfp_flags);
> + if (!rps)
> + goto out_free_r1bio;
> +
> /*
> * Allocate bios : 1 for reading, n-1 for writing
> */
> @@ -132,22 +156,22 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
> need_pages = pi->raid_disks;
> else
> need_pages = 1;
> - for (j = 0; j < need_pages; j++) {
> + for (j = 0; j < pi->raid_disks; j++) {
> + struct resync_pages *rp = &rps[j];
> +
> bio = r1_bio->bios[j];
> - bio->bi_vcnt = RESYNC_PAGES;
> -
> - if (bio_alloc_pages(bio, gfp_flags))
> - goto out_free_pages;
> - }
> - /* If not user-requests, copy the page pointers to all bios */
> - if (!test_bit(MD_RECOVERY_REQUESTED, &pi->mddev->recovery)) {
> - for (i=0; i<RESYNC_PAGES ; i++)
> - for (j=1; j<pi->raid_disks; j++) {
> - struct page *page =
> - r1_bio->bios[0]->bi_io_vec[i].bv_page;
> - get_page(page);
> - r1_bio->bios[j]->bi_io_vec[i].bv_page = page;
> - }
> +
> + if (j < need_pages) {
> + if (resync_alloc_pages(rp, gfp_flags))
> + goto out_free_pages;
> + } else {
> + memcpy(rp, &rps[0], sizeof(*rp));
> + resync_get_all_pages(rp);
> + }
> +
> + rp->idx = 0;
This is the only place the ->idx is initialized, in r1buf_pool_alloc().
The mempool alloc function is suppose to allocate memory, not initialize
it.
If the mempool_alloc() call cannot allocate memory it will use memory
from the pool. If this memory has already been used, then it will no
longer have the initialized value.
In short: you need to initialise memory *after* calling
mempool_alloc(), unless you ensure it is reset to the init values before
calling mempool_free().
https://bugzilla.kernel.org/show_bug.cgi?id=196307
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: My MD is too big to resize ext4.
From: Ram Ramesh @ 2017-07-09 23:09 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Linux Raid
In-Reply-To: <20170708194427.GA15208@metamorpher.de>
On 07/08/2017 02:44 PM, Andreas Klauer wrote:
> On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
>> 1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
>> I can create partitions on the drive instead of on the MD.
>> 2. This means I need to shrink my current md device to smaller size
>> (say 12TB-14TB) - need to check my current
>> active ext4 data size. It is definitely less than 16TB.
>> 3. Repartition the disks to create sd{b-g}2 for the reamining unused
>> 6xnTB area.
>> 4. Created md1 with sd{b-g]2 to get md1.
>> 5. Mount and use md1.
> Should work; you have to shrink the filesystem first, (or stick to
> whatever size it has now), then the md, then the partition.
>
> What makes this risky is that you have to pick the correct sizes.
> When growing you can't go wrong. When shrinking, you have to be
> careful not to shrink too much. Filesystems don't like it at all
> if their end is missing, and md doesn't like it if the block device
> is smaller than what it says in the metadata.
>
> So you have to determine the exact filesystem size (tune2fs -l),
> and take mdadm data offsets into account.
>
> Not necessary if you know what you're doing but if in doubt,
> you can leave a safety margin with each of these steps.
>
> Regards
> Andreas Klauer
I already shrunk ext4 and md as they can be done without having to boot
to a rescue disk. I like to do the last step of partitioning disk using
a rescue disk.
Here is my partition table on one disk (they all are very similar or
identical - each is a 6TB enterprise disk)
> zym [rramesh] 431 > sudo gdisk -l /dev/sdb
> GPT fdisk (gdisk) version 0.8.8
>
>
> Partition table scan:
> MBR: protective
> BSD: not present
> APM: not present
> GPT: present
>
> Found valid GPT with protective MBR; using GPT.
> Disk /dev/sdb: 11721045168 sectors, 5.5 TiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): D5C9B768-D2E5-4DEE-8D89-73A7B631FE28
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 11721045134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 2014 sectors (1007.0 KiB)
>
> Number Start (sector) End (sector) Size Code Name
> 1 2048 11721045134 5.5 TiB FD00 Linux RAID
>
Here is mdadm -E on the same disk
> zym [rramesh] 430 > sudo mdadm -E /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Name : zym:0 (local to host zym)
> Creation Time : Mon Apr 22 00:08:12 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : 2a844ae2:4b028cc3:36095185:7b09f7cc
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Sun Jul 9 17:55:51 2017
> Checksum : 74d5464 - correct
> Events : 290897
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 4
> Array State : AAAAAA ('A' == active, '.' == missing)
I am guessing anything more than 6174015488+2048 as the end sector for
/dev/sdb1 should be ok with mdadm. Let me know if I am not calculating
correctly.
I plan to set /dev/sd{b,c,d,e,f,g}1 to 3TB even (start = 2048, end =
6442452991 size = 6442450944 sectors = 3072GiB)
to get a 12TB md0.
Ramesh
^ permalink raw reply
* Re: [PATCH v3 05/14] md: raid1: don't use bio's vec table to manage resync pages
From: Ming Lei @ 2017-07-10 3:35 UTC (permalink / raw)
To: NeilBrown
Cc: Shaohua Li, Jens Axboe,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT, linux-block,
Christoph Hellwig
In-Reply-To: <87mv8d5ht7.fsf@notabene.neil.brown.name>
On Mon, Jul 10, 2017 at 7:09 AM, NeilBrown <neilb@suse.com> wrote:
> On Fri, Mar 17 2017, Ming Lei wrote:
>
>> Now we allocate one page array for managing resync pages, instead
>> of using bio's vec table to do that, and the old way is very hacky
>> and won't work any more if multipage bvec is enabled.
>>
>> The introduced cost is that we need to allocate (128 + 16) * raid_disks
>> bytes per r1_bio, and it is fine because the inflight r1_bio for
>> resync shouldn't be much, as pointed by Shaohua.
>>
>> Also the bio_reset() in raid1_sync_request() is removed because
>> all bios are freshly new now and not necessary to reset any more.
>>
>> This patch can be thought as a cleanup too
>>
>> Suggested-by: Shaohua Li <shli@kernel.org>
>> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
>> ---
>> drivers/md/raid1.c | 94 +++++++++++++++++++++++++++++++++++++-----------------
>> 1 file changed, 64 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
>> index e30d89690109..0e64beb60e4d 100644
>> --- a/drivers/md/raid1.c
>> +++ b/drivers/md/raid1.c
>> @@ -80,6 +80,24 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
>> #define raid1_log(md, fmt, args...) \
>> do { if ((md)->queue) blk_add_trace_msg((md)->queue, "raid1 " fmt, ##args); } while (0)
>>
>> +/*
>> + * 'strct resync_pages' stores actual pages used for doing the resync
>> + * IO, and it is per-bio, so make .bi_private points to it.
>> + */
>> +static inline struct resync_pages *get_resync_pages(struct bio *bio)
>> +{
>> + return bio->bi_private;
>> +}
>> +
>> +/*
>> + * for resync bio, r1bio pointer can be retrieved from the per-bio
>> + * 'struct resync_pages'.
>> + */
>> +static inline struct r1bio *get_resync_r1bio(struct bio *bio)
>> +{
>> + return get_resync_pages(bio)->raid_bio;
>> +}
>> +
>> static void * r1bio_pool_alloc(gfp_t gfp_flags, void *data)
>> {
>> struct pool_info *pi = data;
>> @@ -107,12 +125,18 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
>> struct r1bio *r1_bio;
>> struct bio *bio;
>> int need_pages;
>> - int i, j;
>> + int j;
>> + struct resync_pages *rps;
>>
>> r1_bio = r1bio_pool_alloc(gfp_flags, pi);
>> if (!r1_bio)
>> return NULL;
>>
>> + rps = kmalloc(sizeof(struct resync_pages) * pi->raid_disks,
>> + gfp_flags);
>> + if (!rps)
>> + goto out_free_r1bio;
>> +
>> /*
>> * Allocate bios : 1 for reading, n-1 for writing
>> */
>> @@ -132,22 +156,22 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
>> need_pages = pi->raid_disks;
>> else
>> need_pages = 1;
>> - for (j = 0; j < need_pages; j++) {
>> + for (j = 0; j < pi->raid_disks; j++) {
>> + struct resync_pages *rp = &rps[j];
>> +
>> bio = r1_bio->bios[j];
>> - bio->bi_vcnt = RESYNC_PAGES;
>> -
>> - if (bio_alloc_pages(bio, gfp_flags))
>> - goto out_free_pages;
>> - }
>> - /* If not user-requests, copy the page pointers to all bios */
>> - if (!test_bit(MD_RECOVERY_REQUESTED, &pi->mddev->recovery)) {
>> - for (i=0; i<RESYNC_PAGES ; i++)
>> - for (j=1; j<pi->raid_disks; j++) {
>> - struct page *page =
>> - r1_bio->bios[0]->bi_io_vec[i].bv_page;
>> - get_page(page);
>> - r1_bio->bios[j]->bi_io_vec[i].bv_page = page;
>> - }
>> +
>> + if (j < need_pages) {
>> + if (resync_alloc_pages(rp, gfp_flags))
>> + goto out_free_pages;
>> + } else {
>> + memcpy(rp, &rps[0], sizeof(*rp));
>> + resync_get_all_pages(rp);
>> + }
>> +
>> + rp->idx = 0;
>
> This is the only place the ->idx is initialized, in r1buf_pool_alloc().
> The mempool alloc function is suppose to allocate memory, not initialize
> it.
>
> If the mempool_alloc() call cannot allocate memory it will use memory
> from the pool. If this memory has already been used, then it will no
> longer have the initialized value.
>
> In short: you need to initialise memory *after* calling
> mempool_alloc(), unless you ensure it is reset to the init values before
> calling mempool_free().
>
> https://bugzilla.kernel.org/show_bug.cgi?id=196307
OK, thanks for posting it out.
Another fix might be to reinitialize the variable(rp->idx = 0) in
r1buf_pool_free().
Or just set it as zero every time when it is used.
But I don't understand why mempool_free() calls pool->free() at the end of
this function, which may cause to run pool->free() on a new allocated buf,
seems a bug in mempool?
--
Ming Lei
^ permalink raw reply
* Re: [PATCH v3 05/14] md: raid1: don't use bio's vec table to manage resync pages
From: Ming Lei @ 2017-07-10 4:13 UTC (permalink / raw)
To: Ming Lei
Cc: NeilBrown, Shaohua Li, Jens Axboe,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT, linux-block,
Christoph Hellwig
In-Reply-To: <CACVXFVP69kLQyr0NOY7zXMZ_bdJYCV7fBbDrKdFF9fBPBTsU-g@mail.gmail.com>
On Mon, Jul 10, 2017 at 11:35:12AM +0800, Ming Lei wrote:
> On Mon, Jul 10, 2017 at 7:09 AM, NeilBrown <neilb@suse.com> wrote:
> > On Fri, Mar 17 2017, Ming Lei wrote:
> >
> >> Now we allocate one page array for managing resync pages, instead
> >> of using bio's vec table to do that, and the old way is very hacky
> >> and won't work any more if multipage bvec is enabled.
> >>
> >> The introduced cost is that we need to allocate (128 + 16) * raid_disks
> >> bytes per r1_bio, and it is fine because the inflight r1_bio for
> >> resync shouldn't be much, as pointed by Shaohua.
> >>
> >> Also the bio_reset() in raid1_sync_request() is removed because
> >> all bios are freshly new now and not necessary to reset any more.
> >>
> >> This patch can be thought as a cleanup too
> >>
> >> Suggested-by: Shaohua Li <shli@kernel.org>
> >> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> >> ---
> >> drivers/md/raid1.c | 94 +++++++++++++++++++++++++++++++++++++-----------------
> >> 1 file changed, 64 insertions(+), 30 deletions(-)
> >>
> >> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> >> index e30d89690109..0e64beb60e4d 100644
> >> --- a/drivers/md/raid1.c
> >> +++ b/drivers/md/raid1.c
> >> @@ -80,6 +80,24 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
> >> #define raid1_log(md, fmt, args...) \
> >> do { if ((md)->queue) blk_add_trace_msg((md)->queue, "raid1 " fmt, ##args); } while (0)
> >>
> >> +/*
> >> + * 'strct resync_pages' stores actual pages used for doing the resync
> >> + * IO, and it is per-bio, so make .bi_private points to it.
> >> + */
> >> +static inline struct resync_pages *get_resync_pages(struct bio *bio)
> >> +{
> >> + return bio->bi_private;
> >> +}
> >> +
> >> +/*
> >> + * for resync bio, r1bio pointer can be retrieved from the per-bio
> >> + * 'struct resync_pages'.
> >> + */
> >> +static inline struct r1bio *get_resync_r1bio(struct bio *bio)
> >> +{
> >> + return get_resync_pages(bio)->raid_bio;
> >> +}
> >> +
> >> static void * r1bio_pool_alloc(gfp_t gfp_flags, void *data)
> >> {
> >> struct pool_info *pi = data;
> >> @@ -107,12 +125,18 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
> >> struct r1bio *r1_bio;
> >> struct bio *bio;
> >> int need_pages;
> >> - int i, j;
> >> + int j;
> >> + struct resync_pages *rps;
> >>
> >> r1_bio = r1bio_pool_alloc(gfp_flags, pi);
> >> if (!r1_bio)
> >> return NULL;
> >>
> >> + rps = kmalloc(sizeof(struct resync_pages) * pi->raid_disks,
> >> + gfp_flags);
> >> + if (!rps)
> >> + goto out_free_r1bio;
> >> +
> >> /*
> >> * Allocate bios : 1 for reading, n-1 for writing
> >> */
> >> @@ -132,22 +156,22 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
> >> need_pages = pi->raid_disks;
> >> else
> >> need_pages = 1;
> >> - for (j = 0; j < need_pages; j++) {
> >> + for (j = 0; j < pi->raid_disks; j++) {
> >> + struct resync_pages *rp = &rps[j];
> >> +
> >> bio = r1_bio->bios[j];
> >> - bio->bi_vcnt = RESYNC_PAGES;
> >> -
> >> - if (bio_alloc_pages(bio, gfp_flags))
> >> - goto out_free_pages;
> >> - }
> >> - /* If not user-requests, copy the page pointers to all bios */
> >> - if (!test_bit(MD_RECOVERY_REQUESTED, &pi->mddev->recovery)) {
> >> - for (i=0; i<RESYNC_PAGES ; i++)
> >> - for (j=1; j<pi->raid_disks; j++) {
> >> - struct page *page =
> >> - r1_bio->bios[0]->bi_io_vec[i].bv_page;
> >> - get_page(page);
> >> - r1_bio->bios[j]->bi_io_vec[i].bv_page = page;
> >> - }
> >> +
> >> + if (j < need_pages) {
> >> + if (resync_alloc_pages(rp, gfp_flags))
> >> + goto out_free_pages;
> >> + } else {
> >> + memcpy(rp, &rps[0], sizeof(*rp));
> >> + resync_get_all_pages(rp);
> >> + }
> >> +
> >> + rp->idx = 0;
> >
> > This is the only place the ->idx is initialized, in r1buf_pool_alloc().
> > The mempool alloc function is suppose to allocate memory, not initialize
> > it.
> >
> > If the mempool_alloc() call cannot allocate memory it will use memory
> > from the pool. If this memory has already been used, then it will no
> > longer have the initialized value.
> >
> > In short: you need to initialise memory *after* calling
> > mempool_alloc(), unless you ensure it is reset to the init values before
> > calling mempool_free().
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=196307
>
> OK, thanks for posting it out.
>
> Another fix might be to reinitialize the variable(rp->idx = 0) in
> r1buf_pool_free().
> Or just set it as zero every time when it is used.
>
> But I don't understand why mempool_free() calls pool->free() at the end of
> this function, which may cause to run pool->free() on a new allocated buf,
> seems a bug in mempool?
Looks I missed the 'return' in mempool_free(), so it is fine.
How about the following fix?
---
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index e1a7e3d4c5e4..d31b06da3e3d 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -242,6 +242,7 @@ static void put_buf(struct r1bio *r1_bio)
struct bio *bio = r1_bio->bios[i];
if (bio->bi_end_io)
rdev_dec_pending(conf->mirrors[i].rdev, r1_bio->mddev);
+ get_resync_pages(bio)->idx = 0;
}
mempool_free(r1_bio, conf->r1buf_pool);
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 797ed60abd5e..c61523768745 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -299,12 +299,21 @@ static void free_r10bio(struct r10bio *r10_bio)
mempool_free(r10_bio, conf->r10bio_pool);
}
-static void put_buf(struct r10bio *r10_bio)
+static void free_r10bio_buf(struct r10bio *r10_bio, struct r10conf *conf)
{
- struct r10conf *conf = r10_bio->mddev->private;
+ int j;
+
+ for (j = conf->copies; j--; )
+ get_resync_pages(r10_bio->devs[j].bio)->idx = 0;
mempool_free(r10_bio, conf->r10buf_pool);
+}
+
+static void put_buf(struct r10bio *r10_bio)
+{
+ struct r10conf *conf = r10_bio->mddev->private;
+ free_r10bio_buf(r10_bio, conf);
lower_barrier(conf);
}
@@ -4383,7 +4392,7 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr,
* on all the target devices.
*/
// FIXME
- mempool_free(r10_bio, conf->r10buf_pool);
+ free_r10bio_buf(r10_bio, conf);
set_bit(MD_RECOVERY_INTR, &mddev->recovery);
return sectors_done;
}
--
Ming
^ permalink raw reply related
* Re: [PATCH v3 05/14] md: raid1: don't use bio's vec table to manage resync pages
From: NeilBrown @ 2017-07-10 4:38 UTC (permalink / raw)
To: Ming Lei, Ming Lei
Cc: Shaohua Li, Jens Axboe,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT, linux-block,
Christoph Hellwig
In-Reply-To: <20170710041304.GB15321@ming.t460p>
[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]
On Mon, Jul 10 2017, Ming Lei wrote:
> On Mon, Jul 10, 2017 at 11:35:12AM +0800, Ming Lei wrote:
>> On Mon, Jul 10, 2017 at 7:09 AM, NeilBrown <neilb@suse.com> wrote:
...
>> >> +
>> >> + rp->idx = 0;
>> >
>> > This is the only place the ->idx is initialized, in r1buf_pool_alloc().
>> > The mempool alloc function is suppose to allocate memory, not initialize
>> > it.
>> >
>> > If the mempool_alloc() call cannot allocate memory it will use memory
>> > from the pool. If this memory has already been used, then it will no
>> > longer have the initialized value.
>> >
>> > In short: you need to initialise memory *after* calling
>> > mempool_alloc(), unless you ensure it is reset to the init values before
>> > calling mempool_free().
>> >
>> > https://bugzilla.kernel.org/show_bug.cgi?id=196307
>>
>> OK, thanks for posting it out.
>>
>> Another fix might be to reinitialize the variable(rp->idx = 0) in
>> r1buf_pool_free().
>> Or just set it as zero every time when it is used.
>>
>> But I don't understand why mempool_free() calls pool->free() at the end of
>> this function, which may cause to run pool->free() on a new allocated buf,
>> seems a bug in mempool?
>
> Looks I missed the 'return' in mempool_free(), so it is fine.
>
> How about the following fix?
It looks like it would probably work, but it is rather unusual to
initialise something just before freeing it.
Couldn't you just move the initialization to shortly after the
mempool_alloc() call. There looks like a good place that already loops
over all the bios....
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox