* Re: [PATCH 2/2] md: raid1/raid10: initialize bvec table via bio_add_page()
From: NeilBrown @ 2017-07-13 22:08 UTC (permalink / raw)
To: Shaohua Li, Ming Lei
Cc: linux-raid, linux-block, Jens Axboe, Christoph Hellwig
In-Reply-To: <20170713163909.whltvlms2zwgevkf@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 4376 bytes --]
On Thu, Jul 13 2017, Shaohua Li wrote:
> On Thu, Jul 13, 2017 at 05:20:52PM +0800, Ming Lei wrote:
>> On Thu, Jul 13, 2017 at 01:09:28PM +1000, NeilBrown wrote:
>> > On Thu, Jul 13 2017, Ming Lei wrote:
>> >
>> > > On Thu, Jul 13, 2017 at 10:01:33AM +1000, NeilBrown wrote:
>> > >> On Wed, Jul 12 2017, Ming Lei wrote:
>> > >>
>> > >> > We will support multipage bvec soon, so initialize bvec
>> > >> > table using the standardy way instead of writing the
>> > >> > talbe directly. Otherwise it won't work any more once
>> > >> > multipage bvec is enabled.
>> > >> >
>> > >> > Acked-by: Guoqing Jiang <gqjiang@suse.com>
>> > >> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> > >> > ---
>> > >> > drivers/md/md.c | 21 +++++++++++++++++++++
>> > >> > drivers/md/md.h | 3 +++
>> > >> > drivers/md/raid1.c | 16 ++--------------
>> > >> > drivers/md/raid10.c | 4 ++--
>> > >> > 4 files changed, 28 insertions(+), 16 deletions(-)
>> > >> >
>> > >> > diff --git a/drivers/md/md.c b/drivers/md/md.c
>> > >> > index 8cdca0296749..cc8dcd928dde 100644
>> > >> > --- a/drivers/md/md.c
>> > >> > +++ b/drivers/md/md.c
>> > >> > @@ -9130,6 +9130,27 @@ void md_reload_sb(struct mddev *mddev, int nr)
>> > >> > }
>> > >> > EXPORT_SYMBOL(md_reload_sb);
>> > >> >
>> > >> > +/* generally called after bio_reset() for reseting bvec */
>> > >> > +void md_bio_reset_resync_pages(struct bio *bio, struct resync_pages *rp,
>> > >> > + int size)
>> > >> > +{
>> > >> > + int idx = 0;
>> > >> > +
>> > >> > + /* initialize bvec table again */
>> > >> > + do {
>> > >> > + struct page *page = resync_fetch_page(rp, idx);
>> > >> > + int len = min_t(int, size, PAGE_SIZE);
>> > >> > +
>> > >> > + /*
>> > >> > + * won't fail because the vec table is big
>> > >> > + * enough to hold all these pages
>> > >> > + */
>> > >> > + bio_add_page(bio, page, len, 0);
>> > >> > + size -= len;
>> > >> > + } while (idx++ < RESYNC_PAGES && size > 0);
>> > >> > +}
>> > >> > +EXPORT_SYMBOL(md_bio_reset_resync_pages);
>> > >>
>> > >> I really don't think this is a good idea.
>> > >> This code is specific to raid1/raid10. It is not generic md code. So
>> > >> it doesn't belong here.
>> > >
>> > > We already added 'struct resync_pages' in drivers/md/md.h, so I think
>> > > it is reasonable to add this function into drivers/md/md.c
>> >
>> > Alternative perspective: it was unreasonable to add "resync_pages" to
>> > md.h.
>> >
>> > >
>> > >>
>> > >> If you want to remove code duplication, then work on moving all raid1
>> > >> functionality into raid10.c, then discard raid1.c
>> > >
>> > > This patch is for avoiding new code duplication, not for removing current
>> > > duplication.
>> > >
>> > >>
>> > >> Or at the very least, have a separate "raid1-10.c" file for the common
>> > >> code.
>> > >
>> > > You suggested it last time, but looks too overkill to be taken. But if
>> > > someone wants to refactor raid1 and raid10, I think it can be a good start,
>> > > but still not belong to this patch.
>> >
>> > You are trying to create common code for raid1 and raid10. This does
>> > not belong in md.c.
>> > If you really want to have a single copy of common code, then it exactly
>> > is the role of this patch to create a place to put it.
>> > I'm not saying you should put all common code in raid1-10.c. Just the
>> > function that you have identified.
>>
>> I really don't want to waste time on this kind of thing, I can do
>> either one frankly.
>>
>> Shaohua, could you share me which way you like to merge? I can do it in
>> either way.
>
> I don't have strong preference, but Neil's suggestion does make the code a
> little better. Of course, only put the function into the raid1-10.c right now.
To make it as easy as possible, I would suggest creating raid1-10.c
containing just this function (and maybe the definitions from md.h),
and declare the function "static" and #include raid1-10.c into raid1.c
and raid10.c. i.e. no worrying about modules and exporting symbols.
Anyone who cares (and that might even be me) could move functionality
gradually out of raid1.c and raid10.c in raid1-10.c. Maybe where would
come a tipping-point where it is easy to just discard raid1.c and raid10.c
and finish the job.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Gionatan Danti @ 2017-07-13 22:34 UTC (permalink / raw)
To: Reindl Harald; +Cc: Roman Mamedov, linux-raid, g.danti
In-Reply-To: <d1255092-73f5-1ca4-0e68-69ff37631a26@thelounge.net>
Il 13-07-2017 23:34 Reindl Harald ha scritto:
> maybe because the disk is, well, not in a good shape and don't know
> that by itself
>
But the kernel *does* know that, as the dmesg entries clearly show.
Basically, some SATA commands timed-out and/or were aborted. As the
kernel reported these erros in dmesg, why do not use these information
to stop a failing disk?
>
> (and no filesystems with checksums won't magically recover
> your data, they just tell you realier they are gone)
>
Checksummed filesystem that integrates their block-level management
(read: ZFS or BTRFS) can recover the missing/corrupted data by the
healthy disks, discarging corrupted data based on the checksum mismatch.
Anyway, this has nothing to do with linux software RAID. I was only
"thinking loud" :)
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8
^ permalink raw reply
* Why can't I re-add my drive after partition shrink?
From: Ram Ramesh @ 2017-07-13 23:17 UTC (permalink / raw)
To: Linux Raid
I am trying to shring my mdadm underlying disks/partitions as a way of
reclaiming space after md shrink operation. Here is my md
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : active raid6 sdb1[6] sdg1[11] sdd1[12] sdf1[8] sde1[9] sdc1[10]
> 12348030976 blocks super 1.2 level 6, 64k chunk, algorithm 2
> [6/6] [UUUUUU]
> bitmap: 2/23 pages [8KB], 65536KB chunk
>
> unused devices: <none>
Here is my mdadm detail about /dev/sdb1
> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
I fail, remove and repartition /dev/sdb1 so that new partition table
looks like this. (No change to data obviously)
> New partition table:
> Number Start (sector) End (sector) Size Code Name
> 1 2048 6442452991 3.0 TiB FD00 Linux RAID
> 2 6442452992 11721045134 2.5 TiB FD00 Linux RAID
Since new /dev/sdb1 is significantly bigger than what mdadm says that it
is using, I thought I could simply re-add the drive. However, I get
> sudo mdadm /dev/md0 --re-add /dev/sdb1
> ***mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible***
Why? What did I miss here? Is it possible to fix this so that I can
repartition other drives and -re-add them without having to go through
full rebuild after each change?
I researched as much as I could on the net and came up with nothing
except some one saying that mdadm keeps something at the end of the disk
regardless of what it says about "Used Dev Size." Is it possible to move
this info so that I could re-add?
Ramesh
^ permalink raw reply
* Re: [opensuse] raid device mounting problem in Leap 42.2
From: Wols Lists @ 2017-07-13 23:18 UTC (permalink / raw)
To: opensuse, linux-raid
In-Reply-To: <d71443c0d631c80af4d09c6098b85359@gmail.hu>
On 13/07/17 23:46, Istvan Gabor wrote:
> On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:
>> On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists <antlists@youngman.org.uk>
>> wrote:
>>>
>>> But I suspect that going back into 12.2, and shrinking the filesystem
>>> (MAKE SURE you shrink the md device, not the sd device)
>>
>> Quite the opposite. Attempting to shrink filesystem that is already
>> effectively corrupted is potentially dangerous. So the right thing
>> here is to stop md, backup superblocks (in case they will be
>
> I am puzzled. Do I need to backup the raid superblock or the device
> superblocks or all? How do I backup the superblock? I googled but only
> find how to recover from superblock backup.
>
The best bet is to ask the linux-raid list what to do.
For the raid list, this is the array I mentioned where the filesystem,
and the partitions the raid was on, were the same size.
So we have a v1.0 mirror where the filesystem on the mirror uses up ALL
the space on the md device, not just the free space that it's supposed
to use, with all the issues that brings like filling up the filesystem
will overwrite the superblock.
So what's the best way to recover? Will just shrinking the filesystem
bring everything back the way it should be, or are we better just
mounting the original filesystem from one disk as if it weren't a
mirror, and then recreating the mirror on the other disk, copying the
data, and then adding the first disk back in to put everything back the
way it should have been?
Cheers,
Wol
^ permalink raw reply
* Re: Why can't I re-add my drive after partition shrink?
From: Anthony Youngman @ 2017-07-13 23:37 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <2ea5ed99-05b5-25d2-c9d4-8ce1f4945eaa@gmail.com>
On 14/07/17 00:17, Ram Ramesh wrote:
> I am trying to shring my mdadm underlying disks/partitions as a way of
> reclaiming space after md shrink operation. Here is my md
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md0 : active raid6 sdb1[6] sdg1[11] sdd1[12] sdf1[8] sde1[9] sdc1[10]
>> 12348030976 blocks super 1.2 level 6, 64k chunk, algorithm 2
>> [6/6] [UUUUUU]
>> bitmap: 2/23 pages [8KB], 65536KB chunk
>>
>> unused devices: <none>
>
> Here is my mdadm detail about /dev/sdb1
>> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
>> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
>> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> I fail, remove and repartition /dev/sdb1 so that new partition table
> looks like this. (No change to data obviously)
>> New partition table:
>> Number Start (sector) End (sector) Size Code Name
>> 1 2048 6442452991 3.0 TiB FD00 Linux RAID
>> 2 6442452992 11721045134 2.5 TiB FD00 Linux RAID
>
> Since new /dev/sdb1 is significantly bigger than what mdadm says that it
> is using, I thought I could simply re-add the drive. However, I get
>> sudo mdadm /dev/md0 --re-add /dev/sdb1
>> ***mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible***
> Why? What did I miss here? Is it possible to fix this so that I can
> repartition other drives and -re-add them without having to go through
> full rebuild after each change?
>
> I researched as much as I could on the net and came up with nothing
> except some one saying that mdadm keeps something at the end of the disk
> regardless of what it says about "Used Dev Size." Is it possible to move
> this info so that I could re-add?
>
I can't say why it won't re-add, but what's at the end of the disk is
the superblock - except it isn't at the end. The 0.9, or v1.0,
superblocks are at the end, but you've got a v1.2 superblock which is
stored at the start - to be precise, 4K in from the front of the partition.
Cheers,
Wol
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Reindl Harald @ 2017-07-14 0:32 UTC (permalink / raw)
To: Gionatan Danti; +Cc: Roman Mamedov, linux-raid
In-Reply-To: <cd37f90b86eb67be4c893b7fdf112692@assyoma.it>
Am 14.07.2017 um 00:34 schrieb Gionatan Danti:
> Il 13-07-2017 23:34 Reindl Harald ha scritto:
>> maybe because the disk is, well, not in a good shape and don't know
>> that by itself
>
> But the kernel *does* know that, as the dmesg entries clearly show.
> Basically, some SATA commands timed-out and/or were aborted. As the
> kernel reported these erros in dmesg, why do not use these information
> to stop a failing disk?
because you won't be that happy when the kernel spits out a disk each
time a random SATA command times out - the 4 RAID10 disks on my
workstation are from 2011 and showed them too several times in the past
while they are just fine
here you go:
http://strugglers.net/~andy/blog/2015/11/09/linux-software-raid-and-drive-timeouts/
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Anthony Youngman @ 2017-07-14 0:52 UTC (permalink / raw)
To: Reindl Harald, Gionatan Danti; +Cc: Roman Mamedov, linux-raid
In-Reply-To: <770b09d3-cff6-b6b2-0a51-5d11e8bac7e9@thelounge.net>
On 14/07/17 01:32, Reindl Harald wrote:
>
>
> Am 14.07.2017 um 00:34 schrieb Gionatan Danti:
>> Il 13-07-2017 23:34 Reindl Harald ha scritto:
>>> maybe because the disk is, well, not in a good shape and don't know
>>> that by itself
>>
>> But the kernel *does* know that, as the dmesg entries clearly show.
>> Basically, some SATA commands timed-out and/or were aborted. As the
>> kernel reported these erros in dmesg, why do not use these information
>> to stop a failing disk?
>
> because you won't be that happy when the kernel spits out a disk each
> time a random SATA command times out - the 4 RAID10 disks on my
> workstation are from 2011 and showed them too several times in the past
> while they are just fine
>
Except, in the context of this thread, the alternative is CORRUPTED
DATA. I certainly know which one I would prefer, and that is a crashed
array!
If a *write* fails, then a failed array may well be the least of the
user's problems - and silent failure merely makes matters worse!
I know, the problem is that linux isn't actually that good at
propagating errors back to user space, and I believe that's a fault of
POSIX. So fixing the problem might be a massive job - indeed I think it is.
But that's no excuse for mocking someone just because they want to be
told that the system has just gone and lost their work for them ...
Oh - and isn't that what raid is *supposed* to do? Kick a disk on a
write failure?
Cheers,
Wol
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Reindl Harald @ 2017-07-14 1:10 UTC (permalink / raw)
To: Anthony Youngman, Gionatan Danti; +Cc: Roman Mamedov, linux-raid
In-Reply-To: <76d8e93f-a9cc-5df4-e086-5d2884a589d0@youngman.org.uk>
Am 14.07.2017 um 02:52 schrieb Anthony Youngman:
> On 14/07/17 01:32, Reindl Harald wrote:
>>
>>
>> Am 14.07.2017 um 00:34 schrieb Gionatan Danti:
>>> Il 13-07-2017 23:34 Reindl Harald ha scritto:
>>>> maybe because the disk is, well, not in a good shape and don't know
>>>> that by itself
>>>
>>> But the kernel *does* know that, as the dmesg entries clearly show.
>>> Basically, some SATA commands timed-out and/or were aborted. As the
>>> kernel reported these erros in dmesg, why do not use these
>>> information to stop a failing disk?
>>
>> because you won't be that happy when the kernel spits out a disk each
>> time a random SATA command times out - the 4 RAID10 disks on my
>> workstation are from 2011 and showed them too several times in the
>> past while they are just fine
>>
> Except, in the context of this thread, the alternative is CORRUPTED
> DATA. I certainly know which one I would prefer, and that is a crashed
> array!
>
> If a *write* fails, then a failed array may well be the least of the
> user's problems - and silent failure merely makes matters worse!
i doubt that you would repeat that if for whatever load condition a
random SATA timeout occours on both disks of a mirror and you lose some
TB of data while in *that case* not silent corruption or anything else
bad would have happened except a short lag
did you really read
http://strugglers.net/~andy/blog/2015/11/09/linux-software-raid-and-drive-timeouts/
or just ignored it on purpose?
> I know, the problem is that linux isn't actually that good at
> propagating errors back to user space, and I believe that's a fault of
> POSIX. So fixing the problem might be a massive job - indeed I think it is.
>
> But that's no excuse for mocking someone just because they want to be
> told that the system has just gone and lost their work for them ...
nobody is mocking someone, i just explained why things are not as simple
as they appear and with a 2 disk mirror they are always complicated in
any error case by lack of quorum
> Oh - and isn't that what raid is *supposed* to do? Kick a disk on a
> write failure?
if things only would be that easy in the real world...
in doubt with a mirrored RAID without data checksums *you have no way*
to guarantee what is the correct data if something flips and "Except, in
the context of this thread" is nice but won't help in general and trying
to handle each and every bordercase with some workarounds would lead nowhere
yes, agreed, silent corruption is bad, hardware lying about data written
is bad, but if things would be that easy all that won't happen and
nobody would have spent time for develop checksummed filesystems
^ permalink raw reply
* Re: Why can't I re-add my drive after partition shrink?
From: NeilBrown @ 2017-07-14 1:35 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <2ea5ed99-05b5-25d2-c9d4-8ce1f4945eaa@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2268 bytes --]
On Thu, Jul 13 2017, Ram Ramesh wrote:
> I am trying to shring my mdadm underlying disks/partitions as a way of
> reclaiming space after md shrink operation. Here is my md
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md0 : active raid6 sdb1[6] sdg1[11] sdd1[12] sdf1[8] sde1[9] sdc1[10]
>> 12348030976 blocks super 1.2 level 6, 64k chunk, algorithm 2
>> [6/6] [UUUUUU]
>> bitmap: 2/23 pages [8KB], 65536KB chunk
>>
>> unused devices: <none>
>
> Here is my mdadm detail about /dev/sdb1
>> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
>> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
>> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> I fail, remove and repartition /dev/sdb1 so that new partition table
> looks like this. (No change to data obviously)
>> New partition table:
>> Number Start (sector) End (sector) Size Code Name
>> 1 2048 6442452991 3.0 TiB FD00 Linux RAID
>> 2 6442452992 11721045134 2.5 TiB FD00 Linux RAID
>
> Since new /dev/sdb1 is significantly bigger than what mdadm says that it
> is using, I thought I could simply re-add the drive. However, I get
>> sudo mdadm /dev/md0 --re-add /dev/sdb1
>> ***mdadm: --re-add for /dev/sdb1 to /dev/md0 is not possible***
> Why? What did I miss here? Is it possible to fix this so that I can
> repartition other drives and -re-add them without having to go through
> full rebuild after each change?
Please report output of "mdadm --examine" on both a device that is active
in the array, and the device that you are trying to add.
Also "mdadm --examine-bitmap" of a device that is active in the array.
NeilBrown
>
> I researched as much as I could on the net and came up with nothing
> except some one saying that mdadm keeps something at the end of the disk
> regardless of what it says about "Used Dev Size." Is it possible to move
> this info so that I could re-add?
>
> Ramesh
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: [opensuse] raid device mounting problem in Leap 42.2
From: NeilBrown @ 2017-07-14 1:40 UTC (permalink / raw)
To: Wols Lists, opensuse, linux-raid
In-Reply-To: <5967FFAD.6030807@youngman.org.uk>
[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]
On Fri, Jul 14 2017, Wols Lists wrote:
> On 13/07/17 23:46, Istvan Gabor wrote:
>> On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:
>>> On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists <antlists@youngman.org.uk>
>>> wrote:
>>>>
>>>> But I suspect that going back into 12.2, and shrinking the filesystem
>>>> (MAKE SURE you shrink the md device, not the sd device)
>>>
>>> Quite the opposite. Attempting to shrink filesystem that is already
>>> effectively corrupted is potentially dangerous. So the right thing
>>> here is to stop md, backup superblocks (in case they will be
>>
>> I am puzzled. Do I need to backup the raid superblock or the device
>> superblocks or all?
You cannot have too many backups.
>> How do I backup the superblock? I googled but only
mdadm --dump=/some/directory list of devices.
>> find how to recover from superblock backup.
>>
> The best bet is to ask the linux-raid list what to do.
>
> For the raid list, this is the array I mentioned where the filesystem,
> and the partitions the raid was on, were the same size.
To the byte? I really like to see the output of commands that report
sizes, rather than have someone claim "were the same size". It isn't
that I don't trust you. I just don't trust humans in general (myself included).
NeilBrown
>
> So we have a v1.0 mirror where the filesystem on the mirror uses up ALL
> the space on the md device, not just the free space that it's supposed
> to use, with all the issues that brings like filling up the filesystem
> will overwrite the superblock.
>
> So what's the best way to recover? Will just shrinking the filesystem
> bring everything back the way it should be, or are we better just
> mounting the original filesystem from one disk as if it weren't a
> mirror, and then recreating the mirror on the other disk, copying the
> data, and then adding the first disk back in to put everything back the
> way it should have been?
>
> Cheers,
> Wol
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Chris Murphy @ 2017-07-14 1:48 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <cd37f90b86eb67be4c893b7fdf112692@assyoma.it>
On Thu, Jul 13, 2017 at 4:34 PM, Gionatan Danti <g.danti@assyoma.it> wrote:
> Il 13-07-2017 23:34 Reindl Harald ha scritto:
>>
>> maybe because the disk is, well, not in a good shape and don't know
>> that by itself
>>
>
> But the kernel *does* know that, as the dmesg entries clearly show.
> Basically, some SATA commands timed-out and/or were aborted. As the kernel
> reported these erros in dmesg, why do not use these information to stop a
> failing disk?
>
>>
>> (and no filesystems with checksums won't magically recover
>> your data, they just tell you realier they are gone)
>>
>
> Checksummed filesystem that integrates their block-level management (read:
> ZFS or BTRFS) can recover the missing/corrupted data by the healthy disks,
> discarging corrupted data based on the checksum mismatch.
>
> Anyway, this has nothing to do with linux software RAID. I was only
> "thinking loud" :)
> Thanks.
>
>
Dealing with device betrayal at a hardware level is a difficult
problem. I'm under the impression the md driver is very intolerant of
write failure and would eject a drive even with a single failed write?
It would seem to be disqualifying for RAID.
Btrfs still tolerates many errors, read and write, so it can still be
a problem there too. But yes it does have an independent way to
unambiguously determine whether file system metadata, or extent data,
is corrupt. It also often keeps two copies of metadata (the file
system itself). Another option (read-only) is dm-verity, but that is
not RAID, it uses forward error correction and cryptographic hash
verification.
--
Chris Murphy
^ permalink raw reply
* Re: Linear device of two arrays
From: NeilBrown @ 2017-07-14 1:57 UTC (permalink / raw)
To: Veljko, linux-raid
In-Reply-To: <e394625b-18d7-d141-3957-3c16c9bc6e44@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]
On Mon, Jul 10 2017, Veljko wrote:
> On 07/10/2017 12:37 AM, NeilBrown wrote:
>> I wasn't clear to me that I needed to chime in.. and the complete lack
>> of details (not even an "mdadm --examine" output), meant I could only
>> answer in vague generalizations.
>> However, seeing you asked.
>> If you really want to have a 'linear' of 2 RAID10s, then
>> 0/ unmount the xfs filesystem
>> 1/ backup the last few megabytes of the device
>> dd if=/dev/mdXX of=/safe/place/backup bs=1M skip=$BIGNUM
>> 2/ create a linear array of the two RAID10s, ensuring the
>> metadata is v1.0, and the dataoffset is zero (should be default with
>> 1.0)
>> mdadm -C /dev/mdZZ -l linear -n 2 -e 1.0 --data-offset=0 /dev/mdXX /dev/mdYY
>> 3/ restore the saved data
>> dd of=/dev/mdZZ if=/safe/place/backup bs=1M seek=$BIGNUM
>> 4/ grow the xfs filesystem
>> 5/ be happy.
>>
>> I cannot comment on the values of "few" and "$BUGNUM" without seeing
>> specifics.
>>
>> NeilBrown
>
> Thanks for your response, Neil!
>
> md0 is boot (raid1), md1 is root (raid10) and md2 is data (raid10) that
> I need to expand. Here are details:
Presumably you also have an md3 raid10 which you want to attach to the
end of md2?
md2 is 5761631232 sectors.
2880815616 kilobytes
2813296.5 (binary)megabytes.
When you include that into a "linear" you will lose a few K from the
end.
It might be sensible to cause the "linear" to use whole stripes from
the raid10, where a stripe is 1M (2 512K chunks).
If you did that, you would lose a little over 1M.
So backup the last 3.5 M of the raid10. This is much more than you need.
ie.
dd if=/dev/md2 of=SOMEWHERE/SAFE bs=1M skip=2813293
(dd treats 'M' as 1024*1024, MB is 1000*1000)
If the file this creates is not 3.5M, then something went wrong. Stop
here.
Just to be safe you might want to backup the first few megabytes. You
won't need this unless something goes wrong
dd if=/dev/md2 of=SOMEWHERE/ELSE bs=1M count=10
Now create the linear from /dev/md2 and /dev/md3(?). Be sure to use
"-e 1.0 --data-offset=0". This creates /dev/md4
Now restore the first backup
dd if=SOMEWHERE/SAFE of=/dev/md4 bs=1M seek=2813293
Be sure to use the same bs= and seek= as you did the first time.
Be sure it is copying from the back and to the new linear raid.
You should now be done. Check your xfs filesystem, and maybe even mount
it and use it.
NeilBrown
>
> # mdadm --detail /dev/md2
> /dev/md2:
> Version : 1.2
> Creation Time : Fri Sep 14 12:40:13 2012
> Raid Level : raid10
> Array Size : 5761631232 (5494.72 GiB 5899.91 GB)
> Used Dev Size : 2880815616 (2747.36 GiB 2949.96 GB)
> Raid Devices : 4
> Total Devices : 4
> Persistence : Superblock is persistent
>
> Update Time : Mon Jul 10 12:32:51 2017
> State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : near=2
> Chunk Size : 512K
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Linear device of two arrays
From: NeilBrown @ 2017-07-14 2:03 UTC (permalink / raw)
To: Veljko, linux-raid
In-Reply-To: <1b1d6d77-c17b-ada0-9a04-c724d34c9c1d@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 711 bytes --]
On Wed, Jul 12 2017, Veljko wrote:
>
> Data offset is not zero on md2 partitions. Is that a dealbreaker?
No.
>
> Would it be than better to reshape the current RAID10 to increase the
> number of devices used from 4 to 8 (as advised by Roman)?
I cannot comment on "better". It would be different. Both approaches
should work. End results would not be identical.
A reshape would take longer, but would leave you with just one array to
manage.
NeilBrown
>
> Regards,
> Veljko
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Linear device of two arrays
From: NeilBrown @ 2017-07-14 2:05 UTC (permalink / raw)
To: Veljko, linux-raid
In-Reply-To: <87o9sn232n.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 455 bytes --]
On Fri, Jul 14 2017, NeilBrown wrote:
>
> Now create the linear from /dev/md2 and /dev/md3(?). Be sure to use
> "-e 1.0 --data-offset=0". This creates /dev/md4
I meant to add:
if you want the linear device to align with raid10 stripes (which seems
elegant, and might slightly help performance is rare cases) you can use
the "--rounding" mdadm option. e.g.
mdadm --create /dev/md4 -l linear --rounding=1M -e 1.0 --data-offset=0 .....
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: [opensuse] raid device mounting problem in Leap 42.2
From: NeilBrown @ 2017-07-14 5:35 UTC (permalink / raw)
To: David C. Rankin, suse; +Cc: linux-raid
In-Reply-To: <6df95e9a-1281-ccc6-0d40-7962d8f42bc5@suddenlinkmail.com>
[-- Attachment #1: Type: text/plain, Size: 5192 bytes --]
On Thu, Jul 13 2017, David C. Rankin wrote:
> On 07/13/2017 08:40 PM, NeilBrown wrote:
>
>>
>> To the byte? I really like to see the output of commands that report
>> sizes, rather than have someone claim "were the same size". It isn't
>> that I don't trust you. I just don't trust humans in general (myself included).
>>
>> NeilBrown
>>
>
> Yep,
>
> Earlier in the post we have:
Thanks. (I'm not subscribed to opensuse@opensuse.org, so I couldn't see
any of this before. Also, people on that list might not see my reply.
Feel free to forward this email to that list.
I'm Ccing to linux-raid as that is the list that I first heard about
this on).
>
> <quote>
>
> It is assembled from /dev/sdb9 and /dev/sdc9.
>
> # sfdisk -l /dev/sdb
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Device Boot Start End Sectors Size Id Type
> /dev/sdb9 298005813 360916289 62910477 30G fd Linux raid autodetect
>
> # sfdisk -l /dev/sdc
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Device Boot Start End Sectors Size Id Type
> /dev/sdc9 298005813 360916289 62910477 30G fd Linux raid autodetect
Device is 62910477 sectors. So 29.998053 GiB or 32.210164 GB
>
> </quote>
>
> and the following at the beginning of the thread
>
> <quote>
>
>
> I have several mdraid RAID1 (mirror) devices I used without
> problem in openSUSE 12.2. In openSUSE Leap 42.2 I can't
> mount some of the same raid devices.
>
> In openSUSE 12.2 I can mount the raid device:
>
> cat /proc/mdstat
>
> md9 : active raid1 sdc9[1] sdb9[0]
> 31455164 blocks super 1.0 [2/2] [UU]
Array is 31455164 KiB, so 29.997982 GiB or 32.210087 GB.
>
> # mount /dev/md9 /mnt -o ro
> #
>
> # df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/md9 30G 28G 364M 99% /mnt
>
"-h" means to use powers of 1024, so this "30G" compares with
the array size of "29.997982 GiB".
I don't know exactly how 'df' formats numbers (and reading the code
makes my head spin), but I wouldn't be surprised if it chose to print
29.997 as 30.
Can we see the output of "df" without the "-h" ('h' stands for 'human'
and we know I don't trust those :-).
>
>
> In openSUSE Leap 42.2 I can't mount the same raid device:
>
> cat /proc/mdstat
>
> md9 : active raid1 sdc9[1] sdb9[0]
> 31455164 blocks super 1.0 [2/2] [UU]
>
>
> # mount /dev/md9 /mnt -o ro
> mount: wrong fs type, bad option, bad superblock on /dev/md9,
> missing codepage or helper program, or other error
>
> In some cases useful info is found in syslog - try
> dmesg | tail or so.
Has there been any report of the output of this "dmesg | tail" command?
>
>
> Why is this and how can I fix it?
>
> </quote>
>
> Glad to see we have a direct pipeline to the master here :)
So I went hunting in the archives (I shouldn't have to..) and found:
> The filesystem size (according to the superblock) is 7863809 blocks
> The physical size of the device is 7863791 blocks
This would be 4K blocks, so
filesystem: 7863809 blocks, 31455236K, 62910472 sectors
device: 7863791 blocks, 31455164K, 62910328 sectors
So the sd[bc]9 is 5 sectors larger than the filesystem.
Hmm. opensuse 12.2... That had a 3.4.6 kernel?
3.4 (and up to 4.3) had separate ext3 and ext4 implementations.
In 4.3, ext3 was discarded and ext4 used to manage ext3 filesystems.
So small changes in behaviour for corner cases are not impossible.
ext4 has a test :
/* check blocks count against device size */
blocks_count = sb->s_bdev->bd_inode->i_size >> sb->s_blocksize_bits;
if (blocks_count && ext4_blocks_count(es) > blocks_count) {
ext4_msg(sb, KERN_WARNING, "bad geometry: block count %llu "
"exceeds size of device (%llu blocks)",
ext4_blocks_count(es), blocks_count);
goto failed_mount;
}
which is presumably failing. If "dmesg | tail" had been run, we
probably would have seen that.
This test has been present since 2.6.30.
I don't think ext3 ever got this test (until it became ext4).
So there is your answer.
The filesystem was created on the member device, instead of on the
array.
The kernel being used didn't check for this inconsistency.
fsck does, but presumably was never run.
The new kernel does check - as it should.
I would suggest:
1/ stop the array (mdadm -S /dev/md9)
2/ run fsck --force /dev/sdb9
3/ run resize2fs /dev/sdb9 29G (or something like that)
4/ "mdadm -C /dev/md9 -l1 -e 1.0 -n2 /dev/sdb9 missing"
5/ "fsck /dev/md9" - if there are errors, stop here.
6/ "resize2fs /dev/md9" (which will make use of all available space)
only continue if you are really sure /dev/md9 looks good
7/ mdadm --zero /dev/sdc9
8/ mdadm /dev/md9 --add /dev/sdc9
9/ wait for resync to complete.
NeilBrown
>
>
> --
> David C. Rankin, J.D.,P.E.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Roman Mamedov @ 2017-07-14 7:22 UTC (permalink / raw)
To: Chris Murphy; +Cc: Linux-RAID
In-Reply-To: <CAJCQCtS=o27zF_sUEv0rTos_0x_7tNMdudBaDDOwxSvTz3zE5w@mail.gmail.com>
On Thu, 13 Jul 2017 19:48:29 -0600
Chris Murphy <lists@colorremedies.com> wrote:
> Btrfs still tolerates many errors, read and write
Actually it was Btrfs which saved me back then. Btrfs was making two copies
of metadata blocks and restored corrupted copies from good ones, and also
signaled me that user files were also affected (via data checksums).
FS checksums do work, and if you have redundancy for the corrupted part (such
as metadata DUP by default, or data DUP (unusual) or data RAID1), allow the FS
to sustain through corruptions, including hardware-caused ones.
--
With respect,
Roman
^ permalink raw reply
* [PATCH v2 0/3] md: three misc changes
From: Ming Lei @ 2017-07-14 8:14 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: NeilBrown, linux-block, Jens Axboe, Christoph Hellwig, Ming Lei
This 1st patch fixes one issue introduced in the following two
commits:
Fixes: f0250618361d(md: raid10: don't use bio's vec table to manage resync pages)
Fixes: 98d30c5812c3(md: raid1: don't use bio's vec table to manage resync pages)
The 2nd one initializes bvec table of bio via bio_add_page() after bio_reset().
The 3rd one moves the common definitation and helpers into raid1-10.c.
V2:
- fix 'page_idx' increasement in patch 1
- move raid1/raid10 common code to raid1-raid10.c, as suggested by Neil
Ming Lei (3):
md: remove 'idx' from 'struct resync_pages'
md: raid1/raid10: initialize bvec table via bio_add_page()
md: raid1-10: move raid1/raid10 common code into raid1-10.c
drivers/md/md.h | 54 ----------------------------------
drivers/md/raid1-10.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++
drivers/md/raid1.c | 31 ++++----------------
drivers/md/raid10.c | 19 ++++--------
4 files changed, 93 insertions(+), 92 deletions(-)
create mode 100644 drivers/md/raid1-10.c
--
2.9.4
^ permalink raw reply
* [PATCH v2 1/3] md: remove 'idx' from 'struct resync_pages'
From: Ming Lei @ 2017-07-14 8:14 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: NeilBrown, linux-block, Jens Axboe, Christoph Hellwig, Ming Lei
In-Reply-To: <20170714081444.32645-1-ming.lei@redhat.com>
bio_add_page() won't fail for resync bio, and the page index for each
bio is same, so remove it.
More importantly the 'idx' of 'struct resync_pages' is initialized in
mempool allocator function, the current way is wrong since mempool is
only responsible for allocation, we can't use that for initialization.
Suggested-by: NeilBrown <neilb@suse.com>
Reported-by: NeilBrown <neilb@suse.com>
Fixes: f0250618361d(md: raid10: don't use bio's vec table to manage resync pages)
Fixes: 98d30c5812c3(md: raid1: don't use bio's vec table to manage resync pages)
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
drivers/md/md.h | 1 -
drivers/md/raid1.c | 6 +++---
drivers/md/raid10.c | 6 +++---
3 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 991f0fe2dcc6..2c780aa8d07f 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -736,7 +736,6 @@ static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio
/* for managing resync I/O pages */
struct resync_pages {
- unsigned idx; /* for get/put page from the pool */
void *raid_bio;
struct page *pages[RESYNC_PAGES];
};
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 3febfc8391fb..0896c772a560 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -170,7 +170,6 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
resync_get_all_pages(rp);
}
- rp->idx = 0;
rp->raid_bio = r1_bio;
bio->bi_private = rp;
}
@@ -2619,6 +2618,7 @@ static sector_t raid1_sync_request(struct mddev *mddev, sector_t sector_nr,
int good_sectors = RESYNC_SECTORS;
int min_bad = 0; /* number of sectors that are bad in all devices */
int idx = sector_to_idx(sector_nr);
+ int page_idx = 0;
if (!conf->r1buf_pool)
if (init_resync(conf))
@@ -2846,7 +2846,7 @@ static sector_t raid1_sync_request(struct mddev *mddev, sector_t sector_nr,
bio = r1_bio->bios[i];
rp = get_resync_pages(bio);
if (bio->bi_end_io) {
- page = resync_fetch_page(rp, rp->idx++);
+ page = resync_fetch_page(rp, page_idx);
/*
* won't fail because the vec table is big
@@ -2858,7 +2858,7 @@ static sector_t raid1_sync_request(struct mddev *mddev, sector_t sector_nr,
nr_sectors += len>>9;
sector_nr += len>>9;
sync_blocks -= (len>>9);
- } while (get_resync_pages(r1_bio->bios[disk]->bi_private)->idx < RESYNC_PAGES);
+ } while (++page_idx < RESYNC_PAGES);
r1_bio->sectors = nr_sectors;
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 5026e7ad51d3..fa8bcf04e791 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -221,7 +221,6 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
resync_get_all_pages(rp);
}
- rp->idx = 0;
rp->raid_bio = r10_bio;
bio->bi_private = rp;
if (rbio) {
@@ -2853,6 +2852,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
sector_t sectors_skipped = 0;
int chunks_skipped = 0;
sector_t chunk_mask = conf->geo.chunk_mask;
+ int page_idx = 0;
if (!conf->r10buf_pool)
if (init_resync(conf))
@@ -3355,7 +3355,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
break;
for (bio= biolist ; bio ; bio=bio->bi_next) {
struct resync_pages *rp = get_resync_pages(bio);
- page = resync_fetch_page(rp, rp->idx++);
+ page = resync_fetch_page(rp, page_idx);
/*
* won't fail because the vec table is big enough
* to hold all these pages
@@ -3364,7 +3364,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
}
nr_sectors += len>>9;
sector_nr += len>>9;
- } while (get_resync_pages(biolist)->idx < RESYNC_PAGES);
+ } while (++page_idx < RESYNC_PAGES);
r10_bio->sectors = nr_sectors;
while (biolist) {
--
2.9.4
^ permalink raw reply related
* [PATCH v2 2/3] md: raid1/raid10: initialize bvec table via bio_add_page()
From: Ming Lei @ 2017-07-14 8:14 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: NeilBrown, linux-block, Jens Axboe, Christoph Hellwig, Ming Lei
In-Reply-To: <20170714081444.32645-1-ming.lei@redhat.com>
We will support multipage bvec soon, so initialize bvec
table using the standardy way instead of writing the
talbe directly. Otherwise it won't work any more once
multipage bvec is enabled.
Acked-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
drivers/md/raid1-10.c | 19 +++++++++++++++++++
drivers/md/raid1.c | 18 ++++--------------
drivers/md/raid10.c | 6 ++++--
3 files changed, 27 insertions(+), 16 deletions(-)
create mode 100644 drivers/md/raid1-10.c
diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
new file mode 100644
index 000000000000..3adb5b9dc4b4
--- /dev/null
+++ b/drivers/md/raid1-10.c
@@ -0,0 +1,19 @@
+/* generally called after bio_reset() for reseting bvec */
+static void md_bio_reset_resync_pages(struct bio *bio, struct resync_pages *rp,
+ int size)
+{
+ int idx = 0;
+
+ /* initialize bvec table again */
+ do {
+ struct page *page = resync_fetch_page(rp, idx);
+ int len = min_t(int, size, PAGE_SIZE);
+
+ /*
+ * won't fail because the vec table is big
+ * enough to hold all these pages
+ */
+ bio_add_page(bio, page, len, 0);
+ size -= len;
+ } while (idx++ < RESYNC_PAGES && size > 0);
+}
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 0896c772a560..fe86ab18961b 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -81,6 +81,8 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
#define raid1_log(md, fmt, args...) \
do { if ((md)->queue) blk_add_trace_msg((md)->queue, "raid1 " fmt, ##args); } while (0)
+#include "raid1-10.c"
+
/*
* 'strct resync_pages' stores actual pages used for doing the resync
* IO, and it is per-bio, so make .bi_private points to it.
@@ -2085,10 +2087,7 @@ static void process_checks(struct r1bio *r1_bio)
/* Fix variable parts of all bios */
vcnt = (r1_bio->sectors + PAGE_SIZE / 512 - 1) >> (PAGE_SHIFT - 9);
for (i = 0; i < conf->raid_disks * 2; i++) {
- int j;
- int size;
blk_status_t status;
- struct bio_vec *bi;
struct bio *b = r1_bio->bios[i];
struct resync_pages *rp = get_resync_pages(b);
if (b->bi_end_io != end_sync_read)
@@ -2097,8 +2096,6 @@ static void process_checks(struct r1bio *r1_bio)
status = b->bi_status;
bio_reset(b);
b->bi_status = status;
- b->bi_vcnt = vcnt;
- b->bi_iter.bi_size = r1_bio->sectors << 9;
b->bi_iter.bi_sector = r1_bio->sector +
conf->mirrors[i].rdev->data_offset;
b->bi_bdev = conf->mirrors[i].rdev->bdev;
@@ -2106,15 +2103,8 @@ static void process_checks(struct r1bio *r1_bio)
rp->raid_bio = r1_bio;
b->bi_private = rp;
- size = b->bi_iter.bi_size;
- bio_for_each_segment_all(bi, b, j) {
- bi->bv_offset = 0;
- if (size > PAGE_SIZE)
- bi->bv_len = PAGE_SIZE;
- else
- bi->bv_len = size;
- size -= PAGE_SIZE;
- }
+ /* initialize bvec table again */
+ md_bio_reset_resync_pages(b, rp, r1_bio->sectors << 9);
}
for (primary = 0; primary < conf->raid_disks * 2; primary++)
if (r1_bio->bios[primary]->bi_end_io == end_sync_read &&
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index fa8bcf04e791..9952721e1cde 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -110,6 +110,8 @@ static void end_reshape(struct r10conf *conf);
#define raid10_log(md, fmt, args...) \
do { if ((md)->queue) blk_add_trace_msg((md)->queue, "raid10 " fmt, ##args); } while (0)
+#include "raid1-10.c"
+
/*
* 'strct resync_pages' stores actual pages used for doing the resync
* IO, and it is per-bio, so make .bi_private points to it.
@@ -2086,8 +2088,8 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
rp = get_resync_pages(tbio);
bio_reset(tbio);
- tbio->bi_vcnt = vcnt;
- tbio->bi_iter.bi_size = fbio->bi_iter.bi_size;
+ md_bio_reset_resync_pages(tbio, rp, fbio->bi_iter.bi_size);
+
rp->raid_bio = r10_bio;
tbio->bi_private = rp;
tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
--
2.9.4
^ permalink raw reply related
* [PATCH v2 3/3] md: raid1-10: move raid1/raid10 common code into raid1-10.c
From: Ming Lei @ 2017-07-14 8:14 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: NeilBrown, linux-block, Jens Axboe, Christoph Hellwig, Ming Lei
In-Reply-To: <20170714081444.32645-1-ming.lei@redhat.com>
No function change, just move 'struct resync_pages' and related
helpers into raid1-10.c
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
drivers/md/md.h | 53 -------------------------------------------
drivers/md/raid1-10.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++
drivers/md/raid1.c | 9 --------
drivers/md/raid10.c | 9 --------
4 files changed, 62 insertions(+), 71 deletions(-)
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 2c780aa8d07f..004a21c214e8 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -729,57 +729,4 @@ static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio
!bdev_get_queue(bio->bi_bdev)->limits.max_write_zeroes_sectors)
mddev->queue->limits.max_write_zeroes_sectors = 0;
}
-
-/* Maximum size of each resync request */
-#define RESYNC_BLOCK_SIZE (64*1024)
-#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
-
-/* for managing resync I/O pages */
-struct resync_pages {
- void *raid_bio;
- struct page *pages[RESYNC_PAGES];
-};
-
-static inline int resync_alloc_pages(struct resync_pages *rp,
- gfp_t gfp_flags)
-{
- int i;
-
- for (i = 0; i < RESYNC_PAGES; i++) {
- rp->pages[i] = alloc_page(gfp_flags);
- if (!rp->pages[i])
- goto out_free;
- }
-
- return 0;
-
-out_free:
- while (--i >= 0)
- put_page(rp->pages[i]);
- return -ENOMEM;
-}
-
-static inline void resync_free_pages(struct resync_pages *rp)
-{
- int i;
-
- for (i = 0; i < RESYNC_PAGES; i++)
- put_page(rp->pages[i]);
-}
-
-static inline void resync_get_all_pages(struct resync_pages *rp)
-{
- int i;
-
- for (i = 0; i < RESYNC_PAGES; i++)
- get_page(rp->pages[i]);
-}
-
-static inline struct page *resync_fetch_page(struct resync_pages *rp,
- unsigned idx)
-{
- if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
- return NULL;
- return rp->pages[idx];
-}
#endif /* _MD_MD_H */
diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
index 3adb5b9dc4b4..9f2670b45f31 100644
--- a/drivers/md/raid1-10.c
+++ b/drivers/md/raid1-10.c
@@ -1,3 +1,65 @@
+/* Maximum size of each resync request */
+#define RESYNC_BLOCK_SIZE (64*1024)
+#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
+
+/* for managing resync I/O pages */
+struct resync_pages {
+ void *raid_bio;
+ struct page *pages[RESYNC_PAGES];
+};
+
+static inline int resync_alloc_pages(struct resync_pages *rp,
+ gfp_t gfp_flags)
+{
+ int i;
+
+ for (i = 0; i < RESYNC_PAGES; i++) {
+ rp->pages[i] = alloc_page(gfp_flags);
+ if (!rp->pages[i])
+ goto out_free;
+ }
+
+ return 0;
+
+out_free:
+ while (--i >= 0)
+ put_page(rp->pages[i]);
+ return -ENOMEM;
+}
+
+static inline void resync_free_pages(struct resync_pages *rp)
+{
+ int i;
+
+ for (i = 0; i < RESYNC_PAGES; i++)
+ put_page(rp->pages[i]);
+}
+
+static inline void resync_get_all_pages(struct resync_pages *rp)
+{
+ int i;
+
+ for (i = 0; i < RESYNC_PAGES; i++)
+ get_page(rp->pages[i]);
+}
+
+static inline struct page *resync_fetch_page(struct resync_pages *rp,
+ unsigned idx)
+{
+ if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
+ return NULL;
+ return rp->pages[idx];
+}
+
+/*
+ * 'strct resync_pages' stores actual pages used for doing the resync
+ * IO, and it is per-bio, so make .bi_private points to it.
+ */
+static inline struct resync_pages *get_resync_pages(struct bio *bio)
+{
+ return bio->bi_private;
+}
+
/* generally called after bio_reset() for reseting bvec */
static void md_bio_reset_resync_pages(struct bio *bio, struct resync_pages *rp,
int size)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index fe86ab18961b..8387eb1540cd 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -84,15 +84,6 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
#include "raid1-10.c"
/*
- * 'strct resync_pages' stores actual pages used for doing the resync
- * IO, and it is per-bio, so make .bi_private points to it.
- */
-static inline struct resync_pages *get_resync_pages(struct bio *bio)
-{
- return bio->bi_private;
-}
-
-/*
* for resync bio, r1bio pointer can be retrieved from the per-bio
* 'struct resync_pages'.
*/
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 9952721e1cde..e2617d0f37dc 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -113,15 +113,6 @@ static void end_reshape(struct r10conf *conf);
#include "raid1-10.c"
/*
- * 'strct resync_pages' stores actual pages used for doing the resync
- * IO, and it is per-bio, so make .bi_private points to it.
- */
-static inline struct resync_pages *get_resync_pages(struct bio *bio)
-{
- return bio->bi_private;
-}
-
-/*
* for resync bio, r10bio pointer can be retrieved from the per-bio
* 'struct resync_pages'.
*/
--
2.9.4
^ permalink raw reply related
* Re: Filesystem corruption on RAID1
From: Gionatan Danti @ 2017-07-14 10:46 UTC (permalink / raw)
To: Reindl Harald; +Cc: Roman Mamedov, linux-raid, g.danti
In-Reply-To: <770b09d3-cff6-b6b2-0a51-5d11e8bac7e9@thelounge.net>
Il 14-07-2017 02:32 Reindl Harald ha scritto:
> because you won't be that happy when the kernel spits out a disk each
> time a random SATA command times out - the 4 RAID10 disks on my
> workstation are from 2011 and showed them too several times in the
> past while they are just fine
>
> here you go:
> http://strugglers.net/~andy/blog/2015/11/09/linux-software-raid-and-drive-timeouts/
Hi, so a premature/preventive drive detachment is not a silver bullet,
and I buy it. However, I would at least expect this behavior to be
configurable. Maybe it is, and I am missing something?
Anyway, what really surprise me is *not* the drive to not be detached,
rather permitting that corruption make its way into real data. I naively
expect that when a WRITE_QUEUED or CACHE_FLUSH command aborts/fails
(which *will* cause data corruption if not properly handled) the I/O
layer has the following possibilities:
a) retry the write/flush. You don't want to retry indefinitely, so the
kernel need some type of counter/threshold; when the counter is reached,
continue with b). This would mask out sporadic errors, while propagating
recurring ones;
b) notify the upper layer that a write error happened. For synchronized
and direct writes it can notify that by simply returning the correct
exit code to the calling function. In this case, the block layer should
return an error to the MD driver, which must act accordlying: for
example, dropping the disk from the array.
c) do nothing. This seems to me by far the worst choice.
If b) is correcly implemented, it should prevent corruption to
accumulate on the drives.
Please also note the *type* of corrupted data: not only user data, but
filesystem journal and metadata also. The latter should be protected by
the using of write barriers / FUAs, so they should be able to stop
themselves *before* corruption.
So I have some very important questions:
- how does MD behave when flushing data to disk?
- does it propagate write barriers?
- when a write barrier fails, is the error propagated to the upper
layers?
Thanks you all.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8
^ permalink raw reply
* Re: Filesystem corruption on RAID1
From: Reindl Harald @ 2017-07-14 10:58 UTC (permalink / raw)
To: Gionatan Danti; +Cc: Roman Mamedov, linux-raid
In-Reply-To: <9eea45ddc0f80f4f4e238b5c2527a1fa@assyoma.it>
Am 14.07.2017 um 12:46 schrieb Gionatan Danti:
> Il 14-07-2017 02:32 Reindl Harald ha scritto:
>> because you won't be that happy when the kernel spits out a disk each
>> time a random SATA command times out - the 4 RAID10 disks on my
>> workstation are from 2011 and showed them too several times in the
>> past while they are just fine
>>
>> here you go:
>> http://strugglers.net/~andy/blog/2015/11/09/linux-software-raid-and-drive-timeouts/
>>
>
> Hi, so a premature/preventive drive detachment is not a silver bullet,
> and I buy it. However, I would at least expect this behavior to be
> configurable. Maybe it is, and I am missing something?
dunno, maybe it is, but it wouldn't be wise because in case of a RAID5
rebuilding after a disk-failure would become even more dangerous on a
large array as it is already
> Anyway, what really surprise me is *not* the drive to not be detached,
> rather permitting that corruption make its way into real data. I naively
> expect that when a WRITE_QUEUED or CACHE_FLUSH command aborts/fails
> (which *will* cause data corruption if not properly handled) the I/O
> layer has the following possibilities:
given that i have seen at least SD-cards confirming over hours sucessful
writes with no single error in the syslog maybe it was one of the rare
cases where the hardware lied and if that is the case you have nearly no
chance on the software layer except verify each write with a uncached
read of the block which would have a unacceptable impact on performance
^ permalink raw reply
* Re: Linear device of two arrays
From: Veljko @ 2017-07-14 13:40 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <87o9sn232n.fsf@notabene.neil.brown.name>
On 07/14/2017 03:57 AM, NeilBrown wrote:
>
> Presumably you also have an md3 raid10 which you want to attach to the
> end of md2?
Yes, I just created it. It's resyncing.
> md2 is 5761631232 sectors.
> 2880815616 kilobytes
> 2813296.5 (binary)megabytes.
>
> When you include that into a "linear" you will lose a few K from the
> end.
> It might be sensible to cause the "linear" to use whole stripes from
> the raid10, where a stripe is 1M (2 512K chunks).
> If you did that, you would lose a little over 1M.
> So backup the last 3.5 M of the raid10. This is much more than you need.
>
> ie.
>
> dd if=/dev/md2 of=SOMEWHERE/SAFE bs=1M skip=2813293
I'm little confused. What I'm backuping is last 3.5M of used space,
right? How is that only ~2.7T? df shows 4.8T of used space.
Rest of the instructions are clear and I'll try it as soon as md3 is
synced.
Thanks Neil!
> (dd treats 'M' as 1024*1024, MB is 1000*1000)
>
> If the file this creates is not 3.5M, then something went wrong. Stop
> here.
>
> Just to be safe you might want to backup the first few megabytes. You
> won't need this unless something goes wrong
> dd if=/dev/md2 of=SOMEWHERE/ELSE bs=1M count=10
>
> Now create the linear from /dev/md2 and /dev/md3(?). Be sure to use
> "-e 1.0 --data-offset=0". This creates /dev/md4
>
> Now restore the first backup
>
> dd if=SOMEWHERE/SAFE of=/dev/md4 bs=1M seek=2813293
>
> Be sure to use the same bs= and seek= as you did the first time.
> Be sure it is copying from the back and to the new linear raid.
>
> You should now be done. Check your xfs filesystem, and maybe even mount
> it and use it.
>
> NeilBrown
>
>>
>> # mdadm --detail /dev/md2
>> /dev/md2:
>> Version : 1.2
>> Creation Time : Fri Sep 14 12:40:13 2012
>> Raid Level : raid10
>> Array Size : 5761631232 (5494.72 GiB 5899.91 GB)
>> Used Dev Size : 2880815616 (2747.36 GiB 2949.96 GB)
>> Raid Devices : 4
>> Total Devices : 4
>> Persistence : Superblock is persistent
>>
>> Update Time : Mon Jul 10 12:32:51 2017
>> State : clean
>> Active Devices : 4
>> Working Devices : 4
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : near=2
>> Chunk Size : 512K
>>
^ permalink raw reply
* Re: Linear device of two arrays
From: NeilBrown @ 2017-07-15 0:12 UTC (permalink / raw)
To: Veljko, linux-raid
In-Reply-To: <1eefd627-1aba-a795-05f0-d2106d3a62a3@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2951 bytes --]
On Fri, Jul 14 2017, Veljko wrote:
> On 07/14/2017 03:57 AM, NeilBrown wrote:
>>
>> Presumably you also have an md3 raid10 which you want to attach to the
>> end of md2?
>
> Yes, I just created it. It's resyncing.
>
>
>> md2 is 5761631232 sectors.
>> 2880815616 kilobytes
>> 2813296.5 (binary)megabytes.
>>
>> When you include that into a "linear" you will lose a few K from the
>> end.
>> It might be sensible to cause the "linear" to use whole stripes from
>> the raid10, where a stripe is 1M (2 512K chunks).
>> If you did that, you would lose a little over 1M.
>> So backup the last 3.5 M of the raid10. This is much more than you need.
>>
>> ie.
>>
>> dd if=/dev/md2 of=SOMEWHERE/SAFE bs=1M skip=2813293
>
> I'm little confused. What I'm backuping is last 3.5M of used space,
> right? How is that only ~2.7T? df shows 4.8T of used space.
You were right to check.
md2 is 5761631232 kilobytes, not sectors.
so 5626593 (binary) megabytes (exactly)
So command should be
>> dd if=/dev/md2 of=SOMEWHERE/SAFE bs=1M skip=5626590
and expect it to create a 3M file.
Use this 'skip' number of the 'seek' number later.
NeilBrown
>
> Rest of the instructions are clear and I'll try it as soon as md3 is
> synced.
>
> Thanks Neil!
>
>> (dd treats 'M' as 1024*1024, MB is 1000*1000)
>>
>> If the file this creates is not 3.5M, then something went wrong. Stop
>> here.
>>
>> Just to be safe you might want to backup the first few megabytes. You
>> won't need this unless something goes wrong
>> dd if=/dev/md2 of=SOMEWHERE/ELSE bs=1M count=10
>>
>> Now create the linear from /dev/md2 and /dev/md3(?). Be sure to use
>> "-e 1.0 --data-offset=0". This creates /dev/md4
>>
>> Now restore the first backup
>>
>> dd if=SOMEWHERE/SAFE of=/dev/md4 bs=1M seek=2813293
>>
>> Be sure to use the same bs= and seek= as you did the first time.
>> Be sure it is copying from the back and to the new linear raid.
>>
>> You should now be done. Check your xfs filesystem, and maybe even mount
>> it and use it.
>>
>> NeilBrown
>>
>>>
>>> # mdadm --detail /dev/md2
>>> /dev/md2:
>>> Version : 1.2
>>> Creation Time : Fri Sep 14 12:40:13 2012
>>> Raid Level : raid10
>>> Array Size : 5761631232 (5494.72 GiB 5899.91 GB)
>>> Used Dev Size : 2880815616 (2747.36 GiB 2949.96 GB)
>>> Raid Devices : 4
>>> Total Devices : 4
>>> Persistence : Superblock is persistent
>>>
>>> Update Time : Mon Jul 10 12:32:51 2017
>>> State : clean
>>> Active Devices : 4
>>> Working Devices : 4
>>> Failed Devices : 0
>>> Spare Devices : 0
>>>
>>> Layout : near=2
>>> Chunk Size : 512K
>>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Why can't I re-add my drive after partition shrink?
From: Ram Ramesh @ 2017-07-15 0:17 UTC (permalink / raw)
To: NeilBrown, Linux Raid
In-Reply-To: <87tw2f2425.fsf@notabene.neil.brown.name>
On 07/13/2017 08:35 PM, NeilBrown wrote:
> <snip
> Please report output of "mdadm --examine" on both a device that is active
> in the array, and the device that you are trying to add.
> Also "mdadm --examine-bitmap" of a device that is active in the array.
>
> NeilBrown
>
>> I researched as much as I could on the net and came up with nothing
>> except some one saying that mdadm keeps something at the end of the disk
>> regardless of what it says about "Used Dev Size." Is it possible to move
>> this info so that I could re-add?
>>
>> Ramesh
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Neil,
Since the problem, I did not want to leave my md in degraded state.
So, I added my drive back and paid the penalty for rebuilding. I have
other disks that need to be resized and *can get you want*. Please let
me know if that is what you meant. If you wanted the current info after
successfully rebuilding the array after a regular add, it is below.
First the device that is currently in after a regular add.
> zym [rramesh] 497 > sudo mdadm --examine /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Name : zym:0 (local to host zym)
> Creation Time : Mon Apr 22 00:08:12 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 6442188800 (3071.88 GiB 3298.40 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 702ca77d:564d69ff:e45d9679:64c314fa
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Fri Jul 14 18:36:37 2017
> Checksum : c5502356 - correct
> Events : 297182
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 4
> Array State : AAAAAA ('A' == active, '.' == missing)
Member disk that has been in the array before /dev/sdb1 resize
> zym [rramesh] 498 > sudo mdadm --examine /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Name : zym:0 (local to host zym)
> Creation Time : Mon Apr 22 00:08:12 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 7e035b56:d1e1882b:e78a08ad:3ba50667
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Fri Jul 14 18:36:37 2017
> Checksum : a5288a4c - correct
> Events : 297182
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 2
> Array State : AAAAAA ('A' == active, '.' == missing)
Now examine bit map on /dev/sdb1 (device that got in after a regular add)
> zym [rramesh] 499 > sudo mdadm --examine-bitmap /dev/sdb1
> Filename : /dev/sdb1
> Magic : 6d746962
> Version : 4
> UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Events : 297182
> Events Cleared : 297182
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 3087007744 (2944.00 GiB 3161.10 GB)
> Bitmap : 47104 bits (chunks), 0 dirty (0.0%)
Ramesh
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox