* Re: Raid5 drive fail during grow and no backup
From: Phillip Susi @ 2014-12-04 19:29 UTC (permalink / raw)
To: Phil Turmel, P. Gautschi; +Cc: Vince, linux-raid
In-Reply-To: <545D8FBA.9090701@turmel.org>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 11/7/2014 10:36 PM, Phil Turmel wrote:
> However, if the device with the bad sector is trying to recover
> longer than the linux low level driver's timeout, bad things^TM
> happen. Specifically, the driver resets the SATA (or SCSI)
> connection and attempts to reconnect. During this brief time, it
> will not accept further I/O, so the write back of the reconstructed
> data fails. Then the device has experienced a *write* error, so MD
> fails the drive. This is the out-of-the-box behavior of
> consumer-grade drives in raid arrays.
What? During the recovery action ( reset and retry ), a write being
issued to the drive should just sit in the request queue until after
the drive finishes being reset; it should not just be failed outright.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUgLX+AAoJENRVrw2cjl5RHb4H+wWzuTFekQMwIoX7Vov5QjLh
XyEmgbwqgtdcnsbQqtnNiQK0k8KVxQDW3xzWkB30PkOjWMfldES3dRjFXuNbZ0r1
FnJeIYbChFBnfJLp/BqHHOnL5YHD81HvENJ4M/OW6t9SpSiFuOieFe7WTEwHoh5t
t9v/J0+x84CQu1q/AF7FRMkLE1fYhZieAMLTyKhbo5TmMm5XSP8eXumMCz+PXmvV
tVN6rYejSozl1wfwa0l4N9jwkyYWLgbzFRIR7PuQNacywFyLhg0WtIPnqjNV6YuL
rAl5VBFbHEn6BwklgxDWkzSIuOIt2ce6KIE0JZtqTGlDajhSUM+ojTPALLFcmLE=
=4mKS
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: Good drive got kicked out - does mdadm assemble across suspend/resume like it does across reboots?
From: Phillip Susi @ 2014-12-04 19:10 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <543ABC82.5020901@gmail.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 10/12/2014 1:38 PM, Ram Ramesh wrote:
> While this is not a disaster, I wonder if my understanding is
> correct? Does this mean mdadm does not scan to assemble as part of
> resume? More importantly, how should I rectify the situation like
> this? Reboot or simply stop md0 and scan in a live system? Further,
> how do I use /dev/sdf1 (that was /dev/sdg1 before this problem)?
> zero-superblock and add back to md0?
Of course it does not reassemble the array after a suspend -- the
point of suspend/resume is that the system picks up exactly where it
left off. Since the array is never torn down, it is not built back
up. In other words, yanking out drives while suspended isn't really
any different than doing it while running -- not a good idea.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUgLGMAAoJENRVrw2cjl5RLOkH/iZVwQ6FC664CN4y2hGiIOYD
FX4VLtRdEDy517sdqm15nv4LTqfGucZjfiEDtwRs5rQZsIyCOelah8RvxYUOTNrJ
xbURswZTaNhvaEG1/gwIlJoQ5cawVetZKeL6PnMWHpSLHQuoIaqe/wDO3pevcTDD
7vggJvYMr8S4aVKnt8/Ksk/E6DVCsGvCuySvHOr3eGecJuLF4CPZYFZy5G0qqRuo
piI918gushh6CgriFOCdBWice3HwK6V1Nb12kjkVY89l0bQlrIRUo6e6aG1c1M9P
eVmQJOzedtQB8qgsAhc2oXEwtuR/VNWNwxNTPTHCND3nk8FcaTDgzE3MMFmi1OY=
=EXWL
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: pretty unstable raid 5 now
From: Matthew M. Dean @ 2014-12-04 11:10 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAPRzDfn33EBkci823=zw=6b3jnf-0YMr1YTU_j7RQ5qseu_fvg@mail.gmail.com>
md/raid:md0: Disk failure on loop3, disabling device.
mdadm --manage /dev/md0 --re-add /dev/loop3
mdadm: Cannot open /dev/loop3: Device or resource busy
# losetup -d /dev/loop3
# losetup -a
/dev/loop1: 0 /media/live1/node1.img
/dev/loop2: 0 /media/live2/node2.img
/dev/loop3: 0 /media/live3/node3.img
what?
# lsof | grep "loop3"
loop3 2577 root cwd DIR 0,15 0 160 /
loop3 2577 root rtd DIR 0,15 0 160 /
loop3 2577 root txt unknown
/proc/2577/exe
# ps | grep "2577"
2577 root 0 SW< [loop3]
# reboot
# mdadm --manage /dev/md0 --re-add /dev/loop3
mdadm: re-added /dev/loop3
root@OpenWrt:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 loop3[3] loop1[0] loop2[1]
3878157312 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
[>....................] recovery = 0.0% (2596/1939078656)
finish=24783.6min speed=1298K/sec
bitmap: 5/15 pages [20KB], 65536KB chunk
WHAT?
On Wed, Dec 3, 2014 at 6:08 PM, Matthew M. Dean <fireculex@gmail.com> wrote:
> So I've been running a raid 5 using an openwrt box with 3 loop files
> and lvm. All was well for over a year. Now /dev/loop3 keeps dropping
> from the raid
>
> The first time it dropped was due to the network drive actually
> locking up, it tried to start with only 1 drive and failed. I had to
> --force the loop files to start, which 2 of them actually did. I could
> no longer --re-add the drive.
>
> Using --add did add the drive to the raid but the sync started at 0%
> so have to wait for it to rebuild. 11mb/sec takes a while on a 1.5
> terabyte array.
>
> after 82% sync the router locked up I think to a oom condition, 128mb
> only. /dev/loop3 was dropped from the array again. I was able to
> --re-add. 0% SYNC AGAIN. ~30 hours wasted. This has never happened.
>
> What has changed?
>
> I upgraded from a 3.10 kernel to 3.14 and from mdadm 3.2.6 to 3.3.2
>
> Aren't bitmap's supposed to fix this? Are they useless now?
^ permalink raw reply
* pretty unstable raid 5 now
From: Matthew M. Dean @ 2014-12-04 2:08 UTC (permalink / raw)
To: linux-raid
So I've been running a raid 5 using an openwrt box with 3 loop files
and lvm. All was well for over a year. Now /dev/loop3 keeps dropping
from the raid
The first time it dropped was due to the network drive actually
locking up, it tried to start with only 1 drive and failed. I had to
--force the loop files to start, which 2 of them actually did. I could
no longer --re-add the drive.
Using --add did add the drive to the raid but the sync started at 0%
so have to wait for it to rebuild. 11mb/sec takes a while on a 1.5
terabyte array.
after 82% sync the router locked up I think to a oom condition, 128mb
only. /dev/loop3 was dropped from the array again. I was able to
--re-add. 0% SYNC AGAIN. ~30 hours wasted. This has never happened.
What has changed?
I upgraded from a 3.10 kernel to 3.14 and from mdadm 3.2.6 to 3.3.2
Aren't bitmap's supposed to fix this? Are they useless now?
^ permalink raw reply
* Re: [dm-devel] The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: zhangxiao @ 2014-12-04 1:26 UTC (permalink / raw)
To: device-mapper development, linux-raid, linux-kernel, agk
In-Reply-To: <20141203141921.GB6993@debian>
On 2014年12月03日 22:19, Joe Thornber wrote:
> On Tue, Dec 02, 2014 at 09:00:17AM -0500, Mike Snitzer wrote:
>> On Tue, Dec 02 2014 at 3:43am -0500,
>> zhangxiao <xiao.zhang@windriver.com> wrote:
>>
>>> Hi Experts,
>>>
>>> See drivers/md/dm-bufio.c as below:
>>> 50 /*
>>> 51 * Buffer hash
>>> 52 */
>>> 53 #define DM_BUFIO_HASH_BITS 20
>>> 54 #define DM_BUFIO_HASH(block) \
>>> 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
>>> 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
>>> "drivers/md/dm-bufio.c" 1854L, 45375C
>>>
>>> DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
>>> use case with limited resources. Do we have any plan to make it more
>>> flexible? Like module parameter or something else?
>>
>> No immediate plans.
>
> That hash table has been replaced with a red/black tree.
>
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.19&id=4e420c452b11edf9d510c8180ac66f529e5b6206
>
>
Great Thanks!
Best Regards
Xiao
^ permalink raw reply
* RAID1 + power fail + unwanted activation
From: Csordás Csaba @ 2014-12-03 16:20 UTC (permalink / raw)
To: linux-raid
Hi,
after a power fail I was afraid so I was thinking about making a full
gzipped image with dd from both members of my level1 array to be able
to go back to the "original" (before the crash) state in case if
*something* goes wrong during rebuild/lvm activation/fsck.
The other machine I put the disk in unfortunately also had mdadm
installed which automatically activated the degraded array (only one
member of the array was connected) at boot time.
Now, if i just put the disks back in the crashed machine, will the
right thing happen while the kernel is resyncing the blocks between
the devices? The internal bitmap was turned on, but the event counter
on the mistakenly activated member of the array might have changed. I
don't know how this -or other things I am not aware of- relates to the
actual reassembly of my RAID1 array. I'm looking for an authoritative
answer to understand the situation.
Csordás Csaba
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: Mike Snitzer @ 2014-12-03 16:17 UTC (permalink / raw)
To: device-mapper development, zhangxiao, linux-raid, linux-kernel,
agk
In-Reply-To: <20141203141921.GB6993@debian>
On Wed, Dec 03 2014 at 9:19am -0500,
Joe Thornber <thornber@redhat.com> wrote:
> On Tue, Dec 02, 2014 at 09:00:17AM -0500, Mike Snitzer wrote:
> > On Tue, Dec 02 2014 at 3:43am -0500,
> > zhangxiao <xiao.zhang@windriver.com> wrote:
> >
> > > Hi Experts,
> > >
> > > See drivers/md/dm-bufio.c as below:
> > > 50 /*
> > > 51 * Buffer hash
> > > 52 */
> > > 53 #define DM_BUFIO_HASH_BITS 20
> > > 54 #define DM_BUFIO_HASH(block) \
> > > 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
> > > 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
> > > "drivers/md/dm-bufio.c" 1854L, 45375C
> > >
> > > DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
> > > use case with limited resources. Do we have any plan to make it more
> > > flexible? Like module parameter or something else?
> >
> > No immediate plans.
>
> That hash table has been replaced with a red/black tree.
>
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.19&id=4e420c452b11edf9d510c8180ac66f529e5b6206
Ah yes, you'd think I'd have remembered that change. Thanks for keeping
me honest Joe!
^ permalink raw reply
* Re: [dm-devel] The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: Joe Thornber @ 2014-12-03 14:19 UTC (permalink / raw)
To: device-mapper development; +Cc: zhangxiao, linux-raid, linux-kernel, agk
In-Reply-To: <20141202140015.GB5985@redhat.com>
On Tue, Dec 02, 2014 at 09:00:17AM -0500, Mike Snitzer wrote:
> On Tue, Dec 02 2014 at 3:43am -0500,
> zhangxiao <xiao.zhang@windriver.com> wrote:
>
> > Hi Experts,
> >
> > See drivers/md/dm-bufio.c as below:
> > 50 /*
> > 51 * Buffer hash
> > 52 */
> > 53 #define DM_BUFIO_HASH_BITS 20
> > 54 #define DM_BUFIO_HASH(block) \
> > 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
> > 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
> > "drivers/md/dm-bufio.c" 1854L, 45375C
> >
> > DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
> > use case with limited resources. Do we have any plan to make it more
> > flexible? Like module parameter or something else?
>
> No immediate plans.
That hash table has been replaced with a red/black tree.
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.19&id=4e420c452b11edf9d510c8180ac66f529e5b6206
^ permalink raw reply
* Re: stray raid10 with 9 hdd with -n3 layout
From: Klaus Thorn @ 2014-12-03 13:08 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <2137843278.261407311736686.JavaMail.root@shiva>
<luvar <at> plaintext.sk> writes:
> They probably need to be warmed up before they provide access to all hdd.
BIOS or EFI or Hardware Controller may have an option to delay boot process
for a few seconds.
> And than I have nine disks which should assemble single raid with raid10,
n3 layout. I have done mdadm
> 1. why does my array assemble automatically?
This is the default. You may be able to prevent this with kernel arguments
or manipulation of the initial ramdisk.
To give you a starting point for research: "raid=noautodetect".
> 2. is there possibility to assemble array in a such way, that array will
"elect" which data is on more disks
> and rewrite last disk if needed?
The default (and to my knowledge the only algorithm available in Linux
software raid) is to choose the disk with the highest event count. The event
counter is part of the meta data saved in each member of a raid.
> 3. is there possibility to assemble array in cooperation with filesystem?
That filesystem will have
> chance to choose blocks from all three disks and choose correct one from
his point of view?
not that I heard of. You could check other filesystems with built-in raid,
though: btrfs and zfs.
> 4. what should I do to have my data OK?
delay assembly, I guess.
^ permalink raw reply
* Re: md raid performance with 3-18-rc3
From: NeilBrown @ 2014-12-03 6:21 UTC (permalink / raw)
To: Manish Awasthi; +Cc: linux-raid
In-Reply-To: <54758B3B.5080907@caviumnetworks.com>
[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]
On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
<manish.awasthi@caviumnetworks.com> wrote:
> Whatever data I have on comparison is attached, I have consolidated this
> from log files to excel. See if this helps.
raid_3_18_performance.xls shows read throughput to be consistently 20% down
on 3.18 compared to 3.6.11.
Writes are a few percent better for 4G/8G files, 20% better for 16G/32G files.
unchanged above that.
Given that you have 8G of RAM, that seems like it could be some change in
caching behaviour, and not necessarily a change in RAID behaviour.
The CPU utilization roughly follows the throughput: 40% higher when write
throughput is 20% better.
Could you check if the value of /proc/sys/vm/dirty_ratio is the same for both
tests. That number has changed occasionally and could affect these tests.
The second file, 3SSDs-perf-2-Cores-3.18-rc1 has the "change" numbers
negative where I expected positive.. i.e. negative mean an increase.
Writes consistently have higher CPU utilisation.
Reads consistently have much lower CPU utilization.
I don't know what that means ... it might not mean anything.
Could you please run the tests between the two kernels *with* RAID. i.e.
directly on an SSD. That will give us a baseline for what changes are caused
by other parts of the kernel (filesystem, block layer, MM, etc). Then we can
see how much change RAID5 is contributing.
The third file, 3SSDs-perf-4Core.xls seems to show significantly reduced
throughput across the board.
CPU utilization is less (better) for writes, but worse for reads. That is
the reverse of what the second file shows.
I might try running some tests across a set of kernel versions and see what I
can come up with.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: md raid performance with 3-18-rc3
From: NeilBrown @ 2014-12-03 5:19 UTC (permalink / raw)
To: Manish Awasthi; +Cc: linux-raid
In-Reply-To: <54758B3B.5080907@caviumnetworks.com>
[-- Attachment #1: Type: text/plain, Size: 4767 bytes --]
On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
<manish.awasthi@caviumnetworks.com> wrote:
>
> On 11/25/2014 08:07 AM, NeilBrown wrote:
> > On Mon, 24 Nov 2014 13:40:06 +0530 Manish Awasthi
> > <manish.awasthi@caviumnetworks.com> wrote:
> >
> >> Hi,
> >>
> >> We benchmarked the md raid driver performance on 3-18-rc3 kernel and
> >> compared the results with that of 3.6.11. The reason for this exercise
> >> is to understand if multithreaded raid driver has any performance
> >> benefits over 3.6.11 which is single threaded. Here are some details
> >> about the setup
> > Thanks for doing this!!!! I love it when people report test results.
> >
> >
> >> System: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz 4 cores (8threads),
> >> 8GB RAM.
> >> Setup: 3 SSDs create a raid5 array
> >> test tool: iozone (only read/re-read, write/re-write tested), blocksize:
> >> 4k-64k, filesize: 1Gig to 200Gig
> >>
> >> Comparison was done for speed of data transfer in kBytes/sec and also
> >> the CPU utilization as reported by iozone.
> >>
> >> raid on 3.18.0-rc3 performed much worse than raid on 3.6.11.
> >>
> >> Read/Write: raid on 3.18.0-rc3 operated at almost half the speed of raid
> >> on 3.6.11
> > That really isn't very good.... Can you try some of the kernels in between
> > and see if there was a single point where performance dropped, or if there
> > were several steps?
> Can you give me some starting point when multithread support for raid
> was added. It might be a good starting point. For instance, I have a
> benchmark on 3.6.11, now I'd like to go to the first kernel that has
> support for multithread raid and then take it from there.
Multi-thread support appeared in 3.12.
> >
> >
> >> CPU Utilization: With md raid on 3.18.0-rc3, the CPU utilization was
> >> less than half of md raid on 3.6.11 on WRITE operations. However, for
> >> READ operations, 3.18-0.rc3 had more CPU utilization than 3.6.11.
> > Can you use "perf" to determine where the extra time is going?
> >
> > perf record
> > run test
> > stop perf
> > perf report
> >
> > or something like that.
> I can do this but as I mentioned below, Its better if I can get to
> understand all the possible tweaks that can be done to get the optimal
> results unless ofcourse you expect 3.18.0 to perform better than that of
> 3.6.11 even if default case without any tweaks.
I have no particular expectations. I like to see concrete measurements and
then try to interpret them.
I prefer to compare default setting (no tweaks) in the first instance.
Because that is what most people will be using.
> >
> >> Also, I noticed that scaling up the CPU cores of the system scales down
> >> the raid througput with 3.18.0-rc3.
> > This is by writing numbers to "group_thread_cnt" ??? Can you provide a simple
> > table comparing thread count to throughput? Or maybe a graph. I love
> > graphs :-)
> I did not tweak anything on the 3.18.0 kernel. I assumed all the
> required support is built-in and did not bother to go into the depth of
> the code as we're still in nascent stages where we are comparing the
> data on specific kernel versions. Can you point me to some text that can
> describe tweaks like "group_thread_cnt" etc?
Multi-threading is disabled by default, so if you haven't explicitly enabled
it, then it cannot be affecting your performance.
If you
echo 8 > /sys/block/mdXXX/md/group_thread_cnt
it will use 8 thread to perform 'xor' calculations and submit IO requests.
> >
> >> I do have detailed logs of the comparison but I'm not sure I should send
> >> those on this mailing list.
> > A few megabytes? Yes. 100Meg? No.
>
> Whatever data I have on comparison is attached, I have consolidated this
> from log files to excel. See if this helps.
Thanks. I'll have a look at the tables and see if anything looks interesting.
Thanks,
NeilBrown
> >
> > If you could put them on a website somewhere that I can browse or download
> > I'll try to have a look.
> >
> >> If my observation aligns with someone else's, then what is really the
> >> gain with multithreaded raid.
> > Some testing shows real improvements. Obviously we cannot test everything
> > and I'm very glad to have extra testing from other people.
> > If we can quantify the regressions and confirm exactly when they occurred, we
> > can start looking for a solution.
> >
> > Thanks a lot!
> >
> > NeilBrown
> >
> >
> >> Manish
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: LVM RAID1 syncing component
From: Joe Lawrence @ 2014-12-02 19:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141202084111.6514ec79@notabene.brown>
On Tue, 2 Dec 2014 08:41:11 +1100
NeilBrown <neilb@suse.de> wrote:
> On Mon, 1 Dec 2014 16:19:47 -0500 Joe Lawrence <joe.lawrence@stratus.com>
> wrote:
>
> > On Thu, 27 Nov 2014 07:41:58 +1100
> > NeilBrown <neilb@suse.de> wrote:
> >
> > > On Mon, 24 Nov 2014 23:07:32 -0500 Joe Lawrence <joe.lawrence@stratus.com>
> > > wrote:
> > >
> > > > Does anyone know how its possible to determine which side of an LVM RAID 1
> > > > is the stale partner during RAID resync?
> > > >
> > > > In ordinary MD RAID, I believe you can check
> > > > /sys/block/md0/md/dev-XXX/state,
> > >
> > > Why do you believe that?
> > >
> > > During a resync (after an unclean shutdown) the devices are indistinguishable.
> > > RAID1 reads all drives and if there is a difference it chooses one data block
> > > to write to the others - always the one with the lowest index number.
> > >
> > > So with md or LVM it is the same: first "first" is "copied" to the "second".
> >
> > Hi Neil,
> >
> > Here's a quick example of my thought-process, where md2 is an in-sync
> > RAID1 of sdq2 and sdr2 with an internal write bitmap:
> >
> > % mdadm --fail /dev/md3 /dev/sdr2
> > % mdadm --remove /dev/md3 /dev/sdr2
>
> You are referring to what I would call "recovery", not "resync"
> (which is why I put "(after an unclean shutdown)" in my answer to make it
> clear what circumstances I was talking about).
>
> resync: fixing things after an unclean shutdown
> recovery: restoring data after a device has been removed an another
> (or possibly the same) added.
>
> I think
>
> dmsetup info
>
> should provide the info you want.
> One of the fields is a sequence of letters 'D', 'a', 'A'.
>
> * Status characters:
> * 'D' = Dead/Failed device
> * 'a' = Alive but not in-sync
> * 'A' = Alive and in-sync
>
> Does that provide the information you wanted?
Yes! When I add a disk back to the array, I see the status characters
you mentioned during _recovery_:
% while [ true ]
do
dmsetup status vg0-lvraid0
sleep 10s
done
0 18857984 raid raid1 2 DA 18857984/18857984 idle 0
0 18857984 raid raid1 2 DA 18857984/18857984 idle 0
0 18857984 raid raid1 2 DA 18857984/18857984 idle 0
0 18857984 raid raid1 2 aA 0/18857984 recover 0
0 18857984 raid raid1 2 aA 0/18857984 recover 0
0 18857984 raid raid1 2 aA 256/18857984 recover 0
0 18857984 raid raid1 2 aA 8519680/18857984 recover 0
0 18857984 raid raid1 2 AA 18857984/18857984 idle 0
So now, determining which disk is which in the raid_set. Can I use a
command like lvs to tie the n-th character status back to a device?
% lvs -a -o name,devices vg0
LV Devices
lvraid0 lvraid0_rimage_0(0),lvraid0_rimage_1(0)
[lvraid0_rimage_0] /dev/sdr1(1)
[lvraid0_rimage_1] /dev/sdt1(1)
[lvraid0_rmeta_0] /dev/sdr1(0)
[lvraid0_rmeta_1] /dev/sdt1(0)
Where the first character represents [lvraid0_rimage_0] and the second
[lvraid0_rimage_1].
Thanks,
-- Joe
^ permalink raw reply
* Re: The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: Mike Snitzer @ 2014-12-02 14:00 UTC (permalink / raw)
To: zhangxiao; +Cc: agk, linux-kernel, NeilBrown, linux-raid, dm-devel
In-Reply-To: <547D7BC6.6070708@windriver.com>
On Tue, Dec 02 2014 at 3:43am -0500,
zhangxiao <xiao.zhang@windriver.com> wrote:
> Hi Experts,
>
> See drivers/md/dm-bufio.c as below:
> 50 /*
> 51 * Buffer hash
> 52 */
> 53 #define DM_BUFIO_HASH_BITS 20
> 54 #define DM_BUFIO_HASH(block) \
> 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
> 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
> "drivers/md/dm-bufio.c" 1854L, 45375C
>
> DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
> use case with limited resources. Do we have any plan to make it more
> flexible? Like module parameter or something else?
No immediate plans.
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-12-02 11:56 UTC (permalink / raw)
To: NeilBrown; +Cc: Mdadm
In-Reply-To: <20141202084611.45f56d6a@notabene.brown>
It works! (Atleast on a sample 5 MB device with 5 x 1MB partitions :-)
will find more space on my drives and do a larger test but don't see
why it shouldn't work)
Here are the following caveats (and questions):
- Neil, like you pointed out, the power of 2 chunk size will probably
need a code change (in the kernel or only in the userspace tool?)
- Any performance or other reasons why a terabyte size chunk may
not be feasible?
- Implications of safe_mode_delay
- Would the metadata be updated on the block device be written to
and the parity device as well?
- If the drive fails which is the same as the drive being written
to, would that lack of metadata updates to the other devices affect
reconstruction?
- Adding new devices (is it possible to move the parity to the disk
being added? How does device addition work for RAID4 ...is it added as
a zero-ed out device with parity disk remaining the same)
On 2 December 2014 at 03:16, NeilBrown <neilb@suse.de> wrote:
> On Mon, 1 Dec 2014 22:04:42 +0530 Anshuman Aggarwal
> <anshuman.aggarwal@gmail.com> wrote:
>
>> On 1 December 2014 at 21:30, Anshuman Aggarwal
>> <anshuman.aggarwal@gmail.com> wrote:
>> > On 26 November 2014 at 11:54, Anshuman Aggarwal
>> > <anshuman.aggarwal@gmail.com> wrote:
>> >> On 25 November 2014 at 04:20, NeilBrown <neilb@suse.de> wrote:
>> >>> On Mon, 24 Nov 2014 12:59:47 +0530 Anshuman Aggarwal
>> >>> <anshuman.aggarwal@gmail.com> wrote:
>> >>>
>> >>>> On 3 November 2014 at 11:22, NeilBrown <neilb@suse.de> wrote:
>> >>>> > On Thu, 30 Oct 2014 20:30:40 +0530 Anshuman Aggarwal
>> >>>> > <anshuman.aggarwal@gmail.com> wrote:
>> >>>> >
>> >>>> >> Would chunksize==disksize work? Wouldn't that lead to the entire
>> >>>> >> parity be invalidated for any write to any of the disks (assuming md
>> >>>> >> operates at a chunk level)...also please see my reply below
>> >>>> >
>> >>>> > Operating at a chunk level would be a very poor design choice. md/raid5
>> >>>> > operates in units of 1 page (4K).
>> >>>>
>> >>>> It appears that my requirement may be met by a partitionable md raid 4
>> >>>> array where the partitions are all on individual underlying block
>> >>>> devices not striped across the block devices. Is that currently
>> >>>> possible with md raid? I dont' see how but such an enhancement could
>> >>>> do all that I had outlined earlier
>> >>>>
>> >>>> Is this possible to implement using RAID4 and MD already?
>> >>>
>> >>> Nearly. RAID4 currently requires the chunk size to be a power of 2.
>> >>> Rounding down the size of your drives to match that could waste nearly half
>> >>> the space. However it should work as a proof-of-concept.
>> >>>
>> >>> RAID0 supports non-power-of-2 chunk sizes. Doing the same thing for
>> >>> RAID4/5/6 would be quite possible.
>> >>>
>> >>>> can the
>> >>>> partitions be made to write to individual block devices such that
>> >>>> parity updates don't require reading all devices?
>> >>>
>> >>> md/raid4 will currently tries to minimize total IO requests when performing
>> >>> an update, but prefer spreading the IO over more devices if the total number
>> >>> of requests is the same.
>> >>>
>> >>> So for a 4-drive RAID4, Updating a single block can be done by:
>> >>> read old data block, read parity, write data, write parity - 4 IO requests
>> >>> or
>> >>> read other 2 data blocks, write data, write parity - 4 IO requests.
>> >>>
>> >>> In this case it will prefer the second, which is not what you want.
>> >>> With 5-drive RAID4, the second option will require 5 IO requests, so the first
>> >>> will be chosen.
>> >>> It is quite trivial to flip this default for testing
>> >>>
>> >>> - if (rmw < rcw && rmw > 0) {
>> >>> + if (rmw <= rcw && rmw > 0) {
>> >>>
>> >>>
>> >>> If you had 5 drives, you could experiment with no code changes.
>> >>> Make the chunk size the largest power of 2 that fits in the device, and then
>> >>> partition to align the partitions on those boundaries.
>> >>
>> >> If the chunk size is almost the same as the device size, I assume the
>> >> entire chunk is not invalidated for parity on writing to a single
>> >> block? i.e. if only 1 block is updated only that blocks parity will be
>> >> read and written and not for the whole chunk? If thats' the case, what
>> >> purpose does a chunk serve in md raid ? If that's not the case, it
>> >> wouldn't work because a single block updation would lead to parity
>> >> being written for the entire chunk, which is the size of the device
>> >>
>> >> I do have more than 5 drives though they are in use currently. I will
>> >> create a small testing partition on each device of the same size and
>> >> run the test on that after ensuring that the drives do go to sleep.
>> >>
>> >>>
>> >>> NeilBrown
>> >>>
>> >
>> > Wouldn't the meta data writes wake up all the disks in the cluster
>> > anyways (defeating the purpose)? This idea will require metadata to
>> > not be written out to each device (is that even possible or on the
>> > cards?)
>> >
>> > I am about to try out your suggestion with the chunk sizes anyways but
>> > thought about the metadata being a major stumbling block.
>> >
>>
>> And it seems to be confirmed that the metadata write is waking up the
>> other drives. On any write to a particular drive the metadata update
>> is accessing all the others.
>>
>> Am I correct in assuming that all metadata is currently written as
>> part of the block device itself and that the external metadata is
>> still embedded in each of the block devices (only the format of the
>> metadata is defined externally?) I guess to implement this we would
>> need to store metadata elsewhere which may be a major development
>> work. Still that may be a flexibility desired in md raid for other
>> reasons...
>>
>> Neil, your thoughts.
>
> This is exactly why I suggested testing with existing code and seeing how far
> you can get. Thanks.
>
> For a full solution we probably do need some code changes here, but for
> further testing you could:
> 1/ make sure there is no bitmap (mdadm --grow --bitmap=none)
> 2/ set the safe_mode_delay to 0
> echo 0 > /sys/block/mdXXX/md/safe_mode_delay
>
> when it won't try to update the metadata until you stop the array, or a
> device fails.
>
> Longer term: it would probably be good to only update the bitmap on the
> devices that are being written to - and to merge all bitmaps when assembling
> the array. Also when there is a bitmap, the safe_mode functionality should
> probably be disabled.
>
> NeilBrown
>
^ permalink raw reply
* The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: zhangxiao @ 2014-12-02 8:43 UTC (permalink / raw)
To: agk, snitzer, linux-kernel, NeilBrown, linux-raid, dm-devel
Hi Experts,
See drivers/md/dm-bufio.c as below:
50 /*
51 * Buffer hash
52 */
53 #define DM_BUFIO_HASH_BITS 20
54 #define DM_BUFIO_HASH(block) \
55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
56 ((1 << DM_BUFIO_HASH_BITS) - 1))
"drivers/md/dm-bufio.c" 1854L, 45375C
DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
use case with limited resources. Do we have any plan to make it more
flexible? Like module parameter or something else?
Thanks
Xiao
^ permalink raw reply
* Re: The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: zhangxiao @ 2014-12-02 8:40 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, linux-kernel, dm-devel
In-Reply-To: <20141202193618.6d3adb22@notabene.brown>
On 2014年12月02日 16:36, NeilBrown wrote:
> On Tue, 2 Dec 2014 16:32:14 +0800 zhangxiao <xiao.zhang@windriver.com> wrote:
>
>> Hi NeilBrown,
>>
>> See drivers/md/dm-bufio.c as below:
>> 50 /*
>> 51 * Buffer hash
>> 52 */
>> 53 #define DM_BUFIO_HASH_BITS 20
>> 54 #define DM_BUFIO_HASH(block) \
>> 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
>> 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
>> "drivers/md/dm-bufio.c" 1854L, 45375C
>>
>>
>> DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
>> use case with limited resources. Do we have any plan to make it more
>> flexible? Like module parameter or something else?
>>
>
> Thanks, but this question should be address to:
>
Thank you for your kindly remind. :-)
Regards
Xiao
> DEVICE-MAPPER (LVM)
> M: Alasdair Kergon <agk@redhat.com>
> M: Mike Snitzer <snitzer@redhat.com>
> M: dm-devel@redhat.com
> L: dm-devel@redhat.com
> W: http://sources.redhat.com/dm
> Q: http://patchwork.kernel.org/project/dm-devel/list/
> T: git git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
> T: quilt http://people.redhat.com/agk/patches/linux/editing/
> S: Maintained
> F: Documentation/device-mapper/
> F: drivers/md/dm*
> F: drivers/md/persistent-data/
> F: include/linux/device-mapper.h
> F: include/linux/dm-*.h
> F: include/uapi/linux/dm-*.h
>
> NeilBrown
>
^ permalink raw reply
* Re: The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: NeilBrown @ 2014-12-02 8:36 UTC (permalink / raw)
To: zhangxiao; +Cc: linux-raid, linux-kernel, dm-devel
In-Reply-To: <547D790E.3070803@windriver.com>
[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]
On Tue, 2 Dec 2014 16:32:14 +0800 zhangxiao <xiao.zhang@windriver.com> wrote:
> Hi NeilBrown,
>
> See drivers/md/dm-bufio.c as below:
> 50 /*
> 51 * Buffer hash
> 52 */
> 53 #define DM_BUFIO_HASH_BITS 20
> 54 #define DM_BUFIO_HASH(block) \
> 55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
> 56 ((1 << DM_BUFIO_HASH_BITS) - 1))
> "drivers/md/dm-bufio.c" 1854L, 45375C
>
>
> DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
> use case with limited resources. Do we have any plan to make it more
> flexible? Like module parameter or something else?
>
Thanks, but this question should be address to:
DEVICE-MAPPER (LVM)
M: Alasdair Kergon <agk@redhat.com>
M: Mike Snitzer <snitzer@redhat.com>
M: dm-devel@redhat.com
L: dm-devel@redhat.com
W: http://sources.redhat.com/dm
Q: http://patchwork.kernel.org/project/dm-devel/list/
T: git git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
T: quilt http://people.redhat.com/agk/patches/linux/editing/
S: Maintained
F: Documentation/device-mapper/
F: drivers/md/dm*
F: drivers/md/persistent-data/
F: include/linux/device-mapper.h
F: include/linux/dm-*.h
F: include/uapi/linux/dm-*.h
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* The DM_BUFIO_HASH_BITS is defined as 20, any plan to make it flexible?
From: zhangxiao @ 2014-12-02 8:32 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel
Hi NeilBrown,
See drivers/md/dm-bufio.c as below:
50 /*
51 * Buffer hash
52 */
53 #define DM_BUFIO_HASH_BITS 20
54 #define DM_BUFIO_HASH(block) \
55 ((((block) >> DM_BUFIO_HASH_BITS) ^ (block)) & \
56 ((1 << DM_BUFIO_HASH_BITS) - 1))
"drivers/md/dm-bufio.c" 1854L, 45375C
DM_BUFIO_HASH_BITS was defined as 20 and it maybe not fit for some
use case with limited resources. Do we have any plan to make it more
flexible? Like module parameter or something else?
Thanks
Xiao
^ permalink raw reply
* Re: LVM RAID1 syncing component
From: NeilBrown @ 2014-12-01 23:28 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
In-Reply-To: <21625.58812.321750.229562@tree.ty.sabi.co.uk>
[-- Attachment #1: Type: text/plain, Size: 1952 bytes --]
On Sat, 29 Nov 2014 15:26:52 +0000 pg@lxra2.for.sabi.co.UK (Peter Grandi)
wrote:
> [ ... ]
>
> > During a resync (after an unclean shutdown) the devices are
> > indistinguishable. RAID1 reads all drives and if there is a
> > difference it chooses one data block to write to the others -
> > always the one with the lowest index number.
>
> Uhhhhh "indistinguishable" and "lowest index number"?
>
> Shouldn't that be "lowest index number among those with the
> highest event count"?
>
> Put another way, couldn't it happen that in a 5-way RAID1 for
> example an unclean shutdown results in 2 drives with the same
> highest event count and 3 drives with lower event counts, and
> then the data page to write is that from the one of the 2 with
> the lowest index number and is written only to the 3 with the
> lower event count?
>
> Also, in case of an «unclean shutdown» resulting in all members
> of a RAID1 set having the same event count, is the resync still
> done? Is it necessary? Or is «unclean shutdown» used here as an
> alias for "not all event counts are the same".
>
> I am asking as to what RAID1 actually does mostly, but also
> perhaps as to what it ought to be doing.
I've told you what it actually does. I think that is what it ought to do.
If you think that maybe it ought to do something differently from what it
does, I suggest you try to come up with a specific scenario where what
actually happens is not optimal, and give clear reasons for why you think
something else is optimal.
To be specific, after an unclean shutdown, the array is assembled from all
devices which have an uptodate event count, and then all blocks are compared
and where a difference is found, data is copies from the lowest index number
block to the others.
"uptodate" in the context of event counts means the event count is equal to,
or one less than, the highest event count found.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 5/8 v2] dm: replace memset by memzero_explicit
From: Mike Snitzer @ 2014-12-01 23:00 UTC (permalink / raw)
To: Milan Broz
Cc: kernel-janitors, linux-kernel, linux-raid, Julia Lawall, dm-devel,
Alasdair Kergon
In-Reply-To: <547CD7B9.5030000@gmail.com>
On Mon, Dec 01 2014 at 4:03pm -0500,
Milan Broz <gmazyland@gmail.com> wrote:
> On 11/30/2014 06:03 PM, Julia Lawall wrote:
> > From: Julia Lawall <Julia.Lawall@lip6.fr>
> >
> > Memset on a local variable may be removed when it is called just before the
> > variable goes out of scope. Using memzero_explicit defeats this
> > optimization. A simplified version of the semantic patch that makes this
> > change is as follows: (http://coccinelle.lip6.fr/)
>
> Ack, but I submitted the same patch a week ago....
>
> https://www.redhat.com/archives/dm-devel/2014-November/msg00084.html
>
> Mike, please could you add this to linux-next tree or you want this
> to go through Herbert's tree?
> (I do not think it is good idea for DM patches.)
I staged it in linux-next for 3.19 inclusion, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.19&id=a722201f9075b28be140bc13f8ec07bf6a42edd4
^ permalink raw reply
* Re: [PATCH] Grow: Fix wrong 'goto' in set_new_data_offset
From: NeilBrown @ 2014-12-01 22:52 UTC (permalink / raw)
To: Pawel Baldysiak; +Cc: linux-raid, artur.paszkiewicz
In-Reply-To: <20141127113524.8874.37994.stgit@gklab-154-222.intel.com>
[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]
On Thu, 27 Nov 2014 12:35:24 +0100 Pawel Baldysiak
<pawel.baldysiak@intel.com> wrote:
> Commit a821c95f114724b38df1ea99b2858178e0ed28ce
> besides introducing additional message, also changed
> direct return to "goto" instruction.
> 'goto release' will cause routine to return with '-1',
> when previously '1' was returned.
> Described behaviour breaks e.g. IMSM reshape process.
> This patch fixes this issue by changing 'goto' to proper one -
> the one that returns '1'.
>
> Signed-off-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
> ---
> Grow.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Grow.c b/Grow.c
> index a0f7526..6ff225a 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -2257,7 +2257,7 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,
> /* Metadata doesn't support data_offset changes */
> pr_err("%s: Metadata version doesn't support"
> " data_offset changes\n", devname);
> - goto release;
> + goto fallback;
> }
> if (before > info2.space_before)
> before = info2.space_before;
Applied, thanks.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: NeilBrown @ 2014-12-01 21:46 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Mdadm
In-Reply-To: <CAK-d5dZMjbhAqgs9GOvswG6crZtrV2uSidRzt_hBJAH=m5rGKw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5522 bytes --]
On Mon, 1 Dec 2014 22:04:42 +0530 Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> On 1 December 2014 at 21:30, Anshuman Aggarwal
> <anshuman.aggarwal@gmail.com> wrote:
> > On 26 November 2014 at 11:54, Anshuman Aggarwal
> > <anshuman.aggarwal@gmail.com> wrote:
> >> On 25 November 2014 at 04:20, NeilBrown <neilb@suse.de> wrote:
> >>> On Mon, 24 Nov 2014 12:59:47 +0530 Anshuman Aggarwal
> >>> <anshuman.aggarwal@gmail.com> wrote:
> >>>
> >>>> On 3 November 2014 at 11:22, NeilBrown <neilb@suse.de> wrote:
> >>>> > On Thu, 30 Oct 2014 20:30:40 +0530 Anshuman Aggarwal
> >>>> > <anshuman.aggarwal@gmail.com> wrote:
> >>>> >
> >>>> >> Would chunksize==disksize work? Wouldn't that lead to the entire
> >>>> >> parity be invalidated for any write to any of the disks (assuming md
> >>>> >> operates at a chunk level)...also please see my reply below
> >>>> >
> >>>> > Operating at a chunk level would be a very poor design choice. md/raid5
> >>>> > operates in units of 1 page (4K).
> >>>>
> >>>> It appears that my requirement may be met by a partitionable md raid 4
> >>>> array where the partitions are all on individual underlying block
> >>>> devices not striped across the block devices. Is that currently
> >>>> possible with md raid? I dont' see how but such an enhancement could
> >>>> do all that I had outlined earlier
> >>>>
> >>>> Is this possible to implement using RAID4 and MD already?
> >>>
> >>> Nearly. RAID4 currently requires the chunk size to be a power of 2.
> >>> Rounding down the size of your drives to match that could waste nearly half
> >>> the space. However it should work as a proof-of-concept.
> >>>
> >>> RAID0 supports non-power-of-2 chunk sizes. Doing the same thing for
> >>> RAID4/5/6 would be quite possible.
> >>>
> >>>> can the
> >>>> partitions be made to write to individual block devices such that
> >>>> parity updates don't require reading all devices?
> >>>
> >>> md/raid4 will currently tries to minimize total IO requests when performing
> >>> an update, but prefer spreading the IO over more devices if the total number
> >>> of requests is the same.
> >>>
> >>> So for a 4-drive RAID4, Updating a single block can be done by:
> >>> read old data block, read parity, write data, write parity - 4 IO requests
> >>> or
> >>> read other 2 data blocks, write data, write parity - 4 IO requests.
> >>>
> >>> In this case it will prefer the second, which is not what you want.
> >>> With 5-drive RAID4, the second option will require 5 IO requests, so the first
> >>> will be chosen.
> >>> It is quite trivial to flip this default for testing
> >>>
> >>> - if (rmw < rcw && rmw > 0) {
> >>> + if (rmw <= rcw && rmw > 0) {
> >>>
> >>>
> >>> If you had 5 drives, you could experiment with no code changes.
> >>> Make the chunk size the largest power of 2 that fits in the device, and then
> >>> partition to align the partitions on those boundaries.
> >>
> >> If the chunk size is almost the same as the device size, I assume the
> >> entire chunk is not invalidated for parity on writing to a single
> >> block? i.e. if only 1 block is updated only that blocks parity will be
> >> read and written and not for the whole chunk? If thats' the case, what
> >> purpose does a chunk serve in md raid ? If that's not the case, it
> >> wouldn't work because a single block updation would lead to parity
> >> being written for the entire chunk, which is the size of the device
> >>
> >> I do have more than 5 drives though they are in use currently. I will
> >> create a small testing partition on each device of the same size and
> >> run the test on that after ensuring that the drives do go to sleep.
> >>
> >>>
> >>> NeilBrown
> >>>
> >
> > Wouldn't the meta data writes wake up all the disks in the cluster
> > anyways (defeating the purpose)? This idea will require metadata to
> > not be written out to each device (is that even possible or on the
> > cards?)
> >
> > I am about to try out your suggestion with the chunk sizes anyways but
> > thought about the metadata being a major stumbling block.
> >
>
> And it seems to be confirmed that the metadata write is waking up the
> other drives. On any write to a particular drive the metadata update
> is accessing all the others.
>
> Am I correct in assuming that all metadata is currently written as
> part of the block device itself and that the external metadata is
> still embedded in each of the block devices (only the format of the
> metadata is defined externally?) I guess to implement this we would
> need to store metadata elsewhere which may be a major development
> work. Still that may be a flexibility desired in md raid for other
> reasons...
>
> Neil, your thoughts.
This is exactly why I suggested testing with existing code and seeing how far
you can get. Thanks.
For a full solution we probably do need some code changes here, but for
further testing you could:
1/ make sure there is no bitmap (mdadm --grow --bitmap=none)
2/ set the safe_mode_delay to 0
echo 0 > /sys/block/mdXXX/md/safe_mode_delay
when it won't try to update the metadata until you stop the array, or a
device fails.
Longer term: it would probably be good to only update the bitmap on the
devices that are being written to - and to merge all bitmaps when assembling
the array. Also when there is a bitmap, the safe_mode functionality should
probably be disabled.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: LVM RAID1 syncing component
From: NeilBrown @ 2014-12-01 21:41 UTC (permalink / raw)
To: Joe Lawrence; +Cc: linux-raid
In-Reply-To: <20141201161947.1d503df9@jlaw-desktop.mno.stratus.com>
[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]
On Mon, 1 Dec 2014 16:19:47 -0500 Joe Lawrence <joe.lawrence@stratus.com>
wrote:
> On Thu, 27 Nov 2014 07:41:58 +1100
> NeilBrown <neilb@suse.de> wrote:
>
> > On Mon, 24 Nov 2014 23:07:32 -0500 Joe Lawrence <joe.lawrence@stratus.com>
> > wrote:
> >
> > > Does anyone know how its possible to determine which side of an LVM RAID 1
> > > is the stale partner during RAID resync?
> > >
> > > In ordinary MD RAID, I believe you can check
> > > /sys/block/md0/md/dev-XXX/state,
> >
> > Why do you believe that?
> >
> > During a resync (after an unclean shutdown) the devices are indistinguishable.
> > RAID1 reads all drives and if there is a difference it chooses one data block
> > to write to the others - always the one with the lowest index number.
> >
> > So with md or LVM it is the same: first "first" is "copied" to the "second".
>
> Hi Neil,
>
> Here's a quick example of my thought-process, where md2 is an in-sync
> RAID1 of sdq2 and sdr2 with an internal write bitmap:
>
> % mdadm --fail /dev/md3 /dev/sdr2
> % mdadm --remove /dev/md3 /dev/sdr2
You are referring to what I would call "recovery", not "resync"
(which is why I put "(after an unclean shutdown)" in my answer to make it
clear what circumstances I was talking about).
resync: fixing things after an unclean shutdown
recovery: restoring data after a device has been removed an another
(or possibly the same) added.
I think
dmsetup info
should provide the info you want.
One of the fields is a sequence of letters 'D', 'a', 'A'.
* Status characters:
* 'D' = Dead/Failed device
* 'a' = Alive but not in-sync
* 'A' = Alive and in-sync
Does that provide the information you wanted?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: LVM RAID1 syncing component
From: Joe Lawrence @ 2014-12-01 21:27 UTC (permalink / raw)
To: Joe Lawrence; +Cc: NeilBrown, linux-raid
In-Reply-To: <20141201161947.1d503df9@jlaw-desktop.mno.stratus.com>
On Mon, 1 Dec 2014 16:19:47 -0500
Joe Lawrence <joe.lawrence@stratus.com> wrote:
> On Thu, 27 Nov 2014 07:41:58 +1100
> NeilBrown <neilb@suse.de> wrote:
>
> > On Mon, 24 Nov 2014 23:07:32 -0500 Joe Lawrence <joe.lawrence@stratus.com>
> > wrote:
> >
> > > Does anyone know how its possible to determine which side of an LVM RAID 1
> > > is the stale partner during RAID resync?
> > >
> > > In ordinary MD RAID, I believe you can check
> > > /sys/block/md0/md/dev-XXX/state,
> >
> > Why do you believe that?
> >
> > During a resync (after an unclean shutdown) the devices are indistinguishable.
> > RAID1 reads all drives and if there is a difference it chooses one data block
> > to write to the others - always the one with the lowest index number.
> >
> > So with md or LVM it is the same: first "first" is "copied" to the "second".
>
> Hi Neil,
>
> Here's a quick example of my thought-process, where md2 is an in-sync
> RAID1 of sdq2 and sdr2 with an internal write bitmap:
>
> % mdadm --fail /dev/md3 /dev/sdr2
> % mdadm --remove /dev/md3 /dev/sdr2
>
> [ ... File I/O to /dev/md3 ... ]
>
> % mdadm -X /dev/sd[qr]2
> Filename : /dev/sdq2
> Magic : 6d746962
> Version : 4
> UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
> Events : 8526 << ECq
> Events Cleared : 8498
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 16768896 (15.99 GiB 17.17 GB)
> Bitmap : 256 bits (chunks), 5 dirty (2.0%)
> Filename : /dev/sdr2
> Magic : 6d746962
> Version : 4
> UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
> Events : 8513 << ECr
> Events Cleared : 8498
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 16768896 (15.99 GiB 17.17 GB)
> Bitmap : 256 bits (chunks), 5 dirty (2.0%)
>
> [ Note that ECq > ECr, which makes sense since sdq was the remaining
> disk standing in the RAID. ]
>
> % mdadm --add /dev/md3 /dev/sdr2
> % mdadm --detail /dev/md3
> /dev/md3:
> Version : 1.2
> Creation Time : Thu Nov 13 15:47:19 2014
> Raid Level : raid1
> Array Size : 16768896 (15.99 GiB 17.17 GB)
> Used Dev Size : 16768896 (15.99 GiB 17.17 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Dec 1 16:07:55 2014
> State : active, degraded, recovering
> Active Devices : 1
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 1
>
> Rebuild Status : 0% complete
>
> Name : dhcp-linux-2192-2025:3
> UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
> Events : 8528
>
> Number Major Minor RaidDevice State
> 0 65 2 0 active sync /dev/sdq2
> 1 65 18 1 spare rebuilding /dev/sdr2
>
> % head /sys/block/md3/md/dev-sd*/state
> ==> /sys/block/md3/md/dev-sdq2/state <==
> in_sync
>
> ==> /sys/block/md3/md/dev-sdr2/state <==
> spare
>
> In this scenario, sdr was re-added to the RAID and with a lower events-
> cleared count. I assume that MD will only need to read the data
> represented by the dirty bitmap bits from the "active sync" disk to the
> "spare rebuilding" disk. Is this not the case?
D'oh! Sorry for the edit, I meant the "events" count and not the "events
cleared" count to determine sync direction. I put the "<<" arrows in
the right place but then used the wrong term everywhere else.
-- Joe
^ permalink raw reply
* Re: LVM RAID1 syncing component
From: Joe Lawrence @ 2014-12-01 21:19 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141127074158.2a3472f3@notabene.brown>
On Thu, 27 Nov 2014 07:41:58 +1100
NeilBrown <neilb@suse.de> wrote:
> On Mon, 24 Nov 2014 23:07:32 -0500 Joe Lawrence <joe.lawrence@stratus.com>
> wrote:
>
> > Does anyone know how its possible to determine which side of an LVM RAID 1
> > is the stale partner during RAID resync?
> >
> > In ordinary MD RAID, I believe you can check
> > /sys/block/md0/md/dev-XXX/state,
>
> Why do you believe that?
>
> During a resync (after an unclean shutdown) the devices are indistinguishable.
> RAID1 reads all drives and if there is a difference it chooses one data block
> to write to the others - always the one with the lowest index number.
>
> So with md or LVM it is the same: first "first" is "copied" to the "second".
Hi Neil,
Here's a quick example of my thought-process, where md2 is an in-sync
RAID1 of sdq2 and sdr2 with an internal write bitmap:
% mdadm --fail /dev/md3 /dev/sdr2
% mdadm --remove /dev/md3 /dev/sdr2
[ ... File I/O to /dev/md3 ... ]
% mdadm -X /dev/sd[qr]2
Filename : /dev/sdq2
Magic : 6d746962
Version : 4
UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
Events : 8526 << ECq
Events Cleared : 8498
State : OK
Chunksize : 64 MB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 16768896 (15.99 GiB 17.17 GB)
Bitmap : 256 bits (chunks), 5 dirty (2.0%)
Filename : /dev/sdr2
Magic : 6d746962
Version : 4
UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
Events : 8513 << ECr
Events Cleared : 8498
State : OK
Chunksize : 64 MB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 16768896 (15.99 GiB 17.17 GB)
Bitmap : 256 bits (chunks), 5 dirty (2.0%)
[ Note that ECq > ECr, which makes sense since sdq was the remaining
disk standing in the RAID. ]
% mdadm --add /dev/md3 /dev/sdr2
% mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Thu Nov 13 15:47:19 2014
Raid Level : raid1
Array Size : 16768896 (15.99 GiB 17.17 GB)
Used Dev Size : 16768896 (15.99 GiB 17.17 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Dec 1 16:07:55 2014
State : active, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Rebuild Status : 0% complete
Name : dhcp-linux-2192-2025:3
UUID : 073511ee:0b0c20e0:662ae8da:b53c7979
Events : 8528
Number Major Minor RaidDevice State
0 65 2 0 active sync /dev/sdq2
1 65 18 1 spare rebuilding /dev/sdr2
% head /sys/block/md3/md/dev-sd*/state
==> /sys/block/md3/md/dev-sdq2/state <==
in_sync
==> /sys/block/md3/md/dev-sdr2/state <==
spare
In this scenario, sdr was re-added to the RAID and with a lower events-
cleared count. I assume that MD will only need to read the data
represented by the dirty bitmap bits from the "active sync" disk to the
"spare rebuilding" disk. Is this not the case?
Regards,
-- Joe
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox