* Re: Is it safe to shutdown while mdadm --grow is in progress.
From: NeilBrown @ 2014-10-12 22:15 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <543AA5B6.6010200@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]
On Sun, 12 Oct 2014 11:00:54 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> Thanks. I appreciate you taking the time to respond. I would like to
> forward this to those that maintain faq/wiki so that some one searches
> gets the info and does not have to bother you. Is that OK?
Sure - anything on a public mailing list is public.
>
> I normally would go for backup before rebuild, but 15TB disk space is
> too much to come up with, without serious $$ and need. Besides, these
> are movies/recordings for htpc and the data is available on
> dvds/internet. So there is inherent lack of importance. In other words,
> I recommend backup as most do, but it looks like every one has some
> justification not to do, and want some one else to assure them that it
> is ok :-)
Quite understandable...
NeilBrown
>
> Thanks and Regards
> Ramesh
>
> On 10/12/2014 03:35 AM, NeilBrown wrote:
> > On Sun, 12 Oct 2014 00:46:14 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> >
> >> I googled this topic and got differing answers. Some old ones report
> >> horror stories and others suggest rebuild continues after reboot
> >> normally. My array is rebuilding and it will take about 6 days to
> >> complete. I am expecting bad weather in a couple of days and would like
> >> to proactively shutdown the machine and reboot. Is this safe?
> >>
> >> If the information is already available in a clear manner, please let me
> >> know.
> >>
> >> uname: Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38
> >> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> >> mdadm - v3.2.5 - 18th May 2012
> >>
> >> Thanks for your help
> >>
> > It is certainly designed to be safe and my experience is that it is.
> > However it is difficult to test all corner cases so it is not impossible that
> > someone what quite work right.
> > If something does go wrong:
> > 1/ don't panic
> > 2/ don't try to --create the array to fix it
> > 3/ report all details to this list and I'm 99.9% sure I can get your array
> > running again with all your data safe.
> > But I suspect it won't come to that.
> >
> > NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* "bitmap file is out of date, doing full recovery"
From: Alexander Lyakas @ 2014-10-12 18:03 UTC (permalink / raw)
To: linux-raid; +Cc: Neil Brown
Hi Neil,
after a 2-drive raid1 unclean shutdown (crash actually), after reboot, we had:
md/raid1:md24: not clean -- starting background reconstruction
md/raid1:md24: active with 2 out of 2 mirrors
md24: bitmap file is out of date (41 < 42) -- forcing full recovery
created bitmap (22 pages) for device md24
md24: bitmap file is out of date, doing full recovery
md24: bitmap initialized from disk: read 2 pages, set 44667 of 44667 bits
The superblock of both drives had event count = 42:
(this is a custom mdadm with some added prints):
mdadm: looking for devices for /dev/md24
mdadm: [/dev/md24] /dev/dm-205: slot=0, events=42,
recovery_offset=N/A, resync_offset=0, comp_size=5854539776
mdadm: [/dev/md24] /dev/dm-206: slot=1, events=42,
recovery_offset=N/A, resync_offset=0, comp_size=5854539776
But the bitmap superblock had lower event count, which resulted in a
full resync. Is this an expected scenario in case of a crash?
For example in md_update_sb, first we call
bitmap_update_sb(mddev->bitmap), which synchronously updates the
bitmap, and only afterwards we go ahead and update our superblocks. So
in this case, the bitmap should not have a lower event count. Is there
some other valid scenario, in which the bitmap can remain with a lower
event count?
Thanks,
Alex.
^ permalink raw reply
* Good drive got kicked out - does mdadm assemble across suspend/resume like it does across reboots?
From: Ram Ramesh @ 2014-10-12 17:38 UTC (permalink / raw)
To: Linux Raid
I did a minor booboo with my mdadm raid6. I suspended (not shutdown, but
suspend-to-ram) the machine to install a *new* disk in a hotswap bay for
a thorough test before usage. This machine also housed a mdadm raid6
array with 4 disks /dev/sd[cdeg]1 (not /dev/sdf1). When I inserted the
new disk, I also rearranged the disks in the hotswap cage. I think that
is a stupid mistake because /dev/sdg1 was also in the cage and now
became /dev/sdf1 and the new disk got the name /dev/sdg1. I did not
think much about it, since I thought mdadm would assemble after a resume
and will find the right disks. I did not bother to check if anything
happened to /dev/md0.
After checking the new disk (now /dev/sdg1) I added /dev/sdg1 to md0 (as
spare) without checking /proc/mdstat. I noticed rebuild started to my
surprise. I was trying to figure out what happened. It looks like resume
after suspend does not assemble md0 as it does for reboot. So, it looked
at the new /dev/sdg1 and found to be not part of md0 and degraded the
array. So when I added it, it thought I am adding a new drive and
proceeded to rebuild to fix the degraded state. A quick check of
/dev/sdf1 (which was /dev/sdg1 before my swap) contains everything
right, but got kicked out of array due to suspend-swap disk-resume issue.
While this is not a disaster, I wonder if my understanding is correct?
Does this mean mdadm does not scan to assemble as part of resume? More
importantly, how should I rectify the situation like this? Reboot or
simply stop md0 and scan in a live system?
Further, how do I use /dev/sdf1 (that was /dev/sdg1 before this
problem)? zero-superblock and add back to md0?
Sorry, if this was already discussed. If so, just let me know and I will
search the archive manually as Google did not find it.
Thanks for your help
Ramesh
^ permalink raw reply
* Re: Is it safe to shutdown while mdadm --grow is in progress.
From: Ram Ramesh @ 2014-10-12 16:00 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux Raid
In-Reply-To: <20141012193555.69907c13@notabene.brown>
Thanks. I appreciate you taking the time to respond. I would like to
forward this to those that maintain faq/wiki so that some one searches
gets the info and does not have to bother you. Is that OK?
I normally would go for backup before rebuild, but 15TB disk space is
too much to come up with, without serious $$ and need. Besides, these
are movies/recordings for htpc and the data is available on
dvds/internet. So there is inherent lack of importance. In other words,
I recommend backup as most do, but it looks like every one has some
justification not to do, and want some one else to assure them that it
is ok :-)
Thanks and Regards
Ramesh
On 10/12/2014 03:35 AM, NeilBrown wrote:
> On Sun, 12 Oct 2014 00:46:14 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
>
>> I googled this topic and got differing answers. Some old ones report
>> horror stories and others suggest rebuild continues after reboot
>> normally. My array is rebuilding and it will take about 6 days to
>> complete. I am expecting bad weather in a couple of days and would like
>> to proactively shutdown the machine and reboot. Is this safe?
>>
>> If the information is already available in a clear manner, please let me
>> know.
>>
>> uname: Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38
>> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> mdadm - v3.2.5 - 18th May 2012
>>
>> Thanks for your help
>>
> It is certainly designed to be safe and my experience is that it is.
> However it is difficult to test all corner cases so it is not impossible that
> someone what quite work right.
> If something does go wrong:
> 1/ don't panic
> 2/ don't try to --create the array to fix it
> 3/ report all details to this list and I'm 99.9% sure I can get your array
> running again with all your data safe.
> But I suspect it won't come to that.
>
> NeilBrown
^ permalink raw reply
* Re: Is it safe to shutdown while mdadm --grow is in progress.
From: NeilBrown @ 2014-10-12 8:35 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <543A15A6.5010508@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]
On Sun, 12 Oct 2014 00:46:14 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> I googled this topic and got differing answers. Some old ones report
> horror stories and others suggest rebuild continues after reboot
> normally. My array is rebuilding and it will take about 6 days to
> complete. I am expecting bad weather in a couple of days and would like
> to proactively shutdown the machine and reboot. Is this safe?
>
> If the information is already available in a clear manner, please let me
> know.
>
> uname: Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> mdadm - v3.2.5 - 18th May 2012
>
> Thanks for your help
>
It is certainly designed to be safe and my experience is that it is.
However it is difficult to test all corner cases so it is not impossible that
someone what quite work right.
If something does go wrong:
1/ don't panic
2/ don't try to --create the array to fix it
3/ report all details to this list and I'm 99.9% sure I can get your array
running again with all your data safe.
But I suspect it won't come to that.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Is it safe to shutdown while mdadm --grow is in progress.
From: Ram Ramesh @ 2014-10-12 5:46 UTC (permalink / raw)
To: Linux Raid
I googled this topic and got differing answers. Some old ones report
horror stories and others suggest rebuild continues after reboot
normally. My array is rebuilding and it will take about 6 days to
complete. I am expecting bad weather in a couple of days and would like
to proactively shutdown the machine and reboot. Is this safe?
If the information is already available in a clear manner, please let me
know.
uname: Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
mdadm - v3.2.5 - 18th May 2012
Thanks for your help
Ramesh
^ permalink raw reply
* Please reply
From: Jose Calvache @ 2014-10-11 19:34 UTC (permalink / raw)
Dear Sir/Madam, Here is a pdf attachment of my proposal to you. Please
read and reply I would be grateful. Jose Calvache
^ permalink raw reply
* Re: /sys/block/md126 still exists even after stopping the array
From: Francis Moreau @ 2014-10-10 19:34 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141009205509.1da5dbd9@notabene.brown>
Hello Neil,
On 10/09/2014 11:55 AM, NeilBrown wrote:
> On Thu, 09 Oct 2014 11:40:25 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
[...]
>>
>> Yes, but reading sysfs is also racy. I was thinking that the advantage
>> of using udev is that it gives me a *consistent* (perhaps outdated)
>> snapshot of the device state.
>>
>
> In what sense do you think sysfs is racy? What exactly do you want to do
> with the udev event?
>
> The event from the kernel to udev only contains the identity of the device
> and the type of event (add,change,remove).
>
> Some drivers add extra 'environment' information.
> - 'dm' adds a 'cookie'.
> - bcache add a 'CACHED_UUID' and 'CACHED_LABEL'
> - libata-acpi adds a 'BAY_EVENT'
>
> but in general there is nothing extra.
>
> If udev adds stuff (which is probably does), it is just as racy as anything
> that you might determine and add yourself.
>
I was wrongly assuming that if I queried the udev DB through its library
it would have returned a consistent image of the state of the device
meaning that each property of the device would be coherent with the others.
But even if that was possible, it would be pointless since all accesses
to the device could still fail.
Thanks for enlighting me :)
^ permalink raw reply
* [PATCH] monitor: fix nullptr dereference when get_md_name() returns NULL
From: Sergey Vidishev @ 2014-10-10 13:23 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, sergeyv
From f35857936a8df2155b9aee3759c91fc19b77842c Mon Sep 17 00:00:00 2001
From: Sergey Vidishev <sergeyv@yandex-team.ru>
Date: Wed, 8 Oct 2014 21:51:03 +0400
Subject: [PATCH] monitor: fix nullptr dereference when get_md_name() returns NULL
Signed-off-by: Sergey Vidishev <sergeyv@yandex-team.ru>
---
This patch against fresh git://neil.brown.name/mdadm.
I'm not subscribed to the list, please CC me in replies.
Monitor.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/Monitor.c b/Monitor.c
index 5cb24fa..1919f99 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -668,6 +668,7 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
{
struct mdstat_ent *mse;
int new_found = 0;
+ char *name;
for (mse=mdstat; mse; mse=mse->next)
if (mse->devnm[0] &&
@@ -678,7 +679,12 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
struct state *st = xcalloc(1, sizeof *st);
mdu_array_info_t array;
int fd;
- st->devname = xstrdup(get_md_name(mse->devnm));
+
+ name = get_md_name(mse->devnm);
+ if (!name)
+ return 0;
+
+ st->devname = xstrdup(name);
if ((fd = open(st->devname, O_RDONLY)) < 0 ||
ioctl(fd, GET_ARRAY_INFO, &array)< 0) {
/* no such array */
--
1.9.1
^ permalink raw reply related
* Re: /sys/block/md126 still exists even after stopping the array
From: NeilBrown @ 2014-10-09 9:55 UTC (permalink / raw)
To: Francis Moreau; +Cc: linux-raid, sebastian.riemer
In-Reply-To: <54365809.90406@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 7037 bytes --]
On Thu, 09 Oct 2014 11:40:25 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:
> On 10/08/2014 01:54 AM, NeilBrown wrote:
> > On Tue, 07 Oct 2014 09:05:43 +0200 Francis Moreau <francis.moro@gmail.com>
> > wrote:
> >
> >> Hi Neil,
> >>
> >> On 09/30/2014 09:43 AM, Francis Moreau wrote:
> >>> Hi Neil,
> >>>
> >>> On 09/29/2014 11:56 PM, NeilBrown wrote:
> >>>> On Mon, 29 Sep 2014 10:45:17 +0200 Francis Moreau <francis.moro@gmail.com>
> >>>> wrote:
> >>>>
> >>>>>> So what were pids 930 and 459?
> >>>>>> One was presumably the "mdadm -Ss" - probably 930.
> >>>>>> Is 459 the "mdadm --monitor" ?? That might be useful hint.
> >>>>>>
> >>>>>
> >>>>> yes.
> >>>>>
> >>>>> [456] is: /sbin/mdadm --monitor --scan --daemonise --syslog
> >>>>> --pid-file=/run/mdadm/mdadm.pid
> >>>>>
> >>>>> and [930] is 'mdamd -Ss'.
> >>>>
> >>>> Good. Please try the patch below.
> >>>>
> >>>
> >>> After applying your patch, this is what I'm getting in syslog:
> >>>
> >>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
> >>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
> >>> [970]
> >>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [972]
> >>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
> >>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
> >>> [972]
> >>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by
> >>> systemd-udevd [971]
> >>> Sep 30 03:40:07 localhost systemd[1]: Cannot add dependency job for unit
> >>> mdmonitor-takeover.service, ignoring: Invalid argument
> >>> Sep 30 03:40:07 localhost systemd[1]: Started Software RAID monitoring
> >>> and management.
> >>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by
> >>> systemd-udevd [971]
> >>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> >>> on md device /dev/md125
> >>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> >>> on md device /dev/md126
> >>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> >>> on md device /dev/md127
> >>> Sep 30 03:40:08 localhost kernel: md125: detected capacity change from
> >>> 1863254016 to 0
> >>> Sep 30 03:40:08 localhost kernel: md: md125 stopped.
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc3>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc3)
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb3>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb3)
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md125 released by mdadm
> >>> [970]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> >>> [466]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [466]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> >>> [466]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> >>> [970]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
> >>> Sep 30 03:40:08 localhost kernel: md126: detected capacity change from
> >>> 67043328 to 0
> >>> Sep 30 03:40:08 localhost kernel: md: md126 stopped.
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc1>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc1)
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb1>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb1)
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> >>> [466]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> >>> [970]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> >>> [970]
> >>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
> >>> Sep 30 03:40:08 localhost kernel: md127: detected capacity change from
> >>> 214564864 to 0
> >>> Sep 30 03:40:08 localhost kernel: md: md127 stopped.
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc2>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc2)
> >>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb2>
> >>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb2)
> >>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> >>> [970]
> >>>
> >>> The ghost device is no more present so your patch seems to have fixed my
> >>> issue. But I must admit I don't really understand what's going on :-/
> >>>
> >>
> >> Since those 'ghost' devices are expected from the MD implementation
> >> point of view, I'm wondering how am I supposed to detect them or maybe
> >> how an application is supposed to recognized online arrays.
> >
> > If your application is looking in /proc/mdstat, then the "ghost" devices will
> > be either "inactive" or not present at all.
> > If your application is looking in /sys/block/md*, then the "ghost" devices
> > will have "clear" or "inactive" in /sys/block/mdXX/md/array_state.
> >
> > If you use the new "CREATE names=yes" line in mdadm.conf (mdadm 3.3 or
> > later), and use kernel 3.17 or later, and use names rather than numbers to
> > identify your arrays (/dev/md/home, /dev/md_root), then the "ghost" problem
> > will be gone, and names in /proc/mdstat will be e.g. "md_home", or "md_root"
> > rather than "md4" or "md127".
> >
> >>
> >> My application uses udev to detect et to get information about new
> >> devices. I don't think the information exported by udev is enough to
> >> figure this out. Also please note that since I rely on udev, I can't
> >> really read information on /sys since this information may be out of
> >> sync with the one returned by udev.
> >
> > If udev reports that an array exists, then it really did exist when udev got
> > the message. By the time your program gets run by udev, it might not exist
> > any more. i.e. udev is always racy.
>
> Yes, but reading sysfs is also racy. I was thinking that the advantage
> of using udev is that it gives me a *consistent* (perhaps outdated)
> snapshot of the device state.
>
In what sense do you think sysfs is racy? What exactly do you want to do
with the udev event?
The event from the kernel to udev only contains the identity of the device
and the type of event (add,change,remove).
Some drivers add extra 'environment' information.
- 'dm' adds a 'cookie'.
- bcache add a 'CACHED_UUID' and 'CACHED_LABEL'
- libata-acpi adds a 'BAY_EVENT'
but in general there is nothing extra.
If udev adds stuff (which is probably does), it is just as racy as anything
that you might determine and add yourself.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: /sys/block/md126 still exists even after stopping the array
From: Francis Moreau @ 2014-10-09 9:40 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, sebastian.riemer
In-Reply-To: <20141008105425.64cd0fed@notabene.brown>
On 10/08/2014 01:54 AM, NeilBrown wrote:
> On Tue, 07 Oct 2014 09:05:43 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> Hi Neil,
>>
>> On 09/30/2014 09:43 AM, Francis Moreau wrote:
>>> Hi Neil,
>>>
>>> On 09/29/2014 11:56 PM, NeilBrown wrote:
>>>> On Mon, 29 Sep 2014 10:45:17 +0200 Francis Moreau <francis.moro@gmail.com>
>>>> wrote:
>>>>
>>>>>> So what were pids 930 and 459?
>>>>>> One was presumably the "mdadm -Ss" - probably 930.
>>>>>> Is 459 the "mdadm --monitor" ?? That might be useful hint.
>>>>>>
>>>>>
>>>>> yes.
>>>>>
>>>>> [456] is: /sbin/mdadm --monitor --scan --daemonise --syslog
>>>>> --pid-file=/run/mdadm/mdadm.pid
>>>>>
>>>>> and [930] is 'mdamd -Ss'.
>>>>
>>>> Good. Please try the patch below.
>>>>
>>>
>>> After applying your patch, this is what I'm getting in syslog:
>>>
>>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
>>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
>>> [970]
>>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [972]
>>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
>>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
>>> [972]
>>> Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by
>>> systemd-udevd [971]
>>> Sep 30 03:40:07 localhost systemd[1]: Cannot add dependency job for unit
>>> mdmonitor-takeover.service, ignoring: Invalid argument
>>> Sep 30 03:40:07 localhost systemd[1]: Started Software RAID monitoring
>>> and management.
>>> Sep 30 03:40:07 localhost kernel: md_release(): md125 released by
>>> systemd-udevd [971]
>>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
>>> on md device /dev/md125
>>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
>>> on md device /dev/md126
>>> Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
>>> on md device /dev/md127
>>> Sep 30 03:40:08 localhost kernel: md125: detected capacity change from
>>> 1863254016 to 0
>>> Sep 30 03:40:08 localhost kernel: md: md125 stopped.
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc3>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc3)
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb3>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb3)
>>> Sep 30 03:40:08 localhost kernel: md_release(): md125 released by mdadm
>>> [970]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
>>> [466]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [466]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
>>> [466]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
>>> [970]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
>>> Sep 30 03:40:08 localhost kernel: md126: detected capacity change from
>>> 67043328 to 0
>>> Sep 30 03:40:08 localhost kernel: md: md126 stopped.
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc1>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc1)
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb1>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb1)
>>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
>>> [466]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
>>> [970]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
>>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
>>> [970]
>>> Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
>>> Sep 30 03:40:08 localhost kernel: md127: detected capacity change from
>>> 214564864 to 0
>>> Sep 30 03:40:08 localhost kernel: md: md127 stopped.
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdc2>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc2)
>>> Sep 30 03:40:08 localhost kernel: md: unbind<vdb2>
>>> Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb2)
>>> Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
>>> [970]
>>>
>>> The ghost device is no more present so your patch seems to have fixed my
>>> issue. But I must admit I don't really understand what's going on :-/
>>>
>>
>> Since those 'ghost' devices are expected from the MD implementation
>> point of view, I'm wondering how am I supposed to detect them or maybe
>> how an application is supposed to recognized online arrays.
>
> If your application is looking in /proc/mdstat, then the "ghost" devices will
> be either "inactive" or not present at all.
> If your application is looking in /sys/block/md*, then the "ghost" devices
> will have "clear" or "inactive" in /sys/block/mdXX/md/array_state.
>
> If you use the new "CREATE names=yes" line in mdadm.conf (mdadm 3.3 or
> later), and use kernel 3.17 or later, and use names rather than numbers to
> identify your arrays (/dev/md/home, /dev/md_root), then the "ghost" problem
> will be gone, and names in /proc/mdstat will be e.g. "md_home", or "md_root"
> rather than "md4" or "md127".
>
>>
>> My application uses udev to detect et to get information about new
>> devices. I don't think the information exported by udev is enough to
>> figure this out. Also please note that since I rely on udev, I can't
>> really read information on /sys since this information may be out of
>> sync with the one returned by udev.
>
> If udev reports that an array exists, then it really did exist when udev got
> the message. By the time your program gets run by udev, it might not exist
> any more. i.e. udev is always racy.
Yes, but reading sysfs is also racy. I was thinking that the advantage
of using udev is that it gives me a *consistent* (perhaps outdated)
snapshot of the device state.
Thanks
^ permalink raw reply
* Re: Extremely High mismatch_cnt on RAID1 system
From: Brassow Jonathan @ 2014-10-09 3:17 UTC (permalink / raw)
To: Ethan Wilson; +Cc: linux-raid
In-Reply-To: <5433E72E.60805@shiftmail.org>
On Oct 7, 2014, at 8:14 AM, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
> On 04/10/2014 15:46, Dennis Grant wrote:
>> Hello all.
>>
>> ...
>>
>> Even after multiple checks, repairs, and rebuilds, the arrays on the
>> bigger drives (/ and /home) are showing insanely high mismatch_cnt
>> values. This has me concerned.
>>
>
> Dennis,
> since nobody more knowledgeable replied, I will try.
>
> Some mismatches on raid1 have been there since always, and nobody ever deeply investigated what they were caused by, nor if they happen on unallocated filesystem space or on real live data. It seems that if LVM is between raid1 and the filesystem then they don't happen anymore, but again nobody is really sure of why.
>
> Recently some changes in the raid1 resync algorithm introduced some bugs that could possibly generate additional mismatches, but if you haven't had resyncs then I am not so sure if such bugs and their fixes are relevant. However the fixes are here:
> https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.14.20
> search for "raid".
>
> You might want to upgrade to kernel 3.14.20, which is probably not what your Ubuntu LTS has currently, then repair the arrays, then see if they grow again.
> Note that you need to do repair and not check:
> echo repair > /sys/block/md0/md/sync_action
> at the next "check" the mismatch_cnt should be 0 (not just after "repair", because that would count the number of mismatches that have been repaired).
>
> I'd say that mismatches in general are pretty worrisome, they shouldn't happen, they are likely to indicate corruption, so if what I said doesn't work, e.g. mismatches grow again, try to report it again on the list and somebody might be able to help further to track down this problem.
The mismatches count can be incremented during operations other than check and repair. I believe its behavior also varies between RAID personalities. However, if you check the ‘last_sync_action’ and see that it was a “check” operation, you are probably safe to assume that the mismatch count has been computed correctly.
Note the following commit:
commit c4a39551451666229b4ea5e8aae8ca0131d00665
Author: Jonathan Brassow <jbrassow@redhat.com>
Date: Tue Jun 25 01:23:59 2013 -0500
MD: Remember the last sync operation that was performed
MD: Remember the last sync operation that was performed
This patch adds a field to the mddev structure to track the last
sync operation that was performed. This is especially useful when
it comes to what is recorded in mismatch_cnt in sysfs. If the
last operation was "data-check", then it reports the number of
descrepancies found by the user-initiated check. If it was a
"repair" operation, then it is reporting the number of
descrepancies repaired. etc.
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>
Relatedly, LVM makes use of the MD RAID personalities to provide its RAID capabilities. It does this by accessing MD through a thin device-mapper target called "dm-raid” - not to be confused with the similarly named userspace application. The above mentioned commit contains a change to the dm-raid module as well, which causes it to report ‘0’ mismatches unless the ‘last_sync_action’ was a “check”. So, for dm-raid (and by extension LVM) the ambiguity in mismatch_count is gone, but the user must be careful when looking at the number for MD.
brassow--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: writing zeros to bad sector results in persistent read error
From: Phillip Susi @ 2014-10-08 17:56 UTC (permalink / raw)
To: Chris Murphy, Wilson Jonathan; +Cc: linux-raid@vger.kernel.org List
In-Reply-To: <BBD6BB5F-9D33-413B-AD74-3DC3EA34D6AB@colorremedies.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 6/9/2014 8:09 PM, Chris Murphy wrote:
> But what I'm reporting is an instance where an ATA Secure Erase
> definitely did not fix up a single one of the bad sectors. Maybe
> that's consistent with the spec, I don't know, but it's not what
> I'd expect seeing as every sector, those with an without LBA's
> assigned, are overwritten. Yet pending sectors were not remapped.
> Further, with all sectors overwritten by software (not merely the
> ATA Secure Erase command) yields no errors yet SMART reports there
> are still pending sectors, yet it's own extended test says there
> are none. I think that's bad behavior. But perhaps I don't
> understand the design and it's actually working as designed.
It sounds like what happened is the secure erase successfully rewrote
the sectors that were already flagged as pending, but did not
decrement the pending count.
FYI, rather than continuing to run a smart selftest to find one
sector, then use dd to fix it, and repeat, it would be much faster to
use the badblocks utility to read and rewrite the whole drive. You
will want to make sure to use the correct sector size, and a
sufficiently large batch size for good performance.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUNXrjAAoJEI5FoCIzSKrw//cH/jgbli22/MmRpRXLOc0YJg8O
npSEI3fusBspMWhWS+a5SGRQQQjrfiK8mK8NkAC1VrX80zI8UcLkrBNVX1NQQ7eP
tgjJJJLN0BeQIk7RtAhO0rxajnZp19bBv7r8oRgWg9PRXrrxZHrXJNxHqUlANNsq
70blruORy3MbTqUk8QU4qXw/y5XduhRyJEX0SDogrQwI0xJqaUWPn5CQPQnKWydr
0q6evfdRVfLC2rg0AbQ1ksj+nRhTRkrUctXuNc/8GL4S6wR77bQwTXlyBn8E8Uec
T6lsCs5J43e2yyRtj3c0ZWcmyuZuwKbO4LHPAA4kYf9faHV/OEWPwlAHHVC1Ggo=
=2OuK
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: writing zeros to bad sector results in persistent read error
From: Phillip Susi @ 2014-10-08 17:51 UTC (permalink / raw)
To: Roger Heflin, Martin K. Petersen
Cc: Chris Murphy, Phil Turmel, Wolfgang Denk, Roman Mamedov,
linux-raid@vger.kernel.org List
In-Reply-To: <CAAMCDef3tO4Ah_3gYJ_6Y0Kh_mPn+z=E5voHFYr-z7_Leuf48Q@mail.gmail.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 6/29/2014 8:51 PM, Roger Heflin wrote:
> All of this is probably the reason that this command exists:
>
> hdparm --write-sector <sectornum>
>
> I believe it directly sends the scsi/ata layer commands.
You end up with the same results as using dd ( with oflag=direct ), it
is just a matter of the path it takes to get there.
With dd, it calls write() to pass the data to the block layer, which
hands it to the scsi layer, which translates it into a scsi
WRITE_10/16 command, which hands it to libata which translates it into
an ata taskfile to be handed to the drive.
With hdparm --write-sector, it builds the ata taskfile, uses the SG_IO
ioctl to hand it to the block layer, which hands it down through the
scsi and libata layers which see that it needs no translation and it
goes to the drive unmodified.
The resulting taskfile the drive actually sees should be the same.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUNXmlAAoJEI5FoCIzSKrwMYwIAJj1c0pBxdOcCQAk4i26802S
/lbPHhY5Xu7wR5KbZSXeEazE/vgTT7mDgjWHoe6Vl9e+Ci90KJxSFgQXNNwcYtuK
V+UFrTyqiKAzfk8VbRj0kwxk1JuXRQesDlwCGUsBkjSO26pdhUVfxwP8I3JcOBQW
uKRmh8PE48iq7kDWQdtxve6IPnAj/VY8AubwRAaVAvZ3xsEUBlf7UAkvA4n3WvWN
mfO1VVWwv4zyZ6bEBoWfjj6//5C0R+q2TrnBDFD9pN/wY4TdAx0gtufiUWx0v5WG
NNzJ9tm5z2rNo/HNi4w1gHm0JLDhSky21sNX7KyY8/1tFjqa3KQT7iQ6vxk4UJM=
=Xv01
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: Bad sequential performance of RAID5 with a lot of disk seeks
From: XiaoNi @ 2014-10-08 9:05 UTC (permalink / raw)
To: P. Gautschi; +Cc: linux-raid
In-Reply-To: <54336FC1.6080306@gautschi.net>
On 10/07/2014 12:44 PM, P. Gautschi wrote:
> I've created a RAID5 on 5 identical SATA disks. Doing some performance
> measurements
> with dd I get a disappointing performance.
> A dd with bs=1M on a btrfs created on md0 transfers about 110 MB/s.
> (both read and write)
> A dd on md0 has the same write speed but only about 20 MB/s on read.
> In all of the tests I hear the disk constantly seeking. This was also
> the case
> during creation of the array.
> I also created a RAID4 to make sure that I doesn't get fooled by the
> stripe layout of RAID5.
> Now I get about 110 MB/s for write and 230 MB/s for read on md0. But
> the constant
> seeking is still present for both read and write and during creation
> of the array.
>
> Why are the disk perform so many seek operations? I think a sequential
> access on md0 should
> cause a sequential access on the individual disk.
Hi P. Gautschi
How can you find the seek operations? Do you use blktrace or other
commands? Can you give the
detail commands and informations?
BTW, I think ever if you use larger chunksize when create raid,
there is few RMW. There is a period to wait for full write.
>
> I have to add that I did something unusual: I created the RAID4/5 with
> a chunk size of 4KiB.
> The idea of this was that when I'm going to use btrfs with the default
> nodesize of 16KiB
> all node write will fill a full stripe and there won't be any RMW at
> all. (both fortunate
> for performance and integrity in a power loss situation.)
> Nevertheless I think a sequential access on the array should cause a
> sequential access on the
> disks for any chunk size if the read/write block size is a exact
> multiple of
> the (numdisks-1)*chunk size.
>
> Is there any explanation for the seeks and how do I get rid of them?
>
> Patrick
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: /sys/block/md126 still exists even after stopping the array
From: NeilBrown @ 2014-10-07 23:54 UTC (permalink / raw)
To: Francis Moreau; +Cc: linux-raid, sebastian.riemer
In-Reply-To: <543390C7.2080104@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6330 bytes --]
On Tue, 07 Oct 2014 09:05:43 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:
> Hi Neil,
>
> On 09/30/2014 09:43 AM, Francis Moreau wrote:
> > Hi Neil,
> >
> > On 09/29/2014 11:56 PM, NeilBrown wrote:
> >> On Mon, 29 Sep 2014 10:45:17 +0200 Francis Moreau <francis.moro@gmail.com>
> >> wrote:
> >>
> >>>> So what were pids 930 and 459?
> >>>> One was presumably the "mdadm -Ss" - probably 930.
> >>>> Is 459 the "mdadm --monitor" ?? That might be useful hint.
> >>>>
> >>>
> >>> yes.
> >>>
> >>> [456] is: /sbin/mdadm --monitor --scan --daemonise --syslog
> >>> --pid-file=/run/mdadm/mdadm.pid
> >>>
> >>> and [930] is 'mdamd -Ss'.
> >>
> >> Good. Please try the patch below.
> >>
> >
> > After applying your patch, this is what I'm getting in syslog:
> >
> > Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
> > Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
> > [970]
> > Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [972]
> > Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by mdadm [970]
> > Sep 30 03:40:07 localhost kernel: md_release(): md125 released by mdadm
> > [972]
> > Sep 30 03:40:07 localhost kernel: md_open(): md125 opened by
> > systemd-udevd [971]
> > Sep 30 03:40:07 localhost systemd[1]: Cannot add dependency job for unit
> > mdmonitor-takeover.service, ignoring: Invalid argument
> > Sep 30 03:40:07 localhost systemd[1]: Started Software RAID monitoring
> > and management.
> > Sep 30 03:40:07 localhost kernel: md_release(): md125 released by
> > systemd-udevd [971]
> > Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> > on md device /dev/md125
> > Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> > on md device /dev/md126
> > Sep 30 03:40:08 localhost mdadm[466]: DeviceDisappeared event detected
> > on md device /dev/md127
> > Sep 30 03:40:08 localhost kernel: md125: detected capacity change from
> > 1863254016 to 0
> > Sep 30 03:40:08 localhost kernel: md: md125 stopped.
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdc3>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc3)
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdb3>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb3)
> > Sep 30 03:40:08 localhost kernel: md_release(): md125 released by mdadm
> > [970]
> > Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
> > Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> > [466]
> > Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [466]
> > Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> > [466]
> > Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
> > Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> > [970]
> > Sep 30 03:40:08 localhost kernel: md_open(): md126 opened by mdadm [970]
> > Sep 30 03:40:08 localhost kernel: md126: detected capacity change from
> > 67043328 to 0
> > Sep 30 03:40:08 localhost kernel: md: md126 stopped.
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdc1>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc1)
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdb1>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb1)
> > Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [466]
> > Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> > [466]
> > Sep 30 03:40:08 localhost kernel: md_release(): md126 released by mdadm
> > [970]
> > Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
> > Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> > [970]
> > Sep 30 03:40:08 localhost kernel: md_open(): md127 opened by mdadm [970]
> > Sep 30 03:40:08 localhost kernel: md127: detected capacity change from
> > 214564864 to 0
> > Sep 30 03:40:08 localhost kernel: md: md127 stopped.
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdc2>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdc2)
> > Sep 30 03:40:08 localhost kernel: md: unbind<vdb2>
> > Sep 30 03:40:08 localhost kernel: md: export_rdev(vdb2)
> > Sep 30 03:40:08 localhost kernel: md_release(): md127 released by mdadm
> > [970]
> >
> > The ghost device is no more present so your patch seems to have fixed my
> > issue. But I must admit I don't really understand what's going on :-/
> >
>
> Since those 'ghost' devices are expected from the MD implementation
> point of view, I'm wondering how am I supposed to detect them or maybe
> how an application is supposed to recognized online arrays.
If your application is looking in /proc/mdstat, then the "ghost" devices will
be either "inactive" or not present at all.
If your application is looking in /sys/block/md*, then the "ghost" devices
will have "clear" or "inactive" in /sys/block/mdXX/md/array_state.
If you use the new "CREATE names=yes" line in mdadm.conf (mdadm 3.3 or
later), and use kernel 3.17 or later, and use names rather than numbers to
identify your arrays (/dev/md/home, /dev/md_root), then the "ghost" problem
will be gone, and names in /proc/mdstat will be e.g. "md_home", or "md_root"
rather than "md4" or "md127".
>
> My application uses udev to detect et to get information about new
> devices. I don't think the information exported by udev is enough to
> figure this out. Also please note that since I rely on udev, I can't
> really read information on /sys since this information may be out of
> sync with the one returned by udev.
If udev reports that an array exists, then it really did exist when udev got
the message. By the time your program gets run by udev, it might not exist
any more.
i.e. udev is always racy.
You should always treat any event from udev as a hint:
"Something happened to this device in the recent past. Lots of other
things might have happened since. The device might not exist any more, or
it might have been replaced with a completely different device. So you
might want to do something, or you might not, but whatever you do - be
careful and don't blame me if things go wrong 'cause I'm just the
messenger."
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: auto assemble in read-only mode?
From: NeilBrown @ 2014-10-07 23:33 UTC (permalink / raw)
To: Errol Neal; +Cc: Linux-RAID
In-Reply-To: <1412685094488232500@businessgrade.com>
[-- Attachment #1: Type: text/plain, Size: 2144 bytes --]
On Tue, 7 Oct 2014 08:31:34 -0400 Errol Neal <eneal@businessgrade.com> wrote:
> On Tue, 10/07/2014 03:15 AM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 7 Oct 2014 02:56:31 -0400 Errol Neal <eneal@businessgrade.com> wrote:
> >
> > > On Mon, 10/06/2014 09:06 AM, Errol Neal <eneal@businessgrade.com> wrote:
> > > > hi folks. is it possible to assemble an md device in read-only mode?
> > > > after an array has been assembled, I know I can mark it as readonly with --misc -o /dev/md0 for example, but only after its been assembled it seems..
> > > >
> > >
> > > Hi again. No thoughts here?
> >
> > Not even 18 hours since you last asked - what do you thing this is? The
> > Internet :-)
>
> HAHA. I tend to feel ignored on large mailing lists lol. Need to see my therapist again ;)
>
> >
> > > I'm need to run a RAID1 device on two independent nodes (shared storage) and I'd like to assemble the device on the "slave" node in read-only mode to prevent any sync actions and etc.
> >
> > Did you try adding the "--readonly" option to your --assemble command?
> > (Always best to report everything you tried and how it failed - requires less
> > guess work).
>
> I did that previously and mdadm told me that "option --readonly not valid in assemble mode". E.g. "mdadm --assemble --scan --readonly".
That tells me use are using a version of mdadm earlier than 3.3.
"--assemble --readonly" support was added for 3.3.
NeilBrown
> The readonly flag seems like it can only be used after a device has been assembled. So for example, after the device has been assembled, I can
> run mdadm --misc -o /dev/md0 but that is not always safe for my use case.
>
>
>
> > I hope you don't expect too much of a filesystem mounted from such an array
> > while the other node is writing to the array...
> >
>
> I don't. It will be read-only until the pacemaker RA promotes the slave.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: UEFI and mdadm questions.
From: Chris Murphy @ 2014-10-07 22:59 UTC (permalink / raw)
To: Linux-RAID Raid
In-Reply-To: <5432BC0A.1030208@turmel.org>
On Oct 6, 2014, at 11:58 AM, Phil Turmel <philip@turmel.org> wrote:
> On 10/05/2014 04:22 PM, Chris Murphy wrote:
>>
>> On Oct 5, 2014, at 2:18 PM, Phil Turmel <philip@turmel.org> wrote:
>
> [trim /]
>
>>> If your BIOS can be configured to try multiple boot images, it
>>> should be possible to have true raid fallback without using
>>> motherboard or hardware raid. (Set up md raid1 with metadata v1.0
>>> of multiple copies of the EFI FAT partition.) I've been meaning to
>>> try this….
>>
>> Problems with this: a.) new Windows 8 hardware might require you boot
>> Windows to get to the feature enabling the firmware setup, because on
>> such hardware USB isn't initialized by default.
>> http://mjg59.dreamwidth.org/24869.html
>>
>> I don't know why we don't have free software to initiate this, but I
>> haven't come across it so far.
>
> Good to know, but totally immaterial to the boot sequence I'm
> recommending. Boot linux of of an EFI FAT and let linux initialize the
> USB hardware in its own good time.
Sure but it means that you can't use the firmware's boot manager to choose anything else. Any other kernel (or copy) only happens in the course of the firmware's fallback mechanisms.
>
> Matthew Garrett's post is really all about how to get linux into the
> Win8 box in the first place. Once there, manipulate the boot sequence
> as you please.
>
>> b.) There's no guarantee the firmware won't write to the ESP, thus
>> rendering the individual md raid members out of sync and without
>> their metadata being updated, i.e. in effect, the logical device they
>> become later, is corrupt. Separately they aren't corrupt, merely out
>> of sync, but you don't have an obvious way of knowing which one.
>
> This is a very good point. In fact, I withdraw my recommendation to
> raid these partitions. Simply have one on every disk the BIOS could
> possibly boot from, and place an EFI bootable kernel in each one (with
> embedded initramfs).
Well, to me that seems more esoteric than just having kernel+initramfs on a conventional md raid1 /boot volume, and having a static (never upgraded or modified) GRUB2 or syslinux point to the modifiable configuration file (basically load a 2nd config file) in the usual location also on /boot. So now kernel upgrades are normal, and the user gets to regress to older kernels on demand should it be necessary and they also get resilient boot.
>
>> c.) strictly speaking any partition with mdadm metadata should have
>> the linux raid partition type GUID set; not the EFI System partition
>> type GUID. Those GUIDs are mutually exclusive.
>
> The former is not true at all--mdadm does not care *at all* what
> partition types are set. Grub might care, but it's moot if you don't
> use Grub. :-)
The partition type isn't for mdadm, it's for other things that might otherwise modify the partition if it's misidentified as an EFI System partition. And it would be a misidentification to set the partition type GUID to EFI System partition, because first this example partition is an md member, and only after assembly is it an EFI system partition. So I still consider them mutually exclusive.
>
>> This is why I'm still not a fan of using mdadm to raid1 an EFI System
>> partition.
>
> One further point: the failure decision tree is nicer if you boot
> directly into a kernel.
>
> 1) Bios locates and attempts to boot from 1st configured kernel image
> 2a) Corrupted image or other disk error blocks complete load of kernel
> image--bios moves to next EFI choice (possibly on a different disk).
> 2b) Successful EFI kernel load, boot encounters missing/corrupt root
> FS--kernel drops to initramfs rescue shell
>
> versus:
>
> 1) Bios locates and attempts to boot from 1st configured grub image
> 2a) Corrupted image or other disk error blocks complete load of
> grub--bios moves to next EFI choice (possibly on a different disk).
> 2b) Successful EFI grub load, grub encounters corrupt config or grub
> module--drop to grub shell
> 2c) Successful EFI grub load, kernel & initramfs load by grub, boot
> encounters missing/corrupt root FS--kernel drops to initramfs rescue shell
>
> I haven't had time to set it up yet, but the clear reduction in points
> of failure is compelling. Faster boot is just icing on the cake.
But we have a lot of experience with the latter, as that's how it's always been on BIOS systems. In a way it's more complicated because BIOS itself didn't really have a meaningful (or complicated) fallback mechanism, that was just up to the boot manager which we (FOSS) control. But with UEFI we don't control this, so exactly how the firmware behaves in failure cases actually needs to be tested on a firmware by firmware basis.
The other thing is this arrangement isn't supported by any distro currently. So setting it up and maintaining is pretty cumbersome.
Chris Murphy--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Extremely High mismatch_cnt on RAID1 system
From: Ethan Wilson @ 2014-10-07 14:23 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <BLU436-SMTP23591E5AC8AD1D0191C134D98A20@phx.gbl>
On 07/10/2014 15:58, Wilson, Jonathan wrote:
> Would mismatches happen if an "assume clean" was used
--assume-clean during creation, then yes, until the first "repair" and
then "check".
> , either for a good
> reason (say to forced a dropped disk back in
I don't think it is possible to force the addition of a disk with
--assume-clean, I think that's an option only for --create
> ) or in error, so that while
> the data on the secondary disk(s) becomes self correcting as new
> writes/updates are performed, to all disks, should the "primary" drive
> fail the second one would contain out of sync data, where it had never
> been (re)written. Although which is "primary" and which is "secondary"
> is I guess not really a good description.
>
> I would have thought that doing a DD to a _FILE_ that fills up the file
> system would also reduce the mismatch count
Yes, except theoretically for raid5 which operates RMW mode, because
that mode propagates existing parity errors if non-full-stripes are written.
But a large file is written sequentially, probably full stripes will be
written, so in that case, yes again.
> , as it would force
> "correct(ing)" data to all the disks, baring reserved file system
> blocks/areas.
Indeed yours is a good way to determine if mismatches are mapped to
existing files or to unused space on the filesystem.
Once all the filesystem space is overwritten with a file, if
mismatch_cnt is still nonzero, the mismatches are evidently located on
files, which means data corruption.
If Dennis tells us that the mismatches count still raises after kernel
upgrade and raid repair (repair itself will bring it to 0), we can
suggest this test, to check for data corruption.
EW
^ permalink raw reply
* Re: Extremely High mismatch_cnt on RAID1 system
From: Wilson, Jonathan @ 2014-10-07 13:58 UTC (permalink / raw)
To: Ethan Wilson; +Cc: linux-raid
In-Reply-To: <5433E72E.60805@shiftmail.org>
On Tue, 2014-10-07 at 15:14 +0200, Ethan Wilson wrote:
> On 04/10/2014 15:46, Dennis Grant wrote:
> > Hello all.
> >
> > ...
> >
> > Even after multiple checks, repairs, and rebuilds, the arrays on the
> > bigger drives (/ and /home) are showing insanely high mismatch_cnt
> > values. This has me concerned.
> >
>
> Dennis,
> since nobody more knowledgeable replied, I will try.
>
> Some mismatches on raid1 have been there since always, and nobody ever
> deeply investigated what they were caused by, nor if they happen on
> unallocated filesystem space or on real live data. It seems that if LVM
> is between raid1 and the filesystem then they don't happen anymore, but
> again nobody is really sure of why.
Would mismatches happen if an "assume clean" was used, either for a good
reason (say to forced a dropped disk back in) or in error, so that while
the data on the secondary disk(s) becomes self correcting as new
writes/updates are performed, to all disks, should the "primary" drive
fail the second one would contain out of sync data, where it had never
been (re)written. Although which is "primary" and which is "secondary"
is I guess not really a good description.
I would have thought that doing a DD to a _FILE_ that fills up the file
system would also reduce the mismatch count, as it would force
"correct(ing)" data to all the disks, baring reserved file system
blocks/areas.
NOTE DD to a FILE on the file system, NOT the raid device, the latter
will DESTROY ALL data!
^ permalink raw reply
* Re: Extremely High mismatch_cnt on RAID1 system
From: Ethan Wilson @ 2014-10-07 13:14 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAOWcSW5+8ZkWAyXL7VTk9JO+yrozFJS8RwroKqtudkiKD-fvXA@mail.gmail.com>
On 04/10/2014 15:46, Dennis Grant wrote:
> Hello all.
>
> ...
>
> Even after multiple checks, repairs, and rebuilds, the arrays on the
> bigger drives (/ and /home) are showing insanely high mismatch_cnt
> values. This has me concerned.
>
Dennis,
since nobody more knowledgeable replied, I will try.
Some mismatches on raid1 have been there since always, and nobody ever
deeply investigated what they were caused by, nor if they happen on
unallocated filesystem space or on real live data. It seems that if LVM
is between raid1 and the filesystem then they don't happen anymore, but
again nobody is really sure of why.
Recently some changes in the raid1 resync algorithm introduced some bugs
that could possibly generate additional mismatches, but if you haven't
had resyncs then I am not so sure if such bugs and their fixes are
relevant. However the fixes are here:
https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.14.20
search for "raid".
You might want to upgrade to kernel 3.14.20, which is probably not what
your Ubuntu LTS has currently, then repair the arrays, then see if they
grow again.
Note that you need to do repair and not check:
echo repair > /sys/block/md0/md/sync_action
at the next "check" the mismatch_cnt should be 0 (not just after
"repair", because that would count the number of mismatches that have
been repaired).
I'd say that mismatches in general are pretty worrisome, they shouldn't
happen, they are likely to indicate corruption, so if what I said
doesn't work, e.g. mismatches grow again, try to report it again on the
list and somebody might be able to help further to track down this problem.
Regards
EW
^ permalink raw reply
* Re: auto assemble in read-only mode?
From: Errol Neal @ 2014-10-07 12:31 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux-RAID
In-Reply-To: <20141007181520.359d939b@notabene.brown>
On Tue, 10/07/2014 03:15 AM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Oct 2014 02:56:31 -0400 Errol Neal <eneal@businessgrade.com> wrote:
>
> > On Mon, 10/06/2014 09:06 AM, Errol Neal <eneal@businessgrade.com> wrote:
> > > hi folks. is it possible to assemble an md device in read-only mode?
> > > after an array has been assembled, I know I can mark it as readonly with --misc -o /dev/md0 for example, but only after its been assembled it seems..
> > >
> >
> > Hi again. No thoughts here?
>
> Not even 18 hours since you last asked - what do you thing this is? The
> Internet :-)
HAHA. I tend to feel ignored on large mailing lists lol. Need to see my therapist again ;)
>
> > I'm need to run a RAID1 device on two independent nodes (shared storage) and I'd like to assemble the device on the "slave" node in read-only mode to prevent any sync actions and etc.
>
> Did you try adding the "--readonly" option to your --assemble command?
> (Always best to report everything you tried and how it failed - requires less
> guess work).
I did that previously and mdadm told me that "option --readonly not valid in assemble mode". E.g. "mdadm --assemble --scan --readonly".
The readonly flag seems like it can only be used after a device has been assembled. So for example, after the device has been assembled, I can
run mdadm --misc -o /dev/md0 but that is not always safe for my use case.
> I hope you don't expect too much of a filesystem mounted from such an array
> while the other node is writing to the array...
>
I don't. It will be read-only until the pacemaker RA promotes the slave.
^ permalink raw reply
* Re: Bad sequential performance of RAID5 with a lot of disk seeks
From: Robin Hill @ 2014-10-07 11:05 UTC (permalink / raw)
To: P. Gautschi; +Cc: linux-raid
In-Reply-To: <20141007123619.Horde.tC2-gMUttC5BYOoaKogzlw9@webmail.gautschi.net>
[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]
On Tue Oct 07, 2014 at 12:36:19PM +0200, P. Gautschi wrote:
> > What does the SMART info show for the drives - are there any reallocated
> > blocks? A large number of those scattered over the disk would certainly
> > cause seeking for both reads and writes.
>
> I will check the SMART this evening but I don't think that this is causing
> the seek. The sound is very constant and for the whole time of syncing
> the array.
> I will also run a dd on the disk to compare.
>
> > It's also worth checking whether there's anything else that would be
> > accessing the disks in the background (monitoring/indexing/etc).
>
> Unlikely because I have not yet created a filesystem after setting up
> the RAID4.
>
> > I can't think of anything else that would be causing reads to seek - SMR
> > disks or write-intent bitmaps would only affect writes.
>
> Exactly
>
> Is there any way or tool to monitor all disk read/write commands - not only
> the count or amount but every access with LBA and length?
>
You can do:
echo 1 > /proc/sys/vm/block_dump
That will write out all disk IO to the kernel log (process ID,
read/write and block offset only though). It can be very verbose,
especially if you have a lot of other things running on the system, but
you should be able to grep out the necessary lines. Echoing 0 will
switch it back off again.
Otherwise there's probably ways to get more specific results via the
kernel auditing system, but that's nothing I've played with.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Bad sequential performance of RAID5 with a lot of disk seeks
From: P. Gautschi @ 2014-10-07 10:36 UTC (permalink / raw)
To: Robin Hill; +Cc: linux-raid
In-Reply-To: <20141007092548.GC18786@cthulhu.home.robinhill.me.uk>
> What does the SMART info show for the drives - are there any reallocated
> blocks? A large number of those scattered over the disk would certainly
> cause seeking for both reads and writes.
I will check the SMART this evening but I don't think that this is causing
the seek. The sound is very constant and for the whole time of syncing
the array.
I will also run a dd on the disk to compare.
> It's also worth checking whether there's anything else that would be
> accessing the disks in the background (monitoring/indexing/etc).
Unlikely because I have not yet created a filesystem after setting up
the RAID4.
> I can't think of anything else that would be causing reads to seek - SMR
> disks or write-intent bitmaps would only affect writes.
Exactly
Is there any way or tool to monitor all disk read/write commands - not only
the count or amount but every access with LBA and length?
Patrick
^ permalink raw reply
* Re: Bad sequential performance of RAID5 with a lot of disk seeks
From: Robin Hill @ 2014-10-07 9:25 UTC (permalink / raw)
To: P. Gautschi; +Cc: linux-raid
In-Reply-To: <20141007095422.Horde.3l6BckPhfY5U8dchaOdLNA1@webmail.gautschi.net>
[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]
On Tue Oct 07, 2014 at 09:54:22AM +0200, P. Gautschi wrote:
> Quoting Robin Hill <robin@robinhill.me.uk>:
> > On Tue Oct 07, 2014 at 06:44:49AM +0200, P. Gautschi wrote:
> >
> >> I've created a RAID5 on 5 identical SATA disks. Doing some
> >> performance measurements
> >> with dd I get a disappointing performance.
> >> A dd with bs=1M on a btrfs created on md0 transfers about 110 MB/s.
> >> (both read and write)
> >> A dd on md0 has the same write speed but only about 20 MB/s on read.
> >> In all of the tests I hear the disk constantly seeking. This was
> >> also the case
> >> during creation of the array.
> >> I also created a RAID4 to make sure that I doesn't get fooled by
> >> the stripe layout of RAID5.
> >> Now I get about 110 MB/s for write and 230 MB/s for read on md0.
> >> But the constant
> >> seeking is still present for both read and write and during
> >> creation of the array.
> >>
> >> Why are the disk perform so many seek operations? I think a
> >> sequential access on md0 should
> >> cause a sequential access on the individual disk.
> >>
> >> I have to add that I did something unusual: I created the RAID4/5
> >> with a chunk size of 4KiB.
> >> The idea of this was that when I'm going to use btrfs with the
> >> default nodesize of 16KiB
> >> all node write will fill a full stripe and there won't be any RMW
> >> at all. (both fortunate
> >> for performance and integrity in a power loss situation.)
> >> Nevertheless I think a sequential access on the array should cause
> >> a sequential access on the
> >> disks for any chunk size if the read/write block size is a exact multiple of
> >> the (numdisks-1)*chunk size.
> >>
> >> Is there any explanation for the seeks and how do I get rid of them?
> >>
> > After creating the arrays did you wait for them to finish syncing? The
> > array is created in degraded mode initially and then rebuilds onto the
> > additional disk (this is the fastest way to do things, unless you know
> > the disks are all zeroed initially). Until this rebuild is complete then
> > it'll be competing with any other disk activity.
> >
> Yes I did wait for the syncing to complete before accessing md0.
> I had the seeks during the syncing and afterwards when reading or
> writing to the array.
>
Hmm, shouldn't be seeking then.
What does the SMART info show for the drives - are there any reallocated
blocks? A large number of those scattered over the disk would certainly
cause seeking for both reads and writes.
It's also worth checking whether there's anything else that would be
accessing the disks in the background (monitoring/indexing/etc).
I can't think of anything else that would be causing reads to seek - SMR
disks or write-intent bitmaps would only affect writes.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox