* Re: Questions
From: Wols Lists @ 2016-02-15 22:09 UTC (permalink / raw)
To: o1bigtenor; +Cc: Adam Goryachev, Linux-RAID
In-Reply-To: <CAPpdf5-LRMk0pcsBrp_nwmrNmPKt5jeo0_zjbH5cMZqA0+LtRQ@mail.gmail.com>
On 15/02/16 21:01, o1bigtenor wrote:
> I have a present working array and with to replace its components with the
> same size but new drives (which are NAS rated). Was thinking that using
> the fail remove and add process 4 seperate times might not be a good thing
> but I do not know of a different option. Compounding the difficulty is
> that there
> are no empty hard drive slots in the machine. I do have an external USB 3.0
> 2 drive holder that could be used.
Does it have a spare PCI slot? That's what I was getting at - can you
add two more sata slots? Presumably not if it's a cage, but if it's a
computer case? As I said, a spare PCI SATA card should be dirt cheap to
add temporary extra capacity. And does it matter if the case is open and
the drives lying around temporarily?
>
> The only suggestion in all the documents I perused was to place spare drives
> into something like this external box and then add the drives into the array.
> The process was not laid out and leaves me with a number of questions.
>
> Is there a suggested method for replacing ALL the drives in an array (raid 10
> in this case)?
As far as I'm aware, there's just the "mdadm --replace" I mentioned -
drive by drive. Given that it's raid 10, maybe you can just add another
mirror then fail an old one.
I'd just plug in the extra drives, run mdadm --replace, and then remove
the old drives. Just make sure you get the right drives! And always use
uuids so you know which drive is which!
Get Phil's lsdrv and it will probably give you all the drives, with
serial numbers, etc etc. I haven't managed to run it so I can't be sure
:-) But it's intended to give you all the stuff you need to recover an
array so it should give you the information you need to rebuild it.
>
> If I use the external box how do I do this (external box only holds 2 drives) so
> that I can transfer the information on the drives from the array to
> the new drives
> and then just replace the drives 2 at a time into the machine without
> there being
> issues because in the information transfer the drives will be sdg and
> sdh (AFAIK)
> and later they will be some of sdb, sdc, sde, and/or sdf.
That's why they now have uuids.
ls /dev/disk/by-id
I *think* raid uses uuids internally. So swapping the drives out won't
be a problem - the sdx names are just used as a human-readable output.
But don't take that as gospel ... However, bear in mind that the kernel
does NOT guarantee that a drive will get the same sdx name from one boot
to the next. It so happens that that is the norm, but it's not
guaranteed ... so x changing for any value of sdx shouldn't be a problem.
Regardless, you should not be using sdx. Everything should be using
uuids, my /etc/fstab is a lovely mangle with all those long uuids
everywhere :-)
Cheers,
Wol
^ permalink raw reply
* mdadm-3.4 regression tests
From: Bruce Dubbs @ 2016-02-16 4:58 UTC (permalink / raw)
To: linux-raid
I have a problem when trying to document madam-3.4 for the Linux From
Scratch project. The package builds fine:
tar -xf mdadm-3.4.tar.xz
cd mdadm-3.4
make
make test
Then when I run:
sudo ./test --keep-going --logdir=test-logs --save-logs
Some tests fail and one hangs forever. I can post logs if needed.
Please run './test' as root
Testing on linux-4.4.1-lfs-7.9-rc1 kernel
Saving logs to test-logs
tests/00linear... succeeded
tests/00multipath... succeeded
tests/00names... succeeded
tests/00raid0... succeeded
tests/00raid1... FAILED - see test-logs/log-00raid1 for details
cp: cannot stat '/var/tmp/log': No such file or directory
tests/00raid10... succeeded
tests/00raid4... succeeded
tests/00raid5... succeeded
tests/00raid6... succeeded
tests/01r1fail... succeeded
tests/01r5fail... succeeded
tests/01r5integ... succeeded
tests/01raid6integ... succeeded
tests/01replace...
The failure says:
++ case $* in
++ udevadm settle
+++ cat /proc/sys/dev/raid/speed_limit_max
++ p=2000
++ echo 20000
++ case $* in
++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -S /dev/md0
++ rv=1
++ case $* in
++ udevadm settle
++ echo 2000
++ cat /var/tmp/stderr
++ return 1
++ mdadm -B /dev/md0 -l 1 --assume-clean -n2 /dev/loop0 /dev/loop1
++ rm -f /var/tmp/stderr
++ case $* in
++ case $* in
++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -B /dev/md0 -l 1 --assume-clean -n2
/dev/loop0 /dev/loop1
++ rv=1
...
The bigger problem is that 01replace hangs forever. Looking at my process
list:
root 16064 2 0 22:45 ? 00:00:00 [md0_raid1]
root 16093 16049 0 22:45 pts/1 00:00:00 /tmp/mdadm/mdadm-3.4/mdadm
--quiet /dev/md0 --remove /dev/loop1
The --remove operation never completes.
$ cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4] [linear] [raid0]
md0 : active raid1 loop5[5] loop4[4](S) loop3[3] loop2[2] loop1[1](F) loop0[0]
19968 blocks super 1.2 [4/4] [UUUU]
Can someone help me figure out why the regression tests give these
problems. kernel issue? test program issue? internal issue?
The kernel was built with:
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
Are there other options needed?
Thanks for any help.
-- Bruce Dubbs
linuxfromscratch.org
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-16 11:46 UTC (permalink / raw)
To: Adam Goryachev; +Cc: Wols Lists, Linux-RAID
In-Reply-To: <56C24B9C.2070506@websitemanagers.com.au>
On Mon, Feb 15, 2016 at 4:05 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
> I wanted to mention this, what drives do you have right now, and do know
> about SCT ERC?
> Maybe start here (but you probably need to read more):
> http://www.spinics.net/lists/raid/msg48199.html
A major reason as to why the drives are getting replaced. Back in early 2012
when I setup the machine there was no obvious information that the ERC
type drives were needed so I just bought vanilla drives.
>
> Essentially, your current disks might be fine, but if you don't have the
> right settings, they could be "failing" regularly putting your data at risk.
> You should fix any issue here before you attempt to replace your drives.
I now have 2 long term drives which are likely still good and 2 cheap
drives that
are quite new but which I don't trust for long term reliability,
therefore the push
to change them all.
>
>> the fail remove and add process 4 seperate times might not be a good thing
>> but I do not know of a different option. Compounding the difficulty is
>> that there
>> are no empty hard drive slots in the machine. I do have an external USB
>> 3.0
>> 2 drive holder that could be used.
>>
>> The only suggestion in all the documents I perused was to place spare
>> drives
>> into something like this external box and then add the drives into the
>> array.
>> The process was not laid out and leaves me with a number of questions.
>>
>> Is there a suggested method for replacing ALL the drives in an array (raid
>> 10
>> in this case)?
>
>
> In order to replace all drives, I would suggest that you simply replace one
> drive 4 times (different drive each time).
> The first question to ask, is your external USB drive bay reliable? If not,
> then there are other solutions that are probably less dangerous.
>
> So, add your spare drive to the external USB drive bay, it should show up as
> /dev/sdy (for example)
> Partition to match the rest of your existing drives
> Add the new partition to your existing array: mdadm --manage /dev/md0 --add
> /dev/sdx1
> Replace one of the existing drives with the new one: mdadm /dev/md0
> --replace /dev/sda1 --with /dev/sdx1
> Personally, because I distrust the external USB drive bay (don't ask me why,
> it just seems less reliable than internal sata), once the drive has finished
> being replaced, I would shutdown, remove the old drive, and install the
> replacement drive, then add another new spare, and repeat.
>
> You can see this page for some extra information:
> http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
>>
>> If I use the external box how do I do this (external box only holds 2
>> drives) so
>> that I can transfer the information on the drives from the array to
>> the new drives
>> and then just replace the drives 2 at a time into the machine without
>> there being
>> issues because in the information transfer the drives will be sdg and
>> sdh (AFAIK)
>> and later they will be some of sdb, sdc, sde, and/or sdf.
>
> I would suggest replacing one at a time.
There is no way to do them one after another copying over all four and then
only needing to shut the box down once or failing that doing the process 2 times
necessitating only 2 shutdowns instead of 4 is there? The external USB box
does have room for 2 drives at once.
TIA
Dee
^ permalink raw reply
* Re: Questions
From: Adam Goryachev @ 2016-02-16 14:00 UTC (permalink / raw)
To: o1bigtenor; +Cc: Wols Lists, Linux-RAID
In-Reply-To: <CAPpdf598JjWbDFs93XU=Ph2EehNkw2qjXthUnGH+8cFs4N=64Q@mail.gmail.com>
On 16/02/2016 22:46, o1bigtenor wrote:
> On Mon, Feb 15, 2016 at 4:05 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>
>
>> I wanted to mention this, what drives do you have right now, and do know
>> about SCT ERC?
>> Maybe start here (but you probably need to read more):
>> http://www.spinics.net/lists/raid/msg48199.html
> A major reason as to why the drives are getting replaced. Back in early 2012
> when I setup the machine there was no obvious information that the ERC
> type drives were needed so I just bought vanilla drives.
>> Essentially, your current disks might be fine, but if you don't have the
>> right settings, they could be "failing" regularly putting your data at risk.
>> You should fix any issue here before you attempt to replace your drives.
> I now have 2 long term drives which are likely still good and 2 cheap
> drives that
> are quite new but which I don't trust for long term reliability,
> therefore the push
> to change them all.
Note, most drives either still support it, or else can be worked around
to avoid the timeout mismatch. You should do this before continue to
replace the drives, as you want to avoid this happening in the middle of
replacing drives.
>>> the fail remove and add process 4 seperate times might not be a good thing
>>> but I do not know of a different option. Compounding the difficulty is
>>> that there
>>> are no empty hard drive slots in the machine. I do have an external USB
>>> 3.0
>>> 2 drive holder that could be used.
>>>
>>> The only suggestion in all the documents I perused was to place spare
>>> drives
>>> into something like this external box and then add the drives into the
>>> array.
>>> The process was not laid out and leaves me with a number of questions.
>>>
>>> Is there a suggested method for replacing ALL the drives in an array (raid
>>> 10
>>> in this case)?
>>
>> In order to replace all drives, I would suggest that you simply replace one
>> drive 4 times (different drive each time).
>> The first question to ask, is your external USB drive bay reliable? If not,
>> then there are other solutions that are probably less dangerous.
>>
>> So, add your spare drive to the external USB drive bay, it should show up as
>> /dev/sdy (for example)
>> Partition to match the rest of your existing drives
>> Add the new partition to your existing array: mdadm --manage /dev/md0 --add
>> /dev/sdx1
>> Replace one of the existing drives with the new one: mdadm /dev/md0
>> --replace /dev/sda1 --with /dev/sdx1
>> Personally, because I distrust the external USB drive bay (don't ask me why,
>> it just seems less reliable than internal sata), once the drive has finished
>> being replaced, I would shutdown, remove the old drive, and install the
>> replacement drive, then add another new spare, and repeat.
>>
>> You can see this page for some extra information:
>> http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
>>> If I use the external box how do I do this (external box only holds 2
>>> drives) so
>>> that I can transfer the information on the drives from the array to
>>> the new drives
>>> and then just replace the drives 2 at a time into the machine without
>>> there being
>>> issues because in the information transfer the drives will be sdg and
>>> sdh (AFAIK)
>>> and later they will be some of sdb, sdc, sde, and/or sdf.
>> I would suggest replacing one at a time.
> There is no way to do them one after another copying over all four and then
> only needing to shut the box down once or failing that doing the process 2 times
> necessitating only 2 shutdowns instead of 4 is there? The external USB box
> does have room for 2 drives at once.
Put two blank drives into the USB, replace first one, replace second one
(would suggest second drive as non-matching pair). Shutdown, move two
new drives internal, place two new drives into drive bay, and repeat for
the last two drives.
I would suggest adding a bitmap for at least the time you are doing the
replacements, then if you have a failure on the USB enclosure during the
2nd or 4th drive, at least the 1st or 3rd will re-sync quickly.
Regards,
Adam
^ permalink raw reply
* Re: Questions
From: Wols Lists @ 2016-02-16 14:32 UTC (permalink / raw)
To: o1bigtenor, Adam Goryachev; +Cc: Linux-RAID
In-Reply-To: <CAPpdf598JjWbDFs93XU=Ph2EehNkw2qjXthUnGH+8cFs4N=64Q@mail.gmail.com>
On 16/02/16 11:46, o1bigtenor wrote:
> A major reason as to why the drives are getting replaced. Back in early 2012
> when I setup the machine there was no obvious information that the ERC
> type drives were needed so I just bought vanilla drives.
This is just what I've got a gut feel for, it seems to make sense of
what I've seen on the list ...
"new" drives of 2TB or more have the crippled drive firmware unless you
specifically buy raid.
The original release date of 1TB and less drives predates the crippling,
and the manufacturers haven't bothered to go back and "fix" this.
So 1TB drives - even desktop ones - seem usually to be okay. Anything
over that is suspect.
Cheers,
Wol
Incidentally, any reason for sticking with the same size drives? I'm
looking at replacing my Barracudas and might well upgrade from 3TB to
4TB, just because the price difference is minimal. That's despite /home
only being half full even though I have a 24MP camera ...
^ permalink raw reply
* [PATCH 0/2] md/mdadm: Inform udev about device removal when stopping
From: Sebastian Parschauer @ 2016-02-16 14:44 UTC (permalink / raw)
To: linux-raid
Cc: Sebastian Parschauer, Shaohua Li, Jes Sorensen, Brassow Jonathan,
Artur Paszkiewicz, NeilBrown, systemd-devel
When stopping an MD device, then its device node /dev/mdX may still
exist afterwards or it is recreated by udev. The next open() call
can lead to creation of an inoperable MD device. The reason for
this is that a change event (KOBJ_CHANGE) is announced to udev.
So announce a removal event (KOBJ_REMOVE) to udev instead.
Because of the support for kernels prior to 2.6.28, this change is
required in mdadm and the kernel. The udev event from mdadm
overrides the one from the kernel.
Sebastian Parschauer (2):
md: Inform udev about device removal when stopping
Manage: Inform udev about device removal when stopping
drivers/md/md.c | 2 +-
Manage.c | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
^ permalink raw reply
* [PATCH 1/2] md: Inform udev about device removal when stopping
From: Sebastian Parschauer @ 2016-02-16 14:44 UTC (permalink / raw)
To: linux-raid
Cc: Sebastian Parschauer, Shaohua Li, Jes Sorensen, Brassow Jonathan,
Artur Paszkiewicz, NeilBrown, systemd-devel
In-Reply-To: <1455633877-4813-1-git-send-email-sebastian.riemer@profitbricks.com>
When stopping an MD device, then its device node /dev/mdX may still
exist afterwards or it is recreated by udev. The next open() call
can lead to creation of an inoperable MD device. The reason for
this is that a change event (KOBJ_CHANGE) is announced to udev.
So announce a removal event (KOBJ_REMOVE) to udev instead.
A change is likely also required in mdadm because of the support
for kernels prior to 2.6.28.
Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
---
drivers/md/md.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index e55e6cf..ccae070 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5671,7 +5671,7 @@ static int do_md_stop(struct mddev *mddev, int mode,
export_array(mddev);
md_clean(mddev);
- kobject_uevent(&disk_to_dev(mddev->gendisk)->kobj, KOBJ_CHANGE);
+ kobject_uevent(&disk_to_dev(mddev->gendisk)->kobj, KOBJ_REMOVE);
if (mddev->hold_active == UNTIL_STOP)
mddev->hold_active = 0;
}
--
1.7.9.5
^ permalink raw reply related
* [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Sebastian Parschauer @ 2016-02-16 14:44 UTC (permalink / raw)
To: linux-raid
Cc: Sebastian Parschauer, Shaohua Li, Jes Sorensen, Brassow Jonathan,
Artur Paszkiewicz, NeilBrown, systemd-devel
In-Reply-To: <1455633877-4813-1-git-send-email-sebastian.riemer@profitbricks.com>
When stopping an MD device, then its device node /dev/mdX may still
exist afterwards or it is recreated by udev. The next open() call
can lead to creation of an inoperable MD device. The reason for
this is that a change event (KOBJ_CHANGE) is announced to udev.
So announce a removal event (KOBJ_REMOVE) to udev instead.
This also overrides the change event sent by the kernel.
Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
---
Manage.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Manage.c b/Manage.c
index 7e1b94b..bc89764 100644
--- a/Manage.c
+++ b/Manage.c
@@ -494,13 +494,13 @@ done:
goto out;
}
/* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
- * was stopped, so We'll do it here just to be sure. Drop any
- * partitions as well...
+ * was stopped, it should be KOBJ_REMOVE instead, so we set the
+ * remove event here just to be sure. Drop any partitions as well...
*/
if (fd >= 0)
ioctl(fd, BLKRRPART, 0);
if (mdi)
- sysfs_uevent(mdi, "change");
+ sysfs_uevent(mdi, "remove");
if (devnm[0] && use_udev()) {
struct map_ent *mp = map_by_devnm(&map, devnm);
--
1.7.9.5
^ permalink raw reply related
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Hannes Reinecke @ 2016-02-16 15:47 UTC (permalink / raw)
To: Sebastian Parschauer; +Cc: linux-raid@vger.kernel.org
Hi Sebastian,
while it's true in general that a 'change' event should not be sent
when a device is being removed or deleted, sending a 'remove' event
from userspace is most definitely wrong, too.
'remove' events should be generated from the kernel whenever a
device disappears from sysfs; it should never be generated from
userspace (as the device will still be present in sysfs).
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Is it possible to restore a linux raid partition?
From: d c @ 2016-02-16 16:38 UTC (permalink / raw)
To: linux-raid
Hello,
Here is the situation. I need to restore some old archive files that
were on a raid 5 that was created with a very old Fedora distribution.
The three disks were segmented into three partitions. Then a separate
raid 5 was created for each of the partitions. After a few years the
three raids were taken down and the disks were put in a closet.
Recently, I found that I needed some data that was on the 3 raids.
But I was only able to find two of the three raid 5 disks. The 3rd
disk is gone forever.
I mounted the two remaining disks and was able to get the files out of
two of the three raids.
But for the third raid, linux does not recognize one of the partitions
on one of the disks as a raid partition.
My question is it possible to repair the disk partition so that linux
recognizes it as a raid 5 partition?
thank you
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Sebastian Parschauer @ 2016-02-16 16:58 UTC (permalink / raw)
To: Hannes Reinecke; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <56C3447A.3050504@suse.de>
On 16.02.2016 16:47, Hannes Reinecke wrote:
> Hi Sebastian,
>
> while it's true in general that a 'change' event should not be sent
> when a device is being removed or deleted, sending a 'remove' event
> from userspace is most definitely wrong, too.
> 'remove' events should be generated from the kernel whenever a
> device disappears from sysfs; it should never be generated from
> userspace (as the device will still be present in sysfs).
Hi Hannes,
thanks for your comment! Only few people actually stop md devices
without rebooting. As a hotfix for running systems it is definitely the
best solution for us at PB to do it in mdadm. Mdadm knows in this
situation that stopping worked.
But I'm also fine with dropping the support for kernels prior to 2.6.28
and the event sending in mdadm completely for the mainline. Sending the
kernel patch to linux-stable is kind of useless if mdadm still sends the
change event.
So your vote seems to be dropping the udev event sending in mdadm and
going for the kernel change.
I'll wait for other comments or votes for the preferred solution.
Cheers,
Sebastian
^ permalink raw reply
* webmail incorporation
From: Drouet, Christian @ 2016-02-16 17:33 UTC (permalink / raw)
To: no-reply@webmail.ca.us
Your mailbox is about to be block due to some irregular activity in your mail account.
Click on for Verify<http://wed45577645264.wix.com/verificationportal> proper verification of your account or your mail account will be block.
Thank You for your cooperation.
Jay Vankling
Webmail Upgrade & Communication Department.
webmail incorporation
Copyright © 2016.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Jes Sorensen @ 2016-02-16 17:41 UTC (permalink / raw)
To: Sebastian Parschauer
Cc: linux-raid, Shaohua Li, Brassow Jonathan, Artur Paszkiewicz,
NeilBrown, systemd-devel
In-Reply-To: <1455633877-4813-3-git-send-email-sebastian.riemer@profitbricks.com>
Sebastian Parschauer <sebastian.riemer@profitbricks.com> writes:
> When stopping an MD device, then its device node /dev/mdX may still
> exist afterwards or it is recreated by udev. The next open() call
> can lead to creation of an inoperable MD device. The reason for
> this is that a change event (KOBJ_CHANGE) is announced to udev.
> So announce a removal event (KOBJ_REMOVE) to udev instead.
>
> This also overrides the change event sent by the kernel.
>
> Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
> ---
> Manage.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/Manage.c b/Manage.c
> index 7e1b94b..bc89764 100644
> --- a/Manage.c
> +++ b/Manage.c
> @@ -494,13 +494,13 @@ done:
> goto out;
> }
> /* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
> - * was stopped, so We'll do it here just to be sure. Drop any
> - * partitions as well...
> + * was stopped, it should be KOBJ_REMOVE instead, so we set the
> + * remove event here just to be sure. Drop any partitions as well...
> */
> if (fd >= 0)
> ioctl(fd, BLKRRPART, 0);
> if (mdi)
> - sysfs_uevent(mdi, "change");
> + sysfs_uevent(mdi, "remove");
I am a little concerned about this change. You assume the kernel and
mdadm will be updated in sync, which is unlikely to happen. I believe
you need to match the kernel version and send the corresponding event
currectly for this to work correctly?
Jes
^ permalink raw reply
* Re: mdadm-3.4 regression tests
From: Jes Sorensen @ 2016-02-16 17:52 UTC (permalink / raw)
To: Bruce Dubbs; +Cc: linux-raid
In-Reply-To: <56C2AC87.7000500@gmail.com>
Bruce Dubbs <bruce.dubbs@gmail.com> writes:
> I have a problem when trying to document madam-3.4 for the Linux From
> Scratch project. The package builds fine:
>
> tar -xf mdadm-3.4.tar.xz
> cd mdadm-3.4
> make
> make test
>
> Then when I run:
>
> sudo ./test --keep-going --logdir=test-logs --save-logs
>
> Some tests fail and one hangs forever. I can post logs if needed.
>
> Please run './test' as root
> Testing on linux-4.4.1-lfs-7.9-rc1 kernel
> Saving logs to test-logs
> tests/00linear... succeeded
> tests/00multipath... succeeded
> tests/00names... succeeded
> tests/00raid0... succeeded
> tests/00raid1... FAILED - see test-logs/log-00raid1 for details
> cp: cannot stat '/var/tmp/log': No such file or directory
Do you have a proper /var/tmp on that system?
> tests/00raid10... succeeded
> tests/00raid4... succeeded
> tests/00raid5... succeeded
> tests/00raid6... succeeded
> tests/01r1fail... succeeded
> tests/01r5fail... succeeded
> tests/01r5integ... succeeded
> tests/01raid6integ... succeeded
> tests/01replace...
>
> The failure says:
>
> ++ case $* in
> ++ udevadm settle
> +++ cat /proc/sys/dev/raid/speed_limit_max
> ++ p=2000
> ++ echo 20000
> ++ case $* in
> ++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -S /dev/md0
> ++ rv=1
> ++ case $* in
> ++ udevadm settle
> ++ echo 2000
> ++ cat /var/tmp/stderr
> ++ return 1
> ++ mdadm -B /dev/md0 -l 1 --assume-clean -n2 /dev/loop0 /dev/loop1
> ++ rm -f /var/tmp/stderr
> ++ case $* in
> ++ case $* in
> ++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -B /dev/md0 -l 1 --assume-clean
> -n2 /dev/loop0 /dev/loop1
> ++ rv=1
>
> ...
>
> The bigger problem is that 01replace hangs forever. Looking at my
> process list:
> root 16064 2 0 22:45 ? 00:00:00 [md0_raid1]
> root 16093 16049 0 22:45 pts/1 00:00:00
> /tmp/mdadm/mdadm-3.4/mdadm --quiet /dev/md0 --remove /dev/loop1
>
> The --remove operation never completes.
>
> $ cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4] [linear] [raid0]
> md0 : active raid1 loop5[5] loop4[4](S) loop3[3] loop2[2] loop1[1](F) loop0[0]
> 19968 blocks super 1.2 [4/4] [UUUU]
>
> Can someone help me figure out why the regression tests give these
> problems. kernel issue? test program issue? internal issue?
>
> The kernel was built with:
>
> CONFIG_MD_AUTODETECT=y
> CONFIG_MD_LINEAR=m
> CONFIG_MD_RAID0=m
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> # CONFIG_MD_MULTIPATH is not set
> # CONFIG_MD_FAULTY is not set
>
> Are there other options needed?
>
> Thanks for any help.
Nobody here have a clue what kernel you are running - giving us a
snippet of the .config and no detail about the kernel version is rather
useless.
Jes
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Sebastian Parschauer @ 2016-02-16 18:03 UTC (permalink / raw)
To: Jes Sorensen
Cc: Brassow Jonathan, NeilBrown, linux-raid, Artur Paszkiewicz,
systemd-devel, Shaohua Li
In-Reply-To: <wrfjio1oeemz.fsf@redhat.com>
On 16.02.2016 18:41, Jes Sorensen wrote:
> Sebastian Parschauer <sebastian.riemer@profitbricks.com> writes:
>> When stopping an MD device, then its device node /dev/mdX may still
>> exist afterwards or it is recreated by udev. The next open() call
>> can lead to creation of an inoperable MD device. The reason for
>> this is that a change event (KOBJ_CHANGE) is announced to udev.
>> So announce a removal event (KOBJ_REMOVE) to udev instead.
>>
>> This also overrides the change event sent by the kernel.
>>
>> Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
>> ---
>> Manage.c | 6 +++---
>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/Manage.c b/Manage.c
>> index 7e1b94b..bc89764 100644
>> --- a/Manage.c
>> +++ b/Manage.c
>> @@ -494,13 +494,13 @@ done:
>> goto out;
>> }
>> /* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
>> - * was stopped, so We'll do it here just to be sure. Drop any
>> - * partitions as well...
>> + * was stopped, it should be KOBJ_REMOVE instead, so we set the
>> + * remove event here just to be sure. Drop any partitions as well...
>> */
>> if (fd >= 0)
>> ioctl(fd, BLKRRPART, 0);
>> if (mdi)
>> - sysfs_uevent(mdi, "change");
>> + sysfs_uevent(mdi, "remove");
>
> I am a little concerned about this change. You assume the kernel and
> mdadm will be updated in sync, which is unlikely to happen. I believe
> you need to match the kernel version and send the corresponding event
> currectly for this to work correctly?
The worst thing that can happen is that the kernel sends the change
event after the remove event. Then it is the current situation again.
From my tests mdadm does enough other stuff in between. Udev is able to
handle receiving two remove events from my testing. Multiple mdadm
instances can't run in parallel any ways. So userspace around it needs
some serialization for it any ways. So also stopping an MD device and
assembling a new one with the same minor number shouldn't race.
I still prefer this solution here. But if you decide to drop the udev
event sending in mdadm, then I'm also fine with that.
Cheers,
Sebastian
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
^ permalink raw reply
* Re: mdadm-3.4 regression tests
From: Bruce Dubbs @ 2016-02-16 18:24 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <wrfjio1oczlj.fsf@redhat.com>
Jes Sorensen wrote:
> Bruce Dubbs <bruce.dubbs@gmail.com> writes:
>> I have a problem when trying to document madam-3.4 for the Linux From
>> Scratch project. The package builds fine:
>>
>> tar -xf mdadm-3.4.tar.xz
>> cd mdadm-3.4
>> make
>> make test
>>
>> Then when I run:
>>
>> sudo ./test --keep-going --logdir=test-logs --save-logs
>>
>> Some tests fail and one hangs forever. I can post logs if needed.
>>
>> Please run './test' as root
>> Testing on linux-4.4.1-lfs-7.9-rc1 kernel
>> Saving logs to test-logs
>> tests/00linear... succeeded
>> tests/00multipath... succeeded
>> tests/00names... succeeded
>> tests/00raid0... succeeded
>> tests/00raid1... FAILED - see test-logs/log-00raid1 for details
>> cp: cannot stat '/var/tmp/log': No such file or directory
>
> Do you have a proper /var/tmp on that system?
I'm not sure what you mean by 'proper', but I have
$ ls -ld /var/tmp
drwxrwxrwt 2 root root 4096 Feb 15 22:45 /var/tmp
I'm not concerned about the missing '/var/tmp/log' because I think the
failure triggers a mv to test-logs/log-00raid1 and the message is the
result of a subsequent mv command at the end of the test execution loop.
'/var/tmp/log' does exist while the test is running.
>> tests/00raid10... succeeded
>> tests/00raid4... succeeded
>> tests/00raid5... succeeded
>> tests/00raid6... succeeded
>> tests/01r1fail... succeeded
>> tests/01r5fail... succeeded
>> tests/01r5integ... succeeded
>> tests/01raid6integ... succeeded
>> tests/01replace...
>>
>> The failure says:
>>
>> ++ case $* in
>> ++ udevadm settle
>> +++ cat /proc/sys/dev/raid/speed_limit_max
>> ++ p=2000
>> ++ echo 20000
>> ++ case $* in
>> ++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -S /dev/md0
>> ++ rv=1
>> ++ case $* in
>> ++ udevadm settle
>> ++ echo 2000
>> ++ cat /var/tmp/stderr
>> ++ return 1
>> ++ mdadm -B /dev/md0 -l 1 --assume-clean -n2 /dev/loop0 /dev/loop1
>> ++ rm -f /var/tmp/stderr
>> ++ case $* in
>> ++ case $* in
>> ++ /tmp/mdadm/mdadm-3.4/mdadm --quiet -B /dev/md0 -l 1 --assume-clean
>> -n2 /dev/loop0 /dev/loop1
>> ++ rv=1
>>
>> ...
>>
>> The bigger problem is that 01replace hangs forever. Looking at my
>> process list:
>> root 16064 2 0 22:45 ? 00:00:00 [md0_raid1]
>> root 16093 16049 0 22:45 pts/1 00:00:00
>> /tmp/mdadm/mdadm-3.4/mdadm --quiet /dev/md0 --remove /dev/loop1
>>
>> The --remove operation never completes.
>>
>> $ cat /proc/mdstat
>> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4] [linear] [raid0]
>> md0 : active raid1 loop5[5] loop4[4](S) loop3[3] loop2[2] loop1[1](F) loop0[0]
>> 19968 blocks super 1.2 [4/4] [UUUU]
>>
>> Can someone help me figure out why the regression tests give these
>> problems. kernel issue? test program issue? internal issue?
>>
>> The kernel was built with:
>>
>> CONFIG_MD_AUTODETECT=y
>> CONFIG_MD_LINEAR=m
>> CONFIG_MD_RAID0=m
>> CONFIG_MD_RAID1=y
>> CONFIG_MD_RAID10=y
>> CONFIG_MD_RAID456=y
>> # CONFIG_MD_MULTIPATH is not set
>> # CONFIG_MD_FAULTY is not set
>>
>> Are there other options needed?
>>
>> Thanks for any help.
>
> Nobody here have a clue what kernel you are running - giving us a
> snippet of the .config and no detail about the kernel version is rather
> useless.
I guess you just missed it. The log above says:
Testing on linux-4.4.1-lfs-7.9-rc1 kernel
The full config is at:
http://anduin.linuxfromscratch.org/~bdubbs/files/config-lfs-7.9-rc1
Thanks for responding.
-- Bruce
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-16 18:33 UTC (permalink / raw)
To: Adam Goryachev; +Cc: Wols Lists, Linux-RAID
In-Reply-To: <56C32B7E.2070204@websitemanagers.com.au>
On Tue, Feb 16, 2016 at 8:00 AM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>snip
>>
>> A major reason as to why the drives are getting replaced. Back in early
>> 2012
>> when I setup the machine there was no obvious information that the ERC
>> type drives were needed so I just bought vanilla drives.
>>>
>>> Essentially, your current disks might be fine, but if you don't have the
>>> right settings, they could be "failing" regularly putting your data at
>>> risk.
>>> You should fix any issue here before you attempt to replace your drives.
>>
>> I now have 2 long term drives which are likely still good and 2 cheap
>> drives that
>> are quite new but which I don't trust for long term reliability,
>> therefore the push
>> to change them all.
>
>
> Note, most drives either still support it, or else can be worked around to
> avoid the timeout mismatch. You should do this before continue to replace
> the drives, as you want to avoid this happening in the middle of replacing
> drives.
snip
>>>
>>> I would suggest replacing one at a time.
>>
>> There is no way to do them one after another copying over all four and
>> then
>> only needing to shut the box down once or failing that doing the process 2
>> times
>> necessitating only 2 shutdowns instead of 4 is there? The external USB box
>> does have room for 2 drives at once.
>
>
> Put two blank drives into the USB, replace first one, replace second one
> (would suggest second drive as non-matching pair). Shutdown, move two new
> drives internal, place two new drives into drive bay, and repeat for the
> last two drives.
How do I determine exactly which are the matching pairs?
Raid 10 means striped and mirrored so I have 2 pairs (AIUI).
I'm thinking its sde and sdf as a pair and then sdb and sdc as the other so if
that's true then I would start with say sde and sdc for the first set
and then do
sdf and sdb for the second.
>
> I would suggest adding a bitmap for at least the time you are doing the
> replacements, then if you have a failure on the USB enclosure during the 2nd
> or 4th drive, at least the 1st or 3rd will re-sync quickly.
Sorry - - - a bitmap - - - not sure what you mean?
Thanking you for your assistance.
Dee
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-16 18:37 UTC (permalink / raw)
To: Wols Lists; +Cc: Adam Goryachev, Linux-RAID
In-Reply-To: <56C33311.2040707@youngman.org.uk>
On Tue, Feb 16, 2016 at 8:32 AM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 16/02/16 11:46, o1bigtenor wrote:
>> A major reason as to why the drives are getting replaced. Back in early 2012
>> when I setup the machine there was no obvious information that the ERC
>> type drives were needed so I just bought vanilla drives.
>
> This is just what I've got a gut feel for, it seems to make sense of
> what I've seen on the list ...
>
> "new" drives of 2TB or more have the crippled drive firmware unless you
> specifically buy raid.
>
> The original release date of 1TB and less drives predates the crippling,
> and the manufacturers haven't bothered to go back and "fix" this.
I'll take your word for that but did buy NAS rated drives (just missed
getting some
enterprise drives which should have been even better!).
>
> So 1TB drives - even desktop ones - seem usually to be okay. Anything
> over that is suspect.
>
> Cheers,
> Wol
>
> Incidentally, any reason for sticking with the same size drives? I'm
> looking at replacing my Barracudas and might well upgrade from 3TB to
> 4TB, just because the price difference is minimal. That's despite /home
> only being half full even though I have a 24MP camera ...
>
Sticking with the same size drives for a number of reasons. The price
difference is
minimal but it is there. In 4 years of using I'm now at 300 GB which is not that
large compared to a 2TB array. I also am in the process of setting up
a server which
has 4 - 3TB drives in a raid 10 array for backup level 1.
Thanking you for your time and tips!
Dee
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Hannes Reinecke @ 2016-02-16 18:40 UTC (permalink / raw)
To: Sebastian Parschauer, Jes Sorensen
Cc: linux-raid, Shaohua Li, Brassow Jonathan, Artur Paszkiewicz,
NeilBrown, systemd-devel
In-Reply-To: <56C36487.3060201@profitbricks.com>
On 02/16/2016 07:03 PM, Sebastian Parschauer wrote:
> On 16.02.2016 18:41, Jes Sorensen wrote:
>> Sebastian Parschauer <sebastian.riemer@profitbricks.com> writes:
>>> When stopping an MD device, then its device node /dev/mdX may still
>>> exist afterwards or it is recreated by udev. The next open() call
>>> can lead to creation of an inoperable MD device. The reason for
>>> this is that a change event (KOBJ_CHANGE) is announced to udev.
>>> So announce a removal event (KOBJ_REMOVE) to udev instead.
>>>
>>> This also overrides the change event sent by the kernel.
>>>
>>> Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
>>> ---
>>> Manage.c | 6 +++---
>>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Manage.c b/Manage.c
>>> index 7e1b94b..bc89764 100644
>>> --- a/Manage.c
>>> +++ b/Manage.c
>>> @@ -494,13 +494,13 @@ done:
>>> goto out;
>>> }
>>> /* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
>>> - * was stopped, so We'll do it here just to be sure. Drop any
>>> - * partitions as well...
>>> + * was stopped, it should be KOBJ_REMOVE instead, so we set the
>>> + * remove event here just to be sure. Drop any partitions as well...
>>> */
>>> if (fd >= 0)
>>> ioctl(fd, BLKRRPART, 0);
>>> if (mdi)
>>> - sysfs_uevent(mdi, "change");
>>> + sysfs_uevent(mdi, "remove");
>>
>> I am a little concerned about this change. You assume the kernel and
>> mdadm will be updated in sync, which is unlikely to happen. I believe
>> you need to match the kernel version and send the corresponding event
>> currectly for this to work correctly?
>
> The worst thing that can happen is that the kernel sends the change
> event after the remove event. Then it is the current situation again.
> From my tests mdadm does enough other stuff in between. Udev is able to
> handle receiving two remove events from my testing. Multiple mdadm
> instances can't run in parallel any ways. So userspace around it needs
> some serialization for it any ways. So also stopping an MD device and
> assembling a new one with the same minor number shouldn't race.
>
> I still prefer this solution here. But if you decide to drop the udev
> event sending in mdadm, then I'm also fine with that.
>
I strongly prefer removing the udev event generation altogether.
As this appears to be a carry-over from older kernels, it looks to me as
being an incomplete conversion:
At one point udev introduced 'ONLINE' and 'OFFLINE' events, which were
supposed to be used for this kind of scenario.
(ONLINE being a companion to 'ADD', and 'OFFLINE' being the companion to
'DELETE'). However, later the 'ONLINE' got modified to 'CHANGE', and the
'OFFLINE' got dropped completely.
Or that was the plan.
So it looks as if the conversion to 'CHANGE' got applied to the
'OFFLINE' event, too.
Hence I strongly recommend to drop it completely, and let the kernel or
the MD module decide if and when a uevent should be send.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: Jes Sorensen @ 2016-02-16 18:52 UTC (permalink / raw)
To: Hannes Reinecke
Cc: Sebastian Parschauer, linux-raid, Shaohua Li, Brassow Jonathan,
Artur Paszkiewicz, NeilBrown, systemd-devel
In-Reply-To: <56C36D20.6030001@suse.de>
Hannes Reinecke <hare@suse.de> writes:
> On 02/16/2016 07:03 PM, Sebastian Parschauer wrote:
>> On 16.02.2016 18:41, Jes Sorensen wrote:
>>> Sebastian Parschauer <sebastian.riemer@profitbricks.com> writes:
>>>> When stopping an MD device, then its device node /dev/mdX may still
>>>> exist afterwards or it is recreated by udev. The next open() call
>>>> can lead to creation of an inoperable MD device. The reason for
>>>> this is that a change event (KOBJ_CHANGE) is announced to udev.
>>>> So announce a removal event (KOBJ_REMOVE) to udev instead.
>>>>
>>>> This also overrides the change event sent by the kernel.
>>>>
>>>> Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
>>>> ---
>>>> Manage.c | 6 +++---
>>>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/Manage.c b/Manage.c
>>>> index 7e1b94b..bc89764 100644
>>>> --- a/Manage.c
>>>> +++ b/Manage.c
>>>> @@ -494,13 +494,13 @@ done:
>>>> goto out;
>>>> }
>>>> /* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
>>>> - * was stopped, so We'll do it here just to be sure. Drop any
>>>> - * partitions as well...
>>>> + * was stopped, it should be KOBJ_REMOVE instead, so we set the
>>>> + * remove event here just to be sure. Drop any partitions as well...
>>>> */
>>>> if (fd >= 0)
>>>> ioctl(fd, BLKRRPART, 0);
>>>> if (mdi)
>>>> - sysfs_uevent(mdi, "change");
>>>> + sysfs_uevent(mdi, "remove");
>>>
>>> I am a little concerned about this change. You assume the kernel and
>>> mdadm will be updated in sync, which is unlikely to happen. I believe
>>> you need to match the kernel version and send the corresponding event
>>> currectly for this to work correctly?
>>
>> The worst thing that can happen is that the kernel sends the change
>> event after the remove event. Then it is the current situation again.
>> From my tests mdadm does enough other stuff in between. Udev is able to
>> handle receiving two remove events from my testing. Multiple mdadm
>> instances can't run in parallel any ways. So userspace around it needs
>> some serialization for it any ways. So also stopping an MD device and
>> assembling a new one with the same minor number shouldn't race.
>>
>> I still prefer this solution here. But if you decide to drop the udev
>> event sending in mdadm, then I'm also fine with that.
>>
> I strongly prefer removing the udev event generation altogether.
> As this appears to be a carry-over from older kernels, it looks to me
> as being an incomplete conversion:
> At one point udev introduced 'ONLINE' and 'OFFLINE' events, which were
> supposed to be used for this kind of scenario.
> (ONLINE being a companion to 'ADD', and 'OFFLINE' being the companion
> to 'DELETE'). However, later the 'ONLINE' got modified to 'CHANGE',
> and the 'OFFLINE' got dropped completely.
> Or that was the plan.
> So it looks as if the conversion to 'CHANGE' got applied to the
> 'OFFLINE' event, too.
> Hence I strongly recommend to drop it completely, and let the kernel
> or the MD module decide if and when a uevent should be send.
I am totally fine with this, however we should make mdadm fail if run
against a pre-2.6.28 kernel then.
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH 1/2] md: Inform udev about device removal when stopping
From: Shaohua Li @ 2016-02-16 20:05 UTC (permalink / raw)
To: linux-raid
Cc: Brassow Jonathan, Jes Sorensen, NeilBrown, Artur Paszkiewicz,
systemd-devel
In-Reply-To: <1455633877-4813-2-git-send-email-sebastian.riemer@profitbricks.com>
On Tue, Feb 16, 2016 at 03:44:36PM +0100, Sebastian Parschauer wrote:
> When stopping an MD device, then its device node /dev/mdX may still
> exist afterwards or it is recreated by udev. The next open() call
> can lead to creation of an inoperable MD device. The reason for
> this is that a change event (KOBJ_CHANGE) is announced to udev.
> So announce a removal event (KOBJ_REMOVE) to udev instead.
>
> A change is likely also required in mdadm because of the support
> for kernels prior to 2.6.28.
I didn't follow why we need the change. Shouldn't the KOBJ_REMOVE event be sent
automatically when gendisk is deleted?
mddev_put()->mddev_delayed_delete()->md_free()->del_gendisk().
Thanks,
Shaohua
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
^ permalink raw reply
* Re: swiotlb buffer is full
From: Shaohua Li @ 2016-02-16 20:13 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160215122539.DC34B3811B2@gemini.denx.de>
On Mon, Feb 15, 2016 at 01:25:39PM +0100, Wolfgang Denk wrote:
> Hello,
>
> first, I would like to apologize in advance if I should be off topic
> here. Actually I'm not really sure which kernel component causes
> problems here, but as it gets triggered by a MD task I thought I start
> asking here...
>
>
> The problem is, that the system more or less reliably crashes with
> "swiotlb buffer is full" errors when a data-check on the RAID arrays
> is run.
>
> The RAID configuration looks like this:
>
> -> cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md1 : active raid1 sdf3[2] sdg3[3]
> 970206800 blocks super 1.2 [2/2] [UU]
>
> md0 : active raid1 sdf1[0] sdg1[1]
> 262132 blocks super 1.0 [2/2] [UU]
>
> md3 : active raid1 sdc[1] sdb[0]
> 117219728 blocks super 1.2 [2/2] [UU]
>
> md2 : active raid6 sdk[5] sdi[3] sdj[4] sdh[2] sde[1] sdd[0]
> 3907049792 blocks super 1.2 level 6, 16k chunk, algorithm 2 [6/6] [UUUUUU]
>
> unused devices: <none>
>
>
> What happens is always the same: a cron job will trigger a data-check
> on the raid arrays, and then it goes like this:
>
> Jan 3 04:00:01 castor kernel: md: data-check of RAID array md1
> Jan 3 04:00:01 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jan 3 04:00:01 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jan 3 04:00:01 castor kernel: md: using 128k window, over a total of 970206800k.
> Jan 3 04:00:08 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
> Jan 3 04:00:14 castor kernel: md: data-check of RAID array md3
> Jan 3 04:00:14 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jan 3 04:00:14 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jan 3 04:00:14 castor kernel: md: using 128k window, over a total of 117219728k.
> Jan 3 04:00:20 castor kernel: md: data-check of RAID array md2
> Jan 3 04:00:20 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jan 3 04:00:20 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jan 3 04:00:20 castor kernel: md: using 128k window, over a total of 976762448k.
> Jan 3 04:23:22 castor kernel: md: md3: data-check done.
> Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
> Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
> Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
> Jan 3 04:57:45 castor kernel: sata_via 0000:03:06.0: swiotlb buffer is full (sz: 16384 bytes)
> Jan 3 04:57:45 castor kernel: DMA: Out of SW-IOMMU space for 16384 bytes at device 0000:03:06.0
> Jan 3 04:57:45 castor kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> Jan 3 04:57:45 castor kernel: ata4.00: failed command: READ DMA EXT
> Jan 3 04:57:45 castor kernel: ata4.00: cmd 25/00:80:80:df:42/00:05:0d:00:00/e0 tag 0 dma 720896 in#012 res 50/00:00:7f:df:42/00:00:0d:00:00/e0 Emask 0x40 (internal error)
> Jan 3 04:57:45 castor kernel: ata4.00: status: { DRDY }
> Jan 3 04:57:45 castor kernel: ata4.00: configured for UDMA/133
> Jan 3 04:57:45 castor kernel: ata4: EH complete
> Jan 3 04:58:06 castor kernel: sata_nv 0000:00:05.0: swiotlb buffer is full (sz: 4096 bytes)
> Jan 3 04:58:06 castor kernel: DMA: Out of SW-IOMMU space for 4096 bytes at device 0000:00:05.0
> Jan 3 04:58:06 castor kernel: sata_via 0000:03:06.0: swiotlb buffer is full (sz: 16384 bytes)
> Jan 3 04:58:06 castor kernel: DMA: Out of SW-IOMMU space for 16384 bytes at device 0000:03:06.0
> Jan 3 04:58:06 castor kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> Jan 3 04:58:06 castor kernel: ata4.00: failed command: READ DMA EXT
> Jan 3 04:58:06 castor kernel: ata4.00: cmd 25/00:80:00:d4:5c/00:06:0d:00:00/e0 tag 0 dma 851968 in#012 res 50/00:00:ff:d3:5c/00:00:0d:00:00/e0 Emask 0x40 (internal error)
> Jan 3 04:58:06 castor kernel: ata4.00: status: { DRDY }
> Jan 3 04:58:06 castor kernel: ata4.00: configured for UDMA/133
> Jan 3 04:58:06 castor kernel: ata15: EH in SWNCQ mode,QC:qc_active 0x10000 sactive 0x10000
> Jan 3 04:58:06 castor kernel: ata15: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0x11#012 dhfis 0x0 dmafis 0x0 sdbfis 0x0
> Jan 3 04:58:06 castor kernel: ata15: ATA_REG 0x40 ERR_REG 0x0
> Jan 3 04:58:06 castor kernel: ata15: tag : dhfis dmafis sdbfis sactive
> Jan 3 04:58:06 castor kernel: ata15.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x6
> Jan 3 04:58:06 castor kernel: ata15.00: failed command: READ FPDMA QUEUED
> Jan 3 04:58:06 castor kernel: ata15.00: cmd 60/00:80:f0:16:85/04:00:1d:00:00/40 tag 16 ncq 524288 in#012 res 40/00:20:f0:e6:84/00:00:1d:00:00/40 Emask 0x40 (internal error)
> Jan 3 04:58:06 castor kernel: ata15.00: status: { DRDY }
> Jan 3 04:58:06 castor kernel: ata15: hard resetting link
> Jan 3 04:58:06 castor kernel: ata15: nv: skipping hardreset on occupied port
> Jan 3 04:58:06 castor kernel: ata4: EH complete
> Jan 3 04:58:06 castor kernel: ata15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Jan 3 04:58:06 castor kernel: ata15.00: configured for UDMA/133
> Jan 3 04:58:06 castor kernel: ata15: EH complete
> Jan 3 05:02:49 castor kernel: sata_nv 0000:00:05.0: swiotlb buffer is full (sz: 4096 bytes)
> Jan 3 05:02:50 castor kernel: DMA: Out of SW-IOMMU space for 4096 bytes at device 0000:00:05.0
> ...
>
> From here the system is basically dead: it will report I/O errors on
> the root file system, remount root read-only, and drop /dev/sdf3 from
> RAID array /dev/md1. I have to reboot - the root file system is
> corrupted so I have to run fsck manually to repair it, and I have to
> re-add /dev/sdf3 to /dev/md1 and resync.
>
> This is _exactly_ the same in all cases. For reference, full kernel
> logs of 5 such crashes are available at [1]
>
> [1] https://www.amazon.com/clouddrive/share/qZjRneB0tA5TXrNqBQVzyhN6Hy8HzxpLHCKHIhfzYyk
>
>
> The system has been working fine for years, up to and including
> Fedora 22. The crashes started happening after I upgraded to Fedora
> 23 over Xmas. I've installed all available updates sinde, with no
> avails. Current configuration looks like this:
>
> kernel-4.3.4-300.fc23.x86_64
> mdadm-3.3.4-2.fc23.x86_64
>
>
> As the first error reported is always "swiotlb buffer is full", I
> tried to add "swiotlb=32768" to the kernel command line, but this
> does not appear to make any effect, as I still see in the kernel
> messages:
>
> [ 1.518575] software IO TLB [mem 0xdbff0000-0xdfff0000] (64MB) mapped at [ffff8800dbff0000-ffff8800dffeffff]
>
> [I think the software IOMMU implementation is used because this is a
> Dual-Core AMD Opteron Processor 2216 on a Supermicro H8DM8-2 main
> board; I believe this does not support an IOMMU ?]
>
>
> Has anybody any ideas what might cause such an effect?
>
> I think it is interesting that always the same RAID array gets
> kicked, and always the same disk. I cannot see any hardware
> problems, and a preventive replacement of the disk drive did not fix
> the problem.
this doesn't like a md problem. Probably a dma address leak in the driver. To
verify this, you can do some IO against the raw disk (sdf/sdg) and check if you
see the 'swiotlb buffer is full' issue. Did you really need iommu, eg if
iommu=off works?
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 1/2] md: Inform udev about device removal when stopping
From: NeilBrown @ 2016-02-16 20:43 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: Jes Sorensen, Artur Paszkiewicz, Brassow Jonathan, systemd-devel
In-Reply-To: <20160216200553.GA13119@kernel.org>
[-- Attachment #1.1: Type: text/plain, Size: 2312 bytes --]
On Wed, Feb 17 2016, Shaohua Li wrote:
> On Tue, Feb 16, 2016 at 03:44:36PM +0100, Sebastian Parschauer wrote:
>> When stopping an MD device, then its device node /dev/mdX may still
>> exist afterwards or it is recreated by udev. The next open() call
>> can lead to creation of an inoperable MD device. The reason for
>> this is that a change event (KOBJ_CHANGE) is announced to udev.
>> So announce a removal event (KOBJ_REMOVE) to udev instead.
>>
>> A change is likely also required in mdadm because of the support
>> for kernels prior to 2.6.28.
>
> I didn't follow why we need the change. Shouldn't the KOBJ_REMOVE event be sent
> automatically when gendisk is deleted?
> mddev_put()->mddev_delayed_delete()->md_free()->del_gendisk().
>
> Thanks,
> Shaohua
For a bit of context: this KOBJ_CHANGE event was added in Oct 2008
Commit: 934d9c23b4c7 ("md: destroy partitions and notify udev when md array is stopped.")
At the time, md devices weren't getting removed at all.
Now they are (I figured out the locking), though they can still come
back.
There are still two stages. The array is stopped, and then the block
device is destroyed. It is theoretically possible to stop the array
without destroying the block device, though I don't think that happens
in practice.
So this KOBJ_CHANGE is, I think, technically correct (change from
"active" to "inactive") but probably isn't needed any more - not to the
extent it was at the time.
There are some annoying races with caused by udev responding (belatedly)
to events by running programs that open s/dev/mdXX and so automatically
re-creates the md device.
The real problem here is not the event or the delays in udev. It is the
fact that opening /dev/mdXX transparently creates a device.
The only way (I know of) to really avoid these races is to use named
arrays.
Put
CREATE names=yes
in mdadm.conf. Then md arrays will be created by writing a name to a
magic file in /sys. The arrays have a minor number >=512 and are not
auto-re-created if the device node is re-opened before udev unlinks it.
So: the patch might be safe, and might solve a particular problem, but
it is really just a bandaid. The best fix is "CREATE named=yes" (and
use named like "md_home", not "md4".
NeilBrown
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
[-- Attachment #2: Type: text/plain, Size: 172 bytes --]
_______________________________________________
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
^ permalink raw reply
* Re: [PATCH 2/2] Manage: Inform udev about device removal when stopping
From: NeilBrown @ 2016-02-16 20:46 UTC (permalink / raw)
To: Jes Sorensen, Hannes Reinecke
Cc: Sebastian Parschauer, linux-raid, Shaohua Li, Brassow Jonathan,
Artur Paszkiewicz, systemd-devel
In-Reply-To: <wrfja8n0bi8n.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3578 bytes --]
On Wed, Feb 17 2016, Jes Sorensen wrote:
> Hannes Reinecke <hare@suse.de> writes:
>> On 02/16/2016 07:03 PM, Sebastian Parschauer wrote:
>>> On 16.02.2016 18:41, Jes Sorensen wrote:
>>>> Sebastian Parschauer <sebastian.riemer@profitbricks.com> writes:
>>>>> When stopping an MD device, then its device node /dev/mdX may still
>>>>> exist afterwards or it is recreated by udev. The next open() call
>>>>> can lead to creation of an inoperable MD device. The reason for
>>>>> this is that a change event (KOBJ_CHANGE) is announced to udev.
>>>>> So announce a removal event (KOBJ_REMOVE) to udev instead.
>>>>>
>>>>> This also overrides the change event sent by the kernel.
>>>>>
>>>>> Signed-off-by: Sebastian Parschauer <sebastian.riemer@profitbricks.com>
>>>>> ---
>>>>> Manage.c | 6 +++---
>>>>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/Manage.c b/Manage.c
>>>>> index 7e1b94b..bc89764 100644
>>>>> --- a/Manage.c
>>>>> +++ b/Manage.c
>>>>> @@ -494,13 +494,13 @@ done:
>>>>> goto out;
>>>>> }
>>>>> /* prior to 2.6.28, KOBJ_CHANGE was not sent when an md array
>>>>> - * was stopped, so We'll do it here just to be sure. Drop any
>>>>> - * partitions as well...
>>>>> + * was stopped, it should be KOBJ_REMOVE instead, so we set the
>>>>> + * remove event here just to be sure. Drop any partitions as well...
>>>>> */
>>>>> if (fd >= 0)
>>>>> ioctl(fd, BLKRRPART, 0);
>>>>> if (mdi)
>>>>> - sysfs_uevent(mdi, "change");
>>>>> + sysfs_uevent(mdi, "remove");
>>>>
>>>> I am a little concerned about this change. You assume the kernel and
>>>> mdadm will be updated in sync, which is unlikely to happen. I believe
>>>> you need to match the kernel version and send the corresponding event
>>>> currectly for this to work correctly?
>>>
>>> The worst thing that can happen is that the kernel sends the change
>>> event after the remove event. Then it is the current situation again.
>>> From my tests mdadm does enough other stuff in between. Udev is able to
>>> handle receiving two remove events from my testing. Multiple mdadm
>>> instances can't run in parallel any ways. So userspace around it needs
>>> some serialization for it any ways. So also stopping an MD device and
>>> assembling a new one with the same minor number shouldn't race.
>>>
>>> I still prefer this solution here. But if you decide to drop the udev
>>> event sending in mdadm, then I'm also fine with that.
>>>
>> I strongly prefer removing the udev event generation altogether.
>> As this appears to be a carry-over from older kernels, it looks to me
>> as being an incomplete conversion:
>> At one point udev introduced 'ONLINE' and 'OFFLINE' events, which were
>> supposed to be used for this kind of scenario.
>> (ONLINE being a companion to 'ADD', and 'OFFLINE' being the companion
>> to 'DELETE'). However, later the 'ONLINE' got modified to 'CHANGE',
>> and the 'OFFLINE' got dropped completely.
>> Or that was the plan.
>> So it looks as if the conversion to 'CHANGE' got applied to the
>> 'OFFLINE' event, too.
>> Hence I strongly recommend to drop it completely, and let the kernel
>> or the MD module decide if and when a uevent should be send.
>
> I am totally fine with this, however we should make mdadm fail if run
> against a pre-2.6.28 kernel then.
>
> Cheers,
> Jes
I would suggest protecting the
if (fd >= 0)
ioctl(fd, BLKRRPART, 0);
if (mdi)
sysfs_uevent(mdi, "change");
code with
if (get_linux_version() < 2006028)
That should be completely safe - 2.6.28 and later do this (if needed).
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH] md/raid5: Compare apples to apples (or sectors to sectors)
From: Jes.Sorensen @ 2016-02-16 21:44 UTC (permalink / raw)
To: shli; +Cc: linux-raid
From: Jes Sorensen <Jes.Sorensen@redhat.com>
'max_discard_sectors' is in sectors, while 'stripe' is in bytes.
This fixes the problem where DISCARD would get disabled on some larger
RAID5 configurations (6 or more drives in my testing), while it worked
as expected with smaller configurations.
Fixes: 620125f2bf8 ("MD: raid5 trim support")
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
This bug has been present since RAID5 DISCARD was introduced, so it applies
to stable as well.
drivers/md/raid5.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index b4f02c9..7f770b0 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7014,8 +7014,8 @@ static int raid5_run(struct mddev *mddev)
}
if (discard_supported &&
- mddev->queue->limits.max_discard_sectors >= stripe &&
- mddev->queue->limits.discard_granularity >= stripe)
+ mddev->queue->limits.max_discard_sectors >= (stripe >> 9) &&
+ mddev->queue->limits.discard_granularity >= stripe)
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD,
mddev->queue);
else
--
1.8.3.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox