* [md:devel 41/50] drivers/md/md.c:1663:33: sparse: restricted __le32 degrades to integer
From: kbuild test robot @ 2015-08-19 6:51 UTC (permalink / raw)
To: Song Liu; +Cc: kbuild-all, NeilBrown, Shaohua Li, linux-raid
tree: git://neil.brown.name/md devel
head: da63d1b483fa5a9604d65974eb442a110ee41e89
commit: 1a8a1e097abc6d2787ef6f2e915d5f18d42a1e6c [41/50] MD: add a new disk role to present write journal device
reproduce:
# apt-get install sparse
git checkout 1a8a1e097abc6d2787ef6f2e915d5f18d42a1e6c
make ARCH=x86_64 allmodconfig
make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
drivers/md/md.c:1501:34: sparse: cast to restricted __le64
>> drivers/md/md.c:1663:33: sparse: restricted __le32 degrades to integer
drivers/md/md.c:1794:40: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1794:40: expected unsigned long long [unsigned] [long] [long long] [usertype] <noident>
drivers/md/md.c:1794:40: got restricted __le64 [usertype] <noident>
drivers/md/md.c:1872:26: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1872:26: expected restricted __le64 [usertype] super_offset
drivers/md/md.c:1872:26: got unsigned long [unsigned] [usertype] sb_start
drivers/md/md.c:8980:39: sparse: restricted __le64 degrades to integer
vim +1663 drivers/md/md.c
1647 }
1648 if (mddev->level != LEVEL_MULTIPATH) {
1649 int role;
1650 if (rdev->desc_nr < 0 ||
1651 rdev->desc_nr >= le32_to_cpu(sb->max_dev)) {
1652 role = MD_DISK_ROLE_SPARE;
1653 rdev->desc_nr = -1;
1654 } else
1655 role = le16_to_cpu(sb->dev_roles[rdev->desc_nr]);
1656 switch(role) {
1657 case MD_DISK_ROLE_SPARE: /* spare */
1658 break;
1659 case MD_DISK_ROLE_FAULTY: /* faulty */
1660 set_bit(Faulty, &rdev->flags);
1661 break;
1662 case MD_DISK_ROLE_JOURNAL: /* journal device */
> 1663 if (!(sb->feature_map & MD_FEATURE_JOURNAL)) {
1664 /* journal device without journal feature */
1665 printk(KERN_WARNING
1666 "md: journal device provided without "
1667 "journal feature, ignoring the device\n");
1668 return -EINVAL;
1669 }
1670 set_bit(Journal, &rdev->flags);
1671 break;
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* [PATCH 2/2] udev: Work around race condition on array activation
From: Thomas Bächler @ 2015-08-18 11:51 UTC (permalink / raw)
To: linux-raid; +Cc: Thomas Bächler
In-Reply-To: <1439898693-936-1-git-send-email-thomas@archlinux.org>
When an array is activated, it is possible for the md/array_state property to be "clean",
while the array is not accessible. This leads to a situation where systemd is told to
activate the block device unit while it is not ready yet.
This causes a race condition in combination with the LVM2 udev rules: Since ID_FS_TYPE is not
set yet, LVM2 does not look at the device. When a change event occurs later, ID_FS_TYPE is set
to LVM2_member and LVM2 sets SYSTEMD_WANTS to call a service to activate the LVM volume.
However, systemd has already activated the device at that time and the change to SYSTEMD_WANTS
is ignored.
To solve this, in addition to checking md/array_state, also check the size property: If size
is 0, the array is not ready and should not be treated as such.
---
udev-md-raid-arrays.rules | 1 +
1 file changed, 1 insertion(+)
diff --git a/udev-md-raid-arrays.rules b/udev-md-raid-arrays.rules
index 90d1aa5..08f3cad 100644
--- a/udev-md-raid-arrays.rules
+++ b/udev-md-raid-arrays.rules
@@ -15,6 +15,7 @@ ENV{DEVTYPE}=="partition", GOTO="md_ignore_state"
ATTR{md/metadata_version}=="external:[A-Za-z]*", ATTR{md/array_state}=="inactive", GOTO="md_ignore_state"
TEST!="md/array_state", ENV{SYSTEMD_READY}="0", GOTO="md_end"
ATTR{md/array_state}=="|clear|inactive", ENV{SYSTEMD_READY}="0", GOTO="md_end"
+ATTR{size}=="0", ENV{SYSTEMD_READY}="0", GOTO="md_end"
LABEL="md_ignore_state"
ENV{SYSTEMD_READY}="1"
--
2.5.0
^ permalink raw reply related
* [PATCH 1/2] udev: Set SYSTEMD_READY to 1 when the array becomes active
From: Thomas Bächler @ 2015-08-18 11:51 UTC (permalink / raw)
To: linux-raid; +Cc: Thomas Bächler
In-Reply-To: <1439898693-936-1-git-send-email-thomas@archlinux.org>
In some cases, the rules set SYSTEMD_READY to 0 in order to tell systemd
to delay activation of the .device unit. It should also be set to 1 when
the array is ready.
---
udev-md-raid-arrays.rules | 2 ++
1 file changed, 2 insertions(+)
diff --git a/udev-md-raid-arrays.rules b/udev-md-raid-arrays.rules
index c95ec7b..90d1aa5 100644
--- a/udev-md-raid-arrays.rules
+++ b/udev-md-raid-arrays.rules
@@ -17,6 +17,8 @@ TEST!="md/array_state", ENV{SYSTEMD_READY}="0", GOTO="md_end"
ATTR{md/array_state}=="|clear|inactive", ENV{SYSTEMD_READY}="0", GOTO="md_end"
LABEL="md_ignore_state"
+ENV{SYSTEMD_READY}="1"
+
IMPORT{program}="BINDIR/mdadm --detail --export $devnode"
ENV{DEVTYPE}=="disk", ENV{MD_NAME}=="?*", SYMLINK+="disk/by-id/md-name-$env{MD_NAME}", OPTIONS+="string_escape=replace"
ENV{DEVTYPE}=="disk", ENV{MD_UUID}=="?*", SYMLINK+="disk/by-id/md-uuid-$env{MD_UUID}"
--
2.5.0
^ permalink raw reply related
* [PATCH 0/2] mdadm: Fix udev rule interaction with systemd
From: Thomas Bächler @ 2015-08-18 11:51 UTC (permalink / raw)
To: linux-raid
Good morning,
I discovered a problem with the interaction between systemd, mdadm and lvm: Every 5th boot or so, my lvm physical volume
on top of a raid1 array would not activate. This turned out to be a race condition with mdadm.
When activating the raid array, two uevents happen: an "ADD" event and a "CHANGE" event. In both cases, the mdadm udev
rules look at the md/array_state file in sysfs. However, I observed that an array with md/array_state == clean may still
be inaccessible. So, in order to analyze the issue, I looked at the contents of the "md/array_state" and the "size" sysfs
files. I discovered that during activation, my raid1 was in the following three states:
1) md/array_state: inactive, size: 0
2) md/array_state: clean, size: 0
3) md/array_state: clean, size: (the real array size)
So, if the ADD event happens in state 1) and the CHANGE event happens in state 3), everything is fine. However, sometimes the
ADD event happens in state 2) and the CHANGE event in state 3). In the latter case, the device is activated in systemd during
the ADD event, but udev internal settings like ID_FS_TYPE are not set (since the device is inaccessible). This causes the
problem described above.
What I don't know is if
a) the transition from state 1) to 3) should be atomic in the kernel.
b) state 2) is legal and this needs to be worked around in udev.
The first patch fixes a problem where ENV{SYSTEMD_READY} is never set to 1 after being set to 0.
The second patch checks whether the array has non-zero size before activating the systemd device unit (in case this is
not a kernel bug).
^ permalink raw reply
* Re: moving arrays from MBR part table install to GPT? Possible?
From: Wilson, Jonathan @ 2015-08-17 15:50 UTC (permalink / raw)
To: Wols Lists; +Cc: David C. Rankin, mdraid
In-Reply-To: <55D1F07A.7090709@youngman.org.uk>
On Mon, 2015-08-17 at 15:32 +0100, Wols Lists wrote:
> On 17/08/15 07:28, David C. Rankin wrote:
> > On 08/16/2015 07:27 AM, Wols Lists wrote:
> >> If you want to add new drives, you can use GPT on them, as I say, if
> >> they're large disks, you'll need to use GPT.
> >>
> >> Sounds like it's a small system, a home system? If you get new drives,
> >> make sure they're proper raid drives, like WD Red. I've got Seagate
> >> Barracudas, which was a mistake ...
> >
> > Thank you for your response. It is my office server, but it runs on 2 1T
> > drives (Carvair Black) Have had good experiences with them. I toss at
> > least 1 barracuda in the trash every 3-4 months.
>
> Well, my experience with Barracudas has been good - they were my
> preferred choice of drive ...
>
> You saying Black rang alarm bells for me, and I've just looked on my
> favourite sales site - it says they are DESKTOP drives!
>
> YOU NEED TO CHECK THEM OUT WITH SMARTCTL!
>
> Check if they support ERC etc. The Barracudas I know don't, the Reds
> various people have said they do. If the blacks don't support it, then
> they'll let you down when you need it. You really don't want to go raid
> 5 or 6 with drives that don't support it. I want to go raid 5, which is
> why I'm gutted to discover I'll have to replace my Barracudas. imho
> they're decent serviceable drives ...
Thats not strictly true....
With a non TLER drive if a problem is found the drive might take a long
time to fix them internally and the device layer time out might be
exceeded.
With TLER the drive will always respond within, usually, 7 seconds.
I have the following embedded in a bash script:
> for x in /sys/block/sd[a-z] ; do echo 360 > $x/device/timeout ; echo -n "$x/device/timeout : " ; cat $x/device/timeout ; done
which is "/sys/block/sdX/device/timeout"
Now if the drive is TLER it will still respond in 7 seconds (either with
data, or an error - no change there), but if I should have any non-TLER
drives the device layer will now wait 360 seconds before timing out. (I
know, a long time which in most cases would cause frozen I/O, seemingly
hung... but hey, who knows the drive might eventually fix its self!)
This time out is just as true for non-raid as raid... if a non raid disk
takes longer to respond than the time out, you potentially lose the
drive... or you've just killed the last I/O commands to it and done a
hard reset. (That said, I think its up to the file system to then deal
with a timed out disk? should mdadm sit on top of the disk its
processing is "disk timed out, kick it".)
> >
> > I'll stick with MBR and see how it goes. In that case I'll just boot the
> > install media and assemble the arrays and see what I end up with.
> >
> >
> Yup. No real point going gpt until you get to 3 and 4TB drives. Once you
> can get linux running, it shouldn't care what sort of drives you have,
> but if you read the gentoo wiki pages about installing on raid
> (disclaimer, I wrote a decent chunk of it), you'll see what a pig it can
> be getting as far as loading linux ...
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: moving arrays from MBR part table install to GPT? Possible?
From: Wols Lists @ 2015-08-17 14:32 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <55D17EF7.8000203@suddenlinkmail.com>
On 17/08/15 07:28, David C. Rankin wrote:
> On 08/16/2015 07:27 AM, Wols Lists wrote:
>> If you want to add new drives, you can use GPT on them, as I say, if
>> they're large disks, you'll need to use GPT.
>>
>> Sounds like it's a small system, a home system? If you get new drives,
>> make sure they're proper raid drives, like WD Red. I've got Seagate
>> Barracudas, which was a mistake ...
>
> Thank you for your response. It is my office server, but it runs on 2 1T
> drives (Carvair Black) Have had good experiences with them. I toss at
> least 1 barracuda in the trash every 3-4 months.
Well, my experience with Barracudas has been good - they were my
preferred choice of drive ...
You saying Black rang alarm bells for me, and I've just looked on my
favourite sales site - it says they are DESKTOP drives!
YOU NEED TO CHECK THEM OUT WITH SMARTCTL!
Check if they support ERC etc. The Barracudas I know don't, the Reds
various people have said they do. If the blacks don't support it, then
they'll let you down when you need it. You really don't want to go raid
5 or 6 with drives that don't support it. I want to go raid 5, which is
why I'm gutted to discover I'll have to replace my Barracudas. imho
they're decent serviceable drives ...
>
> I'll stick with MBR and see how it goes. In that case I'll just boot the
> install media and assemble the arrays and see what I end up with.
>
>
Yup. No real point going gpt until you get to 3 and 4TB drives. Once you
can get linux running, it shouldn't care what sort of drives you have,
but if you read the gentoo wiki pages about installing on raid
(disclaimer, I wrote a decent chunk of it), you'll see what a pig it can
be getting as far as loading linux ...
Cheers,
Wol
^ permalink raw reply
* Re: moving arrays from MBR part table install to GPT? Possible?
From: David C. Rankin @ 2015-08-17 6:28 UTC (permalink / raw)
To: mdraid
In-Reply-To: <55D081C3.4090604@youngman.org.uk>
On 08/16/2015 07:27 AM, Wols Lists wrote:
> If you want to add new drives, you can use GPT on them, as I say, if
> they're large disks, you'll need to use GPT.
>
> Sounds like it's a small system, a home system? If you get new drives,
> make sure they're proper raid drives, like WD Red. I've got Seagate
> Barracudas, which was a mistake ...
Thank you for your response. It is my office server, but it runs on 2 1T drives
(Carvair Black) Have had good experiences with them. I toss at least 1 barracuda
in the trash every 3-4 months.
I'll stick with MBR and see how it goes. In that case I'll just boot the install
media and assemble the arrays and see what I end up with.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: [patch v2 00/11]md: fix raid5 write hole
From: NeilBrown @ 2015-08-17 4:45 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <cover.1439500639.git.shli@fb.com>
On Thu, 13 Aug 2015 14:31:53 -0700 Shaohua Li <shli@fb.com> wrote:
> Hi Neil,
>
> This is the updated patch for the raid5 write hole issue. I thought I addressed
> most of issues from you and Christoph. Please let me know if anything is
> missed. Something not done yet:
Hi Shaohua,
thanks for your persistence and patience. I does look like you have
address everything, though I'm a bit foggy today (recovering from a
cold) so I cannot give it the attention it deserves right now.
Hopefully I'll give you more feedback later in the week. I'll be
offline after that until late September so I'll try really hard to
raising anything important before then.
I'll probably queue these in my "devel" branch and aim for Linux 4.4.
That will give me (and others??) time to test and experiment some more.
I'd be quite happy to see patches on top of these for the write-ahead
caching when I get back in late September, if you happen to have them
ready by then.
>
> - Still use NOFAIL allocation. I don't think 2-element mempool is ok. An
> io_unit will have several bio (> 2), 2 element bioset isn't ok. We can
> dispatch all bio of the io_unit if bio allocation fails, but this will increase
> complexity. I'd prefer using NOFAIL allocation now for simplicity and fix it
> later if necessary
You might need to get a patch accepted which removes the line:
* __GFP_NOFAIL is not to be used in new code.
from mm/page_alloc.c :-(
... though from looking at the code, I think the flag is (currently)
ignored for single-page allocations, they never fail (if they are
allowed to wait).
It's all very confusing.
It can stay for now. It serves as useful documentation at least.
> - Add flag for reshape handling in disk format, but don't support it yet
That looks good, thanks.
Thanks,
NeilBrown
>
> Thanks,
> Shaohua
>
>
> Shaohua Li (9):
> md: override md superblock recovery_offset for journal device
> raid5: export some functions
> raid5: add a new state for stripe log handling
> raid5: add basic stripe log
> raid5: log reclaim support
> raid5: log recovery
> raid5: disable batch with log enabled
> raid5: don't allow resize/reshape with cache(log) support
> raid5: enable log for raid array with cache disk
>
> Song Liu (2):
> MD: replace special disk roles with macros
> MD: add a new disk role to present write journal device
>
> drivers/md/Makefile | 2 +-
> drivers/md/md.c | 44 +-
> drivers/md/md.h | 13 +-
> drivers/md/raid5-cache.c | 1094 ++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid5.c | 137 +++--
> drivers/md/raid5.h | 20 +
> include/uapi/linux/raid/md_p.h | 70 ++-
> 7 files changed, 1320 insertions(+), 60 deletions(-)
> create mode 100644 drivers/md/raid5-cache.c
>
^ permalink raw reply
* Re: [PATCH stable] md/bitmap: return an error when bitmap superblock is corrupt.
From: NeilBrown @ 2015-08-17 0:33 UTC (permalink / raw)
To: Greg KH; +Cc: stable, linux-raid, GuoQing Jiang, Goldwyn Rodrigues, lkml
In-Reply-To: <20150814172351.GB17745@kroah.com>
On Fri, 14 Aug 2015 10:23:51 -0700 Greg KH <greg@kroah.com> wrote:
> On Fri, Aug 14, 2015 at 05:04:21PM +1000, NeilBrown wrote:
> >
> > commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
> > Use separate bitmaps for each nodes in the cluster
> >
> > bitmap_read_sb() validates the bitmap superblock that it reads in.
> > If it finds an inconsistency like a bad magic number or out-of-range
> > version number, it prints an error and returns, but it incorrectly
> > returns zero, so the array is still assembled with the (invalid) bitmap.
> >
> > This means it could try to use a bitmap with a new version number which
> > it therefore does not understand.
> >
> > This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
> > So the patch is suitable for any -stable kernel in that range.
> >
> > Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
> > Cc: stable@vger.kernel.org (v3.5..v4.1)
>
> b97e92574c0 is in 4.1, so do you mean (v3.5..v4.0) here?
>
Yes, I did. Thanks!
v3.5..v4.0.
NeilBrown
^ permalink raw reply
* Re: moving arrays from MBR part table install to GPT? Possible?
From: Wols Lists @ 2015-08-16 12:27 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <55D02E2E.1080909@suddenlinkmail.com>
On 16/08/15 07:31, David C. Rankin wrote:
> All,
>
> I suffered a controller failure on my server housing a
> 2-disk/4-partition raid1 mdraid array under Archlinux with MBR partition
> table. The original motherboard was non-UEFI, of course everything now
> is. I am getting conflicting information whether my existing arrays are
> usable with the new UEFI board. I don't know if this is the best place
> to ask, but heck, there is a lot of collective wisdom regarding mdadm here.
I run gentoo ... and migrated my system from mbr to gpt ...
>
> I am planning to set the BIOS on the new board to Legacy and attempt
> to use my existing arrays by booting the Arch install media, assembling
> the arrays, then chrooting and rebuilding initramfs to preserve the
> existing setup. Is there any reason that can't work?. (I haven't dealt
> with the UEFI fun before)
If your Arch will work on the new hardware, I suspect, actually, that
just transferring the drives across and booting will work (although I
haven't played with UEFI before either).
Your biggest problem is going to be Grub2, I expect.
>
> If that will not work, I'm left with installing to new drives. If I do
> end up with a full install, I'm left with the MBR or GPT partition table
> choice. I will install to 2 new drives that I want to again be setup as
> raid1 arrays with mdadmin. Does mdadmin/mdraid require a MBR partition
> type? Or, can I create a GPT partition table and use mdadm to manage the
> array?
Drives over ?2TB require GPT. It's nothing to do with the software - the
mbr (allegedly) cannot partition larger drives. gfdisk will quite
happily write both a GPT and an MBR to the disk, for the software to use
either. It does warn you, however that "mixing and matching" is
dangerous if they get out of sync.
>
> If GPT is fine, then is there anyway to migrate my existing array to a
> GPT partition table in the new box without having to install to new
> drives with a GPT table and then mount one of the existing drives to
> copy the data to the new array?
Use gfdisk, tell it to write a GPT. Just be careful it doesn't mess up
Grub2 for you ...
If you want to use new drives, once the system is up and running just
use mdadm --add or --replace.
>
> If these questions are already answered in a link somewhere, I
> apologize, I haven't found it. Thanks for any help/advise you can provide.
>
Personally, I think your biggest problem is going to be grub. What I'd
do (and I'm going to have to do it sometime soon) is ...
Put the old drives on the new mobo. Boot using recovery media and make
sure the arrays are okay (shouldn't be a problem).
READ UP ON GRUB! Did I say that was going to be your biggest problem?
Fix grub to boot into your old Arch setup. It might be a good idea to
fit a new (old :-) disk solely for installing grub on until you're happy
everything is okay.
Your old system will now be up and running on the new mobo. Don't bother
switching your old drives to gpt, although you can, as I say, just use
gfdisk to write a GPT. Just be careful it doesn't mess up grub though ...
If you want to add new drives, you can use GPT on them, as I say, if
they're large disks, you'll need to use GPT.
Sounds like it's a small system, a home system? If you get new drives,
make sure they're proper raid drives, like WD Red. I've got Seagate
Barracudas, which was a mistake ...
Cheers,
Wol
^ permalink raw reply
* moving arrays from MBR part table install to GPT? Possible?
From: David C. Rankin @ 2015-08-16 6:31 UTC (permalink / raw)
To: mdraid
All,
I suffered a controller failure on my server housing a 2-disk/4-partition
raid1 mdraid array under Archlinux with MBR partition table. The original
motherboard was non-UEFI, of course everything now is. I am getting conflicting
information whether my existing arrays are usable with the new UEFI board. I
don't know if this is the best place to ask, but heck, there is a lot of
collective wisdom regarding mdadm here.
I am planning to set the BIOS on the new board to Legacy and attempt to use
my existing arrays by booting the Arch install media, assembling the arrays,
then chrooting and rebuilding initramfs to preserve the existing setup. Is there
any reason that can't work?. (I haven't dealt with the UEFI fun before)
If that will not work, I'm left with installing to new drives. If I do end up
with a full install, I'm left with the MBR or GPT partition table choice. I will
install to 2 new drives that I want to again be setup as raid1 arrays with
mdadmin. Does mdadmin/mdraid require a MBR partition type? Or, can I create a
GPT partition table and use mdadm to manage the array?
If GPT is fine, then is there anyway to migrate my existing array to a GPT
partition table in the new box without having to install to new drives with a
GPT table and then mount one of the existing drives to copy the data to the new
array?
If these questions are already answered in a link somewhere, I apologize, I
haven't found it. Thanks for any help/advise you can provide.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* good afternoon
From: vsdfq @ 2015-08-15 15:52 UTC (permalink / raw)
To: bquadranti
good afternoon
rolex,gultar,imac,phone,watch...the shipping is free
web: poazzlo .com
^ permalink raw reply
* Re: Failed to grow
From: Daniel Koch @ 2015-08-14 20:09 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <3161f36377c85c092f4f3a461295e76a@mail.triple6.org>
Fixed.
First make sure to use an overlay for testing:
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
Try this
https://raid.wiki.kernel.org/index.php/RAID_Recovery#Restore_array_by_recreating_.28after_multiple_device_failure.29
It worked so did on the real disks without overlay.
Thanks to frostschutz, SwedeMike and djselbeck from irc ( #linux-raid on
freenode ) for saving me.
---
Viele Grüße,
Daniel Koch
Am 2015-08-14 11:22, schrieb Daniel Koch:
> I crawled the archive of the list and found that --invalid-backup with
> a faked and empty backup might help. Here is the result:
>
> mdadm --assemble --force --run --verbose /dev/md0
> --backup-file=empty.bak --invalid-backup
> mdadm: looking for devices for /dev/md0
> mdadm: no RAID superblock on /dev/sda5
> mdadm: no RAID superblock on /dev/sda2
> mdadm: no RAID superblock on /dev/sda1
> mdadm: no RAID superblock on /dev/sda
> mdadm: /dev/sdn is identified as a member of /dev/md0, slot 9.
> mdadm: /dev/sdj is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdh is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdm is identified as a member of /dev/md0, slot 8.
> mdadm: /dev/sdl is identified as a member of /dev/md0, slot 7.
> mdadm: /dev/sdk is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdi is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sdg is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 10.
> mdadm: /dev/sde is identified as a member of /dev/md0, slot 12.
> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdf is identified as a member of /dev/md0, slot 11.
> mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
> mdadm: :/dev/md0 has an active reshape - checking if critical section
> needs to be restored
> mdadm: Cannot read from empty.bak
> mdadm: No backup metadata on device-12
> mdadm: Failed to find backup of critical section
> mdadm: continuing without restoring backup
> mdadm: added /dev/sdc to /dev/md0 as 1
> mdadm: added /dev/sdg to /dev/md0 as 2
> mdadm: added /dev/sdh to /dev/md0 as 3
> mdadm: added /dev/sdi to /dev/md0 as 4
> mdadm: added /dev/sdj to /dev/md0 as 5
> mdadm: added /dev/sdk to /dev/md0 as 6
> mdadm: added /dev/sdl to /dev/md0 as 7
> mdadm: added /dev/sdm to /dev/md0 as 8
> mdadm: added /dev/sdn to /dev/md0 as 9
> mdadm: added /dev/sdd to /dev/md0 as 10
> mdadm: added /dev/sdf to /dev/md0 as 11
> mdadm: added /dev/sde to /dev/md0 as 12
> mdadm: added /dev/sdb to /dev/md0 as 0
> mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
>
>
> kern.log:
> Aug 14 11:18:20 claw-storage kernel: [ 599.182677] md: md0 stopped.
> Aug 14 11:18:20 claw-storage kernel: [ 599.184459] sdn: unknown
> partition table
> Aug 14 11:18:20 claw-storage kernel: [ 599.185604] sdj: unknown
> partition table
> Aug 14 11:18:20 claw-storage kernel: [ 599.186424] sdm: unknown
> partition table
> Aug 14 11:18:20 claw-storage kernel: [ 599.187869] sdl: unknown
> partition table
> Aug 14 11:18:20 claw-storage kernel: [ 599.189108] sdi: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.213722] sde: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.214073] md: bind<sdc>
> Aug 14 11:18:21 claw-storage kernel: [ 599.214248] md: bind<sdg>
> Aug 14 11:18:21 claw-storage kernel: [ 599.214427] md: bind<sdh>
> Aug 14 11:18:21 claw-storage kernel: [ 599.214438] sdb: sdb1
> Aug 14 11:18:21 claw-storage kernel: [ 599.214620] md: bind<sdi>
> Aug 14 11:18:21 claw-storage kernel: [ 599.214845] md: bind<sdj>
> Aug 14 11:18:21 claw-storage kernel: [ 599.215000] md: bind<sdk>
> Aug 14 11:18:21 claw-storage kernel: [ 599.215140] md: bind<sdl>
> Aug 14 11:18:21 claw-storage kernel: [ 599.215278] sdf: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.215337] md: bind<sdm>
> Aug 14 11:18:21 claw-storage kernel: [ 599.215477] md: bind<sdn>
> Aug 14 11:18:21 claw-storage kernel: [ 599.215588] md: bind<sdd>
> Aug 14 11:18:21 claw-storage kernel: [ 599.216165] md: bind<sdf>
> Aug 14 11:18:21 claw-storage kernel: [ 599.216400] sde: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.216923] md: bind<sde>
> Aug 14 11:18:21 claw-storage kernel: [ 599.217159] sdf: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.217513] md: bind<sdb>
> Aug 14 11:18:21 claw-storage kernel: [ 599.218292] sdl: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.218743] md/raid:md0:
> reshape_position too early for auto-recovery - aborting.
> Aug 14 11:18:21 claw-storage kernel: [ 599.220772] md: pers->run()
> failed ...
> Aug 14 11:18:21 claw-storage kernel: [ 599.222047] sdi: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.222135] md: md0 stopped.
> Aug 14 11:18:21 claw-storage kernel: [ 599.222142] md: unbind<sdb>
> Aug 14 11:18:21 claw-storage kernel: [ 599.222616] sdb: sdb1
> Aug 14 11:18:21 claw-storage kernel: [ 599.223612] sdi: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.224397] sdl: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.225077] sdf: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.225907] sde: unknown
> partition table
> Aug 14 11:18:21 claw-storage kernel: [ 599.226498] sdb: sdb1
> Aug 14 11:18:21 claw-storage kernel: [ 599.232193] md:
> export_rdev(sdb)
> Aug 14 11:18:21 claw-storage kernel: [ 599.232205] md: unbind<sde>
> Aug 14 11:18:21 claw-storage kernel: [ 599.247747] md:
> export_rdev(sde)
> Aug 14 11:18:21 claw-storage kernel: [ 599.247766] md: unbind<sdf>
> Aug 14 11:18:21 claw-storage kernel: [ 599.271766] md:
> export_rdev(sdf)
> Aug 14 11:18:21 claw-storage kernel: [ 599.271774] md: unbind<sdd>
> Aug 14 11:18:21 claw-storage kernel: [ 599.287796] md:
> export_rdev(sdd)
> Aug 14 11:18:21 claw-storage kernel: [ 599.287818] md: unbind<sdn>
> Aug 14 11:18:21 claw-storage kernel: [ 599.319830] md:
> export_rdev(sdn)
> Aug 14 11:18:21 claw-storage kernel: [ 599.319841] md: unbind<sdm>
> Aug 14 11:18:21 claw-storage kernel: [ 599.331804] md:
> export_rdev(sdm)
> Aug 14 11:18:21 claw-storage kernel: [ 599.331816] md: unbind<sdl>
> Aug 14 11:18:21 claw-storage kernel: [ 599.346199] md:
> export_rdev(sdl)
> Aug 14 11:18:21 claw-storage kernel: [ 599.346221] md: unbind<sdk>
> Aug 14 11:18:21 claw-storage kernel: [ 599.355791] md:
> export_rdev(sdk)
> Aug 14 11:18:21 claw-storage kernel: [ 599.355805] md: unbind<sdj>
> Aug 14 11:18:21 claw-storage kernel: [ 599.371779] md:
> export_rdev(sdj)
> Aug 14 11:18:21 claw-storage kernel: [ 599.371794] md: unbind<sdi>
> Aug 14 11:18:21 claw-storage kernel: [ 599.387838] md:
> export_rdev(sdi)
> Aug 14 11:18:21 claw-storage kernel: [ 599.387850] md: unbind<sdh>
> Aug 14 11:18:21 claw-storage kernel: [ 599.403814] md:
> export_rdev(sdh)
> Aug 14 11:18:21 claw-storage kernel: [ 599.403839] md: unbind<sdg>
> Aug 14 11:18:21 claw-storage kernel: [ 599.431842] md:
> export_rdev(sdg)
> Aug 14 11:18:21 claw-storage kernel: [ 599.431852] md: unbind<sdc>
> Aug 14 11:18:21 claw-storage kernel: [ 599.463878] md:
> export_rdev(sdc)
> Aug 14 11:18:21 claw-storage kernel: [ 599.465322] md: bind<sdn>
> Aug 14 11:18:21 claw-storage kernel: [ 599.467200] md: bind<sdj>
> Aug 14 11:18:21 claw-storage kernel: [ 599.469274] md: bind<sde>
> Aug 14 11:18:21 claw-storage kernel: [ 599.471375] md: bind<sdi>
> Aug 14 11:18:21 claw-storage kernel: [ 599.473784] md: bind<sdf>
> Aug 14 11:18:21 claw-storage kernel: [ 599.476780] md: bind<sdl>
> Aug 14 11:18:21 claw-storage kernel: [ 599.480103] md: bind<sdm>
> Aug 14 11:18:21 claw-storage kernel: [ 599.488333] md: bind<sdb>
>
> #cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : inactive sdm[8](S) sdn[9](S) sdj[5](S) sdb[0](S) sdf[11](S)
> sdi[4](S) sde[13](S) sdl[12](S)
> 14161940096 blocks super 1.2
>
> unused devices: <none>
>
> So no luck at all. All disks are accessable using smart which makes me
> think they are operational.
> ---
> Viele Grüße,
>
> Daniel Koch
>
> Am 2015-08-13 17:53, schrieb Daniel Koch:
>> Hello every one,
>>
>> I have a huge problem. I started to grow a array with ( the added
>> device is /dev/sde )
>> # mdadm --grow --raid-devices=13 /dev/md0 --backup-file=md0.bak
>>
>> after that there way 0 disk activity so i checked it with
>> # cat /proc/mdstat
>>
>> everything looked fine but speed was 0K/sec for about 1-2 hours. I
>> decided to reboot the system as grow should continue after that. After
>> that the array wasn't assembled at all. So i wanted to do it myself:
>>
>> # mdadm --assemble /dev/md0 /dev/sd[b-n] --backup-file=md0.bak
>> --verbose
>> mdadm: looking for devices for /dev/md0
>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>> mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 10.
>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 12.
>> mdadm: /dev/sdf is identified as a member of /dev/md0, slot 11.
>> mdadm: /dev/sdg is identified as a member of /dev/md0, slot 2.
>> mdadm: /dev/sdh is identified as a member of /dev/md0, slot 3.
>> mdadm: /dev/sdi is identified as a member of /dev/md0, slot 4.
>> mdadm: /dev/sdj is identified as a member of /dev/md0, slot 5.
>> mdadm: /dev/sdk is identified as a member of /dev/md0, slot 6.
>> mdadm: /dev/sdl is identified as a member of /dev/md0, slot 7.
>> mdadm: /dev/sdm is identified as a member of /dev/md0, slot 8.
>> mdadm: /dev/sdn is identified as a member of /dev/md0, slot 9.
>> mdadm: :/dev/md0 has an active reshape - checking if critical section
>> needs to be restored
>> mdadm: No backup metadata on md0.bak
>> mdadm: No backup metadata on device-12
>> mdadm: Failed to find backup of critical section
>> mdadm: Failed to restore critical section for reshape, sorry.
>>
>> Backup looks broken. What can i do ?
>>
>> # mdadm --version
>> mdadm - v3.3.2 - 21st August 2014
>>
>> # uname -a
>> Linux claw-storage 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
>> (2015-08-04) x86_64 GNU/Linux
>>
>> # du -sch md0.bak
>> 56M md0.bak
>>
>> # mdadm --examine /dev/sd[b-n]
>> /dev/sdb:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : f3d9392c:a9290f8e:4ded7efc:284b4ecf
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 11cb12c0 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdc:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : a5b4533d:d5a4cdc5:e6dcd3aa:4a845044
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 7dfa9159 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdd:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : d39aede2:af92f321:8b6fbd9f:45c73ddf
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : f913b0b - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 10
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sde:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262056 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 13fcdab1:69ce4358:5f1ee2b2:c809b990
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : d976c9a6 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 12
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdf:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 487f70bd:b63c29d6:185aa418:208dc16d
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : a5b479ef - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 11
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdg:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : b798e990:6b50f1a0:3cde1ced:3a20dfaf
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 2053b0c9 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdh:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : bfe03a04:fda138cc:ef11727c:ff1b58d2
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 70ba79db - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdi:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 4fdb8a05:3157bd26:f93ee8d7:fdd42f9f
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 2f151d28 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 4
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdj:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : 6010b774:e02b08ed:29484f97:b03e8a48
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 93158c4c - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 5
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdk:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : afab65ff:a79ed9e7:30ccd918:e2e1c21a
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 6c58c19c - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 6
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdl:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262056 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 8a9031cf:f9b5a888:70363d53:b269047d
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : b3d8bda7 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 7
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdm:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : 1eefdd8c:75b58c0c:f0c555bd:e704e89e
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : 472538a0 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 8
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
>> /dev/sdn:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
>> Name : claw-storage:0 (local to host claw-storage)
>> Creation Time : Tue Jan 14 12:48:49 2014
>> Raid Level : raid6
>> Raid Devices : 13
>>
>> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
>> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
>> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=816 sectors
>> State : clean
>> Device UUID : b8f9a259:c4522982:88538283:497eed43
>>
>> Reshape pos'n : 0
>> Delta Devices : 1 (12->13)
>>
>> Update Time : Thu Aug 13 17:11:05 2015
>> Checksum : f4b8e783 - correct
>> Events : 72402
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 9
>> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
>> replacing)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Problems with Seagate 8TB SMR archive drives
From: Tejas Rao @ 2015-08-14 18:27 UTC (permalink / raw)
To: Jeff Johnson, linux-raid
In-Reply-To: <55CE31CE.8080100@aeoncomputing.com>
The drives are running AR15.
The md device (md8) has XFS on it and I am transferring 33TB of data
using 'cp' command, so I doubt the drives are spinning down as they are
continuously being written to. We plan to use them for archive/backup
purposes only. After the initial ingest (filling up the drives), the
drives will be accessed occasionally for very small reads/writes. It
would be nice though to ingest in a more stable way.
On 8/14/2015 14:22, Jeff Johnson wrote:
> Tejas,
>
> You are probably running firmware AR13 or older. The drives are
> designed for very power efficient archive use and they have a very
> aggressive spin down timer. Once they drop to a slower speed or
> stopped it will take a significant time to get back up to ready. I
> would highly suggest updating to code AR14 or AR15.
>
> The drives aren't really intended for your use model. Perhaps a cron
> job that does a dd read (direct mode) of a hundred MBs or so from the
> md device to dev/null every couple minutes might be enough to keep
> them warm and not spun down.
>
> --Jeff
>
>
> On 8/14/15 10:52 AM, Tejas Rao wrote:
>> I am aware that the Seagate SMR 8tb archive drives are not meant to
>> be used in a RAID environment as they lack TLER/ERC support.
>>
>> We are trying to use these drives with mdraid and are seeing
>> problems. It seems that after writing to these drives for few hours,
>> occasionally some drives stop responding for few minutes and recover
>> on its own later. (This was expected).
>>
>> I have increased the device timeouts to 480 seconds now
>> (sys/block/<device>/device/timeout). The md device is assembled on
>> top of dm-multipath devices and dm-multipath is configured to retry
>> 50 times (no_path_retry=50). I have also changed queue_depth for each
>> device to 1 (NCQ disabled).
>>
>> Usually when I see retries and a drive stops responding, it recovers
>> on its own after few minutes and the md layer does not fail the disk.
>> Occasionally though, the md layer fails the disk after ~15 seconds or
>> so of the drive becoming non-responsive. See below.
>>
>> sdlm and sdfb are the same disk (mpathaw, dm-231,dm-226). Why does
>> the md layer not wait for 480 seconds before failing the disk. As you
>> can see the drive recovered after ~ 8 minutes but the md layer failed
>> it after ~15 seconds.
>>
>> What other tunables can I tune to avoid kicking a drive out early.
>>
>> Aug 14 13:01:12 dc045 kernel: mpt2sas3: log_info(0x31120303):
>> originator(PL), code(0x12), sub_code(0x0303)
>> Aug 14 13:01:12 dc045 kernel: mpt2sas3:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:01:12 dc045 kernel: mpt2sas1:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:05:24 dc045 kernel: mpt2sas3: log_info(0x31120303):
>> originator(PL), code(0x12), sub_code(0x0303)
>> Aug 14 13:05:24 dc045 kernel: mpt2sas3:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:05:24 dc045 kernel: mpt2sas1:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:12:19 dc045 kernel: mpt2sas3: log_info(0x31120303):
>> originator(PL), code(0x12), sub_code(0x0303)
>> Aug 14 13:12:19 dc045 kernel: mpt2sas3:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:12:19 dc045 kernel: mpt2sas1:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:12:35 dc045 kernel: __ratelimit: 2 callbacks suppressed
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
>> 00 55 6a 5e c8 00 00 b8 00
>> Aug 14 13:12:35 dc045 kernel: __ratelimit: 14 callbacks suppressed
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
>> 00 55 6a 5f 80 00 00 10 00
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
>> 00 55 6a 5f 90 00 03 38 00
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(16): 88
>> 00 00 00 00 01 d5 59 50 a8 00 00 00 08 00 00
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(16): 88
>> 00 00 00 00 01 d5 59 50 b0 00 00 00 80 00 00
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(10): 28
>> 00 55 6a 78 00 00 00 50 00
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 7874367664
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 1433040896
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
>> 00 00 00 08 08 00 00 01 00
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdfb, sector 2056
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
>> 00 55 6a 7b e0 00 00 20 00
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdlm, sector 1433041888
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 1433041888
>> Aug 14 13:12:35 dc045 kernel: mpt2sas1:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
>> 00 00 00 00 00 00 00 08 00
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdlm, sector 0
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 2056
>> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
>> uptodate=0
>> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Disk failure on dm-231,
>> disabling device.
>> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Operation continuing on 13
>> devices.
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
>> hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
>> Hardware Error [current]
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>>
>> ASC=0x81 ASCQ=0x0ASC=0x81 ASCQ=0x0
>> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
>> 00 55 6a 62 c8 00 04 00 00
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdlm, sector 1433035464
>> Aug 14 13:12:35 dc045 kernel: mpt2sas3:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 1433035464
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdfb, sector 1433036488
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 1433036488
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdlm, sector 1433037512
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 1433037512
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> sdfb, sector 2128
>> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
>> dm-226, sector 2128
>> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
>> uptodate=0
>> Aug 14 13:12:36 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:12:36 dc045 multipathd: checker failed path 68:320 in map
>> mpathaw
>> Aug 14 13:12:36 dc045 multipathd: mpathaw: remaining active paths: 1
>> Aug 14 13:12:36 dc045 kernel: device-mapper: multipath: Failing path
>> 68:320.
>> Aug 14 13:13:17 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:13:25 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:13:25 dc045 multipathd: checker failed path 129:208 in map
>> mpathaw
>> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
>> max_retries=50
>> Aug 14 13:13:25 dc045 multipathd: mpathaw: remaining active paths: 0
>> Aug 14 13:13:25 dc045 kernel: device-mapper: multipath: Failing path
>> 129:208.
>> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
>> max_retries=50
>> Aug 14 13:13:27 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:13:35 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:13:37 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:13:45 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:13:47 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:13:55 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:13:57 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:05 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:07 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:15 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:17 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:25 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:27 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:35 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:37 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:46 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:48 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:14:56 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:14:58 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:06 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:08 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:16 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:18 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:26 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:28 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:36 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:38 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:46 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:48 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:15:56 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:15:58 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:06 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:08 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:16 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:18 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:26 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:28 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:37 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:39 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:47 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:49 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:16:57 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:16:59 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:07 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:09 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:17 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:19 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:27 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:29 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:37 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:39 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:47 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:49 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:17:57 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:17:59 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:07 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:18:09 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:17 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:18:19 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:28 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:18:30 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:38 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:18:40 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:48 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:18:50 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:18:58 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:00 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:08 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:10 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:18 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:20 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:28 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:30 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:38 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:40 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:48 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:19:50 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:19:58 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:20:00 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:20:08 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:20:11 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:20:19 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:20:21 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:20:29 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is down
>> Aug 14 13:20:31 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is down
>> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: attempting task abort!
>> scmd(ffff881c6e49aec0)
>> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
>> 00 55 6a 72 c8 00 01 68 00
>> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42: handle(0x0037),
>> sas_address(0x5000c5007b2ee20d), phy(18)
>> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42:
>> enclosure_logical_id(0x500093d00104c000), slot(39)
>> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: attempting task abort!
>> scmd(ffff880c35fba7c0)
>> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
>> 00 55 6a 6e c8 00 04 00 00
>> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42: handle(0x0037),
>> sas_address(0x5000c5007b2ee20e), phy(18)
>> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42:
>> enclosure_logical_id(0x500093d00104c000), slot(39)
>> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: task abort: SUCCESS
>> scmd(ffff881c6e49aec0)
>> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: task abort: SUCCESS
>> scmd(ffff880c35fba7c0)
>> Aug 14 13:20:37 dc045 kernel: mpt2sas3:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
>> Aug 14 13:20:37 dc045 kernel: mpt2sas1:
>> _scsih_sas_broadcast_primitive_event: enter: phy number(5), width(8)
>> Aug 14 13:20:39 dc045 multipathd: mpathaw: sdfb - directio checker
>> reports path is up
>> Aug 14 13:20:39 dc045 multipathd: 129:208: reinstated
>> Aug 14 13:20:39 dc045 multipathd: mpathaw: queue_if_no_path enabled
>> Aug 14 13:20:39 dc045 multipathd: mpathaw: Recovered to normal mode
>> Aug 14 13:20:39 dc045 multipathd: mpathaw: remaining active paths: 1
>> Aug 14 13:20:41 dc045 multipathd: mpathaw: sdlm - directio checker
>> reports path is up
>> Aug 14 13:20:41 dc045 multipathd: 68:320: reinstated
>> Aug 14 13:20:41 dc045 multipathd: mpathaw: remaining active paths: 2
>> Aug 14 13:21:12 dc045 kernel: md: unbind<dm-231>
>> Aug 14 13:21:12 dc045 kernel: md: export_rdev(dm-231)
>>
>>
>>
>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply
* Re: Problems with Seagate 8TB SMR archive drives
From: Jeff Johnson @ 2015-08-14 18:22 UTC (permalink / raw)
To: Tejas Rao, linux-raid
In-Reply-To: <55CE2ACA.50203@bnl.gov>
Tejas,
You are probably running firmware AR13 or older. The drives are designed
for very power efficient archive use and they have a very aggressive
spin down timer. Once they drop to a slower speed or stopped it will
take a significant time to get back up to ready. I would highly suggest
updating to code AR14 or AR15.
The drives aren't really intended for your use model. Perhaps a cron job
that does a dd read (direct mode) of a hundred MBs or so from the md
device to dev/null every couple minutes might be enough to keep them
warm and not spun down.
--Jeff
On 8/14/15 10:52 AM, Tejas Rao wrote:
> I am aware that the Seagate SMR 8tb archive drives are not meant to be
> used in a RAID environment as they lack TLER/ERC support.
>
> We are trying to use these drives with mdraid and are seeing problems.
> It seems that after writing to these drives for few hours,
> occasionally some drives stop responding for few minutes and recover
> on its own later. (This was expected).
>
> I have increased the device timeouts to 480 seconds now
> (sys/block/<device>/device/timeout). The md device is assembled on top
> of dm-multipath devices and dm-multipath is configured to retry 50
> times (no_path_retry=50). I have also changed queue_depth for each
> device to 1 (NCQ disabled).
>
> Usually when I see retries and a drive stops responding, it recovers
> on its own after few minutes and the md layer does not fail the disk.
> Occasionally though, the md layer fails the disk after ~15 seconds or
> so of the drive becoming non-responsive. See below.
>
> sdlm and sdfb are the same disk (mpathaw, dm-231,dm-226). Why does the
> md layer not wait for 480 seconds before failing the disk. As you can
> see the drive recovered after ~ 8 minutes but the md layer failed it
> after ~15 seconds.
>
> What other tunables can I tune to avoid kicking a drive out early.
>
> Aug 14 13:01:12 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:01:12 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:01:12 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:05:24 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:05:24 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:05:24 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:19 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:12:19 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:19 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: __ratelimit: 2 callbacks suppressed
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 5e c8 00 00 b8 00
> Aug 14 13:12:35 dc045 kernel: __ratelimit: 14 callbacks suppressed
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 55 6a 5f 80 00 00 10 00
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 5f 90 00 03 38 00
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(16): 88
> 00 00 00 00 01 d5 59 50 a8 00 00 00 08 00 00
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(16): 88
> 00 00 00 00 01 d5 59 50 b0 00 00 00 80 00 00
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(10): 28
> 00 55 6a 78 00 00 00 50 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 7874367664
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433040896
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 00 00 08 08 00 00 01 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 2056
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
> 00 55 6a 7b e0 00 00 20 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433041888
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433041888
> Aug 14 13:12:35 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
> 00 00 00 00 00 00 00 08 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 0
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 2056
> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
> uptodate=0
> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Disk failure on dm-231,
> disabling device.
> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Operation continuing on 13
> devices.
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 62 c8 00 04 00 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433035464
> Aug 14 13:12:35 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433035464
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 1433036488
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433036488
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433037512
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433037512
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 2128
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 2128
> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
> uptodate=0
> Aug 14 13:12:36 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:12:36 dc045 multipathd: checker failed path 68:320 in map
> mpathaw
> Aug 14 13:12:36 dc045 multipathd: mpathaw: remaining active paths: 1
> Aug 14 13:12:36 dc045 kernel: device-mapper: multipath: Failing path
> 68:320.
> Aug 14 13:13:17 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:25 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:25 dc045 multipathd: checker failed path 129:208 in map
> mpathaw
> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
> max_retries=50
> Aug 14 13:13:25 dc045 multipathd: mpathaw: remaining active paths: 0
> Aug 14 13:13:25 dc045 kernel: device-mapper: multipath: Failing path
> 129:208.
> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
> max_retries=50
> Aug 14 13:13:27 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:35 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:37 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:45 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:47 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:55 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:57 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:05 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:07 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:15 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:17 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:25 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:27 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:35 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:37 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:46 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:48 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:56 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:58 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:06 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:08 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:16 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:18 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:26 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:28 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:36 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:38 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:46 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:48 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:56 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:58 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:06 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:08 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:16 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:18 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:26 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:28 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:37 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:39 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:47 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:49 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:57 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:59 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:07 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:09 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:17 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:19 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:27 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:29 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:37 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:39 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:47 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:49 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:57 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:59 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:07 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:09 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:17 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:19 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:28 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:30 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:38 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:40 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:48 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:50 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:58 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:00 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:08 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:10 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:18 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:20 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:28 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:30 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:38 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:40 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:48 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:50 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:58 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:00 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:08 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:11 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:19 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:21 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:29 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:31 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: attempting task abort!
> scmd(ffff881c6e49aec0)
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 55 6a 72 c8 00 01 68 00
> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42: handle(0x0037),
> sas_address(0x5000c5007b2ee20d), phy(18)
> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42:
> enclosure_logical_id(0x500093d00104c000), slot(39)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: attempting task abort!
> scmd(ffff880c35fba7c0)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 6e c8 00 04 00 00
> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42: handle(0x0037),
> sas_address(0x5000c5007b2ee20e), phy(18)
> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42:
> enclosure_logical_id(0x500093d00104c000), slot(39)
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: task abort: SUCCESS
> scmd(ffff881c6e49aec0)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: task abort: SUCCESS
> scmd(ffff880c35fba7c0)
> Aug 14 13:20:37 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:20:37 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(5), width(8)
> Aug 14 13:20:39 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is up
> Aug 14 13:20:39 dc045 multipathd: 129:208: reinstated
> Aug 14 13:20:39 dc045 multipathd: mpathaw: queue_if_no_path enabled
> Aug 14 13:20:39 dc045 multipathd: mpathaw: Recovered to normal mode
> Aug 14 13:20:39 dc045 multipathd: mpathaw: remaining active paths: 1
> Aug 14 13:20:41 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is up
> Aug 14 13:20:41 dc045 multipathd: 68:320: reinstated
> Aug 14 13:20:41 dc045 multipathd: mpathaw: remaining active paths: 2
> Aug 14 13:21:12 dc045 kernel: md: unbind<dm-231>
> Aug 14 13:21:12 dc045 kernel: md: export_rdev(dm-231)
>
>
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
------------------------------
Jeff Johnson
Co-Founder
Aeon Computing
jeff.johnson@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001 f: 858-412-3845
m: 619-204-9061
4170 Morena Boulevard, Suite D - San Diego, CA 92117
High-performance Computing / Lustre Filesystems / Scale-out Storage
^ permalink raw reply
* Re: Problems with Seagate 8TB SMR archive drives
From: Tejas Rao @ 2015-08-14 18:19 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <55CE2ACA.50203@bnl.gov>
What does "md: super_written gets error=-121, uptodate=0" mean? Looking
at the md man page, I see the below. Does the error mean a write failed?
Can we make it so that md waits for the write to complete for 480 seconds?
If the md driver detects a write error on a device in a RAID1, RAID4,
RAID5, RAID6, or RAID10 array, it immediately disables that device
(marking it as faulty) and continues operation on the remaining
devices.
On 8/14/2015 13:52, Tejas Rao wrote:
> I am aware that the Seagate SMR 8tb archive drives are not meant to be
> used in a RAID environment as they lack TLER/ERC support.
>
> We are trying to use these drives with mdraid and are seeing problems.
> It seems that after writing to these drives for few hours,
> occasionally some drives stop responding for few minutes and recover
> on its own later. (This was expected).
>
> I have increased the device timeouts to 480 seconds now
> (sys/block/<device>/device/timeout). The md device is assembled on top
> of dm-multipath devices and dm-multipath is configured to retry 50
> times (no_path_retry=50). I have also changed queue_depth for each
> device to 1 (NCQ disabled).
>
> Usually when I see retries and a drive stops responding, it recovers
> on its own after few minutes and the md layer does not fail the disk.
> Occasionally though, the md layer fails the disk after ~15 seconds or
> so of the drive becoming non-responsive. See below.
>
> sdlm and sdfb are the same disk (mpathaw, dm-231,dm-226). Why does the
> md layer not wait for 480 seconds before failing the disk. As you can
> see the drive recovered after ~ 8 minutes but the md layer failed it
> after ~15 seconds.
>
> What other tunables can I tune to avoid kicking a drive out early.
>
> Aug 14 13:01:12 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:01:12 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:01:12 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:05:24 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:05:24 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:05:24 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:19 dc045 kernel: mpt2sas3: log_info(0x31120303):
> originator(PL), code(0x12), sub_code(0x0303)
> Aug 14 13:12:19 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:19 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: __ratelimit: 2 callbacks suppressed
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 5e c8 00 00 b8 00
> Aug 14 13:12:35 dc045 kernel: __ratelimit: 14 callbacks suppressed
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 55 6a 5f 80 00 00 10 00
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 5f 90 00 03 38 00
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(16): 88
> 00 00 00 00 01 d5 59 50 a8 00 00 00 08 00 00
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(16): 88
> 00 00 00 00 01 d5 59 50 b0 00 00 00 80 00 00
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(10): 28
> 00 55 6a 78 00 00 00 50 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 7874367664
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433040896
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 00 00 08 08 00 00 01 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 2056
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
> 00 55 6a 7b e0 00 00 20 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433041888
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433041888
> Aug 14 13:12:35 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28
> 00 00 00 00 00 00 00 08 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 0
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 2056
> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
> uptodate=0
> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Disk failure on dm-231,
> disabling device.
> Aug 14 13:12:35 dc045 kernel: md/raid:md8: Operation continuing on 13
> devices.
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key :
> Hardware Error [current]
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
> ASCQ=0x0ASC=0x81 ASCQ=0x0
> Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 62 c8 00 04 00 00
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433035464
> Aug 14 13:12:35 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433035464
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 1433036488
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433036488
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdlm, sector 1433037512
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 1433037512
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> sdfb, sector 2128
> Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
> dm-226, sector 2128
> Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121,
> uptodate=0
> Aug 14 13:12:36 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:12:36 dc045 multipathd: checker failed path 68:320 in map
> mpathaw
> Aug 14 13:12:36 dc045 multipathd: mpathaw: remaining active paths: 1
> Aug 14 13:12:36 dc045 kernel: device-mapper: multipath: Failing path
> 68:320.
> Aug 14 13:13:17 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:25 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:25 dc045 multipathd: checker failed path 129:208 in map
> mpathaw
> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
> max_retries=50
> Aug 14 13:13:25 dc045 multipathd: mpathaw: remaining active paths: 0
> Aug 14 13:13:25 dc045 kernel: device-mapper: multipath: Failing path
> 129:208.
> Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
> max_retries=50
> Aug 14 13:13:27 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:35 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:37 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:45 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:47 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:13:55 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:13:57 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:05 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:07 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:15 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:17 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:25 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:27 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:35 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:37 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:46 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:48 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:14:56 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:14:58 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:06 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:08 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:16 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:18 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:26 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:28 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:36 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:38 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:46 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:48 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:15:56 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:15:58 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:06 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:08 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:16 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:18 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:26 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:28 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:37 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:39 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:47 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:49 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:16:57 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:16:59 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:07 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:09 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:17 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:19 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:27 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:29 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:37 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:39 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:47 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:49 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:17:57 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:17:59 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:07 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:09 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:17 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:19 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:28 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:30 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:38 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:40 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:48 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:18:50 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:18:58 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:00 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:08 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:10 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:18 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:20 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:28 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:30 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:38 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:40 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:48 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:19:50 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:19:58 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:00 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:08 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:11 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:19 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:21 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:29 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is down
> Aug 14 13:20:31 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is down
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: attempting task abort!
> scmd(ffff881c6e49aec0)
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a
> 00 55 6a 72 c8 00 01 68 00
> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42: handle(0x0037),
> sas_address(0x5000c5007b2ee20d), phy(18)
> Aug 14 13:20:37 dc045 kernel: scsi target25:0:42:
> enclosure_logical_id(0x500093d00104c000), slot(39)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: attempting task abort!
> scmd(ffff880c35fba7c0)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a
> 00 55 6a 6e c8 00 04 00 00
> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42: handle(0x0037),
> sas_address(0x5000c5007b2ee20e), phy(18)
> Aug 14 13:20:37 dc045 kernel: scsi target27:0:42:
> enclosure_logical_id(0x500093d00104c000), slot(39)
> Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: task abort: SUCCESS
> scmd(ffff881c6e49aec0)
> Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: task abort: SUCCESS
> scmd(ffff880c35fba7c0)
> Aug 14 13:20:37 dc045 kernel: mpt2sas3:
> _scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
> Aug 14 13:20:37 dc045 kernel: mpt2sas1:
> _scsih_sas_broadcast_primitive_event: enter: phy number(5), width(8)
> Aug 14 13:20:39 dc045 multipathd: mpathaw: sdfb - directio checker
> reports path is up
> Aug 14 13:20:39 dc045 multipathd: 129:208: reinstated
> Aug 14 13:20:39 dc045 multipathd: mpathaw: queue_if_no_path enabled
> Aug 14 13:20:39 dc045 multipathd: mpathaw: Recovered to normal mode
> Aug 14 13:20:39 dc045 multipathd: mpathaw: remaining active paths: 1
> Aug 14 13:20:41 dc045 multipathd: mpathaw: sdlm - directio checker
> reports path is up
> Aug 14 13:20:41 dc045 multipathd: 68:320: reinstated
> Aug 14 13:20:41 dc045 multipathd: mpathaw: remaining active paths: 2
> Aug 14 13:21:12 dc045 kernel: md: unbind<dm-231>
> Aug 14 13:21:12 dc045 kernel: md: export_rdev(dm-231)
>
>
>
>
>
>
>
^ permalink raw reply
* Problems with Seagate 8TB SMR archive drives
From: Tejas Rao @ 2015-08-14 17:52 UTC (permalink / raw)
To: linux-raid
I am aware that the Seagate SMR 8tb archive drives are not meant to be
used in a RAID environment as they lack TLER/ERC support.
We are trying to use these drives with mdraid and are seeing problems.
It seems that after writing to these drives for few hours, occasionally
some drives stop responding for few minutes and recover on its own
later. (This was expected).
I have increased the device timeouts to 480 seconds now
(sys/block/<device>/device/timeout). The md device is assembled on top
of dm-multipath devices and dm-multipath is configured to retry 50 times
(no_path_retry=50). I have also changed queue_depth for each device to 1
(NCQ disabled).
Usually when I see retries and a drive stops responding, it recovers on
its own after few minutes and the md layer does not fail the disk.
Occasionally though, the md layer fails the disk after ~15 seconds or so
of the drive becoming non-responsive. See below.
sdlm and sdfb are the same disk (mpathaw, dm-231,dm-226). Why does the
md layer not wait for 480 seconds before failing the disk. As you can
see the drive recovered after ~ 8 minutes but the md layer failed it
after ~15 seconds.
What other tunables can I tune to avoid kicking a drive out early.
Aug 14 13:01:12 dc045 kernel: mpt2sas3: log_info(0x31120303):
originator(PL), code(0x12), sub_code(0x0303)
Aug 14 13:01:12 dc045 kernel: mpt2sas3:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:01:12 dc045 kernel: mpt2sas1:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:05:24 dc045 kernel: mpt2sas3: log_info(0x31120303):
originator(PL), code(0x12), sub_code(0x0303)
Aug 14 13:05:24 dc045 kernel: mpt2sas3:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:05:24 dc045 kernel: mpt2sas1:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:12:19 dc045 kernel: mpt2sas3: log_info(0x31120303):
originator(PL), code(0x12), sub_code(0x0303)
Aug 14 13:12:19 dc045 kernel: mpt2sas3:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:12:19 dc045 kernel: mpt2sas1:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:12:35 dc045 kernel: __ratelimit: 2 callbacks suppressed
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a 00
55 6a 5e c8 00 00 b8 00
Aug 14 13:12:35 dc045 kernel: __ratelimit: 14 callbacks suppressed
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a 00
55 6a 5f 80 00 00 10 00
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a 00
55 6a 5f 90 00 03 38 00
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(16): 88 00
00 00 00 01 d5 59 50 a8 00 00 00 08 00 00
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(16): 88 00
00 00 00 01 d5 59 50 b0 00 00 00 80 00 00
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Read(10): 28 00
55 6a 78 00 00 00 50 00
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 7874367664
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 1433040896
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a 00
00 00 08 08 00 00 01 00
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdfb, sector 2056
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28 00
55 6a 7b e0 00 00 20 00
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdlm, sector 1433041888
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 1433041888
Aug 14 13:12:35 dc045 kernel: mpt2sas1:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Read(10): 28 00
00 00 00 00 00 00 08 00
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdlm, sector 0
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 2056
Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121, uptodate=0
Aug 14 13:12:35 dc045 kernel: md/raid:md8: Disk failure on dm-231,
disabling device.
Aug 14 13:12:35 dc045 kernel: md/raid:md8: Operation continuing on 13
devices.
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Unhandled sense code
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] Sense Key : Hardware
Error [current]
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] <<vendor>> ASC=0x81
ASCQ=0x0ASC=0x81 ASCQ=0x0
Aug 14 13:12:35 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a 00
55 6a 62 c8 00 04 00 00
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdlm, sector 1433035464
Aug 14 13:12:35 dc045 kernel: mpt2sas3:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 1433035464
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdfb, sector 1433036488
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 1433036488
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdlm, sector 1433037512
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 1433037512
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
sdfb, sector 2128
Aug 14 13:12:35 dc045 kernel: end_request: critical target error, dev
dm-226, sector 2128
Aug 14 13:12:35 dc045 kernel: md: super_written gets error=-121, uptodate=0
Aug 14 13:12:36 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:12:36 dc045 multipathd: checker failed path 68:320 in map mpathaw
Aug 14 13:12:36 dc045 multipathd: mpathaw: remaining active paths: 1
Aug 14 13:12:36 dc045 kernel: device-mapper: multipath: Failing path 68:320.
Aug 14 13:13:17 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:13:25 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:13:25 dc045 multipathd: checker failed path 129:208 in map mpathaw
Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
max_retries=50
Aug 14 13:13:25 dc045 multipathd: mpathaw: remaining active paths: 0
Aug 14 13:13:25 dc045 kernel: device-mapper: multipath: Failing path
129:208.
Aug 14 13:13:25 dc045 multipathd: mpathaw: Entering recovery mode:
max_retries=50
Aug 14 13:13:27 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:13:35 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:13:37 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:13:45 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:13:47 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:13:55 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:13:57 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:05 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:07 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:15 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:17 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:25 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:27 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:35 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:37 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:46 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:48 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:14:56 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:14:58 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:06 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:08 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:16 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:18 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:26 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:28 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:36 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:38 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:46 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:48 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:15:56 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:15:58 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:06 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:08 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:16 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:18 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:26 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:28 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:37 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:39 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:47 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:49 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:16:57 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:16:59 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:07 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:09 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:17 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:19 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:27 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:29 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:37 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:39 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:47 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:49 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:17:57 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:17:59 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:07 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:18:09 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:17 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:18:19 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:28 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:18:30 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:38 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:18:40 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:48 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:18:50 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:18:58 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:00 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:08 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:10 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:18 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:20 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:28 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:30 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:38 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:40 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:48 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:19:50 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:19:58 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:20:00 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:20:08 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:20:11 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:20:19 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:20:21 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:20:29 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is down
Aug 14 13:20:31 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is down
Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: attempting task abort!
scmd(ffff881c6e49aec0)
Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: [sdfb] CDB: Write(10): 2a 00
55 6a 72 c8 00 01 68 00
Aug 14 13:20:37 dc045 kernel: scsi target25:0:42: handle(0x0037),
sas_address(0x5000c5007b2ee20d), phy(18)
Aug 14 13:20:37 dc045 kernel: scsi target25:0:42:
enclosure_logical_id(0x500093d00104c000), slot(39)
Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: attempting task abort!
scmd(ffff880c35fba7c0)
Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: [sdlm] CDB: Write(10): 2a 00
55 6a 6e c8 00 04 00 00
Aug 14 13:20:37 dc045 kernel: scsi target27:0:42: handle(0x0037),
sas_address(0x5000c5007b2ee20e), phy(18)
Aug 14 13:20:37 dc045 kernel: scsi target27:0:42:
enclosure_logical_id(0x500093d00104c000), slot(39)
Aug 14 13:20:37 dc045 kernel: sd 25:0:42:0: task abort: SUCCESS
scmd(ffff881c6e49aec0)
Aug 14 13:20:37 dc045 kernel: sd 27:0:42:0: task abort: SUCCESS
scmd(ffff880c35fba7c0)
Aug 14 13:20:37 dc045 kernel: mpt2sas3:
_scsih_sas_broadcast_primitive_event: enter: phy number(7), width(8)
Aug 14 13:20:37 dc045 kernel: mpt2sas1:
_scsih_sas_broadcast_primitive_event: enter: phy number(5), width(8)
Aug 14 13:20:39 dc045 multipathd: mpathaw: sdfb - directio checker
reports path is up
Aug 14 13:20:39 dc045 multipathd: 129:208: reinstated
Aug 14 13:20:39 dc045 multipathd: mpathaw: queue_if_no_path enabled
Aug 14 13:20:39 dc045 multipathd: mpathaw: Recovered to normal mode
Aug 14 13:20:39 dc045 multipathd: mpathaw: remaining active paths: 1
Aug 14 13:20:41 dc045 multipathd: mpathaw: sdlm - directio checker
reports path is up
Aug 14 13:20:41 dc045 multipathd: 68:320: reinstated
Aug 14 13:20:41 dc045 multipathd: mpathaw: remaining active paths: 2
Aug 14 13:21:12 dc045 kernel: md: unbind<dm-231>
Aug 14 13:21:12 dc045 kernel: md: export_rdev(dm-231)
^ permalink raw reply
* [PATCH 3.14 43/44] md/bitmap: return an error when bitmap superblock is corrupt.
From: Greg Kroah-Hartman @ 2015-08-14 17:45 UTC (permalink / raw)
To: linux-kernel, stable@vger.kernel.org, linux-raid
Cc: Greg Kroah-Hartman, stable, NeilBrown, GuoQing Jiang
In-Reply-To: <20150814174401.628233291@linuxfoundation.org>
3.14-stable review patch. If anyone has any objections, please let me know.
------------------
From: NeilBrown <neilb@suse.com>
commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
Use separate bitmaps for each nodes in the cluster
bitmap_read_sb() validates the bitmap superblock that it reads in.
If it finds an inconsistency like a bad magic number or out-of-range
version number, it prints an error and returns, but it incorrectly
returns zero, so the array is still assembled with the (invalid) bitmap.
This means it could try to use a bitmap with a new version number which
it therefore does not understand.
This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
So the patch is suitable for any -stable kernel in that range.
Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
Signed-off-by: NeilBrown <neilb@suse.com>
Reported-by: GuoQing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 2 ++
1 file changed, 2 insertions(+)
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -564,6 +564,8 @@ static int bitmap_read_sb(struct bitmap
if (err)
return err;
+ err = -EINVAL;
+
sb = kmap_atomic(sb_page);
chunksize = le32_to_cpu(sb->chunksize);
^ permalink raw reply
* [PATCH 3.10 34/35] md/bitmap: return an error when bitmap superblock is corrupt.
From: Greg Kroah-Hartman @ 2015-08-14 17:45 UTC (permalink / raw)
To: linux-kernel, stable@vger.kernel.org, linux-raid
Cc: Greg Kroah-Hartman, stable, NeilBrown, GuoQing Jiang
In-Reply-To: <20150814174353.835241087@linuxfoundation.org>
3.10-stable review patch. If anyone has any objections, please let me know.
------------------
From: NeilBrown <neilb@suse.com>
commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
Use separate bitmaps for each nodes in the cluster
bitmap_read_sb() validates the bitmap superblock that it reads in.
If it finds an inconsistency like a bad magic number or out-of-range
version number, it prints an error and returns, but it incorrectly
returns zero, so the array is still assembled with the (invalid) bitmap.
This means it could try to use a bitmap with a new version number which
it therefore does not understand.
This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
So the patch is suitable for any -stable kernel in that range.
Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
Signed-off-by: NeilBrown <neilb@suse.com>
Reported-by: GuoQing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 2 ++
1 file changed, 2 insertions(+)
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -564,6 +564,8 @@ static int bitmap_read_sb(struct bitmap
if (err)
return err;
+ err = -EINVAL;
+
sb = kmap_atomic(sb_page);
chunksize = le32_to_cpu(sb->chunksize);
^ permalink raw reply
* Patch "md/bitmap: return an error when bitmap superblock is corrupt." has been added to the 3.14-stable tree
From: gregkh @ 2015-08-14 17:25 UTC (permalink / raw)
To: neilb, gqjiang, linux-kernel, linux-raid, rgoldwyn; +Cc: stable, stable-commits
In-Reply-To: <20150814170421.60d40eb0@noble>
This is a note to let you know that I've just added the patch titled
md/bitmap: return an error when bitmap superblock is corrupt.
to the 3.14-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
md-bitmap-return-an-error-when-bitmap-superblock-is-corrupt.patch
and it can be found in the queue-3.14 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@vger.kernel.org> know about it.
From neilb@suse.com Fri Aug 14 10:23:56 2015
From: NeilBrown <neilb@suse.com>
Date: Fri, 14 Aug 2015 17:04:21 +1000
Subject: md/bitmap: return an error when bitmap superblock is corrupt.
To: stable@vger.kernel.org, linux-raid <linux-raid@vger.kernel.org>
Cc: GuoQing Jiang <gqjiang@suse.com>, Goldwyn Rodrigues <rgoldwyn@suse.com>, lkml <linux-kernel@vger.kernel.org>
Message-ID: <20150814170421.60d40eb0@noble>
From: NeilBrown <neilb@suse.com>
commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
Use separate bitmaps for each nodes in the cluster
bitmap_read_sb() validates the bitmap superblock that it reads in.
If it finds an inconsistency like a bad magic number or out-of-range
version number, it prints an error and returns, but it incorrectly
returns zero, so the array is still assembled with the (invalid) bitmap.
This means it could try to use a bitmap with a new version number which
it therefore does not understand.
This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
So the patch is suitable for any -stable kernel in that range.
Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
Signed-off-by: NeilBrown <neilb@suse.com>
Reported-by: GuoQing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 2 ++
1 file changed, 2 insertions(+)
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -564,6 +564,8 @@ static int bitmap_read_sb(struct bitmap
if (err)
return err;
+ err = -EINVAL;
+
sb = kmap_atomic(sb_page);
chunksize = le32_to_cpu(sb->chunksize);
Patches currently in stable-queue which might be from neilb@suse.com are
queue-3.14/md-raid1-extend-spinlock-to-protect-raid1_end_read_request-against-inconsistencies.patch
queue-3.14/md-bitmap-return-an-error-when-bitmap-superblock-is-corrupt.patch
queue-3.14/md-use-kzalloc-when-bitmap-is-disabled.patch
^ permalink raw reply
* Patch "md/bitmap: return an error when bitmap superblock is corrupt." has been added to the 3.10-stable tree
From: gregkh @ 2015-08-14 17:25 UTC (permalink / raw)
To: neilb, gqjiang, linux-kernel, linux-raid, rgoldwyn; +Cc: stable, stable-commits
In-Reply-To: <20150814170421.60d40eb0@noble>
This is a note to let you know that I've just added the patch titled
md/bitmap: return an error when bitmap superblock is corrupt.
to the 3.10-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
md-bitmap-return-an-error-when-bitmap-superblock-is-corrupt.patch
and it can be found in the queue-3.10 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@vger.kernel.org> know about it.
From neilb@suse.com Fri Aug 14 10:23:56 2015
From: NeilBrown <neilb@suse.com>
Date: Fri, 14 Aug 2015 17:04:21 +1000
Subject: md/bitmap: return an error when bitmap superblock is corrupt.
To: stable@vger.kernel.org, linux-raid <linux-raid@vger.kernel.org>
Cc: GuoQing Jiang <gqjiang@suse.com>, Goldwyn Rodrigues <rgoldwyn@suse.com>, lkml <linux-kernel@vger.kernel.org>
Message-ID: <20150814170421.60d40eb0@noble>
From: NeilBrown <neilb@suse.com>
commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
Use separate bitmaps for each nodes in the cluster
bitmap_read_sb() validates the bitmap superblock that it reads in.
If it finds an inconsistency like a bad magic number or out-of-range
version number, it prints an error and returns, but it incorrectly
returns zero, so the array is still assembled with the (invalid) bitmap.
This means it could try to use a bitmap with a new version number which
it therefore does not understand.
This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
So the patch is suitable for any -stable kernel in that range.
Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
Signed-off-by: NeilBrown <neilb@suse.com>
Reported-by: GuoQing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 2 ++
1 file changed, 2 insertions(+)
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -564,6 +564,8 @@ static int bitmap_read_sb(struct bitmap
if (err)
return err;
+ err = -EINVAL;
+
sb = kmap_atomic(sb_page);
chunksize = le32_to_cpu(sb->chunksize);
Patches currently in stable-queue which might be from neilb@suse.com are
queue-3.10/md-raid1-extend-spinlock-to-protect-raid1_end_read_request-against-inconsistencies.patch
queue-3.10/md-bitmap-return-an-error-when-bitmap-superblock-is-corrupt.patch
queue-3.10/md-use-kzalloc-when-bitmap-is-disabled.patch
^ permalink raw reply
* Re: [PATCH stable] md/bitmap: return an error when bitmap superblock is corrupt.
From: Greg KH @ 2015-08-14 17:23 UTC (permalink / raw)
To: NeilBrown; +Cc: stable, linux-raid, GuoQing Jiang, Goldwyn Rodrigues, lkml
In-Reply-To: <20150814170421.60d40eb0@noble>
On Fri, Aug 14, 2015 at 05:04:21PM +1000, NeilBrown wrote:
>
> commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49 upstream
> Use separate bitmaps for each nodes in the cluster
>
> bitmap_read_sb() validates the bitmap superblock that it reads in.
> If it finds an inconsistency like a bad magic number or out-of-range
> version number, it prints an error and returns, but it incorrectly
> returns zero, so the array is still assembled with the (invalid) bitmap.
>
> This means it could try to use a bitmap with a new version number which
> it therefore does not understand.
>
> This bug was introduced in 3.5 and fix as part of a larger patch in 4.1.
> So the patch is suitable for any -stable kernel in that range.
>
> Fixes: 27581e5ae01f ("md/bitmap: centralise allocation of bitmap file pages.")
> Cc: stable@vger.kernel.org (v3.5..v4.1)
b97e92574c0 is in 4.1, so do you mean (v3.5..v4.0) here?
thanks,
greg k-h
^ permalink raw reply
* Re: [PATCH] drivers/md/md.c: ignore recovery_offset if bitmap exists
From: Nate Dailey @ 2015-08-14 14:58 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, Jes.Sorensen
In-Reply-To: <55B93B9D.5000103@stratus.com>
I hate to nag... but looking for feedback on this change, which addresses what
seems to me to be a serious bug.
Thanks,
Nate
On 07/29/2015 04:46 PM, Joe Lawrence wrote:
> On 07/28/2015 03:28 PM, Nate Dailey wrote:
>> If a bitmap recovery is interrupted and later restarted, then
>> sectors below the recovery offset, written between interruption
>> and resumption, will not be copied. This results in corruption.
>>
>> See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777511
>> for a script that can be used to repro this.
>>
>> Seems like ignoring the recovery_offset if a bitmap exists is
>> the way to go.
>>
>> Signed-off-by: Nate Dailey <nate.dailey@stratus.com>
>> ---
>> drivers/md/md.c | 24 +++++++++++++-----------
>> 1 file changed, 13 insertions(+), 11 deletions(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 0c2a4e8..79c6285 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -7738,16 +7738,18 @@ void md_do_sync(struct md_thread *thread)
>> else {
>> /* recovery follows the physical size of devices */
>> max_sectors = mddev->dev_sectors;
>> - j = MaxSector;
>> - rcu_read_lock();
>> - rdev_for_each_rcu(rdev, mddev)
>> - if (rdev->raid_disk >= 0 &&
>> - !test_bit(Faulty, &rdev->flags) &&
>> - !test_bit(In_sync, &rdev->flags) &&
>> - rdev->recovery_offset < j)
>> - j = rdev->recovery_offset;
>> - rcu_read_unlock();
>> -
>> + /* we don't use the offset if there's a bitmap */
>> + if (!mddev->bitmap) {
>> + j = MaxSector;
>> + rcu_read_lock();
>> + rdev_for_each_rcu(rdev, mddev)
>> + if (rdev->raid_disk >= 0 &&
>> + !test_bit(Faulty, &rdev->flags) &&
>> + !test_bit(In_sync, &rdev->flags) &&
>> + rdev->recovery_offset < j)
>> + j = rdev->recovery_offset;
>> + rcu_read_unlock();
>> + }
>> /* If there is a bitmap, we need to make sure all
>> * writes that started before we added a spare
>> * complete before we start doing a recovery.
>> @@ -7756,7 +7758,7 @@ void md_do_sync(struct md_thread *thread)
>> * recovery has checked that bit and skipped that
>> * region.
>> */
>> - if (mddev->bitmap) {
>> + else {
>> mddev->pers->quiesce(mddev, 1);
>> mddev->pers->quiesce(mddev, 0);
>> }
>>
> [+cc Ben & Cyril from the Debian bug report]
>
> -- Joe
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Failed to grow
From: Daniel Koch @ 2015-08-14 9:22 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <621f0ebc8080e4ab56eaacde5f9416cf@mail.triple6.org>
I crawled the archive of the list and found that --invalid-backup with a
faked and empty backup might help. Here is the result:
mdadm --assemble --force --run --verbose /dev/md0
--backup-file=empty.bak --invalid-backup
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sda5
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdn is identified as a member of /dev/md0, slot 9.
mdadm: /dev/sdj is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdh is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdm is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sdl is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdk is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdi is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 10.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 12.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 11.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: :/dev/md0 has an active reshape - checking if critical section
needs to be restored
mdadm: Cannot read from empty.bak
mdadm: No backup metadata on device-12
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdg to /dev/md0 as 2
mdadm: added /dev/sdh to /dev/md0 as 3
mdadm: added /dev/sdi to /dev/md0 as 4
mdadm: added /dev/sdj to /dev/md0 as 5
mdadm: added /dev/sdk to /dev/md0 as 6
mdadm: added /dev/sdl to /dev/md0 as 7
mdadm: added /dev/sdm to /dev/md0 as 8
mdadm: added /dev/sdn to /dev/md0 as 9
mdadm: added /dev/sdd to /dev/md0 as 10
mdadm: added /dev/sdf to /dev/md0 as 11
mdadm: added /dev/sde to /dev/md0 as 12
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
kern.log:
Aug 14 11:18:20 claw-storage kernel: [ 599.182677] md: md0 stopped.
Aug 14 11:18:20 claw-storage kernel: [ 599.184459] sdn: unknown
partition table
Aug 14 11:18:20 claw-storage kernel: [ 599.185604] sdj: unknown
partition table
Aug 14 11:18:20 claw-storage kernel: [ 599.186424] sdm: unknown
partition table
Aug 14 11:18:20 claw-storage kernel: [ 599.187869] sdl: unknown
partition table
Aug 14 11:18:20 claw-storage kernel: [ 599.189108] sdi: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.213722] sde: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.214073] md: bind<sdc>
Aug 14 11:18:21 claw-storage kernel: [ 599.214248] md: bind<sdg>
Aug 14 11:18:21 claw-storage kernel: [ 599.214427] md: bind<sdh>
Aug 14 11:18:21 claw-storage kernel: [ 599.214438] sdb: sdb1
Aug 14 11:18:21 claw-storage kernel: [ 599.214620] md: bind<sdi>
Aug 14 11:18:21 claw-storage kernel: [ 599.214845] md: bind<sdj>
Aug 14 11:18:21 claw-storage kernel: [ 599.215000] md: bind<sdk>
Aug 14 11:18:21 claw-storage kernel: [ 599.215140] md: bind<sdl>
Aug 14 11:18:21 claw-storage kernel: [ 599.215278] sdf: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.215337] md: bind<sdm>
Aug 14 11:18:21 claw-storage kernel: [ 599.215477] md: bind<sdn>
Aug 14 11:18:21 claw-storage kernel: [ 599.215588] md: bind<sdd>
Aug 14 11:18:21 claw-storage kernel: [ 599.216165] md: bind<sdf>
Aug 14 11:18:21 claw-storage kernel: [ 599.216400] sde: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.216923] md: bind<sde>
Aug 14 11:18:21 claw-storage kernel: [ 599.217159] sdf: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.217513] md: bind<sdb>
Aug 14 11:18:21 claw-storage kernel: [ 599.218292] sdl: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.218743] md/raid:md0:
reshape_position too early for auto-recovery - aborting.
Aug 14 11:18:21 claw-storage kernel: [ 599.220772] md: pers->run()
failed ...
Aug 14 11:18:21 claw-storage kernel: [ 599.222047] sdi: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.222135] md: md0 stopped.
Aug 14 11:18:21 claw-storage kernel: [ 599.222142] md: unbind<sdb>
Aug 14 11:18:21 claw-storage kernel: [ 599.222616] sdb: sdb1
Aug 14 11:18:21 claw-storage kernel: [ 599.223612] sdi: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.224397] sdl: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.225077] sdf: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.225907] sde: unknown
partition table
Aug 14 11:18:21 claw-storage kernel: [ 599.226498] sdb: sdb1
Aug 14 11:18:21 claw-storage kernel: [ 599.232193] md: export_rdev(sdb)
Aug 14 11:18:21 claw-storage kernel: [ 599.232205] md: unbind<sde>
Aug 14 11:18:21 claw-storage kernel: [ 599.247747] md: export_rdev(sde)
Aug 14 11:18:21 claw-storage kernel: [ 599.247766] md: unbind<sdf>
Aug 14 11:18:21 claw-storage kernel: [ 599.271766] md: export_rdev(sdf)
Aug 14 11:18:21 claw-storage kernel: [ 599.271774] md: unbind<sdd>
Aug 14 11:18:21 claw-storage kernel: [ 599.287796] md: export_rdev(sdd)
Aug 14 11:18:21 claw-storage kernel: [ 599.287818] md: unbind<sdn>
Aug 14 11:18:21 claw-storage kernel: [ 599.319830] md: export_rdev(sdn)
Aug 14 11:18:21 claw-storage kernel: [ 599.319841] md: unbind<sdm>
Aug 14 11:18:21 claw-storage kernel: [ 599.331804] md: export_rdev(sdm)
Aug 14 11:18:21 claw-storage kernel: [ 599.331816] md: unbind<sdl>
Aug 14 11:18:21 claw-storage kernel: [ 599.346199] md: export_rdev(sdl)
Aug 14 11:18:21 claw-storage kernel: [ 599.346221] md: unbind<sdk>
Aug 14 11:18:21 claw-storage kernel: [ 599.355791] md: export_rdev(sdk)
Aug 14 11:18:21 claw-storage kernel: [ 599.355805] md: unbind<sdj>
Aug 14 11:18:21 claw-storage kernel: [ 599.371779] md: export_rdev(sdj)
Aug 14 11:18:21 claw-storage kernel: [ 599.371794] md: unbind<sdi>
Aug 14 11:18:21 claw-storage kernel: [ 599.387838] md: export_rdev(sdi)
Aug 14 11:18:21 claw-storage kernel: [ 599.387850] md: unbind<sdh>
Aug 14 11:18:21 claw-storage kernel: [ 599.403814] md: export_rdev(sdh)
Aug 14 11:18:21 claw-storage kernel: [ 599.403839] md: unbind<sdg>
Aug 14 11:18:21 claw-storage kernel: [ 599.431842] md: export_rdev(sdg)
Aug 14 11:18:21 claw-storage kernel: [ 599.431852] md: unbind<sdc>
Aug 14 11:18:21 claw-storage kernel: [ 599.463878] md: export_rdev(sdc)
Aug 14 11:18:21 claw-storage kernel: [ 599.465322] md: bind<sdn>
Aug 14 11:18:21 claw-storage kernel: [ 599.467200] md: bind<sdj>
Aug 14 11:18:21 claw-storage kernel: [ 599.469274] md: bind<sde>
Aug 14 11:18:21 claw-storage kernel: [ 599.471375] md: bind<sdi>
Aug 14 11:18:21 claw-storage kernel: [ 599.473784] md: bind<sdf>
Aug 14 11:18:21 claw-storage kernel: [ 599.476780] md: bind<sdl>
Aug 14 11:18:21 claw-storage kernel: [ 599.480103] md: bind<sdm>
Aug 14 11:18:21 claw-storage kernel: [ 599.488333] md: bind<sdb>
#cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdm[8](S) sdn[9](S) sdj[5](S) sdb[0](S) sdf[11](S)
sdi[4](S) sde[13](S) sdl[12](S)
14161940096 blocks super 1.2
unused devices: <none>
So no luck at all. All disks are accessable using smart which makes me
think they are operational.
---
Viele Grüße,
Daniel Koch
Am 2015-08-13 17:53, schrieb Daniel Koch:
> Hello every one,
>
> I have a huge problem. I started to grow a array with ( the added
> device is /dev/sde )
> # mdadm --grow --raid-devices=13 /dev/md0 --backup-file=md0.bak
>
> after that there way 0 disk activity so i checked it with
> # cat /proc/mdstat
>
> everything looked fine but speed was 0K/sec for about 1-2 hours. I
> decided to reboot the system as grow should continue after that. After
> that the array wasn't assembled at all. So i wanted to do it myself:
>
> # mdadm --assemble /dev/md0 /dev/sd[b-n] --backup-file=md0.bak
> --verbose
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 10.
> mdadm: /dev/sde is identified as a member of /dev/md0, slot 12.
> mdadm: /dev/sdf is identified as a member of /dev/md0, slot 11.
> mdadm: /dev/sdg is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdh is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdi is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sdj is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdk is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdl is identified as a member of /dev/md0, slot 7.
> mdadm: /dev/sdm is identified as a member of /dev/md0, slot 8.
> mdadm: /dev/sdn is identified as a member of /dev/md0, slot 9.
> mdadm: :/dev/md0 has an active reshape - checking if critical section
> needs to be restored
> mdadm: No backup metadata on md0.bak
> mdadm: No backup metadata on device-12
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
> Backup looks broken. What can i do ?
>
> # mdadm --version
> mdadm - v3.3.2 - 21st August 2014
>
> # uname -a
> Linux claw-storage 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
> (2015-08-04) x86_64 GNU/Linux
>
> # du -sch md0.bak
> 56M md0.bak
>
> # mdadm --examine /dev/sd[b-n]
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=976752816 sectors
> State : clean
> Device UUID : f3d9392c:a9290f8e:4ded7efc:284b4ecf
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 11cb12c0 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdc:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=976752816 sectors
> State : clean
> Device UUID : a5b4533d:d5a4cdc5:e6dcd3aa:4a845044
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 7dfa9159 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdd:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=976752816 sectors
> State : clean
> Device UUID : d39aede2:af92f321:8b6fbd9f:45c73ddf
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : f913b0b - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 10
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sde:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=976752816 sectors
> State : clean
> Device UUID : 13fcdab1:69ce4358:5f1ee2b2:c809b990
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : d976c9a6 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 12
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdf:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=976752816 sectors
> State : clean
> Device UUID : 487f70bd:b63c29d6:185aa418:208dc16d
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : a5b479ef - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 11
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdg:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : b798e990:6b50f1a0:3cde1ced:3a20dfaf
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 2053b0c9 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdh:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : bfe03a04:fda138cc:ef11727c:ff1b58d2
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 70ba79db - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdi:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=976752816 sectors
> State : clean
> Device UUID : 4fdb8a05:3157bd26:f93ee8d7:fdd42f9f
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 2f151d28 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 4
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdj:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : 6010b774:e02b08ed:29484f97:b03e8a48
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 93158c4c - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 5
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdk:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : afab65ff:a79ed9e7:30ccd918:e2e1c21a
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 6c58c19c - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 6
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdl:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=976752816 sectors
> State : clean
> Device UUID : 8a9031cf:f9b5a888:70363d53:b269047d
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : b3d8bda7 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 7
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdm:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : 1eefdd8c:75b58c0c:f0c555bd:e704e89e
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : 472538a0 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 8
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
> /dev/sdn:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 633c1520:61c07f69:26a83b45:b9756587
> Name : claw-storage:0 (local to host claw-storage)
> Creation Time : Tue Jan 14 12:48:49 2014
> Raid Level : raid6
> Raid Devices : 13
>
> Avail Dev Size : 2930015024 (1397.14 GiB 1500.17 GB)
> Array Size : 16115078144 (15368.54 GiB 16501.84 GB)
> Used Dev Size : 2930014208 (1397.14 GiB 1500.17 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=816 sectors
> State : clean
> Device UUID : b8f9a259:c4522982:88538283:497eed43
>
> Reshape pos'n : 0
> Delta Devices : 1 (12->13)
>
> Update Time : Thu Aug 13 17:11:05 2015
> Checksum : f4b8e783 - correct
> Events : 72402
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 9
> Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
> replacing)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: How will mdadm handle a wrongly added drive, when the original comes back on line?
From: Wilson, Jonathan @ 2015-08-14 8:04 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <55BFFC6A.6030006@websitemanagers.com.au>
On Tue, 2015-08-04 at 09:42 +1000, Adam Goryachev wrote:
> On 03/08/15 23:14, Wilson, Jonathan wrote:
> > Due to a bug in the driver for a Marvel chipset 4 port SATA card I think
> > I may have added an empty drive partition into a raid6 array and when I
> > get a new card I it will end up seeing not only the new drive, but also
> > the "missing" drive.
> >
> > Events:
> > Upgraded jessie with latest updates (quite some time since I last did
> > it) and re-booted.
> >
> > A 6 drive raid6 assembled, but all the drives were spare. Stopped the
> > array and did a mdadm --assemble /dev/md6.
> >
> > It assembled with 5 drives, one missing.
> >
> > Tried re-add, which failed, and then -add which completed ok.
> At this point the array should have done a resync to add the 6th drive.
> > Some time later I re-booted and the same problem happened.
> >
> > All drives spare, stopped, assembled, added missing.
> At this point the array should have done a resync to add the 6th drive.
> Whether this is the same "6th" drive or not doesn't matter.
> > Its now working and I have a new card on order due to something going
> > badly wrong with the driver and/or card and/or chipset (Marvel 9230).
> >
> > After some time passed after the second boot, I realised that one of my
> > drives was physically missing. I had a drive ready to go as a genuine
> > spare but not yet added as a spare to mdadm, so in theory it should have
> > been totally empty apart from a partition.
> >
> > Now my problem is that firstly I can not be sure that when I looked
> > at /proc/mdstat/ and saw "all" the drives as spare there might have been
> > a missing one. (On either or both occasions.)
> >
> > In my mdadm.config I don't specify the number of drives in the array,
> > just its name and the UUID.
> >
> > Now my question is: if we call the drives in the array A,B,C,D,E,F and
> > the empty one G.
> >
> > After the first boot I may have added G, so the array would be
> > A,B,C,D,E,G. (F missing from system)
> >
> > After the second boot I may have added F back, so the array would be
> > A,B,C,D,E,F (G missing from system)
> >
> > If after changing the card the system sees A,B,C,D,E,F,G how will mdadm
> > work? Will it fail to assemble as one of the drives is "extra" to the
> > metadata count (I assume even though I don't specify a count in the
> > conf, that internally on the partitions of the disks in the array it
> > knows there should be "6" disks.
> It should reject the "older" 6th drive because the event count will be
> older, and should auto-assemble with all the other drives. The older
> "6th" drive will either be spare, or not added to the array at all, and
> you would need to add it to the array for it to become a spare.
I wanted to thank you for your help with this post some days ago.
The drives had indeed swapped so I had 7 disks in a 6 disk array. After
swapping the 4 port Marvel chipset card for two 2 port ASM1062's
(Interestingly enough these happened to be exactly the same chips as the
additional on board extra satas) the system booted with a nice clean
dmesg log. (No "red line" errors)
The older, original, drive of the array was kicked:
> [ 3.269932] md: bind<sdb4>
> [ 3.272343] md: bind<sda4>
> [ 3.274762] md: raid10 personality registered for level 10
> [ 3.275748] md/raid10:md4: active with 2 out of 2 devices
> [ 3.276617] md4: detected capacity change from 0 to 64390955008
> [ 3.277958] md4: unknown partition table
> [ 3.346450] md: bind<sdn6>
> [ 3.370188] md: bind<sdg6>
> [ 3.372120] md: kicking non-fresh sdh6 from array! <<<<<<<<<<<<<<<<<<<<<
> [ 3.372956] md: unbind<sdh6>
> [ 3.383684] md: export_rdev(sdh6)
> [ 3.452610] raid6: sse2x1 13949 MB/s
> [ 3.520586] raid6: sse2x2 17910 MB/s
> [ 3.588568] raid6: sse2x4 20617 MB/s
> [ 3.656547] raid6: avx2x1 27298 MB/s
> [ 3.724527] raid6: avx2x2 32586 MB/s
> [ 3.792508] raid6: avx2x4 36498 MB/s
> [ 3.793243] raid6: using algorithm avx2x4 (36498 MB/s)
> [ 3.793973] raid6: using avx2x2 recovery algorithm
> [ 3.794829] xor: automatically using best checksumming function:
> [ 3.832495] avx : 43866.000 MB/sec
> [ 3.833351] async_tx: api initialized (async)
> [ 3.834559] md: raid6 personality registered for level 6
> [ 3.835255] md: raid5 personality registered for level 5
> [ 3.835945] md: raid4 personality registered for level 4
> [ 3.836716] md/raid:md6: device sdg6 operational as raid disk 0
> [ 3.837391] md/raid:md6: device sdn6 operational as raid disk 5
> [ 3.838043] md/raid:md6: device sdl6 operational as raid disk 1
> [ 3.838688] md/raid:md6: device sdm6 operational as raid disk 4
> [ 3.839315] md/raid:md6: device sdj6 operational as raid disk 2
> [ 3.839916] md/raid:md6: device sdi6 operational as raid disk 3
> [ 3.840715] md/raid:md6: allocated 0kB
> [ 3.841341] md/raid:md6: raid level 6 active with 6 out of 6 devices, algorithm 2
> [ 3.841963] RAID conf printout:
> [ 3.841963] --- level:6 rd:6 wd:6
> [ 3.841964] disk 0, o:1, dev:sdg6
> [ 3.841965] disk 1, o:1, dev:sdl6
> [ 3.841965] disk 2, o:1, dev:sdj6
> [ 3.841966] disk 3, o:1, dev:sdi6
> [ 3.841966] disk 4, o:1, dev:sdm6
> [ 3.841967] disk 5, o:1, dev:sdn6
> [ 3.842047] created bitmap (22 pages) for device md6
> [ 3.842997] md6: bitmap initialized from disk: read 2 pages, set 0 of 43172 bits
> [ 3.855294] md6: detected capacity change from 0 to 11588669014016
>
The booted drive is now sitting as "inactive" so when I get time I will
clear it and add it as a hot spare.
Thanks again, and thanks to Neil and others for all their hard work in
developing mdadm.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox