* Re: Fwd: I will pay money for the correct RAID recovery instructions
From: John Stoffel @ 2014-10-17 16:14 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GEbNPAWfBL_uPDOuhMb4LtN+_fJ07MD7igOQmVi04mWpA@mail.gmail.com>
Another> Sorry to state the obvious but...... To restore the degraded
Another> array would (based on the info you've posted) likely take
Another> longer then temporarily moving the data to a different set of
Another> drives.
How do you figure this? He hasn't posted any useful data on his array
at all. We know it's six 3Tb disks in RAID10, so that's at most 9Tb
of data. Moving 3Tb (bad disk ddrescue'd onto new disk) is only 3Tb
to move. Then the resync will of course take time.
But if he doesn't have the spare disk space, it's probably the only way.
Another> As time seems to be a major consideration here (likely/possible
Another> failure of sde) then surely the optimal strategy has to be to get the
Another> data off first, then look at the rebuilding the degraded array?
Another> Just my 2c
Another> On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>>
>>
>> Ian,
>>
>> It would also help if you posted the details of your setup using:
>>
>> cat /proc/partitions
>> cat /proc/mdstat
>>
>> mdadm -D /dev/md#
>> - for each of the devices above.
>>
>> mdadm -E /dev/sd<drive><#>
>> - for each disk or partition in the array from above.
>>
>>
>>
>> But the suggestions to ddrescue the going bad drive onto a new disk is
>> a good one. On my debian system, I would do the following:
>>
>> sudo apt-get install gddrescue
>> ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>>
>> and see how that goes.
>>
>> Good luck,
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Another> --
Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Another> the body of a message to majordomo@vger.kernel.org
Another> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Fwd: I will pay money for the correct RAID recovery instructions
From: Another Sillyname @ 2014-10-17 15:13 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAOS+5GGMGU2doP=v1LfeA5CDDmuMj4gxUcNtoK5-gJWHO9CMaw@mail.gmail.com>
Sorry to state the obvious but......
To restore the degraded array would (based on the info you've posted)
likely take longer then temporarily moving the data to a different set
of drives.
As time seems to be a major consideration here (likely/possible
failure of sde) then surely the optimal strategy has to be to get the
data off first, then look at the rebuilding the degraded array?
Just my 2c
On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>
>
> Ian,
>
> It would also help if you posted the details of your setup using:
>
> cat /proc/partitions
> cat /proc/mdstat
>
> mdadm -D /dev/md#
> - for each of the devices above.
>
> mdadm -E /dev/sd<drive><#>
> - for each disk or partition in the array from above.
>
>
>
> But the suggestions to ddrescue the going bad drive onto a new disk is
> a good one. On my debian system, I would do the following:
>
> sudo apt-get install gddrescue
> ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>
> and see how that goes.
>
> Good luck,
> John
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: I will pay money for the correct RAID recovery instructions
From: John Stoffel @ 2014-10-17 14:05 UTC (permalink / raw)
To: Ian Young; +Cc: linux-raid
In-Reply-To: <CANs+QMy7d+kAxazgdk_r+iPtK45+ddSMwEQg6M_6e4giqJgJmw@mail.gmail.com>
Ian,
It would also help if you posted the details of your setup using:
cat /proc/partitions
cat /proc/mdstat
mdadm -D /dev/md#
- for each of the devices above.
mdadm -E /dev/sd<drive><#>
- for each disk or partition in the array from above.
But the suggestions to ddrescue the going bad drive onto a new disk is
a good one. On my debian system, I would do the following:
sudo apt-get install gddrescue
ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
and see how that goes.
Good luck,
John
^ permalink raw reply
* Re: I will pay money for the correct RAID recovery instructions
From: Ian Young @ 2014-10-16 22:08 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20141016202202.GA22898@cthulhu.home.robinhill.me.uk>
Ok, if I can pull this off I owe you a beer.
On Thu, Oct 16, 2014 at 1:22 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Thu Oct 16, 2014 at 12:59:18pm -0700, Ian Young wrote:
>
>> I've been trying to fix a degraded array for a couple of months now
>> and it's getting frustrating enough that I'm willing to put a bounty
>> on the correct solution. The array can start in a degraded state and
>> the data is accessible, so I know this is possible to fix. Any
>> takers? I'll bet someone could use some beer money or a contribution
>> to their web hosting costs.
>>
>> Here's how the system is set up: There are (6) 3 TB drives. Each
>> drive has a BIOS boot partition. The rest of the space on each drive
>> is a large GPT partition that is combined in a RAID 10 array. On top
>> of the array there are four LVM volumes: /boot, /root, swap, and /srv.
>>
>> Here's the problem: /dev/sdf failed. I replaced it but as it was
>> resyncing, read errors on /dev/sde kicked the new sdf out and made it
>> a spare. The array is now in a precarious degraded state. All it
>> would take for the entire array to fail is for /dev/sde to fail, and
>> it's already showing signs that it will. I have tried forcing the
>> array to assemble using /dev/sd[abcde]2 and then forcing it to add
>> /dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
>> /sys/block/md0/md/sync_action" but that finishes immediately and
>> changes nothing.
>>
> If sdf didn't finish syncing then it's no use adding it to the array as
> anything other than a spare. Also, you can't run a check on a degraded
> array (as there's nothing to check against), which is why that's
> finishing immediately.
>
> If sde is giving a read error during rebuild then the solution is to
> stop the array (you'll need to do this via a bootable CD/USB stick I
> guess) and use ddrescue to duplicate sde onto a new disk, The
> read errors may well mean that some can't be copied (though ddrescue
> will try very hard to do so), which may cause file/filesystem corruption
> later. You can then reassemble the (degraded) array with the old sda-sdd
> and the new sde, then add sdf and wait for the array to recover. You
> can then run a fsck on the filesystem to check for any corruption there.
> File corruption is a lot trickier to spot - if you have checksums for
> the files then that's one way, otherwise you may be able to work out
> what files are affected based on the offsets of the missing data (that's
> rather beyond the limits of my knowledge though).
>
> HTH,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: I will pay money for the correct RAID recovery instructions
From: Robin Hill @ 2014-10-16 20:22 UTC (permalink / raw)
To: Ian Young; +Cc: linux-raid
In-Reply-To: <CANs+QMz3jNpAneHLzN5c=EtK+RAOhQJZy=n3P0oYTRm7yNDzjg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2648 bytes --]
On Thu Oct 16, 2014 at 12:59:18pm -0700, Ian Young wrote:
> I've been trying to fix a degraded array for a couple of months now
> and it's getting frustrating enough that I'm willing to put a bounty
> on the correct solution. The array can start in a degraded state and
> the data is accessible, so I know this is possible to fix. Any
> takers? I'll bet someone could use some beer money or a contribution
> to their web hosting costs.
>
> Here's how the system is set up: There are (6) 3 TB drives. Each
> drive has a BIOS boot partition. The rest of the space on each drive
> is a large GPT partition that is combined in a RAID 10 array. On top
> of the array there are four LVM volumes: /boot, /root, swap, and /srv.
>
> Here's the problem: /dev/sdf failed. I replaced it but as it was
> resyncing, read errors on /dev/sde kicked the new sdf out and made it
> a spare. The array is now in a precarious degraded state. All it
> would take for the entire array to fail is for /dev/sde to fail, and
> it's already showing signs that it will. I have tried forcing the
> array to assemble using /dev/sd[abcde]2 and then forcing it to add
> /dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
> /sys/block/md0/md/sync_action" but that finishes immediately and
> changes nothing.
>
If sdf didn't finish syncing then it's no use adding it to the array as
anything other than a spare. Also, you can't run a check on a degraded
array (as there's nothing to check against), which is why that's
finishing immediately.
If sde is giving a read error during rebuild then the solution is to
stop the array (you'll need to do this via a bootable CD/USB stick I
guess) and use ddrescue to duplicate sde onto a new disk, The
read errors may well mean that some can't be copied (though ddrescue
will try very hard to do so), which may cause file/filesystem corruption
later. You can then reassemble the (degraded) array with the old sda-sdd
and the new sde, then add sdf and wait for the array to recover. You
can then run a fsck on the filesystem to check for any corruption there.
File corruption is a lot trickier to spot - if you have checksums for
the files then that's one way, otherwise you may be able to work out
what files are affected based on the offsets of the missing data (that's
rather beyond the limits of my knowledge though).
HTH,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* I will pay money for the correct RAID recovery instructions
From: Ian Young @ 2014-10-16 19:59 UTC (permalink / raw)
To: linux-raid
I've been trying to fix a degraded array for a couple of months now
and it's getting frustrating enough that I'm willing to put a bounty
on the correct solution. The array can start in a degraded state and
the data is accessible, so I know this is possible to fix. Any
takers? I'll bet someone could use some beer money or a contribution
to their web hosting costs.
Here's how the system is set up: There are (6) 3 TB drives. Each
drive has a BIOS boot partition. The rest of the space on each drive
is a large GPT partition that is combined in a RAID 10 array. On top
of the array there are four LVM volumes: /boot, /root, swap, and /srv.
Here's the problem: /dev/sdf failed. I replaced it but as it was
resyncing, read errors on /dev/sde kicked the new sdf out and made it
a spare. The array is now in a precarious degraded state. All it
would take for the entire array to fail is for /dev/sde to fail, and
it's already showing signs that it will. I have tried forcing the
array to assemble using /dev/sd[abcde]2 and then forcing it to add
/dev/sdf2. That still adds sdf2 as a spare. I've tried "echo check >
/sys/block/md0/md/sync_action" but that finishes immediately and
changes nothing.
Can anyone solve this? I'd be happy to pay you for your knowledge.
^ permalink raw reply
* Re: Preventing automatic start of ARRAY on Kubuntu
From: George Rapp @ 2014-10-16 16:19 UTC (permalink / raw)
To: P. Gautschi; +Cc: linux-raid
In-Reply-To: <543FEB86.5070902@gautschi.net>
On Thu, Oct 16, 2014 at 12:00 PM, P. Gautschi <linuxlist@gautschi.net> wrote:
>
> I'm currently experimenting with RAID5 on Kubuntu 14.10 Beta 2.
>
> When the system boots, /dev/md127 is automatically started. This occurs
> whether the line (output after creation of mdadm --detail --scan)
> ARRAY /dev/md0 metadata=1.2 name=test:0 UUID=2e0d5db2:47c49e4b:75c06b8b:11bf68cf
> is in /etc/mdadm/mdadm.conf or not.
>
> I would expected that the array is only started when the line is in the file
> and then as /dev/md0 and not as md127.
>
> My final goal is to prevent the start of the array when it is degraded
> (missing or previously failed disk).
>
> I had problems with a power cable and one disk had failed. While checking
> the cables a second disk failed temporarily. Even though I wasn't writing
> to the fs on the array I ended with and array that could not be assembled
> again, after all disk were working again. (I have tried reboots,
> mdadm --assemble --force and the other tricks from
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> but ended with a corrupt fs.)
>
> I'm now trying to minimize the risk of data loss. Not starting the
> array when degraded (or starting it read-only) is one of my ideas
> but I don't know where and how to implement this.
[re-sending this without the HTML, which the list manager doesn't care for]
Patrick -
Here's what I did (on Fedora, but I suspect the process will be
similar on any distro):
Make whatever changes you need to in /etc/mdadm.conf (for example,
removing /dev/md0 and /dev/md127);
Create a new initramfs to reflect your changes:
# cd /boot
# mv initramfs-$(uname -r).img initramfs-$(uname -r).img.backup
# dracut --mdadmconf /boot/initramfs-$(uname -r).img $(uname -r)
(The dracut command is not provided in Fedora by default, but requires
a package; in Fedora 20, the current package is
dracut-037-11.git20140402.fc20.i686.)
This creates a new initramfs with your updated mdadm.conf. After you
reboot with the new initramfs image, you will have to start whatever
MD array(s) you want manually. At this point, for example, you could
try to start the array without --force, and it would refuse to start
if degraded.
> Another idea is, that the array could switch to read only when
> degraded during runtime. Ideally this would be delayed until the
> fs does a flush/sync to be in a recoverable state. Of course file
> accesses would fail is such a situation but it prefer this to
> the rick of loosing data.
> Is it possible to do this?
My Fedora 20 kernel (3.16.4-200.fc20.i686+PAE) seems to do this
automatically for me. My RAID 5 suffered a disk failure that dropped
it down to one (of three) component devices, and the kernel stopped
writes immediately to prevent data loss. After powering off and
reassembling my array with --force to sync up the event counters, I
didn't lose anything.
Good luck.
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
Work: george.rapp -- at -- hp.com (or) george.rapp.ctr -- at -- dfas.mil
A wise and frugal government, which shall restrain men from injuring
one another, which shall leave them otherwise free to regulate their
own pursuits of industry and improvement, and shall not take from the
mouth of labor the bread it has earned. This is the sum of
good government... - Thomas Jefferson, First Inaugural Address
^ permalink raw reply
* Preventing automatic start of ARRAY on Kubuntu
From: P. Gautschi @ 2014-10-16 16:00 UTC (permalink / raw)
To: linux-raid
I'm currently experimenting with RAID5 on Kubuntu 14.10 Beta 2.
When the system boots, /dev/md127 is automatically started. This occurs
whether the line (output after creation of mdadm --detail --scan)
ARRAY /dev/md0 metadata=1.2 name=test:0 UUID=2e0d5db2:47c49e4b:75c06b8b:11bf68cf
is in /etc/mdadm/mdadm.conf or not.
I would expected that the array is only started when the line is in the file
and then as /dev/md0 and not as md127.
My final goal is to prevent the start of the array when it is degraded
(missing or previously failed disk).
I had problems with a power cable and one disk had failed. While checking
the cables a second disk failed temporarily. Even though I wasn't writing
to the fs on the array I ended with and array that could not be assembled
again, after all disk were working again. (I have tried reboots,
mdadm --assemble --force and the other tricks from
https://raid.wiki.kernel.org/index.php/RAID_Recovery
but ended with a corrupt fs.)
I'm now trying to minimize the risk of data loss. Not starting the
array when degraded (or starting it read-only) is one of my ideas
but I don't know where and how to implement this.
Another idea is, that the array could switch to read only when
degraded during runtime. Ideally this would be delayed until the
fs does a flush/sync to be in a recoverable state. Of course file
accesses would fail is such a situation but it prefer this to
the rick of loosing data.
Is it possible to do this?
Patrick
^ permalink raw reply
* Re: migrate to bad block list
From: NeilBrown @ 2014-10-16 6:54 UTC (permalink / raw)
To: Michael Ryan; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <E1D69AD765F9D841B8DE87D96DD21E4505DFBADD@AEMAILMBX02.lenovo.com>
[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]
On Wed, 15 Oct 2014 17:18:19 +0000 Michael Ryan <mryan@lenovoemc.com> wrote:
>
> Is there any way to migrate an existing array created with mdadm v3.2.5 and using 1.1 metadata to use a bad block list? I'm assuming not as there wouldn't be space reserved for the list, but I need to ask.
>
> Thanks for your response!
mdadm tends to leave a fair bit of unused space on devices so that things
like a bad block list can easily be added.
If you can stop the array, then do that and re-assemble with
--update=bbl
and you should get a bbl added to each device.
If you cannot stop the array, but it has a bitmap, then
you can, for each device:
mdadm /dev/mdX --fail /dev/adevice
mdadm /dev/mdX --remove /dev/adevice
mdadm /dev/mdX --re-add --update=bbl /dev/adevice
I think that should work. The "bblk" is a feature of the device, not of the
whole array. So you can add it to each device.
I haven't actually tested the above I think, so it might be safest to make an
array with loop-back devices and experiment.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* [GIT PULL REQUEST] md updates for 3.18
From: NeilBrown @ 2014-10-16 6:32 UTC (permalink / raw)
To: Linus Torvalds
Cc: lkml, linux RAID, Baldysiak, Pawel, BillStuff, Anton Blanchard,
Chao Yu, Markus Stockhausen
[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]
Hi Linus,
here are some md updates for 3.18.
Thanks,
NeilBrown
The following changes since commit 8e0e99ba64c7ba46133a7c8a3e3f7de01f23bd93:
md/raid5: disable 'DISCARD' by default due to safety concerns. (2014-10-02 13:45:00 +1000)
are available in the git repository at:
git://neil.brown.name/md tags/md/3.18
for you to fetch changes up to b395f75eabb3844c99244928293796ff42feaa3d:
lib/raid6: Add log level to printks (2014-10-14 13:08:29 +1100)
----------------------------------------------------------------
md updates for 3.18
- a few minor bug fixes
- quite a lot of code tidy-up and simplification
- remove PRINT_RAID_DEBUG ioctl. I'm fairly sure
it is unused, and it isn't particularly useful.
----------------------------------------------------------------
Anton Blanchard (1):
lib/raid6: Add log level to printks
Chao Yu (1):
md: avoid potential long delay under pers_lock
Markus Stockhausen (1):
md/raid5: fix init_stripe() inconsistencies
NeilBrown (18):
md/bitmap: always wait for writes on unplug.
md/raid1: minor typos and reformatting.
md: use set_bit/clear_bit instead of shift/mask for bi_flags changes.
md/raid10: another memory leak due to reshape.
md/raid1: process_checks doesn't use its return value.
md: be more relaxed about stopping an array which isn't started.
md: use wait_event() to simplify md_super_wait()
md: discard find_rdev_nr in favour of find_rdev_nr_rcu
md: simplify export_array()
md: Just use RCU when checking for overlap between arrays.
md: don't start resync thread directly from md thread.
md: remove unwanted white space from md.c
md: don't allow "-sync" to be set for device in an active array.
md: remove unnecessary test for MD_MAJOR in md_ioctl()
md: clean up 'exit' labels in md_ioctl().
md: remove MD_BUG()
md: discard PRINT_RAID_DEBUG ioctl
md: move EXPORT_SYMBOL to after function in md.c
drivers/md/bitmap.c | 16 +-
drivers/md/linear.c | 8 +-
drivers/md/md.c | 601 +++++++++++++----------------------------
drivers/md/md.h | 34 ++-
drivers/md/multipath.c | 28 +-
drivers/md/raid0.c | 9 +-
drivers/md/raid1.c | 37 +--
drivers/md/raid1.h | 2 -
drivers/md/raid10.c | 18 +-
drivers/md/raid5.c | 29 +-
drivers/md/raid5.h | 4 +-
include/uapi/linux/raid/md_u.h | 1 -
lib/raid6/algos.c | 12 +-
13 files changed, 266 insertions(+), 533 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* migrate to bad block list
From: Michael Ryan @ 2014-10-15 17:18 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Is there any way to migrate an existing array created with mdadm v3.2.5 and using 1.1 metadata to use a bad block list? I'm assuming not as there wouldn't be space reserved for the list, but I need to ask.
Thanks for your response!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* expression of interest.
From: Daniel Chukwudi @ 2014-10-15 11:16 UTC (permalink / raw)
Good day,
Plans are in advanced stages to convert the sum of 22,586,000.00 to
private investment with your help.More info on your expression of
interest.
Dan
^ permalink raw reply
* Re: Raid5 hang in 3.14.19
From: BillStuff @ 2014-10-14 16:55 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141014124245.69b143e2@notabene.brown>
[snip]
On 10/13/2014 08:42 PM, NeilBrown wrote:
> Write errors start happening.
>
> You should only get a write error if no writes successfully completed to
> in_sync, non-faulty devices.
> It is possible that the write to sdg3 completed before it was marked in-sync,
> and the write to sdh3 completed after it was marked as faulty.
> How long after recovery completes do you fail the next device?
> The logs suggest it is the next second, which could be anywhere from 1msec
> to 1998 msecs.
>
>
> NeilBrown
>
FYI Neil: Running through my logs, I noticed 6 of these failures in my
testing
over the past few days. All "recovery completed, wait a second, fail the
other
drive" cases. Same signature in the logs. Apparently things kept working
until
the filesystem tripped on its journal and fell over.
So the upshot is this seems reasonably reproducible.
-Bill
^ permalink raw reply
* Re: Raid5 hang in 3.14.19
From: BillStuff @ 2014-10-14 14:56 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141014124245.69b143e2@notabene.brown>
On 10/13/2014 08:42 PM, NeilBrown wrote:
> On Sun, 05 Oct 2014 11:05:08 -0500 BillStuff <billstuff2001@sbcglobal.net>
> wrote:
>
>> On 09/30/2014 05:54 PM, NeilBrown wrote:
>>> [removed alot of stuff about raid5 check hanging]
>>> Thanks for the testing! You have included enough information.
>>> I didn't really like that 'sync_starting' variable when I wrote the patch,
>>> but it seemed do the right thing. It doesn't.
>>>
>>> If md_check_recovery() runs again immediately after scheduling the sync
>>> thread to run, it will not have set sync_starting but will find ->sync_thread
>>> is NULL and so will clear MD_RECOVERY_RUNNING. The next time it runs, that
>>> flag is still clear and ->sync_thread is not NULL so it will try to stop the
>>> thread, which deadlocks.
>>>
>>> This patch on top of what you have should fix it... but I might end up
>>> redoing the logic a bit.
>>>
>> Neil,
>>
>> With the second patch, my test has been running well for close to 5 days
>> now,
> That is good news - thanks.
Now two weeks in, this seems to be working ok with the exception of this
write error,
and another "write didn't make it to both disks during recovery" hiccup
I'll describe below.
>> but something odd happened yesterday.
>>
>> It's on a raid1 array (md1) with both ext3 and xfs on LVM, but I suspect
>> it happened silently on my
>> raid5 test also.
>>
>> md1 : active raid1 sdh3[0] sdg3[1]
>> 76959296 blocks [2/2] [UU]
>> [=>...................] check = 9.7% (7532800/76959296)
>> finish=108.8min speed=10628K/sec
>> bitmap: 0/1 pages [0KB], 65536KB chunk
>>
>>
>>
>> Again this is running kernel builds, read / write loops, and remove/
>> (re)add / check loops.
>> At some point while removing a disk from the array, something bad
>> happened, the errors below
>> appeared in the log, and the ext3 filesystem remounted readonly. xfs
>> plowed right on through.
>> There's no evidence of any read or write errors to the member disks,
>> either in logs or on the
>> disks themselves, and the raid1 checks came back with zero mismatches.
>> fsck for the ext3 fs
>> complained a lot but the filesystem came back into a useable state and
>> I've restarted my tests.
>> Below, dm-3 is xfs, dm-1 is ext3. The interesting stuff happens around
>> 12:27:04. This was
>> immediately after (or during?) removing sdh3 from md1.
>>
>> Oct 4 12:26:01 xplane kernel: md: unbind<sdg3>
>> Oct 4 12:26:01 xplane kernel: md: export_rdev(sdg3)
>> Oct 4 12:26:11 xplane kernel: md: bind<sdg3>
> Removed and re-added sdg3
>
>
>> Oct 4 12:26:11 xplane kernel: RAID1 conf printout:
>> Oct 4 12:26:11 xplane kernel: --- wd:1 rd:2
>> Oct 4 12:26:11 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
>> Oct 4 12:26:11 xplane kernel: disk 1, wo:1, o:1, dev:sdg3
> ->add_disk included sdg3 in the array not in-sync.
>
>> Oct 4 12:26:12 xplane kernel: md: recovery of RAID array md1
>> Oct 4 12:26:12 xplane kernel: md: minimum _guaranteed_ speed: 10000
>> KB/sec/disk.
>> Oct 4 12:26:12 xplane kernel: md: using maximum available idle IO
>> bandwidth (but not more than 200000 KB/sec) for recovery.
>> Oct 4 12:26:12 xplane kernel: md: using 128k window, over a total of
>> 76959296k.
> Recovery starts
>
>
>> Oct 4 12:27:03 xplane kernel: md: md1: recovery done.
> Recovery finishes after 51 seconds.
>
>> Oct 4 12:27:03 xplane kernel: RAID1 conf printout:
>> Oct 4 12:27:03 xplane kernel: --- wd:2 rd:2
>> Oct 4 12:27:03 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
>> Oct 4 12:27:03 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
> ->spare_active() marks sdg3 at fully in-sync (not long "wo" - write only).
>
>> Oct 4 12:27:04 xplane kernel: md/raid1:md1: Disk failure on sdh3,
>> disabling device.
> sdh3 fails a second later. Presumably this was requested by your testing
> harness.
>
>
>> Oct 4 12:27:04 xplane kernel: md/raid1:md1: Operation continuing on 1
>> devices.
> raid.c:error() confirms the device will not be ignored.
>
>
>> Oct 4 12:27:05 xplane kernel: quiet_error: 912 callbacks suppressed
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331468
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Write errors start happening.
>
> You should only get a write error if no writes successfully completed to
> in_sync, non-faulty devices.
> It is possible that the write to sdg3 completed before it was marked in-sync,
> and the write to sdh3 completed after it was marked as faulty.
> How long after recovery completes do you fail the next device?
> The logs suggest it is the next second, which could be anywhere from 1msec
> to 1998 msecs.
>
>
> NeilBrown
The script sleeps for 1 second and then fails / removes / adds the
opposite disk. After doing all disks in the array it does a "check",
then starts over. I'm sure there's some script housekeeping time in there,
but it should be right around 1 second.
The other hiccup that happened may be related; hopefully at least
informative.
Half of my test is the fail / remove / add loop. The other half continually
writes files and saves a checksum. Later, when the file is deleted, it
re-checks
the sum. A few days ago this second check failed:
sdg3 in sync, sdh3 (re)added at 10:48:48
Oct 8 10:48:48 xplane kernel: md: bind<sdh3>
Oct 8 10:48:49 xplane kernel: RAID1 conf printout:
Oct 8 10:48:49 xplane kernel: --- wd:1 rd:2
Oct 8 10:48:49 xplane kernel: disk 0, wo:1, o:1, dev:sdh3
Oct 8 10:48:49 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 8 10:48:49 xplane kernel: md: recovery of RAID array md1
Oct 8 10:48:49 xplane kernel: md: minimum _guaranteed_ speed: 10000
KB/sec/disk.
Oct 8 10:48:49 xplane kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for recovery.
Oct 8 10:48:49 xplane kernel: md: using 128k window, over a total of
76959296k.
testfile /mnt/lvtest/tmp/tmpfile02518.0xd6.tmp written at 10:51:47
Oct 8 10:51:54 xplane kernel: md: md1: recovery done.
Oct 8 10:51:54 xplane kernel: RAID1 conf printout:
Oct 8 10:51:54 xplane kernel: --- wd:2 rd:2
Oct 8 10:51:54 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
Oct 8 10:51:54 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
md5sum for /mnt/lvtest/tmp/tmpfile02518.0xd6.tmp different at 11:30
during QC check
Nothing bad showed up in the logs, for disks, filesystems or the array.
From looking at the file, it appears a chunk of it didn't get written
to /dev/sdh3.
When the second QC was run, the read was from /dev/sdh3 and the sum was
different.
I did a "check" and I got mismatch_cnt = 256; looked at the differences
between
/dev/sdg3 and /dev/sdh3 and they're around 50GB into the 73GB (I was
expecting to find
them near the tail end, although this might be less meaningful with the
bitmap.)
I'm now trying to reproduce these two failures, now on 3.14.21 with the
2 resync start
patches, without LVM in the middle.
Thanks always for the help.
-Bill
>
>
>
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331469
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331470
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331471
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331472
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331473
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331474
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331475
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331476
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
>> block 6331477
>> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
>> Oct 4 12:27:05 xplane kernel: Aborting journal on device dm-1.
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_blocks:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
>> ext3_journal_start_sb: Detected aborted journal
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error: remounting
>> filesystem read-only
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
>> ext3_journal_start_sb: Detected aborted journal
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
>> ext3_writeback_write_end: IO failure
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_orphan_add:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
>> Journal has aborted
>> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
>> ext3_dirty_inode: IO failure
>> Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
>> Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
>> Oct 4 12:27:05 xplane kernel: disk 0, wo:1, o:0, dev:sdh3
>> Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
> raid1_remove_disk is called to disconnect the device from the array.
>
>
>> Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
>> Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
>> Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
> which it has now done.
>
>> Oct 4 12:27:14 xplane kernel: md: unbind<sdh3>
>> Oct 4 12:27:14 xplane kernel: md: export_rdev(sdh3)
>> Oct 4 12:27:24 xplane kernel: md: bind<sdh3>
>> Oct 4 12:27:24 xplane kernel: RAID1 conf printout:
>> Oct 4 12:27:24 xplane kernel: --- wd:1 rd:2
>> Oct 4 12:27:24 xplane kernel: disk 0, wo:1, o:1, dev:sdh3
>> Oct 4 12:27:24 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
>> Oct 4 12:27:24 xplane kernel: md: recovery of RAID array md1
>> Oct 4 12:27:24 xplane kernel: md: minimum _guaranteed_ speed: 10000
>> KB/sec/disk.
>> Oct 4 12:27:24 xplane kernel: md: using maximum available idle IO
>> bandwidth (but not more than 200000 KB/sec) for recovery.
>> Oct 4 12:27:24 xplane kernel: md: using 128k window, over a total of
>> 76959296k.
>> Oct 4 12:27:33 xplane kernel: md: md1: recovery done.
>> Oct 4 12:27:33 xplane kernel: RAID1 conf printout:
>> Oct 4 12:27:33 xplane kernel: --- wd:2 rd:2
>> Oct 4 12:27:33 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
>> Oct 4 12:27:33 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
>> Oct 4 12:27:35 xplane kernel: md: data-check of RAID array md1
>> Oct 4 12:27:35 xplane kernel: md: minimum _guaranteed_ speed: 10000
>> KB/sec/disk.
>> Oct 4 12:27:35 xplane kernel: md: using maximum available idle IO
>> bandwidth (but not more than 200000 KB/sec) for data-check.
>> Oct 4 12:27:35 xplane kernel: md: using 128k window, over a total of
>> 76959296k.
>> Oct 4 12:29:59 xplane kernel: __journal_remove_journal_head: freeing
>> b_committed_data
>>
>> It seems like something got out of sync as the disk was being removed,
>> but before the remove completed.
>>
>> Again, this is 3.14.19 with these 8 patches:
>> md/raid1: intialise start_next_window for READ case to avoid hang
>> md/raid1: be more cautious where we read-balance during resync.
>> md/raid1: clean up request counts properly in close_sync()
>> md/raid1: make sure resync waits for conflicting writes to complete.
>> md/raid1: Don't use next_resync to determine how far resync has
>> progressed
>> md/raid1: update next_resync under resync_lock.
>> md/raid1: count resync requests in nr_pending.
>> md/raid1: fix_read_error should act on all non-faulty devices.
>>
>> and the two patches for the check start hang.
>>
>> Any ideas on what happened here?
>>
>> Thanks,
>> Bill
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [GIT PULL] LLVMLinux patches for v3.18
From: Behan Webster @ 2014-10-14 8:59 UTC (permalink / raw)
To: Linus Torvalds
Cc: akpm, bruce.w.allan, d.kasatkin, james.l.morris, john.griffin,
linux-btrfs, linux-crypto, linux-ima-devel, linux-ima-user,
linux-kernel, linux-raid, linux-security-module, neilb, qat-linux,
serge, thomas.lendacky, zohar, agk, clm, davem, dm-devel, fabf,
herbert, jbacik, snitzer, tadeusz.struk
These patches remove the use of VLAIS using a new SHASH_DESC_ON_STACK macro.
Some of the previously accepted VLAIS removal patches haven't used this
macro. I will
push new patches to consistently use this macro in all those older cases
for 3.19
The following changes since commit 2d65a9f48fcdf7866aab6457bc707ca233e0c791:
Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux
(2014-10-14 09:39:08 +0200)
are available in the git repository at:
git://git.linuxfoundation.org/llvmlinux/kernel.git
tags/llvmlinux-for-v3.18
for you to fetch changes up to 4c5c30249452aaebf258751ea4222eba3dd3da4c:
crypto: LLVMLinux: Remove VLAIS usage from crypto/testmgr.c
(2014-10-14 10:51:24 +0200)
----------------------------------------------------------------
LLVMLinux patches for v3.18
----------------------------------------------------------------
Behan Webster (6):
crypto: LLVMLinux: Add macro to remove use of VLAIS in crypto code
crypto: LLVMLinux: Remove VLAIS from crypto/mv_cesa.c
crypto: LLVMLinux: Remove VLAIS from crypto/n2_core.c
crypto: LLVMLinux: Remove VLAIS from crypto/omap_sham.c
crypto: LLVMLinux: Remove VLAIS from crypto/.../qat_algs.c
security, crypto: LLVMLinux: Remove VLAIS from ima_crypto.c
Jan-Simon Möller (5):
crypto: LLVMLinux: Remove VLAIS from crypto/ccp/ccp-crypto-sha.c
crypto, dm: LLVMLinux: Remove VLAIS usage from dm-crypt
crypto: LLVMLinux: Remove VLAIS usage from crypto/hmac.c
crypto: LLVMLinux: Remove VLAIS usage from libcrc32c.c
crypto: LLVMLinux: Remove VLAIS usage from crypto/testmgr.c
Vinícius Tinti (1):
btrfs: LLVMLinux: Remove VLAIS
crypto/hmac.c | 25 ++++++++---------
crypto/testmgr.c | 14 ++++------
drivers/crypto/ccp/ccp-crypto-sha.c | 13 ++++-----
drivers/crypto/mv_cesa.c | 41 ++++++++++++----------------
drivers/crypto/n2_core.c | 11 +++-----
drivers/crypto/omap-sham.c | 28 ++++++++-----------
drivers/crypto/qat/qat_common/qat_algs.c | 31 ++++++++++-----------
drivers/md/dm-crypt.c | 34 ++++++++++-------------
fs/btrfs/hash.c | 16 +++++------
include/crypto/hash.h | 5 ++++
lib/libcrc32c.c | 16 +++++------
security/integrity/ima/ima_crypto.c | 47
+++++++++++++-------------------
12 files changed, 122 insertions(+), 159 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-security-module" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Verify Your Account Else It Will Be Blocked
From: Webmaster @ 2014-10-14 2:17 UTC (permalink / raw)
--
Dear user,
We are undergoing maintenance therefore all accounts must be updated,
this is to reduce the number of dormant accounts.
Accounts not updated in 48 hours will be suspended.
Please follow the hyper link below to update your account
Click Here To Update Account.
http://zimbra1.my3gb.com/
Best Regards,
Administrator
^ permalink raw reply
* Re: [PATCH] lib/raid6: Add log level to printks
From: NeilBrown @ 2014-10-14 1:51 UTC (permalink / raw)
To: Anton Blanchard; +Cc: linux-raid, linux-kernel
In-Reply-To: <20141013230316.1cbe1b7e@kryten>
[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]
On Mon, 13 Oct 2014 23:03:16 +1100 Anton Blanchard <anton@samba.org> wrote:
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
>
> Index: b/lib/raid6/algos.c
> ===================================================================
> --- a/lib/raid6/algos.c
> +++ b/lib/raid6/algos.c
> @@ -121,9 +121,9 @@ static inline const struct raid6_recov_c
> raid6_2data_recov = best->data2;
> raid6_datap_recov = best->datap;
>
> - printk("raid6: using %s recovery algorithm\n", best->name);
> + pr_info("raid6: using %s recovery algorithm\n", best->name);
> } else
> - printk("raid6: Yikes! No recovery algorithm found!\n");
> + pr_err("raid6: Yikes! No recovery algorithm found!\n");
>
> return best;
> }
> @@ -157,18 +157,18 @@ static inline const struct raid6_calls *
> bestperf = perf;
> best = *algo;
> }
> - printk("raid6: %-8s %5ld MB/s\n", (*algo)->name,
> + pr_info("raid6: %-8s %5ld MB/s\n", (*algo)->name,
> (perf*HZ) >> (20-16+RAID6_TIME_JIFFIES_LG2));
> }
> }
>
> if (best) {
> - printk("raid6: using algorithm %s (%ld MB/s)\n",
> + pr_info("raid6: using algorithm %s (%ld MB/s)\n",
> best->name,
> (bestperf*HZ) >> (20-16+RAID6_TIME_JIFFIES_LG2));
> raid6_call = *best;
> } else
> - printk("raid6: Yikes! No algorithm found!\n");
> + pr_err("raid6: Yikes! No algorithm found!\n");
>
> return best;
> }
> @@ -194,7 +194,7 @@ int __init raid6_select_algo(void)
> syndromes = (void *) __get_free_pages(GFP_KERNEL, 1);
>
> if (!syndromes) {
> - printk("raid6: Yikes! No memory available.\n");
> + pr_err("raid6: Yikes! No memory available.\n");
> return -ENOMEM;
> }
>
Applied - thanks.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Raid5 hang in 3.14.19
From: NeilBrown @ 2014-10-14 1:42 UTC (permalink / raw)
To: BillStuff; +Cc: linux-raid
In-Reply-To: <54316C34.6090304@sbcglobal.net>
[-- Attachment #1: Type: text/plain, Size: 10266 bytes --]
On Sun, 05 Oct 2014 11:05:08 -0500 BillStuff <billstuff2001@sbcglobal.net>
wrote:
> On 09/30/2014 05:54 PM, NeilBrown wrote:
> > [removed alot of stuff about raid5 check hanging]
> > Thanks for the testing! You have included enough information.
> > I didn't really like that 'sync_starting' variable when I wrote the patch,
> > but it seemed do the right thing. It doesn't.
> >
> > If md_check_recovery() runs again immediately after scheduling the sync
> > thread to run, it will not have set sync_starting but will find ->sync_thread
> > is NULL and so will clear MD_RECOVERY_RUNNING. The next time it runs, that
> > flag is still clear and ->sync_thread is not NULL so it will try to stop the
> > thread, which deadlocks.
> >
> > This patch on top of what you have should fix it... but I might end up
> > redoing the logic a bit.
> >
> Neil,
>
> With the second patch, my test has been running well for close to 5 days
> now,
That is good news - thanks.
> but something odd happened yesterday.
>
> It's on a raid1 array (md1) with both ext3 and xfs on LVM, but I suspect
> it happened silently on my
> raid5 test also.
>
> md1 : active raid1 sdh3[0] sdg3[1]
> 76959296 blocks [2/2] [UU]
> [=>...................] check = 9.7% (7532800/76959296)
> finish=108.8min speed=10628K/sec
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
>
>
> Again this is running kernel builds, read / write loops, and remove/
> (re)add / check loops.
> At some point while removing a disk from the array, something bad
> happened, the errors below
> appeared in the log, and the ext3 filesystem remounted readonly. xfs
> plowed right on through.
> There's no evidence of any read or write errors to the member disks,
> either in logs or on the
> disks themselves, and the raid1 checks came back with zero mismatches.
> fsck for the ext3 fs
> complained a lot but the filesystem came back into a useable state and
> I've restarted my tests.
> Below, dm-3 is xfs, dm-1 is ext3. The interesting stuff happens around
> 12:27:04. This was
> immediately after (or during?) removing sdh3 from md1.
>
> Oct 4 12:26:01 xplane kernel: md: unbind<sdg3>
> Oct 4 12:26:01 xplane kernel: md: export_rdev(sdg3)
> Oct 4 12:26:11 xplane kernel: md: bind<sdg3>
Removed and re-added sdg3
> Oct 4 12:26:11 xplane kernel: RAID1 conf printout:
> Oct 4 12:26:11 xplane kernel: --- wd:1 rd:2
> Oct 4 12:26:11 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
> Oct 4 12:26:11 xplane kernel: disk 1, wo:1, o:1, dev:sdg3
->add_disk included sdg3 in the array not in-sync.
> Oct 4 12:26:12 xplane kernel: md: recovery of RAID array md1
> Oct 4 12:26:12 xplane kernel: md: minimum _guaranteed_ speed: 10000
> KB/sec/disk.
> Oct 4 12:26:12 xplane kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for recovery.
> Oct 4 12:26:12 xplane kernel: md: using 128k window, over a total of
> 76959296k.
Recovery starts
> Oct 4 12:27:03 xplane kernel: md: md1: recovery done.
Recovery finishes after 51 seconds.
> Oct 4 12:27:03 xplane kernel: RAID1 conf printout:
> Oct 4 12:27:03 xplane kernel: --- wd:2 rd:2
> Oct 4 12:27:03 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
> Oct 4 12:27:03 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
->spare_active() marks sdg3 at fully in-sync (not long "wo" - write only).
> Oct 4 12:27:04 xplane kernel: md/raid1:md1: Disk failure on sdh3,
> disabling device.
sdh3 fails a second later. Presumably this was requested by your testing
harness.
> Oct 4 12:27:04 xplane kernel: md/raid1:md1: Operation continuing on 1
> devices.
raid.c:error() confirms the device will not be ignored.
> Oct 4 12:27:05 xplane kernel: quiet_error: 912 callbacks suppressed
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331468
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Write errors start happening.
You should only get a write error if no writes successfully completed to
in_sync, non-faulty devices.
It is possible that the write to sdg3 completed before it was marked in-sync,
and the write to sdh3 completed after it was marked as faulty.
How long after recovery completes do you fail the next device?
The logs suggest it is the next second, which could be anywhere from 1msec
to 1998 msecs.
NeilBrown
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331469
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331470
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331471
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331472
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331473
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331474
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331475
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331476
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
> block 6331477
> Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
> Oct 4 12:27:05 xplane kernel: Aborting journal on device dm-1.
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_blocks:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
> ext3_journal_start_sb: Detected aborted journal
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error: remounting
> filesystem read-only
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
> ext3_journal_start_sb: Detected aborted journal
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
> ext3_writeback_write_end: IO failure
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_orphan_add:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
> Journal has aborted
> Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
> ext3_dirty_inode: IO failure
> Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
> Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
> Oct 4 12:27:05 xplane kernel: disk 0, wo:1, o:0, dev:sdh3
> Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
raid1_remove_disk is called to disconnect the device from the array.
> Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
> Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
> Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
which it has now done.
> Oct 4 12:27:14 xplane kernel: md: unbind<sdh3>
> Oct 4 12:27:14 xplane kernel: md: export_rdev(sdh3)
> Oct 4 12:27:24 xplane kernel: md: bind<sdh3>
> Oct 4 12:27:24 xplane kernel: RAID1 conf printout:
> Oct 4 12:27:24 xplane kernel: --- wd:1 rd:2
> Oct 4 12:27:24 xplane kernel: disk 0, wo:1, o:1, dev:sdh3
> Oct 4 12:27:24 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
> Oct 4 12:27:24 xplane kernel: md: recovery of RAID array md1
> Oct 4 12:27:24 xplane kernel: md: minimum _guaranteed_ speed: 10000
> KB/sec/disk.
> Oct 4 12:27:24 xplane kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for recovery.
> Oct 4 12:27:24 xplane kernel: md: using 128k window, over a total of
> 76959296k.
> Oct 4 12:27:33 xplane kernel: md: md1: recovery done.
> Oct 4 12:27:33 xplane kernel: RAID1 conf printout:
> Oct 4 12:27:33 xplane kernel: --- wd:2 rd:2
> Oct 4 12:27:33 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
> Oct 4 12:27:33 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
> Oct 4 12:27:35 xplane kernel: md: data-check of RAID array md1
> Oct 4 12:27:35 xplane kernel: md: minimum _guaranteed_ speed: 10000
> KB/sec/disk.
> Oct 4 12:27:35 xplane kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for data-check.
> Oct 4 12:27:35 xplane kernel: md: using 128k window, over a total of
> 76959296k.
> Oct 4 12:29:59 xplane kernel: __journal_remove_journal_head: freeing
> b_committed_data
>
> It seems like something got out of sync as the disk was being removed,
> but before the remove completed.
>
> Again, this is 3.14.19 with these 8 patches:
> md/raid1: intialise start_next_window for READ case to avoid hang
> md/raid1: be more cautious where we read-balance during resync.
> md/raid1: clean up request counts properly in close_sync()
> md/raid1: make sure resync waits for conflicting writes to complete.
> md/raid1: Don't use next_resync to determine how far resync has
> progressed
> md/raid1: update next_resync under resync_lock.
> md/raid1: count resync requests in nr_pending.
> md/raid1: fix_read_error should act on all non-faulty devices.
>
> and the two patches for the check start hang.
>
> Any ideas on what happened here?
>
> Thanks,
> Bill
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* [PATCH] lib/raid6: Add log level to printks
From: Anton Blanchard @ 2014-10-13 12:03 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel
Signed-off-by: Anton Blanchard <anton@samba.org>
---
Index: b/lib/raid6/algos.c
===================================================================
--- a/lib/raid6/algos.c
+++ b/lib/raid6/algos.c
@@ -121,9 +121,9 @@ static inline const struct raid6_recov_c
raid6_2data_recov = best->data2;
raid6_datap_recov = best->datap;
- printk("raid6: using %s recovery algorithm\n", best->name);
+ pr_info("raid6: using %s recovery algorithm\n", best->name);
} else
- printk("raid6: Yikes! No recovery algorithm found!\n");
+ pr_err("raid6: Yikes! No recovery algorithm found!\n");
return best;
}
@@ -157,18 +157,18 @@ static inline const struct raid6_calls *
bestperf = perf;
best = *algo;
}
- printk("raid6: %-8s %5ld MB/s\n", (*algo)->name,
+ pr_info("raid6: %-8s %5ld MB/s\n", (*algo)->name,
(perf*HZ) >> (20-16+RAID6_TIME_JIFFIES_LG2));
}
}
if (best) {
- printk("raid6: using algorithm %s (%ld MB/s)\n",
+ pr_info("raid6: using algorithm %s (%ld MB/s)\n",
best->name,
(bestperf*HZ) >> (20-16+RAID6_TIME_JIFFIES_LG2));
raid6_call = *best;
} else
- printk("raid6: Yikes! No algorithm found!\n");
+ pr_err("raid6: Yikes! No algorithm found!\n");
return best;
}
@@ -194,7 +194,7 @@ int __init raid6_select_algo(void)
syndromes = (void *) __get_free_pages(GFP_KERNEL, 1);
if (!syndromes) {
- printk("raid6: Yikes! No memory available.\n");
+ pr_err("raid6: Yikes! No memory available.\n");
return -ENOMEM;
}
^ permalink raw reply
* mdadm: /dev/sdX2 has wrong raid level.
From: Weedy @ 2014-10-13 2:03 UTC (permalink / raw)
To: linux-raid
My server rebooted for some reason. Possibly disk/cable related.
The superblocks on my raid0s now think they are raid4s. Can I reverse this?
# mdadm --assemble -v -f --config=/etc/mdadm.conf /dev/md1 /dev/sd[ab]2
mdadm: looking for devices for /dev/md1
mdadm: /dev/sda2 has wrong raid level.
mdadm: /dev/sdb2 has wrong raid level.
# mdadm --examine -v /dev/sda2
/dev/sda2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de492a9a:c62a01a3:eb511bbd:ead87a3a
Name : alice:1 (local to host alice)
Creation Time : Tue Jul 31 00:25:03 2012
Raid Level : raid4
Raid Devices : 3
Avail Dev Size : 20240876 (9.65 GiB 10.36 GB)
Array Size : 20239360 (19.30 GiB 20.73 GB)
Used Dev Size : 20239360 (9.65 GiB 10.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=492 sectors
State : active
Device UUID : 436eeacc:5abe6b33:8c9f1de1:9031c001
Update Time : Sat Apr 26 23:56:01 2014
Checksum : c9dfcb49 - correct
Events : 43295
Chunk Size : 32K
Device Role : Active device 0
Array State : AA. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --examine -v /dev/sdb2
/dev/sdb2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de492a9a:c62a01a3:eb511bbd:ead87a3a
Name : alice:1 (local to host alice)
Creation Time : Tue Jul 31 00:25:03 2012
Raid Level : raid4
Raid Devices : 3
Avail Dev Size : 20240876 (9.65 GiB 10.36 GB)
Array Size : 20239360 (19.30 GiB 20.73 GB)
Used Dev Size : 20239360 (9.65 GiB 10.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=492 sectors
State : active
Device UUID : 6a01ee1d:312826ad:4be6e47a:60d5b684
Update Time : Sat Apr 26 23:56:01 2014
Checksum : b15bb2d7 - correct
Events : 43295
Chunk Size : 32K
Device Role : Active device 1
Array State : AA. ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Re: mdadm --grow with --backup-file does not create backupfile?
From: Ram Ramesh @ 2014-10-13 1:49 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux Raid
In-Reply-To: <20141013123914.04b960ed@notabene.brown>
Good to know. Thanks.
Ramesh
On 10/12/2014 08:39 PM, NeilBrown wrote:
> On Sun, 12 Oct 2014 20:26:05 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
>
>> I added a new disk and grew my raid6 md0 with the following command line
>>
>> sudo mdadm --grow --raid-devices=5
>> --backup-file=/root/grow_md0.bkp_file --layout=normalise /dev/md0
>>
>> I got a response that stated that mdadm needed to backup 384K of data.
>> However, when I checked after about 5min there was no backup file. Is
>> this expected?
> Yes.
>
> When increasing the size of an array, the backup file is needed for a tiny
> fraction of a second at the start.
> When decreasing the size it is needed for a tiny fraction if a second at the
> end.
> When the size isn't changed it is needed for the entire time.
>
> ... unless you have a new kernel and a new mdadm, and then the backup file
> isn't needed.
>
> NeilBrown
>
>
>
>> 386 > sudo ls /root
>> <no files. so no output>
>> 387 > sudo ls /root/grow_md0.bkp_file
>> ls: cannot access /root/grow_md0.bkp_file: No such file or directory
>>
>> Ramesh
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: mdadm --grow with --backup-file does not create backupfile?
From: NeilBrown @ 2014-10-13 1:39 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <543B2A2D.9080407@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]
On Sun, 12 Oct 2014 20:26:05 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> I added a new disk and grew my raid6 md0 with the following command line
>
> sudo mdadm --grow --raid-devices=5
> --backup-file=/root/grow_md0.bkp_file --layout=normalise /dev/md0
>
> I got a response that stated that mdadm needed to backup 384K of data.
> However, when I checked after about 5min there was no backup file. Is
> this expected?
Yes.
When increasing the size of an array, the backup file is needed for a tiny
fraction of a second at the start.
When decreasing the size it is needed for a tiny fraction if a second at the
end.
When the size isn't changed it is needed for the entire time.
... unless you have a new kernel and a new mdadm, and then the backup file
isn't needed.
NeilBrown
>
> 386 > sudo ls /root
> <no files. so no output>
> 387 > sudo ls /root/grow_md0.bkp_file
> ls: cannot access /root/grow_md0.bkp_file: No such file or directory
>
> Ramesh
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* mdadm --grow with --backup-file does not create backupfile?
From: Ram Ramesh @ 2014-10-13 1:26 UTC (permalink / raw)
To: Linux Raid
I added a new disk and grew my raid6 md0 with the following command line
sudo mdadm --grow --raid-devices=5
--backup-file=/root/grow_md0.bkp_file --layout=normalise /dev/md0
I got a response that stated that mdadm needed to backup 384K of data.
However, when I checked after about 5min there was no backup file. Is
this expected?
386 > sudo ls /root
<no files. so no output>
387 > sudo ls /root/grow_md0.bkp_file
ls: cannot access /root/grow_md0.bkp_file: No such file or directory
Ramesh
^ permalink raw reply
* Re: "bitmap file is out of date, doing full recovery"
From: NeilBrown @ 2014-10-12 22:24 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: linux-raid
In-Reply-To: <CAGRgLy5zhV49xF=ZwULaF46Xx=8rnWcypTWxxgf0HmEPpCP+yQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]
On Sun, 12 Oct 2014 21:03:57 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
wrote:
> Hi Neil,
> after a 2-drive raid1 unclean shutdown (crash actually), after reboot, we had:
>
> md/raid1:md24: not clean -- starting background reconstruction
> md/raid1:md24: active with 2 out of 2 mirrors
> md24: bitmap file is out of date (41 < 42) -- forcing full recovery
> created bitmap (22 pages) for device md24
> md24: bitmap file is out of date, doing full recovery
> md24: bitmap initialized from disk: read 2 pages, set 44667 of 44667 bits
>
> The superblock of both drives had event count = 42:
> (this is a custom mdadm with some added prints):
> mdadm: looking for devices for /dev/md24
> mdadm: [/dev/md24] /dev/dm-205: slot=0, events=42,
> recovery_offset=N/A, resync_offset=0, comp_size=5854539776
> mdadm: [/dev/md24] /dev/dm-206: slot=1, events=42,
> recovery_offset=N/A, resync_offset=0, comp_size=5854539776
>
> But the bitmap superblock had lower event count, which resulted in a
> full resync. Is this an expected scenario in case of a crash?
No.
>
> For example in md_update_sb, first we call
> bitmap_update_sb(mddev->bitmap), which synchronously updates the
> bitmap, and only afterwards we go ahead and update our superblocks. So
> in this case, the bitmap should not have a lower event count. Is there
> some other valid scenario, in which the bitmap can remain with a lower
> event count?
Not that I can think of.
NeilBrown
>
> Thanks,
> Alex.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Is it safe to shutdown while mdadm --grow is in progress.
From: NeilBrown @ 2014-10-12 22:15 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <543AA5B6.6010200@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]
On Sun, 12 Oct 2014 11:00:54 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> Thanks. I appreciate you taking the time to respond. I would like to
> forward this to those that maintain faq/wiki so that some one searches
> gets the info and does not have to bother you. Is that OK?
Sure - anything on a public mailing list is public.
>
> I normally would go for backup before rebuild, but 15TB disk space is
> too much to come up with, without serious $$ and need. Besides, these
> are movies/recordings for htpc and the data is available on
> dvds/internet. So there is inherent lack of importance. In other words,
> I recommend backup as most do, but it looks like every one has some
> justification not to do, and want some one else to assure them that it
> is ok :-)
Quite understandable...
NeilBrown
>
> Thanks and Regards
> Ramesh
>
> On 10/12/2014 03:35 AM, NeilBrown wrote:
> > On Sun, 12 Oct 2014 00:46:14 -0500 Ram Ramesh <rramesh2400@gmail.com> wrote:
> >
> >> I googled this topic and got differing answers. Some old ones report
> >> horror stories and others suggest rebuild continues after reboot
> >> normally. My array is rebuilding and it will take about 6 days to
> >> complete. I am expecting bad weather in a couple of days and would like
> >> to proactively shutdown the machine and reboot. Is this safe?
> >>
> >> If the information is already available in a clear manner, please let me
> >> know.
> >>
> >> uname: Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38
> >> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> >> mdadm - v3.2.5 - 18th May 2012
> >>
> >> Thanks for your help
> >>
> > It is certainly designed to be safe and my experience is that it is.
> > However it is difficult to test all corner cases so it is not impossible that
> > someone what quite work right.
> > If something does go wrong:
> > 1/ don't panic
> > 2/ don't try to --create the array to fix it
> > 3/ report all details to this list and I'm 99.9% sure I can get your array
> > running again with all your data safe.
> > But I suspect it won't come to that.
> >
> > NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox