* Re: [PATCH 4/6] md-cluster: Defer MD reloading to mddev->thread
From: Goldwyn Rodrigues @ 2015-11-10 3:26 UTC (permalink / raw)
To: NeilBrown, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <87egfyspiu.fsf@notabene.neil.brown.name>
On 11/09/2015 05:48 PM, NeilBrown wrote:
> On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
>
>> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>
>> Reloading of superblock must be performed under reconfig_mutex. However,
>> this cannot be done with md_reload_sb because it would deadlock with
>> the message DLM lock. So, we defer it in md_check_recovery() which is
>> executed by mddev->thread.
>>
>> This introduces a new flag, MD_RELOAD_SB, which if set, will reload the
>> superblock.
>
> I can see no justification for good_device_nr being atomic_t - if you
> can explain what you were trying to achieve I could possible suggest why
> it isn't needed.
Yes, I think it does not need to be atomic.
>
> Also good_device_nr is directly related to MD_RELOAD_SB, so it makes
> sense to put them both in 'struct mddev' - that would save creating a
> new cluster_operation which does very little.
Agree here as well. I got too carried away in keeping cluster-md as
isolated as possible.
>
> so: not applied.
>
> Thanks,
> NeilBrown
>
>
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> drivers/md/md-cluster.c | 12 +++++++++++-
>> drivers/md/md-cluster.h | 1 +
>> drivers/md/md.c | 3 +++
>> drivers/md/md.h | 3 +++
>> 4 files changed, 18 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
>> index a681706..9a36ad6 100644
>> --- a/drivers/md/md-cluster.c
>> +++ b/drivers/md/md-cluster.c
>> @@ -71,6 +71,7 @@ struct md_cluster_info {
>> struct md_thread *recv_thread;
>> struct completion newdisk_completion;
>> unsigned long state;
>> + atomic_t good_device_nr;
>> };
>>
>> enum msg_type {
>> @@ -434,8 +435,10 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
>> static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
>> {
>> struct md_cluster_info *cinfo = mddev->cluster_info;
>> - md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
>> + atomic_set(&cinfo->good_device_nr, le32_to_cpu(msg->raid_slot));
>> + set_bit(MD_RELOAD_SB, &mddev->flags);
>> dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
>> + md_wakeup_thread(mddev->thread);
>> }
>>
>> static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
>> @@ -1047,6 +1050,12 @@ out:
>> return err;
>> }
>>
>> +static int good_device_nr(struct mddev *mddev)
>> +{
>> + struct md_cluster_info *cinfo = mddev->cluster_info;
>> + return atomic_read(&cinfo->good_device_nr);
>> +}
>> +
>> static struct md_cluster_operations cluster_ops = {
>> .join = join,
>> .leave = leave,
>> @@ -1063,6 +1072,7 @@ static struct md_cluster_operations cluster_ops = {
>> .new_disk_ack = new_disk_ack,
>> .remove_disk = remove_disk,
>> .gather_bitmaps = gather_bitmaps,
>> + .good_device_nr = good_device_nr,
>> };
>>
>> static int __init cluster_init(void)
>> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
>> index e75ea26..c699c6c 100644
>> --- a/drivers/md/md-cluster.h
>> +++ b/drivers/md/md-cluster.h
>> @@ -24,6 +24,7 @@ struct md_cluster_operations {
>> int (*new_disk_ack)(struct mddev *mddev, bool ack);
>> int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
>> int (*gather_bitmaps)(struct md_rdev *rdev);
>> + int (*good_device_nr)(struct mddev *mddev);
>> };
>>
>> #endif /* _MD_CLUSTER_H */
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 32ca592..65b6326 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -8184,6 +8184,7 @@ void md_check_recovery(struct mddev *mddev)
>> (mddev->flags & MD_UPDATE_SB_FLAGS & ~ (1<<MD_CHANGE_PENDING)) ||
>> test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
>> test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
>> + test_bit(MD_RELOAD_SB, &mddev->flags) ||
>> (mddev->external == 0 && mddev->safemode == 1) ||
>> (mddev->safemode == 2 && ! atomic_read(&mddev->writes_pending)
>> && !mddev->in_sync && mddev->recovery_cp == MaxSector)
>> @@ -8232,6 +8233,8 @@ void md_check_recovery(struct mddev *mddev)
>> rdev->raid_disk < 0)
>> md_kick_rdev_from_array(rdev);
>> }
>> + if (test_and_clear_bit(MD_RELOAD_SB, &mddev->flags))
>> + md_reload_sb(mddev, md_cluster_ops->good_device_nr(mddev));
>> }
>>
>> if (!mddev->external) {
>> diff --git a/drivers/md/md.h b/drivers/md/md.h
>> index db54341..f89866d 100644
>> --- a/drivers/md/md.h
>> +++ b/drivers/md/md.h
>> @@ -222,6 +222,9 @@ struct mddev {
>> #define MD_STILL_CLOSED 4 /* If set, then array has not been opened since
>> * md_ioctl checked on it.
>> */
>> +#define MD_RELOAD_SB 5 /* Reload the superblock because another node
>> + * updated it.
>> + */
>>
>> int suspended;
>> atomic_t active_io;
>> --
>> 1.8.5.6
--
Goldwyn
^ permalink raw reply
* Re: [PATCH 1/6] md-cluster: Protect communication with mutexes
From: Goldwyn Rodrigues @ 2015-11-10 3:23 UTC (permalink / raw)
To: NeilBrown, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <87mvumsqbp.fsf@notabene.neil.brown.name>
On 11/09/2015 05:31 PM, NeilBrown wrote:
> On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
>
>> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>
>> Communication can happen through multiple threads. It is possible that
>> one thread steps over another threads sequence. So, we use mutexes to
>> protect both the send and receive sequences.
>>
>> We use a new flag CLUSTER_MD_COMM_LOCKED which is used to detect
>> if communication is already locked. This is useful for cases where we need to
>> take and hold the token DLM lock for a while. This bit is set only
>> after locking communication.
>
> I don't understand what this flag is trying to achieve, but I'm fairly
> sure it doesn't achieve it.
Lets consider three specific cases of locking communication channels to
show the conflict:
1. resync_info_update(): communication is locked and release for sending
a message.
2. A regular md_update_sb(): communication is locked in
metadata_update_start() and unlocked in metadata_update_finish() after
writing to disk. In metadata_update_finish(), the sendmsg is called to
send METADATA_UPDATED.
3. An add operation which culminates in a md_update_sb(): Here the
communication is locked before initiating add. If the add is successful,
it results in md_update_sb(). In md_update_sb(), metadata_update_start()
has to check if the communication is already locked. If locked, it
should not lock again.
The flag is used only for case 3. If the communication is already
locked, it should not lock again. This flag is set only after
lock_comm() has executed, but is checked for in metadata_update_start().
This should insure that any of the operations 1, 2 or 3 do not interfere
with each other.
I am not sure if I have made the best effort to explain this. I had a
tough time getting it right (which may or may not be complete).
>
> Maybe if it was test_and_set_bit in metadata_update_start, it might make
> sense. But then I would suggest that clearing the bit be moved to
> unlock_comm()
>
> Do you just want to use mutex_try_lock() to detect if communication is
> already locked?
Consider a race between md_update_sb() and sendmsg() [Case 1. and 2]
mutex_try_lock() may not work in this situation because lock_comm()
could have been called by sendmsg(), which will release it as soon as
the message is sent. In the meantime (while the lock is locked), if a
metadata_update_sb() operation executes. It will not relock the
communication. This will result in the WARN_ON in unlock_comm() since
sendmsg() sequence had already unlocked it.
>
>>
>> Also, Remove stray/ununsed sb_mutex.
>
> I already removed that it mainline - I should have mentioned.
>
> Thanks,
> NeilBrown
>
>
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> drivers/md/md-cluster.c | 26 +++++++++++++++++++++-----
>> 1 file changed, 21 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
>> index b73729b..a93734e 100644
>> --- a/drivers/md/md-cluster.c
>> +++ b/drivers/md/md-cluster.c
>> @@ -47,6 +47,7 @@ struct resync_info {
>> #define MD_CLUSTER_WAITING_FOR_NEWDISK 1
>> #define MD_CLUSTER_SUSPEND_READ_BALANCING 2
>> #define MD_CLUSTER_BEGIN_JOIN_CLUSTER 3
>> +#define MD_CLUSTER_COMM_LOCKED 4
>>
>>
>> struct md_cluster_info {
>> @@ -54,7 +55,8 @@ struct md_cluster_info {
>> dlm_lockspace_t *lockspace;
>> int slot_number;
>> struct completion completion;
>> - struct mutex sb_mutex;
>> + struct mutex recv_mutex;
>> + struct mutex send_mutex;
>> struct dlm_lock_resource *bitmap_lockres;
>> struct dlm_lock_resource *resync_lockres;
>> struct list_head suspend_list;
>> @@ -503,6 +505,7 @@ static void recv_daemon(struct md_thread *thread)
>> struct cluster_msg msg;
>> int ret;
>>
>> + mutex_lock(&cinfo->recv_mutex);
>> /*get CR on Message*/
>> if (dlm_lock_sync(message_lockres, DLM_LOCK_CR)) {
>> pr_err("md/raid1:failed to get CR on MESSAGE\n");
>> @@ -529,6 +532,7 @@ static void recv_daemon(struct md_thread *thread)
>> ret = dlm_unlock_sync(message_lockres);
>> if (unlikely(ret != 0))
>> pr_info("unlock msg failed return %d\n", ret);
>> + mutex_unlock(&cinfo->recv_mutex);
>> }
>>
>> /* lock_comm()
>> @@ -542,20 +546,22 @@ static int lock_comm(struct md_cluster_info *cinfo)
>> {
>> int error;
>>
>> - if (cinfo->token_lockres->mode == DLM_LOCK_EX)
>> - return 0;
>> + mutex_lock(&cinfo->send_mutex);
>>
>> error = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
>> if (error)
>> pr_err("md-cluster(%s:%d): failed to get EX on TOKEN (%d)\n",
>> __func__, __LINE__, error);
>> + mutex_lock(&cinfo->recv_mutex);
>> return error;
>> }
>>
>> static void unlock_comm(struct md_cluster_info *cinfo)
>> {
>> WARN_ON(cinfo->token_lockres->mode != DLM_LOCK_EX);
>> + mutex_unlock(&cinfo->recv_mutex);
>> dlm_unlock_sync(cinfo->token_lockres);
>> + mutex_unlock(&cinfo->send_mutex);
>> }
>>
>> /* __sendmsg()
>> @@ -709,7 +715,8 @@ static int join(struct mddev *mddev, int nodes)
>> init_completion(&cinfo->completion);
>> set_bit(MD_CLUSTER_BEGIN_JOIN_CLUSTER, &cinfo->state);
>>
>> - mutex_init(&cinfo->sb_mutex);
>> + mutex_init(&cinfo->send_mutex);
>> + mutex_init(&cinfo->recv_mutex);
>> mddev->cluster_info = cinfo;
>>
>> memset(str, 0, 64);
>> @@ -839,7 +846,12 @@ static int slot_number(struct mddev *mddev)
>>
>> static int metadata_update_start(struct mddev *mddev)
>> {
>> - return lock_comm(mddev->cluster_info);
>> + struct md_cluster_info *cinfo = mddev->cluster_info;
>> + int ret;
>> + if (test_and_clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state))
>> + return 0;
>> + ret = lock_comm(cinfo);
>> + return ret;
>> }
>>
>> static int metadata_update_finish(struct mddev *mddev)
>> @@ -864,6 +876,7 @@ static int metadata_update_finish(struct mddev *mddev)
>> ret = __sendmsg(cinfo, &cmsg);
>> } else
>> pr_warn("md-cluster: No good device id found to send\n");
>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>> unlock_comm(cinfo);
>> return ret;
>> }
>> @@ -871,6 +884,7 @@ static int metadata_update_finish(struct mddev *mddev)
>> static void metadata_update_cancel(struct mddev *mddev)
>> {
>> struct md_cluster_info *cinfo = mddev->cluster_info;
>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>> unlock_comm(cinfo);
>> }
>>
>> @@ -945,6 +959,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
>> memcpy(cmsg.uuid, uuid, 16);
>> cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
>> lock_comm(cinfo);
>> + set_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>> ret = __sendmsg(cinfo, &cmsg);
>> if (ret)
>> return ret;
>> @@ -964,6 +979,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
>> static void add_new_disk_cancel(struct mddev *mddev)
>> {
>> struct md_cluster_info *cinfo = mddev->cluster_info;
>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>> unlock_comm(cinfo);
>> }
>>
>> --
>> 1.8.5.6
--
Goldwyn
^ permalink raw reply
* Re: Offline array, events count mismatch
From: Guillaume Paumier @ 2015-11-10 3:05 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <56401471.2040605@turmel.org>
Hello Phil and the list,
Le dimanche 8 novembre 2015, 22:35:13 Phil Turmel a écrit :
>
> On 11/08/2015 09:49 PM, Guillaume Paumier wrote:
> >
> > If I understand the documentation [1] correctly, since the event count for
> > sdj is very close to the event count of sd[b,c,d,g,h,i], I should be able
> > to re- assemble the array with these 7 disks using --force, leaving sde
> > and sdf aside. Once the array is assembled, I should be able to re-add
> > sde and sdf, and they will be re-sync'd.
>
> Yes, that is the correct response.
>
> Your situation is common. Please see the thread this weekend started by
> Franscisco Parada.
Thank you for confirming, Phil, and for the additional pointer.
I've re-assembled the array with --force, which cleaned sdj, and then I was
able to re-add the two other disks. The array started rebuilding and recovery
was past 10% when the array failed again.
It seems there was an "unrecoverable read error" on sdj, and now I'm back with
an array where 2 of the disks are marked as spare (sde and sdf, because their
rebuild didn't complete), and sdj is faulty with an event count mismatch of 4,
like before:
/dev/sdb1:
Events : 198704
/dev/sdc1:
Events : 198704
/dev/sdd1:
Events : 198704
/dev/sde1:
Events : 198704
/dev/sdf1:
Events : 198704
/dev/sdg1:
Events : 198704
/dev/sdh1:
Events : 198704
/dev/sdi1:
Events : 198704
/dev/sdj1:
Events : 198700
Below is the output of dmesg with more details on the read error.
Is there any way I can move past this? This error is preventing me from
rebuilding the array, and I'm assuming it would also prevent me from copying
the data off the array without rebuilding, so I'm not sure how to proceed. Any
guidance would be much appreciated.
[88233.712961] md: recovery of RAID array md0
[88233.712965] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[88233.712967] md: using maximum available idle IO bandwidth (but not more
than 200000 KB/sec) for recovery.
[88233.712978] md: using 128k window, over a total of 3907016448k.
[88953.752335] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[88953.752345] ata9.01: BMDMA stat 0x64
[88953.752353] ata9.01: failed command: READ DMA EXT
[88953.752368] ata9.01: cmd 25/00:00:00:fc:e8/00:02:27:00:00/f0 tag 0 dma
262144 in
res 51/40:00:f8:fd:e8/40:00:27:00:00/10 Emask 0x9 (media error)
[88953.752375] ata9.01: status: { DRDY ERR }
[88953.752380] ata9.01: error: { UNC }
[88953.793877] ata9.00: configured for UDMA/33
[88953.799795] ata9.01: configured for UDMA/33
[88953.799855] sd 8:0:1:0: [sdj] Unhandled sense code
[88953.799858] sd 8:0:1:0: [sdj]
[88953.799860] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[88953.799862] sd 8:0:1:0: [sdj]
[88953.799864] Sense Key : Medium Error [current] [descriptor]
[88953.799867] Descriptor sense data with sense descriptors (in hex):
[88953.799868] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[88953.799875] 27 e8 fd f8
[88953.799879] sd 8:0:1:0: [sdj]
[88953.799882] Add. Sense: Unrecovered read error - auto reallocate failed
[88953.799884] sd 8:0:1:0: [sdj] CDB:
[88953.799885] Read(16): 88 00 00 00 00 00 27 e8 fc 00 00 00 02 00 00 00
[88953.799894] end_request: I/O error, dev sdj, sector 669580792
[88953.799898] md/raid:md0: read error not correctable (sector 669578744 on
sdj1).
[88953.799924] ata9: EH complete
[89333.138473] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[89333.138478] ata9.01: BMDMA stat 0x64
[89333.138482] ata9.01: failed command: READ DMA EXT
[89333.138488] ata9.01: cmd 25/00:00:58:6e:3b/00:02:35:00:00/f0 tag 0 dma
262144 in
res 51/40:00:c8:6f:3b/40:00:35:00:00/10 Emask 0x9 (media error)
[89333.138491] ata9.01: status: { DRDY ERR }
[89333.138493] ata9.01: error: { UNC }
[89333.147985] ata9.00: configured for UDMA/33
[89333.153966] ata9.01: configured for UDMA/33
[89333.154022] sd 8:0:1:0: [sdj] Unhandled sense code
[89333.154025] sd 8:0:1:0: [sdj]
[89333.154027] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[89333.154029] sd 8:0:1:0: [sdj]
[89333.154031] Sense Key : Medium Error [current] [descriptor]
[89333.154034] Descriptor sense data with sense descriptors (in hex):
[89333.154035] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[89333.154042] 35 3b 6f c8
[89333.154046] sd 8:0:1:0: [sdj]
[89333.154048] Add. Sense: Unrecovered read error - auto reallocate failed
[89333.154050] sd 8:0:1:0: [sdj] CDB:
[89333.154052] Read(16): 88 00 00 00 00 00 35 3b 6e 58 00 00 02 00 00 00
[89333.154061] end_request: I/O error, dev sdj, sector 893087688
[89333.154064] md/raid:md0: read error not correctable (sector 893085640 on
sdj1).
[89333.154067] md/raid:md0: read error not correctable (sector 893085648 on
sdj1).
[89333.154069] md/raid:md0: read error not correctable (sector 893085656 on
sdj1).
[89333.154071] md/raid:md0: read error not correctable (sector 893085664 on
sdj1).
[89333.154073] md/raid:md0: read error not correctable (sector 893085672 on
sdj1).
[89333.154075] md/raid:md0: read error not correctable (sector 893085680 on
sdj1).
[89333.154077] md/raid:md0: read error not correctable (sector 893085688 on
sdj1).
[89333.154079] md/raid:md0: read error not correctable (sector 893085696 on
sdj1).
[89333.154081] md/raid:md0: read error not correctable (sector 893085704 on
sdj1).
[89333.154083] md/raid:md0: read error not correctable (sector 893085712 on
sdj1).
[89333.154111] ata9: EH complete
[89338.097012] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[89338.097016] ata9.01: BMDMA stat 0x64
[89338.097019] ata9.01: failed command: READ DMA EXT
[89338.097023] ata9.01: cmd 25/00:00:58:70:3b/00:02:35:00:00/f0 tag 0 dma
262144 in
res 51/40:00:60:70:3b/40:00:35:00:00/10 Emask 0x9 (media error)
[89338.097025] ata9.01: status: { DRDY ERR }
[89338.097026] ata9.01: error: { UNC }
[89338.125468] ata9.00: configured for UDMA/33
[89338.131458] ata9.01: configured for UDMA/33
[89338.131489] sd 8:0:1:0: [sdj] Unhandled sense code
[89338.131491] sd 8:0:1:0: [sdj]
[89338.131492] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[89338.131493] sd 8:0:1:0: [sdj]
[89338.131494] Sense Key : Medium Error [current] [descriptor]
[89338.131496] Descriptor sense data with sense descriptors (in hex):
[89338.131497] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[89338.131502] 35 3b 70 60
[89338.131504] sd 8:0:1:0: [sdj]
[89338.131506] Add. Sense: Unrecovered read error - auto reallocate failed
[89338.131507] sd 8:0:1:0: [sdj] CDB:
[89338.131508] Read(16): 88 00 00 00 00 00 35 3b 70 58 00 00 02 00 00 00
[89338.131513] end_request: I/O error, dev sdj, sector 893087840
[89338.131556] ata9: EH complete
[89342.103300] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[89342.103310] ata9.01: BMDMA stat 0x64
[89342.103319] ata9.01: failed command: READ DMA EXT
[89342.103333] ata9.01: cmd 25/00:00:58:72:3b/00:02:35:00:00/f0 tag 0 dma
262144 in
res 51/40:00:58:72:3b/40:00:35:00:00/10 Emask 0x9 (media error)
[89342.103340] ata9.01: status: { DRDY ERR }
[89342.103344] ata9.01: error: { UNC }
[89342.224995] ata9.00: configured for UDMA/33
[89342.230983] ata9.01: configured for UDMA/33
[89342.231022] sd 8:0:1:0: [sdj] Unhandled sense code
[89342.231025] sd 8:0:1:0: [sdj]
[89342.231027] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[89342.231029] sd 8:0:1:0: [sdj]
[89342.231031] Sense Key : Medium Error [current] [descriptor]
[89342.231034] Descriptor sense data with sense descriptors (in hex):
[89342.231035] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[89342.231042] 35 3b 72 58
[89342.231046] sd 8:0:1:0: [sdj]
[89342.231049] Add. Sense: Unrecovered read error - auto reallocate failed
[89342.231051] sd 8:0:1:0: [sdj] CDB:
[89342.231052] Read(16): 88 00 00 00 00 00 35 3b 72 58 00 00 02 00 00 00
[89342.231061] end_request: I/O error, dev sdj, sector 893088344
[89342.231065] raid5_end_read_request: 71 callbacks suppressed
[89342.231067] md/raid:md0: read error not correctable (sector 893086296 on
sdj1).
[89342.231070] md/raid:md0: read error not correctable (sector 893086304 on
sdj1).
[89342.231072] md/raid:md0: read error not correctable (sector 893086312 on
sdj1).
[89342.231074] md/raid:md0: read error not correctable (sector 893086320 on
sdj1).
[89342.231076] md/raid:md0: read error not correctable (sector 893086328 on
sdj1).
[89342.231078] md/raid:md0: read error not correctable (sector 893086336 on
sdj1).
[89342.231080] md/raid:md0: read error not correctable (sector 893086344 on
sdj1).
[89342.231081] md/raid:md0: read error not correctable (sector 893086352 on
sdj1).
[89342.231083] md/raid:md0: read error not correctable (sector 893086360 on
sdj1).
[89342.231085] md/raid:md0: read error not correctable (sector 893086368 on
sdj1).
[89342.231149] ata9: EH complete
[89346.169717] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[89346.169727] ata9.01: BMDMA stat 0x64
[89346.169736] ata9.01: failed command: READ DMA EXT
[89346.169750] ata9.01: cmd 25/00:00:58:74:3b/00:02:35:00:00/f0 tag 0 dma
262144 in
res 51/40:00:58:74:3b/40:00:35:00:00/10 Emask 0x9 (media error)
[89346.169758] ata9.01: status: { DRDY ERR }
[89346.169763] ata9.01: error: { UNC }
[89346.198239] ata9.00: configured for UDMA/33
[89346.204166] ata9.01: configured for UDMA/33
[89346.204232] sd 8:0:1:0: [sdj] Unhandled sense code
[89346.204239] sd 8:0:1:0: [sdj]
[89346.204243] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[89346.204248] sd 8:0:1:0: [sdj]
[89346.204251] Sense Key : Medium Error [current] [descriptor]
[89346.204258] Descriptor sense data with sense descriptors (in hex):
[89346.204261] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[89346.204278] 35 3b 74 58
[89346.204286] sd 8:0:1:0: [sdj]
[89346.204292] Add. Sense: Unrecovered read error - auto reallocate failed
[89346.204296] sd 8:0:1:0: [sdj] CDB:
[89346.204299] Read(16): 88 00 00 00 00 00 35 3b 74 58 00 00 02 00 00 00
[89346.204319] end_request: I/O error, dev sdj, sector 893088856
[89346.204419] ata9: EH complete
[89353.949976] ata9.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[89353.949986] ata9.01: BMDMA stat 0x64
[89353.949994] ata9.01: failed command: READ DMA EXT
[89353.950008] ata9.01: cmd 25/00:90:c8:6f:3b/00:00:35:00:00/f0 tag 0 dma
73728 in
res 51/40:00:e0:6f:3b/40:00:35:00:00/10 Emask 0x9 (media error)
[89353.950016] ata9.01: status: { DRDY ERR }
[89353.950021] ata9.01: error: { UNC }
[89353.994545] ata9.00: configured for UDMA/33
[89354.000539] ata9.01: configured for UDMA/33
[89354.000597] sd 8:0:1:0: [sdj] Unhandled sense code
[89354.000603] sd 8:0:1:0: [sdj]
[89354.000608] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[89354.000612] sd 8:0:1:0: [sdj]
[89354.000616] Sense Key : Medium Error [current] [descriptor]
[89354.000623] Descriptor sense data with sense descriptors (in hex):
[89354.000626] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[89354.000643] 35 3b 6f e0
[89354.000651] sd 8:0:1:0: [sdj]
[89354.000657] Add. Sense: Unrecovered read error - auto reallocate failed
[89354.000661] sd 8:0:1:0: [sdj] CDB:
[89354.000664] Read(16): 88 00 00 00 00 00 35 3b 6f c8 00 00 00 90 00 00
[89354.000684] end_request: I/O error, dev sdj, sector 893087712
[89354.000692] raid5_end_read_request: 118 callbacks suppressed
[89354.000697] md/raid:md0: read error not correctable (sector 893085664 on
sdj1).
[89354.000706] md/raid:md0: Disk failure on sdj1, disabling device.
md/raid:md0: Operation continuing on 6 devices.
[89354.000732] md/raid:md0: read error not correctable (sector 893085672 on
sdj1).
[89354.000737] md/raid:md0: read error not correctable (sector 893085680 on
sdj1).
[89354.000742] md/raid:md0: read error not correctable (sector 893085688 on
sdj1).
[89354.000747] md/raid:md0: read error not correctable (sector 893085696 on
sdj1).
[89354.000751] md/raid:md0: read error not correctable (sector 893085704 on
sdj1).
[89354.000756] md/raid:md0: read error not correctable (sector 893085712 on
sdj1).
[89354.000760] md/raid:md0: read error not correctable (sector 893085720 on
sdj1).
[89354.000765] md/raid:md0: read error not correctable (sector 893085728 on
sdj1).
[89354.000769] md/raid:md0: read error not correctable (sector 893085736 on
sdj1).
[89354.000903] ata9: EH complete
[89354.109105] md: md0: recovery interrupted.
[89354.175670] RAID conf printout:
[89354.175675] --- level:6 rd:9 wd:6
[89354.175677] disk 0, o:1, dev:sdc1
[89354.175679] disk 1, o:1, dev:sdd1
[89354.175680] disk 2, o:1, dev:sdg1
[89354.175681] disk 3, o:1, dev:sdh1
[89354.175682] disk 4, o:1, dev:sdi1
[89354.175683] disk 5, o:0, dev:sdj1
[89354.175684] disk 6, o:1, dev:sdf1
[89354.175685] disk 7, o:1, dev:sde1
[89354.175686] disk 8, o:1, dev:sdb1
[89354.177220] RAID conf printout:
[89354.177221] --- level:6 rd:9 wd:6
[89354.177222] disk 0, o:1, dev:sdc1
[89354.177223] disk 1, o:1, dev:sdd1
[89354.177224] disk 2, o:1, dev:sdg1
[89354.177225] disk 3, o:1, dev:sdh1
[89354.177226] disk 4, o:1, dev:sdi1
[89354.177227] disk 5, o:0, dev:sdj1
[89354.177227] disk 7, o:1, dev:sde1
[89354.177228] disk 8, o:1, dev:sdb1
[89354.177233] RAID conf printout:
[89354.177234] --- level:6 rd:9 wd:6
[89354.177234] disk 0, o:1, dev:sdc1
[89354.177235] disk 1, o:1, dev:sdd1
[89354.177236] disk 2, o:1, dev:sdg1
[89354.177237] disk 3, o:1, dev:sdh1
[89354.177238] disk 4, o:1, dev:sdi1
[89354.177239] disk 5, o:0, dev:sdj1
[89354.177240] disk 7, o:1, dev:sde1
[89354.177241] disk 8, o:1, dev:sdb1
[89354.179575] RAID conf printout:
[89354.179576] --- level:6 rd:9 wd:6
[89354.179577] disk 0, o:1, dev:sdc1
[89354.179578] disk 1, o:1, dev:sdd1
[89354.179579] disk 2, o:1, dev:sdg1
[89354.179580] disk 3, o:1, dev:sdh1
[89354.179581] disk 4, o:1, dev:sdi1
[89354.179582] disk 5, o:0, dev:sdj1
[89354.179583] disk 8, o:1, dev:sdb1
[89354.179585] RAID conf printout:
[89354.179586] --- level:6 rd:9 wd:6
[89354.179587] disk 0, o:1, dev:sdc1
[89354.179588] disk 1, o:1, dev:sdd1
[89354.179589] disk 2, o:1, dev:sdg1
[89354.179589] disk 3, o:1, dev:sdh1
[89354.179590] disk 4, o:1, dev:sdi1
[89354.179591] disk 5, o:0, dev:sdj1
[89354.179592] disk 8, o:1, dev:sdb1
[89354.181443] RAID conf printout:
[89354.181444] --- level:6 rd:9 wd:6
[89354.181445] disk 0, o:1, dev:sdc1
[89354.181446] disk 1, o:1, dev:sdd1
[89354.181447] disk 2, o:1, dev:sdg1
[89354.181448] disk 3, o:1, dev:sdh1
[89354.181449] disk 4, o:1, dev:sdi1
[89354.181450] disk 8, o:1, dev:sdb1
[90001.391680] md0: detected capacity change from 28005493899264 to 0
[90001.391697] md: md0 stopped.
[90001.391717] md: unbind<sdf1>
[90001.396688] md: export_rdev(sdf1)
[90001.396808] md: unbind<sde1>
[90001.403661] md: export_rdev(sde1)
[90001.403726] md: unbind<sdc1>
[90001.412707] md: export_rdev(sdc1)
[90001.412867] md: unbind<sdb1>
[90001.415711] md: export_rdev(sdb1)
[90001.415782] md: unbind<sdj1>
[90001.421708] md: export_rdev(sdj1)
[90001.421783] md: unbind<sdi1>
[90001.424752] md: export_rdev(sdi1)
[90001.424909] md: unbind<sdh1>
[90001.427741] md: export_rdev(sdh1)
[90001.427807] md: unbind<sdg1>
[90001.433745] md: export_rdev(sdg1)
[90001.433812] md: unbind<sdd1>
[90001.436732] md: export_rdev(sdd1)
> You should provide "smartctl -i -A -l scterc /dev/sdX" reports for your
> drives. If you can find an old syslog for when your two worst drives
> fell out, it might help.
Here's the output for the disk with the read error for now, in case it's
useful.
# smartctl -i -A -l scterc /dev/sdj
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.16.7-29-desktop] (SUSE RPM)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST4000VN000-1H4168
Serial Number: Z300NEB5
LU WWN Device Id: 5 000c50 063ed9f94
Firmware Version: SC43
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5900 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Nov 9 18:58:47 2015 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 094 094 006 Pre-fail Always
- 28320486
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always
- 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always
- 73
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always
- 160
7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always
- 17212021570
9 Power_On_Hours 0x0032 079 079 000 Old_age Always
- 19201
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always
- 73
184 End-to-End_Error 0x0032 100 100 099 Old_age Always
- 0
187 Reported_Uncorrect 0x0032 055 055 000 Old_age Always
- 45
188 Command_Timeout 0x0032 100 100 000 Old_age Always
- 0
189 High_Fly_Writes 0x003a 001 001 000 Old_age Always
- 169
190 Airflow_Temperature_Cel 0x0022 065 057 045 Old_age Always
- 35 (Min/Max 30/37)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always
- 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always
- 28
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always
- 73
194 Temperature_Celsius 0x0022 035 043 000 Old_age Always
- 35 (0 18 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always
- 48
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline -
48
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always
- 0
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
--
Guillaume Paumier
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 6/6] md-cluster: Allow spare devices to be marked as faulty
From: NeilBrown @ 2015-11-09 23:51 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1446781819-25571-6-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 966 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> If a spare device was marked faulty, it would not be reflected
> in receiving nodes because it would mark it as activated and continue.
> Continue the operation, so it may be set as faulty.
Yes, that makes sense.
Applied, thanks.
NeilBrown
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md.c | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 4222285..b98cf04 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -9013,7 +9013,6 @@ static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
> ret = remove_and_add_spares(mddev, rdev2);
> pr_info("Activated spare: %s\n",
> bdevname(rdev2->bdev,b));
> - continue;
> }
> /* device faulty
> * We just want to do the minimum to mark the disk
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 5/6] md-cluster: Fix the remove sequence with the new MD reload code
From: NeilBrown @ 2015-11-09 23:49 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Goldwyn Rodrigues, Guoqing Jiang
In-Reply-To: <1446781819-25571-5-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> The remove disk message does not need metadata_update_start(), but
> can be an independent message.
This looks good.
Applied - thanks.
NeilBrown
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> drivers/md/md-cluster.c | 2 +-
> drivers/md/md.c | 9 +--------
> 2 files changed, 2 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 9a36ad6..33f5d7a 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -1018,7 +1018,7 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
> struct md_cluster_info *cinfo = mddev->cluster_info;
> cmsg.type = cpu_to_le32(REMOVE);
> cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
> - return __sendmsg(cinfo, &cmsg);
> + return sendmsg(cinfo, &cmsg);
> }
>
> static int gather_bitmaps(struct md_rdev *rdev)
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 65b6326..4222285 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -6045,15 +6045,11 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
> {
> char b[BDEVNAME_SIZE];
> struct md_rdev *rdev;
> - int ret = -1;
>
> rdev = find_rdev(mddev, dev);
> if (!rdev)
> return -ENXIO;
>
> - if (mddev_is_clustered(mddev))
> - ret = md_cluster_ops->metadata_update_start(mddev);
> -
> if (rdev->raid_disk < 0)
> goto kick_rdev;
>
> @@ -6064,7 +6060,7 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
> goto busy;
>
> kick_rdev:
> - if (mddev_is_clustered(mddev) && ret == 0)
> + if (mddev_is_clustered(mddev))
> md_cluster_ops->remove_disk(mddev, rdev);
>
> md_kick_rdev_from_array(rdev);
> @@ -6073,9 +6069,6 @@ kick_rdev:
>
> return 0;
> busy:
> - if (mddev_is_clustered(mddev) && ret == 0)
> - md_cluster_ops->metadata_update_cancel(mddev);
> -
> printk(KERN_WARNING "md: cannot remove active disk %s from %s ...\n",
> bdevname(rdev->bdev,b), mdname(mddev));
> return -EBUSY;
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 4/6] md-cluster: Defer MD reloading to mddev->thread
From: NeilBrown @ 2015-11-09 23:48 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1446781819-25571-4-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 4619 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> Reloading of superblock must be performed under reconfig_mutex. However,
> this cannot be done with md_reload_sb because it would deadlock with
> the message DLM lock. So, we defer it in md_check_recovery() which is
> executed by mddev->thread.
>
> This introduces a new flag, MD_RELOAD_SB, which if set, will reload the
> superblock.
I can see no justification for good_device_nr being atomic_t - if you
can explain what you were trying to achieve I could possible suggest why
it isn't needed.
Also good_device_nr is directly related to MD_RELOAD_SB, so it makes
sense to put them both in 'struct mddev' - that would save creating a
new cluster_operation which does very little.
so: not applied.
Thanks,
NeilBrown
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 12 +++++++++++-
> drivers/md/md-cluster.h | 1 +
> drivers/md/md.c | 3 +++
> drivers/md/md.h | 3 +++
> 4 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index a681706..9a36ad6 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -71,6 +71,7 @@ struct md_cluster_info {
> struct md_thread *recv_thread;
> struct completion newdisk_completion;
> unsigned long state;
> + atomic_t good_device_nr;
> };
>
> enum msg_type {
> @@ -434,8 +435,10 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
> static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> - md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
> + atomic_set(&cinfo->good_device_nr, le32_to_cpu(msg->raid_slot));
> + set_bit(MD_RELOAD_SB, &mddev->flags);
> dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
> + md_wakeup_thread(mddev->thread);
> }
>
> static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
> @@ -1047,6 +1050,12 @@ out:
> return err;
> }
>
> +static int good_device_nr(struct mddev *mddev)
> +{
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> + return atomic_read(&cinfo->good_device_nr);
> +}
> +
> static struct md_cluster_operations cluster_ops = {
> .join = join,
> .leave = leave,
> @@ -1063,6 +1072,7 @@ static struct md_cluster_operations cluster_ops = {
> .new_disk_ack = new_disk_ack,
> .remove_disk = remove_disk,
> .gather_bitmaps = gather_bitmaps,
> + .good_device_nr = good_device_nr,
> };
>
> static int __init cluster_init(void)
> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
> index e75ea26..c699c6c 100644
> --- a/drivers/md/md-cluster.h
> +++ b/drivers/md/md-cluster.h
> @@ -24,6 +24,7 @@ struct md_cluster_operations {
> int (*new_disk_ack)(struct mddev *mddev, bool ack);
> int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
> int (*gather_bitmaps)(struct md_rdev *rdev);
> + int (*good_device_nr)(struct mddev *mddev);
> };
>
> #endif /* _MD_CLUSTER_H */
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 32ca592..65b6326 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -8184,6 +8184,7 @@ void md_check_recovery(struct mddev *mddev)
> (mddev->flags & MD_UPDATE_SB_FLAGS & ~ (1<<MD_CHANGE_PENDING)) ||
> test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
> test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
> + test_bit(MD_RELOAD_SB, &mddev->flags) ||
> (mddev->external == 0 && mddev->safemode == 1) ||
> (mddev->safemode == 2 && ! atomic_read(&mddev->writes_pending)
> && !mddev->in_sync && mddev->recovery_cp == MaxSector)
> @@ -8232,6 +8233,8 @@ void md_check_recovery(struct mddev *mddev)
> rdev->raid_disk < 0)
> md_kick_rdev_from_array(rdev);
> }
> + if (test_and_clear_bit(MD_RELOAD_SB, &mddev->flags))
> + md_reload_sb(mddev, md_cluster_ops->good_device_nr(mddev));
> }
>
> if (!mddev->external) {
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index db54341..f89866d 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -222,6 +222,9 @@ struct mddev {
> #define MD_STILL_CLOSED 4 /* If set, then array has not been opened since
> * md_ioctl checked on it.
> */
> +#define MD_RELOAD_SB 5 /* Reload the superblock because another node
> + * updated it.
> + */
>
> int suspended;
> atomic_t active_io;
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 3/8] block/floppy.c: handle blk_register_region() return value
From: Grant Grundler @ 2015-11-09 23:44 UTC (permalink / raw)
To: Vishnu Pratap Singh
Cc: Jens Axboe, Andrew Morton, LKML, Jeff Moyer, minchan, ngupta,
sergey.senozhatsky.work, David Miller, neilb, Ulf Hansson, tiwai,
Hannes Reinecke, Ming Lei, jarod, viro, Tejun Heo, adrian.hunter,
Jon Hunter, Grant Grundler, Linux IDE mailing list, linux-raid,
linux-mmc@vger.kernel.org, CPGS, vishu13285, pintu.k, rohit.kr
In-Reply-To: <1446812535-10567-3-git-send-email-vishnu.ps@samsung.com>
On Fri, Nov 6, 2015 at 4:22 AM, Vishnu Pratap Singh
<vishnu.ps@samsung.com> wrote:
> This patch handles blk_register_region() return value.
> Earlier blk_register_region() function doesn't handle error
> cases, now it is added, so the callers of this function
> should also handle it.
>
> Verfied on X86 based ubuntu machine.
> This patch depends on [PATCH 1/8] block/genhd.c: Add error handling
>
> Signed-off-by: Vishnu Pratap Singh <vishnu.ps@samsung.com>
> ---
> drivers/block/floppy.c | 11 +++++++----
> 1 file changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c
> index 331363e..50802a6 100644
> --- a/drivers/block/floppy.c
> +++ b/drivers/block/floppy.c
> @@ -4219,8 +4219,10 @@ static int __init do_floppy_init(void)
> if (err)
> goto out_unreg_blkdev;
>
> - blk_register_region(MKDEV(FLOPPY_MAJOR, 0), 256, THIS_MODULE,
> + err = blk_register_region(MKDEV(FLOPPY_MAJOR, 0), 256, THIS_MODULE,
> floppy_find, NULL, NULL);
> + if (err)
> + goto out_unreg_region;
>
> for (i = 0; i < 256; i++)
> if (ITYPE(i))
> @@ -4250,7 +4252,7 @@ static int __init do_floppy_init(void)
> if (fdc_state[0].address == -1) {
> cancel_delayed_work(&fd_timeout);
> err = -ENODEV;
> - goto out_unreg_region;
> + goto out_fdc_err;
> }
> #if N_FDC > 1
> fdc_state[1].address = FDC2;
> @@ -4261,7 +4263,7 @@ static int __init do_floppy_init(void)
> if (err) {
> cancel_delayed_work(&fd_timeout);
> err = -EBUSY;
> - goto out_unreg_region;
> + goto out_fdc_region;
Please drop this hunk.
> }
>
> /* initialise drive state */
> @@ -4357,8 +4359,9 @@ out_remove_drives:
> out_release_dma:
> if (atomic_read(&usage_count))
> floppy_release_irq_and_dma();
> -out_unreg_region:
> +out_fdc_err:
> blk_unregister_region(MKDEV(FLOPPY_MAJOR, 0), 256);
> +out_unreg_region:
Nit: out_unreg_region should continue to be a label for the
unregister_region call.
My preference would be to add a new label here so something like
"goto out_driver_unregister" works.
I normally would ignore this sort of thing but this patch needs to be
reposted as V2 anyway.
cheers,
grant
> platform_driver_unregister(&floppy_driver);
> out_unreg_blkdev:
> unregister_blkdev(FLOPPY_MAJOR, "fd");
> --
> 1.9.1
>
^ permalink raw reply
* Re: [PATCH 3/6] md-cluster: remove a disk asynchronously from cluster environment
From: NeilBrown @ 2015-11-09 23:43 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Guoqing Jiang, Goldwyn Rodrigues
In-Reply-To: <1446781819-25571-3-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 2965 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> For cluster raid, if one disk couldn't be reach in one node, then
> other nodes would receive the REMOVE message for the disk.
>
> In receiving node, we can't call md_kick_rdev_from_array to remove
> the disk from array synchronously since the disk might still be busy
> in this node. So let's set a ClusterRemove flag on the disk, then
> let the thread to do the removal job eventually.
Thanks.
I've applied this patch.
However
1/ it isn't against mainline.
2/ While the ClusterRemove flag is (currently) only used in a cluster
configuration, the functionality that it represents isn't necessarily
cluster specific. So I would prefer a more generic name (like
AutoRemove).
3/ similarly the test on mddev_is_cluster() in md_check_recovery()
doesn't really search much purpose.
Thanks,
NeilBrown
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 7 +++++--
> drivers/md/md.c | 12 ++++++++++++
> drivers/md/md.h | 1 +
> 3 files changed, 18 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index 3daa464..a681706 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -443,8 +443,11 @@ static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
> struct md_rdev *rdev = md_find_rdev_nr_rcu(mddev,
> le32_to_cpu(msg->raid_slot));
>
> - if (rdev)
> - md_kick_rdev_from_array(rdev);
> + if (rdev) {
> + set_bit(ClusterRemove, &rdev->flags);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + md_wakeup_thread(mddev->thread);
> + }
> else
> pr_warn("%s: %d Could not find disk(%d) to REMOVE\n",
> __func__, __LINE__, le32_to_cpu(msg->raid_slot));
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 44d0342..32ca592 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -8222,6 +8222,18 @@ void md_check_recovery(struct mddev *mddev)
> goto unlock;
> }
>
> + if (mddev_is_clustered(mddev)) {
> + struct md_rdev *rdev;
> + /* kick the device if another node issued a
> + * remove disk.
> + */
> + rdev_for_each(rdev, mddev) {
> + if (test_and_clear_bit(ClusterRemove, &rdev->flags) &&
> + rdev->raid_disk < 0)
> + md_kick_rdev_from_array(rdev);
> + }
> + }
> +
> if (!mddev->external) {
> int did_change = 0;
> spin_lock(&mddev->lock);
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index 2ea0035..db54341 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -172,6 +172,7 @@ enum flag_bits {
> * This device is seen locally but not
> * by the whole cluster
> */
> + ClusterRemove,
> };
>
> #define BB_LEN_MASK (0x00000000000001FFULL)
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 2/6] md-cluster: Avoid the resync ping-pong
From: NeilBrown @ 2015-11-09 23:39 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1446781819-25571-2-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> If a RESYNCING message with (0,0) has been sent before, do not send it
> again. This avoids a resync ping pong between the nodes. We read
> the bitmap lockresource's LVB to figure out the previous value
> of the RESYNCING message.
This seems like a bandaid more than a real fix, but maybe I'm wrong.
I'll apply it for now. If I end of figuring out that there is some
other issue I discuss it then.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index a93734e..3daa464 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -898,8 +898,16 @@ static int resync_start(struct mddev *mddev)
> static int resync_info_update(struct mddev *mddev, sector_t lo, sector_t hi)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> + struct resync_info ri;
> struct cluster_msg cmsg = {0};
>
> + /* do not send zero again, if we have sent before */
> + if (hi == 0) {
> + memcpy(&ri, cinfo->bitmap_lockres->lksb.sb_lvbptr, sizeof(struct resync_info));
> + if (le64_to_cpu(ri.hi) == 0)
> + return 0;
I feel this would look so much better as:
if (hi == 0) {
struct resync_info ri;
memcpy(&ri, cinfo->bitmap_lockres->lksb.sb_lvbptr,
sizeof(ri));
}
so if you happened to resend it like that I'd probably remove what I
have already applied and use this instead.
(make variable more local, use sizeof(var) instead of sizeof(type), and
wrap the long line)
Thanks,
NeilBrown
> + }
> +
> add_resync_info(mddev, cinfo->bitmap_lockres, lo, hi);
> /* Re-acquire the lock to refresh LVB */
> dlm_lock_sync(cinfo->bitmap_lockres, DLM_LOCK_PW);
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 1/6] md-cluster: Protect communication with mutexes
From: NeilBrown @ 2015-11-09 23:31 UTC (permalink / raw)
To: rgoldwyn, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1446781819-25571-1-git-send-email-rgoldwyn@suse.de>
[-- Attachment #1: Type: text/plain, Size: 5333 bytes --]
On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>
> Communication can happen through multiple threads. It is possible that
> one thread steps over another threads sequence. So, we use mutexes to
> protect both the send and receive sequences.
>
> We use a new flag CLUSTER_MD_COMM_LOCKED which is used to detect
> if communication is already locked. This is useful for cases where we need to
> take and hold the token DLM lock for a while. This bit is set only
> after locking communication.
I don't understand what this flag is trying to achieve, but I'm fairly
sure it doesn't achieve it.
Maybe if it was test_and_set_bit in metadata_update_start, it might make
sense. But then I would suggest that clearing the bit be moved to
unlock_comm()
Do you just want to use mutex_try_lock() to detect if communication is
already locked?
>
> Also, Remove stray/ununsed sb_mutex.
I already removed that it mainline - I should have mentioned.
Thanks,
NeilBrown
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/md-cluster.c | 26 +++++++++++++++++++++-----
> 1 file changed, 21 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index b73729b..a93734e 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -47,6 +47,7 @@ struct resync_info {
> #define MD_CLUSTER_WAITING_FOR_NEWDISK 1
> #define MD_CLUSTER_SUSPEND_READ_BALANCING 2
> #define MD_CLUSTER_BEGIN_JOIN_CLUSTER 3
> +#define MD_CLUSTER_COMM_LOCKED 4
>
>
> struct md_cluster_info {
> @@ -54,7 +55,8 @@ struct md_cluster_info {
> dlm_lockspace_t *lockspace;
> int slot_number;
> struct completion completion;
> - struct mutex sb_mutex;
> + struct mutex recv_mutex;
> + struct mutex send_mutex;
> struct dlm_lock_resource *bitmap_lockres;
> struct dlm_lock_resource *resync_lockres;
> struct list_head suspend_list;
> @@ -503,6 +505,7 @@ static void recv_daemon(struct md_thread *thread)
> struct cluster_msg msg;
> int ret;
>
> + mutex_lock(&cinfo->recv_mutex);
> /*get CR on Message*/
> if (dlm_lock_sync(message_lockres, DLM_LOCK_CR)) {
> pr_err("md/raid1:failed to get CR on MESSAGE\n");
> @@ -529,6 +532,7 @@ static void recv_daemon(struct md_thread *thread)
> ret = dlm_unlock_sync(message_lockres);
> if (unlikely(ret != 0))
> pr_info("unlock msg failed return %d\n", ret);
> + mutex_unlock(&cinfo->recv_mutex);
> }
>
> /* lock_comm()
> @@ -542,20 +546,22 @@ static int lock_comm(struct md_cluster_info *cinfo)
> {
> int error;
>
> - if (cinfo->token_lockres->mode == DLM_LOCK_EX)
> - return 0;
> + mutex_lock(&cinfo->send_mutex);
>
> error = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
> if (error)
> pr_err("md-cluster(%s:%d): failed to get EX on TOKEN (%d)\n",
> __func__, __LINE__, error);
> + mutex_lock(&cinfo->recv_mutex);
> return error;
> }
>
> static void unlock_comm(struct md_cluster_info *cinfo)
> {
> WARN_ON(cinfo->token_lockres->mode != DLM_LOCK_EX);
> + mutex_unlock(&cinfo->recv_mutex);
> dlm_unlock_sync(cinfo->token_lockres);
> + mutex_unlock(&cinfo->send_mutex);
> }
>
> /* __sendmsg()
> @@ -709,7 +715,8 @@ static int join(struct mddev *mddev, int nodes)
> init_completion(&cinfo->completion);
> set_bit(MD_CLUSTER_BEGIN_JOIN_CLUSTER, &cinfo->state);
>
> - mutex_init(&cinfo->sb_mutex);
> + mutex_init(&cinfo->send_mutex);
> + mutex_init(&cinfo->recv_mutex);
> mddev->cluster_info = cinfo;
>
> memset(str, 0, 64);
> @@ -839,7 +846,12 @@ static int slot_number(struct mddev *mddev)
>
> static int metadata_update_start(struct mddev *mddev)
> {
> - return lock_comm(mddev->cluster_info);
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> + int ret;
> + if (test_and_clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state))
> + return 0;
> + ret = lock_comm(cinfo);
> + return ret;
> }
>
> static int metadata_update_finish(struct mddev *mddev)
> @@ -864,6 +876,7 @@ static int metadata_update_finish(struct mddev *mddev)
> ret = __sendmsg(cinfo, &cmsg);
> } else
> pr_warn("md-cluster: No good device id found to send\n");
> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
> unlock_comm(cinfo);
> return ret;
> }
> @@ -871,6 +884,7 @@ static int metadata_update_finish(struct mddev *mddev)
> static void metadata_update_cancel(struct mddev *mddev)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
> unlock_comm(cinfo);
> }
>
> @@ -945,6 +959,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
> memcpy(cmsg.uuid, uuid, 16);
> cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
> lock_comm(cinfo);
> + set_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
> ret = __sendmsg(cinfo, &cmsg);
> if (ret)
> return ret;
> @@ -964,6 +979,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
> static void add_new_disk_cancel(struct mddev *mddev)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
> unlock_comm(cinfo);
> }
>
> --
> 1.8.5.6
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* raid6 didnt syncing - Bug?
From: Dragon @ 2015-11-09 22:36 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <trinity-850e4e87-8977-46fa-9382-c97a137130a8-1447108068694@3capp-gmx-bs60>
Hi,
i have debian with kernel 4.2 and wanted to add a new disk to my raid6. i have md0=system in raid1, md1=swap n raid6 and md2=files also in raid6. here are the mdadm -E output: http://pastebin.com/MveRQ2Lv.
i stop md1 and run assamble, but without success. i stop md1 and wait for md2 but no success. at boot i see message "md/raid:md2: reshape: not enough stripes. Needed 512" and run echo and rise from 256 to 600 for stripes, but without success. any idea?
thx
^ permalink raw reply
* Re: RAID6 reshape stalls immediately
From: Phil Turmel @ 2015-11-09 22:26 UTC (permalink / raw)
To: Peter Chubb; +Cc: Anugraha Sinha, linux-raid
In-Reply-To: <84lha6u90k.wl-peter.chubb@nicta.com.au>
On 11/09/2015 05:02 PM, Peter Chubb wrote:
> I just did
> mdadm --grow --continue /dev/md0
>
> and the reshape started up again. I didn't realise that mdadm had to
> keep going to do the reshape -- it must have died when I logged off
> last time. So many other operations are handled entirely in the
> kernel...
mdadm spawns a copy of mdmon to perform these tasks. If the process
hierarchy is strictly enforced, and kills off spawned processes when
the highest parent dies, that's a problem for mdadm. I recall some
discussions about this with systemd and/or cgroups. You might want
to make sure all of your utilities have the latest fixes.
Phil
^ permalink raw reply
* Re: RAID6 reshape stalls immediately
From: Peter Chubb @ 2015-11-09 22:02 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anugraha Sinha, Peter Chubb, linux-raid
In-Reply-To: <5640A1ED.3010205@turmel.org>
>>>>> "Phil" == Phil Turmel <philip@turmel.org> writes:
Phil> On 11/09/2015 08:27 AM, Anugraha Sinha wrote:
>> I am looping in to Phil explicitly for some help here.
>>
>> @Phil, Need some help here!
Phil> I don't know why it's stuck. This has never happened to me :-).
Phil> If omitting the --backup-file fails, I would reboot into a
Phil> rescue CD with a recent kernel and mdadm version. I recommend
Phil> www.sysrescuecd.ord for such things. Then use mdadm --grow
Phil> --continue within the rescue environment.
I just did
mdadm --grow --continue /dev/md0
and the reshape started up again. I didn't realise that mdadm had to
keep going to do the reshape -- it must have died when I logged off
last time. So many other operations are handled entirely in the
kernel...
Peter C
--
Dr Peter Chubb http://www.data61.csiro.au
http://www.ssrg.nicta.com.au Software Systems Research Group/NICTA/Data61
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Phil Turmel @ 2015-11-09 21:13 UTC (permalink / raw)
To: Mathias Mueller; +Cc: linux-raid
In-Reply-To: <a3a91665c4b7cdd70dacc7d8815cc365@pingofdeath.de>
{Resend with list address corrected -- ignore the duplicate, Mathias }
Hi Matthias,
On 11/09/2015 02:36 PM, Mathias Mueller wrote:
>> for x in /dev/sd[b-e]1 ; do echo $x ; dd if=$x bs=4k count=2 |hexdump
>> -C ; done
>
> /dev/sdb1
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |. G.....g.t..]..|
> 00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00 |..G.............|
> 00000420 00 80 00 00 00 80 00 00 00 20 00 00 00 00 00 00 |......... ......|
> 00000430 d9 45 8b 4c 00 00 27 00 53 ef 01 00 01 00 00 00 |.E.L..'.S.......|
> 00000440 2e 44 8b 4c 00 4e ed 00 00 00 00 00 01 00 00 00 |.D.L.N..........|
> 00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00 |............<...|
False positive. (Not an ext4 signature).
> /dev/sdc1
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |. G.....g.t..]..|
> 00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00 |..G.............|
> 00000420 00 80 00 00 00 80 00 00 00 20 00 00 00 00 00 00 |......... ......|
> 00000430 d9 45 8b 4c 00 00 27 00 53 ef 01 00 01 00 00 00 |.E.L..'.S.......|
> 00000440 2e 44 8b 4c 00 4e ed 00 00 00 00 00 01 00 00 00 |.D.L.N..........|
Also a false positive.
> /dev/sdd1
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00000400 00 20 47 07 90 0e 1c 1d 54 9a 74 01 75 aa c1 08 |. G.....T.t.u...|
> 00000410 35 47 33 07 00 00 00 00 02 00 00 00 02 00 00 00 |5G3.............|
> 00000420 00 80 00 00 00 80 00 00 00 20 00 00 0b b2 ca 4d |......... .....M|
> 00000430 7b f0 ca 4d 15 00 21 00 53 ef 01 00 01 00 00 00 |{..M..!.S.......|
> 00000440 9d 7f 3b 4d 00 4e ed 00 00 00 00 00 01 00 00 00 |..;M.N..........|
> 00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00 |............<...|
> 00000460 42 02 00 00 7b 00 00 00 4f a7 6a c3 f8 12 42 1c |B...{...O.j...B.|
> 00000470 8f d2 5b ec 1a e9 fb 4c 64 61 74 61 00 00 00 00 |..[....Ldata....|
> 00000480 00 00 00 00 00 00 00 00 2f 6d 65 64 69 61 2f 64 |......../media/d|
> 00000490 61 74 61 00 d8 dc 83 00 02 88 ff ff 38 da 59 4c |ata.........8.YL|
> 000004a0 00 00 00 00 00 c6 70 c3 01 88 ff ff 00 96 56 23 |......p.......V#|
> 000004b0 02 88 ff ff 40 7b 21 1d 02 88 ff ff dd 75 18 81 |....@{!......u..|
This really is an ext4 signature. Volume label 'data' last
mounted at '/media/data' on Wed, 11 May 2011 15:58:03 GMT.
Given that date and your certainty that your array was v1.2,
this must be garbage left over from a previous filesystem.
> /dev/sde1
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |. G.....g.t..]..|
> 00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00 |..G.............|
> 00000420 00 80 00 00 00 80 00 00 00 20 00 00 0a 88 62 51 |......... ....bQ|
> 00000430 d5 cb 62 51 05 00 24 00 53 ef 01 00 01 00 00 00 |..bQ..$.S.......|
> 00000440 54 d4 81 50 00 4e ed 00 00 00 00 00 01 00 00 00 |T..P.N..........|
> 00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00 |............<...|
Another false positive.
>> Also show the following more picky signature search for each drive:
>> dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep -E '[048c]30 .+ 53 ef'
>
> I will provide this as soon as the processes are finished, seems to take
> some more hours.
We'll just have to wait for this. Your ext4 superblock should be in
there somewhere.
> there is also one more thing I remember, maybe it's helpful: when I was
> starting the reshape, there was this message:
>
> mdadm: Need to backup 8192K of critical section..
If this was an actual error message, it may not have proceeded. We just
don't know yet.
Phil
^ permalink raw reply
* Fwd: Re: broken raid level 5 array caused by user error
From: Jan Kasper @ 2015-11-09 19:51 UTC (permalink / raw)
To: Linux raid
> for x in /dev/sd[b-e]1 ; do echo $x ; dd if=$x bs=4k count=2 |hexdump
> -C ; done
/dev/sdb1
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |.
G.....g.t..]..|
00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00
|..G.............|
00000420 00 80 00 00 00 80 00 00 00 20 00 00 00 00 00 00 |.........
......|
00000430 d9 45 8b 4c 00 00 27 00 53 ef 01 00 01 00 00 00
|.E.L..'.S.......|
00000440 2e 44 8b 4c 00 4e ed 00 00 00 00 00 01 00 00 00
|.D.L.N..........|
00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00
|............<...|
2+0 records in
2+0 records out
8192 bytes (8,2 kB) copied00000460 42 02 00 00 7b 00 00 00 04 9e 2a e7
2f 34 4a a3 |B...{.....*./4J.|
, 0,00133482 s, 6,1 MB/s
00000470 a6 0e fd 70 61 2e 8e b2 00 00 00 00 00 00 00 00
|...pa...........|
00000480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
000004c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 8b 03
|................|
000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000004e0 08 00 00 00 00 00 00 00 00 00 00 00 aa db d1 8b
|................|
000004f0 87 de 46 f7 89 11 66 37 d3 0e 67 3e 01 01 00 00
|..F...f7..g>....|
00000500 00 00 00 00 00 00 00 00 2e 44 8b 4c 0a f3 02 00
|.........D.L....|
00000510 04 00 00 00 00 00 00 00 00 00 00 00 ff 7f 00 00
|................|
00000520 00 80 88 0e ff 7f 00 00 01 00 00 00 ff ff 88 0e
|................|
00000530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08
|................|
00000550 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 1c 00
|................|
00000560 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000570 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00
|................|
00000580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
|.N+.............|
00001010 00 89 e5 62 dc f1 ec 07 df 4b 66 b1 5b 9c bf c7
|...b.....Kf.[...|
00001020 69 6c 6c 2e 64 61 72 6b 73 74 61 72 3a 31 00 00
|ill.darkstar:1..|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001040 5d bc 40 56 00 00 00 00 05 00 00 00 02 00 00 00
|].@V............|
00001050 00 74 dc e8 00 00 00 00 00 04 00 00 04 00 00 00
|.t..............|
00001060 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001080 00 00 04 00 00 00 00 00 71 88 dc e8 00 00 00 00
|........q.......|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000010a0 02 00 00 00 00 00 00 00 42 98 2d b0 dd 3b a0 f6
|........B.-..;..|
000010b0 71 4c 7f 0d e9 84 3b 63 00 00 08 00 48 00 00 00
|qL....;c....H...|
000010c0 93 bc 40 56 00 00 00 00 01 00 00 00 00 00 00 00
|..@V............|
000010d0 ff ff ff ff ff ff ff ff 1a 45 f9 2e 80 00 00 00
|.........E......|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001100 00 00 01 00 02 00 03 00 fe ff fe ff fe ff fe ff
|................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
*
00001400 62 69 74 6d 04 00 00 00 d0 cc 85 d0 4a 6b 3d 7b
|bitm........Jk={|
00001410 b2 4c 69 bb ea 0d b2 aa ef 1c 00 00 00 00 00 00
|.Li.............|
00001420 ef 1c 00 00 00 00 00 00 00 74 e0 e8 00 00 00 00
|.........t......|
00001430 00 00 00 00 00 00 00 04 05 00 00 00 00 00 00 00
|................|
00001440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001770 00 00 00 00 00 00 01 0c 00 00 00 00 00 00 00 00
|................|
00001780 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
000018a0 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000018b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00002000
/dev/sdc1
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |.
G.....g.t..]..|
00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00
|..G.............|
00000420 00 80 00 00 00 80 00 00 00 20 00 00 00 00 00 00 |.........
......|
00000430 d9 45 8b 4c 00 00 27 00 53 ef 01 00 01 00 00 00
|.E.L..'.S.......|
00000440 2e 44 8b 4c 00 4e ed 00 00 00 00 00 01 00 00 00
|.D.L.N..........|
00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00
|............<...|
2+0 records in
2+0 records out
00000460 42 02 00 00 7b 00 00 00 04 9e 2a e7 2f 34 4a a3
|B...{.....*./4J.|
8192 bytes (8,2 kB) copied, 0,00115719 s, 7,1 MB/s
00000470 a6 0e fd 70 61 2e 8e b2 00 00 00 00 00 00 00 00
|...pa...........|
00000480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
000004c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 8b 03
|................|
000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000004e0 08 00 00 00 00 00 00 00 00 00 00 00 aa db d1 8b
|................|
000004f0 87 de 46 f7 89 11 66 37 d3 0e 67 3e 01 01 00 00
|..F...f7..g>....|
00000500 00 00 00 00 00 00 00 00 2e 44 8b 4c 0a f3 02 00
|.........D.L....|
00000510 04 00 00 00 00 00 00 00 00 00 00 00 ff 7f 00 00
|................|
00000520 00 80 88 0e ff 7f 00 00 01 00 00 00 ff ff 88 0e
|................|
00000530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08
|................|
00000550 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 1c 00
|................|
00000560 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000570 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00
|................|
00000580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
|.N+.............|
00001010 00 89 e5 62 dc f1 ec 07 df 4b 66 b1 5b 9c bf c7
|...b.....Kf.[...|
00001020 69 6c 6c 2e 64 61 72 6b 73 74 61 72 3a 31 00 00
|ill.darkstar:1..|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001040 5d bc 40 56 00 00 00 00 05 00 00 00 02 00 00 00
|].@V............|
00001050 00 74 dc e8 00 00 00 00 00 04 00 00 04 00 00 00
|.t..............|
00001060 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001080 00 00 04 00 00 00 00 00 b0 80 dc e8 00 00 00 00
|................|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000010a0 01 00 00 00 00 00 00 00 f4 17 85 d4 2c 5d c0 b3
|............,]..|
000010b0 7b 32 ec 45 25 01 c2 64 00 00 08 00 48 00 00 00
|{2.E%..d....H...|
000010c0 93 bc 40 56 00 00 00 00 01 00 00 00 00 00 00 00
|..@V............|
000010d0 ff ff ff ff ff ff ff ff 9f 40 64 4a 80 00 00 00
|.........@dJ....|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001100 00 00 01 00 02 00 03 00 fe ff fe ff fe ff fe ff
|................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
*
00001400 62 69 74 6d 04 00 00 00 d0 cc 85 d0 4a 6b 3d 7b
|bitm........Jk={|
00001410 b2 4c 69 bb ea 0d b2 aa ef 1c 00 00 00 00 00 00
|.Li.............|
00001420 ef 1c 00 00 00 00 00 00 00 74 e0 e8 00 00 00 00
|.........t......|
00001430 00 00 00 00 00 00 00 04 05 00 00 00 00 00 00 00
|................|
00001440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001770 00 00 00 00 00 00 01 0c 00 00 00 00 00 00 00 00
|................|
00001780 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
000018a0 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000018b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00002000
/dev/sdd1
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000400 00 20 47 07 90 0e 1c 1d 54 9a 74 01 75 aa c1 08 |.
G.....T.t.u...|
00000410 35 47 33 07 00 00 00 00 02 00 00 00 02 00 00 00
|5G3.............|
00000420 00 80 00 00 00 80 00 00 00 20 00 00 0b b2 ca 4d |.........
.....M|
00000430 7b f0 ca 4d 15 00 21 00 53 ef 01 00 01 00 00 00
|{..M..!.S.......|
00000440 9d 7f 3b 4d 00 4e ed 00 00 00 00 00 01 00 00 00
|..;M.N..........|
2+0 records in
2+0 records out
8192 bytes (8,2 kB) copied00000450 00 00 00 00 0b 00 00 00 00 01 00 00
3c 00 00 00 |............<...|
, 0,00132436 s, 6,2 MB/s
00000460 42 02 00 00 7b 00 00 00 4f a7 6a c3 f8 12 42 1c
|B...{...O.j...B.|
00000470 8f d2 5b ec 1a e9 fb 4c 64 61 74 61 00 00 00 00
|..[....Ldata....|
00000480 00 00 00 00 00 00 00 00 2f 6d 65 64 69 61 2f 64
|......../media/d|
00000490 61 74 61 00 d8 dc 83 00 02 88 ff ff 38 da 59 4c
|ata.........8.YL|
000004a0 00 00 00 00 00 c6 70 c3 01 88 ff ff 00 96 56 23
|......p.......V#|
000004b0 02 88 ff ff 40 7b 21 1d 02 88 ff ff dd 75 18 81
|....@{!......u..|
000004c0 ff ff ff ff 38 dd 83 00 00 00 00 00 00 00 8b 03
|....8...........|
000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000004e0 08 00 00 00 00 00 00 00 00 00 00 00 42 fe 0d 59
|............B..Y|
000004f0 47 70 42 40 bf c4 97 e0 10 61 6c 49 01 01 00 00
|GpB@.....alI....|
00000500 00 00 00 00 00 00 00 00 09 54 19 4b 0a f3 02 00
|.........T.K....|
00000510 04 00 00 00 00 00 00 00 00 00 00 00 ff 7f 00 00
|................|
00000520 00 80 88 0e ff 7f 00 00 01 00 00 00 ff ff 88 0e
|................|
00000530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08
|................|
00000550 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 1c 00
|................|
00000560 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000570 00 00 00 00 04 00 00 00 fe 57 e8 70 00 00 00 00
|.........W.p....|
00000580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
|.N+.............|
00001010 00 89 e5 62 dc f1 ec 07 df 4b 66 b1 5b 9c bf c7
|...b.....Kf.[...|
00001020 69 6c 6c 2e 64 61 72 6b 73 74 61 72 3a 31 00 00
|ill.darkstar:1..|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001040 5d bc 40 56 00 00 00 00 05 00 00 00 02 00 00 00
|].@V............|
00001050 00 74 dc e8 00 00 00 00 00 04 00 00 04 00 00 00
|.t..............|
00001060 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001080 00 00 04 00 00 00 00 00 82 74 dc e8 00 00 00 00
|.........t......|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000010a0 00 00 00 00 00 00 00 00 7a 77 95 2a c6 63 f3 1f
|........zw.*.c..|
000010b0 4c 04 34 12 80 ee 4e 6f 00 00 08 00 48 00 00 00
|L.4...No....H...|
000010c0 93 bc 40 56 00 00 00 00 01 00 00 00 00 00 00 00
|..@V............|
000010d0 ff ff ff ff ff ff ff ff ba 59 7c e3 80 00 00 00
|.........Y|.....|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001100 00 00 01 00 02 00 03 00 fe ff fe ff fe ff fe ff
|................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
*
00001400 62 69 74 6d 04 00 00 00 d0 cc 85 d0 4a 6b 3d 7b
|bitm........Jk={|
00001410 b2 4c 69 bb ea 0d b2 aa ef 1c 00 00 00 00 00 00
|.Li.............|
00001420 ef 1c 00 00 00 00 00 00 00 74 e0 e8 00 00 00 00
|.........t......|
00001430 00 00 00 00 00 00 00 04 05 00 00 00 00 00 00 00
|................|
00001440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001770 00 00 00 00 00 00 01 0c 00 00 00 00 00 00 00 00
|................|
00001780 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
000018a0 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000018b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00002000
/dev/sde1
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000400 00 20 47 07 16 10 1c 1d 67 9a 74 01 8b 5d a6 1c |.
G.....g.t..]..|
00000410 f5 1f 47 07 00 00 00 00 02 00 00 00 02 00 00 00
|..G.............|
00000420 00 80 00 00 00 80 00 00 00 20 00 00 0a 88 62 51 |.........
....bQ|
00000430 d5 cb 62 51 05 00 24 00 53 ef 01 00 01 00 00 00
|..bQ..$.S.......|
00000440 54 d4 81 50 00 4e ed 00 00 00 00 00 01 00 00 00
|T..P.N..........|
00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00
|............<...|
00000460 42 02 00 00 7b 00 00 00 a9 23 60 46 3c a1 4a 7b
|B...{....#`F<.J{|
2+0 records in
2+0 records out
00000470 bc 41 b6 98 ef b2 85 1f 00 00 00 00 00 00 00 00
|.A..............|
8192 bytes (8,2 kB) copied00000480 00 00 00 00 00 00 00 00 2f 6d 6e 74
2f 74 6d 70 |......../mnt/tmp|
, 0,0236698 s, 346 kB/s
00000490 00 38 5d 11 83 00 88 ff ff a1 fd 85 4f 00 00 00
|.8].........O...|
000004a0 00 40 25 47 37 01 88 ff ff 40 25 47 37 01 88 ff
|.@%G7....@%G7...|
000004b0 ff 80 62 b4 3c 01 88 ff ff 80 e7 98 79 00 88 ff
|..b.<.......y...|
000004c0 ff 20 7a 07 a0 ff ff ff 00 00 00 00 00 00 8b 03 |.
z.............|
000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000004e0 08 00 00 00 00 00 00 00 00 00 00 00 7c f1 27 08
|............|.'.|
000004f0 ee e9 44 71 97 a4 19 cd 7e 64 9a 8e 01 01 00 00
|..Dq....~d......|
00000500 00 00 00 00 00 00 00 00 88 b2 ca 4d 0a f3 02 00
|...........M....|
00000510 04 00 00 00 00 00 00 00 00 00 00 00 ff 7f 00 00
|................|
00000520 00 80 88 0e ff 7f 00 00 01 00 00 00 ff ff 88 0e
|................|
00000530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08
|................|
00000550 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 1c 00
|................|
00000560 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000570 00 00 00 00 04 00 00 00 3e 47 af 52 00 00 00 00
|........>G.R....|
00000580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
|.N+.............|
00001010 00 89 e5 62 dc f1 ec 07 df 4b 66 b1 5b 9c bf c7
|...b.....Kf.[...|
00001020 69 6c 6c 2e 64 61 72 6b 73 74 61 72 3a 31 00 00
|ill.darkstar:1..|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001040 5d bc 40 56 00 00 00 00 05 00 00 00 02 00 00 00
|].@V............|
00001050 00 74 dc e8 00 00 00 00 00 04 00 00 04 00 00 00
|.t..............|
00001060 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00001080 00 00 04 00 00 00 00 00 b0 80 dc e8 00 00 00 00
|................|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
000010a0 03 00 00 00 00 00 00 00 0e 6f 7c 3b 40 b7 f7 0c
|.........o|;@...|
000010b0 df 43 43 c8 82 18 eb f5 00 00 08 00 48 00 00 00
|.CC.........H...|
000010c0 93 bc 40 56 00 00 00 00 01 00 00 00 00 00 00 00
|..@V............|
000010d0 ff ff ff ff ff ff ff ff 90 1a 13 1e 80 00 00 00
|................|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00001100 00 00 01 00 02 00 03 00 fe ff fe ff fe ff fe ff
|................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
*
00001400 00 00 10 00 10 00 10 00 20 00 10 00 e0 5f 00 20 |........
...._. |
00001410 00 00 05 00 00 00 00 00 00 00 00 00 00 20 2e d9
|............. ..|
00001420 01 00 10 00 11 00 10 00 20 02 10 00 00 80 00 20 |........
...... |
00001430 00 00 05 00 00 00 00 00 00 00 00 00 00 20 aa 85
|............. ..|
00001440 02 00 10 00 12 00 10 00 20 04 10 00 00 80 00 20 |........
...... |
00001450 00 00 05 00 00 00 00 00 00 00 00 00 00 20 c4 24
|............. .$|
00001460 03 00 10 00 13 00 10 00 20 06 10 00 00 80 00 20 |........
...... |
00001470 00 00 05 00 00 00 00 00 00 00 00 00 00 20 1f 84
|............. ..|
00001480 04 00 10 00 14 00 10 00 20 08 10 00 00 80 00 20 |........
...... |
00001490 00 00 05 00 00 00 00 00 00 00 00 00 00 20 1b 26
|............. .&|
000014a0 05 00 10 00 15 00 10 00 20 0a 10 00 00 80 00 20 |........
...... |
000014b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 c0 86
|............. ..|
000014c0 06 00 10 00 16 00 10 00 20 0c 10 00 00 80 00 20 |........
...... |
000014d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 ae 27
|............. .'|
000014e0 07 00 10 00 17 00 10 00 20 0e 10 00 00 80 00 20 |........
...... |
000014f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 75 87
|............. u.|
00001500 08 00 10 00 18 00 10 00 20 10 10 00 00 80 00 20 |........
...... |
00001510 00 00 05 00 00 00 00 00 00 00 00 00 00 20 a5 23
|............. .#|
00001520 09 00 10 00 19 00 10 00 20 12 10 00 00 80 00 20 |........
...... |
00001530 00 00 05 00 00 00 00 00 00 00 00 00 00 20 7e 83
|............. ~.|
00001540 0a 00 10 00 1a 00 10 00 20 14 10 00 00 80 00 20 |........
...... |
00001550 00 00 05 00 00 00 00 00 00 00 00 00 00 20 10 22
|............. ."|
00001560 0b 00 10 00 1b 00 10 00 20 16 10 00 00 80 00 20 |........
...... |
00001570 00 00 05 00 00 00 00 00 00 00 00 00 00 20 cb 82
|............. ..|
00001580 0c 00 10 00 1c 00 10 00 20 18 10 00 00 80 00 20 |........
...... |
00001590 00 00 05 00 00 00 00 00 00 00 00 00 00 20 cf 20
|............. . |
000015a0 0d 00 10 00 1d 00 10 00 20 1a 10 00 00 80 00 20 |........
...... |
000015b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 14 80
|............. ..|
000015c0 0e 00 10 00 1e 00 10 00 20 1c 10 00 00 80 00 20 |........
...... |
000015d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 7a 21
|............. z!|
000015e0 0f 00 10 00 1f 00 10 00 20 1e 10 00 00 80 00 20 |........
...... |
000015f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 a1 81
|............. ..|
00001600 00 00 18 00 10 00 18 00 20 00 18 00 e0 5f 00 20 |........
...._. |
00001610 00 00 05 00 00 00 00 00 00 00 00 00 00 20 38 aa
|............. 8.|
00001620 01 00 18 00 11 00 18 00 20 02 18 00 ff 7b 00 20 |........
....{. |
00001630 00 00 05 00 00 00 00 00 00 00 00 00 00 20 8f d3
|............. ..|
00001640 02 00 18 00 12 00 18 00 20 04 18 00 00 80 00 20 |........
...... |
00001650 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d2 57
|............. .W|
00001660 03 00 18 00 13 00 18 00 20 06 18 00 00 80 00 20 |........
...... |
00001670 00 00 05 00 00 00 00 00 00 00 00 00 00 20 09 f7
|............. ..|
00001680 04 00 18 00 14 00 18 00 20 08 18 00 00 80 00 20 |........
...... |
00001690 00 00 05 00 00 00 00 00 00 00 00 00 00 20 0d 55
|............. .U|
000016a0 05 00 18 00 15 00 18 00 20 0a 18 00 00 80 00 20 |........
...... |
000016b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d6 f5
|............. ..|
000016c0 06 00 18 00 16 00 18 00 20 0c 18 00 00 80 00 20 |........
...... |
000016d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b8 54
|............. .T|
000016e0 07 00 18 00 17 00 18 00 20 0e 18 00 00 80 00 20 |........
...... |
000016f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 63 f4
|............. c.|
00001700 08 00 18 00 18 00 18 00 20 10 18 00 00 80 00 20 |........
...... |
00001710 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b3 50
|............. .P|
00001720 09 00 18 00 19 00 18 00 20 12 18 00 00 80 00 20 |........
...... |
00001730 00 00 05 00 00 00 00 00 00 00 00 00 00 20 68 f0
|............. h.|
00001740 0a 00 18 00 1a 00 18 00 20 14 18 00 00 80 00 20 |........
...... |
00001750 00 00 05 00 00 00 00 00 00 00 00 00 00 20 06 51
|............. .Q|
00001760 0b 00 18 00 1b 00 18 00 20 16 18 00 00 80 00 20 |........
...... |
00001770 00 00 05 00 00 00 00 00 00 00 00 00 00 20 dd f1
|............. ..|
00001780 0c 00 18 00 1c 00 18 00 20 18 18 00 00 80 00 20 |........
...... |
00001790 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d9 53
|............. .S|
000017a0 0d 00 18 00 1d 00 18 00 20 1a 18 00 00 80 00 20 |........
...... |
000017b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 02 f3
|............. ..|
000017c0 0e 00 18 00 1e 00 18 00 20 1c 18 00 00 80 00 20 |........
...... |
000017d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 6c 52
|............. lR|
000017e0 0f 00 18 00 1f 00 18 00 20 1e 18 00 00 80 00 20 |........
...... |
000017f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b7 f2
|............. ..|
00001800 00 00 20 00 10 00 20 00 20 00 20 00 e0 5f 00 20 |.. ... . .
.._. |
00001810 00 00 05 00 00 00 00 00 00 00 00 00 00 20 59 b3
|............. Y.|
00001820 01 00 20 00 11 00 20 00 20 02 20 00 00 80 00 20 |.. ... . .
.... |
00001830 00 00 05 00 00 00 00 00 00 00 00 00 00 20 dd ef
|............. ..|
00001840 02 00 20 00 12 00 20 00 20 04 20 00 00 80 00 20 |.. ... . .
.... |
00001850 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b3 4e
|............. .N|
00001860 03 00 20 00 13 00 20 00 20 06 20 00 00 80 00 20 |.. ... . .
.... |
00001870 00 00 05 00 00 00 00 00 00 00 00 00 00 20 68 ee
|............. h.|
00001880 04 00 20 00 14 00 20 00 20 08 20 00 00 80 00 20 |.. ... . .
.... |
00001890 00 00 05 00 00 00 00 00 00 00 00 00 00 20 6c 4c
|............. lL|
000018a0 05 00 20 00 15 00 20 00 20 0a 20 00 00 80 00 20 |.. ... . .
.... |
000018b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b7 ec
|............. ..|
000018c0 06 00 20 00 16 00 20 00 20 0c 20 00 00 80 00 20 |.. ... . .
.... |
000018d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d9 4d
|............. .M|
000018e0 07 00 20 00 17 00 20 00 20 0e 20 00 00 80 00 20 |.. ... . .
.... |
000018f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 02 ed
|............. ..|
00001900 08 00 20 00 18 00 20 00 20 10 20 00 00 80 00 20 |.. ... . .
.... |
00001910 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d2 49
|............. .I|
00001920 09 00 20 00 19 00 20 00 20 12 20 00 00 80 00 20 |.. ... . .
.... |
00001930 00 00 05 00 00 00 00 00 00 00 00 00 00 20 09 e9
|............. ..|
00001940 0a 00 20 00 1a 00 20 00 20 14 20 00 00 80 00 20 |.. ... . .
.... |
00001950 00 00 05 00 00 00 00 00 00 00 00 00 00 20 67 48
|............. gH|
00001960 0b 00 20 00 1b 00 20 00 20 16 20 00 00 80 00 20 |.. ... . .
.... |
00001970 00 00 05 00 00 00 00 00 00 00 00 00 00 20 bc e8
|............. ..|
00001980 0c 00 20 00 1c 00 20 00 20 18 20 00 00 80 00 20 |.. ... . .
.... |
00001990 00 00 05 00 00 00 00 00 00 00 00 00 00 20 b8 4a
|............. .J|
000019a0 0d 00 20 00 1d 00 20 00 20 1a 20 00 00 80 00 20 |.. ... . .
.... |
000019b0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 63 ea
|............. c.|
000019c0 0e 00 20 00 1e 00 20 00 20 1c 20 00 00 80 00 20 |.. ... . .
.... |
000019d0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 0d 4b
|............. .K|
000019e0 0f 00 20 00 1f 00 20 00 20 1e 20 00 00 80 00 20 |.. ... . .
.... |
000019f0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 d6 eb
|............. ..|
00001a00 00 00 28 00 10 00 28 00 20 00 28 00 e0 5f 00 20 |..(...(.
.(.._. |
00001a10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 4f c0
|............. O.|
00001a20 01 00 28 00 11 00 28 00 20 02 28 00 ff 7b 00 20 |..(...(.
.(..{. |
00001a30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 f8 b9
|............. ..|
00001a40 02 00 28 00 12 00 28 00 20 04 28 00 00 80 00 20 |..(...(.
.(.... |
00001a50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 a5 3d
|............. .=|
00001a60 03 00 28 00 13 00 28 00 20 06 28 00 00 80 00 20 |..(...(.
.(.... |
00001a70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 7e 9d
|............. ~.|
00001a80 04 00 28 00 14 00 28 00 20 08 28 00 00 80 00 20 |..(...(.
.(.... |
00001a90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 7a 3f
|............. z?|
00001aa0 05 00 28 00 15 00 28 00 20 0a 28 00 00 80 00 20 |..(...(.
.(.... |
00001ab0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 a1 9f
|............. ..|
00001ac0 06 00 28 00 16 00 28 00 20 0c 28 00 00 80 00 20 |..(...(.
.(.... |
00001ad0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 cf 3e
|............. .>|
00001ae0 07 00 28 00 17 00 28 00 20 0e 28 00 00 80 00 20 |..(...(.
.(.... |
00001af0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 14 9e
|............. ..|
00001b00 08 00 28 00 18 00 28 00 20 10 28 00 00 80 00 20 |..(...(.
.(.... |
00001b10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 c4 3a
|............. .:|
00001b20 09 00 28 00 19 00 28 00 20 12 28 00 00 80 00 20 |..(...(.
.(.... |
00001b30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 1f 9a
|............. ..|
00001b40 0a 00 28 00 1a 00 28 00 20 14 28 00 00 80 00 20 |..(...(.
.(.... |
00001b50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 71 3b
|............. q;|
00001b60 0b 00 28 00 1b 00 28 00 20 16 28 00 00 80 00 20 |..(...(.
.(.... |
00001b70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 aa 9b
|............. ..|
00001b80 0c 00 28 00 1c 00 28 00 20 18 28 00 00 80 00 20 |..(...(.
.(.... |
00001b90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 ae 39
|............. .9|
00001ba0 0d 00 28 00 1d 00 28 00 20 1a 28 00 00 80 00 20 |..(...(.
.(.... |
00001bb0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 75 99
|............. u.|
00001bc0 0e 00 28 00 1e 00 28 00 20 1c 28 00 00 80 00 20 |..(...(.
.(.... |
00001bd0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 1b 38
|............. .8|
00001be0 0f 00 28 00 1f 00 28 00 20 1e 28 00 00 80 00 20 |..(...(.
.(.... |
00001bf0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 c0 98
|............. ..|
00001c00 00 00 30 00 10 00 30 00 20 00 30 00 e0 5f 00 20 |..0...0.
.0.._. |
00001c10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 75 55
|............. uU|
00001c20 01 00 30 00 11 00 30 00 20 02 30 00 00 80 00 20 |..0...0.
.0.... |
00001c30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 f1 09
|............. ..|
00001c40 02 00 30 00 12 00 30 00 20 04 30 00 00 80 00 20 |..0...0.
.0.... |
00001c50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 9f a8
|............. ..|
00001c60 03 00 30 00 13 00 30 00 20 06 30 00 00 80 00 20 |..0...0.
.0.... |
00001c70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 44 08
|............. D.|
00001c80 04 00 30 00 14 00 30 00 20 08 30 00 00 80 00 20 |..0...0.
.0.... |
00001c90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 40 aa
|............. @.|
00001ca0 05 00 30 00 15 00 30 00 20 0a 30 00 00 80 00 20 |..0...0.
.0.... |
00001cb0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 9b 0a
|............. ..|
00001cc0 06 00 30 00 16 00 30 00 20 0c 30 00 00 80 00 20 |..0...0.
.0.... |
00001cd0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 f5 ab
|............. ..|
00001ce0 07 00 30 00 17 00 30 00 20 0e 30 00 00 80 00 20 |..0...0.
.0.... |
00001cf0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 2e 0b
|............. ..|
00001d00 08 00 30 00 18 00 30 00 20 10 30 00 00 80 00 20 |..0...0.
.0.... |
00001d10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 fe af
|............. ..|
00001d20 09 00 30 00 19 00 30 00 20 12 30 00 00 80 00 20 |..0...0.
.0.... |
00001d30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 25 0f
|............. %.|
00001d40 0a 00 30 00 1a 00 30 00 20 14 30 00 00 80 00 20 |..0...0.
.0.... |
00001d50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 4b ae
|............. K.|
00001d60 0b 00 30 00 1b 00 30 00 20 16 30 00 00 80 00 20 |..0...0.
.0.... |
00001d70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 90 0e
|............. ..|
00001d80 0c 00 30 00 1c 00 30 00 20 18 30 00 00 80 00 20 |..0...0.
.0.... |
00001d90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 94 ac
|............. ..|
00001da0 0d 00 30 00 1d 00 30 00 20 1a 30 00 00 80 00 20 |..0...0.
.0.... |
00001db0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 4f 0c
|............. O.|
00001dc0 0e 00 30 00 1e 00 30 00 20 1c 30 00 00 80 00 20 |..0...0.
.0.... |
00001dd0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 21 ad
|............. !.|
00001de0 0f 00 30 00 1f 00 30 00 20 1e 30 00 00 80 00 20 |..0...0.
.0.... |
00001df0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 fa 0d
|............. ..|
00001e00 00 00 38 00 10 00 38 00 20 00 38 00 e0 5f 00 20 |..8...8.
.8.._. |
00001e10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 63 26
|............. c&|
00001e20 01 00 38 00 11 00 38 00 20 02 38 00 00 80 00 20 |..8...8.
.8.... |
00001e30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 e7 7a
|............. .z|
00001e40 02 00 38 00 12 00 38 00 20 04 38 00 00 80 00 20 |..8...8.
.8.... |
00001e50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 89 db
|............. ..|
00001e60 03 00 38 00 13 00 38 00 20 06 38 00 00 80 00 20 |..8...8.
.8.... |
00001e70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 52 7b
|............. R{|
00001e80 04 00 38 00 14 00 38 00 20 08 38 00 00 80 00 20 |..8...8.
.8.... |
00001e90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 56 d9
|............. V.|
00001ea0 05 00 38 00 15 00 38 00 20 0a 38 00 00 80 00 20 |..8...8.
.8.... |
00001eb0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 8d 79
|............. .y|
00001ec0 06 00 38 00 16 00 38 00 20 0c 38 00 00 80 00 20 |..8...8.
.8.... |
00001ed0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 e3 d8
|............. ..|
00001ee0 07 00 38 00 17 00 38 00 20 0e 38 00 00 80 00 20 |..8...8.
.8.... |
00001ef0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 38 78
|............. 8x|
00001f00 08 00 38 00 18 00 38 00 20 10 38 00 00 80 00 20 |..8...8.
.8.... |
00001f10 00 00 05 00 00 00 00 00 00 00 00 00 00 20 e8 dc
|............. ..|
00001f20 09 00 38 00 19 00 38 00 20 12 38 00 00 80 00 20 |..8...8.
.8.... |
00001f30 00 00 05 00 00 00 00 00 00 00 00 00 00 20 33 7c
|............. 3||
00001f40 0a 00 38 00 1a 00 38 00 20 14 38 00 00 80 00 20 |..8...8.
.8.... |
00001f50 00 00 05 00 00 00 00 00 00 00 00 00 00 20 5d dd
|............. ].|
00001f60 0b 00 38 00 1b 00 38 00 20 16 38 00 00 80 00 20 |..8...8.
.8.... |
00001f70 00 00 05 00 00 00 00 00 00 00 00 00 00 20 86 7d
|............. .}|
00001f80 0c 00 38 00 1c 00 38 00 20 18 38 00 00 80 00 20 |..8...8.
.8.... |
00001f90 00 00 05 00 00 00 00 00 00 00 00 00 00 20 82 df
|............. ..|
00001fa0 0d 00 38 00 1d 00 38 00 20 1a 38 00 ff 7b 00 20 |..8...8.
.8..{. |
00001fb0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 6a 5a
|............. jZ|
00001fc0 0e 00 38 00 1e 00 38 00 20 1c 38 00 00 80 00 20 |..8...8.
.8.... |
00001fd0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 37 de
|............. 7.|
00001fe0 0f 00 38 00 1f 00 38 00 20 1e 38 00 00 80 00 20 |..8...8.
.8.... |
00001ff0 00 00 05 00 00 00 00 00 00 00 00 00 00 20 ec 7e
|............. .~|
00002000
> Also show the following more picky signature search for each drive:
> dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep -E '[048c]30 .+ 53
> ef'
I will provide this as soon as the processes are finished, seems to take
some more hours.
there is also one more thing I remember, maybe it's helpful: when I was
starting the reshape, there was this message:
mdadm: Need to backup 8192K of critical section..
not completely sure if it was exactly 8192k, but it was about 8MB
Thanks a lot!
^ permalink raw reply
* Re: Buffer I/O errors & Kernel OOPS with RAID6
From: Shaohua Li @ 2015-11-09 17:35 UTC (permalink / raw)
To: matt; +Cc: linux-raid
In-Reply-To: <cac386db0aa8fec187d8a2c9f0582934@digitallyhosted.com>
On Mon, Nov 09, 2015 at 11:40:00AM +0000, matt@digitallyhosted.com wrote:
> Hello,
>
> I am experiencing issues with RAID6 on all kernel versions I have tried
> (3.18.12, 4.0.9, 4.1.12).
>
> On 3.18.12, I am getting the following logged to dmesg:
>
> 896.874943] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858058 (offset 16777216 size 1052672 starting block
> 5172953088)
> [ 896.874945] Buffer I/O error on device md4, logical block 5172953088
> [ 896.874947] Buffer I/O error on device md4, logical block 5172953089
> [ 896.874948] Buffer I/O error on device md4, logical block 5172953090
> [ 896.874949] Buffer I/O error on device md4, logical block 5172953091
> [ 896.874950] Buffer I/O error on device md4, logical block 5172953092
> [ 896.874950] Buffer I/O error on device md4, logical block 5172953093
> [ 896.874951] Buffer I/O error on device md4, logical block 5172953094
> [ 896.874952] Buffer I/O error on device md4, logical block 5172953095
> [ 896.874953] Buffer I/O error on device md4, logical block 5172953096
> [ 896.874953] Buffer I/O error on device md4, logical block 5172953097
> [ 897.034829] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 1052672 starting block
> 5172955136)
> [ 897.122306] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 2101248 starting block
> 5172955264)
> [ 897.130547] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 2101248 starting block
> 5172955392)
> [ 897.355966] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 2625536 starting block
> 5172955520)
> [ 897.452464] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858058 (offset 16777216 size 1576960 starting block
> 5172953216)
> [ 897.593480] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 3149824 starting block
> 5172955648)
> [ 897.877728] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 3674112 starting block
> 5172955776)
> [ 898.156331] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858073 (offset 8388608 size 4198400 starting block
> 5172955904)
> [ 898.176687] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
> writing to inode 361858058 (offset 16777216 size 2101248 starting block
> 5172953344)
>
> When this happens, I end up with a file on the array which is partially
> corrupt. For example, if i copied a jpeg file, parts of the image would be
> garbage.
>
> I initially thought that this could be a kernel issue, so I tried two
> further kernel versions (4.0.9 & 4.1.12) and on both, I don't get the above
> messages anymore, instead I get a kernel oops and any process accessing the
> array will get stuck in state D. Here is a typical kernel oops message:
>
> [ 158.138253] BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000120
> [ 158.138391] IP: [<ffffffffa024cc1f>] handle_stripe+0xdc0/0x1e1f [raid456]
> [ 158.138482] PGD 24ff59067 PUD 24fe43067 PMD 0
> [ 158.138646] Oops: 0000 [#1] SMP
> [ 158.138758] Modules linked in: ipv6 binfmt_misc joydev
> x86_pkg_temp_thermal coretemp kvm_intel kvm microcode pcspkr video i2c_i801
> thermal acpi_cpufreq fan battery rtc_cmos backlight processor thermal_sys
> xhci_pci button xts gf128mul aes_x86_64 cbc sha256_generic
> scsi_transport_iscsi multipath linear raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0
> dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log dm_mod
> hid_sunplus hid_sony led_class hid_samsung hid_pl hid_petalynx hid_monterey
> hid_microsoft hid_logitech hid_gyration hid_ezkey hid_cypress hid_chicony
> hid_cherry hid_belkin hid_apple hid_a4tech sl811_hcd usbhid xhci_hcd
> ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd usbcore usb_common
> megaraid_sas megaraid_mbox megaraid_mm megaraid sx8
> [ 158.141809] DAC960 cciss mptsas mptfc scsi_transport_fc mptspi
> scsi_transport_spi mptscsih mptbase sg
> [ 158.142226] CPU: 0 PID: 2017 Comm: md4_raid6 Not tainted 4.1.12-gentoo #1
> [ 158.142272] Hardware name: Supermicro X10SAT/X10SAT, BIOS 2.0 04/21/2014
> [ 158.142323] task: ffff880254267050 ti: ffff880095afc000 task.ti:
> ffff880095afc000
> [ 158.142376] RIP: 0010:[<ffffffffa024cc1f>] [<ffffffffa024cc1f>]
> handle_stripe+0xdc0/0x1e1f [raid456]
> [ 158.142493] RSP: 0018:ffff880095affc18 EFLAGS: 00010202
> [ 158.142554] RAX: 000000000000000d RBX: ffff880095cfac00 RCX:
> 0000000000000002
> [ 158.142617] RDX: 000000000000000d RSI: 0000000000000000 RDI:
> 0000000000001040
> [ 158.142682] RBP: ffff880095affcf8 R08: 0000000000000003 R09:
> 00000000cd920408
> [ 158.142745] R10: 000000000000000d R11: 0000000000000007 R12:
> 000000000000000d
> [ 158.142809] R13: 0000000000000000 R14: 000000000000000c R15:
> ffff8802161f2588
> [ 158.142873] FS: 0000000000000000(0000) GS:ffff88025ea00000(0000)
> knlGS:0000000000000000
> [ 158.142938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 158.143000] CR2: 0000000000000120 CR3: 0000000253ef4000 CR4:
> 00000000001406f0
> [ 158.143062] Stack:
> [ 158.143117] 0000000000000000 ffff880254267050 00000000000147c0
> 0000000000000000
> [ 158.143328] ffff8802161f25d0 0000000effffffff ffff8802161f3670
> ffff8802161f2ef0
> [ 158.143537] 0000000000000000 0000000000000000 0000000000000000
> 0000000c00000000
> [ 158.143747] Call Trace:
> [ 158.143805] [<ffffffffa024dea3>]
> handle_active_stripes.isra.37+0x225/0x2aa [raid456]
> [ 158.143873] [<ffffffffa024e31d>] raid5d+0x363/0x40d [raid456]
> [ 158.143937] [<ffffffff814315bc>] ? schedule+0x6f/0x7e
> [ 158.143998] [<ffffffff81372ae7>] md_thread+0x125/0x13b
> [ 158.144060] [<ffffffff81061b00>] ? wait_woken+0x71/0x71
> [ 158.144122] [<ffffffff813729c2>] ? md_start_sync+0xda/0xda
> [ 158.144185] [<ffffffff81050609>] kthread+0xcd/0xd5
> [ 158.144244] [<ffffffff8105053c>] ? kthread_create_on_node+0x16d/0x16d
> [ 158.144309] [<ffffffff81434f92>] ret_from_fork+0x42/0x70
> [ 158.144370] [<ffffffff8105053c>] ? kthread_create_on_node+0x16d/0x16d
> [ 158.144432] Code: 8c 0f d0 01 00 00 48 8b 49 10 80 e1 10 74 0d 49 8b 4f
> 48 80 e1 40 0f 84 c2 0f 00 00 31 c9 41 39 c8 7e 31 48 8b b4 cd 50 ff ff ff
> <48> 83 be 20 01 00 00 00 74 1a 48 8b be 38 01 00 00 40 80 e7 01
> [ 158.147700] RIP [<ffffffffa024cc1f>] handle_stripe+0xdc0/0x1e1f [raid456]
> [ 158.147801] RSP <ffff880095affc18>
> [ 158.147859] CR2: 0000000000000120
> [ 158.147916] ---[ end trace 536b72bd7c91f068 ]---
>
> In both cases, discs are never flagged as faulty and the array never goes
> into a degraded state.
>
> I have tried posting this in various forums with no solution so far. A post
> with further information can be found here:
> https://forums.gentoo.org/viewtopic-t-1032304.html - In that topic I have
> supplied output from various commands that people have asked me to execute.
> Rather than pasting all the output from these commands here have linked to
> the thread instead.
>
> Any Idea's what could be going on? Any help would be greatly appreciated.
Could you please try a upstream kernel? there are some fixes in error handling
side recently, might be related.
ebda780bce8d58ec0ab
36707bb2e7c6730d79
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Mathias Mueller @ 2015-11-09 15:41 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux raid
In-Reply-To: <5640B8AD.3030800@turmel.org>
> I trimmed the rest because a valid signature was at the very beginning
> of the device. So you had either v0.90 or v1.0 metadata. I would bet
> it's v0.90. I would also bet that the chunk size is 64k.
>
> I suggest you try that first:
>
> mdadm --create --assume-clean --level=5 --raid-devices=4 --chunk=64 \
> --metadata=0.90 /dev/md1 /dev/sd{d,c,b,e}1
>
> Do *not* mount it yet!
>
> Do a non-destructive fsck to see if my guess was accurate:
>
> fsck -n /dev/md1
>
> Generally, if this has many errors, you would stop the array, and try
> the other order combinations (but always with sdd1 first) in turn.
>
> However, you don't say, and we can't tell from the new metadata, how
> far into the reshape mdadm had gotten. If very far, there will be
> many errors no matter what order you try, and your array is toast.
>
> But a result with few fsck errors is almost certainly success,
> and you can mount your array.
>
> BTW, if you recall where you found the suggestion to re-create your
> array, please go back and report there how crazy that is, and point
> them at this mailing list.
>
> Phil
Hi Phil,
thanks for your reply and sorry for my mistakes, I'm new with the
mailinglist.
I tried your suggestion but it did not work, fsck could not find a
superblock. I'm very sure, that I used the metadata v1.2 (and also kind
of sure that I used a chunk size of 512k). Sadly I tried to mount the
array yesterday when I was trying the different device orders.
Is it helpful to post the hexdump of the other devices as well?
Thanks for your Help!
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Phil Turmel @ 2015-11-09 13:50 UTC (permalink / raw)
To: Mikael Abrahamsson, Mathias Mueller; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1511091254290.24520@uplift.swm.pp.se>
On 11/09/2015 06:56 AM, Mikael Abrahamsson wrote:
> On Mon, 9 Nov 2015, Mathias Mueller wrote:
>
>> Please help me, I know I'm stupid and don't deserve it. I really hope,
>> there is a chance for reovering the array.
>
> It's possible. There are numerous threads in the mailing list archives
> with people who have been in the same situation as you are in now.
> Without having a backup of the original superblocks, you're going to
> have to guess offsets. It might make sense to investigate what the
> defaults were for the mdadm you originally created the array with.
Chunks size defaults have varied, too.
I would search the known first drive for an ext4 filesystem signature
and work backwards from there. Show the output of this search:
dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep ' 53 ef ' |head -n 1000
Phil
^ permalink raw reply
* Re: RAID6 reshape stalls immediately
From: Phil Turmel @ 2015-11-09 13:38 UTC (permalink / raw)
To: Anugraha Sinha, Peter Chubb; +Cc: linux-raid
In-Reply-To: <56409F4E.5000508@gmail.com>
On 11/09/2015 08:27 AM, Anugraha Sinha wrote:
> I am looping in to Phil explicitly for some help here.
>
> @Phil,
> Need some help here!
I don't know why it's stuck. This has never happened to me :-).
Neil helped someone recently with a similar problem, and suggested
omitting the backup-file. It isn't needed for --grow operations that
increase capacity. I also recall some discussions about systemd vs.
mdmon issues -- if that applies to you.
If omitting the --backup-file fails, I would reboot into a rescue CD
with a recent kernel and mdadm version. I recommend www.sysrescuecd.ord
for such things. Then use mdadm --grow --continue within the rescue
environment.
Phil
^ permalink raw reply
* Re: RAID6 reshape stalls immediately
From: Anugraha Sinha @ 2015-11-09 13:27 UTC (permalink / raw)
To: Peter Chubb; +Cc: linux-raid, Phil Turmel
In-Reply-To: <84611dw2cw.wl-peter.chubb@nicta.com.au>
Dear Peter,
Apologies for the late reply.
On 11/8/2015 1:18 PM, Peter Chubb wrote:
>
> ls -l /root/raid5-backup
> -rw------- 1 root root 6295552 Nov 4 12:04 raid5-backup
>
> It hasn't changed since the first write when the reshape started.
That means there is no write happening on the root file which was chosen
for backup.
Just a check, for sanity, I hope the root partition is not filled up.
>
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x5
> Array UUID : 6ff23c3d:01042464:77338dc6:710dfaee
> Name : lemma:0 (local to host lemma)
> Creation Time : Tue Oct 27 11:52:53 2015
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 3906748416 (1862.88 GiB 2000.26 GB)
> Array Size : 7813496832 (7451.53 GiB 8001.02 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=0 sectors
> State : clean
> Device UUID : fbd91407:208c66cc:1020f996:d7b1cd5d
>
> Internal Bitmap : 8 sectors from superblock
> Reshape pos'n : 0
> Delta Devices : 1 (5->6)
> New Layout : left-symmetric
>
> Update Time : Sun Nov 8 15:12:08 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : b3ec395a - correct
> Events : 48094
>
> Layout : left-symmetric-6
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
\snip\
> And /proc/mdstat is still:
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid6 sdi1[6] sdh1[5] sdd1[4] sde1[2] sdb1[1] sda1[0]
> 5860122624 blocks super 1.2 level 6, 512k chunk, algorithm 18 [6/5] [UUUU_U]
> [>....................] reshape = 0.0% (0/1953374208) finish=5766.8min speed=5469K/sec
> bitmap: 0/15 pages [0KB], 65536KB chunk
>
> unused devices: <none>
>
\snip\
1. your mdstat says that sdb1 device is down. However, examine says sdb1
seems OK. I am not sure why?
Could you check for the daemon(process) running for md0 and see the
strace of it. Where is it waiting?
Also I would want to check, what have been the last few dmesg in your
system, any updated since the resyncing/reshaping started, as you shared
in the last mail.
I am looping in to Phil explicitly for some help here.
@Phil,
Need some help here!
Regards
Anugraha Sinha
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Mikael Abrahamsson @ 2015-11-09 11:56 UTC (permalink / raw)
To: Mathias Mueller; +Cc: linux-raid
In-Reply-To: <15194c2e14b9a7c3431853dea9dc8b5e@pingofdeath.de>
On Mon, 9 Nov 2015, Mathias Mueller wrote:
> Please help me, I know I'm stupid and don't deserve it. I really hope,
> there is a chance for reovering the array.
It's possible. There are numerous threads in the mailing list archives
with people who have been in the same situation as you are in now. Without
having a backup of the original superblocks, you're going to have to guess
offsets. It might make sense to investigate what the defaults were for the
mdadm you originally created the array with.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Buffer I/O errors & Kernel OOPS with RAID6
From: matt @ 2015-11-09 11:40 UTC (permalink / raw)
To: linux-raid
Hello,
I am experiencing issues with RAID6 on all kernel versions I have tried
(3.18.12, 4.0.9, 4.1.12).
On 3.18.12, I am getting the following logged to dmesg:
896.874943] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error -5
writing to inode 361858058 (offset 16777216 size 1052672 starting block
5172953088)
[ 896.874945] Buffer I/O error on device md4, logical block 5172953088
[ 896.874947] Buffer I/O error on device md4, logical block 5172953089
[ 896.874948] Buffer I/O error on device md4, logical block 5172953090
[ 896.874949] Buffer I/O error on device md4, logical block 5172953091
[ 896.874950] Buffer I/O error on device md4, logical block 5172953092
[ 896.874950] Buffer I/O error on device md4, logical block 5172953093
[ 896.874951] Buffer I/O error on device md4, logical block 5172953094
[ 896.874952] Buffer I/O error on device md4, logical block 5172953095
[ 896.874953] Buffer I/O error on device md4, logical block 5172953096
[ 896.874953] Buffer I/O error on device md4, logical block 5172953097
[ 897.034829] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 1052672 starting
block 5172955136)
[ 897.122306] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 2101248 starting
block 5172955264)
[ 897.130547] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 2101248 starting
block 5172955392)
[ 897.355966] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 2625536 starting
block 5172955520)
[ 897.452464] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858058 (offset 16777216 size 1576960 starting
block 5172953216)
[ 897.593480] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 3149824 starting
block 5172955648)
[ 897.877728] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 3674112 starting
block 5172955776)
[ 898.156331] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858073 (offset 8388608 size 4198400 starting
block 5172955904)
[ 898.176687] EXT4-fs warning (device md4): ext4_end_bio:317: I/O error
-5 writing to inode 361858058 (offset 16777216 size 2101248 starting
block 5172953344)
When this happens, I end up with a file on the array which is partially
corrupt. For example, if i copied a jpeg file, parts of the image would
be garbage.
I initially thought that this could be a kernel issue, so I tried two
further kernel versions (4.0.9 & 4.1.12) and on both, I don't get the
above messages anymore, instead I get a kernel oops and any process
accessing the array will get stuck in state D. Here is a typical kernel
oops message:
[ 158.138253] BUG: unable to handle kernel NULL pointer dereference at
0000000000000120
[ 158.138391] IP: [<ffffffffa024cc1f>] handle_stripe+0xdc0/0x1e1f
[raid456]
[ 158.138482] PGD 24ff59067 PUD 24fe43067 PMD 0
[ 158.138646] Oops: 0000 [#1] SMP
[ 158.138758] Modules linked in: ipv6 binfmt_misc joydev
x86_pkg_temp_thermal coretemp kvm_intel kvm microcode pcspkr video
i2c_i801 thermal acpi_cpufreq fan battery rtc_cmos backlight processor
thermal_sys xhci_pci button xts gf128mul aes_x86_64 cbc sha256_generic
scsi_transport_iscsi multipath linear raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0
dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log dm_mod
hid_sunplus hid_sony led_class hid_samsung hid_pl hid_petalynx
hid_monterey hid_microsoft hid_logitech hid_gyration hid_ezkey
hid_cypress hid_chicony hid_cherry hid_belkin hid_apple hid_a4tech
sl811_hcd usbhid xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_pci
ehci_hcd usbcore usb_common megaraid_sas megaraid_mbox megaraid_mm
megaraid sx8
[ 158.141809] DAC960 cciss mptsas mptfc scsi_transport_fc mptspi
scsi_transport_spi mptscsih mptbase sg
[ 158.142226] CPU: 0 PID: 2017 Comm: md4_raid6 Not tainted
4.1.12-gentoo #1
[ 158.142272] Hardware name: Supermicro X10SAT/X10SAT, BIOS 2.0
04/21/2014
[ 158.142323] task: ffff880254267050 ti: ffff880095afc000 task.ti:
ffff880095afc000
[ 158.142376] RIP: 0010:[<ffffffffa024cc1f>] [<ffffffffa024cc1f>]
handle_stripe+0xdc0/0x1e1f [raid456]
[ 158.142493] RSP: 0018:ffff880095affc18 EFLAGS: 00010202
[ 158.142554] RAX: 000000000000000d RBX: ffff880095cfac00 RCX:
0000000000000002
[ 158.142617] RDX: 000000000000000d RSI: 0000000000000000 RDI:
0000000000001040
[ 158.142682] RBP: ffff880095affcf8 R08: 0000000000000003 R09:
00000000cd920408
[ 158.142745] R10: 000000000000000d R11: 0000000000000007 R12:
000000000000000d
[ 158.142809] R13: 0000000000000000 R14: 000000000000000c R15:
ffff8802161f2588
[ 158.142873] FS: 0000000000000000(0000) GS:ffff88025ea00000(0000)
knlGS:0000000000000000
[ 158.142938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 158.143000] CR2: 0000000000000120 CR3: 0000000253ef4000 CR4:
00000000001406f0
[ 158.143062] Stack:
[ 158.143117] 0000000000000000 ffff880254267050 00000000000147c0
0000000000000000
[ 158.143328] ffff8802161f25d0 0000000effffffff ffff8802161f3670
ffff8802161f2ef0
[ 158.143537] 0000000000000000 0000000000000000 0000000000000000
0000000c00000000
[ 158.143747] Call Trace:
[ 158.143805] [<ffffffffa024dea3>]
handle_active_stripes.isra.37+0x225/0x2aa [raid456]
[ 158.143873] [<ffffffffa024e31d>] raid5d+0x363/0x40d [raid456]
[ 158.143937] [<ffffffff814315bc>] ? schedule+0x6f/0x7e
[ 158.143998] [<ffffffff81372ae7>] md_thread+0x125/0x13b
[ 158.144060] [<ffffffff81061b00>] ? wait_woken+0x71/0x71
[ 158.144122] [<ffffffff813729c2>] ? md_start_sync+0xda/0xda
[ 158.144185] [<ffffffff81050609>] kthread+0xcd/0xd5
[ 158.144244] [<ffffffff8105053c>] ?
kthread_create_on_node+0x16d/0x16d
[ 158.144309] [<ffffffff81434f92>] ret_from_fork+0x42/0x70
[ 158.144370] [<ffffffff8105053c>] ?
kthread_create_on_node+0x16d/0x16d
[ 158.144432] Code: 8c 0f d0 01 00 00 48 8b 49 10 80 e1 10 74 0d 49 8b
4f 48 80 e1 40 0f 84 c2 0f 00 00 31 c9 41 39 c8 7e 31 48 8b b4 cd 50 ff
ff ff <48> 83 be 20 01 00 00 00 74 1a 48 8b be 38 01 00 00 40 80 e7 01
[ 158.147700] RIP [<ffffffffa024cc1f>] handle_stripe+0xdc0/0x1e1f
[raid456]
[ 158.147801] RSP <ffff880095affc18>
[ 158.147859] CR2: 0000000000000120
[ 158.147916] ---[ end trace 536b72bd7c91f068 ]---
In both cases, discs are never flagged as faulty and the array never
goes into a degraded state.
I have tried posting this in various forums with no solution so far. A
post with further information can be found here:
https://forums.gentoo.org/viewtopic-t-1032304.html - In that topic I
have supplied output from various commands that people have asked me to
execute. Rather than pasting all the output from these commands here
have linked to the thread instead.
Any Idea's what could be going on? Any help would be greatly
appreciated.
Kind regards,
Matthew Jones
^ permalink raw reply
* broken raid level 5 array caused by user error
From: Mathias Mueller @ 2015-11-09 11:27 UTC (permalink / raw)
To: linux-raid
Hi Folks,
I'm running a raid level 5 with 4 devices for some years and tried to
grow my array yesterday. I wanted to add two more devices and used the
following commands:
mdadm --add /dev/md0 /dev/sdf1 /dev/sdg1
mdadm --grow --raid-devices=6 /dev/md0
So far, so good. Everything seems to work, but after about 2 hours, the
reshape progress was still at 0.0% and now, my own stupidity kicked in.
I checked the logs via journalctl (I'm running Centos 7) and read
something about "main process died" or similar... then I decided to
reboot.
After reboot, assembling the array failed:
mdadm: Failed to restore critical section for reshape, sorry. Possibly
you needed to specify the --backup-file
But I did not have a backup file and so I panicked and made even worse
decisions.
First I tried to assemble the array using --invalid-backup but it did
not work. I should stop here and ask but I didn't. I read at some board,
that rebuilding the original array with 4 devices will fix my problem. I
did not validate this and entered the suggested command:
mdadm -CR /dev/md0 --metadata=1.2 -n4 -l5 -c512 /dev/sd[bcde]1
--assume-clean
But this did not work (the array assembled but I could not access the
ext4 filesystem), it seems that I assembled it in the wrong device
order, so I also tried different (i.e. all possible) orders, but nothing
helped ( I always used --assume-clean).
I guess this is the perfect guide for how _not_ to do it :(
I continued reading and found this:
http://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using
This gave me some hope and now I wonder, if there is a way to get my
data back, maybe the offset is wrong?
Things I know about the array:
metadata: 1.2
Left-symetric
chunk-size: 512
When I run mdadm --detail /dev/md0 it still shows an array size of 6TB,
the UUID is also still the same
Version : 1.2
Creation Time : Mon Nov 9 00:00:40 2015
Raid Level : raid5
Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Nov 9 00:00:45 2015
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : xxxx
UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
Events : 1
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 65 1 active sync /dev/sde1
2 8 33 2 active sync /dev/sdc1
3 8 17 3 active sync /dev/sdb1
an mdadm --examine gives this results:
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
Name : xxxx
Creation Time : Mon Nov 9 00:00:40 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906766961 (1862.89 GiB 2000.26 GB)
Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=5233 sectors
State : clean
Device UUID : e14f0e2d:a26a7b90:d7dbf780:e2218327
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Nov 9 00:00:45 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7a76b0d6 - correct
Events : 1
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
Name : xxxx
Creation Time : Mon Nov 9 00:00:40 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=3248 sectors
State : clean
Device UUID : d408e617:37f3f0f5:feb5d77f:07e57668
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Nov 9 00:00:45 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a9787e9 - correct
Events : 1
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 3907024002 sectors at 63 (type fd)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
Name : xxxx
Creation Time : Mon Nov 9 00:00:40 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906761858 (1862.89 GiB 2000.26 GB)
Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=130 sectors
State : clean
Device UUID : faf7ec39:e7c0cb77:770a439d:18dc65a0
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Nov 9 00:00:45 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 3d38419 - correct
Events : 1
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d0fdb4e:6111bd7a:96cad2dd:b6a29039
Name : xxx
Creation Time : Mon Nov 9 00:00:40 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 5860142592 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=3248 sectors
State : clean
Device UUID : fe31b351:3559f949:978035ae:616ae615
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Nov 9 00:00:45 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 743a6702 - correct
Events : 1
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
I guess I know the old device order as well, I saved an old boot-log:
md: bind<sde1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<sdd1>
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md/raid:md127: device sdd1 operational as raid disk 0
md/raid:md127: device sdc1 operational as raid disk 1
md/raid:md127: device sdb1 operational as raid disk 2
md/raid:md127: device sde1 operational as raid disk 3
md/raid:md127: allocated 4314kB
md/raid:md127: raid level 5 active with 4 out of 4 devices, algorithm 2
created bitmap (15 pages) for device md127
md127: bitmap initialized from disk: read 1 pages, set 0 of 29809 bits
md127: detected capacity change from 0 to 6001188667392
Please help me, I know I'm stupid and don't deserve it. I really hope,
there is a chance for reovering the array.
Thanks a lot in advance
Mathias
^ permalink raw reply
* [PATCH 2/8] mmc: handle add_disk() return value
From: Vishnu Pratap Singh @ 2015-11-09 5:11 UTC (permalink / raw)
To: axboe, akpm, linux-kernel, jmoyer, minchan, ngupta,
sergey.senozhatsky.work, davem, neilb, ulf.hansson, tiwai, hare,
ming.lei, jarod, viro, tj, adrian.hunter, jonathanh, grundler,
linux-ide, linux-raid, linux-mmc
Cc: cpgs, vishu13285, pintu.k, rohit.kr, Vishnu Pratap Singh
In-Reply-To: <1446812535-10567-2-git-send-email-vishnu.ps@samsung.com>
This patch handles add_disk() return value.
Earlier add_disk() function doesn't handle error
cases, now it is added, so the callers of this function
should also handle it.
Signed-off-by: Vishnu Pratap Singh <vishnu.ps@samsung.com>
--- Verfied on X86 based ubuntu machine.
--- This patch depends on [PATCH 1/8] block/genhd.c: Add error handling
---
drivers/mmc/card/block.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c
index 23b6c8e..543c670 100644
--- a/drivers/mmc/card/block.c
+++ b/drivers/mmc/card/block.c
@@ -2432,7 +2432,10 @@ static int mmc_add_disk(struct mmc_blk_data *md)
int ret;
struct mmc_card *card = md->queue.card;
- add_disk(md->disk);
+ ret = add_disk(md->disk);
+ if (ret)
+ goto add_disk_fail;
+
md->force_ro.show = force_ro_show;
md->force_ro.store = force_ro_store;
sysfs_attr_init(&md->force_ro.attr);
@@ -2468,7 +2471,7 @@ power_ro_lock_fail:
device_remove_file(disk_to_dev(md->disk), &md->force_ro);
force_ro_fail:
del_gendisk(md->disk);
-
+add_disk_fail:
return ret;
}
--
1.9.1
^ permalink raw reply related
* Re: [PATCH 06/32] xen blkback: prepare for bi_rw split
From: Bob Liu @ 2015-11-09 4:00 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-scsi, linux-kernel, linux-raid, dm-devel, mchristi,
linux-fsdevel, xen-devel, drbd-dev
In-Reply-To: <20151107101713.GB12117@infradead.org>
On 11/07/2015 06:17 PM, Christoph Hellwig wrote:
> A little offtopic for this patch, but can some explain this whole
> mess about bios in Xen blkfront? We can happily do partial completions
> at the request later.
>
> Also since the blk-mq conversion the call to blk_end_request_all is
This will be fixed after my next blk-mq patch series which also modified the recover path.
> completely broken, so it doesn't look like this code path is exactly
> well tested.
>
Thanks,
-Bob
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox