* (unknown),
From: citydesk @ 2017-07-18 4:32 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: "EMAIL_40199138625_linux-raid.zip --]
[-- Type: application/zip, Size: 186 bytes --]
^ permalink raw reply
* Re: [PATCH 1/5] md: add a 'md_numa_node' module parameter
From: Zhengyuan Liu @ 2017-07-18 2:03 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, neilb, colyli, liuyun01
In-Reply-To: <20170710172844.hfln24mylpyaoxnw@kernel.org>
2017-07-11 1:28 GMT+08:00 Shaohua Li <shli@kernel.org>:
> On Sat, Jul 01, 2017 at 03:20:54PM +0800, Zhengyuan Liu wrote:
>> This module parameter allows user to control which numa node
>> the memory for mainly structures (e.g. mddev, md_rdev,
>> request_queue, gendisk) is allocated from. The idea came from
>> commit 115485e83f49 ("dm: add 'dm_numa_node' module parameter").
>> If we don't set this parameter while loading module, it won't
>> affect the md default behavior.
>>
>> numa_node_id field is added so personality such as raid0 could
>> inherit the node from mddev, for other personality which has its
>> own handling thread we could add such a module parameter too.
>>
>> This patch includes a bug fix about parameter range check suggested
>> by Coly Li <colyli@suse.de>.
>>
>> Signed-off-by: Zhengyuan Liu <liuzhengyuan@kylinos.cn>
>
> Can you give an example why this is useful?
To be honest, I do't catch clearly performance improvement on my arm64 numa
server with those patch sets. But, theoretically it should make sense especially
for direct IO when all memory allocation binding to a local numa node through
the IO path. I have did a fio test based on raid0 which created from two raw device,
of course I added a numa module parameter to drivers/block/rbd.c too, the iops
got a bit better (about 2%) when md and brd bind to a same node comparing to
no module parameter, but not always better since numa is handled transparently
by memory management in most case.
>
> If we have several raid arrays, we can't control numa node for them separately.
Yes, current patch doesn't consider this scenario. To control them fall into different
nuna node separately, we could make the module parameter writable just like
commit 115485e83f49 does.
> And in the patch 4/5, one array could have two different parameters to control
> memory allocation. this is weird. I have no idea how users use this feature.
>
> If we really want to support numa node, would it be better to deduce the numa
> node from the combination of all underlayer disks?
I think it is feasible, as we could get the device numa node of all md_rdev when creating.
If any two md_rdev get different numa node property, then we can stop keeping going.
Thanks,
Zhengyuan
>
> Thanks,
> Shaohua
>
>> ---
>> drivers/md/md.c | 18 ++++++++++++++----
>> drivers/md/md.h | 2 ++
>> 2 files changed, 16 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 092b48f..fae93db 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -184,6 +184,8 @@ static int start_readonly;
>> */
>> static bool create_on_open = true;
>>
>> +static int md_numa_node = NUMA_NO_NODE;
>> +
>> /* bio_clone_mddev
>> * like bio_clone, but with a local bio set
>> */
>> @@ -588,7 +590,7 @@ static struct mddev *mddev_find(dev_t unit)
>> }
>> spin_unlock(&all_mddevs_lock);
>>
>> - new = kzalloc(sizeof(*new), GFP_KERNEL);
>> + new = kzalloc_node(sizeof(*new), GFP_KERNEL, md_numa_node);
>> if (!new)
>> return NULL;
>>
>> @@ -598,6 +600,7 @@ static struct mddev *mddev_find(dev_t unit)
>> else
>> new->md_minor = MINOR(unit) >> MdpMinorShift;
>>
>> + new->numa_node_id = md_numa_node;
>> mddev_init(new);
>>
>> goto retry;
>> @@ -3387,7 +3390,7 @@ static struct md_rdev *md_import_device(dev_t newdev, int super_format, int supe
>> struct md_rdev *rdev;
>> sector_t size;
>>
>> - rdev = kzalloc(sizeof(*rdev), GFP_KERNEL);
>> + rdev = kzalloc_node(sizeof(*rdev), GFP_KERNEL, md_numa_node);
>> if (!rdev)
>> return ERR_PTR(-ENOMEM);
>>
>> @@ -5256,7 +5259,7 @@ static int md_alloc(dev_t dev, char *name)
>> mddev->hold_active = UNTIL_STOP;
>>
>> error = -ENOMEM;
>> - mddev->queue = blk_alloc_queue(GFP_KERNEL);
>> + mddev->queue = blk_alloc_queue_node(GFP_KERNEL, md_numa_node);
>> if (!mddev->queue)
>> goto abort;
>> mddev->queue->queuedata = mddev;
>> @@ -5264,7 +5267,7 @@ static int md_alloc(dev_t dev, char *name)
>> blk_queue_make_request(mddev->queue, md_make_request);
>> blk_set_stacking_limits(&mddev->queue->limits);
>>
>> - disk = alloc_disk(1 << shift);
>> + disk = alloc_disk_node(1 << shift, md_numa_node);
>> if (!disk) {
>> blk_cleanup_queue(mddev->queue);
>> mddev->queue = NULL;
>> @@ -8969,6 +8972,11 @@ static int __init md_init(void)
>> register_reboot_notifier(&md_notifier);
>> raid_table_header = register_sysctl_table(raid_root_table);
>>
>> + if (md_numa_node < NUMA_NO_NODE)
>> + md_numa_node = NUMA_NO_NODE;
>> + else if (md_numa_node > (num_online_nodes() - 1))
>> + md_numa_node = num_online_nodes() - 1;
>> +
>> md_geninit();
>> return 0;
>>
>> @@ -9252,6 +9260,8 @@ module_param_call(start_ro, set_ro, get_ro, NULL, S_IRUSR|S_IWUSR);
>> module_param(start_dirty_degraded, int, S_IRUGO|S_IWUSR);
>> module_param_call(new_array, add_named_array, NULL, NULL, S_IWUSR);
>> module_param(create_on_open, bool, S_IRUSR|S_IWUSR);
>> +module_param(md_numa_node, int, S_IRUGO);
>> +MODULE_PARM_DESC(md_numa_node, "NUMA node for md device memory allocations");
>>
>> MODULE_LICENSE("GPL");
>> MODULE_DESCRIPTION("MD RAID framework");
>> diff --git a/drivers/md/md.h b/drivers/md/md.h
>> index 991f0fe..5c91033 100644
>> --- a/drivers/md/md.h
>> +++ b/drivers/md/md.h
>> @@ -459,6 +459,8 @@ struct mddev {
>> void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev);
>> struct md_cluster_info *cluster_info;
>> unsigned int good_device_nr; /* good device num within cluster raid */
>> +
>> + int numa_node_id;
>> };
>>
>> enum recovery_flags {
>> --
>> 2.7.4
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* (unknown),
From: citydesk @ 2017-07-17 21:54 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: "EMAIL_976833055_linux-raid.zip --]
[-- Type: application/zip, Size: 3245 bytes --]
^ permalink raw reply
* Re: [PATCH v2 0/3] md: three misc changes
From: Shaohua Li @ 2017-07-17 16:41 UTC (permalink / raw)
To: Ming Lei
Cc: linux-raid, NeilBrown, linux-block, Jens Axboe, Christoph Hellwig
In-Reply-To: <20170714081444.32645-1-ming.lei@redhat.com>
On Fri, Jul 14, 2017 at 04:14:41PM +0800, Ming Lei wrote:
> This 1st patch fixes one issue introduced in the following two
> commits:
> Fixes: f0250618361d(md: raid10: don't use bio's vec table to manage resync pages)
> Fixes: 98d30c5812c3(md: raid1: don't use bio's vec table to manage resync pages)
>
> The 2nd one initializes bvec table of bio via bio_add_page() after bio_reset().
>
> The 3rd one moves the common definitation and helpers into raid1-10.c.
applied, thanks!
> V2:
> - fix 'page_idx' increasement in patch 1
> - move raid1/raid10 common code to raid1-raid10.c, as suggested by Neil
>
> Ming Lei (3):
> md: remove 'idx' from 'struct resync_pages'
> md: raid1/raid10: initialize bvec table via bio_add_page()
> md: raid1-10: move raid1/raid10 common code into raid1-10.c
>
> drivers/md/md.h | 54 ----------------------------------
> drivers/md/raid1-10.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid1.c | 31 ++++----------------
> drivers/md/raid10.c | 19 ++++--------
> 4 files changed, 93 insertions(+), 92 deletions(-)
> create mode 100644 drivers/md/raid1-10.c
>
> --
> 2.9.4
>
^ permalink raw reply
* Re: Linear device of two arrays
From: Veljko @ 2017-07-17 10:16 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <87shhyzhg5.fsf@notabene.neil.brown.name>
On 07/15/2017 02:12 AM, NeilBrown wrote:
> So command should be
>
>>> dd if=/dev/md2 of=SOMEWHERE/SAFE bs=1M skip=5626590
>
> and expect it to create a 3M file.
>
> Use this 'skip' number of the 'seek' number later.
>
> NeilBrown
Thanks, Neil, now it makes more sense.
I tried to create new linear device, but mdadm is complaining about
data-offset:
# mdadm -C /dev/md4 -l linear -n 2 --rounding=1M -e 1.0 --data-offset=0
/dev/md2 /dev/md3
mdadm: invalid data-offset: 0
I'm using Debian 8.8 if it makes any difference.
# mdadm -V
mdadm - v3.3.2 - 21st August 2014
What could be the problem?
Regards,
Veljko
^ permalink raw reply
* [PATCH -v2 1/1] mdadm/test: Add one test case for raid5 reshape
From: Xiao Ni @ 2017-07-17 9:27 UTC (permalink / raw)
To: linux-raid; +Cc: jes.sorensen
This case trys to allow raid5 reshape to use backwards direction.
It changes chunksize after reshape and stop the raid. Then start
the raid again.
Signed-off-by: Xiao Ni <xni@redhat.com>
---
tests/02r5grow | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/tests/02r5grow b/tests/02r5grow
index 386e82e..64c5599 100644
--- a/tests/02r5grow
+++ b/tests/02r5grow
@@ -34,3 +34,16 @@ check nosync
sh tests/testdev $md0 3 $[size/2] 128
mdadm -S $md0
+
+# create a raid5 array and change the chunk
+mdadm -CR $md0 --level raid5 --metadata=1.1 --chunk=32 --raid-disks 3 --size $[size/2] $dev1 $dev2 $dev3
+check wait
+check state UUU
+
+mdadm $md0 --grow --chunk=64
+check reshape
+check wait
+
+mdadm -S $md0
+mdadm -As
+check state UUU
--
2.7.4
^ permalink raw reply related
* Re: Why can't I re-add my drive after partition shrink?
From: Ram Ramesh @ 2017-07-17 5:15 UTC (permalink / raw)
To: NeilBrown, Linux Raid
In-Reply-To: <87pod0ypmh.fsf@notabene.neil.brown.name>
On 07/16/2017 05:37 PM, NeilBrown wrote:
> On Fri, Jul 14 2017, Ram Ramesh wrote:
>
>> On 07/13/2017 08:35 PM, NeilBrown wrote:
>>> <snip
>>> Please report output of "mdadm --examine" on both a device that is active
>>> in the array, and the device that you are trying to add.
>>> Also "mdadm --examine-bitmap" of a device that is active in the array.
>>>
>>> NeilBrown
>>>
>>>> I researched as much as I could on the net and came up with nothing
>>>> except some one saying that mdadm keeps something at the end of the disk
>>>> regardless of what it says about "Used Dev Size." Is it possible to move
>>>> this info so that I could re-add?
>>>>
>>>> Ramesh
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Neil,
>>
>> Since the problem, I did not want to leave my md in degraded state.
>> So, I added my drive back and paid the penalty for rebuilding. I have
>> other disks that need to be resized and *can get you want*. Please let
>> me know if that is what you meant. If you wanted the current info after
>> successfully rebuilding the array after a regular add, it is below.
> I only requested the information because it might help fix, or explain,
> your difficulty. If you don't currently have a difficulty, then I don't
> need to look at any details.
>
> Thanks,
> NeilBrown
Thanks for your time. Yes, I still have the problem as I need to shrink
other 5 disks in the array and I like to re-add rather than add and
rebuild each time.
The host with the array is currently busy, and I will get this info
tomorrow when I attempt the process on my next hard drive.
Ramesh
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Adam Goryachev @ 2017-07-17 0:41 UTC (permalink / raw)
To: Jean-Pierre Human, Anthony Youngman; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <CAH_1uby=5GvrRfE5v_UYNyNev_5ZLWP1Sr48me5siR+Gm_yrGA@mail.gmail.com>
On 17/07/17 02:46, Jean-Pierre Human wrote:
> On Sun, Jul 16, 2017 at 1:03 AM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>>
>> On 15/07/17 19:05, Jean-Pierre Human wrote:
>>> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
>>> <Andreas.Klauer@metamorpher.de> wrote:
>>
>>> Hi Andreas
>>>
>>> Thanks for the prompt response,
>>>
>>> Yes I did not use a partition table. I have always in the past used
>>> partitions but recently noticed a trend to use the device, wont do
>>> that again.
>>>
>>> I am really not sure what wrote those partitions there. I only see
>>> free space in them if I check with fdisk.
>>>
>>> I am sure it was metadata version 1.2
>>>
>>> I will attempt to recover using the overlay, will post my results.
>>
>> Good call. The whole point of overlays is to enable you to write-protect the
>> physical disks. That means (of course) that you can't make matters worse.
>> That's always the worry - that an attempt at a fix will damage the content
>> of the disks and make a recovery harder/impossible.
>>>
>>> Thanks again for your help.
>>>
>> A little more (hopefully) help - on the wiki the last entry in the "When
>> things go wrogn" section is about recovering a trashed array. As it says,
>> it's a work in progress, but it gives you various hints about how to search
>> the disk for clues as to where your data is, and hence what all those things
>> like offsets are.
>>
>> Have you ever added disks to your array? Rebuilt and changed its
>> configuration? This can mess about with the superblock values changing them
>> from the defaults. This is information the list will want.
>>
>> If you have trouble getting overlays to work, search for clues like this on
>> your list and get back here for help. ON NO ACCOUNT do anything that will
>> actually write to your disks unless you are absolutely confident that you've
>> found the correct magic incantation.
>>
>> There are plenty of people here who will help you get your data back - and
>> they've got a good track record! Unfortunately, I'm "learning by
>> documenting" so I'm quite happy to give you hints and guidance, but I can't
>> say for certain what is likely to work.
>>
>> Cheers,
>> Wol
> Hello List
>
> I am happy to say I managed to recover all the data and it was perfectly intact.
>
> To address some of the above questions, the array and server install
> are relatively new 3 months or so. There have been no disk additions,
> replacements or failures. I had issues getting the overlay working
> mainly due to my lack of understanding and time restraints (used dd
> and sfdisk to kinda make backups of the important bits).
>
> Using the same disk order, mdadm version and settings when it was
> initially installed it started perfectly with a --create
> --assume-clean. The existing LVM was also still intact and once the
> ISCSI connection was back online the data was there. I feel I got very
> lucky mainly due to the server being in a roughly constant error free
> state since install.
>
> The odd thing was that after I had the array running and before I had
> started the ISCSI, I had to reboot the server, when it came up the
> newly written superblocks were gone again and I had to --create
> --assume-clean to get it running again. Once I have secured this data
> I will find out why this is happening.
I can't be sure, but I suspect what happened is this:
1) setup the write overlay
2) Do the rapairs/etc
3) Export drive and test
4) Reboot - everything is lost
The reason for this is that the write overlay has been lost by the
reboot, which is the point of the write overlay, it can easily be thrown
away if you accidentally messed up.
What you need to verify now is that:
1) You are not using a write overlay for your current/live data, or else
you will lose it after the next reboot
2) Reboot and make sure everything works properly, so you are not left
in a similar issue next time
Regards,
Adam
--
--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.
^ permalink raw reply
* Re: Advice please re failed Raid6
From: Peter Grandi @ 2017-07-17 0:19 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <9dca5b7a-b60e-0e93-41fd-49d092d8b27b@gmail.com>
> mdadm --create --assume-clean [ ... ]
That's a very dangerous recovery method that you need to get
exactly right or it will cause trouble.
Also it should be used in very rare cases, not routinely to
recover from one missing disk.
> root@keruru:/var/log# mdadm --examine /dev/sd[bedc] >> raid.status
> root@keruru:/var/log# cat raid.status
> /dev/sdb:
> Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
> Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Device UUID : 79e4933f:dfe5923f:5ba03ae7:3efe38eb
> Events : 119
> Chunk Size : 512K
> Device Role : Active device 0
> Array State : AAAA ('A' == active, '.' == missing)
> /dev/sdc:
> Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
> Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Device UUID : f1e1a946:711886a6:2604780f:8eba4a2d
> Events : 119
> Chunk Size : 512K
> Device Role : Active device 1
> /dev/sdd:
> Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
> Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Device UUID : cf3bc8a7:9feed87d:945d8e77:08f7f32d
> Events : 119
> Chunk Size : 512K
> Device Role : Active device 2
> /dev/sde:
> Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
> Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Events : 119
> Chunk Size : 512K
> Device Role : Active device 3
That AAAA meant that the array was fine. The important field of
'--examine' confirm that. The "dirty degraded array" meant most
likely some slight event count difference, usually one just
forces assembly in that case.
The line that worries a bit is this:
Jul 5 21:06:18 keruru mdadm[2497]: RebuildFinished event
detected on md device /dev/md0, component device mismatches
found: 1847058224 (on raid level 6)
That seems to indicate that pretty much every block was a
mismatch. Which would have happened if you put in a blank drive
and then used '--create --assume-clean' instead of '--assemble
--force'. But '--assume-clean" explicitly skips a rebuild, and I
wonder whether you omitted that yoiu have triggered a "repair"
in 'sync_action'. Also the message is reported by 'mdadm', and
it may be that 'mdadm' was running in daemon mode and triggering
a periodic "repair". I can't remember the defaults.
HOWEVER there is a very subtle detail: the order of the devices
from '--examine' is: 0: 'sdb', 1: 'sdc', 2: 'sdd', 3: 'sde' but
you recreated the set in a different order. The order of the
devices does not matter if they have MD superblocks,.but here
you are using '--create' to make new superblocks, and they order
must match exactly the original order.
> root@keruru:/var/log# mdadm --create --assume-clean --level=6 --raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
Probably the best thing you can do is to rerun this with members
"missing /dev/sdc /dev/sdd /dev/sde". and then use 'blkid /dev/md0'
to check whether the data in it is recognized again. If so add
'/dev/sdb'. I did a quick test here of something close to that
and it worked...
^ permalink raw reply
* Re: [PATCH v2 3/3] md: raid1-10: move raid1/raid10 common code into raid1-10.c
From: Ming Lei @ 2017-07-17 0:15 UTC (permalink / raw)
To: Coly Li
Cc: Ming Lei, Shaohua Li,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT, NeilBrown,
linux-block, Jens Axboe, Christoph Hellwig
In-Reply-To: <72dd4c34-f780-e20f-f604-3edf018b138a@suse.de>
On Sun, Jul 16, 2017 at 1:14 PM, Coly Li <colyli@suse.de> wrote:
> On 2017/7/14 下午4:14, Ming Lei wrote:
>> No function change, just move 'struct resync_pages' and related
>> helpers into raid1-10.c
>
> Please give me a hint, I don't find where raid1-10.c is ....
It is introduced in this patch, :-)
--
Ming Lei
^ permalink raw reply
* Re: [PATCH v2 3/3] md: raid1-10: move raid1/raid10 common code into raid1-10.c
From: NeilBrown @ 2017-07-16 22:40 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: linux-block, Jens Axboe, Christoph Hellwig, Ming Lei
In-Reply-To: <20170714081444.32645-4-ming.lei@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5466 bytes --]
On Fri, Jul 14 2017, Ming Lei wrote:
> No function change, just move 'struct resync_pages' and related
> helpers into raid1-10.c
>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
Thanks for doing this!
I'm quite happy with this approach - and with all patches in this
series.
Thanks,
NeilBrown
> ---
> drivers/md/md.h | 53 -------------------------------------------
> drivers/md/raid1-10.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid1.c | 9 --------
> drivers/md/raid10.c | 9 --------
> 4 files changed, 62 insertions(+), 71 deletions(-)
>
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index 2c780aa8d07f..004a21c214e8 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -729,57 +729,4 @@ static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio
> !bdev_get_queue(bio->bi_bdev)->limits.max_write_zeroes_sectors)
> mddev->queue->limits.max_write_zeroes_sectors = 0;
> }
> -
> -/* Maximum size of each resync request */
> -#define RESYNC_BLOCK_SIZE (64*1024)
> -#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
> -
> -/* for managing resync I/O pages */
> -struct resync_pages {
> - void *raid_bio;
> - struct page *pages[RESYNC_PAGES];
> -};
> -
> -static inline int resync_alloc_pages(struct resync_pages *rp,
> - gfp_t gfp_flags)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++) {
> - rp->pages[i] = alloc_page(gfp_flags);
> - if (!rp->pages[i])
> - goto out_free;
> - }
> -
> - return 0;
> -
> -out_free:
> - while (--i >= 0)
> - put_page(rp->pages[i]);
> - return -ENOMEM;
> -}
> -
> -static inline void resync_free_pages(struct resync_pages *rp)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++)
> - put_page(rp->pages[i]);
> -}
> -
> -static inline void resync_get_all_pages(struct resync_pages *rp)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++)
> - get_page(rp->pages[i]);
> -}
> -
> -static inline struct page *resync_fetch_page(struct resync_pages *rp,
> - unsigned idx)
> -{
> - if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
> - return NULL;
> - return rp->pages[idx];
> -}
> #endif /* _MD_MD_H */
> diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
> index 3adb5b9dc4b4..9f2670b45f31 100644
> --- a/drivers/md/raid1-10.c
> +++ b/drivers/md/raid1-10.c
> @@ -1,3 +1,65 @@
> +/* Maximum size of each resync request */
> +#define RESYNC_BLOCK_SIZE (64*1024)
> +#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
> +
> +/* for managing resync I/O pages */
> +struct resync_pages {
> + void *raid_bio;
> + struct page *pages[RESYNC_PAGES];
> +};
> +
> +static inline int resync_alloc_pages(struct resync_pages *rp,
> + gfp_t gfp_flags)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++) {
> + rp->pages[i] = alloc_page(gfp_flags);
> + if (!rp->pages[i])
> + goto out_free;
> + }
> +
> + return 0;
> +
> +out_free:
> + while (--i >= 0)
> + put_page(rp->pages[i]);
> + return -ENOMEM;
> +}
> +
> +static inline void resync_free_pages(struct resync_pages *rp)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++)
> + put_page(rp->pages[i]);
> +}
> +
> +static inline void resync_get_all_pages(struct resync_pages *rp)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++)
> + get_page(rp->pages[i]);
> +}
> +
> +static inline struct page *resync_fetch_page(struct resync_pages *rp,
> + unsigned idx)
> +{
> + if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
> + return NULL;
> + return rp->pages[idx];
> +}
> +
> +/*
> + * 'strct resync_pages' stores actual pages used for doing the resync
> + * IO, and it is per-bio, so make .bi_private points to it.
> + */
> +static inline struct resync_pages *get_resync_pages(struct bio *bio)
> +{
> + return bio->bi_private;
> +}
> +
> /* generally called after bio_reset() for reseting bvec */
> static void md_bio_reset_resync_pages(struct bio *bio, struct resync_pages *rp,
> int size)
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index fe86ab18961b..8387eb1540cd 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -84,15 +84,6 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
> #include "raid1-10.c"
>
> /*
> - * 'strct resync_pages' stores actual pages used for doing the resync
> - * IO, and it is per-bio, so make .bi_private points to it.
> - */
> -static inline struct resync_pages *get_resync_pages(struct bio *bio)
> -{
> - return bio->bi_private;
> -}
> -
> -/*
> * for resync bio, r1bio pointer can be retrieved from the per-bio
> * 'struct resync_pages'.
> */
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 9952721e1cde..e2617d0f37dc 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -113,15 +113,6 @@ static void end_reshape(struct r10conf *conf);
> #include "raid1-10.c"
>
> /*
> - * 'strct resync_pages' stores actual pages used for doing the resync
> - * IO, and it is per-bio, so make .bi_private points to it.
> - */
> -static inline struct resync_pages *get_resync_pages(struct bio *bio)
> -{
> - return bio->bi_private;
> -}
> -
> -/*
> * for resync bio, r10bio pointer can be retrieved from the per-bio
> * 'struct resync_pages'.
> */
> --
> 2.9.4
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Why can't I re-add my drive after partition shrink?
From: NeilBrown @ 2017-07-16 22:37 UTC (permalink / raw)
To: Ram Ramesh, Linux Raid
In-Reply-To: <0341a04d-9aa5-7665-d9ff-527fe9fc8238@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]
On Fri, Jul 14 2017, Ram Ramesh wrote:
> On 07/13/2017 08:35 PM, NeilBrown wrote:
>> <snip
>> Please report output of "mdadm --examine" on both a device that is active
>> in the array, and the device that you are trying to add.
>> Also "mdadm --examine-bitmap" of a device that is active in the array.
>>
>> NeilBrown
>>
>>> I researched as much as I could on the net and came up with nothing
>>> except some one saying that mdadm keeps something at the end of the disk
>>> regardless of what it says about "Used Dev Size." Is it possible to move
>>> this info so that I could re-add?
>>>
>>> Ramesh
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Neil,
>
> Since the problem, I did not want to leave my md in degraded state.
> So, I added my drive back and paid the penalty for rebuilding. I have
> other disks that need to be resized and *can get you want*. Please let
> me know if that is what you meant. If you wanted the current info after
> successfully rebuilding the array after a regular add, it is below.
I only requested the information because it might help fix, or explain,
your difficulty. If you don't currently have a difficulty, then I don't
need to look at any details.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Have I any chance of restoring my Raid6 data?
From: Joe Landman @ 2017-07-16 22:28 UTC (permalink / raw)
To: bogo.mipps, Linux Raid
In-Reply-To: <90ab63b9-3a4b-06fa-d66b-d212ee4ae94f@gmail.com>
On 07/16/2017 06:14 PM, Bogo Mipps wrote:
>
> Since then I've been unable to mount or access any data. Have followed
> instructions as per Linux Raid Wiki's "Recovering a failed software
> RAID" & "RAID Recovery" , but still no success. I've attached the
> results of their suggestions in the attached log file
> "linux_raid_wiki_logs.txt".
Why did you recreate the RAID (though you used --assume-clean) ? Did you
try mounting it before that?
>
> Is there a chance of restoring my data?
Depends ... if you 'fsck -n /dev/md0' does it say there are errors?
>
> P.S. This line looks ominous? <md0: detected capacity change from
> 4000528203776 to 0> !!!
That is because md0 was stopped.
--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
^ permalink raw reply
* Have I any chance of restoring my Raid6 data?
From: Bogo Mipps @ 2017-07-16 22:14 UTC (permalink / raw)
To: Linux Raid
[-- Attachment #1: Type: text/plain, Size: 16259 bytes --]
Have been running a 4 disk Raid 6 setup for over two years without any
issues, until suddenly on June 27 disks on my OpenMediaVault NAS became
100% full, including the NFS mounted volumes in the raid set. There'd
been major disk activity overnight, but foolishly didn't investigate.
Rsnapshot normally backs up two desktop machines onto the raid setup:
the next morning found that one of the backup directories was not on
raid but suddenly was on the root directory of the OMV machine, and
being a total backup (several Gb) this accounted for the 100% reading
for the OMV/NAS machine.
The logs indicated that mdstat had discovered "dirty degraded array"
presumably due to faulty sdb, so had withdrawn that disk, and then
couldn't run the raid set (logs below show)
Bought a new disk and installed on July 4, and raid rebuilt overnight
(see July 5 Rebuild finished below)
Since then I've been unable to mount or access any data. Have followed
instructions as per Linux Raid Wiki's "Recovering a failed software
RAID" & "RAID Recovery" , but still no success. I've attached the
results of their suggestions in the attached log file
"linux_raid_wiki_logs.txt".
Is there a chance of restoring my data?
P.S. This line looks ominous? <md0: detected capacity change from
4000528203776 to 0> !!!
===============Jun 27 16:52:21 keruru kernel: [ 2.912440] md: md0 stopped.
Jun 27 16:52:21 keruru kernel: [ 2.922315] md: bind<sdb>
Jun 27 16:52:21 keruru kernel: [ 2.922508] md: bind<sdc>
Jun 27 16:52:21 keruru kernel: [ 2.922643] md: bind<sde>
Jun 27 16:52:21 keruru kernel: [ 2.922777] md: bind<sdd>
Jun 27 16:52:21 keruru kernel: [ 2.922808] md: kicking non-fresh sdb
from array!
Jun 27 16:52:21 keruru kernel: [ 2.922820] md: unbind<sdb>
Jun 27 16:52:21 keruru kernel: [ 2.927107] md: export_rdev(sdb)
Jun 27 16:52:21 keruru kernel: [ 2.994973] raid6: sse2x1 588 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.062926] raid6: sse2x2 1395 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.130841] raid6: sse2x4 2397 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.130844] raid6: using algorithm sse2x4
(2397 MB/s)
Jun 27 16:52:21 keruru kernel: [ 3.130846] raid6: using ssse3x2 recovery
algorithm
Jun 27 16:52:21 keruru kernel: [ 3.130866] Switched to clocksource tsc
Jun 27 16:52:21 keruru kernel: [ 3.131227] xor: automatically using best
checksumming function:
Jun 27 16:52:21 keruru kernel: [ 3.170797] avx : 6164.000 MB/sec
Jun 27 16:52:21 keruru kernel: [ 3.171121] async_tx: api initialized (async)
Jun 27 16:52:21 keruru kernel: [ 3.172809] md: raid6 personality
registered for level 6
Jun 27 16:52:21 keruru kernel: [ 3.172812] md: raid5 personality
registered for level 5
Jun 27 16:52:21 keruru kernel: [ 3.172815] md: raid4 personality
registered for level 4
Jun 27 16:52:21 keruru kernel: [ 3.173218] md/raid:md0: not clean --
starting background reconstruction
Jun 27 16:52:21 keruru kernel: [ 3.173236] md/raid:md0: device sdd
operational as raid disk 1
Jun 27 16:52:21 keruru kernel: [ 3.173239] md/raid:md0: device sde
operational as raid disk 3
Jun 27 16:52:21 keruru kernel: [ 3.173242] md/raid:md0: device sdc
operational as raid disk 2
Jun 27 16:52:21 keruru kernel: [ 3.173706] md/raid:md0: allocated 0kB
Jun 27 16:52:21 keruru kernel: [ 3.173745] md/raid:md0: cannot start
dirty degraded array.
Jun 27 16:52:21 keruru kernel: [ 3.173811] RAID conf printout:
Jun 27 16:52:21 keruru kernel: [ 3.173814] --- level:6 rd:4 wd:3
Jun 27 16:52:21 keruru kernel: [ 3.173816] disk 1, o:1, dev:sdd
Jun 27 16:52:21 keruru kernel: [ 3.173818] disk 2, o:1, dev:sdc
Jun 27 16:52:21 keruru kernel: [ 3.173820] disk 3, o:1, dev:sde
Jun 27 16:52:21 keruru kernel: [ 3.174025] md/raid:md0: failed to run
raid set.
Jun 27 16:52:21 keruru kernel: [ 3.174071] md: pers->run() failed ...
===============
New disk added - sdb
===============
Jul 5 21:06:18 keruru mdadm[2497]: RebuildFinished event detected on md
device /dev/md0, component device mismatches found: 1847058224 (on raid
level 6)
Jul 6 09:45:52 keruru kernel: [ 1195.390879] raid6: sse2x1 249 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.458735] raid6: sse2x2 476 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.526632] raid6: sse2x4 839 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.526638] raid6: using algorithm
sse2x4 (839 MB/s)
Jul 6 09:45:52 keruru kernel: [ 1195.526644] raid6: using ssse3x2
recovery algorithm
Jul 6 09:45:52 keruru kernel: [ 1195.578970] md: raid6 personality
registered for level 6
Jul 6 09:45:52 keruru kernel: [ 1195.578980] md: raid5 personality
registered for level 5
Jul 6 09:45:52 keruru kernel: [ 1195.578985] md: raid4 personality
registered for level 4
Jul 6 09:45:52 keruru kernel: [ 1195.580003] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 09:45:52 keruru kernel: [ 1195.580012] md/raid:md0: device sde
operational as raid disk 3
Jul 6 09:45:52 keruru kernel: [ 1195.580018] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 09:45:52 keruru kernel: [ 1195.580025] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 09:45:52 keruru kernel: [ 1195.581091] md/raid:md0: allocated 0kB
Jul 6 09:45:52 keruru kernel: [ 1195.581180] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.321957] raid6: sse2x4 1130 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.321964] raid6: using algorithm sse2x4
(1130 MB/s)
Jul 6 09:52:30 keruru kernel: [ 4.321967] raid6: using ssse3x2 recovery
algorithm
Jul 6 09:52:30 keruru kernel: [ 4.368478] md: raid6 personality
registered for level 6
Jul 6 09:52:30 keruru kernel: [ 4.368486] md: raid5 personality
registered for level 5
Jul 6 09:52:30 keruru kernel: [ 4.368490] md: raid4 personality
registered for level 4
Jul 6 09:52:30 keruru kernel: [ 4.369179] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 09:52:30 keruru kernel: [ 4.369185] md/raid:md0: device sde
operational as raid disk 3
Jul 6 09:52:30 keruru kernel: [ 4.369189] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 09:52:30 keruru kernel: [ 4.369194] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 09:52:30 keruru kernel: [ 4.369974] md/raid:md0: allocated 0kB
Jul 6 09:52:30 keruru kernel: [ 4.372062] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 12:56:15 keruru kernel: [ 4.442184] raid6: sse2x1 739 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.510060] raid6: sse2x2 1480 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.577985] raid6: sse2x4 1605 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.577993] raid6: using algorithm sse2x4
(1605 MB/s)
Jul 6 12:56:15 keruru kernel: [ 4.577997] raid6: using ssse3x2 recovery
algorithm
Jul 6 12:56:15 keruru kernel: [ 4.622570] md: raid6 personality
registered for level 6
Jul 6 12:56:15 keruru kernel: [ 4.622577] md: raid5 personality
registered for level 5
Jul 6 12:56:15 keruru kernel: [ 4.622580] md: raid4 personality
registered for level 4
Jul 6 12:56:15 keruru kernel: [ 4.623261] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 12:56:15 keruru kernel: [ 4.623266] md/raid:md0: device sde
operational as raid disk 3
Jul 6 12:56:15 keruru kernel: [ 4.623269] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 12:56:15 keruru kernel: [ 4.623273] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 12:56:15 keruru kernel: [ 4.624064] md/raid:md0: allocated 0kB
Jul 6 12:56:15 keruru kernel: [ 4.624131] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 16:54:43 keruru kernel: [14401.858429] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 16:54:43 keruru kernel: [14401.858442] md/raid:md0: device sde
operational as raid disk 3
Jul 6 16:54:43 keruru kernel: [14401.858449] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 16:54:43 keruru kernel: [14401.858455] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 16:54:43 keruru kernel: [14401.859915] md/raid:md0: allocated 0kB
Jul 6 16:54:43 keruru kernel: [14401.860000] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
linux_raid_wiki_logs.txt
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] >> raid.status
root@keruru:/var/log# cat raid.status
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 79e4933f:dfe5923f:5ba03ae7:3efe38eb
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 9ff2b025 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f1e1a946:711886a6:2604780f:8eba4a2d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 784b0046 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cf3bc8a7:9feed87d:945d8e77:08f7f32d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 197bc63c - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b3323d81:279b7c7b:a0c534ed:46d0e6fc
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 352daaf4 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)
================
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] | egrep 'Event|/dev/sd'
/dev/sdb:
Events : 119
/dev/sdc:
Events : 119
/dev/sdd:
Events : 119
/dev/sde:
Events : 119
===============
root@keruru:/var/log# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@keruru:/var/log# mdadm --assemble --force /dev/md0 /dev/sdb
/dev/sde /dev/sdd /dev/sdc
mdadm: /dev/md0 has been started with 4 drives.
root@keruru:/var/log# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sdb[4] sde[3] sdd[2] sdc[1]
3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2
[4/4] [UUUU]
unused devices: <none>
===============
root@keruru:/var/log# grep Role raid.status
Device Role : Active device 0
Device Role : Active device 1
Device Role : Active device 2
Device Role : Active device 3
===============
root@keruru:/var/log# grep Used raid.status
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6
--raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd
/dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? n
mdadm: create aborted.
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6
--raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd
/dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
===============
root@keruru:/# mount -t ext4 /dev/md0 /mnt/md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
===============
root@keruru:/# dmesg | tail
[448318.800806] --- level:6 rd:4 wd:4
[448318.800812] disk 0, o:1, dev:sdb
[448318.800817] disk 1, o:1, dev:sde
[448318.800822] disk 2, o:1, dev:sdd
[448318.800827] disk 3, o:1, dev:sdc
[448318.800951] md0: detected capacity change from 0 to 4000528203776
[448318.809375] md0: unknown partition table
[448358.704189] EXT4-fs (md0): Unrecognized mount option "\x08" or
missing value
[448358.706680] EXT4-fs (md0): failed to parse options in superblock: \x08
[448358.706690] EXT4-fs (md0): Number of reserved GDT blocks insanely
large: 9216
===============
Jul 11 17:23:15 keruru kernel: [447719.812775] md: export_rdev(sdd)
Jul 11 17:23:19 keruru kernel: [447724.396327] md: md0 stopped.
Jul 11 17:23:19 keruru kernel: [447724.400278] md: bind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.001687] md0: detected capacity
change from 4000528203776 to 0
Jul 11 17:32:29 keruru kernel: [448273.001714] md: md0 stopped.
Jul 11 17:32:29 keruru kernel: [448273.001729] md: unbind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.022972] md: export_rdev(sdc)
Jul 11 17:32:29 keruru kernel: [448273.023143] md: unbind<sde>
Jul 11 17:32:29 keruru kernel: [448273.054889] md: export_rdev(sde)
Jul 11 17:32:29 keruru kernel: [448273.055035] md: unbind<sdd>
Jul 11 17:32:29 keruru kernel: [448273.086870] md: export_rdev(sdd)
Jul 11 17:33:14 keruru kernel: [448318.800827] disk 3, o:1, dev:sdc
===============
[-- Attachment #2: linux_raid_wiki_logs.txt --]
[-- Type: text/plain, Size: 7819 bytes --]
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] >> raid.status
root@keruru:/var/log# cat raid.status
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 79e4933f:dfe5923f:5ba03ae7:3efe38eb
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 9ff2b025 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f1e1a946:711886a6:2604780f:8eba4a2d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 784b0046 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cf3bc8a7:9feed87d:945d8e77:08f7f32d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 197bc63c - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b3323d81:279b7c7b:a0c534ed:46d0e6fc
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 352daaf4 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)
================
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] | egrep 'Event|/dev/sd'
/dev/sdb:
Events : 119
/dev/sdc:
Events : 119
/dev/sdd:
Events : 119
/dev/sde:
Events : 119
===============
root@keruru:/var/log# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@keruru:/var/log# mdadm --assemble --force /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/md0 has been started with 4 drives.
root@keruru:/var/log# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sdb[4] sde[3] sdd[2] sdc[1]
3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
===============
root@keruru:/var/log# grep Role raid.status
Device Role : Active device 0
Device Role : Active device 1
Device Role : Active device 2
Device Role : Active device 3
===============
root@keruru:/var/log# grep Used raid.status
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6 --raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? n
mdadm: create aborted.
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6 --raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
===============
root@keruru:/# mount -t ext4 /dev/md0 /mnt/md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
===============
root@keruru:/# dmesg | tail
[448318.800806] --- level:6 rd:4 wd:4
[448318.800812] disk 0, o:1, dev:sdb
[448318.800817] disk 1, o:1, dev:sde
[448318.800822] disk 2, o:1, dev:sdd
[448318.800827] disk 3, o:1, dev:sdc
[448318.800951] md0: detected capacity change from 0 to 4000528203776
[448318.809375] md0: unknown partition table
[448358.704189] EXT4-fs (md0): Unrecognized mount option "\x08" or missing value
[448358.706680] EXT4-fs (md0): failed to parse options in superblock: \x08
[448358.706690] EXT4-fs (md0): Number of reserved GDT blocks insanely large: 9216
===============
Jul 11 17:23:15 keruru kernel: [447719.812775] md: export_rdev(sdd)
Jul 11 17:23:19 keruru kernel: [447724.396327] md: md0 stopped.
Jul 11 17:23:19 keruru kernel: [447724.400278] md: bind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.001687] md0: detected capacity change from 4000528203776 to 0
Jul 11 17:32:29 keruru kernel: [448273.001714] md: md0 stopped.
Jul 11 17:32:29 keruru kernel: [448273.001729] md: unbind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.022972] md: export_rdev(sdc)
Jul 11 17:32:29 keruru kernel: [448273.023143] md: unbind<sde>
Jul 11 17:32:29 keruru kernel: [448273.054889] md: export_rdev(sde)
Jul 11 17:32:29 keruru kernel: [448273.055035] md: unbind<sdd>
Jul 11 17:32:29 keruru kernel: [448273.086870] md: export_rdev(sdd)
Jul 11 17:33:14 keruru kernel: [448318.800827] disk 3, o:1, dev:sdc
===============
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Jean-Pierre Human @ 2017-07-16 17:53 UTC (permalink / raw)
To: Wols Lists; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <596BA18F.9070101@youngman.org.uk>
On Sun, Jul 16, 2017 at 7:25 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 16/07/17 17:46, Jean-Pierre Human wrote:
>> The documentation on the wiki is very helpful and the most concise I
>> came across. The overlay feature I will spend sometime with to
>> understand as in the future I may need this. It just left me asking a
>> lot of questions like if I have LVM and no filesystem on the device
>> will it work or how should I adopt the setup for that etc. This
>> exercise has opened my eyes to just how resilient and flexible mdadm
>> can be.
>
> As the editor of the wiki, I do my best :-)
>
> Thing is, I don't have lvm or anything like that on my setup. So I'm
> rather chary of writing about it because I don't have the experience - I
> do feel it's a major failing of the wiki. When I rebuild my computer,
> I'm going to set it up as "lvm over raid 5/6", but that's probably going
> to be a little way away :-(
>
> Once I get the opportunity, of course it'll get written up and put there
> as a reference for others :-)
>
> Cheers,
> Wol
Hi Wol
If you would like a VM to help with that, contact me off list.
Thanks again
J-P Human
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Wols Lists @ 2017-07-16 17:25 UTC (permalink / raw)
To: Jean-Pierre Human; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <CAH_1uby=5GvrRfE5v_UYNyNev_5ZLWP1Sr48me5siR+Gm_yrGA@mail.gmail.com>
On 16/07/17 17:46, Jean-Pierre Human wrote:
> The documentation on the wiki is very helpful and the most concise I
> came across. The overlay feature I will spend sometime with to
> understand as in the future I may need this. It just left me asking a
> lot of questions like if I have LVM and no filesystem on the device
> will it work or how should I adopt the setup for that etc. This
> exercise has opened my eyes to just how resilient and flexible mdadm
> can be.
As the editor of the wiki, I do my best :-)
Thing is, I don't have lvm or anything like that on my setup. So I'm
rather chary of writing about it because I don't have the experience - I
do feel it's a major failing of the wiki. When I rebuild my computer,
I'm going to set it up as "lvm over raid 5/6", but that's probably going
to be a little way away :-(
Once I get the opportunity, of course it'll get written up and put there
as a reference for others :-)
Cheers,
Wol
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Jean-Pierre Human @ 2017-07-16 16:46 UTC (permalink / raw)
To: Anthony Youngman; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <f18f569d-88fe-d35d-8c85-319d431af770@youngman.org.uk>
On Sun, Jul 16, 2017 at 1:03 AM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
>
>
> On 15/07/17 19:05, Jean-Pierre Human wrote:
>>
>> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
>> <Andreas.Klauer@metamorpher.de> wrote:
>
>
>> Hi Andreas
>>
>> Thanks for the prompt response,
>>
>> Yes I did not use a partition table. I have always in the past used
>> partitions but recently noticed a trend to use the device, wont do
>> that again.
>>
>> I am really not sure what wrote those partitions there. I only see
>> free space in them if I check with fdisk.
>>
>> I am sure it was metadata version 1.2
>>
>> I will attempt to recover using the overlay, will post my results.
>
>
> Good call. The whole point of overlays is to enable you to write-protect the
> physical disks. That means (of course) that you can't make matters worse.
> That's always the worry - that an attempt at a fix will damage the content
> of the disks and make a recovery harder/impossible.
>>
>>
>> Thanks again for your help.
>>
> A little more (hopefully) help - on the wiki the last entry in the "When
> things go wrogn" section is about recovering a trashed array. As it says,
> it's a work in progress, but it gives you various hints about how to search
> the disk for clues as to where your data is, and hence what all those things
> like offsets are.
>
> Have you ever added disks to your array? Rebuilt and changed its
> configuration? This can mess about with the superblock values changing them
> from the defaults. This is information the list will want.
>
> If you have trouble getting overlays to work, search for clues like this on
> your list and get back here for help. ON NO ACCOUNT do anything that will
> actually write to your disks unless you are absolutely confident that you've
> found the correct magic incantation.
>
> There are plenty of people here who will help you get your data back - and
> they've got a good track record! Unfortunately, I'm "learning by
> documenting" so I'm quite happy to give you hints and guidance, but I can't
> say for certain what is likely to work.
>
> Cheers,
> Wol
Hello List
I am happy to say I managed to recover all the data and it was perfectly intact.
To address some of the above questions, the array and server install
are relatively new 3 months or so. There have been no disk additions,
replacements or failures. I had issues getting the overlay working
mainly due to my lack of understanding and time restraints (used dd
and sfdisk to kinda make backups of the important bits).
Using the same disk order, mdadm version and settings when it was
initially installed it started perfectly with a --create
--assume-clean. The existing LVM was also still intact and once the
ISCSI connection was back online the data was there. I feel I got very
lucky mainly due to the server being in a roughly constant error free
state since install.
The odd thing was that after I had the array running and before I had
started the ISCSI, I had to reboot the server, when it came up the
newly written superblocks were gone again and I had to --create
--assume-clean to get it running again. Once I have secured this data
I will find out why this is happening.
The documentation on the wiki is very helpful and the most concise I
came across. The overlay feature I will spend sometime with to
understand as in the future I may need this. It just left me asking a
lot of questions like if I have LVM and no filesystem on the device
will it work or how should I adopt the setup for that etc. This
exercise has opened my eyes to just how resilient and flexible mdadm
can be.
Thanks again for all the help and advice.
J-P Human
^ permalink raw reply
* Re: [PATCH v2 3/3] md: raid1-10: move raid1/raid10 common code into raid1-10.c
From: Coly Li @ 2017-07-16 5:14 UTC (permalink / raw)
To: Ming Lei
Cc: Shaohua Li, linux-raid, NeilBrown, linux-block, Jens Axboe,
Christoph Hellwig
In-Reply-To: <20170714081444.32645-4-ming.lei@redhat.com>
On 2017/7/14 下午4:14, Ming Lei wrote:
> No function change, just move 'struct resync_pages' and related
> helpers into raid1-10.c
Please give me a hint, I don't find where raid1-10.c is ....
Thanks.
Coly
>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> drivers/md/md.h | 53 -------------------------------------------
> drivers/md/raid1-10.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid1.c | 9 --------
> drivers/md/raid10.c | 9 --------
> 4 files changed, 62 insertions(+), 71 deletions(-)
>
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index 2c780aa8d07f..004a21c214e8 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -729,57 +729,4 @@ static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio
> !bdev_get_queue(bio->bi_bdev)->limits.max_write_zeroes_sectors)
> mddev->queue->limits.max_write_zeroes_sectors = 0;
> }
> -
> -/* Maximum size of each resync request */
> -#define RESYNC_BLOCK_SIZE (64*1024)
> -#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
> -
> -/* for managing resync I/O pages */
> -struct resync_pages {
> - void *raid_bio;
> - struct page *pages[RESYNC_PAGES];
> -};
> -
> -static inline int resync_alloc_pages(struct resync_pages *rp,
> - gfp_t gfp_flags)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++) {
> - rp->pages[i] = alloc_page(gfp_flags);
> - if (!rp->pages[i])
> - goto out_free;
> - }
> -
> - return 0;
> -
> -out_free:
> - while (--i >= 0)
> - put_page(rp->pages[i]);
> - return -ENOMEM;
> -}
> -
> -static inline void resync_free_pages(struct resync_pages *rp)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++)
> - put_page(rp->pages[i]);
> -}
> -
> -static inline void resync_get_all_pages(struct resync_pages *rp)
> -{
> - int i;
> -
> - for (i = 0; i < RESYNC_PAGES; i++)
> - get_page(rp->pages[i]);
> -}
> -
> -static inline struct page *resync_fetch_page(struct resync_pages *rp,
> - unsigned idx)
> -{
> - if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
> - return NULL;
> - return rp->pages[idx];
> -}
> #endif /* _MD_MD_H */
> diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
> index 3adb5b9dc4b4..9f2670b45f31 100644
> --- a/drivers/md/raid1-10.c
> +++ b/drivers/md/raid1-10.c
> @@ -1,3 +1,65 @@
> +/* Maximum size of each resync request */
> +#define RESYNC_BLOCK_SIZE (64*1024)
> +#define RESYNC_PAGES ((RESYNC_BLOCK_SIZE + PAGE_SIZE-1) / PAGE_SIZE)
> +
> +/* for managing resync I/O pages */
> +struct resync_pages {
> + void *raid_bio;
> + struct page *pages[RESYNC_PAGES];
> +};
> +
> +static inline int resync_alloc_pages(struct resync_pages *rp,
> + gfp_t gfp_flags)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++) {
> + rp->pages[i] = alloc_page(gfp_flags);
> + if (!rp->pages[i])
> + goto out_free;
> + }
> +
> + return 0;
> +
> +out_free:
> + while (--i >= 0)
> + put_page(rp->pages[i]);
> + return -ENOMEM;
> +}
> +
> +static inline void resync_free_pages(struct resync_pages *rp)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++)
> + put_page(rp->pages[i]);
> +}
> +
> +static inline void resync_get_all_pages(struct resync_pages *rp)
> +{
> + int i;
> +
> + for (i = 0; i < RESYNC_PAGES; i++)
> + get_page(rp->pages[i]);
> +}
> +
> +static inline struct page *resync_fetch_page(struct resync_pages *rp,
> + unsigned idx)
> +{
> + if (WARN_ON_ONCE(idx >= RESYNC_PAGES))
> + return NULL;
> + return rp->pages[idx];
> +}
> +
> +/*
> + * 'strct resync_pages' stores actual pages used for doing the resync
> + * IO, and it is per-bio, so make .bi_private points to it.
> + */
> +static inline struct resync_pages *get_resync_pages(struct bio *bio)
> +{
> + return bio->bi_private;
> +}
> +
> /* generally called after bio_reset() for reseting bvec */
> static void md_bio_reset_resync_pages(struct bio *bio, struct resync_pages *rp,
> int size)
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index fe86ab18961b..8387eb1540cd 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -84,15 +84,6 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr);
> #include "raid1-10.c"
>
> /*
> - * 'strct resync_pages' stores actual pages used for doing the resync
> - * IO, and it is per-bio, so make .bi_private points to it.
> - */
> -static inline struct resync_pages *get_resync_pages(struct bio *bio)
> -{
> - return bio->bi_private;
> -}
> -
> -/*
> * for resync bio, r1bio pointer can be retrieved from the per-bio
> * 'struct resync_pages'.
> */
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 9952721e1cde..e2617d0f37dc 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -113,15 +113,6 @@ static void end_reshape(struct r10conf *conf);
> #include "raid1-10.c"
>
> /*
> - * 'strct resync_pages' stores actual pages used for doing the resync
> - * IO, and it is per-bio, so make .bi_private points to it.
> - */
> -static inline struct resync_pages *get_resync_pages(struct bio *bio)
> -{
> - return bio->bi_private;
> -}
> -
> -/*
> * for resync bio, r10bio pointer can be retrieved from the per-bio
> * 'struct resync_pages'.
> */
>
^ permalink raw reply
* Re: Advice please re failed Raid6
From: Roman Mamedov @ 2017-07-16 0:58 UTC (permalink / raw)
To: Bogo Mipps; +Cc: linux-raid
In-Reply-To: <9dca5b7a-b60e-0e93-41fd-49d092d8b27b@gmail.com>
Hello,
One thing that I spotted:
Jun 27 16:52:21 keruru kernel: [ 2.922808] md: kicking non-fresh sdb from array!
Jun 27 16:52:21 keruru kernel: [ 3.173236] md/raid:md0: device sdd operational as raid disk 1
Jun 27 16:52:21 keruru kernel: [ 3.173239] md/raid:md0: device sde operational as raid disk 3
Jun 27 16:52:21 keruru kernel: [ 3.173242] md/raid:md0: device sdc operational as raid disk 2
The disk order here was "b d c e"
Jul 6 09:45:52 keruru kernel: [ 1195.580003] md/raid:md0: device sdb operational as raid disk 0
Jul 6 09:45:52 keruru kernel: [ 1195.580012] md/raid:md0: device sde operational as raid disk 3
Jul 6 09:45:52 keruru kernel: [ 1195.580018] md/raid:md0: device sdd operational as raid disk 2
Jul 6 09:45:52 keruru kernel: [ 1195.580025] md/raid:md0: device sdc operational as raid disk 1
But here the order changes to "b c d e"
Unless this is across reboots and your hardware detects disks in a random
order, something weird is going on here.
Also, your array has creation time of "Fri Jun 30 15:42:27 2017", but you
provide no dmesg or logs from that date.
Maybe you forgot to use --assume-clean on one of the attempts (and that's when
you nuked the array entirely)?
Or perhaps --assume-clean also rewrites the creation date, I am not sure.
--
With respect,
Roman
^ permalink raw reply
* Advice please re failed Raid6
From: Bogo Mipps @ 2017-07-15 23:40 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 8466 bytes --]
Hi List
I posted this to the Open Media Vault (my NAS O/S) list a few days ago
without response. A definitive answer would be appreciated.
Have been running a 4 disk Raid 6 setup for over two years without any
issues, until suddenly on June 27 disks on my OMV NAS became 100% full,
including the NFS mounted volumes in the raid set. There'd been major
disk activity overnight, but foolishly didn't investigate.
Rsnapshot normally backs up two desktop machines onto the raid setup:
the next morning found that one of the backup directories was not on
raid but suddenly was on the root directory of the OMV machine, and
being a total backup (several Gb) this accounted for the 100% reading
for the OMV/NAS machine.
The logs indicated that mdstat had discovered "dirty degraded array"
presumably due to faulty sdb, so had withdrawn that disk, and then
couldn't run the raid set (logs below show)
Bought a new disk and installed on July 4, and raid rebuilt overnight
(see July 5 Rebuild finished below)
Since then I've been unable to mount or access any data. Have followed
instructions as per Linux Raid Wiki's "Recovering a failed software
RAID" & "RAID Recovery" , but still no success. I've attached the
results of their suggestions in the attached log file
"linux_raid_wiki_logs.txt".
Any help appreciated - even if it's just to tell me my RAID sets are hosed!
P.S. This line looks ominous? <md0: detected capacity change from
4000528203776 to 0> !!!
===============Jun 27 16:52:21 keruru kernel: [ 2.912440] md: md0 stopped.
Jun 27 16:52:21 keruru kernel: [ 2.922315] md: bind<sdb>
Jun 27 16:52:21 keruru kernel: [ 2.922508] md: bind<sdc>
Jun 27 16:52:21 keruru kernel: [ 2.922643] md: bind<sde>
Jun 27 16:52:21 keruru kernel: [ 2.922777] md: bind<sdd>
Jun 27 16:52:21 keruru kernel: [ 2.922808] md: kicking non-fresh sdb
from array!
Jun 27 16:52:21 keruru kernel: [ 2.922820] md: unbind<sdb>
Jun 27 16:52:21 keruru kernel: [ 2.927107] md: export_rdev(sdb)
Jun 27 16:52:21 keruru kernel: [ 2.994973] raid6: sse2x1 588 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.062926] raid6: sse2x2 1395 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.130841] raid6: sse2x4 2397 MB/s
Jun 27 16:52:21 keruru kernel: [ 3.130844] raid6: using algorithm sse2x4
(2397 MB/s)
Jun 27 16:52:21 keruru kernel: [ 3.130846] raid6: using ssse3x2 recovery
algorithm
Jun 27 16:52:21 keruru kernel: [ 3.130866] Switched to clocksource tsc
Jun 27 16:52:21 keruru kernel: [ 3.131227] xor: automatically using best
checksumming function:
Jun 27 16:52:21 keruru kernel: [ 3.170797] avx : 6164.000 MB/sec
Jun 27 16:52:21 keruru kernel: [ 3.171121] async_tx: api initialized (async)
Jun 27 16:52:21 keruru kernel: [ 3.172809] md: raid6 personality
registered for level 6
Jun 27 16:52:21 keruru kernel: [ 3.172812] md: raid5 personality
registered for level 5
Jun 27 16:52:21 keruru kernel: [ 3.172815] md: raid4 personality
registered for level 4
Jun 27 16:52:21 keruru kernel: [ 3.173218] md/raid:md0: not clean --
starting background reconstruction
Jun 27 16:52:21 keruru kernel: [ 3.173236] md/raid:md0: device sdd
operational as raid disk 1
Jun 27 16:52:21 keruru kernel: [ 3.173239] md/raid:md0: device sde
operational as raid disk 3
Jun 27 16:52:21 keruru kernel: [ 3.173242] md/raid:md0: device sdc
operational as raid disk 2
Jun 27 16:52:21 keruru kernel: [ 3.173706] md/raid:md0: allocated 0kB
Jun 27 16:52:21 keruru kernel: [ 3.173745] md/raid:md0: cannot start
dirty degraded array.
Jun 27 16:52:21 keruru kernel: [ 3.173811] RAID conf printout:
Jun 27 16:52:21 keruru kernel: [ 3.173814] --- level:6 rd:4 wd:3
Jun 27 16:52:21 keruru kernel: [ 3.173816] disk 1, o:1, dev:sdd
Jun 27 16:52:21 keruru kernel: [ 3.173818] disk 2, o:1, dev:sdc
Jun 27 16:52:21 keruru kernel: [ 3.173820] disk 3, o:1, dev:sde
Jun 27 16:52:21 keruru kernel: [ 3.174025] md/raid:md0: failed to run
raid set.
Jun 27 16:52:21 keruru kernel: [ 3.174071] md: pers->run() failed ...
===============
New disk added - sdb
===============
Jul 5 21:06:18 keruru mdadm[2497]: RebuildFinished event detected on md
device /dev/md0, component device mismatches found: 1847058224 (on raid
level 6)
Jul 6 09:45:52 keruru kernel: [ 1195.390879] raid6: sse2x1 249 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.458735] raid6: sse2x2 476 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.526632] raid6: sse2x4 839 MB/s
Jul 6 09:45:52 keruru kernel: [ 1195.526638] raid6: using algorithm
sse2x4 (839 MB/s)
Jul 6 09:45:52 keruru kernel: [ 1195.526644] raid6: using ssse3x2
recovery algorithm
Jul 6 09:45:52 keruru kernel: [ 1195.578970] md: raid6 personality
registered for level 6
Jul 6 09:45:52 keruru kernel: [ 1195.578980] md: raid5 personality
registered for level 5
Jul 6 09:45:52 keruru kernel: [ 1195.578985] md: raid4 personality
registered for level 4
Jul 6 09:45:52 keruru kernel: [ 1195.580003] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 09:45:52 keruru kernel: [ 1195.580012] md/raid:md0: device sde
operational as raid disk 3
Jul 6 09:45:52 keruru kernel: [ 1195.580018] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 09:45:52 keruru kernel: [ 1195.580025] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 09:45:52 keruru kernel: [ 1195.581091] md/raid:md0: allocated 0kB
Jul 6 09:45:52 keruru kernel: [ 1195.581180] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.321957] raid6: sse2x4 1130 MB/s
Jul 6 09:52:30 keruru kernel: [ 4.321964] raid6: using algorithm sse2x4
(1130 MB/s)
Jul 6 09:52:30 keruru kernel: [ 4.321967] raid6: using ssse3x2 recovery
algorithm
Jul 6 09:52:30 keruru kernel: [ 4.368478] md: raid6 personality
registered for level 6
Jul 6 09:52:30 keruru kernel: [ 4.368486] md: raid5 personality
registered for level 5
Jul 6 09:52:30 keruru kernel: [ 4.368490] md: raid4 personality
registered for level 4
Jul 6 09:52:30 keruru kernel: [ 4.369179] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 09:52:30 keruru kernel: [ 4.369185] md/raid:md0: device sde
operational as raid disk 3
Jul 6 09:52:30 keruru kernel: [ 4.369189] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 09:52:30 keruru kernel: [ 4.369194] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 09:52:30 keruru kernel: [ 4.369974] md/raid:md0: allocated 0kB
Jul 6 09:52:30 keruru kernel: [ 4.372062] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 12:56:15 keruru kernel: [ 4.442184] raid6: sse2x1 739 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.510060] raid6: sse2x2 1480 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.577985] raid6: sse2x4 1605 MB/s
Jul 6 12:56:15 keruru kernel: [ 4.577993] raid6: using algorithm sse2x4
(1605 MB/s)
Jul 6 12:56:15 keruru kernel: [ 4.577997] raid6: using ssse3x2 recovery
algorithm
Jul 6 12:56:15 keruru kernel: [ 4.622570] md: raid6 personality
registered for level 6
Jul 6 12:56:15 keruru kernel: [ 4.622577] md: raid5 personality
registered for level 5
Jul 6 12:56:15 keruru kernel: [ 4.622580] md: raid4 personality
registered for level 4
Jul 6 12:56:15 keruru kernel: [ 4.623261] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 12:56:15 keruru kernel: [ 4.623266] md/raid:md0: device sde
operational as raid disk 3
Jul 6 12:56:15 keruru kernel: [ 4.623269] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 12:56:15 keruru kernel: [ 4.623273] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 12:56:15 keruru kernel: [ 4.624064] md/raid:md0: allocated 0kB
Jul 6 12:56:15 keruru kernel: [ 4.624131] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
Jul 6 16:54:43 keruru kernel: [14401.858429] md/raid:md0: device sdb
operational as raid disk 0
Jul 6 16:54:43 keruru kernel: [14401.858442] md/raid:md0: device sde
operational as raid disk 3
Jul 6 16:54:43 keruru kernel: [14401.858449] md/raid:md0: device sdd
operational as raid disk 2
Jul 6 16:54:43 keruru kernel: [14401.858455] md/raid:md0: device sdc
operational as raid disk 1
Jul 6 16:54:43 keruru kernel: [14401.859915] md/raid:md0: allocated 0kB
Jul 6 16:54:43 keruru kernel: [14401.860000] md/raid:md0: raid level 6
active with 4 out of 4 devices, algorithm 2
[-- Attachment #2: linux_raid_wiki_logs.txt --]
[-- Type: text/plain, Size: 7819 bytes --]
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] >> raid.status
root@keruru:/var/log# cat raid.status
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 79e4933f:dfe5923f:5ba03ae7:3efe38eb
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 9ff2b025 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f1e1a946:711886a6:2604780f:8eba4a2d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 784b0046 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cf3bc8a7:9feed87d:945d8e77:08f7f32d
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 197bc63c - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b1e6af5d:e5848ebe:63727445:2ab99719
Name : keruru:0 (local to host keruru)
Creation Time : Fri Jun 30 15:42:27 2017
Raid Level : raid6
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 3906765824 (3725.78 GiB 4000.53 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b3323d81:279b7c7b:a0c534ed:46d0e6fc
Update Time : Wed Jul 5 21:06:18 2017
Checksum : 352daaf4 - correct
Events : 119
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)
================
root@keruru:/var/log# mdadm --examine /dev/sd[bedc] | egrep 'Event|/dev/sd'
/dev/sdb:
Events : 119
/dev/sdc:
Events : 119
/dev/sdd:
Events : 119
/dev/sde:
Events : 119
===============
root@keruru:/var/log# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@keruru:/var/log# mdadm --assemble --force /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/md0 has been started with 4 drives.
root@keruru:/var/log# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sdb[4] sde[3] sdd[2] sdc[1]
3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
===============
root@keruru:/var/log# grep Role raid.status
Device Role : Active device 0
Device Role : Active device 1
Device Role : Active device 2
Device Role : Active device 3
===============
root@keruru:/var/log# grep Used raid.status
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6 --raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? n
mdadm: create aborted.
===============
root@keruru:/var/log# mdadm --create --assume-clean --level=6 --raid-devices=4 --size=1953382912 /dev/md0 /dev/sdb /dev/sde /dev/sdd /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: /dev/sde appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdd appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
mdadm: /dev/sdc appears to be part of a raid array:
level=raid6 devices=4 ctime=Fri Jun 30 15:42:27 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
===============
root@keruru:/# mount -t ext4 /dev/md0 /mnt/md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
===============
root@keruru:/# dmesg | tail
[448318.800806] --- level:6 rd:4 wd:4
[448318.800812] disk 0, o:1, dev:sdb
[448318.800817] disk 1, o:1, dev:sde
[448318.800822] disk 2, o:1, dev:sdd
[448318.800827] disk 3, o:1, dev:sdc
[448318.800951] md0: detected capacity change from 0 to 4000528203776
[448318.809375] md0: unknown partition table
[448358.704189] EXT4-fs (md0): Unrecognized mount option "\x08" or missing value
[448358.706680] EXT4-fs (md0): failed to parse options in superblock: \x08
[448358.706690] EXT4-fs (md0): Number of reserved GDT blocks insanely large: 9216
===============
Jul 11 17:23:15 keruru kernel: [447719.812775] md: export_rdev(sdd)
Jul 11 17:23:19 keruru kernel: [447724.396327] md: md0 stopped.
Jul 11 17:23:19 keruru kernel: [447724.400278] md: bind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.001687] md0: detected capacity change from 4000528203776 to 0
Jul 11 17:32:29 keruru kernel: [448273.001714] md: md0 stopped.
Jul 11 17:32:29 keruru kernel: [448273.001729] md: unbind<sdc>
Jul 11 17:32:29 keruru kernel: [448273.022972] md: export_rdev(sdc)
Jul 11 17:32:29 keruru kernel: [448273.023143] md: unbind<sde>
Jul 11 17:32:29 keruru kernel: [448273.054889] md: export_rdev(sde)
Jul 11 17:32:29 keruru kernel: [448273.055035] md: unbind<sdd>
Jul 11 17:32:29 keruru kernel: [448273.086870] md: export_rdev(sdd)
Jul 11 17:33:14 keruru kernel: [448318.800827] disk 3, o:1, dev:sdc
===============
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Anthony Youngman @ 2017-07-15 23:03 UTC (permalink / raw)
To: Jean-Pierre Human, Andreas Klauer; +Cc: linux-raid
In-Reply-To: <CAH_1ubxjJzNzhV7JCjspPruUv7uJxTwVeYrjOCQv4wUMcfZBrQ@mail.gmail.com>
On 15/07/17 19:05, Jean-Pierre Human wrote:
> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
> <Andreas.Klauer@metamorpher.de> wrote:
> Hi Andreas
>
> Thanks for the prompt response,
>
> Yes I did not use a partition table. I have always in the past used
> partitions but recently noticed a trend to use the device, wont do
> that again.
>
> I am really not sure what wrote those partitions there. I only see
> free space in them if I check with fdisk.
>
> I am sure it was metadata version 1.2
>
> I will attempt to recover using the overlay, will post my results.
Good call. The whole point of overlays is to enable you to write-protect
the physical disks. That means (of course) that you can't make matters
worse. That's always the worry - that an attempt at a fix will damage
the content of the disks and make a recovery harder/impossible.
>
> Thanks again for your help.
>
A little more (hopefully) help - on the wiki the last entry in the "When
things go wrogn" section is about recovering a trashed array. As it
says, it's a work in progress, but it gives you various hints about how
to search the disk for clues as to where your data is, and hence what
all those things like offsets are.
Have you ever added disks to your array? Rebuilt and changed its
configuration? This can mess about with the superblock values changing
them from the defaults. This is information the list will want.
If you have trouble getting overlays to work, search for clues like this
on your list and get back here for help. ON NO ACCOUNT do anything that
will actually write to your disks unless you are absolutely confident
that you've found the correct magic incantation.
There are plenty of people here who will help you get your data back -
and they've got a good track record! Unfortunately, I'm "learning by
documenting" so I'm quite happy to give you hints and guidance, but I
can't say for certain what is likely to work.
Cheers,
Wol
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Jean-Pierre Human @ 2017-07-15 18:05 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170715172643.GA24000@metamorpher.de>
On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Sat, Jul 15, 2017 at 07:09:08PM +0200, Jean-Pierre Human wrote:
>> The array was setup with the below commands:
>> #mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
>> /dev/sdj /dev/sdk /dev/sdl /dev/sdi
>
> So you didn't use a partition table?
>
>> root@store02:~# mdadm --examine /dev/sd[iklj]
>> /dev/sdi:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdj:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdk:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdl:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>
> And then "something" created one. Is that partition table empty?
> Or did it also create and format partitions, that would be worse.
>
> GPT partition table overwrites a bunch of sectors at both start and end.
> So that's where you'll find corruption, depending which mdadm metadata
> version you were using (which can also be located either start or end).
>
> To recover, you'll have to determine the correct RAID level / layout /
> order / data offset. It's best to do this with overlays
>
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> and only write to the real disks once you've found the setting that works.
>
> In the future, consider always using a partition table. Linux doesn't care *
> but the partition table is the most standard way to declare a disk is
> already in use and for what. Without a partition table, any software not
> md-raid aware will see your drive as free, unused, and might format it.
>
> (*) it will happily run anything you like on bare disks
> but it won't do anything to protect you, either
>
> Regards
> Andreas Klauer
Hi Andreas
Thanks for the prompt response,
Yes I did not use a partition table. I have always in the past used
partitions but recently noticed a trend to use the device, wont do
that again.
I am really not sure what wrote those partitions there. I only see
free space in them if I check with fdisk.
I am sure it was metadata version 1.2
I will attempt to recover using the overlay, will post my results.
Thanks again for your help.
Regards
J-P Human
^ permalink raw reply
* Re: Superblocks lost on all disks in array.
From: Andreas Klauer @ 2017-07-15 17:26 UTC (permalink / raw)
To: Jean-Pierre Human; +Cc: linux-raid
In-Reply-To: <CAH_1ubwzXd9C8OBheZv+Z5oFTVt2rG+RATHrhKy6uiVv84y_SA@mail.gmail.com>
On Sat, Jul 15, 2017 at 07:09:08PM +0200, Jean-Pierre Human wrote:
> The array was setup with the below commands:
> #mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
> /dev/sdj /dev/sdk /dev/sdl /dev/sdi
So you didn't use a partition table?
> root@store02:~# mdadm --examine /dev/sd[iklj]
> /dev/sdi:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> /dev/sdj:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> /dev/sdk:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> /dev/sdl:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
And then "something" created one. Is that partition table empty?
Or did it also create and format partitions, that would be worse.
GPT partition table overwrites a bunch of sectors at both start and end.
So that's where you'll find corruption, depending which mdadm metadata
version you were using (which can also be located either start or end).
To recover, you'll have to determine the correct RAID level / layout /
order / data offset. It's best to do this with overlays
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
and only write to the real disks once you've found the setting that works.
In the future, consider always using a partition table. Linux doesn't care *
but the partition table is the most standard way to declare a disk is
already in use and for what. Without a partition table, any software not
md-raid aware will see your drive as free, unused, and might format it.
(*) it will happily run anything you like on bare disks
but it won't do anything to protect you, either
Regards
Andreas Klauer
^ permalink raw reply
* Superblocks lost on all disks in array.
From: Jean-Pierre Human @ 2017-07-15 17:09 UTC (permalink / raw)
To: linux-raid
Hi
I have a 4 disk Raid10 array in an Ubuntu16.04.2 Server (/dev/md2)
which seems to have lost the superblocks on all 4 disks.
The server has two other arrays in it a 3 disk Raid5 (/dev/md1) and
another 4 disk Raid10 (/dev/md0), these arrays are fine. This is not
due to mechanical failure or disk issues. The array did not come up
after a reboot of the server.
A bit more info, the failed array /dev/md2 was 100% assigned to a
logical volume which was shared via ISCSI to a Microsoft server. I
mention this as I have a feeling this could somehow be the reason the
superblocks disappeared? These disks were part of a ZFS pool before
this, I see lsdrv picks this up...
As there are no superblocks I cannot assemble the array. I feel my
only option is to create --assume-clean the array to get new
superblocks written. The server is fairly new and software versions
etc have not changed. I have the original commands I used to create
the array which have the disk order in them.
What I am looking for is any advice on how I should proceed.
Presuming I must --create, Should I use the "file overlay" method to
allow me to roll back? I do not really understand how this works and
if it is relevant for my situation.
Would anyone know what could have caused this?
System info below:
Ubuntu 16.04.2 LTS
The array was setup with the below commands:
#mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
/dev/sdj /dev/sdk /dev/sdl /dev/sdi
#update-initramfs -u
smartctl --xall produced too much output please let me know if you
need this. I see no errors or issues in this output.
root@store02:~# mdadm --examine /dev/sd[iklj]
/dev/sdi:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdj:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdk:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdl:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
root@store02:~# mdadm -Q /dev/sd[iklj]
/dev/sdi: is not an md array
/dev/sdj: is not an md array
/dev/sdk: is not an md array
/dev/sdl: is not an md array
root@store02:~# cat /proc/mdstat
Personalities : [raid10] [raid6] [raid5] [raid4] [linear] [multipath]
[raid0] [raid1]
md0 : active raid10 sdd[3] sdb[2] sde[1] sdc[0]
7813774336 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 2/59 pages [8KB], 65536KB chunk
md1 : active raid5 sdf[1] sdg[3] sda[0]
3906766848 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
unused devices: <none>
root@store02:~/lsdrv# ./lsdrv
PCI [mpt3sas] 02:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
├scsi 0:0:0:0 ATA WDC WD20EFRX-68E {WD-WCC4M2702612}
│└sda 1.82t [8:0] MD raid5 (0/3) (w/ sdf,sdg) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│ ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
├scsi 0:0:1:0 ATA WDC WD4000F9YZ-0 {WD-WMC5D0D0W9D9}
│└sdb 3.64t [8:16] MD raid10,near2 (2/4) (w/ sdc,sdd,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│ │ ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
│ └Mounted as /dev/md0 @ /mnt/md0
├scsi 0:0:2:0 ATA WDC WD4000F9YZ-0 {WD-WMC1F0D9LMDJ}
│└sdc 3.64t [8:32] MD raid10,near2 (0/4) (w/ sdb,sdd,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│ ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:3:0 ATA WDC WD4000F9YZ-0 {WD-WMC5D0D7WP6N}
│└sdd 3.64t [8:48] MD raid10,near2 (3/4) (w/ sdb,sdc,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│ ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:4:0 ATA WDC WD4000F9YZ-0 {WD-WMC5D0D1R7E0}
│└sde 3.64t [8:64] MD raid10,near2 (1/4) (w/ sdb,sdc,sdd) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│ ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:5:0 ATA WDC WD20EFRX-68A {WD-WCC1T0684483}
│└sdf 1.82t [8:80] MD raid5 (1/3) (w/ sda,sdg) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│ ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
├scsi 0:0:6:0 ATA WDC WD20EFRX-68A {WD-WCC1T0681728}
│└sdg 1.82t [8:96] MD raid5 (2/3) (w/ sda,sdf) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│ ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
└scsi 0:x:x:x [Empty]
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
├scsi 1:0:0:0 ATA Maximus-16GB {152403701000}
│└sdh 14.75g [8:112] zfs_member
│ ├sdh1 14.34g [8:113] ext4 {5b7ca57a-9540-47fa-9146-c413d649c5ba}
│ │└Mounted as /dev/sdh1 @ /
│ └sdh2 416.00m [8:114] swap {bfe8ede9-1ca4-4519-9916-3a894708f6a2}
├scsi 2:x:x:x [Empty]
├scsi 3:0:0:0 ATA ST6000NM0115-1YZ {ZAD0RGY5}
│└sdi 5.46t [8:128] Partitioned (gpt)
├scsi 4:0:0:0 ATA ST6000NM0115-1YZ {ZAD0RGCH}
│└sdj 5.46t [8:144] zfs_member
├scsi 5:0:0:0 ATA ST6000NM0115-1YZ {ZAD0RHN3}
│└sdk 5.46t [8:160] zfs_member
└scsi 6:0:0:0 ATA ST6000NM0115-1YZ {ZAD0RGQ6}
└sdl 5.46t [8:176] Partitioned (gpt)
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
└loop7 0.00k [7:7] Empty/Unknown
Any help or advice is appreciated.
Regards
J-P Human
^ permalink raw reply
* Re: Why can't I re-add my drive after partition shrink?
From: Ram Ramesh @ 2017-07-15 0:17 UTC (permalink / raw)
To: NeilBrown, Linux Raid
In-Reply-To: <87tw2f2425.fsf@notabene.neil.brown.name>
On 07/13/2017 08:35 PM, NeilBrown wrote:
> <snip
> Please report output of "mdadm --examine" on both a device that is active
> in the array, and the device that you are trying to add.
> Also "mdadm --examine-bitmap" of a device that is active in the array.
>
> NeilBrown
>
>> I researched as much as I could on the net and came up with nothing
>> except some one saying that mdadm keeps something at the end of the disk
>> regardless of what it says about "Used Dev Size." Is it possible to move
>> this info so that I could re-add?
>>
>> Ramesh
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Neil,
Since the problem, I did not want to leave my md in degraded state.
So, I added my drive back and paid the penalty for rebuilding. I have
other disks that need to be resized and *can get you want*. Please let
me know if that is what you meant. If you wanted the current info after
successfully rebuilding the array after a regular add, it is below.
First the device that is currently in after a regular add.
> zym [rramesh] 497 > sudo mdadm --examine /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Name : zym:0 (local to host zym)
> Creation Time : Mon Apr 22 00:08:12 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 6442188800 (3071.88 GiB 3298.40 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 702ca77d:564d69ff:e45d9679:64c314fa
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Fri Jul 14 18:36:37 2017
> Checksum : c5502356 - correct
> Events : 297182
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 4
> Array State : AAAAAA ('A' == active, '.' == missing)
Member disk that has been in the array before /dev/sdb1 resize
> zym [rramesh] 498 > sudo mdadm --examine /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Name : zym:0 (local to host zym)
> Creation Time : Mon Apr 22 00:08:12 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
> Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
> Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 7e035b56:d1e1882b:e78a08ad:3ba50667
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Fri Jul 14 18:36:37 2017
> Checksum : a5288a4c - correct
> Events : 297182
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 2
> Array State : AAAAAA ('A' == active, '.' == missing)
Now examine bit map on /dev/sdb1 (device that got in after a regular add)
> zym [rramesh] 499 > sudo mdadm --examine-bitmap /dev/sdb1
> Filename : /dev/sdb1
> Magic : 6d746962
> Version : 4
> UUID : 0e9f76b5:4a89171a:a930bccd:78749144
> Events : 297182
> Events Cleared : 297182
> State : OK
> Chunksize : 64 MB
> Daemon : 5s flush period
> Write Mode : Normal
> Sync Size : 3087007744 (2944.00 GiB 3161.10 GB)
> Bitmap : 47104 bits (chunks), 0 dirty (0.0%)
Ramesh
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox