* Re: Recovery after accidental raid5 superblock rewrite
From: Andreas Klauer @ 2017-06-05 23:56 UTC (permalink / raw)
To: Paul Tonelli; +Cc: linux-raid
In-Reply-To: <dfa4dd0d-c68a-4e1b-aac7-649f7f1a72f6@tonel.li>
On Tue, Jun 06, 2017 at 01:24:41AM +0200, Paul Tonelli wrote:
> mdadm --create /dev/md0 --level=5 --assume-clean --raid-devices=3 missing /dev/mapper/sdd /dev/mapper/sdb
You did not specify the --data-offset here?
Check mdadm --examine to make sure which offset it's using.
> xxd -u /dev/mapper/sdc | grep -C 3 'LABELONE'
> >7f001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >7f001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >7f00200: 4C41 4245 4C4F 4E45 0100 0000 0000 0000 LABELONE........
If correct this should appear at the start of /dev/md0 (even w/o adding sdc).
LABELONE should appear on the first drive so this should not be wrong,
however the sdd sdb could still have switched order, and of course
the chunk size could be different (although unlikely according to your log).
> I believe you are right, the issue is still the raid: I have tried
> photorec and most files I have opened look like they have been
> truncated.
Well, files could be fragmented, there's a certain sweet spot
(like - megapixel JPEGs of few megs size) where it's sufficiently
unlikely to be a problem.
I don't know what files you have, if it's movies it would be okay
too if the first few megs of the file were playable.
> - apart from the raid superblock, the disks I use (sdd and sdb) have not
> been erased (as sdc is rebuilt from xor)
It only rebuilds starting from offset. So it should not have covered that
offset if you did not specify it. Check it's not there before you --add.
If it's there then this is after all not the drive you overwrite with dd?
I am confused now.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Paul Tonelli @ 2017-06-05 23:24 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170605092454.GA3651@metamorpher.de>
Hi again Andreas (and anybody else reading)
On 06/05/2017 11:24 AM, Andreas Klauer wrote:
> On Mon, Jun 05, 2017 at 12:58:03AM +0200, Paul Tonelli wrote:
>> I did find this one at that exact offset 26096 = (133169664 - 512)/512
> 26096->260096? Okay.
Yes, sorry this was a typo, the superblock was exactly where you
expected it, I did the following procedure:
```
export $DEVICES="/dev/sdc /dev/sdd /dev/sdb"
/root/create_overlay.sh
mdadm --create /dev/md0 --level=5 --assume-clean --raid-devices=3
missing /dev/mapper/sdd /dev/mapper/sdb
mdadm --add /dev/md0 /dev/mapper/sdc
#sleep 30s
mdadm --stop /dev/md0
xxd -u /dev/mapper/sdc | grep -C 3 'LABELONE'
>7f001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>7f001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>7f00200: 4C41 4245 4C4F 4E45 0100 0000 0000 0000 LABELONE........
>7f00210: B5E5 02EA 2000 0000 4C56 4D32 2030 3031 .... ...LVM2 001
>7f00220: 5132 5A33 3244 6979 506A 3951 5970 7558 Q2Z32DiyPj9QYpuX
>7f00230: 4243 7130 3265 7338 514B 3665 7176 3464 BCq02es8QK6eqv4d
```
I have let it run for a longer time (10 min) and it does not find
another LABELONE after that. (I also tried to find LABELONE on the other
disks without success).
>> Now, after doing a pvcreate and a vgcfgrestore ...
> Should not be necessary if the data on two drives was okay?
> You did leave the dd-overwritten drive out as missing, right?
> Do you have the correct disk order and chunk size and offset?
I believe you are right, the issue is still the raid: I have tried
photorec and most files I have opened look like they have been
truncated. I have checked again the commands used to build the raid, (I
had initially extracted them from bash_history, but I also have a record
with timestamps in /var/log/auth.log because of sudo).
and all the commands match ( I have put all the commands used on the
raid at the end of the mail )
> You have to be 140% sure the RAID itself is running correctly,
> otherwise all other steps are bound to fail.
>
> If you run photorec on the RAID and it manages to recover
> a file that is larger than number of drives * chunk size
> and intact, you can have some confidence the RAID is okay
> in terms of disk order and chunk size - the offset may
> still be off by multiple of stripe alignment but if the
> offset is correct too, file -s /dev/md0 should say LVM PV
> in your case.
Photorec does not: I get only small files when I use photorec. The only
"big" files it recovers are tar (and I believe that is because it does
not check integrity of the files). I have left it running for ~10G of
files were recovered.
From what I understand, finding the "LABELONE" at the correct position
shows:
- apart from the raid superblock, the disks I use (sdd and sdb) have not
been erased (as sdc is rebuilt from xor)
- the offset to build the raid is correct (I find all ext4 backup
superblock when doing the testdisk / checking with e2fsck -b XXX ), with
the issue of the 640 ext4 chunk offset.
- the first disk of the array is sdc (as the "LABELONE" can only be
found on this disk)
so I may need to check the other parame
/dev/md0 should say LVM PV
I believe my best option is to do a script to explore several parameters
using the overlay and see if one finds the correct data. I would use as
parameters:
- disk order
- raid chunk size
Would you test any other parameters ?
In the things I have tried, I used the offset option on an losetup to
align the backup superblocks from the logical volume, e2fsck then runs
directly, but it then refuse to continue because of the wrong volume
size, so I do not think this is the right solution, and the 640 ext4
offset cannot easily be added to lvm, I agree with your previous comment
that it is still on the raid level there are issues.
Thanks (again) for reading me to this point, I will still happily take
any advice, and update once I have written and run my script.
Dump of raid building commands
=====================
part 1
---------
sudo commands which created the array:
```
May 16 17:38:47 redacted sshd[10978]: pam_unix(sshd:session): session
opened for user skoos by (uid=0)
May 16 17:38:47 redacted systemd-logind[1071]: New session 754 of user
skoos.
May 16 17:41:29 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/usr/bin/apt-get install mdadm
May 16 17:41:29 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:41:45 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:41:48 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/mdadm --create --verbose --force
--assume-clean /dev/md0 -
-level=5 --raid-devices=2 /dev/sdb /dev/sdc
May 16 17:41:48 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:42:08 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:42:10 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/mdadm --create --verbose --force
--assume-clean /dev/md0 -
-level=5 --raid-devices=2 /dev/sdb /dev/sdc
May 16 17:42:10 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:42:13 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:42:43 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/pvcreate /dev/md0
May 16 17:42:43 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:42:44 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:42:49 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/vgdisplay
May 16 17:42:49 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:42:49 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:43:02 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/vgcreate vg0 /dev/md0
May 16 17:43:02 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:43:02 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:43:23 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/lvcreate -L 2.6T -n data /dev/vg0
May 16 17:43:23 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:43:23 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:43:32 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/sbin/mkfs.ext4 /dev/vg0/data
May 16 17:43:32 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:43:48 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:45:34 redacted sudo: user : TTY=pts/83 ; PWD=/srv/data ;
USER=root ; COMMAND=/bin/mount /dev/vg0/data /mnt/data
May 16 17:45:34 redacted sudo: pam_unix(sudo:session): session opened
for user root by user(uid=0)
May 16 17:45:35 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 16 17:46:08 redacted sudo: user : TTY=pts/103 ; PWD=/mnt/data ;
USER=root ; COMMAND=/usr/bin/rsync -avH /srv/data/ /mnt/data
```
part 2
---------
command to rebuild the array after reboot (the array was not detected
any longer after reboot, but once the following command was run, it
mounted fine):
```
May 23 13:23:55 redacted sudo: user : TTY=pts/1 ; PWD=/home/user ;
USER=root ; COMMAND=/sbin/mdadm --create --verbose /dev/md0 --level=5
--raid-devices=2 /dev/sdb /dev/sdc
```
commands used to grow with third disk
```
May 24 21:04:10 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --add /dev/md0 /dev/sdd
May 24 21:04:10 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:10 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:04:15 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm /dev/md0
May 24 21:04:15 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:15 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:04:20 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm /dev/md0 --detail
May 24 21:04:20 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:20 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:04:26 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --detail /dev/md0
May 24 21:04:26 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:26 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:04:46 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --gros /dev/md0 /dev/sdd
May 24 21:04:46 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:46 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:04:49 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --grow /dev/md0 /dev/sdd
May 24 21:04:49 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:04:49 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:05:16 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --grow --raid-devices=3 /dev/md0
May 24 21:05:16 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:05:17 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:05:19 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --detail /dev/md0
May 24 21:05:19 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 24 21:05:19 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 24 21:05:41 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --detail /dev/md0
```
part 3
---------
commands used to resize after adding the third disk
```
May 29 09:45:20 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/mdadm --detail /dev/md0
May 29 09:45:20 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:45:21 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:45:37 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/lvextend -L +500G /dev/vg0/data
May 29 09:45:37 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:45:38 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:45:43 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/resize2fs /dev/vg0/data
May 29 09:45:43 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:45:49 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:46:02 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/lvextend -L +500G /dev/vg0/data
May 29 09:46:02 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:46:03 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:46:05 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/resize2fs /dev/vg0/data
May 29 09:46:05 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:46:12 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:46:14 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/lvextend -L +500G /dev/vg0/data
May 29 09:46:14 redacted sudo: pam_unix(sudo:session): session opened
for user root by (uid=0)
May 29 09:46:15 redacted sudo: pam_unix(sudo:session): session closed
for user root
May 29 09:46:16 redacted sudo: root : TTY=pts/30 ; PWD=/root ;
USER=root ; COMMAND=/sbin/resize2fs /dev/vg0/data
```
^ permalink raw reply
* Re: internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: David Brown @ 2017-06-05 16:16 UTC (permalink / raw)
To: CoolCold; +Cc: Linux RAID
In-Reply-To: <CAGqmV7oyh4uTn6c4L5mAtA5tfw5c4jRJebJMP-Ojx07KoSMazQ@mail.gmail.com>
On 05/06/17 14:30, CoolCold wrote:
> Hello!
> Thanks for your proposals.
>
> From my POV, having ~ 100 iops per drive is completely okay, as you
> see from iostat/fio output it's not even near the case.
> Hope this clarifies.
No, that tells me absolutely /nothing/.
Please re-read what I wrote, and think about it. When you can tell us
what you plan to /do/ with your disk array, it will be possible for
people to give you /real/ help and useful advice. As it is, the best
you can get is suggestions for the colour of your bike shed.
mvh.,
David
>
> On Mon, Jun 5, 2017 at 5:55 PM, David Brown <david.brown@hesbynett.no> wrote:
>>
>>
>> On 05/06/17 12:06, CoolCold wrote:
>>>
>>> Hello!
>>> Keep testing the new box and while having not the best sync speed,
>>> it's not the worst thing I found.
>>>
>>> Doing FIO testing, for RAID10 over 20 10k RPM drives, I have very bad
>>> performance, like _45_ iops only.
>>>
>>
>> <snip>
>>
>>>
>>> Any advises would be very helpful.
>>>
>>
>> The best advice I can give you is to take a step back, and try to be clear
>> what problem you are trying to solve here. What is this system supposed to
>> do? What are your requirements? What are your use-cases?
>>
>> Attempting to optimise the sync speed (or IOPS) of a 20 drive RAID10 set is
>> a totally pointless exercise in itself. At best, it would be a torture test
>> for md raid. There is no system in the world where the specifications are
>> "make the fastest 20 drive RAID10 setup" - at least, not from a sane IT
>> management.
>>
>> When you have a clear picture of what you actually /need/, what you /want/,
>> what you /have/, and how you want to use it all - /then/, and only then, is
>> it time to look at possible RAID setups, filesystem arrangements,
>> alternative hardware, etc. Then you ask for advice on this list (or
>> elsewhere - but this list is a good starting point), learning about the pros
>> and cons of a variety of possible arrangements.
>>
>> Once you have collected these ideas, with different balances in performance,
>> safety, space efficiency, features, cost, etc., you can then test them out
>> and look at benchmarks to see if they will work in practice.
>
>
>
^ permalink raw reply
* Re: [PATCH v2] Get failed disk count from array state
From: Jes Sorensen @ 2017-06-05 15:12 UTC (permalink / raw)
To: Tomasz Majchrzak, linux-raid
In-Reply-To: <1496227617-25207-1-git-send-email-tomasz.majchrzak@intel.com>
On 05/31/2017 06:46 AM, Tomasz Majchrzak wrote:
> Recent commit has changed the way failed disks are counted. It breaks
> recovery for external metadata arrays as failed disks are not part of
> the array and have no corresponding entries is sysfs (they are only
> reported for containers) so degraded arrays show no failed disks.
>
> Recent commit overwrites GET_DEGRADED result prior to GET_STATE and it
> is not set again if GET_STATE has not been requested. As GET_STATE
> provides the same information as GET_DEGRADED, the latter is not needed
> anymore. Remove GET_DEGRADED option and replace it with GET_STATE
> option.
>
> Don't count number of failed disks looking at sysfs entries but
> calculate it at the end. Do it only for arrays as containers report
> no disks, just spares.
>
> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> ---
> Incremental.c | 14 ++++----------
> Monitor.c | 4 ++--
> managemon.c | 4 ++--
> mdadm.h | 1 -
> raid6check.c | 2 +-
> sysfs.c | 18 ++++++++----------
> 6 files changed, 17 insertions(+), 26 deletions(-)
>
> v2:
> adapted to apply to latest upstream version
Applied!
Thanks,
Jes
^ permalink raw reply
* Re: [PATCH] Grow: set component size prior to array size
From: Jes Sorensen @ 2017-06-05 15:06 UTC (permalink / raw)
To: Tomasz Majchrzak, linux-raid
In-Reply-To: <1496671784-1612-1-git-send-email-tomasz.majchrzak@intel.com>
On 06/05/2017 10:09 AM, Tomasz Majchrzak wrote:
> It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
> optimization"). For native metadata component size is set in kernel for
> entire disk space. As external metadata supports multiple arrays within
> one disk, the component size is set to array size. If component size is
> not updated prior to array size update, the grow operation fails.
>
> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> ---
> Grow.c | 2 ++
> 1 file changed, 2 insertions(+)
Applied!
Thanks,
Jes
^ permalink raw reply
* [PATCH] Grow: set component size prior to array size
From: Tomasz Majchrzak @ 2017-06-05 14:09 UTC (permalink / raw)
To: linux-raid; +Cc: jes.sorensen, Tomasz Majchrzak
In-Reply-To: <7964b1f5-5092-238b-403e-ed4fba584746@gmail.com>
It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
optimization"). For native metadata component size is set in kernel for
entire disk space. As external metadata supports multiple arrays within
one disk, the component size is set to array size. If component size is
not updated prior to array size update, the grow operation fails.
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
Grow.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Grow.c b/Grow.c
index ecf5ca0..4ecb1d8 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1977,6 +1977,8 @@ int Grow_reshape(char *devname, int fd,
*/
min_csize = 0;
for (mdi = sra->devs; mdi; mdi = mdi->next) {
+ sysfs_set_num(sra, mdi, "size", s->size == MAX_SIZE ? 0
+ : s->size);
if (array.not_persistent == 0 &&
array.major_version == 0 &&
get_linux_version() < 3001000) {
--
1.8.3.1
^ permalink raw reply related
* Re: internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: CoolCold @ 2017-06-05 12:30 UTC (permalink / raw)
To: David Brown; +Cc: Linux RAID
In-Reply-To: <79ff8fa9-4282-4907-bacf-e9083bac3670@hesbynett.no>
Hello!
Thanks for your proposals.
From my POV, having ~ 100 iops per drive is completely okay, as you
see from iostat/fio output it's not even near the case.
Hope this clarifies.
On Mon, Jun 5, 2017 at 5:55 PM, David Brown <david.brown@hesbynett.no> wrote:
>
>
> On 05/06/17 12:06, CoolCold wrote:
>>
>> Hello!
>> Keep testing the new box and while having not the best sync speed,
>> it's not the worst thing I found.
>>
>> Doing FIO testing, for RAID10 over 20 10k RPM drives, I have very bad
>> performance, like _45_ iops only.
>>
>
> <snip>
>
>>
>> Any advises would be very helpful.
>>
>
> The best advice I can give you is to take a step back, and try to be clear
> what problem you are trying to solve here. What is this system supposed to
> do? What are your requirements? What are your use-cases?
>
> Attempting to optimise the sync speed (or IOPS) of a 20 drive RAID10 set is
> a totally pointless exercise in itself. At best, it would be a torture test
> for md raid. There is no system in the world where the specifications are
> "make the fastest 20 drive RAID10 setup" - at least, not from a sane IT
> management.
>
> When you have a clear picture of what you actually /need/, what you /want/,
> what you /have/, and how you want to use it all - /then/, and only then, is
> it time to look at possible RAID setups, filesystem arrangements,
> alternative hardware, etc. Then you ask for advice on this list (or
> elsewhere - but this list is a good starting point), learning about the pros
> and cons of a variety of possible arrangements.
>
> Once you have collected these ideas, with different balances in performance,
> safety, space efficiency, features, cost, etc., you can then test them out
> and look at benchmarks to see if they will work in practice.
--
Best regards,
[COOLCOLD-RIPN]
^ permalink raw reply
* Re: RAID10 performance with 20 drives
From: Pasi Kärkkäinen @ 2017-06-05 11:34 UTC (permalink / raw)
To: CoolCold; +Cc: Nikhil Kshirsagar, Roger Heflin, Roman Mamedov, Linux RAID
In-Reply-To: <CAGqmV7rX7H_sTuJ7RBVHZX+TCTcWAH0K32CimD0oeEB_r32AyQ@mail.gmail.com>
On Mon, Jun 05, 2017 at 04:52:27PM +0700, CoolCold wrote:
> Hello!
> The data i was able to find with lsscsi:
> Enclosure:
> [0:0:24:0] enclosu LSI SAS3x40 0601 -
> state=running queue_depth=254 scsi_level=6 type=13 device_blocked=0 timeout=0
>
> Server https://www.supermicro.com/products/system/2u/2028/ssg-2028r-e1cr24l.cfm
>
OK. So you have a single expander SAS3 backplane. And a Broadcom/LSI SAS 3008 IT-mode HBA, which should be good. Well assuming you're running up-to-date firmware with it.
> Drives are:
> === START OF INFORMATION SECTION ===
> Vendor: TOSHIBA
> Product: AL14SEB18EQ
> Revision: 0101
> User Capacity: 1,800,360,124,416 bytes [1.80 TB]
> Logical block size: 512 bytes
> Physical block size: 4096 bytes
> Lowest aligned LBA: 0
> Rotation Rate: 10500 rpm
> Form Factor: 2.5 inches
> Logical Unit id: 0x500003975840f759
> Serial number: X6K0A0D5FZRC
> Device type: disk
> Transport protocol: SAS
> Local Time is: Mon Jun 5 09:51:56 2017 UTC
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> Temperature Warning: Enabled
>
> (don't think it matters though)
>
Have you tried for example reading from all the disks simultaneously? Just to verify the link(s) between the HBA and the expander backplane is/are working properly, and you get good throughput with simultaneous reads from all disks?
-- Pasi
> On Thu, Jun 1, 2017 at 6:31 PM, Nikhil Kshirsagar <nkshirsa@redhat.com> wrote:
> > Did you check max_sectors_kb ? We found once, in some setups with raid, in
> > particular with HP raid controllers, larger number of drives seemed to
> > reduce the value of max_sectors_kb. Sorry if its already mentioned, I
> > haven't read the entire thread in detail.
> >
> > On Thu, Jun 1, 2017 at 4:50 PM, Roger Heflin <rogerheflin@gmail.com> wrote:
> >>
> >> Also supply an "iostat -x 1 5" since that will show each disks usage.
> >>
> >> vmstat in my experience does not appear to show internal MD disk traffic.
> >>
> >> On Thu, Jun 1, 2017 at 1:33 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> >> > On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
> >> >> Hello!
> >> >> Roman, i've updated the kernel to 4.11 and started "check" action,
> >> >> results are basically the same, output on github
> >> >> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
> >> >> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
> >> >>
> >> >
> >> > You need to provide more details about the actual storage setup.
> >> >
> >> > Like already said/asked for:
> >> >
> >> > - Which HBA are you using?
> >> > - Which PCIe link speed are you using for the HBA?
> >> > - Which driver version for the HBA?
> >> > - Which HBA firmware version?
> >> >
> >> > - How are the disks connected to the HBA ? Direct-connect, or via an
> >> > Expander?
> >> > - If you have an expander, what's the (SAS) link speed/count between the
> >> > HBA(s) and the Expander?
> >> >
> >> >
> >> > -- Pasi
> >> >
> >> >> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
> >> >> > On Wed, 31 May 2017 19:20:10 +0700
> >> >> > CoolCold <coolthecold@gmail.com> wrote:
> >> >> >
> >> >> >> Creation (disable write intent bitmap, with bitmap all is much
> >> >> >> worse):
> >> >> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
> >> >> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
> >> >> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
> >> >> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
> >> >> >>
> >> >> >> kernel:
> >> >> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
> >> >> >> Linux version 3.10.0-327.el7.x86_64
> >> >> >> (builder@kbuilder.dev.centos.org)
> >> >> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
> >> >> >> 19 22:10:57 UTC 2015
> >> >> >>
> >> >> >> So, the question is - why cpu usage is so high and I suppose is a
> >> >> >> limit here?
> >> >> >
> >> >> > Definitely try a newer kernel, 4.4 at the very least; if no changes
> >> >> > then 4.11.
> >> >> >
> >> >> > Also I would suggest to try out larger chunk sizes, such as 512 and
> >> >> > 1024 KB.
> >> >> >
> >> >> > If you plan to use this long-term in production, also read up on the
> >> >> > various
> >> >> > RAID10 data layouts and their benefits and downsides (man md, search
> >> >> > for
> >> >> > "layout"; and search the Internet for benchmarks of all three).
> >> >> >
> >> >> > --
> >> >> > With respect,
> >> >> > Roman
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best regards,
> >> >> [COOLCOLD-RIPN]
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> >> >> in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
> --
> Best regards,
> [COOLCOLD-RIPN]
^ permalink raw reply
* Re: internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: David Brown @ 2017-06-05 10:55 UTC (permalink / raw)
To: CoolCold, Linux RAID
In-Reply-To: <CAGqmV7qRh+_BXAr-qf6vUkiZyzi69E8CiX_vA92tZV2LfDxq_A@mail.gmail.com>
On 05/06/17 12:06, CoolCold wrote:
> Hello!
> Keep testing the new box and while having not the best sync speed,
> it's not the worst thing I found.
>
> Doing FIO testing, for RAID10 over 20 10k RPM drives, I have very bad
> performance, like _45_ iops only.
>
<snip>
>
> Any advises would be very helpful.
>
The best advice I can give you is to take a step back, and try to be
clear what problem you are trying to solve here. What is this system
supposed to do? What are your requirements? What are your use-cases?
Attempting to optimise the sync speed (or IOPS) of a 20 drive RAID10 set
is a totally pointless exercise in itself. At best, it would be a
torture test for md raid. There is no system in the world where the
specifications are "make the fastest 20 drive RAID10 setup" - at least,
not from a sane IT management.
When you have a clear picture of what you actually /need/, what you
/want/, what you /have/, and how you want to use it all - /then/, and
only then, is it time to look at possible RAID setups, filesystem
arrangements, alternative hardware, etc. Then you ask for advice on
this list (or elsewhere - but this list is a good starting point),
learning about the pros and cons of a variety of possible arrangements.
Once you have collected these ideas, with different balances in
performance, safety, space efficiency, features, cost, etc., you can
then test them out and look at benchmarks to see if they will work in
practice.
^ permalink raw reply
* internal write-intent bitmap is horribly slow with RAID10 over 20 drives
From: CoolCold @ 2017-06-05 10:06 UTC (permalink / raw)
To: Linux RAID
Hello!
Keep testing the new box and while having not the best sync speed,
it's not the worst thing I found.
Doing FIO testing, for RAID10 over 20 10k RPM drives, I have very bad
performance, like _45_ iops only.
Drives:
=== START OF INFORMATION SECTION ===
Vendor: TOSHIBA
Product: AL14SEB18EQ
Revision: 0101
User Capacity: 1,800,360,124,416 bytes [1.80 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
Lowest aligned LBA: 0
Rotation Rate: 10500 rpm
Form Factor: 2.5 inches
Logical Unit id: 0x500003975840f759
Serial number: X6K0A0D5FZRC
Device type: disk
Transport protocol: SAS
Local Time is: Mon Jun 5 09:51:56 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Output from fio with internal write-intent bitmap:
Jobs: 1 (f=1): [w(1)] [28.3% done] [0KB/183KB/0KB /s] [0/45/0 iops]
[eta 07m:11s]
array definition:
[root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid10 sdx[19] sdw[18] sdv[17] sdu[16] sdt[15] sds[14]
sdr[13] sdq[12] sdp[11] sdo[10] sdn[9] sdm[8] sdl[7] sdk[6] sdj[5]
sdi[4] sdh[3] sdg[2] sdf[1] sde[0]
17580330880 blocks super 1.2 64K chunks 2 near-copies [20/20]
[UUUUUUUUUUUUUUUUUUUU]
bitmap: 0/66 pages [0KB], 131072KB chunk
Setting journal to be
1) on SSD (separate drives), shows
Jobs: 1 (f=1): [w(1)] [5.0% done] [0KB/18783KB/0KB /s] [0/4695/0 iops]
[eta 09m:31s]
2) to 'none' (disabling) shows
Jobs: 1 (f=1): [w(1)] [14.0% done] [0KB/18504KB/0KB /s] [0/4626/0
iops] [eta 08m:36s]
All bitmap manipulations was on running array,
Sample output from iostat with internal bitmap (array of interest is md1):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdz 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdy 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 47.00 0.00 542.00
23.06 0.67 14.34 0.00 14.34 14.28 67.10
sdh 0.00 0.00 0.00 50.00 0.00 734.00
29.36 0.68 13.66 0.00 13.66 13.60 68.00
sdi 0.00 0.00 0.00 46.00 0.00 478.00
20.78 0.67 14.63 0.00 14.63 14.54 66.90
sdf 0.00 0.00 0.00 47.00 0.00 542.00
23.06 0.67 14.28 0.00 14.28 14.21 66.80
sdl 0.00 0.00 0.00 45.00 0.00 414.00
18.40 0.66 14.78 0.00 14.78 14.69 66.10
sdn 0.00 0.00 0.00 44.00 0.00 350.00
15.91 0.66 14.95 0.00 14.95 14.95 65.80
sdj 0.00 0.00 0.00 46.00 0.00 478.00
20.78 0.65 14.22 0.00 14.22 14.13 65.00
sdk 0.00 0.00 0.00 46.00 0.00 418.00
18.17 0.65 14.35 0.00 14.35 13.98 64.30
sdm 0.00 0.00 0.00 44.00 0.00 350.00
15.91 0.62 14.18 0.00 14.18 14.14 62.20
sdp 0.00 0.00 0.00 43.00 0.00 286.00
13.30 0.62 14.56 0.00 14.56 14.51 62.40
sdg 0.00 0.00 0.00 50.00 0.00 734.00
29.36 0.68 13.58 0.00 13.58 13.52 67.60
sdv 0.00 0.00 0.00 43.00 0.00 346.00
16.09 0.67 15.19 0.00 15.19 15.49 66.60
sdr 0.00 0.00 0.00 44.00 0.00 410.00
18.64 0.66 14.59 0.00 14.59 14.86 65.40
sds 0.00 0.00 0.00 43.00 0.00 286.00
13.30 0.65 15.07 0.00 15.07 15.02 64.60
sdt 0.00 0.00 0.00 42.00 0.00 282.00
13.43 0.65 15.17 0.00 15.17 15.48 65.00
sdo 0.00 0.00 0.00 43.00 0.00 286.00
13.30 0.64 14.93 0.00 14.93 14.88 64.00
sdq 0.00 0.00 0.00 45.00 0.00 414.00
18.40 0.66 14.67 0.00 14.67 14.62 65.80
sdw 0.00 0.00 0.00 43.00 0.00 346.00
16.09 0.66 14.93 0.00 14.93 15.21 65.40
sdu 0.00 0.00 0.00 44.00 0.00 350.00
15.91 0.64 14.55 0.00 14.55 14.50 63.80
sdx 0.00 0.00 0.00 44.00 0.00 350.00
15.91 0.64 14.55 0.00 14.55 14.48 63.70
md127 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
md125 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 41.00 0.00 2624.00
128.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
The same picture on 3.10.0-327.el7.x86_64 and 4.11 kernel.
Any advises would be very helpful.
--
Best regards,
[COOLCOLD-RIPN]
^ permalink raw reply
* Re: RAID10 performance with 20 drives
From: CoolCold @ 2017-06-05 9:52 UTC (permalink / raw)
To: Nikhil Kshirsagar
Cc: Roger Heflin, Pasi Kärkkäinen, Roman Mamedov,
Linux RAID
In-Reply-To: <CAMNNMLEuutwLE8xft+ZEo=ShxRP=rJEq1kzNtVbgE5RBTOYcrQ@mail.gmail.com>
Hello!
The data i was able to find with lsscsi:
Enclosure:
[0:0:24:0] enclosu LSI SAS3x40 0601 -
state=running queue_depth=254 scsi_level=6 type=13 device_blocked=0 timeout=0
Server https://www.supermicro.com/products/system/2u/2028/ssg-2028r-e1cr24l.cfm
Drives are:
=== START OF INFORMATION SECTION ===
Vendor: TOSHIBA
Product: AL14SEB18EQ
Revision: 0101
User Capacity: 1,800,360,124,416 bytes [1.80 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
Lowest aligned LBA: 0
Rotation Rate: 10500 rpm
Form Factor: 2.5 inches
Logical Unit id: 0x500003975840f759
Serial number: X6K0A0D5FZRC
Device type: disk
Transport protocol: SAS
Local Time is: Mon Jun 5 09:51:56 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
(don't think it matters though)
On Thu, Jun 1, 2017 at 6:31 PM, Nikhil Kshirsagar <nkshirsa@redhat.com> wrote:
> Did you check max_sectors_kb ? We found once, in some setups with raid, in
> particular with HP raid controllers, larger number of drives seemed to
> reduce the value of max_sectors_kb. Sorry if its already mentioned, I
> haven't read the entire thread in detail.
>
> On Thu, Jun 1, 2017 at 4:50 PM, Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>> Also supply an "iostat -x 1 5" since that will show each disks usage.
>>
>> vmstat in my experience does not appear to show internal MD disk traffic.
>>
>> On Thu, Jun 1, 2017 at 1:33 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> > On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
>> >> Hello!
>> >> Roman, i've updated the kernel to 4.11 and started "check" action,
>> >> results are basically the same, output on github
>> >> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
>> >> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
>> >>
>> >
>> > You need to provide more details about the actual storage setup.
>> >
>> > Like already said/asked for:
>> >
>> > - Which HBA are you using?
>> > - Which PCIe link speed are you using for the HBA?
>> > - Which driver version for the HBA?
>> > - Which HBA firmware version?
>> >
>> > - How are the disks connected to the HBA ? Direct-connect, or via an
>> > Expander?
>> > - If you have an expander, what's the (SAS) link speed/count between the
>> > HBA(s) and the Expander?
>> >
>> >
>> > -- Pasi
>> >
>> >> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> >> > On Wed, 31 May 2017 19:20:10 +0700
>> >> > CoolCold <coolthecold@gmail.com> wrote:
>> >> >
>> >> >> Creation (disable write intent bitmap, with bitmap all is much
>> >> >> worse):
>> >> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
>> >> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
>> >> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
>> >> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
>> >> >>
>> >> >> kernel:
>> >> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
>> >> >> Linux version 3.10.0-327.el7.x86_64
>> >> >> (builder@kbuilder.dev.centos.org)
>> >> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
>> >> >> 19 22:10:57 UTC 2015
>> >> >>
>> >> >> So, the question is - why cpu usage is so high and I suppose is a
>> >> >> limit here?
>> >> >
>> >> > Definitely try a newer kernel, 4.4 at the very least; if no changes
>> >> > then 4.11.
>> >> >
>> >> > Also I would suggest to try out larger chunk sizes, such as 512 and
>> >> > 1024 KB.
>> >> >
>> >> > If you plan to use this long-term in production, also read up on the
>> >> > various
>> >> > RAID10 data layouts and their benefits and downsides (man md, search
>> >> > for
>> >> > "layout"; and search the Internet for benchmarks of all three).
>> >> >
>> >> > --
>> >> > With respect,
>> >> > Roman
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> [COOLCOLD-RIPN]
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>> >> in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Best regards,
[COOLCOLD-RIPN]
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Andreas Klauer @ 2017-06-05 9:24 UTC (permalink / raw)
To: Paul Tonelli; +Cc: linux-raid
In-Reply-To: <b8b6eb18-2952-86ee-5329-9e4f4c406c3d@tonel.li>
On Mon, Jun 05, 2017 at 12:58:03AM +0200, Paul Tonelli wrote:
> I did find this one at that exact offset 26096 = (133169664 - 512)/512
26096->260096? Okay.
> Now, after doing a pvcreate and a vgcfgrestore ...
Should not be necessary if the data on two drives was okay?
You did leave the dd-overwritten drive out as missing, right?
Do you have the correct disk order and chunk size and offset?
You have to be 140% sure the RAID itself is running correctly,
otherwise all other steps are bound to fail.
If you run photorec on the RAID and it manages to recover
a file that is larger than number of drives * chunk size
and intact, you can have some confidence the RAID is okay
in terms of disk order and chunk size - the offset may
still be off by multiple of stripe alignment but if the
offset is correct too, file -s /dev/md0 should say LVM PV
in your case.
Regards
Andreas Klauer
^ permalink raw reply
* [PATCH] md: fix deadlock between mddev_suspend() and md_write_start()
From: NeilBrown @ 2017-06-05 6:49 UTC (permalink / raw)
To: Shaohua Li; +Cc: Nix, linux-raid
In-Reply-To: <20170530174133.qdj2qsopwug2opp6@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 14893 bytes --]
If mddev_suspend() races with md_write_start() we can deadlock
with mddev_suspend() waiting for the request that is currently
in md_write_start() to complete the ->make_request() call,
and md_write_start() waiting for the metadata to be updated
to mark the array as 'dirty'.
As metadata updates done by md_check_recovery() only happen then
the mddev_lock() can be claimed, and as mddev_suspend() is often
called with the lock held, these threads wait indefinitely for each
other.
We fix this by having md_write_start() abort if mddev_suspend()
is happening, and ->make_request() aborts if md_write_start()
aborted.
md_make_request() can detect this abort, decrease the ->active_io
count, and wait for mddev_suspend().
Reported-by: Nix <nix@esperi.org.uk>
Signed-off-by: NeilBrown <neilb@suse.com>
---
drivers/md/faulty.c | 5 +++--
drivers/md/linear.c | 7 ++++---
drivers/md/md.c | 23 +++++++++++++++++------
drivers/md/md.h | 4 ++--
drivers/md/multipath.c | 8 ++++----
drivers/md/raid0.c | 7 ++++---
drivers/md/raid1.c | 11 +++++++----
drivers/md/raid10.c | 10 ++++++----
drivers/md/raid5.c | 17 +++++++++--------
9 files changed, 56 insertions(+), 36 deletions(-)
diff --git a/drivers/md/faulty.c b/drivers/md/faulty.c
index b0536cfd8e17..06a64d5d8c6c 100644
--- a/drivers/md/faulty.c
+++ b/drivers/md/faulty.c
@@ -170,7 +170,7 @@ static void add_sector(struct faulty_conf *conf, sector_t start, int mode)
conf->nfaults = n+1;
}
-static void faulty_make_request(struct mddev *mddev, struct bio *bio)
+static bool faulty_make_request(struct mddev *mddev, struct bio *bio)
{
struct faulty_conf *conf = mddev->private;
int failit = 0;
@@ -182,7 +182,7 @@ static void faulty_make_request(struct mddev *mddev, struct bio *bio)
* just fail immediately
*/
bio_io_error(bio);
- return;
+ return true;
}
if (check_sector(conf, bio->bi_iter.bi_sector,
@@ -224,6 +224,7 @@ static void faulty_make_request(struct mddev *mddev, struct bio *bio)
bio->bi_bdev = conf->rdev->bdev;
generic_make_request(bio);
+ return true;
}
static void faulty_status(struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/linear.c b/drivers/md/linear.c
index df6f2c98eca7..5f1eb9189542 100644
--- a/drivers/md/linear.c
+++ b/drivers/md/linear.c
@@ -245,7 +245,7 @@ static void linear_free(struct mddev *mddev, void *priv)
kfree(conf);
}
-static void linear_make_request(struct mddev *mddev, struct bio *bio)
+static bool linear_make_request(struct mddev *mddev, struct bio *bio)
{
char b[BDEVNAME_SIZE];
struct dev_info *tmp_dev;
@@ -254,7 +254,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio)
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
md_flush_request(mddev, bio);
- return;
+ return true;
}
tmp_dev = which_dev(mddev, bio_sector);
@@ -292,7 +292,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio)
mddev_check_write_zeroes(mddev, bio);
generic_make_request(bio);
}
- return;
+ return true;
out_of_bounds:
pr_err("md/linear:%s: make_request: Sector %llu out of bounds on dev %s: %llu sectors, offset %llu\n",
@@ -302,6 +302,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio)
(unsigned long long)tmp_dev->rdev->sectors,
(unsigned long long)start_sector);
bio_io_error(bio);
+ return true;
}
static void linear_status (struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 23f4adf3a8cc..da59051545a2 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -277,7 +277,7 @@ static blk_qc_t md_make_request(struct request_queue *q, struct bio *bio)
bio_endio(bio);
return BLK_QC_T_NONE;
}
- smp_rmb(); /* Ensure implications of 'active' are visible */
+check_suspended:
rcu_read_lock();
if (mddev->suspended) {
DEFINE_WAIT(__wait);
@@ -302,7 +302,11 @@ static blk_qc_t md_make_request(struct request_queue *q, struct bio *bio)
sectors = bio_sectors(bio);
/* bio could be mergeable after passing to underlayer */
bio->bi_opf &= ~REQ_NOMERGE;
- mddev->pers->make_request(mddev, bio);
+ if (!mddev->pers->make_request(mddev, bio)) {
+ atomic_dec(&mddev->active_io);
+ wake_up(&mddev->sb_wait);
+ goto check_suspended;
+ }
cpu = part_stat_lock();
part_stat_inc(cpu, &mddev->gendisk->part0, ios[rw]);
@@ -327,6 +331,7 @@ void mddev_suspend(struct mddev *mddev)
if (mddev->suspended++)
return;
synchronize_rcu();
+ wake_up(&mddev->sb_wait);
wait_event(mddev->sb_wait, atomic_read(&mddev->active_io) == 0);
mddev->pers->quiesce(mddev, 1);
@@ -7950,12 +7955,14 @@ EXPORT_SYMBOL(md_done_sync);
* If we need to update some array metadata (e.g. 'active' flag
* in superblock) before writing, schedule a superblock update
* and wait for it to complete.
+ * A return value of 'false' means that the write wasn't recorded
+ * and cannot proceed as the array is being suspend.
*/
-void md_write_start(struct mddev *mddev, struct bio *bi)
+bool md_write_start(struct mddev *mddev, struct bio *bi)
{
int did_change = 0;
if (bio_data_dir(bi) != WRITE)
- return;
+ return true;
BUG_ON(mddev->ro == 1);
if (mddev->ro == 2) {
@@ -7987,9 +7994,13 @@ void md_write_start(struct mddev *mddev, struct bio *bi)
if (did_change)
sysfs_notify_dirent_safe(mddev->sysfs_state);
wait_event(mddev->sb_wait,
- !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags));
+ !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags) && !mddev->suspended);
+ if (test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags)) {
+ percpu_ref_put(&mddev->writes_pending);
+ return false;
+ }
+ return true;
}
-EXPORT_SYMBOL(md_write_start);
/* md_write_inc can only be called when md_write_start() has
* already been called at least once of the current request.
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 0fa1de42c42b..63d342d560b8 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -510,7 +510,7 @@ struct md_personality
int level;
struct list_head list;
struct module *owner;
- void (*make_request)(struct mddev *mddev, struct bio *bio);
+ bool (*make_request)(struct mddev *mddev, struct bio *bio);
int (*run)(struct mddev *mddev);
void (*free)(struct mddev *mddev, void *priv);
void (*status)(struct seq_file *seq, struct mddev *mddev);
@@ -649,7 +649,7 @@ extern void md_wakeup_thread(struct md_thread *thread);
extern void md_check_recovery(struct mddev *mddev);
extern void md_reap_sync_thread(struct mddev *mddev);
extern int mddev_init_writes_pending(struct mddev *mddev);
-extern void md_write_start(struct mddev *mddev, struct bio *bi);
+extern bool md_write_start(struct mddev *mddev, struct bio *bi);
extern void md_write_inc(struct mddev *mddev, struct bio *bi);
extern void md_write_end(struct mddev *mddev);
extern void md_done_sync(struct mddev *mddev, int blocks, int ok);
diff --git a/drivers/md/multipath.c b/drivers/md/multipath.c
index e95d521d93e9..c8d985ba008d 100644
--- a/drivers/md/multipath.c
+++ b/drivers/md/multipath.c
@@ -106,7 +106,7 @@ static void multipath_end_request(struct bio *bio)
rdev_dec_pending(rdev, conf->mddev);
}
-static void multipath_make_request(struct mddev *mddev, struct bio * bio)
+static bool multipath_make_request(struct mddev *mddev, struct bio * bio)
{
struct mpconf *conf = mddev->private;
struct multipath_bh * mp_bh;
@@ -114,7 +114,7 @@ static void multipath_make_request(struct mddev *mddev, struct bio * bio)
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
md_flush_request(mddev, bio);
- return;
+ return true;
}
mp_bh = mempool_alloc(conf->pool, GFP_NOIO);
@@ -126,7 +126,7 @@ static void multipath_make_request(struct mddev *mddev, struct bio * bio)
if (mp_bh->path < 0) {
bio_io_error(bio);
mempool_free(mp_bh, conf->pool);
- return;
+ return true;
}
multipath = conf->multipaths + mp_bh->path;
@@ -141,7 +141,7 @@ static void multipath_make_request(struct mddev *mddev, struct bio * bio)
mddev_check_writesame(mddev, &mp_bh->bio);
mddev_check_write_zeroes(mddev, &mp_bh->bio);
generic_make_request(&mp_bh->bio);
- return;
+ return true;
}
static void multipath_status(struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index d6c0bc76e837..94d9ae9b0fd0 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -548,7 +548,7 @@ static void raid0_handle_discard(struct mddev *mddev, struct bio *bio)
bio_endio(bio);
}
-static void raid0_make_request(struct mddev *mddev, struct bio *bio)
+static bool raid0_make_request(struct mddev *mddev, struct bio *bio)
{
struct strip_zone *zone;
struct md_rdev *tmp_dev;
@@ -559,12 +559,12 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio)
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
md_flush_request(mddev, bio);
- return;
+ return true;
}
if (unlikely((bio_op(bio) == REQ_OP_DISCARD))) {
raid0_handle_discard(mddev, bio);
- return;
+ return true;
}
bio_sector = bio->bi_iter.bi_sector;
@@ -599,6 +599,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio)
mddev_check_writesame(mddev, bio);
mddev_check_write_zeroes(mddev, bio);
generic_make_request(bio);
+ return true;
}
static void raid0_status(struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index e1a7e3d4c5e4..c71739b87ab7 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1321,7 +1321,6 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
* Continue immediately if no resync is active currently.
*/
- md_write_start(mddev, bio); /* wait on superblock update early */
if ((bio_end_sector(bio) > mddev->suspend_lo &&
bio->bi_iter.bi_sector < mddev->suspend_hi) ||
@@ -1550,13 +1549,13 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
wake_up(&conf->wait_barrier);
}
-static void raid1_make_request(struct mddev *mddev, struct bio *bio)
+static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
{
sector_t sectors;
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
md_flush_request(mddev, bio);
- return;
+ return true;
}
/*
@@ -1571,8 +1570,12 @@ static void raid1_make_request(struct mddev *mddev, struct bio *bio)
if (bio_data_dir(bio) == READ)
raid1_read_request(mddev, bio, sectors, NULL);
- else
+ else {
+ if (!md_write_start(mddev,bio))
+ return false;
raid1_write_request(mddev, bio, sectors);
+ }
+ return true;
}
static void raid1_status(struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 797ed60abd5e..52acffa7a06a 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1303,8 +1303,6 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
sector_t sectors;
int max_sectors;
- md_write_start(mddev, bio);
-
/*
* Register the new request and wait if the reconstruction
* thread has put up a bar for new requests.
@@ -1525,7 +1523,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
raid10_write_request(mddev, bio, r10_bio);
}
-static void raid10_make_request(struct mddev *mddev, struct bio *bio)
+static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
{
struct r10conf *conf = mddev->private;
sector_t chunk_mask = (conf->geo.chunk_mask & conf->prev.chunk_mask);
@@ -1534,9 +1532,12 @@ static void raid10_make_request(struct mddev *mddev, struct bio *bio)
if (unlikely(bio->bi_opf & REQ_PREFLUSH)) {
md_flush_request(mddev, bio);
- return;
+ return true;
}
+ if (!md_write_start(mddev, bio))
+ return false;
+
/*
* If this request crosses a chunk boundary, we need to split
* it.
@@ -1553,6 +1554,7 @@ static void raid10_make_request(struct mddev *mddev, struct bio *bio)
/* In case raid10d snuck in to freeze_array */
wake_up(&conf->wait_barrier);
+ return true;
}
static void raid10_status(struct seq_file *seq, struct mddev *mddev)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 861fc9a8d1be..ad1a644a17bc 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5469,7 +5469,6 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
last_sector = bi->bi_iter.bi_sector + (bi->bi_iter.bi_size>>9);
bi->bi_next = NULL;
- md_write_start(mddev, bi);
stripe_sectors = conf->chunk_sectors *
(conf->raid_disks - conf->max_degraded);
@@ -5539,11 +5538,10 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
release_stripe_plug(mddev, sh);
}
- md_write_end(mddev);
bio_endio(bi);
}
-static void raid5_make_request(struct mddev *mddev, struct bio * bi)
+static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
{
struct r5conf *conf = mddev->private;
int dd_idx;
@@ -5559,10 +5557,10 @@ static void raid5_make_request(struct mddev *mddev, struct bio * bi)
int ret = r5l_handle_flush_request(conf->log, bi);
if (ret == 0)
- return;
+ return true;
if (ret == -ENODEV) {
md_flush_request(mddev, bi);
- return;
+ return true;
}
/* ret == -EAGAIN, fallback */
/*
@@ -5572,6 +5570,8 @@ static void raid5_make_request(struct mddev *mddev, struct bio * bi)
do_flush = bi->bi_opf & REQ_PREFLUSH;
}
+ if (!md_write_start(mddev, bi))
+ return false;
/*
* If array is degraded, better not do chunk aligned read because
* later we might have to read it again in order to reconstruct
@@ -5581,18 +5581,18 @@ static void raid5_make_request(struct mddev *mddev, struct bio * bi)
mddev->reshape_position == MaxSector) {
bi = chunk_aligned_read(mddev, bi);
if (!bi)
- return;
+ return true;
}
if (unlikely(bio_op(bi) == REQ_OP_DISCARD)) {
make_discard_request(mddev, bi);
- return;
+ md_write_end(mddev);
+ return true;
}
logical_sector = bi->bi_iter.bi_sector & ~((sector_t)STRIPE_SECTORS-1);
last_sector = bio_end_sector(bi);
bi->bi_next = NULL;
- md_write_start(mddev, bi);
prepare_to_wait(&conf->wait_for_overlap, &w, TASK_UNINTERRUPTIBLE);
for (;logical_sector < last_sector; logical_sector += STRIPE_SECTORS) {
@@ -5730,6 +5730,7 @@ static void raid5_make_request(struct mddev *mddev, struct bio * bi)
if (rw == WRITE)
md_write_end(mddev);
bio_endio(bi);
+ return true;
}
static sector_t raid5_size(struct mddev *mddev, sector_t sectors, int raid_disks);
--
2.12.2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply related
* [PATCH] md: initialise ->writes_pending in personality modules.
From: NeilBrown @ 2017-06-05 6:05 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, dm-devel, Heinz Mauelshagen
[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]
The new per-cpu counter for writes_pending is initialised in
md_alloc(), which is not called by dm-raid.
So dm-raid fails when md_write_start() is called.
Move the initialization to the personality modules
that need it. This way it is always initialised when needed,
but isn't unnecessarily initialized (requiring memory allocation)
when the personality doesn't use writes_pending.
Reported-by: Heinz Mauelshagen <heinzm@redhat.com>
Fixes: 4ad23a976413 ("MD: use per-cpu counter for writes_pending")
Signed-off-by: NeilBrown <neilb@suse.com>
---
drivers/md/md.c | 16 ++++++++++++----
drivers/md/md.h | 1 +
drivers/md/raid1.c | 2 ++
drivers/md/raid10.c | 3 +++
drivers/md/raid5.c | 3 +++
5 files changed, 21 insertions(+), 4 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 10367ffe92e3..23f4adf3a8cc 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5174,6 +5174,18 @@ static void mddev_delayed_delete(struct work_struct *ws)
static void no_op(struct percpu_ref *r) {}
+int mddev_init_writes_pending(struct mddev *mddev)
+{
+ if (mddev->writes_pending.percpu_count_ptr)
+ return 0;
+ if (percpu_ref_init(&mddev->writes_pending, no_op, 0, GFP_KERNEL) < 0)
+ return -ENOMEM;
+ /* We want to start with the refcount at zero */
+ percpu_ref_put(&mddev->writes_pending);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mddev_init_writes_pending);
+
static int md_alloc(dev_t dev, char *name)
{
/*
@@ -5239,10 +5251,6 @@ static int md_alloc(dev_t dev, char *name)
blk_queue_make_request(mddev->queue, md_make_request);
blk_set_stacking_limits(&mddev->queue->limits);
- if (percpu_ref_init(&mddev->writes_pending, no_op, 0, GFP_KERNEL) < 0)
- goto abort;
- /* We want to start with the refcount at zero */
- percpu_ref_put(&mddev->writes_pending);
disk = alloc_disk(1 << shift);
if (!disk) {
blk_cleanup_queue(mddev->queue);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 11f15146ce51..0fa1de42c42b 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -648,6 +648,7 @@ extern void md_unregister_thread(struct md_thread **threadp);
extern void md_wakeup_thread(struct md_thread *thread);
extern void md_check_recovery(struct mddev *mddev);
extern void md_reap_sync_thread(struct mddev *mddev);
+extern int mddev_init_writes_pending(struct mddev *mddev);
extern void md_write_start(struct mddev *mddev, struct bio *bi);
extern void md_write_inc(struct mddev *mddev, struct bio *bi);
extern void md_write_end(struct mddev *mddev);
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index af5056d56878..e1a7e3d4c5e4 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -3063,6 +3063,8 @@ static int raid1_run(struct mddev *mddev)
mdname(mddev));
return -EIO;
}
+ if (mddev_init_writes_pending(mddev) < 0)
+ return -ENOMEM;
/*
* copy the already verified devices into our private RAID1
* bookkeeping area. [whatever we allocate in run(),
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 4343d7ff9916..797ed60abd5e 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3611,6 +3611,9 @@ static int raid10_run(struct mddev *mddev)
int first = 1;
bool discard_supported = false;
+ if (mddev_init_writes_pending(mddev) < 0)
+ return -ENOMEM;
+
if (mddev->private == NULL) {
conf = setup_conf(mddev);
if (IS_ERR(conf))
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 9c4f7659f8b1..861fc9a8d1be 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7108,6 +7108,9 @@ static int raid5_run(struct mddev *mddev)
long long min_offset_diff = 0;
int first = 1;
+ if (mddev_init_writes_pending(mddev) < 0)
+ return -ENOMEM;
+
if (mddev->recovery_cp != MaxSector)
pr_notice("md/raid:%s: not clean -- starting background reconstruction\n",
mdname(mddev));
--
2.12.2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply related
* (unknown),
From: citydesk @ 2017-06-05 4:30 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: 721224187.zip --]
[-- Type: application/zip, Size: 3190 bytes --]
^ permalink raw reply
* Re: [PATCH] md: simplify code with bio_io_error
From: Guoqing Jiang @ 2017-06-05 3:11 UTC (permalink / raw)
To: Christoph Hellwig, Shaohua Li; +Cc: linux-raid
In-Reply-To: <20170603055742.GA19486@infradead.org>
On 06/03/2017 01:57 PM, Christoph Hellwig wrote:
> On Fri, Jun 02, 2017 at 09:29:26AM -0700, Shaohua Li wrote:
>> On Fri, Jun 02, 2017 at 03:49:14PM +0800, Guoqing Jiang wrote:
>>> bio_io_error was introduced in the commit 4246a0b
>>> ("block: add a bi_error field to struct bio"), so
>>> use it to simplify code.
>>>
>>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> Applied, thanks! There are several other places which could use conversion, I added
> Ugg. I have a massive series pending for the block layer that changes
> the error values. Applying these cleanups now will just introduce
> conflicts all over.
>
Hmm, I suppose you mean the thread with subject "dedicated error codes
for the block layer"
which replace bi_error with bi_status. I can resend it on top of the
thread, or maybe you can
add the changes in your patches directly.
Thanks,
Guoqing
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Paul Tonelli @ 2017-06-04 22:58 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170603232919.GA17108@metamorpher.de>
Hello again, things are somewhat improving, I haven't got my data but I
have got things to do:
On 06/04/2017 01:29 AM, Andreas Klauer wrote:
> Good catch. It would probably move the data offset.
>
> # truncate -s 3TB a b c
> # mdadm --create /dev/md42 --level=5 --raid-devices=2 /dev/loop[01]
> # mdadm --examine /dev/loop0
> Data Offset : 262144 sectors
> # mdadm --grow /dev/md42 --raid-devices=3 --add /dev/loop2
> # mdadm --examine /dev/loop0
> Data Offset : 262144 sectors
> New Offset : 260096 sectors
>
> So on re-create you have to find and specify the correct --data-offset.
>
> How to determine the correct data offset? See if you can find LVM magic
> string "LABELONE" in the first 256MiB of the two disks you didn't
> dd-overwrite. That minus 512 bytes should be the correct offset.
>
> # hexdump -C /dev/some-pv
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
> 00000210 55 87 20 ff 20 00 00 00 4c 56 4d 32 20 30 30 31 |U. . ...LVM2 001|
>
> If unlucky it just happened to be in the drive you overwrite.
> In that case you have to xor the two others.
I did find this one at that exact offset 26096 = (133169664 - 512)/512
on the sdc drive (I rebuilt the xor by rebuilding the raid, using
overlays this time, it speeds up things a lot compared to my previous tests)
so this matches the commands. Now, after doing a pvcreate and a
vgcfgrestore using the backup lvm file, e2fsck still refuses to run, but
testdisk now finds a lot more superblocks:
ext4 267262 8729457661 8729190400
ext4 791550 8729981949 8729190400
ext4 1315838 8730506237 8729190400
ext4 1840126 8731030525 8729190400
ext4 6034430 8735224829 8729190400
ext4 6558718 8735749117 8729190400
ext4 12325886 8741516285 8729190400
ext4 20714494 8749904893 8729190400
ext4 32248830 8761439229 8729190400
I also find the following backups superblocks when looping with e2fsck
-b XXX:
using testdisk:
```
33408
98944
164480
230016
...
```
and they always have a +640 difference with what a new ext4 I would
create on this volume doing:
```
mke2fs /dev/vg0/data -n:
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616,
78675968,
102400000, 214990848, 512000000, 550731776, 644972544
```
So I believe I still have an offset issue of 640 sectors or something
like this. Still digging this issue
I have tried to rebuild the ext4 superblock on top of the logical volume
with the "offset" option of mke2fs.ext4, but this did not work (e2fsck
is still not running by itself):
```
mke2fs -E offset=640 -n -S /dev/vg0/data
```
I am still digging around to know where this 640 offset comes from, but
as the last mail made me go forward significantly (thank you Andreas), I
am trying again.
thanks again for reading me to this point.
^ permalink raw reply
* 9 second recovery when re-adding a drive that got kicked out?
From: Marc MERLIN @ 2017-06-04 22:38 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Howdy,
Can you confirm that I understand how the write intent bitmap works, and
that it doesn't cover the entire array, but only a part of it, and once
you overflow it, syncing reverts to syncing the entire array?
I had a raid5 array with 5 6TB drives.
/dev/sdl1 got kicked out due to a bus disk error of some kind.
The drive is fine, it was a cabling issue, so I fixed the cabling,
re-added it, and did
gargamel:~# mdadm -a /dev/md6 /dev/sdl1
Then I saw this:
[ 1001.728134] md: recovery of RAID array md6
[ 1010.975255] md: md6: recovery done.
Before the re-add:
md6 : active raid5 sdk1[5] sdb1[3] sdm1[2] sdj1[1]
23441555456 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [_UUUU]
bitmap: 3/44 pages [12KB], 65536KB chunk
After the re-add (syncing now just to be safe):
md6 : active raid5 sdl1[0] sdj1[1] sdk1[5] sdf1[3] sdm1[2]
23441555456 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] check = 0.8% (49258960/5860388864) finish=569.3min speed=170093K/sec
bitmap: 0/44 pages [0KB], 65536KB chunk
https://raid.wiki.kernel.org/index.php/Mdstat
Explains a bit, I don't think it says how big a page is, but it seems to
be 4KB.
So let's say I have 64MB chuncks, each take 16 bits.
The whole array is 22,892,144MiB
That's 357,689 chunks, or about 700KB (16 bits per chunk) to keep all the
state, but there is 44 pages of 4KB, or 176KB of write intent
state.
The first bitmap line shows 3 pages totallying 12KB, so each page
contains 4KB, or 2048 chunks per page.
Did the above say that I had 6144 chunks that needed to be synced?
If so it would be 6144 * 65536KB = 393,216 MB to write
They were written in 9 seconds, so the sync happened at 43MB/s, which is
believeable.
The part I'm not too clear about is 44 pages of intent isn't enough to
cover all my data.
Is the idea that once I overflow that write intent bitmap, then it
reverts to resyncing the entire array?
I looked at https://raid.wiki.kernel.org/index.php/Write-intent_bitmap
but didn't see anything about that specific bit.
Array details if that helps:
gargamel:~# mdadm --examine /dev/sdl1
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 66bccdfb:afbf9683:fcf1f12e:f2af2dcb
Name : gargamel.svh.merlins.org:6 (local to host gargamel.svh.merlins.org)
Creation Time : Thu Jan 28 14:38:40 2016
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720777728 (5588.90 GiB 6001.04 GB)
Array Size : 23441555456 (22355.61 GiB 24004.15 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : ca4598ba:de585baa:b9935222:e06ac97d
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Jun 4 15:08:45 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d645f600 - correct
Events : 84917
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Andreas Klauer @ 2017-06-03 23:29 UTC (permalink / raw)
To: Paul Tonelli; +Cc: linux-raid
In-Reply-To: <12fc0496-423a-bca6-ac57-ed40958645e7@tonel.li>
On Sun, Jun 04, 2017 at 12:33:43AM +0200, Paul Tonelli wrote:
> I am still searching for a way to put a complete block device (/dev/sdX)
> read-only for these tests, I believe using overlays is the solution.
Yes.
Overlays are extremely useful for data recovery.
It is unfortunate there is no standard tool to manage them easily.
The "overlay functions" in the wiki come close but people don't find
out about it until it's too late.
> Actually, this is one question I have: with mdadm, creating a raid5 with
> two disks and then growing it to 3 creates exactly the same structure as
> creating directly a 3 disk raid5 ? Your message seems to say it is the
> same thing.
Good catch. It would probably move the data offset.
# truncate -s 3TB a b c
# mdadm --create /dev/md42 --level=5 --raid-devices=2 /dev/loop[01]
# mdadm --examine /dev/loop0
Data Offset : 262144 sectors
# mdadm --grow /dev/md42 --raid-devices=3 --add /dev/loop2
# mdadm --examine /dev/loop0
Data Offset : 262144 sectors
New Offset : 260096 sectors
So on re-create you have to find and specify the correct --data-offset.
How to determine the correct data offset? See if you can find LVM magic
string "LABELONE" in the first 256MiB of the two disks you didn't
dd-overwrite. That minus 512 bytes should be the correct offset.
# hexdump -C /dev/some-pv
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 55 87 20 ff 20 00 00 00 4c 56 4d 32 20 30 30 31 |U. . ...LVM2 001|
If unlucky it just happened to be in the drive you overwrite.
In that case you have to xor the two others.
> Is it possible to copy only a raid superblock from one disk to another
> directly using dd ?
Each superblock is unique (differs in device role and checksum at minimum).
So copying superblocks usually is not a thing. Even copying drives can
result in a mess (UUIDs are no longer unique, you have little / no control
which drive will actually be used). This is also a problem you might
encounter with overlays in conjunction with autoassembly/automount magicks
that might be running in the background.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Paul Tonelli @ 2017-06-03 22:33 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170603212011.GA13271@metamorpher.de>
Thank you for your answer and your time.
On 06/03/2017 11:20 PM, Andreas Klauer wrote:
> On Sat, Jun 03, 2017 at 09:46:44PM +0200, Paul Tonelli wrote:
>> I am trying to recover an ext4 partition on lvm2 on raid5.
> Okay, your mail is very long, still unclear in places.
>
> This was all done recently? So we do not have to consider that mdadm
> changed its defaults in regards to metadata versions, offsets, ...?
Correct, no change in version of mdadm, kernel, lvm or any other thing,
mdadm was installed on the machine on the day the raid was created and
it has not been upgraded since (I checked comparing the apt log
timestamps and the lvm metadata files).
> In that case I might have good news for you.
> Provided you didn't screw anything else up.
>
>> ```
>> mdadm --create --verbose --force --assume-clean /dev/md0 --level=5
>> --raid-devices=2 /dev/sdb /dev/sdc
>> ```
> You're not really supposed to do that.
> ( https://unix.stackexchange.com/a/131927/30851 )
I know that, now :-/. This was done before the backups.
>
>> I immediately made backups of the three disks to spares using dd
> This is a key point. If those backups are not good, you have lost.
I did backups (just after erasing the raid superblock ), and still have
them, I have been using them as a reference for all the later tests.
>> I made another mistake during the 3 days I spent trying to recover the
>> data, I switched two disks ids in a dd command and overwrite the first
>> 800Mb or so of disk c:
> Just to confirm, this is somehow not covered by your backups?
Right, this is not covered by my backups. i mistakenly copied from the
disks I was experimenting with to one of the backup and not the opposite
once (third mistake is working too late in the evening)
I am still searching for a way to put a complete block device (/dev/sdX)
read-only for these tests, I believe using overlays is the solution.
>> Part 2: What I tried
>> ====================
> In a data recovery situation there is one thing you should absolutely not do.
> That is writing to your disks. Please use overlays in the future...
> ( https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID#Making_the_harddisks_read-only_using_an_overlay_file )
Point taken, I had done the copy the whole disk way, will try this for
my next tests.
> Your experiments wrote all sorts of nonsense to your disks.
> As stated above, now it all depends on the backups you made...
Appart from the error just on top, I always used a copy of the original,
the originals are still available.
>> - one is missing ~120 GB at the start of the array, I have marked
>> this disk as missing for all my tests
> Maybe good news for you. Provided those backups are still there.
>
> If I understood your story correctly, then this disk has good data.
>
> RAID5 parity is a simple XOR. a XOR b = c
>
> You had a RAID 5 that was fully grown, fully synced.
Actually, this is one question I have: with mdadm, creating a raid5 with
two disks and then growing it to 3 creates exactly the same structure as
creating directly a 3 disk raid5 ? Your message seems to say it is the
same thing.
> You re-created it with the correct drives but wrong disk order.
> This started a sync.
>
> The sync should have done a XOR b = c (only c is written to disk c)
> Wrong order you did c XOR b = a (only a is written to disk a)
>
> It makes no difference. Either way it wrote the data that was already there.
> Merely the data representation (what you got from /dev/md0) was garbage.
>
> As long as you did not write anything to /dev/md0 when you couldn't mount,
> you're good right here. You just have to put the disks in correct order.
>
> Proof:
>
> --- Step 1: RAID Creation ---
>
> # truncate -s 100M a b c
> # losetup --find --show a
> /dev/loop0
> # losetup --find --show b
> /dev/loop1
> # losetup --find --show c
> /dev/loop2
> # mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop0 /dev/loop1 /dev/loop2
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md42 started.
> # mdadm --wait /dev/md42
> # mkfs.ext4 /dev/md42
> # mount /dev/md42 loop/
> # echo I am selling these fine leather jackets... > loop/somefile.txt
> # umount loop/
> # mdadm --stop /dev/md42
>
> --- Step 2: Foobaring it up (wrong disk order) ---
>
> # mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop2 /dev/loop1 /dev/loop0
> mdadm: /dev/loop2 appears to be part of a raid array:
> level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
> mdadm: /dev/loop1 appears to be part of a raid array:
> level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
> mdadm: /dev/loop0 appears to be part of a raid array:
> level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
> Continue creating array? yes
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md42 started.
> # mdadm --wait /dev/md42
> # mount /dev/md42 loop/
> mount: wrong fs type, bad option, bad superblock on /dev/md42,
> missing codepage or helper program, or other error
>
> In some cases useful info is found in syslog - try
> dmesg | tail or so.
> # mdadm --stop /dev/md42
>
> --- Step 3: Pulling the rabbit out of the hat (correct order, even one missing) ---
>
> # mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop0 missing /dev/loop2
> mdadm: /dev/loop0 appears to be part of a raid array:
> level=raid5 devices=3 ctime=Sat Jun 3 23:04:35 2017
> mdadm: /dev/loop2 appears to be part of a raid array:
> level=raid5 devices=3 ctime=Sat Jun 3 23:04:35 2017
> Continue creating array? yes
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md42 started.
> # mount /dev/md42 loop/
> # cat loop/somefile.txt
> I am selling these fine leather jackets...
Thanks you
>> - I am a the point where hiring somebody / a company with better
>> experience than mine to solve this issue is necessary. If yes who would
>> you advise, if this is an allowed question on the mailing list ?
> Oh. I guess should have asked for money first? Damn.
>
> Seriously though. I don't know if the above will solve your issue.
> It is certainly worth a try. And if it doesn't work it probably means
> something else happened... in that case chances of survival are low.
>
> Pictures (if they are small / unfragmented, with identifiable headers,
> i.e. JPEGs not RAWs) can be recovered but not their filenames / order.
for my case, losing even 20% of the pictures is not an issue, the
filename / order / directory tree is more important.
>
> Filesystem with first roughly 2GiB missing... filesystems _HATE_ that.
Thank you, so from what you told me, the next steps should be to :
- start using overlays as described in the wiki this will save me a lot
of time.
- use the correct disk (with only the raid superblock missing)
- use the disk which was partially xor-ed during the sync as this has
no impact on the data
- do not use the disk with the first GB missing
- try rebuilding the raid with these disks by doing all 6 combinations ?
I will try this tomorrow and update depending on the result.
I have gathered a second question from my unsuccessful tests and search:
Is it possible to copy only a raid superblock from one disk to another
directly using dd ? after reading on the wiki that the raid superblock
was 256 bytes long + 2 for each device, I tried:
```
dd if=/dev/sdx of=/dev/sdy count=262 iflag=count_bytes
```
but it did not copy the superblock correctly (mdadm did not find it),
There may be an offset or something missing.
Thank you again for your time, I will try this tomorrow after a good
night sleep. It will be less risky.
> Regards
> Andreas Klauer
^ permalink raw reply
* Re: Recovery after accidental raid5 superblock rewrite
From: Andreas Klauer @ 2017-06-03 21:20 UTC (permalink / raw)
To: Paul Tonelli; +Cc: linux-raid
In-Reply-To: <2009e689-39b7-14fa-44bd-3288f96b9c91@tonel.li>
On Sat, Jun 03, 2017 at 09:46:44PM +0200, Paul Tonelli wrote:
> I am trying to recover an ext4 partition on lvm2 on raid5.
Okay, your mail is very long, still unclear in places.
This was all done recently? So we do not have to consider that mdadm
changed its defaults in regards to metadata versions, offsets, ...?
In that case I might have good news for you.
Provided you didn't screw anything else up.
> ```
> mdadm --create --verbose --force --assume-clean /dev/md0 --level=5
> --raid-devices=2 /dev/sdb /dev/sdc
> ```
You're not really supposed to do that.
( https://unix.stackexchange.com/a/131927/30851 )
> I immediately made backups of the three disks to spares using dd
This is a key point. If those backups are not good, you have lost.
> I made another mistake during the 3 days I spent trying to recover the
> data, I switched two disks ids in a dd command and overwrite the first
> 800Mb or so of disk c:
Just to confirm, this is somehow not covered by your backups?
> Part 2: What I tried
> ====================
In a data recovery situation there is one thing you should absolutely not do.
That is writing to your disks. Please use overlays in the future...
( https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID#Making_the_harddisks_read-only_using_an_overlay_file )
Your experiments wrote all sorts of nonsense to your disks.
As stated above, now it all depends on the backups you made...
> - one is missing ~120 GB at the start of the array, I have marked
> this disk as missing for all my tests
Maybe good news for you. Provided those backups are still there.
If I understood your story correctly, then this disk has good data.
RAID5 parity is a simple XOR. a XOR b = c
You had a RAID 5 that was fully grown, fully synced.
You re-created it with the correct drives but wrong disk order.
This started a sync.
The sync should have done a XOR b = c (only c is written to disk c)
Wrong order you did c XOR b = a (only a is written to disk a)
It makes no difference. Either way it wrote the data that was already there.
Merely the data representation (what you got from /dev/md0) was garbage.
As long as you did not write anything to /dev/md0 when you couldn't mount,
you're good right here. You just have to put the disks in correct order.
Proof:
--- Step 1: RAID Creation ---
# truncate -s 100M a b c
# losetup --find --show a
/dev/loop0
# losetup --find --show b
/dev/loop1
# losetup --find --show c
/dev/loop2
# mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop0 /dev/loop1 /dev/loop2
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# mdadm --wait /dev/md42
# mkfs.ext4 /dev/md42
# mount /dev/md42 loop/
# echo I am selling these fine leather jackets... > loop/somefile.txt
# umount loop/
# mdadm --stop /dev/md42
--- Step 2: Foobaring it up (wrong disk order) ---
# mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop2 /dev/loop1 /dev/loop0
mdadm: /dev/loop2 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
mdadm: /dev/loop1 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
mdadm: /dev/loop0 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Jun 3 23:01:31 2017
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# mdadm --wait /dev/md42
# mount /dev/md42 loop/
mount: wrong fs type, bad option, bad superblock on /dev/md42,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
# mdadm --stop /dev/md42
--- Step 3: Pulling the rabbit out of the hat (correct order, even one missing) ---
# mdadm --create /dev/md42 --level=5 --raid-devices=3 /dev/loop0 missing /dev/loop2
mdadm: /dev/loop0 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Jun 3 23:04:35 2017
mdadm: /dev/loop2 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Jun 3 23:04:35 2017
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# mount /dev/md42 loop/
# cat loop/somefile.txt
I am selling these fine leather jackets...
> - I am a the point where hiring somebody / a company with better
> experience than mine to solve this issue is necessary. If yes who would
> you advise, if this is an allowed question on the mailing list ?
Oh. I guess should have asked for money first? Damn.
Seriously though. I don't know if the above will solve your issue.
It is certainly worth a try. And if it doesn't work it probably means
something else happened... in that case chances of survival are low.
Pictures (if they are small / unfragmented, with identifiable headers,
i.e. JPEGs not RAWs) can be recovered but not their filenames / order.
Filesystem with first roughly 2GiB missing... filesystems _HATE_ that.
Regards
Andreas Klauer
^ permalink raw reply
* Recovery after accidental raid5 superblock rewrite
From: Paul Tonelli @ 2017-06-03 19:46 UTC (permalink / raw)
To: linux-raid
Hello,
I am trying to recover an ext4 partition on lvm2 on raid5. After reading
to find solutions by myself, I tried to go on the freenod irc linux-raid
channel, who advised me to describe my problem here, so here I am.
The first part of the mail describes what led to my issue, the second
part is what I tried to solve it, the third is my current status, the
fourth are the questions. the mail can be read in markdown.
Part I: creation and loss of the array
=============================================
The raid is on 3 sata disks of 3Tb each. It was initialised as:
```
mdadm --create --verbose --force /dev/md0 --level=5 --raid-devices=2
/dev/sdb /dev/sdc
pvcreate /dev/md0
vgcreate vg0 /dev/md0
lvcreate -L 2.5T -n data /dev/vg0 #command guessed from the lvm archives
files
mkfs.ext4 /dev/vg0/data
```
the raid did not initialize correctly at each boot I had to rebuild the
disk using :
```
mdadm --create --verbose --force --assume-clean /dev/md0 --level=5
--raid-devices=2 /dev/sdb /dev/sdc
```
it would then mount without issue (autodetection of the vg and lv worked)
I then extended as follow to add a third disk, (hot plugged to the
system this has its importance). The raid had the time to grow and I was
able to extend everything on top of it:
```
sudo mdadm --add /dev/md0 /dev/sdd
sudo mdadm --grow --raid-devices=3 /dev/md0
sudo lvextend -L +256G /dev/mapper/vg0-data
sudo resize2fs /dev/vg0/data
sudo lvextend -L +256G /dev/mapper/vg0-data
sudo resize2fs /dev/vg0/data
sudo lvresize -L +500G /dev/vg0/data
sudo resize2fs /dev/vg0/data
```
Here the machine crashed for unrelated reasons
The data was not backupped: this was a transition situation where we
regrouped data from several machines and the backup nas was being setup
when this occured (this was the first mistake).
at reboot, I could not reassemble the raid and I did (this was the
second mistake, I had not read the wiki at this time):
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb
/dev/sdc /dev/sdd
I realized my error half an hour later when I could not detect any
volume group or mount anything and immediately stopped the rebuild of
drive sdd which was occuring (it was stopped <5%, so the first 5% of
disk sdd are now wrong).
Actually, between the reboots, the hard drive order had changed (because
the disk had been hotplugged initially), the most probable change is:
sdc became sdd
sdb became sdc
sdd became sdb
I immediately made backups of the three disks to spares using dd (to sde
sdf and sdg) and have been testing different methods to get back my data
ever since without success.
I made another mistake during the 3 days I spent trying to recover the
data, I switched two disks ids in a dd command and overwrite the first
800Mb or so of disk c:
```
dd if=/dev/sdc of=/dev/sdf bs=64k count=12500
```
The data contained on the disks is yml files, pictures (lots of it, with
a specific order) and binary files. Recovering of huge yml (Gb long) and
the structure of the filesystem are the most important data.
Part 2: What I tried
====================
The main test has been to rebuild the raid5 with the different possible
disk orders and try to detect data on it.
I tried several disk orders, restored the physical volume, volume group
and logical volume using:
```
mdadm --create --assume-clean --level=5 --raid-devices=3 /dev/md0
/dev/sdc missing /dev/sdb
pvcreate --uuid "Q2Z32D-iyPj-9QYp-uXBC-q02e-s8QK-6eqv4d" --restorefile
/home/ptonelli/backup_raid/lvm/backup/vg0 /dev/md0
vgcfgrestore vg0 -f /home/ptonelli/backup_raid/lvm/backup/vg0
vgchange -a a vg0
testdisk /dev/vg0/data
```
and did a deep scan of the disks and let it run until it reached 3%. I
got the following data (m for missing):
- sdd(m) sdb sdc : 258046 782334 1306622 1830910 2355198
- sdd(m) sdc sdb : 783358 1306622, 23562222
- sdb sdd(m) sdc : 258046 1307646 1830910
- sdc sdd(m) sdb : 259070 783358 1307646 1831834 235622
- sdb sdc sdd(m) : nothing detected
- sdc sdb sdd(m) : 259070 782334 1831934 235198
I wrote only the ext4 superblocks start position returned by testdisk),
the rest of the data was the same each time and matched the ext4
partition size I am trying to recover.
between each test, I restored the disks with the following method:
```
vgchange -a n vg0
mdadm --stop /dev/md0
dd if=/dev/sde of=/dev/sdb bs=64k count=125000
dd if=/dev/sdf of=/dev/sdc bs=64k count=125000
dd if=/dev/sdg of=/dev/sdd bs=64k count=125000
```
on the most promising orders ([sdc, missing, sdb] and [missing, sdb,
sdc]) i tried to rebuild the ext4 filesystem from the earliest
superblock using):
```
for i in $(seq 0 64000);do echo $i;e2fsck -b $i /dev/vg0/data;done
#and then
e2fsck -b 32XXX /dev/vg0/data -y
```
Each time the superblock was found aroud block 32000 , with a little
difference between the two attempts.
I let it run, it ran fixing/deleting... inodes for 3 hours (from the
output, one out of 10 inodes was modified during the repair), after 3
hours it was still at ~22 000 000 inodes so I guess the disk structure
is incorrect, I expected the repair to be a lot shorter with correct
structure.
I completely restored the disk between and after the tests with dd.
Part 3: current situation
=========================
So What I have:
- all three raid superblocks are screwed and were overwritten without
backup, but I have the commands used to build the initial backup
- I have all the incremental files for the lvm2 structure and the latest
file matches the ext4 superblocks found on the disks
- I have "nearly" complete backup of the three raid5 disks:
- one is good appart from the raid superblock (sdb)
- one is missing ~1 GB at the start (sdc)
- one is missing ~120 GB at the start of the array, I have marked
this disk as missing for all my tests
but I cannot find my data.
additional system info:
the machine is running with an amd64 debian jessie with backports
enabled, mdadm is the standard debian: v3.3.2 - 21st August 2014
I put here the relevant part of the lvm backup and archive files (I can
provide the full files if necessary)
before extension:
```
physical_volumes {
pv0 {
id = "Q2Z32D-iyPj-9QYp-uXBC-q02e-s8QK-6eqv4d"
device = "/dev/md0" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 5860270080 # 2.7289 Terabytes
pe_start = 2048
pe_count = 715364 # 2.7289 Terabytes
}
}
logical_volumes {
data {
id = "OwfU2H-UStb-fkaD-EAvk-fetk-CiOk-xkaWkA"
creation_time = 1494949403 # 2017-05-16 17:43:23 +0200
segment_count = 1
segment1 {
start_extent = 0
extent_count = 681575 # 2.6 Terabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv0", 0
]
```
after extension:
```
pv0 {
id = "Q2Z32D-iyPj-9QYp-uXBC-q02e-s8QK-6eqv4d"
device = "/dev/md0" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 11720538112 # 5.4578 Terabytes
pe_start = 2048
pe_count = 1430729 # 5.4578 Terabytes
}
}
logical_volumes {
data {
creation_time = 1494949403 # 2017-05-16 17:43:23 +0200
segment_count = 1
segment1 {
start_extent = 0
extent_count = 1065575 # 4.06485 Terabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv0", 0
]
```
from the raid wiki, I believe only this information is useful as the
raid superblocks are wrong:
```
PCI [ahci] 00:11.4 SATA controller: Intel Corporation Wellsburg sSATA
Controller [AHCI mode] (rev 05)
├scsi 0:0:0:0 ATA Crucial_CT1050MX {1651150EFB63}
│└sda 978.09g [8:0] Partitioned (gpt)
│ ├sda1 512.00m [8:1] vfat {5AB9-E482}
│ │└Mounted as /dev/sda1 @ /boot/efi
│ ├sda2 29.80g [8:2] ext4 {f8f9eb9a-fc49-4b2b-8c8c-27278dfc7f29}
│ │└Mounted as /dev/sda2 @ /
│ ├sda3 29.80g [8:3] swap {1ea1c6c1-7ec7-49cc-8696-f1fb8fb6e7b0}
│ └sda4 917.98g [8:4] PV LVM2_member 910.83g used, 7.15g free
{TJKWU2-oTcU-mSBC-sGHz-ZTg7-8HoY-u0Tyjj}
│ └VG vg_ssd 917.98g 7.15g free {hguqji-h777-K0yt-gjma-gEbO-HUfw-NU9aRK}
│ └redacted
├scsi 1:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N3EP81NC}
│└sdb 2.73t [8:16] Partitioned (gpt)
│ ├sdb1 2.37g [8:17] ext4 '1.42.6-5691'
{30ef58b3-1e3f-4f33-ade7-7365ebd8c427}
│ ├sdb2 2.00g [8:18] Empty/Unknown
│ └sdb3 2.72t [8:19] Empty/Unknown
├scsi 2:0:0:0 ATA WDC WD30EFRX-68E {WD-WMC4N1087039}
│└sdc 2.73t [8:32] Partitioned (gpt)
└scsi 3:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N3EP8C25}
└sdd 2.73t [8:48] Partitioned (gpt)
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port
SATA Controller [AHCI mode] (rev 05)
├scsi 4:x:x:x [Empty]
├scsi 5:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N0XARKSC}
│└sde 2.73t [8:64] Partitioned (gpt)
│ ├sde1 2.37g [8:65] ext4 '1.42.6-5691'
{30ef58b3-1e3f-4f33-ade7-7365ebd8c427}
│ ├sde2 2.00g [8:66] Empty/Unknown
│ └sde3 2.72t [8:67] Empty/Unknown
├scsi 6:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N7KPUH6U}
│└sdf 2.73t [8:80] Partitioned (gpt)
├scsi 7:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N1TYVTEN}
│└sdg 2.73t [8:96] Partitioned (gpt)
├scsi 8:x:x:x [Empty]
└scsi 9:x:x:x [Empty]
Other Block Devices
└md0 0.00k [9:0] MD vnone () clear, None (None) None {None}
Empty/Unknown
```
I am currently digging on the mailing list archive to find more
information and things to test.
Part 4: Questions
==================
- How much am I screwed ? Do you believe I can still get most of my data
back, what about the ext4 folder tree ?
- what should be my next steps (I would be happy to use any link to
relevant software/procedures).
- Is all the necessary information here or should I gather additional
information before continuing
- I am a the point where hiring somebody / a company with better
experience than mine to solve this issue is necessary. If yes who would
you advise, if this is an allowed question on the mailing list ?
Thank you for reading me down to this point, and thank you for your
answer if you can take the time to answer.
^ permalink raw reply
* Re: [PATCH] md: simplify code with bio_io_error
From: Christoph Hellwig @ 2017-06-03 5:57 UTC (permalink / raw)
To: Shaohua Li; +Cc: Guoqing Jiang, linux-raid
In-Reply-To: <20170602162926.onnaflcf26ahuoti@kernel.org>
On Fri, Jun 02, 2017 at 09:29:26AM -0700, Shaohua Li wrote:
> On Fri, Jun 02, 2017 at 03:49:14PM +0800, Guoqing Jiang wrote:
> > bio_io_error was introduced in the commit 4246a0b
> > ("block: add a bi_error field to struct bio"), so
> > use it to simplify code.
> >
> > Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>
> Applied, thanks! There are several other places which could use conversion, I added
Ugg. I have a massive series pending for the block layer that changes
the error values. Applying these cleanups now will just introduce
conflicts all over.
^ permalink raw reply
* Investment Interest & Offer
From: Seydou Thieba @ 2017-06-02 23:01 UTC (permalink / raw)
To: Recipients
^ permalink raw reply
* Re: Grow set size issue
From: Jes Sorensen @ 2017-06-02 17:48 UTC (permalink / raw)
To: Tomasz Majchrzak; +Cc: NeilBrown, linux-raid
In-Reply-To: <20170602110110.GA29246@proton.igk.intel.com>
On 06/02/2017 07:01 AM, Tomasz Majchrzak wrote:
> On Thu, Mar 30, 2017 at 10:10:32AM -0400, Jes Sorensen wrote:
>> On 03/29/2017 05:34 PM, NeilBrown wrote:
>>> On Thu, Mar 30 2017, jes.sorensen@gmail.com wrote:
>>>
>>>> Hi Neil,
>>>>
>>>> In the below patch you changed the error handling, to make the kernel
>>>> not setting the size of the device being an error. However we still have
>>>> the code in place to handle the error, except it never triggers.
>>>
>>> So we do. I should have removed all of that.
>>> I should have just reverted
>>> Commit: 65a9798b58b4 ("FIX: Detect error and rollback metadata")
>>>
>>>
>>>>
>>>> Question is do you remember the reason for this change? Old kernels not
>>>> allowing it, are there any legitimate reasons for the kernel to refuse
>>>> the size change?
>>>
>>> I needed to go further back to remind myself why we do these size change
>>> at all.
>>> The command being run here is "mdadm --grow /dev/mdX --size=foo"
>>> which has a primary purpose of changing the component_size of the array.
>>> What can happen is that someone makes all the components bigger
>>> (E.g. with LVM) and then uses this command to set --size=max, and it
>>> doesn't work. That is because md doesn't know the devices are bigger.
>>>
>>> You can tell md that devices have changes size by writing to the "size"
>>> attribute.
>>> mdadm doesn't have an option for doing that per-device - you need to
>>> poke into sysfs.
>>>
>>> To make it a bit easier, when you use "--grow --size=foo", mdadm will
>>> always write "foo" to the "size" attribute of each device, just incase
>>> that will be helpful. In the case where the device is now bigger, it
>>> will.
>>>
>>> In the case where the size of the array is being reduced, it is not
>>> permitted to change the "size" of each device until the "component_size"
>>> of the array has changed, so these attempts to change "size" will fail.
>>> But that isn't a problem.
>>>
>>> In short, the attempt to change "size" here is a convenience, and
>>> optimization. It doesn't matter if it fails.
>>>
>>> So please just revert all bits of the above commit that are still
>>> present.
>>
>> Hi Neil,
>>
>> Thanks for the explanation, I'll take the big hammer to the
>> leftovers and get rid of them.
>>
>> Jes
>
> Hi,
>
> The commit 758b327cf5a ("Grow: Remove unnecessary optimization") breaks grow
> operation for external metadata.
>
> mdadm --create /dev/md/imsm0 --metadata=imsm --raid-devices=4 /dev/nvme[0-3]n1 --run
> mdadm --create /dev/md/r5 --level=5 --chunk 64 --size=1G --raid-devices=3 /dev/nvme[0-2]n1 --run --assume-clean
>
> MDADM_EXPERIMENTAL=1 mdadm --grow /dev/md/r5 --size=2G
> mdadm: Cannot set device size for /dev/md/r5: No space left on device
>
> From update_size in MD driver:
>
> if (avail < num_sectors)
> -ENOSPC;
>
> In the above case 'avail' is set to old value (1GiB) and new size requested
> ('num_sectors') is 2GiB.
>
> Shall we revert the recent change then?
Rather than just reverting, track down whats wrong and fix it.
Jes
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox