* Re: Questions
From: Wols Lists @ 2016-02-15 22:09 UTC (permalink / raw)
To: o1bigtenor; +Cc: Adam Goryachev, Linux-RAID
In-Reply-To: <CAPpdf5-LRMk0pcsBrp_nwmrNmPKt5jeo0_zjbH5cMZqA0+LtRQ@mail.gmail.com>
On 15/02/16 21:01, o1bigtenor wrote:
> I have a present working array and with to replace its components with the
> same size but new drives (which are NAS rated). Was thinking that using
> the fail remove and add process 4 seperate times might not be a good thing
> but I do not know of a different option. Compounding the difficulty is
> that there
> are no empty hard drive slots in the machine. I do have an external USB 3.0
> 2 drive holder that could be used.
Does it have a spare PCI slot? That's what I was getting at - can you
add two more sata slots? Presumably not if it's a cage, but if it's a
computer case? As I said, a spare PCI SATA card should be dirt cheap to
add temporary extra capacity. And does it matter if the case is open and
the drives lying around temporarily?
>
> The only suggestion in all the documents I perused was to place spare drives
> into something like this external box and then add the drives into the array.
> The process was not laid out and leaves me with a number of questions.
>
> Is there a suggested method for replacing ALL the drives in an array (raid 10
> in this case)?
As far as I'm aware, there's just the "mdadm --replace" I mentioned -
drive by drive. Given that it's raid 10, maybe you can just add another
mirror then fail an old one.
I'd just plug in the extra drives, run mdadm --replace, and then remove
the old drives. Just make sure you get the right drives! And always use
uuids so you know which drive is which!
Get Phil's lsdrv and it will probably give you all the drives, with
serial numbers, etc etc. I haven't managed to run it so I can't be sure
:-) But it's intended to give you all the stuff you need to recover an
array so it should give you the information you need to rebuild it.
>
> If I use the external box how do I do this (external box only holds 2 drives) so
> that I can transfer the information on the drives from the array to
> the new drives
> and then just replace the drives 2 at a time into the machine without
> there being
> issues because in the information transfer the drives will be sdg and
> sdh (AFAIK)
> and later they will be some of sdb, sdc, sde, and/or sdf.
That's why they now have uuids.
ls /dev/disk/by-id
I *think* raid uses uuids internally. So swapping the drives out won't
be a problem - the sdx names are just used as a human-readable output.
But don't take that as gospel ... However, bear in mind that the kernel
does NOT guarantee that a drive will get the same sdx name from one boot
to the next. It so happens that that is the norm, but it's not
guaranteed ... so x changing for any value of sdx shouldn't be a problem.
Regardless, you should not be using sdx. Everything should be using
uuids, my /etc/fstab is a lovely mangle with all those long uuids
everywhere :-)
Cheers,
Wol
^ permalink raw reply
* Re: Questions
From: Adam Goryachev @ 2016-02-15 22:05 UTC (permalink / raw)
To: o1bigtenor, Wols Lists; +Cc: Linux-RAID
In-Reply-To: <CAPpdf5-LRMk0pcsBrp_nwmrNmPKt5jeo0_zjbH5cMZqA0+LtRQ@mail.gmail.com>
On 16/02/16 08:01, o1bigtenor wrote:
> On Mon, Feb 15, 2016 at 1:50 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 15/02/16 12:12, o1bigtenor wrote:
>>> I am looking at replacing ALL the drives in the array so that I can
>>> reduce the likelihood of these kind of issues for a longer period than
>>> a few months.
>>>
>>> Would you be able to tell me what steps should I be taking to replace
>>> the entire array?
>>>
>>> Should I replace the drives one at a time (sort of just like I did
>>> this time) using the same commands?
>> If you have a spare SATA (I presume they're sata drives) slot, then
>> definitely not. It might be worth buying an add-in card to give you an
>> extra slot, they're only a few quid.
>>
>> Look up "mdadm --replace". That'll keep the old drive in the array until
>> the new one has replaced it, keeping the array fully functional all the
>> time.
>>
>> Doing --fail, --remove, --add places the array in danger until the
>> entire sequence is complete, and if you're doing it several times then
>> you're massively increasing the chances of trouble.
>>
>> In fact, if you get an add-in card, you might be able to replace several
>> drives at once - I don't know - read up and see if it's supported (or not).
> Looked through well over a hundred different pages and only found 1 page
> that even hinted at what I am trying to do and details there were sparse.
>
> Read through the man page (likely one of the best for understandability that
> I have found so far) and it doesn't address the wholesale replacement of all
> the drives in an array. Most of the other pages that I found were what I would
> call old, ie before 2010, as I have found that all too often the software being
> discussed has changed and then the instructions given are not always useful,
> occasionally there is some value in these old pages but only sometimes.
>
> Most often the documentation was focused on replacing a faulty disc.
>
> This is NOT what I am proposing to do.
>
> I have a present working array and with to replace its components with the
> same size but new drives (which are NAS rated). Was thinking that using
I wanted to mention this, what drives do you have right now, and do know
about SCT ERC?
Maybe start here (but you probably need to read more):
http://www.spinics.net/lists/raid/msg48199.html
Essentially, your current disks might be fine, but if you don't have the
right settings, they could be "failing" regularly putting your data at
risk. You should fix any issue here before you attempt to replace your
drives.
> the fail remove and add process 4 seperate times might not be a good thing
> but I do not know of a different option. Compounding the difficulty is
> that there
> are no empty hard drive slots in the machine. I do have an external USB 3.0
> 2 drive holder that could be used.
>
> The only suggestion in all the documents I perused was to place spare drives
> into something like this external box and then add the drives into the array.
> The process was not laid out and leaves me with a number of questions.
>
> Is there a suggested method for replacing ALL the drives in an array (raid 10
> in this case)?
In order to replace all drives, I would suggest that you simply replace
one drive 4 times (different drive each time).
The first question to ask, is your external USB drive bay reliable? If
not, then there are other solutions that are probably less dangerous.
So, add your spare drive to the external USB drive bay, it should show
up as /dev/sdy (for example)
Partition to match the rest of your existing drives
Add the new partition to your existing array: mdadm --manage /dev/md0
--add /dev/sdx1
Replace one of the existing drives with the new one: mdadm /dev/md0
--replace /dev/sda1 --with /dev/sdx1
Personally, because I distrust the external USB drive bay (don't ask me
why, it just seems less reliable than internal sata), once the drive has
finished being replaced, I would shutdown, remove the old drive, and
install the replacement drive, then add another new spare, and repeat.
You can see this page for some extra information:
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
> If I use the external box how do I do this (external box only holds 2 drives) so
> that I can transfer the information on the drives from the array to
> the new drives
> and then just replace the drives 2 at a time into the machine without
> there being
> issues because in the information transfer the drives will be sdg and
> sdh (AFAIK)
> and later they will be some of sdb, sdc, sde, and/or sdf.
I would suggest replacing one at a time.
> If one of the gurus on the list has an already prepared process list
> with the steps
> and commands that would be wonderful. I am thinking that this document would
> very much be appreciated by many others out there that are not on this list.
>
> TIA
>
>
Hope that helps.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-15 21:01 UTC (permalink / raw)
To: Wols Lists; +Cc: Adam Goryachev, Linux-RAID
In-Reply-To: <56C22C20.9090002@youngman.org.uk>
On Mon, Feb 15, 2016 at 1:50 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 15/02/16 12:12, o1bigtenor wrote:
>> I am looking at replacing ALL the drives in the array so that I can
>> reduce the likelihood of these kind of issues for a longer period than
>> a few months.
>>
>> Would you be able to tell me what steps should I be taking to replace
>> the entire array?
>>
>> Should I replace the drives one at a time (sort of just like I did
>> this time) using the same commands?
>
> If you have a spare SATA (I presume they're sata drives) slot, then
> definitely not. It might be worth buying an add-in card to give you an
> extra slot, they're only a few quid.
>
> Look up "mdadm --replace". That'll keep the old drive in the array until
> the new one has replaced it, keeping the array fully functional all the
> time.
>
> Doing --fail, --remove, --add places the array in danger until the
> entire sequence is complete, and if you're doing it several times then
> you're massively increasing the chances of trouble.
>
> In fact, if you get an add-in card, you might be able to replace several
> drives at once - I don't know - read up and see if it's supported (or not).
Looked through well over a hundred different pages and only found 1 page
that even hinted at what I am trying to do and details there were sparse.
Read through the man page (likely one of the best for understandability that
I have found so far) and it doesn't address the wholesale replacement of all
the drives in an array. Most of the other pages that I found were what I would
call old, ie before 2010, as I have found that all too often the software being
discussed has changed and then the instructions given are not always useful,
occasionally there is some value in these old pages but only sometimes.
Most often the documentation was focused on replacing a faulty disc.
This is NOT what I am proposing to do.
I have a present working array and with to replace its components with the
same size but new drives (which are NAS rated). Was thinking that using
the fail remove and add process 4 seperate times might not be a good thing
but I do not know of a different option. Compounding the difficulty is
that there
are no empty hard drive slots in the machine. I do have an external USB 3.0
2 drive holder that could be used.
The only suggestion in all the documents I perused was to place spare drives
into something like this external box and then add the drives into the array.
The process was not laid out and leaves me with a number of questions.
Is there a suggested method for replacing ALL the drives in an array (raid 10
in this case)?
If I use the external box how do I do this (external box only holds 2 drives) so
that I can transfer the information on the drives from the array to
the new drives
and then just replace the drives 2 at a time into the machine without
there being
issues because in the information transfer the drives will be sdg and
sdh (AFAIK)
and later they will be some of sdb, sdc, sde, and/or sdf.
If one of the gurus on the list has an already prepared process list
with the steps
and commands that would be wonderful. I am thinking that this document would
very much be appreciated by many others out there that are not on this list.
TIA
Dee
^ permalink raw reply
* Re: Questions
From: Wols Lists @ 2016-02-15 19:50 UTC (permalink / raw)
To: o1bigtenor, Adam Goryachev; +Cc: Linux-RAID
In-Reply-To: <CAPpdf59u6SPD3=sv1XmKR09jzt6OL8TYLgOj==i3G4775fqbhg@mail.gmail.com>
On 15/02/16 12:12, o1bigtenor wrote:
> I am looking at replacing ALL the drives in the array so that I can
> reduce the likelihood of these kind of issues for a longer period than
> a few months.
>
> Would you be able to tell me what steps should I be taking to replace
> the entire array?
>
> Should I replace the drives one at a time (sort of just like I did
> this time) using the same commands?
If you have a spare SATA (I presume they're sata drives) slot, then
definitely not. It might be worth buying an add-in card to give you an
extra slot, they're only a few quid.
Look up "mdadm --replace". That'll keep the old drive in the array until
the new one has replaced it, keeping the array fully functional all the
time.
Doing --fail, --remove, --add places the array in danger until the
entire sequence is complete, and if you're doing it several times then
you're massively increasing the chances of trouble.
In fact, if you get an add-in card, you might be able to replace several
drives at once - I don't know - read up and see if it's supported (or not).
Cheers,
Wol
^ permalink raw reply
* swiotlb buffer is full
From: Wolfgang Denk @ 2016-02-15 12:25 UTC (permalink / raw)
To: linux-raid
Hello,
first, I would like to apologize in advance if I should be off topic
here. Actually I'm not really sure which kernel component causes
problems here, but as it gets triggered by a MD task I thought I start
asking here...
The problem is, that the system more or less reliably crashes with
"swiotlb buffer is full" errors when a data-check on the RAID arrays
is run.
The RAID configuration looks like this:
-> cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdf3[2] sdg3[3]
970206800 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdf1[0] sdg1[1]
262132 blocks super 1.0 [2/2] [UU]
md3 : active raid1 sdc[1] sdb[0]
117219728 blocks super 1.2 [2/2] [UU]
md2 : active raid6 sdk[5] sdi[3] sdj[4] sdh[2] sde[1] sdd[0]
3907049792 blocks super 1.2 level 6, 16k chunk, algorithm 2 [6/6] [UUUUUU]
unused devices: <none>
What happens is always the same: a cron job will trigger a data-check
on the raid arrays, and then it goes like this:
Jan 3 04:00:01 castor kernel: md: data-check of RAID array md1
Jan 3 04:00:01 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 3 04:00:01 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jan 3 04:00:01 castor kernel: md: using 128k window, over a total of 970206800k.
Jan 3 04:00:08 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
Jan 3 04:00:14 castor kernel: md: data-check of RAID array md3
Jan 3 04:00:14 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 3 04:00:14 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jan 3 04:00:14 castor kernel: md: using 128k window, over a total of 117219728k.
Jan 3 04:00:20 castor kernel: md: data-check of RAID array md2
Jan 3 04:00:20 castor kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 3 04:00:20 castor kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jan 3 04:00:20 castor kernel: md: using 128k window, over a total of 976762448k.
Jan 3 04:23:22 castor kernel: md: md3: data-check done.
Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
Jan 3 04:23:22 castor kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
Jan 3 04:57:45 castor kernel: sata_via 0000:03:06.0: swiotlb buffer is full (sz: 16384 bytes)
Jan 3 04:57:45 castor kernel: DMA: Out of SW-IOMMU space for 16384 bytes at device 0000:03:06.0
Jan 3 04:57:45 castor kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jan 3 04:57:45 castor kernel: ata4.00: failed command: READ DMA EXT
Jan 3 04:57:45 castor kernel: ata4.00: cmd 25/00:80:80:df:42/00:05:0d:00:00/e0 tag 0 dma 720896 in#012 res 50/00:00:7f:df:42/00:00:0d:00:00/e0 Emask 0x40 (internal error)
Jan 3 04:57:45 castor kernel: ata4.00: status: { DRDY }
Jan 3 04:57:45 castor kernel: ata4.00: configured for UDMA/133
Jan 3 04:57:45 castor kernel: ata4: EH complete
Jan 3 04:58:06 castor kernel: sata_nv 0000:00:05.0: swiotlb buffer is full (sz: 4096 bytes)
Jan 3 04:58:06 castor kernel: DMA: Out of SW-IOMMU space for 4096 bytes at device 0000:00:05.0
Jan 3 04:58:06 castor kernel: sata_via 0000:03:06.0: swiotlb buffer is full (sz: 16384 bytes)
Jan 3 04:58:06 castor kernel: DMA: Out of SW-IOMMU space for 16384 bytes at device 0000:03:06.0
Jan 3 04:58:06 castor kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jan 3 04:58:06 castor kernel: ata4.00: failed command: READ DMA EXT
Jan 3 04:58:06 castor kernel: ata4.00: cmd 25/00:80:00:d4:5c/00:06:0d:00:00/e0 tag 0 dma 851968 in#012 res 50/00:00:ff:d3:5c/00:00:0d:00:00/e0 Emask 0x40 (internal error)
Jan 3 04:58:06 castor kernel: ata4.00: status: { DRDY }
Jan 3 04:58:06 castor kernel: ata4.00: configured for UDMA/133
Jan 3 04:58:06 castor kernel: ata15: EH in SWNCQ mode,QC:qc_active 0x10000 sactive 0x10000
Jan 3 04:58:06 castor kernel: ata15: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0x11#012 dhfis 0x0 dmafis 0x0 sdbfis 0x0
Jan 3 04:58:06 castor kernel: ata15: ATA_REG 0x40 ERR_REG 0x0
Jan 3 04:58:06 castor kernel: ata15: tag : dhfis dmafis sdbfis sactive
Jan 3 04:58:06 castor kernel: ata15.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x6
Jan 3 04:58:06 castor kernel: ata15.00: failed command: READ FPDMA QUEUED
Jan 3 04:58:06 castor kernel: ata15.00: cmd 60/00:80:f0:16:85/04:00:1d:00:00/40 tag 16 ncq 524288 in#012 res 40/00:20:f0:e6:84/00:00:1d:00:00/40 Emask 0x40 (internal error)
Jan 3 04:58:06 castor kernel: ata15.00: status: { DRDY }
Jan 3 04:58:06 castor kernel: ata15: hard resetting link
Jan 3 04:58:06 castor kernel: ata15: nv: skipping hardreset on occupied port
Jan 3 04:58:06 castor kernel: ata4: EH complete
Jan 3 04:58:06 castor kernel: ata15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jan 3 04:58:06 castor kernel: ata15.00: configured for UDMA/133
Jan 3 04:58:06 castor kernel: ata15: EH complete
Jan 3 05:02:49 castor kernel: sata_nv 0000:00:05.0: swiotlb buffer is full (sz: 4096 bytes)
Jan 3 05:02:50 castor kernel: DMA: Out of SW-IOMMU space for 4096 bytes at device 0000:00:05.0
...
From here the system is basically dead: it will report I/O errors on
the root file system, remount root read-only, and drop /dev/sdf3 from
RAID array /dev/md1. I have to reboot - the root file system is
corrupted so I have to run fsck manually to repair it, and I have to
re-add /dev/sdf3 to /dev/md1 and resync.
This is _exactly_ the same in all cases. For reference, full kernel
logs of 5 such crashes are available at [1]
[1] https://www.amazon.com/clouddrive/share/qZjRneB0tA5TXrNqBQVzyhN6Hy8HzxpLHCKHIhfzYyk
The system has been working fine for years, up to and including
Fedora 22. The crashes started happening after I upgraded to Fedora
23 over Xmas. I've installed all available updates sinde, with no
avails. Current configuration looks like this:
kernel-4.3.4-300.fc23.x86_64
mdadm-3.3.4-2.fc23.x86_64
As the first error reported is always "swiotlb buffer is full", I
tried to add "swiotlb=32768" to the kernel command line, but this
does not appear to make any effect, as I still see in the kernel
messages:
[ 1.518575] software IO TLB [mem 0xdbff0000-0xdfff0000] (64MB) mapped at [ffff8800dbff0000-ffff8800dffeffff]
[I think the software IOMMU implementation is used because this is a
Dual-Core AMD Opteron Processor 2216 on a Supermicro H8DM8-2 main
board; I believe this does not support an IOMMU ?]
Has anybody any ideas what might cause such an effect?
I think it is interesting that always the same RAID array gets
kicked, and always the same disk. I cannot see any hardware
problems, and a preventive replacement of the disk drive did not fix
the problem.
What else could I do or try?
Thanks in advance.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It is dangerous to be right on a subject on which the established
authorities are wrong. -- Voltaire
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-15 12:12 UTC (permalink / raw)
To: Adam Goryachev; +Cc: Linux-RAID
In-Reply-To: <AE031459-5104-437F-8353-ACF855CB02CC@websitemanagers.com.au>
Greetings
On Sun, Feb 14, 2016 at 6:24 AM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> On 14 February 2016 10:53:48 pm AEDT, o1bigtenor <o1bigtenor@gmail.com> wrote:
>>On Sun, Feb 14, 2016 at 12:34 AM, Adam Goryachev
>><mailinglists@websitemanagers.com.au> wrote:
>>>
>>>
>>> On 14/02/2016 15:28, o1bigtenor wrote:
>>>>
>>>> Greetings
>>>>
>>>> My raid 10 array was the subject of a number of exchanges on this
>>>> board a few months ago.
>>>> With the generous assistance of members here things were
>>reestablished
>>>> and have been working well. Today I had a VirtualBox VM crater and
>>in
>>>> the process cause other system issues. In process to clear the mess
>>a
>>>> number of hard stops (shutting the system off using the button on
>>the
>>>> case) were used. In rebooting I found that one of the drives in the
>>>> array is no longer responding issuing a number of clicks in the boot
>>>> up process with nothing else happening. Even though it is a RAID 10
>>>> array the array is no longer mounted nor available. I have removed
>>the
>>>> faulty drive already. I have an appropriately sized drive available
>>>> that I could place into the machine.
>>>>
>>>> 1. should I reformat the drive (to be placed into the machine)?
>>>> 2. what sequence of commands should I be using for this new drive to
>>>> be included into the array?
>>>> 3. what sequence of commands should I use to remount the array?
>>>
>>>
>>> First thing I would suggest is to let everyone know the status of the
>>> current array, and how to get it working.
>>>
>>> Can you send the output of cat /proc/mdstat and mdadm --misc --detail
>>> /dev/md?
>>
>># cat /proc/mdstat
>>Personalities : [raid10]
>>md0 : active (auto-read-only) raid10 sde1[5] sdc1[4] sdb1[3]
>> 1953518592 blocks super 1.2 512K chunks 2 near-copies [4/3] [U_UU]
>>
>>unused devices: <none>
>>
>># mdadm --misc --detail /dev/md
>>mdadm: /dev/md does not appear to be an md device
>>
>># mdadm --misc --detail /dev/md/0
>>/dev/md/0:
>> Version : 1.2
>> Creation Time : Mon Mar 5 08:26:28 2012
>> Raid Level : raid10
>> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>> Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
>> Raid Devices : 4
>> Total Devices : 3
>> Persistence : Superblock is persistent
>>
>> Update Time : Sat Feb 13 17:21:51 2016
>> State : clean, degraded
>> Active Devices : 3
>>Working Devices : 3
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : near=2
>> Chunk Size : 512K
>>
>> Name : debianbase:0 (local to host debianbase)
>> UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>> Events : 60241
>>
>> Number Major Minor RaidDevice State
>> 5 8 65 0 active sync set-A /dev/sde1
>> 2 0 0 2 removed
>> 4 8 33 2 active sync set-A /dev/sdc1
>> 3 8 17 3 active sync set-B /dev/sdb1
>>
>>
>>>
>>> Assuming the existing array is in a "normal" status, albeit degraded,
>>then
>>> it should be pretty simple to just partition the new drive to match
>>the
>>> other members, and then simply add the new partition to the array
>>(mdadm
>>> --manage /dev/md? --add /dev/sdxy)
>>
>>Drive I wish to use for replacement has had some use.
>>Should I be reformatting it?
>>
>
> Not needed, the resync will overwrite the content. You just need to partition it the same as the other drive members.
>
> Then you can simply add it to the array and it will sync.
>
> Also, you should have access to your stay content already if you just mount it (assuming it contains a file system )
>
> Let us know if you need any more help.
>
Thank you to Mr Adam for his assistance!
Installed the drive and, with a wait time for results, everything is
working very well.
I am looking at replacing ALL the drives in the array so that I can
reduce the likelihood of these kind of issues for a longer period than
a few months.
Would you be able to tell me what steps should I be taking to replace
the entire array?
Should I replace the drives one at a time (sort of just like I did
this time) using the same commands?
If so is there an easy way of mounting the array?
Regards
Dee
^ permalink raw reply
* Re: Questions
From: Adam Goryachev @ 2016-02-14 12:24 UTC (permalink / raw)
To: o1bigtenor; +Cc: Linux-RAID
In-Reply-To: <CAPpdf5_JBJB19B7wG5iuTt_aDZ6Zg0=kgQJtR_REU9JfDN1CxA@mail.gmail.com>
On 14 February 2016 10:53:48 pm AEDT, o1bigtenor <o1bigtenor@gmail.com> wrote:
>On Sun, Feb 14, 2016 at 12:34 AM, Adam Goryachev
><mailinglists@websitemanagers.com.au> wrote:
>>
>>
>> On 14/02/2016 15:28, o1bigtenor wrote:
>>>
>>> Greetings
>>>
>>> My raid 10 array was the subject of a number of exchanges on this
>>> board a few months ago.
>>> With the generous assistance of members here things were
>reestablished
>>> and have been working well. Today I had a VirtualBox VM crater and
>in
>>> the process cause other system issues. In process to clear the mess
>a
>>> number of hard stops (shutting the system off using the button on
>the
>>> case) were used. In rebooting I found that one of the drives in the
>>> array is no longer responding issuing a number of clicks in the boot
>>> up process with nothing else happening. Even though it is a RAID 10
>>> array the array is no longer mounted nor available. I have removed
>the
>>> faulty drive already. I have an appropriately sized drive available
>>> that I could place into the machine.
>>>
>>> 1. should I reformat the drive (to be placed into the machine)?
>>> 2. what sequence of commands should I be using for this new drive to
>>> be included into the array?
>>> 3. what sequence of commands should I use to remount the array?
>>
>>
>> First thing I would suggest is to let everyone know the status of the
>> current array, and how to get it working.
>>
>> Can you send the output of cat /proc/mdstat and mdadm --misc --detail
>> /dev/md?
>
># cat /proc/mdstat
>Personalities : [raid10]
>md0 : active (auto-read-only) raid10 sde1[5] sdc1[4] sdb1[3]
> 1953518592 blocks super 1.2 512K chunks 2 near-copies [4/3] [U_UU]
>
>unused devices: <none>
>
># mdadm --misc --detail /dev/md
>mdadm: /dev/md does not appear to be an md device
>
># mdadm --misc --detail /dev/md/0
>/dev/md/0:
> Version : 1.2
> Creation Time : Mon Mar 5 08:26:28 2012
> Raid Level : raid10
> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
> Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
> Raid Devices : 4
> Total Devices : 3
> Persistence : Superblock is persistent
>
> Update Time : Sat Feb 13 17:21:51 2016
> State : clean, degraded
> Active Devices : 3
>Working Devices : 3
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : near=2
> Chunk Size : 512K
>
> Name : debianbase:0 (local to host debianbase)
> UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
> Events : 60241
>
> Number Major Minor RaidDevice State
> 5 8 65 0 active sync set-A /dev/sde1
> 2 0 0 2 removed
> 4 8 33 2 active sync set-A /dev/sdc1
> 3 8 17 3 active sync set-B /dev/sdb1
>
>
>>
>> Assuming the existing array is in a "normal" status, albeit degraded,
>then
>> it should be pretty simple to just partition the new drive to match
>the
>> other members, and then simply add the new partition to the array
>(mdadm
>> --manage /dev/md? --add /dev/sdxy)
>
>Drive I wish to use for replacement has had some use.
>Should I be reformatting it?
>
Not needed, the resync will overwrite the content. You just need to partition it the same as the other drive members.
Then you can simply add it to the array and it will sync.
Also, you should have access to your stay content already if you just mount it (assuming it contains a file system )
Let us know if you need any more help.
Regards
Adam
^ permalink raw reply
* Re: Questions
From: o1bigtenor @ 2016-02-14 11:53 UTC (permalink / raw)
To: Adam Goryachev; +Cc: Linux-RAID
In-Reply-To: <56C0200B.8030606@websitemanagers.com.au>
On Sun, Feb 14, 2016 at 12:34 AM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
>
> On 14/02/2016 15:28, o1bigtenor wrote:
>>
>> Greetings
>>
>> My raid 10 array was the subject of a number of exchanges on this
>> board a few months ago.
>> With the generous assistance of members here things were reestablished
>> and have been working well. Today I had a VirtualBox VM crater and in
>> the process cause other system issues. In process to clear the mess a
>> number of hard stops (shutting the system off using the button on the
>> case) were used. In rebooting I found that one of the drives in the
>> array is no longer responding issuing a number of clicks in the boot
>> up process with nothing else happening. Even though it is a RAID 10
>> array the array is no longer mounted nor available. I have removed the
>> faulty drive already. I have an appropriately sized drive available
>> that I could place into the machine.
>>
>> 1. should I reformat the drive (to be placed into the machine)?
>> 2. what sequence of commands should I be using for this new drive to
>> be included into the array?
>> 3. what sequence of commands should I use to remount the array?
>
>
> First thing I would suggest is to let everyone know the status of the
> current array, and how to get it working.
>
> Can you send the output of cat /proc/mdstat and mdadm --misc --detail
> /dev/md?
# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[5] sdc1[4] sdb1[3]
1953518592 blocks super 1.2 512K chunks 2 near-copies [4/3] [U_UU]
unused devices: <none>
# mdadm --misc --detail /dev/md
mdadm: /dev/md does not appear to be an md device
# mdadm --misc --detail /dev/md/0
/dev/md/0:
Version : 1.2
Creation Time : Mon Mar 5 08:26:28 2012
Raid Level : raid10
Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sat Feb 13 17:21:51 2016
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Name : debianbase:0 (local to host debianbase)
UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Events : 60241
Number Major Minor RaidDevice State
5 8 65 0 active sync set-A /dev/sde1
2 0 0 2 removed
4 8 33 2 active sync set-A /dev/sdc1
3 8 17 3 active sync set-B /dev/sdb1
>
> Assuming the existing array is in a "normal" status, albeit degraded, then
> it should be pretty simple to just partition the new drive to match the
> other members, and then simply add the new partition to the array (mdadm
> --manage /dev/md? --add /dev/sdxy)
Drive I wish to use for replacement has had some use.
Should I be reformatting it?
Thank you Adam!
Regards
Dee
^ permalink raw reply
* Re: Questions
From: Adam Goryachev @ 2016-02-14 6:34 UTC (permalink / raw)
To: o1bigtenor, Linux-RAID
In-Reply-To: <CAPpdf58O+=Re1dBe1Rmj7jb=9iJkMZZBuZfEX4S32wMXcv6uWQ@mail.gmail.com>
On 14/02/2016 15:28, o1bigtenor wrote:
> Greetings
>
> My raid 10 array was the subject of a number of exchanges on this
> board a few months ago.
> With the generous assistance of members here things were reestablished
> and have been working well. Today I had a VirtualBox VM crater and in
> the process cause other system issues. In process to clear the mess a
> number of hard stops (shutting the system off using the button on the
> case) were used. In rebooting I found that one of the drives in the
> array is no longer responding issuing a number of clicks in the boot
> up process with nothing else happening. Even though it is a RAID 10
> array the array is no longer mounted nor available. I have removed the
> faulty drive already. I have an appropriately sized drive available
> that I could place into the machine.
>
> 1. should I reformat the drive (to be placed into the machine)?
> 2. what sequence of commands should I be using for this new drive to
> be included into the array?
> 3. what sequence of commands should I use to remount the array?
First thing I would suggest is to let everyone know the status of the
current array, and how to get it working.
Can you send the output of cat /proc/mdstat and mdadm --misc --detail
/dev/md?
Assuming the existing array is in a "normal" status, albeit degraded,
then it should be pretty simple to just partition the new drive to match
the other members, and then simply add the new partition to the array
(mdadm --manage /dev/md? --add /dev/sdxy)
Try to take things slowly, as doing the wrong thing might make a simple
recovery into a very sad event (loss of all the data).
Regards,
Adam
Regards,
Adam
^ permalink raw reply
* Questions
From: o1bigtenor @ 2016-02-14 4:28 UTC (permalink / raw)
To: Linux-RAID
Greetings
My raid 10 array was the subject of a number of exchanges on this
board a few months ago.
With the generous assistance of members here things were reestablished
and have been working well. Today I had a VirtualBox VM crater and in
the process cause other system issues. In process to clear the mess a
number of hard stops (shutting the system off using the button on the
case) were used. In rebooting I found that one of the drives in the
array is no longer responding issuing a number of clicks in the boot
up process with nothing else happening. Even though it is a RAID 10
array the array is no longer mounted nor available. I have removed the
faulty drive already. I have an appropriately sized drive available
that I could place into the machine.
1. should I reformat the drive (to be placed into the machine)?
2. what sequence of commands should I be using for this new drive to
be included into the array?
3. what sequence of commands should I use to remount the array?
TIA
Dee
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: John Stoffel @ 2016-02-12 19:57 UTC (permalink / raw)
To: Marc MERLIN; +Cc: Jes Sorensen, Mikael Abrahamsson, linux-raid
In-Reply-To: <20160212190700.GQ13969@merlins.org>
Marc,
Did you look into lvcache instead of bcache? The problem with bcache
is that you can't (as I understand it) remove it from a live volume,
while you can with lvache.
John
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Marc MERLIN @ 2016-02-12 19:07 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Mikael Abrahamsson, linux-raid
In-Reply-To: <wrfja8n62e0c.fsf@redhat.com>
On Fri, Feb 12, 2016 at 09:34:59AM -0500, Jes Sorensen wrote:
> Mikael Abrahamsson <swmike@swm.pp.se> writes:
> > On Thu, 11 Feb 2016, Marc MERLIN wrote:
> >
> >> But can someone confirm that indeed if I have a single md device,
> >> and 4 CPUs, my throughput is not limited by a single CPU decrypting
> >> the device?
> >
> > There have been multiple discussions about this over time, and since
> > you didn't qualify "not good" with a number, it's hard to say if this
> > is your problem or not.
>
> About a year ago when I last received a new laptop, I put btrfs on top
> of luks - it was a rather miserable experience. The system would freeze
> up and be really slow.
>
> I believe the issue was btrfs not acting well with crypto underneath it,
> and there was hallway talk confirming this but I have no hard data. In
> the end I gave up and put my /home on xfs on top of LUKS and used btrfs
> for the uncrypted portions.
>
> It may be the btrfs with crypto issues have been resolved since then.
Thanks all for the feedback and experiences.
"not good" is indeed subjective, I can see the kernel hanging for a few
seconds at a time under heavy I/O and cron jobs taking too long.
I'm indeed going to just rebuild the FS with a new layout, and try
bcache in the process.
I still need to figure out what I should use as best chunk size for
mdadm, whether this is reasonable:
cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain
and see if there are tuning parameters in bcache I should worry about.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Jes Sorensen @ 2016-02-12 14:34 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Marc MERLIN, linux-raid
In-Reply-To: <alpine.DEB.2.02.1602121020570.11524@uplift.swm.pp.se>
Mikael Abrahamsson <swmike@swm.pp.se> writes:
> On Thu, 11 Feb 2016, Marc MERLIN wrote:
>
>> But can someone confirm that indeed if I have a single md device,
>> and 4 CPUs, my throughput is not limited by a single CPU decrypting
>> the device?
>
> There have been multiple discussions about this over time, and since
> you didn't qualify "not good" with a number, it's hard to say if this
> is your problem or not.
About a year ago when I last received a new laptop, I put btrfs on top
of luks - it was a rather miserable experience. The system would freeze
up and be really slow.
I believe the issue was btrfs not acting well with crypto underneath it,
and there was hallway talk confirming this but I have no hard data. In
the end I gave up and put my /home on xfs on top of LUKS and used btrfs
for the uncrypted portions.
It may be the btrfs with crypto issues have been resolved since then.
Cheers,
Jes
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Mikael Abrahamsson @ 2016-02-12 9:30 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-raid
In-Reply-To: <20160211162959.GG13969@merlins.org>
On Thu, 11 Feb 2016, Marc MERLIN wrote:
> But can someone confirm that indeed if I have a single md device, and 4
> CPUs, my throughput is not limited by a single CPU decrypting the
> device?
There have been multiple discussions about this over time, and since you
didn't qualify "not good" with a number, it's hard to say if this is your
problem or not.
If you're getting around 100-300 megabyte/s or something, and "top" shows
high CPU utilization for a few processes and some cores idling (use "1"
and "H" option after starting top to see individual cores), and "iostat -x
5" shows the drives as not topping out, then most likely you're indeed
being hit by dmcrypt not being multithreaded and/or AES-NI not working.
I personally run the same kind of setup as you, I use md->dmcrypt->lvm->fs
as stack. Performance isn't stellar (I just tried and I get 500 megabyte/s
read on a machine I am pretty sure AES-NI is working from the start, but
on another machine where I think AES-NI probably isn't loaded early
enough, I only got 250 megabyte/s).
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Marc MERLIN @ 2016-02-11 17:40 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160211171340.GA7664@EIS>
On Thu, Feb 11, 2016 at 06:13:40PM +0100, Andreas Klauer wrote:
> > gargamel:~# cryptsetup luksDump /dev/md8
> > LUKS header information for /dev/md8
> >
> > Version: 1
> > Cipher name: aes
> > Cipher mode: xts-plain64
> > Hash spec: sha1
> > Payload offset: 3072
> > MK bits: 256
>
> Does the box have AES-NI? What's your 'cryptsetup benchmark' look like?
> Sometimes there can be a problem if the AES-NI module is loaded too late.
> Without AES-NI your performance will suffer either way...
It's a quad core HT CPU
model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
As far as I can tell, AES-NI is working:
gargamel:~# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 420102 iterations per second
PBKDF2-sha256 250137 iterations per second
PBKDF2-sha512 87148 iterations per second
PBKDF2-ripemd160 394795 iterations per second
PBKDF2-whirlpool 125068 iterations per second
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 1.2 MiB/s 1939.8 MiB/s
serpent-cbc 128b 29.8 MiB/s 284.4 MiB/s
twofish-cbc 128b 77.0 MiB/s 339.0 MiB/s
aes-cbc 256b 451.2 MiB/s 1491.7 MiB/s
serpent-cbc 256b 85.7 MiB/s 286.7 MiB/s
twofish-cbc 256b 188.7 MiB/s 358.4 MiB/s
aes-xts 256b 1572.1 MiB/s 1725.3 MiB/s
serpent-xts 256b 272.8 MiB/s 291.2 MiB/s
twofish-xts 256b 289.8 MiB/s 331.7 MiB/s
aes-xts 512b 1355.1 MiB/s 1385.5 MiB/s
serpent-xts 512b 318.4 MiB/s 299.8 MiB/s
twofish-xts 512b 326.6 MiB/s 336.3 MiB/s
> You probably don't want encryption below the RAID; that would mean
> encrypting redundancy and parity so it's even more work to do, doubtful
> whether multicore CPU can offset that to make it worth it. Maybe if
> it's a NAS that has nothing else to do...
It does other work, and I agree that encryption below the raid doesn't
sound like a great idea, which is why I haven't used it so far.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply
* Re: md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Andreas Klauer @ 2016-02-11 17:13 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-raid
In-Reply-To: <20160211162959.GG13969@merlins.org>
> gargamel:~# cryptsetup luksDump /dev/md8
> LUKS header information for /dev/md8
>
> Version: 1
> Cipher name: aes
> Cipher mode: xts-plain64
> Hash spec: sha1
> Payload offset: 3072
> MK bits: 256
Does the box have AES-NI? What's your 'cryptsetup benchmark' look like?
Sometimes there can be a problem if the AES-NI module is loaded too late.
Without AES-NI your performance will suffer either way...
You probably don't want encryption below the RAID; that would mean
encrypting redundancy and parity so it's even more work to do, doubtful
whether multicore CPU can offset that to make it worth it. Maybe if
it's a NAS that has nothing else to do...
Regards
Andreas
^ permalink raw reply
* md raid5 on top of dmcrypt, or dmcrypt on top of md raid5?
From: Marc MERLIN @ 2016-02-11 16:29 UTC (permalink / raw)
To: linux-raid
I have a 5 drive md array with dmcrypt on top, and btrfs on top of that.
Kernel: 4.4
Honestly, the performance is not good, but it's hard to pin down what
exactly is to blame (there could be performance issues with btrfs too).
I have read that dmcrypt is supposed to be multithreaded, and that's
been true for multiple years.
I put dmcrypt on top so that doing a raid sync or rebuild does not
involve encryption/decryption.
But can someone confirm that indeed if I have a single md device, and 4
CPUs, my throughput is not limited by a single CPU decrypting the
device?
And if I'm going to recreate the array (using bcache this time), is
there a better way to recreate it to avoid performance issues?
It is an array with many small files (filesystem backups), so lots of
random non contiguous I/O
Currently, I have:
gargamel:~# mdadm --detail /dev/md8
/dev/md8:
Version : 1.2
Creation Time : Sat Apr 19 23:03:59 2014
Raid Level : raid5
Array Size : 7813523456 (7451.56 GiB 8001.05 GB)
Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Feb 11 08:26:45 2016
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
gargamel:~# cryptsetup luksDump /dev/md8
LUKS header information for /dev/md8
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
Payload offset: 3072
MK bits: 256
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply
* Re: [PATCH 2/3][mdadm] mdadm.h: bswap is already defined in uclibc
From: Maxin B. John @ 2016-02-11 11:26 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid, Maxin B. John
In-Reply-To: <wrfjio1w74od.fsf@redhat.com>
Hi Jes,
On Wed, Feb 10, 2016 at 9:20 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> "Maxin B. John" <maxin.john@gmail.com> writes:
>> Hi Jes,
>>
>> On Mon, Feb 8, 2016 at 6:00 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
>>> "Maxin B. John" <maxin.john@gmail.com> writes:
>>>> From: "Maxin B. John" <maxin.john@intel.com>
>>>>
>>>> Fixes this build error:
>>>>
>>>> | In file included from mdadm.c:28:0:
>>>> | mdadm.h:142:0: error: "bswap_16" redefined [-Werror]
>>>> | #define bswap_16(x) (((x) & 0x00ffU) << 8 | \
>>>> | ^
>>>>
>>>> Signed-off-by: Maxin B. John <maxin.john@intel.com>
>>>> ---
>>>> mdadm.h | 9 +++++++++
>>>> 1 file changed, 9 insertions(+)
>>>
>>> Hi Maxin,
>>>
>>> I am not opposed to this, but I would like to understand why you see
>>> these duplicate defines. What defines it in your build environment?
>>
>>
>> I get the below listed error message with uclibc builds:
>
<snip>
>
> Maxin,
>
> I think it's actually wrong for uClibc to expose those macros with such
> a common name, but rather than picking up outside macros randomly I
> prefer to rename the mdadm ones to be sure we know what we are getting.
That looks reasonable to me.
> Instead of your patch, I applied this one instead. I hope that is fine
> with you.
Yes, this fix looks much better. Thank you!
> Cheers,
> Jes
Best Regards,
Maxin
>
> From dd47b4e0c45fd72b94a9a7d0f0a5046ef9c8d97b Mon Sep 17 00:00:00 2001
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
> Date: Wed, 10 Feb 2016 14:15:38 -0500
> Subject: [PATCH] mdadm.h: rename bswap macros to avoid clash with uClibc
> definitions
>
> uClibc exposes it's own version of bswap_<X> macros. Rather than
> pulling in random macros by change, rename the mdadm ones to make sure
> we know what we are getting.
>
> Reported-by: "Maxin B. John" <maxin.john@gmail.com>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> mdadm.h | 52 ++++++++++++++++++++++++++--------------------------
> 1 file changed, 26 insertions(+), 26 deletions(-)
>
> diff --git a/mdadm.h b/mdadm.h
> index dd02be7..72888e2 100755
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -139,20 +139,20 @@ struct dlm_lksb {
> * and there is no standard conversion function so... */
> /* And dietlibc doesn't think byteswap is ok, so.. */
> /* #include <byteswap.h> */
> -#define bswap_16(x) (((x) & 0x00ffU) << 8 | \
> - ((x) & 0xff00U) >> 8)
> -#define bswap_32(x) (((x) & 0x000000ffU) << 24 | \
> - ((x) & 0xff000000U) >> 24 | \
> - ((x) & 0x0000ff00U) << 8 | \
> - ((x) & 0x00ff0000U) >> 8)
> -#define bswap_64(x) (((x) & 0x00000000000000ffULL) << 56 | \
> - ((x) & 0xff00000000000000ULL) >> 56 | \
> - ((x) & 0x000000000000ff00ULL) << 40 | \
> - ((x) & 0x00ff000000000000ULL) >> 40 | \
> - ((x) & 0x0000000000ff0000ULL) << 24 | \
> - ((x) & 0x0000ff0000000000ULL) >> 24 | \
> - ((x) & 0x00000000ff000000ULL) << 8 | \
> - ((x) & 0x000000ff00000000ULL) >> 8)
> +#define __mdadm_bswap_16(x) (((x) & 0x00ffU) << 8 | \
> + ((x) & 0xff00U) >> 8)
> +#define __mdadm_bswap_32(x) (((x) & 0x000000ffU) << 24 | \
> + ((x) & 0xff000000U) >> 24 | \
> + ((x) & 0x0000ff00U) << 8 | \
> + ((x) & 0x00ff0000U) >> 8)
> +#define __mdadm_bswap_64(x) (((x) & 0x00000000000000ffULL) << 56 | \
> + ((x) & 0xff00000000000000ULL) >> 56 | \
> + ((x) & 0x000000000000ff00ULL) << 40 | \
> + ((x) & 0x00ff000000000000ULL) >> 40 | \
> + ((x) & 0x0000000000ff0000ULL) << 24 | \
> + ((x) & 0x0000ff0000000000ULL) >> 24 | \
> + ((x) & 0x00000000ff000000ULL) << 8 | \
> + ((x) & 0x000000ff00000000ULL) >> 8)
>
> #if !defined(__KLIBC__)
> #if BYTE_ORDER == LITTLE_ENDIAN
> @@ -163,19 +163,19 @@ struct dlm_lksb {
> #define __le32_to_cpu(_x) (unsigned int)(_x)
> #define __le64_to_cpu(_x) (unsigned long long)(_x)
>
> -#define __cpu_to_be16(_x) bswap_16(_x)
> -#define __cpu_to_be32(_x) bswap_32(_x)
> -#define __cpu_to_be64(_x) bswap_64(_x)
> -#define __be16_to_cpu(_x) bswap_16(_x)
> -#define __be32_to_cpu(_x) bswap_32(_x)
> -#define __be64_to_cpu(_x) bswap_64(_x)
> +#define __cpu_to_be16(_x) __mdadm_bswap_16(_x)
> +#define __cpu_to_be32(_x) __mdadm_bswap_32(_x)
> +#define __cpu_to_be64(_x) __mdadm_bswap_64(_x)
> +#define __be16_to_cpu(_x) __mdadm_bswap_16(_x)
> +#define __be32_to_cpu(_x) __mdadm_bswap_32(_x)
> +#define __be64_to_cpu(_x) __mdadm_bswap_64(_x)
> #elif BYTE_ORDER == BIG_ENDIAN
> -#define __cpu_to_le16(_x) bswap_16(_x)
> -#define __cpu_to_le32(_x) bswap_32(_x)
> -#define __cpu_to_le64(_x) bswap_64(_x)
> -#define __le16_to_cpu(_x) bswap_16(_x)
> -#define __le32_to_cpu(_x) bswap_32(_x)
> -#define __le64_to_cpu(_x) bswap_64(_x)
> +#define __cpu_to_le16(_x) __mdadm_bswap_16(_x)
> +#define __cpu_to_le32(_x) __mdadm_bswap_32(_x)
> +#define __cpu_to_le64(_x) __mdadm_bswap_64(_x)
> +#define __le16_to_cpu(_x) __mdadm_bswap_16(_x)
> +#define __le32_to_cpu(_x) __mdadm_bswap_32(_x)
> +#define __le64_to_cpu(_x) __mdadm_bswap_64(_x)
>
> #define __cpu_to_be16(_x) (unsigned int)(_x)
> #define __cpu_to_be32(_x) (unsigned int)(_x)
> --
> 2.5.0
>
^ permalink raw reply
* Re: Reshape stuck immediately, backup file all nulls
From: NeilBrown @ 2016-02-11 4:22 UTC (permalink / raw)
To: Phil Turmel, Björn Augustsson, linux-raid
In-Reply-To: <56ACEE63.6070108@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]
On Sun, Jan 31 2016, Phil Turmel wrote:
> Hi Björn,
>
> On 01/30/2016 07:21 AM, Björn Augustsson wrote:
>
>> The question is, what kind of state am in now?
>
> The kernel is waiting for mdmon (mdadm as a background task) to step
> through the stripes. mdmon died and the kernel will wait forever.
Close, not quite.
'mdmon' is a background task that mdadm *only* uses for externally
managed metadata: IMSM and DDF.
For a reshape like this mdadm needs a background "mdadm" task. It used
to just fork, but in these enlightened days it asks systemd to run that
task.
As has already been observed, that failed due to selinux not
understanding.
So it was an 'mdadm' which exited, rather than an mdmon which died...
Maybe you already worked that out.
The days of backup files should be numbered. New kernels and new mdadm
adjust the data-offset so no back is needed. In that case, no
background mdadm is needed either.
NeilBrown
>
>> And how should I recover?
>> Will just adding a policy to allow access to that file, and then
>> mdadm --grow --continue /dev/md127
>> fix it?
>
> Probably. Others have fixed this by disabling selinux for the reshape.
> Don't forget to specify --backup-file again on the command line. (I
> don't use selinux myself, so I can't be more specific.)
>
> In the future, consider not using a backup file at all -- mdadm
> generally leaves enough dead space on devices to avoid the need.
>
>> Is the broken backup file going to be a problem?
>
> Shouldn't be. Report back if it is.
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: mdadm bad blocks list
From: NeilBrown @ 2016-02-11 4:15 UTC (permalink / raw)
To: Sarah Newman, linux-raid
In-Reply-To: <56B0172B.8020808@prgmr.com>
[-- Attachment #1: Type: text/plain, Size: 986 bytes --]
On Tue, Feb 02 2016, Sarah Newman wrote:
> On 01/27/2016 07:19 PM, NeilBrown wrote:
>> On Thu, Jan 28 2016, Sarah Newman wrote:
>>
>>> Kernel: CentOS 6 Xen4CentOS 3.18.21-17
>>> mdadm: CentOS 6 v3.3.2
>>>
>>> With the above behavior, I consider the bad blocks list to be actively harmful. If it's expected behavior in the current version, please consider
>>> disabling the bad blocks list by default.
>>
>> You can do this yourself by putting
>>
>> CREATE bbl=no
>>
>> in /etc/mdadm.conf. That doesn't help others though.
>
> FYI, I tried adding that line and if it is present when trying to add to an array that currently has a bad blocks list, I get the error message
>
> md: sdi1 does not have a valid v1.0 superblock, not importing!
>
> Is this expected?
No, that looks like a bug.
If would be helpful to see complete "mdadm --examine" output of the
device that you tried to add, and a device that is already in the array.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 2/2] enable bypass raid5 journal for full stripe writes
From: NeilBrown @ 2016-02-11 4:07 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1455151995-240805-2-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 3416 bytes --]
On Thu, Feb 11 2016, Song Liu wrote:
> Summary:
>
> Resending the patch to see whether we can get another chance...
>
> When testing current SATA SSDs as the journal device, we have
> seen 2 challenges: throughput of long sequential writes, and
> SSD life time.
>
> To ease the burn on the SSD, we tested bypassing journal for
> full stripe writes. We understand that bypassing journal will
> re-introduce write hole to the md layer. However, with
> well-designed application and file system, such write holes
> should not result in any data loss.
"should not" isn't very encouraging.
I've always thought that the "write hole" was more of a theoretical than
a practical concern. But if we go to the trouble of preventing them, I
really think we need "will not".
I don't suppose we could ask the filesystem to tag certain requests with
a flag which say "If the system crashes before this write completes, I
promise to re-write the entire region (possibly with different data)
before trusting any data I read from here".
That is what you really need.
But it wouldn't help for in-place overwrites.
>
> Our test systems have 2 RAID-6 array per server and 15 HDDs
> per array. These 2 arrays shared 1 SSD as the journal (2
> partitions). Btrfs is created on both array.
>
> For squential write benchmarks, we observe significant
> performance gain (250MB/s per volume vs. 150M/s) from
> bypassing journal for full stripes.
You need a faster SSD :-)
>
> We all performed power cycle tests on these systems while
> running a write workload. For more than 50 power cycles,
> we have seen zero data loss.
That doesn't fill me with any confidence.
To get even close to a credible test, you would need to instrument the
kernel so that at some (random) point it would never send another IO
request until a reboot, and so that this random point was always
somewhere in the middle of submitting a set of writes to a stripe.
Instrument a kernel to do this, demonstrate that the write-hole causes
problems, then demonstrate that it doesn't with the various patches
applied. Then do this across a range of tests (write-in-place, append,
etc) on a range of filesystems.
Testing for data integrity is *hard*.
And as you say, it may be up to the application to ensure it doesn't use
corrupt data (like a partial write). Testing that is not really
possible.
If you were to make it possible to do this, you would need to be very
explicit about what sort of write-hole corruption could still occur, so
that users could make an informed decision.
(I assume your test had a degraded array... you didn't explicitly say
so, but the test would be meaningless on a non-degraded array so you
probably did).
>
> +static ssize_t
> +r5l_show_bypass_full_stripe(struct mddev *mddev, char *page)
> +{
> + struct r5conf *conf;
> + int ret = 0;
> +
> + spin_lock(&mddev->lock);
> + conf = mddev->private;
> + if (conf) {
> + if (conf->log)
> + ret = sprintf(page, "%d\n",
> + conf->log->bypass_full_stripe);
> + else
> + ret = sprintf(page, "n/a\n");
> + }
Given that this is a boolean, I would much prefer it was consistently a
boolean value, never "n/a".
Not sure if 0/1 Y/N T/F is best, but one of those.
Otherwise the code looks sensible (assuming the idea is sensible, which
I'm not 100% convinced of).
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 1/2] allow md_flush_request to take NULL bio
From: NeilBrown @ 2016-02-11 3:46 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1455151995-240805-1-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]
On Thu, Feb 11 2016, Song Liu wrote:
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index e55e6cf..2997104 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -406,6 +406,8 @@ static void md_submit_flush_data(struct work_struct *ws)
> struct mddev *mddev = container_of(ws, struct mddev, flush_work);
> struct bio *bio = mddev->flush_bio;
>
> + if (!bio)
> + goto out;
> if (bio->bi_iter.bi_size == 0)
> /* an empty barrier - all done */
> bio_endio(bio);
> @@ -415,6 +417,7 @@ static void md_submit_flush_data(struct work_struct *ws)
> }
>
> mddev->flush_bio = NULL;
> +out:
> wake_up(&mddev->sb_wait);
> }
>
I don't think this is safe.
->flush_bio is used for two different purposes.
One is to carry the bio so it can be passed to ->make_request after the
flush.
The other is to ensure exclusive access to mddev->flush_work.
The first thing md_flush_request() does is wait for ->flush_bio to be
NULL. Then it knows that ->flush_work is no longer in use.
With your change you could get md_flush_request trying to INIT_WORK
->flush_work while it is already queued.
Not good.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH 2/2] enable bypass raid5 journal for full stripe writes
From: Song Liu @ 2016-02-11 0:53 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1455151995-240805-1-git-send-email-songliubraving@fb.com>
Summary:
Resending the patch to see whether we can get another chance...
When testing current SATA SSDs as the journal device, we have
seen 2 challenges: throughput of long sequential writes, and
SSD life time.
To ease the burn on the SSD, we tested bypassing journal for
full stripe writes. We understand that bypassing journal will
re-introduce write hole to the md layer. However, with
well-designed application and file system, such write holes
should not result in any data loss.
Our test systems have 2 RAID-6 array per server and 15 HDDs
per array. These 2 arrays shared 1 SSD as the journal (2
partitions). Btrfs is created on both array.
For squential write benchmarks, we observe significant
performance gain (250MB/s per volume vs. 150M/s) from
bypassing journal for full stripes.
We all performed power cycle tests on these systems while
running a write workload. For more than 50 power cycles,
we have seen zero data loss.
To configure the bypass feature:
echo 1 > /sys/block/mdX/md/r5l_bypass_full_stripe
and
echo 0 > /sys/block/mdX/md/r5l_bypass_full_stripe
For file system integrity, the code does not bypass any write
with REQ_FUA.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/raid5.c | 1 +
drivers/md/raid5.h | 2 ++
3 files changed, 77 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 9531f5f..9ec0878 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -97,6 +97,8 @@ struct r5l_log {
bool need_cache_flush;
bool in_teardown;
+
+ int bypass_full_stripe;
};
/*
@@ -446,6 +448,7 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
int reserve;
int i;
int ret = 0;
+ int fua = 0;
if (!log)
return -EAGAIN;
@@ -462,6 +465,8 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
continue;
+ if (test_bit(R5_WantFUA, &sh->dev[i].flags))
+ fua = 1;
write_disks++;
/* checksum is already calculated in last run */
if (test_bit(STRIPE_LOG_TRAPPED, &sh->state))
@@ -471,6 +476,10 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
addr, PAGE_SIZE);
kunmap_atomic(addr);
}
+
+ if (log->bypass_full_stripe && (write_disks == sh->disks) && (!fua))
+ return -EAGAIN; /* bypass journal device */
+
parity_pages = 1 + !!(sh->qd_idx >= 0);
data_pages = write_disks - parity_pages;
@@ -524,7 +533,7 @@ void r5l_write_stripe_run(struct r5l_log *log)
int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio)
{
- if (!log)
+ if (!log || log->bypass_full_stripe)
return -ENODEV;
/*
* we flush log disk cache first, then write stripe data to raid disks.
@@ -1186,6 +1195,69 @@ ioerr:
return ret;
}
+static ssize_t
+r5l_show_bypass_full_stripe(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf;
+ int ret = 0;
+
+ spin_lock(&mddev->lock);
+ conf = mddev->private;
+ if (conf) {
+ if (conf->log)
+ ret = sprintf(page, "%d\n",
+ conf->log->bypass_full_stripe);
+ else
+ ret = sprintf(page, "n/a\n");
+ }
+ spin_unlock(&mddev->lock);
+ return ret;
+}
+
+static ssize_t
+r5l_store_bypass_full_stripe(struct mddev *mddev, const char *page, size_t len)
+{
+ struct r5conf *conf;
+ int err = 0;
+ unsigned int val;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtouint(page, 10, &val))
+ return -EINVAL;
+
+ if (val > 1)
+ val = 1;
+
+ err = mddev_lock(mddev);
+ if (err)
+ return err;
+ mddev_suspend(mddev);
+
+ /*
+ * We do not flush when journal is on, add extra flush for previous writes
+ */
+ if (val == 0)
+ md_flush_request(mddev, NULL);
+
+ conf = mddev->private;
+ if (conf) {
+ if (conf->log) {
+ conf->log->bypass_full_stripe = val;
+ } else
+ err = -EINVAL;
+ } else
+ err = -ENODEV;
+ mddev_resume(mddev);
+ mddev_unlock(mddev);
+ return err ?: len;
+}
+
+struct md_sysfs_entry
+r5l_bypass_full_stripe = __ATTR(r5l_bypass_full_stripe, S_IRUGO | S_IWUSR,
+ r5l_show_bypass_full_stripe,
+ r5l_store_bypass_full_stripe);
+
int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
{
struct r5l_log *log;
@@ -1227,6 +1299,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (!log->meta_pool)
goto out_mempool;
+ log->bypass_full_stripe = 0;
log->reclaim_thread = md_register_thread(r5l_reclaim_thread,
log->rdev->mddev, "reclaim");
if (!log->reclaim_thread)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index b4f02c9..bdf30b1 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -6216,6 +6216,7 @@ static struct attribute *raid5_attrs[] = {
&raid5_group_thread_cnt.attr,
&raid5_skip_copy.attr,
&raid5_rmw_level.attr,
+ &r5l_bypass_full_stripe.attr,
NULL,
};
static struct attribute_group raid5_attrs_group = {
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index a415e1c..6b39a07 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -633,4 +633,6 @@ extern void r5l_stripe_write_finished(struct stripe_head *sh);
extern int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio);
extern void r5l_quiesce(struct r5l_log *log, int state);
extern bool r5l_log_disk_error(struct r5conf *conf);
+
+extern struct md_sysfs_entry r5l_bypass_full_stripe;
#endif
--
2.4.6
^ permalink raw reply related
* [PATCH 1/2] allow md_flush_request to take NULL bio
From: Song Liu @ 2016-02-11 0:53 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index e55e6cf..2997104 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -406,6 +406,8 @@ static void md_submit_flush_data(struct work_struct *ws)
struct mddev *mddev = container_of(ws, struct mddev, flush_work);
struct bio *bio = mddev->flush_bio;
+ if (!bio)
+ goto out;
if (bio->bi_iter.bi_size == 0)
/* an empty barrier - all done */
bio_endio(bio);
@@ -415,6 +417,7 @@ static void md_submit_flush_data(struct work_struct *ws)
}
mddev->flush_bio = NULL;
+out:
wake_up(&mddev->sb_wait);
}
--
2.4.6
^ permalink raw reply related
* Re: Best option for SSD caching on a md raid array?
From: Pasi Kärkkäinen @ 2016-02-10 22:54 UTC (permalink / raw)
To: John Stoffel; +Cc: Thomas Fjellstrom, linux-raid
In-Reply-To: <22203.47103.654951.880508@quad.stoffel.home>
On Wed, Feb 10, 2016 at 05:21:51PM -0500, John Stoffel wrote:
> >>>>> "Pasi" == Pasi Kärkkäinen <pasik@iki.fi> writes:
>
> Pasi> On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
> >>
> Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
> Thomas> off LVM, I would like to add an ssd cache to help with typical
> Thomas> server system loads, and some mixed read/write loads for game
> Thomas> servers.
> >>
> Thomas> I'm wondering what the "best" route for me to go is. There's a
> Thomas> few options that I know of, like flashcache, dm-cache, and
> Thomas> bcache. The question is which of those is better suited to my
> Thomas> use case?
> >>
> >> I looked into both bcache and lvmcache and went with lvmcache because
> >> it made more sense to me. But one restriction is that the cache LVs
> >> must be in the same VGs as the volume to cache. So you might need to
> >> partition your SSD(s) and put the partitions into seperate VGs so that
> >> you can cache one or more LVs.
> >>
> >> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
> >> SSDs. I'm a big believer in mirroring, esp for critical stuff like
> >> this.
> >>
> >> Unfortunately, I don't have a good performance measurements of how
> >> much help this actually gives me. It *feels* faster, but since I
> >> don't have before/after measurements, I could just have wasted a bunch
> >> of money.
> >>
> >> So I'd recommend lvmcache overall, since it does seem to help, but
> >> push for better performance metrics and measurement. Heh, maybe it's
> >> time for me to start writing up lvmcachetop to help measure things.
>
> Pasi> The other day I was benchmarking lvmcache (on CentOS 7.2), but I
> Pasi> couldn't get it to give much advantage for writeback caching use
> Pasi> case.. even when I was using fast nvme/pcie cache device.
>
> What were you using? I should spin up fio and see what I can
> find... or maybe even just some kernel compiles, image manipulation,
> etc.
>
Yeah I was benchmarking small random 4 kB writes, using fio.
iodepth=64 and jobs=12.
> Let's see what we can come up with for benchmarking, since it would be
> a big help! I'm running kernel 4.4-rc7 right now on a Debian Jessie system.
>
> Pasi> It just wouldn't cache writes properly for me. I tried both the
> Pasi> smq and mq, and played with all of the parameters.. it just
> Pasi> wouldn't help much for my use case.
>
> I have both mq and smq compiled into the kernel, so I can't figure out
> how to change between them to test.
>
Hmm, yeah, I think it was documented on the lvmcache man page.
I can't check my notes right now..
> John
-- Pasi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox