* Re: Kernel deadlock during mdadm reshape
From: Bart Van Assche @ 2016-07-28 0:04 UTC (permalink / raw)
To: Michael Shaver, mdraid
In-Reply-To: <bbcfb493-cdee-54cd-99aa-b1f9217b3b43@gmail.com>
On 07/26/2016 07:18 PM, Michael Shaver wrote:
> I am experiencing the exact same problem reported in this thread:
>
> http://www.spinics.net/lists/raid/msg52235.html
>
> Also reported here:
>
> https://forums.gentoo.org/viewtopic-t-1043706.html
>
> And here:
>
> https://bbs.archlinux.org/viewtopic.php?id=212108
>
> I have a raid5 array of 2TB disks currently stuck at 94% of a mdadm
> reshape squeal to a grow operation from 4 disks to 5. In my case, I did
> have a drive drop out of the array during the reshape.
>
> [ ... ]
> [<ffffffff811cb8dd>] truncate_inode_pages_range+0x46d/0x880
> [<ffffffff811cbd05>] truncate_inode_pages+0x15/0x20
> [ ... ]
Hello Michael,
I ran into a similar lockup while retesting the dm-mpath driver. So we
either ran into a different lockup or the lockup was not related to the
md driver. Were you using a distro kernel or a kernel you built
yourself? Anyway, was THP (transparent huge page) support enabled? If
so, can you repeat your test with THP support disabled and see whether
that makes a difference?
Thanks,
Bart.
^ permalink raw reply
* SOLVED [was: Re: Lost RAID6 disks when moving to new PC]
From: Alex Owen @ 2016-07-27 23:27 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
Thanks to Andreas I now have a somewhat functioning RAID array and I'm
pulling data from it on to some new disks.
Here's the final process I had to go through:
- Create an overlay for all six disks. Following the tutorial at
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
only created the overlay for the disks it recognised as being part of
the array. To add the others I manually changed the $DEVICES variable
to include all size disks.
- Then I ran:
mdadm --create /dev/md42 --assume-clean --level=6 --raid-devices=6
--chunk=512 --layout=ls /dev/mapper/sd{e,f,a,b,c,d}
Note the changes from Andreas' suggestion: added the raid-devices
option and removed the data-offset option (not supported on the
version of mdadm I was using on Ubuntu 14.04).
- I was using LVM on top of the array, so to mount it I ran
lvmdiskscan and lvdisplay to find out where the partitions were stored
by lvm
- Then I mounted the LVM partitions and it worked!
Andreas, I really appreciate your help, thank you.
On 27 July 2016 at 12:57, Andreas Klauer <Andreas.Klauer@metamorpher.de> wrote:
> On Wed, Jul 27, 2016 at 11:35:34AM +0100, Alex Owen wrote:
>> The array should be RAID6 on /dev/sd{a-f}.
>
> Full disk raid sucks. Zero advantages, lots of additional risk.
> There are too many programs out there that expect every disk to have
> a partition table, and will use it unasked if it looks unpartitioned.
>
> You seem to have lost your md metadata to some partitioner/installer,
> you're also the third person with this problem in a row. Congrats. ;)
>
>> fdisk -l :
>
> Your fdisk doesn't support GPT, don't use it.
>
>> parted
>> Disk /dev/sd[bcd]: 3001GB
>> Sector size (logical/physical): 512B/4096B
>> Partition Table: gpt
>>
>> Number Start End Size File system Name Flags
>> 1 17.4kB 134MB 134MB Microsoft reserved
>> partition msftres
>> 2 135MB 3001GB 3000GB Basic data partition
>> msftdata
>
> Well, something put GPT partition table on those. GPT overwrites start
> and end of the disk. You're using 1.2 metadata which is located 4K from
> the start, can you show some hexdump for those disks?
>
> hexdump -C -s 4096 -n 4096 /dev/sdb
>
>> And the output of mdadm --examine /dev/sd[a-f]
>>
>> ----------
>>
>> /dev/sda:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
>> Name : NAS:0 (local to host NAS)
>> Creation Time : Sat Jan 23 17:57:37 2016
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
>> Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
>> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : 013740d6:5cc445e7:625f3257:8608daec
>>
>> Update Time : Tue Jul 26 04:09:29 2016
>> Checksum : 14ba6ebd - correct
>> Events : 2949004
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAAAAA ('A' == active, '.' == missing)
>
>> /dev/sdb:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdc:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdd:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>
> Basically what we know is... your disk order for three disks
>
> (/dev/sde = role 0, /dev/sdf = role 1, /dev/sda= role 2)
>
> and what we don't know is the disk order of /dev/sd[bcd].
>
> If the metadata is lost completely, only thing you can do is re-create
> the RAID with all possible orders efa{bcd,bdc,cbd,cdb,dbc,dcb}.
>
> Re-creating is dangerous so you should use an overlay: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> When re-creating you have to specify all variables (level, layout, chunksize,
> data offset, order, ...) since the defaults picked by mdadm might differ
> depending on your mdadm version.
>
> Example command: (untested)
>
> mdadm --create /dev/md42 --assume-clean \
> --level=6 --chunk=512 --data-offset=128M --layout=ls \
> /dev/overlay/sd{e,f,a,b,c,d}
>
> Then you check if it can be mounted, and once mounted if big files
> (larger than chunksize * number of disks) are intact or no. If you
> switch the wrong two disks it may mount but data is garbage anyway.
>
> Regards
> Andreas Klauer
^ permalink raw reply
* Re: RAID5 Performance
From: Adam Goryachev @ 2016-07-27 23:26 UTC (permalink / raw)
To: doug; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAFx4rwR0+wR_0GXh3Y7j1hZyv0_Gy7b4OD-D2+bfxV9MRp+b3Q@mail.gmail.com>
On 27/07/2016 15:36, Doug Dumitru wrote:
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>> My current array looks like this:
>>
>> /dev/md1:
>> Version : 1.2
>> Creation Time : Wed Aug 22 00:47:03 2012
>> Raid Level : raid5
>> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
>> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
>> Raid Devices : 8
>> Total Devices : 8
>> Persistence : Superblock is persistent
>>
>> Update Time : Wed Jul 27 11:32:00 2016
>> State : active
>> Active Devices : 8
>> Working Devices : 8
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Name : san1:1 (local to host san1)
>> UUID : 707957c0:b7195438:06da5bc4:485d301c
>> Events : 2185221
>>
>> Number Major Minor RaidDevice State
>> 7 8 65 0 active sync /dev/sde1
>> 13 8 1 1 active sync /dev/sda1
>> 8 8 81 2 active sync /dev/sdf1
>> 5 8 113 3 active sync /dev/sdh1
>> 9 8 97 4 active sync /dev/sdg1
>> 12 8 17 5 active sync /dev/sdb1
>> 10 8 49 6 active sync /dev/sdd1
>> 11 8 33 7 active sync /dev/sdc1
>>
>> I've configured the following non-standard options:
>>
>> echo 4096 > /sys/block/md1/md/stripe_cache_size
>>
>> The following apply to all SSD's installed:
>> echo noop > $disk/queue/scheduler
>> echo 128 > ${disk}/queue/nr_requests
>>
>> What I can measure (at peak periods) with iostat:
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
>> 0.02 0.15 0.20 0.09 0.15 2.40
>> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
>> 0.02 0.14 0.24 0.05 0.11 1.60
>> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
>> 0.03 0.17 0.09 0.24 0.09 1.60
>> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
>> 0.08 0.36 0.29 0.40 0.34 7.20
>> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
>> 0.08 0.42 0.32 0.51 0.29 5.20
>> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
>> 0.01 0.04 0.09 0.00 0.04 0.80
>> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
>> 0.01 0.06 0.04 0.08 0.04 0.80
>> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
>> 0.18 0.28 0.23 0.32 0.16 9.60
>> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
>> 0.08 38.00 0.00 38.00 20.00 4.00
>> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
>> 0.03 14.00 4.00 24.00 14.00 2.80
>> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
>> 0.09 44.00 0.00 44.00 22.00 4.40
>> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
>> 7.92 41.03 0.00 41.87 4.98 100.00
>> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
>> 0.22 30.29 0.00 30.29 28.57 20.00
>> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
>> 1.72 13.46 0.40 40.18 4.27 57.20
>> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
>> 1.40 40.91 0.00 43.39 28.34 99.20
>> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
>> 0.08 21.00 0.00 28.00 21.00 8.40
>> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> As you can see, the DRBD devices are busy, and slowing down the VM's,
>> looking at the drives on the second server we can see why:
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
>> 0.26 1.47 0.06 2.27 1.45 25.60
>> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
>> 0.45 2.76 0.08 3.93 2.71 44.40
>> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
>> 0.75 4.73 0.00 6.90 4.70 74.80
>> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
>> 0.81 5.21 0.23 7.69 5.13 80.00
>> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
>> 0.94 4.59 0.69 6.85 4.24 86.40
>> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
>> 0.84 5.03 0.21 7.60 4.89 81.60
>> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
>> 0.69 4.27 0.14 6.54 4.22 68.00
>> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
>> 1.21 2.43 1.88 2.90 1.50 74.40
>> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> I've confirmed that the problem is that we have mixed two models of SSD (520
>> series and 530 series), and that the 530 series drives perform significantly
>> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
>> while the other drives are 530 series. So, we will be replacing all of the
>> drives across both systems with 545s series 1000GB SSD's (which I've
>> confirmed will operate same or better than the 520 series, sdc on the first
>> machine above is one of these already).
>>
>> Over the years, I've learned a lot about RAID and optimisation, originally I
>> configured things to optimise for super fast streaming reads and streaming
>> writes, but in practice, the actual work-load is small random read/write,
>> with the writes causing the biggest load.
>>
>> Looking at this:
>> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>> *
>>>
>>> Enhance the queue depth. Standard kernel queue depth is OK for old
>>> single drives with small caches, but not for modern drives or RAID
>>> arrays:
>>>
>>> echo 512 > /sys/block/sda/queue/nr_requests
>>>
>> So my question is should I increase the configured nr_requests above the
>> current 128?
> With your workload, it probably won't matter too much. Really high
> queue depths are great on paper, but hard to actually see.
Is there some way to see if this would help or not?
Would it hurt to increase this (even if it doesn't help)?
>
>> If the chunk size is 64k, and there are 8 drives in total, then the stripe
>> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
>> page suggests the minimum chunk size is 4k ("In any case it must be a
>> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
>> becomes 28k, which means for a random 4k write, we only need to write 28k
>> instead of 448k ?
> This is not how a random write works. If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
> writes. With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes. With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high. Note that the new logic is only
> raid-5. raid-6 cannot use the new logic and has to read the stripe
> from every drive.
Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should
provide a significant performance boost even if nothing else changes.
> The stripe size impacts when the system does can avoid doing a
> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads. I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
So reducing the chunk size will have minimal impact... but reducing it
should still provide some performance boost. Since I'm recreating the
array anyway, what size makes the most sense? 16k or go straight to the
minimum of 4k? Would a smaller chunk size increase the IOPS because we
need to make more (smaller) requests for the same data, potentially from
more drives?
ie, currently, a single read request for 4k will be done by reading one
chunk (64k) from one of the 8 drives (1 IOPS)
currently, a single write request for 4k will be done by reading one
chunk (64k) from 6 drives, and then writing one chunk (64k) to two
drives (8 IOPS)
However, a read (or write) 48k request would be identical to the above,
while a smaller chunk size (4k) would mean:
read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from
2 disks (7 IOPS)
write request - write 8 x 4k (full stripe) (assuming it is stripe
aligned somewhere, but it might not be)
- read 2 x 4k chunks (the only 2 data chunks that
won't be written) + write 6 x 4k chunks
Total of 16 IOPS in the best case, worst case is two partial stripe
writes + 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
Either the above is wrong, or I've just convinced myself that reducing
the chunk size is not a good idea...
> DRBD can saturate GigE without any problem with random 4K writes. I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS. The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach. Some vendors
> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
I have 10G ethernet for the crossover between the two servers, and
another 10G ethernet to connect off to the "clients". Bandwidth
utilisation on either of these is rather low (I think it maxed out at
around 15 to 20%) definitely not anywhere near 100%. My thought here was
on the latency of the connection, but I really didn't have any ideas on
how to measure that, and how to test if it would really help. Also
equipment seems a little less common, and complex...
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
> This is probably not true. If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification. SSDs can be
> bad, but usually not that bad.
>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly. I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves. The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.
My environment is rather small in comparison, it is only around 20 VM's
supporting around 80 users. 5 of the VM's are RDP servers.
>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
> RAID-10 will be faster, but you pay for this with capacity. It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over. Also remember that free
> space on the SSDs only counts if it is actually unallocated. So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity. It is best to leave an empty partition
> that is untouched.
Good point, when I initially provisioned the drives, I only used the
first 400GB, and left 80GB on each drive unpartitioned. As we ran out of
space, I was forced to allocate all of it. The place is to only end up
with 960GB of each 1000GB drive in use, so I could again leave a small
chunk of un-allocated space.
>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll. Most SSDs are also not quite symmetrical in terms of
> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K. But your mileage may vary. If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS. With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
then combined to RAID0 (or linear)? I'd end up with 6TB of usable space
(8 x 1TB - 2 parity) though I'm guessing it is better to upgrade to
kernel 4.4 instead which would basically do the same thing?
> You also need to consider what raid does to the SSD FTL. As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives. raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
Overall drive wear is doing pretty well, it is sitting at around 5% to
8% per year.
Tell me I'm crazy, but one option that I considered is using different
RAID levels. Right now I have RAID51 in that I have RAID5 on each
machine and DRBD (RAID1) between them.
What if I used RAID01 with DRBD between the machines doing the RAID1. In
this way, each machine has RAID0 (across 8 drives), which should provide
maximum performance and storage capacity and DRBD doing RAID1 between
the two machines. It feels rather risky, but perhaps it isn't a terrible
idea?
Slightly better would be RAID10 with DRBD between each pair of drives,
and then RAID0 across the DRBD device. It adds another layer of RAID,
and more complexity, but better security than RAID01...
Regards,
Adam
^ permalink raw reply
* Re: SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: David C. Rankin @ 2016-07-27 23:12 UTC (permalink / raw)
To: mdraid
In-Reply-To: <f0ebae2f-472f-6835-d74b-f3fccd4c6b04@turmel.org>
On 07/27/2016 09:22 AM, Phil Turmel wrote:
> I always place of=/dev/.... last on a dd command line just in case.
>
> Using dd to write near the end of a member device is entirely safe while
> running if the location is after the "Used Data Area"+"Data Offset" of
> the device, as reported by mdadm --examine.
>
> If you are zeroing a backup GPT that has corrupted part of your data
> inside the data area, it doesn't do any additional harm. So don't
> bother using --fail and --remove on the member devices.
I wondered about that, but I figured it was safer to take the drive out of the
array and operate on each separately. One more chunk of good learning that has
come out of this thread for me. Thanks.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: David C. Rankin @ 2016-07-27 23:10 UTC (permalink / raw)
To: mdraid
In-Reply-To: <ac25abf7-b97e-1db4-d227-2ee57a02ed90@youngman.org.uk>
On 07/27/2016 08:04 AM, Anthony Youngman wrote:
> WD Blacks? Do they support SCT/ERC? I think these are desktop drives (like my
> Barracudas) so you WILL get bitten by the timeout problem if anything goes
> wrong. Do you know what you're doing here?
Yes, WD Blacks, and yes, at least for the last 16 years I've managed, somehow,
to provide a complete open-source backend for my law office. So I would answer
the 2nd question in the affirmative as well. You can poo-poo drive X verses
drive Y all you want, but I get a consistent 5 years out of each WD black and
plan on a replacement cycle of 1/2 that. Go with what works for you.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: RAID5 Performance
From: Doug Dumitru @ 2016-07-27 17:38 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
In-Reply-To: <22424.50313.900162.715200@tree.ty.sabi.co.uk>
On Wed, Jul 27, 2016 at 7:26 AM, Peter Grandi <pg@lxra2.for.sabi.co.uk> wrote:
> First a terminology point: you are reporting a *speed* problem,
> not a performance problem. My impression that you are getting
> pretty good performance, given your hardware, configuration and
> workload. The difference between "speed" and "performance" is
> that the first is a simple rate of stuff done per unit of time,
> the second is an envelope embodying several tradeoffs, among
> them with cost. In your question you describe a low rate of
> stuff done per unit of time. :-)
>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
>> on top, DRBD on top, and finally iSCSI on top (and then used
>> as VM raw disks for mostly windows VM's).
>
> A very brave configuration, a shining example of the "syntactic"
> mindset, according to which any arbitrary combination of
> legitimate features must be fine :-).
While you may say that this configuration is very "brave", it is
actually quite common for VDI "appliance" deployments. If you look at
cluster solutions like Ganeti, it is exactly this stack less the iSCSI
(Ganeti runs storage and VM compute on the same nodes). It is also a
"LOT" faster and a lot lower wear than using a file system like ZFS to
create ZVOLs and export those. The other option is to run a
file-system and export files as block devices. This just replaces LVM
with EXT4/XFS, and for static "blobs", LVM is a bunch faster and
safer.
>
> First server DRBD primary disks:
>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55 0.02 0.15 0.20 0.09 0.15 2.40
>> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39 0.02 0.14 0.24 0.05 0.11 1.60
>> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96 0.03 0.17 0.09 0.24 0.09 1.60
>> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96 0.08 0.36 0.29 0.40 0.34 7.20
>> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76 0.08 0.42 0.32 0.51 0.29 5.20
>> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62 0.01 0.04 0.09 0.00 0.04 0.80
>> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53 0.01 0.06 0.04 0.08 0.04 0.80
>> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24 0.18 0.28 0.23 0.32 0.16 9.60
>
> Second server DRBD secondary disks:
>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17 0.26 1.47 0.06 2.27 1.45 25.60
>> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38 0.45 2.76 0.08 3.93 2.71 44.40
>> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40 0.75 4.73 0.00 6.90 4.70 74.80
>> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12 0.81 5.21 0.23 7.69 5.13 80.00
>> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13 0.94 4.59 0.69 6.85 4.24 86.40
>> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29 0.84 5.03 0.21 7.60 4.89 81.60
>> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84 0.69 4.27 0.14 6.54 4.22 68.00
>> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92 1.21 2.43 1.88 2.90 1.50 74.40
>> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38 0.00 0.00 0.00 0.00 0.00 0.00
>
>> I've confirmed that the problem is that we have mixed two
>> models of SSD (520 series and 530 series), and that the 530
>> series drives perform significantly worse (under load) in
>> comparison.
>
> The queue sizes and waiting time on the second server are very
> low (on a somewhat similar system using 4TB disks I see waiting
> times in the 1-5 seconds range, not milliseconds).
The expectation, in terms of performance for VDI is quite high.
vmWare like to say you can get away with 8-12 IOPS per virtual. Most
people think you only get good performance with 100 IOPS per virtual.
The "bad" time for VDI is what is called a "boot storm". Boot, or
reboot, all of the windows clients at the same time and see how long
they take to settle. The IO workload for this is 80%+ 4K random
writes. At 100 IOPS, it takes windows about 2 minutes to boot, so if
you need to support 500 VDI seats from a storage node, that nodes
needs to sustain 500 x 100 = 50,000 IOPS of 4K random writes. This is
so far past what hard disks can do as to be silly. Even with SSDs,
you need reasonably large arrays running RAID-10 to sustain this. If
you want to support 5000 VDI seats like this, then stock Linux just
can't get there, but it can be done.
>
> The impression I get is that there is some issue with DRBD
> latency, because the second server's storage seems to me very
> underutilized. This latency may be related to the flash SSDs
> that you are using, because by default DRBD uses the "C"
> synchronization protocol. Probably if you switched to the "B" or
> even "A" protocols speed could improve, maybe a lot, even if
> performance arguably would be the same or much worse.
>
> Thus the most likely issue here is the 'fsync' problem: for
> "consumerish" SSDs barrier-writes are synchronous, because they
> don't have a battery/capacitor-backed cache, and rather slow for
> small writes, because of the large size of erase blocks, which
> can be mitigated with higher over-provisioning. These have much
> of the story:
On many consumer SSDs, barrier writes are only barriers, and are not
syncs at all. You are guaranteed serialization but not actual
storage. Then again, in a server setup, especially with redundant
power supplies, power loss to the SSDs is rare. You are more
protecting against system hangs and other inter-connectivity issues.
The real system solution is to have some quantity of non volatile DRAM
that you can stage writes (either a PCI-e card like a FlashTec or one
or more nvDIMMs). This is how the "major vendors" deal with sync
writes.
> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
> https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
> http://www.spinics.net/lists/ceph-users/msg25928.html
>
> The 520s seem not too bad, but still a long way from the disks
> with battery/capacity-backed cache.
>
>> the actual work-load is small random read/write, with the
>> writes causing the biggest load.
>
> Here most of the wise comments from the reply from D Dimitru
> apply, to summarize:
>
> * Small writes are a challenging workload for DRBD, regardless
> of other issues.
My comment about DRBD was not that small writes are harder, but that
if your target can keep up with them at low queue depths, then DRBD
can saturate GigE at 4K q=1 on a single thread. So DRBD is not really
the issue, but the latency/IOPS behaviour of the target.
> * Small writes are a very challenging workload for flash SSDs
> without battery/capacitor-backed caches.
Even with battery backup, small writes create garbage collection, so
while batteries may give you some short term bursts, longer term, you
still have to do the writes. A main benefit of batter backup in the
SSDs is that the meta data (mapping information) does not need to be
flushed with the actual data real-time, which makes the FTL algorithms
easier to implement.
> * Parity RAID is a bad idea in general, in particular for
> workloads with many small writes, for they amplify writes via
> RMW.
>
> etc. etc. :-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
WildFire Storage
http://www.wildfire-storage.com
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Peter Grandi @ 2016-07-27 15:12 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <CADzS=arvFWy-9_e8L3ngdrYxwGxVNm=zTqoLusvwKmmkUnBfQQ@mail.gmail.com>
BTW as to in general this question:
> which one is the safest ?
That would be the first option I presented, where the 3x 4TB
disks get split in two, and the existing 3 2TB slices get copied
onto them, as I wrote:
> The least scary option might be:
>
> * Split into two partitions D and E.
> * Block copy the image on C to D1 and E2.
> * Re-partition C also in two.
> * Block copy B to C1.
> * Block copy A to E2.
> * Now we have greatly increased redundancy, as we have the 3x
> 2TB data slices on two separate sets, A, C1, D1 and D2, B,
> E2.
That is about as good as it gets as to safety with the limited
number of drives in play. BTW note that it is really important
not to start the second copy, because it has the same set and
member UUIDs as the other one. But the O.P. must be a very
experienced RAID specialist, or he would not be trying to do
subtle device restructuring like this.
> * Add E1 as a spare to the A, C1, D1 RAID5 set, and start it so
> you end up with a full RAID5 set on A, C1, D1, E1 after
> resync end.
This resync is less dangerous because we have a full (if
degraded) copy of the data on D2, B, E2.
With an extra 4TB drive, even a temporary one, to hold an
offline backup (if the 4.5TB can be compressed to fit), more
dangerous options become less scary.
> * Now that the first RAID5 set is not degraded, you can erase
> the copies on D2, B, E2, and create a second RAID5 set on
> B, C2, D2, E2, which will be empty.
^ permalink raw reply
* Re: Lost RAID6 disks when moving to new PC
From: Alex Owen @ 2016-07-27 15:04 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160727115740.bnwi4hntyjnb4jy3@metamorpher.de>
Thanks for the help. I've created the overlays for all the disks
sd[a-f] and here are the hex dumps. I'm going to try re-creating the
array in all of the six combinations and see what happens.
After this I plan to back everything up to new drives and recreate the
array properly.
Here are the hex dumps from each drive:
a:
00001000 fc 4e 2b a9 01 00 00 00 00 00 00 00 00 00 00 00 |.N+.............|
00001010 b3 77 d9 75 86 be b8 6c 9d a9 f2 1d f7 3b 24 51 |.w.u...l.....;$Q|
00001020 4e 41 53 3a 30 00 00 00 00 00 00 00 00 00 00 00 |NAS:0...........|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00001040 11 bf a3 56 00 00 00 00 06 00 00 00 02 00 00 00 |...V............|
00001050 00 a0 4c 5d 01 00 00 00 00 04 00 00 06 00 00 00 |..L]............|
00001060 00 00 00 00 06 00 00 00 00 80 32 75 05 00 00 00 |..........2u....|
00001070 00 00 00 00 02 00 00 00 00 04 00 00 00 00 00 00 |................|
00001080 00 00 04 00 00 00 00 00 b0 a3 4c 5d 01 00 00 00 |..........L]....|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000010a0 02 00 00 00 00 00 00 00 01 37 40 d6 5c c4 45 e7 |.........7@.\.E.|
000010b0 62 5f 32 57 86 08 da ec 00 00 00 00 00 00 00 00 |b_2W............|
000010c0 69 d4 96 57 00 00 00 00 8c ff 2c 00 00 00 00 00 |i..W......,.....|
000010d0 ff ff ff ff ff ff ff ff bd 6e ba 14 80 00 00 00 |.........n......|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001100 00 00 fe ff 02 00 fe ff 03 00 04 00 01 00 05 00 |................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff |................|
*
00001200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
b:
00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
c:
00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
d:
00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
e:
00001000 fc 4e 2b a9 01 00 00 00 00 00 00 00 00 00 00 00 |.N+.............|
00001010 b3 77 d9 75 86 be b8 6c 9d a9 f2 1d f7 3b 24 51 |.w.u...l.....;$Q|
00001020 4e 41 53 3a 30 00 00 00 00 00 00 00 00 00 00 00 |NAS:0...........|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00001040 11 bf a3 56 00 00 00 00 06 00 00 00 02 00 00 00 |...V............|
00001050 00 a0 4c 5d 01 00 00 00 00 04 00 00 06 00 00 00 |..L]............|
00001060 00 00 00 00 06 00 00 00 00 80 32 75 05 00 00 00 |..........2u....|
00001070 00 00 00 00 02 00 00 00 00 04 00 00 00 00 00 00 |................|
00001080 00 00 04 00 00 00 00 00 b0 a3 4c 5d 01 00 00 00 |..........L]....|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000010a0 00 00 00 00 00 00 00 00 19 09 6a b0 3e 3a ac 62 |..........j.>:.b|
000010b0 7a fd fc 97 86 ee b5 f6 00 00 00 00 00 00 00 00 |z...............|
000010c0 69 d4 96 57 00 00 00 00 8c ff 2c 00 00 00 00 00 |i..W......,.....|
000010d0 ff ff ff ff ff ff ff ff cc 3a f1 b4 80 00 00 00 |.........:......|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001100 00 00 fe ff 02 00 fe ff 03 00 04 00 01 00 05 00 |................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff |................|
*
00001200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
f:
00001000 fc 4e 2b a9 01 00 00 00 00 00 00 00 00 00 00 00 |.N+.............|
00001010 b3 77 d9 75 86 be b8 6c 9d a9 f2 1d f7 3b 24 51 |.w.u...l.....;$Q|
00001020 4e 41 53 3a 30 00 00 00 00 00 00 00 00 00 00 00 |NAS:0...........|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00001040 11 bf a3 56 00 00 00 00 06 00 00 00 02 00 00 00 |...V............|
00001050 00 a0 4c 5d 01 00 00 00 00 04 00 00 06 00 00 00 |..L]............|
00001060 00 00 00 00 06 00 00 00 00 80 32 75 05 00 00 00 |..........2u....|
00001070 00 00 00 00 02 00 00 00 00 04 00 00 00 00 00 00 |................|
00001080 00 00 04 00 00 00 00 00 b0 a3 4c 5d 01 00 00 00 |..........L]....|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000010a0 06 00 00 00 00 00 00 00 dc 42 4c 5a b4 84 5f 55 |.........BLZ.._U|
000010b0 37 3a 04 e0 78 d9 67 23 00 00 00 00 00 00 00 00 |7:..x.g#........|
000010c0 69 d4 96 57 00 00 00 00 8c ff 2c 00 00 00 00 00 |i..W......,.....|
000010d0 ff ff ff ff ff ff ff ff b9 e6 3f c6 80 00 00 00 |..........?.....|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001100 00 00 fe ff 02 00 fe ff 03 00 04 00 01 00 05 00 |................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff |................|
*
00001200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
Thanks,
Alex
On 27 July 2016 at 12:57, Andreas Klauer <Andreas.Klauer@metamorpher.de> wrote:
> On Wed, Jul 27, 2016 at 11:35:34AM +0100, Alex Owen wrote:
>> The array should be RAID6 on /dev/sd{a-f}.
>
> Full disk raid sucks. Zero advantages, lots of additional risk.
> There are too many programs out there that expect every disk to have
> a partition table, and will use it unasked if it looks unpartitioned.
>
> You seem to have lost your md metadata to some partitioner/installer,
> you're also the third person with this problem in a row. Congrats. ;)
>
>> fdisk -l :
>
> Your fdisk doesn't support GPT, don't use it.
>
>> parted
>> Disk /dev/sd[bcd]: 3001GB
>> Sector size (logical/physical): 512B/4096B
>> Partition Table: gpt
>>
>> Number Start End Size File system Name Flags
>> 1 17.4kB 134MB 134MB Microsoft reserved
>> partition msftres
>> 2 135MB 3001GB 3000GB Basic data partition
>> msftdata
>
> Well, something put GPT partition table on those. GPT overwrites start
> and end of the disk. You're using 1.2 metadata which is located 4K from
> the start, can you show some hexdump for those disks?
>
> hexdump -C -s 4096 -n 4096 /dev/sdb
>
>> And the output of mdadm --examine /dev/sd[a-f]
>>
>> ----------
>>
>> /dev/sda:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
>> Name : NAS:0 (local to host NAS)
>> Creation Time : Sat Jan 23 17:57:37 2016
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
>> Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
>> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : 013740d6:5cc445e7:625f3257:8608daec
>>
>> Update Time : Tue Jul 26 04:09:29 2016
>> Checksum : 14ba6ebd - correct
>> Events : 2949004
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAAAAA ('A' == active, '.' == missing)
>
>> /dev/sdb:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdc:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> /dev/sdd:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>
> Basically what we know is... your disk order for three disks
>
> (/dev/sde = role 0, /dev/sdf = role 1, /dev/sda= role 2)
>
> and what we don't know is the disk order of /dev/sd[bcd].
>
> If the metadata is lost completely, only thing you can do is re-create
> the RAID with all possible orders efa{bcd,bdc,cbd,cdb,dbc,dcb}.
>
> Re-creating is dangerous so you should use an overlay: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> When re-creating you have to specify all variables (level, layout, chunksize,
> data offset, order, ...) since the defaults picked by mdadm might differ
> depending on your mdadm version.
>
> Example command: (untested)
>
> mdadm --create /dev/md42 --assume-clean \
> --level=6 --chunk=512 --data-offset=128M --layout=ls \
> /dev/overlay/sd{e,f,a,b,c,d}
>
> Then you check if it can be mounted, and once mounted if big files
> (larger than chunksize * number of disks) are intact or no. If you
> switch the wrong two disks it may mount but data is garbage anyway.
>
> Regards
> Andreas Klauer
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Peter Grandi @ 2016-07-27 14:57 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <5798BAA3.60108@youngman.org.uk>
>>> On Wed, 27 Jul 2016 14:44:03 +0100, Wols Lists <antlists@youngman.org.uk> said:
> On 27/07/16 14:22, Peter Grandi wrote:
>> Also, if the data is just 4.5TB, why not go for just a 3x
>> RAID5 with the 3x 4TB drives, which a usable capacity of 8TB?
>> A degree of redundancy of 1-in-3 is not bad. It would have
>> less IOPS though.
> The problem is that one of the three new 4TB drives is
> currently part of the old - partly failed - array. So without
> buying a fourth 4TB drive, that option isn't on the table.
A and B are the 2TB drives, C, D and E the 4TB drives.
* Copy the 4.5TB if they can be compressed to drive E (it does
not neet to be partitioned).
* Setup drives C and D as a degraded (3-1)x RAID5.
* Create a RAID0 of the 2x 2TB drives.
* Add the RAID0 as the 3rd member to the new RAID5.
* Copy the 4.5TB from drive E to the new RAID5.
* Wait for sync to finish.
* Stop the RAID5 set.
* Copy by 'dd' the RAID0 set's content to drive E.
* Stop the RAID0 set and '--zero' A and B.
* Start the RAID5.
* Reuse A and B for something else.
^ permalink raw reply
* Re: RAID5 Performance
From: Peter Grandi @ 2016-07-27 14:26 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <7b7d730f-2951-ba5f-7f6b-33624b59a02d@websitemanagers.com.au>
First a terminology point: you are reporting a *speed* problem,
not a performance problem. My impression that you are getting
pretty good performance, given your hardware, configuration and
workload. The difference between "speed" and "performance" is
that the first is a simple rate of stuff done per unit of time,
the second is an envelope embodying several tradeoffs, among
them with cost. In your question you describe a low rate of
stuff done per unit of time. :-)
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
> on top, DRBD on top, and finally iSCSI on top (and then used
> as VM raw disks for mostly windows VM's).
A very brave configuration, a shining example of the "syntactic"
mindset, according to which any arbitrary combination of
legitimate features must be fine :-).
First server DRBD primary disks:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24 0.18 0.28 0.23 0.32 0.16 9.60
Second server DRBD secondary disks:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38 0.00 0.00 0.00 0.00 0.00 0.00
> I've confirmed that the problem is that we have mixed two
> models of SSD (520 series and 530 series), and that the 530
> series drives perform significantly worse (under load) in
> comparison.
The queue sizes and waiting time on the second server are very
low (on a somewhat similar system using 4TB disks I see waiting
times in the 1-5 seconds range, not milliseconds).
The impression I get is that there is some issue with DRBD
latency, because the second server's storage seems to me very
underutilized. This latency may be related to the flash SSDs
that you are using, because by default DRBD uses the "C"
synchronization protocol. Probably if you switched to the "B" or
even "A" protocols speed could improve, maybe a lot, even if
performance arguably would be the same or much worse.
Thus the most likely issue here is the 'fsync' problem: for
"consumerish" SSDs barrier-writes are synchronous, because they
don't have a battery/capacitor-backed cache, and rather slow for
small writes, because of the large size of erase blocks, which
can be mitigated with higher over-provisioning. These have much
of the story:
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
http://www.spinics.net/lists/ceph-users/msg25928.html
The 520s seem not too bad, but still a long way from the disks
with battery/capacity-backed cache.
> the actual work-load is small random read/write, with the
> writes causing the biggest load.
Here most of the wise comments from the reply from D Dimitru
apply, to summarize:
* Small writes are a challenging workload for DRBD, regardless
of other issues.
* Small writes are a very challenging workload for flash SSDs
without battery/capacitor-backed caches.
* Parity RAID is a bad idea in general, in particular for
workloads with many small writes, for they amplify writes via
RMW.
etc. etc. :-)
^ permalink raw reply
* Re: SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: Phil Turmel @ 2016-07-27 14:22 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <57985F10.4070706@suddenlinkmail.com>
Hi David,
On 07/27/2016 03:13 AM, David C. Rankin wrote:
> Chris, Phil, All,
>
> Thank you. For anyone else that is faced with the problem where you are using
> whole disks in your raid1 array over the top of unused sub-partitions, here is
> the 5 minute fix.
Glad you got it sorted out.
> How I solved the problem:
>
> (1) do NOT attempt to alter the disk in a partitioning package like fdisk,
> sfdisk, gdisk, parted, etc.. A write after you delete the unused partitions with
> adversely affect the md data and will require a long and painful resync
> depending on the size of your drive.
Correct.
> (2) simply --fail and --remove one drive from the array. My array was
> /dev/md4, and failing and removing /dev/sdd from the array was as simple as:
>
> # mdadm /dev/md4 --fail /dev/sdd
> # mdadm /dev/md4 --remove /dev/sdd
Unnecessary, and you have no redundancy for the duration. Using dd to
wipe the first 4k of a v1.2 array member is entirely safe to do while
running.
> (3) To remove the inadvertent partition on the drive while keeping the raid
> data in tact, you must remove the PMBR and primary Partition tables from the
> drive. You can use `wipefs` or simply use `dd` to overwite the first 4096 bytes
> on the drive with zeros and then the last 1024 bytes before the end of the disk
> to remove the backup GPT header. (I overwote the last 4096 bytes on the disk,
> just to make sure -- I had nothing in the last 100M of the disk, so that seemed
> fine) You can look at the disk geometry reported by gdisk to find the end of the
> disk (the number of logical sectors -- make sure the disk has 512-byte sectors,
> or dd option adjustments will be needed) (then just subtract 8 from that number
> (or 2 if you wish to limit the write to 1024 bytes) and use that as the 'seek'
> offset with 'dd', so
>
> # dd of=/dev/sdd if=/dev/zero bs=4096 count=1
> # dd of=/dev/sdd if=/dev/zero bs=512 count=8 seek=5860533160
I always place of=/dev/.... last on a dd command line just in case.
Using dd to write near the end of a member device is entirely safe while
running if the location is after the "Used Data Area"+"Data Offset" of
the device, as reported by mdadm --examine.
If you are zeroing a backup GPT that has corrupted part of your data
inside the data area, it doesn't do any additional harm. So don't
bother using --fail and --remove on the member devices.
Phil
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Wols Lists @ 2016-07-27 13:44 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
In-Reply-To: <22424.46474.87462.226497@tree.ty.sabi.co.uk>
On 27/07/16 14:22, Peter Grandi wrote:
> Also, if the data is just 4.5TB, why not go for just a 3x RAID5
> with the 3x 4TB drives, which a usable capacity of 8TB? A degree
> of redundancy of 1-in-3 is not bad. It would have less IOPS
> though.
The problem is that one of the three new 4TB drives is currently part of
the old - partly failed - array. So without buying a fourth 4TB drive,
that option isn't on the table.
Cheers,
Wol
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Anthony Youngman @ 2016-07-27 13:27 UTC (permalink / raw)
To: bobzer, Adam Goryachev, Peter Grandi, Linux RAID
In-Reply-To: <CADzS=arvFWy-9_e8L3ngdrYxwGxVNm=zTqoLusvwKmmkUnBfQQ@mail.gmail.com>
On 27/07/16 03:16, bobzer wrote:
> Thank you very much for your help
>
> I realized that there's much better idea than my first idea....
>
> I can plug all my drives the sata plug are not a problem :-)
> I use around 4.5TB on my current raid 5
> I will continue to think on this problem and in the meanwhile try to
> find another disk
>
> i didn't think about :
>>> In general an in-place migration is a very dangerous operation
>>> because it stresses existing hardware a lot plus it uses code
>>> that is rarely used and is quite complex. Given that your
>>> situation is already compromised.
>> But copying everything off will stress it just as much, surely? The
>> alternatives imho are worse ...
> at first i thought about using rsync to do my copy (anyway i don't have a GUI)
> at second i thought that an in-place migration would be a nice and
> safe operation
> now i don't what to think ?
> the 3 4TB disk are new but the 2 2TB are 4 year old and consumer
> grade (the 4TB are seagate NAS HDD)
> so i'm asking myself should I do a copy (with rsync or another tool)
> or an in-place migration ? which one is the safest ?
>
>
Get another 4TB drive?
Firstly, desktop grade is dangerous. You should never have been using
these disks in a raid5 (is that why you lost the other two drives?).
Have you still got those drives? If so, try wiping them with "dd
if=/dev/random of=/dev/dud-drive" (I've said /dev/random rather than
/dev/zero because I'm worried the drive might try and do something
clever). This may get those drives back and usable, although I wouldn't
trust them. IFF that works, you now have your 2 x 2TB array back.
(It'll hopefully fix any dud sectors by overwriting them.)
You can now create a 3 x 4TB raid-5 and copy your data across. If it
doesn't work, you can see why you need to get that next 4TB drive :-)
And once you've got your fourth drive, you have your raid6.
The big problem you have is that your data is over 4TB, and won't fit on
one drive. So there is no way, whether you do an in-place migration, or
a copy-to-new-raid, that you can transfer the data without losing raid
protection somewhere along the line unless you get more disk space.
With that extra 4TB drive, personally I'd migrate in place - as I said
add in the three new 4TB drives, convert the two 2TB drives to a 4TB
raid0 so I can free up the 4TB drive with the 2TB image, then use that
freed drive to replace the raid0.
Cheers,
Wol
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Peter Grandi @ 2016-07-27 13:22 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <CADzS=arvFWy-9_e8L3ngdrYxwGxVNm=zTqoLusvwKmmkUnBfQQ@mail.gmail.com>
[ ... ]
> I can plug all my drives the sata plug are not a problem :-) I
> use around 4.5TB on my current raid 5
That opens up some more options, if it can be cut down to less
than 4TB, which may be possible e.g. by compression (unless most
files are already compressed), for example by doing a full
backup of your existing content as a '.tar.lzo' to one of the
new 4TB drives, which would give you backup, and one from which
after you create a new RAID set, a source to repopulate your
data.
Also, if the data is just 4.5TB, why not go for just a 3x RAID5
with the 3x 4TB drives, which a usable capacity of 8TB? A degree
of redundancy of 1-in-3 is not bad. It would have less IOPS
though.
Another option I did not mention would be to split the 3x 4TB
drives in two, and then create a 3x RAID5 set on one half with a
4TB usable capacity, and a 5x RAID6 set on the other half plus
the 2x 2TB drives, with a 6TB usable capacity.
> at first i thought about using rsync to do my copy
The 'rsync' is good, I use that especially for updates, but
something like this is often better for an initial full copy
because it parallelizes a bit reading and writing:
tar -C $FROM -c --one -b 512 -f - . \
| tar -C $TO -x --preserve-p -b 512 -f -
The alternative of a 'dd' block-by-block physical copy of the
existing members onto suitably sized partitions on the new
drives, which would be faster and safer; but a 'rsync'/'tar'
file-by-file logical copy has the advantage of defragmenting the
destination, at the price of much slower and riskier seek-based
reading of the source.
> in-place migration would be a nice and safe operation now i
> don't what to think ? [ ... ]
Most in-place migrations succeed, even if very very slowly.
The problem happens in the rare but not so uncommon cases where
they fail: if you have a full backup it's not a big problem, if
you don't then it gets rather "challenging".
^ permalink raw reply
* Re: The subarray is loaded container by load_container
From: Xiao Ni @ 2016-07-27 13:19 UTC (permalink / raw)
To: Artur Paszkiewicz, linux-raid
Cc: Jes Sorensen, tomasz.majchrzak, aleksey.obitotskiy,
pawel.baldysiak
In-Reply-To: <4f57d953-8545-4647-49e7-c434e1892fb6@intel.com>
On 07/07/2016 06:16 PM, Artur Paszkiewicz wrote:
> On 07/07/2016 11:13 AM, Xiao Ni wrote:
>> Hi all
>>
>> We encountered one problem at booting time:
>>
>> Run MD devices: mdadm: array /dev/md/OSVOLUME0 now has 2 devices (0 new) [FAILED]
>>
>> mdadm -IRs returns 1 at the booting time. In fact the array is already running. The
>> command mdadm -IRs shouldn't return 1 if all the arrays are running, right?
>>
>> In IncrementalScan function, it scans all the raid in /run/mdadm/map. It contains the
>> subarray too. The return value is 1 from function load_container. So the return value
>> of IncrementalScan rv is set to 1. If all arrays are running already, I think mdadm -IRs
>> should return 0, not 1.
>>
>> Should we checks earlier whether it's a container or not before calling load_container?
>>
>> Best Regards
>> Xiao
>>
> Hi Xiao,
>
> I think you're right that mdadm -IRs should return 0 in this case.
> IncrementalScan should not try loading a container from a member array,
> because that will always fail. Can you check if this fixes the problem
> for you?
>
> diff --git a/Incremental.c b/Incremental.c
> index ba97b00..cc01d41 100644
> --- a/Incremental.c
> +++ b/Incremental.c
> @@ -1347,8 +1347,12 @@ restart:
>
> if (devnm && strcmp(devnm, me->devnm) != 0)
> continue;
> - if (devnm && me->metadata[0] == '/') {
> + if (me->metadata[0] == '/') {
> char *sl;
> +
> + if (!devnm)
> + continue;
> +
> /* member array, need to work on container */
> strncpy(container, me->metadata+1, 32);
> container[31] = 0;
>
> Thanks,
> Artur
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Artur
Sorry for late response. I have been waiting the result from customer
and the customer haven't give me the answer. I tested this in my
environment and this patch can fix this problem.
Best Regards
Xiao
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Anthony Youngman @ 2016-07-27 13:10 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtQpGVf2Tj13o57T4=raF=czYr-V99tEDSkJ-_buk7JtTg@mail.gmail.com>
On 27/07/16 00:18, Chris Murphy wrote:
>> The real question for me is what is the effect of having /dev/sdc1 and
>> >/dev/sdd1 as unused partitions on the drive while I'm using the whole drive. Is
>> >that something that can bite me later?
> It already bit you. All you have to do is forget again that you're not
> using this partition table for anything, and then try to repair it and
> you're back in this same situation. You or someone else who ends up
> managing the drive. So yeah, it's not an in-use valid structure so I'd
> invalidate it so that libblkid unambiguously tells you the only
> signatures that matter onthe drive -> drives are not partitioned, they
> are completely under the control of mdadm, and the logical array from
> those members is ext4 or whatever.
And once you've blown away the GPT, those partitions won't exist anyway.
The only way anything should be able to find any trace of them will be
if you run a forensic data recovery tool.
Cheers,
Wol
^ permalink raw reply
* Re: SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: Anthony Youngman @ 2016-07-27 13:04 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <57985F10.4070706@suddenlinkmail.com>
On 27/07/16 08:13, David C. Rankin wrote:
> In my circumstance, I had partitioned a pair of 3T WD Black drives for use in
> a raid1 array.
WD Blacks? Do they support SCT/ERC? I think these are desktop drives
(like my Barracudas) so you WILL get bitten by the timeout problem if
anything goes wrong. Do you know what you're doing here?
Cheers,
Wol
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Peter Grandi @ 2016-07-27 12:55 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <5797C5CE.6020300@youngman.org.uk>
>> In general an in-place migration is a very dangerous operation
>> because it stresses existing hardware a lot plus it uses code
>> that is rarely used and is quite complex.
> But copying everything off will stress it just as much,
> surely? The alternatives imho are worse ...
* The very different access patterns of a block-by-block copy to
new fresh disks and the migration-in-place having to rewrite by
moving around the contents of all involved disks.
* That in a copy there is always the original to fall back on, a
migration-in-place that fails can be fatal.
I am giving for granted here that one way or another at least one
copy of all the data has to be done; and block-by-block device
sequential copy is probably the best, followed by file-by-file
sequential copy, followed by in-place migration.
Note: RAID5 in-place migration does not mean that existing blocks
stay where they are, it means that (nearly) all blocks get moved,
but inside the existing device set.
So probably the best combination here, because the O.P. does not
currently have any redundancy, is to first ensure there is some
kind of copy of the data and/or redundancy, on the way to getting
a new storage layout. Because *any* whole-content data operation
stresses the storage system.
^ permalink raw reply
* test file disappeared after -f, -r, --add-journal for write-journal device
From: Yi Zhang @ 2016-07-27 12:07 UTC (permalink / raw)
To: linux-raid; +Cc: shli, neilb, Jes.Sorensen
In-Reply-To: <678915290.9410266.1469620858173.JavaMail.zimbra@redhat.com>
Hello maintainer
Here is another strange phenomenon I found after do -f, -r, --add-journal for write-journal device.
Kernel version: 4.7.0-rc7
Steps I used:
mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 7 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 --write-journal /dev/loop0 --bitmap=internal --bitmap-chunk=64M --chunk 512
mdadm --wait /dev/md0
mkfs.ext4 /dev/md0
mount -t ext4 /dev/md0 /mnt/fortest
cp bigfile /mnt/fortest &
wait
md5sum /mnt/fortest/bigfile > md5sum3
mdadm /dev/md0 -f /dev/loop0
mdadm /dev/md0 -r /dev/loop0
umount /dev/md0 -l
mdadm -o /dev/md0
mdadm /dev/md0 --add-journal /dev/loop0
mdadm --wait /dev/md0
mdadm -D /dev/md0
mount /dev/md0 /mnt/fortest
md5sum /mnt/fortest/bigfile > md5sum2 #<----this test file disappeared
kernel log:
<6>[ 3452.198142] md: bind<loop1>
<6>[ 3452.198176] md: bind<loop2>
<6>[ 3452.198202] md: bind<loop3>
<6>[ 3452.198224] md: bind<loop4>
<6>[ 3452.198244] md: bind<loop5>
<6>[ 3452.198265] md: bind<loop6>
<6>[ 3452.198289] md: bind<loop0>
<6>[ 3452.198317] md: bind<loop7>
<6>[ 3452.203183] md/raid:md0: device loop6 operational as raid disk 5
<6>[ 3452.203186] md/raid:md0: device loop5 operational as raid disk 4
<6>[ 3452.203187] md/raid:md0: device loop4 operational as raid disk 3
<6>[ 3452.203188] md/raid:md0: device loop3 operational as raid disk 2
<6>[ 3452.203189] md/raid:md0: device loop2 operational as raid disk 1
<6>[ 3452.203190] md/raid:md0: device loop1 operational as raid disk 0
<6>[ 3452.203684] md/raid:md0: allocated 7548kB
<1>[ 3452.203796] md/raid:md0: raid level 4 active with 6 out of 7 devices, algorithm 0
<7>[ 3452.203800] RAID conf printout:
<7>[ 3452.203801] --- level:4 rd:7 wd:6
<7>[ 3452.203802] disk 0, o:1, dev:loop1
<7>[ 3452.203803] disk 1, o:1, dev:loop2
<7>[ 3452.203804] disk 2, o:1, dev:loop3
<7>[ 3452.203805] disk 3, o:1, dev:loop4
<7>[ 3452.203806] disk 4, o:1, dev:loop5
<7>[ 3452.203807] disk 5, o:1, dev:loop6
<6>[ 3452.203815] md/raid456: discard support disabled due to uncertainty.
<6>[ 3452.203816] Set raid456.devices_handle_discard_safely=Y to override.
<6>[ 3452.203819] md/raid:md0: using device loop0 as journal
<6>[ 3452.204161] created bitmap (1 pages) for device md0
<6>[ 3452.204202] md0: bitmap initialized from disk: read 1 pages, set 8 of 8 bits
<6>[ 3452.271446] md0: detected capacity change from 0 to 3214934016
<7>[ 3452.271474] RAID conf printout:
<7>[ 3452.271477] --- level:4 rd:7 wd:6
<7>[ 3452.271478] disk 0, o:1, dev:loop1
<7>[ 3452.271479] disk 1, o:1, dev:loop2
<7>[ 3452.271480] disk 2, o:1, dev:loop3
<7>[ 3452.271481] disk 3, o:1, dev:loop4
<7>[ 3452.271482] disk 4, o:1, dev:loop5
<7>[ 3452.271483] disk 5, o:1, dev:loop6
<7>[ 3452.271484] disk 6, o:1, dev:loop7
<6>[ 3452.271613] md: recovery of RAID array md0
<6>[ 3452.271616] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
<6>[ 3452.271617] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
<6>[ 3452.271625] md: using 128k window, over a total of 523264k.
<4>[ 3452.484361] md: couldn't update array info. -22
<4>[ 3452.536719] md: couldn't update array info. -22
<4>[ 3452.550794] md: couldn't update array info. -22
<4>[ 3453.276007] md: couldn't update array info. -22
<4>[ 3454.076132] md: couldn't update array info. -22
<4>[ 3454.859359] md: couldn't update array info. -22
<4>[ 3455.638082] md: couldn't update array info. -22
<6>[ 3465.044292] md: md0: recovery done.
<7>[ 3465.996564] RAID conf printout:
<7>[ 3465.996567] --- level:4 rd:7 wd:7
<7>[ 3465.996568] disk 0, o:1, dev:loop1
<7>[ 3465.996569] disk 1, o:1, dev:loop2
<7>[ 3465.996570] disk 2, o:1, dev:loop3
<7>[ 3465.996571] disk 3, o:1, dev:loop4
<7>[ 3465.996572] disk 4, o:1, dev:loop5
<7>[ 3465.996572] disk 5, o:1, dev:loop6
<7>[ 3465.996573] disk 6, o:1, dev:loop7
<6>[ 3470.361924] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
<1>[ 3471.084961] md/raid:md0: Disk failure on loop0, disabling device.
<1>[ 3471.084961] md/raid:md0: Operation continuing on 7 devices.
<7>[ 3471.253666] RAID conf printout:
<7>[ 3471.253669] --- level:4 rd:7 wd:7
<7>[ 3471.253671] disk 0, o:1, dev:loop1
<7>[ 3471.253672] disk 1, o:1, dev:loop2
<7>[ 3471.253672] disk 2, o:1, dev:loop3
<7>[ 3471.253686] disk 3, o:1, dev:loop4
<7>[ 3471.253687] disk 4, o:1, dev:loop5
<7>[ 3471.253688] disk 5, o:1, dev:loop6
<7>[ 3471.253689] disk 6, o:1, dev:loop7
<3>[ 3475.771451] Buffer I/O error on dev md0, logical block 327680, lost sync page write
<3>[ 3475.771458] JBD2: Error -5 detected when updating journal superblock for md0-8.
<3>[ 3475.771461] Aborting journal on device md0-8.
<3>[ 3475.771471] Buffer I/O error on dev md0, logical block 327680, lost sync page write
<3>[ 3475.771475] JBD2: Error -5 detected when updating journal superblock for md0-8.
<6>[ 3476.104774] md: unbind<loop0>
<6>[ 3476.107402] md: export_rdev(loop0)
<3>[ 3476.459554] Buffer I/O error on dev md0, logical block 0, lost sync page write
<2>[ 3476.459563] EXT4-fs error (device md0): ext4_journal_check_start:56: Detected aborted journal
<2>[ 3476.459567] EXT4-fs (md0): Remounting filesystem read-only
<3>[ 3476.459570] EXT4-fs (md0): previous I/O error to superblock detected
<3>[ 3476.459581] Buffer I/O error on dev md0, logical block 0, lost sync page write
<2>[ 3476.459587] EXT4-fs (md0): ext4_writepages: jbd2_start: 9223372036854775807 pages, ino 12; err -30
<3>[ 3476.495441] EXT4-fs (md0): previous I/O error to superblock detected
<3>[ 3476.495500] Buffer I/O error on dev md0, logical block 0, lost sync page write
<2>[ 3476.495508] EXT4-fs error (device md0): ext4_put_super:837: Couldn't clean up the journal
<6>[ 3476.904587] md: bind<loop0>
<6>[ 3476.919462] md/raid:md0: using device loop0 as journal
<6>[ 3476.919531] md: md0 switched to read-write mode.
<6>[ 3477.317614] EXT4-fs (md0): recovery complete
<6>[ 3477.317618] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
Best Regards,
Yi Zhang
^ permalink raw reply
* Re: Lost RAID6 disks when moving to new PC
From: Andreas Klauer @ 2016-07-27 11:57 UTC (permalink / raw)
To: Alex Owen; +Cc: linux-raid
In-Reply-To: <CAFmAqNtzaWBhAkYoNEdyV0VMniEVt0Xv_4mOHoeTsCYbmecReg@mail.gmail.com>
On Wed, Jul 27, 2016 at 11:35:34AM +0100, Alex Owen wrote:
> The array should be RAID6 on /dev/sd{a-f}.
Full disk raid sucks. Zero advantages, lots of additional risk.
There are too many programs out there that expect every disk to have
a partition table, and will use it unasked if it looks unpartitioned.
You seem to have lost your md metadata to some partitioner/installer,
you're also the third person with this problem in a row. Congrats. ;)
> fdisk -l :
Your fdisk doesn't support GPT, don't use it.
> parted
> Disk /dev/sd[bcd]: 3001GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
>
> Number Start End Size File system Name Flags
> 1 17.4kB 134MB 134MB Microsoft reserved
> partition msftres
> 2 135MB 3001GB 3000GB Basic data partition
> msftdata
Well, something put GPT partition table on those. GPT overwrites start
and end of the disk. You're using 1.2 metadata which is located 4K from
the start, can you show some hexdump for those disks?
hexdump -C -s 4096 -n 4096 /dev/sdb
> And the output of mdadm --examine /dev/sd[a-f]
>
> ----------
>
> /dev/sda:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
> Name : NAS:0 (local to host NAS)
> Creation Time : Sat Jan 23 17:57:37 2016
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
> Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 013740d6:5cc445e7:625f3257:8608daec
>
> Update Time : Tue Jul 26 04:09:29 2016
> Checksum : 14ba6ebd - correct
> Events : 2949004
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAAAAA ('A' == active, '.' == missing)
> /dev/sdb:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> /dev/sdc:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> /dev/sdd:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
Basically what we know is... your disk order for three disks
(/dev/sde = role 0, /dev/sdf = role 1, /dev/sda= role 2)
and what we don't know is the disk order of /dev/sd[bcd].
If the metadata is lost completely, only thing you can do is re-create
the RAID with all possible orders efa{bcd,bdc,cbd,cdb,dbc,dcb}.
Re-creating is dangerous so you should use an overlay: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
When re-creating you have to specify all variables (level, layout, chunksize,
data offset, order, ...) since the defaults picked by mdadm might differ
depending on your mdadm version.
Example command: (untested)
mdadm --create /dev/md42 --assume-clean \
--level=6 --chunk=512 --data-offset=128M --layout=ls \
/dev/overlay/sd{e,f,a,b,c,d}
Then you check if it can be mounted, and once mounted if big files
(larger than chunksize * number of disks) are intact or no. If you
switch the wrong two disks it may mount but data is garbage anyway.
Regards
Andreas Klauer
^ permalink raw reply
* WARNING: CPU: 4 PID: 10512 at drivers/md/raid5-cache.c:728 r5l_do_reclaim+0x415/0x430 [raid456]
From: Yi Zhang @ 2016-07-27 11:38 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1382969797.9407100.1469619450040.JavaMail.zimbra@redhat.com>
Hello everyone
I'm testing raid5-cache recently and found below issue on 4.7.0-rc7.
[ 902.701162] md: bind<sdb1>
[ 902.701248] md: bind<sdd1>
[ 902.702566] md: bind<sde1>
[ 902.702625] md: bind<sdf1>
[ 902.703306] md: bind<sdc1>
[ 902.705899] md/raid:md0: not clean -- starting background reconstruction
[ 902.705972] md/raid:md0: device sdf1 operational as raid disk 3
[ 902.705974] md/raid:md0: device sde1 operational as raid disk 2
[ 902.705975] md/raid:md0: device sdd1 operational as raid disk 1
[ 902.705976] md/raid:md0: device sdb1 operational as raid disk 0
[ 902.706312] md/raid:md0: allocated 4374kB
[ 902.706371] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2
[ 902.706372] RAID conf printout:
[ 902.706380] --- level:6 rd:4 wd:4
[ 902.706381] disk 0, o:1, dev:sdb1
[ 902.706382] disk 1, o:1, dev:sdd1
[ 902.706383] disk 2, o:1, dev:sde1
[ 902.706384] disk 3, o:1, dev:sdf1
[ 902.706393] md/raid456: discard support disabled due to uncertainty.
[ 902.706394] Set raid456.devices_handle_discard_safely=Y to override.
[ 902.706396] md/raid:md0: using device sdc1 as journal
[ 902.707354] created bitmap (1 pages) for device md0
[ 902.707401] md0: bitmap initialized from disk: read 1 pages, set 8 of 8 bits
[ 902.781331] md0: detected capacity change from 0 to 1071644672
[ 902.781465] md: resync of RAID array md0
[ 902.781468] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 902.781469] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 902.781475] md: using 128k window, over a total of 523264k.
[ 902.829618] md: couldn't update array info. -22
[ 902.876797] md: couldn't update array info. -22
[ 902.896926] md: couldn't update array info. -22
[ 909.185914] md: md0: resync done.
[ 909.262779] RAID conf printout:
[ 909.262782] --- level:6 rd:4 wd:4
[ 909.262783] disk 0, o:1, dev:sdb1
[ 909.262784] disk 1, o:1, dev:sdd1
[ 909.262785] disk 2, o:1, dev:sde1
[ 909.262786] disk 3, o:1, dev:sdf1
[ 912.795104] md/raid:md0: Disk failure on sdb1, disabling device.
md/raid:md0: Operation continuing on 3 devices.
[ 912.851893] RAID conf printout:
[ 912.851895] --- level:6 rd:4 wd:3
[ 912.851897] disk 0, o:0, dev:sdb1
[ 912.851898] disk 1, o:1, dev:sdd1
[ 912.851899] disk 2, o:1, dev:sde1
[ 912.851900] disk 3, o:1, dev:sdf1
[ 912.855130] RAID conf printout:
[ 912.855133] --- level:6 rd:4 wd:3
[ 912.855135] disk 1, o:1, dev:sdd1
[ 912.855136] disk 2, o:1, dev:sde1
[ 912.855137] disk 3, o:1, dev:sdf1
[ 912.885778] md: unbind<sdb1>
[ 912.894205] md: export_rdev(sdb1)
[ 1548.098231] md: bind<sdb1>
[ 1548.452714] RAID conf printout:
[ 1548.452717] --- level:6 rd:4 wd:3
[ 1548.452718] disk 0, o:1, dev:sdb1
[ 1548.452719] disk 1, o:1, dev:sdd1
[ 1548.452720] disk 2, o:1, dev:sde1
[ 1548.452721] disk 3, o:1, dev:sdf1
[ 1548.486067] ------------[ cut here ]------------
[ 1548.486076] WARNING: CPU: 4 PID: 10512 at drivers/md/raid5-cache.c:728 r5l_do_reclaim+0x415/0x430 [raid456]
[ 1548.486077] Modules linked in: fuse btrfs vfat msdos fat ext4 jbd2 mbcache binfmt_misc raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ipt_REJECT nf_reject_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sb_edac edac_core snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic x86_pkg_temp_thermal snd_hda_intel intel_powerclamp coretemp snd
_hda_codec
[ 1548.486117] kvm_intel kvm snd_hda_core irqbypass crct10dif_pclmul crc32_pclmul snd_hwdep ghash_clmulni_intel snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper hp_wmi iTCO_wdt snd_timer sparse_keymap rfkill cryptd iTCO_vendor_support mei_me ioatdma snd mei shpchp pcspkr sg soundcore i2c_i801 tpm_infineon lpc_ich mfd_core dca nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod nouveau video mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm isci e1000e ata_generic libsas pata_acpi ahci libahci scsi_transport_sas firewire_ohci libata firewire_core ptp crc32c_intel serio_raw pps_core i2c_core crc_itu_t wmi fjes dm_mirror dm_region_hash dm_log dm_mod
[ 1548.486152] CPU: 4 PID: 10512 Comm: md0_resync Tainted: G W 4.7.0-rc7 #1
[ 1548.486153] Hardware name: Hewlett-Packard HP Z620 Workstation/158A, BIOS J61 v03.69 03/25/2014
[ 1548.486155] 0000000000000286 000000002369dfe1 ffff88079d713b40 ffffffff8134caec
[ 1548.486157] 0000000000000000 0000000000000000 ffff88079d713b80 ffffffff8108c351
[ 1548.486159] 000002d8a6ae4d80 000000008f0c904f ffff8807f64b0000 ffff8807c4ab6888
[ 1548.486161] Call Trace:
[ 1548.486166] [<ffffffff8134caec>] dump_stack+0x63/0x87
[ 1548.486169] [<ffffffff8108c351>] __warn+0xd1/0xf0
[ 1548.486172] [<ffffffff8108c48d>] warn_slowpath_null+0x1d/0x20
[ 1548.486175] [<ffffffffa097d805>] r5l_do_reclaim+0x415/0x430 [raid456]
[ 1548.486178] [<ffffffff811f9c50>] ? kfree+0x120/0x170
[ 1548.486181] [<ffffffffa097e33b>] r5l_quiesce+0x7b/0xa0 [raid456]
[ 1548.486183] [<ffffffffa0970da0>] raid5_quiesce+0x50/0x2a0 [raid456]
[ 1548.486186] [<ffffffff810d0250>] ? prepare_to_wait_event+0xf0/0xf0
[ 1548.486189] [<ffffffff8156d00e>] md_do_sync+0xe7e/0xf60
[ 1548.486192] [<ffffffff810c4370>] ? enqueue_entity+0x2a0/0xcc0
[ 1548.486194] [<ffffffff810c201d>] ? update_curr+0xed/0x180
[ 1548.486196] [<ffffffff810c026e>] ? account_entity_dequeue+0xae/0xd0
[ 1548.486197] [<ffffffff810c27f6>] ? dequeue_entity+0x266/0x980
[ 1548.486200] [<ffffffff810b5ef5>] ? check_preempt_curr+0x75/0x90
[ 1548.486202] [<ffffffff810b5f29>] ? ttwu_do_wakeup+0x19/0xe0
[ 1548.486204] [<ffffffff810c2fce>] ? dequeue_task_fair+0xbe/0x830
[ 1548.486206] [<ffffffff810c3995>] ? put_prev_entity+0x35/0x730
[ 1548.486209] [<ffffffff8102c6d9>] ? __switch_to+0x219/0x5c0
[ 1548.486211] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
[ 1548.486214] [<ffffffff81566aa6>] md_thread+0x136/0x150
[ 1548.486216] [<ffffffff81566970>] ? find_pers+0x70/0x70
[ 1548.486219] [<ffffffff810ab698>] kthread+0xd8/0xf0
[ 1548.486221] [<ffffffff816e873f>] ret_from_fork+0x1f/0x40
[ 1548.486223] [<ffffffff810ab5c0>] ? kthread_park+0x60/0x60
[ 1548.486224] ---[ end trace 242e9516e36acb1e ]---
[ 1548.543855] md: recovery of RAID array md0
[ 1548.543857] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1548.543859] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 1548.543864] md: using 128k window, over a total of 523264k.
[ 1548.543868] md: md0: recovery done.
[ 1548.574961] RAID conf printout:
[ 1548.574964] --- level:6 rd:4 wd:4
[ 1548.574966] disk 0, o:1, dev:sdb1
[ 1548.574967] disk 1, o:1, dev:sdd1
[ 1548.574968] disk 2, o:1, dev:sde1
[ 1548.574969] disk 3, o:1, dev:sdf1
Steps I used:
mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 4
/dev/sdb1 /dev/sd[d-f]1 --write-journal /dev/sdc1 --bitmap=internal
--bitmap-chunk=64M --chunk 512
mdadm --wait /dev/md0
mkfs.ext4 /dev/md0
mdadm /dev/md0 -f /dev/sdb1
mdadm /dev/md0 -r /dev/sdb1
mdadm /dev/md0 -a /dev/sdb1
Best Regards
Yi Zhang
Best Regards,
Yi Zhang
^ permalink raw reply
* Lost RAID6 disks when moving to new PC
From: Alex Owen @ 2016-07-27 10:35 UTC (permalink / raw)
To: linux-raid
Hello everyone,
Thank you in advance for any help you might be able to give. I'm
another person with a broken RAID array and potentially lots of lost
data. I'm really stuck and would be very grateful for any help you
might be able to give.
Earlier I tried to move my mdadm array from a Ubuntu Server to a
virtualised Ubuntu Server running on an ESXi host.
I connected them and could see the disks in ESXi, then used
passthrough to get them to the Ubuntu Server instance, but this didn't
work and I could only see three of them.
Then I moved the disks back into the old server and booted it up,
where it failed to see the array on boot, and it still fails to
assemble it.
The array should be RAID6 on /dev/sd{a-f}.
I have backups of the most important 2TB, but there is a lot of data
for which this is the only copy (which I can afford to lose, but
hopefully won't have to).
I've tried:
mdadm --assemble --scan --verbose :
----------
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sdg1
mdadm: no RAID superblock on /dev/sdg
mdadm: /dev/sdf requires wrong number of drives.
mdadm: /dev/sde requires wrong number of drives.
mdadm: no RAID superblock on /dev/sdd2
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdb2
mdadm: no RAID superblock on /dev/sdb1
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sdc2
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sda requires wrong number of drives.
----------
Which gives this in /cat/mdstat:
----------
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
unused devices: <none>
----------
Occasionally it will assemble md0, but with three spares and no data.
fdisk -l :
----------
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
5 heads, 3 sectors/track, 390702211 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 2048 4294967294 2147482623+ 8e Linux LVM
WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x7b8883da
Device Boot Start End Blocks Id System
/dev/sdc1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x7b8883db
Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x453c133d
Device Boot Start End Blocks Id System
/dev/sdd1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
5 heads, 3 sectors/track, 390702211 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sde1 2048 4294967294 2147482623+ 8e Linux LVM
Disk /dev/sdf: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdf doesn't contain a valid partition table
----------
parted -l
Warning: /dev/sda contains GPT signatures, indicating that it has a GPT table.
However, it does not have a valid fake msdos partition table, as it should.
Perhaps it was corrupted -- possibly by a program that doesn't understand GPT
partition tables. Or perhaps you deleted the GPT table, and are now using an
msdos partition table. Is this a GPT partition table?
Yes/No? n
Model: ATA WDC WD30EZRX-00D (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 134MB 134MB Microsoft reserved
partition msftres
2 135MB 3001GB 3000GB Basic data partition
msftdata
Model: ATA WDC WD30EZRX-00D (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 134MB 134MB Microsoft reserved
partition msftres
2 135MB 3001GB 3000GB Basic data partition
msftdata
Model: ATA WDC WD30EZRX-00M (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 134MB 134MB Microsoft reserved
partition msftres
2 135MB 3001GB 3000GB Basic data partition
msftdata
Warning: /dev/sde contains GPT signatures, indicating that it has a
GPT table. However, it does not have a valid fake msdos partition
table, as it should. Perhaps it was corrupted --
possibly by a program that doesn't understand GPT partition tables.
Or perhaps you deleted the GPT table, and are now using an msdos
partition table. Is this a GPT partition table?
Yes/No? n
Error: /dev/sdf: unrecognised disk label
----------
And the output of mdadm --examine /dev/sd[a-f]
----------
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
Name : NAS:0 (local to host NAS)
Creation Time : Sat Jan 23 17:57:37 2016
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 013740d6:5cc445e7:625f3257:8608daec
Update Time : Tue Jul 26 04:09:29 2016
Checksum : 14ba6ebd - correct
Events : 2949004
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
Name : NAS:0 (local to host NAS)
Creation Time : Sat Jan 23 17:57:37 2016
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 19096ab0:3e3aac62:7afdfc97:86eeb5f6
Update Time : Tue Jul 26 04:09:29 2016
Checksum : b4f13acc - correct
Events : 2949004
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : b377d975:86beb86c:9da9f21d:f73b2451
Name : NAS:0 (local to host NAS)
Creation Time : Sat Jan 23 17:57:37 2016
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : dc424c5a:b4845f55:373a04e0:78d96723
Update Time : Tue Jul 26 04:09:29 2016
Checksum : c63fe6b9 - correct
Events : 2949004
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing)
----------
Thank you
^ permalink raw reply
* SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: David C. Rankin @ 2016-07-27 7:13 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtQpGVf2Tj13o57T4=raF=czYr-V99tEDSkJ-_buk7JtTg@mail.gmail.com>
On 07/26/2016 06:18 PM, Chris Murphy wrote:
> To get rid of the backup GPT you'll zero the last two sectors of the
> drive. So first get the total number of sectors from something like
> gdisk -l which gets you this information (in part):
>
> Disk /dev/sda: 1953525168 sectors, 931.5 GiB
>
> And do
> dd if=/dev/zero of=/dev/sda seek=1953525167
>
> That'll erase ..67 and ..68, but the header is in ..67, one sector
> before the last one. Nothing should be in the last sector anyway but
> I'd check first! I don't know if ext4 put something there. And do not
> use the "last usable sector" because that's full 34 sectors from the
> end and there very well may be ext4 metadata in there that you do not
> want to step on with /dev/sdc.
Chris, Phil, All,
Thank you. For anyone else that is faced with the problem where you are using
whole disks in your raid1 array over the top of unused sub-partitions, here is
the 5 minute fix.
In my circumstance, I had partitioned a pair of 3T WD Black drives for use in
a raid1 array. I then created the array, but instead of using the partitions
(sdc1/sdd1), I used the whole disk for the array (sdc/sdd). The array worked
flawlessly for a year, and while collecting partition/geometry info to squirrel
away for disaster recovery, I noticed gdisk -l /dev/sdc complained that the
primary GPT header was corrupt, but the backup was fine. (examples of the gdisk
output can be found earlier in this thread). The robust and flexible mdadm came
through with flying colors. Had I done this correct to begin with, it could have
been completed without a resync (saving several hours)
How I solved the problem:
(1) do NOT attempt to alter the disk in a partitioning package like fdisk,
sfdisk, gdisk, parted, etc.. A write after you delete the unused partitions with
adversely affect the md data and will require a long and painful resync
depending on the size of your drive.
(2) simply --fail and --remove one drive from the array. My array was
/dev/md4, and failing and removing /dev/sdd from the array was as simple as:
# mdadm /dev/md4 --fail /dev/sdd
# mdadm /dev/md4 --remove /dev/sdd
(3) To remove the inadvertent partition on the drive while keeping the raid
data in tact, you must remove the PMBR and primary Partition tables from the
drive. You can use `wipefs` or simply use `dd` to overwite the first 4096 bytes
on the drive with zeros and then the last 1024 bytes before the end of the disk
to remove the backup GPT header. (I overwote the last 4096 bytes on the disk,
just to make sure -- I had nothing in the last 100M of the disk, so that seemed
fine) You can look at the disk geometry reported by gdisk to find the end of the
disk (the number of logical sectors -- make sure the disk has 512-byte sectors,
or dd option adjustments will be needed) (then just subtract 8 from that number
(or 2 if you wish to limit the write to 1024 bytes) and use that as the 'seek'
offset with 'dd', so
# dd of=/dev/sdd if=/dev/zero bs=4096 count=1
# dd of=/dev/sdd if=/dev/zero bs=512 count=8 seek=5860533160
I also wrote over the last 8 sectors at the reported end of sdd1 as well (not
sure if this had any relation to the problem, but I wanted to make sure if there
was any GPT header at the end of the partition, it was zeroed as well)
# dd of=/dev/sdd if=/dev/zero bs=512 count=8 seek=5860328334
(4) then simply --re-add the drive to the array (no resync will be required)
# mdadm /dev/md4 --re-add /dev/sdd
mdadm: re-added /dev/sdd
(5) Now simply repeat the process with /dev/sdc
When you are done, you will have two drives, using the whole disk for the
array without the unintended empty partitions on the drive. Now gdisk reports
correctly, e.g.
# gdisk -l /dev/sdc
GPT fdisk (gdisk) version 1.0.1
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
and you array will be active and clean:
# mdadm -D /dev/md4
/dev/md4:
Version : 1.2
Creation Time : Mon Mar 21 02:27:21 2016
Raid Level : raid1
Array Size : 2930135488 (2794.39 GiB 3000.46 GB)
Used Dev Size : 2930135488 (2794.39 GiB 3000.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Jul 27 01:36:56 2016
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : valkyrie:4 (local to host valkyrie)
UUID : 6e520607:f152d8b9:dd2a3bec:5f9dc875
Events : 7984
Number Major Minor RaidDevice State
0 8 32 0 active sync /dev/sdc
2 8 48 1 active sync /dev/sdd
Thank you again to all that helped.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: RAID5 Performance
From: Doug Dumitru @ 2016-07-27 5:36 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <7b7d730f-2951-ba5f-7f6b-33624b59a02d@websitemanagers.com.au>
On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
> Version : 1.2
> Creation Time : Wed Aug 22 00:47:03 2012
> Raid Level : raid5
> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
> Raid Devices : 8
> Total Devices : 8
> Persistence : Superblock is persistent
>
> Update Time : Wed Jul 27 11:32:00 2016
> State : active
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : san1:1 (local to host san1)
> UUID : 707957c0:b7195438:06da5bc4:485d301c
> Events : 2185221
>
> Number Major Minor RaidDevice State
> 7 8 65 0 active sync /dev/sde1
> 13 8 1 1 active sync /dev/sda1
> 8 8 81 2 active sync /dev/sdf1
> 5 8 113 3 active sync /dev/sdh1
> 9 8 97 4 active sync /dev/sdg1
> 12 8 17 5 active sync /dev/sdb1
> 10 8 49 6 active sync /dev/sdd1
> 11 8 33 7 active sync /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
> 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
> 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
> 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
> 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
> 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
> 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
> 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
> 0.18 0.28 0.23 0.32 0.16 9.60
> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
> 0.08 38.00 0.00 38.00 20.00 4.00
> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
> 0.03 14.00 4.00 24.00 14.00 2.80
> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
> 0.09 44.00 0.00 44.00 22.00 4.40
> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
> 7.92 41.03 0.00 41.87 4.98 100.00
> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
> 0.22 30.29 0.00 30.29 28.57 20.00
> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
> 1.72 13.46 0.40 40.18 4.27 57.20
> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
> 1.40 40.91 0.00 43.39 28.34 99.20
> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
> 0.08 21.00 0.00 28.00 21.00 8.40
> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
> 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
> 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
> 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
> 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
> 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
> 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
> 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
> 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>
>>
>> *
>>
>> Enhance the queue depth. Standard kernel queue depth is OK for old
>> single drives with small caches, but not for modern drives or RAID
>> arrays:
>>
>> echo 512 > /sys/block/sda/queue/nr_requests
>>
> So my question is should I increase the configured nr_requests above the
> current 128?
With your workload, it probably won't matter too much. Really high
queue depths are great on paper, but hard to actually see.
>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?
This is not how a random write works. If you are running raid-5
before the 4.4 kernel, you get the "old" read/modify/write algorithm.
If you write 4K, the system will read 4K from (n-2) drives, add in
your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
writes. With the "new" logic in 4.4, you read the old contents of the
4K plus parity, and re-write the 4k plus parity, so there are 2 reads
and 2 writes. With big arrays, the "new" logic can help quite a bit,
but the chatter rate is still high. Note that the new logic is only
raid-5. raid-6 cannot use the new logic and has to read the stripe
from every drive.
The stripe size impacts when the system does can avoid doing a
read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
the write is exactly on a stripe boundary, and you get lucky and the
background thread does not wake up at just the wrong time, you will do
the write with zero reads. I personally run with very small chunks,
but I have code that always writes perfect stripe writes and stock
file systems don't act that way.
DRBD can saturate GigE without any problem with random 4K writes. I
have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
IOPS. The target arrays needs to keep up, but that is another story.
My testing with DRBD is that it starts to peter out at 10Gig, so if
you want more bandwidth you need some other approach. Some vendors
use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
iSER should give you similar results with RDMA capable ethernet.
Linbit (the people who write DRBD) have a non GPL extension to DRBD
that uses RDMA so you can get more bandwidth that way as well.
> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...
This is probably not true. If the drive really had to update 512K at
a time, then 4K writes would be 128x wear amplification. SSDs can be
bad, but usually not that bad.
>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...
The problem is that the FTL of the SSDs are a black box and as the
array gets bigger, the slowest drive dictates the array performance.
This is why the "big vendors" all map SSDs in the host and avoid or
minimize writing randomly. I know of one vendor install that has 4000
VDI seats (using ESXI as compute hosts) from a single HA pair of 24
SSD shelves. The connection to ESXI is FC and the hosts are HA with
an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
random write IOPS to pull this off, which I think is impossible with
stock parity raid, and very hard with raid-10.
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?
RAID-10 will be faster, but you pay for this with capacity. It is
also a double-edged sword as SSDs themselves run faster if you leave
more free space on them, so RAID-10 absolutely might not be a lot
faster than RAID-5 with some space left over. Also remember that free
space on the SSDs only counts if it is actually unallocated. So you
need to trim the SSDs or start with a secure erased drive and then
never use the full capacity. It is best to leave an empty partition
that is untouched.
> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?
The reads, especially with the pre 4.4 code or with raid-6 definitely
take their toll. Most SSDs are also not quite symmetrical in terms of
performance. If your SSD does 50K read IOPS and 50K write IOPS, it
will probably not do 25K reads and 25K writes concurrently, but
instead stop somewhere around 18K. But your mileage may vary. If you
have 8 drives that do 20 read/write symmetric, with new raid-5, each
4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
your data rate ends up maxing out at 80K IOPS. With the old raid-5
logic, you end up with 6 reads plus two writes per "OP", so you tend
to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
these computations tend to fall apart, so 24 SSD arrays are not 3x
faster than 8 SSD arrays, at least with stock code.
You also need to consider what raid does to the SSD FTL. As you
chatter a drive, its wear goes up and its performance goes down.
Different SSD models can vary wildly, but again the rule of thumb is
keep as much free space as possible on the drives. raid-5 or
mirroring is also 2:1 write amplification (ie, you are writing two
drives) and raid-6 is 3:1, on top of whatever the FTL write
amplification is at the time.
>
> For completeness, PV information:
> PV Name /dev/md1
> VG Name vg0
> PV Size 3.06 TiB / not usable 2.94 MiB
> Allocatable yes
> PE Size 4.00 MiB
> Total PE 801253
> Free PE 33281
> Allocated PE 767972
> PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ
>
> Any advice or assistance would be greatly appreciated.
>
> Regards,
> Adam
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
WileFire Storage. http://www.wildfire-storage.com
^ permalink raw reply
* Re: RAID5 Performance
From: Brad Campbell @ 2016-07-27 3:15 UTC (permalink / raw)
To: Adam Goryachev, linux-raid@vger.kernel.org
In-Reply-To: <7b7d730f-2951-ba5f-7f6b-33624b59a02d@websitemanagers.com.au>
On 27/07/16 10:24, Adam Goryachev wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a
> bit, and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
> DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
> mostly windows VM's).
Wow. More layers than a wedding cake.
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst
> performance results when I was initially setting this up, and seems to
> be what we are getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry
> about chunk/stripe size because we always just write the exact data we
> want, no need to read/compute/write)?
> OR, is that read/compute overhead negligible since I'm using SSD and
> read performance is so quick?
I'll only comment from personal experience with anecdotal evidence.
I have a RAID10 comprised of 6 256GB SSD (3 Intel & 3 Samsung) used as
the backing for multiple VMs (raw files on Ext4).
Initially I played with a number of RAID types when setting up the array
(back in 2012) and found RAID10 offered the best compromise for my use
case. This was based on CPU usage (raid 5 & 6 parity calculations on
*every write*), the need for RMW cycles for small writes and trying to
balance block sizes. None of these things are an issue with RAID10 and
in general I found a measurable reduction in overhead and consequential
performance boost in the VM's.
This is a single machine with a relatively underpowered AMD FX-8350 CPU.
I found that with multiple VM's hitting the disks I was losing enough
CPU time to RAID overhead that it made things noticeably less responsive.
My only issue is half the disks return a deterministic value after
discard and the other half don't, so any raid check operations return a
gazillion mismatches. Not an operational issue, but one worth mentioning
if you were to use different model drives.
Regards,
Brad
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox