* [GIT PULL] MD update for 4.8
From: Shaohua Li @ 2016-07-28 17:36 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid, neilb
Hi,
Please pull MD update for 4.8. The patches are based on 4.7 rc initially, I
merged them to lastest upstream to fix merge conflict against block layer
changes:
- A bunch of patches from Neil Brown to fix RCU usage
- Two performance improvement patches from Tomasz Majchrzak
- Alexey Obitotskiy fixes module refcount issue
- Arnd Bergmann fixes time granularity
- Cong Wang fixes a list corruption issue
- Guoqing Jiang fixes a deadlock in md-cluster
- A null pointer deference fix from me
- Song Liu fixes misuse of raid6 rmw
- Other trival/cleanup fixes from Guoqing Jiang and Xiao Ni
Thanks,
Shaohua
The following changes since commit 194dc870a5890e855ecffb30f3b80ba7c88f96d6:
Add braces to avoid "ambiguous ‘else’" compiler warnings (2016-07-27 20:03:31 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git for-linus
for you to fetch changes up to 3f35e210ed4617a68b6baa9b7ac6c72bf7e313d9:
Merge branch 'mymd/for-next' into mymd/for-linus (2016-07-28 09:34:14 -0700)
----------------------------------------------------------------
Alexey Obitotskiy (1):
Fix kernel module refcount handling
Arnd Bergmann (1):
md: use seconds granularity for error logging
Cong Wang (1):
md: use a mutex to protect a global list
Guoqing Jiang (2):
md-cluster: fix deadlock issue when add disk to an recoverying array
md: simplify the code with md_kick_rdev_from_array
NeilBrown (17):
md: disconnect device from personality before trying to remove it.
md/raid1, raid10: don't recheck "Faulty" flag in read-balance.
md/raid10: fix refounct imbalance when resyncing an array with a replacement device.
md/raid10: add rcu protection in raid10_status.
md/raid10: add rcu protection to rdev access in raid10_sync_request.
md/raid10: add rcu protection to rdev access during reshape.
md/raid10: minor code improvement in fix_read_error()
md/raid10: simplify print_conf a little.
md/raid1: small cleanup in raid1_end_read/write_request
md/raid1: small code cleanup in end_sync_write
md/raid1: add rcu protection to rdev in fix_read_error
md/raid5: add rcu protection to rdev accesses in handle_failed_sync.
md/raid5: add rcu protection to rdev accesses in want_replace
md/raid5: add rcu protection to rdev accesses in raid5_status.
md/multipath: add rcu protection to rdev access in multipath_status.
md: be extra careful not to take a reference to a Faulty device.
md: reduce the number of synchronize_rcu() calls when multiple devices fail.
Shaohua Li (2):
MD: fix null pointer deference
Merge branch 'mymd/for-next' into mymd/for-linus
Song Liu (1):
right meaning of PARITY_ENABLE_RMW and PARITY_PREFER_RMW
Tomasz Majchrzak (3):
raid1/raid10: slow down resync if there is non-resync activity pending
md: add missing sysfs_notify on array_state update
raid10: improve random reads performance
Xiao Ni (1):
MD:Update superblock when err == 0 in size_store
drivers/md/md.c | 74 +++++++++++----
drivers/md/md.h | 10 +-
drivers/md/multipath.c | 29 +++---
drivers/md/raid1.c | 130 ++++++++++++-------------
drivers/md/raid10.c | 250 ++++++++++++++++++++++++++++++-------------------
drivers/md/raid10.h | 3 +-
drivers/md/raid5.c | 45 +++++----
7 files changed, 328 insertions(+), 213 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID5 Performance
From: Peter Grandi @ 2016-07-28 17:20 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <a0921a51-d841-7fbd-0f31-21f5ba8c6f4b@websitemanagers.com.au>
[ ... ]
>> * Replace the flash SSDs with those that are known to deliver
>> high (at least > 10,000 single threaded) small synchronous
>> write IOPS.
> Is there a "known" SSD that you would suggest? My problem is
> that Intel spec sheets seem to suggest that there is little
> performance difference across the range of SSD's, so it's
> really not clear which SSD model I should buy.
The links I wrote earlier have lists:
>>> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>> http://www.spinics.net/lists/ceph-users/msg25928.html
>>> https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
As one of those pages says the Samsung SM863 looks attractive,
but for historical reasons so far I have only seen Intel DCs in
similar use. There discussions of other models in various posts
related to Ceph journal SSD usage.
> Obviously it's not something I can afford to buy one of each
> and test them either.
In addition to the lists above I have justed tested my three
home flash SSDs:
* Micron M4 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 1200.3 s, 341 kB/s
* Samsung 850 Pro 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 1732.93 s, 236 kB/s
* Hynix SK SH910 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 644.742 s, 635 kB/s
So I would not recommend any of them for "small sync writes"
workloads :-), but they are quite good otherwise. I do notice
they are slow on small sync writes when downloading mail, as
each message is duly 'fsync'ed.
BTW as bonus material, I have done on the SH910 an abbreviated
test with block sizes between 4KiB and 1024KiB:
# for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/var/tmp/TEST |& grep copied; done
4k: 4096000 bytes (4.1 MB) copied, 6.23481 s, 657 kB/s
16k: 16384000 bytes (16 MB) copied, 6.29379 s, 2.6 MB/s
64k: 65536000 bytes (66 MB) copied, 6.09223 s, 10.8 MB/s
128k: 131072000 bytes (131 MB) copied, 6.5487 s, 20.0 MB/s
256k: 262144000 bytes (262 MB) copied, 6.93361 s, 37.8 MB/s
512k: 524288000 bytes (524 MB) copied, 7.73957 s, 67.7 MB/s
1024k: 1048576000 bytes (1.0 GB) copied, 12.8671 s, 81.5 MB/s
Note how the time to write 1000 blocks is essentially the same
betweeen 4KiB and 128KiB, which is quite amusing. Probably the
flash-page size is around 256KiB.
For additional bonus value the same on a "fastish" consumer 2TB
disk, a Seagate ST2000DM001:
# for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/fs/sdb6/tmp/TEST |& grep copied; done
4k: 4096000 bytes (4.1 MB) copied, 44.9177 s, 91.2 kB/s
16k: 16384000 bytes (16 MB) copied, 38.131 s, 430 kB/s
64k: 65536000 bytes (66 MB) copied, 35.8263 s, 1.8 MB/s
128k: 131072000 bytes (131 MB) copied, 35.8188 s, 3.7 MB/s
256k: 262144000 bytes (262 MB) copied, 36.6838 s, 7.1 MB/s
512k: 524288000 bytes (524 MB) copied, 37.0612 s, 14.1 MB/s
1024k: 1048576000 bytes (1.0 GB) copied, 42.0844 s, 24.9 MB/s
>> * Relax the requirement for synchronous writes on *both* the
>> primary and secondary DRBD servers, if feeling lucky.
> I have the following entries for DRBD which were suggested by
> linbit (which previously lifted performance from abysmal to
> more than sufficient around 2+ years ago). [ ... ]
That's an inappropriate use of "performance" here:
> disk-barrier no;
> disk-flushes no;
> md-flushes no;
That "feeling lucky" list seems to me to have made performance
lower (in the sense that the performance of writing to
'/dev/null' is zero, even if the speed is really good :->).
With those settings the data sync policy is "disk-drain", which
also involves some waiting, but somewhat dangerous, except "In
case your backing storage device has battery-backed write cache"
(and "device" here means system and host adapter and disk); it
is not clear to me for metadata what "md-flushes no" gives.
BTW if you have battery-backed everything on the secondary side
you could use protocol "B".
However given those it looks likely that the bottleneck is also
on the primary DRBD side.
> Do you have any other suggestions or ideas that might assist?
* Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in space
than RAID10 and enormously raise the chances that a full
stripe-write can happen (it still has the write-hole problem
of parity RAID).
* Make sure the DRBD journal is also on a separate device that
allows fast small sync writes.
Also, I have appended a sample DRBD configuration I have used:
----------------------------------------------------------------
resource r0
{
device /dev/drbd_r0 minor 0;
# A: "local disk and local TCP send buffer"
# B: "local disk and remote buffer cache"
# C: "both local and remote disk"
protocol C;
net
{
# As mentioned on IRC by a DRBD guy, this is not really a
# secret, but more a "unique id" that ensures that replicas
# of different resources don't get accidentally connected.
# Still to be ABR-ized.
shared-secret "xxxxxxxxxxxx";
cram-hmac-alg sha1;
ping-timeout 50;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
# http://article.gmane.org/gmane.linux.network.drbd/18348
# http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html
# https://alteeve.ca/w/AN!Cluster_Tutorial_2_-_Performance_Tuning
# http://fghaas.wordpress.com/2007/06/22/performance-tuning-drbd-setups/
sndbuf-size 0;
rcvbuf-size 0;
max-buffers 16384;
unplug-watermark 16384;
max-epoch-size 16384;
}
syncer
{
csums-alg sha1;
# At 45MB/s takes 6 hour per 1TB.
rate 95M;
use-rle;
}
startup
{
wfc-timeout 15;
degr-wfc-timeout 15;
outdated-wfc-timeout 15;
# Cannot be an address, must be output of 'hostname'.
become-primary-on host-1;
}
on host-1
{
address 192.168.1.11:7788;
disk /dev/md2;
flexible-meta-disk /dev/local0/r0_md;
}
on host-2
{
address 192.168.1.12:7788;
disk /dev/md2;
flexible-meta-disk /dev/local0/r0_md;
}
}
^ permalink raw reply
* Re: RAID5 Performance
From: Adam Goryachev @ 2016-07-28 14:10 UTC (permalink / raw)
To: doug; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAFx4rwTMySfR27gT6You7f2T28Hq8m5Zxme1p43Fv1vtcpt_-Q@mail.gmail.com>
On 28/07/2016 10:11, Doug Dumitru wrote:
> On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>>
>> On 27/07/2016 15:36, Doug Dumitru wrote:
>>
>> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
>> <mailinglists@websitemanagers.com.au> wrote:
>>
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>>
>> This should help your raid-5 array, at least noticeably, provided the
>> new kernel actually has the
>> Facebook Read/Modify/Write new logic included. Based on the version
>> it should. You can very this by doing random writes and looking at
>> iostat. If you see 2 reads and 2 writes for every inbound write, you
>> have the new code. If you see 6 reads and 2 writes for every inbound
>> write, you have the old code.
>>
>> While this sounds huge, the change will be moderated by the behaviour
>> of SSDs. Random writes are much more expensive than read and the new
>> logic only lowers the number of reads. ... and raid-6 is not impacted
>> at all.
Will definitely try this, it seems a pretty simple low-cost, and pretty
low risk option.
>> The stripe size impacts when the system does can avoid doing a
>> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
>> the write is exactly on a stripe boundary, and you get lucky and the
>> background thread does not wake up at just the wrong time, you will do
>> the write with zero reads. I personally run with very small chunks,
>> but I have code that always writes perfect stripe writes and stock
>> file systems don't act that way.
>>
>> So reducing the chunk size will have minimal impact... but reducing it
>> should still provide some performance boost. Since I'm recreating the array
>> anyway, what size makes the most sense? 16k or go straight to the minimum of
>> 4k? Would a smaller chunk size increase the IOPS because we need to make
>> more (smaller) requests for the same data, potentially from more drives?
>>
>> ie, currently, a single read request for 4k will be done by reading one
>> chunk (64k) from one of the 8 drives (1 IOPS)
>> currently, a single write request for 4k will be done by reading one chunk
>> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
>> However, a read (or write) 48k request would be identical to the above,
>> while a smaller chunk size (4k) would mean:
>> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
>> disks (7 IOPS)
>> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
>> somewhere, but it might not be)
>> - read 2 x 4k chunks (the only 2 data chunks that
>> won't be written) + write 6 x 4k chunks
>> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
>> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
> You are confused about what chunk size is. It is not the IO size
> limit. It is just a layout calculation. If your chunk is 64K, then
> 64K is written to one disk before the array moves on to the next disk.
> If you read 4K, then only 4K is read. You never need to read (or
> write) and entire chunk.
>
> Lower chunk sizes are useful if your application does enough long
> writes to reach full stripes. At 64K x 7 drives, this is 448KB. If
> you are writing multi-megabytes, then 64K chunks is a good idea. If
> you are writing 128KB, you might want to go down to 16KB chunks. The
> problem with little chunks is if you read 64K from and array with 16KB
> chunks, you will cut your IO request into four parts. This is
> sometimes faster and sometimes slower. For hard disks, bigger chunks
> seems to be the way to go. For SSDs, smaller. I think 16K is
> probably the lowest reasonable limit unless you have tested your
> workload extensively, and over a long period of time, and have looked
> at drive wear issues.;
I'm not sure here.... since my issue seems to be IOPS, wouldn't
splitting a single IOP (ie, in your example the 64k read) into 4 IOPS (4
x 16k reads), which would seem to exacerbate the issue (not enough IOPS
available).
In which case, it could be beneficial to move to larger chunk sizes, so
even a 128k request can be kept as a single IOP instead of split into 2
? though there must also be an upper limit on the benefits here too?
At the moment, I'm thinking I will just leave the chunk size the same....
>> Either the above is wrong, or I've just convinced myself that reducing the
>> chunk size is not a good idea...
>>
>> DRBD can saturate GigE without any problem with random 4K writes. I
>> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
>> IOPS. The target arrays needs to keep up, but that is another story.
>> My testing with DRBD is that it starts to peter out at 10Gig, so if
>> you want more bandwidth you need some other approach. Some vendors
>> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
>> iSER should give you similar results with RDMA capable ethernet.
>> Linbit (the people who write DRBD) have a non GPL extension to DRBD
>> that uses RDMA so you can get more bandwidth that way as well.
>>
>> I have 10G ethernet for the crossover between the two servers, and another
>> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
>> either of these is rather low (I think it maxed out at around 15 to 20%)
>> definitely not anywhere near 100%. My thought here was on the latency of the
>> connection, but I really didn't have any ideas on how to measure that, and
>> how to test if it would really help. Also equipment seems a little less
>> common, and complex...
> I know that DRBD will not hit 40G. I have actually not done that much
> testing at 10G.
My concern is that even if I solve *this* bottleneck (ie, the 530 model
SSD being too busy), that there will be another bottleneck afterwards
(well, of course there will be, there is always one piece that is
limiting performance). How will I know what/where it is (assuming it
isn't the SSD/raid itself....).
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
>>
>> This is probably not true. If the drive really had to update 512K at
>> a time, then 4K writes would be 128x wear amplification. SSDs can be
>> bad, but usually not that bad.
>>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>>
>> The problem is that the FTL of the SSDs are a black box and as the
>> array gets bigger, the slowest drive dictates the array performance.
>> This is why the "big vendors" all map SSDs in the host and avoid or
>> minimize writing randomly. I know of one vendor install that has 4000
>> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
>> SSD shelves. The connection to ESXI is FC and the hosts are HA with
>> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
>> random write IOPS to pull this off, which I think is impossible with
>> stock parity raid, and very hard with raid-10.
>>
>>
>> My environment is rather small in comparison, it is only around 20 VM's
>> supporting around 80 users. 5 of the VM's are RDP servers.
>>
>>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
>>
>> RAID-10 will be faster, but you pay for this with capacity. It is
>> also a double-edged sword as SSDs themselves run faster if you leave
>> more free space on them, so RAID-10 absolutely might not be a lot
>> faster than RAID-5 with some space left over. Also remember that free
>> space on the SSDs only counts if it is actually unallocated. So you
>> need to trim the SSDs or start with a secure erased drive and then
>> never use the full capacity. It is best to leave an empty partition
>> that is untouched.
>>
>> Good point, when I initially provisioned the drives, I only used the first
>> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
>> was forced to allocate all of it. The place is to only end up with 960GB of
>> each 1000GB drive in use, so I could again leave a small chunk of
>> un-allocated space.
>>
>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
>>
>> The reads, especially with the pre 4.4 code or with raid-6 definitely
>> take their toll. Most SSDs are also not quite symmetrical in terms of
>> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
>> will probably not do 25K reads and 25K writes concurrently, but
>> instead stop somewhere around 18K. But your mileage may vary. If you
>> have 8 drives that do 20 read/write symmetric, with new raid-5, each
>> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
>> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
>> your data rate ends up maxing out at 80K IOPS. With the old raid-5
>> logic, you end up with 6 reads plus two writes per "OP", so you tend
>> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
>> these computations tend to fall apart, so 24 SSD arrays are not 3x
>> faster than 8 SSD arrays, at least with stock code.
>>
>> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
>> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
>> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
>> instead which would basically do the same thing?
>>
>> You also need to consider what raid does to the SSD FTL. As you
>> chatter a drive, its wear goes up and its performance goes down.
>> Different SSD models can vary wildly, but again the rule of thumb is
>> keep as much free space as possible on the drives. raid-5 or
>> mirroring is also 2:1 write amplification (ie, you are writing two
>> drives) and raid-6 is 3:1, on top of whatever the FTL write
>> amplification is at the time.
>>
>> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
>> per year.
>>
>> Tell me I'm crazy, but one option that I considered is using different RAID
>> levels. Right now I have RAID51 in that I have RAID5 on each machine and
>> DRBD (RAID1) between them.
>> What if I used RAID01 with DRBD between the machines doing the RAID1. In
>> this way, each machine has RAID0 (across 8 drives), which should provide
>> maximum performance and storage capacity and DRBD doing RAID1 between the
>> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
>> Slightly better would be RAID10 with DRBD between each pair of drives, and
>> then RAID0 across the DRBD device. It adds another layer of RAID, and more
>> complexity, but better security than RAID01...
> Your 5 to 7% wear per year is pretty safe. I have a pair of systems
> with proprietary code that is saturating dual 10GigE ports looking at
> wearout at 100+ years. Then again, the plastic cases of the drives
> will be dust by then.
Yep, I expect that we will outgrow the capacity of the drives before
they "wear out". I do monitor the drive reported wear values, and alert
on those (each time it drops 10% I get alerts, until I reset the alert
level) so that I won't be suddenly surprised when they hit 10% or
whatever....
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail. They usually "just fall off the bus" with no warning. So
> I dislike skipping redundancy. RAID turned an emergency into a
> mundane task. It is really a cost issue. If you can afford RAID-10
> and extra space, that will work best. I don't think RAID-50 with this
> few drives makes much sense.
>
I'm not sure, but I think I've had one of the 480GB drives fail, and 3
of the smaller 60GB and 80GB drives fail. So far, only the 480G failure
was "catastrophic", the others were still operating . All were replaced
by Intel.
The last one was just reported some large numbers in SMART, so I
questioned Intel, and their advice was replace under warranty, which I did.
I've had many more spinning disks fail over the years though, so I'm
pretty sure SSD's are more reliable, but certainly they do still fail,
and that's one of the reasons for RAID (and backups of course).
Regards,
Adam
Regards,
Adam
^ permalink raw reply
* Re: RAID5 Performance
From: Adam Goryachev @ 2016-07-28 13:57 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
In-Reply-To: <22426.2167.369879.756599@tree.ty.sabi.co.uk>
On 28/07/2016 23:28, Peter Grandi wrote:
> [ ... ]
>> That largely explains why in the tests I have mentioned small
>> sync write IOPS for many "consumerish" flash SSDs top at around
>> 100, instead of the usual > 10,000 for small non-sync writes.
> [ ... ]
>
> To summarize the preceding long discussion:
>
> * The stats reported show a low level of IOPS being carried out.
>
> * The critical part of the workload seems to be synchronous
> small writes.
>
> * Probably then the primary issue is the use of flash SSDs that
> have a limited number of IOPS for small synchronous writes.
>
> * A secondary issue is that RAID5 results in RMW for small
> writes.
>
> There are two possible options:
>
> * Replace the flash SSDs with those that are known to deliver
> high (at least > 10,000 single threaded) small synchronous
> write IOPS.
Is there a "known" SSD that you would suggest? My problem is that Intel
spec sheets seem to suggest that there is little performance difference
across the range of SSD's, so it's really not clear which SSD model I
should buy. Obviously it's not something I can afford to buy one of each
and test them either.
> * Relax the requirement for synchronous writes on *both* the
> primary and secondary DRBD servers, if feeling lucky.
I have the following entries for DRBD which were suggested by linbit
(which previously lifted performance from abysmal to more than
sufficient around 2+ years ago). I guess we are demanding more from the
system now, and we have added the 530 model drives later...
disk-barrier no;
disk-flushes no;
md-flushes no;
I've not configured anything special for LVM/MD/iSCSI/xen in relation to
cache/buffer/etc, and windows has disabled the write back buffer option
(within the VM).
> The third option, which is to change the workload so that it
> does not emit small synchronous writes to the storage layer,
> seems not practical in the context.
>
> Ideally the system would also be switched from RAID5 to RAID10
> to avoid the large penalty on small writes at the RAID level
> too.
>
> That may be considered expensive, but as I wrote:
>
>> [ ... ] it requires a storage layer that has to cover all
>> possible IO workloads optimally, [ ... ]
That will be the final optimisation/fallback option if still needed. I'd
prefer to avoid that as it limits the capacity of the system, and will
obviously cost more.
Do you have any other suggestions or ideas that might assist?
Thanks,
Adam
^ permalink raw reply
* Re: RAID5 Performance
From: Adam Goryachev @ 2016-07-28 13:50 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
In-Reply-To: <22425.63565.439521.592849@tree.ty.sabi.co.uk>
On 28/07/2016 22:19, Peter Grandi wrote:
> [ ... ]
>
>>> A very brave configuration, a shining example of the
>>> "syntactic" mindset, according to which any arbitrary
>>> combination of legitimate features must be fine :-).
>> While you may say that this configuration is very "brave", it
>> is actually quite common for VDI "appliance" deployments. [
>> ... ]
> There are a lot of very "brave" sysadms out there, and often I
> have to clean up after them :-).
>
> But then I am one of those boring people who think that «VDI
> "appliance" deployments» are usually a phenomenally bad idea, as
> it requires a storage layer that has to cover all possible IO
> workloads optimally, as indeed in:
Could I ask what you would "clean up" in the above system? What layers
would you remove/simplify?
At it's simplest, this system should be able to export a block of disk
space which the remote machine can present as a block device to the VM
(using Xen). Keep in mind there are actually multiple of these remote
machines. The method I've chosen is re-written here:
8 x 480GB Intel SSD (mix of 520 and 530 models)
Linux MD RAID5
LVM2
DRBD (takes an LV from each san and joins it together)
iSCSI (exports the block device to the 10Gbps network)
iSCSI (imported to the remove machine 2 x 1Gbps network)
multipathd (join the two iSCSI connections together)
Xen
Also, the two san machines have a second 10Gbps connection directly
between them.
Originally I had DRBD below LVM, but the folks at linbit switched those
two around to improve DRBD performance (multiple smaller DRBD devices is
better than one large, this might have changed since that happened
around 3 years ago).
> > [ ... ] The expectation, in terms of performance for VDI is
> > quite high. vmWare like to say you can get away with 8-12
> > IOPS per virtual. Most people think you only get good
> > performance with 100 IOPS per virtual. [ ... ]
>
> Those 100 random IOPS per VM are a bit "random", but roughly
> translate to one "disk arm" per VM, which is not necessarily
> enough: http://www.sabi.co.uk/blog/15-one.html#150305
>
> [ ... ]
>
>>> The queue sizes and waiting time on the second server are
>>> very low (on a somewhat similar system using 4TB disks I see
>>> waiting times in the 1-5 seconds range, not milliseconds).
>> The expectation, in terms of performance for VDI is quite high.
>> [ ... ]
> Sure, but the point here as to the speed issue is not that the
> SSDs are overwhelmed with IO, as the traffic on them is low and
> has relatively low latency, it is that very few IOPS are getting
> retired.
>
Previously when I was doing lots of tests on the system, I found I could
get great IO using larger block sizes, up to 2.5GB/s read and 1.5GB/s write.
Eventually, I found that using more real-world sized blocks, eg, 1k to
4k, I got abysmal transfer rates.
>>> Thus the most likely issue here is the 'fsync' problem: for
>>> "consumerish" SSDs barrier-writes are synchronous, because
>>> they don't have a battery/capacitor-backed cache, and rather
>>> slow for small writes, because of the large size of erase
>>> blocks, which can be mitigated with higher over-provisioning.
>> On many consumer SSDs, barrier writes are only barriers, and
>> are not syncs at all. You are guaranteed serialization but not
>> actual storage.
> Probably in this case that is irrelevant, because the numbers
> coming out from both the OP's experience and the tests in the
> links I mentioned show that small sync writes seem synchronous
> indeed for the 520/530, resulting in small write rates of around
> 1-5 MB/s, which matches the reported stats.
>
>> Then again, in a server setup, especially with redundant power
>> supplies, power loss to the SSDs is rare. You are more
>> protecting against system hangs and other inter-connectivity
>> issues.
> That is also likely irrelevant here. The firmware in the flash
> SSD does not know about the system setup, and the DRBD is
> probably configured to request synchronous writes on the
> secondary with protocol "C".
Yep, using protocol C at the moment.
> BTW I don't know whether the process(es) writing to the DRBD
> primary also request synchronous writes, but that's hopefully the
> case too, if the VD layer has been configured properly.
>
>> The real system solution is to have some quantity of non
>> volatile DRAM that you can stage writes (either a PCI-e card
>> like a FlashTec or one or more nvDIMMs).
> If this were the case then the VD layer and the DRBD layer could
> be told not to use sync writes, but the numbers reported seem to
> indicate that sync writes are happening.
>
>> This is how the "major vendors" deal with sync writes.
> At the system level, but at the device level the "major vendors"
> put a large capacitor in "enterprise" SSDs for two reasons, one
> of them to allow the persistence of the RAM write buffer, to
> minimize write amplification and erase latency (the other is not
> relevant here).
>
> [ ... ]
>
>>> * Small writes are a very challenging workload for flash SSDs
>>> without battery/capacitor-backed caches.
>> Even with battery backup, small writes create garbage
>> collection, so while batteries may give you some short term
>> bursts,
> That problem is mitigated with bigger overprovisioning in
> "enterprise" class flash SSDs. It can also be done in those of
> the "consumerish" class by partitioning them appropriately, or
> with 'hdparm -N'; but that does not seem to be the case here,
> becase the reported stats show a small number of IOPS with lowish
> queues sizes and not that huge latencies.
Can you advise what numbers I should look for, or worry about, which
would indicate that the problem is (or isn't) a erase cycle delay problem?
>> longer term, you still have to do the writes.
> Unfortunately flash SSDs don't merely have to "do the writes",
> as things are quite different: as I mentioned above the issue is
> the large erase blocks (and the several milliseconds it takes to
> erase one).
>
> In the absence of power backing for the write cache, every sync
> write, for example a 4KiB one, is (usually) stored immediately to
> a flash chip, which means (usually) a lot of write amplification
> because of RMW on the 8MiB (or larger) erase block plus the large
> latency (often near 10 milliseconds) of the erase operation
> before erase block programming.
>
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
>
> Some flash SSDs use an additional SLC buffer with smaller erase
> blocks and lower latency to reduce the problem with flushing sync
> writes directly to MLC etc, and that may explain why the 520s are
> better than the 530s (if the 520s have an SLC buffer, but IIRC
> intel started using an SLC buffer with the 540 series).
I can see the 545s series performs similar (or better, can't tell yet)
than the 520 series, but certainly it is better than the 530.
>
> Flash SSDs have only been popular for around 5 years, so it is
> understandable that some important aspects of their performance
> envelope (like what may happen on sync writes) is not well known
> yet.
>
Thank you, I appreciate all the responses.
So far, I've decided to make the following two changes:
1) Replace all 16 existing SSD's with the 1000GB 545s model, this will
double the capacity, and remove any of the 530 model drives. My concern
was (and is) that making actual use of this double capacity with the
same performance per drive will in effect halve the performance. I will
be able to leave 40G per drive un-partitioned, or partitioned and left
blank whichever is better.... Not sure if 40G per drive is enough to
help with the write/erase problem, but I guess it should be better than
nothing.
I think this change will produce a 40% improvement (potentially, given
the 520 drives are at 40% while the 530 is at 100%)
2) Upgrade to Linux kernel 4.6 from debian backports.
I think this change will give approx 30% improvement, because it will
reduce reads by 4 for each write, but a read is quicker than a write, so
I'm hoping for 30% overall. It sounds like the performance should be
somewhere between current and RAID10.
With the above two improvements, I'm hoping it will be enough to solve
the problem.
At this stage, if it is not enough to solve the problem, my fall-back
option is to convert to RAID10 but it's something I'd prefer to avoid
based on cost, storage capacity, and the fact it is difficult to expand
the existing system past 8 drives (hence capacity)...
I'm not convinced that changing chunk size will offer any benefit
(positive or negative), so will likely leave that as it is (64k chunk size).
Regards,
Adam
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID5 Performance
From: Peter Grandi @ 2016-07-28 13:28 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <22425.63565.439521.592849@tree.ty.sabi.co.uk>
[ ... ]
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
[ ... ]
To summarize the preceding long discussion:
* The stats reported show a low level of IOPS being carried out.
* The critical part of the workload seems to be synchronous
small writes.
* Probably then the primary issue is the use of flash SSDs that
have a limited number of IOPS for small synchronous writes.
* A secondary issue is that RAID5 results in RMW for small
writes.
There are two possible options:
* Replace the flash SSDs with those that are known to deliver
high (at least > 10,000 single threaded) small synchronous
write IOPS.
* Relax the requirement for synchronous writes on *both* the
primary and secondary DRBD servers, if feeling lucky.
The third option, which is to change the workload so that it
does not emit small synchronous writes to the storage layer,
seems not practical in the context.
Ideally the system would also be switched from RAID5 to RAID10
to avoid the large penalty on small writes at the RAID level
too.
That may be considered expensive, but as I wrote:
> [ ... ] it requires a storage layer that has to cover all
> possible IO workloads optimally, [ ... ]
^ permalink raw reply
* Re: RAID5 Performance
From: Anthony Youngman @ 2016-07-28 13:08 UTC (permalink / raw)
To: doug, Adam Goryachev; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAFx4rwTMySfR27gT6You7f2T28Hq8m5Zxme1p43Fv1vtcpt_-Q@mail.gmail.com>
On 28/07/16 01:11, Doug Dumitru wrote:
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail. They usually "just fall off the bus" with no warning. So
> I dislike skipping redundancy. RAID turned an emergency into a
> mundane task. It is really a cost issue. If you can afford RAID-10
> and extra space, that will work best. I don't think RAID-50 with this
> few drives makes much sense.
I came across an article about testing SSDs to destruction. First the
good news - they tended to last much longer than expected. And the bad
news? They typically contain a self-destruct switch. Once they start
failing, a power-cycle will (intentionally) kill them dead. ESPECIALLY
if they're from a major vendor.
So if you don't notice they're dying, or (as in the case of the tester)
you have power problems that tip them over the edge, your data WILL be
gone without warning.
Cheers,
Wol
^ permalink raw reply
* Re: SOLVED [was Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?]
From: Anthony Youngman @ 2016-07-28 12:53 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <57993F58.1050704@suddenlinkmail.com>
On 28/07/16 00:10, David C. Rankin wrote:
> On 07/27/2016 08:04 AM, Anthony Youngman wrote:
>> WD Blacks? Do they support SCT/ERC? I think these are desktop drives (like my
>> Barracudas) so you WILL get bitten by the timeout problem if anything goes
>> wrong. Do you know what you're doing here?
> Yes, WD Blacks, and yes, at least for the last 16 years I've managed, somehow,
> to provide a complete open-source backend for my law office. So I would answer
> the 2nd question in the affirmative as well. You can poo-poo drive X verses
> drive Y all you want, but I get a consistent 5 years out of each WD black and
> plan on a replacement cycle of 1/2 that. Go with what works for you.
>
I'll just say I don't think the past 16 years is a good guide at all ...
(but I will add I'm doing exactly the same as you - two 3TB desktop
drives in a mirror :-).
The timeout problem seems to be relatively recent. MOST 1TB or less
drives don't seem to have an issue. It's bigger drives that will bite you.
Cheers,
Wol
^ permalink raw reply
* Re: RAID5 Performance
From: Peter Grandi @ 2016-07-28 12:19 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <CAFx4rwSRYDACX631eLpX7Jr9jB2u4jbC7dr-0rdi9U1nV9og3Q@mail.gmail.com>
[ ... ]
>> A very brave configuration, a shining example of the
>> "syntactic" mindset, according to which any arbitrary
>> combination of legitimate features must be fine :-).
> While you may say that this configuration is very "brave", it
> is actually quite common for VDI "appliance" deployments. [
> ... ]
There are a lot of very "brave" sysadms out there, and often I
have to clean up after them :-).
But then I am one of those boring people who think that «VDI
"appliance" deployments» are usually a phenomenally bad idea, as
it requires a storage layer that has to cover all possible IO
workloads optimally, as indeed in:
> [ ... ] The expectation, in terms of performance for VDI is
> quite high. vmWare like to say you can get away with 8-12
> IOPS per virtual. Most people think you only get good
> performance with 100 IOPS per virtual. [ ... ]
Those 100 random IOPS per VM are a bit "random", but roughly
translate to one "disk arm" per VM, which is not necessarily
enough: http://www.sabi.co.uk/blog/15-one.html#150305
[ ... ]
>> The queue sizes and waiting time on the second server are
>> very low (on a somewhat similar system using 4TB disks I see
>> waiting times in the 1-5 seconds range, not milliseconds).
> The expectation, in terms of performance for VDI is quite high.
> [ ... ]
Sure, but the point here as to the speed issue is not that the
SSDs are overwhelmed with IO, as the traffic on them is low and
has relatively low latency, it is that very few IOPS are getting
retired.
>> Thus the most likely issue here is the 'fsync' problem: for
>> "consumerish" SSDs barrier-writes are synchronous, because
>> they don't have a battery/capacitor-backed cache, and rather
>> slow for small writes, because of the large size of erase
>> blocks, which can be mitigated with higher over-provisioning.
> On many consumer SSDs, barrier writes are only barriers, and
> are not syncs at all. You are guaranteed serialization but not
> actual storage.
Probably in this case that is irrelevant, because the numbers
coming out from both the OP's experience and the tests in the
links I mentioned show that small sync writes seem synchronous
indeed for the 520/530, resulting in small write rates of around
1-5 MB/s, which matches the reported stats.
> Then again, in a server setup, especially with redundant power
> supplies, power loss to the SSDs is rare. You are more
> protecting against system hangs and other inter-connectivity
> issues.
That is also likely irrelevant here. The firmware in the flash
SSD does not know about the system setup, and the DRBD is
probably configured to request synchronous writes on the
secondary with protocol "C".
BTW I don't know whether the process(es) writing to the DRBD
primary also request synchronous writes, but that's hopefully the
case too, if the VD layer has been configured properly.
> The real system solution is to have some quantity of non
> volatile DRAM that you can stage writes (either a PCI-e card
> like a FlashTec or one or more nvDIMMs).
If this were the case then the VD layer and the DRBD layer could
be told not to use sync writes, but the numbers reported seem to
indicate that sync writes are happening.
> This is how the "major vendors" deal with sync writes.
At the system level, but at the device level the "major vendors"
put a large capacitor in "enterprise" SSDs for two reasons, one
of them to allow the persistence of the RAM write buffer, to
minimize write amplification and erase latency (the other is not
relevant here).
[ ... ]
>> * Small writes are a very challenging workload for flash SSDs
>> without battery/capacitor-backed caches.
> Even with battery backup, small writes create garbage
> collection, so while batteries may give you some short term
> bursts,
That problem is mitigated with bigger overprovisioning in
"enterprise" class flash SSDs. It can also be done in those of
the "consumerish" class by partitioning them appropriately, or
with 'hdparm -N'; but that does not seem to be the case here,
becase the reported stats show a small number of IOPS with lowish
queues sizes and not that huge latencies.
> longer term, you still have to do the writes.
Unfortunately flash SSDs don't merely have to "do the writes",
as things are quite different: as I mentioned above the issue is
the large erase blocks (and the several milliseconds it takes to
erase one).
In the absence of power backing for the write cache, every sync
write, for example a 4KiB one, is (usually) stored immediately to
a flash chip, which means (usually) a lot of write amplification
because of RMW on the 8MiB (or larger) erase block plus the large
latency (often near 10 milliseconds) of the erase operation
before erase block programming.
That largely explains why in the tests I have mentioned small
sync write IOPS for many "consumerish" flash SSDs top at around
100, instead of the usual > 10,000 for small non-sync writes.
Some flash SSDs use an additional SLC buffer with smaller erase
blocks and lower latency to reduce the problem with flushing sync
writes directly to MLC etc, and that may explain why the 520s are
better than the 530s (if the 520s have an SLC buffer, but IIRC
intel started using an SLC buffer with the 540 series).
Flash SSDs have only been popular for around 5 years, so it is
understandable that some important aspects of their performance
envelope (like what may happen on sync writes) is not well known
yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] raid10: increment write counter after bio is split
From: Tomasz Majchrzak @ 2016-07-28 8:28 UTC (permalink / raw)
To: linux-raid; +Cc: shli, aleksey.obitotskiy, pawel.baldysiak, artur.paszkiewicz
md pending write counter must be incremented after bio is split,
otherwise it gets decremented too many times in end bio callback and
becomes negative.
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
Reviewed-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
---
drivers/md/raid10.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index c7de2a5..cfa96b5 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1072,6 +1072,8 @@ static void __make_request(struct mddev *mddev, struct bio *bio)
int max_sectors;
int sectors;
+ md_write_start(mddev, bio);
+
/*
* Register the new request and wait if the reconstruction
* thread has put up a bar for new requests.
@@ -1455,8 +1457,6 @@ static void raid10_make_request(struct mddev *mddev, struct bio *bio)
return;
}
- md_write_start(mddev, bio);
-
do {
/*
--
1.8.3.1
^ permalink raw reply related
* Re: test file disappeared after -f, -r, --add-journal for write-journal device
From: yizhan @ 2016-07-28 7:04 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, neilb, Jes.Sorensen
In-Reply-To: <20160728012225.GB48473@kernel.org>
On 07/28/2016 09:22 AM, Shaohua Li wrote:
> On Wed, Jul 27, 2016 at 08:07:19AM -0400, Yi Zhang wrote:
>> Hello maintainer
>>
>> Here is another strange phenomenon I found after do -f, -r, --add-journal for write-journal device.
>>
>> Kernel version: 4.7.0-rc7
>> Steps I used:
>> mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 7 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 --write-journal /dev/loop0 --bitmap=internal --bitmap-chunk=64M --chunk 512
>> mdadm --wait /dev/md0
>> mkfs.ext4 /dev/md0
>> mount -t ext4 /dev/md0 /mnt/fortest
>> cp bigfile /mnt/fortest &
>> wait
>> md5sum /mnt/fortest/bigfile > md5sum3
>> mdadm /dev/md0 -f /dev/loop0
> since loop0 is the journal device and it fails, we will mark the md0 readonly
> at this point. Filesystem can't do any write. It's possible the fortest/md5sum3
> files not hit disk yet, and are lost. I think your test should do a sync before
> mark journal disk failure.
I add sync operation before mark journal disk failure, this issue cannot
be reproduced any more,
thanks
Yi
>> mdadm /dev/md0 -r /dev/loop0
>> umount /dev/md0 -l
>> mdadm -o /dev/md0
>> mdadm /dev/md0 --add-journal /dev/loop0
>> mdadm --wait /dev/md0
>> mdadm -D /dev/md0
>> mount /dev/md0 /mnt/fortest
>> md5sum /mnt/fortest/bigfile > md5sum2 #<----this test file disappeared
> If a sync is done before journal disk fails and you still see the file
> disappeared, it's a bug.
>
> Thanks,
> Shaohua
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] raid5: fix incorrectly counter of conf->empty_inactive_list_nr
From: ZhengYuan Liu @ 2016-07-28 6:22 UTC (permalink / raw)
To: neilb; +Cc: shli, liuzhengyuang521, linux-raid, linux-kernel, ZhengYuan Liu
The counter conf->empty_inactive_list_nr is only used for determine if the
raid5 is congested which is deal with in function raid5_congested().
It was increased in get_free_stripe() when conf->inactive_list got to be
empty and decreased in release_inactive_stripe_list() when splice
temp_inactive_list to conf->inactive_list. However, this may have a
problem when raid5_get_active_stripe or stripe_add_to_batch_list was called,
because these two functions may call list_del_init(&sh->lru) to delete sh from
"conf->inactive_list + hash" which may cause "conf->inactive_list + hash" to
be empty when atomic_inc_not_zero(&sh->count) got false. So a check should be
done at these two point and increase empty_inactive_list_nr accordingly.
Otherwise the counter may get to be negative number which would influence
async readahead from VFS.
Signed-off-by: ZhengYuan Liu <liuzhengyuan@kylinos.cn>
---
drivers/md/raid5.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 7c53861..1656c44 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -659,6 +659,7 @@ raid5_get_active_stripe(struct r5conf *conf, sector_t sector,
{
struct stripe_head *sh;
int hash = stripe_hash_locks_hash(sector);
+ int inc_empty_inactive_list_flag;
pr_debug("get_stripe, sector %llu\n", (unsigned long long)sector);
@@ -703,7 +704,12 @@ raid5_get_active_stripe(struct r5conf *conf, sector_t sector,
atomic_inc(&conf->active_stripes);
BUG_ON(list_empty(&sh->lru) &&
!test_bit(STRIPE_EXPANDING, &sh->state));
+ inc_empty_inactive_list_flag = 0;
+ if (!list_empty(conf->inactive_list + hash))
+ inc_empty_inactive_list_flag = 1;
list_del_init(&sh->lru);
+ if (list_empty(conf->inactive_list + hash) && inc_empty_inactive_list_flag)
+ atomic_inc(&conf->empty_inactive_list_nr);
if (sh->group) {
sh->group->stripes_cnt--;
sh->group = NULL;
@@ -762,6 +768,7 @@ static void stripe_add_to_batch_list(struct r5conf *conf, struct stripe_head *sh
sector_t head_sector, tmp_sec;
int hash;
int dd_idx;
+ int inc_empty_inactive_list_flag;
if (!stripe_can_batch(sh))
return;
@@ -781,7 +788,11 @@ static void stripe_add_to_batch_list(struct r5conf *conf, struct stripe_head *sh
atomic_inc(&conf->active_stripes);
BUG_ON(list_empty(&head->lru) &&
!test_bit(STRIPE_EXPANDING, &head->state));
+ if (!list_empty(conf->inactive_list + hash))
+ inc_empty_inactive_list_flag = 1;
list_del_init(&head->lru);
+ if (list_empty(conf->inactive_list + hash) && inc_empty_inactive_list_flag)
+ atomic_inc(&conf->empty_inactive_list_nr);
if (head->group) {
head->group->stripes_cnt--;
head->group = NULL;
--
1.9.1
^ permalink raw reply related
* [PATCH 8/8] md-cluster: remove EXPERIMENTAL info
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
We can remove "EXPERIMENTAL" for md-cluster after
several rounds of development and bug fixes.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/Kconfig | 2 +-
drivers/md/md-cluster.c | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index 02a5345..74bc3ca 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -178,7 +178,7 @@ config MD_FAULTY
config MD_CLUSTER
- tristate "Cluster Support for MD (EXPERIMENTAL)"
+ tristate "Cluster Support for MD"
depends on BLK_DEV_MD
depends on DLM
default n
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 0984771..678f0a6 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -1266,7 +1266,6 @@ static struct md_cluster_operations cluster_ops = {
static int __init cluster_init(void)
{
- pr_warn("md-cluster: EXPERIMENTAL. Use with caution\n");
pr_info("Registering Cluster MD functions\n");
register_md_cluster_operations(&cluster_ops, THIS_MODULE);
return 0;
--
2.6.2
^ permalink raw reply related
* [PATCH 7/8] md-cluster: clean related infos of cluster
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
cluster_info and bitmap_info.nodes also need to be
cleared when array is stopped.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 0f046ed..ec46f4e 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5448,12 +5448,14 @@ static void md_clean(struct mddev *mddev)
mddev->degraded = 0;
mddev->safemode = 0;
mddev->private = NULL;
+ mddev->cluster_info = NULL;
mddev->bitmap_info.offset = 0;
mddev->bitmap_info.default_offset = 0;
mddev->bitmap_info.default_space = 0;
mddev->bitmap_info.chunksize = 0;
mddev->bitmap_info.daemon_sleep = 0;
mddev->bitmap_info.max_write_behind = 0;
+ mddev->bitmap_info.nodes = 0;
}
static void __md_stop_writes(struct mddev *mddev)
--
2.6.2
^ permalink raw reply related
* [PATCH 6/8] md-cluster: make resync lock also could be interruptted
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
When one node is perform resync or recovery, other nodes
can't get resync lock and could block for a while before
it holds the lock, so we can't stop array immediately for
this scenario.
To make array could be stop quickly, we check MD_CLOSING
in dlm_lock_sync_interruptible to make us can interrupt
the lock request.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index f3d584e..0984771 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -157,7 +157,8 @@ static int dlm_lock_sync_interruptible(struct dlm_lock_resource *res, int mode,
return ret;
wait_event(res->completion.wait,
- res->completion.done || kthread_should_stop());
+ res->completion.done || kthread_should_stop()
+ || test_bit(MD_CLOSING, &mddev->flags));
if (!res->completion.done) {
/*
* the convert queue contains the lock request when request is
@@ -1026,7 +1027,7 @@ static void metadata_update_cancel(struct mddev *mddev)
static int resync_start(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
- return dlm_lock_sync(cinfo->resync_lockres, DLM_LOCK_EX);
+ return dlm_lock_sync_interruptible(cinfo->resync_lockres, DLM_LOCK_EX, mddev);
}
static int resync_info_update(struct mddev *mddev, sector_t lo, sector_t hi)
--
2.6.2
^ permalink raw reply related
* [PATCH 5/8] md: changes for MD_STILL_CLOSED flag
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
When stop clustered raid while it is pending on resync,
MD_STILL_CLOSED flag could be cleared since udev rule
is triggered to open the mddev. So obviously array can't
be stopped soon and returns EBUSY.
mdadm -Ss md-raid-arrays.rules
set MD_STILL_CLOSED md_open()
... ... ... clear MD_STILL_CLOSED
do_md_stop
We make below changes to resolve this issue:
1. rename MD_STILL_CLOSED to MD_CLOSING since it is set
when stop array and it means we are stopping array.
2. let md_open returns early if CLOSING is set, so no
other threads will open array if one thread is trying
to close it.
3. no need to clear CLOSING bit in md_open because 1 has
ensure the bit is cleared, then we also don't need to
test CLOSING bit in do_md_stop.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 14 ++++++++------
drivers/md/md.h | 5 ++---
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 743cd21..0f046ed 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5567,8 +5567,7 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
mutex_lock(&mddev->open_mutex);
if ((mddev->pers && atomic_read(&mddev->openers) > !!bdev) ||
mddev->sync_thread ||
- test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) ||
- (bdev && !test_bit(MD_STILL_CLOSED, &mddev->flags))) {
+ test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) {
printk("md: %s still in use.\n",mdname(mddev));
if (did_freeze) {
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
@@ -5630,8 +5629,7 @@ static int do_md_stop(struct mddev *mddev, int mode,
if ((mddev->pers && atomic_read(&mddev->openers) > !!bdev) ||
mddev->sysfs_active ||
mddev->sync_thread ||
- test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) ||
- (bdev && !test_bit(MD_STILL_CLOSED, &mddev->flags))) {
+ test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) {
printk("md: %s still in use.\n",mdname(mddev));
mutex_unlock(&mddev->open_mutex);
if (did_freeze) {
@@ -6814,7 +6812,7 @@ static int md_ioctl(struct block_device *bdev, fmode_t mode,
err = -EBUSY;
goto out;
}
- set_bit(MD_STILL_CLOSED, &mddev->flags);
+ set_bit(MD_CLOSING, &mddev->flags);
mutex_unlock(&mddev->open_mutex);
sync_blockdev(bdev);
}
@@ -7063,9 +7061,13 @@ static int md_open(struct block_device *bdev, fmode_t mode)
if ((err = mutex_lock_interruptible(&mddev->open_mutex)))
goto out;
+ if (test_bit(MD_CLOSING, &mddev->flags)) {
+ mutex_unlock(&mddev->open_mutex);
+ return -ENODEV;
+ }
+
err = 0;
atomic_inc(&mddev->openers);
- clear_bit(MD_STILL_CLOSED, &mddev->flags);
mutex_unlock(&mddev->open_mutex);
check_disk_change(bdev);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index dc65ca6..0481f9a 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -201,9 +201,8 @@ struct mddev {
#define MD_CHANGE_PENDING 2 /* switch from 'clean' to 'active' in progress */
#define MD_UPDATE_SB_FLAGS (1 | 2 | 4) /* If these are set, md_update_sb needed */
#define MD_ARRAY_FIRST_USE 3 /* First use of array, needs initialization */
-#define MD_STILL_CLOSED 4 /* If set, then array has not been opened since
- * md_ioctl checked on it.
- */
+#define MD_CLOSING 4 /* If set, we are closing the array, do not open
+ * it then */
#define MD_JOURNAL_CLEAN 5 /* A raid with journal is already clean */
#define MD_HAS_JOURNAL 6 /* The raid array has journal feature set */
#define MD_RELOAD_SB 7 /* Reload the superblock because another node
--
2.6.2
^ permalink raw reply related
* [PATCH 4/8] md-cluster: introduce dlm_lock_sync_interruptible to fix tasks hang
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
When some node leaves cluster, then it's bitmap need to be
synced by another node, so "md*_recover" thread is triggered
for the purpose. However, with below steps. we can find tasks
hang happened either in B or C.
1. Node A create a resyncing cluster raid1, assemble it in
other two nodes (B and C).
2. stop array in B and C.
3. stop array in A.
linux44:~ # ps aux|grep md|grep D
root 5938 0.0 0.1 19852 1964 pts/0 D+ 14:52 0:00 mdadm -S md0
root 5939 0.0 0.0 0 0 ? D 14:52 0:00 [md0_recover]
linux44:~ # cat /proc/5939/stack
[<ffffffffa04cf321>] dlm_lock_sync+0x71/0x90 [md_cluster]
[<ffffffffa04d0705>] recover_bitmaps+0x125/0x220 [md_cluster]
[<ffffffffa052105d>] md_thread+0x16d/0x180 [md_mod]
[<ffffffff8107ad94>] kthread+0xb4/0xc0
[<ffffffff8152a518>] ret_from_fork+0x58/0x90
linux44:~ # cat /proc/5938/stack
[<ffffffff8107afde>] kthread_stop+0x6e/0x120
[<ffffffffa0519da0>] md_unregister_thread+0x40/0x80 [md_mod]
[<ffffffffa04cfd20>] leave+0x70/0x120 [md_cluster]
[<ffffffffa0525e24>] md_cluster_stop+0x14/0x30 [md_mod]
[<ffffffffa05269ab>] bitmap_free+0x14b/0x150 [md_mod]
[<ffffffffa0523f3b>] do_md_stop+0x35b/0x5a0 [md_mod]
[<ffffffffa0524e83>] md_ioctl+0x873/0x1590 [md_mod]
[<ffffffff81288464>] blkdev_ioctl+0x214/0x7d0
[<ffffffff811dd3dd>] block_ioctl+0x3d/0x40
[<ffffffff811b92d4>] do_vfs_ioctl+0x2d4/0x4b0
[<ffffffff811b9538>] SyS_ioctl+0x88/0xa0
[<ffffffff8152a5c9>] system_call_fastpath+0x16/0x1b
The problem is caused by recover_bitmaps can't reliably abort
when the thread is unregistered. So dlm_lock_sync_interruptible
is introduced to detect the thread's situation to fix the problem.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 38 +++++++++++++++++++++++++++++++++++++-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index ea2699e..f3d584e 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -10,6 +10,8 @@
#include <linux/module.h>
+#include <linux/completion.h>
+#include <linux/kthread.h>
#include <linux/dlm.h>
#include <linux/sched.h>
#include <linux/raid/md_p.h>
@@ -141,6 +143,40 @@ static int dlm_unlock_sync(struct dlm_lock_resource *res)
return dlm_lock_sync(res, DLM_LOCK_NL);
}
+/* An variation of dlm_lock_sync, which make lock request could
+ * be interrupted */
+static int dlm_lock_sync_interruptible(struct dlm_lock_resource *res, int mode,
+ struct mddev *mddev)
+{
+ int ret = 0;
+
+ ret = dlm_lock(res->ls, mode, &res->lksb,
+ res->flags, res->name, strlen(res->name),
+ 0, sync_ast, res, res->bast);
+ if (ret)
+ return ret;
+
+ wait_event(res->completion.wait,
+ res->completion.done || kthread_should_stop());
+ if (!res->completion.done) {
+ /*
+ * the convert queue contains the lock request when request is
+ * interrupted, and sync_ast could still be run, so need to
+ * cancel the request and reset completion
+ */
+ ret = dlm_unlock(res->ls, res->lksb.sb_lkid, DLM_LKF_CANCEL, &res->lksb, res);
+ reinit_completion(&res->completion);
+ if (unlikely(ret != 0))
+ pr_info("failed to cancel previous lock request "
+ "%s return %d\n", res->name, ret);
+ return -EPERM;
+ }
+ wait_for_completion(&res->completion);
+ if (res->lksb.sb_status == 0)
+ res->mode = mode;
+ return res->lksb.sb_status;
+}
+
static struct dlm_lock_resource *lockres_init(struct mddev *mddev,
char *name, void (*bastfn)(void *arg, int mode), int with_lvb)
{
@@ -272,7 +308,7 @@ static void recover_bitmaps(struct md_thread *thread)
goto clear_bit;
}
- ret = dlm_lock_sync(bm_lockres, DLM_LOCK_PW);
+ ret = dlm_lock_sync_interruptible(bm_lockres, DLM_LOCK_PW, mddev);
if (ret) {
pr_err("md-cluster: Could not DLM lock %s: %d\n",
str, ret);
--
2.6.2
^ permalink raw reply related
* [PATCH 3/8] md-cluster: remove some unnecessary dlm_unlock_sync
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
Since DLM_LKF_FORCEUNLOCK is used in lockres_free,
we don't need to call dlm_unlock_sync before free
lock resource.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 0a0605f..ea2699e 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -281,7 +281,7 @@ static void recover_bitmaps(struct md_thread *thread)
ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi, true);
if (ret) {
pr_err("md-cluster: Could not copy data from bitmap %d\n", slot);
- goto dlm_unlock;
+ goto clear_bit;
}
if (hi > 0) {
if (lo < mddev->recovery_cp)
@@ -293,8 +293,6 @@ static void recover_bitmaps(struct md_thread *thread)
md_wakeup_thread(mddev->thread);
}
}
-dlm_unlock:
- dlm_unlock_sync(bm_lockres);
clear_bit:
lockres_free(bm_lockres);
clear_bit(slot, &cinfo->recovery_map);
@@ -763,7 +761,6 @@ static int gather_all_resync_info(struct mddev *mddev, int total_slots)
md_check_recovery(mddev);
}
- dlm_unlock_sync(bm_lockres);
lockres_free(bm_lockres);
}
out:
@@ -1173,7 +1170,6 @@ static void unlock_all_bitmaps(struct mddev *mddev)
if (cinfo->other_bitmap_lockres) {
for (i = 0; i < mddev->bitmap_info.nodes - 1; i++) {
if (cinfo->other_bitmap_lockres[i]) {
- dlm_unlock_sync(cinfo->other_bitmap_lockres[i]);
lockres_free(cinfo->other_bitmap_lockres[i]);
}
}
--
2.6.2
^ permalink raw reply related
* [PATCH 2/8] md-cluster: use FORCEUNLOCK in lockres_free
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1469686612-16126-1-git-send-email-gqjiang@suse.com>
For dlm_unlock, we need to pass flag to dlm_unlock as the
third parameter instead of set res->flags.
Also, DLM_LKF_FORCEUNLOCK is more suitable for dlm_unlock
since it works even the lock is on waiting or convert queue.
Acked-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 23 ++++++++---------------
1 file changed, 8 insertions(+), 15 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 41573f1..0a0605f 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -194,25 +194,18 @@ out_err:
static void lockres_free(struct dlm_lock_resource *res)
{
- int ret;
+ int ret = 0;
if (!res)
return;
- /* cancel a lock request or a conversion request that is blocked */
- res->flags |= DLM_LKF_CANCEL;
-retry:
- ret = dlm_unlock(res->ls, res->lksb.sb_lkid, 0, &res->lksb, res);
- if (unlikely(ret != 0)) {
- pr_info("%s: failed to unlock %s return %d\n", __func__, res->name, ret);
-
- /* if a lock conversion is cancelled, then the lock is put
- * back to grant queue, need to ensure it is unlocked */
- if (ret == -DLM_ECANCEL)
- goto retry;
- }
- res->flags &= ~DLM_LKF_CANCEL;
- wait_for_completion(&res->completion);
+ /* use FORCEUNLOCK flag, so we can unlock even the lock is on the
+ * waiting or convert queue */
+ ret = dlm_unlock(res->ls, res->lksb.sb_lkid, DLM_LKF_FORCEUNLOCK, &res->lksb, res);
+ if (unlikely(ret != 0))
+ pr_err("failed to unlock %s return %d\n", res->name, ret);
+ else
+ wait_for_completion(&res->completion);
kfree(res->name);
kfree(res->lksb.sb_lvbptr);
--
2.6.2
^ permalink raw reply related
* [PATCH 1/8] md-cluster: call md_kick_rdev_from_array once ack failed
From: Guoqing Jiang @ 2016-07-28 6:16 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Guoqing Jiang
The new_disk_ack could return failure if WAITING_FOR_NEWDISK
is not set, so we need to kick the dev from array in case
failure happened.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 2ed547f..743cd21 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6092,9 +6092,11 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
export_rdev(rdev);
if (mddev_is_clustered(mddev)) {
- if (info->state & (1 << MD_DISK_CANDIDATE))
- md_cluster_ops->new_disk_ack(mddev, (err == 0));
- else {
+ if (info->state & (1 << MD_DISK_CANDIDATE)) {
+ err = md_cluster_ops->new_disk_ack(mddev, (err == 0));
+ if (err)
+ md_kick_rdev_from_array(rdev);
+ } else {
if (err)
md_cluster_ops->add_new_disk_cancel(mddev);
else
--
2.6.2
^ permalink raw reply related
* Re: test file disappeared after -f, -r, --add-journal for write-journal device
From: Shaohua Li @ 2016-07-28 1:22 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-raid, shli, neilb, Jes.Sorensen
In-Reply-To: <1403590832.9411207.1469621239264.JavaMail.zimbra@redhat.com>
On Wed, Jul 27, 2016 at 08:07:19AM -0400, Yi Zhang wrote:
> Hello maintainer
>
> Here is another strange phenomenon I found after do -f, -r, --add-journal for write-journal device.
>
> Kernel version: 4.7.0-rc7
> Steps I used:
> mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 7 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 --write-journal /dev/loop0 --bitmap=internal --bitmap-chunk=64M --chunk 512
> mdadm --wait /dev/md0
> mkfs.ext4 /dev/md0
> mount -t ext4 /dev/md0 /mnt/fortest
> cp bigfile /mnt/fortest &
> wait
> md5sum /mnt/fortest/bigfile > md5sum3
> mdadm /dev/md0 -f /dev/loop0
since loop0 is the journal device and it fails, we will mark the md0 readonly
at this point. Filesystem can't do any write. It's possible the fortest/md5sum3
files not hit disk yet, and are lost. I think your test should do a sync before
mark journal disk failure.
> mdadm /dev/md0 -r /dev/loop0
> umount /dev/md0 -l
> mdadm -o /dev/md0
> mdadm /dev/md0 --add-journal /dev/loop0
> mdadm --wait /dev/md0
> mdadm -D /dev/md0
> mount /dev/md0 /mnt/fortest
> md5sum /mnt/fortest/bigfile > md5sum2 #<----this test file disappeared
If a sync is done before journal disk fails and you still see the file
disappeared, it's a bug.
Thanks,
Shaohua
^ permalink raw reply
* Re: WARNING: CPU: 4 PID: 10512 at drivers/md/raid5-cache.c:728 r5l_do_reclaim+0x415/0x430 [raid456]
From: Shaohua Li @ 2016-07-28 1:13 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-raid
In-Reply-To: <2099656990.9407283.1469619529240.JavaMail.zimbra@redhat.com>
On Wed, Jul 27, 2016 at 07:38:49AM -0400, Yi Zhang wrote:
>
> Hello everyone
>
> I'm testing raid5-cache recently and found below issue on 4.7.0-rc7.
>
> [ 902.701162] md: bind<sdb1>
> [ 902.701248] md: bind<sdd1>
> [ 902.702566] md: bind<sde1>
> [ 902.702625] md: bind<sdf1>
> [ 902.703306] md: bind<sdc1>
> [ 902.705899] md/raid:md0: not clean -- starting background reconstruction
> [ 902.705972] md/raid:md0: device sdf1 operational as raid disk 3
> [ 902.705974] md/raid:md0: device sde1 operational as raid disk 2
> [ 902.705975] md/raid:md0: device sdd1 operational as raid disk 1
> [ 902.705976] md/raid:md0: device sdb1 operational as raid disk 0
> [ 902.706312] md/raid:md0: allocated 4374kB
> [ 902.706371] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2
> [ 902.706372] RAID conf printout:
> [ 902.706380] --- level:6 rd:4 wd:4
> [ 902.706381] disk 0, o:1, dev:sdb1
> [ 902.706382] disk 1, o:1, dev:sdd1
> [ 902.706383] disk 2, o:1, dev:sde1
> [ 902.706384] disk 3, o:1, dev:sdf1
> [ 902.706393] md/raid456: discard support disabled due to uncertainty.
> [ 902.706394] Set raid456.devices_handle_discard_safely=Y to override.
> [ 902.706396] md/raid:md0: using device sdc1 as journal
> [ 902.707354] created bitmap (1 pages) for device md0
> [ 902.707401] md0: bitmap initialized from disk: read 1 pages, set 8 of 8 bits
> [ 902.781331] md0: detected capacity change from 0 to 1071644672
> [ 902.781465] md: resync of RAID array md0
> [ 902.781468] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 902.781469] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
> [ 902.781475] md: using 128k window, over a total of 523264k.
> [ 902.829618] md: couldn't update array info. -22
> [ 902.876797] md: couldn't update array info. -22
> [ 902.896926] md: couldn't update array info. -22
> [ 909.185914] md: md0: resync done.
> [ 909.262779] RAID conf printout:
> [ 909.262782] --- level:6 rd:4 wd:4
> [ 909.262783] disk 0, o:1, dev:sdb1
> [ 909.262784] disk 1, o:1, dev:sdd1
> [ 909.262785] disk 2, o:1, dev:sde1
> [ 909.262786] disk 3, o:1, dev:sdf1
> [ 912.795104] md/raid:md0: Disk failure on sdb1, disabling device.
> md/raid:md0: Operation continuing on 3 devices.
> [ 912.851893] RAID conf printout:
> [ 912.851895] --- level:6 rd:4 wd:3
> [ 912.851897] disk 0, o:0, dev:sdb1
> [ 912.851898] disk 1, o:1, dev:sdd1
> [ 912.851899] disk 2, o:1, dev:sde1
> [ 912.851900] disk 3, o:1, dev:sdf1
> [ 912.855130] RAID conf printout:
> [ 912.855133] --- level:6 rd:4 wd:3
> [ 912.855135] disk 1, o:1, dev:sdd1
> [ 912.855136] disk 2, o:1, dev:sde1
> [ 912.855137] disk 3, o:1, dev:sdf1
> [ 912.885778] md: unbind<sdb1>
> [ 912.894205] md: export_rdev(sdb1)
> [ 1548.098231] md: bind<sdb1>
> [ 1548.452714] RAID conf printout:
> [ 1548.452717] --- level:6 rd:4 wd:3
> [ 1548.452718] disk 0, o:1, dev:sdb1
> [ 1548.452719] disk 1, o:1, dev:sdd1
> [ 1548.452720] disk 2, o:1, dev:sde1
> [ 1548.452721] disk 3, o:1, dev:sdf1
> [ 1548.486067] ------------[ cut here ]------------
> [ 1548.486076] WARNING: CPU: 4 PID: 10512 at drivers/md/raid5-cache.c:728 r5l_do_reclaim+0x415/0x430 [raid456]
> [ 1548.486077] Modules linked in: fuse btrfs vfat msdos fat ext4 jbd2 mbcache binfmt_misc raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ipt_REJECT nf_reject_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sb_edac edac_core snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic x86_pkg_temp_thermal snd_hda_intel intel_powerclamp coretemp s
nd_hda_codec
> [ 1548.486117] kvm_intel kvm snd_hda_core irqbypass crct10dif_pclmul crc32_pclmul snd_hwdep ghash_clmulni_intel snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper hp_wmi iTCO_wdt snd_timer sparse_keymap rfkill cryptd iTCO_vendor_support mei_me ioatdma snd mei shpchp pcspkr sg soundcore i2c_i801 tpm_infineon lpc_ich mfd_core dca nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod nouveau video mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm isci e1000e ata_generic libsas pata_acpi ahci libahci scsi_transport_sas firewire_ohci libata firewire_core ptp crc32c_intel serio_raw pps_core i2c_core crc_itu_t wmi fjes dm_mirror dm_region_hash dm_log dm_mod
> [ 1548.486152] CPU: 4 PID: 10512 Comm: md0_resync Tainted: G W 4.7.0-rc7 #1
> [ 1548.486153] Hardware name: Hewlett-Packard HP Z620 Workstation/158A, BIOS J61 v03.69 03/25/2014
> [ 1548.486155] 0000000000000286 000000002369dfe1 ffff88079d713b40 ffffffff8134caec
> [ 1548.486157] 0000000000000000 0000000000000000 ffff88079d713b80 ffffffff8108c351
> [ 1548.486159] 000002d8a6ae4d80 000000008f0c904f ffff8807f64b0000 ffff8807c4ab6888
> [ 1548.486161] Call Trace:
> [ 1548.486166] [<ffffffff8134caec>] dump_stack+0x63/0x87
> [ 1548.486169] [<ffffffff8108c351>] __warn+0xd1/0xf0
> [ 1548.486172] [<ffffffff8108c48d>] warn_slowpath_null+0x1d/0x20
> [ 1548.486175] [<ffffffffa097d805>] r5l_do_reclaim+0x415/0x430 [raid456]
> [ 1548.486178] [<ffffffff811f9c50>] ? kfree+0x120/0x170
> [ 1548.486181] [<ffffffffa097e33b>] r5l_quiesce+0x7b/0xa0 [raid456]
> [ 1548.486183] [<ffffffffa0970da0>] raid5_quiesce+0x50/0x2a0 [raid456]
> [ 1548.486186] [<ffffffff810d0250>] ? prepare_to_wait_event+0xf0/0xf0
> [ 1548.486189] [<ffffffff8156d00e>] md_do_sync+0xe7e/0xf60
> [ 1548.486192] [<ffffffff810c4370>] ? enqueue_entity+0x2a0/0xcc0
> [ 1548.486194] [<ffffffff810c201d>] ? update_curr+0xed/0x180
> [ 1548.486196] [<ffffffff810c026e>] ? account_entity_dequeue+0xae/0xd0
> [ 1548.486197] [<ffffffff810c27f6>] ? dequeue_entity+0x266/0x980
> [ 1548.486200] [<ffffffff810b5ef5>] ? check_preempt_curr+0x75/0x90
> [ 1548.486202] [<ffffffff810b5f29>] ? ttwu_do_wakeup+0x19/0xe0
> [ 1548.486204] [<ffffffff810c2fce>] ? dequeue_task_fair+0xbe/0x830
> [ 1548.486206] [<ffffffff810c3995>] ? put_prev_entity+0x35/0x730
> [ 1548.486209] [<ffffffff8102c6d9>] ? __switch_to+0x219/0x5c0
> [ 1548.486211] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
> [ 1548.486214] [<ffffffff81566aa6>] md_thread+0x136/0x150
> [ 1548.486216] [<ffffffff81566970>] ? find_pers+0x70/0x70
> [ 1548.486219] [<ffffffff810ab698>] kthread+0xd8/0xf0
> [ 1548.486221] [<ffffffff816e873f>] ret_from_fork+0x1f/0x40
> [ 1548.486223] [<ffffffff810ab5c0>] ? kthread_park+0x60/0x60
> [ 1548.486224] ---[ end trace 242e9516e36acb1e ]---
> [ 1548.543855] md: recovery of RAID array md0
> [ 1548.543857] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 1548.543859] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [ 1548.543864] md: using 128k window, over a total of 523264k.
> [ 1548.543868] md: md0: recovery done.
> [ 1548.574961] RAID conf printout:
> [ 1548.574964] --- level:6 rd:4 wd:4
> [ 1548.574966] disk 0, o:1, dev:sdb1
> [ 1548.574967] disk 1, o:1, dev:sdd1
> [ 1548.574968] disk 2, o:1, dev:sde1
> [ 1548.574969] disk 3, o:1, dev:sdf1
>
> Steps I used:
>
> mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 4
> /dev/sdb1 /dev/sd[d-f]1 --write-journal /dev/sdc1 --bitmap=internal
> --bitmap-chunk=64M --chunk 512
> mdadm --wait /dev/md0
> mkfs.ext4 /dev/md0
> mdadm /dev/md0 -f /dev/sdb1
> mdadm /dev/md0 -r /dev/sdb1
> mdadm /dev/md0 -a /dev/sdb1
Thanks for the reporting. Looks the assumption .quiesce is called with
reconfig_mutex locked is incorrect. I'll work out a fix.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH] raid5: fix incorrectly counter of conf->empty_inactive_list_nr
From: Shaohua Li @ 2016-07-28 0:19 UTC (permalink / raw)
To: Zhengyuan Liu; +Cc: linux-raid, 刘云, 胡海
In-Reply-To: <CAOOPZo5B8EF0C3+Yg1WVYJx_jerR6zjNjbixpHhvQx4zOJqvxw@mail.gmail.com>
On Wed, Jul 20, 2016 at 05:22:00PM +0800, Zhengyuan Liu wrote:
> The counter conf->empty_inactive_list_nr is only used for determine if
> the raid5 is congested which is deal with in function
> raid5_congested(). It was increased in get_free_stripe() when
> conf->inactive_list got emptied and decreased in
> release_inactive_stripe_list() when splice temp_inactive_list to
> conf->inactive_list. However, this could cause to problem when
> raid5_get_active_stripe was called, this function may call
> list_del_init(&sh->lru) to delete sh from "conf->inactive_list +
> hash" which may cause "conf->inactive_list + hash" to be empty when
> atomic_inc_not_zero(&sh->count) got false.
> I have found conf->empty_inactive_list_nr to be negative number
> during my random test.
> Is there anything out of my thinking?
Good catch! I think you are right, we get a stripe from inactive list and
should decrement the count. Also we should do the same thing for
stripe_add_to_batch_list. Please resend the patch with proper format, see
Documentation/email-clients.txt
Thanks,
Shaohua
^ permalink raw reply
* Re: RAID5 Performance
From: Doug Dumitru @ 2016-07-28 0:11 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <7af0cc98-e395-9446-05eb-a6c0ca20f187@websitemanagers.com.au>
On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
>
> On 27/07/2016 15:36, Doug Dumitru wrote:
>
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
> Version : 1.2
> Creation Time : Wed Aug 22 00:47:03 2012
> Raid Level : raid5
> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
> Raid Devices : 8
> Total Devices : 8
> Persistence : Superblock is persistent
>
> Update Time : Wed Jul 27 11:32:00 2016
> State : active
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : san1:1 (local to host san1)
> UUID : 707957c0:b7195438:06da5bc4:485d301c
> Events : 2185221
>
> Number Major Minor RaidDevice State
> 7 8 65 0 active sync /dev/sde1
> 13 8 1 1 active sync /dev/sda1
> 8 8 81 2 active sync /dev/sdf1
> 5 8 113 3 active sync /dev/sdh1
> 9 8 97 4 active sync /dev/sdg1
> 12 8 17 5 active sync /dev/sdb1
> 10 8 49 6 active sync /dev/sdd1
> 11 8 33 7 active sync /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
> 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
> 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
> 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
> 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
> 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
> 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
> 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
> 0.18 0.28 0.23 0.32 0.16 9.60
> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
> 0.08 38.00 0.00 38.00 20.00 4.00
> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
> 0.03 14.00 4.00 24.00 14.00 2.80
> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
> 0.09 44.00 0.00 44.00 22.00 4.40
> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
> 7.92 41.03 0.00 41.87 4.98 100.00
> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
> 0.22 30.29 0.00 30.29 28.57 20.00
> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
> 1.72 13.46 0.40 40.18 4.27 57.20
> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
> 1.40 40.91 0.00 43.39 28.34 99.20
> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
> 0.08 21.00 0.00 28.00 21.00 8.40
> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
> 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
> 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
> 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
> 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
> 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
> 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
> 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
> 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
> *
>
> Enhance the queue depth. Standard kernel queue depth is OK for old
> single drives with small caches, but not for modern drives or RAID
> arrays:
>
> echo 512 > /sys/block/sda/queue/nr_requests
>
> So my question is should I increase the configured nr_requests above the
> current 128?
>
> With your workload, it probably won't matter too much. Really high
> queue depths are great on paper, but hard to actually see.
>
>
> Is there some way to see if this would help or not?
> Would it hurt to increase this (even if it doesn't help)?
>
>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?
>
> This is not how a random write works. If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
> writes. With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes. With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high. Note that the new logic is only
> raid-5. raid-6 cannot use the new logic and has to read the stripe
> from every drive.
>
> Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should
> provide a significant performance boost even if nothing else changes.
This should help your raid-5 array, at least noticeably, provided the
new kernel actually has the
Facebook Read/Modify/Write new logic included. Based on the version
it should. You can very this by doing random writes and looking at
iostat. If you see 2 reads and 2 writes for every inbound write, you
have the new code. If you see 6 reads and 2 writes for every inbound
write, you have the old code.
While this sounds huge, the change will be moderated by the behaviour
of SSDs. Random writes are much more expensive than read and the new
logic only lowers the number of reads. ... and raid-6 is not impacted
at all.
> The stripe size impacts when the system does can avoid doing a
> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads. I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
>
> So reducing the chunk size will have minimal impact... but reducing it
> should still provide some performance boost. Since I'm recreating the array
> anyway, what size makes the most sense? 16k or go straight to the minimum of
> 4k? Would a smaller chunk size increase the IOPS because we need to make
> more (smaller) requests for the same data, potentially from more drives?
>
> ie, currently, a single read request for 4k will be done by reading one
> chunk (64k) from one of the 8 drives (1 IOPS)
> currently, a single write request for 4k will be done by reading one chunk
> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
> However, a read (or write) 48k request would be identical to the above,
> while a smaller chunk size (4k) would mean:
> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
> disks (7 IOPS)
> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
> somewhere, but it might not be)
> - read 2 x 4k chunks (the only 2 data chunks that
> won't be written) + write 6 x 4k chunks
> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
You are confused about what chunk size is. It is not the IO size
limit. It is just a layout calculation. If your chunk is 64K, then
64K is written to one disk before the array moves on to the next disk.
If you read 4K, then only 4K is read. You never need to read (or
write) and entire chunk.
Lower chunk sizes are useful if your application does enough long
writes to reach full stripes. At 64K x 7 drives, this is 448KB. If
you are writing multi-megabytes, then 64K chunks is a good idea. If
you are writing 128KB, you might want to go down to 16KB chunks. The
problem with little chunks is if you read 64K from and array with 16KB
chunks, you will cut your IO request into four parts. This is
sometimes faster and sometimes slower. For hard disks, bigger chunks
seems to be the way to go. For SSDs, smaller. I think 16K is
probably the lowest reasonable limit unless you have tested your
workload extensively, and over a long period of time, and have looked
at drive wear issues.;
> Either the above is wrong, or I've just convinced myself that reducing the
> chunk size is not a good idea...
>
> DRBD can saturate GigE without any problem with random 4K writes. I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS. The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach. Some vendors
> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
>
> I have 10G ethernet for the crossover between the two servers, and another
> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
> either of these is rather low (I think it maxed out at around 15 to 20%)
> definitely not anywhere near 100%. My thought here was on the latency of the
> connection, but I really didn't have any ideas on how to measure that, and
> how to test if it would really help. Also equipment seems a little less
> common, and complex...
I know that DRBD will not hit 40G. I have actually not done that much
testing at 10G.
> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...
>
> This is probably not true. If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification. SSDs can be
> bad, but usually not that bad.
>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly. I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves. The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.
>
>
> My environment is rather small in comparison, it is only around 20 VM's
> supporting around 80 users. 5 of the VM's are RDP servers.
>
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?
>
> RAID-10 will be faster, but you pay for this with capacity. It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over. Also remember that free
> space on the SSDs only counts if it is actually unallocated. So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity. It is best to leave an empty partition
> that is untouched.
>
> Good point, when I initially provisioned the drives, I only used the first
> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
> was forced to allocate all of it. The place is to only end up with 960GB of
> each 1000GB drive in use, so I could again leave a small chunk of
> un-allocated space.
>
> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?
>
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll. Most SSDs are also not quite symmetrical in terms of
> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K. But your mileage may vary. If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS. With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
>
> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
> instead which would basically do the same thing?
>
> You also need to consider what raid does to the SSD FTL. As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives. raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
>
> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
> per year.
>
> Tell me I'm crazy, but one option that I considered is using different RAID
> levels. Right now I have RAID51 in that I have RAID5 on each machine and
> DRBD (RAID1) between them.
> What if I used RAID01 with DRBD between the machines doing the RAID1. In
> this way, each machine has RAID0 (across 8 drives), which should provide
> maximum performance and storage capacity and DRBD doing RAID1 between the
> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
> Slightly better would be RAID10 with DRBD between each pair of drives, and
> then RAID0 across the DRBD device. It adds another layer of RAID, and more
> complexity, but better security than RAID01...
Your 5 to 7% wear per year is pretty safe. I have a pair of systems
with proprietary code that is saturating dual 10GigE ports looking at
wearout at 100+ years. Then again, the plastic cases of the drives
will be dust by then.
I don't know about you, but I do have SSDs, even from major vendors,
that fail. They usually "just fall off the bus" with no warning. So
I dislike skipping redundancy. RAID turned an emergency into a
mundane task. It is really a cost issue. If you can afford RAID-10
and extra space, that will work best. I don't think RAID-50 with this
few drives makes much sense.
Doug
>
> Regards,
> Adam
>
>
--
Doug Dumitru
EasyCo LLC
^ permalink raw reply
* Re: Kernel deadlock during mdadm reshape
From: Bart Van Assche @ 2016-07-28 0:04 UTC (permalink / raw)
To: Michael Shaver, mdraid
In-Reply-To: <bbcfb493-cdee-54cd-99aa-b1f9217b3b43@gmail.com>
On 07/26/2016 07:18 PM, Michael Shaver wrote:
> I am experiencing the exact same problem reported in this thread:
>
> http://www.spinics.net/lists/raid/msg52235.html
>
> Also reported here:
>
> https://forums.gentoo.org/viewtopic-t-1043706.html
>
> And here:
>
> https://bbs.archlinux.org/viewtopic.php?id=212108
>
> I have a raid5 array of 2TB disks currently stuck at 94% of a mdadm
> reshape squeal to a grow operation from 4 disks to 5. In my case, I did
> have a drive drop out of the array during the reshape.
>
> [ ... ]
> [<ffffffff811cb8dd>] truncate_inode_pages_range+0x46d/0x880
> [<ffffffff811cbd05>] truncate_inode_pages+0x15/0x20
> [ ... ]
Hello Michael,
I ran into a similar lockup while retesting the dm-mpath driver. So we
either ran into a different lockup or the lockup was not related to the
md driver. Were you using a distro kernel or a kernel you built
yourself? Anyway, was THP (transparent huge page) support enabled? If
so, can you repeat your test with THP support disabled and see whether
that makes a difference?
Thanks,
Bart.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox