* Breaks LSI RAID on C600 chipset
From: David F. @ 2015-04-04 0:25 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Loading V3.3.2 Aug 2014 on an asus z9pa-u8 mobo, with the controller
in LSI RAID mode and configured using RAID 1 via the BIOS, will break
the RAID. (no longer configured as RAID). All it takes is starting
mdadm which seems to work for that session but on reboot the RAID is
gone. The z9pa-u8 uses the C602 intel chipset.
Customer reported same on what they called an Intel ESRT2 which
appears to use the same line C600 based chipset. Not sure if in LSI
mode, but presume so. That controller allows you to pick LSI mode,
Intel RAID mode, or just non RAID normal controller mode.
Unfortunately, mine is a production server so I can't do much testing
(even trying to load it will probably break it). Not sure if you can
get your hands on one to test. Loading using DMRAID worked okay
which is how I got around it.
^ permalink raw reply
* [PATCH] md: fix md io stats accounting broken
From: Gu Zheng @ 2015-04-03 0:44 UTC (permalink / raw)
To: neilb; +Cc: sim, linux-raid, linux-kernel, stable
Simon reported the md io stats accounting issue:
"
I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
the other two being write-mostly:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
"
The cause is commit "18c0b223cf9901727ef3b02da6711ac930b4e5d4" uses the
generic_start_io_acct to account the disk stats rather than the open code,
but it also introduced the increase to .in_flight[rw] which is needless to
md. So we re-use the open code here to fix it.
Reported-by: Simon Kirby <sim@hostway.ca>
Cc: <stable@vger.kernel.org> 3.19
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
---
drivers/md/md.c | 6 +++++-
1 files changed, 5 insertions(+), 1 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 717daad..e617878 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -249,6 +249,7 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
const int rw = bio_data_dir(bio);
struct mddev *mddev = q->queuedata;
unsigned int sectors;
+ int cpu;
if (mddev == NULL || mddev->pers == NULL
|| !mddev->ready) {
@@ -284,7 +285,10 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
sectors = bio_sectors(bio);
mddev->pers->make_request(mddev, bio);
- generic_start_io_acct(rw, sectors, &mddev->gendisk->part0);
+ cpu = part_stat_lock();
+ part_stat_inc(cpu, &mddev->gendisk->part0, ios[rw]);
+ part_stat_add(cpu, &mddev->gendisk->part0, sectors[rw], sectors);
+ part_stat_unlock();
if (atomic_dec_and_test(&mddev->active_io) && mddev->suspended)
wake_up(&mddev->sb_wait);
--
1.7.7
^ permalink raw reply related
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: Simon Kirby @ 2015-04-02 19:54 UTC (permalink / raw)
To: NeilBrown; +Cc: Gu Zheng, linux-raid
In-Reply-To: <20150401220725.53ee8f99@notabene.brown>
On Wed, Apr 01, 2015 at 10:07:25PM +1100, NeilBrown wrote:
> On Wed, 1 Apr 2015 15:39:47 +0800 Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
>
> > Hi Neil,
> > One question about md io stats account:
> > I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
> > md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
> > need it? or I missed something?
>
> No, I don't think you are missing anything. md doesn't track 'ticks' - and
> no-one has ever complained.
Hmm, yes, I guess "working" before was "0% utilization". :)
Simon-
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-02 4:07 UTC (permalink / raw)
To: NeilBrown; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150402111941.104d0633@notabene.brown>
On Thu, Apr 02, 2015 at 11:19:41AM +1100, NeilBrown wrote:
> On Wed, 1 Apr 2015 16:40:57 -0700 Shaohua Li <shli@fb.com> wrote:
>
>
> > > Your code does avoid write-hole-protection for fill-stripe-writes, and this
> > > would greatly reduce the number of block that were written multiple times.
> > > However I'm not convinced that is correct.
> > > A reasonable goal is that if the system crashes while writing to a storage
> > > device, then reads should return the old data or not new data, not anything
> > > else. A crash in the middle of a full-stripe-write to a degraded array
> > > could result in some block in the stripe appearing to contain data that is
> > > different to both the old and the new. If you are going to close the whole,
> > > I think it should be done properly.
> >
> > I can do it simpley. But don't think this assumption is true. If you
> > write to a disk range and there is failure, there is nothing guarantee
> > you can either read old data or new data.
>
> If you write a range of blocks to a normal disk and crash during the write,
> each block will contain either the old data or the new data.
> If you write a range to a degraded RAID5 and crash during the write, you
> cannot make that same guarantee.
> I don't know how important this is, but then I don't really know how
> important any of this is.
>
> >
> > >
> > > A combined log would "simply" involve writing every data block and every
> > > compute parity block (with index information) to the log device.
> > > Replaying the log would collect data blocks and flush out those in a stripe
> > > once the parity block(s) for that stripe became available.
> > >
> > > I think this would actually turn into a fairly simple logging mechanism.
> >
> > It's not simple at all. It's unlikely we write data and parity
> > continuously in disk and in the same time. This will make log checkpoint
> > fairly complex.
>
> I don't see any cause for complexity. Let me be more explicit.
>
> I imagine that all data remains in the stripe cache, in memory, until it is
> finally written to the RAID5. So the stripe cache will need to be quite a
> bit bigger.
>
> Every time we get a block that we want to write, either a new data block or a
> a computed parity block, we queue it to the log.
>
> The log works like this:
> - take the first (e.g.) 256 blocks in the queue, create a header to describe
> them, write the header with FUA, then write all the data blocks. If there
> are fewer than 256, just write what we have.
> - when the header write completes, all blocks written *previously* are now
> safe and we can call bio_end_io on data or unlock the stripe for parity.
> - loop back and write some more blocks. If there are no blocks to write,
> write a header which describes an empty set of blocks, and wait for more
> blocks to appear.
Ok, this is similar.
> Each stripe_head needs to track (roughly) where the relevant blocks were
> written so it can release them when the stripe is written.
> I would conceptually divide the log into 32 regions and keep a 32bit number
> with each stripe. When a block is assigned to a region in the log, the
> relevant bit is set for the stripe, and a per-region counter is incremented.
> When a stripe completes its write, the region counters for all the bits are
> cleared. The log cannot progress into a region which has a non-zero counter.
I like this region idea very much. Previously I thought the combined log
is complex because data and parity are not in adjacent disk location,
and can cause fragement, so make checkpoint complex. The region
effectively solves the problem, but a big size region would still have
the fragement issue. We can divide the disk to a lot of equal sized
regions, the region size could be 4k*raid_disks*2 for example. Each
region is a log for exact one stripe. Write will append data to such
log and write is finished. parity append to the log too and then the
region is considered settled down. The downside is meta will use 1
sector even just several bytes are required and this will produce a lot
of small size IO too.
I'm not enthusiastic to use stripe cache though, we can't keep all data
in stripe cache. What we really need is an index.
Thanks,
Shaohua
^ permalink raw reply
* Re: /dev/md0 can't be created
From: Xiao Ni @ 2015-04-02 2:52 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150401135321.2f421b96@notabene.brown>
Hi Neil
I think that is enough to say it's not the problem about md. I really be
thankful for you about the analysis.
Best Regards
Xiao
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, April 1, 2015 10:53:21 AM
> Subject: Re: /dev/md0 can't be created
>
> On Fri, 27 Mar 2015 02:06:21 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> >
> > I'm lucky, I reproduced it again and I ran the command udevadm monitor >
> > monitor before test.
> > I'll put it in as an attachment.
> >
> > I found there is a not whole message in the last line.
> >
> > Xiao
>
> The line:
>
> UDEV [7083.531441] add /devices/virtual/block/md0 (block)
>
> confirms that udev saw an 'add' event for /dev/md0.
>
> That should be enough for udev to create the device node.
> If it doesn't, you need to look into udev and find out why it doesn't.
> I can't really help you there - I don't find udev very easy to debug.
>
> NeilBrown
>
>
> >
> > ----- Original Message -----
> > > From: "Xiao Ni" <xni@redhat.com>
> > > To: "NeilBrown" <neilb@suse.de>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, March 26, 2015 3:36:35 PM
> > > Subject: Re: /dev/md0 can't be created
> > >
> > > Hi Neil
> > >
> > > I can reproduce it now. Do you want me to run udevadm monitor before
> > > the
> > > test?
> > > And I checked the /var/log/messages, it show the information:
> > >
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational
> > > as
> > > raid disk 5
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational
> > > as
> > > raid disk 4
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational
> > > as
> > > raid disk 3
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational
> > > as
> > > raid disk 2
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational
> > > as
> > > raid disk 1
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational
> > > as
> > > raid disk 0
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with
> > > 6
> > > out of 7 devices, algorithm 2
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled
> > > due
> > > to uncertainty.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: Set
> > > raid456.devices_handle_discard_safely=Y to override.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0
> > > to
> > > 1881145344
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000
> > > KB/sec/disk.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO
> > > bandwidth (but not more than 200000 KB/sec) for recovery.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of
> > > 306176k.
> > > Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0,
> > > 10)
> > > failed: No such file or directory
> > >
> > > Xiao
> > >
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Wednesday, March 25, 2015 2:35:29 PM
> > > > Subject: Re: /dev/md0 can't be created
> > > >
> > > > On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com>
> > > > wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I have encountered so many times, the raid device is created
> > > > > successfully, but the directory
> > > > > /dev/md0 can't be created. It can't reproduce 100%.
> > > > >
> > > > > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > > > > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > > > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > > > > loop2[2] loop1[1] loop0[0]
> > > > > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7]
> > > > > [UUUUUUU]
> > > > > bitmap: 0/1 pages [0KB], 65536KB chunk
> > > > >
> > > > > unused devices: <none>
> > > > > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > > > > ls: cannot access /dev/md0: No such file or directory
> > > > >
> > > > > The underline devices are loop devices which are created with big
> > > > > file.
> > > > >
> > > > > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm -
> > > > > v3.3.2
> > > > > - 21st August 2014)
> > > > > I'll try to reproduce this with upstream kernel and mdadm. But I
> > > > > think it
> > > > > shouldn't be the problem about kernel.
> > > > >
> > > > > What do you think I should check for this? And which tool is
> > > > > responsible for creating the directory? Maybe
> > > > > I can add some log to it to find the reason.
> > > > >
> > > >
> > > > /dev/md0 is created by udev.
> > > > Run
> > > > udevadm monitor
> > > >
> > > > to see the events that udev is processing. When and ADD event for
> > > > "md0" is
> > > > processed, /dev/md0 should get created.
> > > >
> > > > NeilBrown
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
>
>
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-02 0:19 UTC (permalink / raw)
To: Shaohua Li; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150401234055.GA3375744@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 3985 bytes --]
On Wed, 1 Apr 2015 16:40:57 -0700 Shaohua Li <shli@fb.com> wrote:
> > Your code does avoid write-hole-protection for fill-stripe-writes, and this
> > would greatly reduce the number of block that were written multiple times.
> > However I'm not convinced that is correct.
> > A reasonable goal is that if the system crashes while writing to a storage
> > device, then reads should return the old data or not new data, not anything
> > else. A crash in the middle of a full-stripe-write to a degraded array
> > could result in some block in the stripe appearing to contain data that is
> > different to both the old and the new. If you are going to close the whole,
> > I think it should be done properly.
>
> I can do it simpley. But don't think this assumption is true. If you
> write to a disk range and there is failure, there is nothing guarantee
> you can either read old data or new data.
If you write a range of blocks to a normal disk and crash during the write,
each block will contain either the old data or the new data.
If you write a range to a degraded RAID5 and crash during the write, you
cannot make that same guarantee.
I don't know how important this is, but then I don't really know how
important any of this is.
>
> >
> > A combined log would "simply" involve writing every data block and every
> > compute parity block (with index information) to the log device.
> > Replaying the log would collect data blocks and flush out those in a stripe
> > once the parity block(s) for that stripe became available.
> >
> > I think this would actually turn into a fairly simple logging mechanism.
>
> It's not simple at all. It's unlikely we write data and parity
> continuously in disk and in the same time. This will make log checkpoint
> fairly complex.
I don't see any cause for complexity. Let me be more explicit.
I imagine that all data remains in the stripe cache, in memory, until it is
finally written to the RAID5. So the stripe cache will need to be quite a
bit bigger.
Every time we get a block that we want to write, either a new data block or a
a computed parity block, we queue it to the log.
The log works like this:
- take the first (e.g.) 256 blocks in the queue, create a header to describe
them, write the header with FUA, then write all the data blocks. If there
are fewer than 256, just write what we have.
- when the header write completes, all blocks written *previously* are now
safe and we can call bio_end_io on data or unlock the stripe for parity.
- loop back and write some more blocks. If there are no blocks to write,
write a header which describes an empty set of blocks, and wait for more
blocks to appear.
Each stripe_head needs to track (roughly) where the relevant blocks were
written so it can release them when the stripe is written.
I would conceptually divide the log into 32 regions and keep a 32bit number
with each stripe. When a block is assigned to a region in the log, the
relevant bit is set for the stripe, and a per-region counter is incremented.
When a stripe completes its write, the region counters for all the bits are
cleared. The log cannot progress into a region which has a non-zero counter.
We choose the size of transactions so that the first block of each region is
a header block. These contain a magic number, a sequence number, and a
checksum together with the addresses of the data/parity blocks. On restart
we read all 32 of these to find out where the log starts and ends. Then we
replay all the blocks into the stripe cache - discarding any that don't come
with the required parity blocks.
So it is a very simple log which is never read exact on crash recovery. It
commits everything ASAP so that the writeout to the array can be lazy and can
gather related blocks and sort address etc with not impact on filesystem
latency.
Does that make sense?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 23:40 UTC (permalink / raw)
To: NeilBrown; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150402085312.5ea3d518@notabene.brown>
On Thu, Apr 02, 2015 at 08:53:12AM +1100, NeilBrown wrote:
> On Mon, 30 Mar 2015 15:25:17 -0700 Shaohua Li <shli@fb.com> wrote:
>
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
> >
> > 3. For log, write will hit to log disk first, then raid disks, and
> > finally IO completion is reported. An optimal way is to report IO
> > completion just after IO hits to log disk to cut write latency. But in
> > that way, read path need query log disk and increase complexity. And
> > since we don't worry about write latency, we choose a simple soltuion.
> > This will be revisited if there is performance issue.
> >
> > This design isn't intrusive for raid5/6. Actully only very few changes
> > of existing code is required.
> >
> > Log looks like jbd. Stripe IO to raid disks will be written to log disk
> > first in atomic way. Several stripe IO will consist a transaction. If
> > all stripes of a transaction are finished, the tranaction can be
> > checkpoint.
> >
> > Basic logic of raid 5/6 write will be:
> > 1. normal raid5/6 steps for a stripe (fetch data, calculate checksum,
> > and etc). log hooks to ops_run_io.
> > 2. stripe is added to a transaction. Write stripe data to log disk (metadata
> > block, stripe data)
> > 3. write commit block to log disk
> > 4. flush log disk cache.
> > 5. stripe is logged now and normal stripe handling continues
> >
> > Transaction checkpoint process:
> > 1. all stripes of a transaction are finished
> > 2. flush disk cache of all raid disks
> > 3. change log super to reflect new log checkpoint position
> > 4. WRITE_FUA log super
> >
> > metadata, data and commit block IO can run in the meaning time, as
> > checksum will be used to make sure their data is correct (like jbd2).
> > Log IO doesn't wait 5s to start like jbd, instead the IO will start
> > every time a metadata block is full. This can cut some latency.
> >
> > Disk layout:
> >
> > |super|metadata|data|metadata| data ... |commitdata|metadata|data| ... |commitdata|
> > super, metadata, commit will use one block
> >
> > This is an initial version, which works but a lot of stuffes are
> > missing:
> > 1. error handling
> > 2. log recovery and impact to raid resync (don't need resync anymore)
> > 3. utility changes
> >
> > The big question is how we report log disk. In this patch, I simply use
> > a spare disk for testing. We need a new raid disk role for log disk.
> >
> > Signed-off-by: Shaohua Li <shli@fb.com>
>
>
> Hi,
> thanks for the proposal and the patch which makes it nice and concrete...
>
> I should start out by saying that I'm not really sold on the importance of
> the issues you are addressing here.
> The "write hole" is certainly of theoretical significance, but I do wonder
> how much practical significance it has. It can only be a problem if you
> have a system failure and a degraded array at the same time, and both of
> those should be very rare event individually...
> I wonder if anyone has *ever* lost data to the "write hole".
We have tens of thousands of machines. Rare event in terms of huge amount
of machines become a normal event :). It's not a significant issue, but
we don't want to take the risk.
> As for write-ahead caching to reduce latency, most writes from Linux are
> async and so would not benefit from that. If you do have a heavily
> synchronous write load, then that can be fixed in the filesystem.
> e.g. with ext3 and an external log to a low-latency device you can get
> low-latency writes which largely mask the latency issues introduced by
> RAID5.
Maybe I should write more about the caching, I didn't because the patch
is about the write hole issue. Any way, the side effect of caching is
write-hole-protection log doesn't need worry about latency. The main
purpose of caching is to produce full stripe write or reduce
read-modify-write if full stripe write is not possible. The caching can
also reduce hard disk spindle seek, because we can sort when flush data
from caching to raid.
> The fact that I'm "not really sold" doesn't mean I am against them ... maybe
> it is just an encouragement for someone to sell them more :-)
>
> While I understand that keeping the two separate might simplify the
> problem, I'm not at all sure it is a good idea. It would mean that every
> data block were written three times - once to the write-ahead log, once to
> the write-hole-protection log, and once to the RAID5.
yes, it the write-ahead log and write-hole-protection log are combined,
one write can be avoid.
> Your code does avoid write-hole-protection for fill-stripe-writes, and this
> would greatly reduce the number of block that were written multiple times.
> However I'm not convinced that is correct.
> A reasonable goal is that if the system crashes while writing to a storage
> device, then reads should return the old data or not new data, not anything
> else. A crash in the middle of a full-stripe-write to a degraded array
> could result in some block in the stripe appearing to contain data that is
> different to both the old and the new. If you are going to close the whole,
> I think it should be done properly.
I can do it simpley. But don't think this assumption is true. If you
write to a disk range and there is failure, there is nothing guarantee
you can either read old data or new data.
>
> A combined log would "simply" involve writing every data block and every
> compute parity block (with index information) to the log device.
> Replaying the log would collect data blocks and flush out those in a stripe
> once the parity block(s) for that stripe became available.
>
> I think this would actually turn into a fairly simple logging mechanism.
It's not simple at all. It's unlikely we write data and parity
continuously in disk and in the same time. This will make log checkpoint
fairly complex.
Thanks,
Shaohua
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-01 21:53 UTC (permalink / raw)
To: Shaohua Li; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150330222459.GA575371@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 5690 bytes --]
On Mon, 30 Mar 2015 15:25:17 -0700 Shaohua Li <shli@fb.com> wrote:
> This is my attempt to fix raid5/6 write hole issue, it's not for merge
> yet, I post it out for comments. Any comments and suggestions are
> welcome!
>
> Thanks,
> Shaohua
>
> We expect a completed raid5/6 stack with reliability and high
> performance. Currently raid5/6 has 2 issues:
>
> 1. read-modify-write for small size IO. To fix this issue, a cache layer
> above raid5/6 can be used to aggregate write to full stripe write.
> 2. write hole issue. A write log below raid5/6 can fix the issue.
>
> We plan to use a SSD to fix the two issues. Here we just fix the write
> hole issue.
>
> 1. We don't try to fix the issues together. A cache layer will do write
> acceleration. A log layer will fix write hole. The seperation will
> simplify things a lot.
>
> 2. Current assumption is flashcache/bcache will be used as the cache
> layer. If they don't work well, we can fix them or add a simple cache
> layer for raid write aggregation later. We also assume cache layer will
> absorb write, so log doesn't worry about write latency.
>
> 3. For log, write will hit to log disk first, then raid disks, and
> finally IO completion is reported. An optimal way is to report IO
> completion just after IO hits to log disk to cut write latency. But in
> that way, read path need query log disk and increase complexity. And
> since we don't worry about write latency, we choose a simple soltuion.
> This will be revisited if there is performance issue.
>
> This design isn't intrusive for raid5/6. Actully only very few changes
> of existing code is required.
>
> Log looks like jbd. Stripe IO to raid disks will be written to log disk
> first in atomic way. Several stripe IO will consist a transaction. If
> all stripes of a transaction are finished, the tranaction can be
> checkpoint.
>
> Basic logic of raid 5/6 write will be:
> 1. normal raid5/6 steps for a stripe (fetch data, calculate checksum,
> and etc). log hooks to ops_run_io.
> 2. stripe is added to a transaction. Write stripe data to log disk (metadata
> block, stripe data)
> 3. write commit block to log disk
> 4. flush log disk cache.
> 5. stripe is logged now and normal stripe handling continues
>
> Transaction checkpoint process:
> 1. all stripes of a transaction are finished
> 2. flush disk cache of all raid disks
> 3. change log super to reflect new log checkpoint position
> 4. WRITE_FUA log super
>
> metadata, data and commit block IO can run in the meaning time, as
> checksum will be used to make sure their data is correct (like jbd2).
> Log IO doesn't wait 5s to start like jbd, instead the IO will start
> every time a metadata block is full. This can cut some latency.
>
> Disk layout:
>
> |super|metadata|data|metadata| data ... |commitdata|metadata|data| ... |commitdata|
> super, metadata, commit will use one block
>
> This is an initial version, which works but a lot of stuffes are
> missing:
> 1. error handling
> 2. log recovery and impact to raid resync (don't need resync anymore)
> 3. utility changes
>
> The big question is how we report log disk. In this patch, I simply use
> a spare disk for testing. We need a new raid disk role for log disk.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
Hi,
thanks for the proposal and the patch which makes it nice and concrete...
I should start out by saying that I'm not really sold on the importance of
the issues you are addressing here.
The "write hole" is certainly of theoretical significance, but I do wonder
how much practical significance it has. It can only be a problem if you
have a system failure and a degraded array at the same time, and both of
those should be very rare event individually...
I wonder if anyone has *ever* lost data to the "write hole".
As for write-ahead caching to reduce latency, most writes from Linux are
async and so would not benefit from that. If you do have a heavily
synchronous write load, then that can be fixed in the filesystem.
e.g. with ext3 and an external log to a low-latency device you can get
low-latency writes which largely mask the latency issues introduced by
RAID5.
The fact that I'm "not really sold" doesn't mean I am against them ... maybe
it is just an encouragement for someone to sell them more :-)
While I understand that keeping the two separate might simplify the
problem, I'm not at all sure it is a good idea. It would mean that every
data block were written three times - once to the write-ahead log, once to
the write-hole-protection log, and once to the RAID5.
Your code does avoid write-hole-protection for fill-stripe-writes, and this
would greatly reduce the number of block that were written multiple times.
However I'm not convinced that is correct.
A reasonable goal is that if the system crashes while writing to a storage
device, then reads should return the old data or not new data, not anything
else. A crash in the middle of a full-stripe-write to a degraded array
could result in some block in the stripe appearing to contain data that is
different to both the old and the new. If you are going to close the whole,
I think it should be done properly.
A combined log would "simply" involve writing every data block and every
compute parity block (with index information) to the log device.
Replaying the log would collect data blocks and flush out those in a stripe
once the parity block(s) for that stripe became available.
I think this would actually turn into a fairly simple logging mechanism.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Wols Lists @ 2015-04-01 20:18 UTC (permalink / raw)
To: Alireza Haghdoost
Cc: Piergiorgio Sartor, Dan Williams, Shaohua Li, Neil Brown,
linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAB-428kMJ7g9RL+P4ZYsMRJVdJc7jeU-wmA1zJGJ4UyKWwSeJA@mail.gmail.com>
On 01/04/15 21:04, Alireza Haghdoost wrote:
> On Wed, Apr 1, 2015 at 2:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>>> Now, how can be assured, in that case, that the "cache"
>>>>> device is safe after the power is restored?
>>> You do sync write-ahead logging on the Flash cache. If it return
>>> successful, you do fire the writes to the RAID. If system crash/fails
>>> during the RAID writes (Write-hole), you just recover data by scanning
>>> write-ahead log in the flash cache and replay the logs into the RAID
>>> drives.
>>>
>> Just to throw something nasty into the mix, I'm not sure whether it's
>> SSDs or SD-cards, but there certainly *was* a spate of corrupted
>> *controllers*.
>>
>> In other words, a power failure would RELIABLY TRASH the device, if it
>> happened at the wrong moment. Hopefully that's been fixed ...
>>
>
> That is certainly true. As Dan mentioned, the cache device it-self
> should be safe against power failure. I agree this is not the case for
> all SSD cards in the market but might be the case for Facebook. I hate
> to say this but It seems these efforts are useful dependent to what
> kind of hardware is deployed for cache device.
>
It would be nice, but probably not possible, to have some form of
black-list of "these devices are unsafe/dangerous". Along the lines of
"mdadm --probe /dev/sda" or whatever, that gets the device type, checks
it, and says "this SSD can be destroyed by a power failure" or "this is
a cheap disk with the timeout problem" or something. But even if someone
did it, the database would probably bit-rot fairly quickly :-(
Cheers,
Wol
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Jens Axboe @ 2015-04-01 20:17 UTC (permalink / raw)
To: Wols Lists, Alireza Haghdoost, Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <551C4DAA.4010701@youngman.org.uk>
On 04/01/2015 01:57 PM, Wols Lists wrote:
> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>> Now, how can be assured, in that case, that the "cache"
>>>> device is safe after the power is restored?
>> You do sync write-ahead logging on the Flash cache. If it return
>> successful, you do fire the writes to the RAID. If system crash/fails
>> during the RAID writes (Write-hole), you just recover data by scanning
>> write-ahead log in the flash cache and replay the logs into the RAID
>> drives.
>>
> Just to throw something nasty into the mix, I'm not sure whether it's
> SSDs or SD-cards, but there certainly *was* a spate of corrupted
> *controllers*.
>
> In other words, a power failure would RELIABLY TRASH the device, if it
> happened at the wrong moment. Hopefully that's been fixed ...
You can't protect against shitty devices. If you care about power fail
events, then you use hw that is specifically tested and vetted for that.
And they do exist. They might just not be the cheapest you can find on
newegg or similar places.
This potential problem isn't specific to what Shaohua is proposing, nor
is it a show stopper for that.
--
Jens Axboe
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Jiang, Dave @ 2015-04-01 20:07 UTC (permalink / raw)
To: Williams, Dan J
Cc: linux-raid@vger.kernel.org, neilb@suse.de, Kernel-team@fb.com,
piergiorgio.sartor@nexgo.de, shli@fb.com, songliubraving@fb.com
In-Reply-To: <CAPcyv4jQZ6jeQpFU+cZ-AXzfQVXQCdK1muxNO35pCdavrggh4Q@mail.gmail.com>
On Wed, 2015-04-01 at 18:46 +0000, Williams, Dan J wrote:
> On Wed, Apr 1, 2015 at 11:36 AM, Piergiorgio Sartor
> <piergiorgio.sartor@nexgo.de> wrote:
> > On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> >> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> >> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> >> > yet, I post it out for comments. Any comments and suggestions are
> >> > welcome!
> >> >
> >> > Thanks,
> >> > Shaohua
> >> >
> >> > We expect a completed raid5/6 stack with reliability and high
> >> > performance. Currently raid5/6 has 2 issues:
> >> >
> >> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> >> > above raid5/6 can be used to aggregate write to full stripe write.
> >> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >> >
> >> > We plan to use a SSD to fix the two issues. Here we just fix the write
> >> > hole issue.
> >> >
> >> > 1. We don't try to fix the issues together. A cache layer will do write
> >> > acceleration. A log layer will fix write hole. The seperation will
> >> > simplify things a lot.
> >> >
> >> > 2. Current assumption is flashcache/bcache will be used as the cache
> >> > layer. If they don't work well, we can fix them or add a simple cache
> >> > layer for raid write aggregation later. We also assume cache layer will
> >> > absorb write, so log doesn't worry about write latency.
> >>
> >> It seems neither bcache nor dm-cache are tackling the write-buffering
> >> problem head on... they still seem to be concerned with some amount of
> >> read caching which I can see as useful for file servers and
> >> workstations, but not necessarily scale out storage.
> >>
> >> I'll try to set aside time to take a look at the patch this week.
> >
> > There is one thing I do not really get.
> >
> > The target is to avoid the "write hole", which happens,
> > for example, when there is a sudden power failure.
> >
> > Now, how can be assured, in that case, that the "cache"
> > device is safe after the power is restored?
>
> If you lose the cache the data-loss damage is greater, but this has
> always been the case with hardware-raid adapters.
>
> > Doesn't this solution just shifts the problem from
> > the array to a different device (SSD, for example)?
> >
> > Speaking of SSD, these are quite "power failure"
> > sensitive, it seems...
>
> Simple, if a cache-device is not itself power-failure safe then it
> should not be used for power-failure protection.
I think this would be a good application for some of the newer
technology coming out such as NVDIMM and persistent memory.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Alireza Haghdoost @ 2015-04-01 20:04 UTC (permalink / raw)
To: Wols Lists
Cc: Piergiorgio Sartor, Dan Williams, Shaohua Li, Neil Brown,
linux-raid, Song Liu, Kernel-team
In-Reply-To: <551C4DAA.4010701@youngman.org.uk>
On Wed, Apr 1, 2015 at 2:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>> Now, how can be assured, in that case, that the "cache"
>>> > device is safe after the power is restored?
>> You do sync write-ahead logging on the Flash cache. If it return
>> successful, you do fire the writes to the RAID. If system crash/fails
>> during the RAID writes (Write-hole), you just recover data by scanning
>> write-ahead log in the flash cache and replay the logs into the RAID
>> drives.
>>
> Just to throw something nasty into the mix, I'm not sure whether it's
> SSDs or SD-cards, but there certainly *was* a spate of corrupted
> *controllers*.
>
> In other words, a power failure would RELIABLY TRASH the device, if it
> happened at the wrong moment. Hopefully that's been fixed ...
>
That is certainly true. As Dan mentioned, the cache device it-self
should be safe against power failure. I agree this is not the case for
all SSD cards in the market but might be the case for Facebook. I hate
to say this but It seems these efforts are useful dependent to what
kind of hardware is deployed for cache device.
--Alireza
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Wols Lists @ 2015-04-01 19:57 UTC (permalink / raw)
To: Alireza Haghdoost, Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <CAB-428nVCN=5au0jONamXORNZ656UZ1Ecgrk-paArruX9tpHJQ@mail.gmail.com>
On 01/04/15 19:46, Alireza Haghdoost wrote:
>> Now, how can be assured, in that case, that the "cache"
>> > device is safe after the power is restored?
> You do sync write-ahead logging on the Flash cache. If it return
> successful, you do fire the writes to the RAID. If system crash/fails
> during the RAID writes (Write-hole), you just recover data by scanning
> write-ahead log in the flash cache and replay the logs into the RAID
> drives.
>
Just to throw something nasty into the mix, I'm not sure whether it's
SSDs or SD-cards, but there certainly *was* a spate of corrupted
*controllers*.
In other words, a power failure would RELIABLY TRASH the device, if it
happened at the wrong moment. Hopefully that's been fixed ...
Cheers.
Wol
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-04-01 19:07 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
In-Reply-To: <CAK2H+ec5Lo0=QaEXa=RrbXCzG5m6nRc7BynSPYEqvCc5Lb4z3g@mail.gmail.com>
On Tue, Mar 31, 2015 at 3:08 PM, Mark Knecht <markknecht@gmail.com> wrote:
> On Tue, Mar 31, 2015 at 1:30 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> On Tue, 31 Mar 2015 08:00:27 -0700
>> Mark Knecht <markknecht@gmail.com> wrote:
>>
>>> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
>>> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
>>> [====>................] resync = 22.8% (110608304/484088160)
>>> finish=78.5min speed=79206K/sec
>>
>> Looks like you are running this without a write intent bitmap, why? It would
>> at least save you from a full resync each time.
>>
>> --
>> With respect,
>> Roman
>
> Honestly, I don't know. The machine was the first I ever built using
> RAID for the system drive and I probably didn't find out about it, or
> was just trying to keep it simple? Not at all sure.
>
> I'll check it out.
>
> Thanks,
> Mark
At this point, having put everything back together, I cannot find any
specific clues about what caused the problem. Possibly the system
might be showing its age. Maybe it's a corner condition in the 3.18
series kernel vs the 3.14 I was running before. No idea. The mdadm
RAID seems to have done it's job wonderfully. At least at the Linux
level there's no evidence at this time that anything was lost. Within
my Windows VMs that were running on the mdadm RAID6 at the time of the
Magic Sys Key reboot there seem to be minimal issues. One Windows
program (Schwab's StreetSmart Edge) lost some of my trading
environment configuration and I needed to recall only that file off
the web server to get the platform configured correctly again.
TradeStation was running in a second VM and showed no issues. Both
trading platforms were live at the time of the lockup so I came out of
this really well, and as I have good backups nothing would have been
lost anyway.
As per Roman's question/suggestion I've added the write intent bitmaps
to all three RAIDs. Nominally only md3 is mounted most of the time
anyway, but it's there to help if this problem continues.
If anyone sees anything else in the data I posted yesterday let me
know but everything mdadm specific seems good at this point.
Cheers,
Mark
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Alireza Haghdoost @ 2015-04-01 18:46 UTC (permalink / raw)
To: Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <20150401183630.GA3103@lazy.lzy>
On Wed, Apr 1, 2015 at 1:36 PM, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
>> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
>> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
>> > yet, I post it out for comments. Any comments and suggestions are
>> > welcome!
>> >
>> > Thanks,
>> > Shaohua
>> >
>> > We expect a completed raid5/6 stack with reliability and high
>> > performance. Currently raid5/6 has 2 issues:
>> >
>> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
>> > above raid5/6 can be used to aggregate write to full stripe write.
>> > 2. write hole issue. A write log below raid5/6 can fix the issue.
>> >
>> > We plan to use a SSD to fix the two issues. Here we just fix the write
>> > hole issue.
>> >
>> > 1. We don't try to fix the issues together. A cache layer will do write
>> > acceleration. A log layer will fix write hole. The seperation will
>> > simplify things a lot.
>> >
>> > 2. Current assumption is flashcache/bcache will be used as the cache
>> > layer. If they don't work well, we can fix them or add a simple cache
>> > layer for raid write aggregation later. We also assume cache layer will
>> > absorb write, so log doesn't worry about write latency.
>>
>> It seems neither bcache nor dm-cache are tackling the write-buffering
>> problem head on... they still seem to be concerned with some amount of
>> read caching which I can see as useful for file servers and
>> workstations, but not necessarily scale out storage.
>>
>> I'll try to set aside time to take a look at the patch this week.
>
> There is one thing I do not really get.
>
> The target is to avoid the "write hole", which happens,
> for example, when there is a sudden power failure.
>
> Now, how can be assured, in that case, that the "cache"
> device is safe after the power is restored?
You do sync write-ahead logging on the Flash cache. If it return
successful, you do fire the writes to the RAID. If system crash/fails
during the RAID writes (Write-hole), you just recover data by scanning
write-ahead log in the flash cache and replay the logs into the RAID
drives.
> Doesn't this solution just shifts the problem from
> the array to a different device (SSD, for example)?
I don't see such a shift. Enterprise hardware RAIDs also used similar
technique to fix the write-hole issue.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Dan Williams @ 2015-04-01 18:46 UTC (permalink / raw)
To: Piergiorgio Sartor
Cc: Shaohua Li, Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401183630.GA3103@lazy.lzy>
On Wed, Apr 1, 2015 at 11:36 AM, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
>> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
>> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
>> > yet, I post it out for comments. Any comments and suggestions are
>> > welcome!
>> >
>> > Thanks,
>> > Shaohua
>> >
>> > We expect a completed raid5/6 stack with reliability and high
>> > performance. Currently raid5/6 has 2 issues:
>> >
>> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
>> > above raid5/6 can be used to aggregate write to full stripe write.
>> > 2. write hole issue. A write log below raid5/6 can fix the issue.
>> >
>> > We plan to use a SSD to fix the two issues. Here we just fix the write
>> > hole issue.
>> >
>> > 1. We don't try to fix the issues together. A cache layer will do write
>> > acceleration. A log layer will fix write hole. The seperation will
>> > simplify things a lot.
>> >
>> > 2. Current assumption is flashcache/bcache will be used as the cache
>> > layer. If they don't work well, we can fix them or add a simple cache
>> > layer for raid write aggregation later. We also assume cache layer will
>> > absorb write, so log doesn't worry about write latency.
>>
>> It seems neither bcache nor dm-cache are tackling the write-buffering
>> problem head on... they still seem to be concerned with some amount of
>> read caching which I can see as useful for file servers and
>> workstations, but not necessarily scale out storage.
>>
>> I'll try to set aside time to take a look at the patch this week.
>
> There is one thing I do not really get.
>
> The target is to avoid the "write hole", which happens,
> for example, when there is a sudden power failure.
>
> Now, how can be assured, in that case, that the "cache"
> device is safe after the power is restored?
If you lose the cache the data-loss damage is greater, but this has
always been the case with hardware-raid adapters.
> Doesn't this solution just shifts the problem from
> the array to a different device (SSD, for example)?
>
> Speaking of SSD, these are quite "power failure"
> sensitive, it seems...
Simple, if a cache-device is not itself power-failure safe then it
should not be used for power-failure protection.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Piergiorgio Sartor @ 2015-04-01 18:36 UTC (permalink / raw)
To: Dan Williams; +Cc: Shaohua Li, Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAPcyv4jqb+OphP-1sy0dqMYG-uNK-CwtNDmOM4YeQU=MXH72Hw@mail.gmail.com>
On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
>
> It seems neither bcache nor dm-cache are tackling the write-buffering
> problem head on... they still seem to be concerned with some amount of
> read caching which I can see as useful for file servers and
> workstations, but not necessarily scale out storage.
>
> I'll try to set aside time to take a look at the patch this week.
There is one thing I do not really get.
The target is to avoid the "write hole", which happens,
for example, when there is a sudden power failure.
Now, how can be assured, in that case, that the "cache"
device is safe after the power is restored?
Doesn't this solution just shifts the problem from
the array to a different device (SSD, for example)?
Speaking of SSD, these are quite "power failure"
sensitive, it seems...
Thanks,
bye,
pg
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* write barrier in RAID5
From: Eric Mei @ 2015-04-01 18:24 UTC (permalink / raw)
To: linux-raid
I can see how FUA is handled, but REQ_FLUSH seems problematic. My
understanding of REQ_FLUSH semantics: all requests before the flush
request must *complete* before the flush request complete. For IO remap
layer like MD, we need to guarantee all previous requests are
*submitted* to component devices before submitting the flush request. Is
that correct?
If so why I don't see the code which make sure the pending write
requests in stripe cache are processed & submitted before
submit_flushes() get called? Thanks
Eric
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 17:14 UTC (permalink / raw)
To: NeilBrown; +Cc: Dan Williams, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401170256.5efebaae@notabene.brown>
On Wed, Apr 01, 2015 at 05:02:56PM +1100, NeilBrown wrote:
> On Tue, 31 Mar 2015 22:53:21 -0700 Shaohua Li <shli@fb.com> wrote:
>
> > On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> > > On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > > > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > > > yet, I post it out for comments. Any comments and suggestions are
> > > > welcome!
> > > >
> > > > Thanks,
> > > > Shaohua
> > > >
> > > > We expect a completed raid5/6 stack with reliability and high
> > > > performance. Currently raid5/6 has 2 issues:
> > > >
> > > > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > > > above raid5/6 can be used to aggregate write to full stripe write.
> > > > 2. write hole issue. A write log below raid5/6 can fix the issue.
> > > >
> > > > We plan to use a SSD to fix the two issues. Here we just fix the write
> > > > hole issue.
> > > >
> > > > 1. We don't try to fix the issues together. A cache layer will do write
> > > > acceleration. A log layer will fix write hole. The seperation will
> > > > simplify things a lot.
> > > >
> > > > 2. Current assumption is flashcache/bcache will be used as the cache
> > > > layer. If they don't work well, we can fix them or add a simple cache
> > > > layer for raid write aggregation later. We also assume cache layer will
> > > > absorb write, so log doesn't worry about write latency.
> > >
> > > It seems neither bcache nor dm-cache are tackling the write-buffering
> > > problem head on... they still seem to be concerned with some amount of
> > > read caching which I can see as useful for file servers and
> > > workstations, but not necessarily scale out storage.
> > >
> > > I'll try to set aside time to take a look at the patch this week.
> >
> > Thanks! The cache layer is definitely what I'll focus on next. bcache
> > supports writeback, I guess we can add an option to skip read data from
> > backing disks for read caching if it's possible. Another option is
> > writting a simple caching just for raid 5/6 write aggregation. We can
> > append all data to a log, and maintain an index in memory. At raid
> > shutdown, we can flush all data to raid disks, the index doesn't need
> > presistent in disk, which makes the caching fairly simple.
>
> Surely if the index doesn't need to persist in disk, then the data doesn't
> either, as without the index you cannot find the data...
I mean not just pure data. We can store tuple (disk offset, length,
data) to disk. index will be used to speed up search. If there is a
crash, we can rebuild the index using the tuple.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH] Fix bitmap offset calculations
From: Goldwyn Rodrigues @ 2015-04-01 14:14 UTC (permalink / raw)
To: Alireza Haghdoost, NeilBrown; +Cc: Linux RAID
In-Reply-To: <CAB-428mbVFcEmT_OryyGxEAhr56Uy0CODrEVEoPUPk34C45_Uw@mail.gmail.com>
Hi Alireza,
On 04/01/2015 08:58 AM, Alireza Haghdoost wrote:
> Does it means if some one use write-intent bitmap without this patch,
> he may end-up with some unsync stripes after system crash or power
> failure and RAID resynchronization ? It seems the bitmaps does not
> record correct address of unsynced blocks due to this bug.
>
> Would you please verify this.
This is for clustered md effort only (which is in Neil's md/for-next
tree). The regular md is unaffected.
In a clustered environment, different nodes use different bitmaps. While
it worked for bitmaps smaller than a page (which is again a
co-incidence), it did not work well for bitmaps which spanned multiple
pages. Each node in the cluster has different start offsets, and the
earlier calculation was incorrect because of conversion from bits to
bytes was inverted.
If you were able to assemble on different nodes, you should be fine with
respect to synchronization of unsynced blocks after a failure.
HTH,
>
> On Tue, Mar 24, 2015 at 9:15 PM, NeilBrown <neilb@suse.de> wrote:
>> On Tue, 24 Mar 2015 11:29:05 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>>
>>> The calculations of bitmap offset is incorrect with respect to bits to bytes
>>> conversion.
>>>
>>> Also, remove an irrelevant duplicate message.
>>>
>>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>> ---
>>> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
>>> index ac79fef..e98db04 100644
>>> --- a/drivers/md/bitmap.c
>>> +++ b/drivers/md/bitmap.c
>>> @@ -575,7 +575,9 @@ re_read:
>>>
>>> sector_div(bm_blocks,
>>> bitmap->mddev->bitmap_info.chunksize >> 9);
>>> - bm_blocks = bm_blocks << 3;
>>> + /* bits to bytes */
>>> + bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
>>> + /* to 4k blocks */
>>> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>>> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
>>> pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>>> @@ -672,9 +674,6 @@ out:
>>> goto out_no_sb;
>>> }
>>> bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
>>> - pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>>> - bitmap->cluster_slot,
>>> - (unsigned long long)bitmap->mddev->bitmap_info.offset);
>>> goto re_read;
>>> }
>>>
>>
>> Applied, thanks.
>>
>> NeilBrown
--
Goldwyn
^ permalink raw reply
* Re: [PATCH] Fix bitmap offset calculations
From: Alireza Haghdoost @ 2015-04-01 13:58 UTC (permalink / raw)
To: NeilBrown; +Cc: Goldwyn Rodrigues, Linux RAID
In-Reply-To: <20150325131502.1a947967@notabene.brown>
Does it means if some one use write-intent bitmap without this patch,
he may end-up with some unsync stripes after system crash or power
failure and RAID resynchronization ? It seems the bitmaps does not
record correct address of unsynced blocks due to this bug.
Would you please verify this.
On Tue, Mar 24, 2015 at 9:15 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 24 Mar 2015 11:29:05 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
>> The calculations of bitmap offset is incorrect with respect to bits to bytes
>> conversion.
>>
>> Also, remove an irrelevant duplicate message.
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
>> index ac79fef..e98db04 100644
>> --- a/drivers/md/bitmap.c
>> +++ b/drivers/md/bitmap.c
>> @@ -575,7 +575,9 @@ re_read:
>>
>> sector_div(bm_blocks,
>> bitmap->mddev->bitmap_info.chunksize >> 9);
>> - bm_blocks = bm_blocks << 3;
>> + /* bits to bytes */
>> + bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
>> + /* to 4k blocks */
>> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
>> pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>> @@ -672,9 +674,6 @@ out:
>> goto out_no_sb;
>> }
>> bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
>> - pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>> - bitmap->cluster_slot,
>> - (unsigned long long)bitmap->mddev->bitmap_info.offset);
>> goto re_read;
>> }
>>
>
> Applied, thanks.
>
> NeilBrown
^ permalink raw reply
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: NeilBrown @ 2015-04-01 11:07 UTC (permalink / raw)
To: Gu Zheng; +Cc: Simon Kirby, linux-raid
In-Reply-To: <551BA0C3.7090004@cn.fujitsu.com>
[-- Attachment #1: Type: text/plain, Size: 3298 bytes --]
On Wed, 1 Apr 2015 15:39:47 +0800 Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
> Hi Neil,
> One question about md io stats account:
> I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
> md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
> need it? or I missed something?
No, I don't think you are missing anything. md doesn't track 'ticks' - and
no-one has ever complained.
NeilBrown
>
> Regards,
> Gu
>
> On 03/26/2015 01:33 AM, Simon Kirby wrote:
>
> > Hello!
> >
> > Please note the following.
> >
> > Thanks,
> >
> > Simon-
> >
> > ----- Forwarded message from NeilBrown <neilb@suse.de> -----
> >
> > Date: Wed, 25 Mar 2015 11:46:18 +1100
> > From: NeilBrown <neilb@suse.de>
> > To: Simon Kirby <sim@hostway.ca>
> > Cc: linux-raid@vger.kernel.org
> > Subject: Re: MD request counter broken on 4.0-rc5 RAID1
> > X-Mailer: Claws Mail 3.10.1-162-g4d0ed6 (GTK+ 2.24.25; x86_64-suse-linux-gnu)
> >
> > On Tue, 24 Mar 2015 17:04:03 -0700 Simon Kirby <sim@hostway.ca> wrote:
> >
> >> Hello :)
> >>
> >> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
> >> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
> >> the other two being write-mostly:
> >>
> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
> >> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
> >> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
> >>
> >> This was working on 3.18. Shall I bisect, or is it obvious?
> >
> > Almost certainly 18c0b223cf9901727ef3b02da6711ac930b4e5d4 in 3.19.
> >
> > It adds 'part_inc_in_flight' and never decrements it.
> >
> > Complain to the author.
> >
> > Thanks,
> > NeilBrown
> >
> >>
> >> It seems to just be leaking as the counters were going up gradually while
> >> it was rebuilding.
> >>
> >> Personalities : [linear] [raid0] [raid1] [raid10]
> >> md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
> >> 530048 blocks [3/3] [UUU]
> >>
> >> md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
> >> 153613440 blocks [3/3] [UUU]
> >>
> >> md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
> >> 2104384 blocks [3/3] [UUU]
> >>
> >> unused devices: <none>
> >>
> >> Simo-
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > ----- End forwarded message -----
> > .
> >
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Please let me know if you need to print color box, display box and labels
From: Jinghao Printing - CHINA @ 2015-04-01 8:48 UTC (permalink / raw)
Hi, this is David Wu from Shanghai, China.
Please let me know if you need color box, display box, corrugated box,
label, hang tag etc.
I will send you the website.
Best regards,
David Wu
^ permalink raw reply
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: Gu Zheng @ 2015-04-01 7:39 UTC (permalink / raw)
To: neilb; +Cc: Simon Kirby, linux-raid
In-Reply-To: <20150325173346.GF8519@hostway.ca>
Hi Neil,
One question about md io stats account:
I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
need it? or I missed something?
Regards,
Gu
On 03/26/2015 01:33 AM, Simon Kirby wrote:
> Hello!
>
> Please note the following.
>
> Thanks,
>
> Simon-
>
> ----- Forwarded message from NeilBrown <neilb@suse.de> -----
>
> Date: Wed, 25 Mar 2015 11:46:18 +1100
> From: NeilBrown <neilb@suse.de>
> To: Simon Kirby <sim@hostway.ca>
> Cc: linux-raid@vger.kernel.org
> Subject: Re: MD request counter broken on 4.0-rc5 RAID1
> X-Mailer: Claws Mail 3.10.1-162-g4d0ed6 (GTK+ 2.24.25; x86_64-suse-linux-gnu)
>
> On Tue, 24 Mar 2015 17:04:03 -0700 Simon Kirby <sim@hostway.ca> wrote:
>
>> Hello :)
>>
>> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
>> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
>> the other two being write-mostly:
>>
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
>> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
>> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
>>
>> This was working on 3.18. Shall I bisect, or is it obvious?
>
> Almost certainly 18c0b223cf9901727ef3b02da6711ac930b4e5d4 in 3.19.
>
> It adds 'part_inc_in_flight' and never decrements it.
>
> Complain to the author.
>
> Thanks,
> NeilBrown
>
>>
>> It seems to just be leaking as the counters were going up gradually while
>> it was rebuilding.
>>
>> Personalities : [linear] [raid0] [raid1] [raid10]
>> md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
>> 530048 blocks [3/3] [UUU]
>>
>> md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
>> 153613440 blocks [3/3] [UUU]
>>
>> md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
>> 2104384 blocks [3/3] [UUU]
>>
>> unused devices: <none>
>>
>> Simo-
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ----- End forwarded message -----
> .
>
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-01 6:02 UTC (permalink / raw)
To: Shaohua Li; +Cc: Dan Williams, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401055309.GA726662@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]
On Tue, 31 Mar 2015 22:53:21 -0700 Shaohua Li <shli@fb.com> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> > On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > > yet, I post it out for comments. Any comments and suggestions are
> > > welcome!
> > >
> > > Thanks,
> > > Shaohua
> > >
> > > We expect a completed raid5/6 stack with reliability and high
> > > performance. Currently raid5/6 has 2 issues:
> > >
> > > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > > above raid5/6 can be used to aggregate write to full stripe write.
> > > 2. write hole issue. A write log below raid5/6 can fix the issue.
> > >
> > > We plan to use a SSD to fix the two issues. Here we just fix the write
> > > hole issue.
> > >
> > > 1. We don't try to fix the issues together. A cache layer will do write
> > > acceleration. A log layer will fix write hole. The seperation will
> > > simplify things a lot.
> > >
> > > 2. Current assumption is flashcache/bcache will be used as the cache
> > > layer. If they don't work well, we can fix them or add a simple cache
> > > layer for raid write aggregation later. We also assume cache layer will
> > > absorb write, so log doesn't worry about write latency.
> >
> > It seems neither bcache nor dm-cache are tackling the write-buffering
> > problem head on... they still seem to be concerned with some amount of
> > read caching which I can see as useful for file servers and
> > workstations, but not necessarily scale out storage.
> >
> > I'll try to set aside time to take a look at the patch this week.
>
> Thanks! The cache layer is definitely what I'll focus on next. bcache
> supports writeback, I guess we can add an option to skip read data from
> backing disks for read caching if it's possible. Another option is
> writting a simple caching just for raid 5/6 write aggregation. We can
> append all data to a log, and maintain an index in memory. At raid
> shutdown, we can flush all data to raid disks, the index doesn't need
> presistent in disk, which makes the caching fairly simple.
Surely if the index doesn't need to persist in disk, then the data doesn't
either, as without the index you cannot find the data...
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox