* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Martin K. Petersen @ 2015-06-25 1:05 UTC (permalink / raw)
To: Jes Sorensen; +Cc: NeilBrown, linux-raid
In-Reply-To: <wrfj616dtlmb.fsf@redhat.com>
>>>>> "Jes" == Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> Or is this some non-SATA/SCSI SSD that has a 'make_request_fn'
>> driver? I think I came across one of those before (NVMe). In that
>> case - the driver needs to be fixed.
Jes> Nope the block layer doesn't merge the requests at this point.
Then something broke it. Discard merging used to work.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Martin K. Petersen @ 2015-06-25 1:03 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Roman Mamedov, neilb, linux-raid
In-Reply-To: <wrfj1th1tlis.fsf@redhat.com>
>>>>> "Jes" == Jes Sorensen <Jes.Sorensen@redhat.com> writes:
Jes> We already disable discard per default on raid456 in a similar
Jes> manner because some of them unreliably reports discard_zeroes_data
Jes> when they in reality don't.
Jes> If there was a way to reliably detect these things it would be
Jes> fine, unfortunately there isn't.
We whitelist drives that do the right thing now.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: 4.0.5: WARNING: CPU: 3 PID: 249 at /home/kernel/COD/linux/mm/backing-dev.c:372 bdi_unregister+0x36/0x40()
From: NeilBrown @ 2015-06-24 23:29 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-kernel, linux-raid
In-Reply-To: <6d6f7bd7a906a79bacf8e622d348efcf@admin.virtall.com>
On Mon, 15 Jun 2015 21:39:03 +0900 Tomasz Chmielewski <tch@virtall.com>
wrote:
> Got this after stopping a RAID-1 array:
>
> [ 626.694737] md: md3 still in use.
> [ 626.694946] md: delaying resync of md3 until md2 has finished (they
> share one or more physical units)
> [ 628.256210] md3: detected capacity change from 3888444473344 to 0
> [ 628.256372] md: md3 stopped.
> [ 628.256383] md: unbind<sdb4>
> [ 628.274852] md: export_rdev(sdb4)
> [ 628.274909] md: unbind<sda4>
> [ 628.282856] md: export_rdev(sda4)
> [ 628.283246] ------------[ cut here ]------------
> [ 628.283258] WARNING: CPU: 3 PID: 249 at
> /home/kernel/COD/linux/mm/backing-dev.c:372 bdi_unregister+0x36/0x40()
Fixed in 4.0.6.
NeilBrown
> [ 628.283261] Modules linked in: intel_rapl iosf_mbi
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64
> lrw gf128mul glue_helper ablk_helper eeepc_wmi asus_wmi ppdev
> sparse_keymap parport_pc cryptd video shpchp 8250_fintek lpc_ich lp
> tpm_infineon wmi mac_hid parport serio_raw btrfs pata_acpi raid10
> raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> raid6_pq raid1 ahci r8169 libahci raid0 mii pata_via multipath linear
> [ 628.283320] CPU: 3 PID: 249 Comm: kworker/3:1 Not tainted
> 4.0.5-040005-generic #201506061639
> [ 628.283322] Hardware name: System manufacturer System Product
> Name/P8H67-M PRO, BIOS 3904 04/27/2013
> [ 628.283328] Workqueue: md_misc mddev_delayed_delete
> [ 628.283341] 0000000000000174 ffff88040867bca8 ffffffff817e4a5d
> 0000000000000007
> [ 628.283347] 0000000000000000 ffff88040867bce8 ffffffff81079227
> ffffffff81adc895
> [ 628.283352] ffff880409197c00 0000000000000000 0000000000000000
> ffff88041f2da900
> [ 628.283365] Call Trace:
> [ 628.283374] [<ffffffff817e4a5d>] dump_stack+0x45/0x57
> [ 628.283381] [<ffffffff81079227>] warn_slowpath_common+0x97/0xe0
> [ 628.283386] [<ffffffff8107928a>] warn_slowpath_null+0x1a/0x20
> [ 628.283390] [<ffffffff811a3f16>] bdi_unregister+0x36/0x40
> [ 628.283397] [<ffffffff813955f8>] del_gendisk+0x108/0x260
> [ 628.283402] [<ffffffff81648eec>] md_free+0x4c/0x70
> [ 628.283408] [<ffffffff813b6b62>] kobject_cleanup+0x82/0x1c0
> [ 628.283413] [<ffffffff813b69f0>] kobject_put+0x30/0x70
> [ 628.283417] [<ffffffff81649c44>] mddev_delayed_delete+0x34/0x40
> [ 628.283422] [<ffffffff81092204>] process_one_work+0x144/0x490
> [ 628.283426] [<ffffffff81092c6e>] worker_thread+0x11e/0x450
> [ 628.283431] [<ffffffff81092b50>] ? create_worker+0x1f0/0x1f0
> [ 628.283436] [<ffffffff81098999>] kthread+0xc9/0xe0
> [ 628.283442] [<ffffffff810988d0>] ? flush_kthread_worker+0x90/0x90
> [ 628.283448] [<ffffffff817f1118>] ret_from_fork+0x58/0x90
> [ 628.283453] [<ffffffff810988d0>] ? flush_kthread_worker+0x90/0x90
> [ 628.283457] ---[ end trace 2c187f15cc11aca4 ]---
>
>
>
^ permalink raw reply
* Re: mdadm/Software RAID problems - another update
From: NeilBrown @ 2015-06-24 23:26 UTC (permalink / raw)
To: Gareth Evans; +Cc: linux-raid
In-Reply-To: <1434896517.3660124.303759113.0AE76093@webmail.messagingengine.com>
On Sun, 21 Jun 2015 15:21:57 +0100 Gareth Evans <donotspam@fastmail.fm>
wrote:
> After a number of tests with VirtualBox this weekend, it seems things work as they should with Fedora 22 but openSUSE 13.2, Debian 8.1 and Ubuntu 14.04.2 all appear to hang on boot when one vdisk is removed and another is attached, even if formatted with a "Linux RAID autodetect"-type partition. CentOS 7 doesn't hang but doesn't have mdadm in its emergency console (I gave up with CentOS after that as not sure how to add commands, if that's possible...) and neither Fedora nor CentOS emergency consoles seem to include fdisk, which isn't too helpful.
>
I cannot comment on the other distros, but I believe that openSUSE
13.2, with all updates installed, handles boot on degraded md arrays
correctly. As I *am* the mdadm maintainer for that distro, I can put
effort there.
If you have specifics of a configuration on openSUSE-13.2 with latest
updates for mdadm and dracut installed, where it doesn't boot correctly,
then I would definitely like to know about them, either here or at
bugzilla.opensuse.org - whichever suits you.
Thanks,
NeilBrown
^ permalink raw reply
* Re: mdadm/Software RAID problems
From: NeilBrown @ 2015-06-24 23:21 UTC (permalink / raw)
To: Gareth Evans; +Cc: linux-raid
In-Reply-To: <1434739736.4161728.300185905.45F829A7@webmail.messagingengine.com>
On Fri, 19 Jun 2015 19:48:56 +0100 Gareth Evans <donotspam@fastmail.fm>
wrote:
> Having failed in my first attempt at setting up encrypted software RAID on Ubuntu 14.04.2, I discovered there seem to be many and various problems with software RAID on Ubuntu and Debian at least.
>
> For example, I have found via testing on KVM and VirtualBox (with and on Ubuntu 14.04.2 and with Debian 8.1) that:
>
> 1. Encrypted RAID doesn't boot after installation (no volume groups found, with unencrypted raided /boot)
> 2. Degraded unencrypted RAID doesn't boot (loops on "Intermittently starting RAID arrays..." with a few lines of related output)
> 3. /etc/initramfs-tools/conf.d/mdadm doesn't seem to exist as the Ubuntu Server Guide suggests it should
> 4. adding bootdegraded=true to grub options at boot seems to have no effect.
>
> The second point above seems particularly concerning because it would seem (unencrypted) software RAID (which works well enough while it's working) fails precisely when it is needed, which rather defeats the point.
>
> I have found lots of bug reports, some still "new" and of "undecided" importance after several years, for various problems, but not many solutions.
>
> I'm not sure if the issues lie with upstream or distro-related mdadm or other packages, or perhaps kernel issues?
Most of the issues you identify lie on the overlap between mdadm and
the boot loader - mkinitramfs or dracut or whatever.
It is distro problem to make these work together. As upstream md and
mdadm maintainer I'm happy to work with distro maintainer to resolve
issues. I can tell you how I think it should work and discuss how it
can work, but I am not in a position to diagnose exactly what is
happening and figure out what to fix.
You really need to follow this up with Debian, though unfortunately I
think the current Debian mdadm maintainer is not able to put in as much
time as he would like.
NeilBrown
>
> Just FYI, I did consider FreeBSD as an alternative which implements ZFS-on-Root disk pools and GELI encryption. After limited testing this seems to work (even with only one disk if you just want to use it for encryption) but I understand ZFS can be corrupted by hardware problems (eg. faulty RAM), and FreeBSD doesn't seem to want to open LUKS containers on-the-fly, such as an encrypted external HD.
>
> I would prefer a Linux solution - any explanations, solutions, tips or advice re linux software RAID would be gratefully received.
>
> With kind regards
> Gareth
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
^ permalink raw reply
* Re: doubt about raid1 and writemostly
From: NeilBrown @ 2015-06-24 22:59 UTC (permalink / raw)
To: Roberto Spadim; +Cc: Linux-RAID
In-Reply-To: <CAH3kUhEdDY2DeAQ1FYaEvsRRdia9kn6V+9VQETxVdh9j-R3TTA@mail.gmail.com>
On Tue, 23 Jun 2015 12:01:33 -0300 Roberto Spadim
<roberto@spadim.com.br> wrote:
> hi guys, i'm with a doubt
> when i use witemostly, how the md handle the slower drive
> (writemostly) for example i have 2 ssd that support 500mb/s write
> running raid1, i included a hdd with +- 100mb/s write with writemostly
> flag
> when i execute a dd if=/dev/zero of=./test , i get near to 200mb/s
> with dd, and using iostat -d 1 -k, i get +- 100mb/s
> there's some kind of configuration to writemostly drive? something
> like 'buffer size' or someting like it?
>
You need to also enable --write-behind if you want md/raid1 to not be
slowed down so much by a slower device.
However this only really helps hide latency. There is nothing that can
be done to help over-all throughput.
When you set --write-behind you can provide a number which is the
number of outstanding requests that can be in flight to the
write-mostly device.
If your writes are "bursty" in nature - so lots of writes together,
then longer pauses with no writes, then this write-behind buffer can
hide the delay caused by the slower device. If you write continuously,
then there is no way that the slower device can keep up and the
over-all throughput will be reduced to match the slowest device.
This functionality was originally written for cases where both devices
had the same throughput, but one was higher latency, due to being
several kilometres away on the other end of an optical fibre.
NeilBrown
^ permalink raw reply
* Re: (R) in mdstat output, clean && degraded
From: NeilBrown @ 2015-06-24 22:52 UTC (permalink / raw)
To: Jared Mauch; +Cc: linux-raid
In-Reply-To: <51364598-D6AA-43E7-B9A1-C3C09255D3D4@puck.nether.net>
On Tue, 23 Jun 2015 14:56:09 -0400 Jared Mauch <jared@puck.nether.net>
wrote:
> I’ve been searching high and low the past few days and have been unable to diagnose what this (R) is in my raid1 mdstat output indicates.
>
> It seems something is ‘stuck’ somehow as I’m not sure how the array is both clean and degraded at the same time.
>
> Some insights are welcome.
>
> kernel 4.0.5-300.fc22 (fedora 22)
>
> /proc/mdstat
>
> md127 : active raid1 sdg1[2](R) sdd1[3]
> 976630464 blocks super 1.2 [2/1] [U_]
> bitmap: 8/8 pages [32KB], 65536KB chunk
Hmm.....
It isn't at all clear to me how you could get into this state, but I
think I can describe the state the array is in.
The array is degraded, but the one working device has been "replaced"
almost completely.
The data has all been copied from sdd1 to sdg1, but sdg1 hasn't been
marked 'faulty' yet. Normally when the 'replace' finishes, the
original gets marked 'faulty' as the new device is being marked
'in-sync'.
Once it is faulty it is removed from the array.
Some how, your replacement device got marked 'in-sync' but the original
didn't get marked 'faulty'.
Currently I believe that all writes are going to both devices, and all
reads are being served by the replacement: sdg1.
You could verify this by looking at io stats (e.g. /proc/diskstats,
though the meanings of the columns aren't obvious...)
You should be able to turn this into a fully functional RAID1 array by:
mdadm /dev/md127 --fail /dev/sdd1
mdadm /dev/md127 --remove /dev/sdd1
mdadm /dev/md127 --re-add /dev/sdd1
When you fail sdd1, sdg1 will change from being a 'replacement' to being
a regular member.
When you --re-add /dev/sdd1 you benefit from the fact that raid1
doesn't really care which device is in which slot (unlike raid5).
So re-adding something marked for slot 0 into slot 1 is perfectly
acceptable.
As the bitmap is present and uptodate, the recovery will be very fast.
I would recommend doing some basic checks for data consistency after
removing sdd1 and before re-adding it. I might be wrong about
something and sdg1 might contain complete garbage - it never hurts to
check :-)
NeilBrown
>
>
> # mdadm -D /dev/md127 ; mdadm -E /dev/sdg1 ; mdadm -E /dev/sdd1
> /dev/md127:
> Version : 1.2
> Creation Time : Sat Jan 24 10:22:05 2015
> Raid Level : raid1
> Array Size : 976630464 (931.39 GiB 1000.07 GB)
> Used Dev Size : 976630464 (931.39 GiB 1000.07 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Tue Jun 23 14:53:50 2015
> State : clean, degraded
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Name : jail-lnx:ssd-array
> UUID : a6277db4:da27d506:916a2c7a:d144aed6
> Events : 9594760
>
> Number Major Minor RaidDevice State
> 2 8 97 0 active sync /dev/sdg1
> 3 8 49 0 active sync /dev/sdd1
> 2 0 0 2 removed
> /dev/sdg1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x11
> Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
> Name : jail-lnx:ssd-array
> Creation Time : Sat Jan 24 10:22:05 2015
> Raid Level : raid1
> Raid Devices : 2
>
> Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
> Array Size : 976630464 (931.39 GiB 1000.07 GB)
> Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=110 sectors
> State : clean
> Device UUID : d5fd7437:1fd04c64:a9327851:b22e8008
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Jun 23 14:53:50 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : fa563013 - correct
> Events : 9594760
>
>
> Device Role : Replacement device 0
> Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdd1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
> Name : jail-lnx:ssd-array
> Creation Time : Sat Jan 24 10:22:05 2015
> Raid Level : raid1
> Raid Devices : 2
>
> Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
> Array Size : 976630464 (931.39 GiB 1000.07 GB)
> Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=110 sectors
> State : clean
> Device UUID : fa65731c:d8c703be:bbf05cfe:c89740f2
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Jun 23 14:53:50 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : cfb0b70c - correct
> Events : 9594760
>
>
> Device Role : Active device 0
> Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Crash flood fills syslog
From: NeilBrown @ 2015-06-24 21:53 UTC (permalink / raw)
To: John Bridges; +Cc: linux-raid
In-Reply-To: <CAJ8DDcPwUqwebvA6yZSJ7igg2pB0VWUOjbcx0xfpb1e8PG_hFA@mail.gmail.com>
On Wed, 24 Jun 2015 10:52:50 -0400 John Bridges
<john.bridges@gmail.com> wrote:
> Messages like this flooded my syslog ballooning it up to 90GB, and
> causing other failures when the system drive had no free space left
> (bad).
>
> Distributor ID: Ubuntu
> Description: Ubuntu 14.04.2 LTS
> Release: 14.04
> Codename: trusty
A bit more context always helps...
Did you do something unusual, or did something unusual happen, shortly
before these started?
It looks like a faulty device was being "replaced" - e.g. via
mdadm --replace
is that correct?
How many devices in the array? RAID5 or RAID6? Was it degraded?
3.13.0 is a little bit old, the 3.13 series is up to 3.13.11 which has
at least one raid5 bugfix in it - it may affect this issue, but I'm not
certain. Is there a kernel upgrade available from ubuntu?
NeilBrown
>
>
> Jun 23 08:25:08 monster kernel: [762785.217575] ------------[ cut here
> ]------------
> Jun 23 08:25:08 monster kernel: [762785.217580] WARNING: CPU: 0 PID:
> 28173 at /build/buildd/linux-3.13.0/drivers/md/raid5.c:3856
> handle_stripe+0x24e4/0x2500 [raid456]()
> Jun 23 08:25:08 monster kernel: [762785.217581] Modules linked in:
> pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF)
> gpio_ich coretemp kvm_intel kvm serio_raw bnep rfcomm bluetooth
> nvidia(POF) i3000_edac parport_pc drm lpc_ich ppdev lp shpchp mac_hid
> edac_core parport xfs btrfs libcrc32c raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor usb_storage pata_acpi
> ahci psmouse libahci e1000e raid6_pq ptp pps_core raid1 raid0
> multipath sata_mv floppy linear
> Jun 23 08:25:08 monster kernel: [762785.217612] CPU: 0 PID: 28173
> Comm: md2_resync Tainted: PF W O 3.13.0-29-generic #53-Ubuntu
> Jun 23 08:25:08 monster kernel: [762785.217613] Hardware name:
> Supermicro PDSM4+/PDSM4+, BIOS 6.00 11/04/2008
> Jun 23 08:25:08 monster kernel: [762785.217615] 0000000000000009
> ffff88017e997af0 ffffffff8171a214 0000000000000000
> Jun 23 08:25:08 monster kernel: [762785.217618] ffff88017e997b28
> ffffffff810676bd ffff88010d90a6b0 0000000000000009
> Jun 23 08:25:08 monster kernel: [762785.217621] 0000000000000009
> ffff8800369e8c00 000000000000000e ffff88017e997b38
> Jun 23 08:25:08 monster kernel: [762785.217624] Call Trace:
> Jun 23 08:25:08 monster kernel: [762785.217627] [<ffffffff8171a214>]
> dump_stack+0x45/0x56
> Jun 23 08:25:08 monster kernel: [762785.217630] [<ffffffff810676bd>]
> warn_slowpath_common+0x7d/0xa0
> Jun 23 08:25:08 monster kernel: [762785.217633] [<ffffffff8106779a>]
> warn_slowpath_null+0x1a/0x20
> Jun 23 08:25:08 monster kernel: [762785.217637] [<ffffffffa0108cd4>]
> handle_stripe+0x24e4/0x2500 [raid456]
> Jun 23 08:25:08 monster kernel: [762785.217640] [<ffffffff8109a7b2>]
> ? default_wake_function+0x12/0x20
> Jun 23 08:25:08 monster kernel: [762785.217644] [<ffffffff810aaeb2>]
> ? autoremove_wake_function+0x12/0x40
> Jun 23 08:25:08 monster kernel: [762785.217646] [<ffffffff810aa7b8>]
> ? __wake_up_common+0x58/0x90
> Jun 23 08:25:08 monster kernel: [762785.217651] [<ffffffffa0104c1e>]
> ? get_active_stripe+0x58e/0x7c0 [raid456]
> Jun 23 08:25:08 monster kernel: [762785.217656] [<ffffffffa0109b9c>]
> sync_request+0x17c/0x3e0 [raid456]
> Jun 23 08:25:08 monster kernel: [762785.217658] [<ffffffff815a40b1>]
> ? is_mddev_idle+0xd1/0x140
> Jun 23 08:25:08 monster kernel: [762785.217661] [<ffffffff815a7393>]
> md_do_sync+0x993/0xdc0
> Jun 23 08:25:08 monster kernel: [762785.217664] [<ffffffff810aaea0>]
> ? prepare_to_wait_event+0x100/0x100
> Jun 23 08:25:08 monster kernel: [762785.217667] [<ffffffff815a3f48>]
> md_thread+0x118/0x130
> Jun 23 08:25:08 monster kernel: [762785.217670] [<ffffffff815a3e30>]
> ? mddev_unlock+0xe0/0xe0
> Jun 23 08:25:08 monster kernel: [762785.217673] [<ffffffff8108b322>]
> kthread+0xd2/0xf0
> Jun 23 08:25:08 monster kernel: [762785.217676] [<ffffffff8108b250>]
> ? kthread_create_on_node+0x1d0/0x1d0
> Jun 23 08:25:08 monster kernel: [762785.217678] [<ffffffff8172ab3c>]
> ret_from_fork+0x7c/0xb0
> Jun 23 08:25:08 monster kernel: [762785.217681] [<ffffffff8108b250>]
> ? kthread_create_on_node+0x1d0/0x1d0
> Jun 23 08:25:08 monster kernel: [762785.217683] ---[ end trace
> 518bf06a77e3600a ]---
> Jun 23 08:25:08 monster kernel: [762785.217694] ------------[ cut here
> ]------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Crash flood fills syslog
From: John Bridges @ 2015-06-24 14:52 UTC (permalink / raw)
To: linux-raid
Messages like this flooded my syslog ballooning it up to 90GB, and
causing other failures when the system drive had no free space left
(bad).
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
Jun 23 08:25:08 monster kernel: [762785.217575] ------------[ cut here
]------------
Jun 23 08:25:08 monster kernel: [762785.217580] WARNING: CPU: 0 PID:
28173 at /build/buildd/linux-3.13.0/drivers/md/raid5.c:3856
handle_stripe+0x24e4/0x2500 [raid456]()
Jun 23 08:25:08 monster kernel: [762785.217581] Modules linked in:
pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF)
gpio_ich coretemp kvm_intel kvm serio_raw bnep rfcomm bluetooth
nvidia(POF) i3000_edac parport_pc drm lpc_ich ppdev lp shpchp mac_hid
edac_core parport xfs btrfs libcrc32c raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor usb_storage pata_acpi
ahci psmouse libahci e1000e raid6_pq ptp pps_core raid1 raid0
multipath sata_mv floppy linear
Jun 23 08:25:08 monster kernel: [762785.217612] CPU: 0 PID: 28173
Comm: md2_resync Tainted: PF W O 3.13.0-29-generic #53-Ubuntu
Jun 23 08:25:08 monster kernel: [762785.217613] Hardware name:
Supermicro PDSM4+/PDSM4+, BIOS 6.00 11/04/2008
Jun 23 08:25:08 monster kernel: [762785.217615] 0000000000000009
ffff88017e997af0 ffffffff8171a214 0000000000000000
Jun 23 08:25:08 monster kernel: [762785.217618] ffff88017e997b28
ffffffff810676bd ffff88010d90a6b0 0000000000000009
Jun 23 08:25:08 monster kernel: [762785.217621] 0000000000000009
ffff8800369e8c00 000000000000000e ffff88017e997b38
Jun 23 08:25:08 monster kernel: [762785.217624] Call Trace:
Jun 23 08:25:08 monster kernel: [762785.217627] [<ffffffff8171a214>]
dump_stack+0x45/0x56
Jun 23 08:25:08 monster kernel: [762785.217630] [<ffffffff810676bd>]
warn_slowpath_common+0x7d/0xa0
Jun 23 08:25:08 monster kernel: [762785.217633] [<ffffffff8106779a>]
warn_slowpath_null+0x1a/0x20
Jun 23 08:25:08 monster kernel: [762785.217637] [<ffffffffa0108cd4>]
handle_stripe+0x24e4/0x2500 [raid456]
Jun 23 08:25:08 monster kernel: [762785.217640] [<ffffffff8109a7b2>]
? default_wake_function+0x12/0x20
Jun 23 08:25:08 monster kernel: [762785.217644] [<ffffffff810aaeb2>]
? autoremove_wake_function+0x12/0x40
Jun 23 08:25:08 monster kernel: [762785.217646] [<ffffffff810aa7b8>]
? __wake_up_common+0x58/0x90
Jun 23 08:25:08 monster kernel: [762785.217651] [<ffffffffa0104c1e>]
? get_active_stripe+0x58e/0x7c0 [raid456]
Jun 23 08:25:08 monster kernel: [762785.217656] [<ffffffffa0109b9c>]
sync_request+0x17c/0x3e0 [raid456]
Jun 23 08:25:08 monster kernel: [762785.217658] [<ffffffff815a40b1>]
? is_mddev_idle+0xd1/0x140
Jun 23 08:25:08 monster kernel: [762785.217661] [<ffffffff815a7393>]
md_do_sync+0x993/0xdc0
Jun 23 08:25:08 monster kernel: [762785.217664] [<ffffffff810aaea0>]
? prepare_to_wait_event+0x100/0x100
Jun 23 08:25:08 monster kernel: [762785.217667] [<ffffffff815a3f48>]
md_thread+0x118/0x130
Jun 23 08:25:08 monster kernel: [762785.217670] [<ffffffff815a3e30>]
? mddev_unlock+0xe0/0xe0
Jun 23 08:25:08 monster kernel: [762785.217673] [<ffffffff8108b322>]
kthread+0xd2/0xf0
Jun 23 08:25:08 monster kernel: [762785.217676] [<ffffffff8108b250>]
? kthread_create_on_node+0x1d0/0x1d0
Jun 23 08:25:08 monster kernel: [762785.217678] [<ffffffff8172ab3c>]
ret_from_fork+0x7c/0xb0
Jun 23 08:25:08 monster kernel: [762785.217681] [<ffffffff8108b250>]
? kthread_create_on_node+0x1d0/0x1d0
Jun 23 08:25:08 monster kernel: [762785.217683] ---[ end trace
518bf06a77e3600a ]---
Jun 23 08:25:08 monster kernel: [762785.217694] ------------[ cut here
]------------
^ permalink raw reply
* [PATCH] Fix read-balancing during node failure
From: Goldwyn Rodrigues @ 2015-06-24 14:30 UTC (permalink / raw)
To: neilb; +Cc: teigland, gqJiang, linux-raid, Goldwyn Rodrigues
During a node failure, We need to suspend read balancing so that the
reads are directed to the first device and stale data is not read.
Suspending writes is not required because these would be recorded and
synced eventually.
A new flag MD_CLUSTER_SUSPEND_READ_BALANCING is set in recover_prep().
area_resyncing() will respond true for the entire devices if this
flag is set and the request type is READ. The flag is cleared
in recover_done().
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Reported-By: David Teigland <teigland@redhat.com>
---
drivers/md/md-cluster.c | 12 +++++++++++-
drivers/md/md-cluster.h | 2 +-
drivers/md/raid1.c | 7 ++++---
3 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index fcfc4b9..0072190 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -44,6 +44,7 @@ struct resync_info {
/* md_cluster_info flags */
#define MD_CLUSTER_WAITING_FOR_NEWDISK 1
+#define MD_CLUSTER_SUSPEND_READ_BALANCING 2
struct md_cluster_info {
@@ -275,6 +276,9 @@ clear_bit:
static void recover_prep(void *arg)
{
+ struct mddev *mddev = arg;
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+ set_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state);
}
static void recover_slot(void *arg, struct dlm_slot *slot)
@@ -307,6 +311,7 @@ static void recover_done(void *arg, struct dlm_slot *slots,
cinfo->slot_number = our_slot;
complete(&cinfo->completion);
+ clear_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state);
}
static const struct dlm_lockspace_ops md_ls_ops = {
@@ -816,12 +821,17 @@ static void resync_finish(struct mddev *mddev)
resync_send(mddev, RESYNCING, 0, 0);
}
-static int area_resyncing(struct mddev *mddev, sector_t lo, sector_t hi)
+static int area_resyncing(struct mddev *mddev, int direction,
+ sector_t lo, sector_t hi)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
int ret = 0;
struct suspend_info *s;
+ if ((direction == READ) &&
+ test_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state))
+ return 1;
+
spin_lock_irq(&cinfo->suspend_lock);
if (list_empty(&cinfo->suspend_list))
goto out;
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index 6817ee0..00defe2 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -18,7 +18,7 @@ struct md_cluster_operations {
int (*metadata_update_start)(struct mddev *mddev);
int (*metadata_update_finish)(struct mddev *mddev);
int (*metadata_update_cancel)(struct mddev *mddev);
- int (*area_resyncing)(struct mddev *mddev, sector_t lo, sector_t hi);
+ int (*area_resyncing)(struct mddev *mddev, int direction, sector_t lo, sector_t hi);
int (*add_new_disk_start)(struct mddev *mddev, struct md_rdev *rdev);
int (*add_new_disk_finish)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 9157a29..b5626d2 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -541,7 +541,7 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
if ((conf->mddev->recovery_cp < this_sector + sectors) ||
(mddev_is_clustered(conf->mddev) &&
- md_cluster_ops->area_resyncing(conf->mddev, this_sector,
+ md_cluster_ops->area_resyncing(conf->mddev, READ, this_sector,
this_sector + sectors)))
choose_first = 1;
else
@@ -1111,7 +1111,8 @@ static void make_request(struct mddev *mddev, struct bio * bio)
((bio_end_sector(bio) > mddev->suspend_lo &&
bio->bi_iter.bi_sector < mddev->suspend_hi) ||
(mddev_is_clustered(mddev) &&
- md_cluster_ops->area_resyncing(mddev, bio->bi_iter.bi_sector, bio_end_sector(bio))))) {
+ md_cluster_ops->area_resyncing(mddev, WRITE,
+ bio->bi_iter.bi_sector, bio_end_sector(bio))))) {
/* As the suspend_* range is controlled by
* userspace, we want an interruptible
* wait.
@@ -1124,7 +1125,7 @@ static void make_request(struct mddev *mddev, struct bio * bio)
if (bio_end_sector(bio) <= mddev->suspend_lo ||
bio->bi_iter.bi_sector >= mddev->suspend_hi ||
(mddev_is_clustered(mddev) &&
- !md_cluster_ops->area_resyncing(mddev,
+ !md_cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector, bio_end_sector(bio))))
break;
schedule();
--
2.1.4
^ permalink raw reply related
* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Jes Sorensen @ 2015-06-24 11:04 UTC (permalink / raw)
To: Roman Mamedov; +Cc: neilb, linux-raid
In-Reply-To: <20150624100014.7d1e9842@natsu>
Roman Mamedov <rm@romanrm.net> writes:
> On Tue, 23 Jun 2015 20:26:12 -0400
> Jes.Sorensen@redhat.com wrote:
>
>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>
>> Neil,
>>
>> I have been hitting issues with discard being ridiculously slow on
>> arrays with certain typs of SSDs that seem to serialize discard
>> processing.
>>
>> This is particularly bad as I have seen systems where the IMSM BIOS
>> defaults to 4KB chunk size, combined with these badly performing
>> drives, it could bump the mkfs on an array from seconds to over 40
>> minutes. Most users will stick to the defaults and then hit the
>> problem during install without understanding why it goes wrong :(
>>
>> The problem is that there is no way to benchmark our way to this or
>> somehow test if a drive performs discard at reasonable speed. I
>> suggest we take an approach similar to that of RAID456 and default to
>> disabling discard, except for the case where the user knows the drives
>> are safe.
>>
>> Thoughts?
>
> It's very unfortunate if you would cripple all the good SSD models because of
> a few bad ones. No one will remember to explicitly put the override to enable
> TRIM, or perhaps even know that it gets disabled in md in the first place. The
> only thing they will later notice is lowered performance and lifespan of their
> SSDs.
We already disable discard per default on raid456 in a similar manner
because some of them unreliably reports discard_zeroes_data when they
in reality don't.
If there was a way to reliably detect these things it would be fine,
unfortunately there isn't.
Jes
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Jes Sorensen @ 2015-06-24 11:02 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150624175556.701b724c@noble>
NeilBrown <neilb@suse.com> writes:
> On Tue, 23 Jun 2015 20:26:12 -0400
> Jes.Sorensen@redhat.com wrote:
>
>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>
>> Neil,
>>
>> I have been hitting issues with discard being ridiculously slow on
>> arrays with certain typs of SSDs that seem to serialize discard
>> processing.
>>
>> This is particularly bad as I have seen systems where the IMSM BIOS
>> defaults to 4KB chunk size, combined with these badly performing
>> drives, it could bump the mkfs on an array from seconds to over 40
>> minutes. Most users will stick to the defaults and then hit the
>> problem during install without understanding why it goes wrong :(
>>
>> The problem is that there is no way to benchmark our way to this or
>> somehow test if a drive performs discard at reasonable speed. I
>> suggest we take an approach similar to that of RAID456 and default to
>> disabling discard, except for the case where the user knows the drives
>> are safe.
>>
>> Thoughts?
>>
>> Cheers,
>> Jes
>>
>> Jes Sorensen (1):
>> raid0: Disable discard per default due to performance uncertainty
>>
>> drivers/md/raid0.c | 20 ++++++++++++++++++++
>> 1 file changed, 20 insertions(+)
>
> RAID1? RAID0?? I hate it when I do that!
>
> Doesn't the scheduler merge adjacent discard requests?
>
> Or is this some non-SATA/SCSI SSD that has a 'make_request_fn' driver?
> I think I came across one of those before (NVMe).
> In that case - the driver needs to be fixed.
Nope the block layer doesn't merge the requests at this point. It
actually triggered an OOM on my test system, which has only 4GB of RAM,
because discard commands are not handled as normal commands and have no
control on how many outstanding commands are in flight.
Jes
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: NeilBrown @ 2015-06-24 7:55 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid
In-Reply-To: <1435105573-1373-1-git-send-email-Jes.Sorensen@redhat.com>
On Tue, 23 Jun 2015 20:26:12 -0400
Jes.Sorensen@redhat.com wrote:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> Neil,
>
> I have been hitting issues with discard being ridiculously slow on
> arrays with certain typs of SSDs that seem to serialize discard
> processing.
>
> This is particularly bad as I have seen systems where the IMSM BIOS
> defaults to 4KB chunk size, combined with these badly performing
> drives, it could bump the mkfs on an array from seconds to over 40
> minutes. Most users will stick to the defaults and then hit the
> problem during install without understanding why it goes wrong :(
>
> The problem is that there is no way to benchmark our way to this or
> somehow test if a drive performs discard at reasonable speed. I
> suggest we take an approach similar to that of RAID456 and default to
> disabling discard, except for the case where the user knows the drives
> are safe.
>
> Thoughts?
>
> Cheers,
> Jes
>
>
> Jes Sorensen (1):
> raid0: Disable discard per default due to performance uncertainty
>
> drivers/md/raid0.c | 20 ++++++++++++++++++++
> 1 file changed, 20 insertions(+)
>
RAID1? RAID0?? I hate it when I do that!
Doesn't the scheduler merge adjacent discard requests?
Or is this some non-SATA/SCSI SSD that has a 'make_request_fn' driver?
I think I came across one of those before (NVMe).
In that case - the driver needs to be fixed.
NeilBrown
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Roman Mamedov @ 2015-06-24 5:00 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: neilb, linux-raid
In-Reply-To: <1435105573-1373-1-git-send-email-Jes.Sorensen@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]
On Tue, 23 Jun 2015 20:26:12 -0400
Jes.Sorensen@redhat.com wrote:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> Neil,
>
> I have been hitting issues with discard being ridiculously slow on
> arrays with certain typs of SSDs that seem to serialize discard
> processing.
>
> This is particularly bad as I have seen systems where the IMSM BIOS
> defaults to 4KB chunk size, combined with these badly performing
> drives, it could bump the mkfs on an array from seconds to over 40
> minutes. Most users will stick to the defaults and then hit the
> problem during install without understanding why it goes wrong :(
>
> The problem is that there is no way to benchmark our way to this or
> somehow test if a drive performs discard at reasonable speed. I
> suggest we take an approach similar to that of RAID456 and default to
> disabling discard, except for the case where the user knows the drives
> are safe.
>
> Thoughts?
It's very unfortunate if you would cripple all the good SSD models because of
a few bad ones. No one will remember to explicitly put the override to enable
TRIM, or perhaps even know that it gets disabled in md in the first place. The
only thing they will later notice is lowered performance and lifespan of their
SSDs.
Also most importantly, shouldn't this be handled in the lower level (individual
block devices), and not in md? There's already a mechanism to blacklist TRIM
on some specific SSD models (see libata-core), maybe there should be a way to
disable it by default. Or if those SSDs you mentioned really make it unusable,
maybe they should be just blacklisted as well.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* [PATCH 1/1] raid0: Disable discard per default due to performance uncertainty
From: Jes.Sorensen @ 2015-06-24 0:26 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
In-Reply-To: <1435105573-1373-1-git-send-email-Jes.Sorensen@redhat.com>
From: Jes Sorensen <Jes.Sorensen@redhat.com>
Some SSDs handle discard requests badly (very slowly). As an example,
mkfs.xfs on a RAID0 array with 4KB chunk size, constructed from Intel
SSDSC2BF18 180GB SSDs, can jump from 1.2secs to 43mins.
There is no reliable way to easily determine whether a device handles
discard at decent speed, and we cannot just benchmark it since it will
destroy data on the drive.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid0.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 4f19837..cc8c3c3 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -25,6 +25,11 @@
#include "raid0.h"
#include "raid5.h"
+static bool devices_discard_performance = false;
+module_param(devices_discard_performance, bool, 0644);
+MODULE_PARM_DESC(devices_discard_performance,
+ "Set to Y if all devices in each array handles discard requests at proper speed");
+
static int raid0_congested(struct mddev *mddev, int bits)
{
struct r0conf *conf = mddev->private;
@@ -277,6 +282,21 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
blk_queue_io_opt(mddev->queue,
(mddev->chunk_sectors << 9) * mddev->raid_disks);
+ /* Unfortunately, some devices have awful discard performance,
+ * especially for small sized requests. This is particularly
+ * bad for RAID0 with a small chunk size resulting in a small
+ * DISCARD requests hitting the underlaying drives.
+ * Only allow DISCARD if the sysadmin confirms that all devices
+ * in use can handle small DISCARD requests at reasonable speed,
+ * by setting a module parameter.
+ */
+ if (!devices_discard_performance) {
+ if (discard_supported) {
+ pr_info("md/raid0: discard support disabled due to performance uncertainty.\n");
+ pr_info("Set raid0.devices_discard_performance=Y to override.\n");
+ }
+ discard_supported = false;
+ }
if (!discard_supported)
queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
else
--
2.4.3
^ permalink raw reply related
* [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Jes.Sorensen @ 2015-06-24 0:26 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
From: Jes Sorensen <Jes.Sorensen@redhat.com>
Neil,
I have been hitting issues with discard being ridiculously slow on
arrays with certain typs of SSDs that seem to serialize discard
processing.
This is particularly bad as I have seen systems where the IMSM BIOS
defaults to 4KB chunk size, combined with these badly performing
drives, it could bump the mkfs on an array from seconds to over 40
minutes. Most users will stick to the defaults and then hit the
problem during install without understanding why it goes wrong :(
The problem is that there is no way to benchmark our way to this or
somehow test if a drive performs discard at reasonable speed. I
suggest we take an approach similar to that of RAID456 and default to
disabling discard, except for the case where the user knows the drives
are safe.
Thoughts?
Cheers,
Jes
Jes Sorensen (1):
raid0: Disable discard per default due to performance uncertainty
drivers/md/raid0.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
--
2.4.3
^ permalink raw reply
* Re: (R) in mdstat output, clean && degraded
From: Jared Mauch @ 2015-06-23 22:23 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid, NeilBrown
In-Reply-To: <5589D918.9000401@turmel.org>
> On Jun 23, 2015, at 6:09 PM, Phil Turmel <philip@turmel.org> wrote:
>
> On 06/23/2015 05:21 PM, Jared Mauch wrote:
>> Let me know what debugging/details might be interesting to collect.
>
> A good sequence of events (from bash history perhaps) with corresponding
> log messages is always helpful. Your initial report was pretty good.
I looked, this seems to have aged out. Plus, bash.. meh.. tcsh :)
>> Based on the Events# being in-sync, I am going to assume that the drives
>> may be operating in a protected-type mode even if the output doesn’t
>> concur.
>
> I think that's unlikely. In replace, one drive responds for sectors
> below the progress mark, and the other drive responds for sectors above it.
>> I can say this state seems to survive reboots.
>
> That suggests you give a vanilla (or near-vanilla) kernel a try with one
> of the various liveCDs that have them. I personally find the one from
> sysrescuecd.org to be most convenient, but I'm a gentoo guy :-)
I used to run SLS back in the day before everyone got all crazy with
the distributions.
> Otherwise, consider adding a third drive to see if it will rebuild and
> break the deadlock.
I’m currently out of physical slots for drives so this poses a challenge,
plus the machine is 250 mi away, but I do have console/IPMI.
> I should have to say, but I hope you have a backup :-)
There is a qcow file on the mirrored SSDs which seems to be ok as I was
able to SCP it off to another host once I found space to toss a 800gb
file. I may even be mailing you from said qcow.
it seems others were able to perhaps recover from not quite exact situations
by stopping and forcing a reassemble, but as copying 800gb would take a few
hours again, I’m nervous to perform such an action. e.g.:
http://ubuntuforums.org/showthread.php?t=1707416
I guess this means when I visit the machine next week I should take the
large external drive array and sas controller card I have here at home
as a checked bag.
- Jared--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] Add MD_FEATURE_WRITE_CACHE to sb->feature_map for write cache
From: Song Liu @ 2015-06-23 22:10 UTC (permalink / raw)
To: linux-raid; +Cc: Song Liu, shli, dan.j.williams, neilb, hch
In write_init_suepr1, if a cache device is presented, turn on
bit MD_FEATURE_WRITE_CACHE.
This is an additional patch to the mdadm patches for write-cache:
http://marc.info/?l=linux-raid&m=143158584528009
This is required for kernel patch v4 by Shaohua:
http://marc.info/?l=linux-raid&m=143509551122609
Signed-off-by: Song Liu <songliubraving@fb.com>
---
super1.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/super1.c b/super1.c
index e229efe..98cd54a 100644
--- a/super1.c
+++ b/super1.c
@@ -125,6 +125,7 @@ struct misc_dev_info {
* backwards anyway.
*/
#define MD_FEATURE_NEW_OFFSET 64 /* new_offset must be honoured */
+#define MD_FEATURE_WRITE_CACHE 256
#define MD_FEATURE_ALL (MD_FEATURE_BITMAP_OFFSET \
|MD_FEATURE_RECOVERY_OFFSET \
|MD_FEATURE_RESHAPE_ACTIVE \
@@ -132,6 +133,7 @@ struct misc_dev_info {
|MD_FEATURE_REPLACEMENT \
|MD_FEATURE_RESHAPE_BACKWARDS \
|MD_FEATURE_NEW_OFFSET \
+ |MD_FEATURE_WRITE_CACHE \
)
static int write_r5l_super1(struct supertype *st, int fd);
@@ -1597,6 +1599,10 @@ static int write_init_super1(struct supertype *st)
for (di = st->info; di; di = di->next) {
if (di->disk.state & (1 << MD_DISK_WRITECACHE))
cache_di = di;
+ sb->feature_map |= MD_FEATURE_WRITE_CACHE;
+ }
+
+ for (di = st->info; di; di = di->next) {
if (di->disk.state & (1 << MD_DISK_FAULTY))
continue;
if (di->fd < 0)
@@ -1733,6 +1739,7 @@ static int write_init_super1(struct supertype *st)
}
sb->sb_csum = calc_sb_1_csum(sb);
+ pr_err("sb->feature_map = %d\n", sb->feature_map);
rv = store_super1(st, di->fd);
if (rv == 0 && (__le32_to_cpu(sb->feature_map) & 1))
rv = st->ss->write_bitmap(st, di->fd);
--
1.8.1
^ permalink raw reply related
* Re: (R) in mdstat output, clean && degraded
From: Phil Turmel @ 2015-06-23 22:09 UTC (permalink / raw)
To: Jared Mauch; +Cc: linux-raid, NeilBrown
In-Reply-To: <F30009F6-7BF5-4786-A3A6-04C5C49B8E0C@puck.nether.net>
On 06/23/2015 05:21 PM, Jared Mauch wrote:
> Let me know what debugging/details might be interesting to collect.
A good sequence of events (from bash history perhaps) with corresponding
log messages is always helpful. Your initial report was pretty good.
> Based on the Events# being in-sync, I am going to assume that the drives
> may be operating in a protected-type mode even if the output doesn’t
> concur.
I think that's unlikely. In replace, one drive responds for sectors
below the progress mark, and the other drive responds for sectors above it.
> I can say this state seems to survive reboots.
That suggests you give a vanilla (or near-vanilla) kernel a try with one
of the various liveCDs that have them. I personally find the one from
sysrescuecd.org to be most convenient, but I'm a gentoo guy :-)
Otherwise, consider adding a third drive to see if it will rebuild and
break the deadlock.
I should have to say, but I hope you have a backup :-)
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH V4 13/13] raid5: skip resync if caching is enabled
From: Shaohua Li @ 2015-06-23 21:38 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
If caching is enabled, the caching layer will guarantee data
consistency, so skip resync for unclean shutdown
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 9faa5c6..a91ff75 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -6998,6 +6998,13 @@ static int run(struct mddev *mddev)
if (mddev->queue)
blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
conf->skip_copy = 1;
+
+ if (mddev->recovery_cp == 0) {
+ printk(KERN_NOTICE
+ "md/raid:%s: skip resync with caching enabled\n",
+ mdname(mddev));
+ mddev->recovery_cp = MaxSector;
+ }
}
return 0;
--
1.8.1
^ permalink raw reply related
* [PATCH V4 12/13] raid5: enable cache for raid array with cache disk
From: Shaohua Li @ 2015-06-23 21:38 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
Now cache is safe to enable for raid array with cache disk, let's do it.
We set the raid array sector size to 4k, so there is no partial write
which current cache code can't handling. This should not a big problem
because main filesystems are ok with 4k sector size.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 75964e9..9faa5c6 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -6340,6 +6340,9 @@ static void free_conf(struct r5conf *conf)
{
if (conf->shrinker.seeks)
unregister_shrinker(&conf->shrinker);
+ if (conf->cache)
+ r5c_exit_cache(conf->cache);
+
free_thread_groups(conf);
shrink_stripes(conf);
raid5_free_percpu(conf);
@@ -6980,6 +6983,23 @@ static int run(struct mddev *mddev)
mddev->queue);
}
+ rdev_for_each(rdev, mddev) {
+ if (test_bit(WriteCache, &rdev->flags)) {
+ char b[BDEVNAME_SIZE];
+ printk(KERN_INFO"md/raid:%s: using device %s as cache\n",
+ mdname(mddev), bdevname(rdev->bdev, b));
+ conf->cache = r5c_init_cache(conf, rdev);
+ break;
+ }
+ }
+
+ if (conf->cache) {
+ /* Make sure all write do overwite in current stage */
+ if (mddev->queue)
+ blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
+ conf->skip_copy = 1;
+ }
+
return 0;
abort:
md_unregister_thread(&mddev->thread);
--
1.8.1
^ permalink raw reply related
* [PATCH V4 11/13] raid5: guarantee cache release stripes in correct way
From: Shaohua Li @ 2015-06-23 21:38 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
When cache trys to flush data to raid, we must be very careful about
stripe release. The issue is if we release a stripe and the stripe is
handling by raid5d and we add a new bio into the stripe later, the
stripe will enter the raid5 state machine several times, which
r5cache_write_parity can't handle well. This could happen if
get_active_stripe() sleeps in adding bio to several stripes. To solve
this issue, we guarantee stripe release after all bio are added to
corresponding stripes. This is a performance win too if the bug happens.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 28 +++++++++-------
drivers/md/raid5.c | 84 +++++++++++++++++++++++++++++++-----------------
drivers/md/raid5.h | 11 ++++++-
3 files changed, 80 insertions(+), 43 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 332230a..7d5292c 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -1830,7 +1830,7 @@ static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
r5c_enter_error_mode(cache, -ENOMEM);
r5c_check_wait_error_mode(cache);
error_mode:
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
}
static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
@@ -1846,7 +1846,7 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
stripe = r5c_get_stripe(cache, stripe_index);
if (!stripe) {
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
return;
}
@@ -1885,7 +1885,7 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
} else
split = bio;
- raid5_make_request(cache->mddev, split);
+ raid5_make_request(cache->mddev, split, NULL);
start = tmp;
}
@@ -1996,7 +1996,7 @@ static void r5c_flush_endio(struct bio *bio, int err)
}
static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
- int start, int pages)
+ int start, int pages, struct raid5_plug_context *pc)
{
sector_t base;
struct bio *bio;
@@ -2031,7 +2031,7 @@ static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
atomic64_inc(&cache->out_cache_rq);
atomic64_add(bio_sectors(bio), &cache->out_cache_sectors);
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, pc);
}
}
@@ -2044,7 +2044,8 @@ static void r5c_put_stripe_dirty(struct r5c_cache *cache,
}
}
-static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe)
+static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe,
+ struct raid5_plug_context *pc)
{
unsigned long *stripe_bits;
int chunk_stripes;
@@ -2073,7 +2074,7 @@ static void r5c_flush_stripe(struct r5c_cache *cache, struct r5c_stripe *stripe)
while (end < cache->stripe_data_pages &&
stripe->data_pages[end])
end++;
- r5c_flush_one(cache, stripe, start, end - start);
+ r5c_flush_one(cache, stripe, start, end - start, pc);
}
r5c_put_stripe_dirty(cache, stripe);
}
@@ -2207,11 +2208,14 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
struct r5c_io_range *range;
u64 seq;
sector_t meta;
- struct blk_plug plug;
size_t size = 0;
+ struct raid5_plug_context pc;
if (list_empty(stripe_list))
return;
+
+ raid5_context_init(&pc);
+
list_sort(NULL, stripe_list, r5c_stripe_list_cmp);
list_for_each_entry(stripe, stripe_list, lru) {
cache->stripe_flush_data[size] =
@@ -2220,11 +2224,11 @@ static void r5c_reclaim_stripe_list(struct r5c_cache *cache,
}
size *= sizeof(__le64);
- blk_start_plug(&plug);
/* step 1: start write to raid */
list_for_each_entry(stripe, stripe_list, lru)
- r5c_flush_stripe(cache, stripe);
- blk_finish_plug(&plug);
+ r5c_flush_stripe(cache, stripe, &pc);
+
+ raid5_context_unplug(&pc, cache->mddev, false);
/* step 2: wait parity write to cache */
list_for_each_entry_reverse(stripe, stripe_list, lru)
@@ -2337,7 +2341,7 @@ static void r5c_reclaim_thread(struct md_thread *thread)
* retry error IO and dispatch it to raid disks directly
**/
while ((bio = bio_list_pop(&cache->retry_bio_list)) != NULL)
- raid5_make_request(cache->mddev, bio);
+ raid5_make_request(cache->mddev, bio, NULL);
if (++cache->retry_cnt < MAX_RETRY) {
cache->next_retry_time = jiffies +
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index f3b22f2..75964e9 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4972,26 +4972,26 @@ static struct stripe_head *__get_priority_stripe(struct r5conf *conf, int group)
return sh;
}
-struct raid5_plug_cb {
- struct blk_plug_cb cb;
- struct list_head list;
- struct list_head temp_inactive_list[NR_STRIPE_HASH_LOCKS];
-};
+void raid5_context_init(struct raid5_plug_context *context)
+{
+ int i;
+ INIT_LIST_HEAD(&context->list);
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
+ INIT_LIST_HEAD(context->temp_inactive_list + i);
+}
-static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
+void raid5_context_unplug(struct raid5_plug_context *context,
+ struct mddev *mddev, bool from_schedule)
{
- struct raid5_plug_cb *cb = container_of(
- blk_cb, struct raid5_plug_cb, cb);
struct stripe_head *sh;
- struct mddev *mddev = cb->cb.data;
struct r5conf *conf = mddev->private;
int cnt = 0;
int hash;
- if (cb->list.next && !list_empty(&cb->list)) {
+ if (context->list.next && !list_empty(&context->list)) {
spin_lock_irq(&conf->device_lock);
- while (!list_empty(&cb->list)) {
- sh = list_first_entry(&cb->list, struct stripe_head, lru);
+ while (!list_empty(&context->list)) {
+ sh = list_first_entry(&context->list, struct stripe_head, lru);
list_del_init(&sh->lru);
/*
* avoid race release_stripe_plug() sees
@@ -5005,15 +5005,38 @@ static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
* case, the count is always > 1 here
*/
hash = sh->hash_lock_index;
- __release_stripe(conf, sh, &cb->temp_inactive_list[hash]);
+ __release_stripe(conf, sh, &context->temp_inactive_list[hash]);
cnt++;
}
spin_unlock_irq(&conf->device_lock);
}
- release_inactive_stripe_list(conf, cb->temp_inactive_list,
+ release_inactive_stripe_list(conf, context->temp_inactive_list,
NR_STRIPE_HASH_LOCKS);
if (mddev->queue)
trace_block_unplug(mddev->queue, cnt, !from_schedule);
+}
+
+static void raid5_context_plug(struct raid5_plug_context *context,
+ struct stripe_head *sh)
+{
+ if (!test_and_set_bit(STRIPE_ON_UNPLUG_LIST, &sh->state))
+ list_add_tail(&sh->lru, &context->list);
+ else
+ release_stripe(sh);
+}
+
+struct raid5_plug_cb {
+ struct blk_plug_cb cb;
+ struct raid5_plug_context context;
+};
+
+static void raid5_unplug(struct blk_plug_cb *blk_cb, bool from_schedule)
+{
+ struct raid5_plug_cb *cb = container_of(
+ blk_cb, struct raid5_plug_cb, cb);
+ struct mddev *mddev = cb->cb.data;
+
+ raid5_context_unplug(&cb->context, mddev, from_schedule);
kfree(cb);
}
@@ -5032,20 +5055,14 @@ static void release_stripe_plug(struct mddev *mddev,
cb = container_of(blk_cb, struct raid5_plug_cb, cb);
- if (cb->list.next == NULL) {
- int i;
- INIT_LIST_HEAD(&cb->list);
- for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
- INIT_LIST_HEAD(cb->temp_inactive_list + i);
- }
+ if (cb->context.list.next == NULL)
+ raid5_context_init(&cb->context);
- if (!test_and_set_bit(STRIPE_ON_UNPLUG_LIST, &sh->state))
- list_add_tail(&sh->lru, &cb->list);
- else
- release_stripe(sh);
+ raid5_context_plug(&cb->context, sh);
}
-static void make_discard_request(struct mddev *mddev, struct bio *bi)
+static void make_discard_request(struct mddev *mddev, struct bio *bi,
+ struct raid5_plug_context *context)
{
struct r5conf *conf = mddev->private;
sector_t logical_sector, last_sector;
@@ -5127,7 +5144,10 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
clear_bit(STRIPE_DELAYED, &sh->state);
if (!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
- release_stripe_plug(mddev, sh);
+ if (context)
+ raid5_context_plug(context, sh);
+ else
+ release_stripe_plug(mddev, sh);
}
remaining = raid5_dec_bi_active_stripes(bi);
@@ -5137,7 +5157,8 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
}
-void raid5_make_request(struct mddev *mddev, struct bio * bi)
+void raid5_make_request(struct mddev *mddev, struct bio * bi,
+ struct raid5_plug_context *context)
{
struct r5conf *conf = mddev->private;
int dd_idx;
@@ -5167,7 +5188,7 @@ void raid5_make_request(struct mddev *mddev, struct bio * bi)
return;
if (unlikely(bi->bi_rw & REQ_DISCARD)) {
- make_discard_request(mddev, bi);
+ make_discard_request(mddev, bi, context);
return;
}
@@ -5295,7 +5316,10 @@ void raid5_make_request(struct mddev *mddev, struct bio * bi)
((bi->bi_rw & REQ_SYNC) || conf->cache) &&
!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
atomic_inc(&conf->preread_active_stripes);
- release_stripe_plug(mddev, sh);
+ if (context)
+ raid5_context_plug(context, sh);
+ else
+ release_stripe_plug(mddev, sh);
} else {
/* cannot get stripe for read-ahead, just give-up */
clear_bit(BIO_UPTODATE, &bi->bi_flags);
@@ -5322,7 +5346,7 @@ static void make_request(struct mddev *mddev, struct bio *bi)
if (conf->cache)
r5c_handle_bio(conf->cache, bi);
else
- raid5_make_request(mddev, bi);
+ raid5_make_request(mddev, bi, NULL);
}
static sector_t raid5_size(struct mddev *mddev, sector_t sectors, int raid_disks);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 25d9014..2d78f1e 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -608,10 +608,19 @@ static inline int algorithm_is_DDF(int layout)
extern void md_raid5_kick_device(struct r5conf *conf);
extern int raid5_set_cache_size(struct mddev *mddev, int size);
+struct raid5_plug_context {
+ struct list_head list;
+ struct list_head temp_inactive_list[NR_STRIPE_HASH_LOCKS];
+};
+void raid5_context_init(struct raid5_plug_context *context);
+void raid5_context_unplug(struct raid5_plug_context *context,
+ struct mddev *mddev, bool from_schedule);
+
void release_stripe(struct stripe_head *sh);
int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *sh);
void r5c_flush_pending_parity(struct r5c_cache *cache);
-void raid5_make_request(struct mddev *mddev, struct bio *bi);
+void raid5_make_request(struct mddev *mddev, struct bio *bi,
+ struct raid5_plug_context *context);
void r5c_handle_bio(struct r5c_cache *cache, struct bio *bi);
struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev);
void r5c_exit_cache(struct r5c_cache *cache);
--
1.8.1
^ permalink raw reply related
* [PATCH V4 10/13] raid5: don't allow resize/reshape with cache support
From: Shaohua Li @ 2015-06-23 21:38 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
If cache support is enabled, don't allow resize/reshape in current
stage. In the future, we can flush all data from cache to raid before
resize/reshape and then allow resize/reshape.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 093611e..f3b22f2 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7186,6 +7186,10 @@ static int raid5_resize(struct mddev *mddev, sector_t sectors)
* worth it.
*/
sector_t newsize;
+ struct r5conf *conf = mddev->private;
+
+ if (conf->cache)
+ return -EINVAL;
sectors &= ~((sector_t)mddev->chunk_sectors - 1);
newsize = raid5_size(mddev, sectors, mddev->raid_disks);
if (mddev->external_size &&
@@ -7237,6 +7241,8 @@ static int check_reshape(struct mddev *mddev)
{
struct r5conf *conf = mddev->private;
+ if (conf->cache)
+ return -EINVAL;
if (mddev->delta_disks == 0 &&
mddev->new_layout == mddev->layout &&
mddev->new_chunk_sectors == mddev->chunk_sectors)
--
1.8.1
^ permalink raw reply related
* [PATCH V4 09/13] raid5: add some sysfs entries
From: Shaohua Li @ 2015-06-23 21:37 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
Add some sysfs entries.
-cache_memory. Control the cache memory size.
-cache_reclaim_batch. Control how many stripes reclaim should run in one
time.
-cache_memory_watermark. The background reclaim runs if cache memory
hits the watermark and stops after hit 1.5x of the watermark.
-cache_disk_watermark. The background reclaim runs if cache disk space
hits the watermark and stops after hit 1.5x of the watermark.
-cache_stat. statistics about cache.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 299 ++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/raid5.c | 3 +
drivers/md/raid5.h | 1 +
3 files changed, 302 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 143f333..332230a 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -352,6 +352,12 @@ struct r5c_cache {
struct kmem_cache *io_range_kc;
struct kmem_cache *stripe_kc;
struct bio_set *bio_set;
+
+ atomic64_t in_cache_rq;
+ atomic64_t out_cache_rq;
+ atomic64_t in_cache_sectors;
+ atomic64_t out_cache_sectors;
+ atomic64_t read_cache_sectors;
};
/* reclaim reason */
@@ -405,6 +411,12 @@ static inline int r5l_page_blocks(struct r5l_log *log, int pages)
return pages << log->page_block_shift;
}
+static inline int r5l_max_flush_stripes(struct r5l_log *log)
+{
+ return (log->block_size - sizeof(struct r5l_flush_block)) /
+ sizeof(__le64);
+}
+
static u32 r5l_calculate_checksum(struct r5l_log *log, u32 crc,
void *buf, size_t size, bool data)
{
@@ -1804,6 +1816,9 @@ static void r5c_write_bio(struct r5c_cache *cache, struct bio *bio)
stripe->existing_pages += new_pages;
r5c_unlock_stripe(cache, stripe, &flags);
+ atomic64_inc(&cache->in_cache_rq);
+ atomic64_add(bio_sectors(bio), &cache->in_cache_sectors);
+
if (r5l_queue_bio(&cache->log, bio, r5c_bio_task_end, io_range,
reserved_blocks))
goto put_error;
@@ -1852,6 +1867,8 @@ static void r5c_read_bio(struct r5c_cache *cache, struct bio *bio)
split = bio;
r5c_copy_bio(split, &stripe->data_pages[start], true);
+ atomic64_add(bio_sectors(split),
+ &cache->read_cache_sectors);
bio_endio(split, 0);
@@ -2010,6 +2027,10 @@ static void r5c_flush_one(struct r5c_cache *cache, struct r5c_stripe *stripe,
bio->bi_end_io = r5c_flush_endio;
bio->bi_rw = WRITE;
atomic_inc(&stripe->pending_bios);
+
+ atomic64_inc(&cache->out_cache_rq);
+ atomic64_add(bio_sectors(bio), &cache->out_cache_sectors);
+
raid5_make_request(cache->mddev, bio);
}
}
@@ -3310,6 +3331,278 @@ static int r5c_shrink_cache_memory(struct r5c_cache *cache, unsigned long size)
return 0;
}
+static ssize_t r5c_show_cache_memory(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->max_pages << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory(struct mddev *mddev, const char *page,
+ size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+ LIST_HEAD(page_list);
+ u64 i;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new > cache->max_pages) {
+ i = cache->max_pages;
+ while (i < new) {
+ struct page *page = alloc_page(GFP_KERNEL);
+
+ if (!page)
+ break;
+ list_add(&page->lru, &page_list);
+ i++;
+ }
+
+ spin_lock_irq(&cache->pool_lock);
+ list_splice(&page_list, &cache->page_pool);
+ cache->free_pages += i - cache->max_pages;
+ cache->max_pages = i;
+ cache->total_pages = i;
+ r5c_calculate_watermark(cache);
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+ }
+ r5c_shrink_cache_memory(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory = __ATTR(cache_memory,
+ S_IRUGO | S_IWUSR, r5c_show_cache_memory, r5c_store_cache_memory);
+
+/*
+ * we reclaim stripes in a batch way, so we must make sure there are enough
+ * stripe cache. Otherwise, reclaim will deadlock to wait some stripe caches
+ * free, but such stripe caches don't even run since reclaim is waitting
+ * */
+int r5c_min_stripe_cache_size(struct r5c_cache *cache)
+{
+ struct r5conf *conf = cache->mddev->private;
+ return (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) *
+ cache->reclaim_batch;
+}
+
+static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch)
+{
+ struct mddev *mddev = cache->mddev;
+ struct r5conf *conf = mddev->private;
+ int size;
+
+ size = (cache->stripe_parity_pages << PAGE_SECTOR_SHIFT) * batch;
+ if (size > cache->reserved_space) {
+ cache->reserved_space = size;
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks = r5l_sector_to_block(&cache->log,
+ cache->reserved_space) + 1;
+ mutex_unlock(&cache->log.io_mutex);
+ r5c_wake_wait_reclaimer(cache,
+ RECLAIM_DISK_BACKGROUND);
+ } else {
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks -= r5l_sector_to_block(&cache->log,
+ cache->reserved_space - size);
+ mutex_unlock(&cache->log.io_mutex);
+ cache->reserved_space = size;
+ }
+
+ size = (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) * batch;
+
+ mddev_lock(mddev);
+ if (size > conf->max_nr_stripes)
+ raid5_set_cache_size(mddev, size);
+ mddev_unlock(mddev);
+
+ cache->reclaim_batch = batch;
+}
+
+static ssize_t r5c_show_cache_reclaim_batch(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%d\n", cache->reclaim_batch);
+}
+
+static ssize_t r5c_store_cache_reclaim_batch(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+
+ if (new > r5l_max_flush_stripes(&cache->log))
+ new = r5l_max_flush_stripes(&cache->log);
+
+ if (new != cache->reclaim_batch)
+ r5c_set_reclaim_batch(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_reclaim_batch =
+ __ATTR(cache_reclaim_batch, S_IRUGO | S_IWUSR,
+ r5c_show_cache_reclaim_batch, r5c_store_cache_reclaim_batch);
+
+static ssize_t r5c_show_cache_stat(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld %lld %lld %lld %lld\n",
+ (u64)atomic64_read(&cache->in_cache_rq),
+ (u64)atomic64_read(&cache->in_cache_sectors),
+ (u64)atomic64_read(&cache->out_cache_rq),
+ (u64)atomic64_read(&cache->out_cache_sectors),
+ (u64)atomic64_read(&cache->read_cache_sectors));
+}
+
+static struct md_sysfs_entry r5c_cache_stat =
+ __ATTR(cache_stat, S_IRUGO, r5c_show_cache_stat, NULL);
+
+static ssize_t r5c_show_cache_disk_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->log.low_watermark *
+ cache->log.block_size);
+}
+
+static ssize_t r5c_store_cache_disk_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ struct r5l_log *log = &cache->log;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new /= log->block_size;
+
+ if (new * 3 / 2 >= log->total_blocks)
+ return -EINVAL;
+
+ mutex_lock(&log->io_mutex);
+ log->low_watermark = new;
+ log->high_watermark = new * 3 / 2;
+ mutex_unlock(&log->io_mutex);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_disk_watermark =
+ __ATTR(cache_disk_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_disk_watermark, r5c_store_cache_disk_watermark);
+
+static ssize_t r5c_show_cache_memory_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->low_watermark << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new * 2 >= cache->max_pages)
+ return -EINVAL;
+
+ spin_lock_irq(&cache->pool_lock);
+ cache->low_watermark = new;
+ cache->high_watermark = new << 1;
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory_watermark =
+ __ATTR(cache_memory_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_memory_watermark, r5c_store_cache_memory_watermark);
+
+static int r5c_init_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ int ret;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj, &r5c_cache_memory.attr,
+ NULL);
+ if (ret)
+ return ret;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ if (ret)
+ goto err_reclaim;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ if (ret)
+ goto disk_watermark;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+ if (ret)
+ goto stat;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+ if (ret)
+ goto memory_watermark;
+ return 0;
+memory_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+stat:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+disk_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+err_reclaim:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ return ret;
+}
+
+static void r5c_exit_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_stat.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+}
+
static void r5c_free_cache_data(struct r5c_cache *cache)
{
struct r5c_stripe *stripe;
@@ -3420,8 +3713,11 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
cache->reclaim_thread->timeout = CHECKPOINT_TIMEOUT;
r5c_shrink_cache_memory(cache, cache->max_pages);
-
+ if (r5c_init_sysfs(cache))
+ goto err_sysfs;
return cache;
+err_sysfs:
+ md_unregister_thread(&cache->reclaim_thread);
err_page:
r5c_free_cache_data(cache);
@@ -3440,6 +3736,7 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
void r5c_exit_cache(struct r5c_cache *cache)
{
+ r5c_exit_sysfs(cache);
md_unregister_thread(&cache->reclaim_thread);
r5l_exit_log(&cache->log);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index bcd6c1f..093611e 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5906,6 +5906,9 @@ raid5_set_cache_size(struct mddev *mddev, int size)
if (size <= 16 || size > 32768)
return -EINVAL;
+ if (conf->cache && size < r5c_min_stripe_cache_size(conf->cache))
+ size = r5c_min_stripe_cache_size(conf->cache);
+
conf->min_nr_stripes = size;
while (size < conf->max_nr_stripes &&
drop_one_stripe(conf))
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 534e5be..25d9014 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -618,4 +618,5 @@ void r5c_exit_cache(struct r5c_cache *cache);
void r5c_write_start(struct mddev *mddev, struct bio *bi);
void r5c_write_end(struct mddev *mddev, struct bio *bi);
void r5c_quiesce(struct r5conf *conf, int state);
+int r5c_min_stripe_cache_size(struct r5c_cache *cache);
#endif
--
1.8.1
^ permalink raw reply related
* [PATCH V4 08/13] raid5: cache recovery support
From: Shaohua Li @ 2015-06-23 21:37 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1435094582.git.shli@fb.com>
After a unclean shutdown, cache must be recovered. According to reclaim
process, we take different actions for stripes in different states.
-stripe without parity: read data to memory
-stripe with parity but not listed in flush_start block: read data to
memory and discard parity
-stripe with parity and listed in flush_start but not in flush_end
block: read data/parity to memory, recovery (flush them to raid disks)
-stripe listed in flush_end block: discard all its data and parity
The recovery process is quite straightforward. We just read all
data/parity into memory, do sanity check and flush stripes which reclaim
was flushint them to raid disks at crash.
There is one thing we must be careful about that recovery should use
parity stored in log instead of that calculated by raid5 stripe state
machine, since after crash, the raid array isn't in consistent state,
calculated parity could be wrong.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 640 ++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 639 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 35657cd..143f333 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -288,6 +288,13 @@ enum {
STRIPE_DEAD = 4, /* stripe can be reused */
};
+/* stripe recovery state */
+enum {
+ RECOVERY_NO_FLUSH = 0,
+ RECOVERY_FLUSH_START = 1, /* stripe in a start flush block */
+ RECOVERY_FLUSH_END = 2,
+};
+
#define STRIPE_LOCK_BITS 8
struct r5c_cache {
struct mddev *mddev;
@@ -1535,7 +1542,12 @@ static void r5c_put_stripe(struct r5c_stripe *stripe)
kmem_cache_free(cache->io_range_kc, range);
}
r5c_put_pages(cache, stripe->data_pages, cache->stripe_data_pages);
- BUG_ON(stripe->parity_pages);
+ if (stripe->parity_pages) {
+ r5c_put_pages(cache, stripe->parity_pages,
+ cache->stripe_parity_pages);
+ kfree(stripe->parity_pages);
+ stripe->parity_pages = NULL;
+ }
r5c_unlock_stripe(cache, stripe, &flags2);
@@ -2401,6 +2413,26 @@ static void r5c_write_one_stripe_parity(struct r5c_cache *cache,
stripe_offset = sector_div(stripe_index, cache->chunk_size);
stripe_offset >>= PAGE_SECTOR_SHIFT;
+ /*
+ * in recovery, raid array isn't in consistent state, we shouldn't use
+ * calculated parity but those stored in log
+ * */
+ if (cache->in_recovery) {
+ stripe_offset *= cache->parity_disks;
+
+ copy_highpage(sh->dev[sh->pd_idx].page,
+ stripe->parity_pages[stripe_offset]);
+ if (sh->qd_idx >= 0)
+ copy_highpage(sh->dev[sh->qd_idx].page,
+ stripe->parity_pages[stripe_offset + 1]);
+
+ if (!atomic_dec_return(&stripe->dirty_stripes)) {
+ stripe->state = STRIPE_PARITY_DONE;
+ wake_up(r5c_stripe_waitq(cache, stripe));
+ }
+ return;
+ }
+
if (cache->error_state) {
r5c_put_stripe_dirty(cache, stripe);
return;
@@ -2454,8 +2486,614 @@ int r5c_write_parity(struct r5c_cache *cache, struct stripe_head *head_sh)
return 0;
}
+struct r5c_load_ctx {
+ struct r5c_cache *cache;
+ struct list_head io_ranges;
+ struct list_head stripes_with_parity;
+ struct list_head stripes_without_parity;
+ atomic_t io_cnt;
+ wait_queue_head_t io_wq;
+};
+
+/*
+ * If a stripe has parity in the log and is listed in a flush start block, the
+ * stripe is being flushed to raid. If the stripe is listed in flush end block,
+ * the striep is fully flushed to raid, all its data/parity can be discarded.
+ * Otherwise, the stripe must be recovered. If a stripe has parity data and is
+ * not listed in flush start block, it's not flushed to raid, its parity can be
+ * discarded.
+ *
+ * Note super block might point to anywhere. d(data), p(parity), f(flush start),
+ * F(flush end), ^(superblock pointed log tail). assume 4d 2p:
+ *
+ * d d d d p p f F
+ * ^, normal case
+ * ^ stripe must be in F, ignore data, parity
+ * ^ stripe must be in F, ignore parity
+ * ^ stripe in f/F can be ingored
+ * d d d d p
+ * ^, f/F should not exsist if p isn't enough
+ * d d d d p p
+ * ^, invalid case, superblock is updated after F
+ *
+ * Each reclaim thread records its own flushed stripes and multiple reclaim
+ * threads can't handle one stripe in the meaning time. Multiple reclaim threads
+ * don't change the behavior.
+ * */
+static int r5c_add_flush_start(struct r5c_load_ctx *ctx,
+ struct r5l_flush_block *fb, u64 seq)
+{
+ struct r5c_stripe *stripe;
+ int size;
+ int i;
+ u64 stripe_index;
+
+ size = le32_to_cpu(fb->header.meta_size) -
+ sizeof(struct r5l_flush_block);
+ size /= sizeof(__le64);
+
+ for (i = 0; i < size; i++) {
+ stripe_index = le64_to_cpu(fb->flush_stripes[i]);
+ /* superblock skips the stripe */
+ stripe = r5c_search_stripe(ctx->cache, stripe_index);
+ if (!stripe)
+ continue;
+ stripe->recovery_state = RECOVERY_FLUSH_START;
+ }
+ return 0;
+}
+
+static int r5c_add_flush_end(struct r5c_load_ctx *ctx,
+ struct r5l_flush_block *fb, u64 seq)
+{
+ struct r5c_cache *cache = ctx->cache;
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ int size;
+ int i;
+ u64 stripe_index;
+
+ size = le32_to_cpu(fb->header.meta_size) -
+ sizeof(struct r5l_flush_block);
+ size /= sizeof(__le64);
+
+ for (i = 0; i < size; i++) {
+ stripe_index = le64_to_cpu(fb->flush_stripes[i]);
+ stripe = r5c_search_stripe(ctx->cache, stripe_index);
+ /* superblock skips the stripe */
+ if (!stripe)
+ continue;
+ if (stripe->recovery_state != RECOVERY_FLUSH_START)
+ return -EINVAL;
+
+ list_del(&stripe->lru);
+ /* essentially r5c_put_stripe */
+ radix_tree_delete(&cache->stripe_tree, stripe->raid_index);
+
+ while (!list_empty(&stripe->io_ranges)) {
+ range = list_first_entry(&stripe->io_ranges,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ list_del(&range->log_sibling);
+
+ kfree(range->checksum);
+ kmem_cache_free(cache->io_range_kc, range);
+ }
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kmem_cache_free(cache->io_range_kc, range);
+ }
+
+ kmem_cache_free(cache->stripe_kc, stripe);
+ }
+ return 0;
+}
+
+static int r5c_add_data(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t raid_start,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset;
+
+ r5c_sector_stripe_index_offset(ctx->cache, raid_start, &index,
+ &offset);
+
+ stripe = r5c_create_get_stripe(ctx->cache, index);
+ if (stripe->recovery_state != RECOVERY_NO_FLUSH)
+ return -EINVAL;
+
+ range = kmem_cache_alloc(ctx->cache->io_range_kc, GFP_NOIO);
+ if (!range)
+ return -ENOMEM;
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = raid_start;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ list_add_tail(&range->stripe_sibling, &stripe->io_ranges);
+ list_add_tail(&range->log_sibling, &ctx->io_ranges);
+
+ if (list_empty(&stripe->lru))
+ list_add_tail(&stripe->lru, &ctx->stripes_without_parity);
+
+ r5c_put_stripe(stripe);
+ return 0;
+}
+
+static int r5c_add_parity(struct r5c_load_ctx *ctx, u64 seq,
+ sector_t meta_start, sector_t data_start, sector_t stripe_sect,
+ ssize_t sectors, u32 *checksum)
+{
+ struct r5c_stripe *stripe;
+ struct r5c_io_range *range;
+ sector_t index = stripe_sect;
+
+ sector_div(index, ctx->cache->log.chunk_size);
+ /* superblock skips this stripe */
+ stripe = r5c_search_stripe(ctx->cache, index);
+ if (!stripe)
+ return 0;
+ if (stripe->recovery_state != RECOVERY_NO_FLUSH)
+ return -EINVAL;
+
+ range = kmem_cache_alloc(ctx->cache->io_range_kc, GFP_NOIO);
+ if (!range)
+ return -ENOMEM;
+
+ range->seq = seq;
+ range->meta_start = meta_start;
+ range->data_start = data_start;
+ range->raid_start = stripe_sect;
+ range->data_sectors = sectors;
+ range->stripe = stripe;
+ range->checksum = checksum;
+
+ if (list_empty(&stripe->parity_list))
+ list_move_tail(&stripe->lru, &ctx->stripes_with_parity);
+ list_add_tail(&range->stripe_sibling, &stripe->parity_list);
+
+ return 0;
+}
+
+static void r5c_free_parity_ranges(struct r5c_stripe *stripe, bool skip_parity)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+
+ while (!list_empty(&stripe->parity_list)) {
+ range = list_first_entry(&stripe->parity_list,
+ struct r5c_io_range, stripe_sibling);
+ list_del(&range->stripe_sibling);
+
+ kfree(range->checksum);
+ kmem_cache_free(cache->io_range_kc, range);
+ }
+ if (skip_parity)
+ return;
+ if (stripe->parity_pages) {
+ r5c_put_pages(cache, stripe->parity_pages,
+ cache->stripe_parity_pages);
+ kfree(stripe->parity_pages);
+ stripe->parity_pages = NULL;
+ }
+}
+
+/*
+ * parities of some stripes aren't fully in log disk, such stripes havedn't
+ * started to be flushed to log yet, ignore parities of such stripes.
+ **/
+static void r5c_analyse_log(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe, *tmp;
+ int recovery_state;
+ LIST_HEAD(list);
+
+ list_for_each_entry_safe(stripe, tmp, &ctx->stripes_with_parity, lru) {
+ recovery_state = stripe->recovery_state;
+ stripe->recovery_state = RECOVERY_NO_FLUSH;
+
+ if (recovery_state == RECOVERY_NO_FLUSH)
+ list_move_tail(&stripe->lru, &list);
+ }
+ if (list_empty(&list))
+ return;
+ list_for_each_entry(stripe, &list, lru)
+ r5c_free_parity_ranges(stripe, false);
+ list_splice_tail(&list, &ctx->stripes_without_parity);
+ printk(KERN_ERR
+ "md: %s, parities of some stripes aren't fully in cache, "
+ "discard them\n",
+ mdname(ctx->cache->mddev));
+}
+
+static void r5l_fetch_endio(struct bio *bio, int err)
+{
+ struct r5c_load_ctx *ctx = bio->bi_private;
+
+ bio_put(bio);
+ if (atomic_dec_and_test(&ctx->io_cnt))
+ wake_up(&ctx->io_wq);
+}
+
+static int r5l_fetch_one_range(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe, struct page **pages,
+ ssize_t page_cnt, sector_t sec, bool data)
+{
+ struct r5l_log *log = &ctx->cache->log;
+ struct bio *bio;
+ int page_index = 0;
+ int nvec;
+ int i;
+
+again:
+ nvec = min_t(int, page_cnt - page_index,
+ bio_get_nr_vecs(r5l_bdev(log)));
+ bio = bio_kmalloc(GFP_NOIO, nvec);
+ bio->bi_iter.bi_sector = sec;
+ bio->bi_end_io = r5l_fetch_endio;
+ bio->bi_private = ctx;
+ bio->bi_bdev = r5l_bdev(log);
+ atomic_inc(&ctx->io_cnt);
+
+ /* FIXME: we might use too many memory */
+ for (i = 0; i < nvec; i++) {
+ pages[page_index] = alloc_page(GFP_NOIO | __GFP_HIGHMEM);
+ if (!bio_add_page(bio, pages[page_index], PAGE_SIZE, 0)) {
+ __free_page(pages[page_index]);
+ pages[page_index] = NULL;
+ break;
+ }
+ page_index++;
+ if (data)
+ stripe->existing_pages++;
+ ctx->cache->total_pages++;
+ }
+
+ sec = r5l_block_to_sector(log, r5l_ring_add(log,
+ r5l_sector_to_block(log, sec), r5l_page_blocks(log, i)));
+ r5l_submit_bio(log, READ, bio);
+
+ if (page_index < page_cnt)
+ goto again;
+ return 0;
+}
+
+static int r5l_fetch_one_stripe(struct r5c_load_ctx *ctx,
+ struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5c_io_range *range;
+ u64 index;
+ int offset, max, start, range_start;
+
+ /* new data is at the tail */
+ list_for_each_entry_reverse(range, &stripe->io_ranges, stripe_sibling) {
+ r5c_sector_stripe_index_offset(cache, range->raid_start,
+ &index, &offset);
+ offset >>= PAGE_SECTOR_SHIFT;
+ range_start = offset;
+ max = (range->data_sectors >> PAGE_SECTOR_SHIFT) + offset;
+again:
+ while (offset < max && stripe->data_pages[offset])
+ offset++;
+ start = offset;
+ while (offset < max && !stripe->data_pages[offset])
+ offset++;
+ if (start >= max)
+ continue;
+ r5l_fetch_one_range(ctx, stripe, &stripe->data_pages[start],
+ offset - start,
+ range->data_start + ((start - range_start) <<
+ PAGE_SECTOR_SHIFT), true);
+ if (offset < max)
+ goto again;
+ }
+
+ if (list_empty(&stripe->parity_list))
+ return 0;
+
+ stripe->parity_pages = kmalloc(cache->stripe_parity_pages *
+ sizeof(struct page *), GFP_NOIO | __GFP_ZERO);
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ r5l_fetch_one_range(ctx, stripe, &stripe->parity_pages[offset],
+ cache->parity_disks, range->data_start, false);
+ }
+ return 0;
+}
+
+static int r5l_fetch_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe;
+ struct blk_plug plug;
+
+ blk_start_plug(&plug);
+ list_for_each_entry(stripe, &ctx->stripes_without_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ list_for_each_entry(stripe, &ctx->stripes_with_parity, lru) {
+ r5l_fetch_one_stripe(ctx, stripe);
+ }
+ blk_finish_plug(&plug);
+
+ atomic_dec(&ctx->io_cnt);
+ wait_event(ctx->io_wq, atomic_read(&ctx->io_cnt) == 0);
+
+ return 0;
+}
+
+static u32 r5l_calculate_page_checksum(struct r5l_log *log, struct page *page)
+{
+ u32 csum;
+ void *addr = kmap_atomic(page);
+ csum = r5l_calculate_checksum(log, log->uuid_checksum_data,
+ addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ return csum;
+}
+
+static int r5l_check_one_stripe_checksum(struct r5c_stripe *stripe)
+{
+ struct r5c_cache *cache = stripe->cache;
+ struct r5l_log *log = &cache->log;
+ struct r5c_io_range *range;
+ struct page *page;
+ u64 index;
+ int offset;
+ int ret = 0;
+
+ /* FIXME: we currently ignore all data parity check */
+ list_for_each_entry(range, &stripe->io_ranges, stripe_sibling) {
+ kfree(range->checksum);
+ range->checksum = NULL;
+ }
+
+ if (list_empty(&stripe->parity_list)) {
+ r5c_free_parity_ranges(stripe, false);
+ return 0;
+ }
+
+ list_for_each_entry(range, &stripe->parity_list, stripe_sibling) {
+ index = range->raid_start;
+ offset = sector_div(index, cache->chunk_size);
+ offset >>= PAGE_SECTOR_SHIFT;
+ offset *= cache->parity_disks;
+
+ page = stripe->parity_pages[offset];
+
+ if (le32_to_cpu(range->checksum[0]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ if (cache->parity_disks > 1) {
+ page = stripe->parity_pages[offset + 1];
+ if (le32_to_cpu(range->checksum[1]) !=
+ r5l_calculate_page_checksum(log, page)) {
+ ret = -EINVAL;
+ break;
+ }
+ }
+ }
+ /* If parity checksum is wrong, we don't recovery the stripe */
+ r5c_free_parity_ranges(stripe, !ret ? true : false);
+ return ret;
+}
+
+static int r5l_check_stripes_checksum(struct r5c_load_ctx *ctx)
+{
+ struct r5c_stripe *stripe, *tmp;
+
+ while (!list_empty(&ctx->stripes_without_parity)) {
+ stripe = list_first_entry(&ctx->stripes_without_parity,
+ struct r5c_stripe, lru);
+ list_del_init(&stripe->lru);
+
+ r5l_check_one_stripe_checksum(stripe);
+ }
+
+ list_for_each_entry_safe(stripe, tmp, &ctx->stripes_with_parity, lru) {
+ /* checksum error, don't recover the stripe */
+ if (r5l_check_one_stripe_checksum(stripe)) {
+ list_del_init(&stripe->lru);
+ }
+ }
+
+ return 0;
+}
+
+static int r5c_recover_stripes(struct r5c_load_ctx *ctx)
+{
+ struct r5c_cache *cache = ctx->cache;
+ LIST_HEAD(list);
+ int i;
+
+ r5l_check_stripes_checksum(ctx);
+
+ list_splice_tail(&ctx->io_ranges, &ctx->cache->log_list);
+
+ if (list_empty(&ctx->stripes_with_parity))
+ return 0;
+
+ /*
+ * we already read all data/metadata in memory, just flush them to raid
+ * disks.
+ *
+ * But note for parity, we should use that stored in cache disk instead
+ * of that we calculated in recovery, since raid array isn't in
+ * consistent state. The in_recovery is for this purpose.
+ * */
+ cache->in_recovery = 1;
+
+ while (!list_empty(&ctx->stripes_with_parity)) {
+ i = 0;
+ /*
+ * Can't handle large stripe list because of the size
+ * limitation of flush_end block. We don't really need
+ * flush_end though, but adding it makes we can share the
+ * reclaim code
+ * */
+ while (i < RECLAIM_BATCH &&
+ !list_empty(&ctx->stripes_with_parity)) {
+ list_move_tail(ctx->stripes_with_parity.next,
+ &list);
+ i++;
+ }
+ r5c_reclaim_stripe_list(cache, &list);
+ BUG_ON(!list_empty(&list));
+ }
+
+ cache->in_recovery = 0;
+ return 0;
+}
+
+static void *r5l_read_meta_block(struct r5l_log *log, r5blk_t block,
+ u64 expected_seq, struct page **retpage)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5l_meta_header *header;
+ u32 crc, stored_crc;
+
+ if (!sync_page_io(log->rdev, r5l_block_to_sector(log, block),
+ log->block_size, page, READ, false))
+ return NULL;
+
+ header = page_address(page);
+ stored_crc = le32_to_cpu(header->checksum);
+ header->checksum = 0;
+
+ if (le32_to_cpu(header->magic) != R5LOG_MAGIC ||
+ le64_to_cpu(header->seq) != expected_seq ||
+ le64_to_cpu(header->position) != block)
+ goto error;
+ if (le32_to_cpu(header->type) != R5LOG_TYPE_META &&
+ le32_to_cpu(header->type) != R5LOG_TYPE_FLUSH_START &&
+ le32_to_cpu(header->type) != R5LOG_TYPE_FLUSH_END)
+ goto error;
+
+ crc = r5l_calculate_checksum(log, log->uuid_checksum_meta,
+ header, log->block_size, false);
+ if (stored_crc != crc)
+ goto error;
+
+ if (le32_to_cpu(header->meta_size) > log->block_size)
+ goto error;
+ *retpage = page;
+ return header;
+error:
+ __free_page(page);
+ return NULL;
+}
+
static int r5l_load_log(struct r5l_log *log)
{
+ r5blk_t last_cp = log->last_checkpoint;
+ u64 last_seq = log->last_cp_seq;
+ r5blk_t payload_block;
+ struct page *page = NULL;
+ struct r5l_meta_header *header;
+ void *meta;
+ struct r5l_meta_payload *payload;
+ struct r5c_load_ctx ctx;
+ int offset;
+
+ ctx.cache = r5l_cache(log);
+ INIT_LIST_HEAD(&ctx.io_ranges);
+ INIT_LIST_HEAD(&ctx.stripes_with_parity);
+ INIT_LIST_HEAD(&ctx.stripes_without_parity);
+ atomic_set(&ctx.io_cnt, 1);
+ init_waitqueue_head(&ctx.io_wq);
+
+again:
+ if (page) {
+ __free_page(page);
+ page = NULL;
+ }
+ header = r5l_read_meta_block(log, last_cp, last_seq, &page);
+ if (!header)
+ goto finish;
+ offset = sizeof(struct r5l_meta_header);
+
+ if (le32_to_cpu(header->type) == R5LOG_TYPE_FLUSH_START) {
+ r5c_add_flush_start(&ctx, (struct r5l_flush_block *)header,
+ last_seq);
+ last_cp = r5l_ring_add(log, last_cp, 1);
+ last_seq++;
+ goto again;
+ }
+
+ if (le32_to_cpu(header->type) == R5LOG_TYPE_FLUSH_END) {
+ r5c_add_flush_end(&ctx, (struct r5l_flush_block *)header,
+ last_seq);
+ last_cp = r5l_ring_add(log, last_cp, 1);
+ last_seq++;
+ goto again;
+ }
+
+ meta = header;
+ payload = meta + offset;
+ payload_block = r5l_ring_add(log, last_cp, 1);
+
+ while (offset < cpu_to_le32(header->meta_size)) {
+ u16 type = le16_to_cpu(payload->payload_type);
+ u16 entries = le32_to_cpu(payload->blocks) >>
+ log->page_block_shift;
+ u32 *checksum;
+
+ checksum = kmalloc(sizeof(u32) * entries, GFP_KERNEL);
+ memcpy(checksum, payload->data_checksum, sizeof(u32) * entries);
+ if (type == R5LOG_PAYLOAD_DATA) {
+ r5c_add_data(&ctx, last_seq,
+ r5l_block_to_sector(log, last_cp),
+ r5l_block_to_sector(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ } else {
+ r5c_add_parity(&ctx, last_seq,
+ r5l_block_to_sector(log, last_cp),
+ r5l_block_to_sector(log, payload_block),
+ le64_to_cpu(payload->location),
+ entries << PAGE_SECTOR_SHIFT, checksum);
+ }
+ payload_block = r5l_ring_add(log, payload_block,
+ r5l_page_blocks(log, entries));
+ offset += sizeof(struct r5l_meta_payload) +
+ entries * sizeof(u32);
+ payload = meta + offset;
+ }
+
+ last_seq++;
+ last_cp = payload_block;
+ goto again;
+finish:
+ if (page)
+ __free_page(page);
+
+ r5c_analyse_log(&ctx);
+
+ r5l_fetch_stripes(&ctx);
+
+ log->seq = last_seq;
+ log->log_start = last_cp;
+ r5c_recover_stripes(&ctx);
+
return 0;
}
--
1.8.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox