* [PATCH 1/1] MDADM:Check mdinfo->reshape_active more times before calling Grow_continue
From: Xiao Ni @ 2016-06-12 9:15 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid
Hi
When reshaping a 3 drives raid5 to 4 drives raid5, there is a chance that
it can't start the reshape. If the disks are not enough to have spaces for
relocating the data_offset, it needs to call start_reshape and then run
mdadm --grow --continue by systemd. But mdadm --grow --continue fails
because it checkes that info->reshape_active is 0.
The info->reshape_active is got from the superblock of underlying devices.
Function start_reshape write reshape to /sys/../sync_action. Before writing
latest superblock to underlying devices, mdadm --grow --continue is called.
There is a chance info->reshape_active is 0. We should wait for superblock
updating more time before calling Grow_continue.
Regards
Xiao
---
Grow.c | 68 +++++++++++++++++++++++++++++++++++++-----------------------------
1 file changed, 38 insertions(+), 30 deletions(-)
diff --git a/Grow.c b/Grow.c
index f184d9c..1b0e4d1 100755
--- a/Grow.c
+++ b/Grow.c
@@ -4788,6 +4788,7 @@ int Grow_continue_command(char *devname, int fd,
dprintf("Grow continue is run for ");
if (st->ss->external == 0) {
int d;
+ int cnt = 5;
dprintf_cont("native array (%s)\n", devname);
if (ioctl(fd, GET_ARRAY_INFO, &array.array) < 0) {
pr_err("%s is not an active md array - aborting\n", devname);
@@ -4799,36 +4800,43 @@ int Grow_continue_command(char *devname, int fd,
* FIXME we should really get what we need from
* sysfs
*/
- for (d = 0; d < MAX_DISKS; d++) {
- mdu_disk_info_t disk;
- char *dv;
- int err;
- disk.number = d;
- if (ioctl(fd, GET_DISK_INFO, &disk) < 0)
- continue;
- if (disk.major == 0 && disk.minor == 0)
- continue;
- if ((disk.state & (1 << MD_DISK_ACTIVE)) == 0)
- continue;
- dv = map_dev(disk.major, disk.minor, 1);
- if (!dv)
- continue;
- fd2 = dev_open(dv, O_RDONLY);
- if (fd2 < 0)
- continue;
- err = st->ss->load_super(st, fd2, NULL);
- close(fd2);
- if (err)
- continue;
- break;
- }
- if (d == MAX_DISKS) {
- pr_err("Unable to load metadata for %s\n",
- devname);
- ret_val = 1;
- goto Grow_continue_command_exit;
- }
- st->ss->getinfo_super(st, content, NULL);
+ do {
+ for (d = 0; d < MAX_DISKS; d++) {
+ mdu_disk_info_t disk;
+ char *dv;
+ int err;
+ disk.number = d;
+ if (ioctl(fd, GET_DISK_INFO, &disk) < 0)
+ continue;
+ if (disk.major == 0 && disk.minor == 0)
+ continue;
+ if ((disk.state & (1 << MD_DISK_ACTIVE)) == 0)
+ continue;
+ dv = map_dev(disk.major, disk.minor, 1);
+ if (!dv)
+ continue;
+ fd2 = dev_open(dv, O_RDONLY);
+ if (fd2 < 0)
+ continue;
+ err = st->ss->load_super(st, fd2, NULL);
+ close(fd2);
+ if (err)
+ continue;
+ break;
+ }
+ if (d == MAX_DISKS) {
+ pr_err("Unable to load metadata for %s\n",
+ devname);
+ ret_val = 1;
+ goto Grow_continue_command_exit;
+ }
+ st->ss->getinfo_super(st, content, NULL);
+
+ if (!content->reshape_active)
+ sleep(3);
+ else
+ break;
+ } while (cnt-- > 0);
} else {
char *container;
--
2.4.3
^ permalink raw reply related
* [PATCH 1/1] MD:Update superblock when err == 0 in size_store
From: Xiao Ni @ 2016-06-12 9:18 UTC (permalink / raw)
To: shli; +Cc: linux-raid
Hi
This is a simple check before updating the superblock. It should update
the superblock when update_size return 0.
Regards
Xiao
---
drivers/md/md.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c068f17..0332a13 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -4181,7 +4181,8 @@ size_store(struct mddev *mddev, const char *buf, size_t len)
return err;
if (mddev->pers) {
err = update_size(mddev, sectors);
- md_update_sb(mddev, 1);
+ if (err == 0)
+ md_update_sb(mddev, 1);
} else {
if (mddev->dev_sectors == 0 ||
mddev->dev_sectors > sectors)
--
2.4.3
^ permalink raw reply related
* Re: [GIT PULL REQUEST] md fixes for 4.3
From: Alexander Lyakas @ 2016-06-13 10:52 UTC (permalink / raw)
To: Neil Brown; +Cc: Jes Sorensen, linux-raid
In-Reply-To: <8737xrhylj.fsf@notabene.neil.brown.name>
Hello Neil,Jes,
Question about the patch
" md/raid1: Avoid raid1 resync getting stuck"
Is this patch relevant for kernel 3.18? In Linus's tree I don't see
this patch being tagged for stable[1].
Thanks,
Alex.
[1]
commit e8ff8bf09ff49733534ff3cee91bde030186055f
Author: Jes Sorensen <Jes.Sorensen@redhat.com>
Date: Wed Sep 16 10:20:05 2015 -0400
md/raid1: Avoid raid1 resync getting stuck
close_sync() needs to set conf->next_resync to a large, but safe value
below MaxSector and use it to determine whether or not to set
start_next_window in wait_barrier()
Solution suggested by Neil Brown.
Reported-by: Nate Dailey <nate.dailey@stratus.com>
Tested-by: Xiao Ni <xni@redhat.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.com>
On Sun, Oct 4, 2015 at 10:39 AM, Neil Brown <neil@brown.name> wrote:
>
> Hi Linus,
> a few md bug fixes.
> Thanks,
> NeilBrown
>
> The following changes since commit bcee19f424a0d8c26ecf2607b73c690802658b29:
>
> Merge branch 'for-4.3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup (2015-09-21 18:26:54 -0700)
>
> are available in the git repository at:
>
> git://neil.brown.name/md tags/md/4.3-fixes
>
> for you to fetch changes up to da6fb7a9e5bd6f04f7e15070f630bdf1ea502841:
>
> md/bitmap: don't pass -1 to bitmap_storage_alloc. (2015-10-02 17:24:13 +1000)
>
> ----------------------------------------------------------------
> Assorted fixes for md in 4.3-rc
>
> Two tagged for -stable
> One is really a cleanup to match and improve kmemcache interface.
>
> ----------------------------------------------------------------
> Jes Sorensen (1):
> md/raid1: Avoid raid1 resync getting stuck
>
> Julia Lawall (1):
> md: drop null test before destroy functions
>
> NeilBrown (4):
> md: wait for pending superblock updates before switching to read-only
> md/raid5: don't index beyond end of array in need_this_block().
> md/raid0: apply base queue limits *before* disk_stack_limits
> md/bitmap: don't pass -1 to bitmap_storage_alloc.
>
> Shaohua Li (2):
> raid5: update analysis state for failed stripe
> md: clear CHANGE_PENDING in readonly array
>
> drivers/md/bitmap.c | 3 ++-
> drivers/md/md.c | 5 +++++
> drivers/md/multipath.c | 3 +--
> drivers/md/raid0.c | 12 ++++++------
> drivers/md/raid1.c | 11 ++++-------
> drivers/md/raid10.c | 9 +++------
> drivers/md/raid5.c | 11 +++++++----
> 7 files changed, 28 insertions(+), 26 deletions(-)
^ permalink raw reply
* RAID1: deadlock between freeze_array and blk plug?
From: Alexander Lyakas @ 2016-06-13 11:02 UTC (permalink / raw)
To: Neil Brown, Jes Sorensen, linux-raid
Hello Neil, Jes,
I wonder if the following deadlock is possible:
- Caller calls blk_start_plug and wants to submit two WRITE bios
- First bio successfully calls wait_barrier() and is appended to
plug->pending list
- Now somebody does freeze_array()
- freeze_array() unconditionally sets:
conf->array_frozen = 1;
and starts waiting for conf->nr_pending to go down
- Second WRITE bio calls wait_barrier, but it will wait for
"!conf->array_frozen" until it can proceed
- Now we have a deadlock: first bio will not be submitted because it
sits on the plug list of the caller, and caller is stuck in
wait_barrier, so it cannot do blk_finish_plug.
I am about to try to reproduce it on kernel 3.18, but looking at the
latest Linus tree, I don't see something preventing this from
happening either. Am I missing something?
Thanks,
Alex.
^ permalink raw reply
* Re: [PATCH] mdadm --detail --scan causes SIGABRT
From: Nikhil Kshirsagar @ 2016-06-13 12:32 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <11EABA5C-C5DA-4498-B1A0-7E1406B323B0@redhat.com>
Hi Jes,
yes those devices exist,
/dev/oczpcie_11_0_ssd: using cached size 6251381753 sectors
The devices were created by the OCZ driver.
Thanks,
Nikhil.
On 06/10/2016 11:42 PM, Nikhil Kshirsagar wrote:
> I think the sosreport should contain the info of the raid devices even though mdadm crashes so we don't see that output in there , there is proc/mdstat and other raid related info there. Let me check.
>
> If nothing else I can ask them for proc/mdstat and mdadm -detail again with the working test binary i sent them.
>
> -Nikhil.
>
>
>> On 10-Jun-2016, at 11:18 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
>>
>> Nikhil Kshirsagar <nkshirsa@redhat.com> writes:
>>> Hi Jes ,
>>>
>>> Would it help to examine the core file ? It's present on the machine
>>> and location specified in the bz comments . That's how I saw the data
>>> structure that had the issue . Indeed there are other device names
>>> where the name does not overflow since they are 32 bytes (which is why
>>> I chose this value) or where the name *does* get truncated. However
>>> this truncation does not seem to happen for de->d_name which is then
>>> copied into dev->sys_name.
>>>
>>> As for allocating an appropriate size on the heap instead of a static
>>> array it does make sense and I can correct the fix to do that but
>>> there are lots of other device names which are static arrays. So which
>>> ones do we change ?
>> This is the tricky part, sys_name is used in a lot of places in
>> different ways.
>>
>> Do you know if they have a /dev/oczpcie_11_0_ssd on the system, and if
>> they do, how did that device get created?
>>
>> Cheers,
>> Jes
^ permalink raw reply
* Re: [GIT PULL REQUEST] md fixes for 4.3
From: Jes Sorensen @ 2016-06-13 13:31 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: Neil Brown, linux-raid
In-Reply-To: <CAGRgLy6f22pJscGab3i28BZGR6QXUP9sq1ygYD2m25yLt-cV5g@mail.gmail.com>
Alexander Lyakas <alex.bolshoy@gmail.com> writes:
> Hello Neil,Jes,
>
> Question about the patch
> " md/raid1: Avoid raid1 resync getting stuck"
>
> Is this patch relevant for kernel 3.18? In Linus's tree I don't see
> this patch being tagged for stable[1].
Hi Alex,
I am not 200% sure as it's been a long time. Looking at it, I think it
falls into the category of has been around forever bugs and the patch
should be relevant for 3.18 too.
Cheers,
Jes
> [1]
> commit e8ff8bf09ff49733534ff3cee91bde030186055f
> Author: Jes Sorensen <Jes.Sorensen@redhat.com>
> Date: Wed Sep 16 10:20:05 2015 -0400
>
> md/raid1: Avoid raid1 resync getting stuck
>
> close_sync() needs to set conf->next_resync to a large, but safe value
> below MaxSector and use it to determine whether or not to set
> start_next_window in wait_barrier()
>
> Solution suggested by Neil Brown.
>
> Reported-by: Nate Dailey <nate.dailey@stratus.com>
> Tested-by: Xiao Ni <xni@redhat.com>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> Signed-off-by: NeilBrown <neilb@suse.com>
>
>
>
>
> On Sun, Oct 4, 2015 at 10:39 AM, Neil Brown <neil@brown.name> wrote:
>>
>> Hi Linus,
>> a few md bug fixes.
>> Thanks,
>> NeilBrown
>>
>> The following changes since commit bcee19f424a0d8c26ecf2607b73c690802658b29:
>>
>> Merge branch 'for-4.3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup (2015-09-21 18:26:54 -0700)
>>
>> are available in the git repository at:
>>
>> git://neil.brown.name/md tags/md/4.3-fixes
>>
>> for you to fetch changes up to da6fb7a9e5bd6f04f7e15070f630bdf1ea502841:
>>
>> md/bitmap: don't pass -1 to bitmap_storage_alloc. (2015-10-02 17:24:13 +1000)
>>
>> ----------------------------------------------------------------
>> Assorted fixes for md in 4.3-rc
>>
>> Two tagged for -stable
>> One is really a cleanup to match and improve kmemcache interface.
>>
>> ----------------------------------------------------------------
>> Jes Sorensen (1):
>> md/raid1: Avoid raid1 resync getting stuck
>>
>> Julia Lawall (1):
>> md: drop null test before destroy functions
>>
>> NeilBrown (4):
>> md: wait for pending superblock updates before switching to read-only
>> md/raid5: don't index beyond end of array in need_this_block().
>> md/raid0: apply base queue limits *before* disk_stack_limits
>> md/bitmap: don't pass -1 to bitmap_storage_alloc.
>>
>> Shaohua Li (2):
>> raid5: update analysis state for failed stripe
>> md: clear CHANGE_PENDING in readonly array
>>
>> drivers/md/bitmap.c | 3 ++-
>> drivers/md/md.c | 5 +++++
>> drivers/md/multipath.c | 3 +--
>> drivers/md/raid0.c | 12 ++++++------
>> drivers/md/raid1.c | 11 ++++-------
>> drivers/md/raid10.c | 9 +++------
>> drivers/md/raid5.c | 11 +++++++----
>> 7 files changed, 28 insertions(+), 26 deletions(-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] raid1/raid10: slow down resync if there is non-resync activity pending
From: Tomasz Majchrzak @ 2016-06-13 13:51 UTC (permalink / raw)
To: linux-raid
A performance drop of mkfs has been observed on RAID10 during resync
since commit 09314799e4f0 ("md: remove 'go_faster' option from
->sync_request()"). Resync sends so many IOs it slows down non-resync
IOs significantly (few times). Add a short delay to a resync. The
previous long sleep (1s) has proven unnecessary, even very short delay
brings performance right.
The change also applied to raid1. The problem has not been observed on
raid1, however it shares barriers code with raid10 so it might be an
issue for some setup too.
Suggested-by: NeilBrown <neilb@suse.com>
Link: http://lkml.kernel.org/r/20160609134555.GA9104@proton.igk.intel.com
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
drivers/md/raid1.c | 7 +++++++
drivers/md/raid10.c | 7 +++++++
2 files changed, 14 insertions(+)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 39fb21e..03c5349 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2535,6 +2535,13 @@ static sector_t raid1_sync_request(struct mddev *mddev, sector_t sector_nr,
return sync_blocks;
}
+ /*
+ * If there is non-resync activity waiting for a turn,
+ * then let it though before starting on this new sync request.
+ */
+ if (conf->nr_waiting)
+ schedule_timeout_uninterruptible(1);
+
/* we are incrementing sector_nr below. To be safe, we check against
* sector_nr + two times RESYNC_SECTORS
*/
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index e3fd725..8a4791e 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2912,6 +2912,13 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
max_sector > (sector_nr | chunk_mask))
max_sector = (sector_nr | chunk_mask) + 1;
+ /*
+ * If there is non-resync activity waiting for a turn,
+ * then let it though before starting on this new sync request.
+ */
+ if (conf->nr_waiting)
+ schedule_timeout_uninterruptible(1);
+
/* Again, very different code for resync and recovery.
* Both must result in an r10bio with a list of bios that
* have bi_end_io, bi_sector, bi_bdev set,
--
1.8.3.1
^ permalink raw reply related
* Re: [GIT PULL REQUEST] md fixes for 4.3
From: Alexander Lyakas @ 2016-06-13 14:54 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Neil Brown, linux-raid
In-Reply-To: <wrfjziqpqkpz.fsf@redhat.com>
Hi Jes,
Thanks for your response. Since you mention that Nate from your team
bisected it to commit 79ef3a8aa1cb1523cc231c9a90a278333c21f761, which
is the raid1 barrier rework, this has not really been around forever.
For example, kernel 3.8 doesn't have this code.
Anyways, Neil mentions that two patches are tagged for stable, but not
sure which ones and which stable kernels he means.
Thanks,
Alex.
On Mon, Jun 13, 2016 at 4:31 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> Alexander Lyakas <alex.bolshoy@gmail.com> writes:
>> Hello Neil,Jes,
>>
>> Question about the patch
>> " md/raid1: Avoid raid1 resync getting stuck"
>>
>> Is this patch relevant for kernel 3.18? In Linus's tree I don't see
>> this patch being tagged for stable[1].
>
> Hi Alex,
>
> I am not 200% sure as it's been a long time. Looking at it, I think it
> falls into the category of has been around forever bugs and the patch
> should be relevant for 3.18 too.
>
> Cheers,
> Jes
>
>> [1]
>> commit e8ff8bf09ff49733534ff3cee91bde030186055f
>> Author: Jes Sorensen <Jes.Sorensen@redhat.com>
>> Date: Wed Sep 16 10:20:05 2015 -0400
>>
>> md/raid1: Avoid raid1 resync getting stuck
>>
>> close_sync() needs to set conf->next_resync to a large, but safe value
>> below MaxSector and use it to determine whether or not to set
>> start_next_window in wait_barrier()
>>
>> Solution suggested by Neil Brown.
>>
>> Reported-by: Nate Dailey <nate.dailey@stratus.com>
>> Tested-by: Xiao Ni <xni@redhat.com>
>> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
>> Signed-off-by: NeilBrown <neilb@suse.com>
>>
>>
>>
>>
>> On Sun, Oct 4, 2015 at 10:39 AM, Neil Brown <neil@brown.name> wrote:
>>>
>>> Hi Linus,
>>> a few md bug fixes.
>>> Thanks,
>>> NeilBrown
>>>
>>> The following changes since commit bcee19f424a0d8c26ecf2607b73c690802658b29:
>>>
>>> Merge branch 'for-4.3-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup (2015-09-21 18:26:54 -0700)
>>>
>>> are available in the git repository at:
>>>
>>> git://neil.brown.name/md tags/md/4.3-fixes
>>>
>>> for you to fetch changes up to da6fb7a9e5bd6f04f7e15070f630bdf1ea502841:
>>>
>>> md/bitmap: don't pass -1 to bitmap_storage_alloc. (2015-10-02 17:24:13 +1000)
>>>
>>> ----------------------------------------------------------------
>>> Assorted fixes for md in 4.3-rc
>>>
>>> Two tagged for -stable
>>> One is really a cleanup to match and improve kmemcache interface.
>>>
>>> ----------------------------------------------------------------
>>> Jes Sorensen (1):
>>> md/raid1: Avoid raid1 resync getting stuck
>>>
>>> Julia Lawall (1):
>>> md: drop null test before destroy functions
>>>
>>> NeilBrown (4):
>>> md: wait for pending superblock updates before switching to read-only
>>> md/raid5: don't index beyond end of array in need_this_block().
>>> md/raid0: apply base queue limits *before* disk_stack_limits
>>> md/bitmap: don't pass -1 to bitmap_storage_alloc.
>>>
>>> Shaohua Li (2):
>>> raid5: update analysis state for failed stripe
>>> md: clear CHANGE_PENDING in readonly array
>>>
>>> drivers/md/bitmap.c | 3 ++-
>>> drivers/md/md.c | 5 +++++
>>> drivers/md/multipath.c | 3 +--
>>> drivers/md/raid0.c | 12 ++++++------
>>> drivers/md/raid1.c | 11 ++++-------
>>> drivers/md/raid10.c | 9 +++------
>>> drivers/md/raid5.c | 11 +++++++----
>>> 7 files changed, 28 insertions(+), 26 deletions(-)
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/1] MD:Update superblock when err == 0 in size_store
From: Shaohua Li @ 2016-06-13 18:47 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <1465723080-9375-1-git-send-email-xni@redhat.com>
On Sun, Jun 12, 2016 at 05:18:00PM +0800, Xiao Ni wrote:
> Hi
>
> This is a simple check before updating the superblock. It should update
> the superblock when update_size return 0.
Applied, thanks!
^ permalink raw reply
* Re: [PATCH] raid1/raid10: slow down resync if there is non-resync activity pending
From: Shaohua Li @ 2016-06-13 18:48 UTC (permalink / raw)
To: Tomasz Majchrzak; +Cc: linux-raid
In-Reply-To: <1465825879-2059-1-git-send-email-tomasz.majchrzak@intel.com>
On Mon, Jun 13, 2016 at 03:51:19PM +0200, Tomasz Majchrzak wrote:
> A performance drop of mkfs has been observed on RAID10 during resync
> since commit 09314799e4f0 ("md: remove 'go_faster' option from
> ->sync_request()"). Resync sends so many IOs it slows down non-resync
> IOs significantly (few times). Add a short delay to a resync. The
> previous long sleep (1s) has proven unnecessary, even very short delay
> brings performance right.
>
> The change also applied to raid1. The problem has not been observed on
> raid1, however it shares barriers code with raid10 so it might be an
> issue for some setup too.
Applied, thanks!
^ permalink raw reply
* [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel
This patch-set adds DAX support to device-mapper dm-linear devices
used by LVM. It works with LVM commands as follows:
- Creation of a logical volume with all DAX capable devices (such
as pmem) sets the logical volume DAX capable as well.
- Once a logical volume is set to DAX capable, the volume may not
be extended with non-DAX capable devices.
The direct_access interface is added to dm and dm-linear to map
a request to a target device.
- Patch 1-2 introduce GENHD_FL_DAX flag to indicate DAX capability.
- Patch 3-4 add direct_access functions to dm and dm-linear.
- Patch 5-6 set GENHD_FL_DAX to dm when all targets are DAX capable.
---
Toshi Kani (6):
1/6 genhd: Add GENHD_FL_DAX to gendisk flags
2/6 block: Check GENHD_FL_DAX for DAX capability
3/6 dm: Add dm_blk_direct_access() for mapped device
4/6 dm-linear: Add linear_direct_access()
5/6 dm, dm-linear: Add dax_supported to dm_target
6/6 dm: Enable DAX support for mapper device
---
drivers/block/brd.c | 2 +-
drivers/md/dm-linear.c | 19 +++++++++++++++++++
drivers/md/dm-table.c | 12 +++++++++---
drivers/md/dm.c | 38 ++++++++++++++++++++++++++++++++++++--
drivers/md/dm.h | 7 +++++++
drivers/nvdimm/pmem.c | 2 +-
drivers/s390/block/dcssblk.c | 1 +
fs/block_dev.c | 5 +++--
include/linux/device-mapper.h | 15 +++++++++++++++
include/linux/genhd.h | 2 +-
10 files changed, 93 insertions(+), 10 deletions(-)
^ permalink raw reply
* [PATCH 1/6] genhd: Add GENHD_FL_DAX to gendisk flags
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel, Martin Schwidefsky, Heiko Carstens,
linux-s390
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Currently, presence of direct_access() in block_device_operations
indicates support of DAX on its block device. Because
block_device_operations is instantiated with 'const', this DAX
capablity may not be enabled conditinally.
In preparation for supporting DAX to device-mapper devices, add
GENHD_FL_DAX to gendisk flags to indicate that this block device
supports DAX. This will allow to set the DAX capability based on
how mapped device is composed.
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: <linux-s390@vger.kernel.org>
---
drivers/block/brd.c | 2 +-
drivers/nvdimm/pmem.c | 2 +-
drivers/s390/block/dcssblk.c | 1 +
include/linux/genhd.h | 2 +-
4 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index c04bd9b..43a6765 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -518,7 +518,7 @@ static struct brd_device *brd_alloc(int i)
disk->fops = &brd_fops;
disk->private_data = brd;
disk->queue = brd->brd_queue;
- disk->flags = GENHD_FL_EXT_DEVT;
+ disk->flags = GENHD_FL_EXT_DEVT | GENHD_FL_DAX;
sprintf(disk->disk_name, "ram%d", i);
set_capacity(disk, rd_size * 2);
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 608fc44..3847449 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -295,7 +295,7 @@ static int pmem_attach_disk(struct device *dev,
disk->fops = &pmem_fops;
disk->queue = q;
- disk->flags = GENHD_FL_EXT_DEVT;
+ disk->flags = GENHD_FL_EXT_DEVT | GENHD_FL_DAX;
nvdimm_namespace_disk_name(ndns, disk->disk_name);
disk->driverfs_dev = dev;
set_capacity(disk, (pmem->size - pmem->pfn_pad - pmem->data_offset)
diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c
index bed53c4..60b1857 100644
--- a/drivers/s390/block/dcssblk.c
+++ b/drivers/s390/block/dcssblk.c
@@ -616,6 +616,7 @@ dcssblk_add_store(struct device *dev, struct device_attribute *attr, const char
dev_info->gd->queue = dev_info->dcssblk_queue;
dev_info->gd->private_data = dev_info;
dev_info->gd->driverfs_dev = &dev_info->dev;
+ dev_info->gd->flags = GENHD_FL_DAX;
blk_queue_make_request(dev_info->dcssblk_queue, dcssblk_make_request);
blk_queue_logical_block_size(dev_info->dcssblk_queue, 4096);
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 359a8e4..9dc3a3d 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -131,7 +131,7 @@ struct hd_struct {
};
#define GENHD_FL_REMOVABLE 1
-/* 2 is unused */
+#define GENHD_FL_DAX 2
#define GENHD_FL_MEDIA_CHANGE_NOTIFY 4
#define GENHD_FL_CD 8
#define GENHD_FL_UP 16
^ permalink raw reply related
* [PATCH 2/6] block: Check GENHD_FL_DAX for DAX capability
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel, linux-fsdevel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Now that GENHD_FL_DAX is set to all drivers supporting DAX,
change bdev_direct_access() and __blkdev_get() to check this
GENHD_FL_DAX flag.
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: <linux-fsdevel@vger.kernel.org>
---
fs/block_dev.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 71ccab1..61935ee 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -493,7 +493,7 @@ long bdev_direct_access(struct block_device *bdev, struct blk_dax_ctl *dax)
if (size < 0)
return size;
- if (!ops->direct_access)
+ if (!(bdev->bd_disk->flags & GENHD_FL_DAX) || !ops->direct_access)
return -EOPNOTSUPP;
if ((sector + DIV_ROUND_UP(size, 512)) >
part_nr_sects_read(bdev->bd_part))
@@ -1287,7 +1287,8 @@ static int __blkdev_get(struct block_device *bdev, fmode_t mode, int for_part)
bdev->bd_disk = disk;
bdev->bd_queue = disk->queue;
bdev->bd_contains = bdev;
- if (IS_ENABLED(CONFIG_BLK_DEV_DAX) && disk->fops->direct_access)
+ if (IS_ENABLED(CONFIG_BLK_DEV_DAX) &&
+ disk->flags & GENHD_FL_DAX)
bdev->bd_inode->i_flags = S_DAX;
else
bdev->bd_inode->i_flags = 0;
^ permalink raw reply related
* [PATCH 3/6] dm: Add dm_blk_direct_access() for mapped device
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Change mapped device to implement direct_access function,
dm_blk_direct_access(), which calls a target direct_access
function. 'struct target_type' is extended to have target
direct_access interface. This function limits direct
accessible size to the dm_target's limit with max_io_len().
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
---
drivers/md/dm.c | 29 +++++++++++++++++++++++++++++
include/linux/device-mapper.h | 10 ++++++++++
2 files changed, 39 insertions(+)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 1b2f962..6e9f958 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1473,6 +1473,34 @@ int dm_set_target_max_io_len(struct dm_target *ti, sector_t len)
}
EXPORT_SYMBOL_GPL(dm_set_target_max_io_len);
+static long dm_blk_direct_access(struct block_device *bdev, sector_t sector,
+ void __pmem **kaddr, pfn_t *pfn, long size)
+{
+ struct mapped_device *md = bdev->bd_disk->private_data;
+ struct dm_table *map;
+ struct dm_target *ti;
+ int srcu_idx;
+ long len, ret = -EIO;
+
+ map = dm_get_live_table(md, &srcu_idx);
+ if (!map)
+ return ret;
+
+ ti = dm_table_find_target(map, sector);
+ if (!dm_target_is_valid(ti))
+ goto out;
+
+ len = max_io_len(sector, ti) << SECTOR_SHIFT;
+ size = min(len, size);
+
+ if (ti->type->direct_access)
+ ret = ti->type->direct_access(ti, sector, kaddr, pfn, size);
+
+out:
+ dm_put_live_table(md, srcu_idx);
+ return min(ret, size);
+}
+
/*
* A target may call dm_accept_partial_bio only from the map routine. It is
* allowed for all bio types except REQ_FLUSH.
@@ -3721,6 +3749,7 @@ static const struct block_device_operations dm_blk_dops = {
.open = dm_blk_open,
.release = dm_blk_close,
.ioctl = dm_blk_ioctl,
+ .direct_access = dm_blk_direct_access,
.getgeo = dm_blk_getgeo,
.pr_ops = &dm_pr_ops,
.owner = THIS_MODULE
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index 0830c9e..16e6c8c 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -116,6 +116,15 @@ typedef void (*dm_io_hints_fn) (struct dm_target *ti,
*/
typedef int (*dm_busy_fn) (struct dm_target *ti);
+/*
+ * Returns:
+ * < 0 : error
+ * >= 0 : the number of bytes accessible at the address
+ */
+typedef long (*dm_direct_access_fn) (struct dm_target *ti, sector_t sector,
+ void __pmem **kaddr, pfn_t *pfn,
+ long size);
+
void dm_error(const char *message);
struct dm_dev {
@@ -162,6 +171,7 @@ struct target_type {
dm_busy_fn busy;
dm_iterate_devices_fn iterate_devices;
dm_io_hints_fn io_hints;
+ dm_direct_access_fn direct_access;
/* For internal device-mapper use. */
struct list_head list;
^ permalink raw reply related
* [PATCH 4/6] dm-linear: Add linear_direct_access()
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Change dm-linear to implement direct_access function,
linear_direct_access(), which maps sector and calls
direct_access function of its target device.
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
---
drivers/md/dm-linear.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
index 05c35aa..49bd7d2 100644
--- a/drivers/md/dm-linear.c
+++ b/drivers/md/dm-linear.c
@@ -141,6 +141,22 @@ static int linear_iterate_devices(struct dm_target *ti,
return fn(ti, lc->dev, lc->start, ti->len, data);
}
+static long linear_direct_access(struct dm_target *ti, sector_t sector,
+ void __pmem **kaddr, pfn_t *pfn, long size)
+{
+ struct linear_c *lc;
+ struct block_device *tbdev;
+ const struct block_device_operations *tops;
+ sector_t tsector;
+
+ lc = ti->private;
+ tbdev = lc->dev->bdev;
+ tops = tbdev->bd_disk->fops;
+ tsector = linear_map_sector(ti, sector);
+
+ return tops->direct_access(tbdev, tsector, kaddr, pfn, size);
+}
+
static struct target_type linear_target = {
.name = "linear",
.version = {1, 2, 1},
@@ -151,6 +167,7 @@ static struct target_type linear_target = {
.status = linear_status,
.prepare_ioctl = linear_prepare_ioctl,
.iterate_devices = linear_iterate_devices,
+ .direct_access = linear_direct_access,
};
int __init dm_linear_init(void)
^ permalink raw reply related
* [PATCH 5/6] dm, dm-linear: Add dax_supported to dm_target
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Extend 'struct dm_target' to have dax_supported bit, which allows
dm-table to check if a dm-target supports DAX.
Change dm-linear to set this bit when its target physical device
supports DAX.
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
---
drivers/md/dm-linear.c | 2 ++
include/linux/device-mapper.h | 5 +++++
2 files changed, 7 insertions(+)
diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
index 49bd7d2..6fdbbc8 100644
--- a/drivers/md/dm-linear.c
+++ b/drivers/md/dm-linear.c
@@ -59,6 +59,8 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv)
ti->num_flush_bios = 1;
ti->num_discard_bios = 1;
ti->num_write_same_bios = 1;
+ if (lc->dev->bdev->bd_disk->flags & GENHD_FL_DAX)
+ ti->dax_supported = 1;
ti->private = lc;
return 0;
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index 16e6c8c..9b72989 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -290,6 +290,11 @@ struct dm_target {
* Set if this target does not return zeroes on discarded blocks.
*/
bool discard_zeroes_data_unsupported:1;
+
+ /*
+ * Set if the target supports DAX (direct access).
+ */
+ bool dax_supported:1;
};
/* Each target can link one of these into the table */
^ permalink raw reply related
* [PATCH 6/6] dm: Enable DAX support for mapper device
From: Toshi Kani @ 2016-06-13 22:21 UTC (permalink / raw)
To: agk, snitzer, dan.j.williams
Cc: ross.zwisler, viro, axboe, toshi.kani, linux-nvdimm, dm-devel,
linux-raid, linux-kernel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Add a new dm type, DM_TYPE_DAX_BIO_BASED, which indicates
that mapped device supports DAX and is bio based. This new
type is used to assure that all target devices have DAX support
and remain that way after GENHD_FL_DAX is set to mapped device.
At initial table load, GENHD_FL_DAX is set to mapped device
when setting DM_TYPE_DAX_BIO_BASED to the type. Any subsequent
table load to the mapped device must have the same type, or else
it fails per the check in table_load().
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
---
drivers/md/dm-table.c | 12 +++++++++---
drivers/md/dm.c | 9 +++++++--
drivers/md/dm.h | 7 +++++++
3 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 626a5ec..81138e7 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -833,7 +833,7 @@ static bool __table_type_request_based(unsigned table_type)
static int dm_table_set_type(struct dm_table *t)
{
unsigned i;
- unsigned bio_based = 0, request_based = 0, hybrid = 0;
+ unsigned bio_based = 0, request_based = 0, hybrid = 0, num_dax = 0;
bool use_blk_mq = false;
struct dm_target *tgt;
struct dm_dev_internal *dd;
@@ -849,6 +849,9 @@ static int dm_table_set_type(struct dm_table *t)
else
bio_based = 1;
+ if (tgt->dax_supported)
+ num_dax++;
+
if (bio_based && request_based) {
DMWARN("Inconsistent table: different target types"
" can't be mixed up");
@@ -870,7 +873,10 @@ static int dm_table_set_type(struct dm_table *t)
if (bio_based) {
/* We must use this table as bio-based */
- t->type = DM_TYPE_BIO_BASED;
+ if (num_dax && num_dax == t->num_targets)
+ t->type = DM_TYPE_DAX_BIO_BASED;
+ else
+ t->type = DM_TYPE_BIO_BASED;
return 0;
}
@@ -978,7 +984,7 @@ static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *
return -EINVAL;
}
- if (type == DM_TYPE_BIO_BASED)
+ if (dm_type_bio_based(type))
for (i = 0; i < t->num_targets; i++) {
tgt = t->targets + i;
per_io_data_size = max(per_io_data_size, tgt->per_io_data_size);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 6e9f958..41b8912 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2492,7 +2492,7 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
if (md->bs) {
/* The md already has necessary mempools. */
- if (dm_table_get_type(t) == DM_TYPE_BIO_BASED) {
+ if (dm_type_bio_based(dm_table_get_type(t))) {
/*
* Reload bioset because front_pad may have changed
* because a different table was loaded.
@@ -2659,6 +2659,9 @@ void dm_set_md_type(struct mapped_device *md, unsigned type)
{
BUG_ON(!mutex_is_locked(&md->type_lock));
md->type = type;
+
+ if (type == DM_TYPE_DAX_BIO_BASED)
+ dm_disk(md)->flags |= GENHD_FL_DAX;
}
unsigned dm_get_md_type(struct mapped_device *md)
@@ -2837,7 +2840,7 @@ out_kfree_tag_set:
static unsigned filter_md_type(unsigned type, struct mapped_device *md)
{
- if (type == DM_TYPE_BIO_BASED)
+ if (dm_type_bio_based(type))
return type;
return !md->use_blk_mq ? DM_TYPE_REQUEST_BASED : DM_TYPE_MQ_REQUEST_BASED;
@@ -2867,6 +2870,7 @@ int dm_setup_md_queue(struct mapped_device *md, struct dm_table *t)
}
break;
case DM_TYPE_BIO_BASED:
+ case DM_TYPE_DAX_BIO_BASED:
dm_init_normal_md_queue(md);
blk_queue_make_request(md->queue, dm_make_request);
/*
@@ -3573,6 +3577,7 @@ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, unsigned t
switch (type) {
case DM_TYPE_BIO_BASED:
+ case DM_TYPE_DAX_BIO_BASED:
cachep = _io_cache;
pool_size = dm_get_reserved_bio_based_ios();
front_pad = roundup(per_io_data_size, __alignof__(struct dm_target_io)) + offsetof(struct dm_target_io, clone);
diff --git a/drivers/md/dm.h b/drivers/md/dm.h
index 13a758e..6d22667 100644
--- a/drivers/md/dm.h
+++ b/drivers/md/dm.h
@@ -39,6 +39,13 @@
#define DM_TYPE_BIO_BASED 1
#define DM_TYPE_REQUEST_BASED 2
#define DM_TYPE_MQ_REQUEST_BASED 3
+#define DM_TYPE_DAX_BIO_BASED 4
+
+/*
+ * To check whether the DM type is bio-based or not.
+ */
+#define dm_type_bio_based(type) (((type) == DM_TYPE_BIO_BASED) || \
+ ((type) == DM_TYPE_DAX_BIO_BASED))
/*
* List of devices that a metadevice uses and should open/close.
^ permalink raw reply related
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-13 22:57 UTC (permalink / raw)
To: Toshi Kani
Cc: agk, dan.j.williams, ross.zwisler, viro, axboe, linux-nvdimm,
dm-devel, linux-raid, linux-kernel
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
On Mon, Jun 13 2016 at 6:21pm -0400,
Toshi Kani <toshi.kani@hpe.com> wrote:
> This patch-set adds DAX support to device-mapper dm-linear devices
> used by LVM. It works with LVM commands as follows:
> - Creation of a logical volume with all DAX capable devices (such
> as pmem) sets the logical volume DAX capable as well.
> - Once a logical volume is set to DAX capable, the volume may not
> be extended with non-DAX capable devices.
>
> The direct_access interface is added to dm and dm-linear to map
> a request to a target device.
>
> - Patch 1-2 introduce GENHD_FL_DAX flag to indicate DAX capability.
> - Patch 3-4 add direct_access functions to dm and dm-linear.
> - Patch 5-6 set GENHD_FL_DAX to dm when all targets are DAX capable.
>
> ---
> Toshi Kani (6):
> 1/6 genhd: Add GENHD_FL_DAX to gendisk flags
> 2/6 block: Check GENHD_FL_DAX for DAX capability
> 3/6 dm: Add dm_blk_direct_access() for mapped device
> 4/6 dm-linear: Add linear_direct_access()
> 5/6 dm, dm-linear: Add dax_supported to dm_target
> 6/6 dm: Enable DAX support for mapper device
Thanks a lot for doing this. I recently added it to my TODO so your
patches come at a great time.
I'll try to get to reviewing/testing your work by the end of this week.
Mike
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Dan Williams @ 2016-06-13 23:18 UTC (permalink / raw)
To: Toshi Kani
Cc: Alasdair Kergon, Mike Snitzer, Ross Zwisler, Al Viro, Jens Axboe,
linux-nvdimm@lists.01.org, dm-devel, linux-raid,
linux-kernel@vger.kernel.org
In-Reply-To: <1465856497-19698-1-git-send-email-toshi.kani@hpe.com>
Thanks Toshi!
On Mon, Jun 13, 2016 at 3:21 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
> This patch-set adds DAX support to device-mapper dm-linear devices
> used by LVM. It works with LVM commands as follows:
> - Creation of a logical volume with all DAX capable devices (such
> as pmem) sets the logical volume DAX capable as well.
> - Once a logical volume is set to DAX capable, the volume may not
> be extended with non-DAX capable devices.
I don't mind this, but it seems a policy decision that the kernel does
not need to make. A sufficiently sophisticated user could cope with
DAX being available at varying LBAs. Would it be sufficient to move
this policy decision to userspace tooling?
> The direct_access interface is added to dm and dm-linear to map
> a request to a target device.
I had dm-linear and md-raid0 support on my list of things to look at,
did you have raid0 in your plans?
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-13 23:59 UTC (permalink / raw)
To: dan.j.williams@intel.com
Cc: linux-kernel@vger.kernel.org, agk@redhat.com,
linux-raid@vger.kernel.org, snitzer@redhat.com,
viro@zeniv.linux.org.uk, axboe@kernel.dk,
linux-nvdimm@lists.01.org, ross.zwisler@linux.intel.com,
dm-devel@redhat.com
In-Reply-To: <CAPcyv4jdM1phR=kGoP2-7tfsVvbNe2C6hHNS5TD28ALGZQQTSw@mail.gmail.com>
On Mon, 2016-06-13 at 16:18 -0700, Dan Williams wrote:
> Thanks Toshi!
>
> On Mon, Jun 13, 2016 at 3:21 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
> >
> > This patch-set adds DAX support to device-mapper dm-linear devices
> > used by LVM. It works with LVM commands as follows:
> > - Creation of a logical volume with all DAX capable devices (such
> > as pmem) sets the logical volume DAX capable as well.
> > - Once a logical volume is set to DAX capable, the volume may not
> > be extended with non-DAX capable devices.
>
> I don't mind this, but it seems a policy decision that the kernel does
> not need to make. A sufficiently sophisticated user could cope with
> DAX being available at varying LBAs. Would it be sufficient to move
> this policy decision to userspace tooling?
I think this is a kernel restriction. When a block device is declared as
DAX capable, it should mean that the whole device is DAX capable. So, I
think we need to assure the same to a mapped device.
In LVM level, a volume group may contain both DAX and non-DAX capable
devices. There is no restriction for creating/extending a volume group.
> > The direct_access interface is added to dm and dm-linear to map
> > a request to a target device.
>
> I had dm-linear and md-raid0 support on my list of things to look at,
> did you have raid0 in your plans?
Yes, I hope to extend further and raid0 is a good candidate.
Thanks,
-Toshi
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Dan Williams @ 2016-06-14 0:02 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: linux-kernel@vger.kernel.org, agk@redhat.com,
linux-raid@vger.kernel.org, snitzer@redhat.com,
viro@zeniv.linux.org.uk, axboe@kernel.dk,
linux-nvdimm@lists.01.org, ross.zwisler@linux.intel.com,
dm-devel@redhat.com
In-Reply-To: <1465861755.3504.185.camel@hpe.com>
On Mon, Jun 13, 2016 at 4:59 PM, Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> On Mon, 2016-06-13 at 16:18 -0700, Dan Williams wrote:
>> Thanks Toshi!
>>
>> On Mon, Jun 13, 2016 at 3:21 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
>> >
>> > This patch-set adds DAX support to device-mapper dm-linear devices
>> > used by LVM. It works with LVM commands as follows:
>> > - Creation of a logical volume with all DAX capable devices (such
>> > as pmem) sets the logical volume DAX capable as well.
>> > - Once a logical volume is set to DAX capable, the volume may not
>> > be extended with non-DAX capable devices.
>>
>> I don't mind this, but it seems a policy decision that the kernel does
>> not need to make. A sufficiently sophisticated user could cope with
>> DAX being available at varying LBAs. Would it be sufficient to move
>> this policy decision to userspace tooling?
>
> I think this is a kernel restriction. When a block device is declared as
> DAX capable, it should mean that the whole device is DAX capable. So, I
> think we need to assure the same to a mapped device.
Hmm, but we already violate this with badblocks. The device is DAX
capable, but certain LBAs will return an error if direct_access is
attempted.
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Dan Williams @ 2016-06-14 7:30 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: linux-kernel@vger.kernel.org, agk@redhat.com,
linux-raid@vger.kernel.org, snitzer@redhat.com,
viro@zeniv.linux.org.uk, axboe@kernel.dk,
linux-nvdimm@lists.01.org, ross.zwisler@linux.intel.com,
dm-devel@redhat.com
In-Reply-To: <CAPcyv4hnKRc6qrPPuKo1r3=YmuAAaeSSv7xBOpDMAy5UuUhoHw@mail.gmail.com>
On Mon, Jun 13, 2016 at 5:02 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Mon, Jun 13, 2016 at 4:59 PM, Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
>> On Mon, 2016-06-13 at 16:18 -0700, Dan Williams wrote:
>>> Thanks Toshi!
>>>
>>> On Mon, Jun 13, 2016 at 3:21 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
>>> >
>>> > This patch-set adds DAX support to device-mapper dm-linear devices
>>> > used by LVM. It works with LVM commands as follows:
>>> > - Creation of a logical volume with all DAX capable devices (such
>>> > as pmem) sets the logical volume DAX capable as well.
>>> > - Once a logical volume is set to DAX capable, the volume may not
>>> > be extended with non-DAX capable devices.
>>>
>>> I don't mind this, but it seems a policy decision that the kernel does
>>> not need to make. A sufficiently sophisticated user could cope with
>>> DAX being available at varying LBAs. Would it be sufficient to move
>>> this policy decision to userspace tooling?
>>
>> I think this is a kernel restriction. When a block device is declared as
>> DAX capable, it should mean that the whole device is DAX capable. So, I
>> think we need to assure the same to a mapped device.
>
> Hmm, but we already violate this with badblocks. The device is DAX
> capable, but certain LBAs will return an error if direct_access is
> attempted.
Nevermind, for this to be useful we would need to fallback to regular
mmap for a portion of the linear span. That's different than the
badblocks case.
^ permalink raw reply
* Re: [PATCH 1/4] raid1: Add a filed array_frozen to indicate whether raid in freeze state.
From: Alexander Lyakas @ 2016-06-14 9:33 UTC (permalink / raw)
To: majianpeng, linux-raid, NeilBrown; +Cc: Jes Sorensen
In-Reply-To: <201308281940120331425@gmail.com>
Hello Jianpeng Ma, Neil,
This commit seems to introduce a severe regression, leading to IO
getting stuck on the whole array.
Previously, the condition of wait_barrier() was:
wait_event_lock_irq(conf->wait_barrier,
!conf->barrier ||
(conf->nr_pending &&
current->bio_list &&
!bio_list_empty(current->bio_list)),
conf->resync_lock);
This means that if current->bio_list is not empty, we will not wait
for the barrier to drop.
But right now the condition is:
wait_event_lock_irq(conf->wait_barrier,
!conf->array_frozen &&
(!conf->barrier ||
(conf->nr_pending &&
current->bio_list &&
!bio_list_empty(current->bio_list))),
conf->resync_lock);
This means that if somebody calls freeze_array (for example,
md_do_sync), then freeze_array unconditionally sets:
conf->array_frozen = 1;
And then if somebody calls wait_barrier and its current->bio_list is
not empty, it will still wait, because array_frozen=1. But
freeze_array is waiting for the bio complete, but this bio sits in
current->bio_list and will never get submitted, because we are waiting
in wait_barrier. DEADLOCK.
For me it happens on kernel 3.18, when raid1.c::make_request() submits
a READ bio, which has raid1_end_read_request completion routine. But
this completion routine never gets called, and I also confirmed that
no IO is stuck on lower levels.
Additional notes:
# Please see my email titled "RAID1: deadlock between freeze_array and
blk plug?" in http://permalink.gmane.org/gmane.linux.raid/52693. The
problem described there also stems from the fact that now
freeze_array() sets array_frozen=1 unconditionally. And wait_barrier()
will always wait if array_frozen!=0.
# Also now freeze_array() is non-reentrant. Meaning that if two
threads call freeze_array in parallel, it will not work, because
array_frozen can only be 1 or 0. Example, when freeze_array can be
called by two threads:
- We have 3-way RAID1 with one drive missing. We want to recover this drive.
- we have an in-flight READ request
- md_do_sync starts recovery and calls mddev->pers->quiesce(mddev, 1),
which waits for the in-flight READ to complete
- meanwhile the READ bio fails
- so raid1d calls handle_read_error, which calls freeze_array
Result: we have two freeze_array calls in parallel. But we don't know
to account for them properly, because array_frozen can only be 1 or 0.
Please advise how to fix this problem. Please kindly note that the fix
should apply to kernel 3.18; which is a longterm kernel that we use.
Thanks,
Alex.
On Wed, Aug 28, 2013 at 2:40 PM, majianpeng <majianpeng@gmail.com> wrote:
> Because the following patch will rewrite the contend between nornal IO
> and resync IO. So we used a parameter to indicate whether raid is in freeze
> array.
>
> Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
> ---
> drivers/md/raid1.c | 15 +++++++--------
> drivers/md/raid1.h | 1 +
> 2 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index d60412c..92a6d29 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -829,6 +829,7 @@ static void raise_barrier(struct r1conf *conf)
>
> /* Now wait for all pending IO to complete */
> wait_event_lock_irq(conf->wait_barrier,
> + !conf->array_frozen &&
> !conf->nr_pending && conf->barrier < RESYNC_DEPTH,
> conf->resync_lock);
>
> @@ -860,10 +861,11 @@ static void wait_barrier(struct r1conf *conf)
> * count down.
> */
> wait_event_lock_irq(conf->wait_barrier,
> - !conf->barrier ||
> + !conf->array_frozen &&
> + (!conf->barrier ||
> (conf->nr_pending &&
> current->bio_list &&
> - !bio_list_empty(current->bio_list)),
> + !bio_list_empty(current->bio_list))),
> conf->resync_lock);
> conf->nr_waiting--;
> }
> @@ -884,8 +886,7 @@ static void freeze_array(struct r1conf *conf, int extra)
> {
> /* stop syncio and normal IO and wait for everything to
> * go quite.
> - * We increment barrier and nr_waiting, and then
> - * wait until nr_pending match nr_queued+extra
> + * We wait until nr_pending match nr_queued+extra
> * This is called in the context of one normal IO request
> * that has failed. Thus any sync request that might be pending
> * will be blocked by nr_pending, and we need to wait for
> @@ -895,8 +896,7 @@ static void freeze_array(struct r1conf *conf, int extra)
> * we continue.
> */
> spin_lock_irq(&conf->resync_lock);
> - conf->barrier++;
> - conf->nr_waiting++;
> + conf->array_frozen = 1;
> wait_event_lock_irq_cmd(conf->wait_barrier,
> conf->nr_pending == conf->nr_queued+extra,
> conf->resync_lock,
> @@ -907,8 +907,7 @@ static void unfreeze_array(struct r1conf *conf)
> {
> /* reverse the effect of the freeze */
> spin_lock_irq(&conf->resync_lock);
> - conf->barrier--;
> - conf->nr_waiting--;
> + conf->array_frozen = 0;
> wake_up(&conf->wait_barrier);
> spin_unlock_irq(&conf->resync_lock);
> }
> diff --git a/drivers/md/raid1.h b/drivers/md/raid1.h
> index 0ff3715..331a98a 100644
> --- a/drivers/md/raid1.h
> +++ b/drivers/md/raid1.h
> @@ -65,6 +65,7 @@ struct r1conf {
> int nr_waiting;
> int nr_queued;
> int barrier;
> + int array_frozen;
>
> /* Set to 1 if a full sync is needed, (fresh device added).
> * Cleared when a sync completes.
> --
> 1.8.1.2
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Jeff Moyer @ 2016-06-14 13:50 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: dan.j.williams@intel.com, linux-kernel@vger.kernel.org,
agk@redhat.com, linux-raid@vger.kernel.org, snitzer@redhat.com,
viro@zeniv.linux.org.uk, axboe@kernel.dk,
linux-nvdimm@lists.01.org, ross.zwisler@linux.intel.com,
dm-devel@redhat.com
In-Reply-To: <1465861755.3504.185.camel@hpe.com>
"Kani, Toshimitsu" <toshi.kani@hpe.com> writes:
>> I had dm-linear and md-raid0 support on my list of things to look at,
>> did you have raid0 in your plans?
>
> Yes, I hope to extend further and raid0 is a good candidate.
dm-flakey would allow more xfstests test cases to run. I'd say that's
more important than linear or raid0. ;-)
Also, the next step in this work is to then decide how to determine on
what numa node an LBA resides. We had discussed this at a prior
plumbers conference, and I think the consensus was to use xattrs.
Toshi, do you also plan to do that work?
Cheers,
Jeff
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-14 15:41 UTC (permalink / raw)
To: Jeff Moyer
Cc: Kani, Toshimitsu, axboe@kernel.dk, linux-nvdimm@lists.01.org,
linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org,
dm-devel@redhat.com, viro@zeniv.linux.org.uk,
dan.j.williams@intel.com, ross.zwisler@linux.intel.com,
agk@redhat.com
In-Reply-To: <x49fusf282h.fsf@segfault.boston.devel.redhat.com>
On Tue, Jun 14 2016 at 9:50am -0400,
Jeff Moyer <jmoyer@redhat.com> wrote:
> "Kani, Toshimitsu" <toshi.kani@hpe.com> writes:
>
> >> I had dm-linear and md-raid0 support on my list of things to look at,
> >> did you have raid0 in your plans?
> >
> > Yes, I hope to extend further and raid0 is a good candidate.
>
> dm-flakey would allow more xfstests test cases to run. I'd say that's
> more important than linear or raid0. ;-)
Regardless of which target(s) grow DAX support the most pressing initial
concern is getting the DM device stacking correct. And verifying that
IO that cross pmem device boundaries are being properly split by DM
core (via drivers/md/dm.c:__split_and_process_non_flush()'s call to
max_io_len).
My hope is to nail down the DM core and its dependencies in block etc.
Doing so in terms of dm-linear doesn't seem like wasted effort
considering you told me it'd be useful to have for pmem devices.
> Also, the next step in this work is to then decide how to determine on
> what numa node an LBA resides. We had discussed this at a prior
> plumbers conference, and I think the consensus was to use xattrs.
> Toshi, do you also plan to do that work?
How does the associated NUMA node relate to this? Does the
DM requests_queue need to be setup to only allocate from the NUMA node
the pmem device is attached to? I recently added support for this to
DM. But there will likely be some code need to propagate the NUMA node
id accordingly.
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox