* [RESEND PATCH 1/3] bcache: Remove redundant parameter for cache_alloc()
From: Yijing Wang @ 2016-06-22 2:10 UTC (permalink / raw)
To: axboe, Kent Overstreet
Cc: Eric Wheeler, Coly Li, linux-bcache, linux-raid, linux-kernel,
Yijing Wang
Cache_sb is not used in cache_alloc, and we have copied
sb info to cache->sb already, remove it.
Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
drivers/md/bcache/super.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index f5dbb4e..aecaace 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -1803,7 +1803,7 @@ void bch_cache_release(struct kobject *kobj)
module_put(THIS_MODULE);
}
-static int cache_alloc(struct cache_sb *sb, struct cache *ca)
+static int cache_alloc(struct cache *ca)
{
size_t free;
struct bucket *b;
@@ -1858,7 +1858,7 @@ static int register_cache(struct cache_sb *sb, struct page *sb_page,
if (blk_queue_discard(bdev_get_queue(ca->bdev)))
ca->discard = CACHE_DISCARD(&ca->sb);
- ret = cache_alloc(sb, ca);
+ ret = cache_alloc(ca);
if (ret != 0)
goto err;
--
1.7.1
^ permalink raw reply related
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-21 18:17 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
axboe-b10kYP2dOMg@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466523280.3504.262.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jun 21 2016 at 11:44am -0400,
Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
> > On Mon, Jun 20 2016 at 6:22pm -0400,
> > Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > >
> > > On Mon, Jun 20 2016 at 5:28pm -0400,
> > > Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> > >
> :
> > > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
> > > branch.
> > >
> > > No longer seeing any corruption in my test that was using partitions to
> > > span pmem devices with a dm-linear device.
> > >
> > > Jens, any chance you'd be open to picking up the first 2 patches in this
> > > series? Or would you like to see them folded or something different?
> >
> > I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
> > rather than establish GENHD_FL_DAX on the genhd?
> >
> > It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
> > check for a queue flag.
>
> I think GENHD_FL_DAX is more appropriate since DAX does not use a request
> queue, except for protecting the underlining device being disabled while
> direct_access() is called (b2e0d1625e19).
The devices in question have a request_queue. All bio-based device have
a request_queue.
I don't have a big problem with GENHD_FL_DAX. Just wanted to point out
that such block device capabilities are generally advertised in terms of
a QUEUE_FLAG.
> About protecting direct_access, this patch assumes that the underlining
> device cannot be disabled until dtr() is called. Is this correct? If not,
> I will need to call dax_map_atomic().
One of the big design considerations for DM that a DM device can be
suspended (with or without flush) and any new IO will be blocked until
the DM device is resumed.
So ideally DM should be able to have the same capability even if using
DAX.
But that is different than what commit b2e0d1625e19 is addressing. For
DM, I wouldn't think you'd need the extra protections that
dax_map_atomic() is providing given that the underlying block device
lifetime is managed via DM core's dm_get_device/dm_put_device (see also:
dm.c:open_table_device/close_table_device).
^ permalink raw reply
* Making spare active without sync?
From: Wakko Warner @ 2016-06-21 17:47 UTC (permalink / raw)
To: linux-raid
I have an array that a drive dropped out due to a write error. I don't know
if the drive is still good or not. Another drive also has some read errors.
I zero'd the superblock of the member device and added it back in. When it
hit the errors, of course it kicks both out of the array. I've already
recovered most of the data by forcing to assemble with the members that were
active (including the member that failed 2nd). The data that is in the bad
spot isn't important and is recoverable by other means, but I'd like to try
this anyway.
I've already created a dm snapshot target for these 4 drives. What I'd like
to do is make the drive that I zero'd the superblock on active without doing
any sync/resync. Is this possible to do? Or do I need to recreate the
array? I don't care if the actual array does south right now.
Here's an mdadm -E on one member (All other members have the same
information regarding the array, ie offsets)
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : d2524c57:00804101:5bbb59b4:871db7a6
Name : kame:2 (local to host kame)
Creation Time : Sat Sep 24 09:17:18 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907028896 (1863.02 GiB 2000.40 GB)
Array Size : 5860543296 (5589.05 GiB 6001.20 GB)
Used Dev Size : 3907028864 (1863.02 GiB 2000.40 GB)
Super Offset : 3907029152 sectors
Unused Space : before=0 sectors, after=280 sectors
State : clean
Device UUID : 0638ac21:63fe9f2b:2aca8835:c65d4ac5
Internal Bitmap : -8 sectors from superblock
Update Time : Mon Jun 20 00:30:30 2016
Checksum : 6a1ca36f - correct
Events : 54245
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
--
Microsoft has beaten Volkswagen's world record. Volkswagen only created 22
million bugs.
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-21 16:56 UTC (permalink / raw)
To: dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
axboe-b10kYP2dOMg@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <CAPcyv4ht8B7dHe1ckv5d=bOrRzCy3=ZDVSTD0rRsak_LYD8r8g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, 2016-06-21 at 09:45 -0700, Dan Williams wrote:
> On Tue, Jun 21, 2016 at 9:35 AM, Kani, Toshimitsu <toshi.kani@hpe.com>
> wrote:
> > On Tue, 2016-06-21 at 09:25 -0700, Dan Williams wrote:
> > > On Tue, Jun 21, 2016 at 8:44 AM, Kani, Toshimitsu <toshi.kani@hpe.com>
> > > wrote:
> > > >
:
> > > > I think GENHD_FL_DAX is more appropriate since DAX does not use a
> > > > request queue, except for protecting the underlining device being
> > > > disabled while direct_access() is called (b2e0d1625e19).
> > > >
> > > > About protecting direct_access, this patch assumes that the
> > > > underlining device cannot be disabled until dtr() is called. Is this
> > > > correct? If not, I will need to call dax_map_atomic().
> > >
> > > Kernel internal usages of dax should be using dax_map_atomic() to
> > > safely resolve device removal races.
> >
> > Will do. In such case, shall I move dax_[un]map_atomic() to block_dev.c
> > and rename them to bdev_dax_[un]map_atomic()?
>
> Sounds good to me. I know Jeff and Christoph don't like the current
> calling convention of passing in a structure. Just note that they
> might ask you to change it back to a list of parameters if it moves to
> bdev_dax_map_atomic().
OK, I will change it back to a list of parameters as well.
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Dan Williams @ 2016-06-21 16:45 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
axboe-b10kYP2dOMg@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466526342.3504.270.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jun 21, 2016 at 9:35 AM, Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Tue, 2016-06-21 at 09:25 -0700, Dan Williams wrote:
>> On Tue, Jun 21, 2016 at 8:44 AM, Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
>> wrote:
>> >
>> > On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
>> > >
>> > > On Mon, Jun 20 2016 at 6:22pm -0400,
>> > > Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> > > >
>> > > > On Mon, Jun 20 2016 at 5:28pm -0400,
>> > > > Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
>> > :
>> > > > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
>> > > > branch.
>> > > >
>> > > > No longer seeing any corruption in my test that was using partitions
>> > > > to span pmem devices with a dm-linear device.
>> > > >
>> > > > Jens, any chance you'd be open to picking up the first 2 patches in
>> > > > this series? Or would you like to see them folded or something
>> > > > different?
>> > >
>> > > I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
>> > > rather than establish GENHD_FL_DAX on the genhd?
>> > >
>> > > It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
>> > > check for a queue flag.
>> >
>> > I think GENHD_FL_DAX is more appropriate since DAX does not use a request
>> > queue, except for protecting the underlining device being disabled while
>> > direct_access() is called (b2e0d1625e19).
>> >
>> > About protecting direct_access, this patch assumes that the underlining
>> > device cannot be disabled until dtr() is called. Is this correct? If
>> > not, I will need to call dax_map_atomic().
>>
>> Kernel internal usages of dax should be using dax_map_atomic() to
>> safely resolve device removal races.
>
> Will do. In such case, shall I move dax_[un]map_atomic() to block_dev.c and
> rename them to bdev_dax_[un]map_atomic()?
Sounds good to me. I know Jeff and Christoph don't like the current
calling convention of passing in a structure. Just note that they
might ask you to change it back to a list of parameters if it moves to
bdev_dax_map_atomic().
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-21 16:35 UTC (permalink / raw)
To: dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
axboe-b10kYP2dOMg@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <CAPcyv4gFREc94ANuFD_Lyddx3iqRTN2UDebgeJe3LqPL8xrVzg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, 2016-06-21 at 09:25 -0700, Dan Williams wrote:
> On Tue, Jun 21, 2016 at 8:44 AM, Kani, Toshimitsu <toshi.kani@hpe.com>
> wrote:
> >
> > On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
> > >
> > > On Mon, Jun 20 2016 at 6:22pm -0400,
> > > Mike Snitzer <snitzer@redhat.com> wrote:
> > > >
> > > > On Mon, Jun 20 2016 at 5:28pm -0400,
> > > > Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> > :
> > > > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
> > > > branch.
> > > >
> > > > No longer seeing any corruption in my test that was using partitions
> > > > to span pmem devices with a dm-linear device.
> > > >
> > > > Jens, any chance you'd be open to picking up the first 2 patches in
> > > > this series? Or would you like to see them folded or something
> > > > different?
> > >
> > > I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
> > > rather than establish GENHD_FL_DAX on the genhd?
> > >
> > > It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
> > > check for a queue flag.
> >
> > I think GENHD_FL_DAX is more appropriate since DAX does not use a request
> > queue, except for protecting the underlining device being disabled while
> > direct_access() is called (b2e0d1625e19).
> >
> > About protecting direct_access, this patch assumes that the underlining
> > device cannot be disabled until dtr() is called. Is this correct? If
> > not, I will need to call dax_map_atomic().
>
> Kernel internal usages of dax should be using dax_map_atomic() to
> safely resolve device removal races.
Will do. In such case, shall I move dax_[un]map_atomic() to block_dev.c and
rename them to bdev_dax_[un]map_atomic()?
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Dan Williams @ 2016-06-21 16:25 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
axboe-b10kYP2dOMg@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466523280.3504.262.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jun 21, 2016 at 8:44 AM, Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
>> On Mon, Jun 20 2016 at 6:22pm -0400,
>> Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> >
>> > On Mon, Jun 20 2016 at 5:28pm -0400,
>> > Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
>> >
> :
>> > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
>> > branch.
>> >
>> > No longer seeing any corruption in my test that was using partitions to
>> > span pmem devices with a dm-linear device.
>> >
>> > Jens, any chance you'd be open to picking up the first 2 patches in this
>> > series? Or would you like to see them folded or something different?
>>
>> I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
>> rather than establish GENHD_FL_DAX on the genhd?
>>
>> It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
>> check for a queue flag.
>
> I think GENHD_FL_DAX is more appropriate since DAX does not use a request
> queue, except for protecting the underlining device being disabled while
> direct_access() is called (b2e0d1625e19).
>
> About protecting direct_access, this patch assumes that the underlining
> device cannot be disabled until dtr() is called. Is this correct? If not,
> I will need to call dax_map_atomic().
Kernel internal usages of dax should be using dax_map_atomic() to
safely resolve device removal races.
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-21 15:50 UTC (permalink / raw)
To: axboe-b10kYP2dOMg@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466523280.3504.262.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, 2016-06-21 at 09:34 -0600, Kani, Toshimitsu wrote:
> On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
> >
> > On Mon, Jun 20 2016 at 6:22pm -0400,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> > >
> > > On Mon, Jun 20 2016 at 5:28pm -0400,
> > > Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> > >
> :
> > >
> > > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
> > > branch.
> > >
> > > No longer seeing any corruption in my test that was using partitions
> > > to span pmem devices with a dm-linear device.
> > >
> > > Jens, any chance you'd be open to picking up the first 2 patches in
> > > this series? Or would you like to see them folded or something
> > > different?
> >
> > I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
> > rather than establish GENHD_FL_DAX on the genhd?
> >
> > It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
> > check for a queue flag.
>
> I think GENHD_FL_DAX is more appropriate since DAX does not use a request
> queue, except for protecting the underlining device being disabled while
> direct_access() is called (b2e0d1625e19).
Forgot to mention that there are bdev_dax_supported() and bdev_dax_capable()
interfaces that can be called from upper layers. They both call
bdev_direct_access() which checks GENHD_FL_DAX.
Thanks,
-Toshi
> About protecting direct_access, this patch assumes that the underlining
> device cannot be disabled until dtr() is called. Is this correct? If
> not, I will need to call dax_map_atomic().
>
> Thanks,
> -Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-21 15:44 UTC (permalink / raw)
To: axboe-b10kYP2dOMg@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160621134147.GA26392-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Tue, 2016-06-21 at 09:41 -0400, Mike Snitzer wrote:
> On Mon, Jun 20 2016 at 6:22pm -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
> >
> > On Mon, Jun 20 2016 at 5:28pm -0400,
> > Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> >
:
> > Looks good, I folded it in and tested it to work. Pushed to my 'wip'
> > branch.
> >
> > No longer seeing any corruption in my test that was using partitions to
> > span pmem devices with a dm-linear device.
> >
> > Jens, any chance you'd be open to picking up the first 2 patches in this
> > series? Or would you like to see them folded or something different?
>
> I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
> rather than establish GENHD_FL_DAX on the genhd?
>
> It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
> check for a queue flag.
I think GENHD_FL_DAX is more appropriate since DAX does not use a request
queue, except for protecting the underlining device being disabled while
direct_access() is called (b2e0d1625e19).
About protecting direct_access, this patch assumes that the underlining
device cannot be disabled until dtr() is called. Is this correct? If not,
I will need to call dax_map_atomic().
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-21 13:41 UTC (permalink / raw)
To: Kani, Toshimitsu, axboe-b10kYP2dOMg
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160620222236.GA22461-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Mon, Jun 20 2016 at 6:22pm -0400,
Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> On Mon, Jun 20 2016 at 5:28pm -0400,
> Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
>
> >
> > Hi Mike,
> >
> > Can you fold the following patch to the dm-linear patch?
> >
> > Thanks,
> > -Tsohi
> >
> > ------
> > Subject: [PATCH] dm-linear: Fix partition handling for DAX
> >
> > Partition handling was missing in linear_direct_access().
> > Call bdev_direct_access(), instead of directly calling
> > target direct_access function.
> >
> > Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
> > ---
> > drivers/md/dm-linear.c | 14 ++++++++++----
> > 1 file changed, 10 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
> > index 325aa06..38323e4 100644
> > --- a/drivers/md/dm-linear.c
> > +++ b/drivers/md/dm-linear.c
> > @@ -148,10 +148,16 @@ static long linear_direct_access(struct dm_target *ti,
> > sector_t sector,
> > {
> > struct linear_c *lc = ti->private;
> > struct block_device *bdev = lc->dev->bdev;
> > - const struct block_device_operations *bd_ops = bdev->bd_disk->fops;
> > -
> > - return bd_ops->direct_access(bdev, linear_map_sector(ti, sector),
> > - kaddr, pfn, size);
> > + struct blk_dax_ctl dax = {
> > + .sector = linear_map_sector(ti, sector),
> > + .size = size,
> > + };
> > + long ret;
> > +
> > + ret = bdev_direct_access(bdev, &dax);
> > + *kaddr = dax.addr;
> > + *pfn = dax.pfn;
> > + return ret;
> > }
> >
> > static struct target_type linear_target = {
>
> Looks good, I folded it in and tested it to work. Pushed to my 'wip'
> branch.
>
> No longer seeing any corruption in my test that was using partitions to
> span pmem devices with a dm-linear device.
>
> Jens, any chance you'd be open to picking up the first 2 patches in this
> series? Or would you like to see them folded or something different?
I'm now wondering if we'd be better off setting a new QUEUE_FLAG_DAX
rather than establish GENHD_FL_DAX on the genhd?
It'd be quite a bit easier to allow upper layers (e.g. XFS and ext4) to
check for a queue flag.
^ permalink raw reply
* Re: unbalanced RAID5 / performance issues
From: Adam Goryachev @ 2016-06-21 2:29 UTC (permalink / raw)
To: Jens-U. Mozdzen; +Cc: linux-raid
In-Reply-To: <20160620104455.Horde.Cu9WNhgOSFjDC3hgl4ZCm01@www3.nde.ag>
On 20/06/16 18:44, Jens-U. Mozdzen wrote:
> Hi Adam,
>
> Zitat von Adam Goryachev <adam@websitemanagers.com.au>:
>> Hi,
>>
>> I have a RAID5 array which consists of 8 x Intel 480GB SSD, single
>> partition on each covering 100% of the drive.
>> [...]
>> I'm finding that the underlying disk utilisation is "uneven" ie, one
>> or two disks is used a lot more heavily than the others. This is best
>> seen with iostat:
>> iostat -x -N /dev/sd? 5
>> This will show 5 second averages... so we should expect the average
>> utilisation of all disks to be equal ( I expect, I am probably wrong).
>> Ignoring the first output, since that is values since the system was
>> booted, I've copied three sample from after that.
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdf 128.00 194.00 86.80 141.20 897.70 1289.60
>> 19.19 0.04 0.18 0.16 0.20 0.13 2.96
>> sdh 110.80 138.60 83.40 139.20 808.80 1063.20
>> 16.82 0.08 0.34 0.21 0.42 0.31 6.96
>> sde 120.80 162.00 90.60 117.80 866.40 1073.60
>> 18.62 0.09 0.42 0.12 0.65 0.38 7.84
>> sdb 141.80 184.60 110.60 130.60 1104.30 1219.20
>> 19.27 0.04 0.15 0.14 0.16 0.11 2.64
>> sda 126.00 153.80 89.80 120.40 921.00 1048.00
>> 18.73 0.13 0.61 0.14 0.96 0.57 12.08
>> sdg 132.20 168.40 113.00 122.80 1037.60 1116.80
>> 18.27 0.05 0.21 0.28 0.15 0.15 3.60
>> sdd 122.20 180.80 99.80 135.60 958.40 1219.20
>> 18.50 0.04 0.16 0.20 0.13 0.10 2.40
>> sdc 112.80 178.60 87.40 115.20 824.00 1128.80
>> 19.28 0.17 0.85 0.43 1.17 0.75 15.20
>> [...]
>> As you can see, sdc (and sda) has a much higher utilisation compared
>> to all the other drives, but we can see the actual reads/writes are
>> similar across all drives.
>
> looking at those numbers, it might not be the (effective) utilization
> that's higher, but the time the SSDs spend handling the requests.
>
> As you already ruled out model issues for sda, further probable causes
> that I'd check might be
>
> - a different firmware level for sda
All the Series 520 drives are running identical firmware (checked with
smartctl) but I can't confirm if that is the latest firmware or not, I
can find the intel tool to upgrade the firmware, but it doesn't specify
what the current firmware version is for this model.
> - disk problems (anything useful in the SMART numbers?)
No, what started all this is I did find some unusual numbers on one
disk, but that was a 160GB SSD used for the OS itself, not part of the
array, and it has now been replaced (purchased a new one, but Intel will
replace the old one eventually). All other drives SMART details look
reasonable....
> - connection issues (are all disks connected to the same (type of)
> controller?)
All disks are connected to the same controller....
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
>
> I can't comment on the RAID parameter questions, though.
I just get the feeling that specific drives are being "worked" harder
than others, and I'm not sure why.
I'm considering moving to either RAID10 or RAID50 in the future to try
to improve performance, but I'm honestly not sure that this is really
the problem anyway. By my calculations, if I double the number of
drives, and move to RAID10, then I can double the read performance and
improve write performance (I'm not exactly sure of the math here, how
does one calculate write performance on RAID5 when you need to do
read/modify/write?), alternatively, RAID50 (with 16 drives, with 4
drives in 4 RAID5 sub-arrays) should also double read performance, but
also improve write performance compared to the current, but not as much
as RAID10 would. Although RAID50 will give more storage capacity than
the RAID10....
I think my real issue is perhaps latency, and that the real "bottleneck"
is at the DRBD layer rather than raid, but I'm trying to optimise each
part that doesn't look right as I go.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: unbalanced RAID5 / performance issues
From: Adam Goryachev @ 2016-06-20 23:41 UTC (permalink / raw)
To: Andreas Klauer, Jens-U. Mozdzen; +Cc: linux-raid
In-Reply-To: <20160620092600.GA3549@metamorpher.de>
On 20/06/16 19:26, Andreas Klauer wrote:
> On Mon, Jun 20, 2016 at 10:44:55AM +0200, Jens-U. Mozdzen wrote:
>> Zitat von Adam Goryachev <adam@websitemanagers.com.au>:
>>> As you can see, sdc (and sda) has a much higher utilisation compared
>>> to all the other drives, but we can see the actual reads/writes are
>>> similar across all drives.
>> looking at those numbers, it might not be the (effective) utilization
>> that's higher, but the time the SSDs spend handling the requests.
> sdc also happens to be the last drive in your array.
>
> When creating raid5, the initial sync will overwrite this drive completely.
> Are you using fstrim / discard? Without TRIM this SSD might consider itself
> completely full and take longer for new writes.
I'm fairly certain that all drives have been completely written to by
now. The system is around 4 years old, and we do approx 200GB or more of
writes per day....
I'm also fairly certain that TRIM is not working through the entire stack:
Windows 2012R2
Xen GPLPV drivers (old ones)
Xen 4.1
Linux open-iSCSI 2.0.873
Linux iscsitarget (iet) 1.4.20.3+svn502-1
DRBD 8.4.x
LVM2
Linux MD RAID5
Partitions
SSD
I never really tried to test for TRIM support through the stack, but I'd
be shocked if it was working.....
> Also there might be an issue with SF-2281 controller used by these SSDs:
>
> http://www.anandtech.com/show/5508/intel-ssd-520-review-cherryville-brings-reliability-to-sandforce/7
>
> They state that even after TRIM the SSD does not return to
> its prime condition...
The performance seems better on the 520 series (older series) than the
530 one.... I'm not sure which chipset/firmware the 530 series use, but
I would have expected it to be better...
Looking at the spec sheets for each I see:
Model Seq Read Seq Write Random Read Random Write
2.5" 480GB 520Series 540MB/s 490MB/s 41KIOPS 80KIOPS
2.5" 480GB 530Series 550MB/s 520MB/s 50KIOPS 42KIOPS
For some reason, maybe the Random Write IOPS being almost half is
causing the problems?
> Apart from that, double check that your partitions are aligned.
> This is usually the case but may be a huge problem if overlooked.
All of the drives are partitioned identically:
Disk /dev/sdh: 480 GB, 480101368320 bytes
255 heads, 63 sectors/track, 58369 cylinders, total 937697985 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdh1 64 937697984 468848961 fd Lnx RAID auto
Not sure if that is "correctly aligned". I note that on newer
systems/drives I see partitions starting at 2048 instead of 64, but I
think that is just to allow extra space for grub/etc...
I think I'll try to swap the single 530 drive with another one if I dare
(means dropping redundancy on the array during the re-sync....)
My main concern is that it could be due to the way the array is
configured, ie, chunk size/etc, but it does also seem to be related to
the model number of the drive.
BTW, the array has been grown a couple of times, it wasn't created new
with all 8 drives, so originally, sdc wasn't the last drive, it is
probably the most recently added drive though.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-20 22:22 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466457467.3504.249.camel-ZPxbGqLxI0U@public.gmane.org>
On Mon, Jun 20 2016 at 5:28pm -0400,
Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Mon, 2016-06-20 at 14:01 -0600, Kani, Toshimitsu wrote:
> > On Mon, 2016-06-20 at 15:52 -0400, Mike Snitzer wrote:
> > >
> > > On Mon, Jun 20 2016 at 3:40pm -0400,
> > > Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > >
> :
> > > > If I don't use XFS, and only issue IO directly to the /dev/pmem/lv, I
> > > > don't see this corruption.
> > >
> > > I did the same test with ext4 instead of xfs and it resulted in the same
> > > type of systemic corruption (lvm2 metadata corrupted too):
> > >
> :
> > I will look into the issue.
>
> Hi Mike,
>
> Can you fold the following patch to the dm-linear patch?
>
> Thanks,
> -Tsohi
>
> ------
> Subject: [PATCH] dm-linear: Fix partition handling for DAX
>
> Partition handling was missing in linear_direct_access().
> Call bdev_direct_access(), instead of directly calling
> target direct_access function.
>
> Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
> ---
> drivers/md/dm-linear.c | 14 ++++++++++----
> 1 file changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
> index 325aa06..38323e4 100644
> --- a/drivers/md/dm-linear.c
> +++ b/drivers/md/dm-linear.c
> @@ -148,10 +148,16 @@ static long linear_direct_access(struct dm_target *ti,
> sector_t sector,
> {
> struct linear_c *lc = ti->private;
> struct block_device *bdev = lc->dev->bdev;
> - const struct block_device_operations *bd_ops = bdev->bd_disk->fops;
> -
> - return bd_ops->direct_access(bdev, linear_map_sector(ti, sector),
> - kaddr, pfn, size);
> + struct blk_dax_ctl dax = {
> + .sector = linear_map_sector(ti, sector),
> + .size = size,
> + };
> + long ret;
> +
> + ret = bdev_direct_access(bdev, &dax);
> + *kaddr = dax.addr;
> + *pfn = dax.pfn;
> + return ret;
> }
>
> static struct target_type linear_target = {
Looks good, I folded it in and tested it to work. Pushed to my 'wip'
branch.
No longer seeing any corruption in my test that was using partitions to
span pmem devices with a dm-linear device.
Jens, any chance you'd be open to picking up the first 2 patches in this
series? Or would you like to see them folded or something different?
Mike
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-20 21:28 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466452883.3504.244.camel-ZPxbGqLxI0U@public.gmane.org>
On Mon, 2016-06-20 at 14:01 -0600, Kani, Toshimitsu wrote:
> On Mon, 2016-06-20 at 15:52 -0400, Mike Snitzer wrote:
> >
> > On Mon, Jun 20 2016 at 3:40pm -0400,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> >
:
> > > If I don't use XFS, and only issue IO directly to the /dev/pmem/lv, I
> > > don't see this corruption.
> >
> > I did the same test with ext4 instead of xfs and it resulted in the same
> > type of systemic corruption (lvm2 metadata corrupted too):
> >
:
> I will look into the issue.
Hi Mike,
Can you fold the following patch to the dm-linear patch?
Thanks,
-Tsohi
------
Subject: [PATCH] dm-linear: Fix partition handling for DAX
Partition handling was missing in linear_direct_access().
Call bdev_direct_access(), instead of directly calling
target direct_access function.
Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
---
drivers/md/dm-linear.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
index 325aa06..38323e4 100644
--- a/drivers/md/dm-linear.c
+++ b/drivers/md/dm-linear.c
@@ -148,10 +148,16 @@ static long linear_direct_access(struct dm_target *ti,
sector_t sector,
{
struct linear_c *lc = ti->private;
struct block_device *bdev = lc->dev->bdev;
- const struct block_device_operations *bd_ops = bdev->bd_disk->fops;
-
- return bd_ops->direct_access(bdev, linear_map_sector(ti, sector),
- kaddr, pfn, size);
+ struct blk_dax_ctl dax = {
+ .sector = linear_map_sector(ti, sector),
+ .size = size,
+ };
+ long ret;
+
+ ret = bdev_direct_access(bdev, &dax);
+ *kaddr = dax.addr;
+ *pfn = dax.pfn;
+ return ret;
}
static struct target_type linear_target = {
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related
* Re:In foreign trade workers should Take the initiative to develop a good understanding of customers around the world!
From: ts9 @ 2016-06-20 21:25 UTC (permalink / raw)
To: linux-raid
做外贸主动出击，搜索开发你的全球客户
不能总是守株待兔，要主动出击，新人不能总是等着询盘过来，要通过各种渠道去搜索客户，发开发信，怎么搜索，google很强大。但当google，雅虎，必应等各大知名搜索引擎集于一体会有怎么样的体验？
我司是做一款外贸客户外发软件的，集各大知名搜索引擎于一体，关键字式搜全球优质客户、更有客户管理系统让你轻松维护老客户，拓展新客户。独家专利技术，一键海量群发邮件，独有的邮件群发，邮件退订技术，防屏蔽，防封号，提高邮到达率，提升客户询盘机率。如果你还在迷茫如何才能更快，更便捷，更质高，你不妨联系我了解一下。五大模块：客户搜素-邮件营销-邮件跟踪-邮件退订-客户管理，28个优化功能一路护航你轻松出订单。
想了解更多主动式外贸开发客户，请加 QQ------> 3246075707 直接验证客户信息 是否真实、有效
也可加微信号：sunsesoftsam
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-20 20:11 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160620195217.GB21657-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Mon, 2016-06-20 at 15:52 -0400, Mike Snitzer wrote:
> On Mon, Jun 20 2016 at 3:40pm -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
> >
> > # dd if=/dev/zero of=/mnt/dax/meh bs=1024K oflag=direct
> > [11729.754671] XFS (dm-4): Metadata corruption detected at
> > xfs_agf_read_verify+0x70/0x120 [xfs], xfs_agf block 0x45a808
> > [11729.766423] XFS (dm-4): Unmount and run xfs_repair
> > [11729.771774] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> > [11729.778869] ffff8800b8038000: 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 00 00 00 ................
> > [11729.788582] ffff8800b8038010: 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 00 00 00 ................
> > [11729.798293] ffff8800b8038020: 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 00 00 00 ................
> > [11729.808002] ffff8800b8038030: 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 00 00 00 ................
> > [11729.817715] XFS (dm-4): metadata I/O error: block 0x45a808
> > ("xfs_trans_read_buf_map") error 117 numblks 8
> >
> > When this XFS corruption occurs corruption then also manifests in lvm2's
> > metadata:
> >
> > # vgremove pmem
> > Do you really want to remove volume group "pmem" containing 1 logical
> > volumes? [y/n]: y
> > Do you really want to remove active logical volume lv? [y/n]: y
> > Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
> > WARNING: Failed to write an MDA of VG pmem.
> > Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> > WARNING: Failed to write an MDA of VG pmem.
> > Failed to write VG pmem.
> > Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> > Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
> >
> > If I don't use XFS, and only issue IO directly to the /dev/pmem/lv, I
> > don't see this corruption.
> I did the same test with ext4 instead of xfs and it resulted in the same
> type of systemic corruption (lvm2 metadata corrupted too):
>
> [12816.407147] EXT4-fs (dm-4): DAX enabled. Warning: EXPERIMENTAL, use at
> your own risk
> [12816.416123] EXT4-fs (dm-4): mounted filesystem with ordered data mode.
> Opts: dax
> [12816.766855] EXT4-fs error (device dm-4): ext4_mb_generate_buddy:758:
> group 9, block bitmap and bg descriptor inconsistent: 32768 vs 32395 free
> clusters
> [12816.782016] EXT4-fs error (device dm-4): ext4_mb_generate_buddy:758:
> group 10, block bitmap and bg descriptor inconsistent: 32768 vs 16384 free
> clusters
> [12816.797491] JBD2: Spotted dirty metadata buffer (dev = dm-4, blocknr =
> 0). There's a risk of filesystem corruption in case of system crash.
>
> # vgremove pmem
> Do you really want to remove volume group "pmem" containing 1 logical
> volumes? [y/n]: y
> Do you really want to remove active logical volume lv? [y/n]: y
> Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
> WARNING: Failed to write an MDA of VG pmem.
> Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> WARNING: Failed to write an MDA of VG pmem.
> Failed to write VG pmem.
> Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
I will look into the issue.
Thanks for the testing!
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-20 19:52 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160620194026.GA21657-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Mon, Jun 20 2016 at 3:40pm -0400,
Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> # dd if=/dev/zero of=/mnt/dax/meh bs=1024K oflag=direct
> [11729.754671] XFS (dm-4): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], xfs_agf block 0x45a808
> [11729.766423] XFS (dm-4): Unmount and run xfs_repair
> [11729.771774] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [11729.778869] ffff8800b8038000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [11729.788582] ffff8800b8038010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [11729.798293] ffff8800b8038020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [11729.808002] ffff8800b8038030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [11729.817715] XFS (dm-4): metadata I/O error: block 0x45a808 ("xfs_trans_read_buf_map") error 117 numblks 8
>
> When this XFS corruption occurs corruption then also manifests in lvm2's
> metadata:
>
> # vgremove pmem
> Do you really want to remove volume group "pmem" containing 1 logical volumes? [y/n]: y
> Do you really want to remove active logical volume lv? [y/n]: y
> Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
> WARNING: Failed to write an MDA of VG pmem.
> Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> WARNING: Failed to write an MDA of VG pmem.
> Failed to write VG pmem.
> Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
> Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
>
> If I don't use XFS, and only issue IO directly to the /dev/pmem/lv, I
> don't see this corruption.
I did the same test with ext4 instead of xfs and it resulted in the same
type of systemic corruption (lvm2 metadata corrupted too):
[12816.407147] EXT4-fs (dm-4): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
[12816.416123] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: dax
[12816.766855] EXT4-fs error (device dm-4): ext4_mb_generate_buddy:758: group 9, block bitmap and bg descriptor inconsistent: 32768 vs 32395 free clusters
[12816.782016] EXT4-fs error (device dm-4): ext4_mb_generate_buddy:758: group 10, block bitmap and bg descriptor inconsistent: 32768 vs 16384 free clusters
[12816.797491] JBD2: Spotted dirty metadata buffer (dev = dm-4, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
# vgremove pmem
Do you really want to remove volume group "pmem" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume lv? [y/n]: y
Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
WARNING: Failed to write an MDA of VG pmem.
Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
WARNING: Failed to write an MDA of VG pmem.
Failed to write VG pmem.
Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-20 19:40 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
sandeen-H+wXaHxf7aLQT0dZR+AlfA,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1466446861.3504.243.camel-ZPxbGqLxI0U@public.gmane.org>
On Mon, Jun 20 2016 at 2:31pm -0400,
Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> On Mon, 2016-06-20 at 14:00 -0400, Mike Snitzer wrote:
> >
> > I rebased your patches on linux-dm.git's 'for-next' (which includes what
> > I've already staged for the 4.8 merge window). And I folded/changed
> > some of the DM patches so that there are only 2 now (1 for DM core and 1
> > for dm-linear). Please see the 4 topmost commits in my 'wip' here:
> >
> > http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=wip
> >
> > Feel free to pick these patches up to use as the basis for continued
> > work or re-posting of this set.. either that or I could post them as v2
> > on your behalf.
> >
> > As for testing, I've verified that basic IO works to a pmem-based DM
> > linear device and that mixed table types are rejected as expected.
>
> Great! I will send additional patch, add DAX support to dm-stripe, on top of
> these once I finish my testing.
I did some further testing and am seeing some XFS corruption when
testing a DM linear device that spans multiple pmem devices. I created
2 partitions ontop of /dev/pmem0 (which I created using the howto from
https://nvdimm.wiki.kernel.org). Then I did:
# pvcreate /dev/pmem0p1
# pvcreate /dev/pmem0p2
# vgcreate pmem /dev/pmem0p1 /dev/pmem0p2
# lvcreate -L 2.9G -n lv pmem
# lsblk /dev/pmem0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
pmem0 259:0 0 6G 0 disk
├─pmem0p1 259:1 0 1G 0 part
│ └─pmem-lv 253:4 0 2.9G 0 lvm
└─pmem0p2 259:2 0 2G 0 part
└─pmem-lv 253:4 0 2.9G 0 lvm
# dmsetup table pmem-lv
0 4186112 linear 259:2 2048
4186112 1900544 linear 259:1 2048
# mkfs.xfs /dev/pmem/lv
# mount -o dax -t xfs /dev/pmem/lv /mnt/dax
[11452.212034] XFS (dm-4): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
[11452.220323] XFS (dm-4): Mounting V4 Filesystem
[11452.226526] XFS (dm-4): Ending clean mount
# dd if=/dev/zero of=/mnt/dax/meh bs=1024K oflag=direct
[11729.754671] XFS (dm-4): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], xfs_agf block 0x45a808
[11729.766423] XFS (dm-4): Unmount and run xfs_repair
[11729.771774] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
[11729.778869] ffff8800b8038000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[11729.788582] ffff8800b8038010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[11729.798293] ffff8800b8038020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[11729.808002] ffff8800b8038030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[11729.817715] XFS (dm-4): metadata I/O error: block 0x45a808 ("xfs_trans_read_buf_map") error 117 numblks 8
When this XFS corruption occurs corruption then also manifests in lvm2's
metadata:
# vgremove pmem
Do you really want to remove volume group "pmem" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume lv? [y/n]: y
Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
WARNING: Failed to write an MDA of VG pmem.
Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
WARNING: Failed to write an MDA of VG pmem.
Failed to write VG pmem.
Incorrect metadata area header checksum on /dev/pmem0p2 at offset 4096
Incorrect metadata area header checksum on /dev/pmem0p1 at offset 4096
If I don't use XFS, and only issue IO directly to the /dev/pmem/lv, I
don't see this corruption.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Kani, Toshimitsu @ 2016-06-20 18:31 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160620180043.GA21261-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Mon, 2016-06-20 at 14:00 -0400, Mike Snitzer wrote:
> On Mon, Jun 13 2016 at 6:57pm -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
> >
> > On Mon, Jun 13 2016 at 6:21pm -0400,
> > Toshi Kani <toshi.kani@hpe.com> wrote:
> >
> > >
> > > This patch-set adds DAX support to device-mapper dm-linear devices
> > > used by LVM. It works with LVM commands as follows:
> > > - Creation of a logical volume with all DAX capable devices (such
> > > as pmem) sets the logical volume DAX capable as well.
> > > - Once a logical volume is set to DAX capable, the volume may not
> > > be extended with non-DAX capable devices.
> > >
> > > The direct_access interface is added to dm and dm-linear to map
> > > a request to a target device.
> > >
> > > - Patch 1-2 introduce GENHD_FL_DAX flag to indicate DAX capability.
> > > - Patch 3-4 add direct_access functions to dm and dm-linear.
> > > - Patch 5-6 set GENHD_FL_DAX to dm when all targets are DAX capable.
> > >
> > > ---
> > > Toshi Kani (6):
> > > 1/6 genhd: Add GENHD_FL_DAX to gendisk flags
> > > 2/6 block: Check GENHD_FL_DAX for DAX capability
> > > 3/6 dm: Add dm_blk_direct_access() for mapped device
> > > 4/6 dm-linear: Add linear_direct_access()
> > > 5/6 dm, dm-linear: Add dax_supported to dm_target
> > > 6/6 dm: Enable DAX support for mapper device
> > Thanks a lot for doing this. I recently added it to my TODO so your
> > patches come at a great time.
> >
> > I'll try to get to reviewing/testing your work by the end of this week.
>
> I rebased your patches on linux-dm.git's 'for-next' (which includes what
> I've already staged for the 4.8 merge window). And I folded/changed
> some of the DM patches so that there are only 2 now (1 for DM core and 1
> for dm-linear). Please see the 4 topmost commits in my 'wip' here:
>
> http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=wip
>
> Feel free to pick these patches up to use as the basis for continued
> work or re-posting of this set.. either that or I could post them as v2
> on your behalf.
>
> As for testing, I've verified that basic IO works to a pmem-based DM
> linear device and that mixed table types are rejected as expected.
Great! I will send additional patch, add DAX support to dm-stripe, on top of
these once I finish my testing.
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 0/6] Support DAX for device-mapper dm-linear devices
From: Mike Snitzer @ 2016-06-20 18:00 UTC (permalink / raw)
To: Toshi Kani
Cc: axboe-tSWWG44O7X1aa/9Udqfwiw, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn, agk-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <20160613225756.GA18417-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Mon, Jun 13 2016 at 6:57pm -0400,
Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> On Mon, Jun 13 2016 at 6:21pm -0400,
> Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
>
> > This patch-set adds DAX support to device-mapper dm-linear devices
> > used by LVM. It works with LVM commands as follows:
> > - Creation of a logical volume with all DAX capable devices (such
> > as pmem) sets the logical volume DAX capable as well.
> > - Once a logical volume is set to DAX capable, the volume may not
> > be extended with non-DAX capable devices.
> >
> > The direct_access interface is added to dm and dm-linear to map
> > a request to a target device.
> >
> > - Patch 1-2 introduce GENHD_FL_DAX flag to indicate DAX capability.
> > - Patch 3-4 add direct_access functions to dm and dm-linear.
> > - Patch 5-6 set GENHD_FL_DAX to dm when all targets are DAX capable.
> >
> > ---
> > Toshi Kani (6):
> > 1/6 genhd: Add GENHD_FL_DAX to gendisk flags
> > 2/6 block: Check GENHD_FL_DAX for DAX capability
> > 3/6 dm: Add dm_blk_direct_access() for mapped device
> > 4/6 dm-linear: Add linear_direct_access()
> > 5/6 dm, dm-linear: Add dax_supported to dm_target
> > 6/6 dm: Enable DAX support for mapper device
>
> Thanks a lot for doing this. I recently added it to my TODO so your
> patches come at a great time.
>
> I'll try to get to reviewing/testing your work by the end of this week.
I rebased your patches on linux-dm.git's 'for-next' (which includes what
I've already staged for the 4.8 merge window). And I folded/changed
some of the DM patches so that there are only 2 now (1 for DM core and 1
for dm-linear). Please see the 4 topmost commits in my 'wip' here:
http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=wip
Feel free to pick these patches up to use as the basis for continued
work or re-posting of this set.. either that or I could post them as v2
on your behalf.
As for testing, I've verified that basic IO works to a pmem-based DM
linear device and that mixed table types are rejected as expected.
Mike
^ permalink raw reply
* Re: unbalanced RAID5 / performance issues
From: Andreas Klauer @ 2016-06-20 9:26 UTC (permalink / raw)
To: Jens-U. Mozdzen; +Cc: Adam Goryachev, linux-raid
In-Reply-To: <20160620104455.Horde.Cu9WNhgOSFjDC3hgl4ZCm01@www3.nde.ag>
On Mon, Jun 20, 2016 at 10:44:55AM +0200, Jens-U. Mozdzen wrote:
> Zitat von Adam Goryachev <adam@websitemanagers.com.au>:
> > As you can see, sdc (and sda) has a much higher utilisation compared
> > to all the other drives, but we can see the actual reads/writes are
> > similar across all drives.
>
> looking at those numbers, it might not be the (effective) utilization
> that's higher, but the time the SSDs spend handling the requests.
sdc also happens to be the last drive in your array.
When creating raid5, the initial sync will overwrite this drive completely.
Are you using fstrim / discard? Without TRIM this SSD might consider itself
completely full and take longer for new writes.
Also there might be an issue with SF-2281 controller used by these SSDs:
http://www.anandtech.com/show/5508/intel-ssd-520-review-cherryville-brings-reliability-to-sandforce/7
They state that even after TRIM the SSD does not return to
its prime condition...
Apart from that, double check that your partitions are aligned.
This is usually the case but may be a huge problem if overlooked.
Regards
Andreas Klauer
^ permalink raw reply
* Re: unbalanced RAID5 / performance issues
From: Jens-U. Mozdzen @ 2016-06-20 8:44 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <57678E2D.9080705@websitemanagers.com.au>
Hi Adam,
Zitat von Adam Goryachev <adam@websitemanagers.com.au>:
> Hi,
>
> I have a RAID5 array which consists of 8 x Intel 480GB SSD, single
> partition on each covering 100% of the drive.
> [...]
> I'm finding that the underlying disk utilisation is "uneven" ie, one
> or two disks is used a lot more heavily than the others. This is
> best seen with iostat:
> iostat -x -N /dev/sd? 5
> This will show 5 second averages... so we should expect the average
> utilisation of all disks to be equal ( I expect, I am probably wrong).
> Ignoring the first output, since that is values since the system was
> booted, I've copied three sample from after that.
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdf 128.00 194.00 86.80 141.20 897.70 1289.60
> 19.19 0.04 0.18 0.16 0.20 0.13 2.96
> sdh 110.80 138.60 83.40 139.20 808.80 1063.20
> 16.82 0.08 0.34 0.21 0.42 0.31 6.96
> sde 120.80 162.00 90.60 117.80 866.40 1073.60
> 18.62 0.09 0.42 0.12 0.65 0.38 7.84
> sdb 141.80 184.60 110.60 130.60 1104.30 1219.20
> 19.27 0.04 0.15 0.14 0.16 0.11 2.64
> sda 126.00 153.80 89.80 120.40 921.00 1048.00
> 18.73 0.13 0.61 0.14 0.96 0.57 12.08
> sdg 132.20 168.40 113.00 122.80 1037.60 1116.80
> 18.27 0.05 0.21 0.28 0.15 0.15 3.60
> sdd 122.20 180.80 99.80 135.60 958.40 1219.20
> 18.50 0.04 0.16 0.20 0.13 0.10 2.40
> sdc 112.80 178.60 87.40 115.20 824.00 1128.80
> 19.28 0.17 0.85 0.43 1.17 0.75 15.20
> [...]
> As you can see, sdc (and sda) has a much higher utilisation compared
> to all the other drives, but we can see the actual reads/writes are
> similar across all drives.
looking at those numbers, it might not be the (effective) utilization
that's higher, but the time the SSDs spend handling the requests.
As you already ruled out model issues for sda, further probable causes
that I'd check might be
- a different firmware level for sda
- disk problems (anything useful in the SMART numbers?)
- connection issues (are all disks connected to the same (type of)
controller?)
I can't comment on the RAID parameter questions, though.
Regards,
Jens
--
Jens-U. Mozdzen voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15 mobile : +49-179-4 98 21 98
D-22423 Hamburg e-mail : jmozdzen@nde.ag
Vorsitzende des Aufsichtsrates: Angelika Mozdzen
Sitz und Registergericht: Hamburg, HRB 90934
Vorstand: Jens-U. Mozdzen
USt-IdNr. DE 814 013 983
^ permalink raw reply
* unbalanced RAID5 / performance issues
From: Adam Goryachev @ 2016-06-20 6:33 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi,
I have a RAID5 array which consists of 8 x Intel 480GB SSD, single
partition on each covering 100% of the drive.
md1 : active raid5 sde1[7] sdc1[11] sdd1[10] sdb1[12] sdg1[9] sdh1[5]
sdf1[8] sda1[6]
3281935552 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8]
[UUUUUUUU]
/dev/md1:
Version : 1.2
Creation Time : Wed Aug 22 00:47:03 2012
Raid Level : raid5
Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Mon Jun 20 16:02:10 2016
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : san1:1 (local to host san1)
UUID : 707957c0:b7195438:06da5bc4:485d301c
Events : 2092476
Number Major Minor RaidDevice State
7 8 65 0 active sync /dev/sde1
6 8 1 1 active sync /dev/sda1
8 8 81 2 active sync /dev/sdf1
5 8 113 3 active sync /dev/sdh1
9 8 97 4 active sync /dev/sdg1
12 8 17 5 active sync /dev/sdb1
10 8 49 6 active sync /dev/sdd1
11 8 33 7 active sync /dev/sdc1
I'm finding that the underlying disk utilisation is "uneven" ie, one or
two disks is used a lot more heavily than the others. This is best seen
with iostat:
iostat -x -N /dev/sd? 5
This will show 5 second averages... so we should expect the average
utilisation of all disks to be equal ( I expect, I am probably wrong).
Ignoring the first output, since that is values since the system was
booted, I've copied three sample from after that.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdf 128.00 194.00 86.80 141.20 897.70 1289.60
19.19 0.04 0.18 0.16 0.20 0.13 2.96
sdh 110.80 138.60 83.40 139.20 808.80 1063.20
16.82 0.08 0.34 0.21 0.42 0.31 6.96
sde 120.80 162.00 90.60 117.80 866.40 1073.60
18.62 0.09 0.42 0.12 0.65 0.38 7.84
sdb 141.80 184.60 110.60 130.60 1104.30 1219.20
19.27 0.04 0.15 0.14 0.16 0.11 2.64
sda 126.00 153.80 89.80 120.40 921.00 1048.00
18.73 0.13 0.61 0.14 0.96 0.57 12.08
sdg 132.20 168.40 113.00 122.80 1037.60 1116.80
18.27 0.05 0.21 0.28 0.15 0.15 3.60
sdd 122.20 180.80 99.80 135.60 958.40 1219.20
18.50 0.04 0.16 0.20 0.13 0.10 2.40
sdc 112.80 178.60 87.40 115.20 824.00 1128.80
19.28 0.17 0.85 0.43 1.17 0.75 15.20
sdf 97.00 147.80 107.40 139.80 911.30 1084.00
16.14 0.04 0.15 0.14 0.15 0.11 2.72
sdh 104.80 139.20 99.00 133.60 901.60 1024.00
16.56 0.03 0.13 0.15 0.12 0.10 2.24
sde 97.60 124.00 98.20 109.40 889.60 868.00
16.93 0.03 0.15 0.08 0.21 0.12 2.48
sdb 91.80 144.60 96.00 117.00 839.80 983.20
17.12 0.03 0.13 0.15 0.12 0.12 2.48
sda 73.80 106.40 94.80 120.00 762.20 837.60
14.90 0.12 0.58 0.10 0.95 0.55 11.76
sdg 97.00 143.80 104.80 114.60 894.50 968.80
16.99 0.06 0.29 0.11 0.45 0.28 6.16
sdd 88.40 140.80 93.00 121.00 770.90 980.00
16.36 0.09 0.41 0.16 0.61 0.40 8.56
sdc 92.60 137.00 94.40 106.20 830.70 908.00
17.33 0.21 1.07 0.48 1.59 0.90 18.00
sdf 71.60 138.60 91.60 137.40 813.80 1040.00
16.19 0.08 0.33 0.12 0.47 0.30 6.96
sdh 87.20 137.20 99.20 124.60 927.10 983.20
17.07 0.03 0.14 0.21 0.08 0.12 2.64
sde 85.40 126.60 84.20 102.20 830.50 850.40
18.04 0.02 0.08 0.11 0.05 0.06 1.12
sdb 90.40 153.00 94.40 117.00 907.40 1019.20
18.23 0.02 0.11 0.13 0.10 0.08 1.68
sda 77.60 134.40 84.40 121.40 813.10 958.40
17.22 0.13 0.65 0.13 1.01 0.62 12.72
sdg 101.80 140.60 109.20 112.20 1038.30 946.40
17.93 0.06 0.28 0.22 0.34 0.25 5.44
sdd 90.00 131.20 83.40 111.20 810.60 907.20
17.65 0.02 0.11 0.12 0.10 0.07 1.36
sdc 85.40 136.00 83.00 101.80 817.70 888.80
18.47 0.23 1.27 0.61 1.81 1.13 20.80
As you can see, sdc (and sda) has a much higher utilisation compared to
all the other drives, but we can see the actual reads/writes are similar
across all drives.
Trying to find/explain the differences in performance, I originally
assumed one drive was being "targeted" more heavily, perhaps due to a
bad configuration (eg, chunk size, resulting in all filesystem
read/write on the same physical disk (plus checksum)).
However, it also seems that one drive is a different model:
sda: Model Family: Intel 520 Series SSDs
sdb: Model Family: Intel 520 Series SSDs
sdc: Model Family: Intel 530 Series SSDs
sdd: Model Family: Intel 520 Series SSDs
sde: Model Family: Intel 520 Series SSDs
sdf: Model Family: Intel 520 Series SSDs
sdg: Model Family: Intel 520 Series SSDs
sdh: Model Family: Intel 520 Series SSDs
The disk sector sizes:
512 bytes logical/physical across all disks/drives.
Except we see that sda is the same model as the rest and also seems to
be affected, though not as much as sdc.
So, should I try to swap sdc with another drive of the same model (520
series)?
Is there something else I can do to better optimise the array?
Should I migrate to RAID50 with 12 drives, or RAID10 with 16 drives
(which would also add 480G capacity)?
Would moving to RAID6 help (I doubt it)?
I don't think there is a single threaded CPU issue, since using top and
watching each individual CPU, I don't see the idle reduce to zero), plus
rsync is using most CPU not the md1_raid5 thread.
Should I use a smaller chunk size to better "spread" the load across the
disks? Would a value of 4k be better (the minimum value permitted
apparently)? Or would it be better to go the other way, and increase the
chunk size even more (I think this would help with throughput, but not
so random workload like we are getting).
Some other details that might be relevant:
Linux san1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4
(2016-02-29) x86_64 GNU/Linux
free
total used free shared buffers cached
Mem: 7902324 3287360 4614964 1203468 196836 2440864
-/+ buffers/cache: 649660 7252664
Swap: 3939324 23436 3915888
vmstat doesn't show anything for si or so, so RAM doesn't seem to be a
problem
I'm using lvm on top of the md1 device, and each LV is used by DRBD,
which is then exported with iSCSI to another linux box, and then used as
the block device for Xen Windows machines.
Any other suggestions on where to look, or additional information I
should provide?
Regards,
Adam
--
Adam Goryachev
Website Managers
P: +61 2 8304 0000 adam@websitemanagers.com.au
F: +61 2 8304 0001 www.websitemanagers.com.au
^ permalink raw reply
* [PATCH] md: use seconds granularity for error logging
From: Arnd Bergmann @ 2016-06-17 15:33 UTC (permalink / raw)
To: Shaohua Li
Cc: y2038, Arnd Bergmann, NeilBrown, Guoqing Jiang, Goldwyn Rodrigues,
Jens Axboe, Hannes Reinecke, linux-raid, linux-kernel
The md code stores the exact time of the last error in the
last_read_error variable using a timespec structure. It only
ever uses the seconds portion of that though, so we can
use a scalar for it.
There won't be an overflow in 2038 here, because it already
used monotonic time and 32-bit is enough for that, but I've
decided to use time64_t for consistency in the conversion.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
drivers/md/md.c | 3 +--
drivers/md/md.h | 2 +-
drivers/md/raid10.c | 11 +++++------
3 files changed, 7 insertions(+), 9 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 3745b9a7a2d7..ad512ad4610f 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -3179,8 +3179,7 @@ int md_rdev_init(struct md_rdev *rdev)
rdev->data_offset = 0;
rdev->new_data_offset = 0;
rdev->sb_events = 0;
- rdev->last_read_error.tv_sec = 0;
- rdev->last_read_error.tv_nsec = 0;
+ rdev->last_read_error = 0;
rdev->sb_loaded = 0;
rdev->bb_page = NULL;
atomic_set(&rdev->nr_pending, 0);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 3c3412d85e42..20c667579ede 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -99,7 +99,7 @@ struct md_rdev {
atomic_t read_errors; /* number of consecutive read errors that
* we have tried to ignore.
*/
- struct timespec last_read_error; /* monotonic time since our
+ time64_t last_read_error; /* monotonic time since our
* last read error
*/
atomic_t corrected_errors; /* number of corrected read errors,
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 41191e04d565..f8cdd08d0a40 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2170,21 +2170,20 @@ static void recovery_request_write(struct mddev *mddev, struct r10bio *r10_bio)
*/
static void check_decay_read_errors(struct mddev *mddev, struct md_rdev *rdev)
{
- struct timespec cur_time_mon;
+ long cur_time_mon;
unsigned long hours_since_last;
unsigned int read_errors = atomic_read(&rdev->read_errors);
- ktime_get_ts(&cur_time_mon);
+ cur_time_mon = ktime_get_seconds();
- if (rdev->last_read_error.tv_sec == 0 &&
- rdev->last_read_error.tv_nsec == 0) {
+ if (rdev->last_read_error == 0) {
/* first time we've seen a read error */
rdev->last_read_error = cur_time_mon;
return;
}
- hours_since_last = (cur_time_mon.tv_sec -
- rdev->last_read_error.tv_sec) / 3600;
+ hours_since_last = (long)(cur_time_mon -
+ rdev->last_read_error) / 3600;
rdev->last_read_error = cur_time_mon;
--
2.9.0
^ permalink raw reply related
* re: dm: raid456 basic support
From: Dan Carpenter @ 2016-06-17 9:14 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
[ No idea why it's only just now complaining about issues from 2011... ]
Hello NeilBrown,
The patch 9d09e663d550: "dm: raid456 basic support" from Jan 13,
2011, leads to the following static checker warning:
drivers/md/dm-raid.c:1217 parse_raid_params()
warn: no lower bound on 'value'
drivers/md/dm-raid.c
1211 return -EINVAL;
1212 }
1213 if (!value || (value > MAX_SCHEDULE_TIMEOUT)) {
value is an int. MAX_SCHEDULE_TIMEOUT is LONG_MAX. Should it be
INT_MAX? What about negatives?
1214 rs->ti->error = "daemon sleep period out of range";
1215 return -EINVAL;
1216 }
1217 rs->md.bitmap_info.daemon_sleep = value;
1218 } else if (!strcasecmp(key, dm_raid_arg_name_by_flag(CTR_FLAG_DATA_OFFSET))) {
1219 /* Userspace passes new data_offset after having extended the the data image LV */
1220 if (test_and_set_bit(__CTR_FLAG_DATA_OFFSET, &rs->ctr_flags)) {
1221 rs->ti->error = "Only one data_offset argument pair allowed";
1222 return -EINVAL;
1223 }
1224 /* Ensure sensible data offset */
1225 if (value < 0) {
1226 rs->ti->error = "Bogus data_offset value";
1227 return -EINVAL;
1228 }
1229 rs->data_offset = value;
1230 } else if (!strcasecmp(key, dm_raid_arg_name_by_flag(CTR_FLAG_DELTA_DISKS))) {
1231 /* Define the +/-# of disks to add to/remove from the given raid set */
1232 if (test_and_set_bit(__CTR_FLAG_DELTA_DISKS, &rs->ctr_flags)) {
1233 rs->ti->error = "Only one delta_disks argument pair allowed";
1234 return -EINVAL;
1235 }
1236 /* Ensure MAX_RAID_DEVICES and raid type minimal_devs! */
1237 if (!__within_range(abs(value), 1, MAX_RAID_DEVICES - rt->minimal_devs)) {
1238 rs->ti->error = "Too many delta_disk requested";
1239 return -EINVAL;
1240 }
1241
1242 rs->delta_disks = value;
1243 } else if (!strcasecmp(key, dm_raid_arg_name_by_flag(CTR_FLAG_STRIPE_CACHE))) {
1244 if (test_and_set_bit(__CTR_FLAG_STRIPE_CACHE, &rs->ctr_flags)) {
1245 rs->ti->error = "Only one stripe_cache argument pair allowed";
1246 return -EINVAL;
1247 }
1248
1249 if (!rt_is_raid456(rt)) {
1250 rs->ti->error = "Inappropriate argument: stripe_cache";
1251 return -EINVAL;
1252 }
1253
1254 rs->stripe_cache_entries = value;
1255 } else if (!strcasecmp(key, dm_raid_arg_name_by_flag(CTR_FLAG_MIN_RECOVERY_RATE))) {
1256 if (test_and_set_bit(__CTR_FLAG_MIN_RECOVERY_RATE, &rs->ctr_flags)) {
1257 rs->ti->error = "Only one min_recovery_rate argument pair allowed";
1258 return -EINVAL;
1259 }
1260 if (value > INT_MAX) {
^^^^^^^
Here we're using INT_MAX.
1261 rs->ti->error = "min_recovery_rate out of range";
1262 return -EINVAL;
1263 }
1264 rs->md.sync_speed_min = (int)value;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This looks like negatives are intentional... A few lines later as well.
drivers/md/dm-raid.c:1274 parse_raid_params() warn: no lower bound on 'value'
regards,
dan carpenter
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox