* Re: [PATCH v2 1/3] badblocks: Add core badblock management code
From: Verma, Vishal L @ 2015-12-22 22:13 UTC (permalink / raw)
To: James.Bottomley@HansenPartnership.com, neilb@suse.com
Cc: linux-raid@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-block@vger.kernel.org,
jmoyer@redhat.com, axboe@fb.com
In-Reply-To: <87r3if2gij.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 1940 bytes --]
On Tue, 2015-12-22 at 16:34 +1100, NeilBrown wrote:
> On Sat, Dec 05 2015, Verma, Vishal L wrote:
>
> > On Fri, 2015-12-04 at 15:30 -0800, James Bottomley wrote:
> > [...]
> > > > +ssize_t badblocks_store(struct badblocks *bb, const char *page,
> > > > size_t len,
> > > > + int unack)
> > > [...]
> > > > +int badblocks_init(struct badblocks *bb, int enable)
> > > > +{
> > > > + bb->count = 0;
> > > > + if (enable)
> > > > + bb->shift = 0;
> > > > + else
> > > > + bb->shift = -1;
> > > > + bb->page = kmalloc(PAGE_SIZE, GFP_KERNEL);
> > >
> > > Why not __get_free_page(GFP_KERNEL)? The problem with kmalloc of
> > > an
> > > exactly known page sized quantity is that the slab tracker for
> > > this
> > > requires two contiguous pages for each page because of the
> > > overhead.
> >
> > Cool, I didn't know about __get_free_page - I can fix this up too.
> >
>
> I was reminded of this just recently I thought I should clear up the
> misunderstanding.
>
> kmalloc(PAGE_SIZE) does *not* incur significant overhead and certainly
> does not require two contiguous free pages.
> If you "grep kmalloc-4096 /proc/slabinfo" you will note that both
> objperslab and pagesperslab are 1. So one page is used to store each
> 4096 byte allocation.
>
> To quote the email from Linus which reminded me about this
>
> > If you
> > want to allocate a page, and get a pointer, just use "kmalloc()".
> > Boom, done!
>
> https://lkml.org/lkml/2015/12/21/605
>
> There probably is a small CPU overhead from using kmalloc, but no
> memory
> overhead.
Thanks Neil.
I just read the rest of that thread - and I'm wondering if we should
change back to kzalloc here.
The one thing __get_free_page gets us is PAGE_SIZE-aligned memory. Do
you think that would be better for this use? (I can't think of any). If
not, I can send out a new version reverting back to kzalloc.
-Vishal
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: RAID10 Write Performance
From: Marc Smith @ 2015-12-22 19:36 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAHkw+LdtZY=d6kCtaTe5Uu8nNXOgF+yCAjJeHWJ2TrVAuNoDeA@mail.gmail.com>
Solved... appears it was the write-intent bitmap that caused the
performance issues. I discovered if I left the test running longer
than 60 seconds, the performance would eventually climb to where I'd
expect it. I ran 'mdadm --grow --bitmap=none /dev/md0' and now random
write performance is high/good/stable right off the bat.
--Marc
On Fri, Dec 18, 2015 at 1:43 PM, Marc Smith <marc.smith@mcc.edu> wrote:
> Hi,
>
> I'm testing a (24) slot SSD array (Supermicro) with MD RAID. The setup
> consists of the Supermicro chassis, (24) Pliant LB406M SAS SSD drives,
> (3) Avago/LSI SAS3008 SAS HBAs, and (2) Intel Xeon E5-2660 2.60GHz
> processors.
>
> The (24) SSDs are directly connected (pass-through back-plane) to the
> (3) SAS HBAs (eight drives per HBA) with no SAS expanders.
>
> I'm planning to use RAID10 for this system. I started by playing with
> some performance configurations, I'm specifically looking at random IO
> performance.
>
> The test commands I've been using with fio are the following:
> 4K 100% random, 100% READ: fio --bs=4k --direct=1 --rw=randread
> --ioengine=libaio --iodepth=16 --numjobs=16 --name=/dev/md0
> --runtime=60
> 4K 100% random, 100% WRITE: fio --bs=4k --direct=1 --rw=randwrite
> --ioengine=libaio --iodepth=16 --numjobs=16 --name=/dev/md0
> --runtime=60
>
> As a benchmark, I initially tested all twenty-four drives using RAID0;
> using a 8K chunk size and here are the numbers I got:
> 4K random read: 645,233 IOPS
> 4K random write: 309,879 IOPS
>
> Not too shabby... obviously these are just for bench-marking, the plan
> is to use RAID10 for production.
>
> So, I won't go into the specifics of all the tests, but I've tried
> quite a few different RAID10 configurations: Nested RAID 10 (1+0) -
> RAID 0 (stripe) built with RAID 1 (mirror) arrays, Nested RAID 10
> (0+1) - RAID 1 (mirror) built with RAID 0 (stripe) arrays, and
> "Complex" RAID 10 - Near Layout / 2.
>
> All of these yield very similar results using (12) of the disks spread
> across the (3) HBAs. As an example:
> Nested RAID 10 (0+1) - RAID 1 (mirror) built with RAID 0 (stripe) arrays
> For the (2) stripe sets (2 disks per HBA, 6 total per set):
> mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=6
> --chunk=64K /dev/sda1 /dev/sdb1 /dev/sdi1 /dev/sdj1 /dev/sdq1
> /dev/sdr1
> mdadm --create --verbose /dev/md1 --level=stripe --raid-devices=6
> --chunk=64K /dev/sdc1 /dev/sdd1 /dev/sdk1 /dev/sdl1 /dev/sds1
> /dev/sdt1
> For the (1) mirror set (consisting of the 2 stripe sets):
> mdadm --create --verbose /dev/md2 --level=mirror --raid-devices=2
> /dev/md0 /dev/md1
>
> Running the random 4K performance tests described above yields the
> following results for the RAID10 array:
> 4K random read: 276,967 IOPS
> 4K random write: 643 IOPS
>
>
> The read numbers seem in-line with what I expected, but the writes are
> absolutely dismal. I expect them not be where the read numbers are,
> but this is really, really low! I gotta have something configured
> incorrectly, right?
>
> I've experimented with different chunk sizes, and haven't gotten much
> of a change in the write numbers. Again, I've tried several different
> variations of a "RAID10" configuration (nested 1+0, nested 0+1,
> complex using near/2) and all yield very similar results: Good read
> performance, extremely poor write performance.
>
> Even the throughput when doing a sequential test with the writes is
> not where I'd expect it to be, so something definitely seems to be up
> when mixing RAID levels 0 and 1. I didn't explore all the extremes of
> the chunk sizes, so perhaps its as simple as that? I haven't tested
> the "far" and "offset" layouts of RAID10 yet, but I'm not hopeful its
> going to be any different.
>
>
> Here is what I'm using:
> Linux 3.14.57 (vanilla)
> mdadm - v3.3.2 - 21st August 2014
> fio-2.0.13
>
>
> Any ideas or suggestions would be greatly appreciated. Just as a
> simple test, I created a RAID5 volume using (4) of the SSDs and ran
> the same random IO performance tests:
> 4K random read: 169,026 IOPS
> 4K random write: 12,682 IOPS
>
> Not sure with the default RAID5 mdadm creation command that we get any
> write cache, but we're getting ~ 12K IOPS with RAID5. Not great, but
> when compared to the 643 IOPS with RAID10...
>
>
> Thanks in advance!
>
>
> --Marc
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Doug Dumitru @ 2015-12-22 18:33 UTC (permalink / raw)
To: Dallas Clement, Robert Kierski; +Cc: Linux-RAID
In-Reply-To: <CAE9DZUQoNh2uU1h4okY0Fz5wzVv6-ZTnet9tS-dNQRHsLnPvNg@mail.gmail.com>
Robert and Dallas,
The patch is an astonishingly single case and has a few usage caveats.
It only works when IO is precisely aligned on stripe boundaries. If
anything is off-aligned, or even is an aligned case is encountered and
the stripe cache is not empty, the patch special case does not happen.
Second, the patch assumes that your application layer "makes sense"
and will not try to read a block that is in the middle of being
written.
The patch is in use on production servers, but with still more
caveats. It turns off if the array is not clean or if a rebuild or
check is in progress.
Here is "raid5.c" from CentOS 7 with the patch applied:
https://drive.google.com/file/d/0B3T4AZzjEGVkbUYzeVZqbkIzN1E/view?usp=sharing
The modified areas are all inside of #ifdef EASYCO conditionals. I
did not want to post this as a patch here as this is not appropriate
code for general use.
-- Some comments on stripe cache --
The stripe cache is a lot of overhead for this particular case, but
still works quite well compared to the alternatives. Most benchmarks
I see with high-end raid cards cannot get to 1GB/sec either on raid-5
or raid-6.
Moving away from the stripe cache, especially dynamically, might open
up a nasty set of locking semantics.
-- Some comments on the raid background thread --
With most "reasonable" disk sets, the single raid thread is fine for
raid-5 at 1.8GB/sec. If you want to get raid-6 faster, you need more
cores. With my E5-1650 v3 I get just over 8 GB/sec with raid-6, most
of which is the raid-6 parity compute code. Multi-socket E5s might do
a little better, but NUMA throws all sorts of interesting performance
tuning issues at our proprietary layer that is above raid.
-- Some comments on benchmarks --
If you run benchmarks like fio, you will get IO patterns that never
happen "in live datasets". For example, a real file system will never
read a block that is being written. This is a side effect of the file
systems use of pages as cache and writes that come from dirty pages.
Benchmarks just pump random numbers and overlaps are allowed. This
means you must write code that survives the benchmarks, but optimizing
for a benchmark in some areas is dubious.
-- Some comments on RMW and SSDs --
One reason I wrote this patch was to keep SSDs happy. If you write to
SSDs "perfectly" they never degrade and stay at full performance. If
you do any random writing, the SSDs eventually need to do some space
management (garbage collection). Even the 2-3% of RMW that I see
without the patch is enough to cost 3x of SSD wear with some drives.
Doug Dumitru
WildFire Storage
On Tue, Dec 22, 2015 at 8:48 AM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Tue, Dec 22, 2015 at 12:15 AM, Doug Dumitru <doug@easyco.com> wrote:
>> My apologies for diving in so late.
>>
>> I routinely run 24 drive raid-5 sets with SSDs. Chunk is set at 32K
>> and the applications only writes "perfect" 736K "stripes". The SSDs
>> are Samsung 850 pros on dedicated LSI 3008 SAS ports and are at "new"
>> preconditioning (ie, they are at full speed) or just over 500 MB/sec.
>> CPU is a single E5-1650 v3.
>>
>> With stock RAID-5 code, I get about 1.8 GB/sec, q=4.
>>
>> Now this application is writing from kernel space
>> (generic_make_request w/ q waiting for completion callback). There
>> are a lot of RMW operations happening here. I think the raid-5
>> background thread is waking up asynchronously when only a part of the
>> write has been buffered into stripe cache pages. The bio going into
>> the raid layer is a single bio, so nothing is being carved up on the
>> request end. The raid-5 helper thread also saturates a cpu core
>> (which is about as fast as you can get with an E5-1650).
>>
>> If I patch raid5.ko with special case code to avoid the stripe cache
>> and just compute parity and go, the write throughput goes up above
>> 11GB/sec.
>>
>> This is obviously an impossible IO pattern for most applications, but
>> does confirm that the upper limit of (n-1)*bw is "possible", but not
>> with the current stripe cache logic in the raid layer.
>>
>> Doug Dumitru
>> WildFire Storage
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>> If I patch raid5.ko with special case code to avoid the stripe cache
>> and just compute parity and go, the write throughput goes up above
>> 11GB/sec.
>
> Hi Doug. This is really quite astounding and encouraging! Would you
> be willing to share your patch? I am eager to give it a try for RAID
> 5 and 6.
>
>> Now this application is writing from kernel space
>> (generic_make_request w/ q waiting for completion callback). There
>> are a lot of RMW operations happening here. I think the raid-5
>> background thread is waking up asynchronously when only a part of the
>> write has been buffered into stripe cache pages.
>
> I am also anxious to hear from anyone who maintains the stripe cache
> code. I am seeing similar behavior when I monitor writes of perfectly
> stripe-aligned blocks. The # of RMWs are smallish and seem to vary,
> but still I do not expect to see any of them!
--
Doug Dumitru
EasyCo LLC
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-22 16:48 UTC (permalink / raw)
To: doug; +Cc: Linux-RAID
In-Reply-To: <CAFx4rwTUiMo=1+JXpY3P5BhZYSQC8MVYhTJfdJd7gL6r9WSAiA@mail.gmail.com>
On Tue, Dec 22, 2015 at 12:15 AM, Doug Dumitru <doug@easyco.com> wrote:
> My apologies for diving in so late.
>
> I routinely run 24 drive raid-5 sets with SSDs. Chunk is set at 32K
> and the applications only writes "perfect" 736K "stripes". The SSDs
> are Samsung 850 pros on dedicated LSI 3008 SAS ports and are at "new"
> preconditioning (ie, they are at full speed) or just over 500 MB/sec.
> CPU is a single E5-1650 v3.
>
> With stock RAID-5 code, I get about 1.8 GB/sec, q=4.
>
> Now this application is writing from kernel space
> (generic_make_request w/ q waiting for completion callback). There
> are a lot of RMW operations happening here. I think the raid-5
> background thread is waking up asynchronously when only a part of the
> write has been buffered into stripe cache pages. The bio going into
> the raid layer is a single bio, so nothing is being carved up on the
> request end. The raid-5 helper thread also saturates a cpu core
> (which is about as fast as you can get with an E5-1650).
>
> If I patch raid5.ko with special case code to avoid the stripe cache
> and just compute parity and go, the write throughput goes up above
> 11GB/sec.
>
> This is obviously an impossible IO pattern for most applications, but
> does confirm that the upper limit of (n-1)*bw is "possible", but not
> with the current stripe cache logic in the raid layer.
>
> Doug Dumitru
> WildFire Storage
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> If I patch raid5.ko with special case code to avoid the stripe cache
> and just compute parity and go, the write throughput goes up above
> 11GB/sec.
Hi Doug. This is really quite astounding and encouraging! Would you
be willing to share your patch? I am eager to give it a try for RAID
5 and 6.
> Now this application is writing from kernel space
> (generic_make_request w/ q waiting for completion callback). There
> are a lot of RMW operations happening here. I think the raid-5
> background thread is waking up asynchronously when only a part of the
> write has been buffered into stripe cache pages.
I am also anxious to hear from anyone who maintains the stripe cache
code. I am seeing similar behavior when I monitor writes of perfectly
stripe-aligned blocks. The # of RMWs are smallish and seem to vary,
but still I do not expect to see any of them!
^ permalink raw reply
* Re: [PATCH 3/3] raid5: allow r5l_io_unit allocations to fail
From: Christoph Hellwig @ 2015-12-22 15:20 UTC (permalink / raw)
To: NeilBrown; +Cc: Shaohua Li, linux-raid
In-Reply-To: <87vb7s680r.fsf@notabene.neil.brown.name>
> I wonder if we should have a mempool for these io units too.
> We would allocate with GFP_ATOMIC (or similar) so the allocation woult
> fail instead of blocking, but we would then know that an allocation
> could only fail if there was another request in flight. So the place
> where we free an io_unit would be the obviously correct place to trigger
> a retry of the delayed-due-to-mem-allocation-failure stripes.
>
> So I think I would prefer two lists, another mempool, and very well
> defined places to retry the two lists. Is that over-engineering?
How about the variant below (relative to md/for-next)? This implements
the above and passes testing fine:
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 18de1fc..4fa9457 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -75,7 +75,10 @@ struct r5l_log {
struct list_head finished_ios; /* io_units which settle down in log disk */
struct bio flush_bio;
+ struct list_head no_mem_stripes; /* pending stripes, -ENOMEM */
+
struct kmem_cache *io_kc;
+ mempool_t *io_pool;
struct bio_set *bs;
mempool_t *meta_pool;
@@ -287,9 +290,10 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
struct r5l_io_unit *io;
struct r5l_meta_block *block;
- io = kmem_cache_zalloc(log->io_kc, GFP_ATOMIC);
+ io = mempool_alloc(log->io_pool, GFP_ATOMIC);
if (!io)
return NULL;
+ memset(io, 0, sizeof(*io));
io->log = log;
INIT_LIST_HEAD(&io->log_sibling);
@@ -490,24 +494,25 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
mutex_lock(&log->io_mutex);
/* meta + data */
reserve = (1 + write_disks) << (PAGE_SHIFT - 9);
- if (!r5l_has_free_space(log, reserve))
- goto err_retry;
+ if (!r5l_has_free_space(log, reserve)) {
+ spin_lock(&log->no_space_stripes_lock);
+ list_add_tail(&sh->log_list, &log->no_space_stripes);
+ spin_unlock(&log->no_space_stripes_lock);
+
+ r5l_wake_reclaim(log, reserve);
+ goto out_unlock;
+ }
ret = r5l_log_stripe(log, sh, data_pages, parity_pages);
- if (ret)
- goto err_retry;
+ if (ret) {
+ spin_lock_irq(&log->io_list_lock);
+ list_add_tail(&sh->log_list, &log->no_mem_stripes);
+ spin_unlock_irq(&log->io_list_lock);
+ }
out_unlock:
mutex_unlock(&log->io_mutex);
return 0;
-
-err_retry:
- spin_lock(&log->no_space_stripes_lock);
- list_add_tail(&sh->log_list, &log->no_space_stripes);
- spin_unlock(&log->no_space_stripes_lock);
-
- r5l_wake_reclaim(log, reserve);
- goto out_unlock;
}
void r5l_write_stripe_run(struct r5l_log *log)
@@ -559,6 +564,21 @@ static sector_t r5l_reclaimable_space(struct r5l_log *log)
log->next_checkpoint);
}
+static void r5l_run_no_mem_stripe(struct r5l_log *log)
+{
+ struct stripe_head *sh;
+
+ assert_spin_locked(&log->io_list_lock);
+
+ if (!list_empty(&log->no_mem_stripes)) {
+ sh = list_first_entry(&log->no_mem_stripes,
+ struct stripe_head, log_list);
+ list_del_init(&sh->log_list);
+ set_bit(STRIPE_HANDLE, &sh->state);
+ raid5_release_stripe(sh);
+ }
+}
+
static bool r5l_complete_finished_ios(struct r5l_log *log)
{
struct r5l_io_unit *io, *next;
@@ -575,7 +595,8 @@ static bool r5l_complete_finished_ios(struct r5l_log *log)
log->next_cp_seq = io->seq;
list_del(&io->log_sibling);
- kmem_cache_free(log->io_kc, io);
+ mempool_free(io, log->io_pool);
+ r5l_run_no_mem_stripe(log);
found = true;
}
@@ -1189,6 +1210,10 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (!log->io_kc)
goto io_kc;
+ log->io_pool = mempool_create_slab_pool(R5L_POOL_SIZE, log->io_kc);
+ if (!log->io_pool)
+ goto io_pool;
+
log->bs = bioset_create(R5L_POOL_SIZE, 0);
if (!log->bs)
goto io_bs;
@@ -1203,6 +1228,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
goto reclaim_thread;
init_waitqueue_head(&log->iounit_wait);
+ INIT_LIST_HEAD(&log->no_mem_stripes);
+
INIT_LIST_HEAD(&log->no_space_stripes);
spin_lock_init(&log->no_space_stripes_lock);
@@ -1219,6 +1246,8 @@ reclaim_thread:
out_mempool:
bioset_free(log->bs);
io_bs:
+ mempool_destroy(log->io_pool);
+io_pool:
kmem_cache_destroy(log->io_kc);
io_kc:
kfree(log);
^ permalink raw reply related
* RE: best base / worst case RAID 5,6 write speeds
From: Robert Kierski @ 2015-12-22 14:34 UTC (permalink / raw)
To: doug@easyco.com; +Cc: Linux-RAID
In-Reply-To: <CAFx4rwTUiMo=1+JXpY3P5BhZYSQC8MVYhTJfdJd7gL6r9WSAiA@mail.gmail.com>
Hey Doug,
I would be interested in seeing the patch you're talking about. I wonder if that code couldn't be turned on/off with a tuning parameter or module param.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Doug Dumitru
Sent: Tuesday, December 22, 2015 12:16 AM
Cc: Linux-RAID
Subject: Re: best base / worst case RAID 5,6 write speeds
My apologies for diving in so late.
I routinely run 24 drive raid-5 sets with SSDs. Chunk is set at 32K and the applications only writes "perfect" 736K "stripes". The SSDs are Samsung 850 pros on dedicated LSI 3008 SAS ports and are at "new"
preconditioning (ie, they are at full speed) or just over 500 MB/sec.
CPU is a single E5-1650 v3.
With stock RAID-5 code, I get about 1.8 GB/sec, q=4.
Now this application is writing from kernel space (generic_make_request w/ q waiting for completion callback). There are a lot of RMW operations happening here. I think the raid-5 background thread is waking up asynchronously when only a part of the write has been buffered into stripe cache pages. The bio going into the raid layer is a single bio, so nothing is being carved up on the request end. The raid-5 helper thread also saturates a cpu core (which is about as fast as you can get with an E5-1650).
If I patch raid5.ko with special case code to avoid the stripe cache and just compute parity and go, the write throughput goes up above 11GB/sec.
This is obviously an impossible IO pattern for most applications, but does confirm that the upper limit of (n-1)*bw is "possible", but not with the current stripe cache logic in the raid layer.
Doug Dumitru
WildFire Storage
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH v2 1/2] md: bcache: Replace get_seconds with ktime_get_seconds
From: Shraddha Barke @ 2015-12-22 13:53 UTC (permalink / raw)
To: Kent Overstreet, Neil Brown; +Cc: linux-raid, Arnd Bergmann, Shraddha Barke
32-bit systems using function get_seconds will break in the year 2038,
in order to avoid that replace the code with more appropriate
function ktime_get_seconds which is y2038 safe.
Function ktime_get_seconds is used since it uses monotonic instead of
real time and thus will not cause overflow.
Signed-off-by: Shraddha Barke <shraddha.6596@gmail.com>
---
Changes in v2-
Drop use of ktime_get_seconds for last_mount since it is not appropriate.
drivers/md/bcache/super.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index 679a093..f7500cd 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -696,7 +696,7 @@ static void bcache_device_detach(struct bcache_device *d)
SET_UUID_FLASH_ONLY(u, 0);
memcpy(u->uuid, invalid_uuid, 16);
- u->invalidated = cpu_to_le32(get_seconds());
+ u->invalidated = cpu_to_le32(ktime_get_seconds());
bch_uuid_write(d->c);
}
@@ -927,7 +927,7 @@ void bch_cached_dev_detach(struct cached_dev *dc)
int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c)
{
- uint32_t rtime = cpu_to_le32(get_seconds());
+ uint32_t rtime = cpu_to_le32(ktime_get_seconds());
struct uuid_entry *u;
char buf[BDEVNAME_SIZE];
@@ -959,7 +959,7 @@ int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c)
(BDEV_STATE(&dc->sb) == BDEV_STATE_STALE ||
BDEV_STATE(&dc->sb) == BDEV_STATE_NONE)) {
memcpy(u->uuid, invalid_uuid, 16);
- u->invalidated = cpu_to_le32(get_seconds());
+ u->invalidated = cpu_to_le32(ktime_get_seconds());
u = NULL;
}
@@ -1261,7 +1261,7 @@ int bch_flash_dev_create(struct cache_set *c, uint64_t size)
get_random_bytes(u->uuid, 16);
memset(u->label, 0, 32);
- u->first_reg = u->last_reg = cpu_to_le32(get_seconds());
+ u->first_reg = u->last_reg = cpu_to_le32(ktime_get_seconds());
SET_UUID_FLASH_ONLY(u, 1);
u->sectors = size >> 9;
--
2.1.4
^ permalink raw reply related
* [PATCH 2/2] md: Replace get_seconds with ktime_get_seconds
From: Shraddha Barke @ 2015-12-22 13:06 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, Arnd Bergmann, Shraddha Barke
32-bit systems using function get_seconds will break in year 2038,
in order to avoid that replace the code with more appropriate function
ktime_get_seconds which is y2038 safe. Function ktime_get_seconds use
monotonic instead of real time and therefore will not cause overflow
Signed-off-by: Shraddha Barke <shraddha.6596@gmail.com>
---
drivers/md/md.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 807095f..8dd74bb 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2333,7 +2333,7 @@ repeat:
spin_lock(&mddev->lock);
- mddev->utime = get_seconds();
+ mddev->utime = ktime_get_seconds();
if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
force_change = 1;
@@ -6347,13 +6347,13 @@ static int set_array_info(struct mddev *mddev, mdu_array_info_t *info)
/* ensure mddev_put doesn't delete this now that there
* is some minimal configuration.
*/
- mddev->ctime = get_seconds();
+ mddev->ctime = ktime_get_seconds();
return 0;
}
mddev->major_version = MD_MAJOR_VERSION;
mddev->minor_version = MD_MINOR_VERSION;
mddev->patch_version = MD_PATCHLEVEL_VERSION;
- mddev->ctime = get_seconds();
+ mddev->ctime = ktime_get_seconds();
mddev->level = info->level;
mddev->clevel[0] = 0;
--
2.1.4
^ permalink raw reply related
* raid5 async_xor: sleep in atomic
From: Stanislav Samsonov @ 2015-12-22 11:58 UTC (permalink / raw)
To: linux-raid
Hi,
Kernel 4.1.3 : there is some troubling kernel message that shows up
after enabling CONFIG_DEBUG_ATOMIC_SLEEP and testing DMA XOR
acceleration for raid5:
BUG: sleeping function called from invalid context at mm/mempool.c:320
in_atomic(): 1, irqs_disabled(): 0, pid: 1048, name: md127_raid5
INFO: lockdep is turned off.
CPU: 1 PID: 1048 Comm: md127_raid5 Not tainted 4.1.15.alpine.1-dirty #1
Hardware name: Annapurna Labs Alpine
[<c00169d8>] (unwind_backtrace) from [<c0012a78>] (show_stack+0x10/0x14)
[<c0012a78>] (show_stack) from [<c07462ec>] (dump_stack+0x80/0xb4)
[<c07462ec>] (dump_stack) from [<c00bf2f0>] (mempool_alloc+0x68/0x13c)
[<c00bf2f0>] (mempool_alloc) from [<c041c9b4>]
(dmaengine_get_unmap_data+0x24/0x4c)
[<c041c9b4>] (dmaengine_get_unmap_data) from [<c03a8084>]
(async_xor_val+0x60/0x3a0)
[<c03a8084>] (async_xor_val) from [<c058e4c0>] (raid_run_ops+0xb70/0x1248)
[<c058e4c0>] (raid_run_ops) from [<c05915d4>] (handle_stripe+0x1068/0x22a8)
[<c05915d4>] (handle_stripe) from [<c0592ae4>]
(handle_active_stripes+0x2d0/0x3dc)
[<c0592ae4>] (handle_active_stripes) from [<c059300c>] (raid5d+0x384/0x5b0)
[<c059300c>] (raid5d) from [<c059db6c>] (md_thread+0x114/0x138)
[<c059db6c>] (md_thread) from [<c0042d54>] (kthread+0xe4/0x104)
[<c0042d54>] (kthread) from [<c000f658>] (ret_from_fork+0x14/0x3c)
The reason is that async_xor_val() in crypto/async_tx/async_xor.c is
called in atomic context (preemption disabled) by raid_run_ops(). Then
it calls dmaengine_get_unmap_data() an then mempool_alloc() with
GFP_NOIO flag - this allocation type might sleep under some condition.
Checked latest kernel 4.3 and it has exactly same flow.
Any advice regarding this issue?
Thanks,
Slava Samsonov
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Doug Dumitru @ 2015-12-22 6:15 UTC (permalink / raw)
Cc: Linux-RAID
In-Reply-To: <CAFx4rwT8xgwZ0OWaLLsZvhMskiwmY54MzHgnnEPaswByeRrXxQ@mail.gmail.com>
My apologies for diving in so late.
I routinely run 24 drive raid-5 sets with SSDs. Chunk is set at 32K
and the applications only writes "perfect" 736K "stripes". The SSDs
are Samsung 850 pros on dedicated LSI 3008 SAS ports and are at "new"
preconditioning (ie, they are at full speed) or just over 500 MB/sec.
CPU is a single E5-1650 v3.
With stock RAID-5 code, I get about 1.8 GB/sec, q=4.
Now this application is writing from kernel space
(generic_make_request w/ q waiting for completion callback). There
are a lot of RMW operations happening here. I think the raid-5
background thread is waking up asynchronously when only a part of the
write has been buffered into stripe cache pages. The bio going into
the raid layer is a single bio, so nothing is being carved up on the
request end. The raid-5 helper thread also saturates a cpu core
(which is about as fast as you can get with an E5-1650).
If I patch raid5.ko with special case code to avoid the stripe cache
and just compute parity and go, the write throughput goes up above
11GB/sec.
This is obviously an impossible IO pattern for most applications, but
does confirm that the upper limit of (n-1)*bw is "possible", but not
with the current stripe cache logic in the raid layer.
Doug Dumitru
WildFire Storage
^ permalink raw reply
* Re: [PATCH v2 1/3] badblocks: Add core badblock management code
From: NeilBrown @ 2015-12-22 5:34 UTC (permalink / raw)
To: Verma, Vishal L, James.Bottomley@HansenPartnership.com
Cc: linux-raid@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-block@vger.kernel.org,
jmoyer@redhat.com, axboe@fb.com
In-Reply-To: <1449273524.16905.103.camel@intel.com>
[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]
On Sat, Dec 05 2015, Verma, Vishal L wrote:
> On Fri, 2015-12-04 at 15:30 -0800, James Bottomley wrote:
> [...]
>> > +ssize_t badblocks_store(struct badblocks *bb, const char *page,
>> > size_t len,
>> > + int unack)
>> [...]
>> > +int badblocks_init(struct badblocks *bb, int enable)
>> > +{
>> > + bb->count = 0;
>> > + if (enable)
>> > + bb->shift = 0;
>> > + else
>> > + bb->shift = -1;
>> > + bb->page = kmalloc(PAGE_SIZE, GFP_KERNEL);
>>
>> Why not __get_free_page(GFP_KERNEL)? The problem with kmalloc of an
>> exactly known page sized quantity is that the slab tracker for this
>> requires two contiguous pages for each page because of the overhead.
>
> Cool, I didn't know about __get_free_page - I can fix this up too.
>
I was reminded of this just recently I thought I should clear up the
misunderstanding.
kmalloc(PAGE_SIZE) does *not* incur significant overhead and certainly
does not require two contiguous free pages.
If you "grep kmalloc-4096 /proc/slabinfo" you will note that both
objperslab and pagesperslab are 1. So one page is used to store each
4096 byte allocation.
To quote the email from Linus which reminded me about this
> If you
> want to allocate a page, and get a pointer, just use "kmalloc()".
> Boom, done!
https://lkml.org/lkml/2015/12/21/605
There probably is a small CPU overhead from using kmalloc, but no memory
overhead.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [MDADM PATCH] Check and remove bitmap first when reshape to raid0
From: NeilBrown @ 2015-12-22 4:20 UTC (permalink / raw)
To: Xiao Ni; +Cc: Jes.Sorensen, linux-raid
In-Reply-To: <1450753774-4358-1-git-send-email-xni@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]
On Tue, Dec 22 2015, Xiao Ni wrote:
> If reshape one raid device with bitmap to raid0, the reshape progress will
> start. But it'll fail and lose some components. So it should remove bitmap
> first.
>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> ---
> Grow.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/Grow.c b/Grow.c
> index 6dfb9c9..db4daa8 100755
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1590,6 +1590,15 @@ int Grow_reshape(char *devname, int fd,
> pr_err("Cannot increase raid-disks on this array beyond %d\n", st->max_devs);
> return 1;
> }
> + if (s->level == 0 &&
> + (array.state & (1<<MD_SB_BITMAP_PRESENT)) &&
> + !(array.state & (1<<MD_SB_CLUSTERED))) {
> + array.state &= ~(1<<MD_SB_BITMAP_PRESENT);
> + if (ioctl(fd, SET_ARRAY_INFO, &array)!= 0) {
> + pr_err("failed to remove internal bitmap.\n");
> + return 1;
> + }
> + }
>
> /* in the external case we need to check that the requested reshape is
> * supported, and perform an initial check that the container holds the
applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-22 4:13 UTC (permalink / raw)
To: Tejas Rao
Cc: Scott Sinno, linux-raid,
Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <5678A908.6070401@bnl.gov>
[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]
On Tue, Dec 22 2015, Tejas Rao wrote:
> Each GPFS disk (block device) has a list of servers associated with it.
> When the first storage server fails (expired disk lease), the storage
> node is expelled and a different server which also sees the shared
> storage will do I/O.
In that case something probably could be made to work with md/raid5
using much of the cluster support developed for md/raid1.
The raid5 module would take a cluster lock that covered some region of
the array and would not need to release it until a fail-over happened.
So there would be little performance penalty.
The simplest approach would be to lock the whole array. This would
preclude the possibility of different partitions being accessed from
different nodes. Maybe that is not a problem. If it were, a solution
could probably be found but there would be little point searching for a
solution before a clear need was presented.
>
> In the future ,we would prefer to use linux raid (RAID6) in a shared
> environment shielding us against server failures. Unfortunately we can
> only do this after Redhat supports such an environment with linux raid.
> Currently they do not support this even in an active/passive environment
> (only one server can have a md device assembled and active regardless).
Obviously that is something you would need to discuss with Redhat.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [MDADM PATCH] Check and remove bitmap first when reshape to raid0
From: Xiao Ni @ 2015-12-22 3:09 UTC (permalink / raw)
To: neilb; +Cc: Jes.Sorensen, linux-raid
If reshape one raid device with bitmap to raid0, the reshape progress will
start. But it'll fail and lose some components. So it should remove bitmap
first.
Signed-off-by: Xiao Ni <xni@redhat.com>
---
Grow.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/Grow.c b/Grow.c
index 6dfb9c9..db4daa8 100755
--- a/Grow.c
+++ b/Grow.c
@@ -1590,6 +1590,15 @@ int Grow_reshape(char *devname, int fd,
pr_err("Cannot increase raid-disks on this array beyond %d\n", st->max_devs);
return 1;
}
+ if (s->level == 0 &&
+ (array.state & (1<<MD_SB_BITMAP_PRESENT)) &&
+ !(array.state & (1<<MD_SB_CLUSTERED))) {
+ array.state &= ~(1<<MD_SB_BITMAP_PRESENT);
+ if (ioctl(fd, SET_ARRAY_INFO, &array)!= 0) {
+ pr_err("failed to remove internal bitmap.\n");
+ return 1;
+ }
+ }
/* in the external case we need to check that the requested reshape is
* supported, and perform an initial check that the container holds the
--
2.4.3
^ permalink raw reply related
* Re: clustered MD - beyond RAID1
From: Tejas Rao @ 2015-12-22 2:33 UTC (permalink / raw)
To: Aaron Knister; +Cc: NeilBrown, Scott Sinno, linux-raid
In-Reply-To: <5678AC55.7070606@nasa.gov>
On 12/21/2015 20:50, Aaron Knister wrote:
> Hi Tejas et al,
>
> I'm fairly confident in saying that GPFS can have many servers actively
> writing to a given NSD (LUN) at any given time. In our production
> environment the NSDs have 6 servers defined and clients more or less
> write to whichever one their little hearts desire. Do you think it's
> possible that the explicit primary/secondary concept is from an older
> version of GPFS? I'm not sure what the locking granularity is for
> NSDs/disks, but even if it's a single GPFS FS block and that block size
> corresponds to the stripe width of the array I'm pretty nervous relying
> on that assumption for data integrity :)
>
> The use case here is creating effectively highly available block storage
> from shared JBODs for use by VMs on the servers as well as to be
> exported to other nodes. The filesystem we're using for this is actually
> GPFS. The intent was to use RAID6 in an active/active fashion on two
> nodes sharing a common set of disks. The active/active was in an effort
> to simplify the configuration.
You are probably not defining the NSD parameter "servers=ServerList". If
this parameter is not defined, GPFS assumes that the disks are SAN
attached to all the NSD nodes, in this case there is no
primary/secondary server. Of-course there is no risk of data integrity
even if the "servers" parameter is not defined.
>
> I'm curious now, Redhat doesn't support SW raid failover? I did some
> googling and found this:
>
> https://access.redhat.com/solutions/231643
>
> While I can't read the solution I have to figure that they're now
> supporting that. I might actually explore that for this project.
https://access.redhat.com/solutions/410203
This article states that md raid is not supported in RHEL6/7 under any
circumstances, including active/passive modes.
>
> -Aaron
>
> On 12/21/15 8:09 PM, Tejas Rao wrote:
>> Each GPFS disk (block device) has a list of servers associated with it.
>> When the first storage server fails (expired disk lease), the storage
>> node is expelled and a different server which also sees the shared
>> storage will do I/O.
>>
>> There is a "leaseRecoveryWait" parameter which tells the filesystem
>> manager to wait for few seconds to allow the expelled node to complete
>> any I/O in flight to the shared storage device to avoid any out of order
>> i/O. After this wait time, the filesystem manager completes recovery on
>> the failed node, replaying journal logs, freeing up shared tokens/locks
>> etc. After the recovery is complete a different storage node will do
>> I/O. There is a concept of primary/secondary servers for a given block
>> device. The secondary server will only do I/O when the primary server
>> has failed and this has been confirmed.
>>
>> See "servers=ServerList" in man page for mmcrnsd. ( I don't think I am
>> allowed to send web links)
>>
>> We currently have 10's of petabytes in production using linux md raid.
>> We are currently not sharing md devices, only hardware raid block
>> devices are shared. In our experience hardware raid controllers are
>> expensive. Linux raid has worked well over the years and performance is
>> very good as GPFS coalesces I/O in large filesystem blocksize blocks
>> (8MB) and if aligned properly eliminate RMW (doing full stripe writes)
>> and the need for NVRAM (unless someone is doing POSIX fsync).
>>
>> In the future ,we would prefer to use linux raid (RAID6) in a shared
>> environment shielding us against server failures. Unfortunately we can
>> only do this after Redhat supports such an environment with linux raid.
>> Currently they do not support this even in an active/passive environment
>> (only one server can have a md device assembled and active regardless).
>>
>> Tejas.
>>
>> On 12/21/2015 17:03, NeilBrown wrote:
>> > On Tue, Dec 22 2015, Tejas Rao wrote:
>> >
>> >> GPFS guarantees that only one node will write to a linux block device
>> >> using disk leases.
>> >
>> > Do you have a reference to documentation explaining that?
>> > A few moments searching the internet suggests that a "disk lease" is
>> > much like a heart-beat. A node uses it to say "I'm still alive, please
>> > don't ignore me". I could find no evidence that only one node could
>> > hold a disk lease at any time.
>> >
>> > NeilBrown
>>
>
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Alireza Haghdoost @ 2015-12-22 2:29 UTC (permalink / raw)
To: Tejas Rao
Cc: NeilBrown, Scott Sinno, Linux RAID,
Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <5678A908.6070401@bnl.gov>
On Mon, Dec 21, 2015 at 7:36 PM, Tejas Rao <raot@bnl.gov> wrote:
>
> We currently have 10's of petabytes in production using linux md raid. We
> are currently not sharing md devices, only hardware raid block devices are
> shared. In our experience hardware raid controllers are expensive. Linux
> raid has worked well over the years and performance is very good as GPFS
> coalesces I/O in large filesystem blocksize blocks (8MB) and if aligned
> properly eliminate RMW (doing full stripe writes) and the need for NVRAM
> (unless someone is doing POSIX fsync).
>
Full Stripe Write does not eliminate the need for NVRAM. Stored data
under MD RAID-6 is vulnerable to write-hole data corruption issue.
Yes, hardware RAID controllers are expensive because they would
provide more data reliability and high-availibiltiy that might not be
useful for this use case.
--Alireza
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Aaron Knister @ 2015-12-22 1:50 UTC (permalink / raw)
To: Tejas Rao, NeilBrown; +Cc: Scott Sinno, linux-raid
In-Reply-To: <5678A2B9.6070008@bnl.gov>
[-- Attachment #1: Type: text/plain, Size: 3912 bytes --]
Hi Tejas et al,
I'm fairly confident in saying that GPFS can have many servers actively
writing to a given NSD (LUN) at any given time. In our production
environment the NSDs have 6 servers defined and clients more or less
write to whichever one their little hearts desire. Do you think it's
possible that the explicit primary/secondary concept is from an older
version of GPFS? I'm not sure what the locking granularity is for
NSDs/disks, but even if it's a single GPFS FS block and that block size
corresponds to the stripe width of the array I'm pretty nervous relying
on that assumption for data integrity :)
The use case here is creating effectively highly available block storage
from shared JBODs for use by VMs on the servers as well as to be
exported to other nodes. The filesystem we're using for this is actually
GPFS. The intent was to use RAID6 in an active/active fashion on two
nodes sharing a common set of disks. The active/active was in an effort
to simplify the configuration.
I'm curious now, Redhat doesn't support SW raid failover? I did some
googling and found this:
https://access.redhat.com/solutions/231643
While I can't read the solution I have to figure that they're now
supporting that. I might actually explore that for this project.
-Aaron
On 12/21/15 8:09 PM, Tejas Rao wrote:
> Each GPFS disk (block device) has a list of servers associated with it.
> When the first storage server fails (expired disk lease), the storage
> node is expelled and a different server which also sees the shared
> storage will do I/O.
>
> There is a "leaseRecoveryWait" parameter which tells the filesystem
> manager to wait for few seconds to allow the expelled node to complete
> any I/O in flight to the shared storage device to avoid any out of order
> i/O. After this wait time, the filesystem manager completes recovery on
> the failed node, replaying journal logs, freeing up shared tokens/locks
> etc. After the recovery is complete a different storage node will do
> I/O. There is a concept of primary/secondary servers for a given block
> device. The secondary server will only do I/O when the primary server
> has failed and this has been confirmed.
>
> See "servers=ServerList" in man page for mmcrnsd. ( I don't think I am
> allowed to send web links)
>
> We currently have 10's of petabytes in production using linux md raid.
> We are currently not sharing md devices, only hardware raid block
> devices are shared. In our experience hardware raid controllers are
> expensive. Linux raid has worked well over the years and performance is
> very good as GPFS coalesces I/O in large filesystem blocksize blocks
> (8MB) and if aligned properly eliminate RMW (doing full stripe writes)
> and the need for NVRAM (unless someone is doing POSIX fsync).
>
> In the future ,we would prefer to use linux raid (RAID6) in a shared
> environment shielding us against server failures. Unfortunately we can
> only do this after Redhat supports such an environment with linux raid.
> Currently they do not support this even in an active/passive environment
> (only one server can have a md device assembled and active regardless).
>
> Tejas.
>
> On 12/21/2015 17:03, NeilBrown wrote:
> > On Tue, Dec 22 2015, Tejas Rao wrote:
> >
> >> GPFS guarantees that only one node will write to a linux block device
> >> using disk leases.
> >
> > Do you have a reference to documentation explaining that?
> > A few moments searching the internet suggests that a "disk lease" is
> > much like a heart-beat. A node uses it to say "I'm still alive, please
> > don't ignore me". I could find no evidence that only one node could
> > hold a disk lease at any time.
> >
> > NeilBrown
>
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 842 bytes --]
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Tejas Rao @ 2015-12-22 1:36 UTC (permalink / raw)
To: NeilBrown
Cc: Scott Sinno, linux-raid,
Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <8737uv4fz6.fsf@notabene.neil.brown.name>
Each GPFS disk (block device) has a list of servers associated with it.
When the first storage server fails (expired disk lease), the storage
node is expelled and a different server which also sees the shared
storage will do I/O.
There is a "leaseRecoveryWait" parameter which tells the filesystem
manager to wait for few seconds to allow the expelled node to complete
any I/O in flight to the shared storage device to avoid any out of order
i/O. After this wait time, the filesystem manager completes recovery on
the failed node, replaying journal logs, freeing up shared tokens/locks
etc. After the recovery is complete a different storage node will do
I/O. There is a concept of primary/secondary servers for a given block
device. The secondary server will only do I/O when the primary server
has failed and this has been confirmed.
See "servers=ServerList" in man page for mmcrnsd. ( I don't think I am
allowed to send web links)
We currently have 10's of petabytes in production using linux md raid.
We are currently not sharing md devices, only hardware raid block
devices are shared. In our experience hardware raid controllers are
expensive. Linux raid has worked well over the years and performance is
very good as GPFS coalesces I/O in large filesystem blocksize blocks
(8MB) and if aligned properly eliminate RMW (doing full stripe writes)
and the need for NVRAM (unless someone is doing POSIX fsync).
In the future ,we would prefer to use linux raid (RAID6) in a shared
environment shielding us against server failures. Unfortunately we can
only do this after Redhat supports such an environment with linux raid.
Currently they do not support this even in an active/passive environment
(only one server can have a md device assembled and active regardless).
Tejas.
On 12/21/2015 17:03, NeilBrown wrote:
> On Tue, Dec 22 2015, Tejas Rao wrote:
>
>> GPFS guarantees that only one node will write to a linux block device
>> using disk leases.
>
> Do you have a reference to documentation explaining that?
> A few moments searching the internet suggests that a "disk lease" is
> much like a heart-beat. A node uses it to say "I'm still alive, please
> don't ignore me". I could find no evidence that only one node could
> hold a disk lease at any time.
>
> NeilBrown
>
>
>> Only a node with a disk lease has the right to submit
>> I/O and disk leases expire every 30 secs and needs to be renewed. Lustre
>> and other distributed file systems have other ways of handing this.
>>
>> Using md devices in a shared/clustered environment is something not
>> supported by Redhat on RHEL6 or RHEL7 kernels, so this is something we
>> would not try in our production environments.
>>
>> Tejas.
>>
>> On 12/21/2015 15:47, NeilBrown wrote:
>>> On Tue, Dec 22 2015, Tejas Rao wrote:
>>>
>>>> What if the application is doing the locking and making sure that only 1
>>>> node writes to a md device at a time? Will this work? How are rebuilds
>>>> handled? This would be helpful with distributed filesystems like
>>>> GPFS/lustre etc.
>>>>
>>> You would also need to make sure that the filesystem only wrote from a
>>> single node at a time (or access the block device directly). I doubt
>>> GPFS/lustre make any promise like that, but I'm happy to be educated.
>>>
>>> rebuilds are handled by using a cluster-wide lock to block all writes to
>>> a range of addresses while those stripes are repaired.
>>>
>>> NeilBrown
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-21 23:09 UTC (permalink / raw)
To: Adam Goryachev, Tejas Rao, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <56787D34.7020108@websitemanagers.com.au>
[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]
On Tue, Dec 22 2015, Adam Goryachev wrote:
> On 22/12/15 09:03, NeilBrown wrote:
>> On Tue, Dec 22 2015, Tejas Rao wrote:
>>
>>> On 12/21/2015 15:47, NeilBrown wrote:
>>>> On Tue, Dec 22 2015, Tejas Rao wrote:
>>>>
>>>>> What if the application is doing the locking and making sure that only 1
>>>>> node writes to a md device at a time? Will this work? How are rebuilds
>>>>> handled? This would be helpful with distributed filesystems like
>>>>> GPFS/lustre etc.
>>>>>
>>>> You would also need to make sure that the filesystem only wrote from a
>>>> single node at a time (or access the block device directly). I doubt
>>>> GPFS/lustre make any promise like that, but I'm happy to be educated.
>>>>
>>>> rebuilds are handled by using a cluster-wide lock to block all writes to
>>>> a range of addresses while those stripes are repaired.
>>>>
>>>> NeilBrown
>
> My understanding of MD level cross host RAID was that it would not
> magically create cluster aware filesystems out of non-cluster aware
> filesystems. ie, you wouldn't be able to use the same multi-host RAID
> device on multiple hosts concurrently with ext3.
This is correct. The expectation is that clustered md/raid1 would be
used with a cluster-aware filesystem such as ocfs2 or gpfs. Certainly
not with ext3 or similar.
>
> IMHO, if it was able to behave similar to DRBD, then that would be
> perfect (ie, enforce only a single node can write at a time (unless you
> specifically set it for multi-node write)). The benefit should be that
> you can lose a node without losing your data. After you lose that node,
> you can then "do something" to use the remaining node to access the data
> (eg, mount it, export with iscsi/nfs, etc).
There is a lot of similarity between DRBD and clustered md/raid1.
I don't know the current state of DRBD but it initially assumed each
storage device was local to a single node and so sent data over the
network (i.e. over IP) to "remote" devices.
clustered md/raid1 assumes that all storage is equally accessible to all
nodes (over a 'storage area network', which may still be IP).
So yes: if you lose a node you should not lose functionality.
>
> Currently, this is what I use DRBD for, previously, I've used NBD + MD
> RAID1 to do the same thing. One question though is what advantage
> multi-host MD RAID might have over the existing in-kernel DRBD ? Are
> there plans which show why this is going to be better, have better
> performance, features, etc?
I'm not the driving force behind clustered md/raid1 so I am not
completely familiar with the motivation, but I believe DRBD doesn't, or
didn't, make best possible use of the storage network when every storage
device is connected to every compute node. It is expected that clustered
md/raid1 will.
I *think* DRBD is primarily for pair of nodes (though there is some
multi-node support). clustered md/raid1 is designed to work with
multiple nodes - however big your cluster is.
(DRBD 9.0 appears to support multi-node configurations. I haven't
researched the details)
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Adam Goryachev @ 2015-12-21 22:29 UTC (permalink / raw)
To: NeilBrown, Tejas Rao, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <8737uv4fz6.fsf@notabene.neil.brown.name>
On 22/12/15 09:03, NeilBrown wrote:
> On Tue, Dec 22 2015, Tejas Rao wrote:
>
>> On 12/21/2015 15:47, NeilBrown wrote:
>>> On Tue, Dec 22 2015, Tejas Rao wrote:
>>>
>>>> What if the application is doing the locking and making sure that only 1
>>>> node writes to a md device at a time? Will this work? How are rebuilds
>>>> handled? This would be helpful with distributed filesystems like
>>>> GPFS/lustre etc.
>>>>
>>> You would also need to make sure that the filesystem only wrote from a
>>> single node at a time (or access the block device directly). I doubt
>>> GPFS/lustre make any promise like that, but I'm happy to be educated.
>>>
>>> rebuilds are handled by using a cluster-wide lock to block all writes to
>>> a range of addresses while those stripes are repaired.
>>>
>>> NeilBrown
My understanding of MD level cross host RAID was that it would not
magically create cluster aware filesystems out of non-cluster aware
filesystems. ie, you wouldn't be able to use the same multi-host RAID
device on multiple hosts concurrently with ext3.
IMHO, if it was able to behave similar to DRBD, then that would be
perfect (ie, enforce only a single node can write at a time (unless you
specifically set it for multi-node write)). The benefit should be that
you can lose a node without losing your data. After you lose that node,
you can then "do something" to use the remaining node to access the data
(eg, mount it, export with iscsi/nfs, etc).
Currently, this is what I use DRBD for, previously, I've used NBD + MD
RAID1 to do the same thing. One question though is what advantage
multi-host MD RAID might have over the existing in-kernel DRBD ? Are
there plans which show why this is going to be better, have better
performance, features, etc?
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-21 22:03 UTC (permalink / raw)
To: Tejas Rao, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <56786EA4.2020209@bnl.gov>
[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]
On Tue, Dec 22 2015, Tejas Rao wrote:
> GPFS guarantees that only one node will write to a linux block device
> using disk leases.
Do you have a reference to documentation explaining that?
A few moments searching the internet suggests that a "disk lease" is
much like a heart-beat. A node uses it to say "I'm still alive, please
don't ignore me". I could find no evidence that only one node could
hold a disk lease at any time.
NeilBrown
> Only a node with a disk lease has the right to submit
> I/O and disk leases expire every 30 secs and needs to be renewed. Lustre
> and other distributed file systems have other ways of handing this.
>
> Using md devices in a shared/clustered environment is something not
> supported by Redhat on RHEL6 or RHEL7 kernels, so this is something we
> would not try in our production environments.
>
> Tejas.
>
> On 12/21/2015 15:47, NeilBrown wrote:
>> On Tue, Dec 22 2015, Tejas Rao wrote:
>>
>>> What if the application is doing the locking and making sure that only 1
>>> node writes to a md device at a time? Will this work? How are rebuilds
>>> handled? This would be helpful with distributed filesystems like
>>> GPFS/lustre etc.
>>>
>> You would also need to make sure that the filesystem only wrote from a
>> single node at a time (or access the block device directly). I doubt
>> GPFS/lustre make any promise like that, but I'm happy to be educated.
>>
>> rebuilds are handled by using a cluster-wide lock to block all writes to
>> a range of addresses while those stripes are repaired.
>>
>> NeilBrown
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: Tejas Rao @ 2015-12-21 21:27 UTC (permalink / raw)
To: NeilBrown, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <87bn9j4jhr.fsf@notabene.neil.brown.name>
GPFS guarantees that only one node will write to a linux block device
using disk leases. Only a node with a disk lease has the right to submit
I/O and disk leases expire every 30 secs and needs to be renewed. Lustre
and other distributed file systems have other ways of handing this.
Using md devices in a shared/clustered environment is something not
supported by Redhat on RHEL6 or RHEL7 kernels, so this is something we
would not try in our production environments.
Tejas.
On 12/21/2015 15:47, NeilBrown wrote:
> On Tue, Dec 22 2015, Tejas Rao wrote:
>
>> What if the application is doing the locking and making sure that only 1
>> node writes to a md device at a time? Will this work? How are rebuilds
>> handled? This would be helpful with distributed filesystems like
>> GPFS/lustre etc.
>>
> You would also need to make sure that the filesystem only wrote from a
> single node at a time (or access the block device directly). I doubt
> GPFS/lustre make any promise like that, but I'm happy to be educated.
>
> rebuilds are handled by using a cluster-wide lock to block all writes to
> a range of addresses while those stripes are repaired.
>
> NeilBrown
^ permalink raw reply
* Re: [MDADM PATCH] Check and remove bitmap first when reshape to raid0
From: NeilBrown @ 2015-12-21 21:06 UTC (permalink / raw)
To: Xiao Ni; +Cc: Jes.Sorensen, linux-raid
In-Reply-To: <1450686229-6157-1-git-send-email-xni@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]
On Mon, Dec 21 2015, Xiao Ni wrote:
> If reshape one raid device with bitmap to raid0, the reshape progress will
> start. But it'll fail and lose some components. So it should remove bitmap
> first.
>
> And it shouldn't close fd when try to call open_dev_excl. Because it's an
> input argument. It should be closed in the function who open it.
I thought I mentioned this before, but please don't include two very
different things in the same patch. It is sometimes OK if one is just
a line or two. But most of this patch has got nothing to do with the
subject line.
So this has to be two patches. Except that you have have to try rather
hard to convince me that the second patch actually has any value. I
cannot see that it improves the code in any important way.
>
> There are some places that return(1) directly but don't free subarray and
> close cfd in Grow_reshape. So add fallback: to do free(subarray) and
> close(cfd).
>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> ---
> Grow.c | 37 ++++++++++++++++++++++++++++---------
> 1 file changed, 28 insertions(+), 9 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index 6dfb9c9..f892023 100755
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1591,6 +1591,17 @@ int Grow_reshape(char *devname, int fd,
> return 1;
> }
>
> + if (s->level == 0 && array.state & (1<<MD_SB_BITMAP_PRESENT)) {
> + array.state &= ~(1<<MD_SB_BITMAP_PRESENT);
> + if (ioctl(fd, SET_ARRAY_INFO, &array)!= 0) {
> + if (array.state & (1<<MD_SB_CLUSTERED))
> + pr_err("failed to remove clustered bitmap.\n");
> + else
> + pr_err("failed to remove internal bitmap.\n");
> + return 1;
> + }
> + }
Removing an internal bitmap for conversion to RAID0 is sensible.
Removing the clustered bitmap isn't so sensible (I'm glad you noted that
difference). Disabling cluster support is a much bigger change than
just removing an internal bitmap which is really just an optimisation.
So please only auto-remove the bitmap when it is a regular internal
bitmap.
Thanks,
NeilBrown
> +
> /* in the external case we need to check that the requested reshape is
> * supported, and perform an initial check that the container holds the
> * pre-requisite spare devices (mdmon owns final validation)
> @@ -1603,7 +1614,6 @@ int Grow_reshape(char *devname, int fd,
> cfd = open_dev_excl(st->container_devnm);
> } else {
> container = st->devnm;
> - close(fd);
> cfd = open_dev_excl(st->devnm);
> fd = cfd;
> }
> @@ -1619,8 +1629,8 @@ int Grow_reshape(char *devname, int fd,
> if (rv) {
> pr_err("Cannot read superblock for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto fallback;
> }
>
> /* check if operation is supported for metadata handler */
> @@ -1644,8 +1654,8 @@ int Grow_reshape(char *devname, int fd,
> pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
> devname, container);
> sysfs_free(cc);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto fallback;
> }
> }
> sysfs_free(cc);
> @@ -1665,7 +1675,8 @@ int Grow_reshape(char *devname, int fd,
> s->raiddisks - array.raid_disks,
> s->raiddisks - array.raid_disks == 1 ? "" : "s",
> array.spare_disks + added_disks);
> - return 1;
> + rv = 1;
> + goto fallback;
> }
>
> sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
> @@ -1678,17 +1689,20 @@ int Grow_reshape(char *devname, int fd,
> } else {
> pr_err("failed to read sysfs parameters for %s\n",
> devname);
> - return 1;
> + rv = 1;
> + goto fallback;
> }
> frozen = freeze(st);
> if (frozen < -1) {
> /* freeze() already spewed the reason */
> sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto fallback;
> } else if (frozen < 0) {
> pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
> sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto fallback;
> }
>
> /* ========= set size =============== */
> @@ -2084,6 +2098,11 @@ release:
> sysfs_free(sra);
> if (frozen > 0)
> unfreeze(st);
> +fallback:
> + if (cfd > -1)
> + close(cfd);
> + if (subarray)
> + free(subarray);
> return rv;
> }
>
> --
> 2.4.3
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 3/3] Add new journal to array that does not have journal
From: NeilBrown @ 2015-12-21 20:56 UTC (permalink / raw)
To: linux-raid; +Cc: dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450725823-1832511-4-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 4971 bytes --]
On Tue, Dec 22 2015, Song Liu wrote:
I've applied the first two patches - thanks.
This one bothers me.
> This patch enables adding journal to an array that does not have journal before.
>
> To add the journal, the array should finish resync. Otherwise, mdadm complains:
>
> [root] mdadm --add-journal /dev/md0 /dev/sdb8 -vvv
> mdadm: /dev/md0 has active resync, please retry after resync is done.
I'm reasonably happy with not adding a journal while a resync is
happening (be nice if you could though).
However this should be an option in the --grow subcommand, for consistency.
>
> The array need to be restarted for the journal to work:
>
> [root] mdadm --add-journal /dev/md0 /dev/sdb8
> mdadm: Making /dev/md0 readonly before adding journal...
> mdadm: Added new journal to /dev/md0.
> mdadm: Please restart the array to make it live.
I'm not at all happy with adding a journal and it not really being
added.
The best case is to add the journal and have it be live straight away.
Why can't we do that.
The second best option is to add the journal as part of --assemble,
e.g. with --update=journal
>
> [root] mdadm --stop /dev/md*
> mdadm: stopped /dev/md0
>
> [root] mdadm -A /dev/md0 /dev/sdb[12358]
> mdadm: device 8 in /dev/md0 has wrong state in superblock, but /dev/sdb8 seems ok
> mdadm: /dev/md0 has been started with 4 drives and 1 journal.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> Assemble.c | 5 +----
> Manage.c | 32 ++++++++++++++++++++++++++------
> 2 files changed, 27 insertions(+), 10 deletions(-)
>
> diff --git a/Assemble.c b/Assemble.c
> index a7cd163..4ddd650 100644
> --- a/Assemble.c
> +++ b/Assemble.c
> @@ -1556,10 +1556,7 @@ try_again:
> */
> if (content->array.level != LEVEL_MULTIPATH) {
> if (devices[j].i.disk.state & (1<<MD_DISK_JOURNAL)) {
> - if (content->journal_device_required)
> - journalcnt++;
> - else /* unexpected journal, mark as faulty */
> - devices[j].i.disk.state |= (1<<MD_DISK_FAULTY);
> + journalcnt++;
> } else if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
> if (!(devices[j].i.disk.state
> & (1<<MD_DISK_FAULTY))) {
> diff --git a/Manage.c b/Manage.c
> index 7e1b94b..da14b99 100644
> --- a/Manage.c
> +++ b/Manage.c
> @@ -740,6 +740,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> struct supertype *dev_st = NULL;
> int j;
> mdu_disk_info_t disc;
> + int new_journal = 0;
>
> if (!get_dev_size(tfd, dv->devname, &ldsize)) {
> if (dv->disposition == 'M')
> @@ -935,19 +936,33 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> if (dv->disposition == 'j') {
> struct mdinfo mdi;
> struct mdinfo *mdp;
> + struct mdstat_ent *mds, *m;
> + int percent = -1;
> +
> + mds = mdstat_read(0, 0);
> + for (m = mds; m; m = m->next)
> + if (strcmp(m->devnm, fd2devnm(fd)) == 0)
> + percent = m->percent;
> + free_mdstat(mds);
> +
> + if (percent > 0) {
> + pr_err("%s has active resync, please retry after resync is done.\n", devname);
> + return -1;
> + }
This is a rather clumsy way to test if a resync is happening. It is all
we could manage years ago, but today we can just read the sync_action
file from sysfs. If that isn't "idle", then assume some
resync/recovery/reshape is happening.
>
> mdp = sysfs_read(fd, NULL, GET_ARRAY_STATE);
>
> if (strncmp(mdp->sysfs_array_state, "readonly", 8) != 0) {
> - pr_err("%s is not readonly, cannot add journal.\n", devname);
> - return -1;
> + pr_err("Making %s readonly before adding journal...\n", devname);
> + if (Manage_ro(devname, fd, 1)) {
> + pr_err("Please retry.\n");
> + return -1;
> + }
?? Why does the array have to be read-only to add a journal?
That would prevent you adding a journal to an array with a mounted
filesystem.
Thanks,
NeilBrown
> }
>
> tst->ss->getinfo_super(tst, &mdi, NULL);
> - if (mdi.journal_device_required == 0) {
> - pr_err("%s does not support journal device.\n", devname);
> - return -1;
> - }
> + if (mdi.journal_device_required == 0)
> + new_journal = 1;
> disc.raid_disk = 0;
> }
>
> @@ -1064,6 +1079,11 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> close(container_fd);
> } else {
> tst->ss->free_super(tst);
> + if (new_journal) {
> + pr_err("Added new journal to %s. \n", devname);
> + pr_err("Please restart the array to make it live.\n");
> + return 1;
> + }
> if (ioctl(fd, ADD_NEW_DISK, &disc)) {
> if (dv->disposition == 'j')
> pr_err("Failed to hot add %s as journal, "
> --
> 2.4.6
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: clustered MD - beyond RAID1
From: NeilBrown @ 2015-12-21 20:47 UTC (permalink / raw)
To: Tejas Rao, Scott Sinno, linux-raid
Cc: Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
In-Reply-To: <567850C4.30108@bnl.gov>
[-- Attachment #1: Type: text/plain, Size: 646 bytes --]
On Tue, Dec 22 2015, Tejas Rao wrote:
> What if the application is doing the locking and making sure that only 1
> node writes to a md device at a time? Will this work? How are rebuilds
> handled? This would be helpful with distributed filesystems like
> GPFS/lustre etc.
>
You would also need to make sure that the filesystem only wrote from a
single node at a time (or access the block device directly). I doubt
GPFS/lustre make any promise like that, but I'm happy to be educated.
rebuilds are handled by using a cluster-wide lock to block all writes to
a range of addresses while those stripes are repaired.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox