* Re: Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: Sean Harris @ 2015-03-19 0:58 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150318121054.066b963b@notabene.brown>
Thanks Neil,
I'd like to try this but I'm having trouble installing it on Debian 5 (lenny).
When I try to make install, it complains:
***** Parent of /run/mdadm does not exist. Maybe set different RUN_DIR=
***** e.g. make RUN_DIR=/dev/.mdadm
***** or set CHECK_RUN_DIR=0
make: *** [check_rundir] Error 1
cat /etc/debian_version
5.0.10
The default run directory apparently is not there ie: /run/mdadm
That directory exists on Debian 7 but not Debian 5.
I admittedly do not know what to set that variable to on Lenny. Can you point me to what to look for?
Thanks,
Sean
> On Mar 17, 2015, at 6:10 PM, NeilBrown <neilb@suse.de> wrote:
>
> mdadm --assemble /dev/md10 --update=metadata ...list.of.devices..
^ permalink raw reply
* Re: Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: NeilBrown @ 2015-03-19 1:03 UTC (permalink / raw)
To: Sean Harris; +Cc: linux-raid
In-Reply-To: <DDB9AAD7-E69F-4D82-9A4D-2200E1D4C81D@backblaze.com>
[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]
On Wed, 18 Mar 2015 17:58:05 -0700 Sean Harris <sean@backblaze.com> wrote:
> Thanks Neil,
> I'd like to try this but I'm having trouble installing it on Debian 5 (lenny).
>
> When I try to make install, it complains:
>
> ***** Parent of /run/mdadm does not exist. Maybe set different RUN_DIR=
> ***** e.g. make RUN_DIR=/dev/.mdadm
> ***** or set CHECK_RUN_DIR=0
> make: *** [check_rundir] Error 1
>
> cat /etc/debian_version
> 5.0.10
>
> The default run directory apparently is not there ie: /run/mdadm
> That directory exists on Debian 7 but not Debian 5.
>
> I admittedly do not know what to set that variable to on Lenny. Can you point me to what to look for?
Does /dev/.mdadm exist? If it does, try
make RUN_DIR=/dev/.mdadm
If not, just use
make CHECK_RUN_DIR=0
You don't need access to the rundir to assemble some devices.
And don't "make install". Just "make", and then run
./mdadm ....
or copy mdadm to some where (e.g. /sbin/mdadm-new) and only use mdadm-new
when you particularly want the new features.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: Guoqing Jiang @ 2015-03-19 3:50 UTC (permalink / raw)
To: NeilBrown; +Cc: jgq516, rgoldwyn, linux-raid
In-Reply-To: <20150318095712.1954b958@notabene.brown>
Hi Neil,
NeilBrown wrote:
> On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
>
>
>> From: Guoqing Jiang <gqjiang@suse.com>
>>
>> The bm_blocks is modified by commit fe60ce (md/bitmap: use
>> sector_div for sector_t divisions), but it makes bm_blocks
>> has different value which is changed from like "a/b" to "a%b",
>> need to correct this to make sure cluster-md still works.
>>
>
> One of us is confused here.
>
> This code is trying to find the start of the bitmap relevant to this host in
> a table of multiple bitmaps. So it first needs to find out the size of each
> bitmap. It then multiples the size by the index number of this host to get
> an offset.
>
>
Thanks for detailed description, it really helps. I quoted related lines
from bitmap.c.
574 sector_t bm_blocks;
575 sector_t resync_sectors =
bitmap->mddev->resync_max_sectors;
576
577 bm_blocks = sector_div(resync_sectors,
578
bitmap->mddev->bitmap_info.chunksize >> 9);
579 bm_blocks = bm_blocks << 3;
580 bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
581 bitmap->mddev->bitmap_info.offset +=
bitmap->cluster_slot * (bm_blocks << 3);
> So it take the total number of sectors (resync_max_sectors), divides by the
> chunksize (in sectors) to get a number of chunks. This is the number of bits.
>
>
L577 is supposed to do above job.
> Then it should div-round-up by 8 to get a number of bytes.
>
I guess what you mean is about L579, while it used "<<3" rather than
">>3" now.
> Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
> are always 4K aligned.
>
L580 did the job.
> Then this number is multiplied by 8 (or shifted by 3) to get a number of
> sectors to add to the start of the table.
>
L581 is for this, right? Is the shifted by 3 is to match the bitmap
format for each
nodes? Seems the relationship between slot and the bitmap region of the node
is like n <-----> [8*nK, 8*(n+1)K]. How about the following changes?
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 501f83f..b2a241b 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -571,12 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
re_read:
/* If cluster_slot is set, the cluster is setup */
if (bitmap->cluster_slot >= 0) {
- sector_t bm_blocks;
- sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
+ sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
- bm_blocks = sector_div(resync_sectors,
-
bitmap->mddev->bitmap_info.chunksize >> 9);
- bm_blocks = bm_blocks << 3;
+ sector_div(bm_blocks,
bitmap->mddev->bitmap_info.chunksize >> 9);
+ bm_blocks = bm_blocks >> 3;
bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
> is wrong because it uses sector_div in a way which destroys
> resync_max_sectors.
> And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
> convert from bits to bytes.
>
> commit f9209a323547f054c7439a3bf67c45e64a054bd
> removes the abuse of sector_div, which is good, but uses a simple "a/b"
> division, which isn't allowed in the kernel.
>
> commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
> then does the right thing with sector_div, but the "<< 3" is still the wrong
> way around.
>
> If you still think your code is correct, please explain in detail why.
>
> Goldwyn: if you agree that "<< 3" should be ">> 3" or even
> DIV_ROUND_UP_SECTOR_T( , 8);
> please send a patch. If you don't think so, please explain why.
>
>
But anyway, it is better wait for Goldwyn's back from vacation, :)
Thanks,
Guoqing
^ permalink raw reply related
* [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: Eric Mei @ 2015-03-19 5:39 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
From: Eric Mei <eric.mei@seagate.com>
When array is degraded, read data landed on failed drives will result in
reading rest of data in a stripe. So a single sequential read would
result in same data being read twice.
This patch is to avoid chunk aligned read for degraded array. The
downside is to involve stripe cache which means associated CPU overhead
and extra memory copy.
Signed-off-by: Eric Mei <eric.mei@seagate.com>
---
drivers/md/raid5.c | 15 ++++++++++++---
1 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..763c64a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
unsigned int chunk_sectors = mddev->chunk_sectors;
unsigned int bio_sectors = bvm->bi_size >> 9;
- if ((bvm->bi_rw & 1) == WRITE)
- return biovec->bv_len; /* always allow writes to be
mergeable */
+ /*
+ * always allow writes to be mergeable, read as well if array
+ * is degraded as we'll go through stripe cache anyway.
+ */
+ if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
+ return biovec->bv_len;
if (mddev->new_chunk_sectors < mddev->chunk_sectors)
chunk_sectors = mddev->new_chunk_sectors;
@@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
struct bio * bi)
md_write_start(mddev, bi);
- if (rw == READ &&
+ /*
+ * If array is degraded, better not do chunk aligned read because
+ * later we might have to read it again in order to reconstruct
+ * data on failed drives.
+ */
+ if (rw == READ && mddev->degraded == 0 &&
mddev->reshape_position == MaxSector &&
chunk_aligned_read(mddev,bi))
return;
--
1.7.1
^ permalink raw reply related
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: NeilBrown @ 2015-03-19 6:02 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <550A60FF.3050902@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2473 bytes --]
On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
> From: Eric Mei <eric.mei@seagate.com>
>
> When array is degraded, read data landed on failed drives will result in
> reading rest of data in a stripe. So a single sequential read would
> result in same data being read twice.
>
> This patch is to avoid chunk aligned read for degraded array. The
> downside is to involve stripe cache which means associated CPU overhead
> and extra memory copy.
>
> Signed-off-by: Eric Mei <eric.mei@seagate.com>
> ---
> drivers/md/raid5.c | 15 ++++++++++++---
> 1 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index cd2f96b..763c64a 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
> unsigned int chunk_sectors = mddev->chunk_sectors;
> unsigned int bio_sectors = bvm->bi_size >> 9;
>
> - if ((bvm->bi_rw & 1) == WRITE)
> - return biovec->bv_len; /* always allow writes to be
> mergeable */
> + /*
> + * always allow writes to be mergeable, read as well if array
> + * is degraded as we'll go through stripe cache anyway.
> + */
> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
> + return biovec->bv_len;
>
> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
> chunk_sectors = mddev->new_chunk_sectors;
> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
> struct bio * bi)
>
> md_write_start(mddev, bi);
>
> - if (rw == READ &&
> + /*
> + * If array is degraded, better not do chunk aligned read because
> + * later we might have to read it again in order to reconstruct
> + * data on failed drives.
> + */
> + if (rw == READ && mddev->degraded == 0 &&
> mddev->reshape_position == MaxSector &&
> chunk_aligned_read(mddev,bi))
> return;
Thanks for the patch.
However this sort of patch really needs to come with some concrete
performance numbers. Preferably both sequential reads and random reads.
I agree that sequential reads are likely to be faster, but how much faster
are they?
I imagine that this might make random reads a little slower. Does it? By
how much?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-19 13:43 UTC (permalink / raw)
To: Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, riel, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel
In-Reply-To: <20150318132650.3336261c58829f49a9af8675@linux-foundation.org>
On Wed, Mar 18, 2015 at 01:26:50PM -0700, Andrew Morton wrote:
> On Mon, 16 Mar 2015 16:25:25 -0400 Dan Williams <dan.j.williams@intel.com> wrote:
>
> > Avoid the impending disaster of requiring struct page coverage for what
> > is expected to be ever increasing capacities of persistent memory. In
> > conversations with Rik van Riel, Mel Gorman, and Jens Axboe at the
> > recently concluded Linux Storage Summit it became clear that struct page
> > is not required in many places, it was simply convenient to re-use.
> >
> > Introduce helpers and infrastructure to remove struct page usage where
> > it is not necessary. One use case for these changes is to implement a
> > write-back-cache in persistent memory for software-RAID. Another use
> > case for the scatterlist changes is RDMA to a pfn-range.
>
> Those use-cases sound very thin. If that's all we have then I'd say
> "find another way of implementing those things without creating
> pageframes for persistent memory".
>
> IOW, please tell us much much much more about the value of this change.
Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
want to be able to do any kind of I/O directly to persistent memory,
and I think we do, we need to do one of:
1. Construct struct pages for persistent memory
1a. Permanently
1b. While the pages are under I/O
2. Teach the I/O layers to deal in PFNs instead of struct pages
3. Replace struct page with some other structure that can represent both
DRAM and PMEM
I'm personally a fan of #3, and I was looking at the scatterlist as
my preferred data structure. I now believe the scatterlist as it is
currently defined isn't sufficient, so we probably end up needing a new
data structure. I think Dan's preferred method of replacing struct
pages with PFNs is actually less instrusive, but doesn't give us as
much advantage (an entirely new data structure would let us move to an
extent based system at the same time, instead of sticking with an array
of pages). Clearly Boaz prefers 1a, which works well enough for the
8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
What's your preference? I guess option 0 is "force all I/O to go
through the page cache and then get copied", but that feels like a nasty
performance hit.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-19 15:54 UTC (permalink / raw)
To: Matthew Wilcox, Andrew Morton
Cc: linux-arch, axboe, riel, hch, linux-nvdimm, Dave Hansen,
linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
<>
>
> Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> want to be able to do any kind of I/O directly to persistent memory,
> and I think we do, we need to do one of:
>
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
> 2. Teach the I/O layers to deal in PFNs instead of struct pages
> 3. Replace struct page with some other structure that can represent both
> DRAM and PMEM
>
> I'm personally a fan of #3, and I was looking at the scatterlist as
> my preferred data structure. I now believe the scatterlist as it is
> currently defined isn't sufficient, so we probably end up needing a new
> data structure. I think Dan's preferred method of replacing struct
> pages with PFNs is actually less instrusive, but doesn't give us as
> much advantage (an entirely new data structure would let us move to an
> extent based system at the same time, instead of sticking with an array
> of pages). Clearly Boaz prefers 1a, which works well enough for the
> 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>
> What's your preference? I guess option 0 is "force all I/O to go
> through the page cache and then get copied", but that feels like a nasty
> performance hit.
Thanks Matthew, you have summarized it perfectly.
I think #1b might have merit, as well. I have a very surgical small
"hack" that we can do with allocating on demand pages before IO.
It involves adding a new MEMORY_MODEL policy that is derived from
SPARSEMEM but lets you allocate individual pages on demand. And a new
type of page say call it GP_emulated_page.
(Tell me if you find this interesting. It is 1/117 in size of both
#2 or #3)
In anyway please reconsider a configurable #1a for people that do
not mind sacrificing 1.2% of their pmem for real pages.
Even at 6G page-structs with 400G pmem, people would love some of the stuff
this gives them today. just few examples: direct_access from within a VM to
an host defined pmem, is trivial with no extra code with my two simple #1a
patches. RDMA memory brick targets, network shared memory FS and so on, the
list will always be bigger then any of #1b #2 or #3. Yes for people that
want to sacrifice the extra cost.
In the Kernel it was always about choice and diversity. And what does it
costs us. Nothing. Two small simple patches and a Kconfig option.
Note that I made it in such a way that if pmem is configured without
use of pages, then the mm code is *not* configured-in automatically.
We can even add a runtime option that even if #1a is enabled, for certain
pmem device may not want pages allocated. And so choose at runtime rather
than compile time.
I think this will only farther our cause and let people advance with
their research and development with great new ideas about use of pmem.
Then once there is a great demand for #1a and those large 512G devices
come out, we can go the #1b or #3 route and save them the extra 1.2%
memory, but once they have the appetite for it. (And Andrews question
becomes clear)
Our two ways need not be "either-or". They can be "have both". I think
choice is a good thing for us here. Even with #3 available #1a still has
merit in some configurations and they can co exist perfectly.
Please think about it?
Thanks
Boaz
^ permalink raw reply
* SMRFFS - EXT4, and a few stack changes
From: Adrian Palmer @ 2015-03-19 15:59 UTC (permalink / raw)
To: Linux Filesystem Development List, ext4 development, fstests,
linux-raid, linux-scsi, linux-ide
Hi all;
As some have been asking, I've posted some proof-of-concept code at
(https://github.com/Seagate/SMR_FS-EXT4).
Let me know any feedback. We're in the process of planning out the
schedule moving forward.
I'm spamming everyone because of the extent of the changes we've touched.
I'm also copying the RAID list because I want to start a discussion
about planning RAID solutions on SMR.
Adrian Palmer
Firmware Engineer II
R&D Firmware
Seagate, Longmont Colorado
720-684-1307
adrian.palmer@seagate.com
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Christoph Hellwig @ 2015-03-19 18:17 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On Thu, Mar 19, 2015 at 09:43:13AM -0400, Matthew Wilcox wrote:
> Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> want to be able to do any kind of I/O directly to persistent memory,
> and I think we do, we need to do one of:
>
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
> 2. Teach the I/O layers to deal in PFNs instead of struct pages
> 3. Replace struct page with some other structure that can represent both
> DRAM and PMEM
>
> I'm personally a fan of #3, and I was looking at the scatterlist as
> my preferred data structure. I now believe the scatterlist as it is
> currently defined isn't sufficient, so we probably end up needing a new
> data structure. I think Dan's preferred method of replacing struct
> pages with PFNs is actually less instrusive, but doesn't give us as
> much advantage (an entirely new data structure would let us move to an
> extent based system at the same time, instead of sticking with an array
> of pages). Clearly Boaz prefers 1a, which works well enough for the
> 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>
> What's your preference? I guess option 0 is "force all I/O to go
> through the page cache and then get copied", but that feels like a nasty
> performance hit.
In addition to the options there's also a time line. At least for the
short term where we want to get something going 1a seems like the
absolutely be option. It works perfectly fine for the lots of small
capacity dram-like nvdimms, and it works funtionally fine for the
special huge ones, although the resource use for it is highly annoying.
If it turns out to be too annoying we can also offer a no I/O possible
option for them in the short run.
In the long run option 2) sounds like a good plan to me, but not as a
parallel I/O path, but as the main one. Doing so will in fact give us
options to experiment with 3). Given that we're moving towards an
increasinly huge page using world replacing the good old struct page
with something extent-like and/or temporary might be needed for dram
as well in the future.
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-19 19:31 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman,
linux-fsdevel
In-Reply-To: <20150319181725.GA17411@infradead.org>
On Thu, Mar 19, 2015 at 11:17:25AM -0700, Christoph Hellwig wrote:
> On Thu, Mar 19, 2015 at 09:43:13AM -0400, Matthew Wilcox wrote:
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
> > 3. Replace struct page with some other structure that can represent both
> > DRAM and PMEM
>
> In addition to the options there's also a time line. At least for the
> short term where we want to get something going 1a seems like the
> absolutely be option. It works perfectly fine for the lots of small
(assuming "best option")
> capacity dram-like nvdimms, and it works funtionally fine for the
> special huge ones, although the resource use for it is highly annoying.
> If it turns out to be too annoying we can also offer a no I/O possible
> option for them in the short run.
>
> In the long run option 2) sounds like a good plan to me, but not as a
> parallel I/O path, but as the main one. Doing so will in fact give us
> options to experiment with 3). Given that we're moving towards an
> increasinly huge page using world replacing the good old struct page
> with something extent-like and/or temporary might be needed for dram
> as well in the future.
Dan's patches don't actually make it a "parallel I/O path", that was
Boaz's mischaracterisation. They move all scatterlists and bios over
to using PFNs, at least on architectures which have been converted.
Speaking of architectures not being converted, it is really past time for
architectures to be switched to supporting SG chaining. It was introduced
in 2007, and not having it generically available causes problems for
the crypto layer, as well as making further enhancements more tricky.
Assuming 'select ARCH_HAS_SG_CHAIN' is sufficient to tell, the following
architectures do support it:
arm arm64 ia64 powerpc s390 sparc x86
which means the following architectures are 8 years delinquent in
adding support:
alpha arc avr32 blackfin c6x cris frv hexagon m32r m68k metag microblaze
mips mn10300 nios2 openrisc parisc score sh tile um unicore32 xtensa
Perhaps we could deliberately make asm-generic/scatterlist.h not build
for architectures that don't select it in order to make them convert ...
^ permalink raw reply
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: Eric Mei @ 2015-03-19 19:41 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150319170215.7d6dfd60@notabene.brown>
On 2015-03-19 12:02 AM, NeilBrown wrote:
> On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
>
>> From: Eric Mei <eric.mei@seagate.com>
>>
>> When array is degraded, read data landed on failed drives will result in
>> reading rest of data in a stripe. So a single sequential read would
>> result in same data being read twice.
>>
>> This patch is to avoid chunk aligned read for degraded array. The
>> downside is to involve stripe cache which means associated CPU overhead
>> and extra memory copy.
>>
>> Signed-off-by: Eric Mei <eric.mei@seagate.com>
>> ---
>> drivers/md/raid5.c | 15 ++++++++++++---
>> 1 files changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index cd2f96b..763c64a 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
>> unsigned int chunk_sectors = mddev->chunk_sectors;
>> unsigned int bio_sectors = bvm->bi_size >> 9;
>>
>> - if ((bvm->bi_rw & 1) == WRITE)
>> - return biovec->bv_len; /* always allow writes to be
>> mergeable */
>> + /*
>> + * always allow writes to be mergeable, read as well if array
>> + * is degraded as we'll go through stripe cache anyway.
>> + */
>> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
>> + return biovec->bv_len;
>>
>> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
>> chunk_sectors = mddev->new_chunk_sectors;
>> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
>> struct bio * bi)
>>
>> md_write_start(mddev, bi);
>>
>> - if (rw == READ &&
>> + /*
>> + * If array is degraded, better not do chunk aligned read because
>> + * later we might have to read it again in order to reconstruct
>> + * data on failed drives.
>> + */
>> + if (rw == READ && mddev->degraded == 0 &&
>> mddev->reshape_position == MaxSector &&
>> chunk_aligned_read(mddev,bi))
>> return;
>
> Thanks for the patch.
>
> However this sort of patch really needs to come with some concrete
> performance numbers. Preferably both sequential reads and random reads.
>
> I agree that sequential reads are likely to be faster, but how much faster
> are they?
> I imagine that this might make random reads a little slower. Does it? By
> how much?
>
> Thanks,
> NeilBrown
>
Hi Neil,
Sorry I should have done the test in first place.
Following test are done on a enterprise storage node with Seagate 6T SAS
drives and Xeon E5-2648L CPU (10 cores, 1.9Ghz), 10 disks MD RAID6 8+2,
chunk size 128 KiB.
I use FIO, using direct-io with various bs size, enough queue depth,
tested sequential and 100% random read against 3 array config: 1)
optimal, as baseline; 2) degraded; 3) degraded with this patch. Kernel
version is 4.0-rc3.
Each individual test I only did once so there might be some variations,
but we just focus on big trend.
Sequential Read:
bs=(KiB) optimal(MiB/s) degraded(MiB/s) degraded-with-patch (MiB/s)
1024 1608 656 995
512 1624 710 956
256 1635 728 980
128 1636 771 983
64 1612 1119 1000
32 1580 1420 1004
16 1368 688 986
8 768 647 953
4 411 413 850
Random Read:
bs=(KiB) optimal(IOPS) degraded(IOPS) degraded-with-patch (IOPS)
1024 163 160 156
512 274 273 272
256 426 428 424
128 576 592 591
64 726 724 726
32 849 848 837
16 900 970 971
8 927 940 929
4 948 940 955
Some notes:
* In sequential + optimal, as bs size getting smaller, the FIO thread
become CPU bound.
* In sequential + degraded, there's big increase when bs is 64K and
32K, I don't have explanation.
* In sequential + degraded-with-patch, the MD thread mostly become CPU
bound.
If you want to we can discuss specific data point in those data. But in
general it seems with this patch, we have more predictable and in most
cases significant better sequential read performance when array is
degraded, and almost no noticeable impact on random read.
Performance is a complicated thing, the patch works well for this
particular configuration, but may not be universal. For example I
imagine testing on all SSD array may have very different result. But I
personally think in most cases IO bandwidth is more scarce resource than
CPU.
Eric
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Andrew Morton @ 2015-03-19 19:59 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <550AF127.9010504@plexistor.com>
On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
> <>
> >
> > Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> > want to be able to do any kind of I/O directly to persistent memory,
> > and I think we do, we need to do one of:
> >
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
> > 3. Replace struct page with some other structure that can represent both
> > DRAM and PMEM
> >
> > I'm personally a fan of #3, and I was looking at the scatterlist as
> > my preferred data structure. I now believe the scatterlist as it is
> > currently defined isn't sufficient, so we probably end up needing a new
> > data structure. I think Dan's preferred method of replacing struct
> > pages with PFNs is actually less instrusive, but doesn't give us as
> > much advantage (an entirely new data structure would let us move to an
> > extent based system at the same time, instead of sticking with an array
> > of pages). Clearly Boaz prefers 1a, which works well enough for the
> > 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
> >
> > What's your preference? I guess option 0 is "force all I/O to go
> > through the page cache and then get copied", but that feels like a nasty
> > performance hit.
>
> Thanks Matthew, you have summarized it perfectly.
>
> I think #1b might have merit, as well.
It would be interesting to see what a 1b implementation looks like and
how it performs. We already allocate a bunch of temporary things to
support in-flight IO (bio, request) and allocating pageframes on the
same basis seems a fairly logical fit.
It is all a bit of a stopgap, designed to shoehorn
direct-io-to-dax-mapped-memory into the existing world. Longer term
I'd expect us to move to something more powerful, but it's unclear what
that will be at this time, so a stopgap isn't too bad?
This is all contingent upon the prevalence of machines which have vast
amounts of nv memory and relatively small amounts of regular memory.
How confident are we that this really is the future?
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Dan Williams @ 2015-03-19 20:59 UTC (permalink / raw)
To: Andrew Morton
Cc: Boaz Harrosh, linux-arch, Jens Axboe, riel, linux-raid,
linux-nvdimm, Dave Hansen, linux-kernel@vger.kernel.org,
Christoph Hellwig, Mel Gorman, linux-fsdevel
In-Reply-To: <20150319125917.6cc2bf02687aab542027d8ac@linux-foundation.org>
On Thu, Mar 19, 2015 at 12:59 PM, Andrew Morton
<akpm@linux-foundation.org> wrote:
> On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
>
>> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
>> <>
>> >
>> > Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
>> > want to be able to do any kind of I/O directly to persistent memory,
>> > and I think we do, we need to do one of:
>> >
>> > 1. Construct struct pages for persistent memory
>> > 1a. Permanently
>> > 1b. While the pages are under I/O
>> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
>> > 3. Replace struct page with some other structure that can represent both
>> > DRAM and PMEM
>> >
>> > I'm personally a fan of #3, and I was looking at the scatterlist as
>> > my preferred data structure. I now believe the scatterlist as it is
>> > currently defined isn't sufficient, so we probably end up needing a new
>> > data structure. I think Dan's preferred method of replacing struct
>> > pages with PFNs is actually less instrusive, but doesn't give us as
>> > much advantage (an entirely new data structure would let us move to an
>> > extent based system at the same time, instead of sticking with an array
>> > of pages). Clearly Boaz prefers 1a, which works well enough for the
>> > 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>> >
>> > What's your preference? I guess option 0 is "force all I/O to go
>> > through the page cache and then get copied", but that feels like a nasty
>> > performance hit.
>>
>> Thanks Matthew, you have summarized it perfectly.
>>
>> I think #1b might have merit, as well.
>
> It would be interesting to see what a 1b implementation looks like and
> how it performs. We already allocate a bunch of temporary things to
> support in-flight IO (bio, request) and allocating pageframes on the
> same basis seems a fairly logical fit.
At least for block-i/o it seems the only place we really need struct
page infrastructure is for kmap(). Given we already need a kmap_pfn()
solution for option 2 a "dynamic allocation" stop along that
development path may just naturally fall out.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 15:56 UTC (permalink / raw)
To: Boaz Harrosh, Matthew Wilcox, Boaz Harrosh
Cc: axboe, linux-arch, linux-raid, linux-nvdimm, Dave Hansen,
linux-kernel, hch, Linus Torvalds, Al Viro, linux-fsdevel,
Andrew Morton, mgorman
In-Reply-To: <55098DFE.8080502@plexistor.com>
On 03/18/2015 10:38 AM, Boaz Harrosh wrote:
> On 03/18/2015 03:06 PM, Matthew Wilcox wrote:
>>> I'm not the one afraid of hard work, if it was for a good cause, but for what?
>>> really for what? The block layer, and RDMA, and networking, and spline, and what
>>> ever the heck any one wants to imagine to do with pmem, already works perfectly
>>> stable. right now!
>>
>> The overhead. Allocating a struct page for every 4k page in a 400GB DIMM
>> (the current capacity available from one NV-DIMM vendor) occupies 6.4GB.
>> That's an unacceptable amount of overhead.
>>
>
> So lets fix the stacks to work nice with 2M pages. That said we can
> allocate the struct page also from pmem if we need to. The fact remains
> that we need state down the different stacks and this is the current
> design over all.
Fixing the stack to work with 2M pages will be just as invasive,
and just as much work as making it work without a struct page.
What state do you need, exactly?
The struct page in the VM is mostly used for two things:
1) to get a memory address of the data
2) refcounting, to make sure the page does not go away
during an IO operation, copy, etc...
Persistent memory cannot be paged out so (2) is not a concern, as
long as we ensure the object the page belongs to does not go away.
There are no seek times, so moving it around may not be necessary
either, making (1) not a concern.
The only case where (1) would be a concern is if we wanted to move
data in persistent memory around for better NUMA locality. However,
persistent memory DIMMs are on their way to being too large to move
the memory, anyway - all we can usefully do is detect where programs
are accessing memory, and move the programs there.
What state do you need that is not already represented?
1.5% overhead isn't a whole lot, but it appears to be unnecessary.
If you have a convincing argument as to why we need a struct page,
you might want to articulate it in order to convince us.
--
All rights reversed
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 16:21 UTC (permalink / raw)
To: Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
Michael Tsirkin and I have been doing some thinking about what
it would take to allocate struct pages per 2MB area permanently,
and allocate additional struct pages for 4kB pages on demand,
when a 2MB area is broken up into 4kB pages.
This should work for both DRAM and persistent memory.
I am still not convinced it is worthwhile to have struct pages
for persistent memory though, but I am willing to change my mind.
--
All rights reversed
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Wols Lists @ 2015-03-20 17:32 UTC (permalink / raw)
To: Andrew Morton, Boaz Harrosh
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <20150319125917.6cc2bf02687aab542027d8ac@linux-foundation.org>
On 19/03/15 19:59, Andrew Morton wrote:
> This is all contingent upon the prevalence of machines which have vast
> amounts of nv memory and relatively small amounts of regular memory.
> How confident are we that this really is the future?
Somewhat off-topic, but it's also the past. I can't help thinking of the
early Pick machines, which treated backing store as one giant permanent
virtual memory. Back when 300Mb hard drives were HUGE.
Cheers,
Wol
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-20 20:31 UTC (permalink / raw)
To: Rik van Riel
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <550C490E.1080708@redhat.com>
On Fri, Mar 20, 2015 at 12:21:34PM -0400, Rik van Riel wrote:
> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
>
> Michael Tsirkin and I have been doing some thinking about what
> it would take to allocate struct pages per 2MB area permanently,
> and allocate additional struct pages for 4kB pages on demand,
> when a 2MB area is broken up into 4kB pages.
Ah! I've looked at that a couple of times as well. I asked our database
performance team what impact freeing up the memmap would have on their
performance. They told me that doubling the amount of memory generally
resulted in approximately a 40% performance improvement. So freeing up
1.5% additional memory would result in about 0.6% performance improvement,
which I thought was probably too small a return on investment to justify
turning memmap into a two-level data structure.
Persistent memory might change that calculation somewhat ... but I'm
not convinced. Certainly, if we already had the ability to allocate
'struct superpage', I wouldn't be pushing for page-less I/Os, I'd just
allocate these data structures for PM. Even if they were 128 bytes in
size, that's only a 25MB overhead per 400GB NV-DIMM, which feels quite
reasonable to me.
> This should work for both DRAM and persistent memory.
>
> I am still not convinced it is worthwhile to have struct pages
> for persistent memory though, but I am willing to change my mind.
There's a lot of code out there that relies on struct page being PAGE_SIZE
bytes. I'm cool with replacing 'struct page' with 'struct superpage'
[1] in the biovec and auditing all of the code which touches it ... but
that's going to be a lot of code! I'm not sure it's less code than
going directly to 'just do I/O on PFNs'.
[1] Please, somebody come up with a better name!
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 21:08 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <20150320203136.GM4003@linux.intel.com>
On 03/20/2015 04:31 PM, Matthew Wilcox wrote:
> On Fri, Mar 20, 2015 at 12:21:34PM -0400, Rik van Riel wrote:
>> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>>
>>> 1. Construct struct pages for persistent memory
>>> 1a. Permanently
>>> 1b. While the pages are under I/O
>>
>> Michael Tsirkin and I have been doing some thinking about what
>> it would take to allocate struct pages per 2MB area permanently,
>> and allocate additional struct pages for 4kB pages on demand,
>> when a 2MB area is broken up into 4kB pages.
>
> Ah! I've looked at that a couple of times as well. I asked our database
> performance team what impact freeing up the memmap would have on their
> performance. They told me that doubling the amount of memory generally
> resulted in approximately a 40% performance improvement. So freeing up
> 1.5% additional memory would result in about 0.6% performance improvement,
> which I thought was probably too small a return on investment to justify
> turning memmap into a two-level data structure.
Agreed, it should not be done for memory savings alone, but only
if it helps improve all kinds of other things.
>> This should work for both DRAM and persistent memory.
>>
>> I am still not convinced it is worthwhile to have struct pages
>> for persistent memory though, but I am willing to change my mind.
>
> There's a lot of code out there that relies on struct page being PAGE_SIZE
> bytes. I'm cool with replacing 'struct page' with 'struct superpage'
> [1] in the biovec and auditing all of the code which touches it ... but
> that's going to be a lot of code! I'm not sure it's less code than
> going directly to 'just do I/O on PFNs'.
Totally agreed here. I see absolutely no advantage to teaching the
IO layer about a "struct superpage" when it could operate on PFNs
just as easily.
--
All rights reversed
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Wols Lists @ 2015-03-20 21:17 UTC (permalink / raw)
To: Matthew Wilcox, Rik van Riel
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <20150320203136.GM4003@linux.intel.com>
On 20/03/15 20:31, Matthew Wilcox wrote:
> Ah! I've looked at that a couple of times as well. I asked our database
> performance team what impact freeing up the memmap would have on their
> performance. They told me that doubling the amount of memory generally
> resulted in approximately a 40% performance improvement. So freeing up
> 1.5% additional memory would result in about 0.6% performance improvement,
> which I thought was probably too small a return on investment to justify
> turning memmap into a two-level data structure.
Don't get me started on databases! This is very much a relational
problem, other databases don't suffer like this.
(imho relational theory is totally inappropriate for an engineering
problem, like designing a database engine ...)
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: NeilBrown @ 2015-03-20 22:37 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: jgq516, rgoldwyn, linux-raid
In-Reply-To: <550A476D.4060707@suse.com>
[-- Attachment #1: Type: text/plain, Size: 4639 bytes --]
On Thu, 19 Mar 2015 11:50:05 +0800 Guoqing Jiang <GQJiang@suse.com> wrote:
> Hi Neil,
>
> NeilBrown wrote:
> > On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
> >
> >
> >> From: Guoqing Jiang <gqjiang@suse.com>
> >>
> >> The bm_blocks is modified by commit fe60ce (md/bitmap: use
> >> sector_div for sector_t divisions), but it makes bm_blocks
> >> has different value which is changed from like "a/b" to "a%b",
> >> need to correct this to make sure cluster-md still works.
> >>
> >
> > One of us is confused here.
> >
> > This code is trying to find the start of the bitmap relevant to this host in
> > a table of multiple bitmaps. So it first needs to find out the size of each
> > bitmap. It then multiples the size by the index number of this host to get
> > an offset.
> >
> >
> Thanks for detailed description, it really helps. I quoted related lines
> from bitmap.c.
>
> 574 sector_t bm_blocks;
> 575 sector_t resync_sectors =
> bitmap->mddev->resync_max_sectors;
> 576
> 577 bm_blocks = sector_div(resync_sectors,
> 578
> bitmap->mddev->bitmap_info.chunksize >> 9);
> 579 bm_blocks = bm_blocks << 3;
> 580 bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> 581 bitmap->mddev->bitmap_info.offset +=
> bitmap->cluster_slot * (bm_blocks << 3);
>
> > So it take the total number of sectors (resync_max_sectors), divides by the
> > chunksize (in sectors) to get a number of chunks. This is the number of bits.
> >
> >
> L577 is supposed to do above job.
> > Then it should div-round-up by 8 to get a number of bytes.
> >
> I guess what you mean is about L579, while it used "<<3" rather than
> ">>3" now.
> > Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
> > are always 4K aligned.
> >
> L580 did the job.
> > Then this number is multiplied by 8 (or shifted by 3) to get a number of
> > sectors to add to the start of the table.
> >
> L581 is for this, right? Is the shifted by 3 is to match the bitmap
> format for each
> nodes? Seems the relationship between slot and the bitmap region of the node
> is like n <-----> [8*nK, 8*(n+1)K]. How about the following changes?
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index 501f83f..b2a241b 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -571,12 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
> re_read:
> /* If cluster_slot is set, the cluster is setup */
> if (bitmap->cluster_slot >= 0) {
> - sector_t bm_blocks;
> - sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
> + sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
>
> - bm_blocks = sector_div(resync_sectors,
> -
> bitmap->mddev->bitmap_info.chunksize >> 9);
> - bm_blocks = bm_blocks << 3;
> + sector_div(bm_blocks,
> bitmap->mddev->bitmap_info.chunksize >> 9);
Yes, of course. sector_div returns the remainder doesn't it!
I was thinking that it returned the quotient and set the first arg to the
remainder - and wonder why you wanted the remainder :-(
I've updated the patch to do the right thing and credited you.
Thanks.
> + bm_blocks = bm_blocks >> 3;
> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> > So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
> > is wrong because it uses sector_div in a way which destroys
> > resync_max_sectors.
> > And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
> > convert from bits to bytes.
> >
> > commit f9209a323547f054c7439a3bf67c45e64a054bd
> > removes the abuse of sector_div, which is good, but uses a simple "a/b"
> > division, which isn't allowed in the kernel.
> >
> > commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
> > then does the right thing with sector_div, but the "<< 3" is still the wrong
> > way around.
> >
> > If you still think your code is correct, please explain in detail why.
> >
> > Goldwyn: if you agree that "<< 3" should be ">> 3" or even
> > DIV_ROUND_UP_SECTOR_T( , 8);
> > please send a patch. If you don't think so, please explain why.
> >
> >
> But anyway, it is better wait for Goldwyn's back from vacation, :)
I'll leave the other change (<<3 or >>8) until then.
thanks,
NeilBrown
>
> Thanks,
> Guoqing
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Why people recommend to disable NCQ for MD RAID ?
From: Alireza Haghdoost @ 2015-03-20 23:52 UTC (permalink / raw)
To: Linux RAID
I have read multiple blog post that people recommend to disable
NCQ/TCQ features of individual drives that is in the RAID group [1] in
a hope to improve RAID speed. I guess there should be some observation
in the field that results such a conclusion. However, this idea does
not make sense in theory. While drives can handle multiple outstanding
scsi commands at the same time, why don't we take advantage of this to
increase the disk access throughput ?
Thanks
Alireza
[1] http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html
^ permalink raw reply
* is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-21 11:01 UTC (permalink / raw)
To: linux-raid
I am a newbie of mdadm. I have a question but find no answer in
document or google for last 10 days.
The question is : RAID1 made by mdadm is full sync? for example, I
have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
commit an IO to the RAID1 disk (md127), it will return back to me when
all the two disk commit successfully or it will return back
to me once just one of the disk successfully commit.
I have test with xfs and ext4 with sync option, and it seems that two
disk have lots of commit difference after reboot the server. is that
means mdadm return success when one of the disk is commit
successfully?
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: NeilBrown @ 2015-03-22 3:20 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqhy0Bao1hAn3R-KtXWOjYb8O2LDo+E1jNdktWdduUL4vw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]
On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
wrote:
> I am a newbie of mdadm. I have a question but find no answer in
> document or google for last 10 days.
>
> The question is : RAID1 made by mdadm is full sync? for example, I
> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
> commit an IO to the RAID1 disk (md127), it will return back to me when
> all the two disk commit successfully or it will return back
> to me once just one of the disk successfully commit.
The write request will not return until it has been submitted to all, and
returned by, all working devices.
>
> I have test with xfs and ext4 with sync option, and it seems that two
> disk have lots of commit difference after reboot the server. is that
> means mdadm return success when one of the disk is commit
> successfully?
That certainly shouldn't happen. I would need more details of the experiment
that you performed.
NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-22 5:00 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150322142033.0af10b1a@notabene.brown>
Thanks for reply.
I have create a raid1 with two fusion io PCIe flash disk:
mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
/dev/fioa2 /dev/mapper/mpathc
/dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
load disk export from server B.
After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
and we will run mysql binlog on it.
In order to avoid data loss of mysql binlog we have set
sync_binlog=1. so every sql commit will call fsync() to flush to disk.
according to your description. if we reboot the server A, the two disk
data on different server will be the same.
but after the server A restarted, we assemble the two disk on two
server, data is different on the two server, disk on server B lost
more than one sql commit.
I have checked it with strace 'mysqld' on Server A.
I found a sql commit and fsync() on binlog file handle on server A but
this sql can not find in assembled disk on server B.
I also test it with two SAS disk, Server B still has more than one sql
commit lost.
2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
> On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
> wrote:
>
>> I am a newbie of mdadm. I have a question but find no answer in
>> document or google for last 10 days.
>>
>> The question is : RAID1 made by mdadm is full sync? for example, I
>> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
>> commit an IO to the RAID1 disk (md127), it will return back to me when
>> all the two disk commit successfully or it will return back
>> to me once just one of the disk successfully commit.
>
> The write request will not return until it has been submitted to all, and
> returned by, all working devices.
>
>>
>> I have test with xfs and ext4 with sync option, and it seems that two
>> disk have lots of commit difference after reboot the server. is that
>> means mdadm return success when one of the disk is commit
>> successfully?
>
> That certainly shouldn't happen. I would need more details of the experiment
> that you performed.
>
> NeilBrown
>
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: NeilBrown @ 2015-03-22 5:38 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqg2wp4VMCA8O0b2rUfbGDGXZY=mFvj1hA=tWaVkdCMXtA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]
On Sun, 22 Mar 2015 13:00:54 +0800 lingli tang <tanglingli001@gmail.com>
wrote:
> Thanks for reply.
>
> I have create a raid1 with two fusion io PCIe flash disk:
> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
> /dev/fioa2 /dev/mapper/mpathc
> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
> load disk export from server B.
>
> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
> and we will run mysql binlog on it.
> In order to avoid data loss of mysql binlog we have set
> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>
> according to your description. if we reboot the server A, the two disk
> data on different server will be the same.
> but after the server A restarted, we assemble the two disk on two
> server, data is different on the two server, disk on server B lost
> more than one sql commit.
What exactly do you mean by "reboot"??
Is this a clean shutdown or do you remove the power or something like that.
If you remove the power, then it is very possible that some requests will
have been submitted to one device but not the other.
If you have a clean shutdown, then the two devices should be identical.
NeilBrown
>
> I have checked it with strace 'mysqld' on Server A.
> I found a sql commit and fsync() on binlog file handle on server A but
> this sql can not find in assembled disk on server B.
>
> I also test it with two SAS disk, Server B still has more than one sql
> commit lost.
>
>
> 2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
> > On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
> > wrote:
> >
> >> I am a newbie of mdadm. I have a question but find no answer in
> >> document or google for last 10 days.
> >>
> >> The question is : RAID1 made by mdadm is full sync? for example, I
> >> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
> >> commit an IO to the RAID1 disk (md127), it will return back to me when
> >> all the two disk commit successfully or it will return back
> >> to me once just one of the disk successfully commit.
> >
> > The write request will not return until it has been submitted to all, and
> > returned by, all working devices.
> >
> >>
> >> I have test with xfs and ext4 with sync option, and it seems that two
> >> disk have lots of commit difference after reboot the server. is that
> >> means mdadm return success when one of the disk is commit
> >> successfully?
> >
> > That certainly shouldn't happen. I would need more details of the experiment
> > that you performed.
> >
> > NeilBrown
> >
> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox