* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Andrew Morton @ 2015-03-19 19:59 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <550AF127.9010504@plexistor.com>
On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
> <>
> >
> > Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> > want to be able to do any kind of I/O directly to persistent memory,
> > and I think we do, we need to do one of:
> >
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
> > 3. Replace struct page with some other structure that can represent both
> > DRAM and PMEM
> >
> > I'm personally a fan of #3, and I was looking at the scatterlist as
> > my preferred data structure. I now believe the scatterlist as it is
> > currently defined isn't sufficient, so we probably end up needing a new
> > data structure. I think Dan's preferred method of replacing struct
> > pages with PFNs is actually less instrusive, but doesn't give us as
> > much advantage (an entirely new data structure would let us move to an
> > extent based system at the same time, instead of sticking with an array
> > of pages). Clearly Boaz prefers 1a, which works well enough for the
> > 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
> >
> > What's your preference? I guess option 0 is "force all I/O to go
> > through the page cache and then get copied", but that feels like a nasty
> > performance hit.
>
> Thanks Matthew, you have summarized it perfectly.
>
> I think #1b might have merit, as well.
It would be interesting to see what a 1b implementation looks like and
how it performs. We already allocate a bunch of temporary things to
support in-flight IO (bio, request) and allocating pageframes on the
same basis seems a fairly logical fit.
It is all a bit of a stopgap, designed to shoehorn
direct-io-to-dax-mapped-memory into the existing world. Longer term
I'd expect us to move to something more powerful, but it's unclear what
that will be at this time, so a stopgap isn't too bad?
This is all contingent upon the prevalence of machines which have vast
amounts of nv memory and relatively small amounts of regular memory.
How confident are we that this really is the future?
^ permalink raw reply
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: Eric Mei @ 2015-03-19 19:41 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150319170215.7d6dfd60@notabene.brown>
On 2015-03-19 12:02 AM, NeilBrown wrote:
> On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
>
>> From: Eric Mei <eric.mei@seagate.com>
>>
>> When array is degraded, read data landed on failed drives will result in
>> reading rest of data in a stripe. So a single sequential read would
>> result in same data being read twice.
>>
>> This patch is to avoid chunk aligned read for degraded array. The
>> downside is to involve stripe cache which means associated CPU overhead
>> and extra memory copy.
>>
>> Signed-off-by: Eric Mei <eric.mei@seagate.com>
>> ---
>> drivers/md/raid5.c | 15 ++++++++++++---
>> 1 files changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index cd2f96b..763c64a 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
>> unsigned int chunk_sectors = mddev->chunk_sectors;
>> unsigned int bio_sectors = bvm->bi_size >> 9;
>>
>> - if ((bvm->bi_rw & 1) == WRITE)
>> - return biovec->bv_len; /* always allow writes to be
>> mergeable */
>> + /*
>> + * always allow writes to be mergeable, read as well if array
>> + * is degraded as we'll go through stripe cache anyway.
>> + */
>> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
>> + return biovec->bv_len;
>>
>> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
>> chunk_sectors = mddev->new_chunk_sectors;
>> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
>> struct bio * bi)
>>
>> md_write_start(mddev, bi);
>>
>> - if (rw == READ &&
>> + /*
>> + * If array is degraded, better not do chunk aligned read because
>> + * later we might have to read it again in order to reconstruct
>> + * data on failed drives.
>> + */
>> + if (rw == READ && mddev->degraded == 0 &&
>> mddev->reshape_position == MaxSector &&
>> chunk_aligned_read(mddev,bi))
>> return;
>
> Thanks for the patch.
>
> However this sort of patch really needs to come with some concrete
> performance numbers. Preferably both sequential reads and random reads.
>
> I agree that sequential reads are likely to be faster, but how much faster
> are they?
> I imagine that this might make random reads a little slower. Does it? By
> how much?
>
> Thanks,
> NeilBrown
>
Hi Neil,
Sorry I should have done the test in first place.
Following test are done on a enterprise storage node with Seagate 6T SAS
drives and Xeon E5-2648L CPU (10 cores, 1.9Ghz), 10 disks MD RAID6 8+2,
chunk size 128 KiB.
I use FIO, using direct-io with various bs size, enough queue depth,
tested sequential and 100% random read against 3 array config: 1)
optimal, as baseline; 2) degraded; 3) degraded with this patch. Kernel
version is 4.0-rc3.
Each individual test I only did once so there might be some variations,
but we just focus on big trend.
Sequential Read:
bs=(KiB) optimal(MiB/s) degraded(MiB/s) degraded-with-patch (MiB/s)
1024 1608 656 995
512 1624 710 956
256 1635 728 980
128 1636 771 983
64 1612 1119 1000
32 1580 1420 1004
16 1368 688 986
8 768 647 953
4 411 413 850
Random Read:
bs=(KiB) optimal(IOPS) degraded(IOPS) degraded-with-patch (IOPS)
1024 163 160 156
512 274 273 272
256 426 428 424
128 576 592 591
64 726 724 726
32 849 848 837
16 900 970 971
8 927 940 929
4 948 940 955
Some notes:
* In sequential + optimal, as bs size getting smaller, the FIO thread
become CPU bound.
* In sequential + degraded, there's big increase when bs is 64K and
32K, I don't have explanation.
* In sequential + degraded-with-patch, the MD thread mostly become CPU
bound.
If you want to we can discuss specific data point in those data. But in
general it seems with this patch, we have more predictable and in most
cases significant better sequential read performance when array is
degraded, and almost no noticeable impact on random read.
Performance is a complicated thing, the patch works well for this
particular configuration, but may not be universal. For example I
imagine testing on all SSD array may have very different result. But I
personally think in most cases IO bandwidth is more scarce resource than
CPU.
Eric
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-19 19:31 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman,
linux-fsdevel
In-Reply-To: <20150319181725.GA17411@infradead.org>
On Thu, Mar 19, 2015 at 11:17:25AM -0700, Christoph Hellwig wrote:
> On Thu, Mar 19, 2015 at 09:43:13AM -0400, Matthew Wilcox wrote:
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
> > 3. Replace struct page with some other structure that can represent both
> > DRAM and PMEM
>
> In addition to the options there's also a time line. At least for the
> short term where we want to get something going 1a seems like the
> absolutely be option. It works perfectly fine for the lots of small
(assuming "best option")
> capacity dram-like nvdimms, and it works funtionally fine for the
> special huge ones, although the resource use for it is highly annoying.
> If it turns out to be too annoying we can also offer a no I/O possible
> option for them in the short run.
>
> In the long run option 2) sounds like a good plan to me, but not as a
> parallel I/O path, but as the main one. Doing so will in fact give us
> options to experiment with 3). Given that we're moving towards an
> increasinly huge page using world replacing the good old struct page
> with something extent-like and/or temporary might be needed for dram
> as well in the future.
Dan's patches don't actually make it a "parallel I/O path", that was
Boaz's mischaracterisation. They move all scatterlists and bios over
to using PFNs, at least on architectures which have been converted.
Speaking of architectures not being converted, it is really past time for
architectures to be switched to supporting SG chaining. It was introduced
in 2007, and not having it generically available causes problems for
the crypto layer, as well as making further enhancements more tricky.
Assuming 'select ARCH_HAS_SG_CHAIN' is sufficient to tell, the following
architectures do support it:
arm arm64 ia64 powerpc s390 sparc x86
which means the following architectures are 8 years delinquent in
adding support:
alpha arc avr32 blackfin c6x cris frv hexagon m32r m68k metag microblaze
mips mn10300 nios2 openrisc parisc score sh tile um unicore32 xtensa
Perhaps we could deliberately make asm-generic/scatterlist.h not build
for architectures that don't select it in order to make them convert ...
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Christoph Hellwig @ 2015-03-19 18:17 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On Thu, Mar 19, 2015 at 09:43:13AM -0400, Matthew Wilcox wrote:
> Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> want to be able to do any kind of I/O directly to persistent memory,
> and I think we do, we need to do one of:
>
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
> 2. Teach the I/O layers to deal in PFNs instead of struct pages
> 3. Replace struct page with some other structure that can represent both
> DRAM and PMEM
>
> I'm personally a fan of #3, and I was looking at the scatterlist as
> my preferred data structure. I now believe the scatterlist as it is
> currently defined isn't sufficient, so we probably end up needing a new
> data structure. I think Dan's preferred method of replacing struct
> pages with PFNs is actually less instrusive, but doesn't give us as
> much advantage (an entirely new data structure would let us move to an
> extent based system at the same time, instead of sticking with an array
> of pages). Clearly Boaz prefers 1a, which works well enough for the
> 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>
> What's your preference? I guess option 0 is "force all I/O to go
> through the page cache and then get copied", but that feels like a nasty
> performance hit.
In addition to the options there's also a time line. At least for the
short term where we want to get something going 1a seems like the
absolutely be option. It works perfectly fine for the lots of small
capacity dram-like nvdimms, and it works funtionally fine for the
special huge ones, although the resource use for it is highly annoying.
If it turns out to be too annoying we can also offer a no I/O possible
option for them in the short run.
In the long run option 2) sounds like a good plan to me, but not as a
parallel I/O path, but as the main one. Doing so will in fact give us
options to experiment with 3). Given that we're moving towards an
increasinly huge page using world replacing the good old struct page
with something extent-like and/or temporary might be needed for dram
as well in the future.
^ permalink raw reply
* SMRFFS - EXT4, and a few stack changes
From: Adrian Palmer @ 2015-03-19 15:59 UTC (permalink / raw)
To: Linux Filesystem Development List, ext4 development, fstests,
linux-raid, linux-scsi, linux-ide
Hi all;
As some have been asking, I've posted some proof-of-concept code at
(https://github.com/Seagate/SMR_FS-EXT4).
Let me know any feedback. We're in the process of planning out the
schedule moving forward.
I'm spamming everyone because of the extent of the changes we've touched.
I'm also copying the RAID list because I want to start a discussion
about planning RAID solutions on SMR.
Adrian Palmer
Firmware Engineer II
R&D Firmware
Seagate, Longmont Colorado
720-684-1307
adrian.palmer@seagate.com
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-19 15:54 UTC (permalink / raw)
To: Matthew Wilcox, Andrew Morton
Cc: linux-arch, axboe, riel, hch, linux-nvdimm, Dave Hansen,
linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
<>
>
> Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> want to be able to do any kind of I/O directly to persistent memory,
> and I think we do, we need to do one of:
>
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
> 2. Teach the I/O layers to deal in PFNs instead of struct pages
> 3. Replace struct page with some other structure that can represent both
> DRAM and PMEM
>
> I'm personally a fan of #3, and I was looking at the scatterlist as
> my preferred data structure. I now believe the scatterlist as it is
> currently defined isn't sufficient, so we probably end up needing a new
> data structure. I think Dan's preferred method of replacing struct
> pages with PFNs is actually less instrusive, but doesn't give us as
> much advantage (an entirely new data structure would let us move to an
> extent based system at the same time, instead of sticking with an array
> of pages). Clearly Boaz prefers 1a, which works well enough for the
> 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>
> What's your preference? I guess option 0 is "force all I/O to go
> through the page cache and then get copied", but that feels like a nasty
> performance hit.
Thanks Matthew, you have summarized it perfectly.
I think #1b might have merit, as well. I have a very surgical small
"hack" that we can do with allocating on demand pages before IO.
It involves adding a new MEMORY_MODEL policy that is derived from
SPARSEMEM but lets you allocate individual pages on demand. And a new
type of page say call it GP_emulated_page.
(Tell me if you find this interesting. It is 1/117 in size of both
#2 or #3)
In anyway please reconsider a configurable #1a for people that do
not mind sacrificing 1.2% of their pmem for real pages.
Even at 6G page-structs with 400G pmem, people would love some of the stuff
this gives them today. just few examples: direct_access from within a VM to
an host defined pmem, is trivial with no extra code with my two simple #1a
patches. RDMA memory brick targets, network shared memory FS and so on, the
list will always be bigger then any of #1b #2 or #3. Yes for people that
want to sacrifice the extra cost.
In the Kernel it was always about choice and diversity. And what does it
costs us. Nothing. Two small simple patches and a Kconfig option.
Note that I made it in such a way that if pmem is configured without
use of pages, then the mm code is *not* configured-in automatically.
We can even add a runtime option that even if #1a is enabled, for certain
pmem device may not want pages allocated. And so choose at runtime rather
than compile time.
I think this will only farther our cause and let people advance with
their research and development with great new ideas about use of pmem.
Then once there is a great demand for #1a and those large 512G devices
come out, we can go the #1b or #3 route and save them the extra 1.2%
memory, but once they have the appetite for it. (And Andrews question
becomes clear)
Our two ways need not be "either-or". They can be "have both". I think
choice is a good thing for us here. Even with #3 available #1a still has
merit in some configurations and they can co exist perfectly.
Please think about it?
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-19 13:43 UTC (permalink / raw)
To: Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, riel, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel
In-Reply-To: <20150318132650.3336261c58829f49a9af8675@linux-foundation.org>
On Wed, Mar 18, 2015 at 01:26:50PM -0700, Andrew Morton wrote:
> On Mon, 16 Mar 2015 16:25:25 -0400 Dan Williams <dan.j.williams@intel.com> wrote:
>
> > Avoid the impending disaster of requiring struct page coverage for what
> > is expected to be ever increasing capacities of persistent memory. In
> > conversations with Rik van Riel, Mel Gorman, and Jens Axboe at the
> > recently concluded Linux Storage Summit it became clear that struct page
> > is not required in many places, it was simply convenient to re-use.
> >
> > Introduce helpers and infrastructure to remove struct page usage where
> > it is not necessary. One use case for these changes is to implement a
> > write-back-cache in persistent memory for software-RAID. Another use
> > case for the scatterlist changes is RDMA to a pfn-range.
>
> Those use-cases sound very thin. If that's all we have then I'd say
> "find another way of implementing those things without creating
> pageframes for persistent memory".
>
> IOW, please tell us much much much more about the value of this change.
Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
want to be able to do any kind of I/O directly to persistent memory,
and I think we do, we need to do one of:
1. Construct struct pages for persistent memory
1a. Permanently
1b. While the pages are under I/O
2. Teach the I/O layers to deal in PFNs instead of struct pages
3. Replace struct page with some other structure that can represent both
DRAM and PMEM
I'm personally a fan of #3, and I was looking at the scatterlist as
my preferred data structure. I now believe the scatterlist as it is
currently defined isn't sufficient, so we probably end up needing a new
data structure. I think Dan's preferred method of replacing struct
pages with PFNs is actually less instrusive, but doesn't give us as
much advantage (an entirely new data structure would let us move to an
extent based system at the same time, instead of sticking with an array
of pages). Clearly Boaz prefers 1a, which works well enough for the
8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
What's your preference? I guess option 0 is "force all I/O to go
through the page cache and then get copied", but that feels like a nasty
performance hit.
^ permalink raw reply
* Re: [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: NeilBrown @ 2015-03-19 6:02 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <550A60FF.3050902@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2473 bytes --]
On Wed, 18 Mar 2015 23:39:11 -0600 Eric Mei <meijia@gmail.com> wrote:
> From: Eric Mei <eric.mei@seagate.com>
>
> When array is degraded, read data landed on failed drives will result in
> reading rest of data in a stripe. So a single sequential read would
> result in same data being read twice.
>
> This patch is to avoid chunk aligned read for degraded array. The
> downside is to involve stripe cache which means associated CPU overhead
> and extra memory copy.
>
> Signed-off-by: Eric Mei <eric.mei@seagate.com>
> ---
> drivers/md/raid5.c | 15 ++++++++++++---
> 1 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index cd2f96b..763c64a 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
> unsigned int chunk_sectors = mddev->chunk_sectors;
> unsigned int bio_sectors = bvm->bi_size >> 9;
>
> - if ((bvm->bi_rw & 1) == WRITE)
> - return biovec->bv_len; /* always allow writes to be
> mergeable */
> + /*
> + * always allow writes to be mergeable, read as well if array
> + * is degraded as we'll go through stripe cache anyway.
> + */
> + if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
> + return biovec->bv_len;
>
> if (mddev->new_chunk_sectors < mddev->chunk_sectors)
> chunk_sectors = mddev->new_chunk_sectors;
> @@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
> struct bio * bi)
>
> md_write_start(mddev, bi);
>
> - if (rw == READ &&
> + /*
> + * If array is degraded, better not do chunk aligned read because
> + * later we might have to read it again in order to reconstruct
> + * data on failed drives.
> + */
> + if (rw == READ && mddev->degraded == 0 &&
> mddev->reshape_position == MaxSector &&
> chunk_aligned_read(mddev,bi))
> return;
Thanks for the patch.
However this sort of patch really needs to come with some concrete
performance numbers. Preferably both sequential reads and random reads.
I agree that sequential reads are likely to be faster, but how much faster
are they?
I imagine that this might make random reads a little slower. Does it? By
how much?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [PATCH] md/raid5: don't do chunk aligned read on degraded array.
From: Eric Mei @ 2015-03-19 5:39 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
From: Eric Mei <eric.mei@seagate.com>
When array is degraded, read data landed on failed drives will result in
reading rest of data in a stripe. So a single sequential read would
result in same data being read twice.
This patch is to avoid chunk aligned read for degraded array. The
downside is to involve stripe cache which means associated CPU overhead
and extra memory copy.
Signed-off-by: Eric Mei <eric.mei@seagate.com>
---
drivers/md/raid5.c | 15 ++++++++++++---
1 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..763c64a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4180,8 +4180,12 @@ static int raid5_mergeable_bvec(struct mddev *mddev,
unsigned int chunk_sectors = mddev->chunk_sectors;
unsigned int bio_sectors = bvm->bi_size >> 9;
- if ((bvm->bi_rw & 1) == WRITE)
- return biovec->bv_len; /* always allow writes to be
mergeable */
+ /*
+ * always allow writes to be mergeable, read as well if array
+ * is degraded as we'll go through stripe cache anyway.
+ */
+ if ((bvm->bi_rw & 1) == WRITE || mddev->degraded)
+ return biovec->bv_len;
if (mddev->new_chunk_sectors < mddev->chunk_sectors)
chunk_sectors = mddev->new_chunk_sectors;
@@ -4656,7 +4660,12 @@ static void make_request(struct mddev *mddev,
struct bio * bi)
md_write_start(mddev, bi);
- if (rw == READ &&
+ /*
+ * If array is degraded, better not do chunk aligned read because
+ * later we might have to read it again in order to reconstruct
+ * data on failed drives.
+ */
+ if (rw == READ && mddev->degraded == 0 &&
mddev->reshape_position == MaxSector &&
chunk_aligned_read(mddev,bi))
return;
--
1.7.1
^ permalink raw reply related
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: Guoqing Jiang @ 2015-03-19 3:50 UTC (permalink / raw)
To: NeilBrown; +Cc: jgq516, rgoldwyn, linux-raid
In-Reply-To: <20150318095712.1954b958@notabene.brown>
Hi Neil,
NeilBrown wrote:
> On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
>
>
>> From: Guoqing Jiang <gqjiang@suse.com>
>>
>> The bm_blocks is modified by commit fe60ce (md/bitmap: use
>> sector_div for sector_t divisions), but it makes bm_blocks
>> has different value which is changed from like "a/b" to "a%b",
>> need to correct this to make sure cluster-md still works.
>>
>
> One of us is confused here.
>
> This code is trying to find the start of the bitmap relevant to this host in
> a table of multiple bitmaps. So it first needs to find out the size of each
> bitmap. It then multiples the size by the index number of this host to get
> an offset.
>
>
Thanks for detailed description, it really helps. I quoted related lines
from bitmap.c.
574 sector_t bm_blocks;
575 sector_t resync_sectors =
bitmap->mddev->resync_max_sectors;
576
577 bm_blocks = sector_div(resync_sectors,
578
bitmap->mddev->bitmap_info.chunksize >> 9);
579 bm_blocks = bm_blocks << 3;
580 bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
581 bitmap->mddev->bitmap_info.offset +=
bitmap->cluster_slot * (bm_blocks << 3);
> So it take the total number of sectors (resync_max_sectors), divides by the
> chunksize (in sectors) to get a number of chunks. This is the number of bits.
>
>
L577 is supposed to do above job.
> Then it should div-round-up by 8 to get a number of bytes.
>
I guess what you mean is about L579, while it used "<<3" rather than
">>3" now.
> Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
> are always 4K aligned.
>
L580 did the job.
> Then this number is multiplied by 8 (or shifted by 3) to get a number of
> sectors to add to the start of the table.
>
L581 is for this, right? Is the shifted by 3 is to match the bitmap
format for each
nodes? Seems the relationship between slot and the bitmap region of the node
is like n <-----> [8*nK, 8*(n+1)K]. How about the following changes?
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 501f83f..b2a241b 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -571,12 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
re_read:
/* If cluster_slot is set, the cluster is setup */
if (bitmap->cluster_slot >= 0) {
- sector_t bm_blocks;
- sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
+ sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
- bm_blocks = sector_div(resync_sectors,
-
bitmap->mddev->bitmap_info.chunksize >> 9);
- bm_blocks = bm_blocks << 3;
+ sector_div(bm_blocks,
bitmap->mddev->bitmap_info.chunksize >> 9);
+ bm_blocks = bm_blocks >> 3;
bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
> is wrong because it uses sector_div in a way which destroys
> resync_max_sectors.
> And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
> convert from bits to bytes.
>
> commit f9209a323547f054c7439a3bf67c45e64a054bd
> removes the abuse of sector_div, which is good, but uses a simple "a/b"
> division, which isn't allowed in the kernel.
>
> commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
> then does the right thing with sector_div, but the "<< 3" is still the wrong
> way around.
>
> If you still think your code is correct, please explain in detail why.
>
> Goldwyn: if you agree that "<< 3" should be ">> 3" or even
> DIV_ROUND_UP_SECTOR_T( , 8);
> please send a patch. If you don't think so, please explain why.
>
>
But anyway, it is better wait for Goldwyn's back from vacation, :)
Thanks,
Guoqing
^ permalink raw reply related
* Re: Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: NeilBrown @ 2015-03-19 1:03 UTC (permalink / raw)
To: Sean Harris; +Cc: linux-raid
In-Reply-To: <DDB9AAD7-E69F-4D82-9A4D-2200E1D4C81D@backblaze.com>
[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]
On Wed, 18 Mar 2015 17:58:05 -0700 Sean Harris <sean@backblaze.com> wrote:
> Thanks Neil,
> I'd like to try this but I'm having trouble installing it on Debian 5 (lenny).
>
> When I try to make install, it complains:
>
> ***** Parent of /run/mdadm does not exist. Maybe set different RUN_DIR=
> ***** e.g. make RUN_DIR=/dev/.mdadm
> ***** or set CHECK_RUN_DIR=0
> make: *** [check_rundir] Error 1
>
> cat /etc/debian_version
> 5.0.10
>
> The default run directory apparently is not there ie: /run/mdadm
> That directory exists on Debian 7 but not Debian 5.
>
> I admittedly do not know what to set that variable to on Lenny. Can you point me to what to look for?
Does /dev/.mdadm exist? If it does, try
make RUN_DIR=/dev/.mdadm
If not, just use
make CHECK_RUN_DIR=0
You don't need access to the rundir to assemble some devices.
And don't "make install". Just "make", and then run
./mdadm ....
or copy mdadm to some where (e.g. /sbin/mdadm-new) and only use mdadm-new
when you particularly want the new features.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: Sean Harris @ 2015-03-19 0:58 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150318121054.066b963b@notabene.brown>
Thanks Neil,
I'd like to try this but I'm having trouble installing it on Debian 5 (lenny).
When I try to make install, it complains:
***** Parent of /run/mdadm does not exist. Maybe set different RUN_DIR=
***** e.g. make RUN_DIR=/dev/.mdadm
***** or set CHECK_RUN_DIR=0
make: *** [check_rundir] Error 1
cat /etc/debian_version
5.0.10
The default run directory apparently is not there ie: /run/mdadm
That directory exists on Debian 7 but not Debian 5.
I admittedly do not know what to set that variable to on Lenny. Can you point me to what to look for?
Thanks,
Sean
> On Mar 17, 2015, at 6:10 PM, NeilBrown <neilb@suse.de> wrote:
>
> mdadm --assemble /dev/md10 --update=metadata ...list.of.devices..
^ permalink raw reply
* Re: dm log userspace: use mempool_create_kmalloc_pool()
From: Mike Snitzer @ 2015-03-18 22:30 UTC (permalink / raw)
To: David Rientjes
Cc: Alasdair Kergon, Neil Brown, linux-kernel, linux-raid, dm-devel
In-Reply-To: <20150311125852.GA28768@redhat.com>
On Wed, Mar 11 2015 at 8:58am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:
> On Sat, Mar 07 2015 at 7:43pm -0500,
> David Rientjes <rientjes@google.com> wrote:
>
> > Mempools created for kmalloc caches should use
> > mempool_create_kmalloc_pool().
> >
> > Cc: Alasdair Kergon <agk@redhat.com>
> > Cc: Mike Snitzer <snitzer@redhat.com>
> > Cc: Neil Brown <neilb@suse.de>
> > Signed-off-by: David Rientjes <rientjes@google.com>
>
> Thanks David, I'll get this queued up.
Turns out I already fixed this a different way about a month ago and
staged it for 4.1, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=db57216e710ea5c517f1030fafe7eb21ac938dc5
You'll be interested to note this portion of that commit's header:
"This _should_ eliminate DM's need for io_schedule_timeout() in
mempool_alloc(); so io_schedule() should be sufficient now."
I'll defer to you on whether you'd like to pursue switching to
io_schedule() in mempool_alloc().
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Andrew Morton @ 2015-03-18 20:26 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel, linux-arch, axboe, riel, linux-nvdimm, Dave Hansen,
linux-raid, mgorman, hch, linux-fsdevel, Matthew Wilcox
In-Reply-To: <20150316201640.33102.33761.stgit@dwillia2-desk3.amr.corp.intel.com>
On Mon, 16 Mar 2015 16:25:25 -0400 Dan Williams <dan.j.williams@intel.com> wrote:
> Avoid the impending disaster of requiring struct page coverage for what
> is expected to be ever increasing capacities of persistent memory. In
> conversations with Rik van Riel, Mel Gorman, and Jens Axboe at the
> recently concluded Linux Storage Summit it became clear that struct page
> is not required in many places, it was simply convenient to re-use.
>
> Introduce helpers and infrastructure to remove struct page usage where
> it is not necessary. One use case for these changes is to implement a
> write-back-cache in persistent memory for software-RAID. Another use
> case for the scatterlist changes is RDMA to a pfn-range.
Those use-cases sound very thin. If that's all we have then I'd say
"find another way of implementing those things without creating
pageframes for persistent memory".
IOW, please tell us much much much more about the value of this change.
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Dan Williams @ 2015-03-18 15:35 UTC (permalink / raw)
To: Boaz Harrosh
Cc: linux-kernel@vger.kernel.org, Jens Axboe, Christoph Hellwig,
Al Viro, Andrew Morton, Linus Torvalds, linux-arch, riel,
linux-nvdimm@lists.01.org, Dave Hansen, linux-raid, mgorman,
linux-fsdevel, Matthew Wilcox
In-Reply-To: <550957B9.5050803@gmail.com>
On Wed, Mar 18, 2015 at 3:47 AM, Boaz Harrosh <openosd@gmail.com> wrote:
> On 03/16/2015 10:25 PM, Dan Williams wrote:
>> Avoid the impending disaster of requiring struct page coverage for what
>> is expected to be ever increasing capacities of persistent memory.
>
> If you are saying "disaster", than we need to believe you. Or is there
> a scientific proof for this.
The same Moore's Law based extrapolation that Dave Chinner did to
determine that major feature development on XFS may cease in 5 - 7
years. In Dave's words we're looking ahead to "lots and fast". Given
the time scale of getting kernel changes out to end users in an
enterprise kernel update the "dynamic page struct allocation" approach
is already insufficient.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-18 14:38 UTC (permalink / raw)
To: Matthew Wilcox, Boaz Harrosh
Cc: axboe, linux-arch, riel, linux-raid, linux-nvdimm, Dave Hansen,
linux-kernel, hch, Linus Torvalds, Al Viro, linux-fsdevel,
Andrew Morton, mgorman
In-Reply-To: <20150318130641.GD4003@linux.intel.com>
On 03/18/2015 03:06 PM, Matthew Wilcox wrote:
> On Wed, Mar 18, 2015 at 12:47:21PM +0200, Boaz Harrosh wrote:
>> God! Look at this endless list of files and it is only the very beginning.
>> It does not even work and touches only 10% of what will need to be touched
>> for this to work, and very very marginally at that. There will always be
>> "another subsystem" that will not work. For example NUMA how will you do
>> NUMA aware pmem? and this is just a simple example. (I'm saying NUMA
>> because our tests show a huge drop in performance if you do not do
>> NUMA aware allocation)
>
> You're very entertaining, but please, tone down your emails and stick
> to facts. The BIOS presents the persistent memory as one table entry
> per NUMA node, so you get one block device per NUMA node. There's no
> mixing of memory from different NUMA nodes within a single filesystem,
> unless you have a filesystem that uses multiple block devices.
>
Not current BIOS, if we have them contiguous then they are presented as
one range. (DDR3 BIOS). But I agree it is a bug and in our configuration
we separate them to different pmem devices.
Yes I meant a "filesystem that uses multiple block devices"
>> I'm not the one afraid of hard work, if it was for a good cause, but for what?
>> really for what? The block layer, and RDMA, and networking, and spline, and what
>> ever the heck any one wants to imagine to do with pmem, already works perfectly
>> stable. right now!
>
> The overhead. Allocating a struct page for every 4k page in a 400GB DIMM
> (the current capacity available from one NV-DIMM vendor) occupies 6.4GB.
> That's an unacceptable amount of overhead.
>
So lets fix the stacks to work nice with 2M pages. That said we can
allocate the struct page also from pmem if we need to. The fact remains
that we need state down the different stacks and this is the current
design over all.
I hate it that you introduce a double design a pfn-or-page and the
combinations of them. It is ugliness to much for my guts. I would
like a unified design. that runs all over the stack. Already we have
too much duplication to my taste, and would love to see more
unification and not more splitting.
But the most important for me is do we have to sacrifice the short
term to the long term. Such a massive change as you are proposing
it will take years. for a theoretical 400GB DIMM. What about the
4G DIMM now in peoples hands, need they wait?
(Though I still do not agree with your design)
I love the SPARSE model of the "section" and the page being it's
own identity relative to virtual & PFN of the section. We could
think of a much smaller page-struct that only takes a ref-count
and flags and have bigger page type for regular use, separate the
low common part of the page, lay down clear rules about its use,
and an high part that's per user. But let us think of a unified
design through out. (most members of page are accessed through
wrappers it would be relatively easy to split)
And let us not sacrifice the now for the "far tomorrow", we should
be able to do this incrementally, wasting more space now and saving
later.
[We can even invent a sizeless page you know how we encode
the section ID directly into the 64 bit address of the page,
So we can have a flag at the section that says this is a
zero-size page section and the needed info is stored at
the section object. But I still think you will need state
per page and that we do need a minimal size.
]
[BTW: The only 400GB DIMM I know of is a real flash, and not directly
mapped to CPU, OK maybe read only, but the erase/write makes it
logical-to-physical managed and not directly accessed
]
And a personal note. I mean only to entertain. If any one feels
I "toned-up", please forgive me. I meant no such thing. As a rule
if I come across strong then please just laugh and don't take me
seriously. I only mean scientific soundness.
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-18 13:06 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Dan Williams, linux-kernel, axboe, hch, Al Viro, Andrew Morton,
Linus Torvalds, linux-arch, riel, linux-nvdimm, Dave Hansen,
linux-raid, mgorman, linux-fsdevel
In-Reply-To: <550957B9.5050803@gmail.com>
On Wed, Mar 18, 2015 at 12:47:21PM +0200, Boaz Harrosh wrote:
> God! Look at this endless list of files and it is only the very beginning.
> It does not even work and touches only 10% of what will need to be touched
> for this to work, and very very marginally at that. There will always be
> "another subsystem" that will not work. For example NUMA how will you do
> NUMA aware pmem? and this is just a simple example. (I'm saying NUMA
> because our tests show a huge drop in performance if you do not do
> NUMA aware allocation)
You're very entertaining, but please, tone down your emails and stick
to facts. The BIOS presents the persistent memory as one table entry
per NUMA node, so you get one block device per NUMA node. There's no
mixing of memory from different NUMA nodes within a single filesystem,
unless you have a filesystem that uses multiple block devices.
> I'm not the one afraid of hard work, if it was for a good cause, but for what?
> really for what? The block layer, and RDMA, and networking, and spline, and what
> ever the heck any one wants to imagine to do with pmem, already works perfectly
> stable. right now!
The overhead. Allocating a struct page for every 4k page in a 400GB DIMM
(the current capacity available from one NV-DIMM vendor) occupies 6.4GB.
That's an unacceptable amount of overhead.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 3/7] dma-mapping: allow archs to optionally specify a ->map_pfn() operation
From: Boaz Harrosh @ 2015-03-18 11:21 UTC (permalink / raw)
To: Dan Williams, linux-kernel
Cc: axboe, linux-raid, riel, linux-nvdimm, hch, mgorman,
linux-fsdevel
In-Reply-To: <20150316202543.33102.12409.stgit@dwillia2-desk3.amr.corp.intel.com>
On 03/16/2015 10:25 PM, Dan Williams wrote:
> This is in support of enabling block device drivers to perform DMA
> to/from persistent memory which may not have a backing struct page
> entry.
>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
> arch/Kconfig | 3 +++
> include/asm-generic/dma-mapping-common.h | 30 ++++++++++++++++++++++++++++++
> include/linux/dma-debug.h | 23 +++++++++++++++++++----
> include/linux/dma-mapping.h | 8 +++++++-
> lib/dma-debug.c | 4 ++--
> 5 files changed, 61 insertions(+), 7 deletions(-)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 05d7a8a458d5..80ea3e124494 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -203,6 +203,9 @@ config HAVE_DMA_ATTRS
> config HAVE_DMA_CONTIGUOUS
> bool
>
> +config HAVE_DMA_PFN
> + bool
> +
> config GENERIC_SMP_IDLE_THREAD
> bool
>
> diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
> index 3378dcf4c31e..58fad817e51a 100644
> --- a/include/asm-generic/dma-mapping-common.h
> +++ b/include/asm-generic/dma-mapping-common.h
> @@ -17,9 +17,15 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
>
> kmemcheck_mark_initialized(ptr, size);
> BUG_ON(!valid_dma_direction(dir));
> +#ifdef CONFIG_HAVE_DMA_PFN
> + addr = ops->map_pfn(dev, page_to_pfn_typed(virt_to_page(ptr)),
> + (unsigned long)ptr & ~PAGE_MASK, size,
> + dir, attrs);
> +#else
Yes our beloved Kernel is full of #ifdef(s) in the middle of the code
Very beautiful
> addr = ops->map_page(dev, virt_to_page(ptr),
> (unsigned long)ptr & ~PAGE_MASK, size,
> dir, attrs);
> +#endif
> debug_dma_map_page(dev, virt_to_page(ptr),
> (unsigned long)ptr & ~PAGE_MASK, size,
> dir, addr, true);
> @@ -68,6 +74,29 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg
> ops->unmap_sg(dev, sg, nents, dir, attrs);
> }
>
> +#ifdef CONFIG_HAVE_DMA_PFN
> +static inline dma_addr_t dma_map_pfn(struct device *dev, __pfn_t pfn,
> + size_t offset, size_t size,
> + enum dma_data_direction dir)
> +{
> + struct dma_map_ops *ops = get_dma_ops(dev);
> + dma_addr_t addr;
> +
> + BUG_ON(!valid_dma_direction(dir));
> + addr = ops->map_pfn(dev, pfn, offset, size, dir, NULL);
> + debug_dma_map_pfn(dev, pfn, offset, size, dir, addr, false);
> +
> + return addr;
> +}
> +
> +static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
> + size_t offset, size_t size,
> + enum dma_data_direction dir)
> +{
> + kmemcheck_mark_initialized(page_address(page) + offset, size);
> + return dma_map_pfn(dev, page_to_pfn_typed(page), offset, size, dir);
> +}
> +#else
And in the middle of source code files
> static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
> size_t offset, size_t size,
> enum dma_data_direction dir)
> @@ -82,6 +111,7 @@ static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
>
> return addr;
> }
> +#endif /* CONFIG_HAVE_DMA_PFN */
>
> static inline void dma_unmap_page(struct device *dev, dma_addr_t addr,
> size_t size, enum dma_data_direction dir)
> diff --git a/include/linux/dma-debug.h b/include/linux/dma-debug.h
> index fe8cb610deac..eb3e69c61e5e 100644
> --- a/include/linux/dma-debug.h
> +++ b/include/linux/dma-debug.h
> @@ -34,10 +34,18 @@ extern void dma_debug_init(u32 num_entries);
>
> extern int dma_debug_resize_entries(u32 num_entries);
>
> -extern void debug_dma_map_page(struct device *dev, struct page *page,
> - size_t offset, size_t size,
> - int direction, dma_addr_t dma_addr,
> - bool map_single);
> +extern void debug_dma_map_pfn(struct device *dev, __pfn_t pfn, size_t offset,
> + size_t size, int direction, dma_addr_t dma_addr,
> + bool map_single);
> +
> +static inline void debug_dma_map_page(struct device *dev, struct page *page,
> + size_t offset, size_t size,
> + int direction, dma_addr_t dma_addr,
> + bool map_single)
> +{
> + return debug_dma_map_pfn(dev, page_to_pfn_typed(page), offset, size,
> + direction, dma_addr, map_single);
> +}
>
> extern void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
>
> @@ -109,6 +117,13 @@ static inline void debug_dma_map_page(struct device *dev, struct page *page,
> {
> }
>
> +static inline void debug_dma_map_pfn(struct device *dev, __pfn_t pfn,
> + size_t offset, size_t size,
> + int direction, dma_addr_t dma_addr,
> + bool map_single)
> +{
> +}
> +
> static inline void debug_dma_mapping_error(struct device *dev,
> dma_addr_t dma_addr)
> {
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index c3007cb4bfa6..6411621e4179 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -26,11 +26,17 @@ struct dma_map_ops {
>
> int (*get_sgtable)(struct device *dev, struct sg_table *sgt, void *,
> dma_addr_t, size_t, struct dma_attrs *attrs);
> -
> +#ifdef CONFIG_HAVE_DMA_PFN
> + dma_addr_t (*map_pfn)(struct device *dev, __pfn_t pfn,
> + unsigned long offset, size_t size,
> + enum dma_data_direction dir,
> + struct dma_attrs *attrs);
> +#else
And in the middle of structures
> dma_addr_t (*map_page)(struct device *dev, struct page *page,
> unsigned long offset, size_t size,
> enum dma_data_direction dir,
> struct dma_attrs *attrs);
> +#endif
> void (*unmap_page)(struct device *dev, dma_addr_t dma_handle,
> size_t size, enum dma_data_direction dir,
> struct dma_attrs *attrs);
> diff --git a/lib/dma-debug.c b/lib/dma-debug.c
> index 9722bd2dbc9b..a447730fff97 100644
> --- a/lib/dma-debug.c
> +++ b/lib/dma-debug.c
> @@ -1250,7 +1250,7 @@ out:
> put_hash_bucket(bucket, &flags);
> }
>
> -void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
> +void debug_dma_map_pfn(struct device *dev, __pfn_t pfn, size_t offset,
> size_t size, int direction, dma_addr_t dma_addr,
> bool map_single)
> {
> @@ -1268,7 +1268,7 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
>
> entry->dev = dev;
> entry->type = dma_debug_page;
> - entry->pfn = page_to_pfn(page);
> + entry->pfn = pfn.pfn;
> entry->offset = offset,
> entry->dev_addr = dma_addr;
> entry->size = size;
>
This is exactly what I meant. It is not only two different code paths it is
two different compilation paths. This is a maintenance nightmare. And a sure
bit rot.
Real nice for nothing
Thanks, but no thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-18 10:47 UTC (permalink / raw)
To: Dan Williams, linux-kernel, axboe, hch, Al Viro, Andrew Morton,
Linus Torvalds
Cc: linux-arch, riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman,
linux-fsdevel, Matthew Wilcox
In-Reply-To: <20150316201640.33102.33761.stgit@dwillia2-desk3.amr.corp.intel.com>
On 03/16/2015 10:25 PM, Dan Williams wrote:
> Avoid the impending disaster of requiring struct page coverage for what
> is expected to be ever increasing capacities of persistent memory.
If you are saying "disaster", than we need to believe you. Or is there
a scientific proof for this.
Actually what you are proposing below, is the "real disaster".
(I do hope it is not impending)
> In conversations with Rik van Riel, Mel Gorman, and Jens Axboe at the
> recently concluded Linux Storage Summit it became clear that struct page
> is not required in many places, it was simply convenient to re-use.
>
> Introduce helpers and infrastructure to remove struct page usage where
> it is not necessary. One use case for these changes is to implement a
> write-back-cache in persistent memory for software-RAID. Another use
> case for the scatterlist changes is RDMA to a pfn-range.
>
> This compiles and boots, but 0day-kbuild-robot coverage is needed before
> this set exits "RFC". Obviously, the coccinelle script needs to be
> re-run on the block updates for kernel.next. As is, this only includes
> the resulting auto-generated-patch against 4.0-rc3.
>
> ---
>
> Dan Williams (6):
> block: add helpers for accessing a bio_vec page
> block: convert bio_vec.bv_page to bv_pfn
> dma-mapping: allow archs to optionally specify a ->map_pfn() operation
> scatterlist: use sg_phys()
> x86: support dma_map_pfn()
> block: base support for pfn i/o
>
> Matthew Wilcox (1):
> scatterlist: support "page-less" (__pfn_t only) entries
>
>
> arch/Kconfig | 3 +
> arch/arm/mm/dma-mapping.c | 2 -
> arch/microblaze/kernel/dma.c | 2 -
> arch/powerpc/sysdev/axonram.c | 2 -
> arch/x86/Kconfig | 12 +++
> arch/x86/kernel/amd_gart_64.c | 22 ++++--
> arch/x86/kernel/pci-nommu.c | 22 ++++--
> arch/x86/kernel/pci-swiotlb.c | 4 +
> arch/x86/pci/sta2x11-fixup.c | 4 +
> arch/x86/xen/pci-swiotlb-xen.c | 4 +
> block/bio-integrity.c | 8 +-
> block/bio.c | 83 +++++++++++++++------
> block/blk-core.c | 9 ++
> block/blk-integrity.c | 7 +-
> block/blk-lib.c | 2 -
> block/blk-merge.c | 15 ++--
> block/bounce.c | 26 +++----
> drivers/block/aoe/aoecmd.c | 8 +-
> drivers/block/brd.c | 2 -
> drivers/block/drbd/drbd_bitmap.c | 5 +
> drivers/block/drbd/drbd_main.c | 4 +
> drivers/block/drbd/drbd_receiver.c | 4 +
> drivers/block/drbd/drbd_worker.c | 3 +
> drivers/block/floppy.c | 6 +-
> drivers/block/loop.c | 8 +-
> drivers/block/nbd.c | 8 +-
> drivers/block/nvme-core.c | 2 -
> drivers/block/pktcdvd.c | 11 ++-
> drivers/block/ps3disk.c | 2 -
> drivers/block/ps3vram.c | 2 -
> drivers/block/rbd.c | 2 -
> drivers/block/rsxx/dma.c | 3 +
> drivers/block/umem.c | 2 -
> drivers/block/zram/zram_drv.c | 10 +--
> drivers/dma/ste_dma40.c | 5 -
> drivers/iommu/amd_iommu.c | 21 ++++-
> drivers/iommu/intel-iommu.c | 26 +++++--
> drivers/iommu/iommu.c | 2 -
> drivers/md/bcache/btree.c | 4 +
> drivers/md/bcache/debug.c | 6 +-
> drivers/md/bcache/movinggc.c | 2 -
> drivers/md/bcache/request.c | 6 +-
> drivers/md/bcache/super.c | 10 +--
> drivers/md/bcache/util.c | 5 +
> drivers/md/bcache/writeback.c | 2 -
> drivers/md/dm-crypt.c | 12 ++-
> drivers/md/dm-io.c | 2 -
> drivers/md/dm-verity.c | 2 -
> drivers/md/raid1.c | 50 +++++++------
> drivers/md/raid10.c | 38 +++++-----
> drivers/md/raid5.c | 6 +-
> drivers/mmc/card/queue.c | 4 +
> drivers/s390/block/dasd_diag.c | 2 -
> drivers/s390/block/dasd_eckd.c | 14 ++--
> drivers/s390/block/dasd_fba.c | 6 +-
> drivers/s390/block/dcssblk.c | 2 -
> drivers/s390/block/scm_blk.c | 2 -
> drivers/s390/block/scm_blk_cluster.c | 2 -
> drivers/s390/block/xpram.c | 2 -
> drivers/scsi/mpt2sas/mpt2sas_transport.c | 6 +-
> drivers/scsi/mpt3sas/mpt3sas_transport.c | 6 +-
> drivers/scsi/sd_dif.c | 4 +
> drivers/staging/android/ion/ion_chunk_heap.c | 4 +
> drivers/staging/lustre/lustre/llite/lloop.c | 2 -
> drivers/xen/biomerge.c | 4 +
> drivers/xen/swiotlb-xen.c | 29 +++++--
> fs/btrfs/check-integrity.c | 6 +-
> fs/btrfs/compression.c | 12 ++-
> fs/btrfs/disk-io.c | 4 +
> fs/btrfs/extent_io.c | 8 +-
> fs/btrfs/file-item.c | 8 +-
> fs/btrfs/inode.c | 18 +++--
> fs/btrfs/raid56.c | 4 +
> fs/btrfs/volumes.c | 2 -
> fs/buffer.c | 4 +
> fs/direct-io.c | 2 -
> fs/exofs/ore.c | 4 +
> fs/exofs/ore_raid.c | 2 -
> fs/ext4/page-io.c | 2 -
> fs/f2fs/data.c | 4 +
> fs/f2fs/segment.c | 2 -
> fs/gfs2/lops.c | 4 +
> fs/jfs/jfs_logmgr.c | 4 +
> fs/logfs/dev_bdev.c | 10 +--
> fs/mpage.c | 2 -
> fs/splice.c | 2 -
> include/asm-generic/dma-mapping-common.h | 30 ++++++++
> include/asm-generic/memory_model.h | 4 +
> include/asm-generic/scatterlist.h | 6 ++
> include/crypto/scatterwalk.h | 10 +++
> include/linux/bio.h | 24 +++---
> include/linux/blk_types.h | 21 +++++
> include/linux/blkdev.h | 2 +
> include/linux/dma-debug.h | 23 +++++-
> include/linux/dma-mapping.h | 8 ++
> include/linux/scatterlist.h | 101 ++++++++++++++++++++++++--
> include/linux/swiotlb.h | 5 +
> kernel/power/block_io.c | 2 -
> lib/dma-debug.c | 4 +
> lib/swiotlb.c | 20 ++++-
> mm/iov_iter.c | 22 +++---
> mm/page_io.c | 8 +-
> net/ceph/messenger.c | 2 -
God! Look at this endless list of files and it is only the very beginning.
It does not even work and touches only 10% of what will need to be touched
for this to work, and very very marginally at that. There will always be
"another subsystem" that will not work. For example NUMA how will you do
NUMA aware pmem? and this is just a simple example. (I'm saying NUMA
because our tests show a huge drop in performance if you do not do
NUMA aware allocation)
Al, Jens, Christoph Andrew. Think of the immediate stability nightmare and
the long term torture to maintain two code paths. Two set of tests, and
the combinatorial explosions of tests.
I'm not the one afraid of hard work, if it was for a good cause, but for what?
really for what? The block layer, and RDMA, and networking, and spline, and what
ever the heck any one wants to imagine to do with pmem, already works perfectly
stable. right now!
We have set up RDMA pmem target without a single line of extra code,
and the RDMA client was trivial to write. We are sending down block layer
BIOs from pmem from day one, and even iscsi NFS and any kind of networking
directly from pmem, for almost a year now.
All it takes is two simple patches to mm that creates a pages-section
for pmem. The Kernel DOCs do says that a page is a construct that keeps track
of the sate of a physical page in memory. A memory mapped pmem is perfectly
that, and it has state that needs tracking just the same, Say that converted
block layers of yours now happens to be an iscsi and goes through the network
stack, it starts to need ref-counting, flags ... It has state.
Matthew Dan. I don't get it. Don't you guys at Intel have nothing to do? why
change half the Kernel? for what? to achieve what? all your wildest dreams
about pmem are right here already. What is it that you guys want to do with
this code that we cannot already do? And I can show you two tons of things
you cannot do with this code that we can already do. With two simple patches.
If it is stability that you are concerned with, "what if a pmem-page gets
to the wrong mm subsystem?" There are a couple small hardening patches and
and extra page-flag allocated, that can make the all thing foolproof. Though
up until now I have not encountered any problem.
> 103 files changed, 658 insertions(+), 335 deletions(-)
Please look, this is only the beginning. And does not even work. Let us come
back to our senses. As true hackers lets do the minimum effort to achieve new
heights. All it really takes to do all this is 2 little patches.
Cheers
Boaz
^ permalink raw reply
* Re: Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: NeilBrown @ 2015-03-18 1:10 UTC (permalink / raw)
To: Sean Harris; +Cc: linux-raid
In-Reply-To: <149825A6-89BC-4804-915B-5241EEB15300@backblaze.com>
[-- Attachment #1: Type: text/plain, Size: 5833 bytes --]
On Tue, 17 Mar 2015 15:49:36 -0700 Sean Harris <sean@backblaze.com> wrote:
> Hi Neil and company,
>
> I have roughly a hundred machines that are still on Debian 5 that are slated for OS upgrades, and a great many of those are running software raid with the 00.90 version superblock. I even have a few machines on debian 7 with the .90 superblock because the boot drive for the machine had been replaced at some point.
>
> I would like to explore the idea of upgrading/converting the raid superblock on many of those machines to version 1.0, mostly to keep the environment as uniform as possible, and standardize training for my SysAdmins on re-assembling failed raid arrays.
> But I am not sure if it is a safe and necessary, or even a worthwhile endeavor.
Safe: there is no intrinsic reason for it not to be safe. Obviously any
change brings risks, but they can be managed with care.
Necessary: It is only necessary from the technical perspective of md if you
want to use some features that are only available with 1.x metadata.
This included bad-block-logs and improved flexibility for reshaping
RAID5/6/10 arrays.
Worthwhile: That is up to you. As you say, consistency can be good and
better familiarity for sysadmins is not a bad thing.
>
> I wanted to test this on a few non-production machines, but I am not quite sure if I need to specify the --size parameter in the create statement in the attempt.
>
> I have a bit of a mixed environment, but very early on, the machines initialized with the .90 superblock were all created in the same way ie:
>
> /sbin/mdadm --create /dev/md10 --bitmap=internal --auto=yes -l 6 -n 15 /dev/sd{b,e,h,k,n,q,t,w,z,ac,af,ai,al,ao,ar}1
>
I recommend not doing it this way. If you use mdadm-3.3 or later you can
mdadm --assemble /dev/md10 --update=metadata ...list.of.devices..
and it will update the metadata from 0.90 to 1.0 and start the array for you.
>
> Most of the machines I am interested in upgrading are running Debian 5, with LVM and ext4 volumes.
> Some machines use drive partitions, but we stopped using them some time ago and went with the simple notion that if a drive brand was marketed as 1TB, we assumed a size of 1,000,000,000,000 bytes and created the array with that size. That way, any drive manufacturer could be used as a replacement spare without worrying about the variation of their respective drive sizes.
>
> I think it is safe to upgrade the superblock from .90 to 1.0 because from what I have read, it is the same size, and in the same place as the .90 superblock.
Not correct. The 0.90 superblock is 4K in size and between 64K and 128K from
the end of the devices. The 1.0 superblock is 512 bytes and between 4K and
8K from the end of the device.
So the 1.0 superblock uses less space than the 0.90. But it does reside
entirely with space that is reserved when the 0.90 metadata is in use.
> But I don't know if having drives with or without partitions would make a difference, or if the conversion would need a specific version of mdadm utilized.
> (My hunch is that any layers on top of the raid array are irrelevant to the superblock conversion).
That's correct.
>
> All arrays are raid 6 with the same number of devices.
> I think what I need to run is something akin to
>
> /sbin/mdadm --create /dev/md10 -l6 -c64 --metadata=1.0 --assume-clean -n 15 /dev/sd{b,e,h,k,n,q,t,w,z,ac,af,ai,al,ao,ar}1
That would probably work, but again, "--assemble --update=metadata" is your
friend.
>
> Where md10 is the array, the raid level is 6, 64k chunk size, assume-clean (for a known good pre-existing array), 15 devices (all specified in order), and a metadata of 1.0.
>
> But I am not sure if I need to insure that --size is calculated and included in the conversion, or if the re-creation of the superblock simply takes care of it.
It probably doesn't matter, certainly not with a chunk size of 64K or larger.
>
> The reason I am unsure on this point is that with version 1.0 superblock arrays, I always include the --size parameter when re-creating the array with a specified list of devices, which is normally the Used_Dev_Size of the array, divided by 2, (since the value reported by mdadm is 512 byte sectors).
>
> I am including some information from my test pod. In the case below, there are only 9 devices per array. (But all of my production conversions will be with 15 devices).
>
> So here are my questions:
> Am I correct in presuming the conversion will work and simply put-in-place an upgraded superblock?
> Also, that it is unnecessary to specify --size for the create command (in this case).
> And also, that I need not worry about the version of mdadm used for the conversion? (in my case it will almost exclusively be version v2.6.7.2 on Debian 5 , or mdadm - v3.2.5 - 18th May 2012 on Debian 7.)
> Other than reading the superblock version after the conversion, is there any other way of verifying the conversion was successful without causing harm?
> (my thoughts were to run FULL raid checks and filesystem checks, and some crc checks to some files).
>
> Also, assuming I am on the right track and nothing can go wrong, are there any specific recommendations you have for going ahead with the conversions? Or even a recommendation to avoid this endeavor?
>
> Thanks for your feedback,
>
> Sean Harris
>
>
> Example from my test machine:
>
> debian_version
> 5.0.10
> proc/version
> Linux version 2.6.32-bpo.5-amd64 (Debian 2.6.32-35~bpo50+1) (norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Wed Jul 20 09:10:04 UTC 2011
> mdadm -V
> mdadm - v2.6.7.2 - 14th November 2008
Grab and compile the latest mdadm, and use that to update the metadata.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: NeilBrown @ 2015-03-17 22:57 UTC (permalink / raw)
To: jgq516; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
In-Reply-To: <1426560030-16152-1-git-send-email-jgq516@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2954 bytes --]
On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> The bm_blocks is modified by commit fe60ce (md/bitmap: use
> sector_div for sector_t divisions), but it makes bm_blocks
> has different value which is changed from like "a/b" to "a%b",
> need to correct this to make sure cluster-md still works.
One of us is confused here.
This code is trying to find the start of the bitmap relevant to this host in
a table of multiple bitmaps. So it first needs to find out the size of each
bitmap. It then multiples the size by the index number of this host to get
an offset.
So it take the total number of sectors (resync_max_sectors), divides by the
chunksize (in sectors) to get a number of chunks. This is the number of bits.
Then it should div-round-up by 8 to get a number of bytes.
Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
are always 4K aligned.
Then this number is multiplied by 8 (or shifted by 3) to get a number of
sectors to add to the start of the table.
So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
is wrong because it uses sector_div in a way which destroys
resync_max_sectors.
And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
convert from bits to bytes.
commit f9209a323547f054c7439a3bf67c45e64a054bd
removes the abuse of sector_div, which is good, but uses a simple "a/b"
division, which isn't allowed in the kernel.
commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
then does the right thing with sector_div, but the "<< 3" is still the wrong
way around.
If you still think your code is correct, please explain in detail why.
Goldwyn: if you agree that "<< 3" should be ">> 3" or even
DIV_ROUND_UP_SECTOR_T( , 8);
please send a patch. If you don't think so, please explain why.
Thanks,
NeilBrown
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> This is against for-next branch.
>
> drivers/md/bitmap.c | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index 501f83f..ea9c685 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -571,11 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
> re_read:
> /* If cluster_slot is set, the cluster is setup */
> if (bitmap->cluster_slot >= 0) {
> - sector_t bm_blocks;
> - sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
> + sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
>
> - bm_blocks = sector_div(resync_sectors,
> - bitmap->mddev->bitmap_info.chunksize >> 9);
> + sector_div(bm_blocks,
> + bitmap->mddev->bitmap_info.chunksize >> 9);
> bm_blocks = bm_blocks << 3;
> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Converting a .90 raid superblock to version 1.0 and whether--size parameter is needed
From: Sean Harris @ 2015-03-17 22:49 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
In-Reply-To: <21CFD7CC-7A05-4298-BD77-E1287B9F78D5@backblaze.com>
Hi Neil and company,
I have roughly a hundred machines that are still on Debian 5 that are slated for OS upgrades, and a great many of those are running software raid with the 00.90 version superblock. I even have a few machines on debian 7 with the .90 superblock because the boot drive for the machine had been replaced at some point.
I would like to explore the idea of upgrading/converting the raid superblock on many of those machines to version 1.0, mostly to keep the environment as uniform as possible, and standardize training for my SysAdmins on re-assembling failed raid arrays.
But I am not sure if it is a safe and necessary, or even a worthwhile endeavor.
I wanted to test this on a few non-production machines, but I am not quite sure if I need to specify the --size parameter in the create statement in the attempt.
I have a bit of a mixed environment, but very early on, the machines initialized with the .90 superblock were all created in the same way ie:
/sbin/mdadm --create /dev/md10 --bitmap=internal --auto=yes -l 6 -n 15 /dev/sd{b,e,h,k,n,q,t,w,z,ac,af,ai,al,ao,ar}1
Most of the machines I am interested in upgrading are running Debian 5, with LVM and ext4 volumes.
Some machines use drive partitions, but we stopped using them some time ago and went with the simple notion that if a drive brand was marketed as 1TB, we assumed a size of 1,000,000,000,000 bytes and created the array with that size. That way, any drive manufacturer could be used as a replacement spare without worrying about the variation of their respective drive sizes.
I think it is safe to upgrade the superblock from .90 to 1.0 because from what I have read, it is the same size, and in the same place as the .90 superblock.
But I don't know if having drives with or without partitions would make a difference, or if the conversion would need a specific version of mdadm utilized.
(My hunch is that any layers on top of the raid array are irrelevant to the superblock conversion).
All arrays are raid 6 with the same number of devices.
I think what I need to run is something akin to
/sbin/mdadm --create /dev/md10 -l6 -c64 --metadata=1.0 --assume-clean -n 15 /dev/sd{b,e,h,k,n,q,t,w,z,ac,af,ai,al,ao,ar}1
Where md10 is the array, the raid level is 6, 64k chunk size, assume-clean (for a known good pre-existing array), 15 devices (all specified in order), and a metadata of 1.0.
But I am not sure if I need to insure that --size is calculated and included in the conversion, or if the re-creation of the superblock simply takes care of it.
The reason I am unsure on this point is that with version 1.0 superblock arrays, I always include the --size parameter when re-creating the array with a specified list of devices, which is normally the Used_Dev_Size of the array, divided by 2, (since the value reported by mdadm is 512 byte sectors).
I am including some information from my test pod. In the case below, there are only 9 devices per array. (But all of my production conversions will be with 15 devices).
So here are my questions:
Am I correct in presuming the conversion will work and simply put-in-place an upgraded superblock?
Also, that it is unnecessary to specify --size for the create command (in this case).
And also, that I need not worry about the version of mdadm used for the conversion? (in my case it will almost exclusively be version v2.6.7.2 on Debian 5 , or mdadm - v3.2.5 - 18th May 2012 on Debian 7.)
Other than reading the superblock version after the conversion, is there any other way of verifying the conversion was successful without causing harm?
(my thoughts were to run FULL raid checks and filesystem checks, and some crc checks to some files).
Also, assuming I am on the right track and nothing can go wrong, are there any specific recommendations you have for going ahead with the conversions? Or even a recommendation to avoid this endeavor?
Thanks for your feedback,
Sean Harris
Example from my test machine:
debian_version
5.0.10
proc/version
Linux version 2.6.32-bpo.5-amd64 (Debian 2.6.32-35~bpo50+1) (norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Wed Jul 20 09:10:04 UTC 2011
mdadm -V
mdadm - v2.6.7.2 - 14th November 2008
mdadm -D /dev/md10
/dev/md10:
Version : 00.90
Creation Time : Tue Jul 26 14:59:59 2011
Raid Level : raid6
Array Size : 6768384448 (6454.83 GiB 6930.83 GB)
Used Dev Size : 966912064 (922.12 GiB 990.12 GB)
Raid Devices : 9
Total Devices : 9
Preferred Minor : 10
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Mar 11 15:23:13 2015
State : active
Active Devices : 9
Working Devices : 9
Failed Devices : 0
Spare Devices : 0
Chunk Size : 64K
UUID : fed610e8:0c049d1e:5ab03808:ccff85b7
Events : 0.137116
Number Major Minor RaidDevice State
0 8 161 0 active sync /dev/sdk1
1 65 145 1 active sync /dev/sdz1
2 65 65 2 active sync /dev/sdu1
3 65 241 3 active sync /dev/sdaf1
4 66 65 4 active sync /dev/sdak1
5 8 81 5 active sync /dev/sdf1
6 8 241 6 active sync /dev/sdp1
7 66 145 7 active sync /dev/sdap1
8 8 1 8 active sync /dev/sda1
mdadm -E /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : fed610e8:0c049d1e:5ab03808:ccff85b7
Creation Time : Tue Jul 26 14:59:59 2011
Raid Level : raid6
Used Dev Size : 966912064 (922.12 GiB 990.12 GB)
Array Size : 6768384448 (6454.83 GiB 6930.83 GB)
Raid Devices : 9
Total Devices : 9
Preferred Minor : 10
Update Time : Wed Mar 11 17:24:13 2015
State : clean
Internal Bitmap : present
Active Devices : 9
Working Devices : 9
Failed Devices : 0
Spare Devices : 0
Checksum : b88cefe6 - correct
Events : 137124
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 1 8 active sync /dev/sda1
0 0 8 161 0 active sync /dev/sdk1
1 1 65 145 1 active sync /dev/sdz1
2 2 65 65 2 active sync /dev/sdu1
3 3 65 241 3 active sync /dev/sdaf1
4 4 66 65 4 active sync /dev/sdak1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 241 6 active sync /dev/sdp1
7 7 66 145 7 active sync /dev/sdap1
8 8 8 1 8 active sync /dev/sda1
smartctl -i /dev/sda
smartctl 5.41.patched_20110818 2011-06-09 r3365 [x86_64-linux-2.6.32-bpo.5-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green
Device Model: WDC WD10EACS-65D6B0
Serial Number: WD-WCAU42439281
LU WWN Device Id: 5 0014ee 201f62eb1
Firmware Version: 01.01A01
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Wed Mar 11 17:28:06 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
fdisk -l /dev/sda
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 1 120375 966912156 fd Linux raid autodetect
lvscan
ACTIVE '/dev/vg0070035/lv0070035' [6.30 TB] inherit
ACTIVE '/dev/vg0070034/lv0070034' [6.30 TB] inherit
ACTIVE '/dev/vg0070033/lv0070033' [6.30 TB] inherit
ACTIVE '/dev/vg0070032/lv0070032' [6.30 TB] inherit
ACTIVE '/dev/vg0070031/lv0070031' [6.30 TB] inherit
^ permalink raw reply
* chunk size K, KB, KiB
From: Chris Murphy @ 2015-03-17 17:48 UTC (permalink / raw)
To: linux-raid
> man mdadm:
> -c, --chunk=
> Specify chunk size of kibibytes. The default when creating an array is
> 512KB.
[...snip...]
>A suffix of 'M' or 'G' can be given to indicate Megabytes or Gigabytes respectively.
Is this actually 512KiB as in 1024 512 byte sectors? Or is is 1000 512
byte sectors? I suppose either one is plausible.
SNIA's DDF reference says strip_size (synonym for chunk and stripe depth) is
Table 27 (following page) [1], see the strip_size examples. All of
those values are clearly base 2 values, but they're wrongly labeled
with the base 10 SI prefix.
Therefore I'm pretty sure the mdadm man page is correct saying
"Specify chunk size of kibibytes" and subsequently should instead use
512KiB and 64KiB respectively.
[1]
http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf
--
Chris Murphy
^ permalink raw reply
* Re: [RFC PATCH 2/7] block: convert bio_vec.bv_page to bv_pfn
From: Dan Williams @ 2015-03-17 15:53 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Al Viro, linux-kernel@vger.kernel.org, linux-arch, Jens Axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman,
Christoph Hellwig, linux-fsdevel
In-Reply-To: <20150317130254.GC4003@linux.intel.com>
On Tue, Mar 17, 2015 at 6:02 AM, Matthew Wilcox <willy@linux.intel.com> wrote:
> On Mon, Mar 16, 2015 at 11:05:33PM +0000, Al Viro wrote:
>> > diff --git a/mm/iov_iter.c b/mm/iov_iter.c
>> > index 827732047da1..be9a7c5b8703 100644
>> > --- a/mm/iov_iter.c
>> > +++ b/mm/iov_iter.c
>> > @@ -61,7 +61,7 @@
>> > __p = i->bvec; \
>> > __v.bv_len = min_t(size_t, n, __p->bv_len - skip); \
>> > if (likely(__v.bv_len)) { \
>> > - __v.bv_page = __p->bv_page; \
>> > + __v.bv_pfn = __p->bv_pfn; \
>> > __v.bv_offset = __p->bv_offset + skip; \
>> > (void)(STEP); \
>> > skip += __v.bv_len; \
>> > @@ -72,7 +72,7 @@
>> > __v.bv_len = min_t(size_t, n, __p->bv_len); \
>> > if (unlikely(!__v.bv_len)) \
>> > continue; \
>> > - __v.bv_page = __p->bv_page; \
>> > + __v.bv_pfn = __p->bv_pfn; \
>> > __v.bv_offset = __p->bv_offset; \
>> > (void)(STEP); \
>> > skip = __v.bv_len; \
>> > @@ -369,7 +369,7 @@ size_t copy_to_iter(void *addr, size_t bytes, struct iov_iter *i)
>> > iterate_and_advance(i, bytes, v,
>> > __copy_to_user(v.iov_base, (from += v.iov_len) - v.iov_len,
>> > v.iov_len),
>> > - memcpy_to_page(v.bv_page, v.bv_offset,
>> > + memcpy_to_page(bvec_page(&v), v.bv_offset,
>>
>> How had memcpy_to_page(NULL, ...) worked for you?
>
> static inline struct page *bvec_page(const struct bio_vec *bvec)
> {
> - return bvec->bv_page;
> + return pfn_to_page(bvec->bv_pfn.pfn);
> }
>
> (yes, more work to be done here to make copy_to_iter work to a bvec that
> is actually targetting a page-less address, but these are RFC patches
> showing the direction we're heading in while keeping current code working)
>
Right, the next item to tackle is kmap() and kmap_atomic() before we
can start converting paths to be "native" pfn-only.
^ permalink raw reply
* Re: [RFC PATCH 2/7] block: convert bio_vec.bv_page to bv_pfn
From: Matthew Wilcox @ 2015-03-17 13:02 UTC (permalink / raw)
To: Al Viro
Cc: Dan Williams, linux-kernel, linux-arch, axboe, riel, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel
In-Reply-To: <20150316230533.GH29656@ZenIV.linux.org.uk>
On Mon, Mar 16, 2015 at 11:05:33PM +0000, Al Viro wrote:
> > diff --git a/mm/iov_iter.c b/mm/iov_iter.c
> > index 827732047da1..be9a7c5b8703 100644
> > --- a/mm/iov_iter.c
> > +++ b/mm/iov_iter.c
> > @@ -61,7 +61,7 @@
> > __p = i->bvec; \
> > __v.bv_len = min_t(size_t, n, __p->bv_len - skip); \
> > if (likely(__v.bv_len)) { \
> > - __v.bv_page = __p->bv_page; \
> > + __v.bv_pfn = __p->bv_pfn; \
> > __v.bv_offset = __p->bv_offset + skip; \
> > (void)(STEP); \
> > skip += __v.bv_len; \
> > @@ -72,7 +72,7 @@
> > __v.bv_len = min_t(size_t, n, __p->bv_len); \
> > if (unlikely(!__v.bv_len)) \
> > continue; \
> > - __v.bv_page = __p->bv_page; \
> > + __v.bv_pfn = __p->bv_pfn; \
> > __v.bv_offset = __p->bv_offset; \
> > (void)(STEP); \
> > skip = __v.bv_len; \
> > @@ -369,7 +369,7 @@ size_t copy_to_iter(void *addr, size_t bytes, struct iov_iter *i)
> > iterate_and_advance(i, bytes, v,
> > __copy_to_user(v.iov_base, (from += v.iov_len) - v.iov_len,
> > v.iov_len),
> > - memcpy_to_page(v.bv_page, v.bv_offset,
> > + memcpy_to_page(bvec_page(&v), v.bv_offset,
>
> How had memcpy_to_page(NULL, ...) worked for you?
static inline struct page *bvec_page(const struct bio_vec *bvec)
{
- return bvec->bv_page;
+ return pfn_to_page(bvec->bv_pfn.pfn);
}
(yes, more work to be done here to make copy_to_iter work to a bvec that
is actually targetting a page-less address, but these are RFC patches
showing the direction we're heading in while keeping current code working)
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox