* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 17:22 UTC (permalink / raw)
To: Dan Williams, Andrew Morton
Cc: Boaz Harrosh, linux-arch, Jens Axboe, riel, linux-raid,
linux-nvdimm, Dave Hansen, linux-kernel@vger.kernel.org,
Christoph Hellwig, Mel Gorman, linux-fsdevel
In-Reply-To: <CAPcyv4j=M8V_36C-HhiJM7MHzNLFcpP=nec=LHnob5+qZ4xgYw@mail.gmail.com>
On 03/19/2015 10:59 PM, Dan Williams wrote:
>
> At least for block-i/o it seems the only place we really need struct
> page infrastructure is for kmap(). Given we already need a kmap_pfn()
> solution for option 2 a "dynamic allocation" stop along that
> development path may just naturally fall out.
Really? what about networked block-io, RDMA, FcOE emulated targets,
mmaped pointers. virtual-machine bdev drivers
Block layer sits in the middle of the stack not at the low end as you
make it appear. There are lots of below the bio subsystems that tie into
a page struct, which will now stop to operate, unless you do:
pfn_to_page() which means a page-less pfn will now crash or will need
to be rejected so any where you have a
if (page_less_pfn())
... /* Fail or do some other code like copy */
else
page = pfn_to_page()
Is a double code path in the Kernel and is a nightmare to maintain.
(I'm here for you believe me ;-) )
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 17:06 UTC (permalink / raw)
To: Rik van Riel, Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <550C8C47.5090002@redhat.com>
On 03/20/2015 11:08 PM, Rik van Riel wrote:
> On 03/20/2015 04:31 PM, Matthew Wilcox wrote:
<>
>> There's a lot of code out there that relies on struct page being PAGE_SIZE
>> bytes. I'm cool with replacing 'struct page' with 'struct superpage'
>> [1] in the biovec and auditing all of the code which touches it ... but
>> that's going to be a lot of code! I'm not sure it's less code than
>> going directly to 'just do I/O on PFNs'.
>
> Totally agreed here. I see absolutely no advantage to teaching the
> IO layer about a "struct superpage" when it could operate on PFNs
> just as easily.
>
Or teaching 'struct page' to be variable length, This is already so at
bio and sg level so you fixed nothing.
Moving to pfn's only means that all this unnamed code above that
"relies on struct page being PAGE_SIZE" is now not allowed to
interfaced with bio and sg list. Which in current code and in Dan's patches
means two tons of BUG_ONS and return -ENOTSUPP . For all these
subsystems below the bio and sglist that operate on page_structs
Say the "relies on struct page being PAGE_SIZE" is such an hard
work, which is not at all at the bio and sg-list level, will
it not be worth while fixing this instead of alienating the all
Kernel from the IO subsystem.
But I believe it is the much much smaller change? Specially considering
Networking, RDMA shared memory ...
Cheers
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 16:46 UTC (permalink / raw)
To: Christoph Hellwig, Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
riel, linux-nvdimm, Dave Hansen, linux-raid, mgorman,
linux-fsdevel
In-Reply-To: <20150319181725.GA17411@infradead.org>
On 03/19/2015 08:17 PM, Christoph Hellwig wrote:
<>
>
> In addition to the options there's also a time line. At least for the
> short term where we want to get something going 1a seems like the
> absolutely be option. It works perfectly fine for the lots of small
> capacity dram-like nvdimms, and it works funtionally fine for the
> special huge ones, although the resource use for it is highly annoying.
> If it turns out to be too annoying we can also offer a no I/O possible
> option for them in the short run.
>
Finally some voice in the dessert.
> In the long run option 2) sounds like a good plan to me, but not as a
> parallel I/O path, but as the main one. Doing so will in fact give us
> options to experiment with 3). Given that we're moving towards an
> increasinly huge page using world replacing the good old struct page
> with something extent-like and/or temporary might be needed for dram
> as well in the future.
Why ? why not just make page mean page_size(page) and mostly even that
is not needed.
Any changes to bio will only solve bio. And will push the problem to
the next subsystem.
Fix the PAGE_SIZE problem and you fixed it for all subsystems, not only
bio. And I believe it is the smaller change by far.
Because in most places PAGE_SIZE just means MIN_PAGE_SIZE when we try
calculate some array sizes for storage of a given "io-length", this
is surly 4k, but then when the actual run time is preformed we usually
have a length specifier like bv_len. (And the few places that do not are
easy to fix I believe)
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 16:24 UTC (permalink / raw)
To: Matthew Wilcox, Rik van Riel
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <20150320203136.GM4003@linux.intel.com>
On 03/20/2015 10:31 PM, Matthew Wilcox wrote:
<>
>
> There's a lot of code out there that relies on struct page being PAGE_SIZE
> bytes.
Not so much really. Not at the lower end of the stack. You can actually feed
a
vp = kmalloc(64K);
bv_page = virt_to_page(vp)
bv_len = 64k
And feed that to an hard drive. It works.
The only last stronghold of PAGE_SIZE is at the page-cache and page-fault
granularity where the minimum is the better. But it should not be hard
to clean up the lower end of the stack. Even introduce a:
page_size(page)
You will find that every subsystem that can work with a sub-page size
similar to above bv_len. Will also work well with bigger than PAGE_SIZE
bv_len equivalent.
Only the BUG_ONs need to convert to page_size(page) instead of PAGE_SIZE
> I'm cool with replacing 'struct page' with 'struct superpage'
> [1] in the biovec and auditing all of the code which touches it ... but
> that's going to be a lot of code! I'm not sure it's less code than
> going directly to 'just do I/O on PFNs'.
>
struct page already knows how to be a super-page. with the THP mechanics.
All a page_size(page) needs is a call to its section, we do not need any
added storage at page-struct. (And we can cache this as a flag we actually
already have a flag)
It looks like you are very trigger happy to change
"biovec and auditing all of the code which touches it"
I believe long long term your #1b is the correct "full audit" path:
Page Is the virtual-2-page-2-physical descriptor + state.
It is variable size
> [1] Please, somebody come up with a better name!
sure struct page *page.
The one to kill is PAGE_SIZE. In most current code it can just be MIN_PAGE_SIZE
and CACHE_PAGE_SIZE == MIN_PAGE_SIZE. Only novelty is enhance of the split_huge_page
in the case of "page-fault-granularity".
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 15:51 UTC (permalink / raw)
To: Rik van Riel, Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <550C490E.1080708@redhat.com>
On 03/20/2015 06:21 PM, Rik van Riel wrote:
> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>
>> 1. Construct struct pages for persistent memory
>> 1a. Permanently
>> 1b. While the pages are under I/O
>
> Michael Tsirkin and I have been doing some thinking about what
> it would take to allocate struct pages per 2MB area permanently,
> and allocate additional struct pages for 4kB pages on demand,
> when a 2MB area is broken up into 4kB pages.
>
> This should work for both DRAM and persistent memory.
>
My thoughts as well, this need *not* be a huge evasive change. Is however
a careful surgery in very core code. And lots of sleepless scary nights
and testing to make sure all the side effects are wrinkled out.
BTW: Basic core block code may very well work with:
bv_page, bv_len > PAGE_SIZE bv_offset > PAGE_SIZE.
Meaning bv_page-pfn is contiguous in physical space (and virtual
of course). So much so that there are already rumors that this suppose
to be supported, and there are already out-of-tree drivers that use
this today by kmalloc a page-order and feeding BIOs with bv_len=64K
But going out of block-layer and say to networking say via iscsi and
this breaks pretty fast. Lets fix that then lets introduce a:
page_size(page)
page already knows its size (ie belonging to a 2M THP)
> I am still not convinced it is worthwhile to have struct pages
> for persistent memory though, but I am willing to change my mind.
>
If we want copy-less, we need a common memory descriptor career. Today this
is page-struct. So for me your above statement means:
"still not convinced I care about copy-less pmem"
Otherwise you either enhance what you have today or devise a new
system, which means change the all Kernel.
Lastly: Why does pmem need to wait out-of-tree. Even you say above that
machines with lots of DRAM can enjoy the HUGE-to-4k split. So why
not let pmem waist 4k pages like everyone else and fix it as above
down the line, both for pmem and ram. And save both ways.
Why do we need to first change the all Kernel, then have pmem. Why not
use current infra structure, for good or for worth, and incrementally
do better.
May I call you on the phone to try and work things out. I believe the
huge page thing + 4k on demand is not a very big change, as long as
struct page *page is left as is, everywhere.
But may *now* carry a different physical/virtual contiguous payload
bigger then 4k. Is not the PAGE_SIZE the real bug? lets fix that problem.
Thanks
Boaz
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: Adam Goryachev @ 2015-03-22 12:51 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqherPsEAPNrmYJ7o7DWP90gO4vv-sbmxCmthh_QU8qpcw@mail.gmail.com>
On 22/03/2015 23:29, lingli tang wrote:
> Thanks very much.
> I will try DRBD later
> But I want to figure this out.
>
> I have export disk using tgtd and load disk on another server using
> iscsiadm with infiniband of iser protocol.
> Does ISCSI/Iser have any cache on it.
Can you test that by removing the local disk from the MD array, or
changing your test so writes are directly to the remote device. Then run
the test, shutdown, and check the remote disk to see if it has all the
expected data, or still only some of the expected data. This will remove
MD as a suspect. Continue to try and get "closer" to the remote until
you can find the culprit. You might also use tcpdump or similar to sniff
the network, which will tell you if the expected data is being sent to
the remote (and when).
Sorry, I don't know anywhere near enough to comment on things like
infiniband/iser, but these are the steps I would look into. Hope that it
is helpful.
PS, I do use DRBD, and iSCSI, and it has been working well in my
environment for the last year or so, I have no commercial
interest/benefit from you using it, just a happy customer.
Regards,
Adam
>
> 2015-03-22 15:28 GMT+08:00 Adam Goryachev <mailinglists@websitemanagers.com.au>:
>>
>> On 22/03/2015 16:00, lingli tang wrote:
>>> Thanks for reply.
>>>
>>> I have create a raid1 with two fusion io PCIe flash disk:
>>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>>> /dev/fioa2 /dev/mapper/mpathc
>>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>>> load disk export from server B.
>>>
>>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on
>>> /data1
>>> and we will run mysql binlog on it.
>>> In order to avoid data loss of mysql binlog we have set
>>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>>
>>> according to your description. if we reboot the server A, the two disk
>>> data on different server will be the same.
>>> but after the server A restarted, we assemble the two disk on two
>>> server, data is different on the two server, disk on server B lost
>>> more than one sql commit.
>>>
>>> I have checked it with strace 'mysqld' on Server A.
>>> I found a sql commit and fsync() on binlog file handle on server A but
>>> this sql can not find in assembled disk on server B.
>>>
>>> I also test it with two SAS disk, Server B still has more than one sql
>>> commit lost.
>> Sounds like you might be better using something like DRBD (www.drbd.org)
>> which has different modes, one of which will do what you are asking (not
>> respond until both systems have confirmed the data is written to the local
>> disk).
>>
>> In your current case, even if md is correctly writing to both underlying
>> 'devices' you have multiple layers under one of the devices, so you should
>> confirm that *all* of those layers are properly passing through the data
>> without any caching/etc.
>>
>> Regards,
>> Adam
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-22 12:29 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <550E6F02.3090800@websitemanagers.com.au>
Thanks very much.
I will try DRBD later
But I want to figure this out.
I have export disk using tgtd and load disk on another server using
iscsiadm with infiniband of iser protocol.
Does ISCSI/Iser have any cache on it.
2015-03-22 15:28 GMT+08:00 Adam Goryachev <mailinglists@websitemanagers.com.au>:
>
>
> On 22/03/2015 16:00, lingli tang wrote:
>>
>> Thanks for reply.
>>
>> I have create a raid1 with two fusion io PCIe flash disk:
>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>> /dev/fioa2 /dev/mapper/mpathc
>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>> load disk export from server B.
>>
>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on
>> /data1
>> and we will run mysql binlog on it.
>> In order to avoid data loss of mysql binlog we have set
>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>
>> according to your description. if we reboot the server A, the two disk
>> data on different server will be the same.
>> but after the server A restarted, we assemble the two disk on two
>> server, data is different on the two server, disk on server B lost
>> more than one sql commit.
>>
>> I have checked it with strace 'mysqld' on Server A.
>> I found a sql commit and fsync() on binlog file handle on server A but
>> this sql can not find in assembled disk on server B.
>>
>> I also test it with two SAS disk, Server B still has more than one sql
>> commit lost.
>
> Sounds like you might be better using something like DRBD (www.drbd.org)
> which has different modes, one of which will do what you are asking (not
> respond until both systems have confirmed the data is written to the local
> disk).
>
> In your current case, even if md is correctly writing to both underlying
> 'devices' you have multiple layers under one of the devices, so you should
> confirm that *all* of those layers are properly passing through the data
> without any caching/etc.
>
> Regards,
> Adam
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 11:53 UTC (permalink / raw)
To: Rik van Riel, Matthew Wilcox, Boaz Harrosh
Cc: axboe, linux-arch, linux-raid, linux-nvdimm, Dave Hansen,
linux-kernel, hch, Linus Torvalds, Al Viro, linux-fsdevel,
Andrew Morton, mgorman
In-Reply-To: <550C4318.8010200@redhat.com>
On 03/20/2015 05:56 PM, Rik van Riel wrote:
> On 03/18/2015 10:38 AM, Boaz Harrosh wrote:
>> On 03/18/2015 03:06 PM, Matthew Wilcox wrote:
>
>>>> I'm not the one afraid of hard work, if it was for a good cause, but for what?
>>>> really for what? The block layer, and RDMA, and networking, and spline, and what
>>>> ever the heck any one wants to imagine to do with pmem, already works perfectly
>>>> stable. right now!
>>>
>>> The overhead. Allocating a struct page for every 4k page in a 400GB DIMM
>>> (the current capacity available from one NV-DIMM vendor) occupies 6.4GB.
>>> That's an unacceptable amount of overhead.
>>>
>>
>> So lets fix the stacks to work nice with 2M pages. That said we can
>> allocate the struct page also from pmem if we need to. The fact remains
>> that we need state down the different stacks and this is the current
>> design over all.
>
> Fixing the stack to work with 2M pages will be just as invasive,
> and just as much work as making it work without a struct page.
>
> What state do you need, exactly?
>
It is not me that needs state it is the Kernel. Let me show you
what I can do now that uses state (and pages).
block layer sends a bio via iscsi, in turn it goes around and
sends it via networking stack. Here page-ref is used as well
as all kind of page based management. (This is half the Kernel
converted right here)
Same thing but iser & RDMA. Same thing to a null-target, via
the target stack, maybe via path-threw.
Another big example:
At user-mode application I mmap a portion of pmem, I then
use the libvirt API to designate a named shared-memory object.
At vm I use the same API to retrieve a pointer to that pmem
region and boom, I'm persistent. (Same can be done between
two VMs)
mmap(pmem) send it to network, to encryption, direct_io
RDMA, anything copyless.
So many subsystem use page_lock page->lru page-ref and are
written to receive and manage pages. I do not like to be
excluded from these systems, and I would very much hate
to re-write them. block layer is an example.
> The struct page in the VM is mostly used for two things:
> 1) to get a memory address of the data
> 2) refcounting, to make sure the page does not go away
> during an IO operation, copy, etc...
>
> Persistent memory cannot be paged out so (2) is not a concern, as
> long as we ensure the object the page belongs to does not go away.
> There are no seek times, so moving it around may not be necessary
> either, making (1) not a concern.
>
I lost you sorry. I'm not sure what you meant here?
Yes kmap/kunmap is mute. I do not see any use for highmem and
any 32bitness with this thing.
refcounting is used sure, even with pmem see above. Actually
relaying on refcounting existence can solve us some stuff at
the pmem management level, which exist today. (RDMA while truncate)
> The only case where (1) would be a concern is if we wanted to move
> data in persistent memory around for better NUMA locality. However,
> persistent memory DIMMs are on their way to being too large to move
> the memory, anyway - all we can usefully do is detect where programs
> are accessing memory, and move the programs there.
>
So actually I have hands on experience with this very problem.
We have observed that NUMA kills us. Now going through memory_add_physaddr_to_nid()
loop for every 4k operation was a pain, but caching it on page_to_nid()
(As part of flags in 64bit) is very nice optimization, we do NUMA aware block
allocation and it preforms much better. (Never like a single node but magnitude
better then without)
> What state do you need that is not already represented?
>
Most of these subsystem you guys are focused on it is mostly read-only
state. Except page-ref. But never the less the page has added information
describing the pfn. Like nid mapping->ops flags etc ...
And it is also a stop gap of translation.
give me a page I now the pfn and vaddr, give me a pfn I know page
give me a vaddr I know the page. So I can move between all these domains.
Now I am sure that in hindsight we might have devised better structures
and abstractions that could carry all this information in a more abstract
and convenient way, throughout the Kernel. But for now this basic object
is a page and is passed around like in a relay-race. Each subsystem with
its own page based meta-structure. The only real global token is
page-struct.
You are saying: "not already represented" ? I'm saying exactly, sir
it is already represented as a page-struct. Anything else is in the
far far future. (if at all)
> 1.5% overhead isn't a whole lot, but it appears to be unnecessary.
>
unnecessary, in a theoretical future with every single Kernel
subsystem changed (maybe for the better I'm not saying). And this
future is not even at all clear what it is.
But for current code structure it is very much necessary. For the
very long present days, it is not 1.5% with or without. It is
need-to-copy or direct(-1.5%)
[For me it is not even the performance of a memcpy which exacly halves
my pmem performance, it is the latency and the extra nightmare locking
and management to keep in sync two copies of the same thing]
> If you have a convincing argument as to why we need a struct page,
> you might want to articulate it in order to convince us.
>
The must simple convincing argument there is. "Existing code". Apparently
page was needed, maybe we can all think of much better constructs. But
for now this is what the Kernel is based on. Until such time that we
better it it is there.
Since when we refrain from new technologies and new fixtures because
"A major cleanup is needed". I'm all for all the great
"change-every-file in Kernel" ideas some guys have, but while at it
also change the small patch I added to support pmem.
For me pmem is now, at clients systems. and I chose direct(-1.5%)
over need-to-copy. Because it gives me the performance, and most
important, latency that sales my products. What is your timetable?
Cheers
Boaz
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-22 11:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150322163809.407c6ddb@notabene.brown>
Yes, I just issue 'reboot' on Server A.
But I am curious about why 'some' request will lost to other server.
Is It should be only one request lost(the last IO committed )
according to full sync strategy.
2015-03-22 13:38 GMT+08:00 NeilBrown <neilb@suse.de>:
> On Sun, 22 Mar 2015 13:00:54 +0800 lingli tang <tanglingli001@gmail.com>
> wrote:
>
>> Thanks for reply.
>>
>> I have create a raid1 with two fusion io PCIe flash disk:
>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>> /dev/fioa2 /dev/mapper/mpathc
>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>> load disk export from server B.
>>
>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
>> and we will run mysql binlog on it.
>> In order to avoid data loss of mysql binlog we have set
>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>
>> according to your description. if we reboot the server A, the two disk
>> data on different server will be the same.
>> but after the server A restarted, we assemble the two disk on two
>> server, data is different on the two server, disk on server B lost
>> more than one sql commit.
>
> What exactly do you mean by "reboot"??
> Is this a clean shutdown or do you remove the power or something like that.
>
> If you remove the power, then it is very possible that some requests will
> have been submitted to one device but not the other.
> If you have a clean shutdown, then the two devices should be identical.
>
> NeilBrown
>
>
>>
>> I have checked it with strace 'mysqld' on Server A.
>> I found a sql commit and fsync() on binlog file handle on server A but
>> this sql can not find in assembled disk on server B.
>>
>> I also test it with two SAS disk, Server B still has more than one sql
>> commit lost.
>>
>>
>> 2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
>> > On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
>> > wrote:
>> >
>> >> I am a newbie of mdadm. I have a question but find no answer in
>> >> document or google for last 10 days.
>> >>
>> >> The question is : RAID1 made by mdadm is full sync? for example, I
>> >> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
>> >> commit an IO to the RAID1 disk (md127), it will return back to me when
>> >> all the two disk commit successfully or it will return back
>> >> to me once just one of the disk successfully commit.
>> >
>> > The write request will not return until it has been submitted to all, and
>> > returned by, all working devices.
>> >
>> >>
>> >> I have test with xfs and ext4 with sync option, and it seems that two
>> >> disk have lots of commit difference after reboot the server. is that
>> >> means mdadm return success when one of the disk is commit
>> >> successfully?
>> >
>> > That certainly shouldn't happen. I would need more details of the experiment
>> > that you performed.
>> >
>> > NeilBrown
>> >
>> >
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >
>
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 10:30 UTC (permalink / raw)
To: Andrew Morton
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <20150319125917.6cc2bf02687aab542027d8ac@linux-foundation.org>
On 03/19/2015 09:59 PM, Andrew Morton wrote:
> On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
>
>> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
>> <>
>>>
>>> Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
>>> want to be able to do any kind of I/O directly to persistent memory,
>>> and I think we do, we need to do one of:
>>>
>>> 1. Construct struct pages for persistent memory
>>> 1a. Permanently
>>> 1b. While the pages are under I/O
>>> 2. Teach the I/O layers to deal in PFNs instead of struct pages
>>> 3. Replace struct page with some other structure that can represent both
>>> DRAM and PMEM
>>>
>>> I'm personally a fan of #3, and I was looking at the scatterlist as
>>> my preferred data structure. I now believe the scatterlist as it is
>>> currently defined isn't sufficient, so we probably end up needing a new
>>> data structure. I think Dan's preferred method of replacing struct
>>> pages with PFNs is actually less instrusive, but doesn't give us as
>>> much advantage (an entirely new data structure would let us move to an
>>> extent based system at the same time, instead of sticking with an array
>>> of pages). Clearly Boaz prefers 1a, which works well enough for the
>>> 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>>>
>>> What's your preference? I guess option 0 is "force all I/O to go
>>> through the page cache and then get copied", but that feels like a nasty
>>> performance hit.
>>
>> Thanks Matthew, you have summarized it perfectly.
>>
>> I think #1b might have merit, as well.
>
> It would be interesting to see what a 1b implementation looks like and
> how it performs. We already allocate a bunch of temporary things to
> support in-flight IO (bio, request) and allocating pageframes on the
> same basis seems a fairly logical fit.
There is a couple of ways we can do this, they are all kind of
"hacks" to me, along the line of how transparent huge pages is an
hack, a very nice one at that, and every one that knows me knows
I love hacks, be so it is never the less.
So it is all about designating the page to mean something else
at a set of a flag.
And actually the transparent-huge-pages is the core of this.
because there is already a switch on core page operations when
it is present. (for example get/put_page )
And because we do not want to allocate pages inline, as part of a
section, we also need a bit of a memory_model.h new define.
(May this can avoided I need to stare harder on this)
>
> It is all a bit of a stopgap, designed to shoehorn
> direct-io-to-dax-mapped-memory into the existing world. Longer term
> I'd expect us to move to something more powerful, but it's unclear what
> that will be at this time, so a stopgap isn't too bad?
>
I'd bet real huge-pages is the long term. The one stop gap for
huge-pages is that no one wants to dirty a full 2M for two changed
bytes. 4k is kind of the IO performance granularity we all calculate
for. This can be solved in couple of ways, all very invasive to lots
of Kernel areas.
Lots of times the problem is "where do you start?"
>
> This is all contingent upon the prevalence of machines which have vast
> amounts of nv memory and relatively small amounts of regular memory.
> How confident are we that this really is the future?
>
One thing you guys are ignoring is that the 1.5% "waste" can come
from nv-memory. If real ram is scarce and nv-ram is hips cheep,
just allocate the pages from nvram then.
Do not forget that very soon after the availability of real
nvram, I mean not the backed up one, but the real like mram
or reram. Lots of machines will be 100% nv-ram + sram caches.
This is nothing to do with storage speed, it is to do with
power consumption. The machine shuts-off and picks up exactly
where it was. (Even at power on they consume much less, no refreshes)
In those machine a partition of storage say, the swap partition, will
be the volatile memory section of the machine, zeroed out on boot and
used as RAM.
So this future above does not exist. Pages can just be allocated
from the cheapest memory you have and be done with it.
(BTW all this can already be done now, I have demonstrated it
in the lab, a reserved NvDIMM memory region is memory_hot_plugged
and is there after used as regular RAM)
Thanks
Boaz
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: Adam Goryachev @ 2015-03-22 7:28 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqg2wp4VMCA8O0b2rUfbGDGXZY=mFvj1hA=tWaVkdCMXtA@mail.gmail.com>
On 22/03/2015 16:00, lingli tang wrote:
> Thanks for reply.
>
> I have create a raid1 with two fusion io PCIe flash disk:
> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
> /dev/fioa2 /dev/mapper/mpathc
> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
> load disk export from server B.
>
> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
> and we will run mysql binlog on it.
> In order to avoid data loss of mysql binlog we have set
> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>
> according to your description. if we reboot the server A, the two disk
> data on different server will be the same.
> but after the server A restarted, we assemble the two disk on two
> server, data is different on the two server, disk on server B lost
> more than one sql commit.
>
> I have checked it with strace 'mysqld' on Server A.
> I found a sql commit and fsync() on binlog file handle on server A but
> this sql can not find in assembled disk on server B.
>
> I also test it with two SAS disk, Server B still has more than one sql
> commit lost.
Sounds like you might be better using something like DRBD (www.drbd.org)
which has different modes, one of which will do what you are asking (not
respond until both systems have confirmed the data is written to the
local disk).
In your current case, even if md is correctly writing to both underlying
'devices' you have multiple layers under one of the devices, so you
should confirm that *all* of those layers are properly passing through
the data without any caching/etc.
Regards,
Adam
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: NeilBrown @ 2015-03-22 5:38 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqg2wp4VMCA8O0b2rUfbGDGXZY=mFvj1hA=tWaVkdCMXtA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]
On Sun, 22 Mar 2015 13:00:54 +0800 lingli tang <tanglingli001@gmail.com>
wrote:
> Thanks for reply.
>
> I have create a raid1 with two fusion io PCIe flash disk:
> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
> /dev/fioa2 /dev/mapper/mpathc
> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
> load disk export from server B.
>
> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
> and we will run mysql binlog on it.
> In order to avoid data loss of mysql binlog we have set
> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>
> according to your description. if we reboot the server A, the two disk
> data on different server will be the same.
> but after the server A restarted, we assemble the two disk on two
> server, data is different on the two server, disk on server B lost
> more than one sql commit.
What exactly do you mean by "reboot"??
Is this a clean shutdown or do you remove the power or something like that.
If you remove the power, then it is very possible that some requests will
have been submitted to one device but not the other.
If you have a clean shutdown, then the two devices should be identical.
NeilBrown
>
> I have checked it with strace 'mysqld' on Server A.
> I found a sql commit and fsync() on binlog file handle on server A but
> this sql can not find in assembled disk on server B.
>
> I also test it with two SAS disk, Server B still has more than one sql
> commit lost.
>
>
> 2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
> > On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
> > wrote:
> >
> >> I am a newbie of mdadm. I have a question but find no answer in
> >> document or google for last 10 days.
> >>
> >> The question is : RAID1 made by mdadm is full sync? for example, I
> >> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
> >> commit an IO to the RAID1 disk (md127), it will return back to me when
> >> all the two disk commit successfully or it will return back
> >> to me once just one of the disk successfully commit.
> >
> > The write request will not return until it has been submitted to all, and
> > returned by, all working devices.
> >
> >>
> >> I have test with xfs and ext4 with sync option, and it seems that two
> >> disk have lots of commit difference after reboot the server. is that
> >> means mdadm return success when one of the disk is commit
> >> successfully?
> >
> > That certainly shouldn't happen. I would need more details of the experiment
> > that you performed.
> >
> > NeilBrown
> >
> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-22 5:00 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150322142033.0af10b1a@notabene.brown>
Thanks for reply.
I have create a raid1 with two fusion io PCIe flash disk:
mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
/dev/fioa2 /dev/mapper/mpathc
/dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
load disk export from server B.
After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
and we will run mysql binlog on it.
In order to avoid data loss of mysql binlog we have set
sync_binlog=1. so every sql commit will call fsync() to flush to disk.
according to your description. if we reboot the server A, the two disk
data on different server will be the same.
but after the server A restarted, we assemble the two disk on two
server, data is different on the two server, disk on server B lost
more than one sql commit.
I have checked it with strace 'mysqld' on Server A.
I found a sql commit and fsync() on binlog file handle on server A but
this sql can not find in assembled disk on server B.
I also test it with two SAS disk, Server B still has more than one sql
commit lost.
2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
> On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
> wrote:
>
>> I am a newbie of mdadm. I have a question but find no answer in
>> document or google for last 10 days.
>>
>> The question is : RAID1 made by mdadm is full sync? for example, I
>> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
>> commit an IO to the RAID1 disk (md127), it will return back to me when
>> all the two disk commit successfully or it will return back
>> to me once just one of the disk successfully commit.
>
> The write request will not return until it has been submitted to all, and
> returned by, all working devices.
>
>>
>> I have test with xfs and ext4 with sync option, and it seems that two
>> disk have lots of commit difference after reboot the server. is that
>> means mdadm return success when one of the disk is commit
>> successfully?
>
> That certainly shouldn't happen. I would need more details of the experiment
> that you performed.
>
> NeilBrown
>
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: NeilBrown @ 2015-03-22 3:20 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqhy0Bao1hAn3R-KtXWOjYb8O2LDo+E1jNdktWdduUL4vw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]
On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
wrote:
> I am a newbie of mdadm. I have a question but find no answer in
> document or google for last 10 days.
>
> The question is : RAID1 made by mdadm is full sync? for example, I
> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
> commit an IO to the RAID1 disk (md127), it will return back to me when
> all the two disk commit successfully or it will return back
> to me once just one of the disk successfully commit.
The write request will not return until it has been submitted to all, and
returned by, all working devices.
>
> I have test with xfs and ext4 with sync option, and it seems that two
> disk have lots of commit difference after reboot the server. is that
> means mdadm return success when one of the disk is commit
> successfully?
That certainly shouldn't happen. I would need more details of the experiment
that you performed.
NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-21 11:01 UTC (permalink / raw)
To: linux-raid
I am a newbie of mdadm. I have a question but find no answer in
document or google for last 10 days.
The question is : RAID1 made by mdadm is full sync? for example, I
have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
commit an IO to the RAID1 disk (md127), it will return back to me when
all the two disk commit successfully or it will return back
to me once just one of the disk successfully commit.
I have test with xfs and ext4 with sync option, and it seems that two
disk have lots of commit difference after reboot the server. is that
means mdadm return success when one of the disk is commit
successfully?
^ permalink raw reply
* Why people recommend to disable NCQ for MD RAID ?
From: Alireza Haghdoost @ 2015-03-20 23:52 UTC (permalink / raw)
To: Linux RAID
I have read multiple blog post that people recommend to disable
NCQ/TCQ features of individual drives that is in the RAID group [1] in
a hope to improve RAID speed. I guess there should be some observation
in the field that results such a conclusion. However, this idea does
not make sense in theory. While drives can handle multiple outstanding
scsi commands at the same time, why don't we take advantage of this to
increase the disk access throughput ?
Thanks
Alireza
[1] http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html
^ permalink raw reply
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: NeilBrown @ 2015-03-20 22:37 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: jgq516, rgoldwyn, linux-raid
In-Reply-To: <550A476D.4060707@suse.com>
[-- Attachment #1: Type: text/plain, Size: 4639 bytes --]
On Thu, 19 Mar 2015 11:50:05 +0800 Guoqing Jiang <GQJiang@suse.com> wrote:
> Hi Neil,
>
> NeilBrown wrote:
> > On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
> >
> >
> >> From: Guoqing Jiang <gqjiang@suse.com>
> >>
> >> The bm_blocks is modified by commit fe60ce (md/bitmap: use
> >> sector_div for sector_t divisions), but it makes bm_blocks
> >> has different value which is changed from like "a/b" to "a%b",
> >> need to correct this to make sure cluster-md still works.
> >>
> >
> > One of us is confused here.
> >
> > This code is trying to find the start of the bitmap relevant to this host in
> > a table of multiple bitmaps. So it first needs to find out the size of each
> > bitmap. It then multiples the size by the index number of this host to get
> > an offset.
> >
> >
> Thanks for detailed description, it really helps. I quoted related lines
> from bitmap.c.
>
> 574 sector_t bm_blocks;
> 575 sector_t resync_sectors =
> bitmap->mddev->resync_max_sectors;
> 576
> 577 bm_blocks = sector_div(resync_sectors,
> 578
> bitmap->mddev->bitmap_info.chunksize >> 9);
> 579 bm_blocks = bm_blocks << 3;
> 580 bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> 581 bitmap->mddev->bitmap_info.offset +=
> bitmap->cluster_slot * (bm_blocks << 3);
>
> > So it take the total number of sectors (resync_max_sectors), divides by the
> > chunksize (in sectors) to get a number of chunks. This is the number of bits.
> >
> >
> L577 is supposed to do above job.
> > Then it should div-round-up by 8 to get a number of bytes.
> >
> I guess what you mean is about L579, while it used "<<3" rather than
> ">>3" now.
> > Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
> > are always 4K aligned.
> >
> L580 did the job.
> > Then this number is multiplied by 8 (or shifted by 3) to get a number of
> > sectors to add to the start of the table.
> >
> L581 is for this, right? Is the shifted by 3 is to match the bitmap
> format for each
> nodes? Seems the relationship between slot and the bitmap region of the node
> is like n <-----> [8*nK, 8*(n+1)K]. How about the following changes?
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index 501f83f..b2a241b 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -571,12 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
> re_read:
> /* If cluster_slot is set, the cluster is setup */
> if (bitmap->cluster_slot >= 0) {
> - sector_t bm_blocks;
> - sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
> + sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
>
> - bm_blocks = sector_div(resync_sectors,
> -
> bitmap->mddev->bitmap_info.chunksize >> 9);
> - bm_blocks = bm_blocks << 3;
> + sector_div(bm_blocks,
> bitmap->mddev->bitmap_info.chunksize >> 9);
Yes, of course. sector_div returns the remainder doesn't it!
I was thinking that it returned the quotient and set the first arg to the
remainder - and wonder why you wanted the remainder :-(
I've updated the patch to do the right thing and credited you.
Thanks.
> + bm_blocks = bm_blocks >> 3;
> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> > So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
> > is wrong because it uses sector_div in a way which destroys
> > resync_max_sectors.
> > And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
> > convert from bits to bytes.
> >
> > commit f9209a323547f054c7439a3bf67c45e64a054bd
> > removes the abuse of sector_div, which is good, but uses a simple "a/b"
> > division, which isn't allowed in the kernel.
> >
> > commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
> > then does the right thing with sector_div, but the "<< 3" is still the wrong
> > way around.
> >
> > If you still think your code is correct, please explain in detail why.
> >
> > Goldwyn: if you agree that "<< 3" should be ">> 3" or even
> > DIV_ROUND_UP_SECTOR_T( , 8);
> > please send a patch. If you don't think so, please explain why.
> >
> >
> But anyway, it is better wait for Goldwyn's back from vacation, :)
I'll leave the other change (<<3 or >>8) until then.
thanks,
NeilBrown
>
> Thanks,
> Guoqing
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Wols Lists @ 2015-03-20 21:17 UTC (permalink / raw)
To: Matthew Wilcox, Rik van Riel
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <20150320203136.GM4003@linux.intel.com>
On 20/03/15 20:31, Matthew Wilcox wrote:
> Ah! I've looked at that a couple of times as well. I asked our database
> performance team what impact freeing up the memmap would have on their
> performance. They told me that doubling the amount of memory generally
> resulted in approximately a 40% performance improvement. So freeing up
> 1.5% additional memory would result in about 0.6% performance improvement,
> which I thought was probably too small a return on investment to justify
> turning memmap into a two-level data structure.
Don't get me started on databases! This is very much a relational
problem, other databases don't suffer like this.
(imho relational theory is totally inappropriate for an engineering
problem, like designing a database engine ...)
Cheers,
Wol
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 21:08 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <20150320203136.GM4003@linux.intel.com>
On 03/20/2015 04:31 PM, Matthew Wilcox wrote:
> On Fri, Mar 20, 2015 at 12:21:34PM -0400, Rik van Riel wrote:
>> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>>
>>> 1. Construct struct pages for persistent memory
>>> 1a. Permanently
>>> 1b. While the pages are under I/O
>>
>> Michael Tsirkin and I have been doing some thinking about what
>> it would take to allocate struct pages per 2MB area permanently,
>> and allocate additional struct pages for 4kB pages on demand,
>> when a 2MB area is broken up into 4kB pages.
>
> Ah! I've looked at that a couple of times as well. I asked our database
> performance team what impact freeing up the memmap would have on their
> performance. They told me that doubling the amount of memory generally
> resulted in approximately a 40% performance improvement. So freeing up
> 1.5% additional memory would result in about 0.6% performance improvement,
> which I thought was probably too small a return on investment to justify
> turning memmap into a two-level data structure.
Agreed, it should not be done for memory savings alone, but only
if it helps improve all kinds of other things.
>> This should work for both DRAM and persistent memory.
>>
>> I am still not convinced it is worthwhile to have struct pages
>> for persistent memory though, but I am willing to change my mind.
>
> There's a lot of code out there that relies on struct page being PAGE_SIZE
> bytes. I'm cool with replacing 'struct page' with 'struct superpage'
> [1] in the biovec and auditing all of the code which touches it ... but
> that's going to be a lot of code! I'm not sure it's less code than
> going directly to 'just do I/O on PFNs'.
Totally agreed here. I see absolutely no advantage to teaching the
IO layer about a "struct superpage" when it could operate on PFNs
just as easily.
--
All rights reversed
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Matthew Wilcox @ 2015-03-20 20:31 UTC (permalink / raw)
To: Rik van Riel
Cc: Andrew Morton, Dan Williams, linux-kernel, linux-arch, axboe,
linux-nvdimm, Dave Hansen, linux-raid, mgorman, hch,
linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <550C490E.1080708@redhat.com>
On Fri, Mar 20, 2015 at 12:21:34PM -0400, Rik van Riel wrote:
> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
>
> Michael Tsirkin and I have been doing some thinking about what
> it would take to allocate struct pages per 2MB area permanently,
> and allocate additional struct pages for 4kB pages on demand,
> when a 2MB area is broken up into 4kB pages.
Ah! I've looked at that a couple of times as well. I asked our database
performance team what impact freeing up the memmap would have on their
performance. They told me that doubling the amount of memory generally
resulted in approximately a 40% performance improvement. So freeing up
1.5% additional memory would result in about 0.6% performance improvement,
which I thought was probably too small a return on investment to justify
turning memmap into a two-level data structure.
Persistent memory might change that calculation somewhat ... but I'm
not convinced. Certainly, if we already had the ability to allocate
'struct superpage', I wouldn't be pushing for page-less I/Os, I'd just
allocate these data structures for PM. Even if they were 128 bytes in
size, that's only a 25MB overhead per 400GB NV-DIMM, which feels quite
reasonable to me.
> This should work for both DRAM and persistent memory.
>
> I am still not convinced it is worthwhile to have struct pages
> for persistent memory though, but I am willing to change my mind.
There's a lot of code out there that relies on struct page being PAGE_SIZE
bytes. I'm cool with replacing 'struct page' with 'struct superpage'
[1] in the biovec and auditing all of the code which touches it ... but
that's going to be a lot of code! I'm not sure it's less code than
going directly to 'just do I/O on PFNs'.
[1] Please, somebody come up with a better name!
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Wols Lists @ 2015-03-20 17:32 UTC (permalink / raw)
To: Andrew Morton, Boaz Harrosh
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <20150319125917.6cc2bf02687aab542027d8ac@linux-foundation.org>
On 19/03/15 19:59, Andrew Morton wrote:
> This is all contingent upon the prevalence of machines which have vast
> amounts of nv memory and relatively small amounts of regular memory.
> How confident are we that this really is the future?
Somewhat off-topic, but it's also the past. I can't help thinking of the
early Pick machines, which treated backing store as one giant permanent
virtual memory. Back when 300Mb hard drives were HUGE.
Cheers,
Wol
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 16:21 UTC (permalink / raw)
To: Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <20150319134313.GF4003@linux.intel.com>
On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
> 1. Construct struct pages for persistent memory
> 1a. Permanently
> 1b. While the pages are under I/O
Michael Tsirkin and I have been doing some thinking about what
it would take to allocate struct pages per 2MB area permanently,
and allocate additional struct pages for 4kB pages on demand,
when a 2MB area is broken up into 4kB pages.
This should work for both DRAM and persistent memory.
I am still not convinced it is worthwhile to have struct pages
for persistent memory though, but I am willing to change my mind.
--
All rights reversed
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-20 15:56 UTC (permalink / raw)
To: Boaz Harrosh, Matthew Wilcox, Boaz Harrosh
Cc: axboe, linux-arch, linux-raid, linux-nvdimm, Dave Hansen,
linux-kernel, hch, Linus Torvalds, Al Viro, linux-fsdevel,
Andrew Morton, mgorman
In-Reply-To: <55098DFE.8080502@plexistor.com>
On 03/18/2015 10:38 AM, Boaz Harrosh wrote:
> On 03/18/2015 03:06 PM, Matthew Wilcox wrote:
>>> I'm not the one afraid of hard work, if it was for a good cause, but for what?
>>> really for what? The block layer, and RDMA, and networking, and spline, and what
>>> ever the heck any one wants to imagine to do with pmem, already works perfectly
>>> stable. right now!
>>
>> The overhead. Allocating a struct page for every 4k page in a 400GB DIMM
>> (the current capacity available from one NV-DIMM vendor) occupies 6.4GB.
>> That's an unacceptable amount of overhead.
>>
>
> So lets fix the stacks to work nice with 2M pages. That said we can
> allocate the struct page also from pmem if we need to. The fact remains
> that we need state down the different stacks and this is the current
> design over all.
Fixing the stack to work with 2M pages will be just as invasive,
and just as much work as making it work without a struct page.
What state do you need, exactly?
The struct page in the VM is mostly used for two things:
1) to get a memory address of the data
2) refcounting, to make sure the page does not go away
during an IO operation, copy, etc...
Persistent memory cannot be paged out so (2) is not a concern, as
long as we ensure the object the page belongs to does not go away.
There are no seek times, so moving it around may not be necessary
either, making (1) not a concern.
The only case where (1) would be a concern is if we wanted to move
data in persistent memory around for better NUMA locality. However,
persistent memory DIMMs are on their way to being too large to move
the memory, anyway - all we can usefully do is detect where programs
are accessing memory, and move the programs there.
What state do you need that is not already represented?
1.5% overhead isn't a whole lot, but it appears to be unnecessary.
If you have a convincing argument as to why we need a struct page,
you might want to articulate it in order to convince us.
--
All rights reversed
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Dan Williams @ 2015-03-19 20:59 UTC (permalink / raw)
To: Andrew Morton
Cc: Boaz Harrosh, linux-arch, Jens Axboe, riel, linux-raid,
linux-nvdimm, Dave Hansen, linux-kernel@vger.kernel.org,
Christoph Hellwig, Mel Gorman, linux-fsdevel
In-Reply-To: <20150319125917.6cc2bf02687aab542027d8ac@linux-foundation.org>
On Thu, Mar 19, 2015 at 12:59 PM, Andrew Morton
<akpm@linux-foundation.org> wrote:
> On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
>
>> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
>> <>
>> >
>> > Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
>> > want to be able to do any kind of I/O directly to persistent memory,
>> > and I think we do, we need to do one of:
>> >
>> > 1. Construct struct pages for persistent memory
>> > 1a. Permanently
>> > 1b. While the pages are under I/O
>> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
>> > 3. Replace struct page with some other structure that can represent both
>> > DRAM and PMEM
>> >
>> > I'm personally a fan of #3, and I was looking at the scatterlist as
>> > my preferred data structure. I now believe the scatterlist as it is
>> > currently defined isn't sufficient, so we probably end up needing a new
>> > data structure. I think Dan's preferred method of replacing struct
>> > pages with PFNs is actually less instrusive, but doesn't give us as
>> > much advantage (an entirely new data structure would let us move to an
>> > extent based system at the same time, instead of sticking with an array
>> > of pages). Clearly Boaz prefers 1a, which works well enough for the
>> > 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
>> >
>> > What's your preference? I guess option 0 is "force all I/O to go
>> > through the page cache and then get copied", but that feels like a nasty
>> > performance hit.
>>
>> Thanks Matthew, you have summarized it perfectly.
>>
>> I think #1b might have merit, as well.
>
> It would be interesting to see what a 1b implementation looks like and
> how it performs. We already allocate a bunch of temporary things to
> support in-flight IO (bio, request) and allocating pageframes on the
> same basis seems a fairly logical fit.
At least for block-i/o it seems the only place we really need struct
page infrastructure is for kmap(). Given we already need a kmap_pfn()
solution for option 2 a "dynamic allocation" stop along that
development path may just naturally fall out.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Andrew Morton @ 2015-03-19 19:59 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Matthew Wilcox, linux-arch, axboe, riel, hch, linux-nvdimm,
Dave Hansen, linux-kernel, linux-raid, mgorman, linux-fsdevel
In-Reply-To: <550AF127.9010504@plexistor.com>
On Thu, 19 Mar 2015 17:54:15 +0200 Boaz Harrosh <boaz@plexistor.com> wrote:
> On 03/19/2015 03:43 PM, Matthew Wilcox wrote:
> <>
> >
> > Dan missed "Support O_DIRECT to a mapped DAX file". More generally, if we
> > want to be able to do any kind of I/O directly to persistent memory,
> > and I think we do, we need to do one of:
> >
> > 1. Construct struct pages for persistent memory
> > 1a. Permanently
> > 1b. While the pages are under I/O
> > 2. Teach the I/O layers to deal in PFNs instead of struct pages
> > 3. Replace struct page with some other structure that can represent both
> > DRAM and PMEM
> >
> > I'm personally a fan of #3, and I was looking at the scatterlist as
> > my preferred data structure. I now believe the scatterlist as it is
> > currently defined isn't sufficient, so we probably end up needing a new
> > data structure. I think Dan's preferred method of replacing struct
> > pages with PFNs is actually less instrusive, but doesn't give us as
> > much advantage (an entirely new data structure would let us move to an
> > extent based system at the same time, instead of sticking with an array
> > of pages). Clearly Boaz prefers 1a, which works well enough for the
> > 8GB NV-DIMMs, but not well enough for the 400GB NV-DIMMs.
> >
> > What's your preference? I guess option 0 is "force all I/O to go
> > through the page cache and then get copied", but that feels like a nasty
> > performance hit.
>
> Thanks Matthew, you have summarized it perfectly.
>
> I think #1b might have merit, as well.
It would be interesting to see what a 1b implementation looks like and
how it performs. We already allocate a bunch of temporary things to
support in-flight IO (bio, request) and allocating pageframes on the
same basis seems a fairly logical fit.
It is all a bit of a stopgap, designed to shoehorn
direct-io-to-dax-mapped-memory into the existing world. Longer term
I'd expect us to move to something more powerful, but it's unclear what
that will be at this time, so a stopgap isn't too bad?
This is all contingent upon the prevalence of machines which have vast
amounts of nv memory and relatively small amounts of regular memory.
How confident are we that this really is the future?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox