* Re: [virtio-dev] Re: [PATCH v9 2/5] virtio-balloon: VIRTIO_BALLOON_F_BALLOON_CHUNKS
From: Wei Wang @ 2017-04-27 6:31 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev@lists.oasis-open.org, linux-kernel@vger.kernel.org,
qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org,
kvm@vger.kernel.org, linux-mm@kvack.org, david@redhat.com,
Hansen, Dave, cornelia.huck@de.ibm.com, akpm@linux-foundation.org,
mgorman@techsingularity.net, aarcange@redhat.com,
amit.shah@redhat.com, pbonzini@redhat.com,
liliang.opensource@gmail.com
In-Reply-To: <20170426192753-mutt-send-email-mst@kernel.org>
On 04/27/2017 07:20 AM, Michael S. Tsirkin wrote:
> On Wed, Apr 26, 2017 at 11:03:34AM +0000, Wang, Wei W wrote:
>> Hi Michael, could you please give some feedback?
> I'm sorry, I'm not sure feedback on what you are requesting.
Oh, just some trivial things (e.g. use a field in the
header, hdr->chunks to indicate the number of chunks
in the payload) that wasn't confirmed.
I will prepare the new version with fixing the agreed issues,
and we can continue to discuss those parts if you still find
them improper.
>
> The interface looks reasonable now, even though there's
> a way to make it even simpler if we can limit chunk size
> to 2G (in fact 4G - 1). Do you think we can live with this
> limitation?
Yes, I think we can. So, is it good to change to use the
previous 64-bit chunk format (52-bit base + 12-bit size)?
>
> But the code still needs some cleanup.
>
OK. We'll also still to discuss your comments in the patch 05.
Best,
Wei
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v9 5/5] virtio-balloon: VIRTIO_BALLOON_F_MISC_VQ
From: Wei Wang @ 2017-04-27 6:33 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, linux-kernel, qemu-devel, virtualization, kvm,
linux-mm, david, dave.hansen, cornelia.huck, akpm, mgorman,
aarcange, amit.shah, pbonzini, liliang.opensource
In-Reply-To: <20170413194732-mutt-send-email-mst@kernel.org>
On 04/14/2017 01:08 AM, Michael S. Tsirkin wrote:
> On Thu, Apr 13, 2017 at 05:35:08PM +0800, Wei Wang wrote:
>> Add a new vq, miscq, to handle miscellaneous requests between the device
>> and the driver.
>>
>> This patch implemnts the VIRTIO_BALLOON_MISCQ_INQUIRE_UNUSED_PAGES
> implements
>
>> request sent from the device.
> Commands are sent from host and handled on guest.
> In fact how is this so different from stats?
> How about reusing the stats vq then? You can use one buffer
> for stats and one buffer for commands.
>
The meaning of the two vqs is a little different. statq is used for
reporting statistics, while miscq is intended to be used to handle
miscellaneous requests from the guest or host (I think it can
also be used the other way around in the future when other
new features are added which need the guest to send requests
and the host to provide responses).
I would prefer to have them separate, because:
If we plan to combine them, we need to put the previous statq
related implementation under miscq with a new command (I think
we can't combine them without using commands to distinguish
the two features).
In this way, an old driver won't work with a new QEMU or a new
driver won't work with an old QEMU. Would this be considered
as an issue here?
>
>> + miscq_out_hdr->flags = 0;
>> +
>> + for_each_populated_zone(zone) {
>> + for (order = MAX_ORDER - 1; order > 0; order--) {
>> + for (migratetype = 0; migratetype < MIGRATE_TYPES;
>> + migratetype++) {
>> + do {
>> + ret = inquire_unused_page_block(zone,
>> + order, migratetype, &page);
>> + if (!ret) {
>> + pfn = (u64)page_to_pfn(page);
>> + add_one_chunk(vb, vq,
>> + PAGE_CHUNK_TYPE_UNUSED,
>> + pfn,
>> + (u64)(1 << order));
>> + }
>> + } while (!ret);
>> + }
>> + }
>> + }
>> + miscq_out_hdr->flags |= VIRTIO_BALLOON_MISCQ_F_COMPLETE;
> And where is miscq_out_hdr used? I see no add_outbuf anywhere.
>
> Things like this should be passed through function parameters
> and not stuffed into device structure, fields should be
> initialized before use and not where we happen to
> have the data handy.
>
miscq_out_hdr is linear with the payload (i.e. kmalloc(hdr+payload) ).
It is the same as the use of statq - one request in-flight each time.
>
> Also, _F_ is normally a bit number, you use it as a value here.
>
It intends to be a bit number. Bit 0 of flags to indicate the completion
of handling the request.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -mm -v10 1/3] mm, THP, swap: Delay splitting THP during swap out
From: Huang, Ying @ 2017-04-27 7:12 UTC (permalink / raw)
To: Minchan Kim
Cc: Huang, Ying, Andrew Morton, linux-mm, linux-kernel,
Andrea Arcangeli, Ebru Akagunduz, Johannes Weiner, Michal Hocko,
Tejun Heo, Hugh Dickins, Shaohua Li, Rik van Riel, cgroups
In-Reply-To: <20170427053141.GA1925@bbox>
Minchan Kim <minchan@kernel.org> writes:
> On Tue, Apr 25, 2017 at 08:56:56PM +0800, Huang, Ying wrote:
>> From: Huang Ying <ying.huang@intel.com>
>>
>> In this patch, splitting huge page is delayed from almost the first
>> step of swapping out to after allocating the swap space for the
>> THP (Transparent Huge Page) and adding the THP into the swap cache.
>> This will batch the corresponding operation, thus improve THP swap out
>> throughput.
>>
>> This is the first step for the THP swap optimization. The plan is to
>> delay splitting the THP step by step and avoid splitting the THP
>> finally.
>>
>> The advantages of the THP swap support include:
>>
>> - Batch the swap operations for the THP and reduce lock
>> acquiring/releasing, including allocating/freeing the swap space,
>> adding/deleting to/from the swap cache, and writing/reading the swap
>> space, etc. This will help to improve the THP swap performance.
>>
>> - The THP swap space read/write will be 2M sequential IO. It is
>> particularly helpful for the swap read, which usually are 4k random
>> IO. This will help to improve the THP swap performance.
>>
>> - It will help the memory fragmentation, especially when the THP is
>> heavily used by the applications. The 2M continuous pages will be
>> free up after the THP swapping out.
>>
>> - It will improve the THP utilization on the system with the swap
>> turned on. Because the speed for khugepaged to collapse the normal
>> pages into the THP is quite slow. After the THP is split during the
>> swapping out, it will take quite long time for the normal pages to
>> collapse back into the THP after being swapped in. The high THP
>> utilization helps the efficiency of the page based memory management
>> too.
>>
>> There are some concerns regarding THP swap in, mainly because possible
>> enlarged read/write IO size (for swap in/out) may put more overhead on
>> the storage device. To deal with that, the THP swap in should be
>> turned on only when necessary. For example, it can be selected via
>> "always/never/madvise" logic, to be turned on globally, turned off
>> globally, or turned on only for VMA with MADV_HUGEPAGE, etc.
>>
>> In this patch, one swap cluster is used to hold the contents of each
>> THP swapped out. So, the size of the swap cluster is changed to that
>> of the THP (Transparent Huge Page) on x86_64 architecture (512). For
>> other architectures which want such THP swap optimization,
>> ARCH_USES_THP_SWAP_CLUSTER needs to be selected in the Kconfig file
>> for the architecture. In effect, this will enlarge swap cluster size
>> by 2 times on x86_64. Which may make it harder to find a free cluster
>> when the swap space becomes fragmented. So that, this may reduce the
>> continuous swap space allocation and sequential write in theory. The
>> performance test in 0day shows no regressions caused by this.
>
> What about other architecures?
>
> I mean THP page size on every architectures would be various.
> If THP page size is much bigger than 2M, the architecture should
> have big swap cluster size for supporting THP swap-out feature.
> It means fast empty-swap cluster consumption so that it can suffer
> from fragmentation easily which causes THP swap void and swap slot
> allocations slow due to not being able to use per-cpu.
>
> What I suggested was contiguous multiple swap cluster allocations
> to meet THP page size. If some of architecure's THP size is 64M
> and SWAP_CLUSTER_SIZE is 2M, it should allocate 32 contiguos
> swap clusters. For that, swap layer need to manage clusters sort
> in order which would be more overhead in CONFIG_THP_SWAP case
> but I think it's tradeoff. With that, every architectures can
> support THP swap easily without arch-specific something.
That may be a good solution for other architectures. But I am afraid I
am not the right person to work on that. Because I don't know the
requirement of other architectures, and I have no other architectures
machines to work on and measure the performance.
And the swap clusters aren't sorted in order now intentionally to avoid
cache line false sharing between the spinlock of struct
swap_cluster_info. If we want to sort clusters in order, we need a
solution for that.
> If (PAGE_SIZE * 512) swap cluster size were okay for most of
> architecture, just increase it. It's orthogonal work regardless of
> THP swapout. Then, we don't need to manage swap clusters sort
> in order in x86_64 which SWAP_CLUSTER_SIZE is equal to
> THP_PAGE_SIZE. It's just a bonus by side-effect.
Andrew suggested to make swap cluster size = huge page size (or turn on
THP swap optimization) only if we enabled CONFIG_THP_SWAP. So that, THP
swap optimization will not be turned on unintentionally.
We may adjust default swap cluster size, but I don't think it need to be
in this patchset.
> AFAIR, I suggested it but cannot remember why we cannot go with
> this way.
>
>>
>> In the future of THP swap optimization, some information of the
>> swapped out THP (such as compound map count) will be recorded in the
>> swap_cluster_info data structure.
>>
>> The mem cgroup swap accounting functions are enhanced to support
>> charge or uncharge a swap cluster backing a THP as a whole.
>>
>> The swap cluster allocate/free functions are added to allocate/free a
>> swap cluster for a THP. A fair simple algorithm is used for swap
>> cluster allocation, that is, only the first swap device in priority
>> list will be tried to allocate the swap cluster. The function will
>> fail if the trying is not successful, and the caller will fallback to
>> allocate a single swap slot instead. This works good enough for
>> normal cases. If the difference of the number of the free swap
>> clusters among multiple swap devices is significant, it is possible
>> that some THPs are split earlier than necessary. For example, this
>> could be caused by big size difference among multiple swap devices.
>>
>> The swap cache functions is enhanced to support add/delete THP to/from
>> the swap cache as a set of (HPAGE_PMD_NR) sub-pages. This may be
>> enhanced in the future with multi-order radix tree. But because we
>> will split the THP soon during swapping out, that optimization doesn't
>> make much sense for this first step.
>>
>> The THP splitting functions are enhanced to support to split THP in
>> swap cache during swapping out. The page lock will be held during
>> allocating the swap cluster, adding the THP into the swap cache and
>> splitting the THP. So in the code path other than swapping out, if
>> the THP need to be split, the PageSwapCache(THP) will be always false.
>>
>> The swap cluster is only available for SSD, so the THP swap
>> optimization in this patchset has no effect for HDD.
>>
>> With the patch, the swap out throughput improves 11.5% (from about
>> 3.73GB/s to about 4.16GB/s) in the vm-scalability swap-w-seq test case
>> with 8 processes. The test is done on a Xeon E5 v3 system. The swap
>> device used is a RAM simulated PMEM (persistent memory) device. To
>> test the sequential swapping out, the test case creates 8 processes,
>> which sequentially allocate and write to the anonymous pages until the
>> RAM and part of the swap device is used up.
>>
>> [hannes@cmpxchg.org: extensive cleanups and simplifications, reduce code size]
>> Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
>> Cc: Andrea Arcangeli <aarcange@redhat.com>
>> Cc: Ebru Akagunduz <ebru.akagunduz@gmail.com>
>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>> Cc: Michal Hocko <mhocko@kernel.org>
>> Cc: Tejun Heo <tj@kernel.org>
>> Cc: Hugh Dickins <hughd@google.com>
>> Cc: Shaohua Li <shli@kernel.org>
>> Cc: Minchan Kim <minchan@kernel.org>
>> Cc: Rik van Riel <riel@redhat.com>
>> Cc: cgroups@vger.kernel.org
>> Suggested-by: Andrew Morton <akpm@linux-foundation.org> [for config option]
>> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> [for changes in huge_memory.c and huge_mm.h]
>> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
>> ---
>> arch/x86/Kconfig | 1 +
>> include/linux/page-flags.h | 7 +-
>> include/linux/swap.h | 25 ++++-
>> include/linux/swap_cgroup.h | 6 +-
>> mm/Kconfig | 12 +++
>> mm/huge_memory.c | 11 +-
>> mm/memcontrol.c | 50 ++++-----
>> mm/shmem.c | 2 +-
>> mm/swap_cgroup.c | 40 +++++--
>> mm/swap_slots.c | 16 ++-
>> mm/swap_state.c | 114 ++++++++++++--------
>> mm/swapfile.c | 256 ++++++++++++++++++++++++++++++++------------
>> 12 files changed, 375 insertions(+), 165 deletions(-)
>
> < snip >
>
>> --- a/mm/shmem.c
>> +++ b/mm/shmem.c
>> @@ -1290,7 +1290,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
>> SetPageUptodate(page);
>> }
>>
>> - swap = get_swap_page();
>> + swap = get_swap_page(page);
>> if (!swap.val)
>> goto redirty;
>>
>
> If swap is non-ssd, swap.val could be zero. Right?
> If so, could we retry like anonymous page swapout?
This is for shmem, where the THP will be split before goes here. That
is, "page" here is always normal page.
>>
>> -swp_entry_t get_swap_page(void)
>> +swp_entry_t get_swap_page(struct page *page)
>> {
>> swp_entry_t entry, *pentry;
>> struct swap_slots_cache *cache;
>>
>> + entry.val = 0;
>> +
>> + if (PageTransHuge(page)) {
>> + if (hpage_nr_pages(page) == SWAPFILE_CLUSTER)
>> + get_swap_pages(1, true, &entry);
>> + return entry;
>> + }
>> +
>
>
> < snip >
>
>> /**
>> @@ -178,20 +192,12 @@ int add_to_swap(struct page *page, struct list_head *list)
>> VM_BUG_ON_PAGE(!PageLocked(page), page);
>> VM_BUG_ON_PAGE(!PageUptodate(page), page);
>>
>> - entry = get_swap_page();
>> +retry:
>> + entry = get_swap_page(page);
>> if (!entry.val)
>> - return 0;
>> -
>> - if (mem_cgroup_try_charge_swap(page, entry)) {
>> - swapcache_free(entry);
>> - return 0;
>> - }
>> -
>> - if (unlikely(PageTransHuge(page)))
>> - if (unlikely(split_huge_page_to_list(page, list))) {
>> - swapcache_free(entry);
>> - return 0;
>> - }
>> + goto fail;
>
> So, with non-SSD swap, THP page *always* get the fail to get swp_entry_t
> and retry after split the page. However, it makes unncessary get_swap_pages
> call which is not trivial. If there is no SSD swap, thp-swap out should
> be void without adding any performance overhead.
> Hmm, but I have no good idea to do it simple. :(
For HDD swap, the device raw throughput is so low (< 100M Bps
typically), that the added overhead here will not be a big issue. Do
you agree?
Best Regards,
Huang, Ying
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v5 31/32] x86: Add sysfs support for Secure Memory Encryption
From: Dave Young @ 2017-04-27 7:25 UTC (permalink / raw)
To: Dave Hansen
Cc: Tom Lendacky, linux-arch, linux-efi, kvm, linux-doc, x86, kexec,
linux-kernel, kasan-dev, linux-mm, iommu, Thomas Gleixner,
Rik van Riel, Brijesh Singh, Toshimitsu Kani, Arnd Bergmann,
Jonathan Corbet, Matt Fleming, Joerg Roedel,
Radim Krčmář, Konrad Rzeszutek Wilk,
Andrey Ryabinin, Ingo Molnar, Michael S. Tsirkin, Andy Lutomirski,
H. Peter Anvin, Borislav Petkov, Paolo Bonzini,
Alexander Potapenko, Larry Woodman, Dmitry Vyukov
In-Reply-To: <1498ec98-b19d-c47d-902b-a68870a3f860@intel.com>
On 04/21/17 at 02:55pm, Dave Hansen wrote:
> On 04/18/2017 02:22 PM, Tom Lendacky wrote:
> > Add sysfs support for SME so that user-space utilities (kdump, etc.) can
> > determine if SME is active.
> >
> > A new directory will be created:
> > /sys/kernel/mm/sme/
> >
> > And two entries within the new directory:
> > /sys/kernel/mm/sme/active
> > /sys/kernel/mm/sme/encryption_mask
>
> Why do they care, and what will they be doing with this information?
Since kdump will copy old memory but need this to know if the old memory
was encrypted or not. With this sysfs file we can know the previous SME
status and pass to kdump kernel as like a kernel param.
Tom, have you got chance to try if it works or not?
Thanks
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 2/2] dax: fix data corruption due to stale mmap reads
From: Jan Kara @ 2017-04-27 7:26 UTC (permalink / raw)
To: Ross Zwisler
Cc: Jan Kara, Andrew Morton, linux-kernel, Alexander Viro,
Alexey Kuznetsov, Andrey Ryabinin, Anna Schumaker,
Christoph Hellwig, Dan Williams, Darrick J. Wong,
Eric Van Hensbergen, Jens Axboe, Johannes Weiner,
Konrad Rzeszutek Wilk, Latchesar Ionkov, linux-cifs,
linux-fsdevel, linux-mm, linux-nfs, linux-nvdimm, Matthew Wilcox,
Ron Minnich, samba-technical, Steve French, Trond Myklebust,
v9fs-developer
In-Reply-To: <20170426225236.GA25838@linux.intel.com>
On Wed 26-04-17 16:52:36, Ross Zwisler wrote:
> On Wed, Apr 26, 2017 at 10:52:35AM +0200, Jan Kara wrote:
> > On Tue 25-04-17 16:59:36, Ross Zwisler wrote:
> > > On Tue, Apr 25, 2017 at 01:10:43PM +0200, Jan Kara wrote:
> > > <>
> > > > Hum, but now thinking more about it I have hard time figuring out why write
> > > > vs fault cannot actually still race:
> > > >
> > > > CPU1 - write(2) CPU2 - read fault
> > > >
> > > > dax_iomap_pte_fault()
> > > > ->iomap_begin() - sees hole
> > > > dax_iomap_rw()
> > > > iomap_apply()
> > > > ->iomap_begin - allocates blocks
> > > > dax_iomap_actor()
> > > > invalidate_inode_pages2_range()
> > > > - there's nothing to invalidate
> > > > grab_mapping_entry()
> > > > - we add zero page in the radix
> > > > tree & map it to page tables
> > > >
> > > > Similarly read vs write fault may end up racing in a wrong way and try to
> > > > replace already existing exceptional entry with a hole page?
> > >
> > > Yep, this race seems real to me, too. This seems very much like the issues
> > > that exist when a thread is doing direct I/O. One thread is doing I/O to an
> > > intermediate buffer (page cache for direct I/O case, zero page for us), and
> > > the other is going around it directly to media, and they can get out of sync.
> > >
> > > IIRC the direct I/O code looked something like:
> > >
> > > 1/ invalidate existing mappings
> > > 2/ do direct I/O to media
> > > 3/ invalidate mappings again, just in case. Should be cheap if there weren't
> > > any conflicting faults. This makes sure any new allocations we made are
> > > faulted in.
> >
> > Yeah, the problem is people generally expect weird behavior when they mix
> > direct and buffered IO (or let alone mmap) however everyone expects
> > standard read(2) and write(2) to be completely coherent with mmap(2).
>
> Yep, fair enough.
>
> > > I guess one option would be to replicate that logic in the DAX I/O path, or we
> > > could try and enhance our locking so page faults can't race with I/O since
> > > both can allocate blocks.
> >
> > In the abstract way, the problem is that we have radix tree (and page
> > tables) cache block mapping information and the operation: "read block
> > mapping information, store it in the radix tree" is not serialized in any
> > way against other block allocations so the information we store can be out
> > of date by the time we store it.
> >
> > One way to solve this would be to move ->iomap_begin call in the fault
> > paths under entry lock although that would mean I have to redo how ext4
> > handles DAX faults because with current code it would create lock inversion
> > wrt transaction start.
>
> I don't think this alone is enough to save us. The I/O path doesn't currently
> take any DAX radix tree entry locks, so our race would just become:
>
> CPU1 - write(2) CPU2 - read fault
>
> dax_iomap_pte_fault()
> grab_mapping_entry() // newly moved
> ->iomap_begin() - sees hole
> dax_iomap_rw()
> iomap_apply()
> ->iomap_begin - allocates blocks
> dax_iomap_actor()
> invalidate_inode_pages2_range()
> - there's nothing to invalidate
> - we add zero page in the radix
> tree & map it to page tables
>
> In their current form I don't think we want to take DAX radix tree entry locks
> in the I/O path because that would effectively serialize I/O over a given
> radix tree entry. For a 2MiB entry, for example, all I/O to that 2MiB range
> would be serialized.
Note that invalidate_inode_pages2_range() will see the entry created by
grab_mapping_entry() on CPU2 and block waiting for its lock and this is
exactly what stops the race. The invalidate_inode_pages2_range()
effectively makes sure there isn't any page fault in progress for given
range...
Also note that writes to a file are serialized by i_rwsem anyway (and at
least serialization of writes to the overlapping range is required by POSIX)
so this doesn't add any more serialization than we already have.
> > Another solution would be to grab i_mmap_sem for write when doing write
> > fault of a page and similarly have it grabbed for writing when doing
> > write(2). This would scale rather poorly but if we later replaced it with a
> > range lock (Davidlohr has already posted a nice implementation of it) it
> > won't be as bad. But I guess option 1) is better...
>
> The best idea I had for handling this sounds similar, which would be to
> convert the radix tree locks to essentially be reader/writer locks. I/O and
> faults that don't modify the block mapping could just take read-level locks,
> and could all run concurrently. I/O or faults that modify a block mapping
> would take a write lock, and serialize with other writers and readers.
Well, this would be difficult to implement inside the radix tree (not
enough bits in the entry) so you'd have to go for some external locking
primitive anyway. And if you do that, read-write range lock Davidlohr has
implemented is what you describe - well we could also have a radix tree
with rwsems but I suspect the overhead of maintaining that would be too
large. It would require larger rewrite than reusing entry locks as I
suggest above though and it isn't an obvious performance win for realistic
workloads either so I'd like to see some performance numbers before going
that way. It likely improves a situation where processes race to fault the
same page for which we already know the block mapping but I'm not sure if
that translates to any measurable performance wins for workloads on DAX
filesystem.
> You could know if you needed a write lock without asking the filesystem - if
> you're a write and the radix tree entry is empty or is for a zero page, you
> grab the write lock.
>
> This dovetails nicely with the idea of having the radix tree act as a cache
> for block mappings. You take the appropriate lock on the radix tree entry,
> and it has the block mapping info for your I/O or fault so you don't have to
> call into the FS. I/O would also participate so we would keep info about
> block mappings that we gather from I/O to help shortcut our page faults.
>
> How does this sound vs the range lock idea? How hard do you think it would be
> to convert our current wait queue system to reader/writer style locking?
>
> Also, how do you think we should deal with the current PMD corruption? Should
> we go with the current fix (I can augment the comments as you suggested), and
> then handle optimizations to that approach and the solution to this larger
> race as a follow-on?
So for now I'm still more inclined to just stay with the radix tree lock as
is and just fix up the locking as I suggest and go for larger rewrite only
if we can demonstrate further performance wins.
WRT your second patch, if we go with the locking as I suggest, it is enough
to unmap the whole range after invalidate_inode_pages2() has cleared radix
tree entries (*) which will be much cheaper (for large writes) than doing
unmapping entry by entry. So I'd go for that. I'll prepare a patch for the
locking change - it will require changes to ext4 transaction handling so it
won't be completely trivial.
(*) The flow of information is: filesystem block mapping info -> radix tree
-> page tables so if 'filesystem block mapping info' changes, we should go
invalidate corresponding radix tree entries (new entries will already have
uptodate info) and then invalidate corresponding page tables (again once
radix tree has no stale entries, we are sure new page table entries will be
uptodate).
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -v3 0/13] mm: make movable onlining suck less
From: Michal Hocko @ 2017-04-27 7:56 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML, Balbir Singh,
Dan Williams, Heiko Carstens, Martin Schwidefsky, Tobias Regnery,
Yasuaki Ishimatsu
In-Reply-To: <20170421120512.23960-1-mhocko@kernel.org>
Hi all,
Andrew prefers to take this after the merge window so I will repost the
full series then. Any feedback is still highly appreciated of course.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 07/13] mm: consider zone which is not fully populated to have holes
From: Michal Hocko @ 2017-04-27 7:58 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML
In-Reply-To: <20170421120512.23960-8-mhocko@kernel.org>
I plan to fold the following into this patch.
---
^ permalink raw reply
* Re: [PATCH 08/13] mm, compaction: skip over holes in __reset_isolation_suitable
From: Michal Hocko @ 2017-04-27 7:59 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML
In-Reply-To: <20170421120512.23960-9-mhocko@kernel.org>
On Fri 21-04-17 14:05:11, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> __reset_isolation_suitable walks the whole zone pfn range and it tries
> to jump over holes by checking the zone for each page. It might still
> stumble over offline pages, though. Skip those by checking PageReserved.
ups, forgot to update the changelog here. s@PageReserved@pfn_to_online_page()@
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
> mm/compaction.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 613c59e928cb..fb548e4c7bd4 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
>
> cond_resched();
>
> - if (!pfn_valid(pfn))
> + page = pfn_to_online_page(pfn);
> + if (!page)
> continue;
> -
> - page = pfn_to_page(pfn);
> if (zone != page_zone(page))
> continue;
>
> --
> 2.11.0
>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm, zone_device: replace {get, put}_zone_device_page() with a single reference
From: Kirill A. Shutemov @ 2017-04-27 8:33 UTC (permalink / raw)
To: Dan Williams, Ingo Molnar
Cc: akpm, linux-mm, Jérôme Glisse, Logan Gunthorpe,
linux-kernel, Kirill Shutemov
In-Reply-To: <149325431313.40660.7404075559824162131.stgit@dwillia2-desk3.amr.corp.intel.com>
On Wed, Apr 26, 2017 at 05:55:31PM -0700, Dan Williams wrote:
> Kirill points out that the calls to {get,put}_dev_pagemap() can be
> removed from the mm fast path if we take a single get_dev_pagemap()
> reference to signify that the page is alive and use the final put of the
> page to drop that reference.
>
> This does require some care to make sure that any waits for the
> percpu_ref to drop to zero occur *after* devm_memremap_page_release(),
> since it now maintains its own elevated reference.
>
> Cc Ingo Molnar <mingo@redhat.com>
> Cc: Jerome Glisse <jglisse@redhat.com>
> Cc: Logan Gunthorpe <logang@deltatee.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Suggested-by: Kirill Shutemov <kirill.shutemov@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
>
> This patch might fix the regression that we found with the conversion to
> generic get_user_pages_fast() in the x86/mm branch pending for 4.12
> (commit 2947ba054a4d "x86/mm/gup: Switch GUP to the generic
> get_user_page_fast() implementation"). I'll test tomorrow, but in case
> someone can give it a try before I wake up, here's an early version.
+ Ingo.
This works for me with and without GUP revert.
Tested-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> drivers/dax/pmem.c | 2 +-
> drivers/nvdimm/pmem.c | 13 +++++++++++--
There's a trivial conflict in drivers/nvdimm/pmem.c when applied to
tip/master.
> include/linux/mm.h | 14 --------------
> kernel/memremap.c | 22 +++++++++-------------
> mm/swap.c | 10 ++++++++++
> 5 files changed, 31 insertions(+), 30 deletions(-)
>
> diff --git a/drivers/dax/pmem.c b/drivers/dax/pmem.c
> index d4ca19bd74eb..9f2a0b4fd801 100644
> --- a/drivers/dax/pmem.c
> +++ b/drivers/dax/pmem.c
> @@ -43,6 +43,7 @@ static void dax_pmem_percpu_exit(void *data)
> struct dax_pmem *dax_pmem = to_dax_pmem(ref);
>
> dev_dbg(dax_pmem->dev, "%s\n", __func__);
> + wait_for_completion(&dax_pmem->cmp);
> percpu_ref_exit(ref);
> }
>
> @@ -53,7 +54,6 @@ static void dax_pmem_percpu_kill(void *data)
>
> dev_dbg(dax_pmem->dev, "%s\n", __func__);
> percpu_ref_kill(ref);
> - wait_for_completion(&dax_pmem->cmp);
> }
>
> static int dax_pmem_probe(struct device *dev)
> diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
> index 3b3dab73d741..6be0c1253fcd 100644
> --- a/drivers/nvdimm/pmem.c
> +++ b/drivers/nvdimm/pmem.c
> @@ -25,6 +25,7 @@
> #include <linux/badblocks.h>
> #include <linux/memremap.h>
> #include <linux/vmalloc.h>
> +#include <linux/blk-mq.h>
> #include <linux/pfn_t.h>
> #include <linux/slab.h>
> #include <linux/pmem.h>
> @@ -243,6 +244,11 @@ static void pmem_release_queue(void *q)
> blk_cleanup_queue(q);
> }
>
> +static void pmem_freeze_queue(void *q)
> +{
> + blk_mq_freeze_queue_start(q);
> +}
> +
> static void pmem_release_disk(void *__pmem)
> {
> struct pmem_device *pmem = __pmem;
> @@ -301,6 +307,9 @@ static int pmem_attach_disk(struct device *dev,
> if (!q)
> return -ENOMEM;
>
> + if (devm_add_action_or_reset(dev, pmem_release_queue, q))
> + return -ENOMEM;
> +
> pmem->pfn_flags = PFN_DEV;
> if (is_nd_pfn(dev)) {
> addr = devm_memremap_pages(dev, &pfn_res, &q->q_usage_counter,
> @@ -320,10 +329,10 @@ static int pmem_attach_disk(struct device *dev,
> pmem->size, ARCH_MEMREMAP_PMEM);
>
> /*
> - * At release time the queue must be dead before
> + * At release time the queue must be frozen before
> * devm_memremap_pages is unwound
> */
> - if (devm_add_action_or_reset(dev, pmem_release_queue, q))
> + if (devm_add_action_or_reset(dev, pmem_freeze_queue, q))
> return -ENOMEM;
>
> if (IS_ERR(addr))
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 00a8fa7e366a..ce17b35257ac 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -758,19 +758,11 @@ static inline enum zone_type page_zonenum(const struct page *page)
> }
>
> #ifdef CONFIG_ZONE_DEVICE
> -void get_zone_device_page(struct page *page);
> -void put_zone_device_page(struct page *page);
> static inline bool is_zone_device_page(const struct page *page)
> {
> return page_zonenum(page) == ZONE_DEVICE;
> }
> #else
> -static inline void get_zone_device_page(struct page *page)
> -{
> -}
> -static inline void put_zone_device_page(struct page *page)
> -{
> -}
> static inline bool is_zone_device_page(const struct page *page)
> {
> return false;
> @@ -786,9 +778,6 @@ static inline void get_page(struct page *page)
> */
> VM_BUG_ON_PAGE(page_ref_count(page) <= 0, page);
> page_ref_inc(page);
> -
> - if (unlikely(is_zone_device_page(page)))
> - get_zone_device_page(page);
> }
>
> static inline void put_page(struct page *page)
> @@ -797,9 +786,6 @@ static inline void put_page(struct page *page)
>
> if (put_page_testzero(page))
> __put_page(page);
> -
> - if (unlikely(is_zone_device_page(page)))
> - put_zone_device_page(page);
> }
>
> #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
> diff --git a/kernel/memremap.c b/kernel/memremap.c
> index 07e85e5229da..5316efdde083 100644
> --- a/kernel/memremap.c
> +++ b/kernel/memremap.c
> @@ -182,18 +182,6 @@ struct page_map {
> struct vmem_altmap altmap;
> };
>
> -void get_zone_device_page(struct page *page)
> -{
> - percpu_ref_get(page->pgmap->ref);
> -}
> -EXPORT_SYMBOL(get_zone_device_page);
> -
> -void put_zone_device_page(struct page *page)
> -{
> - put_dev_pagemap(page->pgmap);
> -}
> -EXPORT_SYMBOL(put_zone_device_page);
> -
> static void pgmap_radix_release(struct resource *res)
> {
> resource_size_t key, align_start, align_size, align_end;
> @@ -237,6 +225,10 @@ static void devm_memremap_pages_release(struct device *dev, void *data)
> struct resource *res = &page_map->res;
> resource_size_t align_start, align_size;
> struct dev_pagemap *pgmap = &page_map->pgmap;
> + unsigned long pfn;
> +
> + for_each_device_pfn(pfn, page_map)
> + put_page(pfn_to_page(pfn));
>
> if (percpu_ref_tryget_live(pgmap->ref)) {
> dev_WARN(dev, "%s: page mapping is still live!\n", __func__);
> @@ -277,7 +269,10 @@ struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
> *
> * Notes:
> * 1/ @ref must be 'live' on entry and 'dead' before devm_memunmap_pages() time
> - * (or devm release event).
> + * (or devm release event). The expected order of events is that @ref has
> + * been through percpu_ref_kill() before devm_memremap_pages_release(). The
> + * wait for the completion of kill and percpu_ref_exit() must occur after
> + * devm_memremap_pages_release().
> *
> * 2/ @res is expected to be a host memory range that could feasibly be
> * treated as a "System RAM" range, i.e. not a device mmio range, but
> @@ -379,6 +374,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
> */
> list_del(&page->lru);
> page->pgmap = pgmap;
> + percpu_ref_get(ref);
> }
> devres_add(dev, page_map);
> return __va(res->start);
> diff --git a/mm/swap.c b/mm/swap.c
> index 5dabf444d724..01267dda6668 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -97,6 +97,16 @@ static void __put_compound_page(struct page *page)
>
> void __put_page(struct page *page)
> {
> + if (is_zone_device_page(page)) {
> + put_dev_pagemap(page->pgmap);
> +
> + /*
> + * The page belong to device, do not return it to
> + * page allocator.
> + */
> + return;
> + }
> +
> if (unlikely(PageCompound(page)))
> __put_compound_page(page);
> else
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* 4.11.0-rc8+/x86_64 desktop lockup until applications closed
From: Arthur Marsh @ 2017-04-27 9:06 UTC (permalink / raw)
To: Linux Kernel Mailing List; +Cc: linux-mm
In-Reply-To: <md5:RQiZYAYNN/yJzTrY48XZ7w==>
I came home yesterday to discover my desktop KDE/plasma session locked
up until I could shutdown firefox and chromium from a console login.
The desktop then became responsive and I could then restart firefox and
chromium.
the 4GiB swap space was nearly full, but the OOM killer apparently
didn't run.
dmesg showed:
[55363.482931] QXcbEventReader: page allocation stalls for 10048ms,
order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null)
[55363.482942] QXcbEventReader cpuset=/ mems_allowed=0
[55363.482948] CPU: 2 PID: 4092 Comm: QXcbEventReader Not tainted
4.11.0-rc8+ #2670
[55363.482950] Hardware name: System manufacturer System Product
Name/M3A78 PRO, BIOS 1701 01/27/2011
[55363.482951] Call Trace:
[55363.482959] ? dump_stack+0x5c/0x84
[55363.482962] ? warn_alloc+0x112/0x1b0
[55363.482964] ? __alloc_pages_slowpath+0x836/0xde0
[55363.482967] ? ktime_get+0x51/0xd0
[55363.482979] ? scsi_request_fn+0x3d/0x690 [scsi_mod]
[55363.482981] ? __alloc_pages_nodemask+0x1eb/0x230
[55363.482984] ? alloc_pages_vma+0xc4/0x280
[55363.482987] ? __read_swap_cache_async+0x189/0x280
[55363.482990] ? read_swap_cache_async+0x24/0x60
[55363.482991] ? swapin_readahead+0x10d/0x1c0
[55363.482994] ? do_swap_page+0x272/0x720
[55363.482997] ? __handle_mm_fault+0x773/0x10f0
[55363.482999] ? handle_mm_fault+0xe7/0x260
[55363.483002] ? __do_page_fault+0x2d4/0x630
[55363.483005] ? page_fault+0x28/0x30
[55363.483008] ? copy_user_generic_string+0x2c/0x40
[55363.483011] ? copy_page_to_iter+0x91/0x2d0
[55363.483014] ? skb_copy_datagram_iter+0x146/0x270
[55363.483016] ? unix_stream_read_actor+0x1a/0x30
[55363.483018] ? unix_stream_read_generic+0x2f8/0x8c0
[55363.483020] ? _raw_spin_lock+0x13/0x40
[55363.483022] ? _raw_spin_unlock_irq+0x1d/0x40
[55363.483024] ? free_swap_slot+0x4e/0x110
[55363.483026] ? _raw_spin_unlock+0x16/0x40
[55363.483028] ? unix_stream_recvmsg+0x81/0xa0
[55363.483029] ? unix_state_double_unlock+0x40/0x40
[55363.483031] ? SYSC_recvfrom+0xe3/0x170
[55363.483034] ? handle_mm_fault+0xe7/0x260
[55363.483036] ? __do_page_fault+0x301/0x630
[55363.483038] ? entry_SYSCALL_64_fastpath+0x1e/0xad
[55363.483040] Mem-Info:
[55363.483044] active_anon:1479559 inactive_anon:281161 isolated_anon:299
active_file:49213 inactive_file:42134 isolated_file:0
unevictable:4651 dirty:108 writeback:188 unstable:0
slab_reclaimable:11225 slab_unreclaimable:20186
mapped:204768 shmem:145888 pagetables:39859 bounce:0
free:25470 free_pcp:0 free_cma:0
[55363.483050] Node 0 active_anon:5918236kB inactive_anon:1124644kB
active_file:196852kB inactive_file:168536kB unevictable:18604kB
isolated(anon):1196kB isolated(file):0kB mapped:819072kB dirty:432kB
writeback:752kB shmem:583552kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0
all_unreclaimable? no
[55363.483052] Node 0 DMA free:15904kB min:132kB low:164kB high:196kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15992kB managed:15904kB
mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[55363.483056] lowmem_reserve[]: 0 3150 7885 7885
[55363.483059] Node 0 DMA32 free:45556kB min:26948kB low:33684kB
high:40420kB active_anon:2273532kB inactive_anon:542768kB
active_file:99788kB inactive_file:89940kB unevictable:32kB
writepending:440kB present:3391168kB managed:3314260kB mlocked:32kB
slab_reclaimable:8800kB slab_unreclaimable:25976kB kernel_stack:7992kB
pagetables:68028kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[55363.483063] lowmem_reserve[]: 0 0 4734 4734
[55363.483066] Node 0 Normal free:40420kB min:40500kB low:50624kB
high:60748kB active_anon:3644668kB inactive_anon:581672kB
active_file:97068kB inactive_file:78784kB unevictable:18572kB
writepending:0kB present:4980736kB managed:4848692kB mlocked:18572kB
slab_reclaimable:36100kB slab_unreclaimable:54768kB kernel_stack:13544kB
pagetables:91408kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[55363.483069] lowmem_reserve[]: 0 0 0 0
[55363.483072] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
15904kB
[55363.483081] Node 0 DMA32: 422*4kB (UME) 847*8kB (UME) 734*16kB (UME)
338*32kB (UME) 108*64kB (UME) 23*128kB (UME) 9*256kB (M) 3*512kB (M)
2*1024kB (M) 0*2048kB 0*4096kB = 46768kB
[55363.483090] Node 0 Normal: 1293*4kB (UME) 1451*8kB (UME) 845*16kB
(UME) 293*32kB (UME) 36*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 41980kB
[55363.483099] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[55363.483100] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[55363.483101] 251525 total pagecache pages
[55363.483104] 10025 pages in swap cache
[55363.483105] Swap cache stats: add 3287896, delete 3277870, find
405176/629612
[55363.483106] Free swap = 498568kB
[55363.483107] Total swap = 4194288kB
[55363.483108] 2096974 pages RAM
[55363.483109] 0 pages HighMem/MovableOnly
[55363.483110] 52260 pages reserved
[55363.483111] 0 pages hwpoisoned
I'm happy to supply configuation information and run further tests.
Arthur.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 4.11.0-rc8+/x86_64 desktop lockup until applications closed
From: Michal Hocko @ 2017-04-27 9:26 UTC (permalink / raw)
To: Arthur Marsh; +Cc: Linux Kernel Mailing List, linux-mm
In-Reply-To: <ccd5aac8-b24a-713a-db54-c35688905595@internode.on.net>
On Thu 27-04-17 18:36:38, Arthur Marsh wrote:
[...]
> [55363.482931] QXcbEventReader: page allocation stalls for 10048ms, order:0,
> mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null)
Are there more of these stalls?
[...]
> [55363.483040] Mem-Info:
> [55363.483044] active_anon:1479559 inactive_anon:281161 isolated_anon:299
> active_file:49213 inactive_file:42134 isolated_file:0
> unevictable:4651 dirty:108 writeback:188 unstable:0
> slab_reclaimable:11225 slab_unreclaimable:20186
> mapped:204768 shmem:145888 pagetables:39859 bounce:0
> free:25470 free_pcp:0 free_cma:0
There is still quite some page cache to reclaim on the inactive list. So
a progress should have been made. Maybe there was a peak memory
consumption which holded this request back?
[...]
> [55363.483059] Node 0 DMA32 free:45556kB min:26948kB low:33684kB
> high:40420kB active_anon:2273532kB inactive_anon:542768kB
> active_file:99788kB inactive_file:89940kB unevictable:32kB
> writepending:440kB present:3391168kB managed:3314260kB mlocked:32kB
> slab_reclaimable:8800kB slab_unreclaimable:25976kB kernel_stack:7992kB
> pagetables:68028kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [55363.483063] lowmem_reserve[]: 0 0 4734 4734
This zone is not usable due to lowmem_reserve
> [55363.483066] Node 0 Normal free:40420kB min:40500kB low:50624kB
> high:60748kB active_anon:3644668kB inactive_anon:581672kB
> active_file:97068kB inactive_file:78784kB unevictable:18572kB
> writepending:0kB present:4980736kB managed:4848692kB mlocked:18572kB
> slab_reclaimable:36100kB slab_unreclaimable:54768kB kernel_stack:13544kB
> pagetables:91408kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [55363.483069] lowmem_reserve[]: 0 0 0 0
and this one is below min watermark already.
> [55363.483106] Free swap = 498568kB
> [55363.483107] Total swap = 4194288kB
Still ~10% of swap is free so not entirely bad.
The question is whether this is reproducible. If yes then I would
suggest watching /proc/vmstat (every second) and if this doesn't show
anything then try to collect vmscan tracepoints
$ mount -t tracefs none /debug/trace/
$ echo 1 > /debug/trace/events/vmscan/enable
$ cat /debug/trace/trace_pipe > log
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 1/7] mm/hugetlb/migration: Use set_huge_pte_at instead of set_pte_at
From: Naoya Horiguchi @ 2017-04-27 9:39 UTC (permalink / raw)
To: Aneesh Kumar K.V
Cc: akpm@linux-foundation.org, mpe@ellerman.id.au, Anshuman Khandual,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <1492449106-27467-2-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
On Mon, Apr 17, 2017 at 10:41:40PM +0530, Aneesh Kumar K.V wrote:
> The right interface to use to set a hugetlb pte entry is set_huge_pte_at. Use
> that instead of set_pte_at.
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Reviewed-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 2/7] mm/follow_page_mask: Split follow_page_mask to smaller functions.
From: Naoya Horiguchi @ 2017-04-27 9:39 UTC (permalink / raw)
To: Aneesh Kumar K.V
Cc: akpm@linux-foundation.org, mpe@ellerman.id.au, Anshuman Khandual,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <1492449106-27467-3-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
On Mon, Apr 17, 2017 at 10:41:41PM +0530, Aneesh Kumar K.V wrote:
> Makes code reading easy. No functional changes in this patch. In a followup
> patch, we will be updating the follow_page_mask to handle hugetlb hugepd format
> so that archs like ppc64 can switch to the generic version. This split helps
> in doing that nicely.
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Reviewed-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 3/7] mm/hugetlb: export hugetlb_entry_migration helper
From: Naoya Horiguchi @ 2017-04-27 9:41 UTC (permalink / raw)
To: Aneesh Kumar K.V
Cc: akpm@linux-foundation.org, mpe@ellerman.id.au, Anshuman Khandual,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <1492449106-27467-4-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
On Mon, Apr 17, 2017 at 10:41:42PM +0530, Aneesh Kumar K.V wrote:
> We will be using this later from the ppc64 code. Change the return type to bool.
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Reviewed-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Freeing HugeTLB page into buddy allocator
From: Anshuman Khandual @ 2017-04-27 11:06 UTC (permalink / raw)
To: Naoya Horiguchi, Anshuman Khandual
Cc: wujianguo@huawei.com, Linux Memory Management List,
Linux Kernel Mailing List
In-Reply-To: <20170427055457.GA19344@hori1.linux.bs1.fc.nec.co.jp>
On 04/27/2017 11:24 AM, Naoya Horiguchi wrote:
> On Tue, Apr 25, 2017 at 02:27:27PM +0530, Anshuman Khandual wrote:
>> Hello Jianguo,
>>
>> In the commit a49ecbcd7b0d5a1cda, it talks about HugeTLB page being
>> freed into buddy allocator instead of hugepage_freelists. But if
>> I look the code closely for the function unmap_and_move_huge_page()
>> it only calls putback_active_hugepage() which puts the page into the
>> huge page active list to free up the source HugeTLB page after any
>> successful migration. I might be missing something here, so can you
>> please point me where we release the HugeTLB page into buddy allocator
>> directly during migration ?
>
> Hi Anshuman,
>
> As stated in the patch description, source hugetlb page is freed after
> successful migration if overcommit is configured.
>
> The call chain is like below:
>
> soft_offline_huge_page
> migrate_pages
> unmap_and_move_huge_page
> putback_active_hugepage(hpage)
> put_page // refcount is down to 0
> __put_page
> __put_compound_page
> free_huge_page
> if (h->surplus_huge_pages_node[nid])
> update_and_free_page
> __free_pages
>
> So the inline comment
>
> + /* overcommit hugetlb page will be freed to buddy */
>
> might be confusing because at this point the overcommit hugetlb page was
> already freed to buddy.
>
> I hope this will help you.
Surely does. Thanks Naoya.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* block: mempool allocation hangs under OOM. (Re: A pitfall of mempool?)
From: Tetsuo Handa @ 2017-04-27 11:19 UTC (permalink / raw)
To: linux-block, linux-mm; +Cc: axboe, mhocko
In-Reply-To: <201704252022.DFB26076.FMOQVFOtJOSFHL@I-love.SAKURA.ne.jp>
Hello.
I noticed a hang up where kswapd was unable to get memory for bio from mempool
at bio_alloc_bioset(GFP_NOFS) request at
http://lkml.kernel.org/r/201704252022.DFB26076.FMOQVFOtJOSFHL@I-love.SAKURA.ne.jp .
Since there is no mean to check whether kswapd was making progress, I tried
below patch on top of allocation watchdog patch at
http://lkml.kernel.org/r/1489578541-81526-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp .
----------
include/linux/gfp.h | 8 ++++++++
kernel/hung_task.c | 5 ++++-
mm/mempool.c | 9 ++++++++-
mm/page_alloc.c | 6 ++----
4 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 2b1a44f5..cc16050 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -469,6 +469,14 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask,
return __alloc_pages_node(nid, gfp_mask, order);
}
+#ifdef CONFIG_DETECT_MEMALLOC_STALL_TASK
+extern void start_memalloc_timer(const gfp_t gfp_mask, const int order);
+extern void stop_memalloc_timer(const gfp_t gfp_mask);
+#else
+#define start_memalloc_timer(gfp_mask, order) do { } while (0)
+#define stop_memalloc_timer(gfp_mask) do { } while (0)
+#endif
+
#ifdef CONFIG_NUMA
extern struct page *alloc_pages_current(gfp_t gfp_mask, unsigned order);
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8f237c0..7d11e8e 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -17,6 +17,7 @@
#include <linux/sysctl.h>
#include <linux/utsname.h>
#include <linux/oom.h>
+#include <linux/console.h>
#include <linux/sched/signal.h>
#include <linux/sched/debug.h>
@@ -149,7 +150,9 @@ static bool rcu_lock_break(struct task_struct *g, struct task_struct *t)
get_task_struct(g);
get_task_struct(t);
rcu_read_unlock();
- cond_resched();
+ if (console_trylock())
+ console_unlock();
+ //cond_resched();
rcu_read_lock();
can_cont = pid_alive(g) && pid_alive(t);
put_task_struct(t);
diff --git a/mm/mempool.c b/mm/mempool.c
index 47a659d..8b449af 100644
--- a/mm/mempool.c
+++ b/mm/mempool.c
@@ -324,11 +324,14 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask)
gfp_temp = gfp_mask & ~(__GFP_DIRECT_RECLAIM|__GFP_IO);
+ start_memalloc_timer(gfp_temp, -1);
repeat_alloc:
element = pool->alloc(gfp_temp, pool->pool_data);
- if (likely(element != NULL))
+ if (likely(element != NULL)) {
+ stop_memalloc_timer(gfp_temp);
return element;
+ }
spin_lock_irqsave(&pool->lock, flags);
if (likely(pool->curr_nr)) {
@@ -341,6 +344,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask)
* for debugging.
*/
kmemleak_update_trace(element);
+ stop_memalloc_timer(gfp_temp);
return element;
}
@@ -350,13 +354,16 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask)
*/
if (gfp_temp != gfp_mask) {
spin_unlock_irqrestore(&pool->lock, flags);
+ stop_memalloc_timer(gfp_temp);
gfp_temp = gfp_mask;
+ start_memalloc_timer(gfp_temp, -1);
goto repeat_alloc;
}
/* We must not sleep if !__GFP_DIRECT_RECLAIM */
if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {
spin_unlock_irqrestore(&pool->lock, flags);
+ stop_memalloc_timer(gfp_temp);
return NULL;
}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f539752..652ba4f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4008,7 +4008,7 @@ bool memalloc_maybe_stalling(void)
return false;
}
-static void start_memalloc_timer(const gfp_t gfp_mask, const int order)
+void start_memalloc_timer(const gfp_t gfp_mask, const int order)
{
struct memalloc_info *m = ¤t->memalloc;
@@ -4032,7 +4032,7 @@ static void start_memalloc_timer(const gfp_t gfp_mask, const int order)
m->in_flight++;
}
-static void stop_memalloc_timer(const gfp_t gfp_mask)
+void stop_memalloc_timer(const gfp_t gfp_mask)
{
struct memalloc_info *m = ¤t->memalloc;
@@ -4055,8 +4055,6 @@ static void memalloc_counter_fold(int cpu)
}
#else
-#define start_memalloc_timer(gfp_mask, order) do { } while (0)
-#define stop_memalloc_timer(gfp_mask) do { } while (0)
#define memalloc_counter_fold(cpu) do { } while (0)
#endif
----------
These patches reported that mempool_alloc(GFP_NOFS|__GFP_NOWARN|__GFP_NORETRY|__GFP_NOMEMALLOC)
was not able to get memory for more than 15 minutes (and thus unable to submit block I/O
request for reclaiming memory via FS writeback, and thus unable to unblock __GFP_FS
allocation requests waiting for FS writeback, and thus unable to invoke the OOM killer for
reclaiming memory for more than 12 minutes, and thus unable to unblock mempool_alloc(),
and thus silent hang up).
Complete log is at http://I-love.SAKURA.ne.jp/tmp/serial-20170427.txt.xz .
----------
[ 0.000000] Linux version 4.11.0-rc8-next-20170426+ (root@localhost.localdomain) (gcc version 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC) ) #96 SMP Thu Apr 27 09:45:31 JST 2017
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.11.0-rc8-next-20170426+ root=UUID=98df1583-260a-423a-a193-182dade5d085 ro crashkernel=256M security=none sysrq_always_enabled console=ttyS0,115200n8 console=tty0 LANG=en_US.UTF-8
(...snipped...)
[ 588.687385] Out of memory: Kill process 8050 (a.out) score 999 or sacrifice child
[ 588.691029] Killed process 8050 (a.out) total-vm:4168kB, anon-rss:80kB, file-rss:0kB, shmem-rss:0kB
[ 588.699415] oom_reaper: reaped process 8050 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
(...snipped...)
[ 1401.127653] MemAlloc: a.out(8865) flags=0x400040 switches=1816 seq=25 gfp=0x1411240(GFP_NOFS|__GFP_NOWARN|__GFP_NORETRY|__GFP_NOMEMALLOC) order=4294967295 delay=1054846 uninterruptible
[ 1401.134726] a.out D11768 8865 7842 0x00000080
[ 1401.137339] Call Trace:
[ 1401.138802] __schedule+0x403/0x940
[ 1401.140626] schedule+0x3d/0x90
[ 1401.142324] schedule_timeout+0x23b/0x510
[ 1401.144348] ? init_timer_on_stack_key+0x60/0x60
[ 1401.146578] io_schedule_timeout+0x1e/0x50
[ 1401.148618] ? io_schedule_timeout+0x1e/0x50
[ 1401.150689] mempool_alloc+0x18b/0x1c0
[ 1401.152587] ? remove_wait_queue+0x70/0x70
[ 1401.154619] bio_alloc_bioset+0xae/0x230
[ 1401.156580] xfs_add_to_ioend+0x7b/0x260 [xfs]
[ 1401.158750] xfs_do_writepage+0x3d9/0x8f0 [xfs]
[ 1401.160916] write_cache_pages+0x230/0x680
[ 1401.162935] ? xfs_add_to_ioend+0x260/0x260 [xfs]
[ 1401.165169] ? sched_clock+0x9/0x10
[ 1401.166961] ? xfs_vm_writepages+0x55/0xa0 [xfs]
[ 1401.169183] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 1401.171312] do_writepages+0x21/0x30
[ 1401.173115] __filemap_fdatawrite_range+0xc6/0x100
[ 1401.175381] filemap_write_and_wait_range+0x2d/0x70
[ 1401.177705] xfs_file_fsync+0x86/0x2b0 [xfs]
[ 1401.179785] vfs_fsync_range+0x4b/0xb0
[ 1401.181660] ? __audit_syscall_exit+0x21f/0x2c0
[ 1401.183824] do_fsync+0x3d/0x70
[ 1401.185473] SyS_fsync+0x10/0x20
[ 1401.187160] do_syscall_64+0x6c/0x1c0
[ 1401.189004] entry_SYSCALL64_slow_path+0x25/0x25
(...snipped...)
[ 1401.211964] MemAlloc: a.out(8866) flags=0x400040 switches=1825 seq=31 gfp=0x1411240(GFP_NOFS|__GFP_NOWARN|__GFP_NORETRY|__GFP_NOMEMALLOC) order=4294967295 delay=1055258 uninterruptible
[ 1401.219018] a.out D11320 8866 7842 0x00000080
[ 1401.221583] Call Trace:
[ 1401.223032] __schedule+0x403/0x940
[ 1401.224871] schedule+0x3d/0x90
[ 1401.226545] schedule_timeout+0x23b/0x510
[ 1401.228550] ? init_timer_on_stack_key+0x60/0x60
[ 1401.230773] io_schedule_timeout+0x1e/0x50
[ 1401.232791] ? io_schedule_timeout+0x1e/0x50
[ 1401.234890] mempool_alloc+0x18b/0x1c0
[ 1401.236768] ? remove_wait_queue+0x70/0x70
[ 1401.238757] bvec_alloc+0x90/0xf0
[ 1401.240444] bio_alloc_bioset+0x17b/0x230
[ 1401.242425] xfs_add_to_ioend+0x7b/0x260 [xfs]
[ 1401.244566] xfs_do_writepage+0x3d9/0x8f0 [xfs]
[ 1401.246709] write_cache_pages+0x230/0x680
[ 1401.248736] ? xfs_add_to_ioend+0x260/0x260 [xfs]
[ 1401.250944] ? sched_clock+0x9/0x10
[ 1401.252720] ? xfs_vm_writepages+0x55/0xa0 [xfs]
[ 1401.254918] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 1401.257028] do_writepages+0x21/0x30
[ 1401.258832] __filemap_fdatawrite_range+0xc6/0x100
[ 1401.261079] filemap_write_and_wait_range+0x2d/0x70
[ 1401.263383] xfs_file_fsync+0x86/0x2b0 [xfs]
[ 1401.265435] vfs_fsync_range+0x4b/0xb0
[ 1401.267301] ? __audit_syscall_exit+0x21f/0x2c0
[ 1401.269449] do_fsync+0x3d/0x70
[ 1401.271080] SyS_fsync+0x10/0x20
[ 1401.272738] do_syscall_64+0x6c/0x1c0
[ 1401.274574] entry_SYSCALL64_slow_path+0x25/0x25
----------
# ./scripts/faddr2line vmlinux bio_alloc_bioset+0xae/0x230
bio_alloc_bioset+0xae/0x230:
bio_alloc_bioset at block/bio.c:484
# ./scripts/faddr2line vmlinux bio_alloc_bioset+0x17b/0x230
bio_alloc_bioset+0x17b/0x230:
bio_alloc_bioset at block/bio.c:501
# ./scripts/faddr2line vmlinux bvec_alloc+0x90/0xf0
bvec_alloc+0x90/0xf0:
bvec_alloc at block/bio.c:216
Something is wrong with assumptions for mempool_alloc() under memory pressure.
I was expecting for a patch that forces not to wait for FS writeback forever
at shrink_inactive_list() at
http://lkml.kernel.org/r/20170307133057.26182-1-mhocko@kernel.org , but
I think this mempool allocation hang problem should be examined before
we give up and workaround shrink_inactive_list().
Reproducer is shown below. Just write()ing & fsync()ing files on a plain
XFS filesystem ( /dev/sda1 ) on a plain SCSI disk under memory pressure.
No RAID/LVM/loopback etc. are used.
----------
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int main(int argc, char *argv[])
{
static char buffer[128] = { };
char *buf = NULL;
unsigned long size;
unsigned long i;
for (i = 0; i < 1024; i++) {
if (fork() == 0) {
int fd = open("/proc/self/oom_score_adj", O_WRONLY);
write(fd, "1000", 4);
close(fd);
snprintf(buffer, sizeof(buffer), "/tmp/file.%u", getpid());
fd = open(buffer, O_WRONLY | O_CREAT | O_APPEND, 0600);
sleep(1);
while (write(fd, buffer, sizeof(buffer)) == sizeof(buffer))
fsync(fd);
_exit(0);
}
}
for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
char *cp = realloc(buf, size);
if (!cp) {
size >>= 1;
break;
}
buf = cp;
}
sleep(2);
/* Will cause OOM due to overcommit */
for (i = 0; i < size; i += 4096) {
buf[i] = 0;
}
pause();
return 0;
}
----------
Regards.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RESENT PATCH] x86/mem: fix the offset overflow when read/write mem
From: zhongjiang @ 2017-04-27 11:49 UTC (permalink / raw)
To: akpm
Cc: arnd, hannes, kirill, rientjes, mgorman, hughd, riel, linux-mm,
linux-kernel
From: zhong jiang <zhongjiang@huawei.com>
Recently, I found the following issue, it will result in the panic.
[ 168.739152] mmap1: Corrupted page table at address 7f3e6275a002
[ 168.745039] PGD 61f4a1067
[ 168.745040] PUD 61ab19067
[ 168.747730] PMD 61fb8b067
[ 168.750418] PTE 8000100000000225
[ 168.753109]
[ 168.757795] Bad pagetable: 000d [#1] SMP
[ 168.761696] Modules linked in: intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd iTCO_wdt glue_helper cryptd sg iTCO_vendor_support i7core_edac edac_core shpchp lpc_ich i2c_i801 pcspkr mfd_core acpi_cpufreq ip_tables xfs libcrc32c sd_mod igb ata_generic ptp pata_acpi pps_core mptsas ata_piix scsi_transport_sas i2c_algo_bit libata mptscsih i2c_core serio_raw crc32c_intel bnx2 mptbase dca dm_mirror dm_region_hash dm_log dm_mod
[ 168.805983] CPU: 15 PID: 10369 Comm: mmap1 Not tainted 4.11.0-rc2-327.28.3.53.x86_64+ #345
[ 168.814202] Hardware name: Huawei Technologies Co., Ltd. Tecal RH2285 /BC11BTSA , BIOS CTSAV036 04/27/2011
[ 168.825704] task: ffff8806207d5200 task.stack: ffffc9000c340000
[ 168.831592] RIP: 0033:0x7f3e622c5360
[ 168.835147] RSP: 002b:00007ffe2bb7a098 EFLAGS: 00010203
[ 168.840344] RAX: 00007ffe2bb7a0c0 RBX: 0000000000000000 RCX: 00007f3e6275a000
[ 168.847439] RDX: 00007f3e622c5360 RSI: 00007f3e6275a000 RDI: 00007ffe2bb7a0c0
[ 168.854535] RBP: 00007ffe2bb7a4e0 R08: 00007f3e621c3d58 R09: 000000000000002d
[ 168.861632] R10: 00007ffe2bb79e20 R11: 00007f3e622fbcb0 R12: 00000000004005d0
[ 168.868728] R13: 00007ffe2bb7a5c0 R14: 0000000000000000 R15: 0000000000000000
[ 168.875825] FS: 00007f3e62752740(0000) GS:ffff880627bc0000(0000) knlGS:0000000000000000
[ 168.883870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 168.889583] CR2: 00007f3e6275a002 CR3: 0000000622845000 CR4: 00000000000006e0
[ 168.896680] RIP: 0x7f3e622c5360 RSP: 00007ffe2bb7a098
[ 168.901713] ---[ end trace ef98fa9f2a01cbc6 ]---
[ 168.90630 arch/x86/kernel/smp.c:127 native_smp_send_reschedule+0x3f/0x50
[ 168.935410] Modules linked in: intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd iTCO_wdt glue_helper cryptd sg iTCO_vendor_support i7core_edac edac_core shpchp lpc_ich i2c_i801 pcspkr mfd_core acpi_cpufreq ip_tables xfs libcrc32c sd_mod igb ata_generic ptp pata_acpi pps_core mptsas ata_piix scsi_transport_sas i2c_algo_bit libata mptscsih i2c_core serio_raw crc32c_intel bnx2 mptbase dca dm_mirror dm_region_hash dm_log dm_mod
[ 168.979686] CPU: 15 PID: 10369 Comm: mmap1 Tainted: G D 4.11.0-rc2-327.28.3.53.x86_64+ #345
[ 168.989114] Hardware name: Huawei Technologies Co., Ltd. Tecal RH2285 /BC11BTSA , BIOS CTSAV036 04/27/2011
[ 169.000616] Call Trace:
[ 169.003049] <IRQ>
[ 169.005050] dump_stack+0x63/0x84
[ 169.008348] __warn+0xd1/0xf0
[ 169.011297] warn_slowpath_null+0x1d/0x20
[ 169.015282] native_smp_send_reschedule+0x3f/0x50
[ 169.019961] resched_curr+0xa1/0xc0
[ 169.023428] check_preempt_curr+0x70/0x90
[ 169.027415] ttwu_do_wakeup+0x1e/0x160
[ 169.031142] ttwu_do_activate+0x77/0x80
[ 169.034956] try_to_wake_up+0x1c3/0x430
[ 169.038771] default_wake_function+0x12/0x20
[ 169.043019] __wake_up_common+0x55/0x90
[ 169.046833] __wake_up_locked+0x13/0x20
[ 169.050649] ep_poll_callback+0xbb/0x240
[ 169.054550] __wake_up_common+0x55/0x90
[ 169.058363] __wake_up+0x39/0x50
[ 169.061574] wake_up_klogd_work_func+0x40/0x60
[ 169.065994] irq_work_run_list+0x4d/0x70
[ 169.069895] irq_work_tick+0x40/0x50
[ 169.073452] update_process_times+0x42/0x60
[ 169.077612] tick_periodic+0x2b/0x80
[ 169.081166] tick_handle_periodic+0x25/0x70
[ 169.085326] local_apic_timer_interrupt+0x35/0x60
[ 169.090004] smp_apic_timer_interrupt+0x38/0x50
[ 169.094507] apic_timer_interrupt+0x93/0xa0
[ 169.098667] RIP: 0010:panic+0x1f5/0x239
[ 169.102480] RSP: 0000:ffffc9000c343dd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 169.110010] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006
[ 169.117106] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff880627bcdfe0
[ 169.124201] RBP: ffffc9000c343e48 R08: 00000000fffffffe R09: 0000000000000395
[ 169.131298] R10: 0000000000000005 R11: 0000000000000394 R12: ffffffff81a0c475
[ 169.138395] R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000000d
[ 169.145491] </IRQ>
[ 169.147578] ? panic+0x1f1/0x239
[ 169.150789] oops_end+0xb8/0xd0
[ 169.153910] pgtable_bad+0x8a/0x95
[ 169.157294] __do_page_fault+0x3aa/0x4a0
[ 169.161194] do_page_fault+0x30/0x80
[ 169.164750] ? do_syscall_64+0x175/0x180
[ 169.168649] page_fault+0x28/0x30
the following case can reproduce the issue.
int mem_fd = 0;
char rw_buf[1024];
unsigned char * map_base_s;
unsigned long show_addr = 0x100000000000;
unsigned long show_len = 0x10;
if(argc !=2 )
{
printf( "%s show_addr\n", argv[0] );
return 0;
}
else
{
char *stop;
show_addr = strtoul( argv[1], &stop, 0 );
printf("show_addr= 0x%lu\n", show_addr );
}
mem_fd = open(DEV_NAME, O_RDONLY);
if (mem_fd == -1)
{
printf("open %s failed.", DEV_NAME);
return 0;
}
map_base_s = mmap(NULL, show_len, PROT_READ, MAP_SHARED, mem_fd, show_addr);
if ((long)map_base_s == -1)
{
printf("input address map to user space fail!\n");
return 0;
}
else
{
printf("mmap successfull!\n");
}
memcpy( rw_buf, map_base_s, show_len );
The pgoff is enough large, it exceed the size of the real memory.
and the mmap can return the success.
I fix it by checking the conditions. it can make it suitable for
the mapped and use.
Signed-off-by: zhong jiang <zhongjiang@huawei.com>
---
drivers/char/mem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/char/mem.c b/drivers/char/mem.c
index 7e4a9d1..3a765e02 100644
--- a/drivers/char/mem.c
+++ b/drivers/char/mem.c
@@ -55,7 +55,7 @@ static inline int valid_phys_addr_range(phys_addr_t addr, size_t count)
static inline int valid_mmap_phys_addr_range(unsigned long pfn, size_t size)
{
- return 1;
+ return (pfn << PAGE_SHIFT) + size <= __pa(high_memory);
}
#endif
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH] mm: Add additional consistency check
From: Michal Hocko @ 2017-04-27 12:06 UTC (permalink / raw)
To: Kees Cook
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Joonsoo Kim, linux-mm, linux-kernel
In-Reply-To: <20170404113022.GC15490@dhcp22.suse.cz>
On Tue 04-04-17 13:30:22, Michal Hocko wrote:
> On Fri 31-03-17 09:40:28, Kees Cook wrote:
> > As found in PaX, this adds a cheap check on heap consistency, just to
> > notice if things have gotten corrupted in the page lookup.
> >
> > Signed-off-by: Kees Cook <keescook@chromium.org>
>
> NAK without a proper changelog. Seriously, we do not blindly apply
> changes from other projects without a deep understanding of all
> consequences.
This still seems to be in the mmotm tree
http://www.ozlabs.org/~akpm/mmotm/broken-out/mm-add-additional-consistency-check.patch
I hope we have agreed that this is not the right approach to handle
invalid pointers in kfree and rather go soemthing like
http://lkml.kernel.org/r/20170411141956.GP6729@dhcp22.suse.cz
instead.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Question on ___GFP_NOLOCKDEP - Was: Re: [PATCH 1/1] Remove hardcoding of ___GFP_xxx bitmasks
From: Igor Stoppa @ 2017-04-27 12:16 UTC (permalink / raw)
To: Michal Hocko; +Cc: namhyung, linux-mm, linux-kernel
In-Reply-To: <e3fe4d80-10a8-2008-1798-af3893fe418a@huawei.com>
On 26/04/17 18:29, Igor Stoppa wrote:
> On 26/04/17 17:47, Michal Hocko wrote:
[...]
>> Also the current mm tree has ___GFP_NOLOCKDEP which is not addressed
>> here so I suspect you have based your change on the Linus tree.
> I used your tree from kernel.org
I found it, I was using master, instead of auto-latest (is it correct?)
But now I see something that I do not understand (apologies if I'm
asking something obvious).
First there is:
[...]
#define ___GFP_WRITE 0x800000u
#define ___GFP_KSWAPD_RECLAIM 0x1000000u
#ifdef CONFIG_LOCKDEP
#define ___GFP_NOLOCKDEP 0x4000000u
#else
#define ___GFP_NOLOCKDEP 0
#endif
Then:
/* Room for N __GFP_FOO bits */
#define __GFP_BITS_SHIFT (25 + IS_ENABLED(CONFIG_LOCKDEP))
Shouldn't it be either:
___GFP_NOLOCKDEP 0x2000000u
or:
#define __GFP_BITS_SHIFT (25 + 2 * IS_ENABLED(CONFIG_LOCKDEP))
thanks, igor
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Question on ___GFP_NOLOCKDEP - Was: Re: [PATCH 1/1] Remove hardcoding of ___GFP_xxx bitmasks
From: Igor Stoppa @ 2017-04-27 12:18 UTC (permalink / raw)
To: Michal Hocko; +Cc: namhyung, linux-mm, linux-kernel
In-Reply-To: <e3fe4d80-10a8-2008-1798-af3893fe418a@huawei.com>
On 26/04/17 18:29, Igor Stoppa wrote:
> On 26/04/17 17:47, Michal Hocko wrote:
[...]
>> Also the current mm tree has ___GFP_NOLOCKDEP which is not addressed
>> here so I suspect you have based your change on the Linus tree.
> I used your tree from kernel.org
I found it, I was using master, instead of auto-latest (is it correct?)
But now I see something that I do not understand (apologies if I'm
asking something obvious).
First there is:
[...]
#define ___GFP_WRITE 0x800000u
#define ___GFP_KSWAPD_RECLAIM 0x1000000u
#ifdef CONFIG_LOCKDEP
#define ___GFP_NOLOCKDEP 0x4000000u
#else
#define ___GFP_NOLOCKDEP 0
#endif
Then:
/* Room for N __GFP_FOO bits */
#define __GFP_BITS_SHIFT (25 + IS_ENABLED(CONFIG_LOCKDEP))
Shouldn't it be either:
___GFP_NOLOCKDEP 0x2000000u
or:
#define __GFP_BITS_SHIFT (25 + 2 * IS_ENABLED(CONFIG_LOCKDEP))
thanks, igor
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Question on ___GFP_NOLOCKDEP - Was: Re: [PATCH 1/1] Remove hardcoding of ___GFP_xxx bitmasks
From: Michal Hocko @ 2017-04-27 13:35 UTC (permalink / raw)
To: Igor Stoppa, Andrew Morton; +Cc: namhyung, linux-mm, linux-kernel
In-Reply-To: <9929419e-c22e-2a9f-a8a6-ad98d5a9da06@huawei.com>
On Thu 27-04-17 15:16:47, Igor Stoppa wrote:
> On 26/04/17 18:29, Igor Stoppa wrote:
>
> > On 26/04/17 17:47, Michal Hocko wrote:
>
> [...]
>
> >> Also the current mm tree has ___GFP_NOLOCKDEP which is not addressed
> >> here so I suspect you have based your change on the Linus tree.
>
> > I used your tree from kernel.org
>
> I found it, I was using master, instead of auto-latest (is it correct?)
yes
> But now I see something that I do not understand (apologies if I'm
> asking something obvious).
>
> First there is:
>
> [...]
> #define ___GFP_WRITE 0x800000u
> #define ___GFP_KSWAPD_RECLAIM 0x1000000u
> #ifdef CONFIG_LOCKDEP
> #define ___GFP_NOLOCKDEP 0x4000000u
> #else
> #define ___GFP_NOLOCKDEP 0
> #endif
>
> Then:
>
> /* Room for N __GFP_FOO bits */
> #define __GFP_BITS_SHIFT (25 + IS_ENABLED(CONFIG_LOCKDEP))
>
>
>
> Shouldn't it be either:
> ___GFP_NOLOCKDEP 0x2000000u
Yes it should. At the time when this patch was written this value was
used. Later I've removed __GFP_OTHER by 41b6167e8f74 ("mm: get rid of
__GFP_OTHER_NODE") and forgot to refresh this one. Thanks for noticing
this.
Andrew, could you fold the following in please?
---
^ permalink raw reply
* Re: [PATCH -mm -v10 1/3] mm, THP, swap: Delay splitting THP during swap out
From: Johannes Weiner @ 2017-04-27 13:37 UTC (permalink / raw)
To: Huang, Ying
Cc: Minchan Kim, Andrew Morton, linux-mm, linux-kernel,
Andrea Arcangeli, Ebru Akagunduz, Michal Hocko, Tejun Heo,
Hugh Dickins, Shaohua Li, Rik van Riel, cgroups
In-Reply-To: <87mvb21fz1.fsf@yhuang-dev.intel.com>
On Thu, Apr 27, 2017 at 03:12:34PM +0800, Huang, Ying wrote:
> Minchan Kim <minchan@kernel.org> writes:
> > On Tue, Apr 25, 2017 at 08:56:56PM +0800, Huang, Ying wrote:
> >> @@ -178,20 +192,12 @@ int add_to_swap(struct page *page, struct list_head *list)
> >> VM_BUG_ON_PAGE(!PageLocked(page), page);
> >> VM_BUG_ON_PAGE(!PageUptodate(page), page);
> >>
> >> - entry = get_swap_page();
> >> +retry:
> >> + entry = get_swap_page(page);
> >> if (!entry.val)
> >> - return 0;
> >> -
> >> - if (mem_cgroup_try_charge_swap(page, entry)) {
> >> - swapcache_free(entry);
> >> - return 0;
> >> - }
> >> -
> >> - if (unlikely(PageTransHuge(page)))
> >> - if (unlikely(split_huge_page_to_list(page, list))) {
> >> - swapcache_free(entry);
> >> - return 0;
> >> - }
> >> + goto fail;
> >
> > So, with non-SSD swap, THP page *always* get the fail to get swp_entry_t
> > and retry after split the page. However, it makes unncessary get_swap_pages
> > call which is not trivial. If there is no SSD swap, thp-swap out should
> > be void without adding any performance overhead.
> > Hmm, but I have no good idea to do it simple. :(
>
> For HDD swap, the device raw throughput is so low (< 100M Bps
> typically), that the added overhead here will not be a big issue. Do
> you agree?
I fully agree. If you swap to spinning rust, an extra function call
here is the least of your concern.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 1/1] Remove hardcoding of ___GFP_xxx bitmasks
From: Michal Hocko @ 2017-04-27 13:41 UTC (permalink / raw)
To: Igor Stoppa; +Cc: namhyung, linux-mm, linux-kernel
In-Reply-To: <e3fe4d80-10a8-2008-1798-af3893fe418a@huawei.com>
On Wed 26-04-17 18:29:08, Igor Stoppa wrote:
[...]
> If you prefer to have this patch only as part of the larger patchset,
> I'm also fine with it.
I agree that the situation is not ideal. If a larger set of changes
would benefit from this change then it would clearly add arguments...
> Also, if you could reply to [1], that would be greatly appreciated.
I will try to get to it but from a quick glance, yet-another-zone will
hit a lot of opposition...
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 1/1] Remove hardcoding of ___GFP_xxx bitmasks
From: Igor Stoppa @ 2017-04-27 14:06 UTC (permalink / raw)
To: Michal Hocko; +Cc: namhyung, linux-mm, linux-kernel
In-Reply-To: <20170427134158.GI4706@dhcp22.suse.cz>
On 27/04/17 16:41, Michal Hocko wrote:
> On Wed 26-04-17 18:29:08, Igor Stoppa wrote:
> [...]
>> If you prefer to have this patch only as part of the larger patchset,
>> I'm also fine with it.
>
> I agree that the situation is not ideal. If a larger set of changes
> would benefit from this change then it would clearly add arguments...
Ok, then I'll send it out as part of the larger RFC set.
>> Also, if you could reply to [1], that would be greatly appreciated.
>
> I will try to get to it but from a quick glance, yet-another-zone will
> hit a lot of opposition...
The most basic questions, that I hope can be answered with Yes/No =) are:
- should a new zone be added after DMA32?
- should I try hard to keep the mask fitting a 32bit word - at least for
hose who do not use the new zone - or is it ok to just stretch it to 64
bits?
If you could answer these, then I'll have a better idea of what I need
to do to.
TIA, igor
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH man-pages 0/2] some more userfault pages updates
From: Mike Rapoport @ 2017-04-27 14:14 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man,
Mike Rapoport
Hi Michael,
Here are some more updates to {ioctl_}userfaultfd.2 pages.
Mike Rapoport (2):
userfaultfd.2: start documenting non-cooperative events
ioctl_userfaultfd.2: start adding details about userfaultfd features
man2/ioctl_userfaultfd.2 | 53 ++++++++++++++++++-
man2/userfaultfd.2 | 135 ++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 179 insertions(+), 9 deletions(-)
--
1.9.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox