* x86: vmalloc and THP
@ 2014-08-12 5:00 Oren Twaig
2014-08-12 6:07 ` Kirill A. Shutemov
0 siblings, 1 reply; 13+ messages in thread
From: Oren Twaig @ 2014-08-12 5:00 UTC (permalink / raw)
To: linux-kernel@vger.kernel.org, linux-mm; +Cc: Shai Fultheim (Shai@ScaleMP.com)
[-- Attachment #1: Type: text/html, Size: 1562 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
2014-08-12 5:00 x86: vmalloc and THP Oren Twaig
@ 2014-08-12 6:07 ` Kirill A. Shutemov
0 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 6:07 UTC (permalink / raw)
To: Oren Twaig
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> <html style="direction: ltr;">
plain/text, please.
>Hello,
>
>Does memory allocated using vmalloc() will be mapped using huge
>pages either directly or later by THP ?
No. It's neither aligned properly, nor physically contiguous.
>If not, is there any fast way to change this behavior ? Maybe by
>changing the granularity/alignment of such allocations to allow such
>mapping ?
What's the point to use vmalloc() in this case?
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
@ 2014-08-12 6:07 ` Kirill A. Shutemov
0 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 6:07 UTC (permalink / raw)
To: Oren Twaig
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> <html style="direction: ltr;">
plain/text, please.
>Hello,
>
>Does memory allocated using vmalloc() will be mapped using huge
>pages either directly or later by THP ?
No. It's neither aligned properly, nor physically contiguous.
>If not, is there any fast way to change this behavior ? Maybe by
>changing the granularity/alignment of such allocations to allow such
>mapping ?
What's the point to use vmalloc() in this case?
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
2014-08-12 6:07 ` Kirill A. Shutemov
@ 2014-08-12 12:28 ` Eric Dumazet
-1 siblings, 0 replies; 13+ messages in thread
From: Eric Dumazet @ 2014-08-12 12:28 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Oren Twaig, linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote:
> On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> >Does memory allocated using vmalloc() will be mapped using huge
> >pages either directly or later by THP ?
>
> No. It's neither aligned properly, nor physically contiguous.
>
> >If not, is there any fast way to change this behavior ? Maybe by
> >changing the granularity/alignment of such allocations to allow such
> >mapping ?
>
> What's the point to use vmalloc() in this case?
Look at various large hashes we have in the system, all using
vmalloc() :
[ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)
[ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
[ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
I would imagine a performance difference if we were using hugepages.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
@ 2014-08-12 12:28 ` Eric Dumazet
0 siblings, 0 replies; 13+ messages in thread
From: Eric Dumazet @ 2014-08-12 12:28 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Oren Twaig, linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote:
> On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> >Does memory allocated using vmalloc() will be mapped using huge
> >pages either directly or later by THP ?
>
> No. It's neither aligned properly, nor physically contiguous.
>
> >If not, is there any fast way to change this behavior ? Maybe by
> >changing the granularity/alignment of such allocations to allow such
> >mapping ?
>
> What's the point to use vmalloc() in this case?
Look at various large hashes we have in the system, all using
vmalloc() :
[ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)
[ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
[ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
I would imagine a performance difference if we were using hugepages.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
2014-08-12 12:28 ` Eric Dumazet
@ 2014-08-12 15:01 ` Kirill A. Shutemov
-1 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 15:01 UTC (permalink / raw)
To: Eric Dumazet
Cc: Oren Twaig, linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 05:28:52AM -0700, Eric Dumazet wrote:
> On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote:
> > On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> > >If not, is there any fast way to change this behavior ? Maybe by
> > >changing the granularity/alignment of such allocations to allow such
> > >mapping ?
> >
> > What's the point to use vmalloc() in this case?
>
> Look at various large hashes we have in the system, all using
> vmalloc() :
>
> [ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)
> [ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
> [ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
I see lower-order allocation in upstream code. Is it some distribution
tweak?
> I would imagine a performance difference if we were using hugepages.
Okay, it's *probably* a valid point.
The hash tables are only allocated with vmalloc() on NUMA system, if
hashdist=1 (default on NUMA). It does it to distribute memory between
nodes. vmalloc() in NUMA_NO_NODE case will allocate all memory with
0-order page allocations: no physical contiguous memory for hugepage
mappings.
I guess we could teach vmalloc() to interleave between nodes on PMD_SIZE
chunks rather then on PAGE_SIZE if caller asks for big memory allocations.
Although, I'm not sure it it would fit all vmalloc() users.
We also would need to allocate PMD_SIZE-aligned virtual address range
to be able to mapped allocated memory with pmds.
It's *potentially* interesting research project. Any volunteers?
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
@ 2014-08-12 15:01 ` Kirill A. Shutemov
0 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 15:01 UTC (permalink / raw)
To: Eric Dumazet
Cc: Oren Twaig, linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 05:28:52AM -0700, Eric Dumazet wrote:
> On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote:
> > On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
> > >If not, is there any fast way to change this behavior ? Maybe by
> > >changing the granularity/alignment of such allocations to allow such
> > >mapping ?
> >
> > What's the point to use vmalloc() in this case?
>
> Look at various large hashes we have in the system, all using
> vmalloc() :
>
> [ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)
> [ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
> [ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
I see lower-order allocation in upstream code. Is it some distribution
tweak?
> I would imagine a performance difference if we were using hugepages.
Okay, it's *probably* a valid point.
The hash tables are only allocated with vmalloc() on NUMA system, if
hashdist=1 (default on NUMA). It does it to distribute memory between
nodes. vmalloc() in NUMA_NO_NODE case will allocate all memory with
0-order page allocations: no physical contiguous memory for hugepage
mappings.
I guess we could teach vmalloc() to interleave between nodes on PMD_SIZE
chunks rather then on PAGE_SIZE if caller asks for big memory allocations.
Although, I'm not sure it it would fit all vmalloc() users.
We also would need to allocate PMD_SIZE-aligned virtual address range
to be able to mapped allocated memory with pmds.
It's *potentially* interesting research project. Any volunteers?
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
2014-08-12 6:07 ` Kirill A. Shutemov
@ 2014-08-12 16:20 ` Oren Twaig
-1 siblings, 0 replies; 13+ messages in thread
From: Oren Twaig @ 2014-08-12 16:20 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
Hi Kirill,
I saw the thread has developed nicely :), still - wanted to answer your
question
below.
On 8/12/2014 9:07 AM, Kirill A. Shutemov wrote:
> On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
>> <html style="direction: ltr;">
> plain/text, please.
Yes - noticed the html, sent again in plain text.
>> If not, is there any fast way to change this behavior ? Maybe by
>> changing the granularity/alignment of such allocations to allow such
>> mapping ?
> What's the point to use vmalloc() in this case?
I've noticed that some lock/s are using linear addresses which are
located at 0xffffc901922b4500 and from what I understand
from mm.txt (kernel 3.0.101):
*ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
*So I'm not sure who/how/why this lock got allocated there, but obviously
it is using that linear set. No ?
>
---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
@ 2014-08-12 16:20 ` Oren Twaig
0 siblings, 0 replies; 13+ messages in thread
From: Oren Twaig @ 2014-08-12 16:20 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
Hi Kirill,
I saw the thread has developed nicely :), still - wanted to answer your
question
below.
On 8/12/2014 9:07 AM, Kirill A. Shutemov wrote:
> On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote:
>> <html style="direction: ltr;">
> plain/text, please.
Yes - noticed the html, sent again in plain text.
>> If not, is there any fast way to change this behavior ? Maybe by
>> changing the granularity/alignment of such allocations to allow such
>> mapping ?
> What's the point to use vmalloc() in this case?
I've noticed that some lock/s are using linear addresses which are
located at 0xffffc901922b4500 and from what I understand
from mm.txt (kernel 3.0.101):
*ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
*So I'm not sure who/how/why this lock got allocated there, but obviously
it is using that linear set. No ?
>
---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
2014-08-12 16:20 ` Oren Twaig
@ 2014-08-12 21:40 ` Kirill A. Shutemov
-1 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 21:40 UTC (permalink / raw)
To: Oren Twaig
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 07:20:52PM +0300, Oren Twaig wrote:
> >What's the point to use vmalloc() in this case?
> I've noticed that some lock/s are using linear addresses which are
> located at 0xffffc901922b4500 and from what I understand
> from mm.txt (kernel 3.0.101):
> *ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
>
> *So I'm not sure who/how/why this lock got allocated there, but obviously
> it is using that linear set. No ?
It would be nice to know what lock it was, but nothing is inherently wrong
with lock in vmalloc space.
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: x86: vmalloc and THP
@ 2014-08-12 21:40 ` Kirill A. Shutemov
0 siblings, 0 replies; 13+ messages in thread
From: Kirill A. Shutemov @ 2014-08-12 21:40 UTC (permalink / raw)
To: Oren Twaig
Cc: linux-kernel@vger.kernel.org, linux-mm,
Shai Fultheim (Shai@ScaleMP.com)
On Tue, Aug 12, 2014 at 07:20:52PM +0300, Oren Twaig wrote:
> >What's the point to use vmalloc() in this case?
> I've noticed that some lock/s are using linear addresses which are
> located at 0xffffc901922b4500 and from what I understand
> from mm.txt (kernel 3.0.101):
> *ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
>
> *So I'm not sure who/how/why this lock got allocated there, but obviously
> it is using that linear set. No ?
It would be nice to know what lock it was, but nothing is inherently wrong
with lock in vmalloc space.
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 13+ messages in thread
* x86: vmalloc and THP
@ 2014-08-12 5:01 ` Oren Twaig
0 siblings, 0 replies; 13+ messages in thread
From: Oren Twaig @ 2014-08-12 5:01 UTC (permalink / raw)
To: linux-kernel@vger.kernel.org, linux-mm; +Cc: Shai Fultheim (Shai@ScaleMP.com)
Hello,
Does memory allocated using vmalloc() will be mapped using huge pages
either directly or later by THP ?
If not, is there any fast way to change this behavior ? Maybe by
changing the granularity/alignment of such allocations to allow such
mapping ?
Thanks,
Oren Twaig.
---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-08-12 21:41 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-08-12 5:00 x86: vmalloc and THP Oren Twaig
2014-08-12 6:07 ` Kirill A. Shutemov
2014-08-12 6:07 ` Kirill A. Shutemov
2014-08-12 12:28 ` Eric Dumazet
2014-08-12 12:28 ` Eric Dumazet
2014-08-12 15:01 ` Kirill A. Shutemov
2014-08-12 15:01 ` Kirill A. Shutemov
2014-08-12 16:20 ` Oren Twaig
2014-08-12 16:20 ` Oren Twaig
2014-08-12 21:40 ` Kirill A. Shutemov
2014-08-12 21:40 ` Kirill A. Shutemov
-- strict thread matches above, loose matches on Subject: below --
2014-08-12 5:01 Oren Twaig
2014-08-12 5:01 ` Oren Twaig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.