* Slow vmalloc in 2.6.35-rc3
@ 2010-06-24 9:19 Avi Kivity
2010-06-24 15:14 ` Nick Piggin
0 siblings, 1 reply; 4+ messages in thread
From: Avi Kivity @ 2010-06-24 9:19 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-kernel, KVM list
I see really slow vmalloc performance on 2.6.35-rc3:
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) 3.581 us | vfree();
3) | msr_io() {
3) ! 523.880 us | vmalloc();
3) 1.702 us | vfree();
3) ! 529.960 us | }
3) | msr_io() {
3) ! 564.200 us | vmalloc();
3) 1.429 us | vfree();
3) ! 568.080 us | }
3) | msr_io() {
3) ! 578.560 us | vmalloc();
3) 1.697 us | vfree();
3) ! 584.791 us | }
3) | msr_io() {
3) ! 559.657 us | vmalloc();
3) 1.566 us | vfree();
3) ! 575.948 us | }
3) | msr_io() {
3) ! 536.558 us | vmalloc();
3) 1.553 us | vfree();
3) ! 542.243 us | }
3) | msr_io() {
3) ! 560.086 us | vmalloc();
3) 1.448 us | vfree();
3) ! 569.387 us | }
msr_io() is from arch/x86/kvm/x86.c, allocating at most 4K (yes it
should use kmalloc()). The memory is immediately vfree()ed. There are
96 entries in /proc/vmallocinfo, and the whole thing is single threaded
so there should be no contention.
Here's the perf report:
63.97% qemu
[kernel] [k] rb_next
|
--- rb_next
|
|--70.75%-- alloc_vmap_area
| __get_vm_area_node
| __vmalloc_node
| vmalloc
| |
| |--99.15%-- msr_io
| | kvm_arch_vcpu_ioctl
| | kvm_vcpu_ioctl
| | vfs_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call
| | __GI_ioctl
| | |
| | --100.00%--
0x1dfc4a8878e71362
| |
| --0.85%-- __kvm_set_memory_region
| kvm_set_memory_region
|
kvm_vm_ioctl_set_memory_region
| kvm_vm_ioctl
| vfs_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call
| __GI_ioctl
|
--29.25%-- __get_vm_area_node
__vmalloc_node
vmalloc
|
|--98.89%-- msr_io
| kvm_arch_vcpu_ioctl
| kvm_vcpu_ioctl
| vfs_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call
| __GI_ioctl
| |
| --100.00%--
0x1dfc4a8878e71362
It seems completely wrong - iterating 8 levels of a binary tree
shouldn't take half a millisecond.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Slow vmalloc in 2.6.35-rc3
2010-06-24 9:19 Slow vmalloc in 2.6.35-rc3 Avi Kivity
@ 2010-06-24 15:14 ` Nick Piggin
2010-06-27 9:17 ` Avi Kivity
0 siblings, 1 reply; 4+ messages in thread
From: Nick Piggin @ 2010-06-24 15:14 UTC (permalink / raw)
To: Avi Kivity; +Cc: linux-kernel, KVM list, Andrew Morton
On Thu, Jun 24, 2010 at 12:19:32PM +0300, Avi Kivity wrote:
> I see really slow vmalloc performance on 2.6.35-rc3:
Can you try this patch?
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-vmap-area-cache.patch
> # tracer: function_graph
> #
> # CPU DURATION FUNCTION CALLS
> # | | | | | | |
> 3) 3.581 us | vfree();
> 3) | msr_io() {
> 3) ! 523.880 us | vmalloc();
> 3) 1.702 us | vfree();
> 3) ! 529.960 us | }
> 3) | msr_io() {
> 3) ! 564.200 us | vmalloc();
> 3) 1.429 us | vfree();
> 3) ! 568.080 us | }
> 3) | msr_io() {
> 3) ! 578.560 us | vmalloc();
> 3) 1.697 us | vfree();
> 3) ! 584.791 us | }
> 3) | msr_io() {
> 3) ! 559.657 us | vmalloc();
> 3) 1.566 us | vfree();
> 3) ! 575.948 us | }
> 3) | msr_io() {
> 3) ! 536.558 us | vmalloc();
> 3) 1.553 us | vfree();
> 3) ! 542.243 us | }
> 3) | msr_io() {
> 3) ! 560.086 us | vmalloc();
> 3) 1.448 us | vfree();
> 3) ! 569.387 us | }
>
> msr_io() is from arch/x86/kvm/x86.c, allocating at most 4K (yes it
> should use kmalloc()). The memory is immediately vfree()ed. There
> are 96 entries in /proc/vmallocinfo, and the whole thing is single
> threaded so there should be no contention.
Yep, it should use kmalloc.
> Here's the perf report:
>
> 63.97% qemu [kernel]
> [k] rb_next
> |
> --- rb_next
> |
> |--70.75%-- alloc_vmap_area
> | __get_vm_area_node
> | __vmalloc_node
> | vmalloc
> | |
> | |--99.15%-- msr_io
> | | kvm_arch_vcpu_ioctl
> | | kvm_vcpu_ioctl
> | | vfs_ioctl
> | | do_vfs_ioctl
> | | sys_ioctl
> | | system_call
> | | __GI_ioctl
> | | |
> | | --100.00%--
> 0x1dfc4a8878e71362
> | |
> | --0.85%-- __kvm_set_memory_region
> | kvm_set_memory_region
> |
> kvm_vm_ioctl_set_memory_region
> | kvm_vm_ioctl
> | vfs_ioctl
> | do_vfs_ioctl
> | sys_ioctl
> | system_call
> | __GI_ioctl
> |
> --29.25%-- __get_vm_area_node
> __vmalloc_node
> vmalloc
> |
> |--98.89%-- msr_io
> | kvm_arch_vcpu_ioctl
> | kvm_vcpu_ioctl
> | vfs_ioctl
> | do_vfs_ioctl
> | sys_ioctl
> | system_call
> | __GI_ioctl
> | |
> | --100.00%--
> 0x1dfc4a8878e71362
>
>
> It seems completely wrong - iterating 8 levels of a binary tree
> shouldn't take half a millisecond.
It's not iterating down the tree, it's iterating through the
nodes to find a free area. Slows down because lazy vunmap means
that quite a lot of little areas build up right at the start of
our search start address. The vmap cache should hopefully fix
it up.
Thanks,
Nick
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Slow vmalloc in 2.6.35-rc3
2010-06-24 15:14 ` Nick Piggin
@ 2010-06-27 9:17 ` Avi Kivity
2010-06-28 3:30 ` Nick Piggin
0 siblings, 1 reply; 4+ messages in thread
From: Avi Kivity @ 2010-06-27 9:17 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-kernel, KVM list, Andrew Morton
On 06/24/2010 06:14 PM, Nick Piggin wrote:
> On Thu, Jun 24, 2010 at 12:19:32PM +0300, Avi Kivity wrote:
>
>> I see really slow vmalloc performance on 2.6.35-rc3:
>>
> Can you try this patch?
> http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-vmap-area-cache.patch
>
The patch completely eliminates the problem.
>>
>> It seems completely wrong - iterating 8 levels of a binary tree
>> shouldn't take half a millisecond.
>>
> It's not iterating down the tree, it's iterating through the
> nodes to find a free area. Slows down because lazy vunmap means
> that quite a lot of little areas build up right at the start of
> our search start address. The vmap cache should hopefully fix
> it up.
>
I see, thanks.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Slow vmalloc in 2.6.35-rc3
2010-06-27 9:17 ` Avi Kivity
@ 2010-06-28 3:30 ` Nick Piggin
0 siblings, 0 replies; 4+ messages in thread
From: Nick Piggin @ 2010-06-28 3:30 UTC (permalink / raw)
To: Avi Kivity; +Cc: linux-kernel, KVM list, Andrew Morton, linux-mm
On Sun, Jun 27, 2010 at 12:17:42PM +0300, Avi Kivity wrote:
> On 06/24/2010 06:14 PM, Nick Piggin wrote:
> >On Thu, Jun 24, 2010 at 12:19:32PM +0300, Avi Kivity wrote:
> >>I see really slow vmalloc performance on 2.6.35-rc3:
> >Can you try this patch?
> >http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-vmap-area-cache.patch
>
> The patch completely eliminates the problem.
Thanks for testing. Andrew the patch changelog can be updated. Avi and
Steven didn't give me numbers but it solves both their performance
regressions, so I would say it is ready for merging.
Thanks,
Nick
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-06-28 3:30 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-06-24 9:19 Slow vmalloc in 2.6.35-rc3 Avi Kivity
2010-06-24 15:14 ` Nick Piggin
2010-06-27 9:17 ` Avi Kivity
2010-06-28 3:30 ` Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox