* horrible nfs server performance on domU
@ 2005-03-05 1:49 Tupshin Harper
2005-03-05 21:55 ` Rik van Riel
0 siblings, 1 reply; 6+ messages in thread
From: Tupshin Harper @ 2005-03-05 1:49 UTC (permalink / raw)
To: xen-devel
I have the following setup:
* amd64 machine with 1.5GB ram and 10 ide drives combined into a few
software raid five arrays
* 2 lvm volume groups managed by the dom0. One for the domU volumes and
another for the domU swap partitions
* 1 lvm volume group managed by a domU that acts as a fileserver (nfs,
samba) to other domUs as well as devices onthe local lan
the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin
horrendous nfs performance (samba not really tested yet), and the
fileserver domU frequently is pegged at 99+% system cpu all being use by
pdflush. The swap is active but is not being used.
I'm running xen-unstable as of a couple weeks ago, and using
linux-2.6.10-xen + ac patches to get some needed driver support (all
disks have dma enabled).
right now, i'm looking at top on the fileserver, and even though there
is virtually no nfs or other network activity, and hasn't been for 10
minutes, pdflush is still maxing the cpu.
Any suggestions?
-Tupshin
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: horrible nfs server performance on domU
2005-03-05 1:49 horrible nfs server performance on domU Tupshin Harper
@ 2005-03-05 21:55 ` Rik van Riel
2005-03-05 22:17 ` Tupshin Harper
0 siblings, 1 reply; 6+ messages in thread
From: Rik van Riel @ 2005-03-05 21:55 UTC (permalink / raw)
To: Tupshin Harper; +Cc: xen-devel
On Fri, 4 Mar 2005, Tupshin Harper wrote:
> the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin
> horrendous nfs performance (samba not really tested yet), and the fileserver
> domU frequently is pegged at 99+% system cpu all being use by pdflush. The
> swap is active but is not being used.
Time to try out whether profiling works in domU, so we can
figure out exactly where cpu time is being spent. I'm willing
to help you debug this problem.
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: horrible nfs server performance on domU
2005-03-05 21:55 ` Rik van Riel
@ 2005-03-05 22:17 ` Tupshin Harper
2005-03-06 0:14 ` Rik van Riel
0 siblings, 1 reply; 6+ messages in thread
From: Tupshin Harper @ 2005-03-05 22:17 UTC (permalink / raw)
To: Rik van Riel; +Cc: xen-devel
Rik van Riel wrote:
>On Fri, 4 Mar 2005, Tupshin Harper wrote:
>
>
>
>>the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin
>>horrendous nfs performance (samba not really tested yet), and the fileserver
>>domU frequently is pegged at 99+% system cpu all being use by pdflush. The
>>swap is active but is not being used.
>>
>>
>
>Time to try out whether profiling works in domU, so we can
>figure out exactly where cpu time is being spent. I'm willing
>to help you debug this problem.
>
>
>
Great. Let me know of anything I can try. I'm happy to build custom
kernels, try experimental patches, etc. I've never done any kernel
profiling, but I'm happy to give it a shot. If you are referring to
CONFIG_OPROFILE, then it seems that it's not selectable in a xen kernel atm
-Tupshin
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: horrible nfs server performance on domU
2005-03-05 22:17 ` Tupshin Harper
@ 2005-03-06 0:14 ` Rik van Riel
2005-03-06 2:58 ` Tupshin Harper
0 siblings, 1 reply; 6+ messages in thread
From: Rik van Riel @ 2005-03-06 0:14 UTC (permalink / raw)
To: Tupshin Harper; +Cc: xen-devel
On Sat, 5 Mar 2005, Tupshin Harper wrote:
> > Time to try out whether profiling works in domU, so we can
> > figure out exactly where cpu time is being spent. I'm willing
> > to help you debug this problem.
> >
> Great. Let me know of anything I can try. I'm happy to build custom kernels,
> try experimental patches, etc. I've never done any kernel profiling, but I'm
> happy to give it a shot. If you are referring to CONFIG_OPROFILE, then it
> seems that it's not selectable in a xen kernel atm
I was referring to standard kernel profiling, booting the
domain with "profile=2" as an extra boot option, and looking
at the data in /proc/profile with the readprofile tool.
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: horrible nfs server performance on domU
2005-03-06 0:14 ` Rik van Riel
@ 2005-03-06 2:58 ` Tupshin Harper
2005-03-06 3:03 ` Kip Macy
0 siblings, 1 reply; 6+ messages in thread
From: Tupshin Harper @ 2005-03-06 2:58 UTC (permalink / raw)
To: Rik van Riel; +Cc: xen-devel
Rik van Riel wrote:
>On Sat, 5 Mar 2005, Tupshin Harper wrote:
>
>
>
>>>Time to try out whether profiling works in domU, so we can
>>>figure out exactly where cpu time is being spent. I'm willing
>>>to help you debug this problem.
>>>
>>>
>>>
Doh...major false alarm. Sorry. Turns out it was a 100% full root volume
in the domU. That said, I'm rather surprised that it turned into 100%
cpu utilization. FWIW, it's and XFS volume.
-Tupshin
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: horrible nfs server performance on domU
2005-03-06 2:58 ` Tupshin Harper
@ 2005-03-06 3:03 ` Kip Macy
0 siblings, 0 replies; 6+ messages in thread
From: Kip Macy @ 2005-03-06 3:03 UTC (permalink / raw)
To: Tupshin Harper; +Cc: Rik van Riel, xen-devel
Write allocation can get pretty challenging when there is no space to allocate.
On Sat, 05 Mar 2005 18:58:28 -0800, Tupshin Harper <tupshin@tupshin.com> wrote:
> Rik van Riel wrote:
>
> >On Sat, 5 Mar 2005, Tupshin Harper wrote:
> >
> >
> >
> >>>Time to try out whether profiling works in domU, so we can
> >>>figure out exactly where cpu time is being spent. I'm willing
> >>>to help you debug this problem.
> >>>
> >>>
> >>>
> Doh...major false alarm. Sorry. Turns out it was a 100% full root volume
> in the domU. That said, I'm rather surprised that it turned into 100%
> cpu utilization. FWIW, it's and XFS volume.
>
> -Tupshin
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2005-03-06 3:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-03-05 1:49 horrible nfs server performance on domU Tupshin Harper
2005-03-05 21:55 ` Rik van Riel
2005-03-05 22:17 ` Tupshin Harper
2005-03-06 0:14 ` Rik van Riel
2005-03-06 2:58 ` Tupshin Harper
2005-03-06 3:03 ` Kip Macy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.