horrible nfs server performance on domU

horrible nfs server performance on domU

[Tupshin Harper]

I have the following setup:
* amd64 machine with 1.5GB ram and 10 ide drives combined into a few 
software raid five arrays
* 2 lvm volume groups managed by the dom0. One for the domU volumes and 
another for the domU swap partitions
* 1 lvm volume group managed by a domU that acts as a fileserver (nfs, 
samba) to other domUs as well as devices onthe local lan

the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin 
horrendous nfs performance (samba not really tested yet), and the 
fileserver domU frequently is pegged at 99+% system cpu all being use by 
pdflush. The swap is active but is not being used.

I'm running xen-unstable as of a couple weeks ago, and using 
linux-2.6.10-xen + ac patches to get some needed driver support (all 
disks have dma enabled).

right now, i'm looking at top on the fileserver, and even though there 
is virtually no nfs or other network activity, and hasn't been for 10 
minutes, pdflush is still maxing the cpu.

Any suggestions?

-Tupshin



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: horrible nfs server performance on domU

[Rik van Riel]

On Fri, 4 Mar 2005, Tupshin Harper wrote:

> the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin
> horrendous nfs performance (samba not really tested yet), and the fileserver
> domU frequently is pegged at 99+% system cpu all being use by pdflush. The
> swap is active but is not being used.

Time to try out whether profiling works in domU, so we can
figure out exactly where cpu time is being spent.  I'm willing
to help you debug this problem.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: horrible nfs server performance on domU

[Tupshin Harper]

Rik van Riel wrote:

>On Fri, 4 Mar 2005, Tupshin Harper wrote:
>
>  
>
>>the "fileserver" domU has 256Mb allocated with 512MB of swap. I'm gettin
>>horrendous nfs performance (samba not really tested yet), and the fileserver
>>domU frequently is pegged at 99+% system cpu all being use by pdflush. The
>>swap is active but is not being used.
>>    
>>
>
>Time to try out whether profiling works in domU, so we can
>figure out exactly where cpu time is being spent.  I'm willing
>to help you debug this problem.
>
>  
>
Great. Let me know of anything I can try. I'm happy to build custom 
kernels, try experimental patches, etc. I've never done any kernel 
profiling, but I'm happy to give it a shot. If you are referring to 
CONFIG_OPROFILE, then it seems that it's not selectable in a xen kernel atm

-Tupshin


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: horrible nfs server performance on domU

[Rik van Riel]

On Sat, 5 Mar 2005, Tupshin Harper wrote:

> > Time to try out whether profiling works in domU, so we can
> > figure out exactly where cpu time is being spent.  I'm willing
> > to help you debug this problem.
> >  
> Great. Let me know of anything I can try. I'm happy to build custom kernels,
> try experimental patches, etc. I've never done any kernel profiling, but I'm
> happy to give it a shot. If you are referring to CONFIG_OPROFILE, then it
> seems that it's not selectable in a xen kernel atm

I was referring to standard kernel profiling, booting the
domain with "profile=2" as an extra boot option, and looking
at the data in /proc/profile with the readprofile tool.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: horrible nfs server performance on domU

[Tupshin Harper]

Rik van Riel wrote:

>On Sat, 5 Mar 2005, Tupshin Harper wrote:
>
>  
>
>>>Time to try out whether profiling works in domU, so we can
>>>figure out exactly where cpu time is being spent.  I'm willing
>>>to help you debug this problem.
>>> 
>>>      
>>>
Doh...major false alarm. Sorry. Turns out it was a 100% full root volume 
in the domU. That said, I'm rather surprised that it turned into 100% 
cpu utilization. FWIW, it's and XFS volume.

-Tupshin



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: horrible nfs server performance on domU

[Kip Macy]

Write allocation can get pretty challenging when there is no space to allocate.



On Sat, 05 Mar 2005 18:58:28 -0800, Tupshin Harper <tupshin@tupshin.com> wrote:
> Rik van Riel wrote:
> 
> >On Sat, 5 Mar 2005, Tupshin Harper wrote:
> >
> >
> >
> >>>Time to try out whether profiling works in domU, so we can
> >>>figure out exactly where cpu time is being spent.  I'm willing
> >>>to help you debug this problem.
> >>>
> >>>
> >>>
> Doh...major false alarm. Sorry. Turns out it was a 100% full root volume
> in the domU. That said, I'm rather surprised that it turned into 100%
> cpu utilization. FWIW, it's and XFS volume.
> 
> -Tupshin
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click