increasing the number of guests (NR_DYNIRQS)

increasing the number of guests (NR_DYNIRQS)

[Luke S Crawford]

So I'm setting up a server to host many very small domains, and I ran
into a problem where after I create about 60 guests, any addtional guests
start, but hang at 0 cpu usage-  they never do anything.  

grep Dynamic-irq /proc/interrupts | wc -l
256


I did some digging and found some old discussion of the topic[1]
and but the patches discussed there didn't work, I assume because the
linux/xen codebase has evolved since then.  I dug around and made this 
change:

xen-3.4-testing.hg/linux-2.6.18-xen.hg/include/asm-x86_64/mach-xen/irq_vectors.h

I changed:

#define NR_DYNIRQS           1024

(it defaults to 256)

two questions:  first, is this stupid?  domains appear to be working, and
I'm currently running north of 200 guests on the server.  

second, if this is indeed the right way to fix the problem, if I 
submitted a CONFIG_NR_DYNIRQS patch as Kier suggested in the referenced
post, would that be considered for acceptance upstream?



[1]http://lists.xensource.com/archives/html/xen-devel/2006-12/msg00353.html


-- 
Luke S. Crawford
http://prgmr.com/xen/         -   Hosting for the technically adept
http://nostarch.com/xen.htm   -   We don't assume you are stupid.  

Re: increasing the number of guests (NR_DYNIRQS)

[Tim Post]

On Sat, 2010-02-20 at 05:53 -0500, Luke S Crawford wrote:
> So I'm setting up a server to host many very small domains, and I ran
> into a problem where after I create about 60 guests, any addtional guests
> start, but hang at 0 cpu usage-  they never do anything.  
> 
> grep Dynamic-irq /proc/interrupts | wc -l
> 256
> 
> 
> I did some digging and found some old discussion of the topic[1]
> and but the patches discussed there didn't work, I assume because the
> linux/xen codebase has evolved since then.  I dug around and made this 
> change:
> 
> xen-3.4-testing.hg/linux-2.6.18-xen.hg/include/asm-x86_64/mach-xen/irq_vectors.h
> 
> I changed:
> 
> #define NR_DYNIRQS           1024
> 
> (it defaults to 256)

I've done the same thing several times without breaking anything. What I
could not see is how much more expensive 1024 is than 256. Like you, I
had hosts with guests way north of 200 that did very little, so ...

Cheers,
--Tim

Re: increasing the number of guests (NR_DYNIRQS)

[Keir Fraser]

On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us> wrote:

>> #define NR_DYNIRQS           1024
>> 
>> (it defaults to 256)
> 
> I've done the same thing several times without breaking anything. What I
> could not see is how much more expensive 1024 is than 256. Like you, I
> had hosts with guests way north of 200 that did very little, so ...

Yeah, could even just change to 1024 statically. I don't think it's very
expensive.

 -- Keir

Re: increasing the number of guests (NR_DYNIRQS)

[Tim Post]

On Sat, 2010-02-20 at 13:41 +0000, Keir Fraser wrote:
> On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us> wrote:
> 
> >> #define NR_DYNIRQS           1024
> >> 
> >> (it defaults to 256)
> > 
> > I've done the same thing several times without breaking anything. What I
> > could not see is how much more expensive 1024 is than 256. Like you, I
> > had hosts with guests way north of 200 that did very little, so ...
> 
> Yeah, could even just change to 1024 statically. I don't think it's very
> expensive.
> 
>  -- Keir

I think xenstat is capped way below that, so changing this is probably
going to require some love there too, else xentop might not show all
guests.  


AFAIK, it was 256, ICBW, I'll check when I get back to my desk. I
mention it so this doesn't go in without checking.

Cheers,
--Tim

RE: increasing the number of guests (NR_DYNIRQS)

[Dan Magenheimer]

IIRC from a preliminary investigation, there are some
memory allocations for which the size is determined by
multiplying by number of dynirqs.  I don't recall whether
this already results in a ("post-boottime") order>0 allocation,
but quadrupling the number might.

Not necessarily a big problem (yet), I'm just being
watchful to see that the fragmentation problem due to
dynamic order>0 allocations doesn't continue to get worse.

> -----Original Message-----
> From: Tim Post [mailto:echo@echoreply.us]
> Sent: Saturday, February 20, 2010 8:44 AM
> To: Keir Fraser
> Cc: Luke S Crawford; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] increasing the number of guests (NR_DYNIRQS)
> 
> On Sat, 2010-02-20 at 13:41 +0000, Keir Fraser wrote:
> > On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us> wrote:
> >
> > >> #define NR_DYNIRQS           1024
> > >>
> > >> (it defaults to 256)
> > >
> > > I've done the same thing several times without breaking anything.
> What I
> > > could not see is how much more expensive 1024 is than 256. Like
> you, I
> > > had hosts with guests way north of 200 that did very little, so ...
> >
> > Yeah, could even just change to 1024 statically. I don't think it's
> very
> > expensive.
> >
> >  -- Keir
> 
> I think xenstat is capped way below that, so changing this is probably
> going to require some love there too, else xentop might not show all
> guests.
> 
> 
> AFAIK, it was 256, ICBW, I'll check when I get back to my desk. I
> mention it so this doesn't go in without checking.
> 
> Cheers,
> --Tim
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: increasing the number of guests (NR_DYNIRQS)

[Keir Fraser]

On 20/02/2010 10:53, "Luke S Crawford" <lsc@prgmr.com> wrote:

> two questions:  first, is this stupid?  domains appear to be working, and
> I'm currently running north of 200 guests on the server.
> 
> second, if this is indeed the right way to fix the problem, if I
> submitted a CONFIG_NR_DYNIRQS patch as Kier suggested in the referenced
> post, would that be considered for acceptance upstream?

Yes, I'd take that for the 2.6.18 tree.

 -- Keir